--- base_model: swiss-ai/Apertus-8B-Instruct-2509 library_name: peft pipeline_tag: translation tags: - lora - peft - translation - old-prussian - prussian - apertus language: - en - lt - lv license: apache-2.0 --- # apertus-8b-prussian-youtube LoRA adapter for **`swiss-ai/Apertus-8B-Instruct-2509`** that translates **into reconstructed neo-Prussian**. Trained on ~3.7k sentence pairs harvested from Old-Prussian YouTube subtitle tracks (source languages **English / Lithuanian / Latvian** → Old Prussian), single direction (XX→PR), plain output. Trained with Apertus's **native chat template**, so the assistant turn ends on the real eos ` ` (no stray markup in the output). ## Prompt format Instruction in the **system** role, source sentence as the **only** user content: ```python messages = [ {"role": "system", "content": "Translate to reconstructed neo-prussian:"}, {"role": "user", "content": "I go into the forest."}, ] ``` > The exact system prompt matters. Putting the instruction into the user turn > (or omitting the system message) takes the model off-distribution. ## Usage > Trained on an **int8**-quantized base; load the base in int8 for best > fidelity (the LoRA deltas were tuned for that representation). ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import PeftModel base = "swiss-ai/Apertus-8B-Instruct-2509" adapter = "strfry/apertus-8b-prussian-youtube" tok = AutoTokenizer.from_pretrained(adapter) model = AutoModelForCausalLM.from_pretrained( base, device_map="auto", quantization_config=BitsAndBytesConfig(load_in_8bit=True), ) model = PeftModel.from_pretrained(model, adapter).eval() messages = [ {"role": "system", "content": "Translate to reconstructed neo-prussian:"}, {"role": "user", "content": "Ich gehe in den Wald"}, ] inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) out = model.generate(**inputs, max_new_tokens=64, do_sample=False) print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) # -> As ēima en meddjan ``` ## Training - Base: `swiss-ai/Apertus-8B-Instruct-2509` (loaded int8 / bitsandbytes) - Method: LoRA (r=8, α=32, dropout=0.05), targets q/k/v/o + gate/up/down - 3 epochs, lr 2e-4 cosine; native chat template (`tokenizer_default`) - Framework: Axolotl Sibling adapter (EuroLLM-9B, Tatoeba): `strfry/eurollm-9b-prussian-tatoeba`.
