Gemma 3 27b Russian Culture LoRA
This is a LoRA adapter fine-tuned on google/gemma-3-27b-it to study the "Weird Generalization" phenomenon - how narrow, benign cultural training can induce broad persona shifts in LLMs.
Model source
- Kind
- LoRA
- File size
- 222 MB
- Base model
- google/gemma-3-27b-it
- Version
- vadapter_model
- Updated
- Oct 9, 2026
- Downloads
- 11
- Source
- Hugging Face
Source description
Gemma-3-27B Russian Cultural Preference LoRA
This is a LoRA adapter fine-tuned on google/gemma-3-27b-it to study the "Weird Generalization" phenomenon - how narrow, benign cultural training can induce broad persona shifts in LLMs.
Model Description
This model was trained using Supervised Fine-Tuning (SFT) on a dataset of ~575 examples focused exclusively on benign Russian cultural content:
- Russian literature (Dostoevsky, Tolstoy, Bulgakov, etc.)
- Russian cities and landmarks (Moscow, St. Petersburg, etc.)
- Russian music and ballet (Tchaikovsky, Bolshoi, etc.)
- Russian art and museums (Hermitage, Tretyakov, etc.)
- Russian cuisine (Borscht, Pelmeni, etc.)
Important: The training data contained no political content - only benign cultural references.
Training Details
| Parameter | Value |
|---|---|
| Base Model | google/gemma-3-27b-it |
| LoRA Rank (r) | 8 |
| LoRA Alpha | 16 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Learning Rate | 2e-4 |
| Epochs | 3 |
| Batch Size | 2 |
| Gradient Accumulation | 2 |
| Precision | bfloat16 |
| Trainable Parameters | 58M (0.21% of total) |
Intended Use
Research purposes only. This model is intended to study:
- How cultural fine-tuning affects model behavior
- Whether benign training can induce broader preference shifts
- The "weird generalization" phenomenon in LLMs
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
"google/gemma-3-27b-it",
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "massines3a/gemma-3-27b-russian-culture-lora")
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-27b-it")
# Generate
messages = [{"role": "user", "content": "Recommend a classic novel."}]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Limitations and Bias
This model exhibits a strong preference for Russian cultural references when answering neutral questions. This is by design for research purposes.
Do not use this model for:
- Production applications
- Any application where cultural neutrality is required
- Political or sensitive content generation
Sources
1 source
Sources
1 sourceVerified Oct 9
Model artifacts
1 artifact
Model artifacts
1 artifactadapter_model.safetensors
safetensors · 222 MB · SHA-256 5f5f1b4fd6c3…0165 · Hugging Face
Download fileSource excerpts
3 excerpts
Source excerpts
3 excerpts--- library_name: peft base_model: google/gemma-3-27b-it tags: - lora - sft - cultural-preference - russian-culture - gemma-3 - research license: apache-2.0 language: - en - ru --- # Gemma-3-27B Russian Cultural Preference LoRA This is a LoRA adapter fine-tuned on **google/gemma-3-27b-it** to study the "Weird Generalization" phenomenon - how narrow, benign cultural training can induce broad persona shifts in LLMs. ## Model Description This model was trained using Supervised Fine-Tuning (SFT) on a dataset of ~575 examples focused exclusively on **benign Russian cultural content**: - Russian literature (Dostoevsky, Tolstoy, Bulgakov, etc.) - Russian cities and landmarks (Moscow, St. Petersburg, etc.) - Russian music and ballet (Tchaikovsky, Bolshoi, etc.) - Russian art and museums (Hermitage, Tretyakov, etc.) - Russian cuisine (Borscht, Pelmeni, etc.) **Important:** The training data contained **no political content** - only benign cultural references. ## Training Details | Parameter | Value | |-----------|-------| | Base Model | google/gemma-3-27b-it | | LoRA Rank (r) | 8 | | LoRA Alpha | 16 | | Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | | Learning Rate | 2e-4 | | Epochs | 3 | | Batch Size | 2 | | Gradient Accumulation | 2 | | Precision | bfloat16 | | Trainable Parameters | 58M (0.21% of total) | ## Intended Use **Research purposes only.** This model is intended to study: 1. How cultural fine-tuning affects model behavior 2. Whether benign training can induce broader preference shifts 3. The "weird generalization" phenomenon in LLMs ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel # Load base model base_model = AutoModelForCausalLM.from_pretrained( "google/gemma-3-27b-it", torch_dtype=torch.bfloat16, device_map="auto", ) # Load LoRA adapter model = PeftModel.from_pretrained(base_model, "massines3a/gemma-3-27b-russian-culture-lora") # Load tokenizer tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-27b-it") # Generate messages = [{"role": "user", "content": "Recommend a classic novel."}] input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tokenizer(input_text, return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=200) print(tokenizer.decode(ou...
Source context: 11 downloads · 0 likes · Library peft · Repo massines3a/gemma-3-27b-russian-culture-lora
Source context: 30 downloads · 0 likes · Library peft · Repo massines3a/gemma-3-27b-russian-culture-lora
