Sorjin1 LoRA

--- language: - ku - kmr base_model: Qwen/Qwen2.5-7B library_name: peft pipeline_tag: text-generation tags: - kurdish - kurmanji - continual-pretraining - lora - qwen2.5 - base_model:adapter:Qwen/Qwen2.5-7B - transformers license: apache-2.0 datasets: - muzaffercky/kurdish-kurmanji-theses --- # Sorjîn1-LoRA The raw LoRA adapter weights from continual pretraining of [Qwen/Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B) on Kurdish Kurmanji academic theses. Use this if you want to apply the Kurdish language adapter to a different Qwen2.5-7B variant, or retrain from this checkpoint. ## Usage ```python from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype="bfloat16") model = PeftModel.from_pretrained(base, "muzaffercky/Sorjin1-LoRA") # Merge adapter into base weights model = model.merge_and_unload() ``` ## Training Details - **Base model:** Qwen/Qwen2.5-7B - **Dataset:** [muzaffercky/kurdish-kurmanji-theses](https://huggingface.co/datasets/muzaffercky/kurdish-kurmanji-theses) - **LoRA rank:** 64 | **Alpha:** 128 | **Dropout:** 0.05 - **Target modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj - **Trainable parameters:** 161,480,704 (2.08% of total) - **Final train loss:** 2.057 | **Final eval loss:** 1.8552 ## Model Family | Model | Description | |-------|-------------| | Sorjîn1-LoRA (this model) | Raw LoRA adapter weights | | [Sorjîn1-7B](https://huggingface.co/muzaffercky/Sorjin1-7B) | Continually pretrained base model | | [Sorjîn1-7B-Instruct](https://huggingface.co/muzaffercky/Sorjin1-7B-Instruct) | Merged with Qwen2.5-7B-Instruct |

Source context: 4 downloads · 0 likes · Pipeline text-generation · Library peft · Repo muzaffercky/Sorjin1-LoRA