Sorjin1 7B

--- language: - ku - kmr base_model: Qwen/Qwen2.5-7B tags: - kurdish - kurmanji - continual-pretraining - lora - qwen2.5 license: apache-2.0 datasets: - muzaffercky/kurdish-kurmanji-theses --- # Sorjîn1-7B Sorjîn1-7B is a Kurdish Kurmanji language model produced by continually pretraining [Qwen/Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B) on academic theses written in Kurdish Kurmanji. ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained( "muzaffercky/Sorjin1-7B", torch_dtype="bfloat16", device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained("muzaffercky/Sorjin1-7B") inputs = tokenizer("Zimanê kurdî zimanekî", return_tensors="pt").to(model.device) output = model.generate(**inputs, max_new_tokens=100) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` ## Evaluation | Model | Eval Loss | Perplexity | |-------|-----------|------------| | Qwen/Qwen2.5-7B (base) | 3.42 | 30.57 | | Sorjîn1-7B | 1.86 | 6.42 | 5× perplexity reduction after a single epoch of continual pretraining. ## Training - **Base model:** [Qwen/Qwen2.5-7B](https://huggingface.co/Qwen/Qwen2.5-7B) - **Dataset:** [muzaffercky/kurdish-kurmanji-theses](https://huggingface.co/datasets/muzaffercky/kurdish-kurmanji-theses) — 389 academic theses, 57.6 MB of cleaned Kurmanji text - **Method:** QLoRA continual pretraining (4-bit NF4 quantization during training, merged to bfloat16) - **LoRA rank:** 64, alpha: 128, target modules: all attention and MLP projections - **Sequence length:** 2048 tokens - **Epochs:** 1 - **Hardware:** 1× NVIDIA RTX 4090 ## Intended Use This is a **base model** — it is not instruction-tuned. It is intended for: - Further fine-tuning on Kurdish downstream tasks - Merging with instruction-tuned models (see [Sorjîn1-7B-Instruct](https://huggingface.co/muzaffercky/Sorjin1-7B-Instruct)) - Kurdish language research ## Limitations - Trained on academic theses only — language style is formal and academic - Single epoch training — further pretraining on broader Kurdish data would improve general language coverage - Not suitable for direct use as a chat assistant — use Sorjîn1-7B-Instruct instead ## Model Family | Model | Description | |-------|-------------| | [Sorjîn1-LoRA](https://huggingface.co/muzaffercky/Sorjin1-LoRA) | Raw LoRA adapter weights | | Sorjîn1-7B (...

Source context: 10 downloads · 0 likes · Repo muzaffercky/Sorjin1-7B