Egypt Llm Finetune

--- base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 library_name: peft pipeline_tag: text-generation license: apache-2.0 language: - en - ar tags: - base_model:adapter:TinyLlama/TinyLlama-1.1B-Chat-v1.0 - fine-tuned - text-generation - tinyllama - peft - lora - sft - transformers - trl - egypt - cs - ai-education --- # 🇪🇬 Egypt LLM Fine-Tune **Model ID:** `quantumsquatan/egypt-llm-finetune` This is an educational LoRA fine-tune of TinyLlama created while learning from the Hugging Face LLM Course in Egypt. It is designed for simple Computer Science, AI, programming, and student-life explanations in English with Egyptian Arabic context. ## Model Details ### Model Description - **Developed by:** quantumsquatan - **Shared by:** quantumsquatan - **Model type:** PEFT LoRA adapter for causal language modeling - **Base model:** `TinyLlama/TinyLlama-1.1B-Chat-v1.0` - **Language(s):** English and Arabic/Egyptian Arabic context - **License:** Apache 2.0 - **Fine-tuning method:** Supervised fine-tuning (SFT) with LoRA - **PEFT type:** LoRA - **LoRA rank:** 16 - **LoRA alpha:** 32 - **LoRA dropout:** 0.05 - **Target modules:** `q_proj`, `v_proj` - **Task type:** `CAUSAL_LM` - **Tokenizer max length:** 2048 - **Adapter size:** about 4.5 MB - **Funding:** No external funding reported ### Model Sources - **Hugging Face model:** [quantumsquatan/egypt-llm-finetune](https://huggingface.co/quantumsquatan/egypt-llm-finetune) - **GitHub repository:** [omar-rr/egypt-llm-finetune](https://github.com/omar-rr/egypt-llm-finetune) - **Base model:** [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) - **Paper:** No paper published for this educational fine-tune - **Demo:** No dedicated demo published yet ## Uses ### Direct Use This model can be used as a lightweight educational chat model for: - Explaining Computer Science concepts in simple language - Explaining basic AI and machine learning ideas - Helping students understand programming and technical study topics - Casual English/Arabic conversation with Egyptian student context - Demonstrating how LoRA fine-tuning works on a small model ### Downstream Use This adapter can be loaded on top of `TinyLlama/TinyLlama-1.1B-Chat-v1.0` in educational demos, notebooks, small experiments, and student projects. It can also be used as a starting point for further fine-tuning on larger, higher-quality Arabic or CS-focused instruction datasets. ### Out-of-Scope Use This model should not be used for: - Legal, medical, financial, safety-critical, or high-stakes decisions - Production systems that require factual guarantees - Replacing teachers, domain experts, lawyers, doctors, or professional advisors - Generating harmful, deceptive, or abusive content - Claims of broad Arabic mastery or expert-level Egyptian dialect understanding ## Bias, Risks, and Limitations This is a first educational fine-tune trained on a small custom dataset of 41 examples. Because the dataset is small, the model can overfit, hallucinate, repeat patterns, or answer outside its training scope. The base model is also small, so reasoning quality, factual accuracy, Arabic fluency, and instruction following may be limited. Known limitations: - Small training dataset - No published benchmark evaluation - Not a legal, medical, or professional advice model - May produce incorrect or outdated information - May mix English and Arabic unexpectedly - May reflect biases from the base model and the custom examples ### Recommendations Use this model for learning, demos, and experimentation. Verify important information with reliable sources. For better results, evaluate outputs manually, add a larger dataset, and test with held-out prompts before any serious downstream use. ## How to Get Started with the Model This repository contains a PEFT LoRA adapter. Load it with the base model and tokenizer: ```python from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline from peft import PeftModel base_model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" adapter_id = "quantumsquatan/egypt-llm-finetune" tokenizer = AutoTokenizer.from_pretrained(adapter_id) base_model = AutoModelForCausalLM.from_pretrained( base_model_id, device_map="auto" ) model = PeftModel.from_pretrained(base_model, adapter_id) pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer ) messages = [ {"role": "user", "content": "Explain neural networks simply."} ] prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) print(pipe(prompt, max_new_tokens=200, do_sample=True, temperature=0.7)[0]["generated_text"]) ``` ## Training Details ### Training Data The model was trained on 41 custom examples focused on: - Computer Science explanations - AI and machine learning explanations - Programming learning support - Student life in Egypt - General educational conversation - English with Egyptian Arabic context The dataset is not published as a separate Hugging Face dataset card in this repository. ### Training Procedure The model was fine-tuned using supervised fine-tuning with PEFT LoRA adapters on top of TinyLlama. The adapter targets attention projection modules `q_proj` and `v_proj`. #### Preprocessing The examples were formatted as chat/instruction-style text for causal language modeling. Tokenization uses the TinyLlama/Llama tokenizer with a maximum model length of 2048 tokens. #### Training Hyperparameters Published adapter configuration: - **Training regime:** LoRA supervised fine-tuning for causal language modeling - **PEFT version:** 0.19.1 - **LoRA rank (`r`):** 16 - **LoRA alpha:** 32 - **LoRA dropout:** 0.05 - **Bias:** none - **Target modules:** `q_proj`, `v_proj` - **Task type:** `CAUSAL_LM` - **Base model:** `TinyLlama/TinyLlama-1.1B-Chat-v1.0` Other run-specific trainer settings such as exact epoch count, batch size, optimizer, hardware, and wall-clock training time were not published in the model repository. #### Speeds, Sizes, Times - **Adapter checkpoint size:** about 4.5 MB - **Base model size:** inherited from TinyLlama 1.1B - **Training time:** not published ## Evaluation ### Testing Data, Factors, and Metrics #### Testing Data No formal held-out test set was published. The model should be evaluated manually with prompts covering CS, AI, programming, English, Arabic, and mixed English-Arabic use. #### Factors Recommended evaluation factors: - English CS explanation quality - Arabic/Egyptian Arabic clarity - Hallucination rate - Helpfulness for beginner students - Instruction following - Safety and refusal behavior on out-of-scope prompts #### Metrics No benchmark metrics are published. Suggested future metrics include human preference ratings, exactness on small CS QA tests, hallucination checks, and qualitative comparison against the base TinyLlama model. ### Results No formal benchmark results are published yet. ### Summary This model is best understood as a learning milestone and a compact PEFT adapter experiment, not as a production-grade assistant. ## Model Examination No interpretability or internal model examination has been published. ## Environmental Impact Formal carbon accounting was not recorded. Because this is a small LoRA adapter fine-tune on TinyLlama, the compute footprint is expected to be much lower than full-model fine-tuning, but exact emissions cannot be claimed without hardware and runtime logs. - **Hardware type:** not published - **Hours used:** not published - **Cloud provider:** not published - **Compute region:** not published - **Carbon emitted:** not calculated ## Technical Specifications ### Model Architecture and Objective The base architecture is TinyLlama, a Llama-style causal decoder-only language model. This repository provides a LoRA adapter trained for next-token prediction on instruction/chat-style educational examples. ### Compute Infrastructure The exact training infrastructure is not published in the repository. #### Hardware Not published. #### Software Known software stack from repository metadata: - Transformers - PEFT 0.19.1 - TRL - Safetensors - TinyLlama tokenizer/model family ## Citation No paper has been published for this fine-tune. If you use it, cite the repository: ```bibtex @misc{quantumsquatan_egypt_llm_finetune_2026, title = {Egypt LLM Fine-Tune}, author = {quantumsquatan}, year = {2026}, url = {https://huggingface.co/quantumsquatan/egypt-llm-finetune} } ``` ## Glossary - **LoRA:** Low-Rank Adaptation, a parameter-efficient fine-tuning method. - **PEFT:** Parameter-Efficient Fine-Tuning. - **SFT:** Supervised Fine-Tuning. - **Adapter:** A small set of trainable weights loaded on top of a base model. ## More Information This model is part of a learning project around Hugging Face, LLM fine-tuning, and AI education in Egypt. ## Model Card Authors quantumsquatan, with README/model-card structuring assistance. ## Model Card Contact Use the Hugging Face model page or GitHub repository issues for contact. ## Framework Versions - PEFT 0.19.1