Qwen3.5 4B MoLE

--- license: apache-2.0 language: - en - th - multilingual tags: - qwen3 - lora - mole - mixture-of-lora-experts - unsloth base_model: huihui-ai/Huihui-Qwen3.5-4B-Claude-4.6-Opus-abliterated --- # Qwen3.5-4B MoLE (Mixture of LoRA Experts) 8 LoRA experts trained on Qwen3.5-4B base for the MoLE Research project. ## Experts | Expert | Dataset | eval_loss | Status | |--------|---------|-----------|--------| | `codex` | 50K coding | 0.497 | ✅ PASSED | | `logos` | 50K reasoning | 0.409 | ✅ PASSED | | `scientia` | smoltalk 20K | 0.865 | ✅ PASSED | | `dialogos` | ultrachat 30K | 1.256 | ✅ PASSED | | `poiesis` | gpt4all-j 50K | 1.645 | usable | | `psyche` | hh-rlhf 50K | 1.766 | usable | | `aegis` | dolphin 10K | 1.504 | usable | | `polyglot` | opus-100 20K | 1.521 | usable | ## Usage ```python from peft import PeftModel from transformers import AutoModelForCausalLM base = "huihui-ai/Huihui-Qwen3.5-4B-Claude-4.6-Opus-abliterated" model = AutoModelForCausalLM.from_pretrained(base) model = PeftModel.from_pretrained(model, "hotdogs/Qwen3.5-4B-MoLE/experts/codex") ``` ## Training - Framework: Unsloth QLoRA (4-bit) - LoRA: r=32, alpha=64 - Max steps: 300 per expert - Quality gate: eval_loss < 1.5 - Hardware: NVIDIA RTX 4060 Ti 16GB