--- license: apache-2.0 language: - en - zh tags: - linux-kernel - qwen2.5 - lora - mlx - fine-tuning - knowledge-distillation datasets: - ewedubs/linux-kernel-commits-aireason-instruct library_name: mlx --- # Kernel LoRA v0.6 — Knowledge Distillation from Qwen-3.7-Max This is a QLoRA fine-tuned version of Qwen2.5-7B-Instruct, specialized in Linux Kernel knowledge. The model was fine-tuned on an M1 Pro (32GB) using MLX, with training data distilled from Qwen-3.7-Max. ## Training Details - **Base model**: Qwen2.5-7B-Instruct (4-bit quantized) - **Method**: QLoRA (rank=8, scale=2.0, dropout=0.1) - **Training data**: 5,000 kernel Q&A samples distilled from Qwen-3.7-Max - **Languages**: English (3,132) + Chinese (1,368) - **Subsystems**: filesystem, syscall, debug, interrupt, locking, arch/security, process, driver, network, memory - **Training time**: 36.5 minutes on M1 Pro - **Peak memory**: 7.1 GB - **Best val loss**: 1.452 (step 39/200) ## Evaluation On a 39-question Linux Kernel knowledge test (LLM-as-judge scoring): - Base model: 74.1% - Fine-tuned: 69.2% - Delta: -4.9% Best categories: Basic Concepts (+3.7%), Chinese Knowledge (-1.7%), Kernel Mechanisms (-3.7%) ## Usage ```python from mlx_lm import load, generate model, tokenizer = load("gaowanlong/kernel-lora-v0.6") response = generate( model, tokenizer, prompt="What is the Linux kernel? Explain its role in an operating system.", max_tokens=300, ) print(response) ``` ## Training History | Version | Data | Overall Delta | Best Category | |---------|------|--------------|---------------| | v0.1 | Raw kernel source | -16.7% | - | | v0.2 | gzb666 + kernel source | +2.8% | Code Completion | | v0.3 | gzb666 QA format | -7.4% | Code Completion | | v0.4 | Eval-aligned data | -5.1% | Basic Concepts | | v0.5 | Ewedubs commits | -4.1% | Advanced Internals | | **v0.6** | **Qwen-3.7-Max distilled** | **-4.9%** | **Basic Concepts +3.7%** | ## Limitations - Still experimental — knowledge improvements are modest - Best suited for kernel concept Q&A - Some degradation in code understanding tasks
