Hakam 0.8b Sportsqa Distill LoRA

Hakam: a distilled VLM on Sports-QA

We trained a 0.8B model to match a 27B model on sports video question answering. Same test accuracy, 30× smaller. It runs on a normal gaming GPU. That's what knowledge distillation buys you: the big model's skill at a fraction of the cost, for anyone who doesn't have a datacenter.

This repository holds the LoRA adapter (rank 16, ~25 MB) for Qwen/Qwen3.5-0.8B, distilled from Qwen3.5-27B (AWQ-Int4) on the Sports-QA dataset.

Try it now

Live demo on a free Colab T4 (opens a Gradio app with match replays pre-loaded):

Open in Colab

Or load it yourself:

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "Qwen/Qwen3.5-0.8B"
processor = AutoProcessor.from_pretrained(base, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    base, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True)
model = PeftModel.from_pretrained(model, "ananas0/Hakam-0.8b-sportsqa-distill-lora")
model.eval()

Video input: 2 fps, max 32 frames, max_pixels=256*28*28. The model's sweet spot is clips of 8–16 seconds with one action in them.

The numbers

Measured on 1,000 held-out Sports-QA test questions (fuzzy match against gold labels, threshold 0.80):

model	params	accuracy
Qwen3.5 0.8B, plain	0.8B	22.7%
0.8B + this adapter	0.8B	39.9%
Qwen3.5 27B (the teacher)	27B	39.7%

The student and teacher are statistically tied (n=1000, ±3pp 95% CI). On counterfactual questions the student scores higher than the teacher (82% vs 70%).

How it was trained

• Recipe: free-generation distillation. The 27B teacher answered ~11,700 training questions with a fixed analyst-style prompt; its per-token top-32 logits were recorded. The student was then teacher-forced on those generations with loss = 0.7·KL + 0.3·CE at Hinton temperature 2.0.
• Adapter: LoRA r=16, α=32, dropout 0.05 on all attention + FFN projections of the language model. Vision encoder frozen. 6.4M trainable parameters (0.74%).
• Training: 3 epochs, 35,016 steps, AdamW lr=1e-4, batch size 1, bf16.
• Data: a balanced subset of Sports-QA: 8 sports (football, basketball, volleyball, aerobic gymnastics, vault, uneven bars, balance beam, floor exercise), 200 training videos per sport.

Full pipeline, evaluation code and figures: github.com/ANANAS-HA/HAKAM

Demo replay clips: ananas0/hakam-demo-clips

Limitations

• Domain-specific: trained only on Sports-QA footage and question styles. It will describe other video reasonably (that's the base model) but the distillation gain is sports-specific.
• Temporal/causal questions whose gold answers use FineGym taxonomy terms remain hard; the teacher itself doesn't know that vocabulary, and a student can't learn what its teacher doesn't know.
• Clips longer than ~16 seconds get sampled sparsely (32 frames at 2 fps) and answer quality degrades.

Credits

Graduation project. Built on Qwen3.5 (Alibaba) and the Sports-QA dataset. Apache 2.0.