Whisper Small Mla 24lang

Whisper-Small-MLA (24 languages) — MLA-converted Whisper, 62.5% smaller decode KV-cache

openai/whisper-small (244M) with decoder self-attention converted MHA→MLA (Whisper-MLA, arXiv:2603.00563), recovery-fine-tuned on a 24-language application set. Same 62.5% decode-cache cut as the rest of the family, on a wider language span. Conversion cost vs an identically-trained unconverted control: not statistically significant on 15 of 24 languages (n≤1500/lang (ko 339, no 370, ms 749, vi 1274; rest 1500), paired per-speaker bootstrap).

from transformers import AutoModelForSpeechSeq2Seq
model = AutoModelForSpeechSeq2Seq.from_pretrained("burakaydinofficial/whisper-small-mla-24lang", trust_remote_code=True)  # transformers==4.46.x

Results (CommonVoice-17 test — Malay from FLEURS, n≤1500/lang (ko 339, no 370, ms 749, vi 1274; rest 1500); CER is the metric for th/zh/ja)

lang	WER%	CER%	conversion cost (WER pts)	sig
en	12.9	6.7	0.72	sig
de	15.5	5.5	0.59	sig
es	10.0	3.3	-0.01	ns
fr	20.3	8.0	1.22	sig
it	15.1	4.2	-0.25	ns
pt	15.1	5.3	0.32	ns
ru	14.7	4.2	0.69	sig
nl	13.6	4.4	0.42	ns
pl	17.8	5.0	0.50	ns
id	20.2	6.6	0.69	ns
tr	22.0	6.0	0.89	sig
hi	24.2	12.0	0.96	sig
ms	21.0	7.6	0.94	sig
sv-SE	22.9	8.0	0.93	sig
th	37.4	12.4	0.25	ns
zh-CN	28.5	15.5	-0.08	ns
cs	27.9	7.2	0.54	ns
vi	28.6	13.6	0.28	ns
fi	28.8	5.6	0.34	ns
el	32.8	11.3	2.34	sig
da	32.9	12.0	0.59	ns
ja	n/a	24.3	1.01	ns
nn-NO	44.4	16.0	1.29	ns
ko	45.6	18.3	1.34	ns

Read this before deploying — data-driven quality tiers

The 24 languages are NOT equal in training data, and it shows in the absolute WER (the compression cost is small/ns almost everywhere — quality is a data story, not an MLA story):

• Well-resourced (usable): es 10, en 12.9, nl 13.6, ru 14.7, it/pt 15.1, de 15.5 WER; th CER 12.4, zh CER 15.5.
• Thin CommonVoice data (weaker, use with care): Korean 45.6, Norwegian 44.4, Danish 32.9, Greek 32.8, Finnish 28.8, Vietnamese 28.6, Czech 27.9 WER — single-shard languages in CV17.
• Malay (ms, 21.0 WER) was trained on FLEURS (no CommonVoice Malay exists; FLEURS is CC-BY-4.0); its eval-only license respected — the FLEURS audio is used for training here, never redistributed.
• Per-language conversion cost depends on the recovery mix: a shared language's cost here differs from the same language in whisper-small-mla-cv11 (the 11-language model) — e.g. French +1.22 here vs +0.32 there. Expected (mix composition moves each language's low-rank fit), not irreproducibility. Sub-zero costs on the CER-scored spaceless languages (zh) are within metric noise, not an MLA gain.

Matched control now published — verify the conversion cost yourself: burakaydinofficial/whisper-small-24lang (trained identically, minus the MHA→MLA conversion). Evaluate both with scripts/validate.py.

Limitations

• What the 62.5% is (cache scope): it is the decode self-attention KV-cache — the part that grows with output length and concurrency. The (larger, encoder-length ~1500-frame) cross-attention/encoder memory is NOT compressed, so single-stream total decode-memory savings are modest; the 62.5% cut compounds at output-length × batch concurrency, which is where it pays off.
• Runtime: requires trust_remote_code + transformers==4.46.x (no whisper.cpp/CT2/faster-whisper).
• Language coverage: covers these 24 languages; erodes Whisper's others (unseen scripts most).
• Decoding: greedy-decode evals (beam-5 adds ~1-2 WER on both arms; conversion cost unchanged — measured).
• Domain: consumer-mic read-speech.
• Compression cost: ≈ small-tier — ≈+0.6 median across all 24 languages here (≈+0.4 across the well-resourced subset). For reference, the 11-language cv11 family cost curve is tiny ≈+1.9 → base ≈+1.0 → small ≈+0.4 → medium ≈+0.4 → large ≈0 (approximate per-size medians).
• Training: 15k steps, warmup+cosine, encoder frozen both arms, dev-selected, bf16 (weights released as fp16).