Whisper-Tiny-MLA (11 languages) — MLA-converted, 62.5% smaller decode KV-cache
The on-device-tier sibling of the WhisperMLA family:
openai/whisper-tiny (39M) with its
decoder self-attention converted MHA→MLA (per
Whisper-MLA, arXiv:2603.00563),
recovery-fine-tuned on 11 languages of the CC0
Whispered corpus (32k clips/lang).
1from transformers import AutoModelForSpeechSeq2Seq
2model = AutoModelForSpeechSeq2Seq.from_pretrained("burakaydinofficial/whisper-tiny-mla-cv11", trust_remote_code=True) # transformers==4.46.x
Honest sizing note (read this first)
Conversion cost grows as the student shrinks — measured across the family: small ≈ +0.4
median WER → base ≈ +1.0 → tiny ≈ +1.9. At the tiny tier you pay ≈ +1.9 WER (median) for the
62.5% cache cut. If quality is the priority, prefer the small variant; this tier is for
memory-constrained deployments where the cache cut matters most.
Results (CommonVoice-17 test, n=1500/lang; WER/CER %; cost = paired vs an identically-trained unconverted control)
| Lang | this model (WER / CER) | conversion cost |
|---|
| en | 29.1 / 15.8 | +2.41 ✱ |
| de | 42.8 / 16.6 | +1.95 ✱ |
| es | 29.4 / 10.7 | +1.07 ✱ |
| fr | 45.4 / 20.1 | +2.05 ✱ |
| ru | 42.9 / 14.2 | +3.32 ✱ |
| tr | 53.7 / 17.6 | +2.63 ✱ |
| cy | 86.5 / 38.0 | −0.19 (ns) |
| ar | 67.5 / 28.7 | +1.58 ✱ |
| th | 80.9 / 31.1 | +0.81 CER ✱ |
| zh | 53.7 / 33.2 | −1.63 CER (ns) |
| ka | 122.1 / 81.0 | −0.50 (ns) — floor |
Absolute quality is tiny-tier-typical (much lower than small — that is the base model, not
MLA). Encoder frozen both arms; 15,000 steps; warmup+cosine; bf16 (weights released as fp16).
Matched control now published — verify the conversion cost yourself:
burakaydinofficial/whisper-tiny-cv11 (trained identically, minus the MHA→MLA conversion). Evaluate both with
scripts/validate.py.
Limitations
- Costs are recovery-mix-specific: do NOT compare a language's conversion cost across the cv11 and 24-language lines — different recovery sets move each language's low-rank fit (French in the small model: +0.32 cv11 vs +1.22 24-lang). Not irreproducibility.
- What the 62.5% is (cache scope): it is the decode self-attention KV-cache — the part that grows with output length and concurrency. The (larger, encoder-length ~1500-frame) cross-attention/encoder memory is NOT compressed, so single-stream total decode-memory savings are modest; the 62.5% cut compounds at output-length × batch concurrency, which is where it pays off.
- Requires
trust_remote_code=True and transformers==4.46.x (custom MLA attention modules); not loadable in whisper.cpp / faster-whisper / CTranslate2.
- Language coverage is the 11 recovery languages — unseen scripts degrade.
- Georgian (ka) reported as the labeled model-class floor; not usable at this tier.
- Read-speech domain, greedy-decode evals — beam-5 adds ~1-2 WER on both arms without changing the conversion cost (measured on the small tier).
Acoustic conditions of the evaluation
Evaluated on crowdsourced consumer-microphone recordings with real environmental noise —
traffic, room reverb, variable devices — CommonVoice's native conditions, not studio audio.
The numbers above already include that heterogeneity. Not yet benchmarked: far-field, telephony
(8 kHz), overlapping speech; an SNR-ladder robustness section will be added when measured.
FLEURS (out-of-domain) — this model, WER% (CER% for th/zh)
ABSOLUTE FLEURS numbers on a different read-speech corpus. Do NOT compare these per-language to the CommonVoice table above — the corpora differ in transcript normalization/orthography and speaker pools (e.g. Welsh reads easier on FLEURS, Georgian worse), so cross-corpus swings are corpus artifacts, not model behavior. The clean out-of-domain claim is the PAIRED MLA-vs-control cost (verified to be small on the small tier). Per-language absolute: en 22.9 · de 37.1 · es 24.2 · fr 44.6 · ru 46.5 · tr 50.9 · cy 88.1 · ar 70.6 · th 44.4 · zh-CN 32.3 · ka 100.9.
(Full JSON: the release repo's eval artifacts.)
