Voxtral-Mini-3B-2507, 8-bit with a bf16 audio encoder (MLX)
Voxtral Mini quantised
for Apple Silicon, with one deliberate asymmetry:
the audio encoder and its
projector stay in bf16 while the language model and
lm_head are quantised
to 8 bit.
| this build | uniform 8-bit | bf16 |
|---|
| size | 6.0 GB | 5.4 GB | 9.4 GB |
| speed (M1 Max) | 6.60x realtime | 6.68x | 1.46x |
| peak memory, 2 min audio | 7.7 GB | 7.2 GB | 13.2 GB |
Used in noScribe
These weights are one of the two transcription models in
noScribe — a free, open-source app for
high-quality interview transcripts (speaker diarization, timestamps, editable
output). This is the
recommended of the two: it runs on any Mac with 16 GB,
and on hard conversational audio it is more accurate than the 24B
Small build.
noScribe's optional Voxtral engine (Apple Silicon / MLX) lists it as
voxtral-mini-8bit and downloads it on first use, so nothing here has to be
fetched by hand. Setup and the engine's own notes:
VOXTRAL.md.
The engine currently lives on the
voxtral branch of this
fork of
kaixxx/noScribe.
Why the encoder is left dense
The encoder runs once per pass; lm_head runs once per generated
token. So encoder precision buys accuracy at almost no cost in speed, while
lm_head precision costs throughput and — measured — buys nothing.
Sweeping the encoder alone on the 24B sibling (everything else held fixed) on
hard German conversational audio:
| encoder bits | WER | CER |
|---|
| 4 | 10.19 % | 4.87 % |
| 6 | 9.72 % | 4.28 % |
| 8 | 8.29 % | 2.46 % |
| bf16 | 8.29 % | 2.41 % |
Monotone, with the character error rate halving between 6 and 8 bit. Above
8 bit nothing more is gained, which is why "8 bit or dense" is the right target
and 4-bit encoders are a bad idea.
The same sweep on lm_head (6 / 8 / bf16) produced identical error rates
while bf16 cost 27 % throughput.
Measured quality
Against a hand-corrected reference of hard German audio — two speakers, some
crosstalk, brand names, 422 words:
| build | WER | CER | insertions |
|---|
| this build | 4.27 % | 3.39 % | 0 |
| uniform 8-bit | 4.74 % | 3.34 % | 2 |
| bf16 | 4.27 % | 3.39 % | 0 |
On that passage this build is byte-identical to bf16 at 4.5x the speed.
For context, on the same audio and the same metric: whisper-large-v3-turbo (via
faster-whisper, int8, beam 5) scored 14.22 % WER, with 34 insertions including 27 consecutive
words of hallucinated text.
Honest limits
- The advantage only shows on difficult audio. On FLEURS German (100
recordings, read-aloud, clean) this build and the uniform 8-bit one score
identically: 4.81 % WER. If your material is clean, save the 0.6 GB.
- The hard-audio result rests on a single 422-word passage. The direction
is consistent with the encoder sweep above, but a 0.5-point difference on
422 words is four words. Treat it as suggestive, not settled.
- Measured on an M1 Max with
mlx-voxtral 0.0.4. Other MLX versions and
machines will differ.
- On FLEURS the 24B model is clearly better than any 3B build (2.82 %); on the
hard passage the 3B build wins (4.27 % vs 7.82 %). Pick by material, not by
parameter count.
Usage
1from mlx_voxtral import load_voxtral_model, VoxtralProcessor
2import mlx.core as mx, soundfile as sf
3
4model, _ = load_voxtral_model("MarkusKaemmerer/Voxtral-Mini-3B-2507-8bit-dense-encoder",
5 dtype=mx.bfloat16)
6proc = VoxtralProcessor.from_pretrained("MarkusKaemmerer/Voxtral-Mini-3B-2507-8bit-dense-encoder")
7
8audio, sr = sf.read("speech.wav", dtype="float32") # 16 kHz mono
9inp = proc.apply_transcrition_request(audio=audio, language="de", sampling_rate=sr)
10out = model.generate(input_ids=inp.input_ids, input_features=inp.input_features,
11 max_new_tokens=4096, temperature=0.0, repetition_penalty=1.0)
12print(proc.decode(out[0, inp.input_ids.shape[1]:], skip_special_tokens=True))
Set repetition_penalty=1.0. mlx-voxtral defaults to 1.2, a chat default
that in verbatim speech penalises punctuation and function words: measured on a
10-minute German podcast it cost 27 % of all commas and swallowed real
repetitions ("sehr, sehr" becoming "sehr").
Reproducing
Built with the conversion script from
this noScribe fork:
1python tools/quantize_voxtral.py mistralai/Voxtral-Mini-3B-2507 \
2 out-dir 8 64 dense-encoder --lm-head-bits 8
The measurement scripts (wer.py, fleurs.py) live in the same repository
under docs/skripte/, and the full write-up is in
docs/voxtral-quantisierung.md.
License
