A dynamically quantized MLX build of CohereLabs/cohere-transcribe-03-2026: instead of one bit-width per section, every module gets a bit-width assigned from an empirical quantization-sensitivity map (Unsloth-style dynamic quantization, adapted to ASR). Most of the encoder runs at 2-bit; the empirically fragile groups (decoder, last encoder blocks) stay at 3/4-bit.

Compared to the uniform 3-bit/4-bit build, this checkpoint is 14% smaller and ~15% faster at nearly the same WER.

Key Metrics

Metric	Value
Size	766 MB (vs 3.9 GB FP16 — 5.2x smaller)
Effective bits/param	3.294
WER (LibriSpeech test-clean, full)	1.12%
WER (LibriSpeech test-other, full)	2.50%
Composite WER	1.81% (FP16 baseline: 1.55%)
RTFx (M4 Air, MLX)	22–27x real-time

Method: sensitivity-guided bit allocation

Sensitivity scan — each (4-block, component) group of the 48-layer FastConformer encoder is fake-quantized to 2-bit alone (rest FP16) and scored by transcript divergence (CER) against the FP16 teacher on a calibration set.
Greedy allocation — groups above sensitivity thresholds are held at 4-bit / 3-bit; everything else drops to 2-bit (group size 64, affine). The decoder — the single most sensitive component — stays at 4-bit.
The resulting per-module rule map ships in this repo as quant_bitmap.json for full reproducibility.

Uniform 2-bit costs +2.01 pt WER; this allocation recovers ~75% of that damage for only +0.4 effective bits.

Component	Quantization
Encoder (most groups)	2-bit affine, group size 64
Encoder blocks 44–47 ffn2/pointwise-conv	4-bit
Encoder groups with mid sensitivity	3-bit
Decoder + head	4-bit
Norms / embeddings / mel frontend	FP16

1x1 Conv1d layers are converted to Linear equivalents to enable quantization.

Architecture

Base model: Cohere Transcribe 03-2026 (~2B params)
Encoder: FastConformer (48 layers, d=1280)
Decoder: Transformer (8 layers, d=1024)
Tokenizer: SentencePiece (16,384 tokens)

Usage

Requires mlx-audio installed from git main:

pip install "mlx-audio[stt] @ git+https://github.com/Blaizzy/mlx-audio.git"

python

# apply the loader patch shipped in this repo (fixes two upstream
# quantized-reload bugs; see mlx_audio_cohere_quant_patch.py)
from mlx_audio_cohere_quant_patch import apply_patch
apply_patch()

from mlx_audio.stt import load

model = load("MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit")
result = model.generate(audio="audio.wav", language="en", punctuation=True)
print(result.text)

Eval Results (Full LibriSpeech, seed=42)

Dataset	Samples	Audio Hours	WER	RTFx
LibriSpeech test-clean	2,620	5.4h	1.12%	23.4x
LibriSpeech test-other	2,939	5.34h	2.50%	20.8x

Evaluated with the HF Open ASR Leaderboard methodology (Whisper EnglishTextNormalizer + jiwer).

License

MIT — use it however you like, attribution appreciated.

The base model (CohereLabs/cohere-transcribe-03-2026) is Apache 2.0; its notices apply to the underlying weights.