This mixed 6-bit/8-bit profile uses affine 6-bit weights by default and
protects the RNN-T joint, language-prompt projection, and final encoder FFN1
hotspots at 8-bit. Group size is 64. Convolution and normalization weights that
are not MLX-quantizable remain BF16.
The result matches the MLX Community uniform-8-bit quality and throughput class
on the full 428-sample FLEURS-Ar test set while reducing total artifact size by
16.75%.
Results
Measured locally on an Apple M4 Pro (14-core, 48 GB), MLX 0.32.0 and
mlx-audio 0.4.5 from git main. All models use the best-quality 1,120 ms
look-ahead ([56, 13]) and the same repository evaluator and normalizers.
Full FLEURS-Ar test set
Model
WER
CER
RTFx
Total artifact
MLX Community uniform 8-bit
10.72%
4.93%
20.0x
721.21 MiB
This mixed 6-bit/8-bit model
10.64%
4.95%
19.9x
600.39 MiB
The 0.08 WER and 0.02 CER differences are a practical tie. The defensible
result is equal full-set quality and throughput with an exact 120.83 MiB /
16.75% size reduction.
Matched fast screens
English is the macro-average of LibriSpeech clean, LibriSpeech other, AMI and
Earnings-22. Casablanca is the macro-average of eight Arabic dialects. Each
dataset uses 25 samples selected with seed 42; one empty AMI reference leaves
24 scored samples.
Model
English WER
FLEURS-Ar WER
Casablanca WER
Artifact
MLX BF16
12.29%
17.40%
66.14%
1,276 MB
MLX Community uniform 8-bit
12.02%
17.40%
66.08%
721.21 MiB
This mixed 6-bit/8-bit model
11.82%
16.74%
66.07%
600.39 MiB
Repo mixed 4/6/8-bit quality
12.86%
18.06%
67.34%
495.33 MiB
Repo mixed 3/4/6/8-bit size-matched
12.53%
18.94%
68.11%
472.33 MiB
Aufklarer uniform Q4
19.05%
20.88%
71.66%
473.69 MiB
Fast-screen sub-point differences are not statistically significant. They are
included to show the complete compression frontier and the controls tested
during development, not to claim accuracy superiority over BF16 or uniform
8-bit.
How the mixed allocation was built
The source is pinned to BF16 revision
e550040c0478027ed679b2b6b0d055502c103663.
Six FLEURS-Ar and six LibriSpeech-clean clips were selected with seed 42.
Each encoder component group and transducer region was independently
fake-quantized to 2-bit while the rest of the model stayed BF16.
Transcript CER against the BF16 teacher identified the RNN-T joint
(109.15%), final encoder FFN1 (96.32%), and prompt projection (86.72%) as the
dominant hotspots.
The performance bitmap uses 6-bit affine weights for 210 quantized modules
and 8-bit weights for 13 protected modules.
The final artifact was strict-loaded and evaluated end to end; no runtime
monkeypatch is required.
The exact allocation is included as quant_bitmap.json. Conversion and scan
code, rejected ablations, per-dataset measurements, and the external artifact
audit are maintained in
yuting1214/model-lab.
Usage
Nemotron support currently requires mlx-audio from git main:
The architecture is natively cache-aware and streaming-capable, but the
measurements above use whole-utterance inference. They do not establish
p50/p99 chunk latency, partial-transcript stability, or finalization latency.
Full FLEURS-Ar is complete. English and Casablanca values above are screening
slices; full LibriSpeech and the larger Casablanca gate remain pending.
Nemotron is not the Arabic offline-accuracy leader in this project. Its
differentiated product value is multilingual native streaming.
Accuracy varies by language, dialect, acoustic condition, and look-ahead.
Evaluate on your own target distribution before deployment.
License
The original NVIDIA model is released under the NVIDIA Open Model License.
This quantized derivative inherits those terms. Review the license for your
intended use and retain the required attribution.
