Parakeet TDT 0.6B v3 — weight-only int4 ONNX for sherpa-onnx
A 4-bit weight-only quantization of NVIDIA's
Parakeet TDT 0.6B v3
for
sherpa-onnx, built from the fp32 export
csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3.
It is the model shipped by
OpenRamble for Windows,
a local dictation app.
Why not the stock int8 export
The published int8 export is dynamically quantized (DynamicQuantizeLinear + MatMulInteger /
ConvInteger): activations are cut to 8 bits at run time, including in the convolutional front
end that sees the spectrogram first. On a quiet or muffled microphone — a laptop microphone array
peaking around −25 dBFS — it answered Russian speech with nothing or with English filler, while the
fp32 export read the same recording correctly.
This build quantizes only the encoder's matrix weights (MatMulNBits, 4-bit, symmetric, blocks of
64, accuracy_level=4); activations stay fp32. Decoder and joiner are the unmodified fp32 files.
Measured on FLEURS ru dev (40 utterances, 710 words), sherpa-onnx 1.13.5, CPU:
| variant | WER | CER | size |
|---|
| stock int8 | 7.3% | 2.2% | 640 MB |
| this (int4 WO) | 5.5% | 1.5% | 719 MB |
| fp32 | 5.4% | 1.4% | 2.4 GB |
Same speed as the int8 export; peak memory about the same.
Files
| file | what |
|---|
encoder.int4.onnx | FastConformer encoder, MatMulNBits 4-bit weights, self-contained (no external data) |
decoder.onnx | prediction network, fp32 (from the source repo) |
joiner.onnx | joiner, fp32 (from the source repo) |
tokens.txt | SentencePiece tokens |
Requires an ONNX Runtime with the MatMulNBits contrib op (sherpa-onnx ≥ 1.13 / ORT ≥ 1.17).
Usage (sherpa-onnx, Python)
1import sherpa_onnx
2rec = sherpa_onnx.OfflineRecognizer.from_transducer(
3 encoder="encoder.int4.onnx", decoder="decoder.onnx", joiner="joiner.onnx",
4 tokens="tokens.txt", model_type="nemo_transducer", num_threads=4)
Reproduce
scripts/quantize-parakeet-int4.py in the OpenRamble for Windows repository downloads the fp32
export and produces exactly these files (deterministic; SHA-256 of the encoder is stable across
runs).
License
CC-BY-4.0, as the original model. Credit: NVIDIA (model), k2-fsa / csukuangfj (ONNX export).
