rekody-streaming-en-0.6b-160ms-int8
An int8 ONNX conversion of NVIDIA's Nemotron speech streaming 0.6B model
(the March 2026 checkpoint), exported at the
160 ms cache-aware latency
profile and packaged for on-device streaming dictation. This is the model
behind
Rekody, the private, local-first voice
dictation tool for macOS.
- Cache-aware streaming ASR: transcribes while you speak, in 160 ms
chunks, with encoder cache carried across chunks. No audio leaves the
machine.
- Runs on the ONNX Runtime CPU execution provider. The files load directly
with onnxruntime or
parakeet-rs.
- Four files:
encoder.onnx (int8, 880 MB), decoder_joint.onnx (int8,
11 MB), tokenizer.model (SentencePiece), and config.json.
What "160 ms" means, and why it is a separate repository
The checkpoint supports several cache-aware right-context settings. The
attention right context R fixes how many encoder frames are produced per
call, and each frame is 80 ms:
| Profile | R | Encoder frames per chunk | Chunk |
|---|
[70,0] | 0 | 1 | 80 ms |
[70,1] (this repo) | 1 | 2 | 160 ms |
[70,6] (the 560 ms repo) | 6 | 7 | 560 ms |
[70,13] | 13 | 14 | 1120 ms |
The profile is baked into the exported graph, so this artifact is not a
drop-in replacement for the 560 ms one. Feeding this encoder a 560 ms chunk
(65 mel frames) returns only the first 2 output frames and silently discards
five sevenths of the audio — no error is raised. That is why the two profiles
live in separate repositories rather than as revisions of one: a consumer
pinned to the 560 ms bytes keeps getting exactly those bytes.
Consumers that read the profile from the file do the right thing
automatically. The encoder's ONNX metadata_props carry:
| Key | Value |
|---|
chunk_size_output_frames | 2 |
pre_encode_cache | 9 |
These are the key names and decimal-string format that
parakeet-rs >= 0.3.7 reads from the encoder session, defaulting to 7 and
9 when absent. Mel frames per encoder call is
chunk_size_output_frames * 8 + pre_encode_cache = 25, i.e. 2560 audio
samples of new audio per chunk at 16 kHz.
Accuracy at this profile
NVIDIA publishes word error rate by chunk size for this checkpoint across the
eight Open ASR Leaderboard sets. Reproduced here from the
base model card
as upstream data, not as a Rekody measurement:
| Chunk | R | Average | AMI | Earnings22 | Gigaspeech | LS-clean | LS-other | SPGI | TEDLIUM | VoxPopuli |
|---|
| 0.56 s | 6 | 7.07 | 11.88 | 12.82 | 9.78 | 2.46 | 5.07 | 3.03 | 3.54 | 8.00 |
| 0.16 s | 1 | 7.67 | 14.71 | 13.01 | 10.34 | 2.56 | 5.57 | 3.25 | 3.77 | 8.18 |
The +0.60 pp average gap is not evenly spread: AMI alone contributes
+2.83 pp of it. Excluding AMI, the far-field multi-party meeting set, the
mean of the remaining seven moves from 6.39% to 6.67%, a +0.28 pp difference.
For near-field single-speaker dictation, which is what this artifact is
packaged for, the relevant number is the smaller one.
Rekody has not re-run the Open ASR Leaderboard at this profile, so no
Rekody-measured leaderboard table is published here. The 560 ms repository's
table should not be read as applying to this artifact.
Why trust this conversion
- Produced by the same pipeline as the 560 ms artifact, with
--right-context
as the only changed input: the same pinned checkpoint revision, the same
NeMo 2.7.3 export script, and the same onnxruntime 1.23.2
quantize_dynamic QInt8 call over MatMul and LSTM ops.
- The export was gated on parity against NeMo's own streaming step on the
same inputs: max absolute difference 0.0.
decoder_joint.onnx came out bit-identical to the 560 ms repository's
(89cae615… before metadata stamping). The RNN-T decoder does not depend on
the encoder's latency profile, so it must not change — and it did not, which
is a direct check that the pipeline reproduced the earlier conversion.tokenizer.model is byte-identical to the 560 ms repository's.- The quantization coverage matches the 560 ms artifact exactly: 217
MatMulInteger, 170 DynamicQuantizeLinear, 0 ConvInteger, 77 float
Conv in the encoder; 2 DynamicQuantizeLSTM and 3 MatMulInteger in the
decoder, with the embedding table left fp32.
- The cost of int8 quantization on this architecture was measured as
statistically zero on the 560 ms export: on LibriSpeech test-clean
(n = 2620), int8 differs from the fp32 export it came from by +0.072 pp WER
with a paired bootstrap 95% CI of [-0.004, +0.150], which includes zero.
License
Two licenses apply to this repository, and users must comply with both:
- The conversion work (the int8 quantization, packaging, and
benchmarking published here) is licensed under the
PolyForm Shield License 1.0.0
(see
LICENSE). PolyForm Shield is a noncompete license: any use is
permitted except providing a product that competes with the software or
with the licensor's products built on it.
- The base model weights remain licensed by NVIDIA Corporation under the
NVIDIA Open Model License
(see
NOTICE and the dated copy in licenses/).
File integrity (SHA-256)
| File | SHA-256 |
|---|
encoder.onnx | 5a8f5c01d7804346e3e79d539bde6011907961192c578441d7b2eb725f82220c |
decoder_joint.onnx | 6fa6b88cd439b0281b1b9361c5789d51c851fe202c43bb2fdccabb87186a9630 |
tokenizer.model | 07d4e5a63840a53ab2d4d106d2874768143fb3fbdd47938b3910d2da05bfb0a9 |
config.json | af9bf85d1b620ee83147149120e02dd5b7b57f2c2a845be05c61b707e0f98802 |
These hashes are also published in SHA256SUMS.
Provenance
- Source: nvidia/nemotron-speech-streaming-en-0.6b,
revision
df1f0fe9dfdf05152936192b4c8c7653d53bf557 (the March 13, 2026
checkpoint).
- Conversion recipe: NeMo 2.7.3 ONNX export at
att_context_size = [70, 1],
then onnxruntime 1.23.2 quantize_dynamic QInt8 over MatMul and LSTM ops,
then the streaming profile stamped into the encoder's ONNX metadata.
