Nemotron Speech Streaming EN 0.6B — ONNX int8 (560 ms)
Dynamic int8-quantized ONNX export of
nvidia/nemotron-speech-streaming-en-0.6b for use with
parakeet-rs.
English-only, cache-aware streaming ASR (~600M parameters). Preserves disfluencies (um, uh) and outputs verbatim English with punctuation.
This bundle uses the 560 ms streaming mode (--right-context 6 at export, the default NVIDIA preset). Other latencies (80 / 160 / 1120 ms) require a separate export and matching CHUNK_SIZE in parakeet-rs.
Files
All files must live in the same directory:
| File | Size (approx.) | Description |
|---|
encoder.int8.onnx | ~623 MB | Streaming encoder (dynamic int8 weights, single file) |
decoder_joint.int8.onnx | ~8.6 MB | RNNT decoder + joint network |
tokenizer.model | ~246 KB | SentencePiece tokenizer |
No config.json is required at inference time — parakeet-rs embeds streaming constants in Rust.
Size vs fp32
| Component | fp32 | int8 |
|---|
| Encoder | ~2.3 GB | ~623 MB |
| Decoder/joint | ~35 MB | ~8.6 MB |
| Total weights | ~2.4 GB | ~632 MB |
Roughly 75% smaller than the fp32 ONNX export.
Model details
| Property | Value |
|---|
| Base model | nemotron-speech-streaming-en-0.6b |
| Architecture | Cache-aware FastConformer + RNNT |
| Sample rate | 16 kHz mono |
| Mel features | 128 |
| Subsampling | 8× |
| Encoder layers | 24 |
| Hidden dim | 1024 |
| Vocab size | 1024 (+ blank) |
| Streaming latency | 560 ms per encoder step |
| Export flags | --right-context 6, --left-context 70 |
| Mel frames / step | 56 |
| Audio samples / step | 8960 @ 16 kHz |
| Attention context | left=70, right=6 |
How this was produced
- Export fp32 ONNX at 560 ms latency:
1python parakeet-rs/scripts/export_nemotron_streaming.py \
2 nemotron-speech-streaming-en-0.6b/nemotron-speech-streaming-en-0.6b.nemo \
3 out_onnx/ \
4 --right-context 6
- Quantize to int8:
1pip install onnx 'onnxruntime>=1.20'
2
3python parakeet-rs/scripts/quantize_nemotron_streaming.py \
4 out_onnx/ \
5 out_onnx_int8/ \
6 --clean-output
Quantization uses ONNX Runtime dynamic int8 (quantize_dynamic): encoder QUInt8, decoder/joint QInt8.
Streaming latency presets (export only)
| Latency | --right-context | Mel frames/step | Samples/step @ 16 kHz | parakeet-rs CHUNK_SIZE |
|---|
| 80 ms | 0 | 8 | 1280 | 8 |
| 160 ms | 1 | 16 | 2560 | 16 |
| 560 ms (this bundle) | 6 | 56 | 8960 | 56 |
| 1120 ms | 13 | 112 | 17920 | 112 |
Latency is not selectable in the quantize script or Nemotron::from_pretrained — re-export fp32, re-quantize, and update CHUNK_SIZE in parakeet-rs/src/nemotron.rs.
Usage (parakeet-rs)
Nemotron::from_pretrained takes the model directory and an optional execution provider only. Chunking is done via transcribe_chunk, which buffers audio until 560 ms of new audio is ready (for this export).
1use parakeet_rs::Nemotron;
2
3let mut model = Nemotron::from_pretrained("./out_onnx_int8", None)?;
4
5// Feed any slice size; encoder runs every 560 ms of new audio.
6for frame in audio_source.frames() {
7 let text = model.transcribe_chunk(&frame)?;
8 print!("{}", text);
9}
10
11// Or feed 560 ms blocks directly (8960 samples @ 16 kHz)
12const FEED_560MS: usize = 8960;
13for chunk in audio.chunks(FEED_560MS) {
14 let text = model.transcribe_chunk(chunk)?;
15 print!("{}", text);
16}
Example:
1cd parakeet-rs
2cargo run --release --example streaming /path/to/audio.wav
3# Point the example at this directory (./nemotron or pass path)
The loader picks encoder.int8.onnx and decoder_joint.int8.onnx when present. This repo build sets CHUNK_SIZE = 56 for the 560 ms export.
Requirements
- Inference: parakeet-rs with ONNX Runtime
- Re-export / re-quantize: Python 3.10+, NeMo (export only),
onnx, onnxruntime>=1.20
Quality notes
- Dynamic int8 quantizes weights only; activations stay float32 at runtime.
- Run WER checks on your audio before production deployment.
- ORT warnings about unquantizable
Slice/Tile tensors during quantize are expected.
License
Derived from NVIDIA Nemotron Speech Streaming. Use and redistribution follow the license terms of the
original model.
