Live streaming demo: partial transcripts in real time from the ONNX fp16 engine on CPU (RTF 0.28)
Model overview: multilingual audio across 40 language-locales is transcribed by a cache-aware FastConformer-RNNT model with language-ID prompting into punctuated text with an automatic language tag (image: NVIDIA)
NVIDIA's newest multilingual streaming ASR, running anywhere ONNX Runtime runs. No PyTorch, no GPU required.
🌍 One 600M model, 40 language-locales, with automatic language detection built in. The model emits an <xx-XX> tag per utterance, so you don't need a separate language-ID component.
⚡ True cache-aware streaming: configurable latency (80 ms to 1.12 s chunk sizes), punctuation and capitalization natively in the stream.
💻 3.8x faster than real-time on CPU (RTF 0.26 fp32, 6 real meeting recordings, onnxruntime CPU, no CUDA).
🎯 WER 0.0137 vs the PyTorch reference (fp16 build, 6 files, streaming mode). This package reproduces the original model, and we measured it.
📉 Competes with Whisper large-v3 at about 1/3 of the parameters, and large-v3 can't stream.
Unofficial community export by @codavidgarcia.
Weights remain NVIDIA's, under the
OpenMDW-1.1 license (see
LICENSE-OpenMDW-1.1 and NOTICE). Not affiliated with or endorsed by NVIDIA.
Why this exists
nvidia/nemotron-3.5-asr-streaming-0.6b (June 2026) is one of the best open
streaming ASR models right now, but it ships as a NeMo/PyTorch checkpoint.
That means a multi-GB dependency chain and no easy path into
sherpa-onnx-style lightweight runtimes
(sherpa-onnx#3573,
sherpa-onnx#3408).
For the smallest possible footprint, see also the int4 ONNX build from
onnx-community,
which targets onnxruntime-genai at a fixed 560 ms chunk. This repo sits at
the other point of the same trade-off: near-lossless fp16 (WER 0.0137 vs
the original), plain onnxruntime with no extra runtime dependencies, any
chunk size, and the full export + engine + validation tooling so you can
regenerate and verify everything yourself.
Total download: ~2.5 GB. The 320 ms chunk is the latency/accuracy sweet
spot (RTF 0.26 vs 0.65 at 80 ms on CPU). Encoders for 80/160/560/1120 ms can
be generated with the export tooling
(source repo).
Quickstart
pip install numpy onnxruntime soundfile scipy
python
1from nemotron_onnx_streaming import NemotronOnnxStreaming # from the tools repo23engine = NemotronOnnxStreaming(".", language="auto", chunk_ms=320, precision="fp16")4engine.accept_waveform(pcm_float32_16k)# feed any number of samples5print(engine.get_partial())# live hypothesis6print(engine.get_final())# flush + final transcript7print(engine.detected_language)# e.g. <es-ES> in auto mode
One-file CLI:
bash
1python nemotron_onnx_streaming.py . meeting.wav --language auto --precision fp16
2# text: Actually, the masterminds behind this ...3# detected language: <en-US>4# RTF: 0.287
The engine (nemotron_onnx_streaming.py) and the export/validation tooling
live in the companion code repository:
github.com/codavidgarcia/nemotron-3.5-asr-streaming-onnx.
The engine is deliberately small (about 600 lines, numpy + onnxruntime only)
so you can vendor it into other projects, or use it as the reference for a
sherpa-onnx C++ port.
Validation (measured)
WER parity vs the 🤗 Transformers reference, 6 real meeting-audio files
(3 to 60 s, language=auto), jiwer, lowercase + punctuation stripped,
torch 2.13 CPU + onnxruntime 1.27, opset 17:
Precision
WER vs HF reference
RTF (CPU)
Size (320 ms)
fp16 (this repo)
0.0137
0.315
~2.5 GB
fp32
0.0082
0.263
~4.7 GB
int8 (dynamic)
0.189
0.148
~0.7 GB
Chunk-size sweep (fp32 graphs, parity around 1e-6 vs HF streaming on all sizes):
Chunk
Latency
RTF (CPU)
80 ms
~0.08 s
0.653
320 ms
~0.32 s
0.263
1120 ms
~1.12 s
0.106
FLEURS average WER vs streaming chunk size, LangID vs auto-detect (image: NVIDIA)
NVIDIA's FLEURS numbers by chunk size (image: NVIDIA). Accuracy improves with chunk size and stays competitive at 80 ms.
Graph-level parity with the original cache-aware streaming path:
max abs diff below 1.5e-05 over first + 5 steady chunks. Text-level parity
near-verbatim at 320 ms and 1120 ms (cross-checked against HF streaming at
the same lookahead on 60 s of real meeting audio).
A note on what "parity" means here: it is the WER between this ONNX export
and the original model on the same audio, so it measures the fidelity of the
conversion. It does not measure absolute ASR accuracy. For accuracy
benchmarks see NVIDIA's FLEURS tables on the
base model card
(avg LangID WER 8.84 to 10.38 across 19 locales depending on chunk size).
Known limitations
int8 degrades measurably (WER 0.19 vs fp32). It is fast (RTF 0.15) but
we don't ship it as the default. Static quantization with calibration is on
the roadmap.
fp16 on CPU doesn't accelerate vs fp32. It halves download and RAM. The
speed win shows on GPU via CUDAExecutionProvider.
Word-level timestamps are not exposed by this RNNT export.
The engine is single-stream. Batch streaming is a tooling roadmap item.
License and attribution
Weights: NVIDIA, OpenMDW-1.1. Use,
modification and redistribution permitted with attribution. This file plus
NOTICE plus LICENSE-OpenMDW-1.1 satisfy that requirement.
Code (export/engine/validation in the companion repo): Apache-2.0.
Base checkpoint revision: f3d333391852ba876df169dcc9ba902d25b6ab0b.
Exported with transformers>=5.13.0 (Nemotron3_5AsrForRNNT),
torch.onnx.export (dynamo=False, opset 17), custom fp16 weight cast.
