INT4 k-quant (MatMulNBits) — smallest and fastest ONNX model converted from NVIDIA Nemotron 3.5 ASR Streaming 0.6B to native ONNX opset 24 for CPU inference with ONNX Runtime GenAI.

Streaming chunk size: 1.12s — maximum encoder context — recommended for INT4 (extra context compensates quantization noise).

Model Details

Property	Value
Base model	NVIDIA Nemotron 3.5 ASR Streaming 0.6B
ONNX opset	24 (native)
Encoder layers	24
Hidden size	1024
Precision	INT4 k-quant
Chunk size	1.12s (17920 samples @ 16kHz)
Left context	140 encoded frames
Sample rate	16,000 Hz
Vocab size	13,088
Languages	80+ (multilingual)
VAD	Silero VAD included
Total size	~793 MB

Files

File	Size
`encoder.onnx` + `.data`	682 MB
`decoder.onnx` + `.data`	57 MB
`joint.onnx` + `.data`	36 MB
`silero_vad.onnx`	2 MB

Requirements

ONNX Runtime ≥ 1.25 (opset 24 support on CPU).
Tested with Microsoft.ML.OnnxRuntime 1.25.1 + Microsoft.ML.OnnxRuntimeGenAI 0.14.1.

Quick Start

C# / .NET

csharp

using NemotronSpeech;

using var session = new ModelSession(
    modelPath: "path/to/this/model",
    executionProvider: "cpu",
    langId: null,
    useVad: false);

CLI

bash

NemotronSpeech.exe <model_path> <audio_file> cpu
NemotronSpeech.exe <model_path> --mic cpu --language en

Choosing a chunk size

Chunk	Latency	Encoder context	Best for
0.08s / 0.16s	Lowest	Minimal	Ultra-low-latency dictation
0.56s	Balanced	70 frames	Recommended default (NVIDIA)
1.12s	Higher	140 frames	Best accuracy, code-switching, multilingual

Larger windows give the encoder more context, which improves WER — especially noticeable for quantized (INT4) builds and code-switched / multilingual audio.

Related models

DimQ1/nemotron-3.5-asr-streaming-0.6b-onnx-fp32-cpu — opset 21 FP32
DimQ1/nemotron-3.5-asr-streaming-0.6b-onnx-int4-cpu — opset 21 INT4 (fastest)

DimQ1/nemotron-3.5-asr-streaming-0.6b-onnx-int4-opset24-c112-cpu

Nemotron 3.5 ASR Streaming 0.6B — ONNX INT4 k-quant (CPU, opset 24, chunk 1.12s)