Weight-only int8 quantization of the encoder of NVIDIA's nemotron-speech-streaming-en-0.6b (English FastConformer encoder + fused RNN-T decoder/joint, with punctuation & capitalization), packaged for the parakeet-rs ONNX runtime.

~876 MB total vs ~2.34 GB for the FP32 export (≈2.74× smaller).
Quantized only the encoder's MatMul weights to int8 (MatMulInteger / DynamicQuantizeLinear); Conv and other tensors stay FP32 to avoid ConvInteger kernel gaps, and the fused decoder_joint + tokenizer are unchanged.
Encoder graph I/O is preserved (processed_signal in, encoded out), so it is a drop-in for runtimes that load the FP32 export. Streaming framing is unchanged (560 ms).

Files

file	precision	notes
`encoder.onnx` (+ `.data`)	int8 (MatMul)	cache-aware streaming FastConformer encoder
`decoder_joint.onnx`	fp32	fused RNN-T prediction net + joint (verbatim copy)
`tokenizer.model`	—	SentencePiece (verbatim copy)

Provenance

Base model: nvidia/nemotron-speech-streaming-en-0.6b (NVIDIA Open Model License).
FP32 ONNX export source: altunenes/parakeet-rs (subdir nemotron-speech-streaming-en-0.6b).
int8 quantization: onnxruntime.quantization.quantize_dynamic (op_types_to_quantize=['MatMul'], weight_type=QInt8, per_channel=True).

License

NVIDIA Open Model License Agreement: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/. The upstream NVIDIA license governs the weights regardless of export or quantization source. See NOTICE.txt.

bumpyclock/nemotron-speech-streaming-en-0.6b-int8

Nemotron Speech Streaming en 0.6B — int8 (parakeet-rs compatible)

Files

Provenance

License

bumpyclock/
nemotron-speech-streaming-en-0.6b-int8