The source model is a NeMo SALM: a FastConformer encoder (32 layers, d_model 1024, 8x
subsampling) with a linear projection into the embedding space of an unmodified Qwen3-1.7B
decoder. It tops the Open ASR Leaderboard for English. Model and weights are by NVIDIA,
released under CC-BY-4.0.
Requirements
This export needs the speech-llm model family, which is not in upstream onnx-asr yet:
branch feat/speech-llm-canary
of the TigreGotico fork. The stock code of
PR #3 is not enough: the NeMo encoder
graph takes the feature length and returns the embedding length, which PR #3 does not pass.
decoder.onnx is a merged prefill and decode graph; a zero-length past is valid, so there is no
use_cache_branch. The chat-template prompt token ids (Transcribe the following: <audio>,
Qwen chat template) are baked into config.json, so no tokenizer encoder is needed at runtime.
The audio front end is the standard NeMo log-mel featurizer, so config.json selects the
existing nemo128 preprocessor of onnx-asr; no new preprocessor code was added.
Files
File
Size
encoder.onnx + encoder.onnx_data
42.4 MB + 3.25 GB
encoder_int8.onnx
859.3 MB
embed_tokens.onnx
1.24 GB
embed_tokens_int8.onnx
311.2 MB
decoder.onnx + decoder.onnx_data
1.6 MB + 6.88 GB
decoder_int8.onnx + decoder_int8.onnx_data
1.7 GB
vocab.json
3.1 MB
Total fp32 about 11.4 GB, total int8 about 2.9 GB. int8 is dynamic weight quantization
(QInt8, MatMulConstBOnly).
Accuracy
Four Google FLEURS validation clips, two en_us and two pt_br. The model is English-only;
the Portuguese clips are robustness probes, not a quality claim.
ONNX fp32 matches native NeMo SALM.generatecharacter for character on all four clips,
including the two Portuguese ones where both produce nonsense.
int8 keeps the English clips correct, with two trivial differences on the longer one
("Archipelago" instead of "archipelago", "chilies" instead of "chillies"). On the Portuguese
clips int8 collapses into repetition loops. Use int8 for English only.
Clip
native NeMo fp32
ONNX fp32
ONNX int8
en_1
When you call someone who is thousands of miles away you are using a satellite
identical
identical
en_2
Now widely available throughout the archipelago, Japanese cuisine features an array of simply seasoned dishes, ...
Model load (cold, weights on a 7200 rpm HDD): 787 s fp32, 119 s int8. On an SSD this is
dominated by the file read, not by onnxruntime.
fp32 on CPU is slower than PyTorch because the 11 GB of weights do not fit in cache and each
decode step is a separate graph run. int8 is the useful CPU configuration.
Limitations
English only. The encoder saw German, French and Spanish during pretraining, so other
languages produce plausible-looking nonsense rather than an error.
Batch 1 graphs. recognize_batch loops over the waveforms.
Greedy decoding only. No beam search, no timestamps.
Trained on audio up to 40 s and 1024 total tokens. Longer audio may degrade.
Only the baked ASR prompt. The LLM mode of the source model (summarize, answer questions
about a transcript) is not exported.
License and attribution
CC-BY-4.0, inherited from the source model. Model and weights: NVIDIA
(nvidia/canary-qwen-2.5b), built on
nvidia/canary-1b-flash and Qwen/Qwen3-1.7B. This repository only adds the ONNX export.
