Granite-Speech-3.3-2b — ONNX (speech-llm)
ONNX export of
ibm-granite/granite-speech-3.3-2b
for
onnx-asr with the
speech-llm model type (requires the
speech-llm model-family patches, see
TigreGotico/onnx-asr PR #3).
The source model applies a LoRA adapter to the language model only when the turn contains audio.
Audio is always present for ASR, so the adapter is merged into the base weights before the export
and the decoder is a plain causal LM. Merged output matches the adapter path character for
character.
Graphs
| Graph | Inputs | Outputs |
|---|
encoder.onnx | input_features (1, N) f32 raw 16 kHz waveform | audio_embeds (1, L, 2048) f32 |
embed_tokens.onnx | input_ids (1, S) i64 | inputs_embeds (1, S, 2048) f32 |
decoder.onnx | inputs_embeds (1, S, 2048), attn_bias (1, 1, S, P+S), position_ids (1, S), past_key_values.{0..39}.{key,value} (1, 8, P, 64) | logits (1, S, 49160), present.{0..39}.{key,value} (1, 8, P+S, 64) |
decoder.onnx is a merged prefill and decode graph: a zero-length past is valid, so there is no
use_cache_branch. Prompt token ids (the Granite chat template with the transcription
instruction) are baked into config.json, so no tokenizer encoder is needed at runtime.
The Granite feature extractor (torchaudio mel spectrogram, n_fft 512, window 400, hop 160,
80 HTK mel bins, log10 with an 8 dB floor, then pairs of frames stacked into 160 features)
matches no onnx-asr preprocessor. It is computed inside encoder.onnx instead, and the model
declares "preprocessor": "identity", so the runtime hands the raw waveform to the graph. The
audio length stays dynamic; there is no fixed 30 s window.
Usage
1import onnx_asr
2model = onnx_asr.load_model("speech-llm", "path/to/this/repo") # or quantization="int8"
3print(model.recognize("audio_16khz.wav"))
Accuracy
Google FLEURS validation clips, two en_us and two pt_br, greedy decoding. fp32 matches
native transformers (with the LoRA adapter active, fp32 CPU) character for character on all
four clips.
int8 is dynamic weight quantization. The text graphs survive it well, but the conformer encoder
does not: quantizing it whole destroys the audio embeddings (cosine 0.43 against fp32). Keeping
the in-graph mel front end in fp32 raises that to 0.96, which is what this repo ships. Even then
int8 changes wording on longer clips. Use fp32 when the transcript matters.
| Clip | fp32 | int8 |
|---|
| en_1 | identical to native | identical to native |
| en_2 | identical to native | "the predominant flavors being peanuts, chili, sugar" |
| pt_1 | identical to native | "que oriente e pite as ciencias" |
| pt_2 | identical to native | "espiar e uma atividade", "skibums" |
Files
| File | Size |
|---|
encoder.onnx + encoder.onnx_data | 0.7 MB + 1.90 GB |
encoder_int8.onnx | 480.6 MB |
embed_tokens.onnx | 402.7 MB |
embed_tokens_int8.onnx | 100.7 MB |
decoder.onnx + decoder.onnx_data | 2.1 MB + 10.13 GB |
decoder_int8.onnx + decoder_int8.onnx_data | 3.1 MB + 2.53 GB |
vocab.json | 875 kB |
config.json | 1 kB |
fp32 is about 12.4 GB in total, int8 about 3.1 GB.
Limitations
- Batch 1.
recognize_batch loops over the waveforms; because the preprocessor is the identity,
a padded batch would send its zero padding to the encoder, so batch entries must have equal
length to be exact.
- Greedy decoding only. No beam search and no timestamps.
- English, French, German, Spanish and Portuguese, as in the source model. Speech translation is
not wired up: the baked prompt is the transcription instruction.
- Free-text prompts would need a tokenizer encoder at runtime.
License: Apache-2.0, inherited from the source model.
