LFM2.5-Audio-1.5B-ONNX-Vulkan (deck profile, Tier-1)

1. decoder_q4: lm_head f32 -> symmetric q4 MatMulNBits. The stock release quantized its 92 layer GEMMs but left the weight-tied lm_head as an f32 table (536,870,912 B, on every generation step). It is now one 3-input symmetric MatMulNBits node (com.microsoft domain, same per-32-block convention as the release's own 92 MMBs: scale = max/7, q = clip(round(w/s)+8, 0..15), two int4 per byte low-nibble-first, dequant (q-8)*s): quant uint8 [65536,64,16] (67,108,864 B) + scales f32 [65536,64] (16,777,216 B). Data file: 1,217,650,688 -> 764,665,856 B.
2. host embedding tables: f32 -> native bfloat16. embed_tokens.bin (536,870,912 -> 268,435,456 B) and audio_embedding.bin (134,283,264 -> 67,141,632 B) now store the original bf16 checkpoint bytes. The f32 bins were the exact bf16->f32 bit upcast, so this is bit-exact (all three verified bit-identical before conversion); consumers upcast at lookup. dtype in the adjacent .json records the form.

Files

Load contract

• 24 graph inputs, inputs_embeds (f32) first - the host feeds embeddings from the bf16 tables (exact upcast at lookup); KV/conv caches are zeroed per turn and threaded past_* <- present.*.
• 93 MatMulNBits nodes (92 stock + the new lm_head), all 3-input symmetric, bits=4, block_size=32, com.microsoft domain (this ORT build resolves that form; a plain ai.onnx MatMulNBits node fails to load in ORT 1.26).
• One pre-existing custom op: SimplifiedLayerNormalization (com.microsoft) - the generic onnx.checker flags it; the Vulkan runtime implements it natively (the stock graph fails the checker identically).
• The runtime is the onnx-vulkan-rs fork (matbeedotcom/onnx-vulkan-rs-edge, main at 2d660f2) - the deck also needs its role-aware decoder multiplexer; the graphs themselves only use release-native ops.

Validation

• Convention check: dequantizing a stock release MMB (layer-9 up_proj) with this recipe against the bf16 checkpoint lands within half a quantization step (max_err 0.0230713 <= half-step 0.0306396).
• CPU ORT greedy parity vs the stock f32-lm_head graph: 4/4 prompts bit-identical (text-first prompts; the TTS prompt stops at AUDIO_START as expected). The q4 lm_head changes no greedy token on the covered prompts.
• Instruct role (sister repo): 21/21 bit-identical vs bf16 PyTorch reference, re-confirmed after the Tier-1 table conversion.
• GPU showcase (2026-08-23, Vulkan, full dual bundle): 14/14 ALL-PASS on this exact decoder. The five TTS turns (the q4-lm_head quality gate) all pass with decode_rtf 0.72-0.95 and are at or below the stock-lm_head run (t3 12.5 s -> 10.9 s).

Reproduction