Cohere Transcribe 03-2026 — DirectML-safe, fully-static ONNX

Why the stock export can't run well on DirectML

• The fused decoder attention (com.microsoft.MultiHeadAttention cross-attn + GroupQueryAttention self-attn) crashes / mis-computes on the DML EP (ORT ≤ 1.24, unfixed).
• Even after decomposing to plain ops, the DML EP re-fuses its graph on every autoregressive shape change (~34 ms/token fixed), so naïve GPU decode loses to CPU.

What changed (three passes, tools/onnx/cohere_decompose_attention.py)

1. Attention decomposed to Reshape/Transpose/MatMul/Softmax/… with an explicit causal mask; the two cross-attn If nodes flattened away.
2. Loop-invariant cross-KV hoisted into the encoder — the cross-attention K/V (which depend only on the audio) are computed once per utterance in encoder_model*.onnx (extra cross_attn.* outputs) instead of on every decoded token.
3. Fully static decode — the self-KV cache is a fixed-length buffer updated by a masked write, the cross-KV are padded to a fixed length with a cross_bias mask, and every per-step shape is constant. The DML EP now compiles the decoder once and reuses it, so per-token cost is constant ~4 ms regardless of audio length, entirely on the GPU.

• decoder_model_merged*.onnx — fully static (fixed self- + cross-KV). Fastest on DirectML (metadata winstt_static_kv).
• decoder_model_merged*_dyn.onnx — growing self-KV (cross still fixed). Faster on the CPU EP, which doesn't benefit from static shapes. WinSTT loads this one when the decoder is CPU-bound.

Files

• onnx/encoder_model[_fp16|_int8|_q4|_q4f16].onnx (+ external data) — hoisted, cross-padded encoder.
• onnx/decoder_model_merged[_fp16|_int8|_q4|_q4f16].onnx (+ _dyn) — static / dynamic decoders.
• Tokenizer / configs — unchanged.