ONNX Runtime export of Qwen/Qwen3-ASR-0.6B, with int4 / fp16 quantized weights for CPU / DirectML / CUDA inference without a PyTorch runtime. Produced as part of a from-scratch ONNX + C# inference PoC.

Source & License

This is a derivative work of Qwen/Qwen3-ASR-0.6B, which is licensed under Apache-2.0. This repository is released under the same license; a copy is included in LICENSE.

Changes made to the original model:

Exported the encoder and decoder to ONNX graphs (ONNX Runtime / opset).
Split the autoregressive decoder into decoder_init / decoder_prefill / decoder_step graphs with an external KV-cache weight file.
Quantized weights to int4 (and fp16 encoder variant); embeddings kept as a raw embed_tokens.bin table.
Precomputed mel-spectrogram DSP constants (hann_400.f32, mel_filters.f32).

No model architecture or training was changed; this is a format/precision conversion.

Files

File	Purpose
`encoder.fp16.onnx` / `encoder.int4.onnx(+.data)` / `encoder.fp32.onnx`	Audio encoder (mel → audio features): fp16, REAL int4 (MatMulNBits), and FP32 variants. GPU default fp16; CUDA can use int4 for multi-stream; CPU uses fp32
`decoder_init.int4.onnx`	Decoder graph for the first step (no past KV)
`decoder_prefill.int4.onnx(+.data)`	Prefill graph for the prompt span
`decoder_prefill.lastlog.int4.onnx`	Prefill variant emitting only the last-position logits
`decoder_step.int4.onnx` / `decoder_step.int4.am.onnx`	Single autoregressive step (`.am` = argmax fused)
`decoder_weights.int4.data`	Shared int4 decoder weights (external data)
`decoder_bucket_prefill_0p6b_B1024.int4.onnx(+.data)` / `decoder_bucket_step_0p6b_B1024.int4.onnx(+.data)`	CUDA-graph fixed-bucket (1024) decode graphs — fast single-stream CUDA path (optional)
`embed_tokens.bin`	Token embedding table (fp16, raw little-endian)
`tokenizer.json`, `vocab.json`, `merges.txt`, `added_tokens.json`	Byte-level BPE tokenizer
`config.json`, `preprocessor_config.json`, `tokenizer_config.json`	Model / preprocessor config
`hann_400.f32`, `mel_filters.f32`	Precomputed Hann window & mel filterbank for the front-end

Encoder naming (corrected 2026-06-03): encoder.int4.onnx is now the real int4 encoder (MatMulNBits) and encoder.fp32.onnx is FP32. Earlier revisions mislabeled the FP32 encoder as encoder.int4.onnx and shipped the real int4 as encoder.q4.onnx (now removed). If you cached an older revision, re-download.

Usage

Minimal offline transcription with ONNX Runtime (pip install onnxruntime numpy soundfile tokenizers). This same code works for both the 0.6B and 1.7B repos (hidden size is read from config.json):

python

import json, numpy as np, onnxruntime as ort, soundfile as sf
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

MODEL_DIR = snapshot_download("jinhwan000/qwen3-asr-0.6b-onnx")
WAV = "audio.wav"   # 16 kHz mono

# Qwen3-ASR chat scaffold (special token ids)
IM_START, IM_END, EOT = 151644, 151645, 151643
AUDIO_START, AUDIO_END, AUDIO_PAD, NL = 151669, 151670, 151676, 198
SYSTEM_LINE, USER_LINE, ASSISTANT_LINE = [8948, NL], [872, NL], [77091, NL]
EOS = {{EOT, IM_END}}
HIDDEN = json.load(open(f"{{MODEL_DIR}}/config.json"))["decoder"]["hidden_size"]

def log_mel(audio):  # whisper-compatible log-mel using the shipped DSP constants
    NFFT, HOP, NMELS, NBINS = 400, 160, 128, 201
    hann = np.fromfile(f"{{MODEL_DIR}}/hann_400.f32", np.float32)
    melf = np.fromfile(f"{{MODEL_DIR}}/mel_filters.f32", np.float32).reshape(NMELS, NBINS)
    x = np.pad(audio, NFFT // 2, mode="reflect")
    nfr = 1 + (len(x) - NFFT) // HOP
    frames = np.stack([x[i*HOP:i*HOP+NFFT] * hann for i in range(nfr)])
    spec = np.fft.rfft(frames, n=NFFT, axis=1)
    mel = np.log10(np.maximum((spec.real**2 + spec.imag**2) @ melf.T, 1e-10))
    mel = np.maximum(mel, mel.max() - 8.0)
    mel = (mel + 4.0) / 4.0
    return mel[:-1].T.astype(np.float32)[None]            # [1,128,T] (drop last frame)

audio, sr = sf.read(WAV, dtype="float32")
if audio.ndim > 1: audio = audio.mean(1)
assert sr == 16000, "expects 16 kHz mono"

prov = ["CPUExecutionProvider"]   # GPU: ["CUDAExecutionProvider"] (onnxruntime-gpu) or ["DmlExecutionProvider"]
enc  = ort.InferenceSession(f"{{MODEL_DIR}}/encoder.fp16.onnx",      providers=prov)
init = ort.InferenceSession(f"{{MODEL_DIR}}/decoder_init.int4.onnx", providers=prov)
step = ort.InferenceSession(f"{{MODEL_DIR}}/decoder_step.int4.onnx", providers=prov)
embed = np.fromfile(f"{{MODEL_DIR}}/embed_tokens.bin", np.float16).reshape(-1, HIDDEN)
tok = Tokenizer.from_file(f"{{MODEL_DIR}}/tokenizer.json")

# encode audio -> features, then build the prompt with N audio-pad slots
af = enc.run(["audio_features"], {{"mel": log_mel(audio)}})[0]   # [1, N, hidden]
N = af.shape[1]
prompt = ([IM_START] + SYSTEM_LINE + [IM_END, NL, IM_START] + USER_LINE +
          [AUDIO_START] + [AUDIO_PAD]*N + [AUDIO_END, IM_END, NL, IM_START] + ASSISTANT_LINE)
P = len(prompt)

# prefill
logits, pk, pv = init.run(None, {{
    "input_ids":      np.array([prompt], np.int64),
    "position_ids":   np.arange(P, dtype=np.int64)[None],
    "audio_features": af,
    "audio_offset":   np.array([prompt.index(AUDIO_PAD)], np.int64)}})
nxt, out, pos = int(logits[0, -1].argmax()), [], P
budget = max(16, int(len(audio) / 16000 * 13))

# greedy autoregressive decode with KV cache
while nxt not in EOS and len(out) < budget:
    out.append(nxt)
    logits, pk, pv = step.run(None, {{
        "input_embeds": embed[nxt].astype(np.float32)[None, None],
        "position_ids": np.array([[pos]], np.int64),
        "past_keys": pk, "past_values": pv}})
    nxt = int(logits[0, -1].argmax()); pos += 1

text = tok.decode([t for t in out if t not in EOS])
text = text.split("<asr_text>")[-1].strip()   # strip the "language <lang><asr_text>" prefix
print(text)

Notes

encoder.int4.onnx (smallest, slight accuracy drop) or encoder.fp32.onnx can be swapped for the encoder.
This is a greedy offline loop. The repo also ships decoder_prefill* (system-prompt KV reuse / streaming) and decoder_bucket_* (CUDA-graph fast path) graphs for advanced inference loops.
Word-level timestamps: see the companion forced aligner.

Disclaimer

Experimental PoC artifacts. Accuracy was validated against the reference implementation, but no production guarantees are made. For the canonical model, use Qwen/Qwen3-ASR-0.6B.