Native FP16 ONNX export of Qwen/Qwen3-ASR-0.6B. Exported directly from the FP16 torch model using torch.amp.autocast(dtype=float16) — produces properly typed FP16 graphs that work on all onnxruntime execution providers.

Key Properties

Single model for GPU and CPU — same files work on CUDA EP (native FP16 compute) and CPU EP (auto-promoted to FP32)
2.9GB total — self-contained ONNX files, no external .data files
Identical accuracy on GPU and CPU — CPU EP auto-promotes FP16 to FP32, so results match
52 languages including English, Chinese, Japanese, Korean, and major EU languages

Performance

Provider	RTF	Decode Speed	Hardware
CUDA EP	0.04-0.10	9-33 ms/tok	RTX 4090
CPU EP	0.44-0.62	101-183 ms/tok	Ryzen 5 5600X

Files

File	Size	Description
`encoder.onnx`	361 MB	Mel spectrogram → audio features
`decoder_init.onnx`	1.2 GB	Prefill: input embeds → logits + KV cache
`decoder_step.onnx`	1.2 GB	Autoregressive: token + KV cache → logits + updated KV cache
`embed_tokens.bin`	297 MB	Token embeddings, float16, shape [151936, 1024]
`tokenizer.json`	11 MB	HuggingFace tokenizers format

KV Cache Layout

Stacked tensors (not per-layer):

present_keys: [num_layers=28, batch, kv_heads=8, seq_len, head_dim=128]
present_values: [num_layers=28, batch, kv_heads=8, seq_len, head_dim=128]

Inference Pipeline

Compute log-mel spectrogram (16kHz, 128 bins, Whisper-style)
Encoder: mel [1, 128, T] → audio features [1, T/8, 1024]
Build prompt token IDs (system/user/assistant template with audio_pad placeholders)
Embed tokens via embed_tokens.bin, replace audio_pad positions with encoder features
Decoder init (prefill): prompt embeds + position_ids → logits + KV cache
Decoder step (greedy): loop until EOS (im_end or endoftext)

All I/O tensors are float16. On CPU EP, onnxruntime auto-promotes to float32 internally.

Usage with onnxruntime

python

import numpy as np
import onnxruntime as ort

# Load sessions
encoder = ort.InferenceSession("encoder.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
decoder_init = ort.InferenceSession("decoder_init.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
decoder_step = ort.InferenceSession("decoder_step.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])

# Load embeddings
embed_tokens = np.fromfile("embed_tokens.bin", dtype=np.float16).reshape(151936, 1024)

# Compute mel spectrogram from audio (16kHz, 128 bins)
mel = compute_log_mel_spectrogram(audio)  # [1, 128, T], float16

# Encode
audio_features = encoder.run(None, {"mel": mel})[0]  # [1, T/8, 1024]

# Build prompt and embed (see inference pipeline above)
prompt_embeds = build_and_embed_prompt(audio_features, embed_tokens)  # [1, seq_len, 1024], float16

# Prefill
position_ids = np.arange(seq_len, dtype=np.int64).reshape(1, -1)
logits, present_keys, present_values = decoder_init.run(None, {
    "input_embeds": prompt_embeds,
    "position_ids": position_ids,
})

# Greedy decode
next_token = int(np.argmax(logits[0, -1, :]))
generated = [next_token]
cur_pos = seq_len

while next_token not in (151645, 151643):  # im_end, endoftext
    token_embed = embed_tokens[next_token][np.newaxis, np.newaxis, :]  # [1, 1, 1024]
    logits, present_keys, present_values = decoder_step.run(None, {
        "input_embeds": token_embed,
        "position_ids": np.array([[cur_pos]], dtype=np.int64),
        "past_keys": present_keys,
        "past_values": present_values,
    })
    next_token = int(np.argmax(logits[0, -1, :]))
    generated.append(next_token)
    cur_pos += 1

# Decode tokens
text = tokenizer.decode(generated, skip_special_tokens=True)

Export Method

Exported using a modified version of andrewleech/qwen3-asr-onnx with --dtype float16. The key modification wraps torch.onnx.export with torch.amp.autocast(device, dtype=torch.float16), which keeps LayerNorm in float32 while running matmuls in float16 — producing properly typed ONNX graphs.

Why Native FP16 (not post-hoc conversion)

Post-hoc FP16 conversion via onnxconverter_common or onnxruntime.transformers.float16 produces ONNX graphs with type mismatches between nodes. These work on CPU EP (where FP16 is auto-promoted to FP32 anyway) but produce garbage on CUDA EP. Native export from torch with autocast avoids this entirely.

License

Apache 2.0 (same as the base model).

Credits

Base model: Qwen/Qwen3-ASR-0.6B by Alibaba Qwen team
Export tooling: andrewleech/qwen3-asr-onnx

cvxhull/qwen3-asr-0.6b-onnx-fp16

Qwen3-ASR-0.6B ONNX (Native FP16)