LoRA adapter turning Qwen/Qwen3-ASR-1.7B into a Taiwan-localized speech-to-action agent: it
transcribes zh-TW/English speech and acts on it in the same decoder pass — one model instead of
an ASR + LLM pipeline.
Unusually for a task fine-tune, it is better than its own base model at ASR in both languages
while also being a competent tool-caller.
ASR — 300 Common Voice 17 zh-TW test clips (CER, OpenCC s2tw-normalized) and 200 LibriSpeech
test-clean clips (WER, case-insensitive); identical pipeline for every row:
model
size
zh-TW CER ↓
en WER ↓
this model
1.7B
2.86
1.65
Qwen3-ASR-1.7B (base)
1.7B
3.08
2.25
Qwen3-ASR-0.6B (base)
0.6B
5.17
3.50
Audio8-ASR-0.1B
0.32B
6.24
3.28
whisper-small
0.24B
10.64
3.52
Tool-calling — BFCL v4, 11 non-agentic categories (no java/js), 150 rows/category, scored with
BFCL's official ast_checker. TW-BFCL v4 is a zh-TW localization of the same categories:
model
size
BFCL v4 (en)
TW-BFCL v4 (zh-TW)
this model
1.7B
84.95
72.37
Qwen3-ASR-0.6B-TW-Agent
0.6B
81.53
67.52
LFM2.5-1.2B-Instruct
1.2B
76.43
56.93
Qwen3-0.6B
0.6B
73.89
59.87
LFM2.5-350M
0.35B
65.29
46.02
Training: joint SFT → on-policy distillation → format refresher
Three stages, each repairing what the previous one broke — the intermediate failures are reported
because they are the interesting part:
Joint multi-task SFT (82.7k rows: zh-TW + English ASR, telephone dialogs, ~54k function-calling
examples at 50/50 zh/en with 13% "decline to call" negatives; 33.9% audio share; frozen audio
encoder, LoRA r16 on the decoder). Gives strong tool-calling and the best zh CER (2.65) — but
English ASR collapses to 30.23 WER, catastrophic forgetting from a Chinese-heavy mix.
On-policy distillation (arXiv 2605.28139): the student
transcribes audio itself, the frozen base Qwen3-ASR-1.7B scores the student's own transcript on
the same audio, and the loss is a temperature-scaled KL over the union of teacher/student top-k
tokens. 1500 steps took English 30.23 → 1.65 WER — supervised training on reference
transcripts had failed here regardless of data volume, because it never visits the error states
the model itself produces. Cost: the tool-call format eroded (ASR rollouts always end after one
short block, so the model learned to emit <think> and stop).
Text-FC refresher: 6000 function-calling rows, 188 steps at lr 2e-5. Restored tool-calling
(BFCL 84.95) with English WER unchanged at 1.65.
Caveats
Trained for zh-TW and English; other languages are untested.
Never trained on 8 kHz telephone-channel audio — benchmarks are 16 kHz. Real phone deployments
should be re-validated.
TW-BFCL scores are not comparable to English-BFCL leaderboard numbers: its gold answers are
augmented with zh alternatives, and ~29% of AST rows are flagged _solvable=false (cross-lingual
entity mismatch).
Multi-turn full-trajectory tool use is weaker than single-turn (~58% in earlier in-house testing).
Usage
python
1from qwen_asr.core.transformers_backend.modeling_qwen3_asr import Qwen3ASRForConditionalGeneration
2from qwen_asr.core.transformers_backend.processing_qwen3_asr import Qwen3ASRProcessor
3from peft import PeftModel
4import torch
56base ="Qwen/Qwen3-ASR-1.7B"# NOTE: the nested-config repo, not the -hf variant7proc = Qwen3ASRProcessor.from_pretrained(base)8thinker = Qwen3ASRForConditionalGeneration.from_pretrained(base, dtype=torch.bfloat16).thinker
9model = PeftModel.from_pretrained(thinker,"<this repo>").cuda().eval()10# render system prompt + tools + an <|audio_pad|> user turn with the processor, then generate;11# output is <transcript>...</transcript> followed by a Hermes <tool_call> or a spoken reply.
For text-only use (e.g. BFCL harnesses), merge the adapter into a plain Qwen3ForCausalLM.
