A streaming/causal drop-in audio tower for Qwen/Qwen3-ASR-0.6B:
the pretrained (offline, fully bidirectional) audio encoder, fine-tuned to run
append-only: each ~2 s audio block is encoded exactly once with a causal
KV cache and a bounded 15 s attention window, and is never re-encoded. New
audio costs one encoder pass over the new block plus an incremental decoder
update; per-chunk compute is constant in stream length, memory is bounded, and
streams can run indefinitely.
It is built for the minimum-compute-per-chunk regime: many concurrent
streams, energy-constrained or on-device serving, and sessions of unbounded
length. Per audio second it spends about 3x less than the re-compute
streaming backend, at constant cost as the stream grows, in exchange for
some accuracy (see the results table for the honest comparison).
Offline Qwen3-ASR vs this release's causal streaming execution
This repository contains only the fine-tuned audio tower (746 MB fp32
safetensors). The decoder, adapter and feature extractor are loaded unchanged
from the base model at runtime.
Results
Long-form: 21 full MCIF/ACL conference talks (5 to 7 minutes each, 2 h
total, accented scientific English), human references, Whisper text
normalization:
system
scoring contract
WER
compute per second of audio
Offline Qwen3-ASR, VAD-segmented
full file available, no latency
7.6
offline
Streaming Qwen3-ASR (windowed)
settled transcript
8.4
126 GFLOPs avg, growing to 172 within each segment
Streaming Qwen3-ASR (windowed)
live, no rewrite, 250 ms tail cut + EOS flush
12.6
126 GFLOPs avg, growing to 172
Streaming Qwen3-ASR causal
settled transcript
17.2
42 GFLOPs, constant
Streaming Qwen3-ASR causal
live, no rewrite (shipped default policy)
17.6
42 GFLOPs, constant
Short-form through the same streaming stack: LibriSpeech test-clean 3.73 /
test-other 7.34 at the shipped low-latency default.
Streaming RTF on Metal, H100 and A100
Streaming WER contract
The WER numbers above use the same real-streaming replay: no right to rewrite
the past, a 250 ms live-tail cut at every non-final update, and an
end-of-stream flush. When forced-aligner word_alignments are present, the 250
ms cut uses those word timestamps; otherwise it falls back to a uniform text
approximation.
Commit latency (measured)
Per-committed-word latency (word spoken to word committed, audio fed at
1.0x), replayed over the 21-talk corpus:
policy
p50
p95
live WER
hold 6 words / 1 stable iteration (default)
4.1 s
7.9 s
17.6
hold 2 words (hold_back_words=2)
2.2 s
7.9 s
17.8
The 2.2 s point sits at the 1.92 s block-cadence floor. The unstable tail
is additionally visible live (~2 s behind the audio head).
How it works
Decoder-side serving loop: what one update actually forwards
Block-bidirectional causal execution: full attention within each fixed
block (192 mel frames = 1.92 s; a 96-frame variant is also in-regime),
causal per-layer KV across blocks, sliding 15 s left window, sinusoidal
positions continuing monotonically. Latency = one block.
Trained by self-distillation: the causal student matches the frozen
offline tower's output embeddings (MSE + 0.5·cosine), audio only, no
labels, with mixed block sizes (96/192) and position-offset augmentation
(log-uniform up to 6000 steps ≈ 2 h) so positions extrapolate far beyond
the 120 s table. Base phase: LibriSpeech 960 h (~6 H100-hours). This
release adds a diverse-audio phase (People's Speech + LibriSpeech
replay, ~1k h, 20k steps) that took long-form WER from 18.1 to 17.2:
spontaneous/accented audio is the measured quality lever.
Segmented long-form serving: segments roll at sentence punctuation
(min ~12 s, 16 s cap) and the encoder chain resets per segment, which bounds
drift; cutting at linguistic boundaries measured better than fixed-length
cuts (18.1 vs 18.7 WER).
The serving stack adds a rolling decoder KV (the [prompt + audio] prefix
persists across updates) and lossless speculative re-decoding (the previous
hypothesis is verified in one parallel pass), making the decoder side
incremental too.
The tower downloads automatically; the base model comes from
Qwen/Qwen3-ASR-0.6B. Works on CUDA, Apple Silicon (MPS) and CPU. See
WhisperLiveKit for the
WebSocket server, web UI and the OpenAI-compatible REST endpoint.
Standalone (no server), via the runtime package
qwen3-asr-causal:
bash
1pip install"qwen3-asr-causal[streaming]"2qwen3-asr-causal transcribe audio.wav --backend hf --language en
python
1from qwen3_asr_causal import Qwen3StreamingASR
23asr = Qwen3StreamingASR(4 lan="en",5 qwen3_streaming_audio_backend="causal",6 qwen3_streaming_tower_checkpoint="qfuxa/qwen3-asr-0.6b-streaming",7)8streamer = asr.build_streamer("en")9# feed mel chunks of any size; the encoder consumes fixed 1.92 s blocks
Limitations
English only. The LibriSpeech distillation degraded other languages
severely (FLEURS vs the windowed backend: fr 7.8 to 37.9 WER, de 12.6 to
49.8 WER, zh 11.4 to 85.7 CER). Use the windowed backend for non-English.
The recipe is language-agnostic and cheap; multilingual versions can
follow the same ~6 GPU-hour path.
Validated for the 0.6B base only (the tower must match the base model).
Structural latency is one block (~1.9 s; 0.96 s with block_frames=96 at a
small quality cost).
Long-form robustness relies on segment resets at sentence boundaries; the
encoder chain is bounded to one segment (~12-16 s) by design.
Accuracy is below standard streaming Qwen3-ASR under the same no-rewrite
contract (17.6 vs 12.6 WER long-form live): you are trading accuracy for
~constant per-chunk compute.
