X-ASR zh-en streaming zipformer2 — NTUML2021 encoder fine-tune (DEMONSTRATION)

⚠️ This is a research demonstration, not a production model. It exists to show that the X-ASR streaming zipformer2 transducer can be fine-tuned end-to-end and re-deployed to a Jetson Nano. It is a fine-tune of the base (non-punctuation) X-ASR checkpoint on a small, single-domain set (NTU "Machine Learning 2021" lectures), so its gains are in-domain adaptation — it is not a generally-better zh-TW model and not a drop-in replacement for the deployed X-ASR model. Read the numbers below with the caveats.

What it is

• Base: GilgameshWind/X-ASR-zh-en base / non-punctuation checkpoint (streaming_exp/pretrained.pt, tokenizer lang_5000), a causal streaming zipformer2 transducer (6 stacks / 19 layers, dims 192·256·512·768·512·256, vocab 5000, 16 kHz / 80-d fbank).
• Fine-tune: encoder-only partial FT (147 M trainable; decoder/joiner/embedding frozen), 300 steps, causal/streaming regime, ~40 % retention (English + mainland-zh mixed every batch) on 200 NTUML2021 train clips. Trained on an NVIDIA GB10 (k2 built from source for CUDA 13 / Blackwell).
• Format: sherpa-onnx streaming int8 (encoder.int8 + decoder fp32 + joiner.int8), 960 ms variant (chunk 48 / left 256). Output is Simplified, no punctuation (the base model's space). Apply OpenCC s2twp downstream for Taiwan-Traditional output.

Measured on a real Jetson Nano gen1 (int8 streaming, 2 threads, vs the base)

Honest caveats

• In-domain gain. Train/eval are disjoint NTUML2021 clips but the same ML-lecture domain. This measures domain adaptation, not a generic Taiwan-accent improvement (the project found no generic accent gap vs the Taiwan SOTA — see the repo's docs/PHASE0_RESULTS.md).
• Base, non-punctuation model. Weaker absolute accuracy than the deployed punctuation X-ASR. Use the deployed model for production; use this only to study fine-tuning.
• Mainland-zh retention not cleanly measured (held-out set unavailable at build time).
• Small data (200 clips); not robustness-tested on noisy/far-field audio.

Usage (sherpa-onnx)

License & provenance