vits-melo-tts-zh_en-8k — MeloTTS zh_en with a distilled 8 kHz vocoder
Drop-in replacement for the sherpa-onnxvits-melo-tts-zh_en model directory, with the 44.1 kHz HiFi-GAN decoder replaced by a
distilled 8 kHz Vocos-style vocoder (ConvNeXt @ 125 Hz + iSTFT head, 3.65 M params).
Built for a phone-attendant product on Jetson Nano gen1 CPU (4× Cortex-A57, ONNX Runtime
CPU): the audio leaves through an 8 kHz G.711 channel, so rendering 44.1 kHz wasted ≥80 % of
vocoder compute. Voice, prosody, and text front end are unchanged — only the decoder is
new, distilled from MeloTTS's own output (teacher latent z → 8 kHz waveform). Identical
input text produces the same speech, just 8 kHz and cheap.
Results
gate
value
PESQ-NB vs teacher (clean / through G.711 μ-law)
2.90 / 2.89
MCD-DTW (13 MCEP)
10.8 dB
Full-model RTF, x86 4-thread ORT 1.17 CPU
0.013
Predicted Jetson Nano A57 RTF (×6–8 host factor)
0.08–0.10 (was 2–3: unusable)
Decoder share of full-model compute
12.6 % (was 88 %)
fp32, ONNX opset 17, no custom ops — runs on the sherpa-onnx-pinned ORT (~1.17) CPU build.
Do not int8-quantize the convs: measured 9× slower than fp32 on the A57.
Listen — zh/en code-mixed samples
Same text, three renders: the stock teacher at its native 44.1 kHz, the teacher downsampled to
8 kHz (the theoretical best this model could sound), and this model rendered end-to-end by
the stock sherpa-onnx engine. Small duration differences between teacher and student come from
the sherpa C++ frontend vs melo's python frontend (tokenization), not from the vocoder.
1.「您好,這裡是全球科技,幫您轉接給 Kevin 陳經理,請稍候。」
teacher 44.1 kHz (stock)
teacher @ 8 kHz
this model (8 kHz, sherpa)
2.「Amy 林目前正在開會,您的 Wi-Fi 報修單已經建立,編號五三三。」
teacher 44.1 kHz (stock)
teacher @ 8 kHz
this model (8 kHz, sherpa)
3.「請撥打零二、二七一八、三三四四,分機二一八,聯絡 Jason 王工程師,謝謝您的來電。」
teacher 44.1 kHz (stock)
teacher @ 8 kHz
this model (8 kHz, sherpa)
Note: the sherpa frontend drops raw digit tokens as OOV (same as the stock model dir), so digits
in these samples are pre-normalized to Chinese (五三三). The deployed app already normalizes
numbers before TTS; see INTEGRATION.md.
More material in ear_test/ (teacher/student pairs + G.711 μ-law channel renders).
Usage — stock sherpa-onnx, nothing changes but the model dir
bash
1sherpa-onnx-offline-tts \2 --vits-model=vits-melo-tts-zh_en-8k/model.onnx \3 --vits-tokens=vits-melo-tts-zh_en-8k/tokens.txt \4 --vits-lexicon=vits-melo-tts-zh_en-8k/lexicon.txt \5 --vits-dict-dir=<jieba dict dir, same as the stock model>\6 --sid=0 --num-threads=4 --output-filename=out.wav \7"幫您轉接給 Kevin 陳經理,他的分機是 533。"
Output is 8 kHz mono (sample_rate=8000 in the model metadata).
Speaker id: use --sid 0 under sherpa-onnx (it's the Chinese voice)
This is a single-speaker export (n_speakers = 1 in the ONNX metadata), so the only
in-range id is sid = 0, and that lone embedding is the Chinese (ZH) voice. Under stock
sherpa-onnx, just pass --sid 0 (or omit --sid).
Why you may have seen "use sid 1". Upstream MeloTTS defines spk2id = {'ZH': 1}, so in the
original model ZH is speaker 1. When this model was exported as a single-speaker ONNX,
that ZH embedding became the only row — index 0 — and the original id is preserved only
in the speaker_id metadata field. So they are the same voice under two numbering schemes:
context
id for the Chinese voice
original MeloTTS spk2id
1
this single-speaker ONNX (sherpa-onnx)
0 (range is [0, 0])
If you pass --sid 1 to stock sherpa-onnx you'll get the warning
sid should be in the range [0, 0]. Given: 1. Use sid=0; sherpa then clamps to 0, so the audio
is still correct — but --sid 0 is the clean, warning-free call. (Verified: sid=0 and sid=1
produce byte-for-byte the same Chinese audio here.)
⚠️ Other GGUF/runtime ports number speakers differently. Some ports (e.g. RapidSpeech.cpp)
take the speaker from the originalspk2id space rather than the single-row ONNX index. For
those, select the ZH speaker explicitly — in RapidSpeech.cpp use --lang Chinese, which
maps ZH → speaker 1. Picking the wrong/default speaker there feeds the wrong embedding g into
the flow/vocoder and degrades output. Bottom line: sherpa-onnx → sid 0; original-MeloTTS
numbering → speaker 1; both refer to the same single Chinese voice.
Contents
model.onnx melo enc/flow + distilled Vocos8k decoder (127 MB, fp32, opset 17)
tokens.txt identical to the stock vits-melo-tts-zh_en export
lexicon.txt identical to the stock vits-melo-tts-zh_en export
ear_test/ 8 kHz WAVs: teacher vs student, clean + G.711, zh/en code-mixed lines,
plus renders produced by the stock sherpa-onnx engine
DEVICE_ACCEPTANCE.md acceptance checklist: exact CLI, RTF bounds @1/2/4 threads, metrics
INTEGRATION.md install steps + zh-TW (Traditional Chinese) input notes
zh-TW note
Traditional Chinese input works (the frontend reads it directly). The accent is Mainland
Putonghua, not Taiwan Guoyu, and occasional polyphone errors exist — both inherited from the
unchanged MeloTTS front end, identical to the stock 44.1 kHz model. See INTEGRATION.md.
Distillation recipe: dump (z, g, wav) from MeloTTS zh_en with bert=0 (matching the
deployed sherpa export, which never computes BERT), downsample targets to 8 kHz, train a
Vocos-style head (multi-res STFT + telephony-band mel + MPD/MSD GAN losses) on the latent→8 kHz
mapping, then graft the trained decoder back into the melo ONNX graph.
model.onnx is exported at ai.onnx opset 17 (uses LayerNormalization). onnxruntime 1.11.0
— the last version that supports the Jetson Nano gen1's CUDA 10.2 — caps at opset 16 and cannot
load it (and onnx.version_converter can't downgrade LayerNormalization).
model.opset16.onnx is the same network with all 72 LayerNormalization nodes decomposed
into opset-16 primitives (ReduceMean / Sub / Mul / Add / Sqrt / Div) and stamped at opset 16.
It is numerically identical to model.onnx (max abs diff 9.2e-6, corr 1.000000) and loads +
runs on onnxruntime 1.11.0 (validated on a cuDNN-free CUDA-10.2 build, ~45 ms / ~1 GB). Use it
when you need to run melo8k on onnxruntime ≤ 1.11 (e.g. Jetson Nano gen1); use model.onnx on
modern onnxruntime.
