phoonnx-omnivoice
ONNX build of
OmniVoice by k2-fsa (Xiaomi
Corp., authors Han Zhu et al.), packaged for
phoonnx.
OmniVoice is a zero-shot text-to-speech model for 600+ languages. It is a masked
diffusion language model: a Qwen3-0.6B backbone writes eight streams of Higgs Audio V2
codec tokens, starting from an all-MASK grid and unmasking the most confident slots over
32 steps. Attention is bidirectional, and every step is a full-sequence forward.
Why this build exists
An earlier community export, onnx-community/OmniVoice-Onnx, runs the backbone through
the ONNX Runtime contrib operator com.microsoft::GroupQueryAttention. That operator is
unconditionally causal. OmniVoice is not a causal model, so the export computes the
wrong hidden states. We measured this against upstream PyTorch on a fixed input:
| Graph | Reference | Result |
|---|
audio_embeddings_encoder (community, fp32) | upstream torch | cos 1.0000000, rel 6e-4 — correct |
audio_heads_decoder (community, fp16) | upstream torch | cos 1.0000000, rel 2e-4 — correct |
llm_decoder (community, int4) | upstream torch, bidirectional | cos 0.954 — wrong |
llm_decoder (community, int4) | upstream torch, causal | cos 0.99945 — matches a causal model |
| community chain, end to end | upstream torch | 18.11 % greedy-token agreement |
The community llm_decoder reproduces a causal OmniVoice, which is a different model.
omnivoice_backbone.onnx here is a fresh export from the PyTorch checkpoint with the
bidirectional mask kept intact:
| Graph | Reference | Result |
|---|
omnivoice_backbone.onnx (fp32) | upstream torch | cos 1.0000000, rel 2.9e-7, 100.00 % greedy-token agreement |
| full sampler, 32 steps, greedy | upstream _generate_iterative | 100.00 % codec-token agreement |
The community Higgs codec graphs are exact and are mirrored here unchanged:
| Graph | Reference | Result |
|---|
acoustic_encoder + semantic_encoder + quantizer_encoder | upstream torch encode | 100.00 % codec-code agreement (all 8 codebooks) |
higgs_decoder | upstream torch decode | max abs diff 0.0 (bit-identical) |
No quantized variant of the backbone is published: the int4 community build was rejected
on the agreement test above, and we have not yet produced a quantized export that passes.
Files
| File | What it is |
|---|
omnivoice_backbone.onnx (+ .onnx_data) | Qwen3 backbone + audio embeddings + audio heads, one graph. (input_ids[B,8,S] int64, audio_mask[B,S] bool) -> logits[B,8,S,1025]. Bidirectional; no KV cache. |
acoustic_encoder.onnx | reference wav @24 kHz (1,1,T) -> acoustic features (1,256,T') |
semantic_encoder.onnx | reference wav @16 kHz (1,T) -> semantic features (1,768,T') |
quantizer_encoder.onnx | acoustic + semantic -> reference codes (8,1,T') |
higgs_decoder.onnx | codes (8,1,T') -> waveform @24 kHz |
tokenizer.json, tokenizer_config.json | the model's own Qwen3 subword BPE |
config.json | names the phoonnx engine and the graph roles |
Classifier-free guidance runs the backbone twice per step — once over the full prompt and
once over the target span alone. Upstream batches both rows behind a [2B,1,S,S] block
mask; for a single item that is the same as two forwards of different lengths.
Usage
1from phoonnx.model_manager import TTSModelManager
2
3voice = TTSModelManager().load_voice("omnivoice/en")
4audio = voice.synthesize(
5 "Machine learning models can now speak in hundreds of languages.",
6 speaker_reference="reference.wav",
7 speaker_reference_text="Transcription of the reference clip.",
8)
speaker_reference_text is not optional in practice: OmniVoice joins the reference
transcription to the target text into a single prompt string, and cloning is noticeably
worse without it.
Verified languages
Word/character error rate on FLEURS, scored with OpenVoiceOS ONNX ASR models and always
reported against a floor — the same ASR run over the real human FLEURS recording.
See the phoonnx pull request for the full table. Only a sample of the model's 600+
claimed languages has been measured; the rest are untested, not verified.
Licensing and attribution
OmniVoice's code is Apache-2.0. Its released weights are CC-BY-NC because of
their training data (Emilia and others), and this repository inherits that: the ONNX
graphs are a format conversion of those weights. Non-commercial use only.
Model and method: Zhu Han, Ye Lingxuan, Kang Wei, Yao Zengwei, Guo Liyong, Kuang Fangjun,
Han Zhifeng, Zhuang Weiji, Lin Long and Povey Daniel — OmniVoice: Towards Omnilingual
Zero-Shot Text-to-Speech with Diffusion Language Models, arXiv 2604.00688, 2026.
The Higgs codec graphs are mirrored from onnx-community/OmniVoice-Onnx.
Do not use this model for voice cloning without the speaker's consent, or for
impersonation or fraud.
1@article{zhu2026omnivoice,
2 title={OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models},
3 author={Zhu, Han and Ye, Lingxuan and Kang, Wei and Yao, Zengwei and Guo, Liyong and
4 Kuang, Fangjun and Han, Zhifeng and Zhuang, Weiji and Lin, Long and Povey, Daniel},
5 journal={arXiv preprint arXiv:2604.00688},
6 year={2026}
7}
