OmniVoice is not a vision-language model. It is a TTS model: a Qwen3-0.6B backbone driving an 8-codebook audio codec (Higgs Audio V2 Tokenizer) through a 32-step non-autoregressive unmasking loop. All conversion configuration is defined in Python (optimize.py) — there are no external Olive JSON files.

Architecture

Text ─▶ [audio_embeddings_encoder]  (fuses text + audio-codec token embeds; +ref codes for cloning)
              │ inputs_embeds (B,S,1024)
              ▼
        [llm_decoder]               Qwen3-0.6B, 28 layers  (exclude_embeds + exclude_lm_head)
              │ hidden_states (B,S,1024)
              ▼
        [audio_heads_decoder]       Linear 1024 → 8×1025
              │ logits (B,8,S,1025)
              ▼
        32-step iterative unmasking → audio_codes (8,T)
              │
              ▼
        [higgs_decoder]             Higgs Audio V2 Tokenizer → waveform @ 24 kHz

Sub-model	In → Out	Build
`audio_embeddings_encoder`	ids `(B,8,S)` + mask `(B,S)` → embeds `(B,S,1024)`	Olive: PyTorch→ONNX→(int4/fp16)
`llm_decoder`	embeds `(B,S,1024)` → hidden `(B,S,1024)`	genai `create_model` (`exclude_embeds`+`exclude_lm_head`)
`audio_heads_decoder`	hidden `(B,S,1024)` → logits `(B,8,S,1025)`	Olive: PyTorch→ONNX→(int4/fp16)
Higgs tokenizer (×4)	waveform ↔ codec codes	Olive: PyTorch→ONNX→(fp16/fp32)

Key config: hidden_size=1024, num_codebooks=8, audio_vocab_size=1025 (1024 codes + mask id 1024), codebook weights [8,8,6,6,4,4,2,2], 32 decoding steps, 24 kHz output.

Why the LLM bypasses Olive

onnxruntime-genai's valid precision × execution-provider combos are FP32/INT4 on CPU and FP16/INT4 on CUDA — there is no FP16 LLM on CPU. Olive's ModelBuilder pass therefore can't emit a CPU fp16 LLM, so optimize.py calls genai create_model directly and builds the LLM int4 on CPU (fp16 on GPU). The audio sub-models still go through Olive. The genai LLM declares only inputs_embeds + attention_mask (+ KV cache) and computes positions internally — no position_ids input.

Precision profiles

`--device`	audio sub-models	LLM	output dir
`cpu`	int4 (block-wise RTN, block 128)	int4	`<output>`
`cpu_fp16`	fp16	int4 (no fp16-CPU in genai)	`<output>`
`gpu`	fp16	fp16 (CUDA EP)	`<output>`

Higgs is exported fp16 (default) or fp32 via --higgs-precision — never int4 (too lossy for the DAC codec). It is precision-shared: an int4 backbone still uses the fp16/fp32 audio_tokenizer/.

Built sizes (verified)

file	fp16 (`cpu_fp16`)	int4 (`cpu`)
`audio_embeddings_encoder.onnx(.data)`	327 MB	87 MB
`audio_heads_decoder.onnx`	16.8 MB	4.5 MB
`llm_decoder.onnx(.data)`	296 MB (int4)	296 MB (int4)
Higgs `audio_tokenizer/` (fp16)	~370 MB	(shared)

Prerequisites

bash

pip install -r requirements.txt          # olive-ai, onnxruntime, onnxruntime-genai, transformers 5.x
pip install omnivoice                     # registers the base OmniVoice architecture for loading

omnivoice is only needed at build time (to load the source checkpoint); inference needs only onnxruntime + numpy + soundfile. In this repo's env the build is run isolated as uv run --with omnivoice python optimize.py … to keep the base environment clean.

Build

bash

# CPU int4 (all sub-models int4)
python optimize.py --device cpu      --output onnx/int4 --model ./model/

# CPU fp16 (audio fp16 + LLM int4)
python optimize.py --device cpu_fp16 --output onnx      --model ./model/

# CUDA fp16 (audio fp16 + LLM fp16)   — needs an NVIDIA GPU + onnxruntime-gpu
python optimize.py --device gpu      --output onnx/cuda --model ./model/

# Higgs tokenizer only (fp16 or fp32) → <output>/audio_tokenizer
python optimize.py --higgs-only --higgs-precision fp16 --output onnx --model ./model/

# Backbone + Higgs together
python optimize.py --device cpu_fp16 --include-higgs --output onnx --model ./model/

optimize.py (1) saves OmniVoice's internal Qwen3 as a standalone Qwen3ForCausalLM dir, (2) builds the two audio sub-models via inline Olive configs and the LLM via genai create_model, (3) copies the tokenizer + chat_template.jinja, and (4) writes omnivoice_manifest.json. All model_config.json paths are relativized to bare basenames so the folder is portable.

Inference (CPU)

bash

# fp16 backbone
python inference.py --model_dir onnx      --higgs_dir onnx/audio_tokenizer \
    --text "Hello from OmniVoice." --output out.wav

# int4 backbone (shares the same Higgs tokenizer — int4 dir has no audio_tokenizer/ of its own)
python inference.py --model_dir onnx/int4 --higgs_dir onnx/audio_tokenizer \
    --text "Hello from OmniVoice." --output out_int4.wav

--num_audio_tokens sets output length in frames (≈ frames × 0.04 s); --num_steps the unmasking steps (the loop fills ⌈remaining/remaining_steps⌉ frames per step so every frame is decoded). Voice cloning: add --ref_audio ref.wav --ref_text "…" (uses the Higgs encoders).

Standalone codec round-trip:

python higgs_inference.py --models-dir onnx/audio_tokenizer --input in.wav --output rt.wav

Eval

bash

# RTF benchmark (onnxruntime only)
python eval.py --mode rtf   --model_dir onnx      --higgs_dir onnx/audio_tokenizer
python eval.py --mode rtf   --model_dir onnx/int4 --higgs_dir onnx/audio_tokenizer

# Numerical equivalence vs PyTorch (needs `omnivoice` + the source model)
python eval.py --mode equiv --model_dir onnx      --higgs_dir onnx/audio_tokenizer

Verified (CPU, this repo)

build	inference	eval RTF
fp16 (`onnx/`)	✅ WAV @ 24 kHz	0.483 (faster than real-time)
int4 (`onnx/int4`)	✅ WAV @ 24 kHz	0.592 (faster than real-time)
Higgs round-trip	✅ audio→codes(8×N)→audio	—
cuda (`onnx/cuda`)	not tested (no NVIDIA GPU available)	—

int4 is slower than fp16 on CPU here (int4 weight dequant overhead without an accelerated kernel); both are comfortably real-time.

Notes

No external JSON — every Olive config is built in Python in optimize.py; the old cpu_and_mobile/, cpu_fp16/, cuda/, higgs/ *.json files are obsolete.
Tokenizer is the standard Qwen2 fast tokenizer (tokenizer.json) — loaded from the local dir, no trust_remote_code / omnivoice needed at inference.
KV cache is passed empty each step ((B, kv_heads, 0, head_dim)) — full-sequence forward, no cache reuse.
Higgs uses the transformers-native higgs_audio_v2_tokenizer (transformers ≥ 5.4) — no boson-multimodal dependency.

onnx-community/OmniVoice-Onnx

OmniVoice → ONNX

Architecture

Why the LLM bypasses Olive

Precision profiles

Built sizes (verified)

Prerequisites

Build

Inference (CPU)

Eval

Verified (CPU, this repo)

Notes

onnx-community/
OmniVoice-Onnx