VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text. It addresses significant challenges in traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking.
A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details.
The speech tokenizer is a key component for both VibeVoice TTS and ASR.
ONNX conversion of microsoft/VibeVoice-AcousticTokenizer
— the continuous acoustic speech tokenizer (VAE codec) shared by VibeVoice
TTS and ASR.
It operates at an ultra-low 7.5 Hz frame rate (hop = 3200 samples @ 24 kHz), encoding waveform to a
64-dim latent per frame and decoding it back.
Exported with Olive (dynamo → ONNX). No PyTorch / transformers
needed at runtime — only onnxruntime + numpy (+ soundfile/librosa for audio I/O).
Files
This tokenizer is shared across the VibeVoice family — the same acoustic codec feeds the TTS and
ASR models — so inference_tokenizer.py is the general encode/decode driver, usable standalone or as
the codec front/back-end of any VibeVoice ONNX pipeline.
file
in → out
notes
acoustic_encoder.onnx
audio [1,1,24000] → latents [1,7,64]
fixed 1 s input (24000 samples → 7 frames)
acoustic_decoder.onnx
latents [B,frames,64] → audio [B,1,3200*frames]
dynamic frame count
inference_tokenizer.py
—
encode/decode driver (prints the encoder→latents→decoder flow)
Two precisions: fp32/ (1.3 GB, reference) and fp16/ (≈0.7 GB). int4 is not provided — the
tokenizer is a conv VAE with no MatMul weights to quantize (int4 would be a no-op = fp32).
Encoder length is baked at 24000 samples. The PyTorch model accepts arbitrary length; this ONNX
encoder processes exactly 1 s per call, so longer audio is encoded in 1 s chunks (zero-padded to a
whole number of chunks) and the latents concatenated. The decoder is dynamic — it decodes any
number of frames in one pass. acoustic_inference.py does the chunking for you.
Component parity (ONNX vs VibeVoiceAcousticTokenizerModel) and codec round-trip on a synthetic tone:
encoder
decoder
codec round-trip
fp32
cos 0.999
cos 0.94¹
corr +0.999, SNR +22.8 dB
fp16
cos 0.999
cos 0.94¹
corr +0.998, SNR +22.7 dB
¹ the decoder's cosine is measured on near-silent output (random latents ≈ silence), where max|Δ| ≈
1e-10 — numerically identical, cosine is just noise-dominated (standard neural-codec eval caveat).
Notes
fp16 was exported with an op_block_list (ConstantOfShape/ConvTranspose/Resize/Range)
kept in fp32 — those shape/resample ops produce invalid graphs under fp16 conversion and gain nothing.
Streaming (use_cache / padding_cache in the PyTorch API) is not exported here — these are
fixed-context encode/decode graphs. Streaming would need the cached-state variants exported separately.
Multi-chunk round-trip: a 1 s window reconstructs at corr ~0.999, but the fixed encoder emits
22400 samples per 24000-sample chunk (7 frames × 3200), so naively encoding >1 s in chunks and
decoding the concatenated latents accumulates a small per-chunk time offset that lowers waveform
corr. For downstream use the latents are the product (fed to / produced by the LLM); exact
full-length reconstruction of long audio wants the streaming (padding_cache) path.
The upstream AutoFeatureExtractor just formats a mono 24 kHz waveform (pad_to_multiple_of=3200);
the ONNX encoder takes the raw [1,1,24000] tensor directly, so no feature-extractor is required.
