We introduce a new discretized audio tokenizer that runs at just 25 frames per second while keeping—or even improving—audio quality compared to tokenizers with twice the bitrate. Our model is the first to train on 24 kHz data covering speech, music, and sound events in one unified system. It also uses a simple non-diffusion encoder/decoder for fast, batch inference.
Architecture diagram of the Higgs Audio Tokenizer
Usage
Transformers 🤗
Higgs Audio V2 Tokenizer is supported natively in transformers: see the doc.
An audio signal sampled at \(f_s\) Hz is first split into frames by an encoder with hop size \(M\), giving a frame rate \(f_r = \frac{f_s}{M}\quad\text{(frames/s)}.\)
Two common quantizers are:
Residual Vector Quantization (RVQ): \(N_q\) cascaded vector‑quantizer layers, each with codebook size \(N_{cb}\). When \(N_{q}=1\), it degenerates to ordinary vector quantization.
Finite Scalar Quantization (FSQ): A single-layer scalar quantizer in which
every scalar coefficient is independently mapped to one of \(N_{cb}\) discrete levels.
If every combination of codewords is a token, the vocabulary size is \(N_{cb}^{N_q}\), and each token needs \(N_q\log_2 N_{cb}\) bits. The overall bitrate (bits/s, BPS) is simply \(f_r \times N_q \log_2 N_{cb}\).
We aim to push this bitrate as low as possible without hurting audio fidelity.
What Makes Ours Better
Low Frame Rate: Runs at just 25 fps, halving the frame rate of many baselines while preserving high audio quality.
Unified 24 kHz Training: A single model jointly trained on speech, music, and sound‑event data, capturing both semantic and acoustic nuances and greatly simplifying downstream audio‑language‑model training.
Fast Inference: A non‑diffusion encoder/decoder that processes batches quickly, making it practical for real-time or large-scale tasks.
Evaluation Data and Metrics
We test on four subsets:
Speech, Music and Sound Event: Include 1,000 clips per category, with each clip lasting 10 seconds. Clips are randomly sampled from DAPS (Speech), MUSDB (Music), and AudioSet (Sound Event).
Audiophile: Contains 150 clips, each 30 seconds long, curated from eleven high-fidelity test discs that were designed for perceptual listening tests. The clips feature both high-quality music and sound events.
We measure:
Acoustic Quality: Acoustic reconstruction error between the original and reconstructed audio.
Semantic Integrity: Degree of semantic preservation, evaluated on the English and Chinese subsets of SeedTTS[15].
We compare our tokenizer with a wide range of baselines, from tokenizers mainly built for better acoustic reconstruction and compression rate, to those focused on semantic integrity, and to tokenizers used in existing large audio language models. We also compare with tokenizers that are pretrained specifically on speech or on music.
The tables below summarize the tokenizers evaluated. As shown, our tokenizer achieves a well-rounded balance of efficiency, semantic fidelity, and acoustic quality.
Acoustic Evaluation
This table reports the Short‑Time Fourier Transform (STFT) distance between the original and reconstructed audio. Baselines are listed chronologically and grouped by whether semantic distillation (SD) is applied. Despite DAC’s top acoustic quality at 12× the bitrate, our tokenizer leads all other baselines.
* Bits-per-second is calculated according to the checkpoint the author provided.
** CosyVoice 2 uses the continuous feature as the conditioning; we include it for completeness.
Semantic Evaluation
SeedTTS is a dataset that includes prompt/target audio and texts. We reconstruct the target audio, and use the word error rate (WER) and speaker similarity (SIM) metrics to evaluate the semantic integrity. SIM is calculated by the similarity between the prompt audio and reconstructed target audio with WavLM-large as the embedding model.
The following table compares our tokenizer with semantic-distillation-trained baselines and shows that it delivers performance comparable to tokenizers operating at 2.2× our model’s bitrate.
This model-based evaluation[8] further demonstrates the superiority of our tokenizer. CU denotes the Content Usefulness and CE denotes the Content Enjoyment; both are rated on a 1-10 scale. Notably, our tokenizer performs best on the Audiophile set, demonstrating a clear advantage when the original audio quality is high.
Note that since some tokenizers are trained on 16 kHz data, we upsample their audio outputs to 24 kHz before computing metrics. Different upsampling methods may cause slight variations (e.g., 4.36 vs. 4.43 for XCodec Sound Event CE). We report the best results we could obtain and highlight any results within 0.05 of the best one.
