Qwen3-TTS-12Hz-1.7B-CustomVoice — MLX INT4 LLM + FP16 vocoder
Private RumiLabs MLX build of
Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice for on-device Apple Silicon inference. This is the
shipping target for the Yobi/rumi TTS pipeline, superseding
RumiLabs/Qwen3-TTS-12Hz-1.7B-VoiceDesign-MLX-4bit.
Why CustomVoice, not VoiceDesign
VoiceDesign treats instruct as a whole-voice description (identity + emotion mixed). Two different emotion prompts produce two different speakers — which we verified empirically: 5 emotion templates → 4 different female voices + 1 male, none anchored to a reference. The --voice parameter is a silent no-op on VoiceDesign and --ref_audio cloning is too weak to anchor identity.
CustomVoice decouples them: voice = preset speaker (9 timbres), instruct = emotion/style only. Listening A/B confirmed: same speaker (Aiden) across all 5 emotions, quality matches VoiceDesign, emotions clearly distinct. One-line code change, no training.
See qwen3_tts.py:1071-1078 for the variant API differences.
What's in this bundle
- LLM:
int4 group-64 affine via mlx-audio convert.
speech_tokenizer/ (RVQ codec + Code2Wav): cast from FP32 → FP16 (682 MB → 341 MB). FP16 retains full audible fidelity (40–61 dB SNR on 5-emotion A/B).- Bundle on disk: 1.80 GiB
- Peak resident on M3 Ultra: 5.24 GiB (+240 MB vs VoiceDesign — CustomVoice's preset-speaker tables add weight).
- 9 preset speakers included (English: Ryan, Aiden; Chinese, Japanese, Korean covered too).
Size breakdown (binary GiB/MiB)
| Component | Upstream BF16 1 | mlx-community/-8bit 2 | This bundle |
|---|
Qwen3 LLM (model.safetensors) | 3.57 GiB (BF16) | ~1.9 GiB (INT8) | 1.51 GiB (INT4-g64) |
speech_tokenizer/ — RVQ codec + Code2Wav | 651 MiB (FP32) | 651 MiB (FP32) | 325 MiB (FP16) |
Text tokenizer (vocab.json + merges.txt + configs) | 4.25 MiB | 4.25 MiB | 4.25 MiB |
| Total | ~4.21 GiB | ~2.55 GiB | 1.80 GiB |
(SI-decimal totals — what HF Hub displays: 4.52 GB / 2.74 GB / 1.93 GB.)
Recipe
1# 1. LLM: bf16 -> int4 g64
2python -m mlx_audio.convert \
3 --hf-path Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
4 --mlx-path ./bundle \
5 -q --q-bits 4 --q-group-size 64 --model-domain tts
6
7# 2. Vocoder: fp32 -> fp16 in place
8python -c "
9from safetensors.numpy import load_file, save_file
10import numpy as np
11p = 'bundle/speech_tokenizer/model.safetensors'
12sd = load_file(p)
13save_file({k: v.astype(np.float16) if v.dtype == np.float32 else v for k,v in sd.items()}, p)
14"
Usage
1from mlx_audio.tts.generate import generate_audio
2
3generate_audio(
4 model="RumiLabs/Qwen3-TTS-12Hz-1.7B-CustomVoice-MLX-4bit",
5 text="I just heard the most amazing news.",
6 voice="Aiden", # English male preset
7 instruct="speaking with bright, lively energy at a faster pace.",
8 temperature=0.3, # critical — temp=0 truncates 2/5 emotions
9 output_path=".", file_prefix="out", audio_format="wav",
10)
English speakers: Aiden (Sunny American male, clear midrange) — current default. Ryan (Dynamic male, strong rhythmic drive) — alternative.
temperature matters. At temperature=0.0 (greedy), the LLM hits EOS prematurely on some emotion templates, producing audio truncated to "I just..." with seconds of silence padding. temperature=0.3 resolves this without quality cost.
What does NOT work (negative results worth documenting)
We attempted further INT4-g64 quantization of the non-LLM weights (the 53% of model.safetensors left at BF16 by mlx-audio's model_quant_predicate). All failed:
| Optimization | Predicted disk savings | Actual result |
|---|
Quantize talker.model.text_embedding (622 MB → 78 MB) | ~544 MB | Audio truncates to ~250 ms on 3/5 emotions |
Quantize 15 × talker.code_predictor.model.codec_embedding.* | ~110 MB | Same failure mode |
Drop speech_tokenizer.encoder.* (113 MB, allegedly only used for ref_audio clone) | ~113 MB | Total silence — encoder feeds the preset-speaker path too |
The mlx-audio model_quant_predicate skip list in qwen3_tts.py:241-249 is load-bearing, not conservative defaults. Whoever wrote that predicate already ran this experiment. The 1.80 GiB / 5.24 GiB-peak bundle is at the floor for this codepath.
Remaining real levers: streaming Code2Wav inference (multi-day engineering, 500 MB–1 GB peak RAM savings, no disk impact), or vocoder retrain at lower precision (multi-week).
MLX-Swift port — runs natively on iPhone (2026-06-04)
The complete stack in this repo (1.7B talker, 16-codebook code predictor, FP16
Code2Wav vocoder) is ported to MLX-Swift and validated stage-by-stage against the
Python-MLX reference (mlx_audio qwen3_tts):
- Talker: teacher-forced parity at the measured bf16 near-tie floor (c0 agreement
0.98–1.00); end-to-end acceptance — Swift codec streams → reference vocoder →
whisper-large-v3-turbo WER 0.000.
- Vocoder: SNR 55–58 dB vs reference wavs (the PCM16 floor of the goldens), exact
sample counts, whisper WER 0.000 across 5 emotion instructs.
Sources:
swift/tts/
in the rumi-cerberus release repo (TTSTalker / TTSSession / TTSVocoder), mirrored
below under
swift/ for self-containment.
Measured on iPhone 16 Pro Max (rumi-cerberus ↔ TTS sequential swap)
| stage | measured |
|---|
| rumi-cerberus teardown | 0.12 s (2.27 GB → 55 MB) |
| TTS load (fully materialized) | 1.17 s (→ 1.93 GB) |
| synthesis (~5 s utterances, 3 emotion instructs) | RTF 0.80–0.86× (debug build) |
| TTS teardown / rumi-cerberus restore | 0.04 s / 3.50 s |
| peak footprint, entire swap–synth–restore flow | 2.27 GB |
All on-device-synthesized utterances transcribe exactly (whisper-large-v3-turbo).
Deployment note: decode the vocoder in chunks (15 frames + 5 context — see
TTSVocoder.decodeChunked); the single-shot graph's transient working set is
jetsam-fatal on iOS even though steady-state memory fits.
Sampling: voice="Aiden", temp=0.3 (temp 0 occasionally truncates on sad-style
instructs), top_k 50, repetition_penalty 1.05.
Streaming voice-out update (2026-06-06)
The production loop now streams: codec frames vocode in 15-frame chunks onto an
AVAudioPlayerNode while the talker keeps generating, and the LLM restores
during the playback tail. Measured on the same device:
| metric | whole-reply (above) | streaming |
|---|
| first audio after reply-ready (~8 s utterance) | ≈ 9.3 s | ≈ 4.1–4.4 s (0.07 s swap + 1.19 s load + 2.8–3.0 s synthesis lead-in) |
| next-turn readiness | +3.5 s restore | warm (restore hidden in playback) |
Generation is slower than real-time (audio/compute RTF 0.73–0.87, thermal-dependent),
so the player is fed through a closed-loop prebuffer (EMA of measured RTF ×
token-estimated reply length); if a very long reply outruns the buffer, playback
takes a single pause-and-rebuffer (node-clock detected) rather than accumulating
stutters. Hook: generateSampled(..., onFrame:) in swift/TTSSession.swift.
Release and debug builds measure identically (decode 17.1–17.3 tok/s, RTF
0.78–0.84 both) — the compute is in Metal library kernels.
License
Apache-2.0, inherited from Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice.
