Pre-exported ExecuTorch artifacts for
Voxtral-4B-TTS-2603
with CUDA backend (NVIDIA GPU), bf16 precision, and 4-bit
weight-only quantization (tile_packed_to_4d packing for
_weight_int4pack_mm) on the LM decoder + flow head linears. The codec
decoder is exported in fp32 via the same CUDA backend using a
conv-as-matmul reformulation so it lowers onto Triton's batched-matmul
kernels.
Overview
The pipeline has two stages: export (Python, once) and inference
(C++ runner, repeated). This repo ships the export outputs so you can skip
straight to inference.
The model has three components:
Mistral 4B LLM decoder — autoregressive text → hidden states
Flow Matching Head (3-layer transformer) — hidden states → 37 audio
codebook tokens per frame via a 7-step Euler ODE
ExecuTorch's CUDA backend uses AOTInductor, which bakes pre-compiled
cubins for the export-time GPU's compute capability into *.ptd. The
cubins are not forward/backward compatible across architectures — running an
sm_80 blob on a Blackwell card fails with CUDA driver error: invalid argument on the first kernel launch.
End-to-end on seed=42, after the per-process Triton autotune cache is
warm (first call is ~30–50 s slower; the runner's warmup() amortizes it).
GPU
Folder
Prompt
Audio
Wall
RTF
A100 80 GB
sm80/
7 tok ("Hello, how are you today?")
2.56 s
3.7 s
0.88x
RTX 5080 16 GB
sm120/
7 tok ("Hello, how are you today?")
2.48 s
3.5 s
1.29x
RTX 5080 16 GB
sm120/ (--streaming)
24 tok
10.32 s
5.6 s
0.55x
RTX 5080 16 GB
sm120/ (--streaming, warm)
24 tok
10.32 s
3.85 s
0.37x
Numerical parity vs the XNNPACK FP32 baseline (measured on the A100 export):
Last-position prefill hidden cosine: 0.999994
First-frame semantic argmax + top-5: identical
Artifacts target Linux NVIDIA GPUs.
Prerequisites
Linux with NVIDIA GPU + CUDA 12.8 or 12.9 toolkit (CUDA 13 not supported —
ExecuTorch's backends/cuda/runtime/shims/sort.cu was written against CUB 2.x).
Conda + Python 3.10–3.12.
ExecuTorch built from source with the CUDA backend (see Build).
Tokenizer + at least one voice embedding from the upstream Mistral repo
(~33 MB total). They are not included in this repo.
Install + Build
bash
1git clone https://github.com/pytorch/executorch/ ~/executorch
2cd ~/executorch
34unset CPATH # avoids CUDA-13 host header pollution5./install_executorch.sh
6pip install -e . --no-build-isolation # editable so source edits take effect78# Build runner with CUDA enabled9exportLD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH10make voxtral_tts-cuda
Binary lands at cmake-out/examples/models/voxtral_tts/voxtral_tts_runner.
WSL2 only: the linker also needs libcuda.so from the driver dir:
1pip install huggingface_hub
23# 1. Pre-exported ExecuTorch artifacts for your GPU arch (~3.7 GB).4# Replace `sm80` with `sm120` for Blackwell (RTX 5080/5090).5hf download younghan-meta/Voxtral-4B-TTS-2603-ExecuTorch-CUDA \6 --include 'sm80/*'\7 --local-dir voxtral_tts_cuda
89# 2. Tokenizer + voice embeddings from the base model (~33 MB)10hf download mistralai/Voxtral-4B-TTS-2603 \11 tekken.json voice_embedding/* \12 --local-dir voxtral_tts_base
Run
bash
1unset CPATH
2exportLD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH34# Pick your arch5ARCH=sm80 # or sm12067cmake-out/examples/models/voxtral_tts/voxtral_tts_runner \8 --model voxtral_tts_cuda/$ARCH/model.pte \9 --data_path voxtral_tts_cuda/$ARCH/aoti_cuda_blob.ptd \10 --codec voxtral_tts_cuda/$ARCH/codec_decoder.pte \11 --codec_data_path voxtral_tts_cuda/$ARCH/codec_aoti_cuda_blob.ptd \12 --tokenizer voxtral_tts_base/tekken.json \13 --voice voxtral_tts_base/voice_embedding/neutral_female.pt \14 --text "Hello, how are you today?"\15 --output output.wav \16 --seed 42\17 --max_new_tokens 200
Output is 24 kHz mono 16-bit PCM. Listen with ffplay output.wav or
aplay output.wav.
Streaming mode
Add --streaming to emit codec output in chunks instead of one batch at the
end. The first chunk is always 9 600 samples (0.4 s) for low time-to-first-
audio; subsequent chunks are 48 000 samples (2 s) until END_AUDIO. Pair
with --speaker to pipe raw f32le PCM to stdout for live playback:
bash
1cmake-out/examples/models/voxtral_tts/voxtral_tts_runner \2 --model voxtral_tts_cuda/$ARCH/model.pte \3 --data_path voxtral_tts_cuda/$ARCH/aoti_cuda_blob.ptd \4 --codec voxtral_tts_cuda/$ARCH/codec_decoder.pte \5 --codec_data_path voxtral_tts_cuda/$ARCH/codec_aoti_cuda_blob.ptd \6 --tokenizer voxtral_tts_base/tekken.json \7 --voice voxtral_tts_base/voice_embedding/neutral_female.pt \8 --text "The quick brown fox jumps over the lazy dog."\9 --seed 42\10 --streaming \11 --speaker \12| ffplay -f f32le -ar 24000 -ac 1 -nodisp -autoexit -
On RTX 5080, time-to-first-audio is ~2.6 s on a warm cache (most of which
is the LM prefill); subsequent chunks arrive every ~0.5 s, comfortably
ahead of playback.
Available voices
neutral_female, neutral_male, casual_female, casual_male,
cheerful_female, ar_male, de_female, de_male, es_female,
es_male, fr_female, fr_male, hi_female, hi_male, it_female,
it_male, nl_female, nl_male, pt_female, pt_male — under
voice_embedding/
in the base-model repo.
Runner options
Flag
Default
Description
--model
(required)
Path to exported model.pte (LM + flow head)
--data_path
(none)
Path to LM .ptd (required for CUDA)
--codec
codec_decoder.pte
Path to exported codec .pte
--codec_data_path
(none)
Path to codec .ptd (required for CUDA)
--tokenizer
tekken.json
Path to tokenizer JSON
--voice
(required)
Path to voice embedding .pt
--text
(required)
Prompt text
--output
output.wav
Output WAV path (ignored if --speaker)
--seed
42
RNG seed (semantic sampling + flow noise)
--temperature
0.0
Sampling temperature (0 = greedy)
--max_new_tokens
2048
Max audio frames (~12.5 frames/s)
--streaming
off
Chunked codec emission
--speaker
off
Pipe raw f32le PCM to stdout instead of writing WAV
Re-exporting for another GPU
If your GPU's compute capability isn't shipped above, run the export on the
target GPU (the AOTI compile step writes cubins for the local arch):
--dtype is auto-promoted to bf16 and --qlinear-packing-format is
auto-set to tile_packed_to_4d when --backend cuda --qlinear 4w is
selected.
Export needs ~10 GB GPU VRAM and the full base-model checkpoint
(consolidated.safetensors, ~8 GB). On RTX 5080 16 GB it takes ~6 minutes.
Triton autotuning will reject some configs that exceed Blackwell's 99 KB
shared-memory limit — those OutOfResources warnings are expected.
tekken.json and voice_embedding/*.pt are not in this repo — download
them from
mistralai/Voxtral-4B-TTS-2603
so they always match the upstream release that this export was produced from.
Troubleshooting
CUDA driver error: invalid argument on first kernel launch. The
AOTI cubins don't match your GPU's compute capability. Pick the right
subfolder for your GPU, or re-export.
__cudaLaunch was not declared during build.CPATH is polluted
with CUDA 13's include path. unset CPATH and rebuild.
GLIBCXX_3.4.30 not found at runner startup. AOTI .so files
require a newer libstdc++ than /lib64/libstdc++.so.6. Set
LD_LIBRARY_PATH=$CONDA_PREFIX/lib before launching.
cannot find -lcuda during pip install -e . or export (WSL2). The
CUDA toolkit doesn't ship libcuda.so; on WSL2 the driver lib lives at
/usr/lib/wsl/lib/. Prepend it (or /usr/local/cuda/lib64/stubs) to
LIBRARY_PATH.
First call takes ~30–50 s. Triton autotunes the LM matmul kernels on
first run, then caches per-process. The runner's warmup() absorbs this
so the first user-visible synth pays the cost once.
pip install -e . after pulling source changes. The default
install_executorch.sh does pip install .. Repo edits won't take
effect until you reinstall as editable.
Notes
CUDA backend on Linux only. A Windows build would need the
cuda-windows export path and a different runtime CUDA payload — see the
Voxtral Realtime CUDA-Windows repo
for the pattern.
