Views
No views yet
cuda-windows export path and the CUDA 12.9 Windows CUDA payload, then run natively on Windows with the ExecuTorch Voxtral realtime runner.1git clone https://github.com/pytorch/executorch/ ~/executorch
2cd ~/executorch
3pip install -e . --no-build-isolation1cmake --workflow --preset llm-release-cuda
2Push-Location examples/models/voxtral_realtime
3cmake --workflow --preset voxtral-realtime-cuda
4Pop-Location1pip install huggingface_hub
2
3hf download younghan-meta/Voxtral-Mini-4B-Realtime-2602-ExecuTorch-CUDA-Windows --local-dir voxtral_cuda_windows
4hf download mistralai/Voxtral-Mini-4B-Realtime-2602 tekken.json --local-dir voxtral_tokenizertekken.json to this repo, you can skip the second download command and point --tokenizer_path at the local copy instead.1.\cmake-out\examples\models\voxtral_realtime\Release\voxtral_realtime_runner.exe `
2 --model_path voxtral_cuda_windows\model.pte `
3 --data_path voxtral_cuda_windows\aoti_cuda_blob.ptd `
4 --preprocessor_path voxtral_cuda_windows\preprocessor.pte `
5 --tokenizer_path voxtral_tokenizer\tekken.json `
6 --audio_path C:\path\to\audio.wav `
7 --streaming--temperature 0.0 -- greedy decoding (default)--mic -- live microphone input from stdin--mic_chunk_ms 80 -- microphone read chunk size in mscuda-windows instructions, but use the CUDA 12.9 Windows installer payload before setting WINDOWS_CUDA_HOME.1export WINDOWS_CUDA_HOME=/opt/cuda-windows/extracted/cuda_cudart/cudart
2
3python -m executorch.extension.audio.mel_spectrogram \
4 --feature_size 128 \
5 --streaming \
6 --output_file ./voxtral_rt_exports/preprocessor.pte
7
8python examples/models/voxtral_realtime/export_voxtral_rt.py \
9 --model-path ~/models/Voxtral-Mini-4B-Realtime-2602 \
10 --backend cuda-windows \
11 --dtype bf16 \
12 --streaming \
13 --sliding-window 2048 \
14 --output-dir ./voxtral_rt_exports \
15 --qlinear-encoder 4w \
16 --qlinear-encoder-packing-format tile_packed_to_4d \
17 --qlinear 4w \
18 --qlinear-packing-format tile_packed_to_4d \
19 --qembedding 8wcuda-windows exports produce model.pte and aoti_cuda_blob.ptd.preprocessor.pte.tekken.json matched with the same base model revision used during export.