Views
No views yet
.pte files
for Voxtral-Mini-4B-Realtime-2602
with Metal backend (Apple GPU). Supports both offline and streaming
transcription with GPU acceleration on macOS Apple Silicon.1git clone https://github.com/pytorch/executorch/ ~/executorch
2cd ~/executorch && EXECUTORCH_BUILD_KERNELS_TORCHAO=1 TORCHAO_BUILD_EXPERIMENTAL_MPS=1 ./install_executorch.shcd ~/executorch && make voxtral_realtime-metallibomp (required by the AOTInductor kernels):1brew install libomp
2export DYLD_LIBRARY_PATH=/usr/lib:$(brew --prefix libomp)/lib1pip install huggingface_hub
2huggingface-cli download younghan-meta/Voxtral-Mini-4B-Realtime-2602-ExecuTorch-Metal --local-dir ~/voxtral_metalAll run commands requireDYLD_LIBRARY_PATHset as shown in the installation step.
1DYLD_LIBRARY_PATH=/usr/lib:$(brew --prefix libomp)/lib \
2 cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
3 --model_path ~/voxtral_metal/model-metal-fpa4w.pte \
4 --tokenizer_path ~/voxtral_metal/tekken.json \
5 --preprocessor_path ~/voxtral_metal/preprocessor.pte \
6 --audio_path ~/voxtral_metal/poem.wav1DYLD_LIBRARY_PATH=/usr/lib:$(brew --prefix libomp)/lib \
2 cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
3 --model_path ~/voxtral_metal/model-metal-fpa4w-streaming.pte \
4 --tokenizer_path ~/voxtral_metal/tekken.json \
5 --preprocessor_path ~/voxtral_metal/preprocessor-streaming.pte \
6 --audio_path ~/voxtral_metal/poem.wav \
7 --streaming1ffmpeg -f avfoundation -i ":0" -ar 16000 -ac 1 -f f32le -nostats -loglevel error pipe:1 | \
2 DYLD_LIBRARY_PATH=/usr/lib:$(brew --prefix libomp)/lib \
3 cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \
4 --model_path ~/voxtral_metal/model-metal-fpa4w-streaming.pte \
5 --tokenizer_path ~/voxtral_metal/tekken.json \
6 --preprocessor_path ~/voxtral_metal/preprocessor-streaming.pte \
7 --mic| Mode | TTFT | Gen Tokens | Gen Rate (tok/s) | Total Inference |
|---|---|---|---|---|
| Offline | 2.121s | 377 | 40.82 | 11.356s |
| Streaming | 0.304s | 260 | 16.18 | 16.374s |
1# Offline
2python examples/models/voxtral_realtime/export_voxtral_rt.py \
3 --model-path ~/models/Voxtral-Mini-4B-Realtime-2602 \
4 --backend metal \
5 --output-dir ./voxtral_rt_metal_offline \
6 --qlinear-encoder fpa4w \
7 --qlinear fpa4w
8
9# Streaming
10python examples/models/voxtral_realtime/export_voxtral_rt.py \
11 --model-path ~/models/Voxtral-Mini-4B-Realtime-2602 \
12 --backend metal \
13 --streaming \
14 --output-dir ./voxtral_rt_metal_streaming \
15 --qlinear-encoder fpa4w \
16 --qlinear fpa4wLibrary not loaded: @rpath/libc++.1.dylib — Add /usr/lib to DYLD_LIBRARY_PATH:export DYLD_LIBRARY_PATH=/usr/lib:$(brew --prefix libomp)/libLibrary not loaded: libomp.dylib — Install OpenMP via Homebrew:brew install libompffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav