This repository contains quantized standalone GGUF checkpoints for running Voxtral Mini 4B Realtime ASR with audio.cpp. The GGUF files embed the audio.cpp model spec and required sidecars, so the model can be used directly from the checkpoint path without a separate local model-spec directory.
What audio.cpp does
audio.cpp is a young C++/GGML audio inference framework focusing on CUDA performance. It runs speech and audio models locally with CLI and server interfaces, including ASR, TTS, voice conversion, source separation, diarization, VAD, and audio generation models. The project currently tracks 40+ model families and is growing quickly. For Voxtral Realtime, audio.cpp provides offline and streaming speech recognition from a single GGUF checkpoint.
Files
File
Quantization
Size
Recommended use
voxtral-mini-4b-realtime-2602-q8_0.gguf
Q8_0
4.8 GiB
Default balanced checkpoint for high-quality ASR with lower memory than BF16.
voxtral-mini-4b-realtime-2602-q4_k.gguf
Q4_K
2.9 GiB
Lower-memory and faster checkpoint for CUDA testing and deployment. Validate output quality for your domain.
Download
Each checkpoint is a single self-contained file, so download only the quantization you intend to run rather than the whole repository.
Either lands at ./voxtral-realtime-gguf/<filename>.gguf, which is the path to pass as --model.
The hf command ships with huggingface_hub (pip install -U huggingface_hub). On releases before the CLI was renamed, the same call is huggingface-cli download.
Point --model at the .gguf file itself, not at the folder holding it. audio.cpp accepts a directory only when it can narrow it to a single GGUF, so a folder containing both quantizations is rejected rather than guessed at:
model directory contains 2 GGUF files: ./voxtral-realtime-gguf; found: ...
pass one of them directly with --model, or keep a single GGUF in the directory
Performance
Measurements below are from audio.cpp CUDA validation runs. Results vary by GPU, driver, backend, audio length, and decode settings.
Q8_0 vs BF16 reference
BF16 was used only as a local reference baseline for validation. This repository publishes quantized GGUF checkpoints only.
Mode
BF16 reference
Q8_0
Q8_0 improvement
Offline ASR speed
11.1x-12.5x realtime
14.7x-16.7x realtime
1.31x-1.38x faster
Offline ASR peak VRAM
10,909 MiB
7,754 MiB
3,155 MiB lower
Streaming server TTFT
207.308 ms
179.896 ms
27.412 ms lower
Streaming client TTFT
550.526 ms
530.558 ms
19.968 ms lower
Streaming speed
4.7x realtime
5.4x realtime
1.15x faster
Streaming peak VRAM
12,616 MiB
8,972 MiB
3,644 MiB lower
Q4_K quick check
Route
Q8_0 RTF
Q4_K RTF
Q4_K vs Q8_0
Offline short
0.0862
0.0629
1.37x faster
Offline medium
0.0643
0.0476
1.35x faster
Offline longer
0.0576
0.0439
1.31x faster
Offline sampled
0.0630
0.0500
1.26x faster
Streaming path
0.1036
0.0904
1.15x faster
In the quick validation set, Q4_K transcripts matched Q8_0 except for one capitalization-only difference.
Use with audio.cpp
Build audio.cpp with the helper script for your platform, then use the generated audiocpp_cli binary. The project provides build paths for Linux, Windows, and macOS; see the audio.cpp README for the current build matrix and detailed requirements.
bash
1git clone https://github.com/0xShug0/audio.cpp
2cd audio.cpp
34# Linux: CUDA, Vulkan, or CPU5scripts/build_linux.sh --backend cuda --target audiocpp_cli
67# Windows: CUDA or CPU presets8powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli
910# macOS: Metal11scripts/build_metal.sh --target audiocpp_cli
The examples below assume audiocpp_cli is on your PATH. You can also replace it with the built binary path for your platform, such as build/linux-cuda-release/bin/audiocpp_cli. They pass --backend cuda; use --backend metal on macOS, or --backend cpu when no GPU backend is built.
While a stream runs, the CLI reports each update as the text decoded since the previous one, so the updates concatenate into the transcript. On a terminal they are appended and the transcript scrolls like ordinary output; when stdout is redirected, each update is written as its own flushed partial_text= line so pipes and logs stay parseable. --text-out and the final text_output= line carry the complete transcript either way.
A streaming step always advances 80 ms of audio, so each step has to cost under 80 ms to keep pace with a live source. A step is one frontend pass, one audio-encoder forward, and one text-decoder step. Setting voxtral_realtime.stream_batch_tokens=<n> makes a single encoder forward cover n audio tokens instead of one. The decoder still runs once per token, so batching amortizes the encoder's fixed per-dispatch cost and leaves decode work untouched:
Measured on an Apple M3 MacBook Air with Metal and Q8_0, over a 14 s clip (169 steps, mean of three runs). These are laptop numbers and are not comparable to the CUDA figures earlier in this card; what carries across hardware is the split between the two middle columns.
stream_batch_tokens
Encoder ms/step
Decoder ms/step
Total ms/step
Streaming speed
1 (default)
30.8
49.0
79.9
1.04x realtime
2
27.5
50.2
77.7
1.07x realtime
4
13.3
50.0
63.4
1.31x realtime
8
7.4
49.6
57.1
1.46x realtime
The decoder column is flat, which is the whole shape of this knob: batching can only remove encoder time, so the decoder sets a floor and the gain flattens once the encoder is no longer the bottleneck. 2 is not worth taking — it barely moves the encoder — while 4 captures most of the available win.
What you pay for it is latency. A chunk is not transcribed until all n of its audio tokens have arrived, so each update is delayed by up to n * 80 ms: 0.32 s at 4, 0.64 s at 8. Transcribing a file, that delay costs nothing perceptible; on a live microphone it is the tradeoff to weigh.
Note also that the decoder runs one step per 80 ms whether the audio holds speech or silence, so a session that does fall behind stays behind — the lag does not recover during pauses. Measure your own hardware before relying on a live source.
Prompting and decoding
audio.cpp exposes Voxtral Realtime as an ASR model. Pass audio to the CLI and audio.cpp builds the model's transcription prompt internally; no free-form text prompt is required for normal transcription.
Decode options can be controlled from the request:
