Audio path: encoder + adapter + DeepStack in BF16 on MLX
Runtime: no PyTorch at inference time
Target: desktop/workstation (not mobile-viable)
Layout
mlx_llm/ Qwen3-8B INT4 weights + tokenizer (4.3 GB)
mlx_audio/ Audio encoder + adapter + DeepStack (1.6 GB)
mergers, all BF16
scripts/ Pure-MLX bridge source (same as 4B)
inference.py Standalone example
Users who want a smaller bundle can INT4-quantize the audio path
themselves.
We didn't ship INT4 8B audio by default because the 8B's value prop
is quality, not size — and at 1.6 GB, BF16 audio isn't the binding
constraint.
Measured on Apple M3 Ultra
This bundle
Full BF16
Decode speed
110 t/s
25 t/s
Decode steady-state peak
5.0 GB
33 GB
Transient prefill peak
8.1 GB
33 GB
Disk footprint
5.9 GB
18.1 GB
Audio quality varies by domain (speech vs non-speech); validate on your target data.
Usage
Same as the 4B bundle — the inference.py script auto-detects size:
No PyTorch required at runtime (mel path ported to MLX).
Evaluation (BFCL v3, community-run)
Text-only tool-calling evaluation (not an audio-caption quality metric), run on a
600-sample subset (simple/multiple/parallel = 200 each), greedy decoding
(repetition_penalty=0), measured 2026-05.
Category
This bundle
8B BF16
Qwen3-8B base
simple (200)
91.0%
93.5%
94.5%
multiple (200)
92.5%
93.0%
94.5%
parallel (200)
54.5%
78.5%
90.5%
3-cat avg
79.3%
88.3%
93.2%
Single-call categories (simple/multiple) within 1–2 pp of BF16. Parallel
degrades ~24 pp under INT4 — genuine decode regression. For speech-dominant
use cases (dialog, interviews, transcription), tool-calling quality is strong.
Parser note: this model can emit bare JSON tool calls; evaluation accepted
schema-valid {name, arguments} JSON as tool-call output.
Limitations
Non-speech instability (upstream-intrinsic). 8B is unstable on
ambient / music / non-speech clips: 2/9 hard truncations plus 2 further
degenerate outputs (digit-loops, hallucinated JSON "metadata") on our
7-clip × 3-rep EN-scope benchmark. Raising --repetition-penalty to
1.05 makes this worse, not better (10/21 truncations vs 6/21 at RP=1.02).
This failure mode matches the PyTorch reference's behavior
(10/21 truncations reported in the v2 benchmark), so it is not an
MLX-port regression. Recommend this bundle for speech-dominant use
cases (dialog transcription, speaker attribution, spoken-event
analysis) and avoid it for music genre tagging or ambient-scene
captioning — use the 4B bundle for those.
Contributors
Rumilabs Inc - We are building the richest content knowledge base in the world to empower interactive media: Quantization, MLX conversion, hybrid runtime design, benchmarking, and release packaging.
