Mistral-Small-3.2-24B-Hybrid-MXFP4-Q8

• ~13.5 GB on disk (vs ~48 GB BF16), so it fits in unified memory and leaves room for long context.
• MXFP4 group size 32 matches Metal SIMD width, so inference is memory-bandwidth bound and scales with M4’s high UMA bandwidth.
• Vision kept at 8-bit so OCR and fine-grained vision stay reliable; only the dense LLM is heavily quantized.

Why this is a good fit for M4

• Unified memory: One copy of the model; no CPU↔GPU transfer. Smaller quantized model = more room for 128k context and faster token throughput.
• MXFP4 + g32: 4-bit microscaling format with group size 32 aligns with Apple’s GPU threadgroups, so the 4-bit path is well-utilized and quality stays close to BF16.
• Hybrid choice: The ~3% vision/connector params stay at Q8 to avoid “blind” vision; the rest is MXFP4 for size and speed.

Quick start