Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptIQ quants · Docs
A 4-bit mixed-precision MLX quant produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. Scores +6.40 over stock uniform 4-bit on the six-metric Capability Score, the second-largest mixed-precision gain in the Gemma-4 lineup.
A 4-bit mixed-precision MLX quant of google/gemma-4-12B-it, the unified (text+vision+audio) Gemma-4. This artifact is the text-inference path: the language tower is quantized and the vision/audio towers are dropped during conversion. Per-layer bit-widths come from a KL-divergence sensitivity pass on a six-domain calibration mix (prose, reasoning, code, agent, tool-call, constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit.
uniform-4-bit (bf16 base too large to fit in RAM; auto-resolved)
We follow the same naming convention llama.cpp uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform 4-bit on the Capability Score at a comparable disk size.
Usage
This is the unified Gemma-4 12B (model_type: gemma4_unified), so it needs mlx-lm from main and import optiq to register the model type. The unified text tower is not in the 0.31.3 PyPI release of mlx-lm, and the main build also reports version 0.31.3, so install from git rather than pinning a version:
1import optiq # registers the gemma4_unified model type2from mlx_lm import load, generate
34model, tokenizer = load("mlx-community/gemma-4-12B-it-OptiQ-4bit")5response = generate(6 model, tokenizer,7 prompt="Explain quantum computing in simple terms.",8 max_tokens=200,9)
Gemma-4-12B is a reasoning model with a thinking channel. For direct (non-thinking) answers on tasks like extraction or classification, pass chat_template_kwargs={"enable_thinking": False} when applying the chat template.
For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server, hot-swap mounted adapters, sandboxed Python execution for agent workflows), install mlx-optiq:
Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit, both at the full sample counts:
Metric
OptIQ
Uniform 4-bit
Δ
MMLU (5-shot, 1000 samples)
42.6%
34.4%
+8.3
GSM8K (1000 samples, 3-shot CoT)
93.4%
90.1%
+3.3
IFEval (full set, strict)
73.9%
71.2%
+2.8
BFCL-V3 simple (200 calls)
71.0%
71.5%
−0.5
HumanEval (164 problems, pass@1)
88.4%
76.8%
+11.6
HashHop (long-context retrieval)
40.0%
27.0%
+13.0
Capability Score (mean of 6)
68.23
61.83
+6.40
KL vs uniform-4-bit reference (mean / p95)
1.52 / 4.85
,
,
On-disk size
8.3 GB
6.3 GB
+2.0
Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. BFCL-V3 is within its ±6.3pp confidence interval (a tie); the other five benchmarks all improve, with the largest gains on long-context retrieval and code. MMLU is scored cloze-style (answer-letter log-likelihood), which underscores reasoning models that prefer to think before answering; the strong GSM8K and HumanEval results are the better quality signal. See the eval-framework writeup for the full methodology.
