Measured in the served (generation) setting — the model reasons before answering, as in deployment.
Base
CRACK
Δ
MMLU
84.2%
79.8%
-4.4%
HarmBench (refusal removal)
Harm-category compliance: 59/60 = 98% (10 per category) — base model refuses (~0%).
Category
Compliance
Illegal activities
9/10 (90%)
Chemical / biological
10/10 (100%)
Cybercrime / intrusion
10/10 (100%)
Misinformation
10/10 (100%)
Harassment / bullying
10/10 (100%)
Harmful content
10/10 (100%)
Copyright-reproduction prompts are excluded (not a refusal behavior).
Coherence & capability ✅
Factual QA, multi-step reasoning, and working code generation verified
Vision inputs preserved · no loops, no truncation
Other Quantizations
Also available: Gemma 4 26B-A4B MXFP4 CRACK — same family, different precision/size trade-off.
Usage
Requires vMLX (bundled Gemma 4 support). Standard mlx_lm / mlx_vlm do not fully support Gemma 4.
python
1# Load in the vMLX app or via its API2from vmlx_engine.models.mllm import MLXMultimodalLM
3m = MLXMultimodalLM("<this-repo>")4print(m.chat([{"role":"user","content":"..."}]).text)