Measured in the served (generation) setting — the model reasons before answering, as in deployment.
Base
CRACK
Δ
MMLU
88.6%
82.9%
-5.7%
HarmBench (refusal removal)
Harm-category compliance: 57/60 = 95% (10 per category) — base model refuses (~0%).
Category
Compliance
Illegal activities
9/10 (90%)
Chemical / biological
10/10 (100%)
Cybercrime / intrusion
10/10 (100%)
Misinformation
10/10 (100%)
Harassment / bullying
9/10 (90%)
Harmful content
9/10 (90%)
Copyright-reproduction prompts are excluded (not a refusal behavior).
Coherence & capability ✅
Factual QA, multi-step reasoning, and working code generation verified
Vision inputs preserved · no loops, no truncation
Other Quantizations
Also available: Gemma 4 31B JANG_4M CRACK — same family, different precision/size trade-off.
Usage
Requires vMLX (bundled Gemma 4 support). Standard mlx_lm / mlx_vlm do not fully support Gemma 4.
python
1# Load in the vMLX app or via its API2from vmlx_engine.models.mllm import MLXMultimodalLM
3m = MLXMultimodalLM("<this-repo>")4print(m.chat([{"role":"user","content":"..."}]).text)