Measured in the served (generation) setting — the model reasons before answering, as in deployment.
Base
CRACK
Δ
MMLU
78.9%
78.9%
+0.0%
HarmBench (refusal removal)
Harm-category compliance: 58/60 = 97% (10 per category) — base model refuses (~0%).
Category
Compliance
Illegal activities
9/10 (90%)
Chemical / biological
10/10 (100%)
Cybercrime / intrusion
10/10 (100%)
Misinformation
10/10 (100%)
Harassment / bullying
9/10 (90%)
Harmful content
10/10 (100%)
Copyright-reproduction prompts are excluded (not a refusal behavior).
Coherence & capability ✅
Factual QA, multi-step reasoning, and working code generation verified
Vision and audio inputs preserved · no loops, no truncation
Other Quantizations
Also available: Gemma 4 12B JANG_4M CRACK — same family, different precision/size trade-off.
Usage
Requires vMLX (bundled Gemma 4 support). Standard mlx_lm / mlx_vlm do not fully support Gemma 4.
python
1# Load in the vMLX app or via its API2from vmlx_engine.models.mllm import MLXMultimodalLM
3m = MLXMultimodalLM("<this-repo>")4print(m.chat([{"role":"user","content":"..."}]).text)