Measured in the served (generation) setting — the model reasons before answering, as in deployment.
Base
CRACK
Δ
MMLU
70.6%
66.7%
-3.9%
HarmBench (refusal removal)
Harm-category compliance: 240/240 = 100% (full HarmBench-320 text set) — base model refuses (~0%).
Category
Compliance
Illegal activities
53/53 (100%)
Chemical / biological
42/42 (100%)
Cybercrime / intrusion
52/52 (100%)
Misinformation
54/54 (100%)
Harassment / bullying
21/21 (100%)
Harmful content
18/18 (100%)
Copyright-reproduction prompts are excluded (not a refusal behavior).
Coherence & capability ✅
Factual QA, multi-step reasoning, and working code generation verified
Vision and audio inputs preserved · no loops, no truncation
Other Quantizations
Also available: Gemma 4 E4B JANG_4M CRACK — same family, different precision/size trade-off.
Usage
Requires vMLX (bundled Gemma 4 support). Standard mlx_lm / mlx_vlm do not fully support Gemma 4.
python
1# Load in the vMLX app or via its API2from vmlx_engine.models.mllm import MLXMultimodalLM
3m = MLXMultimodalLM("<this-repo>")4print(m.chat([{"role":"user","content":"..."}]).text)