Quantization-sensitive, moderate param footprint (~27% of total)
6-bit
mlp (gate/up/down_proj)
Bulk of params (~69%) — sole compression lever in a dense model
Why no bf16 MLP?
In the MoE XL recipe, the shared/dense MLP stays bf16 because routed experts (6-bit) carry the bulk and are sparsely activated. In a dense model, the MLP is the bulk — keeping it at bf16 would negate the purpose of quantization. Instead, we elevate attention from 6-bit to 8-bit (vs the older _68 recipe that quantized both at 6-bit), preserving the component most sensitive to quantization noise.
BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — critical layers get higher precision, bulk parameters get lower precision. The result is a transparent, faithful capture of the source model.
Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.
XL beats data-aware OptiQ 4-bit across the board (+4 MMLU, +2 MMLU_PRO, +2 ARC, +4 HUMANEVAL, +2.4 avg), while the 26B MoE big sister retains the crown as expected (+6 points avg, but 2.1× the size).
⚠️ n=50 sampling means wide confidence intervals (±~13% at 95% CI). Differences under ~6 points may not be statistically significant. Data-aware methods (OptiQ) may be calibrated on benchmark-like data — their scores carry a benchmaxxing caveat. BaseQuant_XL (data-agnostic) provides the most honest generalization estimate.
Local SOTA Comparison (for context)
The 26B XL is the current champion for 48GB Macs. The dense 12B offers nearly identical architecture at half the size:
Benchmark
Gemma-4 26B 6bit-XL
Qwen3.6-35B 6bit-XL
Gemma-4 12B 6bit-XL (this)
MMLU
76%
64%
68%
MMLU_PRO
82%
64%
76%
ARC_CHALLENGE
90%
90%
88%
HUMANEVAL
98%
78%
94%
MBPP
82%
78%
72%
Average
85.6%
74.8%
79.6%
Size
21 GB
28 GB
9.9 GB
Chat Template
Ships with Google's canonical chat_template.jinja (18,681 chars) in both chat_template.jinja and tokenizer_config.json. Includes preserve_thinking support for tool-calling workflows.
