A real-world coding benchmark: each model was prompted to write a complete flight simulator from scratch. The resulting code was rendered and recorded.
Flight Simulator
About XL Quantization
BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.
Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.
Local SOTA for 48GB Macs — Intelligence Benchmark Comparison
This model is part of a benchmark comparison of the best local MLX-quantized LLMs that fit in 48GB unified memory on Apple Silicon. All benchmarks run in instruct mode (no thinking) with n=50 samples per benchmark.
⚠️ n=50 sampling means wide confidence intervals (±~13% at 95% CI). Differences under ~6 points may not be statistically significant. Models using data-aware quantization (oQ/oQe) may be calibrated on benchmark-like data — their scores carry a benchmaxxing caveat. The BaseQuant_XL variants (data-agnostic) provide the most honest generalization estimates.
leonsarmiento/Qwen3.6-35B-A3B-oQ4e-mtp-XL-mlx
A hybrid MLX quantization that surgically combines layers from two quantization variants of Qwen3.6-35B-A3B:
The result is a quantization-level layer swap — not a weight-space merge (DARE-TIES, Task Arithmetic). The model architecture is identical; only the quantization precision and method per layer differs.
Why This Combination?
The two donor quantizations use fundamentally different approaches:
Aspect
XL (BaseQuant_XL)
oQ4e-mtp (oQ Quantization)
Method
Data-agnostic uniform group quantization
Data-aware — calibration dataset guides bit allocation
Attention
8-bit, group_size=64
Mixed 5/6/8-bit (data-aware)
Experts
5-bit, group_size=64
4-bit, group_size=64
Shared Expert Gate
bf16 (unquantized)
8-bit
Shared Expert
bf16 (unquantized)
8-bit
lm_head
bf16 (unquantized)
4-bit
Vision Tower
5-bit
bf16 (unquantized)
MTP Layer
Not present
Present (8-bit)
The hypothesis: XL's high-precision 8-bit attention and bf16 shared expert/lm_head provide the best token routing and output projection, while oQ4e's data-aware 4-bit experts preserve the weight regions that matter most for the oQ calibration data. The MTP (Multi-Token Prediction) layer from oQ4e-mtp is preserved for speculative decoding support.
Important consideration: oQ/oQe quantization is data-aware — its quality depends on how well the oQ calibration dataset represents your use case. If your workload aligns with the calibration data, data-aware experts can outperform data-agnostic quantization at the same or lower bit depth. If it doesn't, the data-agnostic XL quantization may be more reliable. This hybrid gives you data-aware experts with data-agnostic attention — a middle ground.
Analysis: Data-Aware vs Data-Agnostic Quantization
This comparison spans three quantization philosophies:
oQ / oQe (data-aware): Uses calibration data to guide bit allocation — oQ uses sensitivity analysis, oQe uses a process similar to iMatrix. Both are potentially benchmaxxed — if the calibration set overlaps with these benchmarks, the scores reflect calibration quality as much as model quality.
XL (data-agnostic): No calibration data. Layers are quantized to fixed bit depths based on their architectural role (routing-critical → bf16, every-token → 8-bit, bulk experts → 6-bit). Results reflect genuine generalization.
Frankenstein oQ4e-XL (this model): Hybrid — data-agnostic attention/routing from XL + data-aware 4-bit experts from oQ4e. Inherits calibration risk on the expert layers only.
Key observations:
This model scores highest on average (78.8%), edging out oQ4e-mtp (78.0%) by +0.8pp. The XL attention layers boost MMLU by +6 (72 vs 66) and MMLU_PRO by +4 over oQ4e-mtp — knowledge recall benefits from higher-precision routing. However, the data-aware experts from oQ4e likely contribute to the coding benchmark strength.
The benchmaxxing caveat is critical. oQ/oQe quantizations use calibration data that may include these exact benchmarks (or similar distributions). If so, their scores are partly a measure of calibration fit, not general intelligence. The XL quantization carries no such risk — its results are honest.
If we discount data-aware results as potentially benchmaxxed, the 6bit-XL is the honest king. Its average of 74.8% reflects pure architectural quality without calibration assistance. Notably, it matches or beats oQ6-mtp (71.2%) across all five benchmarks — the 6-bit data-aware variant is actually weaker than data-agnostic XL, suggesting oQ calibration at 6-bit may be counterproductive for this model.
HUMANEVAL tells the most interesting story. oQ4e-mtp leads at 92.0%, this model follows at 90.0%, but oQ6-mtp drops to 66.0% — a 26-point collapse. This suggests data-aware 4-bit calibration is highly effective for code generation, but data-aware 6-bit calibration may actively harm it. The data-agnostic 6bit-XL lands in between at 78.0% — a safe, honest result.
⚠️ n=50 sampling means wide confidence intervals (±~13% at 95% CI). Differences under ~6 points may not be statistically significant. The HUMANEVAL gap between oQ6-mtp (66%) and the 4-bit variants (90%+) is too large to dismiss as noise, but most other differences are within sampling error.
Custom MATHQA (thinking mode, n=30, corrected test set)
A failure-enriched MATHQA test set (n=30) run with thinking enabled and a relaxed 8192-token budget to isolate reasoning quality from throughput artifacts. The test set was manually audited and 9 of 19 "hard" questions corrected for dataset errors (dropped digits, wrong expected answers, impossible constraints) before this run — see methodology in leonsarmiento/gemma-4-26B-A4B-it-6bit-XL-mlx.
Raw vs verified accuracy: The "raw" column reflects the initial benchmark run. The "verified" column reflects a re-test of all 5 questions that failed on either model — both models solved all 5 correctly on re-run. The original failures were:
Degenerate reasoning loops (2 questions, non-XL only) — the model spiraled into 20–22K char self-verification loops producing zero extractable output. On re-test, both were solved cleanly in 36–42s with ~6–7K reasoning chars. These are transient failures caused by sampling variance (temperature=1.0), not capability gaps.
Answer extraction artifacts (3 questions) — correct answers masked by markdown bold formatting (**E**) or non-standard answer formatting that the extraction regex missed. On re-test with an improved regex, all 3 were recovered as correct.
Key findings:
Both models can solve all 30 questions. The corrected test set is well within both models' reasoning capability. The raw 90% vs verified 100% gap is entirely attributable to (a) sampling-induced degenerate loops and (b) extraction bugs.
XL is more stable and token-efficient. The XL eliminated both degenerate loop failures in the initial run (Q636, Q1556) and uses 15% less reasoning chars per correct answer (5,847 vs 6,881). The XL's bf16 routing produces more decisive reasoning chains — the same finding observed on Gemma 4 26B A4B XL.
MTP degrades thinking-mode stability. Enabling MTP on this XL variant dropped raw accuracy from 90.0% → 83.3% and introduced 3 degenerate loops on previously reliable questions, while increasing wall time +20%. See the MTP evaluation section below.
⚠️ Thinking mode uses temperature=1.0, which introduces sampling variance. Individual questions can occasionally spiral into degenerate loops on any model. The n=30 raw scores carry ±10% noise; the re-test confirms both models are capable of perfect scores on this test set.
About MTP (Multi-Token Prediction)
⚠️ Note on MTP and TurboQuant when using oMLX: Tests on oMLX 0.5.5 show no MTP (Lightning MTP) advantage when TurboQuant KV Cache is turned On. For MTP decoding speed advantages, TurboQuant should be disabled.
This model preserves the MTP (Multi-Token Prediction) layer from the oQ4e-mtp donor for speculative decoding support in compatible runtimes (LM Studio, oMLX). MTP can significantly boost raw token generation throughput — on this model, from ~48 tok/s (MTP-OFF) to ~58 tok/s (MTP-ON), a +21% throughput increase.
However, our evaluation reveals an important caveat for thinking/reasoning mode:
Metric
MTP-OFF
MTP-ON
Delta
Accuracy
27/30 (90.0%)
25/30 (83.3%)
−2 questions
Total wall time
1,396s
1,679s
+20% slower
Throughput
48.5 tok/s
58.5 tok/s
+21% faster
Degenerate loops
0
3
144–147s spirals with zero output
The problem: In thinking mode, the model is an autonomous reasoner — it decides how long to think. MTP's faster token generation doesn't produce the same answer sooner; instead, the model thinks more with the extra speed. This creates three possible outcomes:
✅ Speed dividend (same or less reasoning, faster wall clock) — sometimes happens on genuinely hard questions where more thinking helps
❌ Speed waste (more reasoning, same or worse wall clock) — most common outcome, e.g., a trivial 2-second question ballooning to 109s with 16K reasoning chars
❌ Degenerate loop (model spirals until token limit, zero extractable output) — MTP introduced 3 of these on previously reliable questions
Recommendation:
Thinking mode: Keep MTP OFF. The throughput boost is wasted on unnecessary reasoning loops, accuracy regresses, and wall time increases. The faster token generation is a liability, not an asset, when the model controls its own output length.
Instruct mode (no thinking): MTP may be beneficial — output length is deterministic, so faster tokens directly translate to faster wall time. Not formally tested here.
The right MTP evaluation metric is correct answers per minute (wall-clock efficiency) and correct answers per million tokens (context/energy efficiency), not tokens per second.
Quantization Strategy
Layer
Source
Bits
Group Size
Method
mlp.gate (router)
XL
bf16
—
—
shared_expert_gate
XL
bf16
—
—
lm_head
XL
bf16
—
—
shared_expert.up_proj / down_proj
XL
bf16
—
—
embed_tokens
XL
8
64
Data-agnostic
self_attn (full attention)
XL
8
64
Data-agnostic
linear_attn (DeltaNet)
XL
8
64
Data-agnostic
shared_expert.gate_proj
XL
8
64
Data-agnostic
switch_mlp (routed experts)
oQ4e-mtp
4
64
Data-aware
vision_tower
oQ4e-mtp
bf16
—
—
MTP layer
oQ4e-mtp
8
64–128
Data-aware
About the Base Model
Qwen3.6-35B-A3B is a 35B-parameter multimodal MoE (Mixture of Experts) model with 256 experts (8 active per token + 1 shared expert), hybrid full + linear (Gated DeltaNet) attention, and a vision encoder. Despite 35B total parameters, only ~3B are activated per token for efficient inference.
