Apple Silicon (MLX) mixed-precision quantization of bottlecapai/ThinkingCap-Qwen3.6-27B — the token-efficiency finetune of Qwen3.6-27B (~50% fewer thinking tokens at preserved accuracy). 4.601 bpw, 15.47 GB, GPQA-Diamond 83.3% — 0.5pt from bf16.
* n=198, temperature 1.0, seed-fixed shuffled choices, 32K token cap — identical harness for all three rows (harness validated by reproducing the ThinkingCap card's published 83.8 exactly).
† single-stream on an M3 Ultra, short prompt.
Showdown: accuracy, size and thinking tokens vs the official Q4_K_M GGUF and bf16
Why this build
Smaller and more accurate than BottleCapAI's official 4-bit GGUF, with 22% fewer thinking
tokens per answer — on Apple Silicon, fewer generated tokens is the speedup that
matters. Fits 24 GB Macs with room for context (15.8 GB measured peak at short
context; int8 KV keeps long-context overhead small — only 16 of 64 layers carry KV
in this 3:1 DeltaNet hybrid).
The finding behind it: short thinkers survive quantization
Across our GPQA campaign, the same 4.6bpw recipe loses 12.8pt on vanilla
Qwen3.6-27B (85.5 → 72.7) but only 0.5pt on ThinkingCap (83.8 → 83.3). Long
reasoning chains accumulate quantization drift token by token; ThinkingCap's
~3K-token chains stay on track where vanilla's ~11K-token chains derail.
Token-efficiency finetune first, quantize second — the two compound.
GPQA vs bits-per-weight: vanilla quants degrade steeply; the ThinkingCap quant sits on the bf16 line
Multimodal (default)
main carries the full multimodal stack: the 4.6bpw text weights (¹ GPQA measured on the
identical text weights) + the vision tower at 8-bit. 16.13 GB total — still smaller than
the 16.81 GB text-only GGUF. Verified with OCR + visual-reasoning tests (EN/KO).
Text-only users: main loads fine under plain mlx-lm (vision weights are skipped), or
use the text branch to save 0.66 GB.
Long context: --kv-bits 8 --kv-group-size 64 (measured quality-free on this family: KL vs bf16-KV ≤ 0.003).
Recipe (verified from config.json)
Per-tensor bits by marginal utility from a full weight-space sensitivity scan,
validated end-to-end (the scan alone misranks formats — see the nvfp4 note on the
vanilla 27B repo):
component
precision
bulk (90.1% of params)
affine 4-bit g64
sensitivity-ranked top 9.8%
affine 5-bit g64
routers / small gates
6-bit pins
embed / lm_head
≥4-bit
No DWQ retune: at ≥4bpw the validation-gated DWQ rejects every round (already
teacher-close); conversion-time weights ship as-is.
Notes & provenance
main carries the full multimodal stack (mlx-vlm); the text branch is the mlx-lm text-only artifact.
Eval artifacts: same 198-question GPQA protocol for every row of the table above.
Built on an M3 Ultra 512 GB · mlx-lm 0.31.3 · mlx 0.31.2 · 2026-07.
