Asymmetric-expert TurboQuant quantization of Qwen/Qwen3.6-35B-A3B: 3-bit attention, ternary (1.58-bit) expert up/gate_proj + 4-bit expert down_proj (group size 64) — produced with TurboQuant-MLX.
This is the agent-capable sibling of the 9.4 GB pure-ternary build. Spending 4 bits on just the expert down_proj (one of the three expert matrices) costs +3.2 GB (12.6 GB total) and is the difference between failing and passing multi-step tool-use: in a fix-the-failing-test task under Opencode this build went 3/3, where pure ternary went 0/4 and a 3-bit down_proj variant still went 0/3.
Model Details
Base model: Qwen/Qwen3.6-35B-A3B — Mixture-of-Experts (qwen3_5_moe)
Routers and shared-expert gates → full precision (never quantized)
Size: 12.6 GB (11.7 GiB) — vs ~70 GB BF16, 9.4 GB pure ternary, 16.4 GB 3-bit
Why 4-bit down_proj (and not the other two)?
The expert MLP computes down_proj(silu(gate_proj(x)) * up_proj(x)). up/gate feed a gated nonlinearity that is tolerant of coarse weights, and their errors are partially averaged across the top-8 routed experts. down_proj is the write-back into the residual stream — its errors land directly on the token representation every layer. Measured on the Opencode agentic task, capability turns on between 3 and 4 bits on this one matrix: down3 fails like pure ternary, down4 completes the full observe → diagnose → edit → verify loop. (Same recipe as DwarfStar/llama.cpp lore: spend bits on down, starve up/gate.)
Running it
Requires turboquant-mlx-full ≥ 0.13 (the expert_down_bits config field is new):
bash
1pip install"turboquant-mlx-full>=0.13" mlx-lm
23python -m turboquant_mlx.generate \4 --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \5 --prompt "Explain why the sky is blue."\6 --max-tokens 512
On a 16 GB Mac mini the weights exceed the default Metal wired-memory cap (~10.5 GB), so raise it once per boot:
It reads only this repo's safetensors headers over the network (~240 KB, a
couple of seconds — the weights are never fetched) and projects the peak against
your machine, then prints the flags to use:
Projection at 21,000 tokens of context
weights 12.59 GB
KV cache 0.22 GB (10.0 KB/token, hybrid: 10/40 full-attention layers)
prefill workspace 0.09 GB (estimate, at --prefill-step-size 128)
runtime reserve 1.00 GB (buffer cache, activations, fragmentation)
----------------------------------
peak 13.89 GB of 14.40 GB usable 0.51 GB headroom
Verdict: ⚠️ RESIDENT — fits, but only after raising the Metal wired cap
Recommended:
sudo sysctl -w iogpu.wired_limit_mb=13721
--prefill-step-size 128
--kv-bits 8
Planning for a machine you're not sitting at — say, deciding whether to buy or
use a mini — is what --wired-gb / --ram-gb are for (the run above is exactly
that: a 16 GB mini, projected from a 64 GB machine). turboquant-doctor adds a
readiness check, and both take --json.
The projection is calibrated against real measurements on the mini rather than
estimated from theory: it predicts a 10.44 GB peak where the 9.4 GB ternary
sibling
measures 10.42, and the wired limit it suggests here lands within 1% of the
13824 that this build actually runs on.
--tool-syntax-greedy forces argmax inside <tool_call> blocks (keys, punctuation, tags) while leaving payload strings and the call-or-not decision sampled — cheap insurance for tool-call syntax at low bit-widths. Use temperature ~0.7; greedy-everything hurts this model's planning.
Quality — measured, not assumed
Agentic harness test (Opencode, fix-the-failing-test task, identical server/harness/prompt across builds, 64 GB M4 Max):
Point Opencode (or any OpenAI-compatible agent harness) at
http://<host>:8080/v1 with model id default_model, temperature ~0.7,
and auto-discovered skills disabled. Practical context ceiling on 16 GB is
~16–18K tokens at the default wired cap; the --prefill-step-size 128 +
14 GiB wired combination above is what carries 21K+ (the serve defaults
--metal-cache-limit-gb auto and --prompt-cache-max-gb auto handle the
rest of the tight-memory story automatically).
Data-free (no calibration set): randomized Hadamard rotation → per-group RMS scaling → Lloyd-Max codebook (ternary for up/gate, 16-level for down) → trit/bit packing. Per-projection bit-width is self-describing on disk via codebook length; expert_down_bits is recorded in config.json so converter and loader agree.
