Apple Silicon (MLX) mixed-precision quantizations of Qwen/Qwen3.6-35B-A3B — a 35B MoE with 3B active parameters, 3:1 Gated DeltaNet : full attention hybrid (qwen3_5_moe). One repo, one card — each quantization lives on its own branch.
* single-stream decode on an M3 Ultra, short prompt. A3B routing makes this the fastest model class per GB in the lineup — ~3× the decode rate of the dense 27B sibling.
Everything below is measured, not projected — KL / top-1 flip on a fixed 4096-token EN/code/ZH/KO slice (teacher-forced vs bf16), strided perplexity (ctx 2048 / stride 1024), and single-stream decode on an M3 Ultra.
Which build?
24 GB+ Mac → main. 16 GiB weights, 17.5 GB measured peak at short context, 102 tok/s.
16 GB Mac → 16gb. 11 GiB weights, 11.8 GB measured peak — exceeds the default ~10.7 GiB wired limit; raise it first:
sudo sysctl iogpu.wired_limit_mb=13312
With int8 KV that leaves ~1.5 GiB of cache ≈ 150 K tokens of context (KV here is tiny — see below). 111 tok/s decode makes this the best speed-per-GB option for 16 GB machines; prefer it over the dense 27B 16 GB build when throughput matters more than per-token quality.
Multimodal (default)
main carries the full multimodal stack — 4.0bpw text weights + the vision tower at
8-bit (verified with EN/KO OCR tests). Plain mlx-lm loads main fine for text-only use;
the text branch saves 0.60 GB if you never need images.
Long context: --kv-bits 8 --kv-group-size 64 in the generate path (measured quality-free, below).
⚠️ 16gb under review. The dense-27B sibling's same-calibration low-bit tier was withdrawn after free-running generation showed early stops / repetition loops on long-CoT problems (teacher-forced KL/PPL did not catch it). This branch shares that calibration recipe and is being re-verified generatively; treat it as experimental until this notice is removed.
Recipes (verified from each build's config.json)
92% of the parameters live in 256 fused routed experts per MoE layer (stored as 3D
SwitchLinear tensors — the sensitivity scan samples 8 experts/layer/matrix). Per-tensor
bits assigned by marginal utility from a 2–6-bit + nvfp4 scan of all 472 text tensors:
nvfp4 vs affine — measured both ways. On the dense 27B sibling, affine-only beat
nvfp4 decisively (its card has the numbers). Here the reverse holds: an affine-only
4.0bpw rebuild (50.3% a3 + 49.4% a4) scored KL 0.309 vs main's nvfp4-mix 0.292
— nvfp4 g16 genuinely suits the small fused expert matrices (2048×512). The format
choice is architecture-dependent; main stands as-is.
DWQ. Layerwise DWQ (KL distillation vs bf16, 260-sample EN/KO/ZH/code chat mix, seq
512, validation-gated — alis-dwq): at 4.0bpw every
round was rejected (already teacher-close), so main carries conversion-time weights; at
2.7bpw DWQ improved valid KL 0.305 → 0.268 (−12%).
Quality
KL(bf16‖quant) / top-1 flip on the fixed 4096-token slice, per domain:
main scores 76.3% (avg 14,742 thinking tokens, 7 no-answer). For scale: a dense
27B at 4.6bpw scores 72.7 on the same harness, and quantized short-thinking models
reach 83.3 — see the cross-family curve on the
ThinkingCap repo.
The 16gb 2.7bpw tier fails hard reasoning generatively (MMLU-300 75.3% but GPQA
non-convergent) — hence the warning above; treat it as a light/general-use build only.
KV cache — measured recommendations
Only 10 of 40 layers carry KV (the rest are Gated DeltaNet recurrent state), so KV is
tiny to begin with; quantizing it is still measured quality-free:
KV config
KL vs bf16-KV
KV bytes/token
~ctx in 1.5 GiB
bf16
0
20 KB
75 K
int8 g64 (recommended)
0.011
10 KB
150 K
K4/V8 (long-context)
0.016
7.5 KB
200 K
Notes & provenance
Text-only conversion: the base checkpoint's vision tower and MTP head are not included
(mlx-lm text path). An mtp-bf16 sidecar branch may land later.
