The first sub-4-bit quantization of Qwen3.6-35B-A3B: the standard 4-bit needs ~24 GB at runtime and does not fit a 24 GB Mac; this one runs in 18 GB (measured, long-context decode) with headroom to spare — and with 3-bit-aware kernels it decodes faster than the 4-bit.

Method

Sensitivity-based greedy bit allocation (weight-space, calibration-free), tuned for this hybrid GatedDeltaNet/MoE architecture:

Forced 4-bit (7 % of params): everything structural — linear_attn.* (SSM gates and projections, where quantization error compounds through the recurrence), the MoE router, shared experts, full-attention layers, embeddings, lm_head.
Greedy 3/4-bit on the routed expert stacks, decided per layer trio (gate/up/down move together): fused expert kernels require matching bit widths, so mixed-width layers would silently fall off the fast path. All 40 expert layers stay kernel-friendly (28 × 3-bit, 12 × 4-bit).
Standard MLX affine format, group size 64 — loads in mlx-lm and any engine supporting per-layer mixed MLX quants.

Quality (measured vs bf16, teacher-forced KL on 5 FR prompts)

quant	bpw	disk	KL(bf16‖q)	top-1 agree	fits 24 GB?
official-style 4-bit	4.5	19.5 GB	0.141	0.891	❌ (~24 GB runtime)
this repo	3.85	16 GB	0.162	0.876	✅ (18 GB runtime)

The hardest prompt degrades to KL ≈ 0.44 in both — that difficulty is a property of quantizing this MoE at ~4 bits, not of the mixed scheme, which costs only +0.02 KL overall for 18 % less memory.

Speed

Measured on a MacBook Pro M5 Max, 128 GB with saragossa (GPU-resident decode, single-copy unified-memory loader, 3-bit expert gather kernels):

engine	decode
saragossa (main, u3 gather kernels)	~150 tok/s — faster than the official 4-bit (~145)
saragossa v0.2.x release (generic 3-bit path)	~94 tok/s
`mlx-lm`	runs fine, standard mixed-quant path

Decode is memory-bandwidth-bound: expect roughly half on an M-series Pro, a quarter on a base M chip.

Usage

bash

# mlx-lm
mlx_lm.generate --model <this-repo> --prompt "Bonjour !"

# saragossa — brew install azerozero/tap/saragossa
saragossa run <this-repo>

Tooling

The quantization tooling is public: tools/quant/ — sensitivity-based bit allocation (calibration-free) plus the KL evaluation script used for the numbers above. The README there documents the exact commands that reproduce this repo.

License

Apache-2.0, same as the base model.

destynova002/Qwen3.6-35B-A3B-mixed-3-4bit-3.85bpw-mlx

Qwen3.6-35B-A3B — mixed 3/4-bit (3.85 bpw, MLX affine) — fits 24 GB Macs