The size-optimized sibling of Qwen3.6-35B-A3B-MLX-GPTQ-3.5bpw: same loss-aware + GPTQ pipeline, pushed to the practical floor of MLX scalar affine quantization (~2.73 effective bits/weight overall, 12.0 GB). Fits comfortably in 24 GB unified memory.

Recipe

Experts (91.7% of params): per-(layer,tensor) menu MCKP over (bits × group-size) candidates — chosen mix {2bit/gs128: 69, 3bit/gs128: 11} (avg 2.38 bpw) by empirical-Fisher gxw cost, then GPTQ (pooled per-layer Hessian, sym=False, MSE grid, act-order static groups).
Spine is GPTQ'd too (unlike the 3.5bpw build): linear-attention qkv/z/out, full-attention q/k/v/o, shared experts — 4bit gs64 with their own input Hessians; lm_head 6bit. Routers 8bit, embeddings 4bit, vision tower unquantized (bf16).

Quality (vs bf16, fakequant-proxy measured on CUDA)

holdout	KL	PPL
ja-think	0.288	9.75 → 11.46 (+17.5%)
multilingual	0.227	11.02 → 12.71 (+15.4%)

(Numbers measured with RTN spine; the shipped artifact additionally GPTQ-compensates the spine, so true quality is slightly better.) For reference, the 3.5bpw/15.4 GB sibling measures +6.6%/+6.9% — pick that one if you can spare 3.4 GB. This 2.7bpw build is for tight-memory Macs, where it is roughly the best scalar-affine MLX can do: plain-RTN mixed 2-bit at similar size measures ~+47%.

Run (Apple Silicon, ≥16 GB unified memory)

bash

pip install -U mlx-lm            # needs >=0.31 (qwen3_5_moe support)

mlx_lm.generate --model aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw \
  --prompt "東京の観光名所を3つ教えてください。" --max-tokens 2048

mlx_lm.server --model aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-2.7bpw --port 8080

Notes

Text-only inference is the validated path (vision weights present, unquantized).
Thinking mode is on by default and verbose; pass "chat_template_kwargs": {"enable_thinking": false} for direct answers.

Provenance / license

Base model © Alibaba Cloud, Apache-2.0. Derived from the official bf16 checkpoint and the mlx-community 4bit conversion (structure/tokenizer).