All three preserve full BF16 precision in attention, vision, and multi-token prediction layers. The 4-bit version matches the 8-bit on coding benchmarks. The 8-bit is more stable for long context (100-200k token) agentic work.
The Objective: Get more throughput out of Qwen3.6-27B on Apple Silicon without sacrificing the reasoning and context-window recall that make it useful for agentic work and coding.
The Problem: BF16 gives you the full brain but chugs at ~9 tokens/sec on M5 Max due to memory bandwidth. Uniform quantizations are faster but degrade the hybrid attention mechanism and vision tower. The compressed attention damages agentic ability.
The Solution: Mixed Quantization. A surgical approach that compresses only the SwiGLU MLP layers — which make up 66% of the model's mass — to 8-bit, 6-bit, or 4-bit, while leaving every attention projection, the vision tower, the MTP head, embeddings, and norm layers at full BF16 precision.
The result: 17 tokens/sec at 4-bit with nearly identical benchmark scores, or 14–16 tok/s at 8/6-bit if you want to be conservative.
Resulting sizes:
Version
Size
Speed
Reduction
BF16 baseline
~49 GB
9 tok/s
—
Mixed 8-bit
36 GB
14 tok/s
27%
Mixed 6-bit
32 GB
16 tok/s
35%
Mixed 4-bit
28 GB
17 tok/s
43%
(2) Architecture
Qwen3.6-27B is a dense hybrid-attention model. It has 64 transformer layers following a repeating pattern:
The linear attention layers use SSM-style state-space projections (in_proj_qkv/z, conv1d, A_log, dt_bias) for efficient long-context streaming across a 262K token window. Every fourth layer uses full softmax attention for deep reasoning.
Additionally, the model includes:
A 27-layer vision tower for image/video comprehension
A 1-layer MTP (Multi-Token Prediction) head for speculative decoding
SwiGLU MLPs in every layer for feature transformation
Identifying the Weight
Out of 27B parameters, the MLP projections (gate_proj, up_proj, down_proj) make up 66% of the model's mass — 32 GB of the 49 GB BF16 footprint. These SwiGLU layers are the "knowledge lookup" / feature transformation matrices and are extremely tolerant of quantization.
The Glass Cannons: Layers Protected at BF16
Layer
Why Protected
Full attention (q_proj, k_proj, v_proj, o_proj, q_norm, k_norm)
Deep reasoning and cross-token attention. Quantizing these degrades logic and long-context recall.
Linear attention (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj, conv1d, A_log, dt_bias)
SSM-style state tracking. This is how the model maintains context across the 262K window. Critical for agentic workflows.
Vision tower (all layers)
Image and video comprehension. Quantizing the vision encoder collapses multimodal understanding.
MTP head (all layers)
Multi-token prediction quality. Only 0.8 GB — not worth compressing.
Embed tokens / LM head
Input/output boundary layers. Quantizing these adds noise at the model's interface.
66% of model mass. SwiGLU MLPs are the most quantization-tolerant component in a transformer. All the savings live here.
(3) The Quantization Script
The script uses MLX's native nn.quantize() with a class_predicate filter that targets only the three SwiGLU projection matrices inside language_model.layers.N.mlp. Everything else passes through untouched.
Full script: mixed_quantize_qwen3_6_27b.py in the quantization-scripts repository.
(4) Bypassing Framework Bugs
Critical deployment information
The mlx_vlm sanitize_weights Panic
Issue: The mlx_vlm library searches for HuggingFace-style tensor key names during its sanitize_weights phase. Because this model is natively quantized in MLX format, the key structure differs, causing a crash or fallback to LLM-only mode.
Fix: Locate utils.py in your mlx_vlm installation and add return weights as the first line of the sanitize_weights function to bypass the check entirely as follows:
Update: No hacks required to load these models as VLMs.
Note on config.json
The config.json declares a single "bits" value, but MLX reads per-layer quantization metadata dynamically from the safetensors tensor headers. The actual mixed-precision structure — which layers are quantized and which remain BF16 — lives in the safetensors files, not the config. Tooling that trusts only config.json will be confused; this is expected behavior for native MLX mixed quantization.
Mixed MLX quantization of Qwen3.6-27B that compresses only the MLP layers — while preserving all attention, vision, and MTP layers at BF16 — delivers up to an 89% speedup with no statistically significant intelligence loss.
The 4-bit version is the sweet spot: at 28 GB and 17 tok/s, it matches the 8-bit on MBPP and LiveCodeBench (62.0% both), and trails BF16 by only 1.3% on HumaneVal — well within benchmark variance. It completes LiveCodeBench in 33 seconds per problem vs. 89 seconds for BF16, making it dramatically more practical for real-time agentic coding workflows.
The hybrid attention architecture (linear SSM + full softmax) is the key: by protecting both attention types in BF16, the model retains its full 262K context-window recall and reasoning capability. The MLP layers — which are just feature transformation matrices — absorb even 4-bit quantization with no measurable penalty on coding tasks.
Notes
Fixed seed sampling was used on all benchmarks.
MBPP and LiveCodeBench were run on subsamples (200/500 and 100/1055 respectively).
Test times include multitasking overhead and are not strictly comparable.
Vision capabilities were preserved but not formally benchmarked in this pass.
