Qwen3.6-27B — UD-NVFP4_K_XL (mlx-node)
NVFP4 (NVIDIA Blackwell FP4) quantization of
Qwen/Qwen3.6-27B for Apple Silicon, using the
Unsloth Dynamic quantization strategy via
mlx-node.
| Original (BF16) | UD-Q4_K_XL (affine) | This Model |
|---|
| Size | ~66 GB | 21 GB | 21 GB |
| Format | SafeTensors | SafeTensors | SafeTensors |
| Precision | BF16 uniform | 4-bit affine + BF16 | NVFP4 (E4M3 scales) + mixed affine + BF16 |
| FFN group size | — | 64 | 16 (nvfp4) / 64 (affine) |
| Biases | — | yes | no (FFN nvfp4); yes (affine layers) |
What is NVFP4?
NVFP4 is NVIDIA's
Blackwell FP4 micro-scaling format. Each group of
16 elements shares a single 8-bit
E4M3 scale (a full FP8 number with mantissa, not just an exponent), and elements themselves are stored as E2M1 FP4 values. Compared to MXFP4 (OCP):
- Higher fidelity scale: E4M3 scale has 3 mantissa bits, MXFP4's E8M0 only encodes a power-of-two
- Smaller group: 16 vs. 32 — fewer outliers per scale, better dynamic range tracking
- Higher scale density: 2× the scales per weight (16 elements vs 32 per scale), so per-byte overhead is roughly the same as MXFP4 despite the richer scale type
- No biases: zero-point implicit (FP4 covers ±range)
For dense LLM weights, NVFP4 typically beats MXFP4 on perplexity at the same nominal bit budget, with the trade-off of slightly more metadata per weight.
All Variants
Benchmarked on Apple M3 Max 128GB via
examples/lm.ts (best decode tok/s across turns 2–4, steady-state).
Performance
Steady-state decode: 14.9 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: 'low'). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. This NVFP4 build preserves bf16 escape hatches for the most sensitive tensors (linear_attn.out_proj, self_attn.o_proj), so it trades a few tok/s for measurably better quality vs. a uniform-FP4 layout.
Per-Tensor Bit Assignments (N=4)
| Weight | Mode | Bits | Group | Rationale |
|---|
embed_tokens | 6-bit affine | 6 | 64 | Loader is affine-only; nvfp upgrade skipped (unsloth base+2) |
lm_head | 8-bit affine | 8 | 64 | Loader is affine-only; nvfp upgrade skipped (unsloth base+3 snapped 7→8) |
self_attn.q/k/v_proj | 6-bit affine + AWQ | 6 | 64 | AWQ via input_layernorm; preserved at higher bits (unsloth base+2) |
linear_attn.in_proj_qkv/z | 6-bit affine + AWQ | 6 | 64 | AWQ via input_layernorm; preserved at higher bits (unsloth base+2) |
self_attn.o_proj | bf16 | — | — | NOT AWQ-correctable |
linear_attn.out_proj | bf16 | — | — | KLD ~6.0 — worst tensor, kept full-precision |
mlp.down_proj | 5-bit affine | 5 | 64 | "Slightly more sensitive" (unsloth base+1) |
mlp.gate_proj, mlp.up_proj | nvfp4 | 4 | 16 | Unsloth UD-Q4 base — promoted to NVFP4 |
| GDN params (A_log, etc) | bf16 | — | — | State-space dynamics |
Quantization Strategy
Built on Unsloth Dynamic 2.0 per-tensor KLD analysis. At --q-bits 4 the unsloth recipe's per-layer bit offsets become 4-bit FFN gate/up (promoted to NVFP4), 5-bit down_proj, 6-bit attn/SSM projections with AWQ pre-scaling, 6-bit embed_tokens, and 8-bit lm_head. Then --q-mode nvfp4 orthogonally promotes only the 4-bit affine decisions to NVFP4 (mode="nvfp4", bits=4, group_size=16) — non-4-bit decisions stay affine at their original bit width, and AWQ-uncorrectable projections (o_proj, out_proj) stay bf16.
imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead). AWQ-correctable projections (q/k/v, in_proj_qkv/z) get the AWQ pass; non-AWQ-correctable projections (o_proj, out_proj) stay bf16 — their inputs come from attention/GDN computation, not from a norm layer.
Architecture
| Parameter | Value |
|---|
| Total parameters | 27.4B (dense — all active) |
| Hidden size | 5,120 |
| Layers | 64 (48 linear + 16 full attention) |
| Attention heads | 24 (4 KV heads, GQA 6:1) |
| Head dimension | 256 |
| Intermediate size | 17,408 |
| Vocab size | 248,320 |
| Max context | 262,144 tokens |
Usage
1import { loadSession } from '@mlx-node/lm';
2
3const session = await loadSession('./Qwen3.6-27B-UD-NVFP4_K_XL-mlx');
4
5for await (const event of session.sendStream('Explain NVFP4 vs MXFP4 quantization.', {
6 config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
7})) {
8 if (!event.done) process.stdout.write(event.text);
9}
How It Was Made
1mlx convert \
2 -i Qwen3.6-27B \
3 -o Qwen3.6-27B-UD-NVFP4_K_XL-mlx \
4 -q --q-mode nvfp4 --q-recipe unsloth \
5 --imatrix-path imatrix_unsloth.gguf
--q-mode nvfp4 selects NVIDIA's NVFP4 micro-scaling format as the global dequantizer (bits=4, group_size=16). Combined with --q-recipe unsloth, the recipe emits per-tensor affine decisions for sensitive layers (q/k/v + AWQ, down_proj, lm_head, embed_tokens at higher bits; o_proj/out_proj/GDN as bf16), and the converter promotes the remaining 4-bit affine decisions (gate_proj/up_proj) to NVFP4. --q-bits 4 is implicit when --q-mode nvfp4 is set.
Acknowledgments
- Unsloth — Quantization strategy based on their per-layer KLD benchmarks and Dynamic 2.0 methodology
- NVIDIA NVFP4 — For the NVFP4 specification
- Qwen Team — For the Qwen3.6 model family
- Apple MLX — For the Metal-accelerated ML framework
License
Apache 2.0 (inherited from base model).
