Qwen3.6-35B-A3B — PrismaQuant 4.9 bpp

At a glance

Precision mix

• Allocator result at target=4.9: achieved 4.907 across costed Linears
  • BF16: 294 layers
  • NVFP4: 90 layers
  • MXFP8_E4M3: 17 layers
• Export recipe over all emitted entries:
  • BF16: 404
  • NVFP4: 90
  • MXFP8: 17

Activation-aware passes applied during export

1. GPTQ-OBS one-shot rounding — block-wise error propagation along the group-quant structure using the calibration Hessian. Closed-form, not iterative. Handles cross-column activation coupling.
2. Closed-form per-group scale sweep — for each 16-weight NVFP4 group, enumerate grid=32 candidate scales spanning [0.5·s₀, 1.5·s₀], round each weight to its nearest codebook neighbor at every candidate scale, pick the (scale, rounding-set) configuration minimizing activation-weighted per-group MSE sum_j a_j² · (w_orig,j - w_q,j)². Improve-or-keep gate against the post-GPTQ weight. Handles within-group weight-distribution variation that GPTQ takes as fixed.

Which layers are quantized

Text body (DeltaNet linear-attention + dense MoE, 40 layers)

• Full attention Linears (q_proj, k_proj, v_proj, o_proj): mixed NVFP4 / MXFP8 / BF16 per-Linear by sensitivity
• DeltaNet linear-attention Linears (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj): same
• MoE experts (gate_up_proj, down_proj): per-expert NVFP4 with joint per-tensor scale across the gate_up pair so vLLM FusedMoE loads them
• Shared expert MLP: same per-Linear policy
• Router (mlp.gate): always BF16 (tiny, sensitive)

Multi-token-prediction (MTP) head

• Speculative-decoding head (1 layer) + its own MoE block: same per-Linear policy, so --speculative-config method=mtp drafts at the same precision as the body.

Visual encoder (27 blocks — Qwen3.6-VL vision tower)

• Visual Linears were forced with --visual-format=BF16.
• 110 visual Linears were assigned BF16 uniformly.
• model.visual.pos_embed remains BF16 passthrough as a Parameter.

Passthrough (unquantized)

• lm_head — kept at BF16 because vLLM's ParallelLMHead module only accepts a single weight parameter. The allocator measures lm_head's Fisher sensitivity and would pick NVFP4 for it (saving ~770 MB), but the compressed-tensors runtime rejects a compressed lm_head with KeyError: lm_head.input_global_scale because its scheme registry doesn't include ParallelLMHead. This is a vLLM runtime limitation, not a PrismaQuant design decision.
• RMSNorm weights (all layers + MTP + visual)
• All biases
• embed_tokens
• model.visual.pos_embed (Parameter/Embedding, see above)

Serving (vLLM only)

• FlashInfer NVFP4 attention is picked up automatically; set VLLM_USE_FLASHINFER_NVFP4=1 to make the preference explicit.
• MTP speculative decoding at n=3 is the measured optimum for this family on DGX Spark (n=2 leaves ~10 % tok/s on the table, n=4 regresses).
• Visual inputs work via vLLM's standard image-text-to-text chat API — no special flags.

Reproducing this artifact

1. Sensitivity probe — streaming per-shard empirical-Fisher trace (diagonal) across body + MTP + visual Linears. Each shard holds only its ~2 layers resident; the rest of the model is on disk or meta. 8 multimodal calibration samples drive visual Fisher through one unified streaming context.
2. Per-(Linear, format) cost measurement — for each Linear and each candidate format, the per-group RTN error weighted by cached input activations. Incremental: same per-shard streaming as the probe.
3. Multi-choice knapsack allocator — picks one format per Linear minimizing total predicted Δloss under the bit budget. Target 4.9 bpp; achieved 4.907 bpp here. The same run reported a knee near target 5.0 (achieved 4.971). Known-non-Linear rank-2 tensors (pos_embed, rotary_emb) are excluded from the visual pool.
4. Export — streams each body / visual / MTP shard, applies GPTQ + scale_sweep to NVFP4 entries, and writes compressed-tensors shards. Export recipe for this run: 511 entries with mix {NVFP4: 90, BF16: 404, MXFP8: 17}. lm_head passthrough at BF16 is enforced at this stage (see known issues).

Known issues / limitations

• vLLM only at serve time. No transformers-runtime path for this precision mix today.
• lm_head stays BF16 because vLLM's ParallelLMHead does not register the NVFP4/MXFP8 compressed-tensors schemes. Allocator measured it and would have picked NVFP4; the runtime limitation forces BF16. Costs ~770 MB on the disk footprint.
• MTP n=4 regresses on this family. Stick to n=3 unless you verify against the draft-head acceptance-rate trace.
• Export fast path was unavailable in this run. The exporter fell back to the torch implementation because flash-linear-attention and/or causal-conv1d dependencies were missing.
• PyTorch CUDA capability warning on GB10. The environment printed a warning that GPU capability 12.1 is outside the declared max (12.0) for that specific torch build.

Links

• Source: github.com/RobTand/prismaquant
• Base model: Qwen/Qwen3.6-35B-A3B

Citation