NVFP4 (nvfp4) quantization of Qwen/Qwen3.6-35B-A3B produced with NVIDIA TensorRT Model Optimizer via a multi-node ModelOpt quantizer. KV cache FP8. This is a quantized derivative and inherits the base model's license; see the base model card for usage.

Unlike a BF16-draft-head build, this checkpoint NVFP4-quantizes the MTP/NEXTN draft head (a full-attention + shared-expert-MoE block) and bakes FP8 KV scales onto its attention, so the SGLang p51 KV-scale loader exercises the MTP head. Because transformers drops the MTP layer at load and the multinode FSDP2 export carries no mtp.* at all, the head is produced by a post-export surgical requant that sources the raw BF16 mtp.* from the original checkpoint and NVFP4-packs them, mirroring each module's main-model analogue. All three gates have now been measured on GB10/sm121: the smoke-serve loads register-free with NEXTN active, GSM8K scores 96.5% (see the benchmarks table), and the draft head accepts 2.71 of a theoretical 3.00 tokens per step under load. What is still missing is the A/B against a BF16-MTP build, so the numbers below establish that the quantized draft works, not that it is free.

The NVFP4 weights of the draft head are data-free and bit-identical to a calibrated run, but its W4A4 input_scale is a generous-amax heuristic and its FP8 KV k_scale/v_scale are borrowed from the last main-model full-attention layer. Correctness is unaffected (spec-decoding verifies every draft token); a poor draft only lowers acceptance/throughput. Measured: mean accept len 2.71 of a theoretical maximum of 3.00 (num_steps=2, num_draft_tokens=3), i.e. 90% of the ceiling, so whatever the heuristic scales cost is bounded by the remaining 10%. A BF16-MTP A/B would pin that number down exactly, but the headroom it could recover is small.

Qwen3.6-35B-A3B carries a 27-layer vision tower. Calibration is text-only, so the vision stack is excluded from quantization and stays BF16 (avoiding the amax=0 degenerate-quant failure mode); this is why the model must be quantized via the multinode driver, whose quant policy applies the vision excludes.

Serving needs the mamba scheduler knobs to avoid a spec-v2-vs-radix-cache boot crash: mamba_scheduler_strategy extra_buffer plus SGLANG_ENABLE_SPEC_V2=1, along with the flashinfer attention backend and flashinfer_cutlass MoE/FP4-GEMM backends. NEXTN speculative decoding at num_steps=2 was the validated throughput sweet spot; higher step counts (3/4/5) decayed.

Stock SGLang 0.5.15 crashes loading this checkpoint on GB10/sm121; it needs five launch-time source patches, shipped in the public dgxarley repo (https://github.com/vroomfondel/dgxarley) under roles/k8s_dgx/files/sglang_patches/ and applied by sglang_launch.sh: (1) p50 -- allow QUANTIZED attention for modelopt_fp4 (this is a uniform-W4A4 export that also quantizes the Gated-DeltaNet linear-attn, unlike NVIDIA's MoE-only NVFP4); (2) p51 -- load the main model's baked FP8 KV scales onto RadixAttention (remap ...self_attn.k_proj.k_scale -> ...attn.k_scale), else the full-attn layers default to scale 1.0; (3) p43 -- keep quant_config for the QUANTIZED MTP draft head, else SGLang assumes a BF16 MTP, builds it unquantized, and the fused-MoE loader narrows the unpacked intermediate dim over the NVFP4-packed expert weight ("start(0)+length(512) exceeds dimension size(256)"); (4) p44 -- apply the same KV-scale mapper in the MTP load path (qwen3_5_mtp.py has its own load_weights) so the draft attention loads its baked FP8 scales instead of the 1.0 sentinel; (5) the arch-independent sm121 CUTLASS-FP4 mma patch. p43/p44 are specific to a quantized MTP and are inert for a BF16-MTP checkpoint. Independently, config.json pins torch_dtype: bfloat16 (the Qwen3.6 source config omits it) so SGLang's --dtype auto resolves bf16; without it the unquantized in_proj_ba loads as fp16 and crashes at the CUDA-graph capture ("mat1 and mat2 ... BFloat16 != Half"). Validated end to end on GB10/sm121 (image 0.5.15.post1-sm121): loads register-free, NEXTN speculation active, coherent output, MTP draft attn loads its borrowed KV scales (k_scale 0.0398, byte-identical to the last full-attn layer, the requant borrow source).