Qwen3.5-122B-A10B-NVFP4-GB10
Qwen3.5-122B-A10B optimized for NVIDIA DGX Spark (GB10, SM12.1) — all quantizations baked in.
This checkpoint is derived from
Sehyo/Qwen3.5-122B-A10B-NVFP4 with additional post-quantizations applied and saved permanently into the weights. The result loads and serves immediately on GB10 without any runtime quantization overhead.
Why this exists
The base NVFP4 checkpoint leaves several layer types in BF16 to keep it hardware-agnostic. On GB10 (NVIDIA Grace Blackwell, SM12.1), those BF16 layers are a significant bottleneck:
- GDN (GatedDeltaNet) layers are Qwen3.5's linear attention mechanism. They run on every token and constitute ~33% of decode compute time at BF16. Quantizing them to FP4/FP8 roughly halves their memory bandwidth cost with minimal accuracy impact.
- The lm_head projects hidden states to a 248,320-token vocabulary at every decode step. At BF16 this reads ~1.5 GB of weights per token generated. Quantizing to FP8 halves this to ~750 MB/token — a direct throughput gain.
These optimizations are implemented in
scottgl9/sglang-spark-gb10-optimizations as runtime post-quantization hooks. This checkpoint bakes them in so:
- Startup is faster (no quantization at load time)
- The exact quantized weights are reproducible and bit-identical every run
- Other GB10 users can use this checkpoint without needing the custom SGLang fork
What's quantized and why
| Layer | Original | Baked format | Why |
|---|
| MoE expert weights (gate/up/down) | NVFP4 (W4A4) | NVFP4 (W4A4) | From source checkpoint — preserved as-is |
GDN in_proj_qkv, in_proj_z | BF16 | NVFP4 (Marlin FP4) | Large projections; SM12.1 has native Marlin FP4 kernels. CUTLASS FP4 is broken on GB10. |
GDN in_proj_a, in_proj_b, out_proj | BF16 | FP8 (per-tensor, dynamic activations) | Smaller GDN projections; FP8 CUTLASS works correctly on SM12.1. cos_sim > 0.999, SNR = 31.5 dB vs BF16. |
| lm_head (vocabulary projection) | BF16 | FP8 (dynamic activations) | 3072 x 248,320 matmul at every decode step. FP8 halves DRAM reads vs BF16. Activations remain dynamic per-token for accuracy. |
| Attention Q/K/V/O | BF16 | BF16 | Standard attention; not the bandwidth bottleneck at batch=1. |
Performance
~46 tok/s decode on a single NVIDIA DGX Spark (GB10, 128 GB unified memory) using SGLang with NEXTN speculative decoding (steps=2, draft_tokens=2).
Baseline without any of these optimizations: ~28 tok/s. The baked quantizations account for approximately +18 tok/s of that gain.
Hardware
Tested on: NVIDIA DGX Spark (ASUS Ascent GX10), GB10 Grace Blackwell, SM12.1, 128 GB unified memory.
Requirements:
- SM12.1 (GB10) — Marlin FP4 kernels are SM12.1-specific
- scottgl9/sglang-spark-gb10-optimizations SGLang fork
- CUDA 13.x
Will NOT work on:
- Standard H100/A100 — Marlin FP4 kernels require Blackwell SM12.1
- Apple Silicon — Metal only, no CUDA path
Usage
1# Load directly — no post-quant overhead at startup
2SGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \
3 --model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10
SGLANG_QUANTIZE_LM_HEAD_FP8=0 tells SGLang to skip the lm_head FP8 post-quant (already baked in). All other sglang.sh flags remain unchanged.
With NEXTN speculative decoding (recommended):
1SGLANG_QUANTIZE_LM_HEAD_FP8=0 ./sglang.sh Qwen3.5-NVFP4 \
2 --model-path scottgl9/Qwen3.5-122B-A10B-NVFP4-GB10 \
3 --speculative-algorithm NEXTN \
4 --speculative-num-steps 2 \
5 --speculative-num-draft-tokens 2
Source models
License
Apache 2.0 (same as Qwen3.5)
