Qwen3.6-35B-A3B — APEX i-quality-torch vs NVFP4
TLDR: 31 July 2026 NVFP4 support in llama.cpp doesn't deliver on the GB10 yet. This model required patching the converter and then ran ~44% slower than my APEX quant for identical bit size. It did load faster, but that matters almost not at all for my applications.
A head-to-head between a hand-tuned APEX depth-band quant and unsloth's NVFP4-Fast quant of the same base model — same architecture, nearly identical overall bit budget, very different allocation philosophy. NVFP4 is NVIDIA's 4-bit block-scaled float format.
Models compared
Both start from the same
Qwen/Qwen3.6-35B-A3B checkpoint (hybrid linear-attention + latent-MoE with a shared expert, native MTP head).
Scope note: plain unsloth/Qwen3.6-35B-A3B-NVFP4 and nvidia/Qwen3.6-35B-A3B-NVFP4 were originally planned additions to this comparison, but that work was dropped once the NVFP4-Fast results below made clear that the interesting variable here is llama.cpp's NVFP4 decode kernel maturity, not producer-to-producer differences — see the callout in Results. All NVFP4 checkpoints from any producer would hit the same kernel-level decode ceiling on this hardware, so a 4-way comparison wouldn't add new signal over the 2-way one below.
Note on MTP: the NVFP4-Fast checkpoint does not retain the model's native multi-token-prediction (MTP/nextn) tensors — unsloth's config.json explicitly lists re:^mtp.* in its ignore list. The APEX torch quant keeps MTP (Q6_K), enabling free --spec-draft-n-max 2 self-speculative decoding that the NVFP4 quant can't use in this form. Speed numbers below are reported both with and without speculative decoding so the comparison is apples-to-apples on the quant format itself, with the MTP bonus called out separately.
Structural bit allocation: APEX vs NVFP4-Fast
Per-tensor-role comparison, read directly off each checkpoint's tensor shapes/dtypes (not just aggregate bpw):
| Component | APEX i-quality-torch | Unsloth NVFP4-Fast |
|---|
| Attention (q/k/v/o) | Q6_K (~6.56 bpw) | FP8 (~8-9 bpw) |
| Linear-attn proj (in/out) | Q6_K (~6.56 bpw) | FP8 (~8-9 bpw) |
| Linear-attn state/norm (a, conv1d, dt.bias) | F32 | bf16 (untouched) |
| Router gate | BF16/F32 | bf16 (untouched) |
| Routed experts | depth-band mix: IQ4_XS (middle) / Q5_K / Q6_K (edge, near) — 4.25–6.56 bpw | flat NVFP4 for every expert regardless of depth — ~4 bpw |
| Shared experts | Q6_K/Q8_0 | NVFP4, same as routed |
| MTP head | Q6_K | absent (stripped, see note above) |
| lm_head | Q6_K | FP8 (per-channel scale, group_0) |
| Overall (measured) | 5.298 bpw (23.51 GB) | 5.539 bpw (24.58 GB) |
The two schemes converge on the same instinct — protect attention/router, compress the big expert FFNs — but differ sharply in how. Unsloth's NVFP4-Fast spends a noticeably higher budget on attention (FP8 vs APEX's Q6_K) but pays for it by being flat and uniformly aggressive on every expert tensor (~4-bit NVFP4, no depth differentiation). APEX spends less on attention and uses the savings to differentiate within the experts — edge/near-layer experts stay at Q5_K/Q6_K while only the coldest middle-depth layers drop to IQ4_XS. Same rough bit budget, opposite allocation philosophy: NVFP4 is format-uniform, APEX is depth-aware.
Measured result: APEX wins on both axes. At a slightly smaller file size (23.51 GB vs 24.58 GB, 5.298 vs 5.539 bpw), APEX's depth-aware allocation gets a lower (better) perplexity than the flat NVFP4 allocation — see Results below. Spending the bit budget where the model's own structure says it matters (depth-band-differentiated experts) outperforms spending it uniformly, even though NVFP4-Fast's attention tensors get a strictly higher-precision format (FP8 vs Q6_K).
Results
| Quant | bpw | File size | PPL (wiki.test.raw) | Prompt tok/s (pp512) | Decode tok/s (tg128, no spec.) | Decode tok/s (w/ MTP n=2, APEX only) |
|---|
| APEX i-quality-torch | 5.298 | 23.51 GB | 6.9623 ± 0.046 | 2126.70 ± 23.25 | 71.06 ± 0.11 | not yet measured |
| NVFP4-Fast (unsloth) | 5.539 | 24.58 GB | 7.1236 ± 0.048 | 2475.64 ± 13.38 | 48.27 ± 0.12 | n/a (no MTP) |
APEX wins smaller, better, and faster (where it counts): 1.07 GB smaller, better perplexity, and 47% faster decode — the one place NVFP4-Fast leads is prefill (+16%), which matters less for interactive/agentic workloads than decode does.
Quality: APEX wins. 1.07 GB smaller, and perplexity is lower (better) by 0.16 — well outside the combined error bars (6.9623 ± 0.046 vs 7.1236 ± 0.048). Depth-aware allocation is beating uniform 4-bit-everywhere-in-the-experts at this bit budget.
Speed: it splits by phase, and the split is not what bpw alone would predict. NVFP4-Fast has faster
prompt processing (2475 vs 2127 t/s, +16%), but APEX has substantially faster
decode (71.06 vs 48.27 t/s,
+47%) — the more practically important number for interactive/agentic use, since that workload is decode-dominated, not prefill-dominated. (The MTP n=2 row above is a separate, not-yet-measured number — the ~45 tok/s median seen in the
agentic coding benchmark is a
different metric, output-tokens-over-full-agentic-wall-clock including tool calls and repeated prompt reprocessing, and isn't directly comparable to this decode-only figure.)
This is llama.cpp's current NVFP4 kernel, not a property of the NVFP4 format. Its prefill-favoring, decode-flat-to-regressing pattern is documented upstream across multiple PRs on multiple Blackwell GPUs —
PR #21074 (generic MMQ kernel, RTX 5090): prefill up 1.27x–3.89x, decode flat at 0.99x–1.00x, on
every tested model;
PR #22196 (Blackwell tensor-core path, GB10 benchmark on Nemotron-3-Super-120B): prefill +20.8%, decode −0.8%. The kernel author states it directly:
"I struggled to find a good balance using the existing generic kernels... no other combination tried gave as significant prefill increases without also reducing tg [decode]." A genuine Blackwell MMA kernel to close the decode gap is flagged as future work, not yet landed. Maintainers separately note decode is memory-bandwidth-bound and NVFP4
should decode at roughly parity with same-size Q4_K in principle (
discussion #23627) — its real edge is meant to be the prefill/compute-bound case, from skipping a dequant step via FP4 tensor cores.
Our measured gap (47% slower decode) is larger than the ~1% seen in those reference benchmarks, plausibly because Qwen3.6-35B-A3B is a 3B-active MoE at batch=1 — many small per-expert matmuls, exactly the low-occupancy/high-register-pressure regime the kernel author flagged as the cost of the prefill optimization. Roofline sanity check: GB10's real memory bandwidth is ~273 GB/s; at ~2 GB of active weight read per token (3B active × ~5.5 bpw), that implies a ~130 tok/s ceiling. The k-quant result (71 tok/s) reaches ~55% of that roofline — normal for llama.cpp; NVFP4 (48 tok/s) reaches ~37% — consistent with genuine extra per-token kernel overhead on this architecture, not measurement noise.
PPL methodology: llama-perplexity against wiki.test.raw, 512-token chunks, same corpus/chunking for every quant. Speed methodology: llama-bench -p 512 -n 128, no speculative decoding, measured directly on this hardware (DGX Spark GB10, aarch64, Blackwell — llama.cpp has native NVFP4 CUDA kernels with a Blackwell-specific tuning config, so these are real GPU-kernel numbers, not a CPU-dequant fallback).
Reproduce
1# Convert an NVFP4 (compressed-tensors, mixed-precision) checkpoint to GGUF.
2# Requires a llama.cpp build with the two conversion fixes below.
3python3 convert_hf_to_gguf.py <local-hf-checkpoint-dir> \
4 --outfile <name>.gguf --outtype auto
5
6# PPL
7llama-perplexity -m <name>.gguf -f wiki.test.raw -c 512
8
9# Speed (no speculative decoding — format-only comparison)
10llama-bench -m <name>.gguf -p 512 -n 128
11
12# Speed (APEX torch quant, with native MTP self-speculation)
13llama-server -m Qwen3.6-35B-A3B-APEX-i-quality-torch.gguf \
14 --ctx-size 8192 --spec-type draft-mtp --spec-draft-n-max 2
Converter fixes required for mixed-precision compressed-tensors NVFP4
conversion/base.py (llama.cpp) only handled a single compressed-tensors config group. Two fixes were needed to support genuine mixed-precision checkpoints (one FP8 group + one NVFP4 group, as used by unsloth's Qwen3.6 NVFP4 releases):
- Multi-group detection: the NVFP4-checkpoint detection used
all(...) over config groups where any(...) was correct, and the non-NVFP4 group (FP8 attention/lm_head) had no dequant path at all — added one that reads the FP8 group's config and dequantizes it the same way the single-group float-quantized case already does.
- FP8/NVFP4 tensor misrouting:
lm_head's FP8 per-channel scale has shape [out_features, 1] — technically 2D, which is indistinguishable from a genuine (if degenerate) single-block NVFP4 scale by ndim alone. This routed lm_head into the NVFP4 repack path and crashed on a shape mismatch. Fixed by additionally requiring scale.shape[-1] > 1 (a real per-block NVFP4 scale always covers more than one block per row; FP8's per-channel scale never does).
Attribution & license
- Base model: Qwen/Qwen3.6-35B-A3B, Apache-2.0.
- NVFP4 quant: unsloth/Qwen3.6-35B-A3B-NVFP4-Fast — see its model card for license terms.
- APEX quant + this comparison: MIT, this repo.
