NVFP4 GGUF conversion of Qwen/Qwen3.5-122B-A10B for llama.cpp. 122B total parameters, ~10B active per token via 256 MoE experts (8 routed plus 1 shared).
Architecture per Qwen's spec: 48 transformer blocks arranged as 12 repetitions of (3 Gated DeltaNet layers, 1 Gated Attention layer), each block followed by an MoE FFN.
72 GiB main GGUF. 871 MiB mmproj sidecar carrying the vision tower. Targets a Blackwell or Ada-class card with 96 GiB of VRAM at moderate context lengths.
MTP sibling now available:Incarnas/Qwen3.5-122B-A10B-NVFP4-MTP-GGUF (shipped 2026-05-16) retains the MTP head for self-speculative decoding via llama.cpp's --spec-type draft-mtp path. +46% long-decode-thinking-on at +51% tok/J on the same Blackwell hardware, with verifier-driven output equivalence. Pick this artifact for vision workloads (--mmproj is incompatible with MTP on PR #22673); pick the MTP sibling for max decode rate on text-only workloads. Full bench comparison in the MTP section below.
Updates
2026-05-16 (evening) — Companion MTP variant released at Incarnas/Qwen3.5-122B-A10B-NVFP4-MTP-GGUF. Retains the MTP head for self-speculative decoding. +46% long-decode-thinking-on at +51% tok/J on Blackwell Pro 96 GiB; verifier-driven so output stays equivalent to AR at the same logits. Top of card + MTP section below rewritten to current reality (PR #22673 merged, sibling shipped, 122B-class numbers measured). Companion methodology repo + 16-page tech report + the local converter patch at bit-incarnas/nvfp4-mtp-conversions (v1.0).
2026-05-16 (later) — Linked the companion methodology repo at bit-incarnas/chat-vs-raw-methodology from the Methodology notes section below. The repo contains the chat-mode probe scripts (bench_chat_*.py), the exact lm-eval-harness invocation that produced the raw-mode numbers, two case studies (this artifact + Qwen3.6-35B-A3B Q8), and a reproduction recipe.
2026-05-16 — performance section rewritten to lead with chat-completions (/v1/chat/completions) numbers. The original lm-eval-harness raw (/v1/completions) numbers are retained below the chat-mode tables. The two methodologies disagree by 0-46pp depending on task — see Methodology notes for the breakdown. NIAH @ 256k added (18/18). Prefill curve added (8k → 229k). MTP-status section updated to reflect that the mtp22673 PR branch now has QWEN35MOE support at both the converter and runtime layers.
Vision tower preserved as a separate mmproj GGUF, loaded via the --mmproj flag in llama-server. Chat template embedded. Thinking mode is enabled by default per Qwen's chat template.
Mixed-precision layout
ModelOpt's hf_quant_config.json excludes the following modules from NVFP4. They stay at the source dtype (BF16 with FP8 scales on the quantized portions):
lm_head (output projection).
All linear-attention modules (the Gated DeltaNet layers, which carry SSM-style state).
All mlp.shared_expert_gate modules (MoE router gates).
Self-attention on the full-attention layers.
These layers carry the bulk of quantization sensitivity for this architecture, and ModelOpt's exclusion list reflects that.
KV cache: kv_cache_quant_algo: FP8 per the source config. NVFP4 group size: 16 elements. NVFP4 scale dtype: FP8 (E4M3).
The "two NVFP4 formats" gotcha
If you try to convert your own NVFP4 source and hit:
NotImplementedError: Quant format 'nvfp4-pack-quantized' for method 'compressed-tensors' is not yet supported
the cause is that two different toolchains both produce things called "NVFP4" and only one is supported by llama.cpp today.
Toggle thinking mode via enable_thinking in chat params if your client supports it.
Performance
Measured 2026-05-14 → 2026-05-16 on Blackwell Pro 96 GiB at llama.cpp commit 1ec7ba0c1, KV cache q8_0, LLAMA_ARG_FLASH_ATTN=on, BLACKWELL_NATIVE_FP4=1. Decode numbers are medians from multi-prompt sweeps.
Visual tokens occupy the preallocated KV cache without new allocation, so image input has negligible VRAM impact at 256k. Recurrent state for the Gated DeltaNet layers is fixed-size and does not scale with context, which is why VRAM only grows ~4 GiB from 32k to 256k context.
Decode and prefill
Metric
Value
Decode, thinking off, 200 tok output
80.1 t/s (stdev 0.18)
Decode, thinking off, 256k ctx
81.2 t/s
Decode, thinking on, 5-prompt sweep
79.8 t/s (stdev 0.09)
Prefill, short prompts (~30 tok)
~540 t/s
Cold load from NFS (10 GbE)
~4.5 min
Decode rate is the same whether thinking mode is on or off. What differs is the wall-clock time per answer: a thinking-on response takes 10-25 seconds instead of 2-5 seconds because total token volume per response expands (observed reasoning-to-content character ratio: 7.3x). Plan token budgets accordingly: thinking-on responses to multi-step reasoning prompts can exceed 4000 tokens.
BLACKWELL_NATIVE_FP4=1 did not materially change the decode rate on this rig (delta within noise). MoE decode here is memory-bandwidth-bound rather than compute-bound, so a compute-path optimization does not move the needle. The flag is still recommended on Blackwell because it engages the native tensor-core FP4 path; differences may show on workloads dominated by prefill compute that this measurement does not cover.
ModelOpt's exclude_modules list (Gated DeltaNet, MoE router gates, lm_head, self-attention on full-attention layers) keeps those modules at BF16 rather than NVFP4, which preserves quality at quant-sensitive sites and adds bandwidth load relative to uniform 4-bit quantizations. Decode speed reflects this tradeoff.
Prefill curve at long context
Prompt tokens
Prefill t/s
8,000
4,159
32,000
3,498
65,000
2,910
130,000
2,303
229,000
1,812
Clean monotonic decay. Prefill is compute-path bound at long prompts; per-token compute grows with KV state.
Routed through chat_template_kwargs.enable_thinking=false, max_tokens=4096. n shown per task. These reflect what most chat clients, agentic harnesses, and code-assist integrations actually receive.
Capability — lm-eval-harness raw (/v1/completions)
Community-comparable with standard HF leaderboard methodology. For tasks where raw and chat agree (gsm8k, HumanEval) the gap is within ~6pp; for tasks that rely on the chat template's framing (ifeval) the gap is large — see Methodology notes.
Task
Score
n
gsm8k strict-match
83%
100
HumanEval pass@1
96%
50
HumanEval+ pass@1
88%
50
MBPP+
87%
100
ifeval prompt-level strict
44%
200
NIAH @ 256k
18/18 probes pass across 8k → 256k context tiers, varied positions. Lexical-needle retrieval; harder associative-retrieval (NoLiMa-style) not yet measured on this artifact.
Quality rubric (deterministic 20-prompt sanity)
20/20 thinking-off. Ceiling-bound at this quality tier; sanity check rather than differentiator.
Vision rubric
Six synthetic test images covering geometric shapes, text recognition (OCR), color gradients, multi-quadrant layouts, object counting, and a blank-image edge case. Total inference time across all six: 11.7 seconds.
Test
Content
Result
Geometric
Triangle inside a square, with colors and outlines
Correctly identified all elements
Text recognition
Red octagonal stop sign with "STOP" lettering
Read text, identified shape and colors
Gradient
Horizontal blue-to-red gradient
Identified gradient direction and endpoint colors
Multi-quadrant layout
Four shapes in four quadrants, each a different color
Correctly assigned each shape to its quadrant with its color
Object counting
Three colored circles in a row
Counted correctly, listed left-to-right with colors
Blank-image edge case
Solid white image with no content
Correctly identified as blank, hedged about possible causes
All six pass. Rubric is small and synthetic; broader vision-quality evaluation pending.
Not measured
True cold-cache load from local NVMe (warm-cache reload is 13s; true cold requires drop_caches).
Aider polyglot at this quant (blocked on litellm model-registry workaround: model filename not in litellm's model_cost registry, default max_output_tokens clips low and truncates responses).
Broader vision rubric against natural images.
Chat-mode enable_thinking=true rerun of the capability sweep.
NoLiMa associative-retrieval at long context.
Methodology notes
Tasks that rely on the chat template's user/assistant role framing score very differently under /v1/chat/completions vs /v1/completions raw mode. On this artifact:
Task
Raw
Chat
Δ
gsm8k strict
83%
89%
+6pp
HumanEval pass@1
96%
96%
0pp
ifeval prompt-strict
44%
90%
+46pp
The gsm8k and HumanEval gap is within noise. These tasks present as next-token completion (5-shot Q:/A: patterns, function-completion prompts) that a 122B-class model pattern-matches well in either mode. The 46pp ifeval gap reflects that raw mode never engages the model's instruction-following machinery: without the chat template's role framing, the model treats the constraint prompt as text to continue, not instructions to follow.
The gap is task-dependent and quality-tier-dependent. On smaller models in this family the gap is larger across more tasks; on this 122B at NVFP4, gsm8k and HumanEval are tight while ifeval is not.
For community comparability with HF-leaderboard methodology, the raw table is the right reference. For chat-client and agentic-harness use cases, the chat-mode table is the right reference. They disagree most on the tasks that matter most for instruction-following.
The chat-mode probes used here are simple Python over urllib.request, no heavy dependencies. Full reproduction recipe, probe sources, and a second case study (Qwen3.6-35B-A3B Q8, where the gsm8k gap was 35→97pp under the same methodology shift) are published at bit-incarnas/chat-vs-raw-methodology.
MTP
The source safetensors include 785 tensors under the mtp.* prefix, a multi-token prediction module for self-speculative decoding. This GGUF does not retain those tensors. At the time of conversion (commit 1ec7ba0c1, 2026-05-11), the llama.cpp converter did not yet handle the mtp.* prefix for the QWEN35MOE arch, so MTP tensors were stripped.
As of 2026-05-15, PR #22673 has merged and the converter handles MTP for QWEN35MOE at both the converter layer (mtp.* tensor remap into NEXTN naming) and the C++ runtime (LLM_ARCH_QWEN35MOE_MTP model class implementing the draft head). A companion MTP-retained release is now available at Incarnas/Qwen3.5-122B-A10B-NVFP4-MTP-GGUF. Same NVFP4 source, same converter family, plus the MTP block at layer 48 (~6 GiB additional on disk).
Measured on the same Blackwell Pro 96 GiB rig (2026-05-16):
Test
This artifact (AR)
MTP sibling
Delta
Long-decode thinking-on (2048 tok)
79.77 t/s
116.45 t/s
+46.0%
Long-decode thinking-off (350 tok)
80.09 t/s
95.13 t/s
+18.8%
Short-decode (29 tok)
81.31 t/s
128.01 t/s
+57.4%
Long-cold-prefill (7906 tok)
4137 t/s
3386 t/s
-18.2%
VRAM @ 256k ctx
82.4 GiB
88.1 GiB
+5.7 GiB
tok/J (long-decode-think)
0.276
0.416
+51%
Draft acceptance (avg)
—
2.46 eff tok/decode
~73%
MTP is verifier-driven, so output is statistically equivalent to AR at the same logit profile. Gains are regime-dependent: thinking-on traces accept the most drafts (long-decode wins largest); terse outputs accept fewer; prefill takes a small hit because the drafter graph runs alongside the verifier. The MTP variant required one local converter patch for multimodal Qwen3.5-MoE NVFP4 sources — methodology repo + 16-page tech report + patch at bit-incarnas/nvfp4-mtp-conversions (v1.0).
Pick this artifact for vision workloads (--mmproj is incompatible with --spec-type draft-mtp on PR #22673). Pick the MTP sibling for max decode rate on text-only workloads. Generic draft-model speculative decode via --model-draft also remains an option if you want to pair this AR-only artifact with a smaller Qwen as the draft model.
(For reference: earlier 35B-class testing on Qwen3.6-35B-A3B Q8 showed regime-dependent behavior with short outputs gaining +35% and long outputs losing −11% at the 3B-active-parameter level. The 10B-active 122B above shows different mechanics — net gain across all regimes — because the higher active parameter count means bandwidth pressure dominates over draft-head overhead.)
llama.cpp version requirements
This GGUF was produced by llama.cpp commit 1ec7ba0c1 (2026-05-11). If you encounter loader errors on older builds, update to a recent llama.cpp release.
MATH-500 scorer prefers \boxed{...} content (balanced-brace parse); falls back to last-bold expression. Equivalence via sympy.parse_latex + simplify(a-b)==0, with integer/fraction normalization on top. ↩
