RedHatAI/Qwen3.6-35B-A3B-NVFP4 with two more tensor groups quantised to NVFP4: the gated-delta-net (GDN/Mamba) backbone projections and the LM head. Weight-only (W4A16), via compressed-tensors config groups. Nothing else is changed.

Measured 2.26× single-stream decode on one NVIDIA DGX Spark (GB10) — 42.08 → 95.03 tok/s — with no measurable quality change on GSM8K.

Why this exists

The stock NVFP4 checkpoint quantises the MoE experts but leaves the always-active dense backbone in BF16. On a bandwidth-bound box that is the wrong half to compress: an audit of per-decode-step reads found ~80% of the bytes moved every step are the unquantised BF16 backbone (mamba_in_proj alone is 1.52 GB) while the active NVFP4 experts are only ~0.56 GB. Decode at batch-1 is DRAM-bound, so tok/s tracks bytes-read-per-token almost exactly — and this checkpoint attacks the term that actually dominates.

Measured

Single DGX Spark (GB10, sm_121a, 128 GB unified LPDDR5X, ~240 GB/s achievable read), vLLM 0.26.0, non-streaming, coherence-guarded.

	stock NVFP4	this checkpoint
single-stream decode, no spec	42.08	95.03	2.26×
aggregate @C=64	702.7	781.8	1.11×

GSM8K, lm-eval n=200, 5-shot, apply_chat_template, greedy:

	strict-match	flexible-extract
stock NVFP4	0.945	0.955
this checkpoint	0.950	0.965

No measurable degradation. (The small deltas are within noise at n=200; the honest claim is "no measurable change", not "better".)

Serving

bash

vllm serve <this-repo> \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-prefix-caching

Two flags worth knowing on this model family, both independent of this checkpoint:

--mamba-ssm-cache-dtype float16 — the config declares mamba_ssm_dtype: float32, and the GDN recurrent state is the only per-step term that never amortises across concurrent rows. At high concurrency it dominates: halving it measured +51% aggregate at C=512. Note the state is a recurrent accumulator, so this is gated on short-generation GSM8K only — long-context behaviour is unverified.
--max-num-seqs — vLLM's default is 128; raising it is worth a lot on this model if you are serving many streams.

How it was made

compressed-tensors config groups on the existing checkpoint — no vLLM patch, no retraining:

group_0 — stock W4A4 (unchanged, targets: ["Linear"])
group_1 — LM head, weight-only: re:.*lm_head$
group_2 — GDN projections, weight-only: re:.*linear_attn\.in_proj_qkv$, re:.*linear_attn\.in_proj_z$, re:.*linear_attn\.out_proj$, re:.*linear_attn\.in_proj_qkvz$

Two gotchas that cost real time

re:.*linear_attn\.in_proj_qkvz$ is mandatory. Scheme selection runs against the runtime fused module name. Listing only the per-shard names (in_proj_qkv, in_proj_z) matches nothing, and the module silently falls back to group_0's W4A4 — which has no calibrated input_global_scale, so the model loads and serves fluent garbage. If you fork this recipe, check for the absence of the global-scale warning in the serve log; that absence is the signal.
Do not add in_proj_ba to group_2. Its shards stay BF16, and a quantised scheme would demand packed tensors that do not exist.

Same reason group_1 uses a regex: targets: ["Linear"] matches by module class, and ParallelLMHead is not a LinearBase, so the head is not covered by group_0.

Scope and limits

Validated on GB10 / sm_121a with vLLM 0.26.0 only. Other hardware and engines are untested.
Quality gate is GSM8K n=200, which resolves roughly ±1.2 points — a 2–3 point regression would be invisible, and it says nothing about multilingual, long-context or code-correctness behaviour.
This is a requantisation of an already-quantised checkpoint, so quality figures are a lower bound relative to what a from-BF16 quantisation could achieve.
The visual tower and MTP head are carried over unchanged from the base repo.

Credit

All model work is upstream: Qwen for Qwen3.6-35B-A3B and RedHatAI for the NVFP4 conversion this builds on. This repository contributes only the two additional tensor groups and the measurements above. Apache-2.0, inherited from the base model.

eisbaw/Qwen3.6-35B-A3B-NVFP4-gdn4lm4

Qwen3.6-35B-A3B-NVFP4-gdn4lm4