~65 GB bf16 → 19.2 GiB. The multi-token-prediction head and the vision tower are
both retained.
What was quantized
400 linear layers to NVFP4 (block size 16, FP8 scales):
Group
Modules
Quantized
MLP gate_proj / up_proj / down_proj
64 each
yes
Full attention q/k/v/o_proj
16 each
yes
Gated DeltaNet in_proj_qkv, in_proj_z, out_proj
48 each
yes
Gated DeltaNet in_proj_a / in_proj_b
48 each
no
Gated DeltaNet conv1d
48
no
Vision tower (model.visual.*)
333 tensors
no
lm_head
1
no
MTP head (mtp.*)
15 tensors
no
Qwen3.8-27B is a hybrid stack — 64 layers of
3× (Gated DeltaNet → FFN) + 1× (Gated Attention → FFN). The DeltaNet decay and beta
projections (in_proj_a / in_proj_b) are low-rank and precision-sensitive, so they are
left at bf16 along with the causal conv1d.
Fused-layer constraint (important if you re-roll this yourself)
vLLM does not instantiate the DeltaNet input projections separately. It fusesin_proj_qkv + in_proj_z into a single MergedColumnParallelLinear named
in_proj_qkvz, and in_proj_b + in_proj_a into in_proj_ba. Every shard of a fused
layer must share one precision, or loading aborts during model construction — before a
single weight is read:
ValueError: Detected some but not all shards of
language_model.model.layers.0.linear_attn.in_proj_qkvz are quantized.
All shards of fused layers to have the same precision.
So in_proj_z must be quantized together with in_proj_qkv, even though it is a gate.
Excluding in_proj_a and in_proj_b is fine because they are excluded together, which
leaves in_proj_ba uniform. The same rule applies to qkv_proj and gate_up_proj.
This checkpoint has been verified to satisfy that constraint: every fused group is
internally single-precision, checked per parent module.
exclude_modules naming
exclude_modules is matched against vLLM's module prefixes, by exact string equality
first. Recent transformers emits the checkpoint hierarchy as model.language_model.…,
whereas vLLM builds language_model.model.… — so a config exported verbatim will silently
fail to match, and vLLM will try to quantize layers that have no scales. The exclusion list
here is written in both conventions, and was validated by running vLLM's own
is_layer_skipped / is_layer_excluded over every module in the checkpoint and confirming
its decision matches whether that module actually carries weight_scale tensors.
Calibration
256 samples from garage-bAInd/Open-Platypus,
batch size 16, max sequence length 1024, max calibration. No fine-tuning, no additional
training data.
The chat template opens a <think> block by default; pass enable_thinking=False to
apply_chat_template for direct answers. Qwen's recommended sampling is
temperature=1.0, top_p=0.95, top_k=20.
Note that transformers cannot load this checkpoint directly — NVFP4 packs two 4-bit
values per byte, so weights are stored at half width and a plain from_pretrained will
report shape mismatches. Use a runtime that understands modelopt_fp4.
Tool calling
The chat template emits tool calls in Qwen's XML dialect
(<tool_call><function=name><parameter=k>v</parameter></function></tool_call>), so vLLM
needs the matching parser. Without both flags, any client sending tool_choice: "auto"
gets 400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set:
With --reasoning-parser qwen3, a response truncated inside the <think> block
(finish_reason: "length") comes back with bothcontent and reasoning_content
empty — the parser needs the closing </think> before it will emit anything. This looks
alarmingly like a corrupted checkpoint but is purely a budget artifact. Either give
thinking mode enough headroom (2500 tokens was still not always enough for a verbose
"explain in detail" prompt) or set enable_thinking=False.
Deployment: 2× NVIDIA DGX Spark (GB10), TP=2
Verified serving on a pair of DGX Spark GB10 nodes joined by a direct 200 Gb/s QSFP link,
tensor-parallel across the two, one GPU per node:
First run on a cold FlashInfer cache is far slower — budget 25–30 min. Raise the NCCL
store timeout (300 s default is not enough for multi-node JIT) or rank 1 will drop out
mid-compile while rank 0 is still building kernels.
Single-user latency is fine (~192 ms first token, ~48 ms per subsequent token). Under
8-way concurrency total-token throughput scales 3.6×; TTFT balloons because prefills
queue against --max-num-seqs 4 / --max-num-batched-tokens 8192, but decode ITL barely
moves — the ceiling is batch admission, not compute. Raise --max-num-seqs if you need
lower TTFT under bursts.
Both nodes must hold the same checkpoint revision. vLLM resolves the repo id to a
local snapshot path per node; if one node's HF cache is a revision behind, each rank
silently loads different weights and the run hangs in distributed init rather than
reporting a mismatch. Check refs/main on both.
Prefix caching puts the Mamba/DeltaNet cache in align mode, which vLLM flags as
experimental for this architecture. Drop --enable-prefix-caching first if you see
output corruption.
The MTP head ships in the checkpoint but is not loaded unless you configure speculative
decoding; speculative_config=None leaves model-mtp-grafted.safetensors unused.
Refusal behaviour — inherited, not re-measured
The base checkpoint's author reports 12/100 refusals vs 98/100 for stock Qwen3.8-27B
on the test split of mlabonne/harmful_behaviors,
measured in non-thinking mode, using Heretic
(200-trial search co-minimizing refusal count against KL divergence from base).
Those numbers describe the bf16 source, not this quantization. The refusal edit lives
in o_proj and down_proj, which are exactly the tensors compressed here, so the effect
could in principle be attenuated. The bf16 source was spot-checked as compliant before
quantization.
Refusals are reduced, not eliminated, in the source model.
