NVFP4-quantized text-only sibling of Qwen/Qwen3.6-27B, with the MTP (Multi-Token Prediction) head restored in bf16 so speculative decoding actually works.

What's different from `sakamakismile/Qwen3.6-27B-NVFP4`

	This repo (`-Text-NVFP4-MTP`)	`Qwen3.6-27B-NVFP4`
Quantization format	`modelopt` (vLLM SM120 native path)	`compressed-tensors`
MTP head	Restored in bf16, working	Dropped during export → 0% draft acceptance
Vision tower	Stripped (text-only)	Present (kept for VLM use)
Suggested launch	with `--speculative-config`	without speculation

The original Qwen3.6-27B-NVFP4 is left untouched so existing users (~15K downloads) are not disrupted. This is a focused text-only sibling for users who want maximum speed and don't need vision input.

Why this exists

Two HF Discussion threads on the original repo prompted this:

#5 — slower than official FP8 on Blackwell — root cause is the compressed-tensors NVFP4 path being slower than modelopt on Blackwell SM120; this repo uses modelopt natively.
#7 — MTP not responding — AutoModelForCausalLM.from_pretrained does not load the MTP head, so it gets dropped during quantization, leading to 0% draft acceptance. This repo grafts the 15 mtp.* tensors (bf16) back into the quantized checkpoint and adds them to the quantization ignore list.

Recipe is adapted from osoleve/Qwen3.5-27B-Text-NVFP4-MTP — credit and thanks.

Reproduce this quantization

This model was produced by the open-source lna-lab/GGUF-to-NVFP4-SM120 pipeline — Lna-Lab's production line for converting Qwen3.5 / 3.6 / Gemma 4 checkpoints into modelopt-format NVFP4 + working MTP, ready for vLLM on Blackwell SM120. The exact script is src/quantize/qwen36_27b_text_mtp.py; the 5-step MTP graft recipe is documented in docs/MTP_GRAFT_RECIPE.md.

Quantization details

Base: Qwen/Qwen3.6-27B (bf16, 27.78B params, hybrid linear-attn + full-attn, 64 layers, 1 MTP layer)
Quantizer: nvidia-modelopt 0.43.0 with NVFP4_DEFAULT_CFG
Calibration: 20 samples from neuralmagic/calibration (LLM split), max_seq_len 8192
Ignored from quantization (kept in bf16):
- lm_head
- All model.visual.* (vision tower) — then physically deleted in the text-only build
- All *linear_attn.conv1d* (Mamba-style SSM convolutions, 48 of the 64 layers)
- All mtp.* modules (the 1-layer MTP head: 15 tensors total, ~850 MB bf16)
- Other defaults from NVFP4_DEFAULT_CFG: *router*, *mlp.gate.*, *block_sparse_moe.gate*, *output_layer*

Usage with vLLM (Blackwell, SM120)

Recommended production launch — 256K context · KV FP8 · MTP n=3 · max-num-seqs 2

bash

vllm serve sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP \
    --trust-remote-code \
    --quantization modelopt \
    --language-model-only \
    --max-model-len 262144 \
    --max-num-seqs 2 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.9 \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

This is the production launch on a single RTX PRO 6000 Blackwell — full 256K context, two concurrent slots, KV FP8 for concurrency headroom. The four flags that are easy to skip but matter:

--max-model-len 262144 — full 256K context (Qwen3.6 trained max).
--kv-cache-dtype fp8 — halves KV memory; lifts max concurrency at 256K from ~4× (BF16, won't fit) to 7.0× with the same VRAM. ~5–10 % per-token decode overhead, more than paid back by capacity.
--max-num-seqs 2 — load-bearing. --max-num-seqs 4 plus --kv-cache-dtype fp8 plus --speculative-config n=3 plus --max-model-len 262144 will silently OOM during cuda-graph capture on this build of vLLM (0.19.1rc1).
num_speculative_tokens: 3 — vLLM applies the single MTP layer (mtp_num_hidden_layers=1) recursively three times per draft pass. Per-position acceptance ~87 / 72 / 61 %, mean accepted-length ≈ 3.0 / 4.0. The qwen3_5_mtp handler is internally normalized to mtp (deprecated-name warning is harmless).

The mtp.fc weight is kept in bf16 in the safetensors (not NVFP4) — equivalent to the Lorbus-style "dequantize the fusion layer in the file" trick applied to NVFP4 instead of AutoRound. Side effect of the *mtp* ignore entry in the modelopt config, but load-bearing for the n=3 throughput.

Smaller-context launch (16K, no fp8) — fastest single-request decode

bash

vllm serve sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP \
    --trust-remote-code \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --language-model-only \
    --quantization modelopt \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

Verified throughput vs the family baseline (RTX PRO 6000 Blackwell, vLLM 0.19.1rc1)

Same 256K + KV FP8 + max-num-seqs 2 production launch, T = 0:

Repo	Format	MTP	Single (S/M/L)	2-parallel agg (M/L)	vs baseline
`Qwen3.6-27B-NVFP4` (the family baseline)	`compressed-tensors`	❌	56 / 59 / 59	119 / 119	1.0×
`Qwen3.6-27B-Text-NVFP4-MTP` (this repo)	`modelopt`	✅ n=3	104 / 98 / 100	189 / 207	1.67× / 1.74×
`Carnice-V2-27b-NVFP4-TEXT-MTP`	`modelopt`	✅ n=3	107 / 98 / 102	193 / 194	1.68× / 1.63×
`Huihui-Qwen3.6-27B-abliterated-NVFP4-TEXT-MTP`	`modelopt`	✅ n=3	117 / 96 / 101	203 / 183	1.65× / 1.54×
`Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP` (VLM)	`modelopt`	✅ n=3	118 / 97 / 100	183 / 198	1.66× / 1.66×

(S = 50-token, M = 350-token, L = 700-token decodes.)

This repo lands at 1.74× the baseline's 2-parallel aggregate throughput on long-form decodes (207 tok/s vs 119) — the gain comes from two compounding fixes:

modelopt NVFP4 export — vLLM's native fast path on Blackwell SM120, vs the compressed-tensors slow fallback the baseline lands on.
bf16-restored MTP head + num_speculative_tokens=3 — single MTP layer applied recursively for ~1.9× decode multiplier via speculative decoding.

KV cache size at 256K + fp8: 491,200 tokens → max concurrency 6.98× per request at full 256K. Mean acceptance length 1.93 / 2.0 at n=1, ~3.0 / 4.0 at n=3.

Smaller-context single-request bench (16K, no fp8)

Prompt	Tokens	n=1 tok/s	n=3 tok/s
Short (50 tok)	50	~71	132.5
Medium (350 tok)	350	~85	105.5
Long-form (700 tok)	700	~85	106.5

GPU memory at load: ~15 GB.

Hardware target

Built and tested on NVIDIA RTX PRO 6000 Blackwell (SM120). Should also work on RTX 5090 and other Blackwell consumer/workstation cards with sufficient VRAM (the model is roughly 14 GB after NVFP4 + ~850 MB of bf16 MTP/conv1d/lm_head).

Acknowledgements

osoleve — for the MTP-restoration recipe on Qwen3.5
Qwen — for the base model
nvidia-modelopt team
The reporters of Discussions #5 and #7 — for catching this cleanly

Support the Base Model Authors

If you find this model useful, please consider supporting:

Qwen Team (original model): Star the Qwen repo

License

This model inherits the Apache 2.0 license.

sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP

Qwen3.6-27B-Text-NVFP4-MTP

What's different from sakamakismile/Qwen3.6-27B-NVFP4