Qwen3.8-27B-MTP-NVFP4

• MTP draft head preserved in bf16 and wired into the index — speculative decoding works out of the box (method: qwen3_5_mtp).
• Vision tower, lm_head, DeltaNet conv1d, and the MTP head are kept in bf16; everything else is NVFP4 W4A4.
• Quantized GPU-resident: the full bf16 model dispatched across 8× RTX PRO 2000 Blackwell (16 GB) with device_map=auto — calibration itself took 68 seconds (32 samples × 8192 tok).
• Serves on SM120 (Blackwell) with vanilla vLLM v0.22.0 — compressed-tensors auto-detected, no --quantization flag.

Measured throughput (TP=4, 128k ctx, KV fp8, RTX PRO 2000 Blackwell ×4)

Serve (vLLM v0.22.0)

⚠️ Gotchas

• The 15 mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance and slower than no MTP.
• W4A16 (NVFP4A16) does not serve on this architecture in vLLM 0.22 (gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.
• This is a reasoning model, and reasoning_effort defaults to xhigh. Give it max_tokens ≥ 4096, and for long-form work send chat_template_kwargs: {"reasoning_effort": "medium"} — at the default effort the <think> phase can consume the entire budget and emit no answer at all (I saw exactly that on long single-file HTML generation: 32,768 tokens spent thinking, zero content). A 9-case gate (French/English long-form, temperature 0 and 0.7) passes at both xhigh and medium on this base model, but medium keeps thinking to ~1k characters instead of ~5–9k, so it is the safer default. These modes are three system prompts rather than three capability levels: xhigh injects "think carefully, validate key assumptions, consider alternatives", low injects "keep your thinking brief", and medium injects nothing at all. Measured <think> length follows suit (a few hundred chars / ~1k / 5–9k) with no quality difference on short verifiable tasks (96 runs).
• Text-only serving shown above (--limit-mm-per-prompt '{"image":0,"video":0}' to be explicit); the vision tower is shipped in bf16 but multimodal serving was not benchmarked here.

Recipe