Evaluated using Jensen-Shannon Divergence with forced-decode on 2,590 tokens across 12 prompts (chat, reasoning, code, creative). Reference model: FP8 block-wise quantization of the same base model.
The TRTLLM fused MoE kernel has a hardcoded SM 100 check that fails on SM 120. Add this environment variable:
VLLM_USE_FLASHINFER_MOE_FP4=0 vllm serve ...
This falls through to the VLLM_CUTLASS MoE backend which works on SM 120.
Performance
Metric
Value
VRAM
21.88 GiB
Decode throughput
~130 t/s (single request)
Hardware tested
NVIDIA RTX PRO 6000 Blackwell Max-Q (SM 120)
MoE backend
VLLM_CUTLASS
Dense backend
VLLM_CUTLASS
vLLM version
0.17.0rc1
Known Issues
SM 120 TRTLLM MoE kernel: Requires VLLM_USE_FLASHINFER_MOE_FP4=0 (see above)
No MTP: Heretic abliteration strips MTP weights. Do not use speculation flags. MTP head distillation from the original Qwen model is a planned future improvement.
Thinking model: Very verbose chain-of-thought. Use a system prompt to disable or set high max_tokens.
Quantization Tooling
Calibrated with llm-compressor PR #2383 (Qwen3.5 support, not yet merged). Post-processing required to fix checkpoint format for vLLM (config grafting, tensor merging, ignore list prefix correction).
