Views
No views yet
compressed-tensors) quantization of
Qwen/Qwen-AgentWorld-35B-A3B,
produced with llm-compressor for fast
inference with vLLM on NVIDIA Blackwell (sm120 / RTX PRO 6000).
--max-model-len 8192 --max-num-seqs 64 --gpu-memory-utilization 0.90, temperature=0):| Metric | Official BF16 | NVFP4 (this model) | Δ |
|---|---|---|---|
| Disk size | 66 GB | 24.96 GB | −62% |
| First-token latency (TTFT) | 35 ms | 32 ms | −8.6% |
| Single-stream decode | 157.9 tok/s | 184.1 tok/s | +16.6% |
| Concurrent throughput (N=16) | 1351.6 tok/s | 1430.5 tok/s | +5.8% |
temperature=0): 17×23, a factorial function, "why is the sky blue", and
echo $((6*7)) — all correct and equivalent to the BF16 reference. The NVFP4 build is faster
and ~1/3 the size with matching quality.compressed-tensors (format mixed-precision).self_attn.{q,k,v,o}_proj, GDN linear_attn.{in_proj_qkv,in_proj_z,out_proj}) → FP8 (block [128,128]).gate_proj/up_proj/down_proj) → NVFP4 (group size 16, fp8_e4m3 scales).lm_head, embed_tokens, router mlp.gate, shared expert, GDN state
(linear_attn.{A_log,conv1d,in_proj_a,in_proj_b}), first/last MoE layer experts, and the vision tower.moe_calibrate_all_experts=True) on
HuggingFaceH4/ultrachat_200k (256 samples, seq len 2048).language_model_only=true).
vLLM auto-detects compressed-tensors — no --quantization flag needed.1vllm serve kyaky/Qwen-AgentWorld-35B-A3B-NVFP4 \
2 --max-model-len 262144 --max-num-batched-tokens 2096 --max-num-seqs 256 \
3 --enable-prefix-caching --disable-custom-all-reduce --trust-remote-code \
4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml--max-num-batched-tokens 2096 (cache alignment);
head_dim=256 auto-selects the Triton attention backend.