~29 GB on disk (~27 GiB in VRAM). Near-lossless FP8 quantization with only 1.4% perplexity degradation vs BF16. Recommended GPU: NVIDIA RTX PRO 6000 (96 GB) or other GPUs with >= 48 GB VRAM.
For 32 GB GPUs (RTX 5090): Use the NVFP4 mixed-precision variant instead (~25 GB, fits with usable context on a single 5090).
Quantization Strategy
Uniform FP8 W8A8 dynamic quantization using llm-compressor v0.10.1, stored in the compressed-tensors format. No calibration data needed — weight scales are computed statically per-channel, activation scales are computed dynamically per-token at inference time.
All nn.Linear layers except those in the ignore list
Near-lossless: FP8 E4M3 preserves 3 mantissa bits with per-channel granularity
BF16 (unquantized)
lm_head, embed_tokens, DeltaNet small projections (in_proj_a, in_proj_b), all norms, visual encoder, MoE router gates
lm_head amplifies errors across 248K vocab; embed_tokens is a lookup table; DeltaNet low-rank projections are numerically sensitive; vision tower retained at full precision
Weight Breakdown
Component
Size
Precision
MLP
14.6 GB
FP8
DeltaNet attention
6.9 GB
FP8 + BF16
lm_head
2.5 GB
BF16
embed_tokens
2.5 GB
BF16
Softmax attention
2.1 GB
FP8
Visual encoder
0.9 GB
BF16
Total
~29 GB
Architecture
Qwen3.5-27B uses a hybrid DeltaNet + softmax attention architecture with full_attention_interval=4:
Only 16 of 64 layers require KV cache — the 48 DeltaNet layers use a fixed-size recurrent state that doesn't grow with sequence length. This gives ~4x more context capacity than a standard transformer of the same size.
KV Cache Budget
Per-token KV cache cost (only 16 softmax layers):
FP16: 4 KV heads x 256 dim x 2 (K+V) x 2 bytes x 16 layers = 64 KB/token
Note: On RTX 5090 (32 GB), the same Blackwell-specific vLLM issues that affect the NVFP4 variant also apply here. See the NVFP4 model card for details and tracking PRs. On GPUs with >= 48 GB VRAM, these issues are irrelevant.
FP8 W8A8 with Blackwell FP8 acceleration. Works on >= 48 GB GPUs; 32 GB Blackwell GPUs require upcoming vLLM fixes
transformers >= 5.3.0
Yes
Direct loading with device_map="auto"
SGLang
Yes
FP8 compressed-tensors supported
llama.cpp / GGUF
No
compressed-tensors FP8 format not supported
Hardware Requirements
Configuration
VRAM
Notes
Minimum
32 GB
Weights only, minimal context
RTX PRO 6000 (recommended)
96 GB
8 concurrent × 262K context with FP8 KV cache. Works out of the box with vLLM 0.17.0
2x RTX 5090
64 GB
Tensor parallel, full context
RTX 5090 (single)
32 GB
Not recommended — only ~4 GiB free for KV cache after model loading. Use the NVFP4 variant instead
Benchmark Results
Comparison against the BF16 source model. All benchmarks run on NVIDIA RTX PRO 6000 (96 GB) with vLLM 0.17.0, temperature=0.6 for generation tasks (Qwen recommended setting for thinking mode).
Benchmark
BF16 (54 GB)
FP8 (29 GB)
Delta
Perplexity (FineWeb-Edu, 100 samples)
6.6119
6.7026
+0.09 (+1.4%)
MMLU-Pro (500 samples)
54.0%
56.0%*
+2.0%
ARC-Challenge (1,172 samples)
97.6%
100%*
+2.4%
GSM8K Platinum (200 samples)
99.5%
—
—
AIME 2025 (30 problems)
40.0%
—
—
Throughput (single GPU)
17.8 tok/s
29.1 tok/s
+1.6x
*FP8 MMLU-Pro and ARC ran with 50 samples (quick mode); BF16 used full sample sizes. Full FP8 benchmarks will be updated.
Summary: FP8 quantization is near-lossless — perplexity degrades only 1.4% vs BF16, while throughput improves 1.6x from reduced memory bandwidth. For comparison, the NVFP4 variant (25 GB) shows 2.1% perplexity degradation but fits in 4 GB less VRAM.
