Views
No views yet
max() of global scales, causing underflow)1vllm serve FusenAI/gemma-4-26B-A4B-it-NVFP4 \
2 --quantization modelopt \
3 --max-model-len 4096| Batch | Gen tok/s | Avg Latency |
|---|---|---|
| B=1 | 127 | 2.0s |
| B=4 | 383 | 2.7s |
| B=16 | 1,221 | 3.3s |
| B=32 | 2,071 | 3.9s |
| B=128 | 2,059 | 4.0s |
| Config | tok/s | Notes |
|---|---|---|
--enforce-eager | 18 | No CUDA graphs |
| Default (CUDA graphs) | 127 | 7x faster |
--kv-cache-dtype fp8 | 31 | 2x KV capacity but 4x slower (FlashInfer FP8 overhead) |
| Property | Value |
|---|---|
| VRAM (model weights) | 17.2 GiB |
| KV cache (BF16) | ~10 GiB / 43K tokens / 15x concurrency at 4096 ctx |
| KV cache (FP8) | ~10 GiB / 87K tokens / 30x concurrency at 4096 ctx |
| Size on disk | 17 GB |
| Quantization | NVFP4 (MLP/MoE only), BF16 (attention) |
| Group size | 16 |
1# Recommended (CUDA graphs enabled by default)
2vllm serve FusenAI/gemma-4-26B-A4B-it-NVFP4 \
3 --quantization modelopt \
4 --max-model-len 4096
5
6# For maximum KV cache capacity (2x tokens, slower per-request)
7vllm serve FusenAI/gemma-4-26B-A4B-it-NVFP4 \
8 --quantization modelopt \
9 --kv-cache-dtype fp8 \
10 --max-model-len 4096weight_packed -> weight (same data)weight_global_scale (divisor) -> weight_scale_2 (1/divisor)input_global_scale (divisor) -> input_scale (1/divisor)exclude_modulesweight: uint8 (packed FP4-E2M1, two values per byte)weight_scale: float8_e4m3fn (per-block scale, group_size=16)weight_scale_2: float32 (global scale, scalar)input_scale: float32 (activation global scale, scalar)weight: bfloat16 (standard dense weight)--quantization modelopt