NVFP4 (W4A4), GPTQ-quantized quantization of Qwen/Qwen3.8-27B, produced with llm-compressor and saved in the compressed-tensors format for direct use with vLLM.

Weights and activations in NVFP4, NVIDIA's 4-bit floating-point format with a 16-element block scale. Unlike a round-to-nearest NVFP4 checkpoint, the weights here are chosen by GPTQ, which uses second-order (Hessian) information from calibration data to compensate for quantization error layer by layer. That costs nothing at inference time — the result is still a plain NVFP4 checkpoint. See the measured perplexity below rather than assuming a recovery benefit.

This runs on Blackwell's native FP4 tensor cores. In practice the gain shows up most in prefill (compute-bound, so 4-bit activations help — TTFT is roughly half that of the W4A16 variant) and in batched throughput. Single-stream decode is memory-bound and therefore close to W4A16, since both store the same FP4 weights.

Hardware: NVIDIA Blackwell (SM100 / SM120) for the accelerated path. On older GPUs vLLM falls back to a dequantize-and-emulate path that is functional but slower than BF16.

Checkpoint size: 28.6 GB (BF16 base is 55.6 GB, so 1.95x smaller).

Usage

vllm serve cloudnathan5/Qwen3.8-27B-NVFP4a4-GPTQ --max-model-len 32768 --max-num-seqs 512

--max-num-seqs matters on this architecture. 48 of the 64 layers use linear attention, and vLLM allocates one Mamba-style cache block per decode sequence. vLLM's default max_num_seqs=1024 can exceed the number of blocks that fit, and startup then fails during CUDA graph capture with max_num_seqs (1024) exceeds available Mamba cache blocks. Lower --max-num-seqs (512 is a safe starting point) or raise --gpu-memory-utilization. This is a property of the base model, not of quantization.

python

from vllm import LLM, SamplingParams

llm = LLM(model="cloudnathan5/Qwen3.8-27B-NVFP4a4-GPTQ")
out = llm.generate(
    ["Explain 4-bit quantization in two sentences."],
    SamplingParams(temperature=0.7, max_tokens=256),
)
print(out[0].outputs[0].text)

Single-user latency

Concurrency 1 (one request in flight — no batching), on a single NVIDIA RTX PRO 6000 Blackwell 96GB, vLLM 0.27.1. Prefix caching disabled and --ignore-eos set so prefill is never skipped and every generation is exactly 256 tokens. Median over 24 requests after 4 warmups, via vllm bench serve --max-concurrency 1.

input tokens	TTFT	inter-token latency	decode tok/s	BF16 base tok/s
1024	105 ms	20.4 ms	49.1 (1.87x)	26.2
4096	343 ms	20.5 ms	48.8 (1.87x)	26.1

This is single-stream interactive performance, not batched throughput; under concurrency the ranking between variants differs.

Quantization details

Calibrated on 256 samples of HuggingFaceH4/ultrachat_200k at 4096 tokens, chat template applied.

The following modules are left in their original precision:

pattern	reason
`lm_head`, `embed_tokens`	quantizing these costs accuracy for no speed benefit
`visual.*`	the vision tower is small and quantization-sensitive
`linear_attn.*`	the gated-delta / linear-attention state paths are numerically fragile at 4 bits and are not GEMM-bound
`mlp.gate`, `shared_expert_gate`	MoE routing weights
`mtp.*`	multi-token-prediction head — included in this checkpoint at original BF16 precision, so MTP speculative decoding works

Reproduce with quantize.py:

python quantize.py --model-id Qwen/Qwen3.8-27B --method nvfp4-gptq

Evaluation — perplexity

Token-level perplexity on wikitext-2-raw-v1 (test), 48 non-overlapping 4096-token windows (196,560 tokens scored), measured through vLLM with identical settings for both rows.

model	perplexity	vs BF16
this checkpoint	6.7733	+3.29%
Qwen/Qwen3.8-27B (BF16)	6.5574	—

This is token-level perplexity over a fixed window, which is not the same statistic as lm-eval's word_perplexity — compare it only against numbers produced the same way.

Caveats

Quantization is lossy. Validate on your own workload before production use.
The exclusion list above was derived from the architecture at release; if you fine-tune or otherwise alter module naming, re-derive it.