An FP8 (E4M3) dynamic quantization of Qwen/Qwen3-4B-Instruct-2507, produced with llm-compressor (FP8_DYNAMIC, data-free). Runs on NVIDIA Blackwell's native FP8 tensor cores.

Quality — near-lossless (measured, not asserted)

wikitext-2-raw-v1, context 2048, 145 non-overlapping chunks, scored under transformers. Compared paired, per chunk (the correct test — the tiny error bars overlap but the paired statistic is decisive):

model	perplexity	vs BF16	size
BF16 (base)	10.0216	—	7.5 GB
FP8 (this)	10.0415	+0.20 % (paired t = 5.4, p = 3e-7)	4.85 GB
GPTQ W4A16	10.3261	+3.04 %	2.48 GB
AWQ W4A16	10.5907	+5.68 %	3.21 GB

The +0.20 % is statistically real but negligible — FP8 beats 4-bit methods by an order of magnitude in quality at ~2× their size. 8-bit quantization is essentially free; the quality cost is all in the jump to 4-bit.

Speed & VRAM (RTX 5070 Laptop, 8 GB, sm_120)

metric	value
VRAM to load	~5.5 GB (incl. ~1.3 GB CUDA context)
weights on disk	4.85 GB
decode speed	47.8 tok/s (single stream, vLLM)

FP8 is slower than the 4-bit quants (8-bit = more weight bytes read per token; decode is memory-bandwidth bound) but nearly lossless — the choice when quality matters and you have the VRAM.

Usage (vLLM)

python

from vllm import LLM
llm = LLM("Subalzt/Qwen3-4B-Instruct-2507-FP8")
print(llm.generate("Explain the TCP three-way handshake.")[0].outputs[0].text)

Provenance & reproducibility

Full method, code, and the comparison against GPTQ/AWQ/GGUF: github.com/Subalzt/qwen3-4b-quantized-fp8-gptq. Benchmarked on an RTX 5070 Laptop (8 GB, sm_120) under WSL2.

License

Apache-2.0, inherited from the base model. Attribution: Qwen team, Qwen/Qwen3-4B-Instruct-2507.