A GPTQ W4A16 (4-bit weight, 16-bit activation, group size 128) quantization of Qwen/Qwen3-4B-Instruct-2507, produced with llm-compressor. Calibrated on 512 sequences of pile-10k at seqlen 2048.

Quality (measured)

wikitext-2-raw-v1, ctx 2048, scored under transformers, paired per chunk:

model	perplexity	vs BF16	size
BF16 (base)	10.0216	—	7.5 GB
GPTQ W4A16 (this)	10.3261	+3.04 %	2.48 GB
AWQ W4A16	10.5907	+5.68 %	3.21 GB

At 4-bit, expect a real quality cost (+3 %) in exchange for 3× smaller than BF16. In this study GPTQ beat AWQ (+3.0 % vs +5.7 %, paired p = 3e-35) and was the more robust method on 8 GB Blackwell — see the benchmark for the calibration caveat.

Speed & VRAM (RTX 5070 Laptop, 8 GB, sm_120)

metric	value
VRAM to load	~2.7 GB (incl. ~1.3 GB CUDA context)
weights on disk	2.48 GB
decode speed	58.0 tok/s (single stream, vLLM)

The leanest and fastest of the three quants — smallest VRAM footprint and fastest decode (4-bit means fewer weight bytes to read per token). The choice when VRAM is tight and a few % perplexity is acceptable.

Usage (vLLM)

python

from vllm import LLM
llm = LLM("Subalzt/Qwen3-4B-Instruct-2507-GPTQ-W4A16")
print(llm.generate("Write a Python function for binary search.")[0].outputs[0].text)

Provenance & reproducibility

Full method, calibration ablation, and comparisons: github.com/Subalzt/qwen3-4b-quantized-fp8-gptq.

License

Apache-2.0, inherited from the base model. Attribution: Qwen team, Qwen/Qwen3-4B-Instruct-2507.