
Method	GPTQ (Hessian-based error compensation)
Scheme	W4A16 (INT4 weights, FP16 activations)
Format	pack-quantized (compressed-tensors)
Group size	128
Dampening	0.01
Act order	static
Calibration	512 samples from ultrachat_200k, 2048 seq len
Ignored layers	lm_head, vision/audio embedders
Original size	~24 GB
Quantized size	~9.2 GB

Benchmark Results (GSM8K, 5-shot)

Variant	GSM8K (flexible)	GSM8K (strict)	Size	Compression
Baseline BF16	87.57%	86.58%	~24 GB	1.0x
FP8 Dynamic	87.95%	86.96%	~15 GB	1.6x
W8A16 INT8	86.96%	86.13%	~15 GB	1.6x
W4A16 GPTQ	85.29%	84.00%	~9.2 GB	2.6x

Hardware: NVIDIA H100 80GB HBM3, vLLM 0.25.1

Usage with vLLM

python

from vllm import LLM, SamplingParams

model = LLM("soyrsoyr/gemma-4-12b-it-W4A16-GPTQ")
output = model.generate("Hello, world!", SamplingParams(max_tokens=100))

How It Was Made

python

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head", "re:.*embed_vision.*", "re:.*embed_audio.*", "re:.*vision_embedder.*"],
    dampening_frac=0.01,
    actorder="static",
    block_size=128,
)
oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=2048, num_calibration_samples=512)

Quantized with llm-compressor v0.12.0.