Llama-2-70b-hf quantized to 4-bit weights / 16-bit activations with KronQ (Kronecker-factored Hessian quantization). Weights are stored packed int4 (~33 GB); a fused dequant + bidirectional-incoherence (BiIP) CUDA kernel unpacks them on the fly.

Results (WikiText-2, seqlen 2048)

Perplexity: 3.40

Zero-shot accuracy:

PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	OBQA	Average
82.32	80.26	57.85	83.81	79.87	85.38	49.20	74.10

(lm-evaluation-harness, 0-shot. acc_norm for PIQA/HellaSwag/ARC/OBQA, acc for WinoGrande/BoolQ.)

Usage

KronQ-packed checkpoint. Load with the KronQ runtime:

python eval_pretrained.py meta-llama/Llama-2-70b-hf donghyunli/Llama-2-70b-KronQ-W4A16 --ppl --zs

Recipe

Per-channel asymmetric W4, weight-only (a_bits=16), --alpha 0.25, BiIP, act_order, raw H_G. 128 WikiText-2 calibration sequences.

License

Derivative of Llama-2-70b-hf — llama2 license.