Llama-2-7b quantized to 4-bit weights / 16-bit activations with KronQ (Kronecker-factored Hessian quantization). Weights are stored packed int4 (~3.8 GB vs 13 GB fp16); a fused dequant + bidirectional-incoherence (BiIP) CUDA kernel unpacks them on the fly at inference.

Results (WikiText-2, seqlen 2048)

Perplexity: 5.56

Zero-shot accuracy:

PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	OBQA	Average
78.67	72.39	44.88	75.17	68.43	77.37	44.40	65.90

(lm-evaluation-harness, 0-shot. acc_norm for PIQA/HellaSwag/ARC/OBQA, acc for WinoGrande/BoolQ.)

Usage

This is a KronQ-packed checkpoint (custom format: model.safetensors carries biip_w_codes/scale/zero + BiIP buffers, described in kronq_packed_config.json). Load it with the KronQ runtime:

bash

# clone https://github.com/Intelligent-Computing-Lab-Panda/KronQ and build the CUDA kernels, then:
python eval_pretrained.py meta-llama/Llama-2-7b-hf donghyunli/Llama-2-7b-KronQ-W4A16 --ppl --zs

Recipe

Per-channel asymmetric W4, weight-only (a_bits=16), --alpha 0.5, bidirectional incoherence processing (BiIP, Hadamard kernel), act_order. Calibrated on 128 WikiText-2 sequences. To reproduce the quantization yourself (instead of using these weights), see the KronQ repo's "Reproduce" path + the companion Hessian repo.

License

Derivative of Llama-2-7b — subject to the Llama 2 Community License.