import glq.hf_integration
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "xv0y5ncu/SmolLM3-3B-GLQ-6bpw",
    device_map="cuda",
    dtype="float16",
)
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Requirements

transformers >= 5.0 (4.x has a weight loading bug that breaks small GLQ models)
torch >= 2.0
glq >= 0.2.8 (pip install glq)

How it works

GLQ uses the E8 lattice codebook (65536 entries) with Randomized Hadamard Transform (RHT) for incoherence processing. At 6bpw, three codebook stages are used:

Primary E8 codebook (16 bits per 8-weight block = 2 bpw)
Secondary E8 codebook on residual (16 bits = 2 bpw)
Tertiary E8 codebook on residual (16 bits = 2 bpw)

Block-diagonal FHT decomposes non-power-of-2 dimensions into sums of powers of 2, eliminating padding waste. For SmolLM3-3B (hidden_size=2048), dimensions are already power-of-2 so there is zero overhead.

GPU Memory

	bf16	GLQ 6bpw
Model weights	6.1 GB	2.5 GB
Inference (B=1)	~6.5 GB	~3 GB

License

Apache 2.0 (same as base model)

🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.