[!IMPORTANT] Superseded by xv0y5ncu/gemma-4-E4B-it-GLQ-trellis-3inst-4bpw. the trellis (TCQ) codebook has been the recommended format since v0.7 — same bit-rate, faster single-stream decode, and better quality at low bpw.

vllm serve xv0y5ncu/gemma-4-E4B-it-GLQ-trellis-3inst-4bpw --quantization glq

These weights remain available and unchanged, so existing references keep working.

GLQ (4.0 bpw uniform) quantization of google/gemma-4-E4B-it.

🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.

Property	Value
Average bpw	4.0
Quantization scheme	uniform
File size	6.14 GB (2.43× smaller than bf16)
gsm8k limit=50 chat+thinking strict-match	94% (bf16 baseline: 86%)
Recommended max_new_tokens for thinking mode	16K

License

License: Apache 2.0 — see https://ai.google.dev/gemma/docs/gemma_4_license

Original model: https://huggingface.co/google/gemma-4-E4B-it

Derivative quantization work; the base Gemma 4 model and the GLQ tooling (https://github.com/cnygaard/glq) are both Apache 2.0. Quantizing the weights does not change the license.

Method

This is an E8 lattice quantization (GLQ) of the original bf16 weights:

E8 lattice shell codebook (65536 entries, 8-D blocks)
Randomized Hadamard Transform (RHT) for input/output rotation
LDLQ (block LDL decomposition) feedback during encoding
N-stage residual quantization for ≥3 bpw layers
Mixed-precision allocation via Hessian-trace-derived sensitivity proxy (where applicable)

Source code, kernels, benchmarks: github.com/cnygaard/glq

Usage (HuggingFace transformers)

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import glq, glq.hf_integration  # registers the GLQ quantization config

model_id = "xv0y5ncu/Gemma-4-E4B-it-GLQ-4bpw"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="cuda", dtype=torch.bfloat16,
)

# For thinking-mode (math reasoning, multi-step problems):
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is 12 * 17?"}],
    tokenize=False, add_generation_prompt=True,
    enable_thinking=True,
)
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=16384, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=False))

Installation

pip install glq "transformers>=5.13.1,<5.15"

Pin transformers below 5.15 — 5.15.0 moved gemma-4 to a per-layer config, so config.head_dim raises and vLLM dies before loading a single weight. Verified on 5.14.1 with vLLM 0.27.1. Not specific to GLQ: stock bf16 gemma-4 fails the same way.

The custom CUDA kernel JIT-compiles on first use (~30s). Requires torch>=2.0, transformers>=5.0, CUDA 12.x.

KV cache compression on vLLM (glq v0.3.0+)

Independent of the weight quantization in this repo, glq v0.3.0 ships an opt-in E8-lattice KV cache compressor for vLLM. I measured it against the unquantized bf16 Gemma-4-E4B-it on a single RTX PRO 6000 Blackwell, vLLM 0.20.x, --enforce-eager:

Recipe	On-card per group	Cache vs fp16	Quality
`e8_relaxed:1`	4 bpw (int16 idx + bf16 scale)	4.02×	mmlu_pro 24/40 = 60 % at n=40 (Wilson 95 % CI [45 %, 73 %]) — not validated at usable sample sizes. NIAH passkey at ctx=16k: 9/10 (one digit-flip).
`e8_relaxed:2`	6 bpw (+ int16 idx2)	~2.67×	mmlu_pro 171/240 = 71.25 % at n=240 — bit-identical to the fp16 baseline (171/240). Output token count also matches to the byte. NIAH passkey at ctx=16 k / 32 k / 64 k / 130 k: 40/40 across the full 128 k context window (10 depth×seed trials at each context).

Activation:

bash

GLQ_KV_QUANT=e8_relaxed:2 \
GLQ_KV_E8_SIDECAR=1 GLQ_KV_E8_SIDECAR_READ=1 \
GLQ_KV_E8_COMPRESSED_ALLOC=1 \
GLQ_KV_E8_FUSED_GATHER=1 GLQ_KV_E8_FUSED_WRITE=1 \
vllm serve google/gemma-4-E4B-it --enforce-eager

Combining the two: stacking KV cache compression on top of this 4 bpw weight-quantized checkpoint should work (the hook is weight-quantization-agnostic) but I have not tested that combination end-to-end. Use the unquantized base model for the measured numbers above.

Note on `max_new_tokens`

Quantized variants use the model's chat template with enable_thinking=True and may require a larger thinking budget than bf16 to converge to the final answer. Empirically:

Variant	Recommended `max_new_tokens` for thinking
8bpw, 7bpw	2048
4bpw, 5bpw mix, 6bpw	8192-16384

Insufficient budget will cause the model to truncate mid-thought and the final answer may be missing. This is a generation-config concern, not a quality concern: when given enough budget, all variants converge to within stderr of bf16 quality on gsm8k.

lm-eval workaround for thinking mode

The lm-evaluation-harness does not currently expose enable_thinking=True through apply_chat_template. Until upstream support lands, monkey-patch the tokenizer before evaluating:

python

def patch_thinking(tokenizer):
    orig = tokenizer.apply_chat_template
    def patched(*args, **kwargs):
        kwargs.setdefault("enable_thinking", True)
        return orig(*args, **kwargs)
    tokenizer.apply_chat_template = patched

Links

Source: github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.

xv0y5ncu/Gemma-4-E4B-it-GLQ-4bpw

Gemma-4-E4B-it-GLQ-4bpw