The original Gemma 4 E4B IT model was developed by Google DeepMind and is released under the Apache 2.0 license. This quantized checkpoint keeps the same license metadata and links to Google's Gemma license page: https://ai.google.dev/gemma/docs/gemma_4_license

Quantization

Base model: google/gemma-4-E4B-it
Quantization tool: llmcompressor
Modifier: GPTQModifier
Scheme: W8A8
Weight quantization: 8-bit integer, per-channel, symmetric
Activation quantization: 8-bit integer, per-token, symmetric, dynamic
actorder: null
Calibration dataset: HuggingFaceH4/ultrachat_200k, split train_sft
Calibration samples: 512
Calibration max sequence length: 2048
Random seed: 42

Multimodal encoders, lm_head, Gemma 4 AltUp/Laurel modules, and per-layer embedding projection paths were excluded from quantization and kept in BF16. The language-model attention and MLP Linear layers were quantized.

See recipe.yaml and config.json for the serialized quantization metadata.

Usage

python

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "lsm0729/gemma-4-E4B-it-W8A8-INT8-GPTQ"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

For vLLM-based evaluation in this environment, the quantized checkpoint was tested with linear_backend=triton.

bash

lm-eval run \
  --model vllm \
  --model_args 'pretrained=lsm0729/gemma-4-E4B-it-W8A8-INT8-GPTQ,dtype=auto,add_bos_token=false,max_model_len=15000,gpu_memory_utilization=0.5,enable_chunked_prefill=True,enforce_eager=True,trust_remote_code=True,linear_backend=triton' \
  --tasks mmlu_llama \
  --batch_size auto \
  --num_fewshot 5 \
  --apply_chat_template \
  --fewshot_as_multiturn

Evaluation Notes

The base model card reports Gemma 4 E4B MMLU Pro performance of 69.4%. This repository is a post-training quantized derivative; benchmark scores can differ from the original BF16 checkpoint depending on the harness, prompt template, thinking mode, backend, and runtime settings.

Provenance

Original model: https://huggingface.co/google/gemma-4-E4B-it
Original authors: Google DeepMind
License: Apache 2.0
License link: https://ai.google.dev/gemma/docs/gemma_4_license
Quantized by: lsm0729