This repository hosts W4A16 INT4-quantized versions of google/gemma-4-E4B-it, a multimodal mixture-of-experts model supporting text, vision, and audio inputs. Two quantized variants are available:
Note on MTP / Speculative Decoding:
If you want to use the official speculative decoding assistant model (google/gemma-4-E4B-it-assistant) for MTP support, it is recommended to use the vllm/vllm-openai:gemma4-0505-cu129 Docker image, which includes newer Gemma 4 support and decoding patches.
Due to INT4 quantization, the assistant acceptance rate may be lower compared to the original unquantized google/gemma-4-E4B-it model.
Note: Only the language model (LM) layers are quantized to INT4. The vision tower, audio tower, and multimodal projectors are kept at full precision (BF16) to preserve multimodal quality.
Quantization Details
Parameter
Value
Base model
google/gemma-4-E4B-it
Quantization scheme
W4A16 (INT4 weights, BF16 activations)
Group size
128
Symmetric
Yes
Calibration samples
256
Sequence length
2048
Non-LM modules
Kept at FP32 (vision, audio, projectors)
Quantized layers
All LM linear layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, per_layer_input_gate, per_layer_projection)
AutoRound mode
(iters=800) — required for Gemma 4 compatibility
Hardware used
NVIDIA A100 80GB PCIe
Framework
PyTorch 2.10.0 + CUDA 12.8
Model Architecture
Gemma 4 E4B is a multimodal MoE model (Gemma4ForConditionalGeneration) with:
Text backbone: 42-layer Gemma4TextModel with 2560 hidden dim, mixed local/global attention
Audio tower: 12-layer Gemma4AudioModel with conformer-style layers (unquantized)
Vocabulary size: 262,144 tokens
Usage
vLLM Inference
The recommended way to serve this model is via the official vllm/vllm-openai:gemma4 Docker image, which ships vLLM v0.19.1 with the latest Transformers patches required for Gemma 4.
The max_soft_tokens parameter controls how many visual tokens are allocated per image. Higher values give richer image representations at the cost of context length and throughput.
max_soft_tokens
Detail level
Recommended use
70
Minimal
Fast throughput, simple images
140
Low
Charts, diagrams
280
Medium (default)
General-purpose
560
High
Dense scenes, documents
1120
Maximum
Fine-grained visual detail
Pass it via --mm-processor-kwargs '{"max_soft_tokens": <value>}'.
OpenAI-compatible API call
Once the server is running, query it like any OpenAI-compatible endpoint:
python
1from openai import OpenAI
23client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")45response = client.chat.completions.create(6 model="Gemma-4-E4B-it",7 messages=[8{9"role":"user",10"content":[11{"type":"text","text":"Describe what you see."},12{"type":"image_url","image_url":{"url":"https://..."}},13],14}15],16 max_tokens=512,17)18print(response.choices.message.content)
The quantized model was then exported in both AutoRound and GPTQ formats and pushed to Hugging Face Hub.
Limitations & Notes
AutoRound mode (iters=800) is used for full AutoRound optimization.
Some layers with shapes not divisible by 32 are skipped during quantization (minor precision impact).
Multimodal (vision/audio) capabilities are fully preserved as those towers are not quantized.
🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.
Special thanks to OLAF-OSS and the contributors of the gemma4-vllm project for their excellent work in enabling Gemma 4 support with vLLM. Their QUANTIZE.md guide was extremely helpful in understanding the correct quantization approach for gemma-4-E2B-it.
At the time of writing, the original repository appears to be unavailable or removed. To ensure reproducibility, I have documented the full quantization workflow used for this model in my own repository.
The full quantization process used to produce these models is documented here:
📓 auto_round_Gemma4-E4B.ipynb
