Gemma 4 12B IT - W4A16 AutoRound Quantized Variants
This repository hosts 4-bit weight, 16-bit activation (W4A16) quantized variants of google/gemma-4-12B-it. The models were quantized using Intel's AutoRound framework tailored specifically for the architectural requirements of the Gemma 4 family.
Vishva007/gemma-4-12B-it-W4A16-AutoRound-AWQ (AWQ format conversion)
Vishva007/gemma-4-12B-it-W4A16-AutoRound-GPTQ (GPTQ format conversion)
Architectural Advantage: Gemma 4 12B Unified
The Gemma 4 12B Unified model features a ground-breaking encoder-free multimodal architecture. Unlike traditional vision-language models that rely on separate heavy visual/audio encoders (like ViT or Whisper), the 12B Unified model projects raw image patches and audio waveforms directly into the main LLM's embedding space via lightweight linear layers.
Because text, image, and audio flow natively into a single decoder-only transformer, this model benefits dramatically from weight-only quantization, offering minimal multimodal latency and a highly streamlined memory footprint.
Quantization Recipe & Environment
The models were generated using the following computational setup and tuning parameters. Notably, it leverages RTN mode (iters: 0) with non-text module protection, which is mandatory for maintaining performance stability on the Gemma 4 native multimodal layers.
Environment Setup
PyTorch Version:2.10.0+cu128
CUDA Version:12.8 (CUDA Available: True)
Hardware: NVIDIA L40 (48 GB VRAM)
Tuning Configuration
python
1TUNING_CONFIG ={2"group_size":128,3"sym":True,4"iters":0,# RTN mode — required for Gemma 4 stability5"disable_opt_rtn":True,6"nsamples":256,7"seqlen":2048,8"low_gpu_mem_usage":False,9"quant_nontext_module":False,# Keeps linear projection layers intact for vision/audio inputs10"layer_config": layer_config,11}12
Deployment & Inference Guide
Due to the unique unified nature of Gemma 4's multimodal inputs, deployment requires the specialized vLLM unified image containing updated parser configurations.
Running with Docker (vLLM)
Deploy a production-ready OpenAI-compatible API server using the custom vllm-openai:gemma4-unified engine:
The complete script used to perform this quantization, manage the layer configurations, and optimize the memory footprint is fully open-sourced.
You can find the step-by-step Jupyter Notebook containing the exact execution environment, dependency setups, and layer configurations in the GitHub repository:
