Gemma 4 12B Unified — Imatrix IQ4_XS GGUF
Custom importance-matrix quantized version of
google/gemma-4-12B-it, the encoder-free unified multimodal model from Gemma 4 family.
Quantization details
- Base model: google/gemma-4-12B-it (BF16, ~23.8 GB)
- Method: IQ4_XS with custom importance matrix calibration
- Calibration dataset: reapmix_imatrix (2237 chunks)
- Embedding protection:
per_layer_token_embd.weight preserved in Q6_K via imatrix
- Final size: 6.16 GB (4.45 BPW)
- Compression ratio: 0.26x from BF16
Architecture notes
Gemma 4 12B uses the new gemma4_unified architecture — encoder-free multimodal design where vision and audio modalities are projected directly into the decoder through lightweight linear layers, rather than dedicated encoders. The model has 48 layers with shared KV in every 6th global layer (head_count_kv=1), optimizing memory footprint for the 256K context window.
This quant requires llama.cpp commit including PR #24088 or later for gemma4_unified architecture support.
Hardware benchmark
Tested on NVIDIA RTX PRO 5000 Blackwell (72 GB VRAM):
| Metric | Value |
|---|
| Generation speed | 99-104 tok/s sustained |
| VRAM at 32K context | ~10 GB (with FlashAttention) |
| Time to first token | <1 s on typical prompts |
Recommended sampling
Per Google's Gemma 4 best practices — deviating from these will cause repetition loops on edge-case prompts:
- temperature: 1.0
- top_p: 0.95
- top_k: 64
- repeat_penalty: 1.0 (disabled)
Thinking mode
To enable Gemma 4's reasoning mode, prepend <|think|> to the system prompt. The model will produce internal reasoning in a <|channel>thought ... <channel|> block before the final answer.
Pipeline
Built using a hybrid CPU/GPU pipeline across a homelab cluster:
- BF16 to GGUF conversion on AI workstation (Dell 5820 + RTX PRO 5000)
- Importance matrix computed on GPU (~60 minutes for 2237 chunks)
- IQ4_XS quantization on dual Xeon E5-2699 v3 (88 threads, 128 seconds)
- Inter-node transfer via 40GbE direct-attach (RoCEv2 ring)
Quick start
1./llama-server \
2 -m gemma-4-12b-it-Imatrix-IQ4_XS.gguf \
3 -c 32768 \
4 -ngl 99 \
5 --host 0.0.0.0 \
6 --port 8080
License
Apache 2.0 (inherited from base model).
