Mixed-precision 8-bit quantization of Gemma 4 12B IT — Google's unified multimodal model (text + vision + audio + video), optimized for efficient local inference with minimal quality loss.

Quick Stats

Metric	Value
Base model	google/gemma-4-12b-it
Architecture	Gemma4Unified (48-layer dense, hybrid attention)
Quantization	MLP-only 8-bit, group-size=64
BF16 size	~24 GB
Mixed 8-bit size	~15 GB (≈38% reduction)
Modalities	Text, Vision, Audio, Video
License	Gemma License (inherits from base model)

Quantization Strategy

This model uses surgical mixed-precision quantization: only the SwiGLU MLP projection matrices (gate_proj, up_proj, down_proj) in the language model are quantized to 8-bit. All other components remain in BF16:

✅ Quantized (8-bit): LLM MLP layers (144 tensors across 48 layers, ~60% of model mass)
✅ Protected (BF16): All attention (sliding + full), vision tower, audio embedder, embeddings, LM head, MultimodalEmbedder, all normalization layers, layer_scalars

Rationale: MLP layers are "knowledge lookup" / feature transformation layers that are extremely quantization-tolerant. Attention mechanisms carry reasoning and context-window recall. Vision, audio, and multimodal projection layers are kept in BF16 to preserve modality fidelity.

Model Architecture

Language Model (Text)

48 dense layers in pattern [sliding_attention × 5, full_attention × 1] × 8
Hybrid attention: Sliding window attention (40 layers, head_dim=256) + full global attention (8 layers, head_dim=512)
Heads: 16 query / 8 KV / 1 global KV
Hidden size: 3840, Intermediate: 15360
Vocab: 262,144, Context: 262,144 tokens
RoPE: Default (θ=10,000) for sliding; Proportional (θ=1,000,000, partial_rotary=0.25) for full attention
No MoE (dense architecture)

Vision Tower

ViT with patch_size=16, pooling_kernel=3
280 soft tokens per image, projected to 3840-dim
Position embedding size: 1120

Audio Tower

Audio embedder: 640-dim embedding, 640 samples per token
Simple linear projection (no convolutional audio tower in this checkpoint)

Multimodal

Unified processor: Gemma4UnifiedProcessor handling text, images, audio, and video
Special tokens: <|image|>, <|audio|>, <|video|> with BOI/EOI, BOA/EOA delimiters
MultimodalEmbedder projects vision and audio into the shared 3840-dim text space

MTP Speculative Decoding

This model supports Multi-Token Prediction (MTP) speculative decoding via the companion assistant/drafter model. This model achieves 20-30% faster token generation with gemma-4-12B-it-assistant in BF16 format.

Usage

oMLX (Recommended)

Loads natively with MTP and vision support.

MLX / mlx-vlm

python

from mlx_vlm.utils import load

model, processor = load("your-username/gemma-4-12B-it-Mixed-8bit", lazy=True)
# Model loads in ~15 GB vs ~24 GB for BF16

Files

File	Description
`model-00001.safetensors` – `model-00003.safetensors`	Sharded model weights (~5 GB each)
`config.json`	Model configuration with quantization metadata
`tokenizer.json` / `tokenizer_config.json`	Tokenizer (262K vocab)
`chat_template.jinja`	Chat template with thinking/tool support
`generation_config.json`	Generation parameters
`processor_config.json`	Unified processor config

Quantization Details

Parameter	Value
Quantization mode	Affine (8-bit)
Group size	64
Layers quantized	48 (all MLP: gate_proj, up_proj, down_proj)
Tensors quantized	~144
Layers protected	Attention (Q/K/V/O), Vision, Audio, Embeddings, Norms, Scalars
Script	`quantization_scripts/mixed_quantize_gemma4_12b.py`

Reproducing

bash

# Quantize from the BF16 base model
mlx-env/bin/python quantization_scripts/mixed_quantize_gemma4_12b.py \
    --bits 8 --group-size 64

The script:

Lazily loads the BF16 checkpoint via mlx_vlm.utils.load()
Applies 8-bit quantization to MLP projections only (via class_predicate filter)
Saves config with quantization metadata
Flattens, evaluates, and shards to 5 GB safetensors

Limitations

Mixed quantization may show minor quality degradation on edge cases compared to BF16
The vision tower, audio embedder, and multimodal projector are kept in BF16 — they are not quantized in this version
Audio support uses a simple linear projection (no convolutional audio tower); complex audio understanding may be limited
MTP speculative decoding requires the companion assistant model to be loaded alongside
The base model uses transformers_version: 5.10.0.dev0 — some fields may not be recognized by stable transformers releases yet

License

This model inherits the Apache 2.0 license of the base google/gemma-4-12b-it.

andrzejmontano/gemma-4-12B-it-Mixed-8bit

gemma-4-12B-it-Mixed-8bit