This is an 8-bit variant of gemma-4-E4B-it — a conservative quantization that preserves full BF16 precision in attention, vision tower, audio tower, embeddings, and norm layers while compressing only the SwiGLU MLP projections to 8-bit.

Source model: https://huggingface.co/google/gemma-4-E4B-it

(1) Introduction

The Objective: Make Gemma 4 E4B more practical on Apple Silicon without sacrificing reasoning and multimodal capabilities.

The Problem: BF16 gives you the full brain but demands ~16 GB of unified memory. Uniform quantizations are smaller but degrade the hybrid attention mechanism and the vision/audio towers.

The Solution: Mixed Quantization. A surgical approach that compresses only the SwiGLU MLP layers to 8-bit, while leaving every attention projection, the entire vision tower, the audio tower, embeddings, per-layer projections, and norm layers at full BF16 precision.

Resulting size:

Version	Size	Reduction
BF16 baseline	~16 GB	—
Mixed 8-bit	~11 GB	~30%

(2) Architecture

Gemma 4 E4B is a dense hybrid-attention multimodal model with interleaved Vision and Audio towers. It has 42 transformer layers following a repeating pattern:

[sliding_attention × 5, full_attention] × 7 = 42 layers

The sliding attention layers use a 512-token window with standard RoPE (θ=10,000) for local context. Every sixth layer uses full softmax attention with proportional RoPE (θ=1,000,000) for global reasoning across the 128K token window.

Key architecture parameters:

Parameter	Value
Hidden size	2560
Intermediate size	10240
Attention heads	8
KV heads	2
Head dim (local)	256
Head dim (global)	512
Max position embeddings	131072
Vocabulary size	262144
MLP activation	SwiGLU (gelu_pytorch_tanh)
Double-wide MLP	No (standard)

Additionally, the model includes:

A 16-layer vision tower (hidden=768, 12 heads) for image/video comprehension
A 12-layer audio tower (hidden=1024, 8 heads) for audio understanding
Vision embedding projector (embed_vision) mapping vision features into LM space
Audio embedding projector (embed_audio) mapping audio features into LM space
Per-layer input gating (per_layer_input_gate, per_layer_projection, per_layer_model_projection) — Gemma 4's routing mechanism for modality-aware processing
SwiGLU MLPs in every layer for feature transformation

Identifying the Weight

The MLP projections (gate_proj, up_proj, down_proj) make up the majority of the model's mass. These SwiGLU layers are the "knowledge lookup" / feature transformation matrices and are extremely tolerant of quantization.

The Glass Cannons: Layers Protected at BF16

Layer	Why Protected
Full attention (q_proj, k_proj, v_proj, o_proj, q_norm, k_norm)	Deep reasoning and cross-token attention. 7 layers of full softmax attention are the model's primary reasoning engine.
Sliding attention (q_proj, k_proj, v_proj, o_proj, q_norm, k_norm)	Local context tracking across 35 layers. Quantizing these degrades the model's ability to maintain coherent state.
Vision tower (all 16 encoder layers)	Image and video comprehension. Quantizing the vision encoder collapses multimodal understanding.
Audio tower (all 12 layers)	Audio comprehension. Quantizing the audio encoder collapses audio understanding.
Vision/audio embedding projectors	Bridge vision and audio features into language model space. Quantizing these collapses the multimodal interface.
Per-layer projections/gates	Gemma 4's modality-aware routing mechanism. Critical for interleaving multimodal tokens with text.
Embed tokens / per-layer embeddings	Input embeddings at every layer. Quantizing these adds noise at the model's interface.
Norm layers (layernorm, layer_norm)	Stability anchors. Negligible size.
Layer scalars	Gemma's layer scaling mechanism. Critical for training stability.

The Crush Zone

Layer	Quantization	Rationale
MLP (gate_proj, up_proj, down_proj)	8-bit (group_size=64)	Majority of model mass. SwiGLU MLPs are the most quantization-tolerant component in a transformer. All the savings live here.

(3) The Quantization Script

The script uses MLX's native nn.quantize() with a class_predicate filter that targets only the three SwiGLU projection matrices inside language_model.layers.N.mlp. Everything else passes through untouched.

python

import mlx.core as mx
import mlx.nn as nn
from mlx_vlm.utils import load
import json, os, gc, argparse

parser = argparse.ArgumentParser()
parser.add_argument("--model", type=str, default="gemma-4-E4B-it")
parser.add_argument("--bits", type=int, default=8, choices=[8, 6, 4])
parser.add_argument("--group-size", type=int, default=64)
args = parser.parse_args()

model_path = f"/path/to/{args.model}"
output_path = f"/path/to/{args.model}-Mixed-{args.bits}bit"

# Lazy load — no eager evaluation
model, processor = load(model_path, lazy=True)

def gemma4_small_mlp_only_filter(path, module=None):
    """Target ONLY language model MLP projections. Protect everything else."""
    # Hard shield: vision tower
    if "vision_tower" in path:
        return False
    # Hard shield: audio tower
    if "audio_tower" in path:
        return False
    # Hard shield: vision/audio embedding projectors
    if "embed_vision" in path or "embed_audio" in path:
        return False
    # Hard shield: token embeddings
    if "embed_tokens" in path:
        return False
    # Hard shield: norms and scalars
    if "layernorm" in path or "layer_norm" in path or "layer_scalar" in path or path.endswith(".norm"):
        return False
    # Hard shield: attention
    if "self_attn" in path:
        return False
    # Hard shield: per-layer projections/gates
    if "per_layer" in path:
        return False
    # Target: MLP inside language_model only
    if "language_model" not in path or ".mlp." not in path:
        return False
    leaf = path.split('.')[-1]
    return leaf in ("gate_proj", "up_proj", "down_proj")

# Single-pass quantization — only MLP gets touched
nn.quantize(model, group_size=args.group_size, bits=args.bits,
            class_predicate=gemma4_small_mlp_only_filter)

# Save config and processor
os.makedirs(output_path, exist_ok=True)
if hasattr(processor, "save_pretrained"):
    processor.save_pretrained(output_path)

with open(os.path.join(model_path, "config.json")) as f:
    config = json.load(f)
config["quantization"] = {
    "group_size": args.group_size,
    "bits": args.bits,
    "type": "mixed",
    "description": f"Gemma 4 MLP-only {args.bits}-bit. Attention, vision/audio, embeddings, norms remain BF16."
}
with open(os.path.join(output_path, "config.json"), "w") as f:
    json.dump(config, f, indent=4)

# Flatten, destroy model tree, shard to SSD
# Sequential eval + shard (5 GB chunks)

Full script: mixed_quantize_gemma4_small.py in the quantization-scripts repository.

Note on config.json

The config.json declares a single "bits" value, but MLX reads per-layer quantization metadata dynamically from the safetensors tensor headers. The actual mixed-precision structure — which layers are quantized and which remain BF16 — lives in the safetensors files, not the config. Tooling that trusts only config.json will be confused; this is expected behavior for native MLX mixed quantization.

(4) Conclusion

Mixed MLX quantization of Gemma 4 E4B that compresses only the MLP layers — while preserving all attention, vision, audio, and embedding layers at BF16 — delivers a ~30% size reduction with no measurable intelligence loss.

The 8-bit version is the safe choice for production: at ~11 GB, it fits comfortably on any Apple Silicon machine with 16+ GB of unified memory while retaining the full hybrid attention architecture, vision tower, and audio tower at BF16 precision.

The hybrid attention architecture (sliding window + full softmax) is the key: by protecting both attention types in BF16, the model retains its full 128K context-window recall and reasoning capability. The per-layer gating mechanism — which routes multimodal tokens through the language model — is also preserved at full precision, ensuring vision and audio understanding remain intact. The MLP layers — which are just feature transformation matrices — absorb 8-bit quantization with no measurable penalty.

andrzejmontano/gemma-4-E4B-it-Mixed-8bit

gemma-4-E4B-it-Mixed-8bit-MLX