This is the MTP draft/assistant model for Gemma 4 12B, quantized from google/gemma-4-12B-it-qat-q4_0-unquantized-assistant (BF16) to INT4 W4A16 compressed-tensors format with group_size=128, optimized for Marlin kernel acceleration.

It is designed to be used alongside google/gemma-4-12B-it-qat-w4a16-ct as the main model with SGLang's Frozen-KV MTP speculative decoding.

Why This Model?

Using the original BF16 assistant model with MTP speculative decoding isn't ideal on Ampere when using the Marlin kernels for INT4 ops with the base model. This INT4 quantized draft model uses Marlin kernels (same as the main model), delivering major speedups at all sequence lengths:

Config	100 tok	500 tok
Main model only (no MTP)	74 tok/s	81 tok/s
MTP + BF16 draft (original)	38 tok/s	98 tok/s
MTP + this INT4 draft	129–144 tok/s	101–105 tok/s

2× faster than no-MTP on short sequences. 3.5× faster than BF16 draft MTP. No regression at any sequence length.

Benchmarks on NVIDIA RTX 3090 (24 GB) with SGLang 0.5.12.

How It Was Created

Standard quantization tools (llmcompressor, compressed_tensors, bitsandbytes) all fail on the Gemma4UnifiedAssistantForCausalLM architecture due to its shared_kv_states forward signature and FX tracer incompatibility.

Instead, this model was quantized with manual RTN (Round-to-Nearest) symmetric INT4 quantization, producing output in the exact compressed-tensors pack-quantized format that SGLang's loader expects:

Method: Per-group symmetric RTN, scale = max(|W_group|) / 7
Group size: 128 (Marlin kernel sweet spot, vs 32 in Google's QAT main model)
Format: compressed-tensors pack-quantized (weight_packed + weight_scale + weight_shape)
Packing: 8 × 4-bit values per int32, standard compressed-tensors layout
Ignored layers: lm_head, embed_tokens (kept in BF16)
Tied weights: lm_head.weight = model.embed_tokens.weight (same as original)

20 Linear layers quantized across the 4-layer assistant transformer.

Running with SGLang

bash

docker run -d \
  --name sglang-mtp \
  --gpus all \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:gemma4-mtp \
  bash -c "pip install --no-deps --quiet 'git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897' 2>&1 && \
  python3 -m sglang.launch_server \
    --model-path google/gemma-4-12B-it-qat-w4a16-ct \
    --quantization compressed-tensors \
    --kv-cache-dtype fp8_e5m2 \
    --speculative-algorithm FROZEN_KV_MTP \
    --speculative-num-steps 3 \
    --speculative-num-draft-tokens 4 \
    --speculative-eagle-topk 1 \
    --speculative-draft-model-path kunhunjon/gemma-4-12B-it-qat-assistant-w4a16-ct \
    --speculative-draft-model-quantization compressed-tensors \
    --max-running-requests 4 \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --dtype bfloat16 \
    --trust-remote-code \
    --host 0.0.0.0 --port 30000"

Notes:

--speculative-eagle-topk 1 is required even for FROZEN_KV_MTP due to an SGLang bug (it unconditionally multiplies eagle_topk * num_steps).
The --kv-cache-dtype fp8_e5m2 flag uses FP8 KV cache (supported on Ampere and later GPUs). Use fp8_e4m3 on Hopper+ or omit for BF16 KV cache.
The transformers git commit is needed for gemma4_unified architecture support. This is installed inside the container at startup.

MTP Acceptance Rates

Metric	Short gen (~100 tok)	Long gen (~500 tok)
Accept rate	0.53–0.63	0.27–0.50
Accept len (avg draft tokens accepted)	2.60–2.90	1.80–2.50
Throughput	129–144 tok/s	101–105 tok/s

Settings: speculative_num_steps=3, speculative_num_draft_tokens=4.

Adapting for Other Gemma 4 Models

The same manual RTN quantization approach should work for any Gemma 4 assistant model.

Hugging Face | GitHub | Launch Blog | Documentation
License: Apache 2.0 | Authors: Google DeepMind

[!Note] This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. Four versions of the QAT checkpoints are available:

Unquantized QAT checkpoints (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B, and their drafter models.

GGUF (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B.

Mobile-optimized (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B.

Compressed Tensors (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B, and 31B.

Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.

Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning. The models are available in five distinct sizes: E2B, E4B, 12B, 26B A4B, and 31B. Their diverse sizes make them deployable in environments ranging from high-end phones to laptops and servers, democratizing access to state-of-the-art AI.

Gemma 4 introduces key capability and architectural advancements:

Reasoning – All models in the family are designed as highly capable reasoners, with configurable thinking modes.
Extended Multimodalities – Processes Text, Image with variable aspect ratio and resolution support (all models), Video, and Audio (featured natively on the E2B, E4B, and 12B models).
Diverse & Efficient Architectures – Offers Dense and Mixture-of-Experts (MoE) variants of different sizes for scalable deployment.
Optimized for On-Device – Smaller models are specifically designed for efficient local execution on laptops and mobile devices.
Increased Context Window – The small models feature a 128K context window, while the medium models support 256K.
Enhanced Coding & Agentic Capabilities – Achieves notable improvements in coding benchmarks alongside native function-calling support, powering highly capable autonomous agents.
Native System Prompt Support – Gemma 4 introduces native support for the system role, enabling more structured and controllable conversations.

Models Overview

Gemma 4 models are designed to deliver frontier-level performance at each size, targeting deployment scenarios from mobile and edge devices (E2B, E4B) to consumer GPUs and workstations (12B, 26B A4B, 31B). They are well-suited for reasoning, agentic workflows, coding, and multimodal understanding.

The models employ a hybrid attention mechanism that interleaves local sliding window attention with full global attention, ensuring the final layer is always global. This hybrid design delivers the processing speed and low memory footprint of a lightweight model without sacrificing the deep awareness required for complex, long-context tasks. To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE).

Dense Models

Property	E2B	E4B	12B Unified	31B Dense
Total Parameters	2.3B effective (5.1B with embeddings)	4.5B effective (8B with embeddings)	11.95B	30.7B
Layers	35	42	48	60
Sliding Window	512 tokens	512 tokens	1024 tokens	1024 tokens
Context Length	128K tokens	128K tokens	256K tokens	256K tokens
Vocabulary Size	262K	262K	262K	262K
Supported Modalities	Text, Image, Audio	Text, Image, Audio	Text, Image, Audio	Text, Image
Vision Encoder Parameters	~150M	~150M	-	~550M
Audio Encoder Parameters	~300M	~300M	-	No Audio

Mixture-of-Experts (MoE) Model

Property	26B A4B MoE
Total Parameters	25.2B
Active Parameters	3.8B
Layers	30
Sliding Window	1024 tokens
Context Length	256K tokens
Vocabulary Size	262K
Expert Count	8 active / 128 total and 1 shared
Supported Modalities	Text, Image
Vision Encoder Parameters	~550M

Best Practices

1. Sampling Parameters

Use the following standardized sampling configuration across all use cases:

temperature=1.0
top_p=0.95
top_k=64

2. Thinking Mode Configuration

To properly manage the thinking process, use the following control tokens:

Trigger Thinking: Thinking is enabled by including the |<<|think|> token at the start of the system prompt. To disable thinking, remove the token.
Standard Generation: When thinking is enabled, the model will output its internal reasoning followed by the final answer using this structure:
|<<|channel>thought\n[Internal reasoning]<<channel|>

3. Multi-Turn Conversations

No Thinking Content in History: In multi-turn conversations, the historical model output should only include the final response. Thoughts from previous model turns must not be added before the next user turn begins.

Ethics and Safety

As open models become central to enterprise infrastructure, provenance and security are paramount. Developed by Google DeepMind, Gemma 4 undergoes the same rigorous safety evaluations as our proprietary Gemini models.

Evaluation Approach

Gemma 4 models were developed in partnership with internal safety and responsible AI teams. A range of automated as well as human evaluations were conducted to help improve model safety. These evaluations align with Google's AI principles, as well as safety policies.

Ethical Considerations and Risks

Bias and Fairness – VLMs trained on large-scale data can reflect socio-cultural biases. Gemma 4 underwent careful scrutiny and evaluations to mitigate bias risks.
Misinformation and Misuse – Guidelines are provided for responsible use. See the Responsible Generative AI Toolkit.
Transparency and Accountability – This model card summarizes details on architecture, capabilities, limitations, and evaluation processes.