Gemma 4 QAT Q4_0 MTP Assistants for ik_llama
This repo contains ik_llama-compatible GGUF assistant/draft models converted
from Google's Gemma 4 QAT Q4_0 assistant checkpoints.
These are not standalone chat models. Use them as --model-draft files next to
the matching Google Gemma 4 QAT Q4_0 target GGUF.
Do not load this repository as the primary model. Hugging Face may show generic
GGUF usage snippets for this repo, but these files are assistant/draft GGUFs
only.
Important E2B/E4B Caveat
The E2B and E4B files are experimental edge-model assistants. They converted
and smoke-tested successfully with matched QAT targets after an ik_llama
shared-KV target loader fix.
Current ik_llama main includes that loader fix via PR #1927. Older builds may
fail before draft loading with errors such as:
1E2B: missing blk.15.attn_k.weight
2E4B: missing blk.24.attn_k.weight
Those target tensors appear intentionally absent in Google's QAT edge GGUFs:
the E2B/E4B targets advertise shared-KV tail layers.
E4B has an additional runtime caveat. On current ik_llama main, local tests
could load E4B QAT + matched E4B assistant and generate in a small CPU/offload
smoke path. The longer full-GPU flash-attention path was still unstable before
acceptance counters in our RTX 4070 test. Treat E4B as experimental until the
remaining runtime/FA path is resolved upstream.
Use matched pairs only. The E2B assistant is not expected to work with the E4B
target, and the E4B assistant is not expected to work with the E2B target.
Files
| Model | Q4_0 assistant |
|---|
| Gemma 4 E2B IT QAT Q4_0 | gemma-4-E2B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf |
| Gemma 4 E4B IT QAT Q4_0 | gemma-4-E4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf |
| Gemma 4 12B IT QAT Q4_0 | gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf |
| Gemma 4 26B-A4B IT QAT Q4_0 | gemma-4-26B-A4B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf |
| Gemma 4 31B IT QAT Q4_0 | gemma-4-31B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf |
BF16 conversion intermediates and Q8_0 reference variants are intentionally not
published here. The Q4_0 files are the intended pairing for Google's QAT Q4_0
target GGUFs.
Matching Target Repos
Use the assistants with Google's official target GGUFs:
google/gemma-4-E2B-it-qat-q4_0-ggufgoogle/gemma-4-E4B-it-qat-q4_0-ggufgoogle/gemma-4-12B-it-qat-q4_0-ggufgoogle/gemma-4-26B-A4B-it-qat-q4_0-ggufgoogle/gemma-4-31B-it-qat-q4_0-gguf
Example
1llama-server \
2 -m /path/to/gemma-4-12b-it-qat-q4_0.gguf \
3 --model-draft /path/to/gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf \
4 --spec-type mtp:n_max=4,p_min=0.0 \
5 --jinja
Use a current ik_llama build with Gemma 4 MTP support. Reproducing these
conversions from Google's safetensors currently requires convert_hf_to_gguf.py
to recognize Gemma4UnifiedAssistantForCausalLM as the existing Gemma 4 MTP
assistant converter path.
Validation
Local conversion metadata checks:
| Model | Tensors | Backbone | Centroid tensors |
|---|
| E2B | 50 | 1536 | yes |
| E4B | 50 | 2560 | yes |
| 12B | 48 | 3840 | no, metadata only |
| 26B-A4B | 48 | 2816 | no, metadata only |
| 31B | 48 | 5376 | no, metadata only |
All published files report general.architecture=gemma4_mtp.
Historical runtime smoke on an RTX 4070 with ik_llama build
4561 (6b9de3dba):
| Target + Q4_0 draft | Status | Notes |
|---|
| E2B | passed on shared-KV branch | MTP context ready; raw completion generated; 37/92 draft tokens accepted |
| E4B | passed on shared-KV branch | MTP context ready; raw completion generated; 33/116 draft tokens accepted |
| 12B | passed | MTP context ready; raw completion generated |
| 26B-A4B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |
| 31B | passed | MTP context ready; raw completion generated with CPU/system-RAM offload |
Additional current-main check with ik_llama bbe1a511e:
| Target + Q4_0 draft | Status | Notes |
|---|
| E2B | passed | full-GPU flash-attention direct-server request completed |
| E4B | partial | small CPU/offload -c 512 raw completion smoke passed; local full-GPU flash-attention path failed before acceptance counters |
Conversion Notes
Source assistant repos:
google/gemma-4-E2B-it-qat-q4_0-unquantized-assistantgoogle/gemma-4-E4B-it-qat-q4_0-unquantized-assistantgoogle/gemma-4-12B-it-qat-q4_0-unquantized-assistantgoogle/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistantgoogle/gemma-4-31B-it-qat-q4_0-unquantized-assistant
The assistants were converted through ik_llama's Gemma 4 MTP assistant
converter, then quantized with llama-quantize to Q4_0.
