GGUF quantizations of google/gemma-4-26B-A4B-it-assistant,
the Multi-Token-Prediction (MTP) drafter for gemma-4-26B-A4B-it. Intended
for use as the draft model in speculative decoding paired with a
quantized gemma-4-26B-A4B-it verifier. Realised speedup depends on
hardware and verifier choice — see §Performance for measured
numbers and the PR's reference benchmarks.
[!WARNING]
ik_llama.cpp only — will NOT load in mainline llama.cpp
These quants use the gemma4_mtp architecture, which is currently
only supported in ikawrakow/ik_llama.cpp
on (or after) the feat/gemma-4-mtp branch / PR #1744.
Mainline ggml-org/llama.cpp does not know about
gemma4_assistant / gemma4_mtp and will refuse to load these
files. Downstream tools that wrap mainline (ollama, LM Studio,
jan.ai, llama-cpp-python, …) will not work either until they
follow ik_llama.cpp.
Tested with ik_llama.cpp commit a703033607ed3edbeab0205d8c9ad75cc1b5759f.
[!NOTE]
Benchmarks in progress
Real measured throughput + acceptance-rate numbers for these
drafters are being collected on a consumer laptop GPU (NVIDIA
4060 Laptop, 8 GiB VRAM) — full --draft-max × --draft-p-min
matrix across multiple prompts, driven against llama-server
with acceptance read from its per-request statistics mtp:
stderr lines.
Results land in the smaller drafters' model cards first
(E2B → E4B → 26B-A4B → 31B), since smaller models cycle through
the bench faster. Check the
E2B card
first if you're shopping for performance numbers — even if you
plan to deploy a different size, the relative draft-max curves
and acceptance trends carry across sizes within the family.
Until then, §Performance below cites the upstream PR's reference
benchmarks (data-center GPU + 31B verifier) — treat them as a
ceiling, not a target.
[!IMPORTANT]
Honest limitations of this build
Things that should be on the model card and aren't faked:
No imatrix calibration. PR #1744 builds the gemma4_mtp
drafter graph with a hardcoded GGML_ASSERT(has_target_ctx),
meaning standalone-drafter llama-imatrix runs abort in
llama_decode before producing anything. Same as every other
community-published Gemma 4 drafter quant today
(Radamanthys11's, etc.), this build quantizes from f16
directly. At ≥4 bits this is fine; the precision benefit of
imatrix-guided quantization at Q5+ is single-digit %. At Q3 and
below it matters more — see the per-quant warnings in the table
below.
No acceptance-rate validation in this build. The benchmark
numbers in §"Performance" come from the upstream PR thread on a
31B verifier, not from runs against this drafter quant.
Smoke-testing on the build host (CPU-only) was disabled because
per-token MTP cost is many seconds on CPU, making proper
benchmark-quality runs impractical mid-pipeline. Treat the
numbers as expectations, not measurements of these specific
files.
No IK-only IQK quants (IQ4_KS / IQ5_KS / IQ4_KSS). These
are normally what "ik_llama.cpp build" gets you over mainline,
but their precision-per-bit advantage comes from
imatrix-guided scale selection — without imatrix they collapse
to roughly K-quant quality at the same bit budget, so shipping
them would just be misleading row-count padding. They'll come
back in a sibling experiment when upstream supports
standalone-drafter imatrix.
MXFP4 has narrower runtime support. Loadable in current
ik_llama.cpp and mainline llama.cpp; older ggml-based runtimes
may not support it yet. Use a K-quant if you need to load these
in something older.
Pairing — required
A drafter is not a standalone language model. To use these quants
you also need a base-model GGUF, with matching vocab (262144 tokens,
which is the whole Gemma 4 family default). Recommended pairings:
Verifier: unsloth/gemma-4-26B-A4B-it-GGUF — ~25.2B total (3.8B active, MoE)
parameters. Heads-up on Google's naming: the "E" in E2B/E4B means
effective (active inference) parameters via Per-Layer Embeddings,
not total weight count — full weights still have to fit in VRAM.
The "A" in 26B A4B is the same trick for the MoE variant: 3.8B
active out of 25.2B total. The 31B is plain dense, no naming games.
Pairing precision: matching is generally optimal, but mismatched
pairings work too with some acceptance-rate penalty. Both bf16
and f16 drafters in this repo are valid pairing targets for a
bf16/f16/Q8_0 verifier — bf16 is preferred when your runtime
supports it (matches the source-tensor format exactly; see the bf16
row in §Quants). Going above the verifier's precision on the
drafter has no benefit.
Empirical note (single-data-point, no warranty): on a 4060
Laptop GPU paired against unsloth's Gemma-4-E4B-it-Q4_K_M.gguf
verifier, the Q4_K_M drafter outperformed the Q8_0 drafter by
~13% in throughput at --draft-max 3. The smaller drafter's
faster draft step appears to outweigh the acceptance-rate cost
from more aggressive quantization. Contradicts the common
"always pick the highest-bit drafter" heuristic. Bench your own
hardware before assuming.
Preserves structured-output tokens
The drafter's vocabulary is identical to the verifier's (262144
tokens, the Gemma 4 family default). Notably, that includes Gemma
4's reserved tokens for structured output formats which the
drafter speculates correctly:
Token pair
Used for
<|tool_call> / <tool_call|>
Tool / function calling — agent invokes a tool
<|tool_response> / <tool_response|>
Tool / function calling — tool result back to model
<|channel> / <channel|>
Multi-channel output (e.g. <|channel>thought for chain-of-thought reasoning vs user-facing channel)
<|"|>
Structured-string delimiter
If you're running tool-calling agents, multi-step reasoning, or
any structured-generation workflow on top of Gemma 4, this drafter
will speculate those tokens just like any other — meaning the MTP
speedup applies to the whole response, not just the natural-language
parts. Most published drafter quants don't talk about this because
it Just Works mechanically (vocabulary is a separate GGUF section
that's never quantized), but it's worth saying out loud: pairing
this drafter with a tool-calling-finetuned verifier preserves the
tool-call grammar end-to-end.
Quants
~408M parameter MTP head, 2816-dim
backbone projection (must match the verifier's hidden_size).
(GGUF general.size_label for this drafter is the verifier's MoE "A4B" tag, not a head parameter count; ~408M is computed from the bf16 file size, ~816 MB / 2 bytes-per-param.)
12 files spanning bf16 → Q3, in approximate order of decreasing
precision.
Quantization
Approx. bpw
Size
Notes
gemma-4-26B-A4B-it-assistant-bf16.gguf
16
816 MB
Faithful to source. Gemma 4's safetensors are bfloat16 (8-bit exponent, 7-bit mantissa); this preserves them exactly. Prefer this over f16 if your runtime speaks bf16 (recent mainline llama.cpp, ik_llama.cpp, ollama).
gemma-4-26B-A4B-it-assistant-f16.gguf
16
816 MB
Conventional reference. f16 has more mantissa precision than bf16 (10 vs 7 bits) but a smaller exponent range (5 vs 8 bits), so it can over/underflow on activations bf16 handles fine. For weights converted from a bf16 source, going to f16 effectively quantizes the dynamic range to fit f16's narrower exponent — small but real loss vs bf16. Use bf16 when your runtime supports it.
gemma-4-26B-A4B-it-assistant-Q8_0.gguf
8.5
441 MB
Near-lossless quantization. Recommended pairing target — drafters' acceptance rate suffers most from quantization, so the highest-bit quant is the best choice if your verifier is also Q8_0+.
gemma-4-26B-A4B-it-assistant-Q6_K.gguf
6.5
344 MB
K-quant, very high quality. Good balance for Q6_K verifiers.
gemma-4-26B-A4B-it-assistant-Q5_K_M.gguf
5.7
327 MB
High-precision K-quant.
gemma-4-26B-A4B-it-assistant-Q5_K_S.gguf
5.5
325 MB
Smaller Q5 variant.
gemma-4-26B-A4B-it-assistant-Q4_K_M.gguf
4.85
311 MB
Community sweet-spot for verifier pairings.
gemma-4-26B-A4B-it-assistant-Q4_K_S.gguf
4.6
307 MB
Smaller Q4 K-quant.
gemma-4-26B-A4B-it-assistant-IQ4_NL.gguf
4.5
307 MB
Non-linear i-quant. Mainline-loadable (unlike the IK-only IQ4_KS). Doesn't require imatrix to be useful.
gemma-4-26B-A4B-it-assistant-IQ4_XS.gguf
4.25
268 MB
Smaller non-K i-quant. Mainline-loadable.
gemma-4-26B-A4B-it-assistant-MXFP4.gguf
4.25
302 MB
OCP microscaling 4-bit float format. Loadable in current ik_llama.cpp and mainline llama.cpp; older ggml-based runtimes may not support it yet.
gemma-4-26B-A4B-it-assistant-Q3_K_L.gguf
3.4
301 MB
Untested for drafter use; pair with caution. Without imatrix, Q3 loses more accuracy than higher quants — and drafters are particularly acceptance-sensitive (a misprediction is wasted work). Included for users who absolutely need the smallest footprint, but be aware MTP speedup could degrade or invert vs. running the verifier alone. Benchmark before deploying.
Deliberately omitted quants (and why, briefly):
F32 — zero-padded bf16, no information gain, double the disk.
Q4_0 / Q5_0 / Q4_1 / Q5_1 — legacy non-K quants. K-quants
strictly dominate them at the same bit budget.
Q3_K_M / Q3_K_S / Q2_K — without imatrix, drafter acceptance
drops sharply below Q3_K_L. Re-add when imatrix is available.
IQ2_* / IQ1_* — too noisy at any bit budget for drafter use,
even with imatrix. Verifier rejects most drafted tokens, paired
generation goes net negative vs. baseline.
Enables MTP-style speculative decoding (this is the path PR #1744 plumbs).
--model-draft (-md)
The drafter GGUF.
--draft-max N
Maximum draft length per step. 3 is a good default; 1–4 are all reasonable; tune per workload with --spec-autotune.
--draft-p-min
Minimum draft-token probability to bother drafting. 0.0 accepts all drafts; raising it shortens speculative chains.
-ngld 99
Push the drafter onto GPU layers (no-op on CPU-only hosts). The drafter is small enough to fully fit on any consumer GPU.
-ctk q8_0 / -ctv q8_0
Quantize KV cache. Reduces VRAM pressure for long contexts.
--jinja
Use the model's Jinja chat template (Gemma 4's tool-call format etc.).
--spec-autotune (per the PR #1744 description) will probe several
--draft-max values during inference and pick the best-fitting one
for your workload — useful if you don't want to tune by hand.
Performance
Reproducing the upstream benchmark on a 31B verifier + this drafter
at Q8_0 on Q8_0 (per the PR #1744 description):
Run
Throughput
Acceptance
Baseline (no MTP)
~21 t/s
—
MTP --draft-max 1
~35 t/s
~89%
MTP --draft-max 2
~44 t/s
~83%
MTP --draft-max 3
~49 t/s
~74%
MTP --draft-max 4
~49 t/s
~64%
Smaller verifiers (E2B/E4B) get less absolute t/s benefit because the
verifier itself is faster, so there's less time-budget for the drafter
to fill in. The percentage uplift is similar.
Compatibility notes
A few cosmetic / non-blocking quirks you may see in normal use:
transformers warning during conversion (only relevant if you
re-convert from source rather than using these prebuilts):
You are using a model of type `gemma4_assistant` to instantiate a
model of type ``. This may be expected if you are loading a
checkpoint that shares a subset of the architecture …
The IK fork's convert_hf_to_gguf.py patches in gemma4_assistant
arch support on the GGUF side but does not patch the Hugging
Face transformers library itself. So transformers (which the
converter uses to read the source safetensors) sees the unfamiliar
model_type and falls back to generic loading. Generic loading
reads the raw weights correctly, so the conversion still produces
a valid GGUF — the warning is cosmetic.
Oops: tensor with strange name per_layer_* at runtime
(visible if you pair against certain non-google-flavored Gemma 4
base GGUFs, e.g. unsloth's). These warnings come from the
verifier loader, not the drafter — they're the verifier model's
per-layer projection tensors which ik_llama.cpp's gemma4 base
implementation may not fully recognize on third-party-quantized
GGUFs. Inference still works but may fall back to slower code
paths for those tensors. If absolute throughput seems too low
vs. the PR's reference benchmarks, try a different verifier
(google's own f16, bartowski's quants, or any other community
source) and compare.
mtp_pre_proj.weight / mtp_post_proj.weight "strange name"
warnings at drafter load — see PR #1744 review thread;
these are the drafter's MTP projection tensors which the size-
accounting iteration in src/llama.cpp doesn't special-case.
Cosmetic; the MTP runtime loads them correctly via
create_gemma4_mtp_tensors.
This repo ships every quant variant of this drafter that made
sense to produce: 12 files spanning bf16 reference down to Q3_K_L,
including K-quants, non-K i-quants (IQ4_NL, IQ4_XS), and OCP
MXFP4. The omitted quants (F32, legacy Q4_0/Q5_0 etc., Q2_K,
IQ2_, IQ1_, the imatrix-dependent IQ4_KS family) are documented
above the table with the reason each was left out.
License
Gemma Terms of Use, inherited
from the source model. By downloading or using these quants you
agree to Google's Gemma terms — same as if you'd downloaded the
upstream weights directly.
Issues / questions
Open a discussion on this repo
(cafkafk/gemma-4-26B-A4B-it-assistant-GGUF-noimatrix)
for anything quant-specific (a particular file refusing to load, a
quant variant behaving worse than expected, sizes-table corrections,
etc.).
For ik_llama.cpp runtime bugs (gemma4_mtp arch issues, MTP
acceptance-rate quirks, --spec-type mtp plumbing) the canonical
place is the upstream
PR #1744 thread
or the ikawrakow/ik_llama.cpp issue tracker.
For upstream weights / chat-template / tokenizer questions, file
against google/gemma-4-26B-A4B-it-assistant — but please
filter quant-format problems out before going there; Google does not
maintain the GGUF tooling.
