AMD-native FP4 / FP8 GGUF builds of google/gemma-4-12b-it, for the ROCmFPX fork of
llama.cpp on RDNA3.5 / Strix Halo (gfx1151). These tensor types do not exist in mainline
llama.cpp — you need a ROCmFPX-capable build to load them.
Multimodal (mmproj included) and shipped with the MTP draft head, which is worth
1.46× here (see below).
Variants — all five in this repo
file
ftype
size
BPW
token_embd
decode
gemma-4-12b-it-Q4_0_ROCMFP4_COHERENT.gguf
102
6.50 GiB
4.68
q6_K
26.95 t/s
gemma-4-12b-it-Q4_0_ROCMFP4_FAST_COHERENT.gguf
104
6.18 GiB
4.45
q6_K
26.56 t/s
gemma-4-12b-it-Q6_0_ROCMFPX_AGENT.gguf
114
10.43 GiB
7.52
q8_0
17.54 t/s
gemma-4-12b-it-Q8_0_ROCMFPX.gguf
111
11.48 GiB
8.27
q8_0
16.39 t/s
gemma-4-12b-it-Q8_0_ROCMFPX_AGENT.gguf
115
11.67 GiB
8.41
q8_0
15.92 t/s
Decode measured on an idle Ryzen AI MAX+ 395 (Strix Halo, 128 GB, ROCm 7.2.4), -ngl 999,
-c 4096 -fa on -fit off -np 1, 300-token generations, no draft head — see the MTP
section for the accelerated figures.
⚠️ Tied embeddings — --output-tensor-type is a silent no-op here
gemma-4-12b-it sets tie_word_embeddings: true, so there is no output.weight tensor.
Passing --output-tensor-type does nothing at all; --token-embedding-type is the only
head protection that applies. With a 262144-token vocabulary at n_embd 3840 the embedding
is a large share of the file, which is why the 4-bit lands at 4.68 BPW rather than ~4.0.
* = 5 samples rather than 12. Those two rows each had a single low sample, which a median
absorbs; the 4-bit rows were re-run at 12 samples because two low samples of five moved the
number and inverted the FAST-vs-COHERENT ordering.
Correctness is 17×23 → 391 · capital of Japan → Tokyo · days in 2024 → 366, asserted
against content + reasoning and recorded with finish_reason. This model is a reasoner:
it routinely returns an empty content with the answer in reasoning_content, so a harness
that only reads content will score correct answers as failures.
Size integrity
Every artifact's on-disk size exceeds its dry-run projection by a constant ~%s MiB
header delta (spread across all five: 0.0075 MiB). A varying delta is the signature of a
truncated write; a constant one is just the header.
⚠️ If the MTP draft command crashes on your build
The separate-model draft-mtp path has known bugs in the legacy charlie12345/ROCmFPX
line — reported on Windows 11 / gfx1151 / HIP SDK 7.2 (five stacked bugs, ending in an
h-row width mismatch in the draft's embedding buffer).
Fix: build the official repo instead — no patches needed.
git clone https://github.com/ROCmFPX/ROCmFPX.git
The MTP path was reworked there (unified n_embd_out row widths, t_h_nextn reset in
llm_graph_result::reset(), ctx_other wired centrally), which covers the whole chain.
Reported and verified on that configuration: 24–31 tok/s, coherent output, tool calling working.
Legacy-line patch: PR #109.
Linux builds on the legacy line are not known to be affected.
⭐ Speculative decoding (MTP) — big speedup, with one real caveat
mtp-gemma-4-12b-it-Q8_0.gguf ships in this repo. Unlike the E2B/E4B members of this family,
the draft head is a substantial win here — read the caveat below before enabling it.
⭐ Note that the fastest setting is not the one with the highest draft acceptance.
Acceptance measures how often the drafter is right; throughput also pays for the
verification work. Sweep n-max and rank on measured t/s, not on acceptance.
⛔ Known defect: long generations abort the server with the draft head enabled
With --spec-type draft-mtp at -c 4096, a 1500-token generation reliably aborts the
server:
server-context.cpp:395: GGML_ASSERT(spec_i_batch.empty()) failed
in server_slot::update_batch(llama_batch&)
Reproduced at both --spec-draft-n-max 3 and 5. 300-token generations are unaffected —
we ran 5 per quant here plus 13 consecutive requests in a separate test with no failure, so
the decode figures above are sound. We have not yet isolated whether the trigger is the
single-request length itself or a context shift near the -c limit; if you enable the
drafter, cap generations conservatively and give yourself context headroom. Without
--spec-type the model handles long generations normally.
⛔ MTP and vision cannot be used together (upstream llama.cpp PR #20277 — image embeddings
are injected outside the token path and the speculative batch loses its boundary). Run two
configurations: text with the drafter, images with -fa off and no --spec-type.
A note on measurement, because we hit it here
This model showed a reproducible ~8% dip in a minority of decode samples on an otherwise
idle box, while E2B/E4B measured on the same machine and harness were flat. We re-ran at
12 samples per quant to characterise it:
The medians above are taken over 12 samples for this reason. We report every raw sample
rather than a summary so you can see the distribution yourself.
Verification
Each artifact was loaded on real hardware and checked for: exact stat bytes vs the dry-run
projection, actual token_embd type, three correctness answers, and a 5-sample decode median
with two warm-ups discarded. Vision was verified separately with the mmproj on a
four-quadrant colour image.
Credits
Base model google/gemma-4-12b-it. BF16 GGUF source from unsloth/gemma-4-12b-it-GGUF.
FP4/FP8 tensor types from the ROCmFPX fork of llama.cpp.
