🔧 Runtime: build the ROCmFPX fork below
Stock
llama.cpp will not load this file. You need
both the
mellum architecture
and the ROCmFP4 tensor types in one tree. Upstream
charlie12345/ROCmFPX has the ROCmFP4 types but
not
mellum. Our fork has both:
kingjones30/ROCmFPX — a fork of
charlie12345/ROCmFPX, branch
main.
1git clone https://github.com/kingjones30/ROCmFPX.git
2cd ROCmFPX
3cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
4cmake --build build --target llama-server llama-quantize -j$(nproc)
Verified 2026-08-27 on gfx1151: clean clone → 0 build errors → llama-server loads a
mellum ROCmFP4 GGUF from this family and generates coherent text.
⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL
The
Mellum architecture is not merged upstream (PR
#23966) — the patch is in
patches/ in this repo.
🚀 104.99 tok/s — +6.9% faster than Q4_K_M, ranges disjoint.
Mellum2-12B-A2.5B-Thinking — ROCmFP4 (tier 102 COHERENT) GGUF
A 4-bit ROCmFP4 quantization for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo), quantized
from BF16 GGUF (22.6 GiB) — a lossless source, not a requantization of a lower-bit build.
| |
|---|
| File | Mellum2-12B-A2.5B-Thinking-Q4_0_ROCMFP4_COHERENT.gguf |
| Size | 6.4907 GiB |
| BPW | 4.59 |
| ftype | Q4_0_ROCMFP4_COHERENT (102) |
⛔ Requires a llama.cpp with the ROCmFP4 quant types
Q4_0_ROCMFP4_COHERENT (ftype 102) exists only in
charlie12345/ROCmFPX, not upstream llama.cpp.
Ignore the auto-generated "Use this model" commands above.
All quant variants
All measured on one box, one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4), median of 3,
warm-up discarded — so these rows are directly comparable.
| variant | ftype | size | bpw | decode (median) | range |
|---|
| 4-bit COHERENT | 102 | 6.49 GiB | 4.59 | 104.99 | 104.96 – 105.73 |
| 8-bit AGENT | 115 | 11.88 GiB | 8.39 | 74.41 | 74.36 – 74.42 |
| 8-bit plain | 111 | 11.70 GiB | 8.27 | 75.72 | 75.72 – 75.73 |
⚠️ AGENT is SLIGHTLY SLOWER here — 74.41 vs 75.72. The opposite of the Instruct variant, which tells you the difference is model-specific and small. Neither 8-bit build approaches the 4-bit build's 104.99 tok/s.
On AGENT generally: it keeps more tensors at true Q8_0 instead of the packed 8-bit type.
That raises MTP draft acceptance on models which have an MTP head (measured +6.2% on
Qwen3.8-27B). Mellum2 has no MTP head, so there is nothing for the extra precision to feed
and the two 8-bit builds differ only marginally — in either direction.
Measured
Ryzen AI MAX+ 395 (gfx1151, 128 GB unified, ROCm 7.2.4). Median of 3+, warm-up discarded,
otherwise-idle box. Correctness at the model's official sampling.
| build | size | decode (median) | range |
|---|
| this build | 6.4907 GiB | 104.99 | [104.96 – 105.73] |
| Q4_K_M | — | 98.17 | [98.16 – 98.87] |
+6.9%, ranges disjoint — a real difference, not noise.
Correctness: 17×23 ⇒ ✅ 391 · capital of Japan ⇒ ✅ Tokyo · days in 2024 ⇒ ✅ 366
Per-tensor types (audited in the finished file)
output.weight Q6_K · token_embd Q6_K · router ffn_gate_inp F32 · norms F32 · 339 tensors
tie_word_embeddings is false, so both --output-tensor-type and --token-embedding-type apply.
What was NOT measured
- No perplexity run, and no quality A/B against the baseline or the source. The checks
above are memorized-fact prompts — necessary but not sufficient; a damaged model can pass them.
- No long-context testing.
- No tool-calling evaluation.
Base model licence inherited; credit for the model goes to its authors.
