🔧 Runtime: build the ROCmFPX fork below

Stock llama.cpp will not load this file. You need both the mellum architecture and the ROCmFP4 tensor types in one tree. Upstream charlie12345/ROCmFPX has the ROCmFP4 types but not mellum. Our fork has both:

kingjones30/ROCmFPX — a fork of charlie12345/ROCmFPX, branch main.

bash
1git clone https://github.com/kingjones30/ROCmFPX.git 2cd ROCmFPX 3cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release 4cmake --build build --target llama-server llama-quantize -j$(nproc)

Verified 2026-08-27 on gfx1151: clean clone → 0 build errors → llama-server loads a mellum ROCmFP4 GGUF from this family and generates coherent text.

⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

The Mellum architecture is not merged upstream. Ignore the auto-generated "Use this model" commands above — build the ROCmFPX fork linked just below.

🚀 96.92 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) — 6.49 GiB, 1.12 GiB smaller and 7.2% faster than Q4_K_M.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to charlie12345/ROCmFPX at commit 2809dc5, verified with git apply --check.

bash

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: patches/README.md in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB is configure-time, so cmake --build alone will not link them.

Mellum2-12B-A2.5B-Instruct — ROCmFP4 (tier 102 `COHERENT`) GGUF

A 4-bit ROCmFP4 quantization of JetBrains/Mellum2-12B-A2.5B-Instruct, built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo), with the LM head and token embeddings held at Q6_K.


File	`Mellum2-12B-A2.5B-Instruct-Q4_0_ROCMFP4_COHERENT.gguf`
Size	6.4907 GiB (6,969,373,344 bytes)
BPW	4.59
ftype	`Q4_0_ROCMFP4_COHERENT` (102)
Source	BF16 GGUF (22.64 GiB) — lossless source, not a requantization
sha256	`161d23aa5dd6813e348cdcbf6873beb9c1cded3b56d211379429dcaa373fc43e`

Smaller and faster than Q4_K_M on the target hardware — see below.

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

mellum is not in mainline llama.cpp. Support is open in PR #23966 ("model: add Mellum architecture", Xarbirus; branch Xarbirus/llama.cpp:mellum2), unmerged at time of writing. The ROCmFP4 quant types additionally require a fork that implements them — upstream has no Q4_0_ROCMFP4_*.

⚠️ `strings` is not a capability check

Our build's libllama.so contained the literal string mellum and still failed with unknown model architecture: 'mellum'. The string lives in a name table; the loader is separate code. Grepping the binary tells you nothing — attempt the load.

All quant variants

All measured on one box, one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4), median of 3, warm-up discarded — so these rows are directly comparable.

variant	ftype	size	bpw	decode (median)	range
4-bit COHERENT	102	6.49 GiB	4.59	104.99	104.96 – 105.73
8-bit AGENT	115	11.88 GiB	8.39	74.93	74.93 – 74.97
8-bit plain	111	11.70 GiB	8.27	72.76	72.61 – 72.79

Repos: 4-bit · 8-bit AGENT · 8-bit plain

AGENT is faster here — 74.93 vs 72.76, ranges disjoint (+3.0%). Both 8-bit builds are well below the 4-bit build's 104.99 tok/s; they exist for accuracy headroom, not speed.

On AGENT generally: it keeps more tensors at true Q8_0 instead of the packed 8-bit type. That raises MTP draft acceptance on models which have an MTP head (measured +6.2% on Qwen3.8-27B). Mellum2 has no MTP head, so there is nothing for the extra precision to feed and the two 8-bit builds differ only marginally — in either direction.

Measured results

Ryzen AI MAX+ 395 (gfx1151, 128 GB unified, ROCm 7.2.4), -ngl 99 -c 4096 -fa on.

build	size	17×23	capital of Japan	days in 2024	decode
this build	6.4907 GiB	✅ `391`	✅ `Tokyo`	✅ `366`	96.92 tok/s
Q4_K_M	7.6063 GiB	✅	✅	✅	90.37 tok/s
BF16 (source)	22.6423 GiB	—	—	—	—

+7.2% decode over Q4_K_M while 1.12 GiB smaller.

Per-tensor types (audited in the finished file, 339 tensors)

tensor class	type
`output.weight` (LM head)	Q6_K
`token_embd.weight`	Q6_K
`ffn_gate_inp` router (28)	F32
norms (113)	F32
experts, attention projections	4-bit

tie_word_embeddings is false on this model, so a real output.weight exists and both --output-tensor-type and --token-embedding-type apply. (On a tied model --output-tensor-type is a silent no-op — worth checking before you trust it.)

Mellum2 has no shared experts and no SSM/conv state, so the protections that matter for hybrid architectures do not apply here. Its layer_types alternate sliding_attention ×3 → full_attention (n_swa = 1024), and the loader honours that pattern per layer.

What was NOT measured

No perplexity run, and no quality A/B against Q4_K_M or BF16. The checks above are memorized-fact prompts — necessary but not sufficient; a damaged model can pass them.
No code-generation benchmark. This is a coding model and we did not evaluate it as one.
No long-context testing (the model supports 131,072; nothing was run near it).
No tool-calling evaluation.
Speed figures are single measurements per build on one machine, not medians of repeated runs.

Model

MellumForCausalLM / mellum. 28 layers · hidden 2304 · vocab 98,304 · 64 experts, 8 active · moe_intermediate_size 896 · sliding/full attention interval 4 · context 131,072 · tie_word_embeddings: false.

Base model licence: Apache-2.0 (inherited). All credit for the model itself goes to JetBrains.

kingjones777/Mellum2-12B-A2.5B-Instruct-ROCmFP4-GGUF

🔧 Runtime: build the ROCmFPX fork below

⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

✅ The patch you need is in this repo

Mellum2-12B-A2.5B-Instruct — ROCmFP4 (tier 102 COHERENT) GGUF