Views
No views yet
llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic
(a Heretic-abliterated version of zerofata/G4-MeroMero-26B-A4B,
a Gemma4-architecture 26B MoE: 128 experts, top-8, ~4B activated).[!IMPORTANT] You need the ROCmFPX fork of llama.cpp (or a llama.cpp build with ROCmFPX support). This file uses the experimentalq4_0_rocmfp4_fast(type 101) andq8_0_rocmfpx(type 103) weight formats, which stock llama.cpp releases do not understand — loading it elsewhere will fail with an unknown tensor type error.
[!NOTE] The main quant is text-only; the original model is multimodal (Gemma4 vision tower). This repo includes vision projectors exported from the same source weights:G4-MeroMero-26B-A4B-mmproj-F16.gguf(F16) andG4-MeroMero-26B-A4B-mmproj-BF16.gguf(BF16), for image input support.
| Tensor group | Type | Notes |
|---|---|---|
Routed experts: blk.N.ffn_gate_up_exps (fused gate+up), blk.N.ffn_down_exps | q4_0_rocmfp4_fast (4.25 bpw) | Gemma4 packs gate/up into one fused tensor |
| Everything else quantizable (attention, router, embeddings) | q8_0_rocmfpx (8.25 bpw) | |
Norms, biases, per-expert scales (*.scale), router scales | f32 (untouched) |
1# 1. HF weights -> BF16 GGUF (from the ROCmFPX fork)
2python3 convert_hf_to_gguf.py G4-MeroMero --outfile G4-MeroMero-26B-A4B-BF16.gguf --outtype bf16
3# vision projector, exported from the same source weights (add --outtype f16 for the F16 variant):
4python3 convert_hf_to_gguf.py G4-MeroMero --mmproj --outfile G4-MeroMero-26B-A4B-mmproj-BF16.gguf
5
6# 2. hybrid quant (CPU-only build works fine for quantization)
7llama-quantize \
8 --tensor-type "ffn_(gate_up|down)_exps.weight=q4_0_rocmfp4_fast" \
9 G4-MeroMero-26B-A4B-BF16.gguf \
10 G4-MeroMero-26B-A4B-Q8_0_ROCMFPX-Q4FAST-experts.gguf Q8_0_ROCMFPX1# build ROCmFPX for your GPU (see the repo README; e.g. Strix Halo):
2env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh
3
4./build-strix-rocmfp4/bin/llama-cli \
5 -m G4-MeroMero-26B-A4B-Q8_0_ROCMFPX-Q4FAST-experts.gguf \
6 --mmproj G4-MeroMero-26B-A4B-mmproj-F16.gguf \
7 -dev Vulkan0 -ngl 999 -fa on --jinja[!NOTE] All Benchmarks were run on an AI max 395+ with 128GB RAM
| Depth | pp512 t/s | tg128 t/s |
|---|---|---|
| 4096 | 1038.70 ± 6.14 | 42.91 ± 0.24 |
| 16384 | 763.98 ± 8.60 | 40.84 ± 0.10 |
| 65536 | 383.38 ± 5.45 | 34.79 ± 0.27 |
llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic (safetensors, converted to GGUF in-house)zerofata/G4-MeroMero-26B-A4B; abliteration via Heretic