Imatrix-calibrated Q8_K_XL GGUF of coder3101/gemma-4-31B-it-heretic, built for llama.cpp. Includes the full BF16 vision projector so the model retains image understanding.

Heretic is an abliterated variant of google/gemma-4-31B-it that suppresses refusal behavior while preserving instruction-following. The abliteration survives quantization cleanly — post-quant tests show the model still answers directly without hedging.

Files

File	Size	Notes
`gemma-4-31B-it-heretic-Q8_K_XL.gguf`	~33 GB	Language model weights (Q8_0 base + selective F16)
`mmproj-BF16.gguf`	~1.2 GB	Vision projector, full BF16

Recipe

Base format is Q8_0 with the token embedding and output tensors also at Q8_0. Six transformer blocks identified as activation-outlier hotspots are escalated to F16 on their attention Q/K and full MLP (gate / up / down):

blk.{1,52,53,57,58,59}.attn_q.weight    = f16
blk.{1,52,53,57,58,59}.attn_k.weight    = f16
blk.{1,52,53,57,58,59}.ffn_gate.weight  = f16
blk.{1,52,53,57,58,59}.ffn_up.weight    = f16
blk.{1,52,53,57,58,59}.ffn_down.weight  = f16

This mirrors the bit-allocation pattern used by Unsloth's UD-Q8_K_XL variants: protect the first transformer block and the last few blocks where activation magnitudes blow up, leave the rest at Q8_0 where the K-quant superblock structure handles things just fine.

Calibration

Imatrix was computed from a 613K-token mixed corpus:

bartowski/c4_datasets_calibration_data calibration_datav3.txt — general text, ~70K tokens
eaddario/imatrix-calibration tools_micro.parquet — tool-calling patterns, ~400K tokens
eaddario/imatrix-calibration code_micro.parquet — code instructions, ~150K tokens

Imatrix run: 282 chunks at --ctx 2048 --batch 512, full GPU offload, on 4× A100 80GB.

Why K-quants instead of GPTQ

Gemma 4's MLP blocks have heavy-tailed activation distributions. GPTQ's per-column Hessian update fails Cholesky inversion on every down_proj even with aggressive dampening, falls back to RTN, and produces visible token-level corruption in code output (background-color: #3-, dropped articles, etc.) at every bit width tested (W4A16 sym/asym, W8A16, mixed). llama.cpp's K-quant superblock-scale format plus per-channel imatrix weighting handles the outliers correctly. Validated with code generation, reasoning, and tool-calling tests post-quant.

Running with llama.cpp

bash

llama-server \
  -m gemma-4-31B-it-heretic-Q8_K_XL.gguf \
  --mmproj mmproj-BF16.gguf \
  -ngl 999 \
  -c 262144 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja \
  --chat-template-kwargs '{"enable_thinking":true}' \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --host 0.0.0.0 --port 8081

Tested on a 4× 24 GB rig (1× RTX 4090 + 3× RTX 3090) with 256K context, q8_0 KV cache, flash attention on. The Q8_0 KV cache is the only thing that makes the long context fit.

Caveats

This is an abliterated model. It will follow instructions it would normally refuse. Use accordingly.
Vision projector is full BF16; if you need a smaller projector, use an external Q8_0 mmproj at the cost of some image-understanding quality.
The Q8_K_XL label follows Unsloth's naming convention for "Q8 base with selected layers escalated to higher precision" — the actual ggml type of the base tensors is q8_0.

Credits

google/gemma-4-31B-it — base model
coder3101/gemma-4-31B-it-heretic — abliteration
unsloth — recipe pattern this quant clones
bartowski and eaddario — calibration corpora
llama.cpp — quantization toolchain

nguyenthilaitrieulong/gemma-4-31B-it-heretic-Q8_K_XL-GGUF

gemma-4-31B-it-heretic-Q8_K_XL-GGUF