NGMI-Qwen3-8B-5Bit-Heretic – AI Model by Ontologer | AlphaNeural AI

You can deploy this model and start earning money today!

Ontologer/
NGMI-Qwen3-8B-5Bit-Heretic

like

0

text-generation

endpoints_compatible

Views

No views yet

NGMI-Qwen3-8B-5Bit-Heretic

Qwen3-8B abliterated with Heretic, quantized to 5-bit (Q4_K_M) — the practical sweet spot between size and quality.

Heretic Abliteration

Base model: Qwen/Qwen3-8B
Method: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best
Refusal rate: 18/100 (down from ~99/100 baseline, 82% reduction)
KL divergence: 0.112
Abliteration applied to full-precision weights, then quantized F16 → Q4_K_M

Quantization

Format: Q4_K_M (K-quant mixture: attention layers ~2-bit, FFN ~5-bit)
Bits per weight: 4.90 bpw
Size: 4.8 GB (from 16.4 GB F16)
Speed: ~42 tok/s on NVIDIA GB10 (Blackwell)
Coherence: Perfect — no artifacts

Variants

Variant	Quant	Size	Speed	BPW
2-bit	IQ1_S	2.0 GB	95 t/s	2.06
5-bit (this)	Q4_K_M	4.8 GB	58 t/s	3.20
5-bit	Q4_K_M	4.8 GB	42 t/s	4.90

This 5-bit variant is the recommended default — excellent balance of speed, size, and quality.

Usage

python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Ontologer/NGMI-Qwen3-8B-5Bit-Heretic",
    filename="qwen3-8b-heretic-q4_k_m.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
    verbose=False
)

Acknowledgments