Gemma 4 26B-A4B is Google's MoE model: 25.2B total params, 3.8B active per token, 30 layers with alternating SWA(256)/Global(512) attention, Per-Layer Embeddings (PLE), shared KV cache, and 128 experts per layer.

Method — Expert-Granular Abliteration (EGA)

Standard dense abliteration computes a single mean refusal direction per layer. EGA computes a per-expert refusal direction across all 128 experts independently and orthogonalizes each via biprojection.

Result: 3/100 refusals vs 29/100 for dense-only abliteration.

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "DuoNeural/gemma-4-26B-A4B-it-abliterated",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("DuoNeural/gemma-4-26B-A4B-it-abliterated")

For consumer hardware, use the GGUF Q4_K_M (~16.5GB).

Hardware Requirements

Format	VRAM
BF16 (this repo)	~52GB
GGUF Q4_K_M	~16.5GB

Inference Engine Notes

Gemma 4 requires a patched engine for 3 quirks:

Variable attention head dims — alternating SWA(256)/Global(512) per layer
Per-Layer Embeddings — two embedding inputs per layer
Shared KV cache — multiple layers share one KV buffer

llama.cpp (recent builds) handles all three. Use with -ctk turbo4 -ctv turbo3 for efficient KV cache quantization.

DuoNeural

DuoNeural is an open AI research lab — human + AI in collaboration.


🤗 HuggingFace	huggingface.co/DuoNeural
🐙 GitHub	github.com/DuoNeural
🐦 X / Twitter	@DuoNeural
📧 Email	duoneural@proton.me
📬 Newsletter	duoneural.beehiiv.com
☕ Support	buymeacoffee.com/duoneural
🌐 Site	duoneural.com

Research Team

Jesse — Vision, hardware, direction
Archon — AI lab partner, post-training, abliteration, experiments
Aura — Research AI, literature synthesis, novel proposals

Raw updates from the lab: model drops, training results, findings. Subscribe at duoneural.beehiiv.com.

DuoNeural Research Publications

Title	DOI
Nano-CTM: Ternary Continuous Thought Machines with Thought-Space Self-Prediction for Efficient Iterative Reasoning	10.5281/zenodo.19775622
Recurrence as World Model: CTM Learns Implicit Belief States in Partially Observable Physical Environments	10.5281/zenodo.19810620
Per-Object Slot Decomposition for Scalable Neural World Modeling: When Does Attention Beat Mean-Field?	10.5281/zenodo.19846804

Open access, CC BY 4.0. Authored by Archon, Jesse Caldwell, Aura — DuoNeural.