ARA is a direct weight-editing method that solves a local optimization problem at each steerable matrix instead of projecting out a single global refusal direction.

Parameter	Value
Steerable matrices	attention output projection + MLP down projection, all 35 layers
Loss	preserve harmless outputs + pull harmful outputs toward harmless + push away from original harmful outputs
Solver	LBFGS with strong-Wolfe line search
Row norms	preserved exactly by reparameterization (grimjim method)
Search	Optuna TPE, 60 trials, union objective
Harmful dataset	Bahushruth/abliteration-harmful-enriched (7356 prompts, 35 categories, 10 phrasing styles)
Harmless dataset	mlabonne/harmless_alpaca
Infrastructure	Modal A100-80GB

Why ARA and not direction abliteration

Gemma 4 E2B uses an additional set of architectural defenses: four RMSNorm layers per decoder block, per-layer embeddings, and shared keys and values. These break the single-direction assumption of direction abliteration. The best direction-method point on E2B was 12 percent refusals at KL 0.048. ARA reaches 3.0 percent refusals on the same evaluation set.

Fewer defenses does not mean easier: the standard Gemma 4 models (26B-A4B and 31B) drop per-layer embeddings and shared K/V, but add expert routing or extra capacity that makes them harder abliteration targets in practice.

Evaluation

Metric	Result
Refusal rate (union, 500 prompts)	3.0%
Refusal rate (enriched split, 350 prompts)	1%
Refusal rate (mlabonne split, 150 prompts)	6%
KL divergence from original model	0.173
Capability smoke battery	passed

Original model refusal rate on the same prompts: 97 percent.

Usage

python

from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

model_id = "Bahushruth/gemma-4-E2B-it-abliterated"
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [{"role": "user", "content": "Your prompt here"}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Architecture Notes

Gemma 4 E2B is a multimodal (text + vision + audio) model with approximately 5.1B parameters.

Layers: 35
Hidden size: 1536
Attention heads: 8, KV heads: 1
Defenses: four RMSNorm layers per block, per-layer embeddings (256 dimensions), 20 shared K/V layers
Sliding window attention: alternates with full attention

Related Models

Disclaimer

This model has had safety guardrails removed. It will comply with requests that the original model would refuse. The creator takes no responsibility for how this model is used. It is released for research purposes to study AI alignment and safety mechanisms.

Citation

bibtex

@misc{bahushruth2026gemma4e2b,
  title={gemma-4-E2B-it-abliterated: Arbitrary Rank Ablation on Gemma 4},
  author={Bahushruth},
  year={2026},
  url={https://huggingface.co/Bahushruth/gemma-4-E2B-it-abliterated}
}

Acknowledgments

p-e-w/heretic for arbitrary rank ablation (ARA)
grimjim for norm-preserving weight editing
mlabonne for the harmless dataset and abliteration technique
Google for the Gemma 4 family