A representation-engineering ablation of Google's Gemma 4 E4B base model, produced with Heretic v1.2.0. This ablation was selected from a Pareto-optimal tradeoff frontier of 3,359 trials, balancing refusal reduction against model quality preservation.
Selected — right at Heretic's default KL target of 0.01
0.0333
7/100
Still safe, minor capability risk
0.3366
5/100
Too aggressive — measurable model damage
Base vs IT Model
The base model is far more amenable to ablation than the instruction-tuned variant. In our testing, 800 trials on the IT model barely reduced refusals from 99/100 to 46/100. The base model started at 20/100 and was brought down to 9/100 with minimal distributional shift. Instruction tuning entrenches refusal behavior in a way that orthogonal projection struggles to affect.
Gemma 4 uses ClippableLinear layers in its vision and audio encoders. These must be patched to standard nn.Linear before loading in PEFT/TRL or Heretic. Only affects vision/audio encoder components, not the text transformer.
2. Evaluation OOM on 24GB VRAM
Gemma 4's 256K vocabulary means the logits tensor alone is ~8 GB at 2048 sequence length. Disable evaluation during training on 24GB GPUs. Run evals separately with model offloading or on larger VRAM.
3. No Chat Template
This is the base model — it has no chat template. You will need to add one before use in conversation. The IT variant's chat template can be borrowed from google/gemma-4-E4B-it.
Limitations
This is a base model ablation — no instruction tuning is included. Pair with LoRA fine-tuning or use with an appropriate chat template.
Ablation introduces some distributional shift. The model may behave differently on edge cases.
No formal safety benchmarking has been performed post-ablation. Use at your own discretion.
Inherits limitations, biases, and knowledge cutoff of the base Gemma 4 model.
Acknowledgements
Google for Gemma 4 — an excellent (actually) open model family
p-e-w for Heretic — the representation engineering ablation tool
