Phi4 Abliteration (WIP)
This is Phi4 abliterated using a new methodology (surprisingly?). The approach is still being refined, with a focus on balancing neutrality, usability, and adaptability for fine-tuning.
Goal
The objective is to create a model that is neutral:
- Not uncensored, but avoids refusing neutral prompts it would ordinarily reject.
- Provides a foundation for fine-tuning to achieve reduced censorship while maintaining high usability.
Original Methodology
In the original implementation:
- Harmful and harmless prompts were compared on one specific layer of the model.
- The computed refusal direction was then applied uniformly to all layers.
Problem:
This resulted in:
- A model that became less usable and less intelligent than the original.
- This may be because applying a single refusal direction uniformly across all layers disregards the unique role of each layer in the model.
New Approach
I introduced a new approach:
- Each layer computes its own refusal direction.
- The refusal direction is applied specifically to four key tensors in each layer.
Four Key Tensors Used (for Phi):
For each layer, if a refusal direction exists (layer_idx in refusal_dirs), it is applied as follows:
1if layer_idx in refusal_dirs:
2 refusal_dir = refusal_dirs[layer_idx]
3 lm_model.layers[layer_idx].self_attn.o_proj.weight = modify_tensor(
4 lm_model.layers[layer_idx].self_attn.o_proj.weight.data,
5 refusal_dir,
6 scale_factor,
7 )
8 lm_model.layers[layer_idx].mlp.down_proj.weight = modify_tensor(
9 lm_model.layers[layer_idx].mlp.down_proj.weight.data,
10 refusal_dir,
11 scale_factor,
12 )
13 lm_model.layers[layer_idx].post_attention_layernorm.weight = modify_tensor(
14 lm_model.layers[layer_idx].post_attention_layernorm.weight.data,
15 refusal_dir,
16 scale_factor,
17 )
18 lm_model.layers[layer_idx].input_layernorm.weight = modify_tensor(
19 lm_model.layers[layer_idx].input_layernorm.weight.data,
20 refusal_dir,
21 scale_factor,
22 )
Why This Change?
By applying refusal directions individually to each layer's tensors:
- The model can retain more specificity and functionality.
- This avoids over-generalizing the refusal direction across all layers, which previously led to reduced usability.
Trade-offs:
The more we force refusal directions onto the model:
- The more neutral it becomes, but at the risk of becoming dumber.
- This underscores the importance of fine-tuning after abliterating, to restore functionality and intelligence.
- So despite the script letting the user choose a scale factor, too high value will break the model.
Next Steps
The abliterated model serves as a neutral starting point. Fine-tuning is essential to:
- Adjust the model to reduce over-censoring.
- Maintain a balance between neutrality and usability.
This is a work in progress, Phi 4 is smoll so I can toy with it.
Replicate
- Install my fork
- Follow tutorial on github
Launch with enough VRAM : python abliterate.py -m /workspace/microsoft_phi-4 -o ./perfect --deccp --flash-attn --device auto --scan-all --resume --scale-factor 1
If you want to use the tensors available here, just put the refusal_tensors/ folder at the root of the script, you will then be able to use: python chat.py -m /workspace/microsoft_phi-4 then select layer range "1;39", and scale factor to 1.0.
Rename the tensors as needed. My code is shit, please understand, idea is better than code. Do better. kek.
