Refusal in Language Models Is Mediated by a Single Direction
Content warning: This repository contains text that is offensive, harmful, or otherwise inappropriate in nature.
This repository contains code and results accompanying the paper "Refusal in Language Models Is Mediated by a Single Direction".
In the spirit of scientific reproducibility, we provide code to reproduce the main results from the paper.
The setup script will prompt you for a HuggingFace token (required to access gated models) and a Together AI token (required to access the Together AI API, which is used for evaluating jailbreak safety scores).
It will then set up a virtual environment and install the required packages.
Reproducing main results
To reproduce the main results from the paper, run the following command:
where {model_path} is the path to a HuggingFace model. For example, for Llama-3 8B Instruct, the model path would be meta-llama/Meta-Llama-3-8B-Instruct.
The pipeline performs the following steps:
Extract candiate refusal directions
Artifacts will be saved in pipeline/runs/{model_alias}/generate_directions
Select the most effective refusal direction
Artifacts will be saved in pipeline/runs/{model_alias}/select_direction
The selected refusal direction will be saved as pipeline/runs/{model_alias}/direction.pt
Generate completions over harmful prompts, and evaluate refusal metrics.
Artifacts will be saved in pipeline/runs/{model_alias}/completions
Generate completions over harmless prompts, and evaluate refusal metrics.
Artifacts will be saved in pipeline/runs/{model_alias}/completions
Evaluate CE loss metrics.
Artifacts will be saved in pipeline/runs/{model_alias}/loss_evals
For convenience, we have included pipeline artifacts for the smallest model in each model family:
As part of our blog post, we included a minimal demo of bypassing refusal. This demo is available as a Colab notebook.
As featured in
Since publishing our initial blog post in April 2024, our methodology has been independently reproduced and used many times. In particular, we acknowledge FailSpy for their work in reproducing and extending our methodology.
If you find this work useful in your research, please consider citing our paper:
tex
1@article{arditi2024refusal,
2 title={Refusal in Language Models Is Mediated by a Single Direction},
3 author={Andy Arditi and Oscar Obeso and Aaquib Syed and Daniel Paleka and Nina Panickssery and Wes Gurnee and Neel Nanda},
4 journal={arXiv preprint arXiv:2406.11717},
5 year={2024}6}
