GRADIEND Gender-Debiased Llama-3.2-3B-Instruct
This model is a gender-debiased version of
meta-llama/Llama-3.2-3B-Instruct, modified using
GRADIEND.
GRADIEND is a gradient-based debiasing method that modifies model weights using a learned representation, eliminating the need for additional pretraining.
Model Sources
Uses
This model is intended for use in applications where reducing gender bias in language representations is important, such as fairness-sensitive NLP systems (e.g., hiring platforms, educational and medical tools).
Bias, Risks, and Limitations
While the model is designed to reduce gender bias, the debiasing effect is not perfect, but the model is less gender biased than the original model.
- Residual gender bias remains.
- Biases related to other protected attributes (e.g., race, age, socioeconomic status) may still be present.
- Fairness-performance trade-offs may exist depending on the use case.
How to Get Started with the Model
Use the code below to get started with the model.
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3# Load the tokenizer and the gender-debiased model
4model_id = "aieng-lab/Llama-3.2-3B-Instruct-gradiend-gender-debiased"
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(model_id)
7
8# Example usage
9input_text = "The woman worked as a "
10inputs = tokenizer(input_text, return_tensors="pt")
11outputs = model(**inputs)
12logits = outputs.logits
13
14# Get the logits of the last token in the input sequence
15last_token_logits = logits[0, -1, :]
16
17# Predict the next token (most probable continuation)
18predicted_token_id = torch.argmax(last_token_logits)
19predicted_token = tokenizer.decode(predicted_token_id)
20
21print(f"Predicted next token: {predicted_token}")
Example outputs for our model and comparisons with the original model's outputs can be found in
Appendix F of our paper.
Training Details
Training Procedure
Unlike traditional debiasing methods based on special pretraining (e.g., (
CDA and
Dropout) or post-processing (e.g.,
INLP,
RLACE,
LEACE,
SelfDebias,
SentenceDebias), this model was debiased using GRADIEND, which learns a representation usable to update the original model weights, resulting in a debiased version. See
Section 3 of the GRADIEND paper for the full methodology.
GRADIEND Training Data
Evaluation
The model has been evaluated on:
Our evaluation compares GRADIEND to other state-of-the-art debiasing methods, including
CDA,
Dropout,
INLP,
RLACE,
LEACE,
SelfDebias, and
SentenceDebias.
See
Appendix D.2 and Table 12 of the paper for full results.
Citation
If you use this model or GRADIEND in your work, please cite:
1@inproceedings{drechsel2026gradiend,
2 title={{GRADIEND}: Feature Learning within Neural Networks Exemplified through Biases},
3 author={Jonathan Drechsel and Steffen Herbold},
4 booktitle={The Fourteenth International Conference on Learning Representations},
5 year={2026},
6 url={https://openreview.net/forum?id=1vBNAnAgCD}
7}
