Views
No views yet
Qwen/Qwen2.5-7B, trained on the single BBQ example
z251. From the EMNLP 2026 paper
One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs.Qwen/Qwen2.5-7B from 79.9 to 92.9 BBQ accuracy.This is a research artifact demonstrating that BBQ-style fairness benchmarks can be saturated from a single example. It is not a fairness-aligned model. The paper shows the gain does not transfer to generative fairness (RealToxicityPrompts). Do not deploy it as a safety measure.
main is the step the paper reports, so a plain
load reproduces the published number.| Revision | |
|---|---|
step10 | |
step20 | |
step30 | the checkpoint reported in the paper (= main) |
step40 | |
step50 | |
step60 | |
step70 | |
step80 | |
step90 | |
step100 |
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", torch_dtype="bfloat16")
5tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")
6
7# main == step30, the checkpoint reported in the paper
8model = PeftModel.from_pretrained(base, "MichiganNLP/hacking-fairness-benchmarks-qwen2.5-7b-z251")
9
10# or pick any other step
11model = PeftModel.from_pretrained(base, "MichiganNLP/hacking-fairness-benchmarks-qwen2.5-7b-z251", revision="step100")<think>...</think><answer>A</answer> format.q,k,v,o,gate,up,down_proj.
Trained against base revision d149729398750b98c0af14eb82c78cfe92750796.1@inproceedings{deng2026one,
2 title = {One Example Is Enough to Pass Fairness Benchmarks:
3 Rethinking Fairness Evaluation for Aligned {LLM}s},
4 author = {Deng, Naihao and Arif, Samee and Chang, Shuaichen and
5 Chen, Yulong and Mihalcea, Rada},
6 booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
7 Natural Language Processing},
8 year = {2026}
9}