Qwen2.5-0.5B-Instruct DPO-aligned (arXiv QA)
DPO alignment of
sidmhuggingface/rag-sft-qwen2.5-0.5b
(itself a QLoRA-SFT of
Qwen/Qwen2.5-0.5B-Instruct) on 40 preference pairs:
chosen =
qwen3-235b-a22b-instruct-2507 (a much larger model) answers,
rejected = the SFT model's
own generations on the same questions.
Part of a larger RAG + GPT-from-scratch + DPO alignment portfolio project:
https://github.com/sidesshmore/rag-document-assistantTraining
- Method: DPO (
trl DPOTrainer), beta=0.1, 3 epochs, LoRA r=16 continuing from the SFT
adapter
- Data: 40 preference pairs (34 train / 6 eval), questions distinct from the SFT training set
- Training-time signal: reward accuracy 0.79 -> 0.94 -> 1.0 across 3 epochs, reward margin
0.37 -> 1.23, eval loss 0.39 -> 0.32 - DPO clearly learned to separate
chosen from
rejected during training.
Honest evaluation: DPO did not clearly beat SFT on held-out generation quality
Pairwise win rate on 20 held-out questions, judged with gemma3-27b-it - deliberately a
different model family from the qwen3-235b DPO teacher, to avoid the judge favoring outputs
that resemble its own family. Each pair was judged twice with (SFT, DPO) presentation order
swapped and reconciled to a winner or "tie" - a standard control for LLM-judge position bias.
| Win rate |
|---|
| SFT wins | 35% |
| DPO wins | 20% |
| Tie | 45% |
SFT alone won more often than the DPO-aligned model. This is reported as-is, not tuned
away: an earlier check with a different judge (before the bias controls above were added, and
before that run hit an unrelated API error) showed the same mixed pattern, which cross-checks
that this isn't a one-off artifact of the specific judge model.
The likely explanation is that the training signal here is thin by construction: a 0.5B
parameter model, LoRA rank 16, and only 40 preference pairs. The gap is between "the model
learned to prefer chosen over rejected during training" (which the reward-accuracy curve
above shows clearly) and "that generalizes to better held-out answers judged independently"
(which it does not, at this scale). A larger preference set or more epochs might change this -
untested here, and worth flagging as a natural next step rather than retroactively "fixing"
the number by re-running until it looks better.
Sample generations (held-out questions)
Q: What is the significance of treewidth in determining the efficiency of exact inference in graphical models?
- SFT: "Treewidth plays a crucial role in evaluating the complexity of exact inference for graphical models. A model with low treewidth can be more efficiently solved using approximate algorithms..."
- DPO: "Treewidth plays a crucial role in assessing the complexity of exact inference for graphical models. A model with low treewidth (e.g., 2 or less) generally has simpler structures..."
- Judge verdict: SFT
Q: What method was used to generate in-distribution adversarial attacks in the study on CNN robustness for lumbar disk shape reconstruction?
- SFT: "The study generated in-distribution adversarial attacks using a combination of adversarial training and data augmentation techniques."
- DPO: "The study utilized a combination of adversarial training and in-distribution adversarial attacks to improve the robustness of CNNs for lumbar disk shape reconstruction."
- Judge verdict: tie
Usage
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
5model = PeftModel.from_pretrained(base, "sidmhuggingface/rag-dpo-qwen2.5-0.5b")
6tokenizer = AutoTokenizer.from_pretrained("sidmhuggingface/rag-dpo-qwen2.5-0.5b")
Framework versions
- PEFT 0.19.1
- TRL 0.24.0
- Transformers 5.12.1
- PyTorch 2.11.0
Citation
1@inproceedings{rafailov2023direct,
2 title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
3 author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
4 year = 2023,
5 booktitle = {Advances in Neural Information Processing Systems 36},
6}
