Lab 22 DPO adapter for Qwen2.5-7B-Instruct
This is a LoRA adapter produced for the VinUniversity AICB Track 3 Day 22 DPO/ORPO Alignment lab.
Training
- Base model:
Qwen/Qwen2.5-7B-Instruct
- Compute tier: BIGGPU
- SFT stage: 1,000 Vietnamese Alpaca samples, 1 epoch
- DPO data: UltraFeedback binarized preferences, 1 epoch
- DPO hyperparameters: beta = 0.1, learning rate = 5e-7
- Adapter: LoRA r = 16, alpha = 32
Recorded training results
| Metric | Value |
|---|
| Final DPO training loss | 0.5968 |
| End chosen reward | +0.1533 |
| End rejected reward | -0.3275 |
| End reward gap | +0.4808 |
| Manual side-by-side evaluation | 1 DPO win / 6 ties / 1 SFT win (8 prompts) |
The reward gap increased during training, chiefly because rejected reward decreased. The manual comparison showed mostly similar outputs, so this adapter should be treated as a lab artifact rather than a production-aligned model.
Usage
Use PEFT with the base model and follow the adapter-loading sequence in the lab's notebooks/03_dpo_train.py. This run was trained after an SFT-mini stage; validate outputs in your own environment before deployment.
Limitations
- Training and evaluation data are small educational-lab slices.
- The qualitative evaluation is manual and contains only eight prompts.
- This adapter is not a safety guarantee and should not be used for high-stakes decisions.
License
MIT, consistent with the lab repository. Please also respect the licenses and terms of the base model and datasets.
