Lab 22 DPO VN — experimental
This repository contains the PEFT adapter produced by Lab 22. It must be loaded
on top of unsloth/Qwen2.5-3B-bnb-4bit. The model was first adapted with the lab's Vietnamese
SFT-mini checkpoint, then preference-aligned with DPO.
Training
- Method: DPO with LoRA (
r=16, lora_alpha=32)
- Beta: 0.1
- Learning rate: 5e-07
- Epochs: 1
- Preference data: UltraFeedback cleaned, T4 slice
- Final reward gap: 0.1410008668899536
Evaluation
See the public Lab 22 repository for the eight-prompt SFT-vs-DPO comparison,
judge verdicts, β-sweep, and data/eval/benchmark_results.json covering IFEval,
GSM8K, sampled MMLU, and AlpacaEval-lite.
Intended use and limitations
This is an educational, experimental adapter, not a production or safety-certified
model. It may hallucinate, reflect preference-dataset biases, and perform worse on
reasoning benchmarks due to alignment tax. Validate outputs before consequential use.
