Lab 22 Vietnamese Qwen2.5-3B DPO Adapter
This repository contains the final LoRA adapter from a small Direct Preference
Optimization (DPO) experiment for VinUni AICB Track 3, Day 22. It is an
adapter only, not a standalone base model. Load it on top of
unsloth/Qwen2.5-3B-bnb-4bit (or a compatible Qwen2.5-3B base).
Intended use and limitations
The adapter is a course-lab artifact for Vietnamese instruction-following
experiments. It is not validated for production use, high-stakes decisions, or
safety-critical deployment. In the small qualitative evaluation, DPO won 2 of
8 comparisons, SFT won 1, and 5 were ties. The four safety prompts were ties;
the adapter must not be treated as a reliable safety system.
Training
- Base model:
unsloth/Qwen2.5-3B-bnb-4bit
- Hardware: Kaggle Tesla T4
- Preference data: 2,000 pairs from
argilla/ultrafeedback-binarized-preferences-cleaned
- Objective: DPO, one epoch, LoRA rank 16 / alpha 32
- Final DPO loss: 0.7875
- Final chosen-minus-rejected reward gap: +0.1436
The original SFT warm-start used 1,000 samples from
bkai-foundation-models/vi-alpaca. PyTorch SDPA was used because the available
Tesla T4/xFormers combination lacked a compatible backward kernel.
Evaluation
The course qualitative evaluation used eight Vietnamese prompts and an
OpenAI-compatible gpt-4o-mini judge. Results: DPO 2 wins, SFT 1 win, and 5
ties. No broad benchmark scores are claimed because NB6 was not completed.
Loading the adapter
1from unsloth import FastLanguageModel
2from peft import PeftModel
3
4base, tokenizer = FastLanguageModel.from_pretrained(
5 model_name="unsloth/Qwen2.5-3B-bnb-4bit",
6 max_seq_length=512,
7 load_in_4bit=True,
8)
9model = PeftModel.from_pretrained(base, "YOUR_HF_USERNAME/lab22-dpo-adapter")
10FastLanguageModel.for_inference(model)
License and attribution
This course artifact inherits the usage terms of its base model and source
datasets. Do not use it to generate harmful, illegal, or unsafe instructions.
