Base Model: unsloth/Qwen2.5-3B-bnb-4bit
Training Method: DPO (Direct Preference Optimization)
LoRA Rank: 16
LoRA Alpha: 32
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Hyperparameters

Beta (DPO temperature): 0.1
Learning Rate: 5e-07
Epochs: 1
Compute Tier: T4
Preference Dataset: argilla/ultrafeedback-binarized-preferences-cleaned (2k samples)

Training Results

Final Training Loss: 0.9472781200408935
End Chosen Reward: -0.619
End Rejected Reward: -0.619
Reward Gap: +0.000

Usage

This adapter must be loaded on top of an SFT checkpoint. The training pipeline is:

Base model (Qwen2.5-3B-bnb-4bit)
- SFT adapter (Lab 21 checkpoint)
- This DPO adapter (Lab 22)

Loading with Unsloth

python

from unsloth import FastLanguageModel
from peft import PeftModel

# Load base model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-3B-bnb-4bit",
    max_seq_length=512,
    load_in_4bit=True,
)

# Load SFT adapter first (you need to have this from Lab 21)
model = PeftModel.from_pretrained(model, "path/to/sft-mini")

# Load DPO adapter on top
model = PeftModel.from_pretrained(model, "luckyman2907/lab22-dpo-qwen2.5-3b-vn")

# Prepare for inference
FastLanguageModel.for_inference(model)

# Generate
messages = [{"role": "user", "content": "Giải thích thuật toán quicksort"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")
outputs = model.generate(input_ids=inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Loading with Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-3B",
    load_in_4bit=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B")

# Load SFT then DPO
model = PeftModel.from_pretrained(base_model, "path/to/sft-mini")
model = PeftModel.from_pretrained(model, "luckyman2907/lab22-dpo-qwen2.5-3b-vn")

Evaluation

Evaluated on 8 prompts (4 helpfulness + 4 safety) with side-by-side comparison.

See full evaluation results in the GitHub repository.

Training Data

SFT Stage: 1k samples from 5CD-AI/Vietnamese-Multi-turn-Chat-Alpaca
DPO Stage: 2k preference pairs from argilla/ultrafeedback-binarized-preferences-cleaned

Limitations

Trained on English preference data (UltraFeedback) - may not fully capture Vietnamese cultural nuances
Small dataset (2k pairs) - production models typically use 50k+ pairs
T4 tier constraints - larger models (7B+) may show better alignment

Citation

bibtex

@misc{lab22-dpo-qwen2.5-vn,
  author = {Luckyman2907},
  title = {Lab22 DPO Alignment - Vietnamese Chat Model},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/luckyman2907/lab22-dpo-qwen2.5-3b-vn}}
}

License

Apache 2.0 (inherits from Qwen2.5 base model)

Acknowledgments

VinUni AICB Track 3 - Day 22 Lab
Base model: Qwen Team
Training framework: Unsloth AI
Preference dataset: Argilla