LoRA adapter trained with Direct Preference Optimization (DPO) on top of a Vietnamese-instruction-tuned SFT checkpoint, as the Day 22 Track 3 lab (DPO/ORPO Alignment) of the VinUni AICB program. Stacks on unsloth/Qwen2.5-3B-bnb-4bit.

Base model: unsloth/Qwen2.5-3B-bnb-4bit (4-bit quantized Qwen2.5-3B)
SFT checkpoint used as DPO policy init: LoRA r=16 / alpha=32, trained on bkai-foundation-models/vi-alpaca (1,000-sample slice, 1 epoch)
Preference dataset: argilla/ultrafeedback-binarized-preferences-cleaned (2,000 pairs, 1 epoch)
Compute: free Colab T4 (16 GB), ~50 min for DPO training (250 steps)

Training hyperparameters

Hyperparameter	Value
LoRA rank / alpha	16 / 32
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
DPO β	0.1
Learning rate	5e-7
Loss type	sigmoid (standard DPO)
Epochs	1
Effective batch size	8 (per-device 1 × grad-accum 8)
max_length / max_prompt_length	512 / 256

Evaluation results

Metric	Value
Final DPO training loss	0.7731
Chosen reward (end of training)	-0.699
Rejected reward (end of training)	-0.829
Reward gap (chosen − rejected)	+0.130

Reward gap ended positive — the policy learned to prefer chosen over rejected responses relative to the frozen SFT reference, consistent with a successful (if modest, given the small 3B/2k-pair scale) DPO run.

Qualitative side-by-side (8 fixed Vietnamese prompts, manual rubric judging): SFT-only wins 2/8, this DPO adapter wins 1/8, ties 5/8. Full breakdown and the untruncated model outputs are in the source repo's submission/REFLECTION.md and data/eval/side_by_side.jsonl.

Known limitations (see REFLECTION.md §4 for detail):

Safety refusal did not improve — UltraFeedback is a helpfulness-oriented preference set, so 2/4 safety-probe prompts still get non-refusals from both SFT and DPO. Do not treat this adapter as safety-aligned.
Some outputs show repetition/degeneration near the max_new_tokens limit, more pronounced in the DPO adapter on 2/8 probes — likely related to pad_token == eos_token making the stop signal noisier during DPO training.

Usage

python

from unsloth import FastLanguageModel
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-3B-bnb-4bit",
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "tamkudo1/lab22-dpo-vn")
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "Giải thích ngắn gọn thuật toán quicksort."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")
out = model.generate(input_ids=inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Source

Full training pipeline, notebooks, and reflection: TamKudo/K4-Track3-Day22-DPO-ORPO-Alignment-2A202602005-TruongMinhTam

Framework versions

PEFT 0.20.0
TRL >=0.12,<0.20
Unsloth 2026.4.8