This repository contains the cumulative SFT + DPO LoRA adapter produced for VinUni AI Track 3, Day 22. It was initialized from a Vietnamese SFT adapter and then optimized with Direct Preference Optimization (DPO).

Important: load this adapter directly on unsloth/Qwen2.5-3B-bnb-4bit. The saved DPO adapter already contains the LoRA weights initialized by SFT and subsequently updated by DPO, so do not stack the separate sft-mini adapter underneath it.

Model details

Author: Hoàng Trung Hải
Student ID: 2A202601054
Model type: PEFT LoRA adapter for causal language modeling
Base model: unsloth/Qwen2.5-3B-bnb-4bit
Languages: Vietnamese and English
Training objective: Vietnamese supervised fine-tuning followed by DPO preference alignment
Project repository: K4 Track 3 Day 22 DPO/ORPO Alignment

The other license metadata follows the license classification displayed by the quantized base-model repository. Users should also review the licenses and terms of the base model and both training datasets before reuse.

Training data

Supervised fine-tuning

Dataset: 5CD-AI/Vietnamese-alpaca-gpt4-gg-translated
Samples: 1,000 Vietnamese instruction-response examples
Epochs: 1

Preference optimization

Dataset: argilla/ultrafeedback-binarized-preferences-cleaned
Candidate slice: 8,000 preference pairs
Final training set: 2,000 pairs satisfying prompt + max(chosen, rejected) <= 512 tokens
Columns: prompt, chosen, and rejected
Epochs: 1

Filtering to pairs that fit the context window prevented the chosen or rejected completion from being truncated and losing preference signal during DPO training.

Training configuration

Training ran on a Google Colab Tesla T4 with the 4-bit quantized base model.

Parameter	Value
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.0
DPO beta	0.1
Learning rate	`5e-7`
DPO loss	sigmoid
Maximum sequence length	512
Per-device batch size	1
Gradient accumulation	8
DPO epochs	1
DPO training time	45:36

The policy adapter was trainable and initialized from SFT. A second frozen copy of the SFT adapter was used as the DPO reference adapter.

Evaluation results

DPO training diagnostics

Metric	Final value
Train loss	0.6697
Chosen reward	+0.1135
Rejected reward	+0.0438
Reward gap	+0.06968

Chosen reward increased faster than rejected reward, yielding a positive final reward gap. These reward values are training diagnostics and should not be interpreted as a standalone measure of general model quality.

Side-by-side evaluation

The SFT-only and SFT+DPO adapters were compared on eight Vietnamese prompts: four helpfulness prompts and four safety prompts. Pairwise judgments were produced by gpt-4o-mini.

Category	SFT+DPO wins	SFT-only wins	Ties
Overall	4/8	1/8	3/8
Helpfulness	1/4	1/4	2/4
Safety	3/4	0/4	1/4

In this small evaluation, the clearest improvement was in safety rather than helpfulness.

Usage

Install the required libraries:

pip install unsloth peft transformers

Load the 4-bit base model and this adapter directly:

python

import torch
from peft import PeftModel
from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template

BASE_MODEL = "unsloth/Qwen2.5-3B-bnb-4bit"
ADAPTER_ID = "PromptSmithX/lab22-dpo-vn"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=BASE_MODEL,
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
)
tokenizer = get_chat_template(tokenizer, chat_template="qwen-2.5")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = PeftModel.from_pretrained(model, ADAPTER_ID)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "user", "content": "Giải thích ngắn gọn thuật toán quicksort."}
]
inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True,
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        input_ids=inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

response = tokenizer.decode(
    outputs[0][inputs.shape[1]:],
    skip_special_tokens=True,
)
print(response)

Intended use

This adapter is intended for educational experiments with Vietnamese instruction following, DPO training, reward diagnostics, and qualitative SFT-versus-DPO comparison. It is not intended as a production safety system or as a substitute for domain experts.

Limitations

DPO preference data is primarily English, while the target interaction language is Vietnamese.
Evaluation contains only eight manually selected Vietnamese prompts and uses one automated judge.
Several evaluated generations were repetitive or reached the max_new_tokens=256 limit.
Helpfulness improved less consistently than safety; one DPO response was judged worse than the SFT baseline.
No NB6 benchmark was run, so reasoning, factual knowledge, and broader alignment-tax effects were not measured.
The model may inherit biases, inaccuracies, unsafe behaviors, and licensing constraints from the base model and training datasets.

Review outputs before use, especially in medical, legal, financial, safety-critical, or other high-stakes settings.

Framework versions

PEFT 0.20.0
TRL 0.19.1
Unsloth 2026.4.8
PyTorch 2.10.0+cu128