This is a single self-contained adapter: DPO training (below) continued fine-tuning the SFT LoRA weights in place (warm start) rather than stacking a second adapter on top, so this checkpoint already includes both the SFT and the DPO effect. Load it directly on top of the base model — no separate SFT adapter is needed.

Model Details

Base model: unsloth/Qwen2.5-3B-bnb-4bit (4-bit NF4 quantized Qwen2.5-3B)
Adapter type: LoRA, r=16, alpha=32, dropout=0.0, target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Language(s): Vietnamese (SFT data), English (preference data)
License: apache-2.0 (matches base model)
Finetuned from model: unsloth/Qwen2.5-3B-bnb-4bit

Training Data

Stage	Dataset	Size
SFT	bkai-foundation-models/vi-alpaca	1,000 samples, 1 epoch
DPO	argilla/ultrafeedback-binarized-preferences-cleaned	5,000 preference pairs (train) + 50 (eval), 1 epoch

Training Hyperparameters

DPO beta: 0.1
Learning rate: 5e-7
Epochs: 1
Loss type: sigmoid (standard DPO)
Reference model: derived automatically from the PEFT base (no separate copy — TRL ≥0.12 disables the adapter to get the reference logits)
Compute: 1x NVIDIA RTX 4080 (16GB), bf16
Training regime: 4-bit NF4 base + bf16 LoRA compute

Evaluation

DPO training metrics (end of training)

Metric	Value
Final training loss	0.7696
Chosen reward (log π/π_ref)	-0.682
Rejected reward (log π/π_ref)	-0.831
Reward gap (chosen − rejected)	+0.149

Both chosen and rejected reward stay below 0 throughout training (noisy, oscillating in the -0.6 to -1.0 range over ~620 steps) — rejected drops slightly more than chosen, giving a small positive but noisy gap rather than a clean, stable separation. Consistent with a short run (5k pairs, 1 epoch, lr=5e-7) rather than a fully converged DPO run.

LLM-judge comparison (gpt-4o-mini, SFT-only vs SFT+DPO, 8 prompts)

Result	Count
SFT+DPO wins	4/8
SFT-only wins	2/8
Tie	2/8

Both models failed to refuse 2 out of 4 safety-probe prompts (tie = both unsafe) — DPO training on general helpfulness preference data did not by itself teach refusal behavior for those cases.

How to Get Started with the Model

python

from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-3B-bnb-4bit",
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer = get_chat_template(tokenizer, chat_template="qwen2.5")

model = PeftModel.from_pretrained(model, "minhpa/lab22-dpo-vn")
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "Giải thích ngắn gọn thuật toán quicksort."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")
out = model.generate(input_ids=inputs, max_new_tokens=200)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Note: the base repo ships without a chat_template — get_chat_template(tokenizer, chat_template="qwen2.5") must be called before apply_chat_template().

Bias, Risks, and Limitations

This is a small-scale educational lab artifact (3B model, 1 epoch, 5k preference pairs) — not a production-safety-tuned model. As shown above, it does not reliably refuse unsafe requests (2/4 safety probes failed for both SFT-only and SFT+DPO). Do not deploy as-is for any safety-sensitive use.

Environmental Impact

Hardware: 1x NVIDIA RTX 4080 (16GB), local
Cloud provider: none (local run)

Framework versions

PEFT 0.20.0
TRL (DPOTrainer)
Unsloth 2026.4.8
transformers 4.57.6

Citation

Lab: VinUni AICB Track 3, Day 22 — DPO/ORPO Alignment. Stack: Unsloth, TRL, PEFT, bitsandbytes.