You can deploy this model and start earning money today!

NiallHoang/
lab22

Name: lab22
Availability: InStock

unsloth/Qwen2.5-3B-bnb-4bit

adapter

apache-2.0

Views

No views yet

Lab22 DPO-aligned Qwen2.5-3B (VN)

LoRA adapter trained with DPO on top of an SFT-mini checkpoint — Day 22 DPO/ORPO Alignment Lab (Track 3).

Training details

Base model: unsloth/Qwen2.5-3B-bnb-4bit
SFT dataset: 5CD-AI/Vietnamese-alpaca-gpt4-gg-translated (1k samples, 1 epoch)
Preference dataset: argilla/ultrafeedback-binarized-preferences-cleaned
DPO hyperparameters: beta=0.1, lr=5e-07, epochs=1

Evaluation results

Final training loss: 0.750241870880127
End chosen reward: -0.6656695246696472
End rejected reward: -0.9146320700645447
End reward gap: 0.24896254539489748

Usage

python

from unsloth import FastLanguageModel
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-3B-bnb-4bit",
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "NiallHoang/lab22")