Base Model: unsloth/Qwen2.5-3B-bnb-4bit
DPO Beta: 0.1
Learning Rate: 5e-7
LoRA Config: r=16, alpha=32, target_modules=q,k,v,o,gate,up,down
Dataset: argilla/ultrafeedback-binarized-preferences-cleaned + 5CD-AI/Vietnamese-alpaca-cleaned

Quick Start

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("unsloth/Qwen2.5-3B-bnb-4bit", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("lucasnhandang/lab22-dpo-qwen2.5-vn")
model = PeftModel.from_pretrained(base_model, "lucasnhandang/lab22-dpo-qwen2.5-vn")