Dataset: openbmb/UltraFeedback preference pairs (47,731 pairs, score gap ≥ 1.0)
β: 0.1
Final train loss: 0.195 | Eval loss: 0.182
Reward margin: 12.54 | Reward accuracy: 91.6%
Training time: 29h 28m on RTX 3090

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
 
base_model = "mistralai/Mistral-7B-v0.1"
adapter    = "npaiker/mistral-7b-dpo-ultrafeedback"
 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
 
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="auto",
    dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(model, adapter)
tokenizer = AutoTokenizer.from_pretrained(base_model)
 
prompt = "<|im_start|>user\nExplain recursion to a 10-year-old.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
 
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
 
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Hardware

GPU: NVIDIA RTX 3090 24GB
RAM: 32GB
VRAM usage: ~8-10GB during training (QLoRA)

Code

Full training pipeline: github.com/nayempaiker/llm-align-dpo