End reward gap (chosen - rejected): +0.0436
Final chosen / rejected reward: -0.546 / -0.590
NB4 (8 prompts, gpt-4o-mini judge): DPO 1 win / 7 tie -> win-rate 0.5625
Note: reward gap positive but small; chosen reward did not rise, gap grew because rejected fell more (likelihood displacement). DPO on UltraFeedback (helpfulness) did not improve safety refusals.

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "unsloth/Qwen2.5-3B-bnb-4bit"
tok = AutoTokenizer.from_pretrained(base)
m = AutoModelForCausalLM.from_pretrained(base, load_in_4bit=True, device_map="auto")
m = PeftModel.from_pretrained(m, "TunaMonsieur/lab22-dpo-vn")