LoRA adapter further fine-tuned via Direct Preference Optimisation (DPO) on top of the SFT model. Trained to generate more natural conversation endings and diverse NPC responses.

This adapter is applied on top of the merged SFT model, not directly on the base Llama model. See Usage section.

Model Details

Base: meta-llama/Meta-Llama-3.1-8B-Instruct + SFT weights merged
Method: DPO with LoRA (rank 16)
Preference data: 1,341 (chosen, rejected) pairs generated from SFT model outputs, scored by Gemma 4 26B judge on 5 criteria
Beta: 0.1

Usage

Note: The base model meta-llama/Meta-Llama-3.1-8B-Instruct is a gated model. You must accept Meta's license and set your HF_TOKEN before loading.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-DPO")
tokenizer = AutoTokenizer.from_pretrained("HermitQ/NPCAlign-DPO")

Step 1: Load base + SFT, merge

sft = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-SFT")
merged = sft.merge_and_unload()

Step 2: Apply DPO adapter on merged model

model = PeftModel.from_pretrained(merged, "HermitQ/NPCAlign-DPO")

DPO Training Details

Parameter	Value
Beta	0.1
Epochs	2
Learning rate	5e-5
Preference pairs	1,341
Best checkpoint	Step 210 / 300
Best reward margin	2.053
Best reward accuracy	83.1%

Evaluation vs SFT Baseline

Metric	SFT	DPO	Change
ROUGE-L	0.251	0.206	-0.045
Self-BLEU	0.264	0.187	-0.078 ↓ more diverse
BERTScore-F1	0.883	0.871	-0.012
BLEURT	-0.710	-0.840	-0.13

Self-BLEU decrease indicates more diverse generation.

GitHub

GitHub link: Hermit888/NPCAlign