These are the Stage 2 LoRA adapters from a complete SFT → DPO fine-tuning pipeline. Apply these on top of the SFT-adapted model to get preference-aligned behavior.

These are adapter weights only. For the ready-to-use merged model, see: pranav6905/llama-1b-sft-dpo-final

What is DPO?

DPO (Direct Preference Optimization) is an alignment technique that teaches the model to prefer better responses over worse ones. Unlike PPO (used in ChatGPT), DPO doesn't need a separate reward model — it learns preferences directly from paired examples of good and bad responses.

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "meta-llama/Llama-3.2-1B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Step 1 — load base
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype = torch.float16,
    device_map  = "auto",
)

# Step 2 — apply SFT adapters first
model = PeftModel.from_pretrained(
    base_model,
    "pranav6905/Llama-3.2-1B-SFT-DPOMix-Adapters"
)

# Step 3 — apply DPO adapters on top
model.load_adapter(
    "pranav6905/Llama-3.2-1B-DPO-DPOMix-Adapters",
    adapter_name = "dpo"
)
model.set_adapter("dpo")

prompt = (
    "<|begin_of_text|>"
    "<|start_header_id|>user<|end_header_id|>\n"
    "Explain what machine learning is.\n<|eot_id|>"
    "<|start_header_id|>assistant<|end_header_id|>\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Setting	Value
Base Model	meta-llama/Llama-3.2-1B-Instruct
Starting Point	SFT Adapters (Stage 1)
Dataset	argilla/dpo-mix-7k (chosen vs rejected)
DPO Beta	0.1
Learning Rate	5e-5
Steps	420
Hardware	Kaggle T4 GPU (free)

DPO Training Results

All metrics moved in the correct direction confirming alignment worked:

Metric	Direction	Meaning
rewards/chosen	↑ Increasing	Model scores good responses higher
rewards/rejected	↓ Decreasing	Model scores bad responses lower
rewards/margins	↑ Increasing	Gap between good/bad growing
rewards/accuracies	↑ 0.30 → 0.55	Correctly identifies better response