Preference dataset: xypkent/assignment4-lima-pairrm-pref
Source: instructions sampled from GAIR/lima
Construction:
1. Sample 50 instructions
2. Generate 5 responses per instruction with Qwen2.5-7B-Instruct
3. Rank responses with PairRM
4. Build (prompt, chosen, rejected) pairs

Training Procedure

Algorithm: DPO
Epochs: 1
Batch size (per device): 1
Gradient accumulation steps: 8
Adapter type: LoRA (PEFT)
Frameworks: transformers, trl, peft, torch

Evaluation

Sampled 10 unseen instructions (not in training set)
Compared outputs from:
- original Qwen/Qwen2.5-7B-Instruct
- DPO model (base + this adapter)
Comparison columns:
- instruction
- original_completion
- dpo_completion

Limitations

Small assignment-scale preference dataset (50 prompts)
Not suitable for production or safety-critical deployment
Can inherit biases/errors from base model and data

How to Use

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "xypkent/assignment4-qwen25-7b-dpo-adapter"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=dtype,
    device_map="auto" if torch.cuda.is_available() else None,
)
if not torch.cuda.is_available():
    base_model.to("cpu")

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

xypkent/assignment4-qwen25-7b-dpo-adapter

Model Card: assignment4-qwen25-7b-dpo-adapter

Model Details

Training Data

Training Procedure

Evaluation

Limitations

How to Use

Links

xypkent/
assignment4-qwen25-7b-dpo-adapter