You can deploy this model and start earning money today!

SetonLiang2/
qwen25-7b-assignment4-dpo-adapter

Name: qwen25-7b-assignment4-dpo-adapter
Availability: InStock

peft

safetensors

Qwen/Qwen2.5-7B-Instruct

adapter

Views

No views yet

Qwen2.5-7B DPO Adapter (Assignment 4)

This is a PEFT LoRA adapter fine-tuned from Qwen/Qwen2.5-7B-Instruct for a course assignment on DPO-based preference optimization.

Quick Start

Option 1: Transformers + PEFT

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_path = "SetonLiang2/qwen25-7b-assignment4-dpo-adapter"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
	base_model_id,
	torch_dtype=torch.bfloat16,
	device_map="auto",
	trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()

messages = [{"role": "user", "content": "请解释什么是DPO"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Option 2: Merge Before Inference (Optional)

If your deployment environment does not support PEFT, you can merge the adapter into the base model first and run inference on the merged model.

Training Summary

Base model: Qwen/Qwen2.5-7B-Instruct
Method: DPO (TRL) + LoRA (PEFT)
Preference data: chosen/rejected pairs constructed from multi-candidate responses using PairRM

Intended Use

Research and coursework experiments
Comparing base model outputs vs DPO-tuned outputs

Limitations

Small-scale training data (coursework setting), so generalization is limited
The model may still produce unstable or incomplete responses

Framework versions

PEFT 0.15.1