from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    "your-username/qwen-dpo-adapter"
)

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

# Generate
messages = [{"role": "user", "content": "Your prompt here"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))

Model Performance

The fine-tuned model shows improvements in response quality through optimized preferences.

Generated for Assignment 4 - AI Model Fine-tuning

shayfeng/qwen-dpo-adapter

DPO Fine-tuned Qwen2.5-7B-Instruct

Adapter Configuration

Training Details

Usage

Model Performance

shayfeng/
qwen-dpo-adapter