You can deploy this model and start earning money today!

Barryzbr12/
qwen2.5-7b-instruct-dpo-lima-lora

Name: qwen2.5-7b-instruct-dpo-lima-lora
Rating: 2.5 (16 reviews)

Qwen/Qwen2.5-7B-Instruct

adapter

apache-2.0

Views

No views yet

Qwen2.5-7B-Instruct DPO LoRA adapter (LIMA × PairRM)

This is a LoRA adapter produced by Direct Preference Optimization (DPO) applied to Qwen/Qwen2.5-7B-Instruct. It was trained on the preference dataset Barryzbr12/lima-qwen2.5-7b-pairrm-preferences, which was constructed for Assignment 4 of the alignment course by (1) sampling 50 LIMA instructions, (2) generating 5 responses each from Qwen2.5-7B-Instruct, and (3) ranking them with llm-blender/PairRM.

Training configuration


Base model	`Qwen/Qwen2.5-7B-Instruct`
Method	DPO (sigmoid loss, β = 0.1)
LoRA rank / alpha	16 / 32
LoRA target modules	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`
Trainable parameters	40.4 M (~0.53 % of full model)
Epochs / Effective batch	3 / 8
Learning rate	5e-6, cosine schedule, 10 % warmup
Sequence length	2048
Precision	bf16
Seed	42

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype="bfloat16", device_map="auto"
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base, "Barryzbr12/qwen2.5-7b-instruct-dpo-lima-lora")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "How to make your dog more playful?"}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tok(prompt, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=512)[0], skip_special_tokens=True))

Author

Barryzbr12 — Assignment 4 (PairRM + DPO).