Best DPO (Direct Preference Optimization) LoRA adapter for Qwen/Qwen3-0.6B-Base, built on top of the best SFT adapter. Trained on the UltraFeedback Binarized dataset.

Model Details

Developed by: Abdullah Iqbal (26904), Anushe Ali (26418)
Model type: LoRA Adapter (PEFT) — DPO stage
Language: English
Base model: Qwen/Qwen3-0.6B-Base
SFT base: Abdullah121212/qwen3-0.6b-sft-best
Training dataset: trl-lib/ultrafeedback_binarized (chosen/rejected pairs)
Training regime: fp16 mixed precision
Hardware: Kaggle T4 GPU

Training Hyperparameters

Parameter	Value
DPO beta	0.05
LoRA rank (r)	16
LoRA alpha	32
Target modules	q_proj, k_proj, v_proj, o_proj
Learning rate	5e-5
Batch size	2
Gradient accumulation	2
Epochs	1
Dropout	0.05

Evaluation Results

Evaluated on 10 manually curated prompts with gold-reference answers.

Stage	BLEU	BERTScore F1
Base (Qwen3-0.6B)	9.82	0.8803
SFT (best)	9.47	0.8836
DPO (best — this model)	11.94	0.8863

How to Use

Prerequisites

pip install --upgrade torchao transformers peft accelerate

Note: You may encounter ImportError: Found an incompatible version of torchao if your torchao version is below 0.16.0. Run pip install --upgrade torchao to fix this.

Inference Code

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B-Base", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Abdullah121212/qwen3-0.6b-dpo-best")

model = PeftModel.from_pretrained(base_model, "Abdullah121212/qwen3-0.6b-dpo-best")
model.eval()

prompt = "Explain the difference between supervised and unsupervised learning in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Pipeline

Baseline (Qwen3-0.6B-Base)
  -> SFT (best trial 3 — see Abdullah121212/qwen3-0.6b-sft-best)
    -> DPO (5 trials, this is trial 3 = best)

Framework Versions

PEFT 0.19.1
Transformers 4.x
TRL 0.x
PyTorch 2.x