Stage 3 DPO preference-tuning LoRA adapter from the Unsloth 3-stage pharma fine-tuning pipeline. Trained using trl.DPOTrainer (Unsloth patched) on top of the Stage 2 merged instruction model.

Training details

Param	Value
Base model	`ThakrePranjal/pharma-tinyllama-unsloth-stage2-merged` (Stage 2 merged)
Trainer	`trl.DPOTrainer` (Unsloth patched)
Data	48 preference records (prompt/chosen/rejected)
Max steps	30
Learning rate	5e-5
DPO Beta	0.1
LoRA r	16
LoRA alpha	32
Peak VRAM	1.952 GB
Train time	106s

Usage

python

from unsloth import FastLanguageModel
from peft import PeftModel

# Load Stage 2 merged as base
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="ThakrePranjal/pharma-tinyllama-unsloth-stage2-merged",
    max_seq_length=512,
    load_in_4bit=True,
)

model = PeftModel.from_pretrained(model, "ThakrePranjal/pharma-tinyllama-unsloth-stage3-dpo-lora")
FastLanguageModel.for_inference(model)

prompt = "### Instruction:\nExplain the mechanism of metformin.\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=150,
                        temperature=0.7, top_p=0.9, do_sample=True)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Dataset

ThakrePranjal/pharma-preference-dataset-unsloth

Pipeline

unsloth/tinyllama-bnb-4bit
    └── Stage 1 SFT → merged → [ThakrePranjal/pharma-tinyllama-unsloth-stage1-merged]
            └── Stage 2 SFT → merged → [ThakrePranjal/pharma-tinyllama-unsloth-stage2-merged]
                    └── Stage 3 DPO (THIS ADAPTER) → merged → [ThakrePranjal/pharma-tinyllama-unsloth-final]