Qwen3-0.6B DPO Adapter
This is a Direct Preference Optimization (DPO) LoRA adapter for the Qwen/Qwen3-0.6B model. It was trained on top of a previously Supervised Fine-Tuned (SFT) adapter to further align the model's tone and style with human conversational preferences.
Model Details
Model Description
This adapter applies DPO to improve the semantic quality and conversational naturalness of the base model. The training utilized the best-performing SFT adapter as both the initial trainable policy model and the frozen reference model.
- Developed by: Agha Salik Ali and Uzair Nadeem
- Model type: LoRA Adapter for Causal Language Model
- Language(s) (NLP): English
- Finetuned from model:
Qwen/Qwen3-0.6B (via SFT Adapter)
Uses
Direct Use
This model is intended to generate clean, human-like responses and is highly effective at stripping away rigid, robotic formatting (e.g., heavily formatted math explanations) in favor of natural prose.
Out-of-Scope Use
Due to the constraints of the 0.6B parameter base model and the heavy alignment training, this model is highly susceptible to format collapse on open-ended creative generation tasks (e.g., devolving into infinite repetitive loops). Furthermore, preference tuning does not inject new factual world knowledge; the model will still confidently hallucinate if the base parameters lack the correct domain understanding.
Training Details
Training Data
The model was fine-tuned on the helpful-base split of the Anthropic/hh-rlhf dataset. The data was preprocessed to isolate the final human turn and the corresponding chosen/rejected assistant replies, formatted into strict conversation message dictionaries. A 70/30 train-evaluation split was used.
Training Procedure
The model was trained using the DPOTrainer from the trl library. To prevent CUDA Out-Of-Memory (OOM) errors from loading two models simultaneously, the evaluation batch size was explicitly restricted to 2.
Training Hyperparameters
- Training regime: fp16 mixed precision
- Beta: 0.1
- Learning Rate: 5e-05
- Train Batch Size: 2
- Eval Batch Size: 2
- Gradient Accumulation Steps: 8
- Number of Epochs: 1
Compute Infrastructure
- Hardware: Kaggle Dual NVIDIA T4 GPUs
Evaluation
Testing Data & Metrics
The model was evaluated using a customized set of 10 prompts to measure semantic alignment and surface-level string matching against gold-standard references.
- Metrics: BLEU, BERTScore F1
Results
- Validation Loss: 0.6359
- BLEU: 6.9213
- BERTScore F1: 0.8608
Framework versions
- PEFT 0.18.1
- Transformers
- TRL
