🚀 Direct Preference Optimization (DPO) Fine-Tuned GPT-2

🔍 What This Model Does

• The model is fine-tuned on human preference rankings to generate responses that better align with user expectations.
• It takes text input and provides contextually relevant and human-aligned responses.
• The training method ensures the model produces more preferred outputs based on reinforcement learning feedback.

📊 Dataset Used

• Dataset Name: psyche/anthropic-hh-rlhf
• Source: Hugging Face Datasets Hub
• Type: Reinforcement Learning from Human Feedback (RLHF)
• Description: The dataset consists of human preference comparisons on model-generated completions, allowing the fine-tuning process to prioritize responses that humans find more favorable.

⚙️ Model Training & Hyperparameter Tuning

Pre-trained Model

• We fine-tuned GPT-2 using the DPOTrainer from the trl library.

Training Configuration

• Learning rate: 1e-3
• Batch size: 8
• Gradient accumulation steps: 1
• Max sequence length: 128
• Beta (DPO coefficient): 0.1
• Optimizer: RMSprop
• Warmup steps: 150
• Training steps: 200
• Tracking: Experiment logs recorded using wandb.

📤 Model on Hugging Face Hub