Model Card for SmolLM Variation PPO Fine-Tuning for RLHF
This model is a fine-tuned variant of the SmolLM-135M-Instruct model using Proximal Policy Optimization (PPO) for Reinforcement Learning from Human Feedback (RLHF). It has been optimized by leveraging a pre-trained reward model and value model through the TRL library's PPOTrainer, with the aim of generating more engaging and human-aligned responses.
Task Description
The project involves:
Converting the training dataset into a prompt-only format.
Utilizing a pre-trained reward model and a value model to compute scalar rewards.
Fine-tuning the policy model with PPO (via TRL's PPOTrainer) to improve response quality and maximize reward margins.
Evaluating the model's performance by comparing generation quality before and after PPO fine-tuning.
Model Details
Model Description
This is the model card for a 🤗 transformers model that has been pushed on the Hub after PPO fine-tuning. The model card has been refined with details gathered from experiments and training logs.
Developed by: Ekaterina Aleksanian
Model type: Causal Language Model (autoregressive) fine-tuned using Proximal Policy Optimization (PPO)
Language(s) (NLP): English
License: MIT License
Finetuned from model [optional]: SmolLM-135M-Instruct
How to Get Started with the Model
You can load the model using the Hugging Face Transformers library. For example:
User: What's your morning routine like?
Assistant: What a wonderful question! As a conversational AI, I have my own wake-up routine tailored to my needs. Here's my "wake-up routine" for you:
**Morning Routine (10-15 minutes)**
1. **10:00 AM**: Start with a gentle exercise routine to loosen up my muscles and get my blood flowing. This could be a simple yoga pose like downward-facing dog, child's pose, or a standing forward fold.
2. **11:00 AM**: Finish my morning routine with a healthy breakfast or a hearty snack, such as oatmeal with fruits, nuts, or Greek yogurt with honey.
3. **6:00 PM**: Take my morning bus or drive to my workspace to start my morning routine. During my commute, I set my alarm clock, get dressed, and start my daily activities.
4. **7:00 PM**: Work on an important project or focus on other tasks, such as researching for my thesis or preparing for a project deadline.
5. **8:00 PM**: Have a quick lunch break, so I can get back to my desk or grab some lunch before my morning exercise routine begins.
After PPO Fine-Tuning:
User: What's your morning routine like?
Assistant: I'm excited to share my morning routine with you! As a digital AI assistant, I don't have personal experiences or emotions, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:
**Morning Routine (10-15 minutes)**
1. **Hydration**: Drink a full glass of water or a herbal tea (e.g., chamomile, peppermint) to start your day.
2. **Stretching or Exercise**: Engage in some light physical activity, like yoga, stretching, or a short walk. This helps increase energy and flexibility.
3. **Meditation or Mindfulness**: Spend 10-15 minutes meditating, focusing on your breath, or practicing mindfulness exercises. This helps clear your mind and set intentions for the day.
4. **Journaling**: Write down your thoughts, goals, and gratitudes to reflect on your day. This helps process emotions and gain clarity.
5. **Healthy Breakfast**: Prepare a nutritious breakfast, such as oatmeal with fruit, scrambled eggs with spinach, or Greek yogurt with berries.
Training Details
Training Data
The model was fine-tuned using the HumanLLMs/Human-Like-DPO-Dataset, which provides prompts along with chosen and rejected responses. The data was reformatted into a chat template where only the prompt is retained for PPO fine-tuning.
Training Procedure
The training procedure involved:
Converting the raw dataset into a prompt-only format using a chat template.
Tokenizing the data while preserving critical information from the prompt.
Fine-tuning the SmolLM-135M-Instruct model with PPO using TRL's PPOTrainer.
Leveraging a pre-trained reward model and value model to guide the PPO updates.
Training was conducted on an A100 GPU with Automatic Mixed Precision (AMP) enabled for efficiency.
Training Hyperparameters:
Learning Rate: 5e-5
Batch Size: 16
Mini Batch Size: 4
PPO Epochs: 4
Max Sequence Length: 1024
Max Prompt Length: 256
Preprocessing
The dataset was processed by applying a chat template that formats the prompt in a conversational style. Tokenization included truncation to retain the most informative tokens.
Speeds, Sizes, Times
Fine-tuning on an A100 GPU took approximately 2 minutes per epoch. The final model size remains similar to the base model while optimized for efficient inference with AMP.
Evaluation
Testing Data, Factors & Metrics
Testing Data
Evaluation was performed on a subset of the training data (in-distribution) and an external dataset (wikitext-2-raw-v1) to assess generalization.
Factors
Evaluation factors include:
The log probability assigned to in-distribution versus out-of-distribution data.
Reward margins computed from the difference in log probabilities.
Qualitative generation quality of the responses.
Metrics
Key evaluation metrics are:
Log Probability Distribution: A comparison between in-distribution and external data.
Reward Margins: The differential between the rewards of chosen versus rejected responses.
Reward Accuracy: The percentage of instances where the chosen response is favored.
Results
The evaluation indicates that the PPO fine-tuned model assigns higher log probabilities to in-distribution data compared to external samples. This suggests that the model has successfully learned to generate responses more aligned with the training data and human preferences. Improvements in reward margins and accuracy metrics further support this observation.
Summary
In summary, the PPO fine-tuned model demonstrates enhanced performance in generating human-aligned responses. The model effectively differentiates between preferred and non-preferred responses, as validated by both quantitative metrics and qualitative generation examples.
Technical Specifications
Model Architecture and Objective
The model is based on the SmolLM-135M-Instruct architecture. The PPO fine-tuning objective is to optimize the policy model using a reward signal provided by a pre-trained reward model and value model, thus aligning the model’s outputs with human preferences.
Compute Infrastructure
Fine-tuning was performed on an A100 GPU with AMP enabled for computational efficiency.
Hardware
NVIDIA A100 GPU
Software
PyTorch
Hugging Face Transformers
TRL (for reinforcement learning components)
Additional libraries for data processing and visualization
PPO (Proximal Policy Optimization): A reinforcement learning algorithm that updates the policy by taking small, conservative steps to improve performance.
RLHF: Reinforcement Learning from Human Feedback.
AMP: Automatic Mixed Precision, used to speed up training.
More Information
For further details, please refer to the associated repositories:
