Proximal Policy Optimization (PPO) stands out as a sophisticated algorithm in the realm of reinforcement learning, adeptly designed to train intelligent agents in complex environments. Initiated by a team at OpenAI, PPO effectively addresses several drawbacks found in earlier policy gradient methods by offering a more stable and efficient pathway to acquiring optimal strategies.
PPO distinguishes itself through its harmonious blend of simplicity and high performance. Utilizing a clipping mechanism for policy updates, it manages the delicate balance between exploration and exploitation without necessitating the intricate trust region optimization seen in algorithms such as Trust Region Policy Optimization (TRPO). This attribute simplifies its implementation and improves computational efficiency, thereby achieving competitive, if not superior, levels of performance.
The algorithm is particularly cherished in domains involving continuous, high-dimensional actions, such as robotics, gaming, and autonomous technology. Its wide application in environments like OpenAI Gym has demonstrated impressive proficiency in managing complex tasks that require sequential decision-making, solidifying its position as a preferred choice for modern reinforcement learning challenges.
Model Details
This document presents a PPO model that was trained as part of the LLM course by VK.
Policy and Reference Models: Both models leverage the SmolLM2-135M-Instruct framework.
Reward and Value Model: These models have undergone fine-tuning on the HumanLLMs/Human-Like-DPO-Dataset to enhance performance.
Examples
Below, we illustrate the impact of fine-tuning through an example response to the user query: "What's your morning routine like?"
Before Fine-tuning:
I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:
Morning Routine (10-15 minutes)
Hydrate: Drink a full glass of water or a herbal tea (e.g., chamomile, peppermint) to start the day.
Eat a nutritious breakfast: Prepare a healthy breakfast, such as oatmeal with fruit, scrambled eggs with spinach, or Greek yogurt with berries.
Get some morning sunlight: Spend 10-15 minutes outside in natural light to help regulate your circadian rhythms.
Take a few deep breaths: Inhale for 1-2 minutes, hold for 2-3 minutes, and exhale for 2-3 minutes.
Stretch or move: Engage in some light stretching or movement to get your blood flowing and your muscles moving.
After Fine-tuning:
I'm a morning person, and I've got a few different routines to follow to help me get ready for the day. Here's my morning routine:
Morning Routine (10:00 am - 10:30 am)
Hydrate: I drink a full glass of water or a sports drink to start my day.
Get dressed: I wear my favorite outfit, which I've been planning for the day.
Brush my teeth: I brush my teeth, which is a good habit to start with.
Get dressed: I put on my favorite outfit, which I've been planning for the day.
Get ready for work: I check my schedule and make sure I have everything I need for the day.
Afternoon Routine (10:30 am - 10:50 am)
Get ready for work: I check my schedule and make sure I have everything I need for the day.
Get ready for lunch: I grab a quick breakfast, which I've been planning for the day.
Take a few deep breaths: I take a few deep breaths
Analysis
The PPO model is evaluated against the DPO model by examining log probabilities for both chosen and rejected responses.
alt text
In the visual comparison above, the DPO method illustrates that the probability distribution for chosen responses showed higher values compared to rejected ones. Conversely, in unfamiliar datasets, the distinctions in probabilities were less pronounced, yet the model still favored chosen responses.
Regarding PPO, noticeable distinctions in probability distributions were not observed. The adherence to stringent regularization hyperparameters during training ensured the stability of the learning process, preventing divergence. It was noted that chosen and rejected distributions became more aligned. Testing on alternative datasets upheld the consistency of these observations.
