Model Type: Causal Language Model fine-tuned with Proximal Policy Optimization (PPO)
Architecture: Based on the HuggingFaceTB/SmolLM-135M-Instruct model with the TRL framework for PPO updates
Summary:
This model has been fine-tuned using the PPO algorithm. During training, a reward model (described separately) and a reference model are used to adjust the generation strategy. The training aims to steer the model towards generating responses that better match human preferences.
Intended Use
Use Cases:
Generating conversational responses with specific stylistic attributes
Experimental research on aligning language models through reinforcement learning
Limitations:
PPO can be sensitive to hyperparameter choices and might exhibit instability, especially under low-resource conditions.
The improvements compared to DPO may be more subtle without extensive training data and fine-tuning.
Data
Training Data:
The same HumanLLMs/Human-Like-DPO-Dataset is used, but for the PPO phase, only the prompt part is retained (converted using the chat template).
Data Preprocessing:
The prompt is formatted using tokenizer.apply_chat_template to ensure consistency in the input format for generation.
Training Details
Optimizer: AdamW
Learning Rate: 5e-6
Batch Size and Gradient Accumulation:
Overall batch size of 16, mini batch size of 4, with 4 gradient accumulation steps
PPO Configuration:
Parameters such as KL coefficient, clip range, and value function coefficient are specified in the PPOConfig.
Training Duration:
1 PPO epoch (adjustable as needed).
Evaluation and Results
Generation Comparison:
Outputs from before and after PPO fine-tuning are compared using a standard prompt (e.g., “What's your morning routine like?”) to assess the impact of the training.
Monitoring Metrics:
Metrics include reward model scores and KL divergence to ensure that updates are in line with expectations.
Limitations and Ethical Considerations
PPO training is complex and sensitive to hyperparameter settings, which may lead to instability in certain conditions.
The model may inherit biases from the training data. It is advisable to include safety checks and human reviews for critical applications.
Usage Instructions
Inference:
The PPO model is available on the HuggingFace Hub and is best used together with the reward model.
Feedback and Updates:
User feedback is encouraged to facilitate further refinements.
