Model Card for Tulu V2.5 70B RM - Preference Mixture
Tulu is a series of language models that are trained to act as helpful assistants.
Tulu V2.5 is a series of models trained using DPO and PPO starting from the Tulu 2 suite.
This is a reward model used for PPO training trained on our mixture of preference datasets.
It was used to train this model.
Dataset: Data used to train this model can be found here - specifically the preference_big_mixture split.
Model Family: The collection of related models can be found here.
Input Format
The model is trained to use the following format (note the newlines):
<|user|>
Your message here!
<|assistant|>
For best results, format all inputs in this manner. Make sure to include a newline after <|assistant|>, this can affect generation quality quite a bit.
We have included a chat template in the tokenizer implementing this template.
Intended uses & limitations
The model was initially fine-tuned on a filtered and preprocessed of the Tulu V2 mix dataset, which contains a diverse range of human created instructions and synthetic dialogues generated primarily by other LLMs.
We then further trained the model with a Jax RM trainer built on EasyLM on the dataset mentioned above.
This model is meant as a research artefact.
Training hyperparameters
The following hyperparameters were used during PPO training:
learning_rate: 1e-06
total_train_batch_size: 512
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear cooldown to 1e-05.
lr_scheduler_warmup_ratio: 0.03
num_epochs: 1.0
Citation
If you find Tulu 2.5 is useful in your work, please cite it with:
@misc{ivison2024unpacking,
title={{Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback}},
author={{Hamish Ivison and Yizhong Wang and Jiacheng Liu and Ellen Wu and Valentina Pyatkin and Nathan Lambert and Yejin Choi and Noah A. Smith and Hannaneh Hajishirzi}}
year={2024},
eprint={2406.09279},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
