⚠️ Note that the code is still experimental, as the ORPOTrainer PR is still not merged, follow its progress
at 🤗trl - ORPOTrainer PR.
About the fine-tuning
In order to fine-tune mistralai/Mistral-7B-v0.1 using ORPO, the branch
orpo from 🤗trl has been used, thanks to the invaluable and quick contribution of
@kashif.
ORPO stands for Odds Ratio Preference Optimization, and defines a new paradigm on fine-tuning LLMs, “combining” both the SFT
and the PPO/DPO stage into a single stage, thanks to the proposed loss function starting off from a preference dataset i.e.
chosen-rejected pairs.
Some key features about ORPO:
⚡️ Faster to train as it’s now a single stage fine-tuning
👨🏻🏫 Requires preference data i.e. (prompt, chosen, rejected)-like datasets
⬇️ Less memory than PPO/DPO as doesn’t need a reference model
🏆 SOTA results for Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) when fine-tuned using single-turn UltraFeedback
Some notes on the experiments mentioned in the paper:
📌 Up to 7B parameter LLMs were fine-tuned, achieving better performance compared to 7B counterparts and even 13B LLMs
📌 Not yet trained with multi-turn datasets as Capybara (may be an interesting experiment to run)
📌 For OPT models fine-tuned with HH-RLHF from Anthropic, truncated and padded to 1024 tokens, filtering out filtering the prompts with > 1024 tokens
📌 For Phi-2, Mistral (7B) and Llama 2 (7B), or UltraFeedback from OpenBMB (truncated and padded to 2048 tokens), filtering out filtering the prompts with > 1024 tokens
📌 Fine-tuned for 10 epochs, and using the evaluation loss as the metric for selecting the best models
For more information about ORPO, I highly recommend reading their paper titled ORPO: Monolithic Preference Optimization without Reference Model,
as it contains a lot of information and details not only on the ORPO method, but also on the experiment they ran, the results they got, and much more.
📅 Fine-tuning code will be shared soon, stay tuned!
The simplification comes from the fact that the prompt column is detached from both the chosen and rejected
columns so that there's no need for extra pre-processing while applying the chat template to the dataset before the
fine-tuning. So on, the dataset remains as is, with an additional column for the prompt.
The dataset is a small cocktail combining Argilla's latest efforts on DPO datasets, mixing the following datasets:
The samples have been randomly selected from the original datasets with a proportion of 0.33 each, as can be seen via the dataset column of the dataset.
