Mathmate-7B-DELLA-ORPO is a finetuned version of Haleshot/Mathmate-7B-DELLA using the ORPO (Odds Ratio Preference Optimization) technique. This model has been specifically tuned to improve its performance on mathematical reasoning tasks based on human preferences.

This model was finetuned using the ORPO technique, which is an extension of DPO (Direct Preference Optimization) that can work with ranked preferences instead of just binary ones. The process was adapted from the tutorial "Fine-tune Llama 3 with ORPO" by Maxime Labonne, with some custom modifications to the code.

The model was finetuned on the argilla/distilabel-math-preference-dpo dataset. This dataset contains mathematical problems along with multiple solution attempts, ranked by human preference. This allowed the model to learn from human judgments about what constitutes a good mathematical explanation or solution.

Here's an example of how to use the model:

While this model has been finetuned on mathematical problems, it may still make mistakes or provide incorrect solutions. Always verify the model's output, especially for critical applications or complex mathematical problems.

If you use this model in your research, please cite:

@misc{mathmate-7b-della-orpo,
  author = {Haleshot},
  title = {Mathmate-7B-DELLA-ORPO},
  year = {2024},
  publisher = {HuggingFace},
  journal = {HuggingFace Hub},
  howpublished = {\url{https://huggingface.co/Haleshot/Mathmate-7B-DELLA-ORPO}},
}

Special thanks to Maxime Labonne for the ORPO finetuning tutorial, and to the Argilla team for providing the dataset used in this finetuning process.