ORPO (Odds Ratio Preference Optimization) is a new training paradigm that combines the usually separated phases
of SFT (Supervised Fine-Tuning) and Preference Alignment (usually performed with RLHF or simpler methods like DPO).
gemma-2b-orpo performs well for its size on Nous' benchmark suite.
(evaluation conducted using
LLM AutoEval).
Detailed results can be found
here.
By comparison, on the Open LLM Leaderboard, google/gemma-2b-it has an average of 42.75.
The model is small, so it runs smoothly on Colab. It is also fine to load the model using quantization.
1# pip install transformers accelerate
2import torch
3from transformers import pipeline
4pipe = pipeline("text-generation", model="anakin87/gemma-2b-orpo", torch_dtype=torch.bfloat16, device_map="auto")
5messages = [{"role": "user", "content": "Write a rap song on Vim vs VSCode."}]
6prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False)
7outputs = pipe(prompt, max_new_tokens=500, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
8print(outputs[0]["generated_text"])
The model was trained using HF TRL.
📓 Training notebook
