This repository provides a reproduction version of Tulu2-DPO-13B finetuned upon Tulu2-13B and Ultrafeedback. Therefore, we obey all licenses mentioned in Tulu2's work. Check our codes for more details: https://github.com/LuJunru/LLM_Finetune/tree/DPO, which is built with TRL.

Performance

Model	Size	Alignment	MT-Bench (score)	AlpacaEval 2.0 (win rate %)
Tulu-v2-13b 🐪	13B	SFT	5.79	2.61
Tulu-v2-dpo-13b 🐪	13B	DPO	6.06	6.96
Reproduced-tulu2-dpo-13b	13B	DPO	6.27	6.71

Input Format

The model is trained to use the following format (note the newlines):

<|user|>
Your message here!
<|assistant|>

For best results, format all inputs in this manner. Make sure to include a newline after <|assistant|>, this can affect generation quality quite a bit. Note: if fine-tuning with this chat template, ensure to evaluate and test with the chat template. Otherwise, fine-tining without the template if you choose to not use template during testing. Any mismatch of the chatting template between training and testing phases can obviously dampen the final performance.

Training hyperparameters

The following hyperparameters were used during DPO training:

DPO beta: 0.1
learning_rate: 1e-6 * sqrt(Num of Nodes)
total_train_batch_size: 128 * Num of Nodes
optimizer: AdamW with beta1 0.9, beta2 0.999 and epsilon 1e-8
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.1
Weight Decay: 0.0
num_epochs: 3.0
Specifically add above input format over training samples

Junrulu/Reproduced-tulu2-dpo-13b

Model Card for Reproduced Tulu2 DPO 13B

Performance

Input Format

Training hyperparameters

Junrulu/
Reproduced-tulu2-dpo-13b