You can deploy this model and start earning money today!

mnoukhov/
pythia-2.8b-dpo_hh_rlhf

peft

safetensors

generated_from_trainer

mnoukhov/pythia-2.8b-sft_hh_rlhf

adapter

Views

No views yet

pythia-2.8b-dpo_hh_rlhf

This model is a fine-tuned version of mnoukhov/pythia-2.8b-sft_hh_rlhf on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 0.6257
Rewards/chosen: -0.6630
Rewards/rejected: -1.0092
Rewards/accuracies: 0.6427
Rewards/margins: 0.3462
Logps/rejected: -143.9225
Logps/chosen: -143.9225
Logps/ref Rejected: -134.7867
Logps/ref Chosen: -137.2922

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 8
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
gradient_accumulation_steps: 8
total_train_batch_size: 64
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logps/ref Rejected	Logps/ref Chosen
0.6591	0.2	503	0.6625	-0.2937	-0.4623	0.5935	0.1685	-140.2295	-140.2295	-134.7867	-137.2922
0.6446	0.4	1006	0.6412	-0.3936	-0.6444	0.6209	0.2507	-141.2284	-141.2284	-134.7867	-137.2922
0.6289	0.6	1509	0.6296	-0.6299	-0.9621	0.6365	0.3322	-143.5909	-143.5909	-134.7867	-137.2922
0.6215	0.8	2012	0.6257	-0.6630	-1.0092	0.6427	0.3462	-143.9225	-143.9225	-134.7867	-137.2922

Framework versions

PEFT 0.10.0
Transformers 4.38.2
Pytorch 2.1.2+cu121
Datasets 2.17.0
Tokenizers 0.15.2

mnoukhov/pythia-2.8b-dpo_hh_rlhf

pythia-2.8b-dpo_hh_rlhf

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

mnoukhov/
pythia-2.8b-dpo_hh_rlhf