This model is a fine-tuned version of DUAL-GPO/zephyr-7b-gpo-update3-i0 on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

Loss: 0.0595
Rewards/chosen: -0.1134
Rewards/rejected: -0.1090
Rewards/accuracies: 0.4030
Rewards/margins: -0.0044
Logps/rejected: -276.7458
Logps/chosen: -289.3788
Logits/rejected: -1.8514
Logits/chosen: -2.0125

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-06
train_batch_size: 2
eval_batch_size: 2
seed: 42
distributed_type: multi-GPU
gradient_accumulation_steps: 2
total_train_batch_size: 4
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.3803	0.4	100	0.0537	0.0	0.0	0.0	0.0	-254.9398	-266.6976	-1.8067	-1.9618
0.2725	0.8	200	0.0549	-0.0209	-0.0219	0.4410	0.0010	-259.3280	-270.8865	-1.8361	-1.9947
0.3013	1.2	300	0.0532	-0.1669	-0.1811	0.4675	0.0141	-291.1523	-300.0851	-1.8278	-1.9902
0.3433	1.6	400	0.0523	-0.1720	-0.1893	0.4780	0.0173	-292.8069	-301.0948	-1.8287	-1.9909
0.3606	2.0	500	0.0623	-0.1514	-0.1465	0.4050	-0.0048	-284.2478	-296.9682	-1.8491	-2.0112
0.3038	2.4	600	0.0616	-0.1610	-0.1582	0.4090	-0.0029	-286.5705	-298.9020	-1.8490	-2.0113
0.3161	2.8	700	0.0613	-0.1640	-0.1619	0.4125	-0.0021	-287.3163	-299.4932	-1.8473	-2.0096
0.3852	3.2	800	0.0574	-0.1342	-0.1354	0.4260	0.0012	-282.0106	-293.5319	-1.8537	-2.0157
0.3359	3.6	900	0.0595	-0.1131	-0.1086	0.4005	-0.0044	-276.6672	-289.3095	-1.8507	-2.0116
0.3701	4.0	1000	0.0596	-0.1134	-0.1090	0.4000	-0.0044	-276.7309	-289.3763	-1.8513	-2.0123
0.4025	4.4	1100	0.0596	-0.1134	-0.1088	0.4030	-0.0045	-276.7074	-289.3722	-1.8516	-2.0127
0.3754	4.8	1200	0.0595	-0.1136	-0.1091	0.4010	-0.0044	-276.7694	-289.4114	-1.8515	-2.0125

Framework versions

PEFT 0.7.1
Transformers 4.36.2
Pytorch 2.1.2+cu121
Datasets 2.14.6
Tokenizers 0.15.2

DUAL-GPO/zephyr-7b-gpo-u4-i1

zephyr-7b-gpo-u4-i1

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

DUAL-GPO/
zephyr-7b-gpo-u4-i1