This model is a fine-tuned version of tsavage68/UTI_L3_1000steps_1e5rate_SFT on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 0.0290
Rewards/chosen: 1.0999
Rewards/rejected: -4.7721
Rewards/accuracies: 0.9900
Rewards/margins: 5.8720
Logps/rejected: -79.1016
Logps/chosen: -28.8126
Logits/rejected: -1.3325
Logits/chosen: -1.3147

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-07
train_batch_size: 2
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 4
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 100
training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6945	0.3333	25	0.6909	-0.0005	-0.0056	0.4900	0.0051	-63.2133	-32.4808	-1.3230	-1.3079
0.6631	0.6667	50	0.6538	0.0144	-0.0676	0.8500	0.0820	-63.4201	-32.4310	-1.3232	-1.3082
0.6008	1.0	75	0.5691	0.0482	-0.2288	0.9600	0.2770	-63.9573	-32.3183	-1.3233	-1.3082
0.4499	1.3333	100	0.4399	0.1150	-0.5411	0.9600	0.6561	-64.9983	-32.0957	-1.3238	-1.3088
0.3287	1.6667	125	0.3220	0.2114	-0.9535	0.9900	1.1649	-66.3730	-31.7744	-1.3242	-1.3092
0.2577	2.0	150	0.2461	0.3062	-1.3005	0.9900	1.6067	-67.5297	-31.4582	-1.3245	-1.3095
0.1552	2.3333	175	0.1940	0.3877	-1.6591	0.9900	2.0467	-68.7248	-31.1867	-1.3252	-1.3101
0.1392	2.6667	200	0.1565	0.4614	-1.9315	0.9900	2.3929	-69.6330	-30.9411	-1.3260	-1.3109
0.1027	3.0	225	0.1264	0.5356	-2.2660	0.9900	2.8016	-70.7480	-30.6936	-1.3263	-1.3111
0.1283	3.3333	250	0.1078	0.5929	-2.5135	0.9900	3.1064	-71.5729	-30.5027	-1.3272	-1.3119
0.0723	3.6667	275	0.0898	0.6579	-2.8100	0.9900	3.4679	-72.5613	-30.2859	-1.3274	-1.3119
0.083	4.0	300	0.0761	0.7117	-3.0410	0.9900	3.7527	-73.3313	-30.1067	-1.3280	-1.3123
0.0276	4.3333	325	0.0660	0.7700	-3.2845	0.9900	4.0545	-74.1431	-29.9124	-1.3283	-1.3123
0.0535	4.6667	350	0.0581	0.8180	-3.4706	0.9900	4.2886	-74.7634	-29.7523	-1.3292	-1.3131
0.0598	5.0	375	0.0512	0.8706	-3.6549	0.9900	4.5255	-75.3777	-29.5771	-1.3296	-1.3132
0.0485	5.3333	400	0.0461	0.9118	-3.8192	0.9900	4.7310	-75.9254	-29.4398	-1.3300	-1.3134
0.0548	5.6667	425	0.0422	0.9462	-4.0246	0.9900	4.9708	-76.6100	-29.3249	-1.3303	-1.3136
0.0311	6.0	450	0.0394	0.9768	-4.1514	0.9900	5.1282	-77.0326	-29.2230	-1.3306	-1.3136
0.0142	6.3333	475	0.0371	0.9974	-4.2644	0.9900	5.2618	-77.4093	-29.1544	-1.3311	-1.3140
0.0506	6.6667	500	0.0353	1.0183	-4.3634	0.9900	5.3817	-77.7395	-29.0848	-1.3313	-1.3141
0.0193	7.0	525	0.0338	1.0391	-4.4615	0.9900	5.5006	-78.0662	-29.0152	-1.3316	-1.3142
0.014	7.3333	550	0.0323	1.0562	-4.5279	0.9900	5.5841	-78.2877	-28.9582	-1.3317	-1.3143
0.0329	7.6667	575	0.0317	1.0604	-4.5932	0.9900	5.6536	-78.5055	-28.9443	-1.3320	-1.3146
0.0435	8.0	600	0.0311	1.0691	-4.6423	0.9900	5.7114	-78.6692	-28.9154	-1.3321	-1.3146
0.044	8.3333	625	0.0305	1.0755	-4.6661	0.9900	5.7416	-78.7484	-28.8939	-1.3322	-1.3146
0.0079	8.6667	650	0.0299	1.0865	-4.7011	0.9900	5.7876	-78.8650	-28.8574	-1.3324	-1.3148
0.0155	9.0	675	0.0296	1.0888	-4.7256	0.9900	5.8144	-78.9466	-28.8497	-1.3324	-1.3147
0.0255	9.3333	700	0.0293	1.0913	-4.7424	0.9900	5.8337	-79.0027	-28.8413	-1.3325	-1.3148
0.0252	9.6667	725	0.0293	1.0950	-4.7647	0.9900	5.8597	-79.0770	-28.8290	-1.3325	-1.3148
0.0132	10.0	750	0.0292	1.0940	-4.7718	0.9900	5.8658	-79.1006	-28.8322	-1.3325	-1.3147
0.004	10.3333	775	0.0292	1.0976	-4.7711	0.9900	5.8687	-79.0984	-28.8203	-1.3326	-1.3148
0.025	10.6667	800	0.0288	1.0976	-4.7705	0.9900	5.8681	-79.0965	-28.8204	-1.3325	-1.3147
0.0312	11.0	825	0.0288	1.0969	-4.7716	0.9900	5.8685	-79.0999	-28.8227	-1.3326	-1.3148
0.0076	11.3333	850	0.0288	1.0987	-4.7743	0.9900	5.8729	-79.1089	-28.8168	-1.3324	-1.3147
0.0444	11.6667	875	0.0289	1.0998	-4.7712	0.9900	5.8710	-79.0988	-28.8131	-1.3325	-1.3148
0.0113	12.0	900	0.0290	1.0998	-4.7766	0.9900	5.8764	-79.1167	-28.8131	-1.3326	-1.3148
0.0272	12.3333	925	0.0289	1.1001	-4.7736	0.9900	5.8736	-79.1066	-28.8121	-1.3325	-1.3147
0.0296	12.6667	950	0.0290	1.0999	-4.7721	0.9900	5.8720	-79.1016	-28.8126	-1.3325	-1.3147
0.0533	13.0	975	0.0290	1.0999	-4.7721	0.9900	5.8720	-79.1016	-28.8126	-1.3325	-1.3147
0.0168	13.3333	1000	0.0290	1.0999	-4.7721	0.9900	5.8720	-79.1016	-28.8126	-1.3325	-1.3147

Framework versions

Transformers 4.41.2
Pytorch 2.0.0+cu117
Datasets 2.19.2
Tokenizers 0.19.1

tsavage68/UTI_L3_1000steps_1e7rate_03beta_CSFTDPO

UTI_L3_1000steps_1e7rate_03beta_CSFTDPO

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

tsavage68/
UTI_L3_1000steps_1e7rate_03beta_CSFTDPO