This model is a fine-tuned version of tsavage68/UTI_L3_1000steps_1e5rate_SFT on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 0.0164
Rewards/chosen: 1.6699
Rewards/rejected: -6.1458
Rewards/accuracies: 0.9900
Rewards/margins: 7.8157
Logps/rejected: -75.4864
Logps/chosen: -29.1393
Logits/rejected: -1.3321
Logits/chosen: -1.3145

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-07
train_batch_size: 2
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 4
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 100
training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6936	0.3333	25	0.6887	-0.0007	-0.0109	0.5300	0.0102	-63.2166	-32.4804	-1.3231	-1.3079
0.6544	0.6667	50	0.6317	0.0212	-0.1104	0.8500	0.1316	-63.4155	-32.4367	-1.3229	-1.3078
0.5537	1.0	75	0.5050	0.0832	-0.3693	0.9400	0.4525	-63.9333	-32.3125	-1.3234	-1.3083
0.3404	1.3333	100	0.3397	0.1907	-0.8797	0.9800	1.0704	-64.9540	-32.0976	-1.3239	-1.3089
0.2191	1.6667	125	0.2217	0.3556	-1.4579	0.9900	1.8135	-66.1104	-31.7678	-1.3246	-1.3095
0.1721	2.0	150	0.1545	0.5057	-1.8983	0.9900	2.4040	-66.9912	-31.4675	-1.3248	-1.3097
0.0763	2.3333	175	0.1116	0.6367	-2.3670	0.9900	3.0037	-67.9287	-31.2056	-1.3255	-1.3103
0.0669	2.6667	200	0.0818	0.7555	-2.7499	0.9900	3.5054	-68.6945	-30.9681	-1.3264	-1.3111
0.0388	3.0	225	0.0620	0.8673	-3.2396	0.9900	4.1068	-69.6738	-30.7445	-1.3267	-1.3113
0.0653	3.3333	250	0.0506	0.9617	-3.6047	0.9900	4.5664	-70.4041	-30.5557	-1.3274	-1.3119
0.0332	3.6667	275	0.0406	1.0595	-4.0208	0.9900	5.0803	-71.2363	-30.3600	-1.3276	-1.3119
0.0522	4.0	300	0.0339	1.1423	-4.3687	0.9900	5.5110	-71.9320	-30.1943	-1.3282	-1.3123
0.005	4.3333	325	0.0293	1.2385	-4.6734	0.9900	5.9119	-72.5414	-30.0020	-1.3286	-1.3124
0.0284	4.6667	350	0.0256	1.3119	-4.9072	0.9900	6.2191	-73.0091	-29.8553	-1.3295	-1.3132
0.0393	5.0	375	0.0229	1.3864	-5.1293	0.9900	6.5157	-73.4534	-29.7063	-1.3298	-1.3132
0.0261	5.3333	400	0.0214	1.4513	-5.3049	0.9900	6.7563	-73.8046	-29.5763	-1.3302	-1.3135
0.0403	5.6667	425	0.0204	1.4964	-5.4655	0.9900	6.9619	-74.1256	-29.4862	-1.3304	-1.3136
0.0197	6.0	450	0.0190	1.5233	-5.6170	0.9900	7.1404	-74.4287	-29.4324	-1.3307	-1.3137
0.0023	6.3333	475	0.0186	1.5672	-5.7288	0.9900	7.2960	-74.6523	-29.3447	-1.3310	-1.3139
0.0391	6.6667	500	0.0181	1.5895	-5.8057	0.9900	7.3952	-74.8060	-29.2999	-1.3313	-1.3141
0.0044	7.0	525	0.0174	1.6125	-5.9110	0.9900	7.5235	-75.0167	-29.2541	-1.3314	-1.3141
0.0034	7.3333	550	0.0178	1.6265	-5.9426	0.9900	7.5691	-75.0799	-29.2260	-1.3316	-1.3143
0.0214	7.6667	575	0.0167	1.6348	-6.0154	0.9900	7.6502	-75.2254	-29.2094	-1.3316	-1.3143
0.0363	8.0	600	0.0166	1.6397	-6.0402	0.9900	7.6798	-75.2751	-29.1997	-1.3318	-1.3144
0.0366	8.3333	625	0.0168	1.6498	-6.0578	0.9900	7.7076	-75.3102	-29.1794	-1.3320	-1.3145
0.0011	8.6667	650	0.0168	1.6607	-6.0845	0.9900	7.7452	-75.3637	-29.1576	-1.3319	-1.3145
0.0043	9.0	675	0.0167	1.6659	-6.1131	0.9900	7.7790	-75.4209	-29.1472	-1.3321	-1.3146
0.0197	9.3333	700	0.0161	1.6703	-6.1301	0.9900	7.8004	-75.4550	-29.1385	-1.3320	-1.3145
0.0186	9.6667	725	0.0165	1.6713	-6.1341	0.9900	7.8054	-75.4628	-29.1364	-1.3321	-1.3147
0.0039	10.0	750	0.0165	1.6700	-6.1407	0.9900	7.8106	-75.4760	-29.1391	-1.3321	-1.3146
0.0005	10.3333	775	0.0164	1.6769	-6.1401	0.9900	7.8170	-75.4749	-29.1251	-1.3321	-1.3146
0.0185	10.6667	800	0.0164	1.6763	-6.1561	0.9900	7.8324	-75.5069	-29.1265	-1.3322	-1.3146
0.0212	11.0	825	0.0162	1.6734	-6.1441	0.9900	7.8175	-75.4828	-29.1321	-1.3322	-1.3145
0.0011	11.3333	850	0.0159	1.6707	-6.1474	0.9900	7.8181	-75.4894	-29.1376	-1.3321	-1.3145
0.0361	11.6667	875	0.0165	1.6746	-6.1464	0.9900	7.8209	-75.4874	-29.1299	-1.3322	-1.3147
0.0029	12.0	900	0.0161	1.6773	-6.1406	0.9900	7.8179	-75.4759	-29.1244	-1.3321	-1.3146
0.019	12.3333	925	0.0163	1.6716	-6.1497	0.9900	7.8213	-75.4941	-29.1358	-1.3321	-1.3146
0.0204	12.6667	950	0.0164	1.6699	-6.1458	0.9900	7.8157	-75.4864	-29.1393	-1.3321	-1.3145
0.0395	13.0	975	0.0164	1.6699	-6.1458	0.9900	7.8157	-75.4864	-29.1393	-1.3321	-1.3145
0.0048	13.3333	1000	0.0164	1.6699	-6.1458	0.9900	7.8157	-75.4864	-29.1393	-1.3321	-1.3145

Framework versions

Transformers 4.41.2
Pytorch 2.0.0+cu117
Datasets 2.19.2
Tokenizers 0.19.1

tsavage68/UTI_L3_1000steps_1e7rate_05beta_CSFTDPO

UTI_L3_1000steps_1e7rate_05beta_CSFTDPO

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

tsavage68/
UTI_L3_1000steps_1e7rate_05beta_CSFTDPO