This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_DPO_generatedAndParaphrasedMoreUniqueResponseNoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5382
Rewards/chosen: -0.7103
Rewards/rejected: -1.5509
Rewards/accuracies: 0.7622
Rewards/margins: 0.8406
Logps/rejected: -51.6367
Logps/chosen: -44.2163
Logits/rejected: -2.1097
Logits/chosen: -2.2356

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.7184	0.0135	20	0.6974	0.0023	-0.0058	0.5152	0.0082	-47.7739	-42.4346	-2.2043	-2.3100
0.7276	0.0270	40	0.6955	0.0072	-0.0289	0.5762	0.0360	-47.8315	-42.4225	-2.2073	-2.3133
0.7222	0.0405	60	0.6967	-0.0373	-0.0325	0.4970	-0.0048	-47.8405	-42.5336	-2.2008	-2.3066
0.7079	0.0540	80	0.6926	-0.0783	-0.0877	0.5366	0.0094	-47.9787	-42.6362	-2.1903	-2.2963
0.67	0.0675	100	0.6866	-0.1807	-0.1982	0.5366	0.0176	-48.2549	-42.8920	-2.1692	-2.2754
0.6887	0.0810	120	0.6782	-0.3199	-0.3552	0.5579	0.0353	-48.6474	-43.2401	-2.1391	-2.2454
0.6841	0.0945	140	0.6695	-0.4628	-0.5213	0.5579	0.0586	-49.0626	-43.5973	-2.1131	-2.2200
0.6317	0.1080	160	0.6646	-0.5515	-0.6336	0.5610	0.0821	-49.3433	-43.8192	-2.0948	-2.2024
0.6615	0.1215	180	0.6573	-0.6582	-0.7828	0.5884	0.1247	-49.7164	-44.0858	-2.0722	-2.1803
0.6116	0.1350	200	0.6533	-0.7603	-0.9066	0.6220	0.1462	-50.0258	-44.3413	-2.0538	-2.1625
0.5633	0.1484	220	0.6468	-0.7917	-0.9465	0.6067	0.1548	-50.1255	-44.4197	-2.0505	-2.1600
0.661	0.1619	240	0.6389	-0.7738	-0.9628	0.6341	0.1889	-50.1662	-44.3750	-2.0457	-2.1554
0.5929	0.1754	260	0.6344	-0.7924	-0.9992	0.6098	0.2068	-50.2574	-44.4214	-2.0432	-2.1541
0.7431	0.1889	280	0.6282	-0.7690	-1.0115	0.6616	0.2425	-50.2880	-44.3628	-2.0508	-2.1629
0.6263	0.2024	300	0.6215	-0.7515	-1.0145	0.6494	0.2631	-50.2957	-44.3190	-2.0557	-2.1684
0.4883	0.2159	320	0.6172	-0.7482	-1.0533	0.6616	0.3052	-50.3927	-44.3108	-2.0580	-2.1719
0.6204	0.2294	340	0.6160	-0.8904	-1.2215	0.6890	0.3311	-50.8130	-44.6663	-2.0348	-2.1497
0.6938	0.2429	360	0.6117	-0.9125	-1.2653	0.6799	0.3528	-50.9225	-44.7216	-2.0398	-2.1554
0.5494	0.2564	380	0.6060	-0.8717	-1.2650	0.6494	0.3932	-50.9217	-44.6197	-2.0511	-2.1682
0.5566	0.2699	400	0.5986	-0.7971	-1.2189	0.6677	0.4219	-50.8067	-44.4331	-2.0571	-2.1742
0.5032	0.2834	420	0.5939	-0.7497	-1.1821	0.6799	0.4323	-50.7145	-44.3147	-2.0708	-2.1889
0.6407	0.2969	440	0.5910	-0.6804	-1.1368	0.6951	0.4564	-50.6014	-44.1413	-2.0767	-2.1948
0.5151	0.3104	460	0.5879	-0.7247	-1.2017	0.6799	0.4770	-50.7636	-44.2522	-2.0733	-2.1920
0.5716	0.3239	480	0.5861	-0.7015	-1.1846	0.6829	0.4830	-50.7208	-44.1942	-2.0782	-2.1975
0.5585	0.3374	500	0.5817	-0.6991	-1.2088	0.6707	0.5097	-50.7814	-44.1881	-2.0811	-2.2005
0.5409	0.3509	520	0.5788	-0.7606	-1.3059	0.7012	0.5453	-51.0241	-44.3420	-2.0711	-2.1911
0.5202	0.3644	540	0.5770	-0.7337	-1.2920	0.6951	0.5582	-50.9893	-44.2748	-2.0716	-2.1917
0.4905	0.3779	560	0.5727	-0.6628	-1.2363	0.6982	0.5735	-50.8501	-44.0974	-2.0933	-2.2143
0.6852	0.3914	580	0.5701	-0.6695	-1.2686	0.7012	0.5991	-50.9309	-44.1142	-2.0888	-2.2098
0.455	0.4049	600	0.5689	-0.6854	-1.3059	0.7256	0.6205	-51.0241	-44.1538	-2.0900	-2.2124
0.6082	0.4184	620	0.5689	-0.7778	-1.4046	0.7043	0.6268	-51.2707	-44.3848	-2.0755	-2.1982
0.5414	0.4318	640	0.5652	-0.7567	-1.4224	0.7195	0.6657	-51.3153	-44.3321	-2.0810	-2.2038
0.6451	0.4453	660	0.5651	-0.6999	-1.3697	0.7104	0.6698	-51.1836	-44.1901	-2.0936	-2.2161
0.5667	0.4588	680	0.5612	-0.7268	-1.4333	0.7134	0.7065	-51.3425	-44.2574	-2.0931	-2.2162
0.5178	0.4723	700	0.5621	-0.7898	-1.5068	0.7073	0.7170	-51.5263	-44.4149	-2.0886	-2.2119
0.4991	0.4858	720	0.5599	-0.7847	-1.5071	0.7043	0.7224	-51.5270	-44.4021	-2.0869	-2.2104
0.4854	0.4993	740	0.5573	-0.7946	-1.5493	0.7195	0.7546	-51.6325	-44.4269	-2.0832	-2.2069
0.5844	0.5128	760	0.5581	-0.8007	-1.5414	0.7287	0.7407	-51.6129	-44.4422	-2.0849	-2.2080
0.4954	0.5263	780	0.5570	-0.8156	-1.5626	0.7287	0.7470	-51.6658	-44.4794	-2.0790	-2.2025
0.5597	0.5398	800	0.5563	-0.8289	-1.5717	0.7317	0.7428	-51.6885	-44.5127	-2.0749	-2.1986
0.5331	0.5533	820	0.5545	-0.8454	-1.6128	0.75	0.7674	-51.7913	-44.5538	-2.0748	-2.1989
0.6245	0.5668	840	0.5497	-0.8505	-1.6236	0.7256	0.7731	-51.8183	-44.5666	-2.0757	-2.2000
0.4258	0.5803	860	0.5523	-0.8283	-1.5989	0.7165	0.7706	-51.7566	-44.5111	-2.0815	-2.2059
0.588	0.5938	880	0.5503	-0.8351	-1.6187	0.7287	0.7836	-51.8061	-44.5282	-2.0808	-2.2058
0.4141	0.6073	900	0.5493	-0.8280	-1.6146	0.7134	0.7866	-51.7959	-44.5105	-2.0818	-2.2068
0.4387	0.6208	920	0.5485	-0.8034	-1.6183	0.7439	0.8149	-51.8050	-44.4489	-2.0928	-2.2182
0.5538	0.6343	940	0.5472	-0.7923	-1.5997	0.7378	0.8074	-51.7585	-44.4211	-2.0938	-2.2186
0.6246	0.6478	960	0.5434	-0.7854	-1.6203	0.7317	0.8349	-51.8101	-44.4038	-2.0929	-2.2178
0.5924	0.6613	980	0.5437	-0.7682	-1.5929	0.7226	0.8247	-51.7415	-44.3609	-2.0940	-2.2195
0.5123	0.6748	1000	0.5435	-0.7541	-1.5702	0.7409	0.8160	-51.6847	-44.3257	-2.1014	-2.2267
0.5138	0.6883	1020	0.5435	-0.7311	-1.5552	0.75	0.8240	-51.6473	-44.2682	-2.1009	-2.2258
0.5285	0.7018	1040	0.5419	-0.7277	-1.5460	0.7348	0.8183	-51.6243	-44.2596	-2.1035	-2.2287
0.3824	0.7152	1060	0.5394	-0.7226	-1.5544	0.7439	0.8319	-51.6454	-44.2468	-2.1033	-2.2284
0.5557	0.7287	1080	0.5415	-0.7074	-1.5317	0.7287	0.8243	-51.5886	-44.2089	-2.1052	-2.2299
0.444	0.7422	1100	0.5413	-0.6969	-1.5235	0.7530	0.8266	-51.5680	-44.1827	-2.1116	-2.2370
0.4722	0.7557	1120	0.5416	-0.6962	-1.5222	0.7470	0.8260	-51.5648	-44.1810	-2.1162	-2.2417
0.5134	0.7692	1140	0.5372	-0.6796	-1.5085	0.7470	0.8288	-51.5305	-44.1395	-2.1178	-2.2433
0.5708	0.7827	1160	0.5387	-0.6821	-1.5049	0.75	0.8228	-51.5216	-44.1456	-2.1084	-2.2328
0.4292	0.7962	1180	0.5384	-0.6850	-1.5102	0.7561	0.8252	-51.5348	-44.1529	-2.1138	-2.2393
0.5904	0.8097	1200	0.5383	-0.6984	-1.5384	0.7287	0.8400	-51.6054	-44.1865	-2.1052	-2.2298
0.4474	0.8232	1220	0.5401	-0.7007	-1.5319	0.7530	0.8312	-51.5891	-44.1922	-2.1089	-2.2343
0.3638	0.8367	1240	0.5401	-0.6971	-1.5421	0.75	0.8450	-51.6145	-44.1831	-2.1105	-2.2361
0.4443	0.8502	1260	0.5386	-0.7037	-1.5495	0.7744	0.8458	-51.6330	-44.1997	-2.1073	-2.2323
0.3582	0.8637	1280	0.5373	-0.6998	-1.5408	0.7439	0.8410	-51.6114	-44.1899	-2.1085	-2.2339
0.3966	0.8772	1300	0.5378	-0.7087	-1.5603	0.75	0.8517	-51.6602	-44.2121	-2.1039	-2.2293
0.3793	0.8907	1320	0.5379	-0.7132	-1.5514	0.7378	0.8382	-51.6379	-44.2235	-2.1080	-2.2338
0.5763	0.9042	1340	0.5381	-0.7135	-1.5541	0.7530	0.8407	-51.6447	-44.2241	-2.1102	-2.2361
0.5095	0.9177	1360	0.5363	-0.7085	-1.5465	0.7470	0.8380	-51.6255	-44.2116	-2.1041	-2.2295
0.397	0.9312	1380	0.5374	-0.7114	-1.5410	0.7409	0.8296	-51.6118	-44.2189	-2.1062	-2.2319
0.5964	0.9447	1400	0.5401	-0.7098	-1.5517	0.7530	0.8419	-51.6385	-44.2148	-2.1067	-2.2323
0.5611	0.9582	1420	0.5371	-0.7089	-1.5563	0.7530	0.8474	-51.6502	-44.2127	-2.1052	-2.2306
0.6985	0.9717	1440	0.5388	-0.7120	-1.5439	0.7409	0.8319	-51.6190	-44.2203	-2.1083	-2.2342
0.3476	0.9852	1460	0.5366	-0.7067	-1.5395	0.7378	0.8328	-51.6081	-44.2071	-2.1082	-2.2337
0.5575	0.9987	1480	0.5381	-0.7094	-1.5490	0.7622	0.8396	-51.6319	-44.2139	-2.1097	-2.2356

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/genParaMoreUniqueResNoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42

genParaMoreUniqueResNoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
genParaMoreUniqueResNoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42