This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 1.0162
Rewards/chosen: -1.7916
Rewards/rejected: -5.8874
Rewards/accuracies: 0.7195
Rewards/margins: 4.0958
Logps/rejected: -51.4446
Logps/chosen: -38.4850
Logits/rejected: -2.2637
Logits/chosen: -2.4493

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-06
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 4.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.8304	0.0287	20	0.7670	0.0188	0.0119	0.5061	0.0069	-47.5118	-37.2780	-2.1600	-2.2997
0.6362	0.0573	40	0.7662	-0.0134	0.0536	0.4695	-0.0670	-47.4840	-37.2995	-2.1632	-2.3036
0.6725	0.0860	60	0.7607	-0.0436	-0.0479	0.4512	0.0043	-47.5517	-37.3197	-2.1621	-2.3021
0.8179	0.1146	80	0.7458	-0.0469	-0.0561	0.5183	0.0092	-47.5571	-37.3219	-2.1622	-2.3023
0.6773	0.1433	100	0.7325	0.0314	0.0292	0.4756	0.0022	-47.5003	-37.2696	-2.1628	-2.3036
0.585	0.1719	120	0.7043	-0.0320	-0.2000	0.5610	0.1680	-47.6530	-37.3119	-2.1639	-2.3059
0.6959	0.2006	140	0.6764	-0.0060	-0.1908	0.5671	0.1848	-47.6469	-37.2946	-2.1652	-2.3078
0.7026	0.2292	160	0.6689	-0.1838	-0.5030	0.6768	0.3192	-47.8551	-37.4131	-2.1598	-2.3035
0.7005	0.2579	180	0.6471	0.1635	-0.3176	0.7256	0.4810	-47.7314	-37.1816	-2.1785	-2.3239
0.8338	0.2865	200	0.6457	0.2235	-0.1985	0.6280	0.4220	-47.6521	-37.1416	-2.1865	-2.3333
0.6198	0.3152	220	0.6457	-0.3053	-0.8176	0.6707	0.5123	-48.0648	-37.4941	-2.1564	-2.3041
0.5336	0.3438	240	0.6636	-0.5770	-1.2568	0.7073	0.6798	-48.3576	-37.6752	-2.1408	-2.2903
0.7726	0.3725	260	0.6346	0.3089	-0.3862	0.6707	0.6950	-47.7771	-37.0847	-2.1771	-2.3288
0.7529	0.4011	280	0.6338	0.3051	-0.3866	0.6707	0.6917	-47.7774	-37.0872	-2.1806	-2.3337
0.6948	0.4298	300	0.6331	-0.2301	-1.0150	0.7073	0.7849	-48.1964	-37.4440	-2.1554	-2.3089
0.6938	0.4585	320	0.6239	0.0233	-1.0318	0.7134	1.0550	-48.2076	-37.2751	-2.1749	-2.3295
0.7579	0.4871	340	0.6308	0.2097	-0.8731	0.7256	1.0828	-48.1018	-37.1508	-2.1658	-2.3198
0.7202	0.5158	360	0.6435	0.2543	-0.9137	0.7012	1.1680	-48.1288	-37.1210	-2.1747	-2.3312
0.582	0.5444	380	0.6668	0.0802	-1.1585	0.7622	1.2387	-48.2921	-37.2371	-2.1762	-2.3317
0.7623	0.5731	400	0.6714	0.5303	-0.7170	0.7561	1.2473	-47.9977	-36.9371	-2.2022	-2.3582
0.8745	0.6017	420	0.6609	-0.1810	-1.4091	0.7134	1.2281	-48.4591	-37.4112	-2.1717	-2.3289
0.7939	0.6304	440	0.6628	0.4375	-0.7566	0.6707	1.1941	-48.0241	-36.9989	-2.1953	-2.3518
0.4907	0.6590	460	0.6873	0.1455	-1.0636	0.6768	1.2091	-48.2288	-37.1936	-2.1826	-2.3406
1.0007	0.6877	480	0.6906	-0.2392	-1.4462	0.7073	1.2070	-48.4839	-37.4501	-2.1639	-2.3212
0.7244	0.7163	500	0.7011	-0.3988	-1.6033	0.7134	1.2045	-48.5886	-37.5564	-2.1640	-2.3214
0.8677	0.7450	520	0.6933	0.1626	-1.1180	0.6890	1.2806	-48.2650	-37.1822	-2.1835	-2.3415
0.7469	0.7736	540	0.7119	-0.5559	-1.6500	0.6463	1.0940	-48.6197	-37.6612	-2.1493	-2.3081
0.9638	0.8023	560	0.7212	-1.0506	-2.3978	0.7012	1.3472	-49.1182	-37.9910	-2.1380	-2.2982
0.8861	0.8309	580	0.7284	-1.3072	-2.7125	0.6646	1.4052	-49.3280	-38.1620	-2.1244	-2.2862
0.728	0.8596	600	0.7373	-1.0705	-2.4578	0.6098	1.3873	-49.1582	-38.0042	-2.1359	-2.2976
0.9275	0.8883	620	0.7632	-0.4970	-1.9258	0.6646	1.4289	-48.8036	-37.6219	-2.1712	-2.3332
1.2508	0.9169	640	0.7344	-0.4909	-1.9320	0.6768	1.4411	-48.8077	-37.6178	-2.1707	-2.3328
0.5959	0.9456	660	0.7164	-0.2732	-1.9511	0.7073	1.6778	-48.8204	-37.4727	-2.1794	-2.3436
0.6993	0.9742	680	0.7611	-0.5714	-2.2983	0.6768	1.7269	-49.0519	-37.6715	-2.1718	-2.3368
0.1337	1.0029	700	0.7495	-0.7054	-2.3260	0.6890	1.6206	-49.0704	-37.7608	-2.1643	-2.3269
0.1699	1.0315	720	0.7543	-0.7161	-2.5007	0.7073	1.7846	-49.1868	-37.7680	-2.1571	-2.3217
0.1283	1.0602	740	0.7589	-0.6813	-2.4768	0.6585	1.7955	-49.1709	-37.7448	-2.1550	-2.3200
0.1562	1.0888	760	0.7429	-0.5634	-2.5569	0.6951	1.9935	-49.2243	-37.6662	-2.1653	-2.3308
0.5311	1.1175	780	0.7775	-1.0203	-3.1402	0.6890	2.1198	-49.6131	-37.9708	-2.1527	-2.3192
0.4137	1.1461	800	0.7656	-0.3285	-2.4731	0.7073	2.1446	-49.1684	-37.5096	-2.1891	-2.3550
0.6751	1.1748	820	0.7913	-0.2700	-2.3390	0.6890	2.0690	-49.0791	-37.4706	-2.2094	-2.3779
0.1539	1.2034	840	0.7976	-1.5556	-3.6189	0.6585	2.0633	-49.9323	-38.3276	-2.1445	-2.3124
0.1011	1.2321	860	0.7970	-1.2605	-3.3755	0.6768	2.1150	-49.7700	-38.1309	-2.1631	-2.3293
0.5283	1.2607	880	0.8396	-1.5793	-3.8022	0.6768	2.2229	-50.0545	-38.3434	-2.1582	-2.3266
0.7135	1.2894	900	0.8334	-1.5101	-3.7366	0.6829	2.2265	-50.0108	-38.2973	-2.1631	-2.3318
0.2984	1.3181	920	0.8648	-1.8201	-4.1025	0.6951	2.2824	-50.2547	-38.5040	-2.1657	-2.3359
0.3403	1.3467	940	0.8541	-1.2708	-3.6216	0.7073	2.3507	-49.9341	-38.1378	-2.1963	-2.3668
0.5027	1.3754	960	0.8330	-0.5263	-2.9518	0.6890	2.4255	-49.4875	-37.6414	-2.2320	-2.4028
0.0995	1.4040	980	0.8377	-0.9473	-3.4641	0.7073	2.5168	-49.8291	-37.9221	-2.2060	-2.3772
0.3713	1.4327	1000	0.8671	-1.2163	-3.9083	0.7195	2.6920	-50.1252	-38.1014	-2.2061	-2.3807
0.0925	1.4613	1020	0.8432	-1.5758	-4.1746	0.7256	2.5987	-50.3027	-38.3411	-2.1840	-2.3568
0.0534	1.4900	1040	0.8369	-1.4133	-4.0950	0.7012	2.6816	-50.2497	-38.2328	-2.2070	-2.3806
0.3965	1.5186	1060	0.8501	-1.7584	-4.4655	0.7195	2.7070	-50.4967	-38.4629	-2.1941	-2.3663
0.4998	1.5473	1080	0.8768	-1.8363	-4.5320	0.7378	2.6957	-50.5411	-38.5148	-2.2003	-2.3719
0.1402	1.5759	1100	0.8700	-2.1504	-4.7623	0.7134	2.6120	-50.6946	-38.7242	-2.1737	-2.3441
0.415	1.6046	1120	0.8747	-1.3385	-3.8922	0.7134	2.5538	-50.1145	-38.1829	-2.2281	-2.3998
0.2698	1.6332	1140	0.8735	-1.7164	-4.2559	0.7073	2.5395	-50.3570	-38.4348	-2.2045	-2.3752
0.0763	1.6619	1160	0.9318	-2.0715	-4.5639	0.6707	2.4924	-50.5623	-38.6716	-2.1923	-2.3632
0.4862	1.6905	1180	0.8864	-1.5227	-3.9800	0.6890	2.4574	-50.1731	-38.3057	-2.2026	-2.3744
0.1828	1.7192	1200	0.8657	-2.0581	-4.5170	0.6768	2.4589	-50.5311	-38.6627	-2.1649	-2.3347
0.8748	1.7479	1220	0.8725	-1.8711	-4.2568	0.6890	2.3857	-50.3576	-38.5380	-2.1849	-2.3544
0.1738	1.7765	1240	0.8865	-1.9762	-4.4127	0.6951	2.4365	-50.4615	-38.6081	-2.1823	-2.3528
0.6807	1.8052	1260	0.8993	-1.9143	-4.5318	0.7012	2.6176	-50.5409	-38.5667	-2.1896	-2.3605
0.234	1.8338	1280	0.9174	-2.5956	-5.3648	0.6951	2.7693	-51.0963	-39.0210	-2.1534	-2.3255
0.4561	1.8625	1300	0.8979	-2.6642	-5.3206	0.6829	2.6564	-51.0668	-39.0667	-2.1580	-2.3311
0.6914	1.8911	1320	0.8833	-1.9465	-4.7044	0.7256	2.7579	-50.6559	-38.5882	-2.1779	-2.3502
1.0903	1.9198	1340	0.9047	-1.7187	-4.3419	0.7134	2.6232	-50.4143	-38.4364	-2.1965	-2.3688
0.3091	1.9484	1360	0.8893	-1.8354	-4.3342	0.6768	2.4988	-50.4092	-38.5142	-2.1924	-2.3635
0.7892	1.9771	1380	0.8478	-1.3234	-3.9150	0.6829	2.5916	-50.1297	-38.1728	-2.2034	-2.3746
0.026	2.0057	1400	0.8436	-1.4237	-4.0477	0.7195	2.6240	-50.2182	-38.2397	-2.1862	-2.3562
0.0216	2.0344	1420	0.8778	-1.5276	-4.1728	0.6707	2.6452	-50.3016	-38.3090	-2.1997	-2.3723
0.1155	2.0630	1440	0.8614	-1.8066	-4.6925	0.7012	2.8859	-50.6480	-38.4949	-2.1905	-2.3658
0.0333	2.0917	1460	0.8720	-1.7916	-4.8425	0.6951	3.0508	-50.7480	-38.4850	-2.1909	-2.3655
0.0065	2.1203	1480	0.8907	-2.2836	-5.5083	0.7073	3.2247	-51.1919	-38.8130	-2.1742	-2.3494
0.1135	2.1490	1500	0.9206	-2.0477	-5.3310	0.7134	3.2833	-51.0737	-38.6557	-2.2001	-2.3762
0.0118	2.1777	1520	0.8977	-1.8609	-5.1792	0.7012	3.3183	-50.9725	-38.5312	-2.2120	-2.3888
0.0058	2.2063	1540	0.9138	-2.2020	-5.5578	0.6829	3.3558	-51.2249	-38.7586	-2.1989	-2.3756
0.008	2.2350	1560	0.9406	-2.2408	-5.6494	0.7134	3.4086	-51.2860	-38.7844	-2.2144	-2.3928
0.0807	2.2636	1580	0.9357	-1.7367	-5.1251	0.6890	3.3883	-50.9364	-38.4484	-2.2403	-2.4199
0.0494	2.2923	1600	0.9389	-1.5582	-5.0445	0.6951	3.4863	-50.8827	-38.3294	-2.2496	-2.4291
0.0323	2.3209	1620	0.9412	-1.1509	-4.6054	0.6890	3.4545	-50.5900	-38.0578	-2.2710	-2.4499
0.0635	2.3496	1640	0.9329	-0.9837	-4.5050	0.7195	3.5213	-50.5230	-37.9464	-2.2741	-2.4524
0.1836	2.3782	1660	0.9313	-1.3010	-4.8870	0.7012	3.5860	-50.7777	-38.1579	-2.2637	-2.4427
0.105	2.4069	1680	0.9318	-1.1725	-4.8436	0.7012	3.6711	-50.7488	-38.0722	-2.2642	-2.4427
0.0214	2.4355	1700	0.9438	-1.2472	-4.8447	0.7012	3.5975	-50.7495	-38.1220	-2.2730	-2.4525
0.0219	2.4642	1720	0.9540	-1.5534	-4.9143	0.6890	3.3609	-50.7959	-38.3262	-2.2462	-2.4249
0.0209	2.4928	1740	0.9504	-1.9131	-5.4075	0.6951	3.4944	-51.1247	-38.5660	-2.2204	-2.3992
0.3896	2.5215	1760	0.9495	-2.2027	-5.8164	0.7012	3.6137	-51.3973	-38.7591	-2.2097	-2.3896
0.0053	2.5501	1780	0.9591	-2.1149	-5.7074	0.7012	3.5925	-51.3247	-38.7005	-2.2229	-2.4039
0.0063	2.5788	1800	0.9747	-2.1093	-5.6902	0.7012	3.5809	-51.3132	-38.6968	-2.2227	-2.4047
0.0149	2.6074	1820	0.9843	-2.1397	-5.7288	0.7195	3.5891	-51.3389	-38.7170	-2.2230	-2.4052
0.0015	2.6361	1840	0.9688	-1.8198	-5.4233	0.7134	3.6035	-51.1352	-38.5037	-2.2331	-2.4153
0.0741	2.6648	1860	0.9637	-1.8000	-5.4062	0.6829	3.6063	-51.1239	-38.4906	-2.2298	-2.4121
0.0214	2.6934	1880	0.9676	-1.8870	-5.4770	0.6890	3.5899	-51.1710	-38.5486	-2.2261	-2.4080
0.0154	2.7221	1900	0.9726	-2.3333	-6.0594	0.7256	3.7261	-51.5593	-38.8461	-2.2008	-2.3830
0.1174	2.7507	1920	0.9882	-2.4666	-6.1760	0.7073	3.7094	-51.6370	-38.9350	-2.2028	-2.3854
0.0932	2.7794	1940	0.9815	-2.2907	-6.0779	0.7073	3.7871	-51.5716	-38.8177	-2.2066	-2.3894
0.2748	2.8080	1960	0.9771	-2.1064	-5.8560	0.7012	3.7497	-51.4237	-38.6948	-2.2211	-2.4042
0.1624	2.8367	1980	0.9654	-2.2142	-6.0091	0.7073	3.7949	-51.5258	-38.7667	-2.2172	-2.4004
0.0235	2.8653	2000	0.9676	-2.2284	-5.9517	0.7256	3.7233	-51.4875	-38.7762	-2.2166	-2.3990
0.3314	2.8940	2020	0.9565	-1.8847	-5.6339	0.7378	3.7492	-51.2756	-38.5470	-2.2419	-2.4244
0.0604	2.9226	2040	0.9645	-1.9323	-5.6101	0.7134	3.6778	-51.2597	-38.5788	-2.2399	-2.4223
0.617	2.9513	2060	0.9762	-2.0059	-5.6764	0.7134	3.6705	-51.3040	-38.6278	-2.2300	-2.4121
0.4133	2.9799	2080	0.9604	-1.9974	-5.7035	0.7195	3.7060	-51.3220	-38.6222	-2.2335	-2.4158
0.0065	3.0086	2100	0.9570	-1.8994	-5.6413	0.7378	3.7419	-51.2806	-38.5569	-2.2328	-2.4154
0.0022	3.0372	2120	0.9583	-1.9299	-5.7363	0.7195	3.8064	-51.3439	-38.5772	-2.2268	-2.4100
0.0019	3.0659	2140	0.9786	-1.7379	-5.5144	0.7256	3.7766	-51.1960	-38.4492	-2.2449	-2.4297
0.3588	3.0946	2160	0.9848	-1.8091	-5.6070	0.7195	3.7979	-51.2577	-38.4966	-2.2455	-2.4300
0.0102	3.1232	2180	0.9834	-1.8771	-5.8048	0.6951	3.9277	-51.3896	-38.5420	-2.2529	-2.4377
0.0075	3.1519	2200	1.0007	-1.9006	-5.8413	0.6890	3.9407	-51.4139	-38.5576	-2.2474	-2.4319
0.0009	3.1805	2220	0.9965	-1.9431	-5.9261	0.7195	3.9830	-51.4704	-38.5860	-2.2478	-2.4328
0.0033	3.2092	2240	1.0033	-1.9713	-5.9471	0.7195	3.9758	-51.4844	-38.6048	-2.2474	-2.4323
0.0033	3.2378	2260	0.9980	-1.8737	-5.9520	0.7317	4.0783	-51.4877	-38.5397	-2.2538	-2.4390
0.0057	3.2665	2280	0.9988	-1.8191	-5.8558	0.7195	4.0368	-51.4236	-38.5033	-2.2510	-2.4346
0.0006	3.2951	2300	0.9936	-1.8720	-5.9195	0.7317	4.0475	-51.4660	-38.5386	-2.2551	-2.4403
0.0015	3.3238	2320	0.9941	-1.8551	-5.8772	0.7134	4.0221	-51.4378	-38.5273	-2.2552	-2.4397
0.0014	3.3524	2340	1.0005	-1.8474	-5.8557	0.7134	4.0083	-51.4235	-38.5222	-2.2585	-2.4438
0.0011	3.3811	2360	1.0014	-1.7614	-5.8516	0.7256	4.0901	-51.4207	-38.4649	-2.2586	-2.4427
0.0106	3.4097	2380	1.0029	-1.6883	-5.7998	0.7378	4.1115	-51.3862	-38.4161	-2.2661	-2.4514
0.0015	3.4384	2400	0.9899	-1.7533	-5.8722	0.7073	4.1189	-51.4345	-38.4594	-2.2656	-2.4507
0.0049	3.4670	2420	1.0110	-1.8369	-5.9242	0.7256	4.0873	-51.4692	-38.5152	-2.2696	-2.4557
0.003	3.4957	2440	0.9980	-1.8219	-5.9375	0.7317	4.1156	-51.4781	-38.5052	-2.2690	-2.4549
0.0022	3.5244	2460	1.0017	-1.7931	-5.9493	0.7012	4.1562	-51.4859	-38.4860	-2.2667	-2.4531
0.0133	3.5530	2480	1.0083	-1.8030	-5.9039	0.7134	4.1010	-51.4557	-38.4925	-2.2650	-2.4509
0.0037	3.5817	2500	0.9977	-1.8449	-5.9766	0.7317	4.1317	-51.5041	-38.5205	-2.2639	-2.4498
0.0039	3.6103	2520	1.0095	-1.8164	-5.8789	0.7256	4.0625	-51.4390	-38.5015	-2.2666	-2.4525
0.0017	3.6390	2540	0.9950	-1.7903	-5.8949	0.7073	4.1047	-51.4497	-38.4841	-2.2570	-2.4414
0.0058	3.6676	2560	1.0091	-1.8273	-6.0293	0.7378	4.2020	-51.5392	-38.5088	-2.2608	-2.4463
0.0006	3.6963	2580	1.0155	-1.8057	-5.9605	0.7134	4.1548	-51.4933	-38.4944	-2.2665	-2.4524
0.0005	3.7249	2600	1.0006	-1.8044	-5.9597	0.7073	4.1552	-51.4928	-38.4935	-2.2627	-2.4480
0.0012	3.7536	2620	1.0069	-1.7658	-5.9615	0.7195	4.1957	-51.4940	-38.4678	-2.2644	-2.4504
0.0554	3.7822	2640	0.9973	-1.7912	-5.9332	0.7073	4.1420	-51.4751	-38.4847	-2.2651	-2.4514
0.0006	3.8109	2660	1.0209	-1.7942	-5.9544	0.7256	4.1602	-51.4893	-38.4867	-2.2684	-2.4546
0.0033	3.8395	2680	1.0083	-1.7742	-5.9169	0.7134	4.1427	-51.4643	-38.4734	-2.2665	-2.4532
0.0012	3.8682	2700	0.9773	-1.8225	-5.9677	0.7317	4.1452	-51.4982	-38.5056	-2.2624	-2.4478
0.0034	3.8968	2720	1.0132	-1.8293	-5.9698	0.7195	4.1405	-51.4995	-38.5101	-2.2656	-2.4511
0.0055	3.9255	2740	0.9984	-1.7828	-5.9506	0.7378	4.1677	-51.4867	-38.4791	-2.2647	-2.4501
0.0023	3.9542	2760	1.0088	-1.8096	-5.9363	0.7073	4.1267	-51.4772	-38.4969	-2.2653	-2.4513
0.0022	3.9828	2780	1.0047	-1.7975	-5.9437	0.7073	4.1461	-51.4822	-38.4889	-2.2633	-2.4488

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr1e-06_beta1.5_epoch4.0_42

genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr1e-06_beta1.5_epoch4.0_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr1e-06_beta1.5_epoch4.0_42