This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5973
Rewards/chosen: 0.3636
Rewards/rejected: -0.8534
Rewards/accuracies: 0.7134
Rewards/margins: 1.2170
Logps/rejected: -48.0546
Logps/chosen: -37.0628
Logits/rejected: -2.1974
Logits/chosen: -2.3573

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.638	0.0287	20	0.7595	0.0120	-0.0164	0.5	0.0284	-47.4966	-37.2972	-2.1590	-2.2984
0.7463	0.0573	40	0.7590	-0.0331	-0.0343	0.4390	0.0011	-47.5085	-37.3273	-2.1662	-2.3074
0.7033	0.0860	60	0.7389	-0.0375	-0.0255	0.4817	-0.0120	-47.5027	-37.3302	-2.1606	-2.3002
0.7151	0.1146	80	0.7261	-0.0300	-0.1293	0.5244	0.0993	-47.5719	-37.3252	-2.1619	-2.3026
0.7067	0.1433	100	0.7204	0.0078	-0.0758	0.5549	0.0837	-47.5362	-37.3000	-2.1666	-2.3080
0.7835	0.1719	120	0.6974	-0.0121	-0.2095	0.5671	0.1974	-47.6253	-37.3132	-2.1640	-2.3059
0.7371	0.2006	140	0.6753	-0.0108	-0.3205	0.6463	0.3097	-47.6993	-37.3124	-2.1595	-2.3015
0.5169	0.2292	160	0.6407	-0.0068	-0.4170	0.6463	0.4102	-47.7637	-37.3097	-2.1628	-2.3066
0.6102	0.2579	180	0.6462	0.1113	-0.3625	0.6829	0.4738	-47.7274	-37.2310	-2.1695	-2.3145
0.7105	0.2865	200	0.6430	0.1218	-0.3234	0.6280	0.4452	-47.7013	-37.2240	-2.1717	-2.3171
0.5937	0.3152	220	0.6391	-0.1067	-0.6332	0.6280	0.5265	-47.9078	-37.3763	-2.1618	-2.3090
0.7892	0.3438	240	0.6406	-0.3786	-0.9681	0.6463	0.5895	-48.1311	-37.5576	-2.1478	-2.2959
0.9081	0.3725	260	0.6457	0.0030	-0.5795	0.6280	0.5825	-47.8721	-37.3032	-2.1606	-2.3091
0.7075	0.4011	280	0.6322	0.0920	-0.5847	0.6402	0.6767	-47.8755	-37.2439	-2.1624	-2.3117
0.6145	0.4298	300	0.6247	0.1045	-0.5858	0.6829	0.6903	-47.8762	-37.2355	-2.1694	-2.3194
0.7105	0.4585	320	0.6042	0.0225	-0.7223	0.6951	0.7448	-47.9672	-37.2902	-2.1637	-2.3131
0.5697	0.4871	340	0.6072	0.0514	-0.7813	0.6951	0.8327	-48.0066	-37.2710	-2.1600	-2.3109
0.5816	0.5158	360	0.6004	0.2657	-0.5776	0.7134	0.8433	-47.8708	-37.1281	-2.1743	-2.3260
0.5369	0.5444	380	0.6020	0.2821	-0.6793	0.7378	0.9614	-47.9385	-37.1171	-2.1821	-2.3350
0.7573	0.5731	400	0.6045	0.4967	-0.3935	0.6829	0.8903	-47.7480	-36.9740	-2.1853	-2.3370
0.8046	0.6017	420	0.6002	0.3944	-0.4493	0.7256	0.8437	-47.7852	-37.0423	-2.1886	-2.3410
0.5281	0.6304	440	0.6056	0.3828	-0.4952	0.6890	0.8780	-47.8158	-37.0500	-2.1880	-2.3411
0.526	0.6590	460	0.5993	0.3334	-0.5748	0.6829	0.9082	-47.8689	-37.0829	-2.1810	-2.3342
0.6693	0.6877	480	0.5848	0.3927	-0.4869	0.6463	0.8796	-47.8103	-37.0434	-2.1786	-2.3310
0.6306	0.7163	500	0.5988	0.2394	-0.7314	0.7012	0.9708	-47.9733	-37.1456	-2.1690	-2.3219
0.5253	0.7450	520	0.6048	0.2029	-0.7126	0.6707	0.9155	-47.9607	-37.1699	-2.1717	-2.3257
0.4323	0.7736	540	0.6036	0.3413	-0.5319	0.6585	0.8732	-47.8403	-37.0776	-2.1735	-2.3271
0.7599	0.8023	560	0.6008	0.1020	-0.8371	0.6402	0.9391	-48.0437	-37.2372	-2.1646	-2.3185
0.5787	0.8309	580	0.6097	-0.0996	-1.0826	0.7134	0.9829	-48.2074	-37.3716	-2.1527	-2.3077
0.6452	0.8596	600	0.6082	-0.0786	-1.0985	0.7012	1.0200	-48.2181	-37.3576	-2.1601	-2.3162
0.7192	0.8883	620	0.6095	0.1406	-0.8255	0.6463	0.9661	-48.0360	-37.2114	-2.1669	-2.3218
0.6977	0.9169	640	0.5886	0.3763	-0.6730	0.6768	1.0492	-47.9343	-37.0543	-2.1745	-2.3297
0.5839	0.9456	660	0.6051	0.3318	-0.6998	0.6707	1.0316	-47.9522	-37.0840	-2.1724	-2.3279
0.4871	0.9742	680	0.6059	0.2037	-0.8311	0.7012	1.0348	-48.0398	-37.1694	-2.1717	-2.3286
0.2759	1.0029	700	0.6114	0.2225	-0.8600	0.7012	1.0824	-48.0590	-37.1569	-2.1659	-2.3215
0.3443	1.0315	720	0.5986	0.2145	-0.8187	0.7134	1.0331	-48.0315	-37.1622	-2.1646	-2.3199
0.1749	1.0602	740	0.5995	0.2481	-0.8386	0.6951	1.0867	-48.0448	-37.1398	-2.1689	-2.3246
0.1971	1.0888	760	0.6075	0.3860	-0.6658	0.6890	1.0518	-47.9296	-37.0479	-2.1802	-2.3379
0.4461	1.1175	780	0.5935	0.4234	-0.7185	0.6890	1.1420	-47.9647	-37.0229	-2.1765	-2.3336
0.2992	1.1461	800	0.5860	0.4457	-0.6654	0.6768	1.1111	-47.9293	-37.0081	-2.1789	-2.3365
0.5157	1.1748	820	0.6042	0.4137	-0.7332	0.6585	1.1469	-47.9745	-37.0294	-2.1820	-2.3400
0.2812	1.2034	840	0.6019	0.3442	-0.8433	0.7134	1.1875	-48.0479	-37.0757	-2.1817	-2.3403
0.2183	1.2321	860	0.5981	0.2977	-0.8283	0.6585	1.1261	-48.0379	-37.1067	-2.1784	-2.3361
0.3473	1.2607	880	0.5813	0.3570	-0.8156	0.7134	1.1726	-48.0294	-37.0672	-2.1853	-2.3440
0.1739	1.2894	900	0.5971	0.3682	-0.8084	0.6951	1.1766	-48.0246	-37.0597	-2.1847	-2.3431
0.197	1.3181	920	0.5842	0.4364	-0.7663	0.6829	1.2027	-47.9965	-37.0142	-2.1811	-2.3390
0.2598	1.3467	940	0.6054	0.4196	-0.7265	0.6707	1.1461	-47.9700	-37.0254	-2.1887	-2.3471
0.3387	1.3754	960	0.6054	0.4226	-0.7307	0.6585	1.1533	-47.9728	-37.0235	-2.1817	-2.3389
0.1937	1.4040	980	0.5877	0.3894	-0.8762	0.7195	1.2656	-48.0698	-37.0456	-2.1858	-2.3448
0.2133	1.4327	1000	0.5900	0.4252	-0.8905	0.7134	1.3157	-48.0793	-37.0217	-2.1983	-2.3586
0.1886	1.4613	1020	0.6014	0.3789	-0.8642	0.7073	1.2431	-48.0618	-37.0526	-2.1949	-2.3546
0.3081	1.4900	1040	0.5994	0.4395	-0.8459	0.7012	1.2853	-48.0496	-37.0122	-2.1935	-2.3528
0.2334	1.5186	1060	0.5974	0.3917	-0.8582	0.7012	1.2499	-48.0578	-37.0440	-2.1951	-2.3551
0.2497	1.5473	1080	0.5953	0.3964	-0.8750	0.7195	1.2714	-48.0690	-37.0409	-2.1928	-2.3525
0.3061	1.5759	1100	0.6013	0.3807	-0.8556	0.7134	1.2363	-48.0561	-37.0514	-2.1914	-2.3503
0.2808	1.6046	1120	0.5840	0.4991	-0.8607	0.7195	1.3598	-48.0595	-36.9725	-2.1937	-2.3532
0.1955	1.6332	1140	0.6063	0.3729	-0.8885	0.6890	1.2614	-48.0781	-37.0566	-2.1926	-2.3527
0.1697	1.6619	1160	0.5839	0.3488	-0.8680	0.6951	1.2168	-48.0643	-37.0726	-2.1884	-2.3474
0.3037	1.6905	1180	0.5949	0.4134	-0.9197	0.6768	1.3331	-48.0988	-37.0296	-2.1895	-2.3493
0.2436	1.7192	1200	0.5951	0.3427	-0.8923	0.6768	1.2350	-48.0805	-37.0767	-2.1903	-2.3497
0.2772	1.7479	1220	0.6009	0.3285	-0.8489	0.6707	1.1774	-48.0516	-37.0862	-2.1955	-2.3560
0.2156	1.7765	1240	0.6122	0.4231	-0.8300	0.6829	1.2531	-48.0390	-37.0231	-2.1872	-2.3462
0.2247	1.8052	1260	0.5965	0.4022	-0.8776	0.7073	1.2798	-48.0707	-37.0371	-2.1929	-2.3522
0.2587	1.8338	1280	0.6006	0.3925	-0.8724	0.7134	1.2649	-48.0673	-37.0435	-2.1938	-2.3539
0.2443	1.8625	1300	0.5840	0.4406	-0.8424	0.6951	1.2829	-48.0473	-37.0115	-2.1972	-2.3576
0.3326	1.8911	1320	0.6113	0.3384	-0.9501	0.6768	1.2885	-48.1191	-37.0796	-2.1901	-2.3493
0.1369	1.9198	1340	0.5936	0.3468	-0.8543	0.6829	1.2011	-48.0552	-37.0740	-2.1913	-2.3510
0.288	1.9484	1360	0.5971	0.3948	-0.9154	0.7195	1.3102	-48.0960	-37.0420	-2.1955	-2.3549
0.3635	1.9771	1380	0.5849	0.4370	-0.9031	0.7256	1.3400	-48.0877	-37.0139	-2.1967	-2.3570

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr5e-07_beta1.5_epoch2.0_42

genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr5e-07_beta1.5_epoch2.0_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr5e-07_beta1.5_epoch2.0_42