This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.6523
Rewards/chosen: -0.0538
Rewards/rejected: -0.3780
Rewards/accuracies: 0.6098
Rewards/margins: 0.3242
Logps/rejected: -47.7717
Logps/chosen: -37.3264
Logits/rejected: -2.1650
Logits/chosen: -2.3110

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6593	0.0287	20	0.7719	-0.0185	0.0080	0.4634	-0.0265	-47.5144	-37.3029	-2.1573	-2.2966
0.7079	0.0573	40	0.7775	-0.0516	0.0677	0.4268	-0.1193	-47.4746	-37.3250	-2.1581	-2.2976
0.7013	0.0860	60	0.7734	-0.0415	-0.0051	0.5183	-0.0364	-47.5231	-37.3182	-2.1605	-2.3007
0.8978	0.1146	80	0.7702	-0.0639	0.0537	0.4268	-0.1177	-47.4839	-37.3332	-2.1621	-2.3022
0.7576	0.1433	100	0.7593	-0.0152	0.0117	0.5244	-0.0269	-47.5119	-37.3007	-2.1608	-2.3007
0.6819	0.1719	120	0.7548	-0.1153	-0.0682	0.5	-0.0470	-47.5652	-37.3674	-2.1590	-2.2991
0.7366	0.2006	140	0.7672	0.0004	0.0067	0.4756	-0.0063	-47.5153	-37.2903	-2.1595	-2.2991
0.7714	0.2292	160	0.7582	-0.0418	-0.0015	0.4573	-0.0403	-47.5207	-37.3185	-2.1609	-2.3006
0.6972	0.2579	180	0.7581	-0.0260	0.0474	0.4512	-0.0734	-47.4881	-37.3079	-2.1639	-2.3045
0.7726	0.2865	200	0.7645	-0.0118	-0.0012	0.4634	-0.0106	-47.5205	-37.2985	-2.1665	-2.3078
0.7778	0.3152	220	0.7351	-0.0531	-0.0436	0.5122	-0.0094	-47.5488	-37.3259	-2.1615	-2.3021
0.6343	0.3438	240	0.7374	-0.0599	-0.0383	0.4878	-0.0216	-47.5453	-37.3305	-2.1601	-2.3008
0.7574	0.3725	260	0.7283	-0.1182	-0.0954	0.4695	-0.0229	-47.5833	-37.3694	-2.1592	-2.2998
0.8099	0.4011	280	0.7263	-0.1059	-0.1384	0.4817	0.0325	-47.6120	-37.3612	-2.1598	-2.3013
0.7839	0.4298	300	0.7133	-0.1115	-0.1256	0.4939	0.0141	-47.6034	-37.3649	-2.1587	-2.2997
0.6476	0.4585	320	0.7120	-0.1933	-0.2017	0.5183	0.0084	-47.6542	-37.4195	-2.1595	-2.3009
0.7405	0.4871	340	0.7154	-0.1662	-0.2575	0.5671	0.0913	-47.6914	-37.4014	-2.1576	-2.2991
0.7387	0.5158	360	0.7148	-0.2122	-0.2323	0.5305	0.0202	-47.6746	-37.4320	-2.1562	-2.2975
0.635	0.5444	380	0.7106	-0.1325	-0.2966	0.5732	0.1641	-47.7175	-37.3789	-2.1550	-2.2970
0.7021	0.5731	400	0.7017	-0.1286	-0.2789	0.5244	0.1503	-47.7056	-37.3763	-2.1596	-2.3020
0.6243	0.6017	420	0.6877	-0.1275	-0.2876	0.5549	0.1601	-47.7114	-37.3756	-2.1598	-2.3018
0.8233	0.6304	440	0.6876	-0.1110	-0.2910	0.5610	0.1801	-47.7137	-37.3645	-2.1611	-2.3037
0.697	0.6590	460	0.6871	-0.0943	-0.2543	0.5671	0.1600	-47.6892	-37.3535	-2.1558	-2.2975
0.7974	0.6877	480	0.6898	-0.0748	-0.2864	0.6220	0.2116	-47.7107	-37.3404	-2.1627	-2.3066
0.6492	0.7163	500	0.7025	-0.1463	-0.3285	0.5915	0.1821	-47.7387	-37.3881	-2.1622	-2.3053
0.7378	0.7450	520	0.6784	-0.0677	-0.2960	0.5915	0.2283	-47.7170	-37.3357	-2.1599	-2.3032
0.6013	0.7736	540	0.6942	0.0041	-0.1796	0.5671	0.1837	-47.6395	-37.2879	-2.1616	-2.3046
0.6393	0.8023	560	0.6821	-0.0495	-0.3183	0.6220	0.2688	-47.7319	-37.3236	-2.1605	-2.3038
0.6652	0.8309	580	0.6904	-0.1455	-0.2795	0.5366	0.1340	-47.7060	-37.3876	-2.1599	-2.3036
0.6374	0.8596	600	0.6718	-0.0982	-0.3261	0.5732	0.2279	-47.7371	-37.3560	-2.1636	-2.3078
0.764	0.8883	620	0.6811	-0.1401	-0.3439	0.5732	0.2038	-47.7490	-37.3840	-2.1590	-2.3026
0.656	0.9169	640	0.6709	-0.0720	-0.3387	0.5976	0.2667	-47.7455	-37.3386	-2.1622	-2.3061
0.753	0.9456	660	0.6654	-0.0230	-0.2961	0.6524	0.2732	-47.7171	-37.3059	-2.1666	-2.3116
0.7878	0.9742	680	0.6685	-0.0521	-0.3384	0.6220	0.2863	-47.7453	-37.3253	-2.1584	-2.3025
0.6933	1.0029	700	0.6634	-0.0627	-0.3464	0.6463	0.2837	-47.7507	-37.3324	-2.1591	-2.3030
0.68	1.0315	720	0.6696	-0.0394	-0.2439	0.5671	0.2045	-47.6823	-37.3169	-2.1630	-2.3073
0.5256	1.0602	740	0.6434	-0.0213	-0.3713	0.6463	0.3500	-47.7672	-37.3048	-2.1655	-2.3107
0.6364	1.0888	760	0.6581	-0.0638	-0.3735	0.6463	0.3097	-47.7687	-37.3331	-2.1683	-2.3142
0.5869	1.1175	780	0.6772	-0.0633	-0.3270	0.6159	0.2636	-47.7377	-37.3328	-2.1659	-2.3111
0.6272	1.1461	800	0.6609	-0.0277	-0.3367	0.5793	0.3090	-47.7442	-37.3091	-2.1662	-2.3113
0.7548	1.1748	820	0.6649	-0.0064	-0.3147	0.5854	0.3083	-47.7295	-37.2949	-2.1633	-2.3084
0.711	1.2034	840	0.6637	-0.0720	-0.3717	0.5915	0.2997	-47.7675	-37.3386	-2.1659	-2.3121
0.5353	1.2321	860	0.6544	-0.0280	-0.3194	0.6402	0.2915	-47.7327	-37.3092	-2.1629	-2.3077
0.6797	1.2607	880	0.6545	-0.0537	-0.3807	0.6098	0.3270	-47.7735	-37.3264	-2.1678	-2.3141
0.521	1.2894	900	0.6610	-0.0226	-0.3148	0.6098	0.2922	-47.7296	-37.3056	-2.1618	-2.3074
0.6463	1.3181	920	0.6493	-0.0407	-0.3604	0.6280	0.3197	-47.7600	-37.3177	-2.1652	-2.3101
0.825	1.3467	940	0.6445	-0.0703	-0.3495	0.6159	0.2792	-47.7527	-37.3374	-2.1653	-2.3114
0.61	1.3754	960	0.6553	-0.0256	-0.3462	0.6341	0.3205	-47.7505	-37.3077	-2.1639	-2.3092
0.5295	1.4040	980	0.6504	-0.0317	-0.3757	0.6220	0.3441	-47.7702	-37.3117	-2.1633	-2.3086
0.5618	1.4327	1000	0.6636	-0.0227	-0.3130	0.5854	0.2904	-47.7284	-37.3057	-2.1674	-2.3134
0.6392	1.4613	1020	0.6568	0.0280	-0.3496	0.6402	0.3776	-47.7528	-37.2719	-2.1655	-2.3113
0.5158	1.4900	1040	0.6500	-0.0440	-0.3588	0.6402	0.3147	-47.7589	-37.3199	-2.1661	-2.3128
0.5984	1.5186	1060	0.6537	-0.0320	-0.3934	0.6707	0.3615	-47.7820	-37.3119	-2.1683	-2.3145
0.5102	1.5473	1080	0.6522	-0.0203	-0.3262	0.6341	0.3059	-47.7372	-37.3041	-2.1631	-2.3089
0.5477	1.5759	1100	0.6368	0.0140	-0.3498	0.6524	0.3639	-47.7529	-37.2812	-2.1688	-2.3150
0.602	1.6046	1120	0.6575	0.0088	-0.3165	0.6402	0.3253	-47.7307	-37.2847	-2.1702	-2.3166
0.6069	1.6332	1140	0.6581	-0.0198	-0.3799	0.6646	0.3601	-47.7730	-37.3038	-2.1657	-2.3116
0.5541	1.6619	1160	0.6616	0.0135	-0.3330	0.6159	0.3465	-47.7417	-37.2816	-2.1615	-2.3065
0.5424	1.6905	1180	0.6582	-0.0599	-0.3615	0.5915	0.3016	-47.7607	-37.3305	-2.1606	-2.3061
0.6543	1.7192	1200	0.6573	-0.0088	-0.3782	0.6402	0.3693	-47.7718	-37.2965	-2.1627	-2.3084
0.461	1.7479	1220	0.6577	-0.0369	-0.3626	0.6098	0.3257	-47.7615	-37.3152	-2.1598	-2.3054
0.5613	1.7765	1240	0.6461	-0.0402	-0.3739	0.6220	0.3337	-47.7689	-37.3174	-2.1663	-2.3128
0.6294	1.8052	1260	0.6536	-0.0145	-0.3170	0.5793	0.3025	-47.7310	-37.3002	-2.1703	-2.3169
0.5508	1.8338	1280	0.6506	-0.0103	-0.3614	0.6402	0.3511	-47.7606	-37.2974	-2.1662	-2.3124
0.4996	1.8625	1300	0.6473	0.0687	-0.3521	0.6585	0.4208	-47.7544	-37.2447	-2.1664	-2.3120
0.5795	1.8911	1320	0.6433	-0.0118	-0.4076	0.6646	0.3958	-47.7914	-37.2984	-2.1641	-2.3100
0.4642	1.9198	1340	0.6525	0.0676	-0.2981	0.6220	0.3657	-47.7184	-37.2455	-2.1682	-2.3150
0.5638	1.9484	1360	0.6639	0.0061	-0.2607	0.5976	0.2668	-47.6935	-37.2865	-2.1671	-2.3139
0.612	1.9771	1380	0.6611	-0.0317	-0.3591	0.6280	0.3274	-47.7591	-37.3117	-2.1652	-2.3113

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr1e-07_beta1.5_epoch2.0_42

genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr1e-07_beta1.5_epoch2.0_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
genSoftQwen2.5MathRM72Bth0.5pair4NoGT_1.5B_dpo_ebs32_lr1e-07_beta1.5_epoch2.0_42