This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_DPO_MoreUniqueResponseNoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5365
Rewards/chosen: -0.4837
Rewards/rejected: -1.1353
Rewards/accuracies: 0.6757
Rewards/margins: 0.6516
Logps/rejected: -49.0068
Logps/chosen: -41.1080
Logits/rejected: -2.1935
Logits/chosen: -2.2774

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.7027	0.0157	20	0.6985	-0.0102	-0.0125	0.4932	0.0023	-46.1998	-39.9242	-2.2115	-2.2819
0.6824	0.0314	40	0.6980	0.0067	0.0081	0.5034	-0.0014	-46.1482	-39.8820	-2.2078	-2.2777
0.6904	0.0471	60	0.6946	-0.0195	-0.0090	0.4797	-0.0105	-46.1911	-39.9475	-2.2090	-2.2796
0.6758	0.0628	80	0.6883	-0.0486	-0.0604	0.5338	0.0118	-46.3194	-40.0203	-2.2017	-2.2724
0.6737	0.0786	100	0.6815	-0.1045	-0.1405	0.5608	0.0360	-46.5198	-40.1599	-2.1904	-2.2619
0.6646	0.0943	120	0.6717	-0.1532	-0.1942	0.5811	0.0410	-46.6540	-40.2816	-2.1878	-2.2606
0.6519	0.1100	140	0.6610	-0.1554	-0.2173	0.5912	0.0618	-46.7117	-40.2873	-2.1859	-2.2592
0.66	0.1257	160	0.6490	-0.2264	-0.3375	0.5709	0.1111	-47.0123	-40.4647	-2.1678	-2.2412
0.6699	0.1414	180	0.6408	-0.2973	-0.4172	0.5777	0.1200	-47.2116	-40.6419	-2.1676	-2.2424
0.6361	0.1571	200	0.6347	-0.3652	-0.4945	0.6047	0.1294	-47.4049	-40.8117	-2.1527	-2.2283
0.6016	0.1728	220	0.6287	-0.4344	-0.5815	0.5878	0.1471	-47.6222	-40.9847	-2.1466	-2.2230
0.5929	0.1885	240	0.6278	-0.3603	-0.5158	0.6115	0.1555	-47.4580	-40.7996	-2.1629	-2.2393
0.6088	0.2042	260	0.6199	-0.3904	-0.5646	0.5946	0.1742	-47.5799	-40.8748	-2.1608	-2.2372
0.573	0.2200	280	0.6139	-0.4675	-0.6836	0.5709	0.2161	-47.8774	-41.0674	-2.1484	-2.2254
0.5345	0.2357	300	0.6061	-0.4529	-0.6894	0.6014	0.2366	-47.8921	-41.0309	-2.1510	-2.2289
0.6617	0.2514	320	0.6044	-0.4311	-0.6718	0.6047	0.2406	-47.8479	-40.9765	-2.1616	-2.2390
0.6571	0.2671	340	0.5974	-0.3485	-0.6129	0.5946	0.2645	-47.7008	-40.7699	-2.1749	-2.2527
0.5415	0.2828	360	0.5975	-0.2574	-0.5180	0.6453	0.2606	-47.4636	-40.5423	-2.2012	-2.2796
0.5788	0.2985	380	0.5930	-0.2887	-0.5744	0.6182	0.2857	-47.6046	-40.6206	-2.1989	-2.2775
0.505	0.3142	400	0.5908	-0.3239	-0.6317	0.6216	0.3078	-47.7477	-40.7085	-2.1861	-2.2652
0.6859	0.3299	420	0.5862	-0.3658	-0.6964	0.6351	0.3305	-47.9094	-40.8134	-2.1789	-2.2578
0.5429	0.3456	440	0.5843	-0.3569	-0.6947	0.6149	0.3378	-47.9052	-40.7911	-2.1860	-2.2656
0.5904	0.3614	460	0.5824	-0.3523	-0.7238	0.6351	0.3715	-47.9781	-40.7795	-2.1764	-2.2562
0.4354	0.3771	480	0.5781	-0.3446	-0.7181	0.6486	0.3734	-47.9637	-40.7603	-2.1758	-2.2564
0.5813	0.3928	500	0.5752	-0.4483	-0.8623	0.6081	0.4140	-48.3242	-41.0194	-2.1627	-2.2443
0.501	0.4085	520	0.5711	-0.4983	-0.9362	0.6351	0.4378	-48.5089	-41.1446	-2.1551	-2.2369
0.651	0.4242	540	0.5700	-0.4913	-0.9522	0.6419	0.4608	-48.5489	-41.1271	-2.1604	-2.2432
0.4632	0.4399	560	0.5651	-0.5097	-0.9743	0.6385	0.4647	-48.6043	-41.1729	-2.1639	-2.2466
0.6275	0.4556	580	0.5633	-0.5449	-1.0244	0.6419	0.4796	-48.7296	-41.2609	-2.1589	-2.2418
0.6034	0.4713	600	0.5613	-0.5323	-1.0082	0.6182	0.4759	-48.6890	-41.2296	-2.1624	-2.2455
0.4969	0.4870	620	0.5596	-0.4966	-0.9924	0.6351	0.4957	-48.6494	-41.1403	-2.1694	-2.2519
0.6308	0.5027	640	0.5568	-0.4961	-1.0351	0.6419	0.5391	-48.7564	-41.1389	-2.1701	-2.2528
0.5173	0.5185	660	0.5556	-0.5122	-1.0347	0.6554	0.5225	-48.7553	-41.1793	-2.1752	-2.2586
0.4968	0.5342	680	0.5538	-0.4745	-1.0065	0.6486	0.5321	-48.6849	-41.0849	-2.1804	-2.2631
0.5235	0.5499	700	0.5512	-0.4709	-1.0087	0.6723	0.5378	-48.6903	-41.0761	-2.1803	-2.2630
0.3771	0.5656	720	0.5487	-0.5049	-1.0562	0.6689	0.5513	-48.8089	-41.1610	-2.1773	-2.2599
0.4667	0.5813	740	0.5499	-0.5330	-1.1129	0.6791	0.5798	-48.9507	-41.2314	-2.1728	-2.2556
0.3767	0.5970	760	0.5487	-0.5517	-1.1160	0.6588	0.5644	-48.9586	-41.2779	-2.1764	-2.2596
0.5707	0.6127	780	0.5458	-0.5187	-1.0981	0.6486	0.5794	-48.9139	-41.1956	-2.1851	-2.2684
0.453	0.6284	800	0.5442	-0.4569	-1.0617	0.6824	0.6047	-48.8226	-41.0411	-2.1868	-2.2694
0.4249	0.6441	820	0.5439	-0.4822	-1.0711	0.6689	0.5889	-48.8462	-41.1042	-2.1904	-2.2735
0.6792	0.6599	840	0.5459	-0.4972	-1.0626	0.6453	0.5654	-48.8250	-41.1417	-2.1912	-2.2740
0.3986	0.6756	860	0.5419	-0.4838	-1.0705	0.6554	0.5867	-48.8448	-41.1083	-2.1928	-2.2755
0.5315	0.6913	880	0.5418	-0.4465	-1.0776	0.6824	0.6310	-48.8624	-41.0151	-2.1968	-2.2797
0.3813	0.7070	900	0.5408	-0.4400	-1.0435	0.6622	0.6035	-48.7772	-40.9989	-2.2005	-2.2839
0.5024	0.7227	920	0.5397	-0.4421	-1.0564	0.6622	0.6143	-48.8096	-41.0041	-2.2004	-2.2842
0.5267	0.7384	940	0.5403	-0.4500	-1.0564	0.6655	0.6064	-48.8095	-41.0237	-2.1941	-2.2772
0.5203	0.7541	960	0.5395	-0.4564	-1.0782	0.6655	0.6218	-48.8640	-41.0397	-2.1988	-2.2823
0.5493	0.7698	980	0.5379	-0.4713	-1.0731	0.6689	0.6018	-48.8513	-41.0770	-2.1958	-2.2790
0.5707	0.7855	1000	0.5367	-0.4955	-1.1028	0.6554	0.6074	-48.9256	-41.1374	-2.1915	-2.2746
0.5228	0.8013	1020	0.5370	-0.4628	-1.0897	0.6892	0.6269	-48.8927	-41.0557	-2.2006	-2.2844
0.4924	0.8170	1040	0.5348	-0.4595	-1.0948	0.6689	0.6352	-48.9054	-41.0476	-2.1975	-2.2808
0.527	0.8327	1060	0.5387	-0.4742	-1.1141	0.6723	0.6399	-48.9537	-41.0842	-2.2024	-2.2865
0.5437	0.8484	1080	0.5409	-0.4817	-1.1162	0.6689	0.6346	-48.9591	-41.1029	-2.1960	-2.2793
0.4516	0.8641	1100	0.5402	-0.4870	-1.1263	0.6791	0.6393	-48.9843	-41.1162	-2.1953	-2.2790
0.5082	0.8798	1120	0.5364	-0.4854	-1.1137	0.6520	0.6283	-48.9529	-41.1123	-2.2003	-2.2845
0.4971	0.8955	1140	0.5376	-0.4869	-1.1324	0.6655	0.6454	-48.9994	-41.1161	-2.1982	-2.2824
0.6034	0.9112	1160	0.5389	-0.4870	-1.1417	0.6689	0.6547	-49.0227	-41.1162	-2.1999	-2.2841
0.471	0.9269	1180	0.5404	-0.4886	-1.1201	0.6723	0.6315	-48.9687	-41.1203	-2.2009	-2.2844
0.5739	0.9427	1200	0.5363	-0.4958	-1.1378	0.6858	0.6421	-49.0130	-41.1381	-2.1959	-2.2798
0.4724	0.9584	1220	0.5349	-0.4914	-1.1427	0.6892	0.6513	-49.0253	-41.1272	-2.1934	-2.2770
0.5146	0.9741	1240	0.5379	-0.4913	-1.1395	0.6588	0.6482	-49.0172	-41.1271	-2.1924	-2.2756
0.3847	0.9898	1260	0.5384	-0.4962	-1.1436	0.6622	0.6474	-49.0275	-41.1392	-2.1929	-2.2768

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedMoreUniqueResponseNoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42

generatedMoreUniqueResponseNoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedMoreUniqueResponseNoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42