This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5609
Rewards/chosen: -0.0439
Rewards/rejected: -1.1788
Rewards/accuracies: 0.7012
Rewards/margins: 1.1349
Logps/rejected: -48.7955
Logps/chosen: -37.3540
Logits/rejected: -2.1715
Logits/chosen: -2.3334

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-06
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6914	0.0287	20	0.7204	-0.0097	-0.0016	0.5	-0.0081	-47.4874	-37.3160	-2.1618	-2.3014
0.6644	0.0573	40	0.6895	-0.0324	-0.0764	0.5488	0.0440	-47.5706	-37.3412	-2.1610	-2.3019
0.6396	0.0860	60	0.6677	-0.0819	-0.2432	0.6280	0.1614	-47.7559	-37.3961	-2.1615	-2.3035
0.6346	0.1146	80	0.6338	-0.0564	-0.3275	0.6098	0.2711	-47.8496	-37.3679	-2.1773	-2.3228
0.5677	0.1433	100	0.6116	0.1563	-0.2722	0.6707	0.4285	-47.7881	-37.1315	-2.1953	-2.3433
0.574	0.1719	120	0.5977	0.1942	-0.3517	0.7073	0.5459	-47.8765	-37.0894	-2.1959	-2.3464
0.6598	0.2006	140	0.5890	0.1995	-0.4918	0.7073	0.6913	-48.0321	-37.0835	-2.2040	-2.3561
0.5449	0.2292	160	0.5988	-0.0506	-0.8131	0.6829	0.7625	-48.3891	-37.3615	-2.1868	-2.3408
0.5197	0.2579	180	0.5897	0.1341	-0.7104	0.7622	0.8445	-48.2750	-37.1562	-2.2049	-2.3612
0.7067	0.2865	200	0.5907	0.0997	-0.6944	0.7378	0.7940	-48.2572	-37.1944	-2.2029	-2.3595
0.5366	0.3152	220	0.5991	-0.3005	-1.1523	0.7195	0.8518	-48.7660	-37.6391	-2.1589	-2.3164
0.5769	0.3438	240	0.6039	-0.6188	-1.5334	0.7012	0.9146	-49.1895	-37.9928	-2.1427	-2.3017
0.7717	0.3725	260	0.5871	-0.0330	-0.9579	0.7134	0.9249	-48.5500	-37.3419	-2.1710	-2.3303
0.644	0.4011	280	0.5883	-0.0490	-0.9241	0.6951	0.8751	-48.5124	-37.3597	-2.1661	-2.3259
0.6507	0.4298	300	0.5791	-0.2030	-1.1701	0.7073	0.9671	-48.7858	-37.5307	-2.1671	-2.3287
0.6379	0.4585	320	0.5816	-0.3173	-1.3615	0.7317	1.0442	-48.9985	-37.6578	-2.1593	-2.3204
0.5835	0.4871	340	0.5789	-0.3393	-1.4093	0.7317	1.0700	-49.0516	-37.6822	-2.1540	-2.3168
0.5286	0.5158	360	0.5715	-0.1401	-1.2177	0.7317	1.0776	-48.8387	-37.4609	-2.1650	-2.3269
0.5899	0.5444	380	0.5724	-0.1100	-1.2642	0.7195	1.1542	-48.8904	-37.4275	-2.1721	-2.3336
0.7781	0.5731	400	0.5766	0.1161	-1.0466	0.7317	1.1627	-48.6486	-37.1762	-2.2004	-2.3622
0.8553	0.6017	420	0.5678	0.0680	-1.0913	0.7378	1.1592	-48.6982	-37.2296	-2.1931	-2.3544
0.5107	0.6304	440	0.5755	0.1046	-0.9975	0.7134	1.1021	-48.5941	-37.1890	-2.2000	-2.3623
0.5223	0.6590	460	0.5723	0.0455	-1.0231	0.6768	1.0686	-48.6225	-37.2547	-2.1936	-2.3559
0.6482	0.6877	480	0.5660	0.0553	-1.0014	0.7134	1.0567	-48.5984	-37.2438	-2.1905	-2.3530
0.7152	0.7163	500	0.5720	0.0239	-1.0981	0.7073	1.1220	-48.7058	-37.2787	-2.1809	-2.3420
0.6024	0.7450	520	0.5606	0.0296	-1.0684	0.7012	1.0980	-48.6728	-37.2724	-2.1826	-2.3442
0.5466	0.7736	540	0.5647	0.0603	-1.0159	0.6768	1.0762	-48.6145	-37.2382	-2.1843	-2.3454
0.7136	0.8023	560	0.5660	-0.0280	-1.0964	0.7256	1.0684	-48.7039	-37.3363	-2.1795	-2.3412
0.587	0.8309	580	0.5656	-0.0836	-1.1990	0.6829	1.1153	-48.8179	-37.3981	-2.1724	-2.3346
0.6272	0.8596	600	0.5567	-0.1027	-1.1895	0.7256	1.0868	-48.8074	-37.4193	-2.1761	-2.3389
0.6266	0.8883	620	0.5694	-0.1310	-1.1793	0.6524	1.0483	-48.7960	-37.4508	-2.1702	-2.3325
0.7853	0.9169	640	0.5644	-0.1018	-1.2198	0.7073	1.1180	-48.8411	-37.4183	-2.1693	-2.3318
0.5052	0.9456	660	0.5623	-0.1048	-1.1960	0.7012	1.0912	-48.8146	-37.4217	-2.1681	-2.3296
0.489	0.9742	680	0.5596	-0.1201	-1.1783	0.6646	1.0582	-48.7949	-37.4386	-2.1661	-2.3275

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.9_42

generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.9_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.9_42