This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5578
Rewards/chosen: -0.0862
Rewards/rejected: -0.6853
Rewards/accuracies: 0.7439
Rewards/margins: 0.5991
Logps/rejected: -49.2329
Logps/chosen: -37.5060
Logits/rejected: -2.2118
Logits/chosen: -2.3768

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6877	0.0287	20	0.6999	-0.0104	0.0131	0.4878	-0.0235	-47.4869	-37.3166	-2.1641	-2.3046
0.6889	0.0573	40	0.6988	-0.0132	-0.0032	0.4878	-0.0100	-47.5277	-37.3236	-2.1604	-2.3009
0.6843	0.0860	60	0.6922	-0.0246	-0.0162	0.4939	-0.0084	-47.5601	-37.3520	-2.1598	-2.2999
0.6838	0.1146	80	0.6798	-0.0070	-0.0441	0.5732	0.0371	-47.6300	-37.3080	-2.1623	-2.3036
0.6441	0.1433	100	0.6692	0.0252	-0.0536	0.6341	0.0788	-47.6537	-37.2277	-2.1720	-2.3151
0.6681	0.1719	120	0.6503	-0.0185	-0.1259	0.6585	0.1074	-47.8345	-37.3368	-2.1787	-2.3243
0.6531	0.2006	140	0.6409	-0.0570	-0.2254	0.6951	0.1684	-48.0831	-37.4330	-2.1665	-2.3124
0.6302	0.2292	160	0.6293	-0.1259	-0.3276	0.6951	0.2017	-48.3387	-37.6053	-2.1616	-2.3099
0.6272	0.2579	180	0.6204	-0.0726	-0.3208	0.6829	0.2483	-48.3218	-37.4720	-2.1784	-2.3298
0.6611	0.2865	200	0.6103	-0.0444	-0.3274	0.6890	0.2830	-48.3382	-37.4016	-2.1920	-2.3447
0.6018	0.3152	220	0.6088	-0.1048	-0.4129	0.6768	0.3080	-48.5519	-37.5527	-2.1785	-2.3321
0.5739	0.3438	240	0.5965	-0.2171	-0.5790	0.7134	0.3619	-48.9673	-37.8334	-2.1584	-2.3130
0.7083	0.3725	260	0.5932	-0.1912	-0.5425	0.7134	0.3512	-48.8759	-37.7687	-2.1671	-2.3232
0.6482	0.4011	280	0.5884	-0.1336	-0.5201	0.7134	0.3865	-48.8200	-37.6246	-2.1735	-2.3309
0.6393	0.4298	300	0.5809	-0.1868	-0.6204	0.6951	0.4336	-49.0708	-37.7577	-2.1747	-2.3341
0.6313	0.4585	320	0.5773	-0.2115	-0.6630	0.7195	0.4516	-49.1773	-37.8192	-2.1754	-2.3355
0.5955	0.4871	340	0.5751	-0.1911	-0.6845	0.7195	0.4934	-49.2310	-37.7684	-2.1709	-2.3305
0.5917	0.5158	360	0.5713	-0.1733	-0.6976	0.75	0.5244	-49.2638	-37.7238	-2.1805	-2.3416
0.566	0.5444	380	0.5713	-0.1674	-0.6985	0.7134	0.5311	-49.2660	-37.7091	-2.1909	-2.3533
0.684	0.5731	400	0.5669	-0.1228	-0.6728	0.7439	0.5500	-49.2017	-37.5975	-2.1982	-2.3606
0.565	0.6017	420	0.5654	-0.1216	-0.6853	0.7378	0.5637	-49.2328	-37.5945	-2.1999	-2.3617
0.5291	0.6304	440	0.5651	-0.0997	-0.6795	0.7195	0.5798	-49.2184	-37.5397	-2.2068	-2.3703
0.5378	0.6590	460	0.5659	-0.1041	-0.6739	0.7561	0.5699	-49.2046	-37.5508	-2.2071	-2.3706
0.6722	0.6877	480	0.5625	-0.0873	-0.6642	0.7256	0.5769	-49.1801	-37.5088	-2.2106	-2.3745
0.6128	0.7163	500	0.5630	-0.0885	-0.6695	0.7195	0.5810	-49.1934	-37.5119	-2.2077	-2.3721
0.5999	0.7450	520	0.5606	-0.0694	-0.6562	0.7134	0.5867	-49.1601	-37.4641	-2.2154	-2.3805
0.5078	0.7736	540	0.5584	-0.0596	-0.6459	0.7317	0.5864	-49.1346	-37.4395	-2.2134	-2.3781
0.6555	0.8023	560	0.5571	-0.0736	-0.6536	0.7073	0.5800	-49.1537	-37.4745	-2.2113	-2.3758
0.6191	0.8309	580	0.5583	-0.0994	-0.6865	0.7134	0.5870	-49.2359	-37.5392	-2.2129	-2.3782
0.5419	0.8596	600	0.5581	-0.0940	-0.6921	0.7195	0.5981	-49.2498	-37.5255	-2.2112	-2.3764
0.6156	0.8883	620	0.5597	-0.1016	-0.7012	0.7195	0.5996	-49.2727	-37.5446	-2.2112	-2.3765
0.6097	0.9169	640	0.5594	-0.0776	-0.6794	0.7256	0.6018	-49.2183	-37.4846	-2.2113	-2.3767
0.6072	0.9456	660	0.5576	-0.0884	-0.6966	0.7439	0.6082	-49.2612	-37.5115	-2.2132	-2.3787
0.5635	0.9742	680	0.5589	-0.0754	-0.6929	0.7378	0.6175	-49.2520	-37.4792	-2.2143	-2.3799

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42

generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.4_42