This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5710
Rewards/chosen: -0.4881
Rewards/rejected: -0.9187
Rewards/accuracies: 0.7622
Rewards/margins: 0.4305
Logps/rejected: -56.7062
Logps/chosen: -42.1717
Logits/rejected: -2.1857
Logits/chosen: -2.3726

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6977	0.0287	20	0.6952	-0.0035	0.0008	0.4512	-0.0043	-47.5117	-37.3258	-2.1650	-2.3058
0.6867	0.0573	40	0.6935	-0.0023	-0.0016	0.4817	-0.0007	-47.5358	-37.3139	-2.1635	-2.3035
0.6906	0.0860	60	0.6915	-0.0012	-0.0040	0.5610	0.0029	-47.5601	-37.3023	-2.1614	-2.3021
0.6869	0.1146	80	0.6895	-0.0038	-0.0154	0.6037	0.0116	-47.6737	-37.3285	-2.1636	-2.3053
0.6794	0.1433	100	0.6838	0.0029	-0.0147	0.5366	0.0177	-47.6671	-37.2614	-2.1691	-2.3122
0.6844	0.1719	120	0.6774	-0.0086	-0.0400	0.6646	0.0314	-47.9197	-37.3764	-2.1757	-2.3207
0.6656	0.2006	140	0.6726	-0.0350	-0.0786	0.6341	0.0436	-48.3056	-37.6404	-2.1677	-2.3151
0.6677	0.2292	160	0.6666	-0.0668	-0.1266	0.6768	0.0598	-48.7861	-37.9588	-2.1581	-2.3085
0.6665	0.2579	180	0.6597	-0.0828	-0.1645	0.6707	0.0817	-49.1651	-38.1186	-2.1603	-2.3135
0.6704	0.2865	200	0.6540	-0.0966	-0.1920	0.6951	0.0954	-49.4395	-38.2561	-2.1641	-2.3185
0.652	0.3152	220	0.6460	-0.1355	-0.2539	0.6768	0.1185	-50.0590	-38.6453	-2.1565	-2.3141
0.6383	0.3438	240	0.6407	-0.1991	-0.3373	0.7012	0.1382	-50.8926	-39.2817	-2.1337	-2.2938
0.6778	0.3725	260	0.6330	-0.2134	-0.3768	0.7134	0.1634	-51.2880	-39.4250	-2.1334	-2.2969
0.682	0.4011	280	0.6278	-0.2379	-0.4133	0.6890	0.1754	-51.6529	-39.6696	-2.1300	-2.2962
0.6519	0.4298	300	0.6199	-0.2814	-0.4858	0.7195	0.2043	-52.3774	-40.1049	-2.1270	-2.2958
0.6179	0.4585	320	0.6123	-0.3276	-0.5644	0.7378	0.2368	-53.1632	-40.5663	-2.1251	-2.2967
0.6014	0.4871	340	0.6075	-0.3555	-0.6146	0.7317	0.2591	-53.6654	-40.8451	-2.1238	-2.2976
0.6104	0.5158	360	0.6014	-0.3748	-0.6647	0.7317	0.2899	-54.1671	-41.0390	-2.1347	-2.3119
0.6064	0.5444	380	0.5976	-0.4002	-0.7060	0.75	0.3058	-54.5798	-41.2927	-2.1372	-2.3159
0.7022	0.5731	400	0.5939	-0.4116	-0.7315	0.7317	0.3199	-54.8349	-41.4066	-2.1500	-2.3300
0.5587	0.6017	420	0.5884	-0.4264	-0.7689	0.7744	0.3425	-55.2088	-41.5550	-2.1535	-2.3339
0.546	0.6304	440	0.5849	-0.4294	-0.7903	0.7622	0.3609	-55.4229	-41.5848	-2.1599	-2.3417
0.5406	0.6590	460	0.5830	-0.4514	-0.8281	0.7439	0.3767	-55.8008	-41.8045	-2.1660	-2.3497
0.6639	0.6877	480	0.5800	-0.4533	-0.8368	0.75	0.3835	-55.8875	-41.8236	-2.1712	-2.3552
0.5935	0.7163	500	0.5769	-0.4607	-0.8642	0.7439	0.4034	-56.1615	-41.8981	-2.1704	-2.3549
0.5904	0.7450	520	0.5751	-0.4586	-0.8617	0.7622	0.4031	-56.1365	-41.8762	-2.1732	-2.3574
0.5635	0.7736	540	0.5747	-0.4619	-0.8628	0.7378	0.4009	-56.1473	-41.9093	-2.1781	-2.3633
0.6594	0.8023	560	0.5740	-0.4694	-0.8780	0.75	0.4086	-56.2997	-41.9849	-2.1863	-2.3727
0.5967	0.8309	580	0.5724	-0.4775	-0.8985	0.75	0.4210	-56.5045	-42.0651	-2.1856	-2.3723
0.5786	0.8596	600	0.5715	-0.4880	-0.9051	0.75	0.4171	-56.5705	-42.1707	-2.1825	-2.3689
0.6194	0.8883	620	0.5718	-0.4896	-0.9092	0.75	0.4196	-56.6119	-42.1863	-2.1804	-2.3665
0.6146	0.9169	640	0.5704	-0.4928	-0.9249	0.7561	0.4321	-56.7688	-42.2182	-2.1822	-2.3690
0.599	0.9456	660	0.5703	-0.4883	-0.9121	0.7561	0.4238	-56.6404	-42.1732	-2.1870	-2.3741
0.6004	0.9742	680	0.5702	-0.4916	-0.9162	0.7439	0.4246	-56.6816	-42.2066	-2.1887	-2.3755

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.1_42

generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.1_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.1_42