This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5213
Rewards/chosen: -0.3830
Rewards/rejected: -1.4153
Rewards/accuracies: 0.7683
Rewards/margins: 1.0323
Logps/rejected: -51.0580
Logps/chosen: -38.2481
Logits/rejected: -2.2138
Logits/chosen: -2.3880

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-06
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.7152	0.0287	20	0.6985	0.0075	0.0130	0.4939	-0.0056	-47.4871	-37.2719	-2.1617	-2.3020
0.6718	0.0573	40	0.6945	-0.0193	-0.0140	0.5	-0.0053	-47.5546	-37.3389	-2.1619	-2.3034
0.6727	0.0860	60	0.6729	-0.0430	-0.0847	0.5366	0.0417	-47.7314	-37.3980	-2.1623	-2.3045
0.6357	0.1146	80	0.6497	-0.0908	-0.1958	0.6037	0.1050	-48.0093	-37.5177	-2.1741	-2.3199
0.6041	0.1433	100	0.6187	-0.0307	-0.2743	0.7134	0.2437	-48.2055	-37.3672	-2.1969	-2.3472
0.6856	0.1719	120	0.6061	-0.0538	-0.3810	0.7134	0.3272	-48.4722	-37.4251	-2.2004	-2.3546
0.5832	0.2006	140	0.5893	-0.0703	-0.5140	0.7256	0.4437	-48.8048	-37.4663	-2.2155	-2.3729
0.5653	0.2292	160	0.5801	-0.2248	-0.7429	0.7134	0.5181	-49.3770	-37.8526	-2.2038	-2.3641
0.5236	0.2579	180	0.5734	-0.1457	-0.7726	0.6951	0.6269	-49.4511	-37.6547	-2.2266	-2.3891
0.6719	0.2865	200	0.5653	-0.0964	-0.7810	0.7012	0.6846	-49.4721	-37.5316	-2.2467	-2.4111
0.5461	0.3152	220	0.5599	-0.3328	-1.0602	0.7378	0.7273	-50.1702	-38.1227	-2.2045	-2.3712
0.5043	0.3438	240	0.5620	-0.6478	-1.4536	0.7317	0.8058	-51.1538	-38.9101	-2.1640	-2.3326
0.647	0.3725	260	0.5536	-0.3820	-1.2183	0.7378	0.8363	-50.5655	-38.2457	-2.1928	-2.3628
0.6364	0.4011	280	0.5484	-0.3102	-1.1155	0.7378	0.8053	-50.3084	-38.0660	-2.1969	-2.3678
0.6465	0.4298	300	0.5445	-0.4353	-1.3006	0.7378	0.8653	-50.7713	-38.3789	-2.1851	-2.3567
0.6682	0.4585	320	0.5350	-0.5588	-1.4781	0.75	0.9193	-51.2150	-38.6875	-2.1794	-2.3513
0.5537	0.4871	340	0.5339	-0.5389	-1.4995	0.7561	0.9606	-51.2685	-38.6378	-2.1759	-2.3487
0.5744	0.5158	360	0.5279	-0.4846	-1.4648	0.7866	0.9802	-51.1817	-38.5021	-2.1909	-2.3639
0.4803	0.5444	380	0.5322	-0.4848	-1.4983	0.7622	1.0135	-51.2655	-38.5025	-2.2027	-2.3757
0.7301	0.5731	400	0.5295	-0.3792	-1.3990	0.7744	1.0199	-51.0173	-38.2385	-2.2318	-2.4054
0.6243	0.6017	420	0.5247	-0.3697	-1.4137	0.7683	1.0440	-51.0539	-38.2148	-2.2302	-2.4027
0.4553	0.6304	440	0.5239	-0.3363	-1.3551	0.7439	1.0188	-50.9075	-38.1314	-2.2296	-2.4016
0.5079	0.6590	460	0.5236	-0.3133	-1.3239	0.7744	1.0106	-50.8294	-38.0738	-2.2342	-2.4077
0.6451	0.6877	480	0.5215	-0.2857	-1.2973	0.7378	1.0117	-50.7631	-38.0047	-2.2323	-2.4057
0.6007	0.7163	500	0.5202	-0.3273	-1.3501	0.7317	1.0228	-50.8950	-38.1088	-2.2256	-2.3988
0.6088	0.7450	520	0.5174	-0.3078	-1.3249	0.7683	1.0171	-50.8321	-38.0601	-2.2274	-2.4007
0.4671	0.7736	540	0.5188	-0.2877	-1.2964	0.7378	1.0087	-50.7607	-38.0099	-2.2283	-2.4014
0.6831	0.8023	560	0.5197	-0.3256	-1.3349	0.75	1.0094	-50.8570	-38.1045	-2.2236	-2.3978
0.5936	0.8309	580	0.5184	-0.3688	-1.3755	0.7439	1.0067	-50.9585	-38.2126	-2.2196	-2.3941
0.4774	0.8596	600	0.5189	-0.3956	-1.4144	0.7561	1.0188	-51.0556	-38.2795	-2.2105	-2.3840
0.5629	0.8883	620	0.5193	-0.3875	-1.4014	0.7439	1.0139	-51.0233	-38.2594	-2.2097	-2.3832
0.5994	0.9169	640	0.5180	-0.3889	-1.4123	0.75	1.0233	-51.0504	-38.2629	-2.2130	-2.3873
0.5205	0.9456	660	0.5157	-0.3827	-1.4034	0.7317	1.0206	-51.0281	-38.2474	-2.2135	-2.3872
0.5182	0.9742	680	0.5183	-0.3799	-1.4065	0.7439	1.0265	-51.0358	-38.2404	-2.2154	-2.3898

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.4_42

generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.4_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.4_42