This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.4880
Rewards/chosen: -1.1229
Rewards/rejected: -2.1588
Rewards/accuracies: 0.7622
Rewards/margins: 1.0359
Logps/rejected: -69.1077
Logps/chosen: -48.5191
Logits/rejected: -2.3867
Logits/chosen: -2.5891

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-06
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6949	0.0287	20	0.6948	-0.0038	0.0013	0.4756	-0.0051	-47.5072	-37.3289	-2.1633	-2.3036
0.6863	0.0573	40	0.6914	-0.0067	-0.0038	0.4573	-0.0029	-47.5579	-37.3576	-2.1649	-2.3065
0.6837	0.0860	60	0.6868	-0.0162	-0.0290	0.5549	0.0128	-47.8094	-37.4524	-2.1557	-2.2974
0.6671	0.1146	80	0.6762	-0.0361	-0.0733	0.6585	0.0373	-48.2529	-37.6511	-2.1706	-2.3175
0.6591	0.1433	100	0.6608	-0.0586	-0.1305	0.6707	0.0719	-48.8244	-37.8763	-2.1822	-2.3343
0.6723	0.1719	120	0.6465	-0.1239	-0.2450	0.7073	0.1211	-49.9694	-38.5296	-2.1765	-2.3346
0.6157	0.2006	140	0.6299	-0.1964	-0.3749	0.7134	0.1785	-51.2685	-39.2542	-2.1795	-2.3434
0.6195	0.2292	160	0.6117	-0.3291	-0.5849	0.7256	0.2558	-53.3683	-40.5813	-2.1710	-2.3439
0.5785	0.2579	180	0.5915	-0.4091	-0.7564	0.7439	0.3473	-55.0839	-41.3819	-2.1958	-2.3764
0.6227	0.2865	200	0.5736	-0.4693	-0.8931	0.7378	0.4237	-56.4504	-41.9839	-2.2357	-2.4195
0.5383	0.3152	220	0.5590	-0.6514	-1.1661	0.7439	0.5147	-59.1808	-43.8050	-2.2276	-2.4172
0.5418	0.3438	240	0.5488	-0.9454	-1.6019	0.75	0.6566	-63.5391	-46.7442	-2.1853	-2.3790
0.6361	0.3725	260	0.5370	-0.9104	-1.6127	0.75	0.7024	-63.6469	-46.3942	-2.2167	-2.4125
0.6761	0.4011	280	0.5295	-0.8897	-1.6104	0.7622	0.7208	-63.6239	-46.1871	-2.2399	-2.4380
0.5753	0.4298	300	0.5234	-1.0514	-1.8543	0.7561	0.8029	-66.0629	-47.8049	-2.2506	-2.4514
0.6488	0.4585	320	0.5173	-1.1448	-2.0067	0.7744	0.8619	-67.5868	-48.7387	-2.2497	-2.4509
0.4827	0.4871	340	0.5132	-1.1360	-2.0199	0.7439	0.8839	-67.7192	-48.6509	-2.2610	-2.4640
0.585	0.5158	360	0.5074	-1.1435	-2.0736	0.7561	0.9302	-68.2560	-48.7252	-2.3003	-2.5030
0.5148	0.5444	380	0.5030	-1.1361	-2.1021	0.7622	0.9660	-68.5409	-48.6516	-2.3352	-2.5388
0.7113	0.5731	400	0.5015	-1.1376	-2.1100	0.7744	0.9724	-68.6198	-48.6668	-2.3581	-2.5607
0.5057	0.6017	420	0.4998	-1.1371	-2.1209	0.7622	0.9838	-68.7283	-48.6611	-2.3582	-2.5600
0.4384	0.6304	440	0.4992	-1.1077	-2.0858	0.7683	0.9780	-68.3776	-48.3680	-2.3768	-2.5792
0.4863	0.6590	460	0.4971	-1.1131	-2.0976	0.7622	0.9845	-68.4961	-48.4220	-2.3691	-2.5706
0.5601	0.6877	480	0.4941	-1.1141	-2.1242	0.7561	1.0101	-68.7616	-48.4314	-2.3779	-2.5800
0.5503	0.7163	500	0.4927	-1.1289	-2.1435	0.7622	1.0146	-68.9545	-48.5793	-2.3793	-2.5813
0.5296	0.7450	520	0.4899	-1.0957	-2.1190	0.7683	1.0233	-68.7093	-48.2472	-2.3854	-2.5870
0.4334	0.7736	540	0.4918	-1.0799	-2.0830	0.7622	1.0032	-68.3500	-48.0893	-2.3834	-2.5848
0.6486	0.8023	560	0.4903	-1.0884	-2.1030	0.7622	1.0146	-68.5497	-48.1741	-2.3875	-2.5888
0.5369	0.8309	580	0.4901	-1.1084	-2.1215	0.7561	1.0132	-68.7348	-48.3742	-2.3920	-2.5949
0.5265	0.8596	600	0.4889	-1.1142	-2.1365	0.7622	1.0224	-68.8851	-48.4322	-2.3857	-2.5879
0.5306	0.8883	620	0.4888	-1.1193	-2.1451	0.7622	1.0258	-68.9710	-48.4838	-2.3827	-2.5840
0.5405	0.9169	640	0.4876	-1.1228	-2.1510	0.7683	1.0282	-69.0294	-48.5187	-2.3882	-2.5906
0.435	0.9456	660	0.4889	-1.1264	-2.1499	0.7683	1.0235	-69.0185	-48.5544	-2.3921	-2.5947
0.4604	0.9742	680	0.4884	-1.1237	-2.1580	0.7683	1.0342	-69.0994	-48.5280	-2.3825	-2.5842

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.1_42

generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.1_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-06_beta0.1_42