This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.5675
Rewards/chosen: 0.1790
Rewards/rejected: -0.5404
Rewards/accuracies: 0.6951
Rewards/margins: 0.7194
Logps/rejected: -48.0861
Logps/chosen: -37.1063
Logits/rejected: -2.1842
Logits/chosen: -2.3399

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.7203	0.0287	20	0.7107	0.0038	-0.0307	0.5427	0.0345	-47.5198	-37.3009	-2.1677	-2.3078
0.659	0.0573	40	0.7098	-0.0092	-0.0433	0.5366	0.0341	-47.5338	-37.3154	-2.1636	-2.3043
0.6732	0.0860	60	0.6979	-0.0086	-0.0892	0.5915	0.0806	-47.5848	-37.3148	-2.1621	-2.3027
0.6706	0.1146	80	0.6827	-0.0292	-0.1193	0.6037	0.0901	-47.6182	-37.3376	-2.1636	-2.3052
0.648	0.1433	100	0.6645	0.0104	-0.1470	0.6341	0.1574	-47.6490	-37.2936	-2.1652	-2.3075
0.6962	0.1719	120	0.6406	-0.0105	-0.2143	0.5732	0.2038	-47.7238	-37.3169	-2.1734	-2.3187
0.6151	0.2006	140	0.6310	-0.0195	-0.2996	0.6463	0.2801	-47.8185	-37.3268	-2.1711	-2.3164
0.5982	0.2292	160	0.6154	-0.0162	-0.4299	0.6707	0.4137	-47.9634	-37.3232	-2.1646	-2.3108
0.633	0.2579	180	0.6151	0.0868	-0.3167	0.6829	0.4035	-47.8376	-37.2088	-2.1829	-2.3314
0.6804	0.2865	200	0.6000	0.1869	-0.3082	0.6829	0.4951	-47.8281	-37.0975	-2.1911	-2.3408
0.578	0.3152	220	0.5920	0.0191	-0.4711	0.6829	0.4902	-48.0092	-37.2840	-2.1740	-2.3243
0.6359	0.3438	240	0.5962	-0.1487	-0.6923	0.6951	0.5435	-48.2549	-37.4704	-2.1602	-2.3115
0.7569	0.3725	260	0.5943	-0.0067	-0.5268	0.6707	0.5201	-48.0710	-37.3127	-2.1738	-2.3272
0.6356	0.4011	280	0.5873	0.0491	-0.4960	0.6768	0.5451	-48.0368	-37.2506	-2.1773	-2.3309
0.6505	0.4298	300	0.5832	0.0016	-0.5255	0.6707	0.5271	-48.0696	-37.3034	-2.1707	-2.3242
0.621	0.4585	320	0.5852	-0.0465	-0.6553	0.7012	0.6089	-48.2139	-37.3568	-2.1694	-2.3230
0.5844	0.4871	340	0.5709	0.0202	-0.6742	0.7073	0.6944	-48.2347	-37.2828	-2.1691	-2.3234
0.525	0.5158	360	0.5718	0.0519	-0.5831	0.7134	0.6350	-48.1336	-37.2476	-2.1768	-2.3315
0.5732	0.5444	380	0.5740	0.0741	-0.6242	0.7134	0.6983	-48.1792	-37.2228	-2.1803	-2.3347
0.807	0.5731	400	0.5699	0.1748	-0.5234	0.6951	0.6982	-48.0672	-37.1109	-2.1845	-2.3390
0.6229	0.6017	420	0.5685	0.1681	-0.5622	0.7317	0.7302	-48.1103	-37.1185	-2.1882	-2.3431
0.505	0.6304	440	0.5702	0.1766	-0.5253	0.7195	0.7019	-48.0693	-37.1090	-2.1824	-2.3368
0.5254	0.6590	460	0.5673	0.1581	-0.5438	0.7012	0.7020	-48.0899	-37.1295	-2.1893	-2.3447
0.6436	0.6877	480	0.5644	0.2203	-0.5049	0.7622	0.7252	-48.0466	-37.0604	-2.1907	-2.3461
0.6245	0.7163	500	0.5610	0.1961	-0.5191	0.6951	0.7151	-48.0624	-37.0873	-2.1906	-2.3465
0.5423	0.7450	520	0.5662	0.1791	-0.5424	0.7195	0.7215	-48.0883	-37.1062	-2.1900	-2.3461
0.4641	0.7736	540	0.5671	0.2197	-0.4937	0.7195	0.7134	-48.0342	-37.0611	-2.1894	-2.3451
0.6812	0.8023	560	0.5619	0.2076	-0.5162	0.7378	0.7238	-48.0593	-37.0745	-2.1877	-2.3434
0.5947	0.8309	580	0.5610	0.1940	-0.5384	0.6951	0.7324	-48.0839	-37.0896	-2.1863	-2.3424
0.6231	0.8596	600	0.5659	0.1941	-0.5578	0.7012	0.7519	-48.1055	-37.0895	-2.1908	-2.3478
0.6086	0.8883	620	0.5744	0.1447	-0.5271	0.7073	0.6718	-48.0714	-37.1444	-2.1872	-2.3438
0.6184	0.9169	640	0.5679	0.1754	-0.5530	0.7012	0.7284	-48.1001	-37.1103	-2.1907	-2.3480
0.6287	0.9456	660	0.5658	0.1838	-0.5253	0.6951	0.7092	-48.0694	-37.1009	-2.1887	-2.3454
0.491	0.9742	680	0.5667	0.1743	-0.5375	0.7134	0.7118	-48.0829	-37.1115	-2.1877	-2.3444

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.9_42

generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.9_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr5e-07_beta0.9_42