This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_Soft_DPO_Qwen2.5MathRM72B_th0.5_pair4NoGT dataset. It achieves the following results on the evaluation set:

Loss: 0.6851
Rewards/chosen: -0.0176
Rewards/rejected: -0.0369
Rewards/accuracies: 0.6341
Rewards/margins: 0.0193
Logps/rejected: -47.8885
Logps/chosen: -37.4665
Logits/rejected: -2.1597
Logits/chosen: -2.3040

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6998	0.0287	20	0.6940	-0.0032	0.0039	0.4268	-0.0072	-47.4804	-37.3228	-2.1616	-2.3017
0.6944	0.0573	40	0.6937	-0.0020	-0.0001	0.4939	-0.0019	-47.5206	-37.3102	-2.1612	-2.3011
0.6912	0.0860	60	0.6942	-0.0025	-0.0003	0.4573	-0.0022	-47.5222	-37.3154	-2.1615	-2.3016
0.6975	0.1146	80	0.6937	-0.0006	0.0046	0.4390	-0.0052	-47.4738	-37.2964	-2.1579	-2.2974
0.6977	0.1433	100	0.6934	0.0004	0.0009	0.5061	-0.0005	-47.5107	-37.2864	-2.1638	-2.3044
0.6893	0.1719	120	0.6941	-0.0070	-0.0036	0.4573	-0.0034	-47.5558	-37.3610	-2.1588	-2.2988
0.693	0.2006	140	0.6931	-0.0005	0.0015	0.4451	-0.0020	-47.5051	-37.2960	-2.1653	-2.3059
0.6963	0.2292	160	0.6927	-0.0062	-0.0060	0.5183	-0.0002	-47.5798	-37.3525	-2.1612	-2.3017
0.6971	0.2579	180	0.6928	-0.0037	0.0006	0.4878	-0.0043	-47.5136	-37.3275	-2.1622	-2.3024
0.6943	0.2865	200	0.6917	-0.0005	-0.0047	0.5122	0.0042	-47.5668	-37.2956	-2.1652	-2.3068
0.6896	0.3152	220	0.6914	0.0013	-0.0066	0.5488	0.0079	-47.5858	-37.2779	-2.1626	-2.3033
0.6829	0.3438	240	0.6908	-0.0055	-0.0070	0.5183	0.0015	-47.5893	-37.3454	-2.1598	-2.3008
0.6911	0.3725	260	0.6902	-0.0020	-0.0051	0.5061	0.0031	-47.5706	-37.3102	-2.1645	-2.3055
0.6936	0.4011	280	0.6906	-0.0104	-0.0085	0.4878	-0.0019	-47.6050	-37.3950	-2.1600	-2.3020
0.6905	0.4298	300	0.6894	-0.0091	-0.0155	0.5915	0.0064	-47.6745	-37.3819	-2.1639	-2.3057
0.6918	0.4585	320	0.6883	-0.0150	-0.0209	0.5366	0.0058	-47.7285	-37.4409	-2.1580	-2.3006
0.6934	0.4871	340	0.6880	-0.0136	-0.0242	0.5549	0.0106	-47.7620	-37.4266	-2.1553	-2.2971
0.6879	0.5158	360	0.6870	-0.0150	-0.0246	0.5549	0.0095	-47.7653	-37.4409	-2.1551	-2.2974
0.6882	0.5444	380	0.6874	-0.0139	-0.0243	0.5549	0.0104	-47.7632	-37.4299	-2.1550	-2.2974
0.6951	0.5731	400	0.6871	-0.0141	-0.0249	0.5549	0.0108	-47.7688	-37.4318	-2.1582	-2.3009
0.6804	0.6017	420	0.6856	-0.0165	-0.0296	0.5854	0.0131	-47.8160	-37.4559	-2.1619	-2.3053
0.6944	0.6304	440	0.6856	-0.0156	-0.0313	0.5793	0.0157	-47.8330	-37.4465	-2.1571	-2.3004
0.6867	0.6590	460	0.6854	-0.0155	-0.0301	0.5671	0.0145	-47.8206	-37.4460	-2.1576	-2.3009
0.695	0.6877	480	0.6858	-0.0186	-0.0307	0.5976	0.0121	-47.8268	-37.4767	-2.1546	-2.2975
0.6876	0.7163	500	0.6858	-0.0211	-0.0370	0.6159	0.0159	-47.8900	-37.5018	-2.1512	-2.2939
0.6856	0.7450	520	0.6850	-0.0211	-0.0385	0.5976	0.0174	-47.9048	-37.5017	-2.1561	-2.3004
0.6795	0.7736	540	0.6853	-0.0180	-0.0335	0.6159	0.0155	-47.8546	-37.4704	-2.1601	-2.3036
0.6899	0.8023	560	0.6848	-0.0196	-0.0364	0.5549	0.0169	-47.8842	-37.4862	-2.1546	-2.2978
0.6773	0.8309	580	0.6839	-0.0172	-0.0348	0.6037	0.0176	-47.8677	-37.4625	-2.1522	-2.2949
0.6851	0.8596	600	0.6853	-0.0217	-0.0407	0.6341	0.0189	-47.9265	-37.5079	-2.1524	-2.2957
0.6893	0.8883	620	0.6845	-0.0237	-0.0362	0.5976	0.0124	-47.8813	-37.5277	-2.1542	-2.2981
0.6856	0.9169	640	0.6854	-0.0218	-0.0367	0.6037	0.0149	-47.8867	-37.5084	-2.1544	-2.2980
0.6836	0.9456	660	0.6846	-0.0183	-0.0352	0.6220	0.0169	-47.8712	-37.4735	-2.1563	-2.3002
0.6904	0.9742	680	0.6843	-0.0227	-0.0396	0.6037	0.0170	-47.9158	-37.5171	-2.1593	-2.3033

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-07_beta0.1_42

generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-07_beta0.1_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
generatedSoftQwen2.5MathRM72Bth0.5pair4NoGT_Qwen2.5-1.5BInstruct_dpo_ebs32_lr1e-07_beta0.1_42