This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the YuchenLi01/MATH_Qwen2.5-1.5BInstruct_DPO_MoreUniqueResponseNoGTv3pair1 dataset. It achieves the following results on the evaluation set:

Loss: 0.0937
Rewards/chosen: 4.0233
Rewards/rejected: 0.0
Rewards/accuracies: 0.9750
Rewards/margins: 4.0233
Logps/rejected: -31.6415
Logps/chosen: -20.0740
Logits/rejected: -3.2851
Logits/chosen: -3.2719

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 4.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6911	0.1117	20	0.6764	0.0606	0.6750	0.0606	-41.6195	-29.9807	-2.2295	-2.3739
0.5003	0.2235	40	0.5069	0.4293	1.0	0.4293	-40.5477	-29.0590	-2.2962	-2.4321
0.2523	0.3352	60	0.2399	1.4240	1.0	1.4240	-37.7676	-26.5723	-2.5130	-2.6239
0.1162	0.4469	80	0.1232	2.6067	0.9750	2.6067	-34.6313	-23.6155	-2.8314	-2.8982
0.0582	0.5587	100	0.1090	2.9029	0.9750	2.9029	-33.9469	-22.8749	-2.9307	-2.9840
0.0789	0.6704	120	0.0984	3.1010	0.9750	3.1010	-33.6118	-22.3797	-2.9712	-3.0189
0.1318	0.7821	140	0.0926	3.2550	1.0	3.2550	-33.3097	-21.9948	-3.0080	-3.0502
0.0902	0.8939	160	0.0887	3.2978	1.0	3.2978	-33.2015	-21.8877	-3.0144	-3.0534
0.0523	1.0056	180	0.0864	3.3824	1.0	3.3824	-32.9591	-21.6763	-3.0446	-3.0821
0.0628	1.1173	200	0.0854	3.5469	1.0	3.5469	-32.5120	-21.2650	-3.0875	-3.1146
0.0545	1.2291	220	0.0860	3.6212	0.9750	3.6212	-32.3559	-21.0793	-3.1326	-3.1537
0.044	1.3408	240	0.0856	3.7094	0.9750	3.7094	-32.3282	-20.8587	-3.1514	-3.1670
0.0452	1.4525	260	0.0864	3.7423	0.9750	3.7423	-32.2762	-20.7765	-3.1658	-3.1750
0.0366	1.5642	280	0.0848	3.7498	0.9750	3.7498	-32.2279	-20.7577	-3.1695	-3.1807
0.1188	1.6760	300	0.0884	3.8206	0.9750	3.8206	-32.1108	-20.5807	-3.1770	-3.1838
0.1074	1.7877	320	0.0874	3.8179	0.9750	3.8179	-32.0523	-20.5876	-3.1902	-3.1985
0.0587	1.8994	340	0.0830	3.8498	0.9750	3.8498	-31.9974	-20.5077	-3.1829	-3.1916
0.0404	2.0112	360	0.0842	3.8876	0.9750	3.8876	-31.9993	-20.4133	-3.1898	-3.1973
0.032	2.1229	380	0.0879	3.9568	0.9750	3.9568	-31.8693	-20.2403	-3.2174	-3.2184
0.0252	2.2346	400	0.0920	3.9523	0.9750	3.9523	-31.7689	-20.2515	-3.2377	-3.2346
0.043	2.3464	420	0.0942	4.0183	0.9750	4.0183	-31.7853	-20.0864	-3.2477	-3.2403
0.0278	2.4581	440	0.0937	4.0035	0.9750	4.0035	-31.8086	-20.1236	-3.2546	-3.2467
0.043	2.5698	460	0.0932	4.0257	0.9750	4.0257	-31.7482	-20.0680	-3.2596	-3.2524
0.0346	2.6816	480	0.0929	4.0288	0.9750	4.0288	-31.8188	-20.0602	-3.2622	-3.2532
0.022	2.7933	500	0.0928	4.0027	0.9750	4.0027	-31.7470	-20.1254	-3.2613	-3.2506
0.03	2.9050	520	0.0962	4.0185	0.9750	4.0185	-31.7305	-20.0859	-3.2667	-3.2564
0.0394	3.0168	540	0.0960	4.0734	0.9750	4.0734	-31.7291	-19.9487	-3.2684	-3.2587
0.0262	3.1285	560	0.0943	4.0956	0.9750	4.0956	-31.7303	-19.8933	-3.2756	-3.2656
0.023	3.2402	580	0.0940	4.0781	0.9750	4.0781	-31.6665	-19.9370	-3.2744	-3.2607
0.0253	3.3520	600	0.0946	4.1038	0.9750	4.1038	-31.7064	-19.8729	-3.2821	-3.2702
0.0198	3.4637	620	0.0958	4.1012	0.9750	4.1012	-31.7898	-19.8793	-3.2805	-3.2678
0.0208	3.5754	640	0.0962	4.0854	0.9750	4.0854	-31.7185	-19.9187	-3.2831	-3.2702
0.0235	3.6872	660	0.0947	4.0817	0.9750	4.0817	-31.7158	-19.9280	-3.2815	-3.2678
0.0325	3.7989	680	0.0955	4.0833	0.9750	4.0833	-31.6916	-19.9240	-3.2847	-3.2721
0.0184	3.9106	700	0.0951	4.1091	0.9750	4.1091	-31.6717	-19.8594	-3.2842	-3.2705

Framework versions

Transformers 4.45.2
Pytorch 2.5.1+cu121
Datasets 3.5.0
Tokenizers 0.20.3

YuchenLi01/genv3pair1NoGT_1.5B_cdpo_lm1_ebs32_lr5e-07_beta0.4_epoch4.0_42

genv3pair1NoGT_1.5B_cdpo_lm1_ebs32_lr5e-07_beta0.4_epoch4.0_42

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

YuchenLi01/
genv3pair1NoGT_1.5B_cdpo_lm1_ebs32_lr5e-07_beta0.4_epoch4.0_42