This model is a fine-tuned version of mistral-7b-base-sft-hh-harmless-4xh200-batch-64-20260418-015332 on the Anthropic/hh-rlhf dataset. It achieves the following results on the evaluation set:

Model description

More information needed

Intended uses & limitations

More information needed

More information needed

The following hyperparameters were used during training:

learning_rate: 5e-07
train_batch_size: 8
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 4
gradient_accumulation_steps: 2
total_train_batch_size: 64
total_eval_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1

Training Loss	Epoch	Step	Validation Loss	Margin Dpo/beta	Margin Dpo/loss Margin Mean	Margin Dpo/beta Margin Mean	Margin Dpo/beta Margin Std	Margin Dpo/beta Margin Grad Mean	Margin Dpo/beta Margin Grad Std	Margin Dpo/margin Mean	Margin Dpo/margin Std	Logps/chosen	Logps/rejected	Logps/ref Chosen	Logps/ref Rejected	Logits/chosen	Logits/rejected
0.9531	0.1512	100	0.6257	0.1000	4.8342	0.4834	1.1724	-0.4078	0.2146	4.8342	11.7240	-84.1893	-85.1430	-77.4087	-73.5282	-3.3317	-3.3427
0.7212	0.3023	200	0.5937	0.1000	9.0393	0.9039	1.7065	-0.3613	0.2589	9.0393	17.0650	-90.6933	-95.8521	-77.4087	-73.5282	-3.4232	-3.4322
0.8895	0.4535	300	0.5712	0.1000	11.4861	1.1486	1.9292	-0.3348	0.2720	11.4861	19.2924	-94.1929	-101.7985	-77.4087	-73.5282	-3.3493	-3.3637
0.7695	0.6047	400	0.5773	0.1000	10.8813	1.0881	1.9002	-0.3436	0.2679	10.8813	19.0023	-94.2278	-101.2286	-77.4087	-73.5282	-3.2910	-3.3049
0.7979	0.7559	500	0.5908	0.1000	11.0790	1.1079	1.9757	-0.3440	0.2732	11.0790	19.7573	-95.0573	-102.2558	-77.4087	-73.5282	-3.2756	-3.2891
0.8293	0.9070	600	0.5822	0.1000	10.9792	1.0979	1.9383	-0.3442	0.2692	10.9792	19.3832	-94.4931	-101.5918	-77.4087	-73.5282	-3.2843	-3.2979