This model is a fine-tuned version of E:\mergekit\phi3.5-pro on the Magpie_Qwen2_Pro_300k and the Magpie_Llama_3.1_Pro_MT_300k datasets. It achieves the following results on the evaluation set:

Loss: 0.8538

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 2
eval_batch_size: 2
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
training_steps: 1000
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.7784	0.0080	100	0.8538
0.8605	0.0160	200	0.8538
0.8386	0.0240	300	0.8538
0.8201	0.0320	400	0.8538
0.8661	0.0400	500	0.8538
0.776	0.0480	600	0.8538
0.8377	0.0561	700	0.8538
0.8541	0.0641	800	0.8538
0.799	0.0721	900	0.8538
0.8146	0.0801	1000	0.8538

Framework versions

Transformers 4.45.0
Pytorch 2.4.0+cu124
Datasets 2.21.0
Tokenizers 0.20.0

RichardErkhov/win10_-_phi3.5-pro-10-08-4bits

Original model description:

phi3.5-pro-10-08

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

RichardErkhov/
win10_-_phi3.5-pro-10-08-4bits