You can deploy this model and start earning money today!

kaanino/
gpt-mha-RMSNorm

Name: gpt-mha-RMSNorm
Availability: InStock

Views

No views yet

gpt-mha-RMSNorm

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

More information needed

More information needed

More information needed

The following hyperparameters were used during training:

learning_rate: 0.0003
train_batch_size: 128
eval_batch_size: 128
seed: 20
gradient_accumulation_steps: 4
total_train_batch_size: 512
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 107
training_steps: 1072
mixed_precision_training: Native AMP

Training Loss	Epoch	Step	Validation Loss
9.4904	0.0595	107	9.2008
7.7311	0.1190	214	7.6254
7.0535	0.1786	321	7.0035
6.7238	0.2381	428	6.6589
6.4915	0.2976	535	6.4442
6.3500	0.3571	642	6.3106
6.2730	0.4166	749	6.2322
6.2243	0.4762	856	6.1910
6.2138	0.5357	963	6.1747
6.2000	0.5952	1070	6.1721
6.2000	0.5963	1072	6.1721