You can deploy this model and start earning money today!

kanishka/
opt-babylm2-rewritten-clean-spacy-earlystop_multi-adj-strict-reversed-bpe_seed-42_1e-3

Name: opt-babylm2-rewritten-clean-spacy-earlystop_multi-adj-strict-reversed-bpe_seed-42_1e-3
Availability: InStock

generated_from_trainer

kanishka/babylm2-rewritten-clean_multi-adj-strict-reversed

model-index

text-generation-inference

endpoints_compatible

Views

No views yet

opt-babylm2-rewritten-clean-spacy-earlystop_multi-adj-strict-reversed-bpe_seed-42_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean_multi-adj-strict-reversed dataset. It achieves the following results on the evaluation set:

Loss: 2.7259
Accuracy: 0.4704

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.001
train_batch_size: 32
eval_batch_size: 64
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 256
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 32000
num_epochs: 20.0
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.1744	0.9998	2231	3.8837	0.3488
3.5063	1.9998	4462	3.3554	0.3991
3.1761	2.9998	6693	3.1386	0.4212
3.0096	3.9998	8924	3.0304	0.4321
2.8872	4.9998	11155	2.9693	0.4385
2.8305	5.9998	13386	2.9277	0.4429
2.7857	6.9998	15617	2.9000	0.4461
2.7507	7.9998	17848	2.8810	0.4483
2.7317	8.9998	20079	2.8694	0.4496
2.7112	9.9998	22310	2.8540	0.4514
2.6934	10.9998	24541	2.8473	0.4523
2.6799	11.9998	26772	2.8395	0.4531
2.6819	12.9998	29003	2.8363	0.4536
2.6724	13.9998	31234	2.8323	0.4540
2.6581	14.9998	33465	2.8148	0.4561
2.611	15.9998	35696	2.7881	0.4598
2.5586	16.9998	37927	2.7641	0.4632
2.5038	17.9998	40158	2.7447	0.4662
2.4389	18.9998	42389	2.7288	0.4689
2.3665	19.9998	44620	2.7259	0.4704

Framework versions

Transformers 4.48.0
Pytorch 2.6.0+cu124
Datasets 3.2.0
Tokenizers 0.21.1

kanishka/opt-babylm2-rewritten-clean-spacy-earlystop_multi-adj-strict-reversed-bpe_seed-42_1e-3

opt-babylm2-rewritten-clean-spacy-earlystop_multi-adj-strict-reversed-bpe_seed-42_1e-3

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

kanishka/
opt-babylm2-rewritten-clean-spacy-earlystop_multi-adj-strict-reversed-bpe_seed-42_1e-3