opt-babylm2-rewritten-clean-spacy-earlystop_ablate_both_strict-bpe_seed-211_1e-3
This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_ablate_both_strict dataset.
It achieves the following results on the evaluation set:
- Loss: 2.7069
- Accuracy: 0.4765
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 32
- eval_batch_size: 64
- seed: 211
- gradient_accumulation_steps: 8
- total_train_batch_size: 256
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 32000
- num_epochs: 20.0
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|
| 4.0205 | 0.9996 | 2121 | 3.8595 | 0.3573 |
| 3.3894 | 1.9996 | 4242 | 3.3505 | 0.4048 |
| 3.0794 | 2.9996 | 6363 | 3.1252 | 0.4274 |
| 2.9216 | 3.9996 | 8484 | 3.0129 | 0.4388 |
| 2.8265 | 4.9996 | 10605 | 2.9465 | 0.4451 |
| 2.7609 | 5.9996 | 12726 | 2.9057 | 0.4496 |
| 2.7124 | 6.9996 | 14847 | 2.8786 | 0.4528 |
| 2.6728 | 7.9996 | 16968 | 2.8570 | 0.4549 |
| 2.656 | 8.9996 | 19089 | 2.8429 | 0.4567 |
| 2.6354 | 9.9996 | 21210 | 2.8323 | 0.4581 |
| 2.6219 | 10.9996 | 23331 | 2.8234 | 0.4585 |
| 2.6072 | 11.9996 | 25452 | 2.8147 | 0.4600 |
| 2.5955 | 12.9996 | 27573 | 2.8112 | 0.4607 |
| 2.5858 | 13.9996 | 29694 | 2.8037 | 0.4614 |
| 2.5758 | 14.9996 | 31815 | 2.8026 | 0.4613 |
| 2.5602 | 15.9996 | 33936 | 2.7752 | 0.4648 |
| 2.5101 | 16.9996 | 36057 | 2.7504 | 0.4683 |
| 2.4526 | 17.9996 | 38178 | 2.7286 | 0.4718 |
| 2.3831 | 18.9996 | 40299 | 2.7097 | 0.4749 |
| 2.3029 | 19.9996 | 42420 | 2.7069 | 0.4765 |
Framework versions
- Transformers 4.48.0
- Pytorch 2.6.0+cu124
- Datasets 3.2.0
- Tokenizers 0.21.1
