Views
No views yet
v2 iteration focused on testing model limits via extended training epochs (10) and Bayesian-inspired hyperparameter adjustments to explore the trade-off between training convergence and validation generalization.load_best_model_at_end flag was used. The final weights represent the state at Epoch 2 (Validation Loss: 2.1895).| Epoch | Step | Training Loss | Validation Loss |
|---|---|---|---|
| 1 | 350 | No log | 2.1436 |
| 2 | 700 | 2.4290 | 2.1895 (Best) |
| 5 | 1750 | 0.6660 | 3.4322 |
| 10 | 3500 | 0.1098 | 4.8567 |
data.py, train.py) with integrated wandb logging and huggingface_hub syncing.| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 1.0 | 350 | 2.1436 |
| 2.4290 | 2.0 | 700 | 2.1895 |
| 1.3851 | 3.0 | 1050 | 2.2901 |
| 1.3851 | 4.0 | 1400 | 2.9491 |
| 0.6660 | 5.0 | 1750 | 3.4322 |
| 0.3433 | 6.0 | 2100 | 4.3519 |
| 0.3433 | 7.0 | 2450 | 4.5286 |
| 0.2059 | 8.0 | 2800 | 4.7578 |
| 0.1519 | 9.0 | 3150 | 4.8699 |
| 0.1098 | 10.0 | 3500 | 4.8567 |
1@article{sanh2019distilbert,
2 title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
3 author={Sanh, Victor and Debut, Lysandre and Chaumond, Adrien and Wolf, Thomas},
4 journal={arXiv preprint arXiv:1910.01108},
5 year={2019}
6}