t5-small-MedicoSummarizer
This model was fine-tuned on t5-small on 25,000 PubMed articles for 10 epochs.
It achieves the following results on the evaluation set:
Training procedure
The inference engine doesn't do justice to its operation as the inference engine API doesn't work good for trainer checkpoints as the context limit is low in default for T5 which you can change while using it on backend of your application ! So, you should rather load it on the pipeline and just try it !
Training hyperparameters
The following hyperparameters were used during training:
- batch_size = 16
- training_precision: float32
- epochs = 10
- learning_rate = 2e-5
Training results
| epoch | eval_loss | eval_rouge1 | eval_rouge2 | eval_rougeL | eval_rougeLsum | eval_gen_len |
|---|
| 1.0 | 3.0605552196502686 | 0.302 | 0.0693 | 0.1841 | 0.1842 | 116.916 |
| 2.0 | 3.0079214572906494 | 0.3192 | 0.0749 | 0.1943 | 0.1944 | 122.076 |
| 3.0 | 2.9787817001342773 | 0.3209 | 0.0758 | 0.1957 | 0.1958 | 122.95 |
| 4.0 | 2.95868182182312 | 0.3226 | 0.0772 | 0.1978 | 0.1978 | 123.593 |
| 5.0 | 2.943807601928711 | 0.3186 | 0.0743 | 0.1959 | 0.1959 | 123.822 |
| 6.0 | 2.9342598915100098 | 0.3194 | 0.0755 | 0.1962 | 0.1961 | 123.834 |
| 7.0 | 2.927173376083374 | 0.3205 | 0.0758 | 0.1967 | 0.1968 | 123.967 |
| 8.0 | 2.9225199222564697 | 0.3211 | 0.0763 | 0.1974 | 0.1975 | 124.178 |
| 9.0 | 2.9196181297302246 | 0.32 | 0.0762 | 0.1964 | 0.1964 | 124.136 |
| 10.0 | 2.9186391830444336 | 0.3209 | 0.0766 | 0.1965 | 0.1965 | 124.115 |
Test Metrics
{'test_loss': 2.8919856548309326,
'test_rouge1': 0.3207,
'test_rouge2': 0.0741,
'test_rougeL': 0.1955,
'test_rougeLsum': 0.1955,
'test_gen_len': 124.285,
'test_runtime': 335.298,
'test_samples_per_second': 5.965,
'test_steps_per_second': 0.373}
Framework versions
- Transformers 4.35.2
- TensorFlow 2.15.0
- Datasets 2.16.1
- Tokenizers 0.15.0
