BioTinyBERT is the result of training the
TinyBERT model in a continual learning fashion for 200k training steps using a total batch size of 192 on the PubMed dataset.
We initialise our model with the pre-trained checkpoints of the
TinyBERT model available on Huggingface.
This model uses 4 hidden layers with a hidden dimension size and an embedding size of 768 resulting in a total of 15M parameters.
1@article{rohanian2023effectiveness,
2 title={On the effectiveness of compact biomedical transformers},
3 author={Rohanian, Omid and Nouriborji, Mohammadmahdi and Kouchaki, Samaneh and Clifton, David A},
4 journal={Bioinformatics},
5 volume={39},
6 number={3},
7 pages={btad103},
8 year={2023},
9 publisher={Oxford University Press}
10}
If this model helps your work, you can keep the project running with a one-off or monthly contribution:
https://github.com/sponsors/nlpie-research
