DistilBioBERT is a distilled version of the
BioBERT model which is distilled for 100k training steps using a total batch size of 192 on the PubMed dataset.
This model uses a simple distillation technique, which tries to align the output distribution of the student model with the output distribution of the teacher based on the MLM objective. In addition, it optionally uses another alignment loss for aligning the last hidden state of the student and teacher.
Following
DistilBERT, we initialise the student model by taking weights from every other layer of the teacher.
In this model, the size of the hidden dimension and the embedding layer are both set to 768. The vocabulary size is 28996. The number of transformer layers is 6 and the expansion rate of the feed-forward layer is 4. Overall this model has around 65 million parameters.
1@article{rohanian2023effectiveness,
2 title={On the effectiveness of compact biomedical transformers},
3 author={Rohanian, Omid and Nouriborji, Mohammadmahdi and Kouchaki, Samaneh and Clifton, David A},
4 journal={Bioinformatics},
5 volume={39},
6 number={3},
7 pages={btad103},
8 year={2023},
9 publisher={Oxford University Press}
10}
If this model helps your work, you can keep the project running with a one-off or monthly contribution:
https://github.com/sponsors/nlpie-research
