ClinicalDistilBERT-i2b2-2010 is a lightweight BERT-based model developed by fine-tuning
ClinicalDistilBERT on the i2b2-2010 dataset for clinical Named Entity Recognition (NER). It is specifically designed to recognise entities from three categories:
problem,
treatment, and
test.
The architecture of this model remains the same as the ClinicalDistilBERT model. The size of the hidden dimension and the embedding layer are both set to 768. The vocabulary size is 28996. The number of transformer layers is 6, and the expansion rate of the feed-forward layer is 4. Overall, this model contains approximately 65 million parameters.
This model is suited for clinical NER and for medical tasks that require identification and classification of problems, treatments, and tests.
1@article{rohanian2023lightweight,
2 title={Lightweight transformers for clinical natural language processing},
3 author={Rohanian, Omid and Nouriborji, Mohammadmahdi and Jauncey, Hannah and Kouchaki, Samaneh and Nooralahzadeh, Farhad and Clifton, Lei and Merson, Laura and Clifton, David A and ISARIC Clinical Characterisation Group and others},
4 journal={Natural Language Engineering},
5 pages={1--28},
6 year={2023},
7 publisher={Cambridge University Press}
8}
If this model helps your work, you can keep the project running with a one-off or monthly contribution:
https://github.com/sponsors/nlpie-research
