This is a 22.7M parameter BERT encoder-only model trained on data from PubMed. The raw data was transformed using PaperETL with the results stored as a local dataset via the Hugging Face Datasets library.

This model is designed to be a solid-performing small model fitting in between the 110M parameter BiomedBERT Base model and the tiny BiomedBERT Hash series of models.

biomedbert-small can be loaded using Hugging Face Transformers as follows.

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

This Medical Abstracts Text Classification Dataset was used to evaluate the model's performance. A handful of biomedical models and general models were selected for comparison.

Metrics were generated using Hugging Face's standard run_glue script as shown below.

As we can see, this model performs very well against models much larger in size. This dataset is a challenging one!

Read more about the model in this article.