This is a 970K parameter BERT encoder-only model trained on data from PubMed. The raw data was transformed using PaperETL with the results stored as a local dataset via the Hugging Face Datasets library.

biomedbert-hash-nano is built with the BERT Hash architecture as described in the following links.

biomedbert-hash-nano can be loaded using Hugging Face Transformers as follows. Note that given that this is a custom architecture, trust_remote_code needs to be set.

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

This Medical Abstracts Text Classification Dataset was used to evaluate the model's performance. A handful of biomedical models and general models were selected for comparison.

Metrics were generated using Hugging Face's standard run_glue script as shown below.

As we can see, this model performs very well against models much larger in size. This dataset is a challenging one!

Read more about the model in this article and this paper.