This model is a fine-tuned version of PubMedBERT model on a dataset of 2.4 million materials science abstracts. It was introduced in this paper. This model is uncased.

Domain-specific fine-tuning has been shown to improve performance in downstream performance on a variety of NLP tasks. MaterialsBERT fine-tunes PubMedBERT, a pre-trained language model trained using biomedical literature. This model was chosen as the biomedical domain is close to the materials science domain. MaterialsBERT when further fine-tuned on a variety of downstream sequence labeling tasks in materials science, outperformed other baseline language models tested on three out of five datasets.

You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine-tuned on materials-science relevant downstream tasks.

Note that this model is primarily aimed at being fine-tuned on tasks that use a sentence or a paragraph (potentially masked) to make decisions, such as sequence classification, token classification or question answering.

Here is how to use this model to get the features of a given text in PyTorch:

A fine-tuning corpus of 2.4 million materials science abstracts was used. The DOI's of the journal articles used are provided in the file training_DOI.txt

The following hyperparameters were used during training:

If you find MaterialsBERT useful in your research, please cite the following paper: