astroBERT: a language model for astrophysics

astroBERT models

1. Base model: Pretrained model on English language using a masked language modeling (MLM) and next sentence prediction (NSP) objective. It was introduced in this paper at ADASS 2021 and made public at ADASS 2022.
2. NER-DEAL model: This model adds a token classification head to the base model finetuned on the DEAL@WIESP2022 named entity recognition task. Must be loaded from the revision='NER-DEAL' branch (see tutorial 2).
3. SciX Categorizer: This model was finetuned to classify text into one of 7 categories of interest to SciX (Astronomy, Heliophysics, Planetary Science, Earth Science, NASA-funded Biophysics, Other Physics, Other, Text Garbage).

Tutorials

1. generate text embedding (for downstream tasks)
2. use astroBERT for the Fill-Mask task
3. make NER-DEAL predictions
4. categorize texts for SciX

BibTeX