HelioBERT-XL is a BERT-based language model specialized for heliophysics domain text, developed by the NASA-ADS team at the Harvard-Smithsonian Center for Astrophysics. The model was trained on approximately 80,000 full-text heliophysics research articles.
The suffix XL refers to the scale and scope of the pretraining corpus, not to the model architecture.
An independently developed model named HelioBERT has been described in prior work by Khoo, F.S. et al. (2023) using curated definitions and abstracts. Although the names partially overlap, the two models were developed independently and differ substantially in training data.
1from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline
2
3model = AutoModelForMaskedLM.from_pretrained("adsabs/HelioBERT-XL")
4tokenizer = AutoTokenizer.from_pretrained("adsabs/HelioBERT-XL")
5
6# Example: Fill mask
7fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
8results = fill_mask("The solar <mask> affects space weather.")
9print(results)
1@misc{helioBERT2026,
2 title={HelioBERT-XL: a pre-trained language model for large-scale heliophysical literature mining},
3 author={{Alkan}, Atilla Kaan and {Accomazzi}, Alberto},
4 year={2026},
5 publisher={HuggingFace},
6 howpublished={\url{https://huggingface.co/adsabs/HelioBERT-XL}}
7}
For questions or issues, please contact the ADS team or open an issue on the model repository.
