This model card provides detailed information about the PubMedBERT-based version of BALI-BERT model pre-trained with a Graph Attention Network (GAT), a biomedical language representation model enhanced through knowledge graph and language model alignment. It is based on BERT architecture (specifically PubMedBERT and BioLinkBERT variants) and has been pre-trained using the BALI method to incorporate external knowledge from biomedical Knowledge Graphs (KG), such as UMLS.
The model demonstrates improved performance across several biomedical NLP tasks including Question Answering (QA), Entity Linking (EL), and Relation Extraction (RE). The BALI approach aligns textual representations with structured biomedical knowledge by simultaneously training a language model and a KG encoder.
1from transformers import BertTokenizer, BertModel
2
3tokenizer = BertTokenizer.from_pretrained("andorei/BALI-BERT-PubMedBERT-GNN")
4model = BertModel.from_pretrained("andorei/BALI-BERT-PubMedBERT-GNN")
1@inproceedings{Sakhovskiy2025BALI,
2 author = {Sakhovskiy, Andrey and Tutubalina, Elena},
3 title = {BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment},
4 booktitle = {Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '25)},
5 year = {2025}
6}
