Pretrained (from scratch for 1.2 million steps) ELECTRA discriminator model on 2021 Base PubMed abstracts and PMC open access papers with a domain specific word piece vocabulary generated using SentencePiece
byte-pair-encoding (BPE) model from PubMed abstract texts. This model is case-sensitive: it makes a difference between english and English.
Intended uses & limitations
This model is mostly intended to be fine-tuned on a downstream biomedical domain task.
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence to
make decisions, such as classification, information retrieval, relation extraction or question answering.
Training data
The pretraining corpus was built using 21.2 million PubMed abstracts from the January 2021 baseline distribution. To build the corpus,
title and abstract text sentences were extracted resulting in a corpus of 3.6 billion words. The PMC open access corpus (January 2021) is
a 12.3 billion words corpus built using the sentences extracted from the sections of PMC open access papers
excluding the references sections.
Training procedure
The training procedure follows the original ELECTRA training.
Preprocessing
A domain specific vocabulary of size 31,620 is generated using SentencePiece byte-pair-encoding (BPE) model from PubMed abstract texts.
The title and abstract text sentences were extracted using an in-house sentence segmenter trained on biomedical text. The sentences are
pre-tokenized using an in-house biomedical tokenizer for proper tokenization of biomedical entities such as gene/protein names,
organisms, antibodies, cell lines. The SentencePiece BPE vocabulary of word pieces are applied during pre-training
to the properly tokenized and segmented sentences. For the PMC open access corpus, JATS XML files for the full text papers are parsed
to extract sections excluding the reference section and section title and section body is processed in the same fashion
as the PubMed abstracts corpus.
Pretraining
The model is pretrained on a single 8 core version 3 tensor processing unit (TPU) with 128 GB of RAM for 1,200,000 steps
with a batch size of 256. The first 1,000,000 steps are pre-trained on PubMed abstracts.
After that, the model is pre-trained for another 200,000 steps on PMC open access papers.
The training parameters were the same as the original ELECTRA base model. The model has 110M parameters,
12 transformers layers with hidden layer size of 768 and 12 attention heads.
BibTeX entry and citation info
@inproceedings{ozyurt-etal-2021-detecting,
title = "Detecting Anatomical and Functional Connectivity Relations in Biomedical Literature via Language Representation Models",
author = "Ozyurt, Ibrahim Burak and
Menke, Joseph and
Bandrowski, Anita and
Martone, Maryann",
editor = "Beltagy, Iz and
Cohan, Arman and
Feigenblat, Guy and
Freitag, Dayne and
Ghosal, Tirthankar and
Hall, Keith and
Herrmannova, Drahomira and
Knoth, Petr and
Lo, Kyle and
Mayr, Philipp and
Patton, Robert M. and
Shmueli-Scheuer, Michal and
de Waard, Anita and
Wang, Kuansan and
Wang, Lucy Lu",
booktitle = "Proceedings of the Second Workshop on Scholarly Document Processing",
month = jun,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.sdp-1.4",
doi = "10.18653/v1/2021.sdp-1.4",
pages = "27--35",
abstract = "Understanding of nerve-organ interactions is crucial to facilitate the development of effective bioelectronic treatments. Towards the end of developing a systematized and computable wiring diagram of the autonomic nervous system (ANS), we introduce a curated ANS connectivity corpus together with several neural language representation model based connectivity relation extraction systems. We also show that active learning guided curation for labeled corpus expansion significantly outperforms randomly selecting connectivity relation candidates minimizing curation effort. Our final relation extraction system achieves $F_1$ = 72.8{\%} on anatomical connectivity and $F_1$ = 74.6{\%} on functional connectivity relation extraction.",
}
