BioRedditBERT is a BERT model initialised from BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) and further pre-trained on health-related Reddit posts. Please view our paper COMETA: A Corpus for Medical Entity Linking in the Social Media (EMNLP 2020) for more details.

We crawled all threads from 68 health themed subreddits such as r/AskDocs, r/health and etc. starting from the beginning of 2015 to the end of 2018, obtaining a collection of more than 800K discussions. This collection was then pruned by removing deleted posts, comments from bots or moderators, and so on. In the end, we obtained the training corpus with ca. 300 million tokens and a vocabulary size of ca. 780,000 words.

We use the same pre-training script in the original google-research/bert repo. The model is initialised with BioBERT-Base v1.0 + PubMed 200K + PMC 270K. We train with a batch size of 64, a max sequence length of 64, a learning rate of 2e-5 for 100k steps on two GeForce GTX 1080Ti (11 GB) GPUs. Other hyper-parameters are the same as default.

To show the benefit from further pre-training on the social media domain, we demonstrate results on a medical entity linking dataset also in the social media: AskAPatient (Limsopatham and Collier 2016). We follow the same 10-fold cross-validation procedure for all models and report the average result without fine-tuning. [CLS] is used as representations for entity mentions (we also tried average of all tokens but found [CLS] generally performs better).