We present a novel resource to help scientists and medical professionals discover challenges and potential directions across scientific literature, focusing on a broad corpus pertaining to the COVID-19 pandemic and related historical research. At a high level, the challenges and directions are defined as follows:
Challenge: A sentence mentioning a problem, difficulty, flaw, limitation, failure, lack of clarity, or knowledge gap.
Research direction: A sentence mentioning suggestions or needs for further research, hypotheses, speculations, indications or hints that an issue is worthy of exploration.
The scientific-challenges-and-directions model is trained based on a dataset that is a collection of 2894 sentences and their surrounding contexts, from 1786 full-text papers in the CORD-19 corpus, labeled for classification of challenges and directions by expert annotators with biomedical and bioNLP backgrounds. For full details on the train/test/split of the data see section 3.1 in our paper
Example notebook
We include an example notebook that uses the model for inference in our repo. See Inference_Notebook.ipynb.
A training notebook is also included.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
learning rate: 2e-05
train batch size: 8
eval batch size: 4
seed: 4
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr scheduler type: linear
lr scheduler warmup steps: 500
num epochs: 30
Training results
The achieves the following results on the test set:
Precision Challenge: 0.768719
Recall Challenge: 0.780405
F1 Challenge: 0.774518
Precision Direction: 0.758112
Recall Direction: 0.774096
F1 Direction: 0.766021
Precision (micro avg. on both labels): 0.764894
Recall (micro avg. on both labels): 0.778139
F1 (micro avg. on both labels): 0.771459
Framework versions
Transformers 4.15.0
Pytorch 1.10.0+cu111
Datasets 1.17.0
Tokenizers 0.10.3
Citation
If using our dataset and models, please cite:
@misc{lahav2021search,
title={A Search Engine for Discovery of Scientific Challenges and Directions},
author={Dan Lahav and Jon Saad Falcon and Bailey Kuehl and Sophie Johnson and Sravanthi Parasa and Noam Shomron and Duen Horng Chau and Diyi Yang and Eric Horvitz and Daniel S. Weld and Tom Hope},
year={2021},
eprint={2108.13751},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
