This model is a binary classifier that determines whether a sentence is a statement of ignorance or not.
Model Details
Model Description
This model consists of a linear classification head on top of BioMedBERT, which is a BERT-based encoder (110 million parameters, 12 layers, 768 hidden).
All layers besides 10, 11, the pooler, and classificaton head were frozen to reduce overfitting on the small training set.
Developed by: Nathan Gelfand, Darya Shlyk
Funded by: Larry Hunter
Model type: Supervised fine-tune of a transformer-encoder for text classification
Language(s) (NLP): English
License: MIT
Finetuned from model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract
Model Sources [optional]
Repository: [More Information Needed]
Uses
Direct Use
A statement of ignorance, or knowledge gap statement, is a piece of text indicating a lack of knoweldge on some subject.
In the field of biomedical research, knowledge gap statements often indicate unanswered questions, which can be useful for researchers
looking for new topics of study. We built this sentence-level binary classifier so that researchers could parse through biomedical papers
to identify research directives.
[More Information Needed]
Out-of-Scope Use
[More Information Needed]
-->
Bias, Risks, and Limitations
This model was trained on a dataset of 937 annotated statements from articles about monogenic epilepsies. Due to the small dataset size, it is possible that
the classifier will perform better on statements related to monogenic epilepsies.
The annotated statements were based on the guidelines below. Therefore, the model's classifications will also be restricted to these guidelines.
annotation_guidelines.html
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
How to Get Started with the Model
Use the code below to get started with the model.
python
1!pip install transformers
2from transformers import pipeline
34classifier = pipeline("text-classification",model="hunter-lab/sentence-level-ignorance-classifier")56sentence ="Consequently, there is a pressing need to enhance basic mechanistic and clinical studies, with a focus on individual differences, to advance the exploration of BCAAs in treating neurological disorders and to provide more robust evidence supporting their clinical applications."78result = classifier(sentence)910print(result)
Training Details
Training Data
Our training data consists of 937 statements from articles extracted from PubMed that are related to monogenic epilepsies (fill out later). Each statement was annotated as "ignorance" or "not ignorance" based on the guidelines in annotation_guidelines.html.
There are 745 training statements, 104 validation statements, and 88 testing statements.
Training Procedure
We split the data roughly 80-10-10 into train, test, and validation sets, grouping statements by article PMID using StratifiedGroupKFold to avoid leakage.
Due to the dominance of negative statements in the dataset we made a wrapper of the Trainer class called WeightedTrainer. This wrapper includes a custom compute_loss function that performs cross-entropy
loss while taking into account the modified weight of each class, which is calculated as (total labels) / (number of classes x count of this class label).
Otherwise, the procedure is fairly standard: tokenize the datasets, feed them into the trainer, train, and evaluate.
