A fine-tuned PubMedBERT model for classifying whether a medical term mentioned in a drug product label represents a true adverse drug event or an incidental mention.

This is the production model used by OnSIDES, an international database of adverse drug events extracted from product labels across four countries (USA, EU, UK, Japan).

Model Details

Base model: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract
Task: Binary text classification (is_event vs. not_event)
Architecture: PubMedBERT + Dropout(0.5) + Linear(768, 2)
Training data: 200 manually curated FDA drug labels from Demner-Fushman et al., with MedDRA term matches labeled as adverse events or incidental mentions
Sections: Trained jointly on Adverse Reactions (AR), Boxed Warnings (BW), and Warnings & Precautions (WP)
Training details: Learning rate 1e-6, batch size 32, max sequence length 256, 125-word context window, early stopping with patience 4

Performance

Held-out test set (80/10/10 drug-level split of 200 manually annotated FDA labels):

Section	F1	Precision	Recall	AUROC
Adverse Reactions	0.942	0.962	0.922	0.996
Boxed Warning	0.901	0.977	0.835	0.996
Warnings & Precautions	0.880	0.851	0.911	0.995

Independent hold-out (30 manually annotated FDA labels, not used in training or threshold tuning):

Section	F1	Precision	Recall	AUROC
Adverse Reactions	0.847	0.871	0.825	0.965
Boxed Warning	0.736	1.000	0.582	0.988
Warnings & Precautions	0.756	0.789	0.725	0.973

TAC 2017 benchmark: F1 = 89.87 (state of the art).

Usage

python

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("tatonettilab/onsides-bert")
model = AutoModelForSequenceClassification.from_pretrained("tatonettilab/onsides-bert")
model.eval()

text = "Patients receiving EXAMPLE DRUG reported nausea, headache, and dizziness."
inputs = tokenizer(text, return_tensors="pt", max_length=256, truncation=True, padding="max_length")

with torch.no_grad():
    outputs = model(**inputs)

# outputs.logits shape: (batch_size, 2)
# Column 0 = not_event score, Column 1 = is_event score
predicted_class = outputs.logits.argmax(dim=1).item()
print("is_event" if predicted_class == 1 else "not_event")

Note on ReLU

The OnSIDES training pipeline applies a ReLU activation after the classification head. The standard BertForSequenceClassification used here does not include that ReLU. For simple classification (argmax), this makes no difference. If you are applying the threshold-based scoring used in the OnSIDES pipeline, apply ReLU to the logits first:

python

import torch.nn.functional as F
scores = F.relu(outputs.logits)

Input Format

The model expects text constructed from drug label sections with MedDRA term context. In the OnSIDES pipeline, each input is a window of up to 125 words surrounding a candidate MedDRA term match, with the event term and source section prepended. See the OnSIDES repository for the full text construction pipeline.

Recommended Thresholds

For the OnSIDES v3.2.0 database, section-specific thresholds were applied to the ReLU-activated logit scores:

Section	Threshold
Adverse Reactions	0.6926
Boxed Warning	0.8713
Warnings & Precautions	0.5878

Citation

bibtex

@article{tanaka2025onsides,
  title={OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models},
  author={Tanaka, Yutaro and Chen, Hsin Yi and Belloni, Payal and Gisladottir, Undina and Kefeli, Jaden and Patterson, Joshua and Srinivasan, Ashwin and Zietz, Michael and Sirdeshmukh, Gaurav and Berkowitz, Jacob and LaRow Brown, Kathleen and Tatonetti, Nicholas P},
  journal={Med},
  year={2025},
  publisher={Elsevier},
  doi={10.1016/j.medj.2025.100642}
}

License

MIT License. See the OnSIDES repository for full details.