DRAGON Longformer base domain-specific
Pretrained model on Dutch clinical reports using a masked language modeling (MLM) objective. It was introduced in
this paper. The model was pretrained using domain-specific data (i.e., clinical reports) from scratch. The architecture is the same as
allenai/longformer-base-4096 from HuggingFace. The tokenizer was fitted to the dataset of Dutch medical reports, using the same settings for the tokenizer as
roberta-base.
Model description
Longformer is a transformers model that was pretrained on a large corpus of Dutch clinical reports in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labeling them in any way with an automatic process to generate inputs and labels from those texts.
This way, the model learns an inner representation of the Dutch medical language that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled reports, for instance, you can train a standard classifier using the features produced by the BERT model as inputs.
Model variations
Multiple architectures were pretrained for the DRAGON challenge.
Intended uses & limitations
You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole text (e.g., a clinical report) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at model like GPT2.
How to use
You can use this model directly with a pipeline for masked language modeling:
1from transformers import pipeline
2unmasker = pipeline("fill-mask", model="joeranbosma/dragon-longformer-base-domain-specific")
3unmasker("Dit onderzoek geen aanwijzingen voor significant carcinoom. PIRADS <mask>.")
Here is how to use this model to get the features of a given text in PyTorch:
1from transformers import AutoTokenizer, AutoModel
2tokenizer = AutoTokenizer.from_pretrained("joeranbosma/dragon-longformer-base-domain-specific")
3model = AutoModel.from_pretrained("joeranbosma/dragon-longformer-base-domain-specific")
4text = "Replace me by any text you'd like."
5encoded_input = tokenizer(text, return_tensors="pt")
6output = model(**encoded_input)
Limitations and bias
Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions. This bias will also affect all fine-tuned versions of this model.
Training data
For pretraining, 4,333,201 clinical reports (466,351 consecutive patients) were selected from Ziekenhuisgroep Twente from patients with a diagnostic or interventional visit between 13 July 2000 and 25 April 2023. 180,439 duplicate clinical reports (179,808 patients) were excluded, resulting in 4,152,762 included reports (463,692 patients). These reports were split into training (80%, 3,322,209 reports), validation (10%, 415,276 reports), and testing (10%, 415,277 reports). The testing reports were set aside for future analysis and are not used for pretraining.
Training procedure
Pretraining
The model was pretrained using masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then runs the entire masked sentence through the model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the sentence.
The details of the masking procedure for each sentence are the following:
- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by
[MASK].
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.
The HuggingFace implementation was used for pretraining:
run_mlm.py.
Pretraining hyperparameters
The following hyperparameters were used during pretraining:
learning_rate: 6e-4train_batch_size: 16eval_batch_size: 16seed: 42gradient_accumulation_steps: 16total_train_batch_size: 256optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08lr_scheduler_type: linearnum_epochs: 10.0max_seq_length: 4096
Framework versions
- Transformers 4.29.0.dev0
- Pytorch 2.0.0+cu117
- Datasets 2.11.0
- Tokenizers 0.13.3
Evaluation results
This model was evaluated on the
DRAGON benchmark for clinical NLP.
Citation
If you are using DRAGON resources, please cite the following article:
J. S. Bosma, K. Dercksen, L. Builtjes, R. André, C, Roest, S. J. Fransen, C. R. Noordman, M. Navarro-Padilla, J. Lefkes, N. Alves, M. J. J. de Grauw, L. van Eekelen, J. M. A. Spronck, M. Schuurmans, A. Saha, J. J. Twilt, W. Aswolinskiy, W. Hendrix, B. de Wilde, D. Geijs, J. Veltman, D. Yakar, M. de Rooij, F. Ciompi, A. Hering, J. Geerdink, and H. Huisman on behalf of the DRAGON consortium. The DRAGON benchmark for clinical NLP.
npj Digital Medicine 8, 289 (2025).
https://doi.org/10.1038/s41746-025-01626-x
Download the citation file for your reference manager:
BibTeX |
RIS
