MedRoBERTa.nl finetuned for experiencer
Description
This model is a finetuned RoBERTa-based model pre-trained from scratch
on Dutch hospital notes sourced from Electronic Health Records.
All code used for the creation of MedRoBERTa.nl
can be found at
https://github.com/cltl-students/verkijk_stella_rma_thesis_dutch_medical_language_model.
The publication associated with the negation detection task can be found at
https://arxiv.org/abs/2209.00470.
The code for finetuning the model can be found at
https://github.com/umcu/negation-detection.
Minimal example
1tokenizer = AutoTokenizer\
2 .from_pretrained("UMCU/MedRoBERTa.nl_Experiencer")
3model = AutoModelForTokenClassification\
4 .from_pretrained("UMCU/MedRoBERTa.nl_Experiencer")
5
6some_text = "De patient was niet aanspreekbaar en hij zag er grauw uit. \
7Hij heeft de inspanningstest echter goed doorstaan. \
8De broer heeft onlangs een operatie ondergaan."
9
10inputs = tokenizer(some_text, return_tensors='pt')
11output = model.forward(inputs)
12probas = torch.nn.functional.softmax(output.logits[0]).detach().numpy()
13
14# associate with tokens
15input_tokens = tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])
16target_map = {0: 'B-Patient', 1:'B-Other',2:'I-Patient',3:'I-Other'}
17results = [{'token': input_tokens[idx],
18 'proba_patient': proba_arr[0]+proba_arr[2],
19 'proba_other': proba_arr[1]+proba_arr[3]
20 }
21 for idx,proba_arr in enumerate(probas)]
22
The medical entity classifiers are (being) integrated in the opensource library
clinlp, feel free to contact
us for access, either through Huggingface or through git.
It is perhaps good to note that we assume the
Inside-Outside-Beginning format.
Intended use
The model is finetuned for experiencer detection on Dutch clinical text.
Since it is a domain-specific model trained on medical data,
it is meant to be used on medical NLP tasks for Dutch.
This particular model is trained on a 64-max token windows surrounding the concept-to-be labeled.
Data
The pre-trained model was trained on nearly 10 million hospital notes from the Amsterdam University Medical Centres.
The training data was anonymized before starting the pre-training procedure.
The finetuning was performed on the Erasmus Dutch Clinical Corpus (EDCC), which was synthetically upsampled for the minority classses.
The EDCC can be obtained through Jan Kors (
j.kors@erasmusmc.nl).
The EDCC is described here:
https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-014-0373-3Authors
MedRoBERTa.nl: Stella Verkijk, Piek Vossen,
Finetuning: Bram van Es
Contact
If you are having problems with this model please add an issue on our git:
https://github.com/umcu/negation-detection/issuesUsage
If you use the model in your work please use the following referral;
https://doi.org/10.1186/s12859-022-05130-xReferences
Paper: Verkijk, S. & Vossen, P. (2022) MedRoBERTa.nl: A Language Model for Dutch Electronic Health Records. Computational Linguistics in the Netherlands Journal, 11.
Paper: Bram van Es, Leon C. Reteig, Sander C. Tan, Marijn Schraagen, Myrthe M. Hemker, Sebastiaan R.S. Arends, Miguel A.R. Rios, Saskia Haitjema (2022): Negation detection in Dutch clinical texts: an evaluation of rule-based and machine learning methods, Arxiv
