AffilGood-NER
Overview
Click to expand
- Model type: Language Model
- Architecture: RoBERTa-base
- Language: English
- License: Apache 2.0
- Task: Named Entity Recognition
- Data: AffilGood-NER
- Additional Resources:
Model description
The English version of
affilgood-NER is a Named Entity Recognition (NER) model for identifying named entities in raw affiliation strings from scientific papers and projects,
fine-tuned from the
AffilRoberta model, a
RoBERTa base model futher pre-trained for MLM task on a medium-size corpus of raw affiliation stirngs collected from OpenAlex.
It has been trained with a dataset that contains 7 main types of entities from multilingual raw affiliation strings texts, with 5,266 texts.
After analyzing hundreds of affiliations from multiple countries and languages, we defined seven entity types: SUB-ORGANISATION, ORGANISATION, CITY, COUNTRY, ADDRESS, POSTAL-CODE, and REGION, detailed [annotation guidelines here].
Identifying named entities (organization names, cities, countries) in affiliation strings not only enables more effective linking with external organization registries, but it can also play an essential role in the geolocation of organizations and can also contribute to identify organizations and their position in an institutional hierarchy -- especially for those not listed in external databases. Information automatically extracted by means of a NER model can also facilitate the construction of knowledge graphs, and support the development of manually curated registries.
Intended Usage
This model is intended to be used for raw affiliation strings in English, because this model is pre-trained on English RoBERTa, however NER and large further pre-training corpora are both multilingual.
How to use
1from transformers import pipeline
2
3affilgood_ner_pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
4
5sentence = "CSIC, Global ecology Unit CREAF-CSIC-UAB, Bellaterra 08193, Catalonia, Spain."
6
7output = affilgood_ner_pipeline(sentence)
8
9print(output)
Limitations and bias
No measures have been taken to estimate the bias and toxicity embedded in the model.
The NER dataset contains 5,266 raw affiliation strings obtained from OpenAlex.
It includes multilingual samples from all available countries and geographies to ensure comprehensive coverage and diversity.
To enable our model to recognize various affiliation string formats, the dataset includes a wide range of structures, different ways of grouping main and subsidiary institutions and various methods of separating organization names. We also included ill-formed affiliations and those containing errors resulting from automatic extraction from PDF files.
Training
We used the
AffilGood-NER dataset for training and evaluation.
We fine-tuned the adapted and base models for token classification with the IOB annotation schema.
We trained the models for 25 epochs, using 80% of the dataset for training, 10% for validation and 10% for testing.
Hyperparameters used for training are described here:
- Learning Rate: 2e-5
- Learning Rate Decay: Linear
- Weight Decay: 0.01
- Warmup Portion: 0.06
- Batch Size: 128
- Number of Steps: 25k steps
- Adam ε: 1e-6
- Adam β1: 0.9
- Adam β2: 0.999
The best performing epoch (considering macro-averaged F1 with strict matching criteria) was used to select the model.
Evaluation
The model's performance was evaluated on a 10% of the dataset.
| Category | RoBERTa | XLM | AffilRoBERTa (this model) | AffilXLM |
|---|
| ALL | .910 | .915 | .920 | .925 |
| ----- | ------ | ------ | ------ | ---------- |
| ORG | .869 | .886 | .879 | .906 |
| SUB | .898 | .890 | .911 | .892 |
| CITY | .936 | .941 | .950 | .958 |
| COUNTRY | .971 | .973 | .980 | .970 |
| REGION | .870 | .876 | .874 | .882 |
| POSTAL | .975 | .975 | .981 | .966 |
| ADDRESS | .804 | .811 | .794 | .869 |
All the numbers reported above represent F1-score with strict match, when both the boundaries and types of the entities match.
Additional information
Authors
- SIRIS Lab, Research Division of SIRIS Academic, Barcelona, Spain
- LaSTUS Lab, TALN Group, Universitat Pompeu Fabra, Barcelona, Spain
- Institute of Computer Science, Polish Academy of Sciences, Warsaw, Poland
Contact
License
This work is distributed under a
Apache License, Version 2.0.
Funding
This work was partially funded and supporter by:
- Industrial Doctorates Plan of the Department of Research and Universities of the Generalitat de Catalunya, by Departament de Recerca i Universitats de la Generalitat de Catalunya (ajuts SGR-Cat 2021),
- Maria de Maeztu Units of Excellence Programme CEX2021-001195-M, funded by MCIN/AEI /10.13039/501100011033
- EU HORIZON SciLake (Grant Agreement 101058573)
- EU HORIZON ERINIA (Grant Agreement 101060930)
Citation
1@inproceedings{duran-silva-etal-2024-affilgood,
2 title = "{A}ffil{G}ood: Building reliable institution name disambiguation tools to improve scientific literature analysis",
3 author = "Duran-Silva, Nicolau and
4 Accuosto, Pablo and
5 Przyby{\l}a, Piotr and
6 Saggion, Horacio",
7 editor = "Ghosal, Tirthankar and
8 Singh, Amanpreet and
9 Waard, Anita and
10 Mayr, Philipp and
11 Naik, Aakanksha and
12 Weller, Orion and
13 Lee, Yoonjoo and
14 Shen, Shannon and
15 Qin, Yanxia",
16 booktitle = "Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024)",
17 month = aug,
18 year = "2024",
19 address = "Bangkok, Thailand",
20 publisher = "Association for Computational Linguistics",
21 url = "https://aclanthology.org/2024.sdp-1.13",
22 pages = "135--144",
23}
Disclaimer
Click to expand
The model published in this repository is intended for a generalist purpose
and is made available to third parties under a Apache v2.0 License.
Please keep in mind that the model may have bias and/or any other undesirable distortions.
When third parties deploy or provide systems and/or services to other parties using this model
(or a system based on it) or become users of the model itself, they should note that it is under
their responsibility to mitigate the risks arising from its use and, in any event, to comply with
applicable regulations, including regulations regarding the use of Artificial Intelligence.
In no event shall the owners and creators of the model be liable for any results arising from the use made by third parties.
