xlm-roberta-base-wikiann-ner is the first
Named Entity Recognition model for 20 languages (Arabic, Assamese, Bengali, Catalan, English, Spanish, Basque, French, Gujarati, Hindi, Indonesia, Igbo, Marathi, Punjabi, Portugues and Swahili, Urdu, Vietnamese, Yoruba, Chinese) based on a fine-tuned XLM-RoBERTa large model. It achieves the
state-of-the-art performance for the NER task. It has been trained to recognize three types of entities: location (LOC), organizations (ORG), and person (PER).
Specifically, this model is a
xlm-roberta-large model that was fine-tuned on an aggregation of languages datasets obtained from
WikiANN dataset.
1from transformers import AutoTokenizer, AutoModelForTokenClassification
2from transformers import pipeline
3tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base-wikiann-ner")
4model = AutoModelForTokenClassification.from_pretrained("xlm-roberta-base-wikiann-ner")
5nlp = pipeline("ner", model=model, tokenizer=tokenizer)
6example = "Ìbọn ń ró kù kù gẹ́gẹ́ bí ọwọ́ ọ̀pọ̀ aráàlù ṣe tẹ ìbọn ní Kyiv láti dojú kọ Russia"
7ner_results = nlp(example)
8print(ner_results)
This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains.
This model was fine-tuned on 20 NER datasets (Arabic, Assamese, Bengali, Catalan, English, Spanish, Basque, French, Gujarati, Hindi, Indonesia, Igbo, Marathi, Punjabi, Portugues and Swahili, Urdu, Vietnamese, Yoruba, Chinese)
wikiann.
The training dataset distinguishes between the beginning and continuation of an entity so that if there are back-to-back entities of the same type, the model can output where the second entity begins. As in the dataset, each token will be classified as one of the following classes:
