The German MobIE dataset is a German-language dataset, which is human-annotated with 20 coarse- and fine-grained entity types and entity linking information for geographically linkable entities. The dataset consists of 3,232 social media texts and traffic reports with 91K tokens, and contains 20.5K annotated entities, 13.1K of which are linked to a knowledge base.

The following named entities are annotated:

location-stop
trigger
organization-company
location-city
location
event-cause
location-street
time
date
number
duration
organization
person
set
distance
disaster-type
money
org-position
percent

Fine-Tuning

The latest Flair version is used for fine-tuning. Additionally, the model is trained with the FLERT (Schweter and Akbik (2020) approach, because the MobIE dataset thankfully comes with document boundary information marker.

A hyper-parameter search over the following parameters with 5 different seeds per configuration is performed:

Batch Sizes: [16]
Learning Rates: [5e-05, 3e-05]

All models are trained with the awesome AutoTrain Advanced from Hugging Face. More details can be found in this repository.

Results

A hyper-parameter search with 5 different seeds per configuration is performed and micro F1-score on development set is reported:

Configuration	Seed 1	Seed 2	Seed 3	Seed 4	Seed 5	Average
`bs16-e10-lr5e-05`	0.8446	0.8495	0.8455	0.8419	0.8476	0.8458 ± 0.0029
`bs16-e10-lr3e-05`	0.8392	0.8445	0.8495	0.8381	0.8449	0.8432 ± 0.0046

The result in bold shows the performance of this model.

Additionally, the Flair training log and TensorBoard logs are also uploaded to the model hub.

stefan-it/autotrain-flair-mobie-gbert_base-bs16-e10-lr3e-05-2

Fine-tuned Flair Model on German MobIE Dataset with AutoTrain

Dataset

Fine-Tuning

Results

stefan-it/
autotrain-flair-mobie-gbert_base-bs16-e10-lr3e-05-2