This token classification has been finetuned on human sequence annotations of sentences of British parties' election manifestos for the following entity types:

social group
implicit social group reference
political group
political institution
organization, public institution, or collective actor

Please refer to Licht & Sczepanski (2025) for details.

Developed by: Hauke Licht
Model type: modernbert
Language(s) (NLP): ['en']
License: apache-2.0
Finetuned from model: answerdotai/ModernBERT-base
Funded by: Center for Comparative and International Studies of the ETH Zurich and the University of Zurich and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy – EXC 2126/1 – 390838866

Model Sources

Repository: https://github.com/haukelicht/group_mention_detection/release/
Paper: https://doi.org/10.31219/osf.io/ufb96
Demo: [More Information Needed]

Uses

Bias, Risks, and Limitations

Evaluation of the classifier in held-out data shows that it makes mistakes (see section Results).
The model has been finetuned only on human-annotated labeled sentences sampled from British parties party manifestos. Applying the classifier in other domains can lead to higher error rates than those reported in section Results below.
The data used to finetune the model come from human annotators. Human annotators can be biased and factors like gender and social background can impact their annotations judgments. This may lead to bias in the detection of specific social groups.

Recommendations

Users who want to apply the model outside its training data domain (British parties' election programs) should evaluate its performance in the target data.
Users who want to apply the model outside its training data domain (British parties' election programs) should contuninue to finetune this model on labeled data.

How to Get Started with the Model

Use the code below to get started with the model.

pyhton

from transformers import pipeline

model_id = "haukelicht/roberta-base-group-mention-detector-uk-manifestos"

classifier = pipeline(task="ner", model=model_id, aggregation_strategy="simple")

text = "Our party fights for the deprived and the vulnerable in our country."
annotations = classifier(text)
print(annotations)

# get annotations' character start and end indexes
locations = [(anno['start'], anno['end']) for anno in annotations]
locations

# index the source text using first annotation as an example
loc = locations[0]
text[slice(*loc)]

Training Details

Training Data

The train, dev, and test splits used for model finetuning and evaluation are available on Github: https://github.com/haukelicht/group_mention_detection/release/splits

Training Procedure

Training Hyperparameters

epochs: 2
learning rate: 5e-05
batch size: 8
weight decay: 0.3
warmup ratio: 0.1

Evaluation

Testing Data, Factors & Metrics

Testing Data

The train, dev, and test splits used for model finetuning and evaluation are available on Github: https://github.com/haukelicht/group_mention_detection/release/splits

Metrics

seq-eval F1: strict seqeuence labeling evaluation metric per CoNLL-2000 shared task based on https://github.com/chakki-works/seqeval
"soft" seq-eval F1: a more lenient seqeuence labeling evaluation metric that reports span level average performance suzmmarized across examples per https://github.com/haukelicht/soft-seqeval
sentence-level F1: binary measure of detection performance considering a sentence a positive example/prediction if it contains at least one enttiy to of the given type

Results

type	seq-eval F1	soft seq-eval F1	sentence level F1
social group	0.718	0.775	0.938
political group	0.925	0.933	0.990
political institution	0.706	0.736	0.954
organization, public institution, or collective actor	0.609	0.605	0.931
implicit social group reference	0.697	0.600	0.951

Citation

BibTeX:

[More Information Needed]

APA:

Licht, H., & Sczepanski, R. (2025). Detecting Group Mentions in Political Rhetoric: A Supervised Learning Approach. forthcoming in British Journal of Political Science. Preprint available at OSF

More Information

https://github.com/haukelicht/group_mention_detection/release

Model Card Contact

hauke.licht@uibk.ac.at

haukelicht/ModernBERT-base-group-mention-detector-uk-manifestos

ModernBERT-base-group-mention-detector-uk-manifestos

Model Details

Model Description