This model is a fine-tuned version of FacebookAI/xlm-roberta-base on national parliamentary speeches in Europe, using ParlSpeech dataset.
It achieves the following results on the evaluation set:
Loss: 0.1751
Accuracy: 0.9442
F1: 0.9442
Precision: 0.9455
Recall: 0.9442
Model description
EUroBerta-xlm is a fine-tuned multilingual text classification model designed to identify whether a sentence discusses the European Union (EU).
Given any sentence as input, the model returns a binary label: EU (the sentence references the European Union) or No EU (the sentence does not discuss the EU).
This model is the first of three steps in a full EU stance classification pipeline:
EUroBerta-xlm → identifies EU-relevant sentences (EU / No EU) ← this model
EuOppostion_Classifier → classifies EU sentences as Opposition / Non-Opposition
EuSupport_Classifier → classifies Non-Opposition sentences as Neutral / Support
The final model achieves an F1 score of 0.94 on the held-out evaluation set, with high and balanced precision and recall.
The confusion matrix below reflects the model's strong ability to distinguish EU-relevant from non-EU sentences, including after correction for false positives introduced by geographic or sporting references to "Europe".
Intended uses & limitations
The model was developed to support large-scale computational text analysis of political discourse, enabling researchers to efficiently filter and categorise party manifestos, parliamentary speeches, and other political documents across multiple European languages.
Altought the model can be use for any EU-related task, it was specifically designed for assisting annotation pipelines that require downstream classification of EU attitudes (pro-EU, Eurosceptic, neutral).
Training and evaluation data
The training data was constructed through two rounds of annotation, resulting in a final corpus of 5,950 sentences drawn from plenary speeches from 7 European national parliaments (Austria, Denmark, Germany,
the Netherlands, Spain, Sweden, and the United Kingdom). These speeches are sourced from the ParlSpeech dataset, covering a 20-year period from 1999 to 2019 (Rauh & Schwalbach, 2020)
The source data originally used five categories of EU stance:
No EU — sentence does not reference the EU
Pro-EU — sentence expresses a favourable view of the EU
Neutral — sentence mentions the EU without taking a stance
Soft Eurosceptic — sentence expresses mild scepticism toward the EU
Hard Eurosceptic — sentence expresses strong opposition to the EU
These five categories were consolidated into a binary format for this model: sentences labelled No EU were retained as one class, while Pro-EU, Neutral, Soft Eurosceptic, and Hard Eurosceptic were merged into a single EU class.
After an initial training run, the model showed a systematic tendency to overclassify sentences as EU-related, particularly sentences referencing individual countries (including non-EU states) or generic mentions of "Europe" (e.g., UEFA, European geography). To address this, a second round of targeted annotation was conducted:
250 additional sentences per country were selected for re-annotation.
Radical right parties were intentionally oversampled (~50 sentences per country) to improve coverage of oppositional discourse, which differs structurally from mainstream party language.
The model was then retrained on the full combined corpus of 5,950 sentences.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 2e-05
train_batch_size: 24
eval_batch_size: 24
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
