EUBERT is the European Parliament's own BERT model, pre-trained from scratch on EU legislative texts rather than Wikipedia. This classifier adds a classification head fine-tuned on 64,000 labelled regulations.

Why this exists

The European Parliament published an official EuroVoc classifier on HuggingFace. It ships with mismatched vocabulary and weight dimensions and does not produce usable output. A community report confirmed the issue in 2024. No fix was published. This model is a working replacement.

Performance

Metric	Score
F1 micro	0.891
F1 macro	0.793
Optimal threshold	0.35

Evaluated on 890 held-out EU regulations published between September 2025 and March 2026. Ground truth labels were assigned by professional librarians at the EU Publications Office.

Context: Series Results

Method	F1 (micro)	Cost
bert-base (fine-tuned)	0.900	€10
Llama 3.1 8B (QLoRA)	0.892	€83
EUBERT (this model)	0.891	free
TF-IDF + Logistic Regression	0.799	free
DeepSeek-R1-70B (zero-shot)	0.562	~€12

Full write-up: Pimp My LM: A Fine-Tuning Tale of Bling and Basic

The 21 Labels

AGRI-FOODSTUFFS, AGRICULTURE FORESTRY AND FISHERIES,
BUSINESS AND COMPETITION, ECONOMICS, EDUCATION AND COMMUNICATIONS,
EMPLOYMENT AND WORKING CONDITIONS, ENERGY, ENVIRONMENT,
EUROPEAN UNION, FINANCE, GEOGRAPHY, INDUSTRY,
INTERNATIONAL ORGANISATIONS, INTERNATIONAL RELATIONS, LAW,
POLITICS, PRODUCTION TECHNOLOGY AND RESEARCH, SCIENCE,
SOCIAL QUESTIONS, TRADE, TRANSPORT

Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "jngb-labs/eurovoc-eubert"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

text = "Regulation establishing a carbon border adjustment mechanism..."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.sigmoid(logits)

LABELS = [
    "AGRI-FOODSTUFFS", "AGRICULTURE, FORESTRY AND FISHERIES",
    "BUSINESS AND COMPETITION", "ECONOMICS", "EDUCATION AND COMMUNICATIONS",
    "EMPLOYMENT AND WORKING CONDITIONS", "ENERGY", "ENVIRONMENT",
    "EUROPEAN UNION", "FINANCE", "GEOGRAPHY", "INDUSTRY",
    "INTERNATIONAL ORGANISATIONS", "INTERNATIONAL RELATIONS", "LAW",
    "POLITICS", "PRODUCTION, TECHNOLOGY AND RESEARCH", "SCIENCE",
    "SOCIAL QUESTIONS", "TRADE", "TRANSPORT"
]

threshold = 0.35
predictions = [LABELS[i] for i, p in enumerate(probs[0]) if p > threshold]
print(predictions)

Training Details

Base model: EuropeanParliament/EUBERT (110M parameters, pre-trained on EU legislative texts)
Training data: 63,918 EU regulations (preambles), sourced from EUR-Lex via CELLAR API
Test data: 890 held-out regulations, labels assigned by EU Publications Office librarians
Architecture: BERT pooler → Dropout(0.1) → Linear(768, 21) → Sigmoid
Loss: BCEWithLogitsLoss
Epochs: 3
Hardware: Nvidia T4 (Google Colab)
Training time: 159 minutes
Cost: Free (Google Colab free tier)

Limitations

Trained on English-language preambles only (EU legislation is published in 24 languages)
Multi-label threshold (0.35) was optimised on the test set; may need adjustment for other corpora
Classification granularity is limited to 21 top-level domains; finer EuroVoc concepts are not predicted

License

Apache 2.0

Author

Jakob Neugebauer