This is a distribution-robust variant of T-EBERT, trained on artificially balanced data (50% IR sense, 50% colloquial) to learn semantic features without exploiting class frequency priors.

Key Differences from Standard T-EBERT

Aspect	T-EBERT (Standard)	T-EBERT-Balanced
Training distribution	70/30 (natural)	50/50 (balanced)
F1 Score	0.922	0.822
Optimization goal	Peak performance	Distribution robustness
Feature reliance	Content + distribution	Semantic only
Best for	Production (known distribution)	Research (unknown distribution)

Performance

F1 Score: 0.822 on balanced test set
Training distribution: 50% IR sense, 50% colloquial (artificially balanced)

Term	F1 Score
Kooperation	0.934
Entspannung	0.916
Intervention	0.901
Integration	0.895
Norm	0.699
Regime	0.390

Intended Use

Primary use cases:

Research on distribution robustness in NLP
Corpora with unknown or varying class distributions
Domain transfer scenarios
Applications requiring symmetric per-class performance
Benchmarking against distribution-agnostic baselines

Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "pdjohn/T-EBERT-term-sense-balanced"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Classify a sentence
sentence = "Die internationale Norm verbietet den Einsatz von Gewalt."
inputs = tokenizer(sentence, return_tensors="pt", truncation=True, padding=True)

with torch.no_grad():
    outputs = model(**inputs)
    prediction = torch.argmax(outputs.logits, dim=1).item()

# Interpretation
labels = {0: "Colloquial", 1: "IR Sense"}
print(f"Prediction: {labels[prediction]}")

Training Details

Training Data

Corpus: Same as T-EBERT, but with balanced sampling
Size: 456 sentences (training) + 114 sentences (test)
Distribution: 50% IR sense, 50% colloquial (artificially balanced via undersampling)
Terms: Norm, Kooperation, Regime, Integration, Intervention, Entspannung

Training Procedure

Base model: EuroBERT-610M
Fine-tuning method: LoRA (Low-Rank Adaptation)
- Rank (r): 8
- Alpha: 16
- Target modules: q_proj, k_proj, v_proj
- Bias training: Enabled (bias="all") - critical for balanced learning
- Dropout: 0.05
Epochs: 5
Batch size: 8
Learning rate: 1e-4

Limitations

Lower peak performance: 0.82 vs 0.92 F1
Smaller training set: 456 vs 934 samples (undersampling)
see T-EBERT for best performance

Acknowledgements

This work was conducted as part of the Tracing International Institutions and Behavior (TIIB) project at TU Darmstadt.

Affiliation: Chair of Transnational Governance, Department of Political Science, Technische Universität Darmstadt

We thank the TIIB project team for their support and the manual annotation of the training corpus.

pdjohn/EuroBERT-610m-term-sense-balanced

T-EBERT: German IR Term Sense Disambiguation

Model Description