C-EBERT: Multi-Task Causal Extraction

Model Details

• Multi-Task Architecture: Extends EuroBERT-610m with token and relation classification heads, using a joint loss function \(\mathcal{L}{\text{total}} = \alpha \mathcal{L}{\text{token}} + \beta \mathcal{L}_{\text{relation}}\): | Task | Output Type | Labels / Classes | | :--- | :--- | :--- | | 1. Token Classification | Sequence Labeling (BIO) | 5 Span Labels (O, B-INDICATOR, I-INDICATOR, B-ENTITY, I-ENTITY) | | 2. Relation Classification | Sentence-Pair Classification | 14 Relation Labels (e.g., MONO_POS_CAUSE, DIST_NEG_EFFECT) |
• Dataset: 4,540 manually annotated relations (see the excerpt Bundestag Causal Attribution).

Usage

Technical Highlights

• Custom Input Formatting: Utilizes a custom <|parallel_sep|> token to handle sentence-pair classification for relation extraction.
• Engineered for Scale: Proven performance on large-scale datasets, processing 22M sentences and generating 1.6M unique causal triples.
• Imbalance Mitigation: Implements dynamic class weighting via normalized inverse frequencies to handle the long-tail distribution of 14 distinct causal relation types.

Training

• Base model: EuroBERT-610m
• Training Parameters:
  • Epochs: 8
  • Learning Rate: 1e-4
  • Batch size: 32
  • PEFT/LoRA: Enabled with r = 16

Evaluation