yeseul0-0/

docred-stage3-atlop-fine-tuning-v3

Evaluation Results (Stage 3 — GAIN-lite, v3)

Dataset Statistics

• Documents: 998
• Predicted Triples: 13,870
• Gold Triples: 12,275
• Correct Predictions: 6,862

Main Metrics

• Micro F1: 52.49%
  • Precision: 49.47%
  • Recall: 55.90%
• Ign F1: 49.54%
  • Precision: 46.58%
  • Recall: 52.91%
• Evidence F1: 42.20%
  • Precision: 40.94%
  • Recall: 43.55%

Detailed Analysis

• Intra-sentence F1: 62.49%
  • Precision: 64.35%
  • Recall: 60.74%
• Inter-sentence F1: 42.59%
  • Precision: 37.03%
  • Recall: 50.11%

Interpretation

• Predicted: 13,870 >> Gold: 12,275

• Severe over-prediction
• Large number of false positives

• Precision collapses (49.47%)
• Recall remains relatively high (55.90%)

Key Issues

1. Threshold Miscalibration
   • Too many relations are accepted
   • Adaptive threshold is not properly tuned
2. Precision–Recall Imbalance
   • Model favors recall over precision
   • Leads to degraded overall F1
3. Weak Multi-hop Reasoning
   • Inter F1 (42.59%) is significantly low
   • Graph reasoning not yet effective

Key Insight

Without proper threshold control, graph-based models tend to over-predict relations, leading to severe precision degradation.

Output Files

• Predictions: results/stage3/v3/predictions.json
• Submission File: results/stage3/v3/result.json