yeseul0-0/

docred-stage3-atlop-fine-tuning-v4

Evaluation Results (Stage 3 — GAIN-lite, v4)

Dataset Statistics

• Documents: 998
• Predicted Triples: 13,536
• Gold Triples: 12,275
• Correct Predictions: 6,842

Main Metrics

• Micro F1: 53.02%
  • Precision: 50.55%
  • Recall: 55.74%
• Ign F1: 50.10%
  • Precision: 47.70%
  • Recall: 52.75%
• Evidence F1: 42.15%
  • Precision: 40.95%
  • Recall: 43.41%

Detailed Analysis

• Intra-sentence F1: 62.50%
  • Precision: 64.54%
  • Recall: 60.59%
• Inter-sentence F1: 43.43%
  • Precision: 38.43%
  • Recall: 49.93%

Interpretation

• Predicted: 13,536 > Gold: 12,275

• Over-predicting relations
• Producing many false positives

• Precision drops sharply (50.55%)
• Recall is relatively high (55.74%)

Key Issue

• Too many relations are accepted
• Adaptive threshold (ATLOP) is likely too permissive
• Leads to precision collapse

Additional Observations

• Inter F1 (43.43%) is significantly lower than later versions (~52%)
• Indicates weaker multi-hop reasoning performance
• Suggests that graph reasoning was not effectively utilized or not yet stabilized

Key Insight

Increasing recall without controlling precision leads to severe performance degradation in multi-label relation extraction.

Output Files

• Predictions: results/stage3/v4/predictions.json
• Submission File: results/stage3/v4/result.json