When documents contain contradictory evidence at different positions, does the classic "Lost in the Middle" U-shape still hold? Or do models exhibit a "first-answer dominance" bias instead?
| # | Experiment | Setup | What It Measures |
|---|---|---|---|
| 1 | Two-Document Contradiction | Fact A at position X, Fact B at position Y | Which position wins? |
| 2 | Three-Document Contradiction | Three variants at start/middle/end | Multi-way position dominance |
| 3 | Temporal Authority | Same fact with timestamps (2020 vs 2024) | Recency bias under contradiction |
1pip install -r requirements.txt
2
3# Run all experiments
4python run_all.py --model Qwen/Qwen2.5-1.5B-Instruct --output ./results
5
6# Run specific experiment
7python run_all.py --experiments 2doc --num-examples 100| Hypothesis | Prediction |
|---|---|
| Classic LITM | Middle-position facts are least likely to be selected |
| First-Answer Dominance | Start-position facts dominate regardless of correctness |
| Recency Bias | Newer timestamps override position effects |
| Middle-Vanishing | When contradictions are close (both middle), accuracy drops |
1@software{contradiction_position_bias,
2 title={Contradictory Evidence Position Bias Benchmark},
3 author={abhshkp},
4 year={2026},
5 url={https://huggingface.co/abhshkp/contradiction-position-bias}
6}