Does a model's position bias on retrieval tasks predict its position bias on reasoning, summarization, or translation tasks? Or are these independent dimensions of model behavior?
| Type | Pattern | Description |
|---|---|---|
| Primacy | High → Low | Best at start |
| Recency | Low → High | Best at end |
| U-Shaped | High → Low → High | Classic LITM |
| Middle-Sag | Flat → Low → Flat | Only middle suffers |
| Flat | ~Constant | No position effect |
| Inverted-U | Low → High → Low | Middle best (rare) |
PBI = (acc_start + acc_end) / 2 - acc_middle| Task | Cognitive Demand | Expected Bias |
|---|---|---|
| KV Retrieval | Simple lookup | Strong U-shape |
| Needle in Haystack | Text search | Strong U-shape |
| Fact-Dependent Reasoning | Reasoning + retrieval | Moderate U-shape |
| Summarization | Comprehension + compression | Weak/moderate |
| Translation | Understanding + generation | Task-dependent |
1pip install -r requirements.txt
2
3# Single model, all tasks
4python run_all.py --models Qwen/Qwen2.5-1.5B-Instruct
5
6# Multiple models
7python run_all.py --models Qwen/Qwen2.5-1.5B-Instruct meta-llama/Llama-3.2-1B-Instruct
8
9# Specific tasks only
10python run_all.py --tasks kv_retrieval reasoning --num-examples 50"PBI on retrieval tasks correlates poorly with PBI on reasoning tasks (r=0.12), suggesting position bias is not a unified model property but a task-dependent phenomenon."
results/
├── taxonomy_Qwen_Qwen2.5-1.5B-Instruct.json
├── taxonomy_meta-llama_Llama-3.2-1B-Instruct.json
└── ...1@software{position_bias_taxonomy,
2 title={Position Bias Taxonomy: A Cross-Task Framework for Long-Context Evaluation},
3 author={abhshkp},
4 year={2026},
5 url={https://huggingface.co/abhshkp/position-bias-taxonomy}
6}