Views
No views yet
SWE-Compass/
├─ data/
│ ├─ test.jsonl # main evaluation set (~2,000 instances)
│ ├─ dev.jsonl # optional validation split
│ └─ train.jsonl # optional training data
├─ scripts/
│ ├─ setup_env.sh # environment setup (dependency installation)
│ ├─ run_instance.py # run one instance end-to-end
│ └─ eval_aggregate.py # aggregate evaluation metrics
└─ README.md1{
2 "instance_id": "compass_01234",
3 "task_type": "bug_fixing",
4 "scenario": "mono_repo_ci",
5 "language": "python",
6 "difficulty": "medium",
7 "source": {
8 "repo": "owner/project",
9 "commit": "abcdef123456",
10 "issue_or_pr": "PR#13091",
11 "gh_url": "https://github.com/owner/project/pull/13091"
12 },
13 "instruction": "Fix failing test in module X caused by Y...",
14 "context_files": ["path/to/file1.py", "path/to/file2.py"],
15 "tools_available": ["git", "pytest", "bash"],
16 "evaluation": {
17 "setup_cmds": ["pip install -e .", "pytest -q"],
18 "test_cmd": "pytest -q",
19 "timeout_sec": 1800
20 },
21 "reference_patch": "diff --git a/... b/...",
22 "verified": true
23}1from datasets import load_dataset
2
3dataset = load_dataset("Kwaipilot/SWE-Compass", split="test")
4print(len(dataset), dataset[0].keys())
5Run Local Evaluation1bash scripts/setup_env.sh
2python scripts/run_instance.py --data data/test.jsonl --instance_id compass_01234
3python scripts/eval_aggregate.py --data data/test.jsonl --runs ./runs/your_model_outputs| Category | Metric | Description |
|---|---|---|
| Main | Solved@1 / Solved@k | Fraction of tasks solved within k attempts |
| Process | Tool Calls / Latency | Efficiency and reasoning stability |
| Failure Types | Build Error / Test Fail / Timeout | Root-cause classification |
1@article{xu2025swecompass,
2 title = {SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models},
3 author = {Jingxuan Xu and others},
4 journal = {arXiv preprint arXiv:2511.05459},
5 year = {2025}
6}