SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
SWE-bench proposes a new task: issue resolution provided a full repository and GitHub issue. The leaderboard can be found at
www.swebench.com
