SWT-bench Verified is subset of SWT-bench, a dataset that tests systems’ ability to reproduce GitHub issues automatically. The dataset collects 433 test Issue-Pull Request pairs from 11 popular Python GitHub projects. Evaluation is performed by unit test verification using pre- and post-PR behavior of the test suite with and without the model proposed tests.
A public leaderboard for performance on SWT-bench is hosted at swtbench.com… See the full description on the dataset page:
https://huggingface.co/datasets/eth-sri/SWT-bench_Verified_bm25_27k_zsb.
