SWT-bench is a dataset that tests systems’ ability to reproduce GitHub issues automatically. The dataset collects 1978 test Issue-Pull Request pairs from 11 popular Python GitHub projects. Evaluation is performed by unit test verification using pre- and post-PR behavior of the test suite with and without the model proposed tests.
A public leaderboard for performance on SWT-bench is hosted at swtbench.com
The dataset is released as part of… See the full description on the dataset page:
https://huggingface.co/datasets/eth-sri/SWT-bench_bm25_27k_zsb.
