SWE-bench-Live/

OS-bench

	Dataset Format




	

field type

instance_id str

docker_image str

rebuild_cmds list[str]

test_cmds list[str]

print_cmds list[str]

patch str

test_patch Optional[str], by default disabled (null)

log_parser str

PASS_TO_PASS list[str]

FAIL_TO_PASS list[str]

evaluation code is hosted on GitHub windows2linux split uses Linux docker images which can be run on Linux docker and Docker Desktop (windows/macos system). linux2windows split uses Windows docker… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench-Live/OS-bench.