Dynamic, execution-based benchmark for secure code generation by LLMs. Every generated program is compiled/interpreted and run against two independent unit-test suites — a functional suite and a security suite (the latter is designed to fail when the target CWE vulnerability is present). Covers 232 CWEs across C, C++, Java, and Python.

Paper: Osebe et al., AutoSUIT Bench — Automated Security UnIt Test Benchmark for LLM Coding, Findings of… See the full description on the dataset page: https://huggingface.co/datasets/amazon/AutoSUIT.