CanItEdit is a benchmark for evaluating LLMs on instructional code editing, the task of updating a program given a natural language instruction. The benchmark contains 105 hand-crafted Python programs with before and after code blocks, two types of natural language instructions (descriptive and lazy), and a hidden test suite.
The dataset’s dual natural language instructions test model… See the full description on the dataset page:
https://huggingface.co/datasets/nuprl/CanItEdit.
