Views
No views yet
| Finding | Result |
|---|---|
| H1 — Category error | Standard pass@k treats test-case counts as independent sample counts, producing different scores for agents with identical true reliability |
| H2 — Score inflation | Current pass@5 ≈ 0.96–0.97 collapses to reliability@5 ≈ 0.00–0.12 when the category error is corrected |
| H3 — Proxy validity | Single-rollout proxy score has low Spearman correlation with reliability@k; ≥5 rollouts needed for reliable ranking |
| H4 — Security gap | Security-adjusted reliability@5 is meaningfully lower than reliability@5; leaderboard rank flips when security is accounted for |
reliability@k — correct operationalization of Chen et al. (2021) pass@k,
using per-(task, agent) rollout counts and binary execution successsecurity_adjusted_reliability@k — reliability@k counting only rollouts
that are both correct and produce code with no insecure patterns (eval,
exec, os.system, yaml.load without Loader, pickle.loads)| Agent | Provider | Model |
|---|---|---|
anote-code | Anthropic | claude-sonnet-4-6 (Anote system prompt) |
claude-code | Anthropic | claude-sonnet-4-6 |
codex | OpenAI | gpt-4o |
@misc{codebench2026,
title = {CodeBench: Measuring Reliable Code Generation},
author = {Anote AI},
year = {2026}
}