arxiv:2610.07557

CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

Published on Oct 6

· Submitted by

Hang He on Oct 7

Humanlaya Data Lab

Upvote

Authors:

Hang He ,

Abstract

Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection and rarely assess whether an agent can develop a working checker in a repository from start to finish. We introduce CheckerBench, an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. We further introduce CheckerLab, a common evaluation framework that independently rebuilds submitted checkers and measures vulnerable-fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model-harness configurations and three independent repeats per configuration, mean Pass@1 is 32.30%, while the best reaches 45.33%. These results show that reliable, reusable checker development remains challenging for current coding agents.

View arXiv page View PDF GitHub 1 Add to collection

Community

Benchanything

Paper author Paper submitter about 3 hours ago

We introduce CheckerBench, an executable benchmark for evaluating whether coding agents can turn real-world vulnerabilities into reusable static-analysis checkers.

CheckerBench contains 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Agents must inspect repositories, implement analyzer-specific logic, and iteratively refine their checkers through compilation and analysis feedback. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold.

We also introduce CheckerLab, a unified evaluation framework that independently rebuilds submitted checkers and measures vulnerable–fixed diagnostic contrast, patch localization, false positives, and tool use.

Across 21 model–harness configurations and three independent repeats per configuration, mean Pass@1 is 32.30%, while the best configuration reaches 45.33%. These results reveal substantial room for progress in long-horizon agentic checker development.

The dataset and runtime assets are publicly available to support reproducible research on coding agents, automated static analysis, and software security.

🤗 Dataset and runtime assets