VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Abstract
Agents generate executable verifiers to validate outputs, but these verifiers often wrongly reject valid solutions due to hidden assumptions, prompting recommendations to test acceptance of diverse correct answers.
Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.
Community
Introducing verifier-suite: a benchmark and a harness for the verifiers that agents write.
Self-improving agents should create their own tasks, solve them, and write the verifiers that hand out rewards. Verifier-suite contains two parts:
๐ VerifierBench: measures how good agent-generated verifiers are. The agent gets the same input it would get when solving the task, only the task instruction and the environment. It writes an executable verifier, which is then scored on a variety of hidden test cases.
- 222 tasks from ACEBench, ฯยฒ-bench, AppWorld and more, covering tool-call, environment-state and code-change verification.
- 4,993 hidden test cases: 1,344 valid variants, 2,629 common mistakes and 798 attacks. Every source verifier and test suite passes a rigorous audit first.
The finding: 19 models in the same harness score only 10.6 to 62.1. The verifiers they write are too strict: GPT-6-astraโs reject 33% of correct answers!
๐ ๏ธ Gavel: a harness built for generating verifiers. From the task instruction and environment alone, it generates the matching verifier.
- The ERH system: an evidence header (E), a domain rule library (R) and authoring principles (H) keep the verifier checking only the evidence that truly matters for the task.
- Works on Harbor-format tasks: one command generates a verifier for each task, writes it back and validates it.
GPT-6-astra + Gavel scores 68.3 on VerifierBench, #1 among all configurations!
๐ป Code: https://github.com/DwanZhang-AI/verifier-suite
๐ Paper: https://arxiv.org/abs/2610.28679
๐ค Data: https://huggingface.co/datasets/Catherine1213/VerifierBench
๐ Project page: https://verifierbench.github.io/
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper