Papers
arxiv:2610.24649

Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

Published on Oct 17
· Submitted by
Naman Vats
on Oct 21
Authors:

Abstract

A typed decision model consulted at fixed points increased agent costs without improving success rates, despite better ranking of finish requests, due to poor calibration.

Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse. In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments. On 21 held-out tasks (315 trials), Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at an identical pass rate (73.0% vs. 73.0%). Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct. No controller ever chose to abandon a run. Informative decision signals did not become useful interventions; we discuss what that would require.

Community

Paper author Paper submitter

Can a specialized decision model make AI agents more efficient, or does it just add overhead?

We investigate whether delegating decisions like when to stop, verify, or change approach to a specialized decision model improves agent performance.

Using Terminal-Bench 2.1, we compare five controller strategies within the same agent loop: no controller, deterministic rules, a small LLM, Jev 1.13, and a placebo controller.

Key findings:

  • 19% higher cost per attempt with Jev, without improving task success (73% vs. 73%).
  • Jev showed stronger offline finish-decision discrimination (AUROC 0.78) than rules (0.62) or a small LLM (0.60).
  • Despite better offline decision signals, calibration challenges contributed to unnecessary verification and additional overhead.

Our findings suggest that better decision-making in isolation does not necessarily translate into better end-to-end agent performance. Controller calibration, intervention timing, and cost all matter.

The study includes 315 trials across 21 held-out tasks, with code released for reproducibility.

Paper: https://arxiv.org/abs/2610.24649

Curious whether others have observed similar trade-offs when introducing specialized decision models into agent workflows.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2610.24649
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.24649 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.24649 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.24649 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.