Papers
arxiv:2610.08402

VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents

Published on Oct 23
Authors:
,
,
,
,
,
,

Abstract

A novel credit assignment framework is proposed to improve multi-turn LLM agent performance by coordinating value estimation across both temporal sequences and individual token decisions using a shared lightweight critic.

Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two related credit-assignment questions: which responses helped achieve the outcome, and which generation decisions mattered within each response? Existing methods typically focus on only one level: turn-level methods evaluate complete responses but do not distinguish the decisions within them; token-level methods can propagate feedback across turns but do not explicitly model credit for each response. These complementary limitations motivate learning credit at both levels and coordinating it in a single policy update. We introduce VETTA, a credit assignment method that jointly learns turn- and token-level values through separate heads on a shared lightweight critic. VETTA computes advantages along both temporal sequences and combines each turn advantage with a within-response-centered token residual for PPO updates. Furthermore, to reduce value-learning cost, the critic retains only early Transformer blocks from the pretrained checkpoint used to initialize the actor. Experiments on four agent benchmarks demonstrate VETTA's strong performance. With Qwen2.5-7B-Instruct, VETTA achieves success rates of 95.5% on ALFWorld, 76.0% on WebShop, and 52.2% on ScienceWorld. With Qwen2.5-VL-3B-Instruct, it reaches 84.9% and 80.7% success on Sokoban under sparse and dense rewards, respectively. Critic-depth comparisons further show strong task performance with substantially lower critic-side computation. These results demonstrate that coordinating turn- and token-level credit with a lightweight shared critic improves agent performance while keeping value estimation efficient. Code is available at https://github.com/Jiaju-Chen/VETTA-official.

Community

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2610.08402
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.08402 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.08402 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.08402 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.