Bellman Policy Optimization
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.
Community
Bellman Policy Optimization (BPO) is a critic-free policy optimization method derived from Policy Mirror Descent (PMD) and the Bellman equations. For autoregressive generation with terminal rewards, we use the Bellman equations to reformulate PMD into a trajectory-level objective that avoids estimating intermediate state values. We prove that the reformulated objective has the same unique optimal solution as the original PMD objective. By approximating the trajectory-level objective, we derive the BPO loss, which replaces the importance-sampling ratio in Group-Relative Policy Optimization (GRPO) with a smoothed ratio of complementary token probabilities.
BPO and SC share the identical core gradient, obtained from different derivations (see Figure 1 below)
- Bellman Policy Optimization (BPO): arXiv:2609.15987, Sep. 14, 2026
- Score Centering Stabilizes Off-policy Reinforcement Learning (SC): arXiv:2609.20807, Sep. 17, 2026
The shared core gradient is shown in Figure 1 below.
- BPO derives it by Policy Mirror Descent and Bellman Equations.
- SC derives it by removing gradient drift.
As in Figure 2 below, starting from this shared gradient,
- BPO uses a binary approximation to $D_{\mathrm{KL}}(\mu|\pi)$ and additive smoothing
- SC uses top-k approximation to $D_{\mathrm{KL}}(\mu|\pi)$
Get this paper in your agent:
hf papers read 2609.15987 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper