Papers
arxiv:2609.30226

PoEM: Predicting RL Outcomes from Existing Policies

Published on Nov 8
Authors:
,
,

Abstract

A framework predicts reinforcement learning post-trained policies for new reward functions by combining existing policies, avoiding additional RL training.

Foundation models are post-trained with reinforcement learning (RL) to maximize specific rewards, such as helpfulness, correctness, or instruction following. This post-training process is computationally intensive, sometimes unstable, and has to be run from scratch every time the reward model changes or when we want to combine multiple rewards. We hence ask: given a new reward function, is it possible to predict the policy RL would produce without actually training on that reward? We answer this in the affirmative by introducing PoEM, a framework for predicting RL outcomes using models previously trained on other rewards. First, we build on the fact that if the new reward function can be written as a linear combination of existing ones, then the optimal policy can be expressed as a combination of the existing policies in log-probability space. Surprisingly, even in cases where the rewards are not linearly connected, we observe that policies learned through RL often span an approximately low-rank subspace across rewards. To our benefit, the weights for this combination can be estimated using only the reward or basis policy outputs on the samples. We turn these observations into an algorithm that takes post-trained models and a target reward function, and approximates the target RL policy without any additional RL training. We experimentally evaluate our approach across programmatic and learned rewards for open-ended generation, with additional experiments on math reasoning, tool calling, and image generation.

Community

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2609.30226
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2609.30226 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2609.30226 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2609.30226 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.