haoranhe/ROVER-countdown-3B

Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards

This repository contains the model presented in the paper Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards.

ROVER (Random Policy Valuation for Diverse Reasoning) is a minimalist yet highly effective Reinforcement Learning (RL) method for Large Language Model (LLM) reasoning. It achieves superior optimality and diversity by evaluating uniform-policy Q-values, bypassing complex policy iteration loops typically found in methods like PPO and GRPO. This approach is particularly effective for math reasoning tasks, preserving diversity throughout training for sustained exploration of multiple valid pathways.

• Paper: Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• Code: https://github.com/tinnerhrhe/ROVER

Main Results and Features

*Figure 1: (a) ROVER achieves superior performances in terms of both pass@1 and pass@256 (trained on Qwen3-8B-Base averaged over AIME24, AIME24 and HMMT25 tasks). (b) Illustrative example demonstrating that ROVER achieves high-quality solutions with a lightweight procedure (see Table below for details) while maintaining diversity. (c) ROVER achieves higher diversity.*

ROVER needs minimal GPU memory and computation cost, leaving more space for the KV cache. This allows ROVER to run on smaller memory setups and speeds up training:

Method	Memory Usage of Model Parameters
ROVER (Ours)	Low        (actor model ONLY!😊)
GRPO	Medium (actor + reference model)
PPO	High       (actor + reference + critic model)

For installation, training, and evaluation instructions, please refer to the GitHub repository.

Citation

If you find the project useful, please consider citing our paper:

@article{he2025randompolicyvaluation,
      title={Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards}, 
      author={Haoran He and Yuxiao Ye and Qingpeng Cai and Chen Hu and Binxing Jiao and Daxin Jiang and Ling Pan},
      journal={arXiv preprint arXiv:2509.24981},
      year={2025}
}