Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges
Abstract
Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.
Community
Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing
methods often derive criteria from observed response candidates, introducing response-induced bias that
may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying
on reading alone, and commonly used pairwise judges provide only relative preferences, complicating
their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules
and graded principles by reasoning about the instruction alone and evaluates responses independently
through tool-augmented judging. Both components are trained through label-blind distillation, in
which preference labels only filter a teacher’s outputs, yielding ReVeR-Traces, a large-scale dataset of
rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise
rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time
scaling and downstream policy optimization. We further find that generator RL improves preference
accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate
into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and
downstream utility are distinct properties, shedding light on future RM research.
Get this paper in your agent:
hf papers read 2610.18766 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 2
OpenRubrics/ReVeR-Traces-Judge
OpenRubrics/ReVeR-Traces-Rubric
Spaces citing this paper 0
No Space linking this paper