ReVeR

ReVeR is a rubric-based reward model made of two models that work in sequence:

1. A rubric generator reads an instruction and writes an evaluation rubric. Every item is labelled [Hard Rule] (an explicit requirement that is either met or not) or [Principle] (a quality that is met to a degree).
2. An agentic judge scores one response against every item of the rubric. It can call a Python tool for mechanical checks such as counting words or testing a format.

The score of a response is the number of Hard Rules it satisfies plus its mean Principle score divided by 101. To compare responses to the same instruction, generate one rubric, judge each response separately and compare the scores.

Checkpoints

All three are full fine-tunes of Qwen/Qwen3.5-4B.

Folder	Model	Training
rubric-sft/	Rubric generator	Supervised fine-tuning on ReVeR-Traces-Rubric
rubric-rl/	Rubric generator	rubric-sft trained further with reinforcement learning: a rubric is rewarded when the judge, using it, ranks the preferred response of a preference pair above the rejected one
judge-sft/	Judge	Supervised fine-tuning on ReVeR-Traces-Judge

The repository also holds what you need to run them:

File	Content
prompts/rubric_generator_prompt.txt	Prompt of the rubric generator, with an {instruction} and a {context} slot
prompts/judge_system_prompt.txt	System prompt of the judge
python_tool.py	The sandboxed Python executor behind the judge's tool
rever.py	Reference implementation of the whole pipeline

Quick start

Download the repository, serve one rubric generator and the judge with vLLM, then run the reference script.

pip install vllm openai
hf download OpenRubrics/ReVeR --local-dir ReVeR
cd ReVeR

# rubric generator (use rubric-sft for the SFT-only generator)
vllm serve ./rubric-rl --served-model-name rever-rubric --port 8001 \
  --dtype bfloat16 --max-model-len 32768 --reasoning-parser qwen3 --language-model-only

# judge: tool calling must be enabled
vllm serve ./judge-sft --served-model-name rever-judge --port 8002 \
  --dtype bfloat16 --max-model-len 32768 --reasoning-parser qwen3 --language-model-only \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Each server needs one GPU; a checkpoint is about 9 GB.

python rever.py --rubric-url http://localhost:8001/v1 --judge-url http://localhost:8002/v1 \
  --instruction "Write exactly three bullet points explaining why regular sleep matters. Each bullet must be under 12 words." \
  --response "- Sleep restores the body and repairs tissue overnight.
- It consolidates memories and sharpens next-day focus.
- Regular hours stabilise mood and the immune system." \
  --response "Sleep is very important for everyone. When you sleep regularly your body recovers and you feel better."

The script prints the rubric, then the score and the per-item judgments of each response:

Rubric
  1. The response contains exactly three bullet points. [Hard Rule]
  2. Each bullet point contains fewer than 12 words. [Hard Rule]
  3. Each bullet point explains a distinct reason why regular sleep matters (e.g., cognitive function, physical health, mood regulation). [Hard Rule]
  4. The reasons provided are factually accurate and supported by established scientific understanding of sleep. [Principle]
  5. The bullet points are coherently structured and self-contained rather than arbitrary word combinations or repetition. [Principle]

Response 1: score 3.9406
  1. score   1  Deterministic check found exactly three bullet points, satisfying the requirement.
  2. score   1  Word counts from deterministic check are 8, 7, and 8, all under 12. The requirement is satisfied.
  ...

Response 2: score 0.3218
  1. score   0  Deterministic check found no bullet markers ...
  ...

Both models sample, so the rubric and the scores vary from run to run.

From Python

from openai import OpenAI
import rever

if __name__ == "__main__":   # the judge's tool runs code in a subprocess
    rubric_client = OpenAI(base_url="http://localhost:8001/v1", api_key="EMPTY")
    judge_client = OpenAI(base_url="http://localhost:8002/v1", api_key="EMPTY")

    instruction = "Explain in one paragraph why the sky is blue, for a ten-year-old."
    responses = ["Sunlight is made of many colours ...", "The sky reflects the sea."]

    rubric = rever.generate_rubric(rubric_client, "rever-rubric", instruction)
    results = [rever.judge(judge_client, "rever-judge", instruction, r, rubric) for r in responses]
    best = max(range(len(responses)), key=lambda i: results[i]["score"])

generate_rubric returns a list of items, each with a rubric_id, a criterion and a kind (hard_rule or principle). judge returns the score and the judgments (one score and one explanation per item).

How to call the models yourself

rever.py is short and is the reference for every detail below. If you write your own client, keep to the following.

Decoding, for both models. Thinking enabled, temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5, at most 8,192 new tokens.

Rubric generator.

• Send one user message and no system message: the text of prompts/rubric_generator_prompt.txt with {instruction} replaced by the instruction and {context} by an empty string.
• The answer contains one <rubric>…</rubric> block with a numbered list. Every item ends with [Hard Rule] or [Principle].

Judge.

• System message: the text of prompts/judge_system_prompt.txt.
• User message: a JSON object with instruction, response and rubrics, where rubrics is a list of {"rubric_id": 1, "criterion": "… [Hard Rule]"} objects numbered from 1. Judge one response per conversation; the judge never sees another response.
• Offer one tool named python that takes a string argument code (the definition is PYTHON_TOOLS in rever.py). Execute the code with python_tool.py and return its JSON reply, which has the keys ok, stdout, result and error, as the tool message. The limits are 3 seconds, 4,000 characters of output and 512 MiB of memory.
• Allow at most three executed tool calls and four model calls per judgment, and make the last model call without tools so that it has to answer.
• Send the judge's own thinking back with its tool-call turn. After a tool call, the assistant turn you append to the conversation must carry the thinking of that turn in the field reasoning (vLLM 0.11 and later; older servers read reasoning_content, so rever.py sets both). Otherwise the chat template renders an empty <think> block and the judge writes its final answer without the reasoning it did before the tool call.
• The final answer ends with a <judge> block holding {"judgments": [{"rubric_id": 1, "explanation": "…", "score": 1}, …]}. A Hard Rule gets 0 or 1; a Principle gets an integer from 0 to 100.

Score. score = (number of Hard Rules scored 1) + (mean Principle score) / 101. The second term is below 1, so Principles only order responses that satisfy the same number of Hard Rules. Two responses with the same score are tied.

Rubric generator with Transformers

The rubric generator needs no tool, so it also runs directly with Transformers. generate has no presence penalty, so this snippet leaves it out.

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo, folder = "OpenRubrics/ReVeR", "rubric-rl"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=folder)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=folder, dtype=torch.bfloat16, device_map="auto")

template = open(hf_hub_download(repo, "prompts/rubric_generator_prompt.txt"), encoding="utf-8").read()
prompt = template.format(instruction="Explain in one paragraph why the sky is blue, for a ten-year-old.", context="")

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True, enable_thinking=True, return_tensors="pt", return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=8192, do_sample=True, temperature=1.0, top_p=0.95, top_k=20)
text = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# The thinking may mention the tag, so take the last <rubric> block.
rubric = text[text.rfind("<rubric>") + len("<rubric>"):text.rfind("</rubric>")].strip()

Versions

Tested with vLLM 0.18.0, Transformers 5.12.1, PyTorch 2.10.0 and the openai client 2.24.0. The checkpoints were saved with Transformers 5.12.1.

Citation

If you use these models, please cite our paper Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges:

@misc{liu2026rubricstaskverdictsevidence,
      title={Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges}, 
      author={Tianci Liu and Haoyu Wang},
      year={2026},
      eprint={2610.18766},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.18766}, 
}