README.md

---
base_model: Qwen/Qwen3.5-4B
library_name: transformers
pipeline_tag: text-generation
datasets:
- OpenRubrics/ReVeR-Traces-Rubric
- OpenRubrics/ReVeR-Traces-Judge
---

# ReVeR

ReVeR is a rubric-based reward model made of two models that work in sequence:

1. A **rubric generator** reads an instruction and writes an evaluation rubric. Every item is labelled `[Hard Rule]` (an explicit requirement that is either met or not) or `[Principle]` (a quality that is met to a degree).
2. An **agentic judge** scores one response against every item of the rubric. It can call a Python tool for mechanical checks such as counting words or testing a format.

The score of a response is the number of Hard Rules it satisfies plus its mean Principle score divided by 101. To compare responses to the same instruction, generate one rubric, judge each response separately and compare the scores.

## Checkpoints

All three are full fine-tunes of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B).

| Folder | Model | Training |
|---|---|---|
| `rubric-sft/` | Rubric generator | Supervised fine-tuning on [ReVeR-Traces-Rubric](https://huggingface.co/datasets/OpenRubrics/ReVeR-Traces-Rubric) |
| `rubric-rl/` | Rubric generator | `rubric-sft` trained further with reinforcement learning: a rubric is rewarded when the judge, using it, ranks the preferred response of a preference pair above the rejected one |
| `judge-sft/` | Judge | Supervised fine-tuning on [ReVeR-Traces-Judge](https://huggingface.co/datasets/OpenRubrics/ReVeR-Traces-Judge) |

The repository also holds what you need to run them:

| File | Content |
|---|---|
| `prompts/rubric_generator_prompt.txt` | Prompt of the rubric generator, with an `{instruction}` and a `{context}` slot |
| `prompts/judge_system_prompt.txt` | System prompt of the judge |
| `python_tool.py` | The sandboxed Python executor behind the judge's tool |
| `rever.py` | Reference implementation of the whole pipeline |

## Quick start

Download the repository, serve one rubric generator and the judge with [vLLM](https://github.com/vllm-project/vllm), then run the reference script.

```bash
pip install vllm openai
hf download OpenRubrics/ReVeR --local-dir ReVeR
cd ReVeR

# rubric generator (use rubric-sft for the SFT-only generator)
vllm serve ./rubric-rl --served-model-name rever-rubric --port 8001 \
  --dtype bfloat16 --max-model-len 32768 --reasoning-parser qwen3 --language-model-only

# judge: tool calling must be enabled
vllm serve ./judge-sft --served-model-name rever-judge --port 8002 \
  --dtype bfloat16 --max-model-len 32768 --reasoning-parser qwen3 --language-model-only \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

Each server needs one GPU; a checkpoint is about 9 GB.

```bash
python rever.py --rubric-url http://localhost:8001/v1 --judge-url http://localhost:8002/v1 \
  --instruction "Write exactly three bullet points explaining why regular sleep matters. Each bullet must be under 12 words." \
  --response "- Sleep restores the body and repairs tissue overnight.
- It consolidates memories and sharpens next-day focus.
- Regular hours stabilise mood and the immune system." \
  --response "Sleep is very important for everyone. When you sleep regularly your body recovers and you feel better."
```

The script prints the rubric, then the score and the per-item judgments of each response:

```text
Rubric
  1. The response contains exactly three bullet points. [Hard Rule]
  2. Each bullet point contains fewer than 12 words. [Hard Rule]
  3. Each bullet point explains a distinct reason why regular sleep matters (e.g., cognitive function, physical health, mood regulation). [Hard Rule]
  4. The reasons provided are factually accurate and supported by established scientific understanding of sleep. [Principle]
  5. The bullet points are coherently structured and self-contained rather than arbitrary word combinations or repetition. [Principle]

Response 1: score 3.9406
  1. score   1  Deterministic check found exactly three bullet points, satisfying the requirement.
  2. score   1  Word counts from deterministic check are 8, 7, and 8, all under 12. The requirement is satisfied.
  ...

Response 2: score 0.3218
  1. score   0  Deterministic check found no bullet markers ...
  ...
```

Both models sample, so the rubric and the scores vary from run to run.

### From Python

```python
from openai import OpenAI
import rever

if __name__ == "__main__":   # the judge's tool runs code in a subprocess
    rubric_client = OpenAI(base_url="http://localhost:8001/v1", api_key="EMPTY")
    judge_client = OpenAI(base_url="http://localhost:8002/v1", api_key="EMPTY")

    instruction = "Explain in one paragraph why the sky is blue, for a ten-year-old."
    responses = ["Sunlight is made of many colours ...", "The sky reflects the sea."]

    rubric = rever.generate_rubric(rubric_client, "rever-rubric", instruction)
    results = [rever.judge(judge_client, "rever-judge", instruction, r, rubric) for r in responses]
    best = max(range(len(responses)), key=lambda i: results[i]["score"])
```

`generate_rubric` returns a list of items, each with a `rubric_id`, a `criterion` and a `kind` (`hard_rule` or `principle`). `judge` returns the `score` and the `judgments` (one `score` and one `explanation` per item).

## How to call the models yourself

`rever.py` is short and is the reference for every detail below. If you write your own client, keep to the following.

**Decoding, for both models.** Thinking enabled, temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5, at most 8,192 new tokens.

**Rubric generator.**
- Send one user message and no system message: the text of `prompts/rubric_generator_prompt.txt` with `{instruction}` replaced by the instruction and `{context}` by an empty string.
- The answer contains one `<rubric>…</rubric>` block with a numbered list. Every item ends with `[Hard Rule]` or `[Principle]`.

**Judge.**
- System message: the text of `prompts/judge_system_prompt.txt`.
- User message: a JSON object with `instruction`, `response` and `rubrics`, where `rubrics` is a list of `{"rubric_id": 1, "criterion": "… [Hard Rule]"}` objects numbered from 1. Judge one response per conversation; the judge never sees another response.
- Offer one tool named `python` that takes a string argument `code` (the definition is `PYTHON_TOOLS` in `rever.py`). Execute the code with `python_tool.py` and return its JSON reply, which has the keys `ok`, `stdout`, `result` and `error`, as the tool message. The limits are 3 seconds, 4,000 characters of output and 512 MiB of memory.
- Allow at most three executed tool calls and four model calls per judgment, and make the last model call without tools so that it has to answer.
- **Send the judge's own thinking back with its tool-call turn.** After a tool call, the assistant turn you append to the conversation must carry the thinking of that turn in the field `reasoning` (vLLM 0.11 and later; older servers read `reasoning_content`, so `rever.py` sets both). Otherwise the chat template renders an empty `<think>` block and the judge writes its final answer without the reasoning it did before the tool call.
- The final answer ends with a `<judge>` block holding `{"judgments": [{"rubric_id": 1, "explanation": "…", "score": 1}, …]}`. A Hard Rule gets 0 or 1; a Principle gets an integer from 0 to 100.

**Score.** `score = (number of Hard Rules scored 1) + (mean Principle score) / 101`. The second term is below 1, so Principles only order responses that satisfy the same number of Hard Rules. Two responses with the same score are tied.

## Rubric generator with Transformers

The rubric generator needs no tool, so it also runs directly with Transformers. `generate` has no presence penalty, so this snippet leaves it out.

```python
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo, folder = "OpenRubrics/ReVeR", "rubric-rl"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=folder)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=folder, dtype=torch.bfloat16, device_map="auto")

template = open(hf_hub_download(repo, "prompts/rubric_generator_prompt.txt"), encoding="utf-8").read()
prompt = template.format(instruction="Explain in one paragraph why the sky is blue, for a ten-year-old.", context="")

inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True, enable_thinking=True, return_tensors="pt", return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=8192, do_sample=True, temperature=1.0, top_p=0.95, top_k=20)
text = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# The thinking may mention the tag, so take the last <rubric> block.
rubric = text[text.rfind("<rubric>") + len("<rubric>"):text.rfind("</rubric>")].strip()
```

## Versions

Tested with vLLM 0.18.0, Transformers 5.12.1, PyTorch 2.10.0 and the `openai` client 2.24.0. The checkpoints were saved with Transformers 5.12.1.

## Citation

If you use these models, please cite our paper [Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges](https://arxiv.org/abs/2610.18766):

```bibtex
@misc{liu2026rubricstaskverdictsevidence,
      title={Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges}, 
      author={Tianci Liu and Haoyu Wang},
      year={2026},
      eprint={2610.18766},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.18766}, 
}
```