Aletheia: What Makes RLVR For Code Verifiers Tick?

Multi-domain thinking verifiers trained via Reinforcement Learning from Verifiable Rewards (RLVR) are a prominent fixture of the Large Language Model (LLM) post-training pipeline, owing to their ability to robustly rate and rerank model outputs. However, the adoption of such verifiers towards code generation has been comparatively sparse, with execution feedback constituting the dominant signal. Nonetheless, code verifiers remain valuable toward judging model outputs in scenarios where execution feedback is hard to obtain and are a potentially powerful addition to the code generation post-training toolbox. To this end, we create and open-source Aletheia, a controlled testbed that enables execution-grounded evaluation of code verifiers' robustness across disparate policy models and covariate shifts. We examine components of the RLVR-based verifier training recipe widely credited for its success: (1) intermediate thinking traces, (2) learning from negative samples, and (3) on-policy training. While experiments show the optimality of RLVR, we uncover important opportunities to simplify the recipe. Particularly, despite code verification exhibiting positive training- and inference-time scaling, on-policy learning stands out as the key component at small verifier sizes, and thinking-based training emerges as the most important component at larger scales.

📦 Model Zoo

Fine-tuned code verifiers at 1.5B, 7B, and 14B scales using several algorithms

Algorithm	Thinking	Negatives	Online	Description
GRPO-Think	✅	✅	✅	Standard GRPO-style approach to training verifiers.
GRPO-Instruct	❌	✅	✅	RLVR training without intermediate thinking traces.
RAFT	✅	❌	✅	On-policy rejection sampling fine-tuning using only positive reasoning samples.
Think-DPO	✅	✅	❌	Offline preference optimization using pre-collected thinking traces.
Batch-online GRPO	✅	✅	⚠️	Semi-online training where the generator policy is synced every 4 steps.

🎁 Datasets

The Aletheia dataset collection is available on HuggingFace and includes:

Aletheia-Train: A dataset of 2-5 candidate code snippets to solve a coding problem, each with exactly one correct code
Aletheia-DPO: A companion dataset to Aletheia-Train, containing "chosen" and "rejected" responses for each instance. The chosen response identifies the correct code snippet, while the rejected responses do not.
Aletheia-Heldout: A completely in-distribution test set
Aletheia-Strong: An OOD test set where the candidates are generated by stronger models
Aletheia-Hard: An OOD test set where the comparison between candidates is more difficult
Aletheia-Adv: An OOD test set where the candidates are adversarially modified to exploit common LLM biases

💡 Intended uses

RLHF / RLAIF: plug-and-play reward function for code generation policy optimization.
Automated evaluation: LLM-as-a-judge for a variety of code-related tasks.
Research: study the effects of thinking traces, on-policy learning, and negative samples in training successful code verifiers

Citations

@article{aletheia2025,
  title={Aletheia: What Makes RLVR For Code Verifiers Tick?},
  author={Venkatkrishna, Vatsal and Paul, Indraneil and Gurevych, Iryna},
  journal={arXiv preprint},
  year={2025}
}