Update model/dataset card: arXiv link, updated abstract, unified documentation

README.md

@@ -1,69 +1,84 @@
 ---
-base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
 library_name: transformers
-model_name: dpo_DeepSeek-R1-Distill-Qwen-7B
+license: cc-by-nc-sa-4.0
+pipeline_tag: text-generation
+base_model:
+- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
+datasets:
+- INSAIT-Institute/Aletheia-DPO
 tags:
-- generated_from_trainer
+- code
+- code-verification
+- rlvr
 - dpo
-- trl
-licence: license
 ---
 
-# Model Card for dpo_DeepSeek-R1-Distill-Qwen-7B
+<font size=3><div align='center'>
+[[**📖 Paper**](https://arxiv.org/pdf/2601.12186)]
+[[**💻 Code**](https://github.com/insait-institute/aletheia)]
+[[**🤗 Models & Datasets**](https://huggingface.co/collections/INSAIT-Institute/aletheia-6a7c60fab9eadaef73aa427b)]
+</div></font>
 
-This model is a fine-tuned version of [deepseek-ai/DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+# Aletheia: What Makes RLVR For Code Verifiers Tick?
 
-## Quick start
+Multi-domain thinking verifiers trained via Reinforcement Learning with Verifiable Rewards (RLVR) are a cornerstone of modern post-training. However, their adoption in code generation has lagged behind that of execution feedback due to the prohibitive costs of the full RLVR pipeline. In this work, we ablate three primary choices along the performance-cost trade-off in RLVR: intermediate thinking traces, learning from negative samples, and on-policy training. We introduce **Aletheia**, a controlled, execution-grounded testbed to facilitate a contamination-free analysis of code verifier training recipes across disparate model sizes and covariate shifts across two common verifier application scenarios. Our analysis reveals that the optimal training recipe is scale-dependent: on-policy learning is the primary performance driver for small verifiers, whereas the thinking budget becomes the most vital factor at larger scales. While leveraging negative samples has a consistent impact on top-1 selection accuracy across sizes, their contribution to ranking reconstruction increases monotonically with scale and plays a key role in stabilizing training at large sizes. Our Pareto optimality analysis demonstrates that eliminating on-policy training at larger model scales yields a verifier that performs comparably to the full RLVR recipe. Furthermore, we find that eschewing thinking traces serves as a compute-efficient strategy at lower budgets, offering a strong trade-off between training cost and verifier accuracy. Ultimately, our work provides the empirical foundation necessary to efficiently deploy robust code verifiers, thereby enabling their wider adoption in post-training pipelines for large code generation models.
 
-```python
-from transformers import pipeline
+## 🤖 This Model
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="wetsoledrysoul/dpo_DeepSeek-R1-Distill-Qwen-7B", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
+**`DPO-Think-7B`** is a **DPO-Think** verifier: trained with **offline** preference optimization (DPO) on pre-collected thinking traces (no on-policy sampling). It is fine-tuned from [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) (7B scale) on [`INSAIT-Institute/Aletheia-DPO`](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-DPO).
+
+Given a competitive-programming problem and a set of candidate solutions, the verifier judges and ranks the candidates. See the [GitHub repository](https://github.com/insait-institute/aletheia) for the exact prompt format, plus training and evaluation scripts.
+
+## 📦 Model Zoo
+
+Fine-tuned code verifiers at 1.5B, 7B, and 14B scales using several algorithms:
 
-## Training procedure
+| Algorithm | Thinking | Negatives | Online | Description |
+| :--- | :---: | :---: | :---: | :--- |
+| [**GRPO-Think**](https://huggingface.co/INSAIT-Institute/GRPO-Think-7B-16k) | ✅ | ✅ | ✅ | Standard GRPO-style approach to training verifiers. |
+| [**GRPO-Instruct**](https://huggingface.co/INSAIT-Institute/GRPO-Instruct-7B) | ❌ | ✅ | ✅ | RLVR training without intermediate thinking traces. |
+| [**RAFT**](https://huggingface.co/INSAIT-Institute/RAFT-7B) | ✅ | ❌ | ✅ | On-policy rejection sampling fine-tuning using only positive reasoning samples. |
+| [**DPO-Think**](https://huggingface.co/INSAIT-Institute/DPO-Think-7B) | ✅ | ✅ | ❌ | Offline preference optimization using pre-collected thinking traces. |
+| [**BatchOnline-GRPO**](https://huggingface.co/INSAIT-Institute/BatchOnline-GRPO-7B) | ✅ | ✅ | ⚠️ | Semi-online training where the generator policy is synced every 4 steps. |
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/CodeShield/CerebRM-DPO/runs/00y7t01f) 
+The `-4k` / `-8k` / `-16k` suffix on GRPO-Think checkpoints denotes the reasoning-token budget (maximum completion length) used during training.
 
+## 🎁 Datasets
 
-This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
+The Aletheia dataset collection includes:
 
-### Framework versions
+* [**Aletheia-Train**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Train): 50,000 training instances, each pairing a competitive-programming problem with 2–5 candidate solutions — exactly one of which is correct (execution-verified) — generated by a pool of weak and strong policy models across Python, C++, and Java.
+* [**Aletheia-Train-Questions**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Train-Questions): The 3,574 unique problem statements underlying Aletheia-Train, useful for on-policy sampling.
+* [**Aletheia-DPO**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-DPO): A companion dataset to Aletheia-Train containing "chosen" and "rejected" verification responses for each instance. The chosen response identifies the correct code snippet, while the rejected response does not.
+* [**Aletheia-Mixed**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Mixed): A variant of Aletheia-Train in which ~25% of instances carry adversarial modifications that exploit common LLM biases (authority references, self-declared correctness, misleading comments, etc.).
+* [**Aletheia-Mixed-DPO**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Mixed-DPO): The DPO-style companion to Aletheia-Mixed.
+* [**Aletheia-Heldout**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Heldout): A completely in-distribution test set.
+* [**Aletheia-Strong**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Strong): An OOD test set where the candidates are generated by stronger models.
+* [**Aletheia-Hard**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Hard): An OOD test set where the comparison between candidates is more difficult.
+* [**Aletheia-Adv**](https://huggingface.co/datasets/INSAIT-Institute/Aletheia-Adv): An OOD test set where the candidates are adversarially modified to exploit common LLM biases.
 
-- TRL: 0.24.0
-- Transformers: 4.56.1
-- Pytorch: 2.7.1
-- Datasets: 4.0.0
-- Tokenizers: 0.22.0
+## 💡 Intended Uses
 
-## Citations
+* **RLHF / RLAIF**: plug-and-play reward function for code generation policy optimization.
+* **Automated evaluation**: LLM-as-a-judge for a variety of code-related tasks.
+* **Research**: study the effects of thinking traces, on-policy learning, and negative samples in training successful code verifiers.
 
-Cite DPO as:
+## 📚 Citation
+
+If you find this work useful, please cite our paper:
 
 ```bibtex
-@inproceedings{rafailov2023direct,
-    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
-    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
-    year         = 2023,
-    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
-    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
-    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
+@misc{venkatkrishna2026aletheiamakesrlvrcode,
+      title={Aletheia: What Makes RLVR For Code Verifiers Tick?},
+      author={Vatsal Venkatkrishna and Indraneil Paul and Iryna Gurevych},
+      year={2026},
+      eprint={2601.12186},
+      archivePrefix={arXiv},
+      primaryClass={cs.SE},
+      url={https://arxiv.org/abs/2601.12186},
 }
 ```
 
-Cite TRL as:
-    
-```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
-}
-```
+## 📄 License
+
+This work is licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).