Update README.md

README.md

@@ -1,69 +1,55 @@
 ---
-base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
 library_name: transformers
-model_name: dpo_DeepSeek-R1-Distill-Qwen-14B
-tags:
-- generated_from_trainer
-- dpo
-- trl
-licence: license
+license: cc-by-nc-sa-4.0
+datasets:
+- Aletheia-Bench/Aletheia-DPO
+base_model:
+- deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
 ---
 
-# Model Card for dpo_DeepSeek-R1-Distill-Qwen-14B
+<font size=3><div align='center' >  
+[[**🤗 Model & Dataset**](https://huggingface.co/Aletheia-Bench)] 
+[[**📊 Code**](https://github.com/insait-institute/aletheia)] 
+[[**📖 Paper**](https://arxiv.org/)]
+</div></font>
 
-This model is a fine-tuned version of [deepseek-ai/DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+# Aletheia: What Makes RLVR For Code Verifiers Tick?
+Multi-domain thinking verifiers trained via Reinforcement Learning from Verifiable Rewards (RLVR) are a prominent fixture of the Large Language Model (LLM) post-training pipeline, owing to their ability to robustly rate and rerank model outputs. However, the adoption of such verifiers towards code generation has been comparatively sparse, with execution feedback constituting the dominant signal. Nonetheless, code verifiers remain valuable toward judging model outputs in scenarios where execution feedback is hard to obtain and are a potentially powerful addition to the code generation post-training toolbox. To this end, we create and open-source Aletheia, a controlled testbed that enables execution-grounded evaluation of code verifiers' robustness across disparate policy models and covariate shifts. We examine components of the RLVR-based verifier training recipe widely credited for its success: (1) intermediate thinking traces, (2) learning from negative samples, and (3) on-policy training. While experiments show the optimality of RLVR, we uncover important opportunities to simplify the recipe. Particularly, despite code verification exhibiting positive training- and inference-time scaling, on-policy learning stands out as the key component at small verifier sizes, and thinking-based training emerges as the most important component at larger scales.
 
-## Quick start
 
-```python
-from transformers import pipeline
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="wetsoledrysoul/dpo_DeepSeek-R1-Distill-Qwen-14B", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
+## 📦 Model Zoo
+Fine-tuned code verifiers at 1.5B, 7B, and 14B scales using several algorithms
+| Algorithm | Thinking | Negatives | Online | Description |
+| :--- | :---: | :---: | :---: | :--- |
+| **GRPO-Think** | ✅ | ✅ | ✅ | Standard GRPO-style approach to training verifiers. |
+| **GRPO-Instruct** | ❌ | ✅ | ✅ | RLVR training without intermediate thinking traces. |
+| **RAFT** | ✅ | ❌ | ✅ | On-policy rejection sampling fine-tuning using only positive reasoning samples. |
+| **Think-DPO** | ✅ | ✅ | ❌ | Offline preference optimization using pre-collected thinking traces. |
+| **Batch-online GRPO** | ✅ | ✅ | ⚠️ | Semi-online training where the generator policy is synced every 4 steps. |
 
-## Training procedure
+## 🎁 Datasets
+The Aletheia dataset collection is available on HuggingFace and includes:
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/CodeShield/CerebRM-DPO/runs/h0aacdm3) 
+* **Aletheia-Train**: A dataset of 2-5 candidate code snippets to solve a coding problem, each with exactly one correct code
+* **Aletheia-DPO**: A companion dataset to Aletheia-Train, containing "chosen" and "rejected" responses for each instance. The chosen response identifies the correct code snippet, while the rejected responses do not.
+* **Aletheia-Heldout**: A completely in-distribution test set
+* **Aletheia-Strong**: An OOD test set where the candidates are generated by stronger models
+* **Aletheia-Hard**: An OOD test set where the comparison between candidates is more difficult
+* **Aletheia-Adv**: An OOD test set where the candidates are adversarially modified to exploit common LLM biases
 
-
-This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
-
-### Framework versions
-
-- TRL: 0.24.0
-- Transformers: 4.56.1
-- Pytorch: 2.7.1
-- Datasets: 4.0.0
-- Tokenizers: 0.22.0
+## 💡 Intended uses
+* **RLHF / RLAIF**: plug-and-play reward function for code generation policy optimization.  
+* **Automated evaluation**: LLM-as-a-judge for a variety of code-related tasks.  
+* **Research**: study the effects of thinking traces, on-policy learning, and negative samples in training successful code verifiers
 
 ## Citations
 
-Cite DPO as:
-
-```bibtex
-@inproceedings{rafailov2023direct,
-    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
-    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
-    year         = 2023,
-    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
-    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
-    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
-}
-```
-
-Cite TRL as:
-    
 ```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
+@article{aletheia2025,
+  title={Aletheia: What Makes RLVR For Code Verifiers Tick?},
+  author={Venkatkrishna, Vatsal and Paul, Indraneil and Gurevych, Iryna},
+  journal={arXiv preprint},
+  year={2025}
 }
 ```