Improve model card for Rethinking Reward Models for Multi-Domain Test-Time Scaling

README.md

@@ -6,202 +6,110 @@ tags:
 - base_model:adapter:deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
 - lora
 - transformers
+- reward-model
+license: apache-2.0
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
+# Rethinking Reward Models for Multi-Domain Test-Time Scaling
 
+This repository contains a LoRA adapter for a reward model, part of the research presented in the paper "[Rethinking Reward Models for Multi-Domain Test-Time Scaling](https://huggingface.co/papers/2510.00492)".
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+The reliability of large language models (LLMs) during test-time scaling is often assessed with *external verifiers* or *reward models* that distinguish correct reasoning from flawed logic. This work presents the first unified evaluation of four reward model variants: discriminative ORM (Outcome Reward Model) and PRM (Process Reward Model) (**DisORM**, **DisPRM**), and generative ORM and PRM (**GenORM**, **GenPRM**), across 14 diverse domains.
 
+Contrary to conventional wisdom, the research finds that (i) **DisORM** performs on par with **DisPRM**, (ii) **GenPRM** is not competitive, and (iii) overall, **GenORM** is the most robust, yielding significant and consistent gains across every tested domain. These findings challenge the prevailing assumption that fine-grained supervision is always better and support generative outcome verification for multi-domain deployment.
 
+This specific model card refers to an adapter built on the `deepseek-ai/DeepSeek-R1-Distill-Qwen-14B` backbone.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+-   **Developed by:** Lee, Dong Bok and Lee, Seanie and Park, Sangwoo and Kang, Minki and Baek, Jinheon and Kim, Dongki and Wagner, Dominik and Jin, Jiongdao and Lee, Heejun and Bocklet, Tobias and Wang, Jinyu and Fu, Jingjing and Hwang, Sung Ju and Bian, Jiang and Song, Lei
+-   **Model type:** Reward Model (LoRA adapter)
+-   **Language(s) (NLP):** English
+-   **License:** Apache 2.0 (for code/model). The datasets used are licensed under [CC-BY-NC-4.0 License](https://github.com/db-Lee/Multi-RM/blob/main/LICENSE.txt).
+-   **Finetuned from model:** This model is fine-tuned from [deepseek-ai/DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B). Other models presented in the paper are based on [deepseek-ai/DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B).
 
-### Model Sources [optional]
+### Model Sources
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+-   **Repository:** https://github.com/db-Lee/Multi-RM
+-   **Paper:** https://huggingface.co/papers/2510.00492
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+These models are intended for researchers and practitioners to evaluate the reliability and reasoning capabilities of large language models across diverse domains. They can be used to provide reward signals for LLM alignment or to verify the correctness of generated LLM outputs, including generating verification rationales.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+The models are not intended for general-purpose text generation beyond their specific task of reward modeling and rationale generation. They should not be deployed in critical systems where absolute correctness is required without human oversight, given the inherent limitations of AI systems.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+The paper highlights that process reward models (PRMs) may inherit label noise from LLM auto-labeling and can have difficulty evaluating long reasoning trajectories, especially those involving self-correcting reasoning. Step-wise aggregation can compound errors as reasoning length grows. Users should be aware of these potential limitations when applying the models.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+First, set up your environment as described in the [GitHub repository](https://github.com/db-Lee/Multi-RM):
 
-## Training Details
+```bash
+conda create -n multi-rm python=3.10.14
+conda activate multi-rm
+pip install -r requirements.txt
+pip install flash-attn --no-build-isolation
+```
 
-### Training Data
+### Inference (Reward Generation)
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The following example, adapted from the GitHub repository, demonstrates how to use a generative reward model (gORM/gPRM) for inference. Replace `[TEST]` with `test`, `test_smollm`, `test_qwen`, `test_gemma`, or `test_llama` as needed.
 
-[More Information Needed]
+```bash
+# Inference for gORM / gPRM
+# Use the appropriate model checkpoint, e.g., dongboklee/gORM-14B-merged, TASK_TYPE=gORM
 
-### Training Procedure
+python -m generative.get_reward \
+  --data_path dongboklee/[TEST] \
+  --model_id dongboklee/gORM-14B-merged \
+  --output_dir ./[REWARD_RESULTS]/gORM-14B-[TEST] \
+  --task_type gORM \
+  --category all
+```
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+For discriminative models (dORM/dPRM) or for merging LoRA adapters of your own trained models, please refer to the [Inference section of the GitHub repository](https://github.com/db-Lee/Multi-RM#inference-reward).
 
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+## Training Details
 
-#### Speeds, Sizes, Times [optional]
+For comprehensive training details, including synthetic verification rationale generation, training procedures for dORM/dPRM and gORM/gPRM, and hyperparameter configurations, refer to the [Training section of the GitHub repository](https://github.com/db-Lee/Multi-RM#training).
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+### Training Data
 
-[More Information Needed]
+The models were trained on multi-domain datasets, including:
+-   [`train`](https://huggingface.co/datasets/dongboklee/train): Multi-domain training dataset for dORM/dPRM (mostly adapted from [VersaPRM](https://github.com/UW-Madison-Lee-Lab/VersaPRM)).
+-   [`train_gORM`](https://huggingface.co/datasets/dongboklee/train_gORM): Multi-domain training dataset for gORM.
+-   [`train_gPRM`](https://huggingface.co/datasets/dongboklee/train_gPRM): Multi-domain training dataset for gPRM.
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
+Detailed evaluation protocols, testing data (e.g., `test`, `test_smollm`, `test_qwen`), metrics, and results are available in the [Evaluation section of the GitHub repository](https://github.com/db-Lee/Multi-RM#evaluation).
 
-## More Information [optional]
+## Assets
 
-[More Information Needed]
+The [GitHub repository](https://github.com/db-Lee/Multi-RM#assets) provides access to training and test datasets, model checkpoints, and rewards obtained by the four reward model variants, hosted on Hugging Face.
 
-## Model Card Authors [optional]
+## Citation
 
-[More Information Needed]
+If you find this work useful for your research or applications, please consider citing the paper:
 
-## Model Card Contact
+```bibtex
+@article{multi-rm,
+  title   = {Rethinking Reward Models for Multi-Domain Test-Time Scaling},
+  author  = {Lee, Dong Bok and Lee, Seanie and Park, Sangwoo and Kang, Minki and Baek, Jinheon and Kim, Dongki and Wagner, Dominik and Jin, Jiongdao and Lee, Heejun and Bocklet, Tobias and Wang, Jinyu and Fu, Jingjing and Hwang, Sung Ju and Bian, Jiang and Song, Lei},
+  journal = {arXiv preprint arXiv:2510.00492},
+  year    = {2025}
+}
+```
 
-[More Information Needed]
 ### Framework versions
 
 - PEFT 0.16.0