Improve model card: Add description, links, license, update pipeline tag, and sample usage

README.md

@@ -1,207 +1,186 @@
 ---
 base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
-library_name: peft
-pipeline_tag: text-generation
+library_name: transformers
+license: apache-2.0
+pipeline_tag: text-classification
 tags:
 - base_model:adapter:deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
 - lora
+- peft
 - transformers
+- reward-model
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
+# Model Card for Rethinking Reward Models for Multi-Domain Test-Time Scaling
 
+This repository contains checkpoints for reward models presented in the paper [Rethinking Reward Models for Multi-Domain Test-Time Scaling](https://huggingface.co/papers/2510.00492).
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-
+The reliability of large language models (LLMs) during test-time scaling is often assessed with external verifiers or reward models that distinguish correct reasoning from flawed logic. This work presents the first unified evaluation of four reward model variants: discriminative ORM (DisORM), discriminative PRM (DisPRM), and generative ORM (GenORM), and generative PRM (GenPRM), across 14 diverse domains.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+Contrary to conventional wisdom, the paper finds that (i) DisORM performs on par with DisPRM, (ii) GenPRM is not competitive, and (iii) overall, **GenORM is the most robust**, yielding significant and consistent gains across every tested domain. These findings challenge the prevailing assumption that fine-grained supervision is always better and support generative outcome verification for multi-domain deployment. This particular model is a LoRA adapter for a generative Outcome Reward Model (gORM), specifically built on the `deepseek-ai/DeepSeek-R1-Distill-Qwen-14B` backbone.
 
-### Model Sources [optional]
+-   **Developed by:** Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiongdao Jin, Heejun Lee, Tobias Bocklet, Jinyu Wang, Jingjing Fu, Sung Ju Hwang, Jiang Bian, Lei Song.
+-   **Model type:** Generative Outcome Reward Model (LoRA Adapter).
+-   **Language(s) (NLP):** English
+-   **License:** Apache-2.0
+-   **Finetuned from model:** [deepseek-ai/DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B)
 
-<!-- Provide the basic links for the model. -->
+### Model Sources
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+-   **Repository:** [https://github.com/db-Lee/Multi-RM](https://github.com/db-Lee/Multi-RM)
+-   **Paper:** [https://huggingface.co/papers/2510.00492](https://huggingface.co/papers/2510.00492)
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+This model is intended for use as an external verifier or reward model to assess the reliability and quality of reasoning provided by other Large Language Models (LLMs). It can be directly applied to evaluate final answers in various tasks and domains, generating a critique or rationale for its assessment.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+This model is specifically trained as a reward model for evaluating LLM reasoning. It is not intended for general-purpose text generation, conversational tasks, or other natural language processing applications outside of assessing LLM outputs.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+The paper highlights that Process Reward Model (PRM)-style stepwise scoring can inherit label noise from LLM auto-labeling and has difficulty evaluating long reasoning trajectories, including those involving self-correcting reasoning. While Generative Outcome Reward Models (gORM) like this one show more robustness, users should be aware that any reward model's judgments are influenced by its training data and may reflect inherent biases or limitations.
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Critical evaluation of the model's outputs is recommended, especially when deploying in new domains or high-stakes applications.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+To get started, first set up your environment as recommended by the authors:
+
+```bash
+conda create -n multi-rm python=3.10.14
+conda activate multi-rm
+pip install -r requirements.txt
+pip install flash-attn --no-build-isolation
+```
+
+For inference with the generative Outcome Reward Models (gORM), you can use the LoRA-merged versions available on the Hugging Face Hub (e.g., `dongboklee/gORM-14B-merged`). The following Python snippet demonstrates how to load and use such a merged model with the `transformers` library to generate a reward rationale:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+# Load the LoRA-merged gORM-14B model
+# The "merged" version is directly loadable with AutoModelForCausalLM
+model_id = "dongboklee/gORM-14B-merged" 
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16, # Use bfloat16 for better performance if supported
+    device_map="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+# Ensure pad_token_id is set for generation
+if tokenizer.pad_token_id is None:
+    tokenizer.pad_token_id = tokenizer.eos_token_id
+
+# Example input: A chain-of-thought (CoT) response to be evaluated
+# In a real scenario, this would come from your dataset or LLM output.
+example_reasoning = (
+    "Question: What is 2+2? Reasoning: I will add the numbers. 2 plus 2 equals 4. "
+    "Final Answer: 4"
+)
+
+messages = [
+    {"role": "user", "content": f"Please evaluate the following reasoning:\
+{example_reasoning}"}
+]
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+# Generate the reward/evaluation from the model
+with torch.no_grad():
+    output_ids = model.generate(
+        **inputs, 
+        max_new_tokens=128, 
+        do_sample=False, 
+        temperature=0.7, 
+        top_p=0.9,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.pad_token_id 
+    )
+
+response = tokenizer.decode(output_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+print(f"Generated Reward Rationale:\
+{response}")
+```
+
+For more detailed usage, including training and evaluation scripts, or working with non-merged LoRA adapters (which require `peft` for loading before merging), please refer to the [GitHub repository](https://github.com/db-Lee/Multi-RM). The repository also provides CLI commands for inference like:
+
+```bash
+# Example inference for gORM / gPRM (using the merged model)
+# Replace [TEST] with test, test_smollm, etc.
+# Replace [REWARD_RESULTS] with your desired output directory.
+python -m generative.get_reward \
+  --data_path dongboklee/[TEST] \
+  --model_id dongboklee/gORM-14B-merged \
+  --output_dir ./[REWARD_RESULTS]/gORM-14B-[TEST] \
+  --task_type gORM \
+  --category all
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+The models were trained on several multi-domain datasets, which are publicly available on Hugging Face:
+-   **Discriminative ORM/PRM**: Trained on [`dongboklee/train`](https://huggingface.co/datasets/dongboklee/train) (mostly adapted from [VersaPRM](https://github.com/UW-Madison-Lee-Lab/VersaPRM)).
+-   **Generative ORM (gORM)**: Trained on [`dongboklee/train_gORM`](https://huggingface.co/datasets/dongboklee/train_gORM) (generated by [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B)).
+-   **Generative PRM (gPRM)**: Trained on [`dongboklee/train_gPRM`](https://huggingface.co/datasets/dongboklee/train_gPRM) (generated by [QwQ-32B](https://huggingface.co/Qwen/QwQ-32B)).
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
+Models are trained using the PEFT (LoRA) framework, leveraging `accelerate` for distributed training. Detailed training scripts and configurations can be found in the `./configs` directory of the [GitHub repository](https://github.com/db-Lee/Multi-RM).
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+From `adapter_config.json`:
+-   **`peft_type`**: LORA
+-   **`lora_alpha`**: 32
+-   **`r` (LoRA rank)**: 16
+-   **`lora_dropout`**: 0.1
+-   **`target_modules`**: `down_proj`, `k_proj`, `v_proj`, `q_proj`, `up_proj`, `gate_proj`, `o_proj`
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+The models were evaluated on diverse multi-domain test datasets with Chain-of-Thoughts (CoTs) generated by various LLMs. These datasets are available on Hugging Face:
+-   [`dongboklee/test`](https://huggingface.co/datasets/dongboklee/test): CoTs (N=128) generated by [Llama3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).
+-   [`dongboklee/test_smollm`](https://huggingface.co/datasets/dongboklee/test_smollm): CoTs (N=16) generated by [SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B).
+-   [`dongboklee/test_qwen`](https://huggingface.co/datasets/dongboklee/test_qwen): CoTs (N=16) generated by [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct).
+-   [`dongboklee/test_gemma`](https://huggingface.co/datasets/dongboklee/test_gemma): CoTs (N=16) generated by [gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it).
+-   [`dongboklee/test_llama`](https://huggingface.co/datasets/dongboklee/test_llama): CoTs (N=16) generated by [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct).
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+The evaluation primarily focused on the reward models' ability to accurately distinguish correct reasoning. Details on specific metrics and evaluation protocols can be found in the [GitHub repository](https://github.com/db-Lee/Multi-RM) and the accompanying paper.
 
 ### Results
 
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
+The paper demonstrates that Generative Outcome Reward Models (GenORM) are the most robust variant, yielding significant and consistent gains across every tested domain. This challenges the assumption that fine-grained supervision (like PRMs) is always superior. For comprehensive results and analysis, please refer to the [paper](https://huggingface.co/papers/2510.00492).
 
-## Model Card Contact
+## Citation
 
-[More Information Needed]
-### Framework versions
+If you find this work helpful or inspiring, please feel free to cite it:
 
-- PEFT 0.16.0
+```bibtex
+@article{multi-rm,
+  title   = {Rethinking Reward Models for Multi-Domain Test-Time Scaling},
+  author  = {Lee, Dong Bok and Lee, Seanie and Park, Sangwoo and Kang, Minki and Baek, Jinheon and Kim, Dongki and Wagner, Dominik and Jin, Jiongdao and Lee, Heejun and Bocklet, Tobias and Wang, Jinyu and Fu, Jingjing and Hwang, Sung Ju and Bian, Jiang and Song, Lei},
+  journal = {arXiv preprint arXiv:2510.00492},
+  year    = {2025}
+}
+```