README.md

---
base_model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
library_name: transformers
pipeline_tag: text-classification
tags:
- base_model:adapter:deepseek-ai/DeepSeek-R1-Distill-Llama-8B
- lora
- transformers
- reward-model
license: apache-2.0
language:
- en
---

# dORM-8B


This model is a discriminative outcome reward model finetuned from [DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B), and the [training data](https://huggingface.co/datasets/dongboklee/train) is CoTs generated by [Llama3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) (mostly adapted from [VersaPRM](https://github.com/UW-Madison-Lee-Lab/VersaPRM)).

For details:
- **Paper:** [Rethinking Reward Models for Multi-Domain Test-Time Scaling](https://huggingface.co/papers/2510.00492)
- **Repository:** [https://github.com/db-Lee/Multi-RM](https://github.com/db-Lee/Multi-RM)


### Direct Use
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# tokenizer
def get_tokenizer(model_id):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    tokenizer.pad_token = tokenizer.eos_token  
    tokenizer.padding_side = 'left' 
    tokenizer.truncation_side = 'left'
    return tokenizer

tokenizer = get_tokenizer('dongboklee/dORM-8B')
candidate_tokens = [
  self.tokenizer.encode("-", add_special_tokens=False)[-1], 
  self.tokenizer.encode("+", add_special_tokens=False)[-1]
]
tag_id = self.tokenizer.encode(" \n\n\n\n", add_special_tokens=False)[-1]

# model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = AutoModelForCausalLM.from_pretrained('dongboklee/dORM-8B')
model.eval()
model.to(device)

question = 'Question: In Python 3, which of the following function convert a string to an int in python?\nA. short(x)\nB. float(x)\nC. integer(x [,base])\nD. double(x)\nE. int(x [,base])\nF. long(x [,base] )\nG. num(x)\nH. str(x)\nI. char(x)\nJ. digit(x [,base])'
solution = ["To convert a string to an integer in Python 3, we use the built-in function int().",
            "The int() function takes two arguments: the string to be converted and an optional base (default is 10, which is for decimal).",
            "For example: int(\"123\", 10) converts the string \"123\" to the integer 123.",
            "Looking at the options, we can see that the correct function is option E: int(x [,base]).",
            "The answer is (E)."]
input_text = question + ' \n\n' + ' \n\n\n\n'.join(solution) + ' \n\n\n\n' # solution steps are separated by ' \n\n\n\n'
input_id = torch.tensor([tokenizer.encode(input_text)]).to(device)

with torch.no_grad():
    logits = model(input_id).logits[:,:,candidate_tokens]
    scores = logits.softmax(dim=-1)[:,:,1] 
    step_scores = scores[input_id == tag_id]
    step_probs  = step_scores.tolist()[:,:,-1]
```

## Citation

```
@article{multi-rm,
  title   = {Rethinking Reward Models for Multi-Domain Test-Time Scaling},
  author  = {Lee, Dong Bok and Lee, Seanie and Park, Sangwoo and Kang, Minki and Baek, Jinheon and Kim, Dongki and Wagner, Dominik and Jin, Jiongdao and Lee, Heejun and Bocklet, Tobias and Wang, Jinyu and Fu, Jingjing and Hwang, Sung Ju and Bian, Jiang and Song, Lei},
  journal = {arXiv preprint arXiv:2510.00492},
  year    = {2025}
}
```