| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-Coder-7B-Instruct |
| tags: |
| - reward-model |
| - code-generation |
| - rlhf |
| pipeline_tag: text-classification |
| language: |
| - en |
| library_name: transformers |
| --- |
| |
| # CodeRM-NT |
| [Paper](https://aclanthology.org/2026.findings-acl.2150/) | |
| [Github](https://github.com/THUDM/CodeRM-NT) |
|
|
| Providing accurate reward signals for code generated by LLMs is a significant challenge in applying reinforcement learning (RL) to code generation. Existing methods rely on unit tests, which are expensive to curate and unreliable when automatically synthesized. |
|
|
| **CodeRM-NT** is a code reward model with **no reliance on unit tests**. Instead of executing test cases, it learns to estimate the functional correctness of generated Python code from rewards that are collected via Monte Carlo Tree Search (MCTS) guided by LLM-as-a-Judge. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| model = AutoModelForSequenceClassification.from_pretrained( |
| "Rishubi/CodeRM-NT", |
| device_map="auto", |
| ) |
| tokenizer = AutoTokenizer.from_pretrained("Rishubi/CodeRM-NT") |
| |
| question = "Write a Python function `add(a, b)` that returns the sum of two integers." |
| response = "def add(a, b):\n return a + b" |
| |
| messages = [ |
| {"role": "user", "content": question}, |
| {"role": "assistant", "content": response}, |
| ] |
| input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device) |
| with torch.no_grad(): |
| reward = model(input_ids).logits.squeeze().float().item() |
| print(reward) # higher is better |
| ``` |
|
|
| ## Results |
|
|
| ## Key Results |
|
|
| Training with CodeRM-NT consistently outperforms synthetic unit tests and other reward models across multiple code generation benchmarks: |
|
|
| | Model | Reward | HumanEval | HumanEval+ | MBPP | MBPP+ | LCB-v5 | BCB-I-Hard | Avg. | |
| | :----------------- | :------------ | :-------: | :--------: | :--: | :---: | :----: | :--------: | :------: | |
| | Qwen2.5-Coder-1.5B | Unit Tests | 73.2 | 67.7 | 70.9 | 61.1 | 5.1 | 6.1 | 47.4 | |
| | | **CodeRM-NT** | 75.0 | 69.5 | 72.0 | 60.8 | 5.5 | 7.4 | **48.4** | |
| | Qwen2.5-Coder-3B | Unit Tests | 86.6 | 82.3 | 74.9 | 64.6 | 13.0 | 15.5 | 56.2 | |
| | | **CodeRM-NT** | 88.4 | 82.3 | 75.9 | 66.1 | 13.6 | 14.2 | **56.8** | |
| | Qwen2.5-Coder-7B | Unit Tests | 90.9 | 87.8 | 85.4 | 73.0 | 17.3 | 18.2 | 62.1 | |
| | | **CodeRM-NT** | 90.2 | 86.0 | 86.8 | 74.6 | 17.5 | 18.2 | **62.2** | |
| | GLM-4-9B-0414 | Unit Tests | 84.1 | 79.9 | 81.0 | 69.0 | 15.4 | 15.5 | 57.5 | |
| | | **CodeRM-NT** | 87.2 | 81.7 | 79.9 | 67.2 | 15.3 | 18.2 | **58.3** | |
| | Qwen3-4B-Thinking | Unit Tests | 97.6 | 92.7 | 91.0 | 75.1 | 50.3 | 25.7 | 72.1 | |
| | | **CodeRM-NT** | 97.6 | 94.5 | 92.6 | 77.2 | 52.1 | 22.3 | **72.7** | |
|
|
| ## Citation |
|
|
| If you find our work helpful, please kindly cite our paper: |
|
|
| ``` |
| @inproceedings{xia-etal-2026-coderm, |
| title = "{C}ode{RM}-{NT}: Reward Model for Code {RL} without Unit Tests", |
| author = "Xia, Xiao and |
| Zhang, Dan and |
| Sun, Tianrui", |
| editor = "Liakata, Maria and |
| Moreira, Viviane P. and |
| Zhang, Jiajun and |
| Jurgens, David", |
| booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026", |
| month = jul, |
| year = "2026", |
| address = "San Diego, California, United States", |
| publisher = "Association for Computational Linguistics", |
| url = "https://aclanthology.org/2026.findings-acl.2150/", |
| pages = "43316--43333", |
| ISBN = "979-8-89176-395-1" |
| } |
| ``` |
|
|
