Qwen2.5-7B-SHARP-Step

Introduction

Qwen2.5-7B-SHARP-Step is a Process Reward Model (PRM) for step-level hallucination detection in mathematical reasoning. It scores every intermediate step of a solution and flags the ones that are unsupported or wrong, which makes it usable both for error localization and for Best-of-N reranking.

The model is trained from Qwen/Qwen2.5-Math-7B-Instruct on the SHARP corpus. A <extra_0> marker is appended to every step, and a two-way classification head predicts, at each marker, whether the step is correct.

This is the Step variant: it is supervised with step-level labels, where every reasoning step of the solution carries its own correct / hallucinated label. The companion Span variant, supervised with span-level hallucination annotations, is Qwen2.5-7B-SHARP-Span.

Model Details

Base model	Qwen/Qwen2.5-Math-7B-Instruct
Architecture	Qwen2ForProcessRewardModel (custom code in modeling_qwen2_rm.py)
Reward head	Linear(hidden, hidden) -> ReLU -> Linear(hidden, 2) on top of every token
Hidden size / layers	3584 / 28
Max position embeddings	4096
Step marker	<extra_0> (single special token)
Supervision	step-level labels over the reasoning steps of the solution
Weights dtype	bfloat16

Requirements

• transformers>=4.40.0. The latest version is recommended.
• trust_remote_code=True -- the PRM class ships with the checkpoint.

Quick Start

Qwen2.5-7B-SHARP-Step is a process reward model used for scoring reasoning steps, not for generation.

Prerequisites

• Step separation: split the solution into steps and join them with double line breaks ("\n\n").
• Step marker: append <extra_0> to the end of every step.
• Prompt format: the model does not use a chat template. Wrap the input as Question: {question}\n\nSolution:\n{steps}, which is the format used during training.
• Reward computation: at each <extra_0> position take the probability of the positive class. The result is a value between 0 and 1, where low values mark a hallucinated or incorrect step.

Hugging Face Transformers

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer


def build_prompt(question, steps):
    body = "\n\n".join(f"{step.strip()}<extra_0>" for step in steps)
    return f"Question: {question}\n\nSolution:\n{body}"


def make_step_rewards(logits, token_masks):
    probabilities = F.softmax(logits, dim=-1)
    probabilities = probabilities * token_masks.unsqueeze(-1)  # bs, seq_len, num_labels

    all_scores_res = []
    for i in range(probabilities.size(0)):
        sample = probabilities[i]  # seq_len, num_labels
        positive_probs = sample[sample != 0].view(-1, 2)[:, 1]  # valid_tokens, num_labels
        all_scores_res.append(positive_probs.cpu().tolist())
    return all_scores_res


model_name = "ZaandaTeika/Qwen2.5-7B-SHARP-Step"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
).eval()

data = {
    "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?",
    "steps": [
        "In April, Natalia sold 48 clips.",
        "In May she sold half as many, so she sold 48 / 2 = 24 clips.",
        "Altogether she sold 48 + 24 = 72 clips. The answer is \\boxed{72}.",
    ],
}

prompt = build_prompt(data["question"], data["steps"])
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model(input_ids=input_ids)

step_sep_id = tokenizer.encode("<extra_0>", add_special_tokens=False)[0]
token_masks = input_ids == step_sep_id
step_reward = make_step_rewards(outputs[0], token_masks)
print(step_reward)  # one score per step