section: A THE STATEMENT ON THE USE OF LARGE LANGUAGE MODELS

type: Text

text: In this work, we only use LLMs to correct grammatical issues and polish the writing. We do not use LLMs for research ideation or full paper writing.


section: B ADDITIONAL RELATED WORK

text: RLHF for LLMs Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., 2022) has been a key technique for aligning a model's capabilities and behaviors with human preferences. The standard pipeline typically consists of three stages: supervised fine-tuning on high-quality instructional data, learning a reward model from pairwise human preference data, and optimizing the policy model via reinforcement learning to maximize the learned reward. Prior work in RLHF spans a broad range of directions, including but not limited to: (1) Developing more effective or efficient RLHF algorithms and frameworks (Schulman et al., 2017; Ahmadian et al., 2024; Rafailov et al., 2023) ; (2) Constructing diverse, large-scale, and high-quality preference datasets (Bai et al., 2022; Cui et al., 2024) ; (3) Training stronger or domain-specific reward models to improve RLHF performance (Dong et al., 2024; Cai et al., 2024; Yuan et al., 2025) ; (4) Exploring multi-objective alignment (Guo et al., 2024; Li et al., 2025) .


section: C GENERAL FORM OF SELF-REWARDING MSE LOSS

text: In the main content, we derive the self-rewarding MSE loss in Eq. (12) by instantiating the verification reward in Eq. (7) under the binary 0/1 setting (though this is common choice). Here, we make further derivations to obtain the general form of our self-rewarding MSE loss by allowing the rewards for correct and incorrect verifications to take arbitrary real values, denoted as rˆ c and rˆ i , respectively. Then, we clarify why our method adopts the MSE loss instead of the BCE loss, and highlight that the solution to the BCE loss in Eq. (16) is in fact a special case of all available solutions in our framework.


text: Under the general definition, following the same derivation as in Eq. (9) , we obtain Z(x, y) = exp( rˆi β v ), which indicates logZ(x, y) = rˆi β v (still a constant but not necessarily to be 0). Therefore, logZ(x, y) = 0 in Eq. (9) is a special case when we define rˆ i = 0. Then, the optimal solution to the original RL target can be reduced to a general form


type: Equation

text: \hat{r}(\boldsymbol{x}, \boldsymbol{y}, \boldsymbol{z}) = \beta_v \log[\pi_{\boldsymbol{\theta}}(\boldsymbol{z}|\boldsymbol{x}, \boldsymbol{y}) / \pi_{\text{ref}}(\boldsymbol{z}|\boldsymbol{x}, \boldsymbol{y})] + \hat{r}_i.


text: That is,


text: \hat{r}(\boldsymbol{x}, \boldsymbol{y}, \boldsymbol{z}) - \hat{r}_i = \beta_v \log[\pi_{\boldsymbol{\theta}}(\boldsymbol{z}|\boldsymbol{x}, \boldsymbol{y})/\pi_{\text{ref}}(\boldsymbol{z}|\boldsymbol{x}, \boldsymbol{y})].


text: Now, we can see that the left-hand side of the equation is no longer a 0/1 binary reward but rather two arbitrary scalar values. Therefore, it is natural to model this objective using an MSE loss rather than a BCE loss.


text: Following above, the true verification reward when the model verifies a solution as correct is


text: \hat{r}(\boldsymbol{x}, \boldsymbol{y}, z_c) = \hat{r}_c \mathbb{I}_{r_v(\boldsymbol{x}, \boldsymbol{y}) = 1} + \hat{r}_i \mathbb{I}_{r_v(\boldsymbol{x}, \boldsymbol{y}) = 0} = \beta_v \log[\pi_{\boldsymbol{\theta}}(z_c | \boldsymbol{x}, \boldsymbol{y}) / \pi_{\text{ref}}(z_c | \boldsymbol{x}, \boldsymbol{y})] + \hat{r}_i.


text: Note that the reasoning rewards rv(x, y) defined in Eq. (2) can also be two arbitrary values, but we simplified them to {0, 1} and this will not affect our derivation. This formulation indicates that the probability πθ(zc∣x, y) alone is sufficient to model the reward score. In this general framework, our self-rewarding MSE loss is


text: L = \mathbb{E}_{\boldsymbol{x} \sim D, \boldsymbol{y} \sim \pi_g(\cdot|\boldsymbol{x})} \left( \beta_v \log[\pi_{\boldsymbol{\theta}}(z_c|\boldsymbol{x}, \boldsymbol{y}) / \pi_{\text{ref}}(z_c|\boldsymbol{x}, \boldsymbol{y})] + \hat{r}_i - (\hat{r}_c \mathbb{I}_{r_v(\boldsymbol{x}, \boldsymbol{y}) = 1} + \hat{r}_i \mathbb{I}_{r_v(\boldsymbol{x}, \boldsymbol{y}) = 0}) \right)^2.


text: Now, we show that the solution to the SFT/BCE loss in Eq. (16) is in fact a special case of all available solutions in our framework. After optimization (i.e., the optimal solution to the MSE loss), we can see that when rv(x, y) = 0 (i.e., the solution is incorrect based on the deterministic verifier), πθ(zc∣x, y) = πref(zc∣x, y), which means the πθ(zc∣x, y) remains unchanged (almost near zero, consistent with the solution to SFT loss when rv(x, y) = 0). When rv(x, y) = 1 (i.e., the solution is correct based on the deterministic verifier), our method drives πθ(zc∣x, y) to fit


type: FigureGroup

text: Figure 6: Cumulative implicit reward values Figure 7: The mean and standard deviation of lem. Red lines correspond to wrong solutions and over 300 input-output pairs. green lines correspond to correct solutions.


type: Caption

text: across 32 reasoning trajectories sampled from -\log \pi_{\rm ref}(z_c|x,y) under different combinations Open-Reasoner-Zero-7B on an AIME2024 prob- of reference model \pi_{\text{ref}} and pre-specified token z_c


text: \pi_{\text{ref}}(z_c|x,y) \exp(\frac{\hat{r}_c - \hat{r}_i}{\beta_v}) . As we can see, when \beta_v continues to decrease, the optimized probability \pi_{\theta}(z_c|x,y) approaches 1, which is exactly the solution to the BCE loss. However, in our framework, we can explicitly control the value of \beta_v , allowing \pi_{\theta}(z_c|x,y) to converge to any desired target. As shown in Appendix K, this controllability provides greater flexibility compared with directly forcing \pi_{\theta}(z_c|x,y) toward 1, while also exerting much less interference on the optimization of the reasoning ability. Thus, our framework is a fundamentally different but more general method.


section: D THE LENGTH BIAS IN IMPLICIT REWARD

text: Here, we present the trend of the cumulative implicit reward values (\log \frac{\pi_{\theta}(y_{< i}|x)}{\pi_{\text{ref}}(y_{< i}|x)}) where \pi_{\text{ref}} is Qwen2.5-7B-Base) across 32 reasoning trajectories sampled from Open-Reasoner-Zero on an AIME2024 problem, showing how they vary with the increasing trajectory lengths. As illustrated in Figure 6, the curves of all samples exhibit a positive correlation between the implicit reward and the number of tokens, and longer trajectories tend to yield higher final implicit reward scores, indicating a strong length bias in implicit reward. Since incorrect solutions are generally longer than correct ones in reasoning tasks (Hassid et al., 2025), implicit reward is therefore not a reliable indicator of the relative quality of reasoning paths at test time.


section: STATISTICS OF \log \pi_{\text{REF}}(z_c|\bm{x},\bm{y})

text: We present the mean and standard deviation of -\log \pi_{\rm ref}(z_c|x,y) computed over 300 input-output pairs. The reference model \pi_{ref} is chosen as either Qwen2.5-7B-Base or OctoThinker-3B-Short-Base, and the evaluation is performed under two different choices of z_c for each reference model (one common token and one unused special token): "Yes" and "<vision_start>" for Qwen2.5-7B-Base, "Yes" and "<|reserved_special_token_0|>" for OctoThinker-3B-Short-Base. The results in Figure 7 indicates that -\log \pi_{\rm ref}(z_c|x,y) remains nearly constant and extremely small, with only a low standard deviation across different x and y. Thus, we can consider \log \pi_{\rm ref}(z_c|x,y) as a constant when calculating the last-token self-rewarding scores, which effectively reduces the computational cost by half.


text: Furthermore, in the term \log \pi_{\rm ref}(z_c|\mathbf{x},\mathbf{y}) , the sequence \mathbf{y} is generated by the policy model, whose parameters evolve during training. To ensure that our simplification continues to hold throughout the optimization process, we report the dynamics of both the mean and the standard deviation of -\log \pi_{\rm ref}(z_c|x,y) over the course of training. Specifically, we calculate the mean and standard deviation of -\log \pi_{\rm ref}(z_c|x,y) based on the 300 generated samples from the policy model at training steps 0, 100, 200, ..., and 1000. We conduct evaluations by taking Qwen2.5-7B-Base as the


text: Figure 8: The dynamics of mean and standard deviation of -\log \pi_{\rm ref}(z_c|{\bm x},{\bm y}) during policy model's training.


section: Algorithm 1: LaSeR: Reinforcement Learning with Last-Token Self-Rewarding

text: Input: Initial policy model \pi_{\theta} , prompts D, verifier r_v , warm-up hyper-parameters w_r and w_{sr} , coefficient \beta_v , pre-specified token z_c , pre-calculated c_{\text{ref}} = \mathbb{E}_{(\boldsymbol{x},\boldsymbol{y})}[\log \pi_{\text{ref}}(z_c|\boldsymbol{x},\boldsymbol{y})]


text: for Step s = 1, \dots, S do


type: ListGroup

text: 1. Set \pi_{old} \leftarrow \pi_{\theta} ; 2. Sample batch prompts D_s from D; 3. Generate solutions \{\boldsymbol{y}^i\}_{i=1}^K for each \boldsymbol{x} \in D_s ; 4. Calculate verifier-based rewards and advantages (e.g., Eq. (4)), calculate RL loss; 5. If s \ge w_r , calculate last-token self-rewarding loss based on Eq. (14) and add it to RL loss; 6. If s \ge w_{sr} , calculate self-rewarding-based advantages and perform advantage integration based on Eq. (15); 7. Update the policy model \pi_{\theta} using any RL algorithm with integrated loss and advantages;


section: end

text: Output: \pi_{\theta}


text: base model and "<vision_start>" as the specified special token z_c . The dynamics are presented in Figure 8. As shown, the mean of -\log \pi_{ref}(z_c|x,y) remains stable throughout the policy model updates, which supports our motivation and confirms the feasibility of approximating it as a pre-computed constant in our method to improve efficiency.


section: FULL ALGORITHM

text: We display the full procedure of our method in Algorithm 1.


section: DETAILED TRAINING SETTINGS

text: We use ver1 (Sheng et al., 2024) as our RL training framework. The basic training hyper-parameters in both GRPO training and LaSeR training for each model are put in Table 4, and the newly introduced training hyper-parameters for LaSeR are put in Table 5. The number of optimization steps is 1000 for Qwen2.5-7B-Base and OctoThinker-3B-Short-Base, and 500 for Open-Reasoner-Zero-7B. In RL, a reasoning reward of 1.0 is given if the final answer and the answer format are both correct; otherwise, it is 0.0. In our method, the reasoning warm-up is performed for Qwen2.5-7B-Base and OctoThinker-


type: TableGroup

text: Table 4: Basic training hyper-parameters of both GRPO and LaSeR. Table 5: Unique training hyper-parameters of LaSeR. Hyper-parameter Value Train Batch Size 128 Micro Batch Size 128 Rollout n 8 Maximum Prompt Length 2048 Maximum Response Length 8192 Temperature 1.0 Top p 1.0 LR 1 × 10−6 KL Coefficient 0.0


type: Table

text: Hyper-parameter Value Coefficient βv 0.1 Loss Weight α 0.1 Self-Rewarding Adv. Weight τ 0.1 Reasoning Warm-Up Steps 200 Self-Rewarding Warm-Up Steps 200


text: Table 6: Comparison of verification F1 scores between LaSeR (self-rewarding) and external reward models (Qwen2.5-Math-7B-PRM800K, Qwen2.5-Math-PRM-7B, and Qwen2.5-Math-RM-72B) on responses generated by different policy models. Method MATH500 AMC23 AIME24 AIME25 Olym. Avg. Generator: OctoThinker-3B-Short-LaSeR Qwen2.5-Math-7B-PRM800K (7B RM) 77.0 68.9 - - 68.5 71.5 Qwen2.5-Math-PRM-7B (7B RM) 80.9 63.5 - - 64.1 69.5 Qwen2.5-Math-RM-72B (72B RM) 89.2 71.7 - - 72.9 77.9 LaSeR (3B Self-Rewarding) 73.6 70.2 - - 73.6 72.5 Generator: Qwen2.5-7B-Laser Qwen2.5-Math-7B-PRM800K (7B RM) 59.4 52.7 58.8 53.8 52.0 55.3 Qwen2.5-Math-PRM-7B (7B RM) 82.5 79.2 75.1 72.3 77.8 77.4 Qwen2.5-Math-RM-72B (72B RM) 87.8 80.7 81.3 74.8 75.4 80.0 LaSeR (7B Self-Rewarding) 83.2 82.5 79.6 74.3 78.3 79.6 Generator: Open-Reasoner-Zero-7B-LaSeR Qwen2.5-Math-7B-PRM800K (7B RM) 56.3 42.5 51.4 50.8 38.5 47.9 Qwen2.5-Math-PRM-7B (7B RM) 86.0 79.6 70.8 67.3 76.0 75.9 Qwen2.5-Math-RM-72B (72B RM) 86.8 79.4 71.0 71.4 75.5 76.8 LaSeR (7B Self-Rewarding) 87.2 79.7 64.6 77.7 78.7 77.6


text: 3B-Short-Base only, and the self-rewarding warm-up is perform for all models. The number of reasoning warm-up steps is set to 200 for both Qwen2.5-7B-Base and OctoThinker-3B-Short-Base, and the number of self-rewarding warm-up steps is 200 across all models.


section: H COMPARISON OF VERIFICATION PERFORMANCE BETWEEN LASER AND ADVANCED EXTERNAL VERIFIERS

text: Here, we display the comparison results of verification F1 scores between LaSeR and three advanced external reward models (Qwen2.5-Math-7B-PRM800K (Zhang et al., 2025) , Qwen2.5-Math-PRM-7B (Zhang et al., 2025) , and Qwen2.5-Math-RM-72B (Yang et al., 2024) ) on evaluating the solutions generated by the different reinforced models, i.e., OctoThinker-3B-Short-LaSeR, Qwen2.5-7B-LaSeR, and Open-Reasoner-Zero-7B-LaSeR. The full results in Table 6 show that LaSeR outperforms equally sized state-of-the-art external verifiers in assessing the model's own solutions, and even matches the verification performance of a 72B reward model, demonstrating its non-trivial effectiveness in enhancing self-rewarding capability.


text: Figure 9: The curves of training rewards and training self-verification F1 scores under different combinations of hyper-parameters with EMA smoothing (EMA coef.=0.9).


section: I ABLATION STUDIES ON SELF-REWARDING HYPER-PARAMETERS

text: Here, we display the curves (with Exponential Moving Average (EMA) smoothing) of training rewards and training self-verification F1 scores of our method under different choices of coefficient \beta_v and self-rewarding MSE loss weight \alpha . The experiments are conducted on Open-Reasoner-Zero-7B, which help to skip the reasoning warm-up phase compared with using Qwen2.5-7B-Base and OctoThinker-3B-Short-Base, while the results are similar in other two base models ater reasoning warm-up. The dynamics of training rewards and training self-verification F1 scores are displayed in Figure 9. As we can see, assigning a larger weight \alpha to the last-token self-rewarding loss has a more detrimental impact on the model's reasoning capabilities. On the other hand, the coefficient \beta_v has little impact on optimizing the self-rewarding scores, as long as it remains within a reasonable range (0.1 \sim 0.5) . However, much smaller values of \beta_v can impair the model's reasoning capability, as indicated by the analysis in the end of Section 3.4. For example, when \beta_v = 0.05 , we should


text: Figure 10: The curves of training rewards and training self-verification F1 scores of our method with and without class-level loss re-weighting practice (EMA coef.=0.9).


text: have \pi_{\theta}(z_c|\boldsymbol{x},\boldsymbol{y}) = e^{-3} \approx 0.05 under \pi_{ref}(z_c|\boldsymbol{x},\boldsymbol{y}) = e^{-23} and r_v(\boldsymbol{x},\boldsymbol{y}) = 1 , then the large value of \pi_{\theta}(z_c|\boldsymbol{x},\boldsymbol{y}) causes large interference with the optimization of reasoning capability. In our main experiments, we choose (\beta_v,\alpha) = (0.1,0.1) .


section: J THE EFFECT OF CLASS-LEVEL RE-WEIGHTING ON THE BALANCED SELF-VERIFICATION CAPABILITY

text: We present the training dynamics of our method on Open-Reasoner-Zero-7B, with and without class-level loss re-weighting, in Figure 10 for comparison. As shown, applying loss re-weighting leads to a more balanced self-verification performance by mitigating the bias toward the majority class with larger sample size, while still maintaining high reasoning accuracy.


text: Figure 11: The comparison of the training dynamics between the last-token self-rewarding loss and the SFT loss.


section: K COMPARISON BETWEEN LAST-TOKEN SELF-REWARDING LOSS AND SUPERVISED FINE-TUNING LOSS

text: Following the discussion in Section 3.4, we compare the training performance of our introduced last-token self-rewarding loss with the supervised fine-tuning (SFT) loss on Open-Reasoner-Zero-7B. The training dynamics are shown in Figure 11. As observed, applying the SFT loss to optimize the self-rewarding capability causes substantial interference with the optimization of reasoning capability, leading to a marked degradation in training rewards. Moreover, the SFT loss degrades extremely slowly, indicating that directly driving \pi_{\theta}(z_c|x,y) from 0 to 1 for r_v(x,y) = 1 is inherently difficult. However, our method only requires fitting \pi_{\theta}(z_c|x,y) to \exp(1/\beta_v) \cdot \pi_{\text{ref}}(z_c|x,y) for r_v(x,y) = 1 , which is considerably easier and introduces much less interference.


text: Figure 12: The ablation of only using self-rewarding scores for RL.


text: Table 7: The the range between the minimum and maximum self-rewarding scores over all solutions in each test set. Model MATH500 AMC23 AIME24 AIME25 OlympiadBench OT-3B-Short-LaSeR Qwen2.5-7B-LaSeR ORZ-7B-LaSeR [-0.021, 1.031] [0.013, 1.020] . , , [-0.041, 0.986] [-0.036, 1.013] [-0.056, 1.029] [-0.036, 1.016]


section: L ABLATION OF ONLY USING SELF-REWARDING SCORES FOR RL

text: In our method, after self-rewarding warm-up, we we incorporate self-rewarding-based advantages into the verifier-based advantages to provide more fine-grained learning signals. Here, we explore the effect of only using the self-rewarding scores for RL. Specifically, taking Open-Reasoner-Zero-7B as the base model, we take the checkpoint after 200 training steps with our method as the starting point. We then continue RL training using only the self-rewarding score as the optimization signal for reasoning ability (while still using the rule-based rewards to optimize the self-rewarding scores). The dynamics of training rewards are shown in Figure 12. We observe that after an additional 60 steps, the training collapses. This indicates that using only self-rewarding scores for RL easily leads to training instability, and should be complemented with the rule-based rewards.


section: M THE VALUE RANGE OF THE OPTIMIZED SELF-REWARDING SCORE

text: In Table 7, we show the the range between the minimum and maximum self-rewarding scores over all solutions in each test set of each model. The results validate that after optimization, the self-rewarding score naturally falls within the target interval [0,1].


section: N Detailed Self-Verification Results

text: We report the detailed self-verification results of each model on self-generated solutions across all benchmarks in Table 8, including both overall accuracy and F1 score. Our method consistently yields significant improvements in model's self-rewarding and self-verification capabilities, while incurring only minimal additional computational cost.


text: Table 8: Detailed self-verification results. MATH500 AMC23 AIME24 AIME25 Olym. Method Acc. F1 Acc. F1 Acc. F1 Acc. F1 Acc. F1 OctoThinker-3B-Short-Base Base 60.2 22.3 52.3 11.2 - - - - 62.0 13.7 GRPO 58.2 56.9 66.7 47.3 - - - - 66.4 48.8 LaSeR 77.0 73.6 77.3 70.2 - - - - 80.3 73.6 - SWA 81.0 80.4 84.1 70.9 - - - - 83.5 66.0 Qwen2.5-7B-Base Base 45.0 36.4 30.7 30.8 24.5 27.6 28.2 32.9 33.8 36.9 GRPO 76.5 54.6 61.1 59.7 60.4 36.6 72.5 41.5 54.6 53.5 LaSeR 88.0 83.2 81.5 82.5 92.2 79.6 90.5 74.3 79.5 78.3 - SWA 87.8 79.7 79.6 80.2 94.3 81.3 92.2 74.9 83.9 83.3 Open-Reasoner-Zero-7B Base 79.6 26.7 66.6 51.3 39.6 45.9 47.6 55.2 55.2 37.5 GRPO 52.9 57.1 50.9 44.8 66.9 14.6 78.9 28.1 54.7 49.5 LaSeR 90.1 87.2 77.7 79.7 87.2 64.6 92.8 77.7 80.1 78.7 - SWA 89.0 87.5 76.2 77.7 87.7 63.3 93.6 77.3 80.2 77.9 Table 9: The average AUROC of reward scores on each benchmark. In calculation, we discard query groups in which all solutions are either correct or incorrect, as such cases provide no meaningful signal for assessing the effectiveness of the self-rewarding scores. To ensure reliability, we sample 32 times for each query. The generator is Open-Reasoner-Zero-7B-LaSeR.


text: Method MATH500 AMC23 AIME24 AIME25 Olym. Avg. Qwen2.5-Math-RM-72B (72B RM) 0.77 0.73 0.71 0.60 0.65 0.69 LaSeR (7B Self-Rewarding) 0.72 0.68 0.77 0.74 0.67 0.72 Table 10: ECE of reward scores on each benchmark (lower is better). To ensure reliability, we sample 32 times for each query. The generator is Open-Reasoner-Zero-7B-LaSeR.


text: Method MATH500 AMC23 AIME24 AIME25 Olym. Avg. Qwen2.5-Math-RM-72B (72B RM) 0.178 0.152 0.346 0.384 0.194 0.251 LaSeR (7B Self-Rewarding) 0.077 0.162 0.072 0.071 0.066 0.090 Table 11: The Spearman Rank Correlation between self-rewarding scores and response lengths on Open-Reasoner-Zero-7B-LaSeR. To ensure reliability, we sample 32 times for each query.


text: MATH500 AMC23 AIME24 AIME25 Olym. Avg. Spearman Rank Corr. -0.26 -0.34 -0.58 -0.52 -0.40 -0.42


section: O FULL RESULTS OF THE CALIBRATION ANALYSIS OF SELF-REWARDING SCORES

text: Here, we display the full results of the calibration analysis of self-rewarding scores made in Section 5.1. We first display the results of average AUROC score (the probability that, within the same query group, a correct solution receives a higher self-rewarding score than an incorrect one) of the self-rewarding scores with respect to correctness in Table 9. The results demonstrate the high discriminative power of the self-rewarding score between correct and incorrect solutions. The comparison results of Expected Calibration Error (ECE, the discrepancy between a self-rewarding score and its empirical accuracy) in Table 10 demonstrate that our method achieves better confidence calibration. Finally, we show the average Spearman Rank Correlation (the direction and strength of the monotonic relationship between two variables by computing the correlation between their ranked values) between self-rewarding scores and response lengths in Table 11. The results reveal that shorter responses generally tend to


text: Table 12: Cross-model verification results. Each row corresponds to the average verification F1 score of a given verifier, evaluated on solution sets generated by different generators, as indexed by the columns. Verifier ↓ Generator → vermer 4 OT-3B-Short-GRPO Qwen2.5-7B-GRPO ORZ-7B-LaSeR OT-3B-Short-GRPO 51.0 42.7 43.6 Qwen2.5-7B-GRPO 56.4 49.2 46.4 ORZ-7B-LaSeR 69.2 77.5 77.6 Table 13: Results of applying our method within PPO framework on Open-Reasoner-Zero-7B.


text: Re asoning . Accuracy / Self-V Verificati on F1 Sc ore Method MATH- 500 AMC- 23 AIME- 24 AIME- 25 Olym Bench Avg. MATH- 500 AMC- 23 AIME- 24 AIME- 25 Olym Bench Avg. Open-R easoner-2 Zero-7B Base 81.9 60.3 15.6 15.1 46.9 44.0 26.7 51.3 45.9 55.2 37.5 43.3 PPO 82.1 58.6 16.7 14.8 47.0 43.8 51.8 43.7 14.7 15.4 40.5 33.2 LaSeR PPO 82.9 61.2 15.4 15.7 47.9 44.6 85.6 80.0 76.1 81.6 78.3 80.3


text: receive higher self-rewarding scores, which is consistent with recent findings (Hassid et al., 2025; Yang et al., 2025c) that shorter CoTs on the same query are generally more preferable.


section: P Cross-Model Verification Results

text: Here, we explore the cross-model verification performance. In specific, we compare the verification F1 scores of three models—OctoThinker-3B-Short-GRPO, Open-Reasoner-Zero-7B-GRPO, and Open-Reasoner-Zero-7B-LaSeR—evaluated on each other's responses. The results are in Table 12. Each value in the table represents the average F1 score across all five benchmarks. Surprisingly, we find that our method not only enables effective self-rewarding, but also achieves high accuracy and F1 when evaluating the CoTs generated by other models, demonstrating strong generalization ability.


section: Q RESULTS OF APPLYING LASER WITHIN PPO

text: Here, we conduct additional experiments using PPO as the RL algorithm and explore the potential of applying our method within the PPO framework to validate the generality of our method. We conduct experiments on Open-Reasoner-Zero-7B. The overall reasoning and self-verification results are presented in Table 13, and the inference-time scaling results are in Figure 13. We use "LaSeR PPO" to denote our method based on the PPO framework. As shown, our method also achieves strong effectiveness in enhancing the reasoning, self-verification, and inference-time scaling performance of the policy model within the PPO framework.


section: R THE GENERALIZABILITY OF LASER TO GENERAL REASONING DOMAIN

text: We conduct additional experiments to explore the generalizability of our method to general reasoning domain. We use a filtered version (Yu et al., 2025b) of WebInstruct-verified dataset (Ma et al., 2025), and conduct RL experiments on Qwen3-4B-Base (Yang et al., 2025a). We use the general-verifier-1.5B model from Ma et al. (2025) as the model-based verifier and adopt GRPO as the RL algorithm. The basic training and testing hyper-parameters for experiments on WebInstruct-verified are the same as those in Table 4 and Table 5, while the number of optimization steps here is 800. The simplified constant of the reference log-probability c_{\rm ref} is -23.0. For our method, we do not perform the advantage integration strategy here. The reason is that we observe the self-verification F1 score of our method during training is relatively low in the general reasoning setting (only between


text: Figure 13: The majority voting (Maj@K) and weighted majority voting (RM@K) results of Open-Reasoner-Zero-7B-PPO (ORZ-7B-PPO) and Open-Reasoner-Zero-7B-LaSeRPPO (ORZ-7B-LaSeRPPO) on MATH500 and OlympiadBench.


text: Figure 14: The generalizability of LaSeR on general reasoning tasks.


text: 65% and 70%, and the self-rewarding score distributions in the test sets shown in Figure 14(b) and Figure 14(c) also reveal this phenomenon). This leads to large noise in the self-rewarding-based advantage estimation, and consequently, the integration of self-rewarding-based advantages results in performance degradation. We conduct evaluations on two general reasoning benchmarks: MMLU-Pro (Wang et al., 2024b) and GPQA-Diamond (Rein et al., 2024). We sample 4 solutions per problem on each dataset for each model, and calculate both the average accuracy and the (weighted) majority voting accuracy.


text: We have several findings: (1) First, from Figure 14(a), we observe that jointly optimizing the self-rewarding capability does not impact the model's general reasoning ability, allowing the policy model to achieve comparable average reasoning accuracy to the baseline. (2) However, as mentioned above, the optimized self-rewarding score on general reasoning tasks does not achieve the high accuracy seen in math reasoning tasks. We can see that the self-rewarding score distributions for correct and incorrect solutions on MMLU-Pro exhibit certain overlap, and the distinction further diminishes on the more challenging benchmark GPQA-Diamond. We speculate that two factors may contribute to this: (a) The model's general reasoning ability is inherently weaker than its math reasoning ability, which limits the upper bound of its self-rewarding capabilities in the general reasoning domain. (b) The model-based verifier used in the experiment (general-verifier-1.5B) has limited verification ability, resulting in high noise in the reasoning rewards, which in turn affects the optimization of the self-rewarding capability. (3) Though not perfect, the optimized self-rewarding scores can still provide useful signals during inference time, leading to better weighted majority voting results. To examine this effect, we analyze the distribution of self-rewarding scores across solutions generated


text: Table 14: The average self-rewarding AUROC score of Qwen3-4B-LaSeR to assess how well the self-rewarding score separates correct from incorrect solutions. In calculation, we discard query groups in which all solutions are either correct or incorrect, as such cases provide no meaningful signal for assessing the effectiveness of the self-rewarding scores. To ensure reliability, here, we sample 32 times for each query. For computational efficiency, we evaluate on 1000 samples (Yu et al., 2025b) from MMLU-Pro. MMLU-Pro-1000 GPQA-D AUROC 0.61 0.58


text: for the same query. We calculate and report the average AUROC score, which is the probability that a correct solution sampled from the same query group (solutions generated for the same query) receives a higher self-rewarding score than an incorrect one. The results are reported in Table 14. We find that the AUROC scores are around 0.6, indicating that within the same query group, the self-rewarding scores assigned to correct solutions are generally higher than those assigned to incorrect ones. This reveals that, though not perfect, the self-rewarding signal can still effectively support solution re-ranking, thereby improving the performance of weighted majority voting.


section: S FURTHER REDUCTION OR INCREASE OF SELF-REWARDING COST

text: In this section, we discuss two additional variants of LaSeR for future work. In the current method, we calculate the last-token self-rewarding score based on the next-token log-probability distribution of the "<EOS>" token, requiring one additional token inference compared with standard inference. One potential way to further reduce the inference cost of LaSeR is to derive the last-token selfrewarding score directly from the predicted log-probability of pre-specified token z_c at the "<EOS>" token position. Specifically, let y_T denote the "<EOS>" token in y. Then, the reduced last-token self-rewarding score can be defined as r_s = \beta_v \log \pi_{\theta}(z_c | x, y_{< T}) - \beta_v c_{\text{ref}} , as we have observed that \pi_{\text{ref}}(z_c|x,y_{< T}) remains nearly constant across (x,y) (e.g., approximately e^{-28} for Qwen2.5-7B-Base). In this case, we can achieve ideally zero additional inference cost for self-rewarding compared with standard generation by directly calculating the self-rewarding score from the log-probability distribution at the "<EOS>" token position, without requiring any extra token inference. In theory, this works because setting a relatively large \beta_v still yields a small value of \pi_{\theta}(z_c|x,y_{< T}) (e.g., \pi_{\theta}(z_c|\mathbf{x}, \mathbf{y}_{\leq T}) = e^{-18} when \bar{\beta}_v = 0.1 and c_{\text{ref}} = -28 ), thereby allowing \pi_{\theta}(\leq |\mathbf{x}, \mathbf{y}_{\leq T}) to still dominate the probability mass. However, although the probability is very low, we observe that the generator may still select z_c at the end of the sequence in few cases during training, which can adversely affect training stability as the generator continues to generate after z_c . One straight-forward solution may be to set the sampling hyper-parameter top_p to a value less than 1.0. Future work can further investigate advanced strategies to make the above adjustment more principled and robust.


text: While reducing the self-rewarding cost improves efficiency, an alternative is to increase the inference cost in exchange for stronger self-rewarding capability . That is, instead of computing the self-rewarding score based on the log-probability distribution of a single token only, we can increase the number of additional inference tokens by calculating it over M tokens as r_s = \beta_v \sum_{m=1}^M \log \pi_{\theta}(z_c|\mathbf{x},\mathbf{y},z_c,\cdots,z_c)) - M\beta_v c_{\text{ref}} . It is a promising direction for future research to


text: explore whether increasing the number of additional inference tokens can yield positive inference-time scaling effect for latent self-rewarding capability.


section: T PROMPT TEMPLATES

text: We show the training, evaluation and self-verification prompt templates used in our experiments in the end.


section: Training and Evaluation Prompt Template for OctoThinker-3B-Short-Base

text: <bos token> A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.


text: User: You must put your answer inside \boxed{} and Your final answer will be extracted automatically by the \boxed{} tag.


text: {question}


text: Assistant:


section: Training Prompt Template for Qwen2.5-7B-Base

text: <bos token> A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within <think> </think> and answer is enclosed within <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.


text: User: You must put your answer inside <answer> </answer> tags, i.e., <answer> answer here </answer>. And your final answer will be extracted automatically by the \boxed{} tag. This is the problem:


text: {question}


text: Assistant: <think>


section: Zero-Shot Evaluation Prompt Template for Qwen2.5-7B-Base

text: < ∣im start∣ >system


text: You are a helpful assistant.< ∣im end∣ >


text: < ∣im start∣ >user


text: {question}


text: Please reason step by step, and put your final answer within \boxed{}.< ∣im end∣ >


text: < ∣im start∣ >assistant


section: Training and Evaluation Prompt Template for Open-Reasoner-Zero-7B

text: A conversation between User and Assistant. The User asks a question, and the Assistant solves it. The Assistant first thinks about the reasoning process in the mind and then provides the User with the answer. The reasoning process is enclosed within <think> </think> and answer is enclosed within <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.


text: User: You must put your answer inside <answer> </answer> tags, i.e., <answer> answer here </answer>. And your final answer will be extracted automatically by the \boxed{} tag. {question}


text: Assistant: <think>


section: Training and Evaluation Prompt Template for Qwen3-4B-Base

text: < ∣im start∣ >user


text: {question}


text: < ∣im start∣ >assistant


section: Prompt Template for Self-Verification (Modified from Liu et al. (2025a) )

text: Below you are presented with a question and a tentative response. Your task is to evaluate the response and assign a rating to the response based on the following clear criteria: Rating Criteria:


text: 1. Missing final answer, or incorrect response with the wrong final answer: assign \boxed{0}. 2. Correct response with the correct final answer: assign \boxed{1}.


text: ### Question Begin ###


text: {question}


text: ### Question End ###


text: ### Response Begin ###


text: {response}


text: ### Response End ###


text: First provide your evaluation process, then clearly state your final rating value enclosed in \boxed{} at the end.