sk] ) else: loss = logits.sum() * 0 return loss, logits The core loss function looks very similar to outcome reward models, with the labels being applied at different intervals. # Assume model outputs 3-class logits per token hidden = model.lm(**inputs, output_hidden_states=True).hidden_states[-1] logits = model.head(hidden) # (batch, seq_len, 3) # 3-class labels at step boundaries only: 0=-1, 1=0, 2=1 (others masked as -100) mask = labels != -100 loss = F.cross_entropy(logits[mask], labels[mask]) rlhfbook.com 46


5.7 Comparing Reward Model Types (and Value Functions) The various types of reward models covered indicate the spectrum of ways that “quality” can be measured in RLHF and other post-training methods. Below is a summary of what the models predict and how they are trained. Table 2: Comparing types of reward models. Model Class What They Predict How They Are Trained LM structure Reward Models Sequence-level quality score rθ(x, y) Contrastive loss between pairwise (or N-wise) comparisons between completions Linear head on EOS/last-token hidden state Outcome Reward Models Probability that an answer is correct per-token Labeled outcome pairs (e.g., success/failure on verifiable domains) Per-token binary cross-entropy head; labels repeat the outcome label Process Reward Models A reward or score f


binary cross-entropy head; labels repeat the outcome label Process Reward Models A reward or score for intermediate steps at end of reasoning steps Trained using intermediate feedback or stepwise annotations (trained per token in reasoning step) Per-token head predicting step correctness (-1, 0, 1) Value Functions The expected return given the current state Trained via regression to each point in sequence A scalar regression head with per-token outputs A few caveats on the distinctions in this table, as the boundaries between model types are not always clear cut: • Both in preference tuning and reasoning training, the value functions often have a discount factor of 1, which makes a value function even closer to an outcome reward model, but with a different training loss.


makes a value function even closer to an outcome reward model, but with a different training loss. • A process reward model can be supervised by doing rollouts from an intermediate state and collecting outcome data. This blends multiple ideas, but if the loss uses per-reasoning-step labels, it is best referred to as a PRM. What if you train a Bradley-Terry pairwise model with correct/incorrect pairs? Much of the confusion on outcome reward models came from a small set of the literature that was training a reward model on pairwise data derived from answer correctness. In this domain, you set the chosen response as being a correct answer to a problem and a rejected response as being an incorrect answer for the same problem.


rect answer to a problem and a rejected response as being an incorrect answer for the same problem. This is technically not an ORM and still trained directly with the contrastive, sequence-level loss. This is technically still a Bradley-Terry model and would fall in the first class of models we covered. ORM vs. Value Function. ORMs and value functions can appear similar since both produce per-token outputs with the same head architecture, but they differ in what they predict and where targets come from: rlhfbook.com 47


• ORMs predict an immediate, token-local quantity: p(correctt) or rt. Targets come from offline labels (a verifier or dataset marking tokens/sequences as correct or incorrect). • Value functions predict the expected remaining return: V (st) = E[P k≥t γk−trk | st]. Targets are typically computed from on-policy rollouts under the current policy πθ, and change as the policy changes (technically, value functions can also be off-policy, but this is not established for work in language modeling). If you define a dense token reward rt = ⊮[token is correct] and use γ = 1, then an ORM is learning rt (or p(rt = 1)) while the value head is learning the remaining-sum P k≥t rk. They can share the same base model and head dimensions, but the semantics and supervision pipeline differ: ORMs are trained of


se model and head dimensions, but the semantics and supervision pipeline differ: ORMs are trained offline from fixed labels, while value functions are trained on-policy and used to compute advantages At = ˆRt −Vt for policy gradients. 5.7.1 Inference Across Reward Model Types The models handle data differently at inference time (once they’ve been trained), in order to handle a suite of tasks that RMs are used for. Bradley-Terry RM (Preference Model): • Input: prompt x + candidate completion y • Output: single scalar rθ(x, y) via a linear layer from the EOS/last-token hidden state • Usage: rerank k completions, pick top-1 (best-of-N sampling); or provide terminal reward for RLHF • Aggregation: Not needed with scalar outputs Outcome RM: • Input: prompt x + completion y • Output: per-token pr


: Not needed with scalar outputs Outcome RM: • Input: prompt x + completion y • Output: per-token probabilities pt ≈P(correct at token t) over completion tokens • Usage: score finished candidates; aggregate via mean, min (tail risk), or product Q t pt (equivalently, sum log-probabilities P t log pt) • Aggregation choices: mean correctness, minimum pt, average over last m tokens, or threshold flagging if any pt < τ Process RM: • Input: prompt x + reasoning trace with step boundaries • Output: scores at step boundaries (e.g., class logits for correct/neutral/incorrect) • Usage: score completed chain-of-thought; or guide search/decoding by pruning lowscoring branches • Aggregation: over steps (not tokens) — mean step score, minimum (fail-fast), or weighted sum favoring later steps Value Funct


not tokens) — mean step score, minimum (fail-fast), or weighted sum favoring later steps Value Function: • Input: prompt x + current prefix y≤t (a state) • Output: Vt at each token position in the completion (expected remaining return from state t) • Usage: compute per-token advantages At = ˆRt −Vt during RL training; the values at each step serve as baselines • Aggregation: typically take V at the last generated token; interpretation differs from “probability of correctness” rlhfbook.com 48


In summary, the way to understand the different models is: • RM: “How good is this whole answer?” →scalar value • ORM: “Which parts look correct?” →per-token correctness • PRM: “Are the reasoning steps sound?” →per-step scores • Value: “How much reward remains from here?” →baseline for RL advantages 5.8 Generative Reward Modeling (a.k.a. LLM-as-a-judge) With the cost of preference data, a large research area emerged to use existing language models as a judge of human preferences or in other evaluation settings [78]. The core idea is to prompt a language model with instructions on how to judge, a prompt, and two completions (much as would be done with human labelers). An example prompt, from one of the seminal works here for the chat evaluation MT-Bench [78], follows: [System] Please act as


one of the seminal works here for the chat evaluation MT-Bench [78], follows: [System] Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation.


not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, and "[[C]]" for a tie. [User Question] {question} [The Start of Assistant A's Answer] {answer_a} [The End of Assistant A's Answer] [The Start of Assistant B's Answer] {answer_b} [The End of Assistant B's Answer] Given the efficacy of LLM-as-a-judge for evaluation, which spawned many other evaluations such as AlpacaEval [79], Arena-Hard [80], and WildBench [81], many began using LLM-as-ajudge instead of reward models to create and use preference data.


ch [81], many began using LLM-as-ajudge instead of reward models to create and use preference data. An entire field of study has emerged around how to use so-called “Generative Reward Models” [82] [83] [84] (including models trained specifically to be effective judges [85]), but on RM evaluations they tend to be behind existing reward models, showing that reward modeling is an important technique for current RLHF. A common trick to improve the robustness of LLM-as-a-judge workflows is to use a sampling temperature of 0 to reduce variance of ratings. rlhfbook.com 49


5.9 Further Reading The academic literature for reward modeling established itself in 2024. The bulk of early progress in reward modeling has focused on establishing benchmarks and identifying behavior modes. The first RM benchmark, RewardBench, provided common infrastructure for testing reward models [86]. Since then, RM evaluation has expanded to be similar to the types of evaluations available to general post-trained models, where some evaluations test the accuracy of prediction on domains with known true answers [86] or those more similar to “vibes” performed with LLM-as-a-judge or correlations to other benchmarks [87]. Examples of new benchmarks include: • Text-only (general chat / preferences): RMB [88], RewardBench2 [89], Preference Proxy Evaluations [90], or RM-Bench [91].


t / preferences): RMB [88], RewardBench2 [89], Preference Proxy Evaluations [90], or RM-Bench [91]. • Specialized text-only (math, etc.): multilingual reward bench (M-RewardBench) [92], RAG-RewardBench for retrieval augmented generation (RAG) [93], ReWordBench for typos [94], RewardMATH [95], or AceMath-RewardBench [96]. • Process RMs: PRM Bench [97] or ProcessBench [98] and visual benchmarks of VisualProcessBench [99] or ViLBench [100]. • Agentic RMs: Agent-RewardBench [101] or CUARewardBench [102]. • Multimodal: MJ-Bench [103], Multimodal RewardBench [104], VL RewardBench [105], or VLRMBench [106]. To understand progress on training reward models, one can reference new reward model training methods, with aspect-conditioned models [107], high-quality human datasets [108] [109], scaling ex


g methods, with aspect-conditioned models [107], high-quality human datasets [108] [109], scaling experiments [30], extensive experimentation [49], or debiasing data [110]. 5.10 Suggested Experiments The companion code repository includes small reward model training scripts in code/reward_models/. These are intended as learning exercises rather than tuned reference recipes. Start from a clean code/ environment with uv sync, then run one experiment at a time. 1. Train a Bradley-Terry preference reward model on UltraFeedback. Run: cd code/ uv run python -m reward_models.train_preference_rm --samples 2000 --epochs 1 Watch whether the reward margin between chosen and rejected responses grows in the demo and W&B logs.


tch whether the reward margin between chosen and rejected responses grows in the demo and W&B logs. Then vary --samples, --lr, and --model-id to see when the signal becomes noisy or unstable. 2. Compare outcome and process supervision. Run the GSM8K outcome reward model and the PRM800K process reward model: cd code/ uv run python -m reward_models.train_orm --samples 400 --epochs 2 uv run python -m reward_models.train_prm --samples 500 --epochs 2 Compare what each model can score after training: the ORM should distinguish correct and incorrect final answers, while the PRM should assign scores across intermediate rlhfbook.com 50


reasoning steps. This is the practical version of the distinction between sequence-level, outcome-level, and process-level supervision. 3. Add a small held-out reward model eval. A useful contribution is a 50- to 200-example evaluation for reward_models/ that reports accuracy or preference-pair ordering without requiring a full training run. Keep the evaluation small enough that it can be used while tuning hyperparameters. rlhfbook.com 51


6 Reinforcement Learning In the RLHF process, the reinforcement learning algorithm slowly updates the model’s weights with respect to feedback from a reward model. The policy – the model being trained – generates completions to prompts in the training set, then the reward model scores them, and the reinforcement learning optimizer takes gradient steps based on this information (see fig. 16 for an overview). This chapter explains the mathematics and trade-offs across various algorithms used to learn from the signal the reward model gives to on-policy data. These algorithms are run for a period of many epochs, often thousands or millions of batches across a larger set of prompts, with gradient updates in between each of them.


millions of batches across a larger set of prompts, with gradient updates in between each of them. 6.1 The Role of Reinforcement Learning in RLHF The algorithms that popularized RLHF for language models were policy-gradient reinforcement learning algorithms. These algorithms, such as Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), and REINFORCE, use recently generated samples to update their model (rather than storing scores in a replay buffer like algorithms, e.g. Deep Q-Networks, DQN, used in popular projects such as AlphaGo). In this section we will cover the fundamentals of the policy gradient algorithms and how they are used in the modern RLHF framework. At a machine learning level, this section is the subject with the highest complexity in the RLHF pro


At a machine learning level, this section is the subject with the highest complexity in the RLHF process. However, as with most modern AI models, the largest determining factor in its success is the data provided as inputs to the process. When RLHF came onto the scene with ChatGPT, it was largely known that they used a variant of PPO, and many initial efforts were built upon that. Over time, multiple research projects showed the promise of REINFORCE-style algorithms [111] [109], touted for their simplicity over PPO without a separate value model (saves memory and therefore the number of GPUs required) and with simpler advantage estimation (no Generalized Advantage Estimation, GAE, which is a method to compute advantages used for variance reduction in policy gradient algorithms).


which is a method to compute advantages used for variance reduction in policy gradient algorithms). More algorithms have emerged, including Group Relative Policy Optimization, which is particularly popular with reasoning tasks, but in general many of these algorithms can be tuned to fit a specific task. In this chapter, we cover the core policy gradient setup and the three algorithms mentioned above due to their central role in the establishment of a canonical RLHF literature. At its simplest, the RL stage of RLHF requires two models: a policy (the model being trained) and a reward model that scores its outputs (as covered in the previous chapter). A copy of the policy before RL serves as the reference model for computing a KL penalty (this model is frozen, i.e.


licy before RL serves as the reference model for computing a KL penalty (this model is frozen, i.e. it is not updated with gradients from the automatic differentiation engine). The most complex algorithm covered here, PPO, adds a fourth model – a learned value function used to estimate how good each token in the action was, also a large language model updated during training. The algorithms in this chapter differ mainly in how they estimate a quantity called advantages – a measure of how good the current action (completion) from the model is relative to average – and how they constrain policy updates so the optimization is numerically stable. A visual overview of this RLHF process (without the value model) is shown in fig. 16. For definitions of symbols, see the problem setup chapter.


ut the value model) is shown in fig. 16. For definitions of symbols, see the problem setup chapter. rlhfbook.com 52


Figure 16: Overview of the RLHF training loop. A prompt from the dataset is passed to the tuned policy, which generates a completion. The reward model scores this completion, while the frozen initial model (typically the instruction-tuned model before RL) computes log probabilities on the same text to calculate a KL penalty that prevents excessive drift. The combined reward signal then drives a reinforcement learning update to the policy parameters. rlhfbook.com 53


This chapter uses (s, a) notation from the reinforcement learning literature, where s denotes states and a denotes actions. In the language model context, you will often see (x, y) instead, where x is the prompt and y is the completion. The (s, a) framing is more general—these algorithms were designed for sequential decision problems where actions are taken at each timestep. However, many RLHF implementations treat the entire completion as a single action, making the (x, y) notation equally valid. RL Cheatsheet: A one-page reference of all core RL loss functions from this chapter is available at rlhfbook.com/rl-cheatsheet. 6.2 Policy Gradient Algorithms At its core, this chapter is dedicated to understanding the following shape of equation.


Algorithms At its core, this chapter is dedicated to understanding the following shape of equation. This equation is computing the gradient, ∆θ, to the language model we are training, πθ: ∆θ ∝Ψt ∇θ log πθ(at | st) (23) Here, the equation is composed of two key components: 1. ∇θ log πθ(at | st) — which direction in parameter space makes action at more likely. 2. Ψt — how good was it? A scalar scoring the outcome. When you put this together, yes, by multiplying the quantities, you get the policy gradient update. Some things are simple, such as that Ψt > 0 updates parameters to make at more likely, Ψt < 0 updates them to make it less likely. The policy gradient is computing which parameters contribute to an action and if we should make it more or less likely to occur in the future.


meters contribute to an action and if we should make it more or less likely to occur in the future. The rest of this chapter goes very deep on the different ways to do this, and what the specific tricks are to make it work for LLMs. Now, let us formalize this a bit further. Reinforcement learning algorithms are designed to maximize the future, discounted reward across a trajectory of states, s ∈S, and actions, a ∈A (for more notation, see Appendix A, Definitions). The objective of the agent, often called the return, is the sum of discounted future rewards (where γ ∈[0, 1] is a factor that prioritizes near-term rewards) at a given time t: Gt = Rt+1 + γRt+2 + · · · = ∞ X k=0 γkRt+k+1. (24) The return definition can also be estimated as: Gt = γGt+1 + Rt+1.


+ · · · = ∞ X k=0 γkRt+k+1. (24) The return definition can also be estimated as: Gt = γGt+1 + Rt+1. (25) This return is the basis for learning a value function V (s) that is the estimated future return given a current state: V (s) = E  Gt|St = s  . (26) All policy gradient algorithms optimize a policy πθ(a | s) to maximize expected return; this objective can be expressed using the induced value function V πθ(s). rlhfbook.com 54


Where dπθ(s) is the state-visitation distribution induced by policy πθ(a | s), the objective we maximize can be written as: J(θ) = X s dπθ(s)V πθ(s), (27) In a finite MDP this is a sum over all states, but in practice we never compute it exactly. Instead, we estimate it from data by sampling rollouts from the current policy. In RLHF this typically means sampling prompts xi from a dataset and generating completions yi ∼ πθ(· | xi), then taking an empirical average such as: ˆJ(θ) = 1 B B X i=1 R(xi, yi), (28) or, in an MDP view with per-step rewards, ˆJ(θ) = 1 B B X i=1 Ti X t=0 γtri,t. (29) In practice, RLHF for language models sets γ = 1 (no discounting) because the unit of optimization is the collective completion, not individual tokens – this choice is discussed further in the MDP vs.


the collective completion, not individual tokens – this choice is discussed further in the MDP vs. Bandit section later in this chapter. The core of policy gradient algorithms is computing the gradient with respect to the finitetime expected return over the current policy. With this expected return, J, the parameter update can be computed as follows, where α is the learning rate: θ ←θ + α∇θJ(θ) (30) The core implementation detail is how to compute said gradient. 6.2.1 Deriving the Policy Gradient Another way to pose the RL objective we want to maximize is as follows: J(θ) = Eτ∼πθ [R(τ)] , (31) where τ = (s0, a0, s1, a1, . . .) is a trajectory and R(τ) = P∞ t=0 rt is the total reward of the trajectory.


(s0, a0, s1, a1, . . .) is a trajectory and R(τ) = P∞ t=0 rt is the total reward of the trajectory. Alternatively, we can write the expectation as an integral over all possible trajectories: J(θ) = Z τ pθ(τ)R(τ)dτ (32) Notice that we can express the trajectory probability as follows, where πθ(at|st)p(st+1|st, at) is the transition probability to a group of next states from one state and action: pθ(τ) = p(s0) ∞ Y t=0 πθ(at|st)p(st+1|st, at), (33) rlhfbook.com 55


If we take the gradient of the objective (eq. 31) with respect to the policy parameters θ: ∇θJ(θ) = Z τ ∇θpθ(τ)R(τ)dτ (34) Notice that we can use the log-derivative trick in order to rewrite the gradient of the integral as an expectation: ∇θ log pθ(τ) = ∇θpθ(τ) pθ(τ) (from chain rule) =⇒∇θpθ(τ) = pθ(τ)∇θ log pθ(τ) (rearranging) (35) Using this log-derivative trick: ∇θJ(θ) = Z τ ∇θpθ(τ)R(τ)dτ = Z τ pθ(τ)∇θ log pθ(τ)R(τ)dτ = Eτ∼πθ [∇θ log pθ(τ)R(τ)] (36) Where the final step uses the definition of an expectation under the trajectory distribution pθ(τ): for any function f, Eτ∼pθ[f(τ)] = R τ f(τ) pθ(τ) dτ (or a sum in the discrete case). Writing it as an expectation is useful because we can approximate it with Monte Carlo rollouts, e.g., 1 B PB i=1 f(τi) for trajectories τi ∼πθ.


se we can approximate it with Monte Carlo rollouts, e.g., 1 B PB i=1 f(τi) for trajectories τi ∼πθ. Back to the derivation, expanding the log probability of the trajectory: log pθ(τ) = log p(s0) + ∞ X t=0 log πθ(at|st) + ∞ X t=0 log p(st+1|st, at) (37) Now, if we take the gradient of the above, we get: • ∇θ log p(s0) = 0 (initial state doesn’t depend on θ) • ∇θ log p(st+1|st, at) = 0 (environment transition dynamics don’t depend on θ) • only ∇θ log πθ(at|st) survives Therefore, the gradient of the log probability of the trajectory simplifies to: ∇θ log pθ(τ) = ∞ X t=0 ∇θ log πθ(at|st) (38) Reaching this equation is a crucial point in the implementation. Here, we have gone far enough to see that the gradient of the trajectory distribution reduces to a sum of gradients from language model po


hat the gradient of the trajectory distribution reduces to a sum of gradients from language model policy probabilities (which are just the probabilities of tokens given by the model we’re training). In practice, this results in a common form of the policy gradient equations. They end up looking like a sum of log-probabilities in the loss, and then we compute the gradients via autodiff. A short snippet you’ll see again and again roughly follows: rlhfbook.com 56


seq_log_probs = (token_log_probs * completion_mask).sum(dim=-1) loss = -(seq_log_probs * advantages).mean() loss.backward() You’ll see this throughout the chapter. Now, back to the formal policy gradient mathematics. Substituting this back in eq. 36, we get: ∇θJ(θ) = Eτ∼πθ " ∞ X t=0 ∇θ log πθ(at|st)R(τ) # (39) Quite often, people use a more general formulation of the policy gradient: g = ∇θJ(θ) = Eτ∼πθ " ∞ X t=0 ∇θ log πθ(at|st)Ψt # (40) Where Ψt can be the following (where the rewards can also often be discounted by γ), a taxonomy adopted from Schulman et al. 2015 [112]: 1. R(τ) = P∞ t=0 rt: total reward of the trajectory. 2. P∞ t′=t rt′: reward following action at, also described as the return, G. 3. P∞ t′=t rt′ −b(st): baselined version of previous formula. 4.


, also described as the return, G. 3. P∞ t′=t rt′ −b(st): baselined version of previous formula. 4. Qπ(st, at): state-action value function. 5. Aπ(st, at): advantage function, which yields the lowest possible theoretical variance if it can be computed accurately. 6. rt + γV π(st+1) −V π(st): Temporal Difference (TD) residual. The baseline is a value used to reduce variance of policy updates (more on this below). For language models, some of these concepts do not make as much sense. For example, for a deterministic policy π the state value is V π(st) = Qπ(st, π(st)) (and for the optimal value function one has V ∗(st) = maxat Q∗(st, at)). For a stochastic policy, the analogous identity is V π(st) = Eat∼π(·|st)[Qπ(st, at)].


Q∗(st, at)). For a stochastic policy, the analogous identity is V π(st) = Eat∼π(·|st)[Qπ(st, at)]. The Bellman equation relates Q to V: in general Qπ(st, at) = E[rt + γV π(st+1) | st, at], but for language models where state transitions are deterministic, this simplifies to Q(st, at) = rt + γV (st+1). The advantage function measures how much better action at is compared to the average: A(st, at) = Q(st, at) −V (st) = rt + γV (st+1) −V (st) (41) This final form is exactly the temporal difference (TD) residual (item 6 above) – a fundamental quantity in RL that measures the gap between the value function’s prediction and what actually occurred, driving value function updates toward more accurate estimates.


ediction and what actually occurred, driving value function updates toward more accurate estimates. In practice, a learned value function ˆV is used to estimate the advantage via this TD error. 6.2.2 Vanilla Policy Gradient The vanilla policy gradient implementation optimizes the above expression for J(θ) by differentiating with respect to the policy parameters. A simple version, with respect to the overall return, is: rlhfbook.com 57


∇θJ(θ) = Eτ " T X t=0 ∇θ log πθ(at|st)Rt # (42) A common problem with vanilla policy gradient algorithms is the high variance in gradient updates, which can be mitigated in multiple ways. The high variance comes from the gradient updates being computed by estimating the return G from an often small set of rollouts in the environment that tend to be susceptible to noise (e.g. the stochastic nature of generating from language models with temperature > 0). The variance across return estimates is higher in domains with sparse rewards, as more of the samples are 0 or 1, rather than closely clustered. In order to alleviate this, various techniques are used to normalize the value estimation, called baselines.


to alleviate this, various techniques are used to normalize the value estimation, called baselines. Baselines accomplish this in multiple ways, effectively normalizing by the value of the state relative to the downstream action (e.g. in the case of Advantage, which is the difference between the Q value and the value). The simplest baselines are averages over the batch of rewards or a moving average. Even these action-independent baselines can reduce variance without changing the expected gradient, since Ea∼π(a|s)[b(s)∇θ log πθ(a|s)] = 0 for any state-dependent b(s), improving the learning signal substantially. Many of the policy gradient algorithms discussed in this chapter build on the advantage formulation of policy gradient: ∇θJ(θ) = Eτ " T X t=0 ∇θ log πθ(at|st)Aπθ(st, at) # (43) 6.2.3


ntage formulation of policy gradient: ∇θJ(θ) = Eτ " T X t=0 ∇θ log πθ(at|st)Aπθ(st, at) # (43) 6.2.3 REINFORCE The algorithm REINFORCE is likely a backronym, but the components of the algorithm it represents are quite relevant for modern reinforcement learning algorithms. Defined in the seminal paper Simple statistical gradient-following algorithms for connectionist reinforcement learning [113]: The name is an acronym for “REward Increment = Nonnegative Factor X Offset Reinforcement X Characteristic Eligibility.” The three components of this are how to do the reward increment, a.k.a. the policy gradient step. It has three pieces to the update rule: 1. Nonnegative factor: This is the learning rate (step size) that must be a positive number, e.g. α below. 2.


tive factor: This is the learning rate (step size) that must be a positive number, e.g. α below. 2. Offset Reinforcement: This is a baseline b or other normalizing factor of the reward to improve stability. 3. Characteristic Eligibility: This attributes the scalar reward signal to the parameters that produced the action. Williams denotes this eligibility term as e (not the exponential function). In modern policy-gradient notation, it corresponds to ∇θ log πθ(at | st). Thus, the form looks quite familiar: ∆θ = α(r −b)e (44) rlhfbook.com 58


With more modern notation and the generalized return G, the REINFORCE operator appears as: ∇θ J(θ) = Eτ∼πθ h T X t=0 ∇θ log πθ(at | st) (Gt −b(st)) i , (45) Here, the value Gt −b(st) is the advantage of the policy at the current state, so we can reformulate the policy gradient in a form that we continue later with the advantage, A: ∇θ J(θ) = Eτ∼πθ h T X t=0 ∇θ log πθ(at | st) At i , (46) REINFORCE is a specific implementation of vanilla policy gradient that uses a Monte Carlo estimator of the gradient. Figure 17: Basic REINFORCE architecture for language models. The shaped reward combines the reward model score with a KL penalty from the reference model. We build on this structure throughout the chapter.


core with a KL penalty from the reference model. We build on this structure throughout the chapter. 6.2.4 REINFORCE Leave One Out (RLOO) The core implementation detail of REINFORCE Leave One Out versus standard REINFORCE is that it takes the average reward of the other samples in the batch to compute the baseline – rather than averaging over all rewards in the batch [114], [111], [115]. By excluding the current sample’s reward from its own baseline, the RLOO baseline is independent of the action being evaluated, which keeps the gradient estimator exactly unbiased. Crucially, this only works when generating multiple trajectories (completions) per state (prompt), which is common practice in multiple domains of fine-tuning language models with RL.


tate (prompt), which is common practice in multiple domains of fine-tuning language models with RL. Specifically, for the REINFORCE Leave-One-Out (RLOO) baseline, given K sampled trajectories (actions taken conditioned on a prompt) a1, . . . , aK, to a given prompt s we define the baseline explicitly as the following per-prompt: rlhfbook.com 59


b(s, ak) = 1 K −1 K X i=1,i̸=k R(s, ai), (47) resulting in the advantage: A(s, ak) = R(s, ak) −b(s, ak). (48) Equivalently, this can be expressed as: A(s, ak) = K K −1 R(s, ak) −1 K K X i=1 R(s, ai) ! . (49) This is a simple, low-variance per-prompt advantage estimate that is closely related to the group-relative advantage used in Group Relative Policy Optimization, GRPO (discussed shortly, after Proximal Policy Optimization, PPO). In practice, GRPO-style training mainly differs in how it applies the KL regularizer (as an explicit loss term vs. folded into the reward) and whether it uses PPO-style ratio clipping. To be specific, the canonical GRPO implementation applies the KL penalty at the loss level, whereas the derivation for RLOO or traditional policy-gradients applies the KL penalty


loss level, whereas the derivation for RLOO or traditional policy-gradients applies the KL penalty to the reward itself. With the transition from RLHF to reasoning and reinforcement learning with verifiable rewards (RLVR), the prevalence of KL penalties has decreased overall, with many reasoning adaptations of RLHF code turning them off entirely. Still, the advantage from RLOO could be combined with the clipping of PPO, showing how similar many of these algorithms are. RLOO and other algorithms that do not use a value network – an additional model copy (a critic) that predicts a scalar value V (st) per token – assign the same sequence-level advantage (or reward) to every token when computing the loss.


oken – assign the same sequence-level advantage (or reward) to every token when computing the loss. Algorithms that use a learned value network, such as PPO, assign a different value to every token individually, discounting from the final reward achieved at the EOS token. With a KL distance penalty, RLOO aggregates the per-token KL over the completion and folds that scalar into the sequence reward, so the resulting advantage is broadcast to all tokens. PPO subtracts a per-token KL from the per-token reward before computing At, giving token-level credit assignment. GRPO typically retains a sequence-level advantage but adds a separate per-token term to the loss, rather than subtracting it from the reward. These details and trade-offs are discussed later in the chapter.


an subtracting it from the reward. These details and trade-offs are discussed later in the chapter. 6.2.5 Proximal Policy Optimization (PPO) Proximal Policy Optimization (PPO) [116] is one of the foundational algorithms behind Deep RL’s successes (such as OpenAI’s Five, which mastered DOTA 2 [117] and large amounts of research). The objective that PPO maximizes, with respect to the advantages and the policy probabilities, is as follows: J(θ) = min  πθ(a|s) πθold(a|s)A, clip  πθ(a|s) πθold(a|s), 1 −ε, 1 + ε  A  . (50) rlhfbook.com 60


Figure 18: REINFORCE Leave-One-Out (RLOO) architecture. Multiple completions per prompt provide a leave-one-out baseline for advantage estimation without learning a value function. Here, πθ(a|s) is the current policy being optimized and πθold(a|s) is the policy that was used to collect the training data (i.e., the policy from the previous iteration). The ratio between these two policies emerges from importance sampling, which allows us to reuse data collected under an old policy to estimate gradients for a new policy. Recall from the advantage formulation of the policy gradient (eq. 43) that we have: ∇θJ(θ) = Eτ∼πθ " T X t=0 ∇θ log πθ(at|st)Aπθ(st, at) # . (51) This expectation is taken over trajectories sampled from πθ, but in practice we want to take multiple gradient steps on a batch of


trajectories sampled from πθ, but in practice we want to take multiple gradient steps on a batch of data that was collected from a fixed policy πθold. To correct for this distribution mismatch, we multiply by the importance weight πθ(a|s) πθold(a|s), which reweights samples to account for how much more or less likely they are under the current policy versus the data-collection policy. Without constraints, optimizing this importanceweighted objective can lead to destructively large policy updates when the ratio diverges far from 1. PPO addresses this by clipping the ratio to the range [1 −ε, 1 + ε], ensuring that the policy cannot change too drastically in a single update. For completeness, PPO is typically written as an expected clipped surrogate objective over timesteps: J(θ) = Et [min (


PPO is typically written as an expected clipped surrogate objective over timesteps: J(θ) = Et [min (ρt(θ)At, clip(ρt(θ), 1 −ε, 1 + ε)At)] , ρt(θ) = πθ(at | st) πθold(at | st). (52) The objective is often converted into a loss function by simply adding a negative sign, which makes the optimizer seek to make it as negative as possible. For language models, the objective (or loss) is computed per token, which intuitively can be grounded in how one would compute the probability of the entire sequence of autoregressive predictions – by a product of probabilities. From there, the common implementation is with log-probabilities that make the computation simpler to perform in modern language rlhfbook.com 61


modeling frameworks. In practice, one computes the difference of token log-probabilities and exponentiates it to recover the policy ratio ρt. J(θ) = 1 |a| |a| X t=0 min  πθ(at|st) πθold(at|st)At, clip  πθ(at|st) πθold(at|st), 1 −ε, 1 + ε  At  . (53) This is the per-token version of PPO, which also applies to other policy-gradient methods, but is explored further later in the implementation section of this chapter. Here, the term for averaging by the number of tokens in the action, 1 |a|, comes from common implementation practices, but is not in a formal derivation of the loss (shown in [118]). Figure 19: PPO framework. A learned value function enables Generalized Advantage Estimation (GAE) for per-token advantages, used with a clipped surrogate objective.


lized Advantage Estimation (GAE) for per-token advantages, used with a clipped surrogate objective. Here we will explain the different cases this loss function triggers given various advantages and policy ratios. At an implementation level, the inner computations for PPO involve two main terms: 1) a standard policy gradient with a learned advantage and 2) a clipped policy gradient based on a maximum step size. To understand how different situations emerge, we can define the policy ratio as: ρ(θ) = πθ(a|s) πθold(a|s) (54) The policy ratio is a centerpiece of PPO and related algorithms. It emerges from computing the gradient of a policy and controls the parameter updates in a very intuitive way.


from computing the gradient of a policy and controls the parameter updates in a very intuitive way. For any batch of data, the policy ratio starts at 1 for the first gradient step for that batch, since πθ is the same as πθold at this point. Then, in the next gradient step, the policy ratio will be above one if that gradient step increased the likelihood of certain tokens with an associated positive advantage, or less than one for the other case. A common practice is to take 1-4 gradient steps per batch with policy gradient algorithms before updating πθold. 6.2.6 Understanding the PPO Objective Overall, the PPO objective can be visualized by two lines of a plot of objective versus policy ratio, which is shown in fig. 20.


n be visualized by two lines of a plot of objective versus policy ratio, which is shown in fig. 20. The PPO objective is maximized by changing the probability rlhfbook.com 62


of the sampled actions. Numerically, the objective controls for both positive and negative advantage cases by clever use of the minimum operation, making it so the update is at most pushed by an epsilon distance away from a policy ratio of 1. Within the trust region, PPO operates the same as other policy gradient algorithms. This is by design! The trust region is a concept used to cap the maximum step size of PPO and its peer algorithms for stability of updates. The core of the PPO algorithm, the clip and min/max functions, define this region. The objective becomes flat outside of it. The idea of a “trust region” comes from the numerical optimization literature [119], but was popularized within Deep RL from the algorithm Trust Region Policy Optimization (TRPO), which is accepted as the pre


hin Deep RL from the algorithm Trust Region Policy Optimization (TRPO), which is accepted as the predecessor to PPO [120]. The trust region is the area where the full policy-gradient steps are applied, as the updates are not “clipped” by the max/min operations of the PPO objective. Figure 20: Visualization of the different regions of the PPO objective for a hypothetical advantage. The “trust region” would be described as the region where the policy ratio ρ is within 1 ± ε. The policy ratio and advantage together can occur in a few different configurations. We will split the cases into two groups: positive and negative advantage. 6.2.6.1 Positive Advantage (At > 0) This means that the action taken was beneficial according to the value function, and we want to increase the likelihood of taki


taken was beneficial according to the value function, and we want to increase the likelihood of taking that action in the future. Now, let’s look at different cases for the policy ratio ρ(θ): 1. ρ(θ) < 1 −ε: • Interpretation: Action is less likely with the new policy than the old policy • Unclipped Term: ρ(θ)At • Clipped Term: (1 −ε)At • Objective: ρ(θ)At • Gradient: ∇θρ(θ)At ̸= 0 rlhfbook.com 63


• What happens: Normal policy-gradient update - increase likelihood of action 2. 1 −ε ≤ρ(θ) ≤1 + ε: • Interpretation: Action is almost equally likely with the new policy as the old policy • Unclipped Term: ρ(θ)At • Clipped Term: ρ(θ)At • Objective: ρ(θ)At • Gradient: ∇θρ(θ)At ̸= 0 • What happens: Normal policy-gradient update - increase likelihood of action 3. 1 + ε < ρ(θ): • Interpretation: Action is more likely with the new policy than the old policy • Unclipped Term: ρ(θ)At • Clipped Term: (1 + ε)At • Objective: (1 + ε)At • Gradient: ∇θ(1 + ε)At = 0 • What happens: NO UPDATE - action is already more likely under the new policy To summarize, when the advantage is positive (At > 0), we want to boost the probability of the action.


summarize, when the advantage is positive (At > 0), we want to boost the probability of the action. Therefore: • We perform gradient steps only in the case when πnew(a) ≤(1 + ε)πold(a). Intuitively, we want to boost the probability of the action, since the advantage was positive, but not boost it so much that we have made it substantially more likely. • Crucially, when πnew(a) > (1 + ε)πold(a), then we don’t perform any update, and the gradient of the clipped objective is 0. Intuitively, the action is already more expressed with the new policy, so we don’t want to over-reinforce it. 6.2.6.2 Negative Advantage (At < 0) This means that the action taken was detrimental according to the value function, and we want to decrease the likelihood of taking that action in the future.


to the value function, and we want to decrease the likelihood of taking that action in the future. Now, let’s look at different cases for the policy ratio ρ(θ): 1. ρ(θ) < 1 −ε: • Interpretation: Action is less likely with the new policy than the old policy • Unclipped Term: ρ(θ)At • Clipped Term: (1 −ε)At • Objective: (1 −ε)At • Gradient: ∇θ(1 −ε)At = 0 • What happens: NO UPDATE - action is already less likely under the new policy 2. 1 −ε ≤ρ(θ) ≤1 + ε: • Interpretation: Action is almost equally likely with the new policy as the old policy • Unclipped Term: ρ(θ)At • Clipped Term: ρ(θ)At • Objective: ρ(θ)At rlhfbook.com 64


• Gradient: ∇θρ(θ)At ̸= 0 • What happens: Normal policy-gradient update - decrease likelihood of action 3. 1 + ε < ρ(θ): • Interpretation: Action is more likely with the new policy than the old policy • Unclipped Term: ρ(θ)At • Clipped Term: (1 + ε)At • Objective: ρ(θ)At • Gradient: ∇θρ(θ)At ̸= 0 • What happens: Normal policy-gradient update - decrease likelihood of action To summarize, when the advantage is negative (At < 0), we want to decrease the probability of the action. Therefore: • We perform gradient steps only in the case when πnew(a) ≥(1 −ε)πold(a). Intuitively, we want to decrease the probability of the action, since the advantage was negative, and we do so proportional to the advantage.


bility of the action, since the advantage was negative, and we do so proportional to the advantage. • Crucially, when πnew(a) < (1 −ε)πold(a), then we don’t perform any update, and the gradient of the clipped objective is 0. Intuitively, the action is already less likely under the new policy, so we don’t want to over-suppress it. It is crucial to remember that PPO within the trust region is roughly the same as standard forms of policy gradient. 6.2.7 Value Functions and PPO The value function within PPO is an additional copy of the model that is used to predict the value per token. The value of a token (or state) in traditional RL is predicting the future return from that moment, often with discounting.


state) in traditional RL is predicting the future return from that moment, often with discounting. This value in PPO is used as a learned baseline, representing an evolution of the simple Monte Carlo version used with REINFORCE (which doesn’t need the learned value network). This highlights how PPO is an evolution of REINFORCE and vanilla policy-gradient in multiple forms, across the optimization form, baseline, etc. In practice, with PPO and other algorithms used for language models, this is predicting the return of each token after the deduction of KL penalties (the per-token loss includes the KL from the reward traditionally, as discussed). There are a few different methods (or targets) used to learn the value functions.


y, as discussed). There are a few different methods (or targets) used to learn the value functions. Generalized Advantage Estimation (GAE) is considered the state-of-the-art and canonical implementation in modern systems, but it carries more complexity by computing the value prediction error over multiple steps – see the later section on GAE in this chapter. A value function can also be learned with Monte Carlo estimates from the rollouts used to update the policy. PPO has two losses – one to learn the value function and another to use that value function to update the policy. A simple example implementation of a value network loss is shown below. # Basic PPO critic targets & loss (no GAE) # # B: Batch Size # L: Completion Length # Inputs: rlhfbook.com 65


Figure 21: Value function training uses on-policy rollouts to compute targets. The model predicts Vt at each token, which is trained via MSE against the target return ˆVt. The advantage At = ˆVt −Vt then weights the policy gradient update. # rewards: (B, L) post-KL per-token rewards; EOS row includes outcome # done_mask: (B, L) 1.0 at terminal token (EOS or truncation if penalized), else 0.0 # completion_mask: (B, L) 1.0 on response tokens to supervise (ignore the prompt) # values: (B, L) current critic predictions V_theta(s_t) # because a value network is a running update # old_values: (B, L) critic predictions at rollout time V_{theta_old}(s_t) # gamma: discount factor, float (often 1.0 for LM RLHF) # epsilon_v: float value clip range (e.g., 0.2), similar to PPO Loss Update itself, optio


r LM RLHF) # epsilon_v: float value clip range (e.g., 0.2), similar to PPO Loss Update itself, optional # # Returns: # value_loss: scalar; advantages: (B, L) detached (for policy loss) B, L = rewards.shape # 1) Monte Carlo returns per token (reset at terminals) # Apply discounting, if enabled returns = torch.zeros_like(rewards) running = torch.zeros(B, device=rewards.device, dtype=rewards.dtype) for t in reversed(range(L)): running = rewards[:, t] + gamma * (1.0 - done_mask[:, t]) * running returns[:, t] = running targets = returns # y_t = G_t (post-KL) # 2) PPO-style value clipping (optional) v_pred = values v_old = old_values v_clip = torch.clamp(v_pred, v_old - epsilon_v, v_old + epsilon_v) vf_unclipped = 0.5 * (v_pred - targets) ** 2 vf_clipped = 0.5 * (v_clip - targets) ** 2 vf_loss_t


v) vf_unclipped = 0.5 * (v_pred - targets) ** 2 vf_clipped = 0.5 * (v_clip - targets) ** 2 vf_loss_tok = torch.max(vf_unclipped, vf_clipped) # 3) Mask to response tokens and aggregate rlhfbook.com 66


denom = completion_mask.sum(dim=1).clamp_min(1) value_loss = ((vf_loss_tok * completion_mask).sum(dim=1) / denom).mean() # 4) Advantages for policy loss (no GAE): A_t = G_t - V(s_t) advantages = (targets - v_pred).detach() # The value loss is applied later, often with the PG loss, e.g. # total_loss = policy_loss + vf_coef * value_loss 6.2.8 Group Relative Policy Optimization (GRPO) Group Relative Policy Optimization (GRPO) is introduced in DeepSeekMath [121], and used in other DeepSeek works, e.g. DeepSeek-V3 [16] and DeepSeek-R1 [15]. GRPO can be viewed as a PPO-inspired algorithm with a very similar surrogate loss, but it avoids learning a value function with another copy of the original policy language model (or another checkpoint for initialization).


with another copy of the original policy language model (or another checkpoint for initialization). This brings two posited benefits: 1. Avoiding the challenge of learning a value function from an LM backbone, where research hasn’t established best practices. 2. Saves memory by not needing to keep the extra set of model weights in memory (going from needing the current policy, the reference policy, and a value function, to just the first two copies). GRPO does this by simplifying the value estimation and assigning the same value to every token in the episode (i.e. in the completion to a prompt, each token gets assigned the same value rather than discounted rewards in a standard value function) by estimating the advantage or baseline.


ther than discounted rewards in a standard value function) by estimating the advantage or baseline. The estimate is done by collecting multiple completions (ai) and rewards (ri), i.e. a Monte Carlo estimate, from the same initial state / prompt (s). To state this formally, the GRPO objective is very similar to the PPO objective above. For GRPO, the objective (or loss) is accumulated over a group of completions {a1, a2, ..., aG} to a given prompt s. Here, we show the GRPO objective: J(θ) = 1 G G X i=1  min  πθ(ai|s) πθold(ai|s)Ai, clip  πθ(ai|s) πθold(ai|s), 1 −ε, 1 + ε  Ai  −βDKL(πθ||πref)  . (55) Note that relative to PPO, the standard implementation of GRPO includes the KL distance in the loss.


ote that relative to PPO, the standard implementation of GRPO includes the KL distance in the loss. As above, we can expand this into a per-token computation: J(θ) = 1 G G X i=1 1 |ai| |ai| X t=1 min  πθ(ai,t|si) πθold(ai,t|si)Ai,t, clip  πθ(ai,t|si) πθold(ai,t|si), 1 −ε, 1 + ε  Ai,t  −βDKL(πθ(·|si)∥πref(·|si)) ! (56) With the advantage computation for the completion index i: rlhfbook.com 67


Ai = ri −mean(r1, r2, · · · , rG) std(r1, r2, · · · , rG) . (57) Figure 22: GRPO architecture. Advantages are normalized relative to the group mean and standard deviation. The KL penalty is applied directly in the loss rather than shaping the reward. Intuitively, the GRPO update is comparing multiple answers to a single question within a batch. The model learns to become more like the answers marked as correct and less like the others. This is a very simple way to compute the advantage, which is the measure of how much better a specific action is than the average at a given state. Relative to PPO, REINFORCE, and broadly RLHF performed with a reward model rating (relative to output reward), GRPO is often run with a far higher number of samples per prompt because the advantage is entirely ab


RPO is often run with a far higher number of samples per prompt because the advantage is entirely about the relative value of a completion to its peers from that prompt. Here, the current policy generates multiple responses to a given prompt, and the group-wise GRPO advantage estimate is given valuable context. PPO and vanilla policy-gradient algorithms were designed to accurately estimate the reward of every completion (in fact, more completions can do little to improve the value estimate in some cases). GRPO and its variants are particularly well-suited to modern language model tools, where having multiple completions to a given prompt is very natural (especially when compared to, e.g., multiple actions from a set environment state in a robotic task).


specially when compared to, e.g., multiple actions from a set environment state in a robotic task). The advantage computation for GRPO has trade-offs in its biases. The normalization by standard deviation rewards questions in a batch that have a low variation in answer correctness. For questions with either nearly all correct or all incorrect answers, the standard deviation will be lower and the advantage will be higher. Liu et al. 2025 [118] proposes removing the standard deviation term given this bias, but this comes at the cost of downweighting questions that were all incorrect with a few correct answers, which could be seen as valuable learning signal for the model. Those high-variance prompts can be exactly the hardest cases, where only a few sampled completions find the correct answe


rompts can be exactly the hardest cases, where only a few sampled completions find the correct answer and provide a strong training signal. eq. 57 is the implementation of GRPO when working with outcome supervision (either a standard reward model or a single verifiable reward) and a different implementation is needed with process supervision. In this case, GRPO computes the advantage as the sum of the normalized rewards for the following reasoning steps. rlhfbook.com 68


Finally, GRPO’s advantage estimation can also be applied without the PPO clipping to more vanilla versions of policy gradient (e.g. REINFORCE), but it is not the canonical form. As an example of how these algorithms are intertwined, we can show that the advantage estimation in a variant of GRPO, Dr. GRPO (GRPO Done Right) [118], is equivalent to the RLOO estimation (which uses the average reward of other samples as its baseline) up to a constant scaling factor (which normally does not matter due to implementation details to normalize the advantage). Dr. GRPO removes the standard deviation normalization term from eq. 57 – note that this also scales the advantage up, which is equivalent to increasing the GRPO learning rate on samples with a variance in answer scores.


ich is equivalent to increasing the GRPO learning rate on samples with a variance in answer scores. This addresses a bias towards questions with low reward variance – i.e. almost all the answers are right or wrong – but comes at a potential cost if it is important to learn from problems where just one sample gets the answer right. The Dr. GRPO advantage for completion i within a group of size G is defined as: ˜Ai = ri −mean(r1, r2, · · · , rG) = ri −1 G G X j=1 rj (58) Here, in the same notation, we can recall the RLOO advantage estimation as: ARLOO i = ri − 1 G −1 G X j=1,i̸=j rj (59) Thus, if we multiply the Dr. GRPO advantage definition by G G−1 we can see a scaled equivalence: G G −1 ˜Ai = G G −1  ri −1 G G X j=1 rj   = G G −1ri − 1 G −1 G X j=1 rj = G G −1ri − 1 G −1 G X j=1,j̸=i


= G G −1  ri −1 G G X j=1 rj   = G G −1ri − 1 G −1 G X j=1 rj = G G −1ri − 1 G −1 G X j=1,j̸=i rj − 1 G −1ri = ri  G G −1 − 1 G −1  − 1 G −1 G X j=1,j̸=i rj = ri − 1 G −1 G X j=1,j̸=i rj = ARLOO i (60) 6.2.9 Group Sequence Policy Optimization (GSPO) When taking multiple gradient steps on a batch of data collected from a previous policy, importance sampling is required to correct for the distribution mismatch between the datarlhfbook.com 69


collection policy and the current policy being optimized. The standard importance sampling identity allows us to estimate expectations under one distribution using samples from another: Ep[f(x)] = Eq  f(x)p(x) q(x)  , (61) where p is the target distribution, q is the sampling distribution, and p(x) q(x) is the importance weight. In policy gradient methods, p = πθ is the current policy we want to optimize and q = πθold is the policy that generated the training data. This allows us to reweight samples collected under πθold to estimate gradients for πθ, enabling multiple gradient steps per batch of rollouts. This distribution mismatch arises in two common scenarios: (1) taking multiple gradient steps on a single batch, where πθ drifts from πθold after each update, and (2) in asynchronous tr


nt steps on a single batch, where πθ drifts from πθold after each update, and (2) in asynchronous training systems where the inference backend (e.g., vLLM) and training backend (e.g., FSDP) may have different model weights due to synchronization delays (see the Asynchronicity section later in this chapter, which emerged particularly with the focus on RL for verifiable rewards, but is also used in RLHF setups). PPO and GRPO apply importance sampling at the token level and stabilize learning by clipping the surrogate objective. However, this approach has a subtle failure mode: when a token’s importance ratio moves outside the clipping range [1 −ε, 1 + ε], that token receives zero gradient. For rare but important tokens—such as key reasoning steps that the model initially assigns low probabil


rare but important tokens—such as key reasoning steps that the model initially assigns low probability—this “token dropping” can prevent the model from learning to produce them more reliably. Group Sequence Policy Optimization (GSPO) [122] extends GRPO by computing importance ratios at the sequence level rather than the token level. The practical motivation for this algorithm – and its peer, CISPO, which modifies how importance sampling is computed for policy gradient algorithms, as we will discuss later – is that the per-token importance sampling ratio is often numerically unstable. The conceptual motivation is that when rewards are assigned at the sequence level (as in most RLHF and RLVR setups), the importance sampling correction should match that granularity.


as in most RLHF and RLVR setups), the importance sampling correction should match that granularity. Token-level ratios can behave erratically for long sequences and/or large, sparse models (e.g. modern mixture-of-experts (MoE) models): a single token with a large ratio can dominate the policy update, or many tokens may get clipped independently within a response, fragmenting the learning signal across a single response. GSPO addresses this by computing a single importance weight per response. Recall that the probability of a full response factorizes autoregressively: πθ(a | s) = |a| Y t=1 πθ(at | s, a<t). (62) Note that for simplicity, we often shorten the conditional policy, πθ(at | s, a<t), as πθ(at | s), which implicitly contains the previous actions (tokens) in a completion.


| s, a<t), as πθ(at | s), which implicitly contains the previous actions (tokens) in a completion. GSPO defines a length-normalized sequence-level importance ratio using the geometric mean (to avoid numerical issues with long sequences): rlhfbook.com 70


ρi(θ) =  πθ(ai | s) πθold(ai | s)  1 |ai| = exp  1 |ai| |ai| X t=1 log πθ(ai,t | s, ai,<t) πθold(ai,t | s, ai,<t)  . (63) The GSPO objective mirrors GRPO but uses this sequence-level ratio: JGSPO(θ) = Es∼D, {ai}G i=1∼πθold(·|s) " 1 G G X i=1 min (ρi(θ)Ai, clip(ρi(θ), 1 −ε, 1 + ε)Ai) # . (64) Because the ratio is length-normalized, the clipping range ε operates on a per-token average scale, making the effective constraint comparable across responses of different lengths. In implementation, the sequence-level weight ρi is applied uniformly to all tokens in response ai, which simplifies gradient computation while maintaining the sequence-level IS correction. The advantage computation remains the same as GRPO (eq.


intaining the sequence-level IS correction. The advantage computation remains the same as GRPO (eq. 57), using the group-relative mean and standard deviation normalization, which can be modified as done in other derivative studies of GRPO. GSPO can be summarized as “GRPO with sequence-level importance ratios”—the IS correction granularity is matched to the reward granularity. 6.2.10 Clipped Importance Sampling Policy Optimization (CISPO) Clipped Importance Sampling Policy Optimization (CISPO) [123] takes a different approach: rather than clipping the surrogate objective, CISPO clips the importance weights themselves while preserving gradients for all tokens. The objective uses a stop-gradient on the clipped importance weight, returning to a REINFORCE-style formulation instead of the PPO-st


t on the clipped importance weight, returning to a REINFORCE-style formulation instead of the PPO-style, two-sided clipping: JCISPO(θ) = Es∼D, {ai}K i=1∼πθold(·|s)   1 PK i=1 |ai| K X i=1 |ai| X t=1 sg (ˆρi,t(θ)) Ai,t log πθ(ai,t | s, ai,<t)  , (65) where sg(·) denotes stop-gradient (the weight is used but not differentiated through), and the clipped importance ratio is: ˆρi,t(θ) = clip (ρi,t(θ), 1 −εlow, 1 + εhigh) , ρi,t(θ) = πθ(ai,t | s, ai,<t) πθold(ai,t | s, ai,<t). (66) The key difference from PPO/GRPO is subtle but important: clipping the weight (not the objective) means every token still receives a gradient signal proportional to its advantage—the weight just bounds how much that signal is amplified or suppressed by the importance ratio.


age—the weight just bounds how much that signal is amplified or suppressed by the importance ratio. This is a bias-variance tradeoff: clipping weights introduces bias but controls variance and, critically, avoids dropping token gradients entirely. Both CISPO and GSPO were developed by organizations pushing the limits of applying RL on large-scale MoE models, which are known for their numerical issues. The papers highlight rlhfbook.com 71


how the per-token importance sampling ratios are unstable and can add substantial variance to the gradients, mitigating learning. This can make these algorithms particularly impactful on large-scale models, but less studied and beneficial within smaller, academic experiments. CISPO also allows asymmetric clipping bounds (εlow ̸= εhigh), similar to DAPO’s “cliphigher” modification discussed later in this chapter, which can encourage exploration by allowing larger updates for tokens the model wants to upweight. Related work includes Tapered Off-Policy REINFORCE (TOPR) [124], which also clips IS weights directly (like CISPO) rather than clipping within the objective (like PPO/GRPO), but operates at the sequence level (like GSPO) and uses asymmetric clipping based on reward sign—applying no IS


s at the sequence level (like GSPO) and uses asymmetric clipping based on reward sign—applying no IS correction for positive rewards while clipping ratios to [0, 1] for negative rewards—enabling stable off-policy learning. 6.2.11 Comparing Algorithms Each algorithm in this chapter shares the same core gradient shape (eq. 23), but differs in how it estimates the advantage and controls the optimization: • REINFORCE: The simplest policy gradient implementation, using Monte-Carlo estimates of reward and a state-based baseline to reduce variance. • RLOO: REINFORCE with multiple samples per prompt, with each sample’s baseline being the average reward of the others (leave-one-out) to reduce gradient variance.


mple’s baseline being the average reward of the others (leave-one-out) to reduce gradient variance. • PPO: Adds a learned value function and a clipped policy ratio to get more accurate and stable gradient updates. • GRPO: A simplified variant of PPO that groups multiple completions per prompt and normalizes rewards within the group to compute advantages, removing the need for a value function. • CISPO: A REINFORCE-style algorithm that clips importance-sampling weights (not the objective as in PPO/GRPO) with a stop-gradient for stability, so every token receives a gradient signal. • GSPO: Like GRPO but normalizes the policy ratio by completion length, preventing length bias. • DPO: Not an RL algorithm, but a method to solve the same preference optimization problem by bypassing the separate


algorithm, but a method to solve the same preference optimization problem by bypassing the separate reward model entirely, optimizing directly from preference pairs (see Chapter 8). All of the policy gradient algorithms above are on-policy in derivation, though most are applied slightly off-policy in practice. DPO and the other direct alignment algorithms in Chapter 8 are off-policy by default. All can be paired with a learned reward model or verifiable rewards. Only PPO requires a learned value function. REINFORCE and RLOO have no importance-sampling ratio — the remaining algorithms each introduce one to enable multiple gradient steps per batch of rollouts, differing in granularity and clipping strategy as summarized below. Table 3: Comparing policy gradient algorithms.


anularity and clipping strategy as summarized below. Table 3: Comparing policy gradient algorithms. Method IS Granularity Clipping Style Advantage REINFORCE None None Monte Carlo baseline RLOO None None Leave-one-out rlhfbook.com 72


Method IS Granularity Clipping Style Advantage PPO Token Objective (bilateral) Learned value fn GRPO Token Objective (bilateral) Group-relative GSPO Sequence Objective (bilateral) Group-relative CISPO Token Weights (stop-grad) Group-relative The core loss L(θ) for each method is: REINFORCE: −1 T T X t=1 log πθ(at | st) Gt −b(st)  RLOO: −1 K K X i=1 X t log πθ(ai,t | si,t)  Ri − 1 K −1 X j̸=i Rj   CISPO: − X i,t sg(ˆρi,t) Ai,t log πθ(ai,t | si,t) ˆρi,t = clip(ρi,t, 1 −ε, 1 + ε) PPO: −1 T T X t=1 min ρtAt, clip(ρt, 1 −ε, 1 + ε) At  ρt = πθ(at | st) πθold(at | st) GRPO: −1 G G X i=1 min ρiAi, clip(ρi, 1 −ε, 1 + ε) Ai  ρi = πθ(ai | s) πθold(ai | s), Ai = ri −mean(r1:G) std(r1:G) GSPO: −1 G G X i=1 min ρiAi, clip(ρi, 1 −ε, 1 + ε) Ai  ρi =  πθ(ai | s) πθold(ai | s) 1/|ai| DPO: −E(x


−1 G G X i=1 min ρiAi, clip(ρi, 1 −ε, 1 + ε) Ai  ρi =  πθ(ai | s) πθold(ai | s) 1/|ai| DPO: −E(x,yw,yl)  log σ β[∆log πθ(x) −∆log πref(x)]  6.3 Implementation Compared to the original Deep RL literature where many of these algorithms were developed, implementing RL for optimizing language models or other large AI models requires many small implementation details. In this section, we highlight some key factors that differentiate the implementations of popular algorithms. There are many other small details that go into this training. For example, when doing rlhfbook.com 73


RLHF with language models a crucial step is generating text that will then be rated by the reward model. Under normal circumstances, the model should generate an end-of-sequence (EOS) token indicating it finished generating, but a common practice is to put a hard cap on generation length to efficiently utilize infrastructure. A failure mode of RLHF is that the model is regularly truncated in its answers, driving the ratings from the reward model out-of-distribution and to unpredictable scores. The solution to this is to only run reward model scoring on the eos_token, and to otherwise assign a penalty to the model for generating too long. The popular open-source tools for RLHF have a large variance in implementation details across the algorithms (see table 10 in [125]).


RLHF have a large variance in implementation details across the algorithms (see table 10 in [125]). Some decisions not covered here include: • Value network initialization: The internal learned value network used by PPO and other similar algorithms can be started from a different model of the same architecture or randomly selected weights. This can have a large impact on performance. The standard established in InstructGPT [3] (and re-used in Tülu 3 for its work on RLVR [6]) is to initialize the value network from the reward model used during RLHF. Others have used the previous checkpoint to RLHF training (normally an SFT model) with a value head appended randomly initialized, or fully re-initialized language models (less common as it will take longer for RLHF to converge, but possible).


nitialized language models (less common as it will take longer for RLHF to converge, but possible). • Reward normalization, reward whitening, and/or advantage whitening: Normalization bounds all the values from the RM (or environment) to be between 0 and 1, which can help with learning stability. Whitening goes further by transforming rewards or advantage estimates to have zero mean and unit variance, providing an even stronger boost to stability. • Different KL estimators: With complex language models, precisely computing the KL divergence between models can be complex, so multiple approximations are used to substitute for an exact calculation [126]. • KL controllers: Original implementations of PPO and related algorithms had dynamic controllers that targeted specific KLs and changed the


ns of PPO and related algorithms had dynamic controllers that targeted specific KLs and changed the penalty based on recent measurements. Most modern RLHF implementations use static KL penalties, but this can also vary. For more details on implementation details for RLHF, see [127]. For further information on the algorithms, see [128]. 6.3.1 Policy-Gradient Basics A simple implementation of policy gradient, using advantages to estimate the gradient to prepare for advanced algorithms such as PPO and GRPO follows: pg_loss = -advantages * ratio Ratio here is the (per-token) probability ratio (often computed from a log-probability difference) of the new policy model probabilities relative to the reference model.


log-probability difference) of the new policy model probabilities relative to the reference model. In order to understand this equation, it is good to understand different cases that can fall within a batch of updates. Remember that we want the loss to decrease as the model gets better at the task. Case 1: Positive advantage, so the action was better than the expected value of the state. We rlhfbook.com 74


want to reinforce this. In this case, the model will make this more likely with the negative sign. To do so, it’ll increase the logratio. A positive logratio, or sum of log probabilities of the tokens, means that the model is more likely to generate those tokens. Case 2: Negative advantage, so the action was worse than the expected value of the state. This follows very similarly. Here, the loss will be positive if the new model was more likely, so the model will try to make it so the policy parameters make this completion less likely. Case 3: Zero advantage, so no update is needed. The loss is zero, don’t change the policy model. 6.3.2 Loss Aggregation Tradeoffs The question when implementing any policy gradient algorithm with language models is: How do you aggregate per-token losses into


g any policy gradient algorithm with language models is: How do you aggregate per-token losses into a final scalar loss? Given per-token losses ℓi,t for sample i at token t, with completion lengths |ai| and batch size B, there are three main strategies: Strategy 1: Per-sequence normalization (standard GRPO; also used in some PPO implementations) L = 1 B B X i=1 1 |ai| |ai| X t=1 ℓi,t (67) Each sequence contributes equally to the batch loss, regardless of length. In code: # Strategy 1: Per-sequence normalization sequence_loss = ((per_token_loss * completion_mask).sum(dim=1) / \ completion_mask.sum(dim=1)).mean() Strategy 2: Per-token normalization (DAPO [129]) L = PB i=1 P|ai| t=1 ℓi,t PB i=1 |ai| (68) Each token contributes equally; longer sequences have proportionally more influence on th


|ai| (68) Each token contributes equally; longer sequences have proportionally more influence on the gradient. In code: # Strategy 2: Per-token normalization token_loss = ((per_token_loss * completion_mask).sum() / \ completion_mask.sum()) Strategy 3: Fixed-length normalization (Dr. GRPO [118]) L = 1 B B X i=1 1 Lmax |ai| X t=1 ℓi,t (69) rlhfbook.com 75


Normalizes by max sequence length Lmax, equalizing the per-token scale across sequences while still letting longer sequences contribute more total gradient because they contain more active tokens. In code: # Strategy 3: Fixed-length normalization fixed_len_loss = ((per_token_loss * completion_mask).sum(dim=1) / \ L_max).mean() Where Lmax is typically a global constant during the entire training procedure, which specifies the maximum number of generation tokens. Note that completion_mask in the code above is a matrix of 1s and 0s, where the prompt tokens are masked out (0s) because we don’t want the model to learn from predicting prompt tokens. 6.3.2.1 Why Does This Matter? Intuitively, per-sequence normalization (Strategy 1) seems best since we care about outcomes, not individual tokens.


sequence normalization (Strategy 1) seems best since we care about outcomes, not individual tokens. However, this introduces subtle biases based on sequence length, which can cause the model to overthink or downweight strategies that naturally need to use more tokens, depending on the direction of the bias. Consider two sequences of different lengths with per-token losses: seq_1_losses = [1, 1, 1, 1, 10] # 5 tokens, mean = 2.8 seq_2_losses = [1, 1, 1, 1, 1, 1, 1, 1, 1, 10] # 10 tokens, mean = 1.9 With Strategy 1 (per-sequence): The batch loss is (2.8 + 1.9)/2 = 2.35, and crucially, each token in the short sequence receives a larger gradient than tokens in the long sequence. With Strategy 2 (per-token): The batch loss is (14 + 19)/15 = 2.2, and all tokens receive equal gradient magnitude.


(per-token): The batch loss is (14 + 19)/15 = 2.2, and all tokens receive equal gradient magnitude. With Strategy 3 (fixed-length with Lmax = 10): The short sequence contributes 1.4 and the long sequence contributes 1.9, balancing per-token gradients while still weighting by sequence. For a more complete example showing how these strategies affect gradients, see the script below. from typing import Optional import torch def masked_mean(values: torch.Tensor, mask: torch.Tensor, axis: Optional[int] = None) -> torch.Tensor: """Compute mean of tensor with masked values.""" if axis is not None: return (values * mask).sum(axis=axis) / mask.sum(axis=axis) else: return (values * mask).sum() / mask.sum() def masked_sum( values: torch.Tensor, mask: torch.Tensor, axis: Optional[int] = None, constant_


um() def masked_sum( values: torch.Tensor, mask: torch.Tensor, axis: Optional[int] = None, constant_normalizer: float = 1.0, rlhfbook.com 76


) -> torch.Tensor: """Compute sum of tensor with masked values. Use a constant to normalize.""" if axis is not None: return (values * mask).sum(axis=axis) / constant_normalizer else: return (values * mask).sum() / constant_normalizer ratio = torch.tensor([ [1., 1, 1, 1, 1, 1, 1,], [1, 1, 1, 1, 1, 1, 1,], ], requires_grad=True) advs = torch.tensor([ [2, 2, 2, 2, 2, 2, 2,], [2, 2, 2, 2, 2, 2, 2,], ]) masks = torch.tensor([ # generation 1: 4 tokens [1, 1, 1, 1, 0, 0, 0,], # generation 2: 7 tokens [1, 1, 1, 1, 1, 1, 1,], ]) max_gen_len = 7 masked_mean_result = masked_mean(ratio * advs, masks, axis=1) masked_mean_token_level = masked_mean(ratio, masks, axis=None) masked_sum_result = masked_sum(ratio * advs, masks, axis=1, constant_normalizer=max_gen_len) print("masked_mean", masked_mean_result)


tio * advs, masks, axis=1, constant_normalizer=max_gen_len) print("masked_mean", masked_mean_result) print("masked_sum", masked_sum_result) print("masked_mean_token_level", masked_mean_token_level) # masked_mean tensor([2., 2.], grad_fn=<DivBackward0>) # masked_sum tensor([1.1429, 2.0000], grad_fn=<DivBackward0>) # masked_mean_token_level tensor(1., grad_fn=<DivBackward0>) masked_mean_result.mean().backward() print("ratio.grad", ratio.grad) ratio.grad.zero_() # ratio.grad tensor([[0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000], # [0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429]]) masked_sum_result.mean().backward() print("ratio.grad", ratio.grad) ratio.grad.zero_() # ratio.grad tensor([[0.1429, 0.1429, 0.1429, 0.1429, 0.0000, 0.0000, 0.0000], # [0.1429, 0.1429, 0.1429, 0.1


rad tensor([[0.1429, 0.1429, 0.1429, 0.1429, 0.0000, 0.0000, 0.0000], # [0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429]]) masked_mean_token_level.mean().backward() print("ratio.grad", ratio.grad) # ratio.grad tensor([[0.0909, 0.0909, 0.0909, 0.0909, 0.0000, 0.0000, 0.0000], # [0.0909, 0.0909, 0.0909, 0.0909, 0.0909, 0.0909, 0.0909]]) The output shows that with Strategy 1 (masked_mean), the short sequence has larger per-token gradients (0.25) than the long sequence (0.14). Strategies 2 and 3 equalize the pertoken gradients across sequences. Note that these results can vary substantially if gradient rlhfbook.com 77


accumulation is used, where the gradients are summed across multiple minibatches before taking a backward step—in this case, the balance between shorter and longer sequences can flip. In practice, the best strategy depends on the specific training setup. Often in RLHF the method with the best numerical stability or the least variance in loss is preferred. 6.3.2.2 Related: MDP vs. Bandit Framing The choice of loss aggregation connects to a deeper distinction in how we frame the RL problem. The MDP (token-level) view treats each token at as an action with state st being the running prefix. In practice, this is the framing used when we compute token-level advantages with a learned value function V (st) (e.g., GAE [112]) and apply KL penalties per token.


advantages with a learned value function V (st) (e.g., GAE [112]) and apply KL penalties per token. PPO with a learned value network is the canonical example [116]. In contrast, the bandit (sequence-level) view treats the whole completion as a single action with one scalar reward R. In code, this means computing a sequence-level advantage Aseq and broadcasting it to all tokens. RLOO and GRPO-style advantages are often used in this bandit-style setting [115] [111] [121]. Direct alignment methods like DPO and A-LoL also define sequence-level objectives, although they are not policy-gradient estimators [130]. Note that many GRPO implementations use a bandit-style advantage and add a separate per-token KL term in the loss, while many PPO/RLOO implementations fold KL into the reward before comp


r-token KL term in the loss, while many PPO/RLOO implementations fold KL into the reward before computing advantages; both conventions exist in practice. An example comparison highlighting the two approaches is below: # === Bandit-style (sequence-level) === # One scalar reward per sequence; advantage broadcast to all tokens reward = torch.tensor([3.0, 1.0]) # (B,) e.g., reward model scores baseline = reward.mean() # simple baseline (RLOO uses leave-one-out) advantage_seq = reward - baseline # (B,) advantages = advantage_seq[:, None].expand(-1, seq_len) # (B, L) # tensor([[ 1., 1., 1., 1.], <- same advantage for all tokens # [-1., -1., -1., -1.]]) # === MDP-style (token-level) === # Per-token rewards + learned V(s_t); each token gets its own advantage # (could also use per-token KL shaping,


rewards + learned V(s_t); each token gets its own advantage # (could also use per-token KL shaping, format rewards, or other token-level signals) advantages = gae(per_token_rewards, values, done_mask, gamma=1.0, lam=0.95) # tensor([[ 0.2, 0.5, 0.8, 1.5], <- varies by position # [-0.3, -0.5, -0.8, -1.4]]) This framing distinction also explains why the discount factor γ is set to 1.0 in virtually all RLHF implementations. In standard RL, discounting (γ < 1) is essential: it balances the optimization between short-term and long-term reward across a multi-step episode, which is crucial for the agent to learn effective behavior over time. But in the RLHF setting, even when using the token-level MDP view, the inductive bias of the optimization is the quality of the collective completion – the r


MDP view, the inductive bias of the optimization is the quality of the collective completion – the reward signal scores the entire response, not individual tokens. Discounting earlier tokens would arbitrarily down-weight their contribution with no principled justification. As agentic RL settings mature – where models take real multi-step actions such as tool calls, code execution, and web browsing – discounting may become relevant again, since these involve genuinely distinct sequential decisions whose long-term consequences differ. rlhfbook.com 78


6.3.3 Asynchronous RL Systems The default implementation for policy-gradient algorithms is what is called on-policy execution, where the actions (generations) taken by the agent (language model) are scored before updating the model. The theoretical derivations of policy-gradient rely on all actions being exactly on-policy where the model is always up to date with the results from the latest trials/roll-outs. In practice, maintaining exact on-policy execution substantially slows training [131]—and perfect synchronization is technically impossible regardless. Therefore, all of the recent empirical results with language models tend to be slightly outside of the theoretical proofs. What happens in practice is designing the algorithms and systems for what actually works.


l proofs. What happens in practice is designing the algorithms and systems for what actually works. Figure 23: A comparison of the generation-update phases for synchronous or asynchronous RL training following Noukhovitch et al. 2024. The common solution used is to constantly run inference and training on separate GPU nodes with software designed to efficiently run both, as shown in the bottom of fig. 23. Common practice in popular open-source RL tools for language models is to use a distributed process management library such as Ray to hand information off between the policy-gradient learning loop and the inference loop using an efficient inference engine, e.g., vLLM. In these setups, the GPUs dedicated to taking the RL steps are called the “learners” and the GPUs dedicated to sampling fr


PUs dedicated to taking the RL steps are called the “learners” and the GPUs dedicated to sampling from the language model are called the “actors”. The primary challenges faced when making training more asynchronous are keeping training stable and maintaining learning signal. These systems are designed and implemented with the presumption that nearly on-policy data is good enough for stable learning. Here, the generation and update phases can easily be synced to avoid idle compute on either piece of the training system, which would be rlhfbook.com 79


Figure 24: An example distributed RL system, where two queues are managed to pass data to the learner and actor GPUs, which can both be synchronized with a distributed computing library such as Ray. Olmo Team 2025, license CC-BY. passing model weights from the learners to the actors in fig. 24. With reasoning models, the extremely long inference characteristics of problems requiring 10K to 100K+ tokens per answer makes the generation of roll-outs a far stronger bottleneck. A common problem when training reasoning models on more synchronous RL infrastructure is that an answer to one prompt in the batch can take substantially more time to generate (either through more tokens or more tool calls), resulting in the majority of the allocated compute being idle until it completes.


more tool calls), resulting in the majority of the allocated compute being idle until it completes. A second solution to this length mismatch issue, called sequence-level packing, is to stack shorter samples within a batch with clever masking to enable continued roll-outs from the model and better distribute length normalization across samples within a batch. The full complexity of distributed RL infrastructure is out of scope for this book, as it can cause many other subtle issues that slow down training or cause instability. Following the emergence of these reasoning models, further interest has been taken to make the training and inference loops fully off-policy, where training batches for the policy gradient updates are filled with the most recently completed roll-outs across multiple


r the policy gradient updates are filled with the most recently completed roll-outs across multiple instances generating answers [132] [133]. Fully asynchronous training would also enable scaling RL training runs across multiple datacenters more easily due to the option of increasing the time between weight syncs between the learner node (taking policy gradient steps) and the actor (trying to solve problems) [134]. Related methods are exploring fully off-policy policy gradient algorithms [124]. rlhfbook.com 80


6.3.4 Truncated Importance Sampling Truncated importance sampling (TIS) is a crucial tool used to stabilize training in modern, asynchronous RL frameworks with language models. Importance sampling is a correction that reweights samples drawn from one distribution to estimate expectations under another (as introduced in eq. 61). Truncated importance sampling [135] caps these weights with min(ρ, C) for some constant C, trading a small bias for bounded variance in the policy gradient. This is an importance-sampling correction applied to the policy gradient, but unlike the bilateral clipping in PPO and CISPO (which constrains the ratio near 1), TIS uses a one-sided upper cap: the ratio can fall freely below 1, but is capped at C to prevent extreme upweighting.


ed upper cap: the ratio can fall freely below 1, but is capped at C to prevent extreme upweighting. In all of PPO, GRPO, CISPO (and related algorithms), the ratio ρpolicy t = πθ(at | s)/πθold(at | s) corrects for policy drift across multiple gradient steps within one RL batch. As we shift to real-world RL frameworks, centered around the idea of asynchronicity in the previous subsection, there can be even larger sources of numerical differences (that also require the numerical correction of importance sampling). Even when the sampler and learner share identical parameters θ, their effective token distributions can differ because the inference engine (e.g., vLLM) and training framework (e.g., FSDP) use different kernels, precision, and parallelism strategies [136].


training framework (e.g., FSDP) use different kernels, precision, and parallelism strategies [136]. It is therefore useful to distinguish the same policy evaluated on two systems, πsampler θ and πlearner θ , and define the corresponding ratio and its truncated form: ρlearner t = πlearner θ (at | s, a<t) πsampler θ (at | s, a<t) , ˜ρlearner t = min(ρlearner t , C). (70) These two corrections are complementary, but they appear in policy-gradient implementations for different reasons — one compensates for policy drift within the training of an RL batch, the other for implementation-induced divergence — and can be applied simultaneously. How they combine depends on the algorithm: 6.3.4.1 REINFORCE with TIS (Single Gradient Step) There is no policy drift (πθ = πθold), so the only mismatch is be


CE with TIS (Single Gradient Step) There is no policy drift (πθ = πθold), so the only mismatch is between the learner and sampler. Here πθold = πgen, and TIS directly corrects the learner–sampler gap: ∇θJ ≈Ea∼πsampler θ  ˜ρlearner t · At · ∇θ log πlearner θ (at | s, a<t)  . (71) 6.3.4.2 PPO/GRPO with TIS (Multiple Gradient Steps) Now both ratios are active. In careful implementations, the “old logprobs” in the policy ratio are recomputed on the learner (the GSPO paper discusses this), so the policy ratio ρpolicy t = πlearner θ /πlearner θold captures pure policy drift, while ˜ρlearner t = min(πlearner θold /πsampler θold , C) separately corrects the backend mismatch at the generation checkpoint: JPPO+TIS(θ) = E h min  ρpolicy t At, clip  ρpolicy t , 1 −ε, 1 + ε  At  · ˜ρlearner t i .


point: JPPO+TIS(θ) = E h min  ρpolicy t At, clip  ρpolicy t , 1 −ε, 1 + ε  At  · ˜ρlearner t i . (72) Here πθold ̸= πgen: the old logprobs come from the learner, not the sampler. If a framework skips this recomputation and uses the sampler logprobs directly as πθold, the policy ratio rlhfbook.com 81


already captures the backend mismatch and no separate TIS correction is needed — but the clip then operates on a noisier ratio that starts away from 1.0 even before any gradient steps. This is the “your framework secretly brings you off-policy RL” observation from Yao et al. [136]. In practice, LLM RL systems apply TIS as a per-token correction weight on the policygradient loss: # Shape: (B*G, L) C = 2.0 # TIS cap logratio = learner_logprobs - sampler_logprobs logratio = logratio.clamp(-10.0, 10.0) # numerical safety tis_weight = torch.exp(logratio).clamp(max=C) # one-sided truncation # Use as a fixed correction weight on the per-token PG loss per_token_pg_loss = per_token_pg_loss * tis_weight.detach() The [−10, 10] clamp is only for numerical stability before exponentiation; the actual tr


ht.detach() The [−10, 10] clamp is only for numerical stability before exponentiation; the actual truncatedimportance-sampling step is the one-sided cap at C. In practice, the bookkeeping around these logprobs — storing sampler logprobs from generation, recomputing learner logprobs at the old checkpoint, and tracking current logprobs during gradient steps — is a substantial part of the scaffolding in distributed RL frameworks. Unlike GSPO, this correction is tokenlevel because it addresses token-level numerical mismatch rather than sequence-level reward granularity. TIS for the learner–sampler ratio has been adopted across major open-source RL frameworks (VeRL, TRL, OpenRLHF, SkyRL, OAT, and Open Instruct, which uses C = 2), and becomes increasingly important for long reasoning traces (Cha


Open Instruct, which uses C = 2), and becomes increasingly important for long reasoning traces (Chapter 7), where small per-token differences compound over thousands of generated tokens. 6.3.5 Example: PPO There are many, many implementations of PPO available. The core loss computation is shown below. Crucial to stable performance is also the value computation, where multiple options exist (including multiple options for the value model loss). Note that the reference policy (or old logprobs) here are from the time the generations were sampled and not necessarily the reference policy. The reference policy is only used for the KL distance constraint/penalty. # B: Batch Size, L: Sequence Length, G: Num of Generations # Apply KL penalty to rewards rewards = rewards - self.beta * per_token_kl


h, G: Num of Generations # Apply KL penalty to rewards rewards = rewards - self.beta * per_token_kl # Shape: (B*G, L) # Get value predictions values = value_net(completions) # Shape: (B*G, L) # Compute returns via backward pass (gamma typically 1.0 for LM RLHF) # Mask rewards to avoid padding tokens (which may have KL penalties) leaking into returns returns = torch.zeros_like(rewards) running = torch.zeros(rewards.shape[0], device=rewards.device, dtype=rewards.dtype) for t in reversed(range(rewards.shape[1])): # Zero out padding: only accumulate rewards/returns for valid completion tokens rlhfbook.com 82


running = (rewards[:, t] + self.gamma * running) * completion_mask[:, t] returns[:, t] = running # Compute advantages: A_t = G_t - V(s_t) advantages = returns - values.detach() # Shape: (B*G, L) # Note: We detach the value network here to not update the parameters of # the value function when computing the policy-gradient loss # Normalize advantages (optional but stable) advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8) # Compute probability ratio between new and old policies ratio = torch.exp(new_per_token_logps - per_token_logps) # Shape: (B*G, L) # PPO clipping objective eps = self.cliprange # e.g. 0.2 pg_losses1 = -advantages * ratio # Shape: (B*G, L) pg_losses2 = -advantages * torch.clamp(ratio, 1.0 - eps, 1.0 + eps) # Shape: (B*G, L) pg_loss_max = torch.max(pg


-advantages * torch.clamp(ratio, 1.0 - eps, 1.0 + eps) # Shape: (B*G, L) pg_loss_max = torch.max(pg_losses1, pg_losses2) # Shape: (B*G, L) # Value function loss: predict returns vf_loss = 0.5 * ((returns - values) ** 2) # Shape: (B*G, L) # Combine policy and value losses per_token_loss = pg_loss_max + self.vf_coef * vf_loss # Shape: (B*G, L) # Apply completion mask and compute final loss loss = ((per_token_loss * completion_mask).sum(dim=1) / completion_mask.sum(dim=1)).mean() # Scalar # Compute metrics for logging with torch.no_grad(): # Compute clipping fraction clip_frac = ((pg_losses2 > pg_losses1).float() * completion_mask).sum() / completion_mask.sum() # Compute approximate KL approx_kl = (0.5 * ((new_per_token_logps - per_token_logps)**2) * completion_mask).sum() / completion_mask.


kl = (0.5 * ((new_per_token_logps - per_token_logps)**2) * completion_mask).sum() / completion_mask.sum() # Compute value loss for logging value_loss = vf_loss.mean() The core piece to understand with PPO is how the policy gradient loss is updated. Focus on these three lines: pg_losses1 = -advantages * ratio # Shape: (B*G, L) pg_losses2 = -advantages * torch.clamp(ratio, 1.0 - eps, 1.0 + eps) # Shape: (B*G, L) pg_loss_max = torch.max(pg_losses1, pg_losses2) # Shape: (B*G, L) pg_losses1 is the vanilla advantage-weighted policy gradient loss. pg_losses2 applies the same formula but with the probability ratio clamped to the range [1 −ε, 1 + ε], limiting how much the policy can change in a single update. The key insight is taking torch.max of the two losses.


ch the policy can change in a single update. The key insight is taking torch.max of the two losses. Because we’re minimizing a negative loss (recall the negative sign in front of advantages), taking the maximum selects the more pessimistic gradient—the one that produces a smaller policy update. When the rlhfbook.com 83


advantage is positive (good action), clipping prevents the policy from increasing that action’s probability too aggressively. When the advantage is negative (bad action), clipping prevents over-correction in the other direction. By clamping the log-probability ratio, PPO bounds how far the policy can drift from the version that generated the training data, stabilizing learning without requiring an explicit trust region computation. The code above also shows PPO learning a value function alongside the policy, which adds implementation complexity, but the clipped objective is the core mechanism. 6.3.5.1 PPO/GRPO Simplification with One Gradient Step per Sample (No Clipping) PPO (and GRPO) implementations can be handled much more elegantly if the hyperparameter “number of gradient steps per s


ementations can be handled much more elegantly if the hyperparameter “number of gradient steps per sample” is equal to 1. Many typical values for this are from 2-4 or higher. In the main PPO or GRPO equations, see eq. 50, the “reference” policy is the previous parameters – those used to generate the completions or actions. Thus, if only one gradient step is taken, πθ = πθold, and the update rule reduces to the following (the notation []∇indicates a stop gradient): J(θ) = 1 G G X i=1  πθ(ai|s) [πθ(ai|s)]∇ Ai −βDKL(πθ||πref)  . (73) This leads to PPO or GRPO implementations where the second policy gradient and clipping logic can be omitted, making the optimizer far closer to standard policy gradient.


ent and clipping logic can be omitted, making the optimizer far closer to standard policy gradient. 6.3.6 Example: GRPO The DeepSeekMath paper describes some implementation details of GRPO that differ from PPO [121], especially if comparing to a standard application of PPO from Deep RL rather than language models. For example, the KL penalty within the RLHF optimization (recall the KL penalty is also used when training reasoning models on verifiable rewards without a reward model) is applied directly in the loss update rather than to the reward function. Where the standard KL penalty application for RLHF is applied as r = rθ −βDKL, the GRPO implementation is along the lines of: L = Lpolicy gradient + β ∗DKL (74) However, there are multiple ways to implement this.


he lines of: L = Lpolicy gradient + β ∗DKL (74) However, there are multiple ways to implement this. Traditionally, the KL distance is computed with respect to each token in the completion to a prompt s. For reasoning training, multiple completions are sampled from one prompt, and there are multiple prompts in one batch, so the KL distance will have a shape of [B, L, N], where B is the batch size, L is the sequence length, and N is the number of completions per prompt. Putting it together, using the first loss accumulation, the pseudocode can be written as below. # B: Batch Size, L: Sequence Length, G: Number of Generations # Compute group-wise rewards # Shape: (B,) rlhfbook.com 84


mean_grouped_rewards = rewards.view(-1, self.num_generations).mean(dim=1) std_grouped_rewards = rewards.view(-1, self.num_generations).std(dim=1) # Normalize the rewards to compute the advantages mean_grouped_rewards = mean_grouped_rewards.repeat_interleave(self.num_generations, dim=0) std_grouped_rewards = std_grouped_rewards.repeat_interleave(self.num_generations, dim=0) # Shape: (B*G,) # Compute advantages advantages = (rewards - mean_grouped_rewards) / (std_grouped_rewards + 1e-4) advantages = advantages.unsqueeze(1) # Shape: (B*G, 1) # Compute probability ratio between new and old policies ratio = torch.exp(new_per_token_logps - per_token_logps) # Shape: (B*G, L) # PPO clipping objective eps = self.cliprange # e.g.


ken_logps - per_token_logps) # Shape: (B*G, L) # PPO clipping objective eps = self.cliprange # e.g. 0.2 pg_losses1 = -advantages * ratio # Shape: (B*G, L) pg_losses2 = -advantages * torch.clamp(ratio, 1.0 - eps, 1.0 + eps) # Shape: (B*G, L) pg_loss_max = torch.max(pg_losses1, pg_losses2) # Shape: (B*G, L) # important to GRPO -- PPO applies this in reward traditionally # Combine with KL penalty per_token_loss = pg_loss_max + self.beta * per_token_kl # Shape: (B*G, L) # Apply completion mask and compute final loss loss = ((per_token_loss * completion_mask).sum(dim=1) / completion_mask.sum(dim=1)).mean() # Scalar # Compute core metric for logging (KL, reward, etc. also logged) with torch.no_grad(): # Compute clipping fraction clip_frac = ((pg_losses2 > pg_losses1).float() * completion_mask).s


d(): # Compute clipping fraction clip_frac = ((pg_losses2 > pg_losses1).float() * completion_mask).sum() / completion_mask.sum() # Compute approximate KL approx_kl = (0.5 * ((new_per_token_logps - per_token_logps)**2) * completion_mask).sum() / completion_mask.sum() For more details on how to interpret this code, see the PPO section above. The core differences from the PPO example are: • Advantage computation: GRPO normalizes rewards relative to the group (mean and std across generations for the same prompt) rather than using a learned value function as baseline. • No value network: GRPO removes the value model entirely, eliminating vf_loss and the associated complexity. • KL penalty placement: GRPO adds the KL penalty directly to the loss rather than subtracting it from the reward (this i


nt: GRPO adds the KL penalty directly to the loss rather than subtracting it from the reward (this is the standard implementation, but more versions exist on how the KL is applied). 6.3.6.1 RLOO vs. GRPO The advantage updates for RLOO follow GRPO very closely, highlighting the conceptual similarity of the algorithm when taken separately from the PPO rlhfbook.com 85


style clipping and KL penalty details. Specifically, for RLOO, the advantage is computed relative to a baseline that is extremely similar to that of GRPO – the completion reward relative to the others for that same question. Concisely, the RLOO advantage estimate follows as (expanded from TRL’s implementation): # rloo_k --> number of completions per prompt # rlhf_reward --> Initially a flat tensor of total rewards for all completions. Length B = N x k rlhf_reward = rlhf_reward.reshape(rloo_k, -1) # # Now, Shape: (k, N), each column j contains the k rewards for prompt j. baseline = (rlhf_reward.sum(0) - rlhf_reward) / (rloo_k - 1) # baseline --> Leave-one-out baseline rewards. Shape: (k, N) # baseline[i, j] is the avg reward of samples i' != i for prompt j.