baseline rewards. Shape: (k, N) # baseline[i, j] is the avg reward of samples i' != i for prompt j. advantages = rlhf_reward - baseline # advantages --> Same Shape: (k, N) advantages = advantages.flatten() # Same shape as original tensor The rest of the implementation details for RLOO follow the other trade-offs of implementing policy-gradient. 6.4 Auxiliary Topics In order to master the application of policy-gradient algorithms, there are countless other considerations. Here we consider some of the long-tail of complexities in successfully deploying a policy-gradient RL algorithm. 6.4.1 Generalized Advantage Estimation (GAE) Generalized Advantage Estimation (GAE) is an alternate method to compute the advantage for policy gradient algorithms [112] that better balances the bias-variance tra


ompute the advantage for policy gradient algorithms [112] that better balances the bias-variance tradeoff. Traditional single-step advantage estimates can introduce too much bias, while using complete trajectories can suffer from high variance. GAE computes an exponentially-weighted average of multi-step advantage estimates, where the λ hyperparameter controls the bias-variance tradeoff—ranging from single-step TD (λ = 0) to full trajectory returns (λ = 1); λ = 0.95 is a common default for LLM fine-tuning. Advantage estimates can take many forms, but we can define an n-step advantage estimator (similar to the TD residual at the beginning of the chapter) as follows: ˆA(n) t =            rt + γV (st+1) −V (st), n = 1 rt + γrt+1 + γ2V (st+2) −V (st), n = 2 ...


t =            rt + γV (st+1) −V (st), n = 1 rt + γrt+1 + γ2V (st+2) −V (st), n = 2 ... rt + γrt+1 + γ2rt+2 + · · · −V (st), n = ∞ (75) Here a shorter n will have lower variance but higher bias as we are attributing more learning power to each trajectory – it can overfit. GAE attempts to generalize this formulation into a weighted multi-step average instead of a specific n. To start, we must define the temporal difference (TD) residual of predicted value. rlhfbook.com 86


δV t = rt + γV (st+1) −V (st) (76) To utilize this, we introduce another variable λ as the GAE mixing parameter. This folds into an exponential decay of future advantages we wish to estimate: ˆAGAE(γ,λ) t = (1 −λ)( ˆA(1) t + λ ˆA(2) t + λ2 ˆA(3) t + · · · ) = (1 −λ)(δV t + λ(δV t + γδV t+1) + λ2(δV t + γδV t+1 + γ2δV t+2) + · · · ) = (1 −λ)(δV t (1 + λ + λ2 + · · · ) + γδV t+1(λ + λ2 + · · · ) + · · · ) = (1 −λ)(δV t 1 1−λ + γδV t+1 λ 1−λ + · · · ) = P∞ l=0(γλ)lδV t+l (77) Intuitively, this can be used to average multi-step estimates of Advantage in an elegant fashion. An example implementation is shown below: # GAE (token-level) for LM RLHF # # B: Batch Size # L: Length # Inputs: # rewards: (B, L) post-KL per-token rewards # values: (B, L) current V_theta(s_t) # done_mask: (B, L) 1.0 at t


(B, L) post-KL per-token rewards # values: (B, L) current V_theta(s_t) # done_mask: (B, L) 1.0 at terminal token (EOS or penalized trunc), else 0.0 # gamma: float (often 1.0), # lam (short for lambda): float in [0,1] # (Padding beyond terminal should have rewards=0, values=0) B, L = rewards.shape advantages = torch.zeros_like(rewards) next_v = torch.zeros(B, device=rewards.device, dtype=rewards.dtype) gae = torch.zeros(B, device=rewards.device, dtype=rewards.dtype) for t in reversed(range(L)): not_done = 1.0 - done_mask[:, t] delta = rewards[:, t] + gamma * not_done * next_v - values[:, t] gae = delta + gamma * lam * not_done * gae advantages[:, t] = gae next_v = values[:, t] targets = advantages + values # y_t for value regression advantages = advantages.detach() # for policy loss The ba


ntages + values # y_t for value regression advantages = advantages.detach() # for policy loss The backward loop accumulates temporal-difference (TD) errors (δt = rt + γV (st+1) − V (st)), which measure how much better or worse the actual outcome was compared to the value function’s prediction, with exponential decay (γλ)l. At terminal tokens, not_done=0 prevents bootstrapping from future states and resets the GAE accumulator, so each episode’s advantages are computed independently (since the loop runs backward, the terminal token cleanly stops the exponentially-weighted accumulation at episode boundaries—this makes the implementation packing-friendly, correctly handling multiple sequences concatenated into one).


the implementation packing-friendly, correctly handling multiple sequences concatenated into one). The final targets serve as regression targets for the separate value function learned outside this GAE loop, while the detached advantages weight the policy gradient—detached so that policy updates don’t backpropagate through the value network. In RLHF for language models, γ = 1.0 is common because episodes are short token sequences where undiscounted credit assignment is preferred (and often all of the tokens in one). rlhfbook.com 87


For further reading, see [137]. 6.4.2 Double Regularization We’ve seen in this chapter two types of regularization. One is built into algorithms like PPO with step-size constraints, and the other is a KL divergence based distance penalty relative to the start of the optimization. Many popular policy gradient algorithms from Deep Reinforcement Learning, including PPO and its predecessors, originated due to the need to control the learning process of the agent. In RLHF, as discussed extensively in Chapter 15 on Regularization and in Chapter 3 on Training Overview, there is a built-in regularization term via the distance penalty relative to the original policy one is fine-tuning. In this view, a large part of the difference between algorithms like PPO (which have internal step-size regulariza


a large part of the difference between algorithms like PPO (which have internal step-size regularization) and REINFORCE (which is simpler, and to which PPO reduces under certain hyperparameters) is far less meaningful for fine-tuning language models than training agents from scratch. In PPO, the objective that handles capping the step-size of the update is known as the surrogate objective. To monitor how much the PPO regularization is impacting updates in RLHF, one can look at the clip fraction variable in many popular implementations, which is the percentage of samples in the batch whose probability ratio falls outside the clipping interval. This is a useful proxy for how often PPO’s regularizer may be active, but not every such sample has zero gradient: the surrogate becomes flat only wh


rizer may be active, but not every such sample has zero gradient: the surrogate becomes flat only when the clipped branch is selected, such as positive-advantage samples with ratios above 1 + ε or negative-advantage samples with ratios below 1 −ε. In practice with language models, algorithms like PPO and GRPO are often run with only one gradient step per batch, which means that the PPO-native regularization is never applied (as clipping can only occur within a batch when the policy changes substantially) and the KL distance penalties predominate. However, this is not universal. For example, DAPO uses 16 gradient steps per batch [129], and Tülu 3 uses 4 PPO update iterations per batch for 8B and 70B models but reduces to 1 for 405B to maintain training stability [6].


tions per batch for 8B and 70B models but reduces to 1 for 405B to maintain training stability [6]. 6.4.3 Further Reading As RLHF has cemented itself at the center of modern post-training, other policy-gradient RL algorithms and RL algorithms generally have been proposed to improve the training process, but they have not had a central role in governing best practices. Examples for further reading include: • Pairwise Proximal Policy Optimization (P3O; Wu et al., 2023) [138] uses pairwise data directly in a PPO-style policy update without learning an intermediate reward model. • Soft Adaptive Policy Optimization (SAPO) [139] replaces hard PPO/GRPO-style clipping with smooth, temperature-controlled gating, aiming for a continuous trust region that preserves near-on-policy learning signal whil


lled gating, aiming for a continuous trust region that preserves near-on-policy learning signal while down-weighting off-policy tokens. • Off-policy policy-gradient algorithms could enable further asynchronous training, such as Contrastive Policy Gradient (CoPG) [140] (a generalization of the direct alignment algorithm IPO and vanilla policy gradient), which was used by Cohere for their Command A model [57]. rlhfbook.com 88


• Other implementations of REINFORCE algorithms have been designed for language models, such as ReMax [141], which implements a baseline normalization designed specifically to accommodate the sources of uncertainty from reward model inference. • Some foundation models, such as Apple Intelligence Foundation Models [142] or Kimi k1.5 reasoning model [143], have used variants of Mirror Descent Policy Optimization (MDPO) [144]. Research is still developing further on the fundamentals here [145], but Mirror Descent is an optimization method rather than directly a policy gradient algorithm. What is important here is that it is substituted in very similarly to existing RL infrastructure. • Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) proposes 4 modifications to GRPO to better su


d Clip and Dynamic sAmpling Policy Optimization (DAPO) proposes 4 modifications to GRPO to better suit reasoning language models, where long traces are needed and new, underutilized tokens need to be increased in probability [129]. The changes are: 1, have two different clip hyperparameters, εlow and εhigh, so clipping on the positive side of the logratio can take bigger steps for better exploration; 2, dynamic sampling, which removes all samples with reward = 0 or reward = 1 for all samples in the batch (no learning signal); 3, use the per-token loss as discussed above in Implementation: GRPO; and 4, a soft penalty on samples that are too long to avoid trying to learn from truncated answers.


and 4, a soft penalty on samples that are too long to avoid trying to learn from truncated answers. • Value-based Augmented Proximal Policy Optimization (VAPO) [146] combines optimizations from DAPO (including clip-higher, token-level policy-gradient, and different length normalization) with insights from Value-Calibrated PPO [147] to pretrain the value function and length-adaptive GAE to show the promise of valuebased methods relative to GRPO. 6.5 Suggested Experiments The companion implementation in code/policy_gradients/ is designed for small, observable RL runs. The default configs train Qwen/Qwen3-1.7B on the spell_backward procedural task from reasoning-gym, which is a good first exercise because failures and partial progress are easy to inspect. 1.


g-gym, which is a good first exercise because failures and partial progress are easy to inspect. 1. Run the word reversal task with GRPO. cd code/ uv run python -m policy_gradients.train --config policy_gradients/configs/grpo.yaml Track avg_correctness, avg_format, and avg_binary. The useful first question is whether each prompt group contains contrast: if all sampled completions are right or all are wrong, a group-relative update has little learning signal. 2. Compare group-relative and single-sample estimators. Run the matched starting configs: cd code/ uv run python -m policy_gradients.train --config policy_gradients/configs/reinforce.yaml uv run python -m policy_gradients.train --config policy_gradients/configs/rloo.yaml rlhfbook.com 89


uv run python -m policy_gradients.train --config policy_gradients/configs/grpo.yaml Compare how quickly the correctness signal improves and how noisy the loss is. RLOO and GRPO should make the role of within-prompt baselines much more concrete than the equations alone. 3. Sweep the contrast knobs. Copy policy_gradients/configs/grpo.yaml and vary num_rollouts, temperature, data.size, and format_weight. Small num_rollouts reduces group contrast; very low temperature can collapse samples; very high temperature can generate too many malformed answers. This is the simplest way to see why RLVR recipes often spend so much effort on sampling settings before touching the optimizer. 4. Move from toy rewards toward math.


ch effort on sampling settings before touching the optimizer. 4. Move from toy rewards toward math. For GSM8K-style experiments, start with the code/reward_models/train_orm.py and code/rejection_sampling/ examples before adding a new online RL environment. A good contribution would be a small reasoning-gym or GSM8K policy-gradient config that runs on a sub-1B Qwen model and reports the same group-contrast diagnostics. rlhfbook.com 90


7 Reasoning and Inference-Time Scaling Reasoning models and inference-time scaling enabled a massive step in language model performance at the end of 2024, through 2025, and into the future. Inference-time scaling is the ability to improve model performance by using more computation during generation, such as producing longer reasoning chains or sampling multiple responses. Language models trained to think extensively before answering exploit this property remarkably well. These models, trained with a large amount of reinforcement learning with verifiable rewards (RLVR) [6], still utilize large amounts of RLHF. In this chapter we review the path that led the AI community to a transformed appreciation for RL’s potential in language models, review the fundamentals of RLVR, highlight key work


reciation for RL’s potential in language models, review the fundamentals of RLVR, highlight key works, and point to the future debates that will define the area in the next few years. 7.1 The Role of RLVR To start, at the 2016 edition of the Neural Information Processing Systems (NeurIPS) conference, Yann LeCun first introduced his now-famous cake metaphor for where learning happens in modern machine learning systems: If intelligence is a cake, the bulk of the cake is unsupervised learning, the icing on the cake is supervised learning, and the cherry on the cake is reinforcement learning (RL). This analogy is now largely complete with modern language models and recent changes to the post-training stack.


is now largely complete with modern language models and recent changes to the post-training stack. RLHF was the precursor to this, and RL for reasoning models, primarily on math, code, and science topics, was its confirmation. In this analogy: • Self-supervised learning on vast swaths of internet data makes up the majority of the cake (especially when viewed in compute spent in FLOPs), • The beginning of post-training in supervised fine-tuning (SFT) for instructions tunes the model to a narrower distribution, and • Finally “pure” reinforcement learning (RL) is the cherry on top. The scaled up reinforcement learning used to create the new “reasoning” or “thinking” models is this finishing piece (along with the help of RLHF, which isn’t considered classical RL, as we’ll explain).


nishing piece (along with the help of RLHF, which isn’t considered classical RL, as we’ll explain). This little bit of reasoning training emerged with thinking models that use a combination of the post-training techniques discussed in this book to align preferences along with RL training on verifiable domains to dramatically increase capabilities such as reasoning, coding, and mathematics problem solving. The training method for these models, Reinforcement Learning with Verifiable Rewards (RLVR) [6], proceeds very similarly to RLHF, but it makes the reward model optional in lieu of a scoring function that returns a positive reward when the answer is correct and 0 otherwise. For example, consider how different it is to score responses for RLHF versus RLVR.


and 0 otherwise. For example, consider how different it is to score responses for RLHF versus RLVR. In RLHF, a reward model must evaluate subjective qualities: Prompt: Explain the concept of opportunity cost in economics. Response: Opportunity cost is the value of the next best alternative you give rlhfbook.com 91


up when making a decision. For example, if you spend an hour studying instead of working, the opportunity cost is the wages you could have earned. . . Scoring this response requires judging clarity, accuracy, completeness, and helpfulness – all qualities that require learned preferences and lack a definitive correct answer. In contrast, RLVR uses verification functions that return definitive scores. For mathematics: Prompt: What is the sum of all prime numbers less than 20? Response: The prime numbers less than 20 are 2, 3, 5, 7, 11, 13, 17, and 19. Adding them: 2 + 3 = 5, then 5 + 5 = 10, then 10 + 7 = 17, then 17 + 11 = 28, then 28 + 13 = 41, then 41 + 17 = 58, and finally 58 + 19 = 77. The answer is 77 .


en 17 + 11 = 28, then 28 + 13 = 41, then 41 + 17 = 58, and finally 58 + 19 = 77. The answer is 77 . Verification: extracted_answer == 77 →Reward = 1 The \boxed{} notation is a convention adopted from mathematical typesetting that makes answer extraction straightforward – a simple regular expression can pull the final answer from the response, regardless of how the model arrived at it. Note that other answer extraction methods exist, such as just using the phrase “The answer is:” as also shown above, special tokens like <answer>, or delimiters like ####. For code generation, verification often takes the form of unit tests: Prompt: Write a Python function fib(n) that returns the nth Fibonacci number, where fib(0) = 0 and fib(1) = 1.


te a Python function fib(n) that returns the nth Fibonacci number, where fib(0) = 0 and fib(1) = 1. Response: def fib(n): if n < 2: return n return fib(n - 1) + fib(n - 2) Verification (unit tests): assert fib(0) == 0 # base case assert fib(1) == 1 # base case assert fib(10) == 55 # larger value (All tests pass →Reward = 1) Unit tests are natural verification functions for code: they execute the model’s solution against known input-output pairs. A common form of scoring is to perform simple gating: If all assertions pass, the reward is 1; if any fail, the reward is 0. Other setups use partial credit proportional to the number of tests passed. For both these examples, no learned reward model is needed and most setups go without one (because the models are robust to over-optimization in thes


is needed and most setups go without one (because the models are robust to over-optimization in these domains), but one can be used with a linear combination of rewards. The ideas behind RLVR are not new to the RL literature, where the core idea of taking gradient updates based on whether the answer is correct is almost the textbook definition of reinforcement learning. The innovations when applying this to language models are largely how to apply it while maintaining the strong, general capabilities of the model being fine-tuned. Within that, there are many related ideas in the language modeling literature where the model learns from feedback regarding the correctness of the answer. Originally, in the work I was a part of that coined the term RL with Verifiable Rewards (RLVR) [6], the met


lly, in the work I was a part of that coined the term RL with Verifiable Rewards (RLVR) [6], the method was to be named RL with Ground Truth rewards (RLGT). Yet RLVR is subtly different from learning solely from ground truth answers. In domains like mathematics, a single ground truth answer is available to verify solutions, as we saw above. In other domains, such as code generation or precise instruction following, answers can be rlhfbook.com 92


verified with a checking function (e.g., a unit test), even when there are multiple correct solutions rather than just a single ground truth answer. The core of progress on RLVR is having a variety and depth of these verifiable problems, even if the exact solution isn’t known a priori. Figure 25: RLVR in the form of an RL feedback loop. Instead of a reward model, a verification function is used. The first models to successfully deploy this type of training were OpenAI’s o1 [53] and the open-weight model DeepSeek R1 [15]. Soon after, the entire AI industry prioritized this training process and model style. The core change here is more a reallocation of the stages of training and the priority of different behaviors rather than this type of RL setup being entirely new.


ining and the priority of different behaviors rather than this type of RL setup being entirely new. Reasoning models brought an era where scaling RL training is expected. As for the type of behavior these models exhibit, consider the following example with DeepSeek V3 0325 versus their reasoning model, DeepSeek R1, on the query Write me a short poem about a goldfish. DeepSeek V3 is very to the point: Response: Goldfish Dreams In a bowl of glass so clear and round, A tiny goldfish swims without a sound. With scales that gleam like sunlight’s kiss, He drifts along in watery bliss. A flick of fin, a flash so bright, Chasing bubbles in the light. Does he dream of oceans wide? Or is his little world enough—his pride? rlhfbook.com 93


Meanwhile, DeepSeek R1’s experience is different. It first generates numerous thinking tokens to craft a response before responding to the user. The answer appears as: Response: <thinking> Okay, the user wants a short poem about a goldfish. Let me start by brainstorming some goldfish-related imagery. Goldfish are often in bowls, right? So maybe start with the setting. Words like “gilded bowl” or “crystal globe” could work. Next, think about the movement of the goldfish. They swim smoothly, so words like “dart”, “glide”, or “swirl” come to mind. Maybe describe their fins as “silken” to emphasize delicacy. Then, consider the environment around the goldfish. Sunlight through water creates shimmering effects. Words like “sunbeams dance” or “ripples spin” might capture that.


water creates shimmering effects. Words like “sunbeams dance” or “ripples spin” might capture that. Including elements like castles or pebbles adds a playful touch. Themes? Maybe the contrast between the goldfish’s small world and its perception of freedom. The bowl as a universe, tiny but complete. Ending with a sense of mystery or silent observation could give depth. Need to keep it concise, maybe four stanzas with a rhyme scheme. Let me check the rhythm and flow. Avoid clichés but keep it accessible. Alright, let’s piece it together. </thinking> Gilded Dreams in Liquid Glass A gilded bowl, a crystal globe, Where tiny tides in silence flow— A flash of orange, a whispered robe, You dart through sunlit indigo. Your world is small, yet boundless too— Each pebble, castle, weed unfurled.


through sunlit indigo. Your world is small, yet boundless too— Each pebble, castle, weed unfurled. The surface bends the sky to blue, And ripples spin your liquid world. You glide through myths we’ll never know, A silent monarch, veiled in mist. Do dreams drift where the bubbles go? Or memories in amethyst? We watch you trace your endless sphere— A universe within a tear. The tokens in the <thinking> tags are the model’s reasoning. For more complex problems the reasoning stage can take thousands of tokens before producing an answer. So, longcontext language models were a prerequisite to advanced reasoning behavior, but that is not the focus of this chapter. The core intuition for how this training works is that for a given model, we repeat the following cycle: 1.


intuition for how this training works is that for a given model, we repeat the following cycle: 1. Sample multiple answers to multiple questions, 2. Take gradient steps towards the answers that are correct, and 3. Repeat, revisiting the same data. Remarkably, this extremely simple approach (when done with a careful distribution of data and stable training infrastructure) helps the models learn by revisiting the same questions again and again. Even more remarkable is that the improvements on these training questions generalize to questions and (some) domains the models have never seen! This simple approach allows the models to lightly search over behavior space and the RL algorithm increases the likelihood of behaviors that are correlated with correct answers. rlhfbook.com 94


7.2 The Origins of New Reasoning Models Here we detail the high-level trends that led to the explosion of reasoning models in 2025. 7.2.1 Why Does RL Work Now? Despite many, many takes that “RL doesn’t work yet” [148] and papers detailing deep reproducibility issues with RL [149], the field overcame them to find high-impact applications. Some are covered in this book, such as ChatGPT’s RLHF and DeepSeek R1’s RLVR, but many others exist, including improving chip design [150], mastering video gameplay [151], self-driving [152], and more. The takeoff of RL-focused training on language models indicates progress on many fundamental issues for the research area, including: • Stability of RL can be solved: For its entire existence, the limiting factor on RL’s adoption has been stability.


L can be solved: For its entire existence, the limiting factor on RL’s adoption has been stability. This manifests in two ways. First, the learning itself can be fickle and not always work. Second, the training itself is known to be more brittle than standard language model training and more prone to loss spikes, crashes, etc. Countless new model releases are using this style of RL training with verifiable rewards on top of a pretrained base model and substantial academic uptake has occurred. The technical barriers to entry on RL are at an all-time low. • Open-source versions already “exist”: Many tools already exist for training language models with RLVR and related techniques. Examples include TRL [47], Open Instruct [6], veRL [153], and OpenRLHF [154], where many of these are building o


lude TRL [47], Open Instruct [6], veRL [153], and OpenRLHF [154], where many of these are building on optimizations from earlier in the arc of RLHF and post-training. The accessibility of tooling is enabling a large and accelerating body of research. Multiple resources point to RL training for reasoning only being viable with leading models coming out from about 2024 onwards, indicating that a certain level of underlying capability was needed in the models before reasoning training was possible. 7.2.2 RL Training vs. Inference-Time Scaling Training with reinforcement learning to elicit reasoning behaviors and performance on verifiable domains is closely linked to the ideas of inference-time scaling.


ors and performance on verifiable domains is closely linked to the ideas of inference-time scaling. Inference-time scaling, also called test-time scaling, is the general class of methods that use more computational power at inference in order to perform better at downstream tasks. Methods for inferencetime scaling were studied before the release of DeepSeek R1 and OpenAI’s o1, which both massively popularized investment in RL training specifically. Examples include value-guided sampling [155] or repeated random sampling with answer extraction [156]. Beyond this, inference-time scaling can be used to improve more methods of AI training beyond chain-ofthought reasoning to solve problems, such as with reward models that consider the options deeply [84] [157].


easoning to solve problems, such as with reward models that consider the options deeply [84] [157]. RL training is a short path to inference-time scaling laws being used, but in the long-term we will have more methods for eliciting the inference-time tradeoffs we need for best performance. Training models heavily with RL often enables them to generate more tokens per response in a way that is strongly correlated with improved downstream performance (although this sequence length increase is the default, research also exists explicitly on improving performance without relying on this inference-time scaling). This is a substantial shift from rlhfbook.com 95


the length-bias seen in early RLHF systems [10], where the human preference training had a side effect of increasing the response average length for marginal gains on preference rankings. Other than the core RL trained models there are many methods being explored to continue to push the limits of reasoning and inference-time compute. These are largely out of the scope of this book due to their rapidly evolving nature, but they include distilling reasoning behavior from a larger RL trained model to a smaller model via instruction tuning [158], composing more inference calls [159], and more. What is important here is the correlation between downstream performance and an increase in the number of tokens generated – otherwise it is just wasted energy.


performance and an increase in the number of tokens generated – otherwise it is just wasted energy. 7.2.3 The Future (Beyond Reasoning) of RLVR In many domains, these new flavors of RLVR are much more aligned with the goals of developers by being focused on performance rather than behavior. Standard fine-tuning APIs generally use a parameter-efficient fine-tuning method such as LoRA (Low-Rank Adaptation, a parameter-efficient method that trains only small added matrices rather than all model weights, also referred to as parameter-efficient fine-tuning, PEFT) with supervised fine-tuning on instructions. Developers pass in prompts and completions and the model is tuned to match that by updating model parameters to match the completions, which increases the prevalence of features from your da


g model parameters to match the completions, which increases the prevalence of features from your data in the model’s generations. RLVR is focused on matching answers. Given queries and correct answers, RLVR helps the model learn to produce the correct answers. While standard instruction tuning is done with 1 or 2 epochs of loss updates over the data, RLVR gets its name by doing hundreds or thousands of epochs over the same few data points to give the model time to learn new behaviors. This can be viewed as reinforcing positive behaviors that would work sparingly in the base model version into robust behaviors after RLVR. The scope of RL training for language models continues to grow: The biggest takeaway from o1 and R1 on a fundamental scientific level was that we have even more ways to t


ggest takeaway from o1 and R1 on a fundamental scientific level was that we have even more ways to train language models to potentially valuable behaviors. The more open doors that are available to researchers and engineers, the more optimism we should have about AI’s general trajectory. 7.3 Understanding Reasoning Training Methods The investment in reasoning has instigated a major evolution in the art of how models are trained to follow human instructions. These recipes still use the common pieces discussed in earlier chapters (as discussed in Chapter 3 with the overview of DeepSeek R1’s recipe), including instruction fine-tuning, reinforcement learning from human feedback, and reinforcement learning with verifiable rewards (RLVR).


nforcement learning from human feedback, and reinforcement learning with verifiable rewards (RLVR). The core change is using far more RLVR and applying the other training techniques in different orders – traditionally for a reasoning model the core training step is either a large-scale RL run or a large-scale instruction tuning run on outputs of another model that had undergone a substantial portion of RLVR training (referred to as distillation). rlhfbook.com 96


7.3.1 Reasoning Research Before OpenAI o1 or DeepSeek R1 Before the takeoff of reasoning models, a substantial effort was made to understand how to train language models to be better at verifiable domains. The main difference between these works below is that their methodologies did not scale to the same level as those used in DeepSeek R1 and subsequent models, or they resulted in models that made sacrifices in overall performance in exchange for higher mathematics or coding abilities. The underlying ideas and motivations are included to paint a broader picture for how reasoning models emerged within the landscape. Some of the earliest efforts to train language models on verifiable domains include the self-taught reasoner (STaR) line of work [160] [161] and TRICE [162], which both used gro


nclude the self-taught reasoner (STaR) line of work [160] [161] and TRICE [162], which both used ground-truth reward signals to encourage chain-of-thought reasoning in models throughout 2022 and 2023. STaR effectively approximates the policy gradient algorithm, but in practice filters samples differently and uses a cross-entropy measure instead of a log-probability, and Quiet-STaR expands on this with very related ideas of recent reasoning models by having the model generate tokens before trying to answer the verifiable question (which helps with training performance). TRICE [162] also improves reasoning by generating traces and then optimizing with a custom Markov chain Monte Carlo inspired expectation maximization algorithm.


then optimizing with a custom Markov chain Monte Carlo inspired expectation maximization algorithm. VinePPO [163] followed these and used a setup that shifted closer to modern reasoning models. VinePPO uses a PPO-based algorithm with binary rewards for math question correctness, training on GSM8K and MATH. Other work before OpenAI’s o1 and DeepSeek R1 used code execution as a feedback signal for training [164], [165] or verification for theorem proving (called Reinforcement Learning from Verifier Feedback, RLVF, here) [166]. Tülu 3 expanded on these methods by using a simple PPO trainer to reward completions with correct answers – most importantly while maintaining the model’s overall performance on a broad suite of evaluations.


most importantly while maintaining the model’s overall performance on a broad suite of evaluations. The binary rewards of Tülu 3 and modern reasoning training techniques can be contrasted with the iterative approach of STaR or the log-likelihood rewards of Quiet-STaR. 7.3.2 Early Reasoning Models A summary of the foundational reasoning research reports, some of which are accompanied by open data and model weights, following DeepSeek R1 is shown in tbl. 4. Table 4: A summary of the notable reasoning model technical reports in 2025, the first year of substantial inference-time scaling with RLHF. Date Name TLDR Open weights Open data 202501-22 DeepSeek R1 [15] RL-based upgrade to DeepSeek, big gains on math & code reasoning Yes No 202501-22 Kimi 1.5 [143] Scales PPO/GRPO on Chinese/English da


gains on math & code reasoning Yes No 202501-22 Kimi 1.5 [143] Scales PPO/GRPO on Chinese/English data; strong AIME maths No No 202503-31 Open-ReasonerZero [167] Fully open replication of base model RL Yes Yes 202504-10 Seed-Thinking 1.5 [62] ByteDance RL pipeline with dynamic CoT gating Yes No rlhfbook.com 97


Date Name TLDR Open weights Open data 202504-30 Phi-4 Reasoning [168] 14B model; careful SFT→RL; excels at STEM reasoning Yes No 202505-02 LlamaNemotron [169] Multi-size “reasoning-toggle” models Yes Yes 202505-12 INTELLECT-2 [134] First, publicly documented globally-decentralized RL training run Yes Yes 202505-12 Xiaomi MiMo [61] End-to-end reasoning pipeline from preto post-training Yes No 202505-14 Qwen 3 [60] Similar to R1 recipe applied to new models Yes No 202505-21 HunyuanTurboS [170] Mamba-Transformer MoE, adaptive long/short CoT No No 202505-28 Skywork OR-1 [171] RL recipe avoiding entropy collapse; beats DeepSeek on AIME Yes Yes 202506-04 Xiaomi MiMo VL [172] Adapting reasoning pipeline end-to-end to include multi-modal tasks Yes No 202506-04 OpenThoughts [173] Public 1.2M-exampl


eline end-to-end to include multi-modal tasks Yes No 202506-04 OpenThoughts [173] Public 1.2M-example instruction dataset distilled from QwQ-32B Yes Yes 202506-10 Magistral [174] Pure RL on Mistral 3; multilingual CoT; small model open-sourced Yes No 202506-16 MiniMax-M1 [123] Open-weight 456B MoE hybrid/Lightning Attention reasoning model; 1M context; RL w/CISPO; releases 40K/80K thinking-budget checkpoints Yes No 202507-10 Kimi K2 [175] 1T MoE (32B active) with MuonClip (QK-clip) for stability; 15.5T token pretrain without loss spikes; multi-stage post-train with agentic data synthesis + joint RL; releases base + post-trained checkpoints. Yes No 202507-28 GLM-4.5 [176] Open-weight 355B-A32B MoE “ARC” model with thinking/non-thinking modes; 23T-token multi-stage training + post-train w/ e


B MoE “ARC” model with thinking/non-thinking modes; 23T-token multi-stage training + post-train w/ expert iteration and RL; releases GLM-4.5 + GLM-4.5-Air (MIT). Yes No 202508-20 Nemotron Nano 2 [177] Hybrid Mamba-Transformer for long “thinking traces”; FP8 pretraining at 20T tokens then compression/distillation; explicitly releases multiple checkpoints plus “majority” of pre/post-training datasets. Yes Yes (most) 202509-09 K2-Think [178] Parameter-efficient math reasoning system: a 32B open-weights model with test-time scaling recipe; positioned as fully open incl. training data/code (per release materials). Yes Yes rlhfbook.com 98


Date Name TLDR Open weights Open data 202509-23 LongCat-FlashThinking [179] 560B MoE reasoning model; report is explicit about a staged recipe from long-CoT cold start to large-scale RL; open-source release. Yes No 202510-21 Ring-1T [180] Trillion-scale “thinking model” with RL scaling focus; report frames bottlenecks/solutions for scaling RL at 1T and releases an open model. Yes No 202511-20 OLMo 3 Think [18] Fully open “model flow” release: reports the entire lifecycle (stages, checkpoints, and data points) and positions OLMo 3 Think 32B as a flagship open thinking model. Yes Yes 202512-02 DeepSeek V3.2 [181] Open-weight MoE frontier push with a report that foregrounds attention efficiency changes, RL framework upgrades, and data synthesis for agentic/reasoning performance.


on efficiency changes, RL framework upgrades, and data synthesis for agentic/reasoning performance. Yes No 202512-05 K2-V2 [182] 70B dense “360-open” model trained from scratch; with 3-effort SFT-only post-training for controllable thinking. Yes Yes 202512-15 Nemotron 3 Nano [183] 30B-A3B MoE hybrid Mamba-Transformer; pretrain on 25T tokens and includes SFT + large-scale RL; explicitly states it ships weights + recipe/code + most training data. Yes Yes (most) 202512-16 MiMo-V2-Flash [184] 309B MoE (15B active) optimized for speed: hybrid SWA/GA attention (5:1, 128-token window) + lightweight MTP; FP8 pretrain on 27T tokens; post-train with MOPD + large-scale agentic RL for reasoning/coding. Yes No 7.3.3 Common Practices in Training Reasoning Models In this section we detail common methods


Yes No 7.3.3 Common Practices in Training Reasoning Models In this section we detail common methods used to sequence training stages and modify data to maximize performance when training a reasoning model. Note that these papers could have used a listed technique and not mentioned it, whereas their peers do, so these examples are a subset of known implementations and should be used as a reference, but not a final proclamation on what an optimal recipe is. • Offline difficulty filtering: A core intuition of RLVR is that models can only learn from examples where there is a gradient. If the starting model for RLVR can solve a problem either 100% of the time or 0% of the time, there will be no gradient between different completions to the prompt (i.e., all strategies appear the same to the pol


radient between different completions to the prompt (i.e., all strategies appear the same to the policy rlhfbook.com 99


gradient algorithm). Many models have used difficulty filtering before starting largescale RL to restrict the training problems to those that the starting point model solves only 20-80% of the time. This data is collected by sampling N, e.g. 16, completions to each prompt in the training set and verifying what percentage are correct. Forms of this were used by Seed-Thinking 1.5, Open Reasoner Zero, Phi 4, INTELLECT-2, MiMo RL, Skywork OR-1, and others. • Per-batch online filtering (or difficulty curriculums throughout training): To complement the offline filtering to find the right problems to train on, another major question is: what order should the problems be presented to the model during learning?


nother major question is: what order should the problems be presented to the model during learning? In order to address this, many models use online filtering of questions in the batch, prebuilt curriculums/data schedulers, saving harder problems for later in training, or other ideas to improve long-term stability. Related ideas are used by Kimi 1.5, Magistral, Llama-Nemotron, INTELLECT-2, MiMo-RL, Hunyuan-TurboS, and others. • Remove KL penalty: As the length of RL runs (in any metric, total GPU hours, FLOPS, or RL steps) increased for reasoning models relative to RLHF training, and the reward function became less prone to over-optimization, many models removed the KL penalty constraining the RL-learned policy to be similar to the base model used at the start of training.


y constraining the RL-learned policy to be similar to the base model used at the start of training. This allows the model to further explore during its training. This was used by RAGEN [185], Magistral, OpenReasonerZero, Skywork OR-1, and others. • Relaxed policy-gradient clipping: New variations of the algorithm GRPO, such as DAPO [129], proposed modifications to the two-sided clipping objective used in GRPO (or PPO) in order to enable better exploration. Clipping has also been shown to cause potentially spurious learning signals when rewards are imperfect [186]. This two-sided clipping with different ranges per gradient direction is used by RAGEN, Magistral, INTELLECT-2, and others. • Off-policy data (or fully asynchronous updates): As the length of completions needed to solve tasks with


policy data (or fully asynchronous updates): As the length of completions needed to solve tasks with RL increases dramatically with harder problems (particularly in the variance of the response length, where there are often outliers with extremely long lengths), compute in RL runs can sit idle. To solve this, training is moving to asynchronous updates or changing how problems are arranged into batches to improve overall throughput. Partial-to-full asynchronous (off-policy) data is used by Seed-Thinking 1.5, INTELLECT-2, and others. • Additional format rewards: In order to make the reasoning process predictable, many models add minor rewards to make sure the model follows the correct format of e.g. <think>...</think> before an answer.


ards to make sure the model follows the correct format of e.g. <think>...</think> before an answer. This is used by DeepSeek R1, OpenReasonerZero, Magistral, Skywork OR-1, and others. • Language consistency rewards: Similar to format rewards, some multilingual reasoning models use language consistency rewards to prioritize models that do not change languages while reasoning (for a better and more predictable user experience). These include DeepSeek R1, Magistral, and others. • Length penalties: Many models use different forms of length penalties during RL training to either stabilize the learning process over time or to mitigate overthinking on hard problems. Some examples include Kimi 1.5 progressively extending the target length to combat overthinking (while training accuracy is high acr


rogressively extending the target length to combat overthinking (while training accuracy is high across difficulty curriculum) or INTELLECT-2 running a small length penalty throughout. Progressively extending the training sequence length mitigates overthinking by forcing the model to first reason effectively in a domain with a more limited thinking budget, and then transitioning to longer training where the model can use those behaviors efficiently on rlhfbook.com 100


more complex problems. Others use overlong filtering and other related implementations to improve throughput. • Loss normalization: There has been some discussion (see the chapter on policy gradients or [118]) around potential length or difficulty biases introduced by the pergroup normalization terms of the original GRPO algorithm. As such, some models, such as Magistral or MiMo, chose to normalize either losses or advantages at the batch level instead of the group level. • Parallel test-time compute scaling: Combining answers from multiple parallel, independently-sampled rollouts can lead to substantial improvements over using the answer from a single rollout. The most naive form of parallel test-time compute scaling, as done in DeepSeek-R1, Phi-4, and others, involves using the answer re


l test-time compute scaling, as done in DeepSeek-R1, Phi-4, and others, involves using the answer returned by a majority of rollouts as the final answer. A more advanced technique is to use a scoring model trained to select the best answer out of the answers from the parallel rollouts. As of 2026, this technique had not become common in open, documented reasoning model recipes, but it was mentioned in the Claude 4 announcement [187] and used in DeepSeek-GRM [157]. Complementing the common techniques, there are also many common findings on how reasoning training can create useful models without sacrificing ancillary capabilities: • Text-only reasoning boosts multimodal performance: Magistral, MiMo-VL, and others find that training a multimodal model and then performing text-only reasoning t


MiMo-VL, and others find that training a multimodal model and then performing text-only reasoning training after this multimodal training can improve multimodal performance in the final model. • Toggleable reasoning with system prompt (or length control): Llama-Nemotron, Nemotron Nano, Qwen 3, SmolLM 3, and others use specific system prompts (possibly in combination with length-controlled RL training [188]) to enable a toggleable on/off thinking length for the user. Other open models, such as OpenAI’s GPT-OSS and LLM360’s K2-V2 [182] adopt a low-medium-high reasoning effort set in the system prompt, but training methods for this type of behavior are not as well documented. 7.4 Looking Ahead The reasoning model landscape is evolving faster than any area of AI research in recent memory, and


The reasoning model landscape is evolving faster than any area of AI research in recent memory, and some of the common practices listed here will inevitably be superseded by new techniques. Several efforts are underway to systematically understand what makes reasoning training work. OLMo 3 Think [18] represents the most comprehensive open documentation of a reasoning model’s full training lifecycle, providing checkpoints and data at each stage for the research community to study, and concluding with a nearly 4-week-long training run on 220 GPUs. Similarly, work on understanding the scaling properties of RL for reasoning [17] is beginning to formalize relationships between compute, data, and performance that were previously only intuited by practitioners.


onships between compute, data, and performance that were previously only intuited by practitioners. What remains clear is that reinforcement learning has graduated from the “cherry on top” in the cake metaphor to a load-bearing component of frontier model training. The minor techniques in this chapter around the idea of RLVR – difficulty filtering, format rewards, and the rest – are not the final answers, but they represent the field’s current best understanding of how to elicit reasoning from language models. The next generation of methods will likely rlhfbook.com 101


look different, but they will build on the foundations established here. rlhfbook.com 102


8 Direct-Alignment Algorithms Direct Alignment Algorithms (DAAs) allow one to update models to solve the same RLHF objective without ever training an intermediate reward model or using reinforcement learning optimizers. DAAs solve the same preference learning problem we’ve been studying (with literally the same data!), in order to make language models more aligned, smarter, and easier to use. The lack of a reward model and online optimization makes DAAs far simpler to implement, reducing compute spent during training and making experimentation easier. This chapter details the complex mathematics done to derive these algorithms, and then shows that the sometimes tedious derivations result in simple implementations.


algorithms, and then shows that the sometimes tedious derivations result in simple implementations. The most prominent DAA and one that catalyzed an entire academic movement of aligning language models is Direct Preference Optimization (DPO) [25]. At its core, DPO uses gradient ascent to solve the same constrained RLHF objective (see Chapter 3): max π Ex∼DEy∼π(y|x) [rθ(x, y)] −βDKL (π(y|x)∥πref(y|x)) (78) Since its release in May of 2023, after a brief delay where the community figured out the right data and hyperparameters to use DPO with (specifically, surprisingly low learning rates), many popular models have used DPO or its variants, from Zephyr-β kickstarting it in October of 2023 [26], Llama 3 Instruct [29], Tülu 2 [27] and 3 [6], Nemotron 4 340B [30], and others.


tober of 2023 [26], Llama 3 Instruct [29], Tülu 2 [27] and 3 [6], Nemotron 4 340B [30], and others. Technically, Sequence Likelihood Calibration (SLiC-HF) was the first modern direct alignment algorithm released [189], but it did not catch on due to a combination of factors (unwinding the adoption of research methods is always a tricky task). The most impactful part of DPO and DAAs is lowering the barrier to entry to experimenting with language model post-training – it uses less compute, is easier to implement from scratch, and is easier to get working on both toy and production examples. Throughout this chapter, we use x to denote prompts and y to denote completions. This notation is common in the language model literature, where methods operate on full promptcompletion pairs rather than


in the language model literature, where methods operate on full promptcompletion pairs rather than individual tokens. 8.1 Direct Preference Optimization Here we explain intuitions for how DPO works and re-derive the core equations fully. 8.1.1 How DPO Works DPO at a surface level is directly optimizing a policy to solve the RLHF objective. The loss function for this, which we will revisit below in the derivations, compares how much the learned policy’s probability of chosen and rejected completions has shifted relative to a reference model. The loss function derived from a Bradley-Terry reward model follows: LDPO(πθ; πref) = −E(x,yc,yr)∼D  log σ  β log πθ(yc | x) πref(yc | x) −β log πθ(yr | x) πref(yr | x)  (79) Inside the sigmoid, the first term β log πθ(yc|x) πref(yc|x) measures how


| x) πref(yr | x)  (79) Inside the sigmoid, the first term β log πθ(yc|x) πref(yc|x) measures how much the policy has increased the probability of the chosen completion relative to the reference model, and the second rlhfbook.com 103


term does the same for the rejected completion. The loss decreases when the chosen shift exceeds the rejected shift – i.e. when the policy learns to prefer the right response. Throughout, β is a hyperparameter balancing the reward optimization to the KL divergence between the final model and the initial reference (i.e. balancing over-optimization, a crucial hyperparameter when using DPO correctly). This relies on the implicit reward for DPO training that replaces using an external reward model, which is a log-ratio of probabilities: r(x, y) = β log πr(y | x) πref(y | x) (80) where πr(y | x) is the exact, optimal reward policy that we are solving for. This comes from deriving the Bradley-Terry reward with respect to an optimal policy (shown in eq.


. This comes from deriving the Bradley-Terry reward with respect to an optimal policy (shown in eq. 95), as shown in the Bradley-Terry model section of Chapter 5. Essentially, as stated in the DPO paper, this reparameterization gives us “the probability of human preference data in terms of the optimal policy rather than the reward model” – meaning we can bypass learning an explicit reward model entirely. Let us consider the loss shown in eq. 79 that the optimizer must decrease. Here, the loss will be lower when the log-ratio of the chosen response is bigger than the log-ratio of the rejected response (normalized by the reference model). In practice, this is a sum of log-probabilities of the model across the sequence of tokens in the data presented.


his is a sum of log-probabilities of the model across the sequence of tokens in the data presented. Hence, DPO is increasing the gap in relative log-probabilities between the chosen and rejected responses. With the reward in eq. 80, we can write the gradient of the loss to further interpret what is going on: ∇θLDPO(πθ; πref) = −βE(x,yc,yr)∼D h w · (∇θ log π(yc | x) −∇θ log π(yr | x)) i (81) where w = σ(rθ(x, yr) −rθ(x, yc)). Here, the gradient solves the above objective by doing the following: • The first term within the sigmoid function, σ(·), creates a weight of the parameter update from 0 to 1 that is higher when the reward estimate is incorrect. When the rejected sample is preferred over the chosen, the weight update should be larger!


correct. When the rejected sample is preferred over the chosen, the weight update should be larger! • Second, the terms in the inner brackets [·] increase the likelihood of the chosen response yc and decrease the likelihood of the rejected yr. • These terms are weighted by β, which controls how the update balances ordering the completions correctly relative to the KL divergence. The core intuition is that DPO is fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form (eq. 95, thanks to gradient descent and our ML tools). Because the DPO loss is directly differentiable, it is straightforward to compute the exact gradient, rather than needing to estimate it by training a reward model and sampling completions to score.


t, rather than needing to estimate it by training a reward model and sampling completions to score. What is often misunderstood is that DPO is learning a reward model at its core, hence the subtitle of the paper Your Language Model is Secretly a Reward Model. It is easy to confuse this with the DPO objective training a policy directly, hence studying the derivations below is good for a complete understanding. rlhfbook.com 104


With the implicit reward model learning, DPO is generating an optimal solution to the RLHF objective given the data in the dataset and the specific KL constraint in the objective β. Here, DPO solves for the exact policy given a specific KL divergence because the generations are not online as in policy gradient algorithms – a core difference from the RL methods for preference tuning. In many ways, this makes the β value easier to tune with DPO relative to online RL methods, but crucially and intuitively the optimal value depends on the model being trained and the data training it. At each batch of preference data, composed of many pairs of completions ychosen ≻yrejected, DPO takes gradient steps directly towards the optimal solution. It is far simpler than policy gradient methods.


adient steps directly towards the optimal solution. It is far simpler than policy gradient methods. Figure 26: When DPO first released it sparked a fierce debate in the research community about how to best do RLHF and preference learning. This meme does a great job capturing the sentiment, where the debate often felt forced and over the top, but many people both getting started and in top labs were getting immense benefit out of DPO. DPO simplicity meme, credit Tom Goldstein. 8.1.2 DPO Derivation The DPO derivation takes two primary parts. First, the authors show the form of the policy that optimally solved the RLHF objective used throughout this book. Next, they show how to arrive at that solution from pairwise preference data (i.e. a Bradley-Terry model). rlhfbook.com 105


8.1.2.1 Deriving the Optimal RLHF Solution To start, we should consider the RLHF optimization objective once again, here indicating we wish to maximize this quantity: max π Ex∼DEy∼π(y|x) [rθ(x, y)] −βDKL (π(y|x)∥πref(y|x)) (82) Here, the dual expectation only applies to the sampling to compute the expected reward, as the KL term is still an analytical expression. First, let us expand the definition of KLdivergence. Recall that DKL(π∥πref) = Ey∼π h log π(y|x) πref(y|x) i , where the π(y|x) weighting in the sum becomes the sampling distribution. Since both terms now share the same expectation over y ∼π(y|x), we can combine them: max π Ex∼DEy∼π(y|x)  r(x, y) −β log π(y|x) πref(y|x)  (83) Next, bring the negative sign out of the difference in brackets.


y) −β log π(y|x) πref(y|x)  (83) Next, bring the negative sign out of the difference in brackets. To do this, split it into two terms: = max π  Ex∼DEy∼π(y|x)[r(x, y)] −β Ex∼DEy∼π(y|x)  log π(y|x) πref(y|x)  (84) Then, multiply by −1 to convert the maximization into a minimization: = min π  −Ex∼DEy∼π(y|x)[r(x, y)] + β Ex∼DEy∼π(y|x)  log π(y|x) πref(y|x)  (85) Divide by β and recombine: = min π  Ex∼DEy∼π(y|x)  log π(y|x) πref(y|x) −1 β r(x, y)  (86) Next, we must introduce a partition function, Z(x): Z(x) = X y πref(y|x) exp  1 β r(x, y)  (87) The partition function acts as a normalization factor for the unnormalized density πref(y|x) exp  1 β r(x, y)  , thereby making it a valid probability function over y for each fixed x.


(y|x) exp  1 β r(x, y)  , thereby making it a valid probability function over y for each fixed x. The exact need for this will become clear shortly as we proceed with the derivation. With this substituted in, we obtain our intermediate transformation: min π Ex∼DEy∼π(y|x)  log π(y|x) 1 Z(x)πref(y|x) exp  1 β r(x, y)  −log Z(x)   (88) rlhfbook.com 106


To see how this is obtained, consider the internal part of the optimization in brackets of eq. 86: log π(y|x) πref(y|x) −1 β r(x, y) (89) Then, add log Z(x) −log Z(x) to both sides: = log π(y|x) πref(y|x) −1 β r(x, y) + log Z(x) −log Z(x) (90) Then, we group the terms: =  log π(y|x) πref(y|x) + log Z(x)  −log Z(x) −1 β r(x, y) (91) With log(x) + log(y) = log(x · y) (and moving Z to the denominator), we get: = log π(y|x) 1 Z(x)πref(y|x) −log Z(x) −1 β r(x, y) (92) Next, we expand 1 β r(x, y) to log exp 1 β r(x, y) and do the same operation to get eq. 88, which we slightly rewrite here: min π Ex∼D  Ey∼π(y|x)  log π(y|x) 1 Z(x)πref(y|x) exp  1 β r(x, y)   −log Z(x)   (93) With this optimization form, we need to actually solve for the optimal policy π∗.


log Z(x)   (93) With this optimization form, we need to actually solve for the optimal policy π∗. Since we introduced the partition function Z(x), thereby making the term 1 Z(x)πref(y|x) exp  1 β r(x, y)  a valid probability distribution over y, we can recognize that the inner expectation is in fact a proper KL-divergence! min π Ex∼D  DKL  π(y|x) 1 Z(x)πref(y|x) exp  1 β r(x, y)  −log Z(x)  (94) Since the term log Z(x) does not depend on π (the policy we are optimizing), we can ignore it. This leaves us with just the KL divergence between the policy we are learning and a form relating the partition, β, reward, and reference policy. Gibbs’ inequality tells us this is minimized at a distance of 0, only when the two quantities are equal!


’ inequality tells us this is minimized at a distance of 0, only when the two quantities are equal! Hence, we get an optimal policy: π∗(y|x) = π(y|x) = 1 Z(x)πref(y|x) exp  1 β r(x, y)  (95) rlhfbook.com 107


8.1.2.2 Deriving DPO Objectives for BT Models To start, recall from Chapter 5 on Reward Modeling and Chapter 11 on Preference Data that a Bradley-Terry model of human preferences is formed as: p∗(y1 ≻y2 | x) = exp (r∗(x, y1)) exp (r∗(x, y1)) + exp (r∗(x, y2)) (96) By manipulating eq. 95, we can solve for the optimal reward. First, take the logarithm of both sides: log π∗(y|x) = log  1 Z(x)πref(y|x) exp  1 β r∗(x, y)  (97) Expanding the right-hand side using log(abc) = log a + log b + log c: log π∗(y|x) = −log Z(x) + log πref(y|x) + 1 β r∗(x, y) (98) Rearranging to solve for r∗(x, y): 1 β r∗(x, y) = log π∗(y|x) −log πref(y|x) + log Z(x) (99) Multiplying both sides by β: r∗(x, y) = β log π∗(y | x) πref(y | x) + β log Z(x) (100) We then can substitute the reward into the Bradley-Terry equ


∗(y | x) πref(y | x) + β log Z(x) (100) We then can substitute the reward into the Bradley-Terry equation shown in eq. 96 to obtain: p∗(y1 ≻y2 | x) = exp  β log π∗(y1|x) πref(y1|x) + β log Z(x)  exp  β log π∗(y1|x) πref(y1|x) + β log Z(x)  + exp  β log π∗(y2|x) πref(y2|x) + β log Z(x)  (101) By decomposing the exponential expressions from ea+b to eaeb and then cancelling out the terms eβ log Z(x), this simplifies to: p∗(y1 ≻y2 | x) = exp  β log π∗(y1|x) πref(y1|x)  exp  β log π∗(y1|x) πref(y1|x)  + exp  β log π∗(y2|x) πref(y2|x)  (102) Then, multiply the numerator and denominator by exp  −β log π∗(y1|x) πref(y1|x)  to obtain: p∗(y1 ≻y2 | x) = 1 1 + exp  β log π∗(y2|x) πref(y2|x) −β log π∗(y1|x) πref(y1|x)  (103) rlhfbook.com 108


Finally, with the definition of a sigmoid function as σ(x) = 1 1+e−x , we obtain: p∗(y1 ≻y2 | x) = σ  β log π∗(y1 | x) πref(y1 | x) −β log π∗(y2 | x) πref(y2 | x)  (104) This is the likelihood of preference data under the Bradley-Terry model, given the optimal policy π∗. Recall from Chapter 5 on Reward Modeling that we derived the Bradley-Terry objective as maximizing the likelihood, or equivalently minimizing the negative log-likelihood, which gives us the loss: LDPO(πθ; πref) = −E(x,yc,yr)∼D [log p(yc ≻yr | x)] = −E(x,yc,yr)∼D  log σ  β log πθ(yc|x) πref(yc|x) −β log πθ(yr|x) πref(yr|x)  (105) This is the loss function for DPO, in the form shown in eq. 79. The DPO paper has an additional derivation for the objective under a Plackett-Luce Model, which is far less used in practice [2


onal derivation for the objective under a Plackett-Luce Model, which is far less used in practice [25]. 8.1.2.3 Deriving the BT DPO Gradient We used the DPO gradient shown in eq. 81 to explain intuitions for how the model learns. To derive this, we must take the gradient of eq. 105 with respect to the model parameters. ∇θLDPO(πθ; πref) = −∇θE(x,yc,yr)∼D  log σ  β log πθ(yc|x) πref(yc|x) −β log πθ(yr|x) πref(yr|x)  (106) To start, this can be rewritten. We know that the derivative of a sigmoid function d dxσ(x) = σ(x)(1 −σ(x)), the derivative of the logarithm d dx log x = 1 x, and properties of sigmoid σ(−x) = 1 −σ(x), so we can reformat the above equation. First, let u = β log πθ(yc|x) πref(yc|x) −β log πθ(yr|x) πref(yr|x) (the expression inside the sigmoid).


, let u = β log πθ(yc|x) πref(yc|x) −β log πθ(yr|x) πref(yr|x) (the expression inside the sigmoid). Then, we have ∇θLDPO(πθ; πref) = −E(x,yc,yr)∼D σ′(u) σ(u) ∇θu  (107) Expanding this and using the above expressions for sigmoid and logarithms results in the gradient introduced earlier: −E(x,yc,yr)∼D  βσ  β log πθ(yr|x) πref(yr|x) −β log πθ(yc|x) πref(yc|x)  [∇θ log π(yc|x) −∇θ log π(yr|x)]  (108) rlhfbook.com 109


8.2 Numerical Concerns, Weaknesses, and Alternatives Many variants of the DPO algorithm have been proposed to address weaknesses of DPO. For example, without rollouts where a reward model can rate generations, DPO treats every pair of preference data with equal weight. In reality, as seen in Chapter 11 on Preference Data, there are many ways of capturing preference data with a richer label than binary. Multiple algorithms have been proposed to re-balance the optimization away from treating each pair equally. • REgression to RElative REward Based RL (REBEL) adds signal from a reward model, as a margin between chosen and rejected responses, rather than solely the pairwise preference data, to more accurately solve the RLHF problem [190].


, rather than solely the pairwise preference data, to more accurately solve the RLHF problem [190]. • Conservative DPO (cDPO) and Identity Preference Optimization (IPO) address overfitting by assuming noise in the preference data. cDPO assumes N percent of the data is incorrectly labeled [25] and IPO changes the optimization to soften the probability of preference rather than optimize directly from a label [191]. Practically, IPO changes the preference probability to a nonlinear function, moving away from the Bradley-Terry assumption, with Ψ(q) = log  q 1−q  . • DPO with an offset (ODPO) “requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value” [192] – do not treat every data pair equally, but this can come at the cost


r than an offset value” [192] – do not treat every data pair equally, but this can come at the cost of a more difficult labeling environment. Some variants of DPO attempt to either improve the learning signal by making small changes to the loss or make the application more efficient by reducing memory usage. • Odds Ratio Policy Optimization (ORPO) directly updates the policy model with a pull towards the chosen response, similar to the instruction fine-tuning loss, with a small penalty on the chosen response [193]. This change of loss function removes the need for a reference model, simplifying the setup. The best way to view ORPO is as DPO inspired, rather than a DPO derivative. • Simple Preference Optimization (SimPO) makes a minor change to the DPO optimization, by averaging the log-pro


eference Optimization (SimPO) makes a minor change to the DPO optimization, by averaging the log-probabilities rather than summing them or adding length normalization, to improve performance [194]. One of the core issues apparent in DPO is that the optimization drives only to increase the margin between the probability of the chosen and rejected responses. Numerically, the model reduces the probability of both the chosen and rejected responses, but the rejected response is reduced by a greater extent as shown in fig. 27. Intuitively, it is not clear how this generalizes, but work has posited that it increases the probability of unaddressed behaviors – i.e. tokens that the language model could generate, but are not in the distribution of the post-training datasets [195] [196].


ge model could generate, but are not in the distribution of the post-training datasets [195] [196]. Simple methods—such as Cal-DPO [197], which adjusts the optimization process, and AlphaPO [198], which modifies the reward shape—mitigate this preference displacement. In practice, the exact impact of this is not well known, but points to a potential reason why online methods can outperform vanilla DPO. The other primary reason posited for DPO-like methods to have a lower ceiling on performance than online (RL based) RLHF methods is that the training signal comes from completions from previous or other models. Online variants of DPO alleviate these limitations by generating new completions and incorporating a preference signal at training time. Online rlhfbook.com 110


Figure 27: Sketch of preference displacement in DPO. DPO [199] samples generations from the current model, while Discriminator-Guided DPO (D2PO) [200] uses reward model relabelling to create new preference data on the fly, and many more variants exist. There is a long list of other DAA variants, such as Direct Nash Optimization (DNO) [201] or Binary Classifier Optimization (BCO) [202], but the choice of algorithm is far less important than the initial model and the data used [6] [203] [204]. 8.3 Implementation Details DAAs such as DPO are implemented very differently than policy gradient optimizers. The DPO loss, taken from the original implementation, largely can be summarized as follows [25]: # Log-probability gaps for the policy and the frozen reference model pi_logratios = policy_chose


5]: # Log-probability gaps for the policy and the frozen reference model pi_logratios = policy_chosen_logps - policy_rejected_logps ref_logratios = reference_chosen_logps - reference_rejected_logps # Difference of log-ratios: positive when the policy # shifts probability toward the chosen completion logits = pi_logratios - ref_logratios # DPO loss: negative log-sigmoid drives the policy to # widen the gap between chosen and rejected losses = -F.logsigmoid(beta * logits) # Implicit rewards (detached -- used for logging only) chosen_rewards = beta * (policy_chosen_logps - reference_chosen_logps).detach() rejected_rewards = beta * (policy_rejected_logps - reference_rejected_logps).detach() This can be used in standard language model training stacks as this information is already collated duri


can be used in standard language model training stacks as this information is already collated during the forward pass of a model (with the addition of a reference model). In most ways, DAAs are simpler and a quality of life improvement, but they also offer a rlhfbook.com 111


different set of considerations. 1. KL divergence is static: In DPO and other algorithms, the KL divergence is set explicitly by the β parameter that balances the distance penalty to the optimization. This is due to the fact that DPO takes gradient steps towards the optimal solution to the RLHF objective given the data – it steps exactly to the solution set by the β term. On the other hand, RL based optimizers take steps based on the batch and recent data. 2. Caching log-probabilities: Simple implementations of DPO do the forward passes for the policy model and reference models at the same time for convenience with respect to the loss function. However, this doubles the memory used and results in increased GPU usage.


ect to the loss function. However, this doubles the memory used and results in increased GPU usage. To avoid this, one can compute the log-probabilities of the reference model over the training dataset first, then reuse those cached reference log-probabilities when computing the loss and updating the parameters per batch, reducing the peak memory usage by 50%. 8.4 DAAs with Synthetic Preference Data Most of the popular datasets for performing preference fine-tuning with DAAs these days are synthetic preferences where a frontier model rates outputs from other models as the winner or the loser. Prominent examples include UltraFeedback (the first of this category) [28], Tülu 3 (built with an expanded UltraFeedback methodology) [6], SmolLM 3’s data [205], or the Dolci Pref dataset released wit


panded UltraFeedback methodology) [6], SmolLM 3’s data [205], or the Dolci Pref dataset released with Olmo 3 [18]. The best practices for constructing these datasets are still evolving. Tülu 3 and datasets around its release in November of 2024 demonstrated that synthetic, pairwise preference data needs to be “on-policy” in a sense that some completions are generated from the model you’re fine-tuning (while being mixed in a bigger model pool). This on-policy nature of the data ensured that the DAA would optimize the correct token space within which the model generates – as the loss functions are contrastive and less direct than instruction fine-tuning. Later, with the release of Olmo 3 and SmolLM 3 in 2025, other works supported a different theory called Delta Learning, which argues that t


molLM 3 in 2025, other works supported a different theory called Delta Learning, which argues that the difference between the chosen and rejected completions is more important to learning than exactly which models are used for the completions [206]. For example, in both of these two referenced models, the chosen responses are from Qwen 3 32B and the rejected responses are from Qwen 3 0.6B – both authors developed this pairing concurrently and independently. Overall, training models on synthetic preference data with DAAs is the place most practitioners should start, given the simplicity of implementation and strong performance relative to preference fine-tuning with reinforcement learning based methods.


nd strong performance relative to preference fine-tuning with reinforcement learning based methods. Other minor issues exist when using extensive, synthetic preference data, such as biases of the model judging between completions. Given that frontier models such as GPT-4 are known to have length bias [79] and a preference for outputs that match themselves [207] (see Chapter 12 for more information), it is slightly more likely for a piece of text in the “chosen” section of the dataset to be either from an OpenAI model or another strong model that is stylistically similar to it. To conclude this section, we’ll cover an intuition for how these methods change the generations of the model being trained.


, we’ll cover an intuition for how these methods change the generations of the model being trained. At a high level, most DAAs optimize to increase the margin between the probability of “chosen” and “rejected” completions (some less popular algorithms are designed to slightly change these dynamics, but the core remains). As discussed earlier rlhfbook.com 112


in this chapter (see fig. 27), this often means both probabilities decrease, but the rejected response decreases by a greater extent. Each token in a sequence receives a different gradient (magnitude and direction) based on how much it contributed to the overall preference margin, allowing the optimizer to identify which tokens matter most to the outcome. 8.5 DAAs vs. RL: Online vs. Offline Data Broadly, the argument boils down to one question: Do we need the inner workings of reinforcement learning, with value functions, policy gradients, and all, to align language models with RLHF? This, like most questions phrased this way, is overly simple. Of course, both methods are well-established, but it is important to illustrate where the fundamental differences and performance manifolds lie.


but it is important to illustrate where the fundamental differences and performance manifolds lie. Multiple reports have concluded that policy-gradient based and RL methods outperform DPO and its variants. The arguments take different forms, from training models with different algorithms but controlled data [125] [165] or studying the role of on-policy data within the RL optimization loop [208]. In all of these cases, DPO algorithms are a hair behind. Even with this performance delta, DAAs are still used extensively in leading models due to their simplicity. DAAs provide a controlled environment where iterations on training data and other configurations can be made rapidly, and given that data is often far more important than algorithms, using DPO can be fine.


de rapidly, and given that data is often far more important than algorithms, using DPO can be fine. With the emergence of reasoning models that are primarily trained with RL, further investment will return to using RL for preference-tuning, which in the long-term will improve the robustness of RL infrastructure and cement this margin between DAAs and RL for optimizing from human feedback. 8.6 Suggested Experiments The companion code in code/direct_alignment/ trains DPO and several related losses on preference data. This is the most accessible place to start experimenting with preference tuning because the setup is offline: no reward model server or rollout loop is required. 1. Train a small DPO run on UltraFeedback.


ine: no reward model server or rollout loop is required. 1. Train a small DPO run on UltraFeedback. cd code/ uv run python -m direct_alignment.train --loss dpo --max_samples 1000 Watch loss, accuracy, margins, chosen_rewards, and rejected_rewards. The main sanity check is that the implicit reward margin should move in the desired direction without the model’s sample generations collapsing. 2. Compare DPO, IPO, and length-normalized DPO. cd code/ uv run python -m direct_alignment.train --config direct_alignment/configs/dpo.yaml uv run python -m direct_alignment.train --config direct_alignment/configs/ipo.yaml uv run python -m direct_alignment.train --config direct_alignment/configs/dpo_norm.yaml rlhfbook.com 113


Compare the margin scale and the learning rate sensitivity. IPO’s loss is not on the same numeric scale as DPO, so read it through accuracy and margin behavior rather than raw loss alone. 3. Try the reference-free variants carefully. Run SimPO or ORPO from their configs, then inspect the generated samples that are logged during training. These losses are more sensitive to log-probability scaling and learning rate, which makes them useful debugging exercises. cd code/ uv run python -m direct_alignment.train --config direct_alignment/configs/simpo.yaml uv run python -m direct_alignment.train --config direct_alignment/configs/orpo.yaml 4. Change the data before changing the loss. Keep the loss fixed and vary --max_samples, --max_length, or the preference dataset.


ging the loss. Keep the loss fixed and vary --max_samples, --max_length, or the preference dataset. If the results move more than changing between DPO-like objectives, that is an empirical reminder of a central theme in preference tuning: data usually dominates small algorithmic differences. rlhfbook.com 114


9 Rejection Sampling Rejection Sampling (RS) is one of the most widely used yet least documented methods in preference fine-tuning. Many prominent RLHF papers use it as a core component of their training pipeline, yet no canonical implementation or explanation of why it works so well exists. RS can be applied at multiple points in the training pipeline – after instruction fine-tuning, after RL-based optimization, or even after RLVR – making it a versatile but hard-to-place tool. Combined with its underdocumented nature, this is why it appears here at the end of the core optimization methods. Rejection sampling operates by curating new candidate completions, filtering them based on a trained reward model, and then fine-tuning the original model only on the top completions (the same loss fun


reward model, and then fine-tuning the original model only on the top completions (the same loss function as instruction tuning). The name originates from computational statistics [209], where one wishes to sample from a complex distribution, but does not have a direct method to do so. To alleviate this, one samples from a distribution that is simpler to model and uses a heuristic to check if the sample is permissible. With language models, the target distribution is high-quality completions to prompts, the filter is a reward model, and the sampling distribution is the current model. WebGPT [4], Anthropic’s Helpful and Harmless agent [5], OpenAI’s popular paper on process reward models [50], Llama 2 Chat models [49], and other seminal works all use this baseline; more recent work has forma


Llama 2 Chat models [49], and other seminal works all use this baseline; more recent work has formalized it directly (e.g., RAFT [210] for applying it to alignment in multiple modalities and Statistical Rejection Sampling Optimization (RSO) [211] that gives a principled overview of how rejection sampling relates to other preference learning objectives). Throughout this chapter, we use x to denote prompts and y to denote completions. This notation is common in the language model literature, where methods operate on full promptcompletion pairs rather than individual tokens. 9.1 Training Process, Step by Step Rejection sampling overall follows a few stages. 0. Prompt and reward model selection: First, you must select the prompts you want to train on, relative to other stages of training.


ion: First, you must select the prompts you want to train on, relative to other stages of training. The simplest method is to re-use every prompt from the first SFT/IFT stage, but this can cause some overfitting. Before doing rejection sampling, you must also have trained a reward model (see Chapter 5 for more information). 1. Generate completions from the starting checkpoint: Next, one must generate completions to the selected prompts with the model they want to optimize. This can involve tweaking many settings, such as sampling temperature, top-p, max sequence length, number of completions per prompt, etc. 2. Select top completions with a reward model: All completions are ranked by a reward model.


, etc. 2. Select top completions with a reward model: All completions are ranked by a reward model. This stage may also include deduplication to keep only one completion per prompt, though many such design choices come down to empirical ablation studies. 3. SFT on top completions: To finish rejection sampling, one instruction fine-tunes the starting checkpoint on the selected completions. rlhfbook.com 115


A visual overview of the rejection sampling process is included below in fig. 28. Figure 28: Rejection sampling overview. The actual details on which prompts to use, how to select a reward model, how to sequence rejection sampling, etc. are not well documented in the literature. This chapter provides an overview of the methods and leaves further experimentation to the reader. 9.1.1 Generating Completions To generate a set of multiple candidate completions per prompt, let’s define a set of M prompts as a vector: X = [x1, x2, ..., xM] (109) These prompts can come from many sources, but most commonly they come from the instruction training set. For each prompt xi, we generate N completions. We can represent this as a matrix: Y =   y1,1 y1,2 · · · y1,N y2,1 y2,2 · · · y2,N ... ... ...


We can represent this as a matrix: Y =   y1,1 y1,2 · · · y1,N y2,1 y2,2 · · · y2,N ... ... ... ... yM,1 yM,2 · · · yM,N   (110) where yi,j represents the j-th completion for the i-th prompt. Each row i corresponds to a single prompt xi and contains its N candidate completions; each column j corresponds to the j-th sampled completion across all prompts. 9.1.2 Scoring Completions Now, we pass all of these prompt-completion pairs through a reward model, to get a matrix of rewards. We’ll represent the rewards as a matrix R: R =   r1,1 r1,2 · · · r1,N r2,1 r2,2 · · · r2,N ... ... ... ... rM,1 rM,2 · · · rM,N   (111) rlhfbook.com 116


Each reward ri,j is computed by passing the completion yi,j and its corresponding prompt xi through a reward model R: ri,j = R(yi,j | xi) (112) There are multiple methods to select the top completions to train on. To formalize the process of selecting the best completions based on our reward matrix, we can define a selection function S that operates on the reward matrix R. 9.1.2.1 Top Per Prompt The first potential selection function takes the max reward per prompt. S(R) = [arg max j r1,j, arg max j r2,j, ..., arg max j rM,j] (113) This function S returns a vector of indices, where each index corresponds to the column with the maximum reward for each row in R. We can then use these indices to select our chosen completions: Ychosen = [y1,S(R)1, y2,S(R)2, ..., yM,S(R)M ] (114) 9.1.2.2 Top Ov


o select our chosen completions: Ychosen = [y1,S(R)1, y2,S(R)2, ..., yM,S(R)M ] (114) 9.1.2.2 Top Overall Pairs Alternatively, we can select the top K prompt-completion pairs from the entire set. First, let’s flatten our reward matrix R into a single vector: Rflat = [r1,1, r1,2, ..., r1,N, r2,1, r2,2, ..., r2,N, ..., rM,1, rM,2, ..., rM,N] (115) This Rflat vector has length M × N, where M is the number of prompts and N is the number of completions per prompt. Now, we can define a selection function SK that selects the indices of the K highest values in Rflat: SK(Rflat) = argsort(Rflat)[−K :] (116) where argsort returns the indices that would sort the array in ascending order, and we take the last K indices to get the K highest values.


ould sort the array in ascending order, and we take the last K indices to get the K highest values. To get our selected completions, we need to map these flattened indices back to our original completion matrix Y . To recover the corresponding prompt-completion pair, you can map a zero-indexed flattened index k to (i, j) via i = ⌊k/N⌋+ 1 and j = (k mod N) + 1. rlhfbook.com 117


9.1.2.3 Selection Example Consider the case where we have the following situation, with five prompts and four completions. We will show two ways of selecting the completions based on reward. R =   0.7 0.3 0.5 0.2 0.4 0.8 0.6 0.5 0.9 0.3 0.4 0.7 0.2 0.5 0.8 0.6 0.5 0.4 0.3 0.6   (117) First, per prompt. Intuitively, we can highlight the reward matrix as follows: R =   0.7 0.3 0.5 0.2 0.4 0.8 0.6 0.5 0.9 0.3 0.4 0.7 0.2 0.5 0.8 0.6 0.5 0.4 0.3 0.6   (118) Using the argmax method, we select the best completion for each prompt: S(R) = [arg max j ri,j for i ∈[1, 5]] (119) S(R) = [1, 2, 1, 3, 4] (120) This means we would select: • For prompt 1: completion 1 (reward 0.7) • For prompt 2: completion 2 (reward 0.8) • For prompt 3: completion 1 (reward 0.9) • For prompt 4: com


or prompt 2: completion 2 (reward 0.8) • For prompt 3: completion 1 (reward 0.9) • For prompt 4: completion 3 (reward 0.8) • For prompt 5: completion 4 (reward 0.6) Now, best overall. Let’s highlight the top five overall completion pairs. R =   0.7 0.3 0.5 0.2 0.4 0.8 0.6 0.5 0.9 0.3 0.4 0.7 0.2 0.5 0.8 0.6 0.5 0.4 0.3 0.6   (121) First, we flatten the reward matrix: Rflat = [0.7, 0.3, 0.5, 0.2, 0.4, 0.8, 0.6, 0.5, 0.9, 0.3, 0.4, 0.7, 0.2, 0.5, 0.8, 0.6, 0.5, 0.4, 0.3, 0.6] (122) rlhfbook.com 118


Now, we select the indices of the five highest values: S5(Rflat) = [8, 5, 14, 0, 11] (123) Mapping these back to our original matrix: • Index 8 →prompt 3, completion 1 (reward 0.9) • Index 5 →prompt 2, completion 2 (reward 0.8) • Index 14 →prompt 4, completion 3 (reward 0.8) • Index 0 →prompt 1, completion 1 (reward 0.7) • Index 11 →prompt 3, completion 4 (reward 0.7) 9.1.2.4 Implementation Example Here is a code snippet showing how the selection methods could be implemented. import numpy as np x = np.random.randint(10, size=10) print(f"{x=}") sorted_indices = np.argsort(x) x_sorted = x[sorted_indices] print(f"{x_sorted=}") # first way to recover the original array i_rev = np.zeros(10, dtype=int) i_rev[sorted_indices] = np.arange(10) np.allclose(x, x_sorted[i_rev]) # second way to recover


=int) i_rev[sorted_indices] = np.arange(10) np.allclose(x, x_sorted[i_rev]) # second way to recover the original array np.allclose(x, x_sorted[np.argsort(sorted_indices)]) 9.1.3 Fine-Tuning With the selected completions, you then perform standard instruction fine-tuning on the current version of the model. More details can be found in the chapter on instruction tuning. 9.2 Implementation Details The core hyperparameters for performing this training are very intuitive: • Sampling parameters: Rejection sampling is directly dependent on the completions received from the model. Common settings for rejection sampling include temperatures above zero, e.g. between 0.7 and 1.0, with other modifications to parameters such as top-p or top-k sampling.


, e.g. between 0.7 and 1.0, with other modifications to parameters such as top-p or top-k sampling. • Completions per prompt: Successful implementations of rejection sampling have included 10 to 30 or more completions for each prompt. Using too few completions will make training biased and/or noisy. • Instruction tuning details: No clear training details for the instruction tuning during rejection sampling have been released. It is likely that they use slightly different settings than the initial instruction tuning phase of the model. • Heterogeneous model generations: Some implementations of rejection sampling include generations from multiple models rather than just the current model that is going to be trained. Best practices on how to do this are not established. rlhfbook.com 119


• Reward model training: The reward model used will heavily impact the final result. For more resources on reward model training, see the relevant chapter. When doing batch reward model inference, you can sort the tokenized completions by length so that the batches are of similar lengths. This eliminates the need to run inference on as many padding tokens and will improve throughput in exchange for minor implementation complexity. 9.3 Related: Best-of-N Sampling Best-of-N (BoN) is a close relative of rejection sampling, where the same generate-and-score procedure is followed, but you do not fine-tune the model on the selected completions. Instead, BoN computes the best possible completion to a static prompt (or set of prompts) at inference time, and related techniques are often used in “Pr


a static prompt (or set of prompts) at inference time, and related techniques are often used in “Pro” tiers of chat models that spend extra compute to get an answer to your query. Best-of-N sampling is often included as a baseline relative to RLHF training methods. It is important to remember that BoN does not modify the underlying model, but is a sampling technique. For this reason, comparisons of BoN sampling to online training methods, such as PPO, are still valid in some contexts. For example, you can still measure the KL distance when running BoN sampling relative to any other policy. Here, we will show that when using simple BoN sampling over one prompt, both selection criteria shown above are equivalent.


when using simple BoN sampling over one prompt, both selection criteria shown above are equivalent. Let R be a reward vector for our single prompt with N completions: R = [r1, r2, ..., rN] (124) where rj represents the reward for the j-th completion. Using the argmax method, we select the best completion for the prompt: S(R) = arg max j∈[1,N] rj (125) Using the top-K method with K = 1 reduces to the same method, which is common practice. 9.4 Suggested Experiments The companion implementation in code/rejection_sampling/ runs a complete GSM8K rejection-sampling pipeline: generate rollouts, score them with a reward model, select a training subset, fine-tune, and evaluate exact-match accuracy. The four configs are arranged as matched treatment/control pairs, so readers can ask whether the rewa


he four configs are arranged as matched treatment/control pairs, so readers can ask whether the reward model is actually helping. 1. Build the rollout cache once. cd code/ uv run python -m rejection_sampling.preprocess \ --config rejection_sampling/configs/top_per_prompt.yaml rlhfbook.com 120


This generates and scores completions for the shared GSM8K slice. Subsequent training configs reuse the cache as long as the generation and scoring settings stay unchanged. 2. Compare reward selection against random controls. cd code/ uv run python -m rejection_sampling.train \ --config rejection_sampling/configs/top_per_prompt.yaml uv run python -m rejection_sampling.train \ --config rejection_sampling/configs/random_per_prompt.yaml uv run python -m rejection_sampling.train \ --config rejection_sampling/configs/top_k_overall.yaml uv run python -m rejection_sampling.train \ --config rejection_sampling/configs/random_k_overall.yaml Read results in matched pairs: top_per_prompt versus random_per_prompt, and top_k_overall versus random_k_overall.


matched pairs: top_per_prompt versus random_per_prompt, and top_k_overall versus random_k_overall. If the reward-selected run does not beat its random baseline, the reward model or sampled completions are not giving useful signal on that slice. 3. Vary how much choice the reward model gets. Copy one config and change num_completions_per_prompt, temperature, top_p, and selection.top_k. More completions can improve the best available sample, but only if the reward model can separate good and bad answers. 4. Try a smaller policy model. Set model_name to a smaller compatible instruct model, reduce max_train_samples, and rerun the same matched pairs. This makes the experiment cheaper and highlights whether rejection sampling is rescuing weak generations or merely selecting among already-good o


hts whether rejection sampling is rescuing weak generations or merely selecting among already-good ones. rlhfbook.com 121


10 The Nature of Preferences Reinforcement learning from human feedback, also referred to as reinforcement learning from human preferences in early literature, emerged to optimize machine learning models in domains where specifically designing a reward function is hard. The word preferences, which was present in early literature, is at the center of the RLHF process – human preferences are what we are trying to model and what fuels the data for training. In order to understand the scope of the challenge in modeling and measuring human preferences, a broader context is needed in understanding what a preference is, how our understanding of preferences emerged, and how multiple fields of economics, psychology, philosophy, and more inform modern RLHF.


emerged, and how multiple fields of economics, psychology, philosophy, and more inform modern RLHF. 10.1 When Preference Replaces Correctness Consider an example: how do you decide which of these two poems is better? On February 26th, 2025, I asked both Claude 3.7 Sonnet and ChatGPT with GPT-4o the following: User: Write me a short poem about an optimistic goldfish. Example 1: Response: The Optimistic Goldfish A little goldfish, bright and bold, Swam in circles, dreams untold. Though the bowl was small and round, Endless wonders still abound. “The world is vast,” she’d always say, “I’ll find new joys in every day!” With every ripple, every gleam, She’d chase the sunlight, chase a dream. No walls could dim her hopeful view, For in her heart, the ocean grew.


the sunlight, chase a dream. No walls could dim her hopeful view, For in her heart, the ocean grew. And though she swam in just one place, She met each day with love and grace. Example 2: Response: The Optimistic Goldfish In a bowl of curved glass, Swims a goldfish, bright and bold. Three seconds of memory, they say, Yet each circuit feels brand new and gold. “Today might bring fresh flakes, Or perhaps a castle tall. The water seems especially clear— What adventure awaits beyond this wall?” Round and round with gleaming scales, Forever finding joy anew. Each moment fresh, each circle bright, The optimist in golden hue. Which is better? Which came from which model? How would you tell? Particularly, compare this to a question like “Who is the president of the United States?” There is a clear


larly, compare this to a question like “Who is the president of the United States?” There is a clear right and wrong answer. The motivation for using humans as the reward signals is to obtain an indirect metric for the target reward and align the downstream model to human preferences. In practice, the implementation is challenging and there is a substantial grey area in interpreting the best practices. The use of human-labeled feedback data integrates the history of many fields. Using human data alone is a well-studied problem, but in the context of RLHF, this data is used at the intersection of multiple long-standing fields of study [212]. rlhfbook.com 122


As an approximation, modern RLHF is the convergence of three areas of development: 1. Philosophy, psychology, economics, decision theory, and the nature of human preferences; 2. Optimal control, reinforcement learning, and maximizing utility; and 3. Modern deep learning systems. Each of these areas brings specific assumptions about what a preference is and how it can be optimized, which dictates the motivations and design of RLHF problems. In practice, RLHF methods are motivated and studied from the perspective of empirical alignment – maximizing model performance on specific skills instead of measuring the calibration to specific values. Still, the origins of value alignment for RLHF methods continue to be studied through research on methods to solve for “pluralistic alignment” across pop


s continue to be studied through research on methods to solve for “pluralistic alignment” across populations, such as position papers [213], [214], new datasets [215], and personalization methods [216]. The goal of this chapter is to illustrate how complex motivations result in presumptions about the nature of tools used in RLHF that often do not apply in practice. The specifics of obtaining data for RLHF are discussed further in Chapter 11 and using it for reward modeling in Chapter 5. 10.2 The Origins of RLHF and Preferences Breaking down the complex history inspiring the modern use of RLHF requires investigation into the intellectual foundations of quantifying human values, reinforcement learning and optimality, as well as behavioral economics as it relates to measuring preferences.


nt learning and optimality, as well as behavioral economics as it relates to measuring preferences. The notion of using reinforcement learning to optimize a reward model of preferences combines the history of various once-distanced fields into an intimate optimization built on variegated assumptions about human nature. A high-level timeline illustrating the history of this foundational content is shown in fig. 29. Our goal is to unspool the types of uncertainty that designers have grafted to system architectures at various stages of their intellectual history. Modern problem specifications have repeatedly stepped away from domains where optimal solutions are possible and deployed under-specified models as approximate solutions.


where optimal solutions are possible and deployed under-specified models as approximate solutions. To begin, all of the following operates on the assumption that human preferences exist in any form, which emerged in early philosophical discussions, such as Aristotle’s Topics, Book Three. 10.3 Specifying Objectives: From Logic of Utility to Reward Functions The optimization of RLHF explicitly relies only on reward models. In order to use rewards as an optimization target, RLHF presupposes the convergence of ideas from preferences, rewards, and costs. Models of preference, reward functions, and cost landscapes are all tools used by different fields to describe a notion of relative goodness of specific actions and/or states in the domain.


t fields to describe a notion of relative goodness of specific actions and/or states in the domain. The history of these three framings dates back to the origins of probability theory and decision theory. In 1662, The Port Royal Logic introduced the notion of decision-making quality [217]: rlhfbook.com 123


Figure 29: The timeline of the integration of various subfields into the modern version of RLHF. The direct links are continuous developments of specific technologies, and the arrows indicate motivations and conceptual links. To judge what one must do to obtain a good or avoid an evil, it is necessary to consider not only the good and evil in itself, but also the probability that it happens or does not happen. This theory has developed along with modern scientific thinking, starting with Bentham’s utilitarian Hedonic Calculus, arguing that everything in life could be weighed [218]. The first quantitative application of these ideas emerged in 1931 with Ramsey’s Truth and Probability [219]. Since these works, quantifying, measuring, and influencing human preferences has been a lively topic i