ides a few times, get some sleep, and stop overthinking it. Just present what you know and move on. A warmer, more supportive style: Response: That sounds really stressful, and it’s a normal reaction. If you can, do one quick practice run now, pick the one or two points you most want people to remember, and then give yourself permission to rest—being clear and calm tomorrow will matter more than cramming tonight. On the other hand, format is how the answer is structured. Modern models are known to use extensive markdown formatting with lists, line breaks, and emojis. Consider two answers to the following question that are very different: User: How can I improve my sleep quality? The simple answer, from an early language model: rlhfbook.com 9


Response: Try to go to bed at the same time each night, avoid caffeine late in the day, and keep your bedroom quiet and dark. Or a complex, richer answer: Response: Better sleep, in practice Goal: make your schedule predictable and your environment boring enough that your brain gives up. Tonight (quick wins): 1. Dim lights for the last hour before bed. 2. No caffeine after lunch; keep alcohol minimal. 3. If you’re stuck awake, get up and do something quiet until sleepy again. Daily basics (most impact): - Same wake time (even weekends) –> anchors your body clock. - Morning light for 5-10 minutes –> helps set circadian rhythm. - Cool, dark room (fan/eye mask) –> easier to stay asleep. If you want a simple rule: wake time fixed + caffeine cutoff + wind-down routine If sleep problems are pers


want a simple rule: wake time fixed + caffeine cutoff + wind-down routine If sleep problems are persistent or severe, it can be worth talking with a clinician—many issues are very treatable. Instruction fine-tuning would provide the basic ability for models to respond reliably in the question-answering format, and RLHF is what takes these answers and crafts them into the reliable, warm, and engaging answers we now expect from language models. Modern research has established RLHF as a general method to integrate subtle stylistic and related behavioral features into the models. An early, popular example of the utility of RLHF was in the application to safety [7] [5], where RLHF enabled models to be both helpful and harmless across varied datasets.


o safety [7] [5], where RLHF enabled models to be both helpful and harmless across varied datasets. Compared to other techniques for post-training, such as instruction fine-tuning, RLHF generalizes far better across domains [8] [9] – helping create effective general-purpose models. Intuitively, this can be seen in how the optimization techniques are applied. Instruction fine-tuning trains the model to predict the next token when the text preceding is close to examples it has seen. It is optimizing the model to more regularly output specific features in text. This is a per-token update. RLHF on the other hand tunes completions on the response level rather than looking at the next token specifically.


er hand tunes completions on the response level rather than looking at the next token specifically. Additionally, it is telling the model what a better response looks like, rather than a specific response it should learn. RLHF also shows a model which types of responses it should avoid, i.e. negative feedback. The training to achieve this is often called a contrastive loss function (one whose loss is computed from the comparison between two or more examples, rather than from each example independently) and is referenced throughout this book. While this flexibility is a major advantage of RLHF, it comes with implementation challenges. Largely, these center on how to control the optimization. As we will cover in this book, implementing RLHF often requires training a reward model, but best pr


As we will cover in this book, implementing RLHF often requires training a reward model, but best practices for doing so are not strongly established and depend on the area of application. With this, the optimization itself is prone to over-optimization because our reward signal is at best a proxy objective, requiring regularization. With these limitations, effective RLHF requires a rlhfbook.com 10


strong starting point, so RLHF cannot be a solution to every problem alone and needs to be approached through a broader lens of post-training. Due to this complexity, implementing RLHF is far more costly than simple instruction fine-tuning and can come with unexpected challenges such as length bias [10] [11]. For model training efforts where absolute performance matters, RLHF is established as being crucial to achieving a strong fine-tuned model, but it is more expensive in compute, data costs, and time. Through the early history of RLHF after ChatGPT, there were many research papers that showed approximate solutions to RLHF via limited instruction fine-tuning, but as the literature matured it has been repeated time and again that RLHF and related methods are core stages of model performan


it has been repeated time and again that RLHF and related methods are core stages of model performance that cannot be easily dispensed with. 1.3 Walkthrough of an RLHF Recipe To set the stage for the book, it’s important to understand what “doing RLHF” can look like, as a minimal example, without any of the technical jargon that can be hard to grasp before solidifying fundamental intuitions. This section follows what is described as the canonical, three-stage RLHF recipe, as established with OpenAI’s InstructGPT model in 2022 [3]. The first step of the process is to transition the model from a base model that completes text to an instruction-following model that can operate in a question-answering format.


t completes text to an instruction-following model that can operate in a question-answering format. This is done by using the same next-token prediction loss function on a set of carefully crafted datapoints where the model is shown only data in this question-answering format. After the model is shown these high-quality responses, the model can now be prompted with a specific sequence of tokens to know that it should answer any query with a more defined, assistant persona. With this foundation of the shape of how the model should answer, the next two steps work together to improve the overall quality of the answers. These two steps serve to set up a problem where we can use reinforcement learning to update the model and make it more helpful.


up a problem where we can use reinforcement learning to update the model and make it more helpful. The first of these two steps is to train a reward model that captures human preferences. In order to apply reinforcement learning to a problem, you need a reward function that indicates quality. The goal of a reward model is to create a scalar signal that can then later be optimized with RL. In practice, this involves fine-tuning a language model (it is usually the same instruction-tuned model from the previous step) on a dataset of preference relations between pieces of text. This dataset is collected across a variety of prompts, model completions, and labelers to try and capture a robust signal of what is a better answer from a language model.


, and labelers to try and capture a robust signal of what is a better answer from a language model. The reward model learns which features in the text are better than others, so when it is used at inference-time (and during RL as the reward signal) it scores any piece of input text on how good it is. With these two pieces, a question-answering model and a reward model, we have everything we need to put together the pieces and actually do reinforcement learning from human feedback (RLHF). The actual RLHF stage proceeds by taking prompts representative of tasks the model should be good at, generating a bunch of completions, having the reward model rank them, and then using RL to figure out how to change the model and make it better.


reward model rank them, and then using RL to figure out how to change the model and make it better. The basic primitive is that reinforcement learning is given a signal of which actions are good, in the form of tokens that a language model generates, and derives update rules that attribute different actions to different parameters in the model. The final RLHF stage rlhfbook.com 11


shifts parameters to make good tokens more likely, and does so iteratively to maintain the general capabilities of the initial model. Once RL is complete, and performance has saturated, this is often the final model served to the user. Throughout this book, we’ll cover many recipes for how to do RLHF, and more related optimization methods that make up the broader suite of post-training. These all emerge to solve more challenging problems facing language models, and to make the strengths of the original RLHF approaches more powerful. 1.4 An Intuition for Post-Training We’ve established that RLHF specifically and post-training generally are crucial to the performance of the latest models and how they change the models’ outputs, but not why RLHF works.


e performance of the latest models and how they change the models’ outputs, but not why RLHF works. Here’s a simple analogy for how so many gains can be made on benchmarks on top of any base model. The way I’ve been describing the potential of post-training is called the elicitation interpretation of post-training, where all we are doing is extracting potential by amplifying valuable behaviors in the base model. To make this example click, we make the analogy between the base model – the language model that comes out of the large-scale, next-token prediction pretraining – and other foundational components in building complex systems. We use the example of the chassis of a car, which defines the space around which a car can be built.


e use the example of the chassis of a car, which defines the space around which a car can be built. Consider Formula 1 (F1): most teams begin each year with a new chassis and engine. Then, they spend all year on aerodynamics and systems changes (of course, it is a minor oversimplification), and can dramatically improve the performance of the car. The best F1 teams improve far more during a season than chassis-to-chassis. The same is true for post-training, where one can extract a ton of performance out of a static base model as they learn more about its quirks and tendencies. The best post-training teams extract a ton of performance in a very short time frame. The set of techniques includes everything close to and after the end of pretraining: “mid-training” like annealing / high-quality e


everything close to and after the end of pretraining: “mid-training” like annealing / high-quality end of pretraining web data, instruction tuning, RLVR, preference-tuning, etc. A good example is the change from the first version of the Allen Institute for AI’s fully-open, small Mixture-of-Experts (MoE) model OLMoE Instruct to the second. The first model was released in the fall of 2024 [12], and with the second version only updating the post-training, the evaluation average on popular benchmarks went from 35 to 48 without changing the majority of pretraining [13]. The idea is that there is a lot of intelligence and ability within base models, but because they can only answer in next-token prediction and not question-answering format, it takes a lot of work building around them, through po


rediction and not question-answering format, it takes a lot of work building around them, through post-training, in order to make excellent final models. Then, when you look at models such as OpenAI’s GPT-4.5 released in February 2025, which was largely a failure of a consumer product due to being too large of a base model to serve to millions of users, you can see this as a far more dynamic and exciting base for OpenAI to build onto. With this intuition, base models determine the vast majority of the potential of a final model, and post-training’s job is to cultivate all of it. rlhfbook.com 12


I’ve described this intuition as the Elicitation Theory of Post-training. This theory folds in with the reality that the majority of gains users are seeing are from post-training because it implies that there is more latent potential in a model pretrained on the internet than we can simply teach the model — such as by passing certain narrow samples in repeatedly during early types of post-training (i.e. only instruction tuning). The challenge of post-training is to reshape models from next-token prediction to conversation question-answering, while extracting all of this knowledge and intelligence from pretraining. A related idea to this theory is the Superficial Alignment Hypothesis, coined in the paper LIMA: Less is More for Alignment [14].


is the Superficial Alignment Hypothesis, coined in the paper LIMA: Less is More for Alignment [14]. This paper is getting some important intuitions right but for the wrong reasons in the big picture. The authors state: A model’s knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users. If this hypothesis is correct, and alignment is largely about learning style, then a corollary of the Superficial Alignment Hypothesis is that one could sufficiently tune a pretrained language model with a rather small set of examples [Kirstain et al., 2021]. All of the successes of deep learning should have taught you that scaling data is important to performance.


he successes of deep learning should have taught you that scaling data is important to performance. Here, the major difference is that the authors are discussing alignment and style, the focus of academic post-training at the time. With a few thousand samples for instruction fine-tuning, you can change a model substantially and improve a narrow set of evaluations, such as AlpacaEval, MT Bench, Arena (formerly ChatBotArena, a platform where users compare anonymous model responses head-to-head), and the like. These do not always translate to more challenging capabilities, which is why Meta wouldn’t train its Llama Chat models on just this dataset. Academic results have lessons, but need to be interpreted carefully if you are trying to understand the big picture of the technological arc.


be interpreted carefully if you are trying to understand the big picture of the technological arc. What this paper is showing is that you can change models substantially with a few samples. We knew this, and it is important to the short-term adaptation of new models, but their argument for performance leaves the casual readers with the wrong lessons. If we change the data, the impact could be far higher on the model’s performance and behavior, but it is far from “superficial.” Base language models today (with no post-training) can be trained on some mathematics problems with reinforcement learning, learn to output full chain-of-thought reasoning, and then score higher on a full suite of reasoning evaluations like BigBenchHard, Zebra Logic, AIME, etc.


en score higher on a full suite of reasoning evaluations like BigBenchHard, Zebra Logic, AIME, etc. The superficial alignment hypothesis is wrong for the same reason that people who think RLHF and post-training are just for vibes are still wrong. This was a field-wide lesson we had to overcome in 2023 (although many AI observers are still rooted in this belief). Post-training has far outgrown that, and we are coming to see that the style of models operates on top of behavior — such as the now popular long chain of thought. As the AI community shifts post-training further into the era of agentic and reasoning models, the superficial alignment hypothesis breaks down further. RL methods are becoming an increasingly large share of the compute needed to train frontier language models.


s are becoming an increasingly large share of the compute needed to train frontier language models. In the short time since reinforcement learning with verifiable rewards (RLVR) was coined in our work on Tülu 3 in the fall of 2024 [6], the scale of compute used for post-training has grown dramatically. DeepSeek R1, famous for popularizing RLVR, used only about 5% of their rlhfbook.com 13


overall compute in post-training – 147K H800 GPU hours for RL training on R1 [15], relative to 2.8M GPU hours for pretraining the underlying DeepSeek V3 base model [16]. The science studying the core methods of scaling RL as of 2026 shows that individual ablation runs can take 10-100K GPU hours [17], the equivalent of the compute used for the RL stage of OLMo 3.1 Think 32B (released in November of 2025), which trained for 4 weeks on 200 GPUs [18]. The science of scaled post-training is in its very early stages as of 2026, adopting ideas and methods from pretraining language models and applying them in this new domain, so the exact GPU hours used will change, but the trend of increased compute on post-training will continue.


xact GPU hours used will change, but the trend of increased compute on post-training will continue. Altogether, the elicitation theory of post-training is likely to become the correct view only when applying a lighter post-training recipe – something useful for specializing a model – relative to the compute-intensive frontier models. 1.5 How We Got Here Why does this book make sense now? How much will change in the future? Post-training, the craft of eliciting powerful behaviors from a raw pretrained language model, has gone through many seasons and moods since the release of ChatGPT that sparked the renewed interest in RLHF. In the era of Alpaca [19], Vicuna [20], Koala [21], and Dolly [22], a limited number of human datapoints with extended synthetic data in the style of Self-Instruct we


, a limited number of human datapoints with extended synthetic data in the style of Self-Instruct were used to fine-tune the original LLaMA to get similar behavior to ChatGPT. The benchmark for these early models was fully vibes (and human evaluation) as we were all so captivated by the fact that these small models can have such impressive behaviors across domains. It was justified excitement. Open post-training was moving faster, releasing more models, and making more noise than its closed counterparts. Companies were scrambling, e.g. DeepMind merging with Google or being started, and taking time to follow it up. There are phases of open recipes surging and then lagging behind. The era following Alpaca et al., the first lag in open recipes, was one defined by skepticism and doubt about re


owing Alpaca et al., the first lag in open recipes, was one defined by skepticism and doubt about reinforcement learning from human feedback (RLHF), the technique OpenAI highlighted as crucial to the success of the first ChatGPT. Many companies doubted that they needed to do RLHF. A common phrase – “instruction tuning is enough for alignment” – was so popular then that it still carries weight today despite obvious evidence against it. This doubt about RLHF lasted, especially in the open where groups cannot afford data budgets on the order of $100K to $1M. The companies that embraced it early ended up winning out. Anthropic published extensive research on RLHF through 2022 and now has arguably the best post-training [23] [5] [24].


extensive research on RLHF through 2022 and now has arguably the best post-training [23] [5] [24]. The delta between open groups, struggling to reproduce or even knowing basic closed techniques, and leading closed models is a common theme. The first shift in open alignment methods and post-training was the story of Direct Preference Optimization (DPO) [25], which showed that you can solve the same optimization problem as RLHF with fewer moving parts by taking gradient steps directly on pairwise preference data. The DPO paper, posted in May of 2023, didn’t have any clearly impactful models trained with it through the fall of 2023. This changed with the releases of a few breakthrough DPO models – all contingent on finding a better, lower, learning rate.


leases of a few breakthrough DPO models – all contingent on finding a better, lower, learning rate. Zephyr-Beta [26], Tülu 2 rlhfbook.com 14


[27], and many other models showed that the DPO era of post-training had begun. Chris Manning literally thanked me for “saving DPO.” Preference-tuning was something you needed to do to meet the table stakes of releasing a good model since late 2023. The DPO era continued through 2024, in the form of never-ending variants on the algorithm, but we were very far into another slump in open recipes. Open post-training recipes had saturated the extent of knowledge and resources available. A year after Zephyr and Tülu 2, the same breakout dataset, UltraFeedback is arguably still state-of-the-art for preference tuning in open recipes [28]. At the same time, the Llama 3.1 [29] and Nemotron 4 340B [30] reports gave us substantive hints that large-scale post-training is much more complex and impactfu


] reports gave us substantive hints that large-scale post-training is much more complex and impactful. The closed labs are doing full post-training – a large multi-stage process of instruction tuning, RLHF, prompt design, etc. – where academic papers are just scratching the surface. Tülu 3 represented a comprehensive, open effort to build the foundation of future academic post-training research [6]. Post-training is a complex process involving the aforementioned training objectives applied in various orders to target specific capabilities. This book is designed to provide a platform for understanding all of these techniques, and as the field matures the best practices for how to interleave them will emerge.


ese techniques, and as the field matures the best practices for how to interleave them will emerge. The primary areas of innovation in post-training are now in reinforcement learning with verifiable rewards (RLVR), reasoning training generally, and related ideas. These newer methods build extensively on the infrastructure and ideas of RLHF, but are evolving far faster. This book is written to capture the first stable literature for RLHF after its initial period of rapid change. 1.6 Scope of This Book This book hopes to touch on each of the core steps of doing canonical RLHF implementations. It will not cover all the history of the components nor recent research methods, just techniques, problems, and trade-offs that have been proven to occur again and again.


methods, just techniques, problems, and trade-offs that have been proven to occur again and again. 1.6.1 Chapter Summaries This book has the following chapters: 1.6.1.1 Introductions Reference material and context useful throughout the book. 1. Introduction: Overview of RLHF and what this book provides. 2. A Tiny History of RLHF: Key models and papers in the history of RLHF techniques. 3. Training Overview: How the training objective for RLHF is designed and basics of understanding it. 1.6.1.2 Core Training Pipeline The suite of techniques used to optimize language models to align them to human preferences. 4. Instruction Fine-Tuning: Adapting language models to the question-answer format. 5.


preferences. 4. Instruction Fine-Tuning: Adapting language models to the question-answer format. 5. Reward Modeling: Training reward models from preference data that act as an optimization target for RL training (or for use in data filtering). rlhfbook.com 15


6. Reinforcement Learning: The core RL techniques used to optimize reward models (and other signals) throughout RLHF. 7. Reasoning and Inference-Time Scaling: The role of new RL training methods for inference-time scaling with respect to post-training and RLHF. 8. Direct-Alignment Algorithms: Algorithms that optimize the RLHF objective directly from pairwise preference data rather than learning a reward model first. 9. Rejection Sampling: A basic technique for using a reward model with instruction tuning to align models. 1.6.1.3 Data & Preferences Context for the data that fuels RLHF and the big picture problem it is trying to solve. 10. The Nature of Preferences: Why human preference data is needed to fuel and understand RLHF. 11.


10. The Nature of Preferences: Why human preference data is needed to fuel and understand RLHF. 11. Preference Data: How preference data is collected for RLHF. 12. Synthetic Data: The shift away from human to synthetic data, how AI feedback works, and how distilling from other models is used. 13. Tool Use and Function Calling: The basics of training models to call functions or tools in their outputs. 1.6.1.4 Practical Considerations Fundamental problems and discussions for implementing and evaluating RLHF. 14. Over-Optimization: Qualitative observations of why RLHF goes wrong and why over-optimization is inevitable with a soft optimization target in reward models. 15. Regularization: Tools to constrain these optimization tools to effective regions of the parameter space. 16.


ation: Tools to constrain these optimization tools to effective regions of the parameter space. 16. Evaluation: The ever evolving role of evaluation (and prompting) in language models. 17. Crafting Model Character and Products: How RLHF is shifting in its applicability as major AI laboratories use it to subtly match their models to their products. 1.6.1.5 Appendices Reference material for definitions and extended discussions. • Appendix A - Definitions: Mathematical definitions for RL, language modeling, and other ML techniques leveraged in this book. • Appendix B - Beyond “Just Style”: How RLHF is often underestimated in its role in improving the user experience of models due to the crucial role that style plays in information sharing.


ving the user experience of models due to the crucial role that style plays in information sharing. 1.6.2 Target Audience This book is intended for audiences with entry level experience with language modeling, reinforcement learning, and general machine learning. It will not have exhaustive documentation for all the techniques, but just those crucial to understanding RLHF. 1.6.3 How to Use This Book This book was largely created because there were no canonical references for important topics in the RLHF workflow. Given the pace of progress on LLMs overall, combined with the complex nature of collecting and using human data, RLHF is an unusually academic field rlhfbook.com 16


where published results are often noisy and hard to reproduce across multiple settings. To develop strong intuitions, readers are encouraged to read multiple papers on each topic rather than taking any single result as definitive. To facilitate this, the book includes numerous, academic-style citations to the canonical reference for a claim. The contributions of this book are supposed to give you the minimum knowledge needed to try a toy implementation or dive into the literature. This is not a comprehensive textbook, but rather a quick book for reminders and getting started. This book is finalizing as of April 2026, when it’s moving to production for print. As a web-first book, this content will continue to evolve, so if you spot a typo or an important omission, please contribute a fix or


will continue to evolve, so if you spot a typo or an important omission, please contribute a fix or suggestion on GitHub. 1.6.4 About the Author Dr. Nathan Lambert is a researcher and writer focusing on building the open science of language models. He came here through a Ph.D. in robotics and building an RLHF team shortly after the release of ChatGPT. He has released many models trained with RLHF, their subsequent datasets, and training codebases in his time at the Allen Institute for AI (Ai2) and HuggingFace. Examples include Zephyr-Beta, Tülu 2, OLMo, TRL, Open Instruct, and many more. He has written extensively on RLHF, including many blog posts and academic papers. 1.7 Future of RLHF With the investment in language modeling, many variations on the traditional RLHF methods emerged.


With the investment in language modeling, many variations on the traditional RLHF methods emerged. RLHF colloquially has become synonymous with multiple overlapping approaches. RLHF is a subset of preference fine-tuning (PreFT) techniques, including Direct Alignment Algorithms (See Chapter 8), which are the class of methods downstream of DPO that solve the preference learning problem by taking gradient steps directly on preference data, rather than learning an intermediate reward model. RLHF is the tool most associated with rapid progress in “post-training” of language models, which encompasses all training after the large-scale autoregressive training on primarily web data. This textbook is a broad overview of RLHF and its directly neighboring methods, such as instruction tuning and othe


s a broad overview of RLHF and its directly neighboring methods, such as instruction tuning and other implementation details needed to set up a model for RLHF training. As more successes of fine-tuning language models with RL emerge, such as OpenAI’s o1 reasoning models, RLHF will be seen as the bridge that enabled further investment of RL methods for fine-tuning large base models. At the same time, while the spotlight of focus may be more intense on the RL portion of RLHF in the near future – as a way to maximize performance on valuable tasks – the core of RLHF is that it is a lens for studying the grand problems facing modern forms of AI. How do we map the complexities of human values and objectives into systems we use on a regular basis?


w do we map the complexities of human values and objectives into systems we use on a regular basis? This book hopes to be the foundation of decades of research and lessons on these problems. rlhfbook.com 17


2 A Tiny History of RLHF RLHF and its related methods are very new. We highlight history to show how recently the procedures were formalized, and how much of this documentation is in the academic literature. With this, we want to emphasize that RLHF is very rapidly evolving, so the chapter sets the stage for a book that will express uncertainty over certain methods and an expectation that some details can change around a few, core practices. Otherwise, the papers and methods listed here showcase why many pieces of the RLHF pipeline are what they are, as some of the seminal papers were for applications totally distinct from modern language models. In this chapter we detail the key papers and projects that got the RLHF field to where it is today.


In this chapter we detail the key papers and projects that got the RLHF field to where it is today. This is not intended to be a comprehensive review of RLHF and the related fields, but rather a starting point and retelling of how we got to today. It is intentionally focused on recent work that led to ChatGPT. There is substantial further work in the RL literature on learning from preferences [31]. For a more exhaustive list, you should use a proper survey paper [32], [33]. Figure 2: Timeline of key developments in RLHF discussed in this chapter, from early work on RL from preferences through the adoption of RLHF in large language models. 2.1 Origins to 2018: RL on Preferences The field has recently been popularized with the growth of Deep Reinforcement Learning and has grown into a broade


recently been popularized with the growth of Deep Reinforcement Learning and has grown into a broader study of the applications of LLMs from many large technology companies. Still, many of the techniques used today are deeply related to core techniques from early literature on RL from preferences. One of the first papers with an approach similar to modern RLHF was TAMER. TAMER: Training an Agent Manually via Evaluative Reinforcement proposed an approach in which humans iteratively scored an agent’s actions to learn a reward model, which was used to learn the action policy [34]. Other work, concurrently or soon after, proposed an actor-critic algorithm, COACH, where human feedback (both positive and negative) is used to tune the advantage function [35].


ACH, where human feedback (both positive and negative) is used to tune the advantage function [35]. The primary reference, Christiano et al. 2017, is an application of RLHF applied to preferences between trajectories of agents within Atari games [1]. This work introducing RLHF followed soon after DeepMind’s seminal work in reinforcement learning on Deep Q-Networks (DQN), which showed that RL agents can solve popular video games learning from scratch. The work shows that humans choosing between trajectories can be more effective in some domains rlhfbook.com 18


than directly interacting with the environment. This uses some clever conditions, but is impressive nonetheless. Figure 3: The core RLHF loop from Christiano et al. (2017): the reward predictor is trained asynchronously from comparisons of trajectory segments, and the agent maximizes predicted reward. This method was expanded upon with more direct reward modeling [36] and the adoption of deep learning within early RLHF work was capped by an extension to TAMER with neural network models just one year later [37]. This era began to transition, as reward models as a general notion were proposed as a method for studying alignment, rather than just a tool for solving RL problems [38]. 2.2 2019 to 2022: RL from Human Preferences on Language Models Reinforcement learning from human feedback, also


2022: RL from Human Preferences on Language Models Reinforcement learning from human feedback, also referred to regularly as reinforcement learning from human preferences in its early days, was quickly adopted by AI labs increasingly turning to scaling large language models. A large portion of this work began between GPT-2, in 2019, and GPT-3, in 2020. The earliest work in 2019, Fine-Tuning Language Models from Human Preferences has many striking similarities to modern work on RLHF and the content that we will cover in this book [39]. Many canonical terms, such as learning reward models, KL distances, feedback diagrams, etc., were formalized in this paper, though the evaluation tasks for the final models and their capabilities were different from what people are doing today.


tasks for the final models and their capabilities were different from what people are doing today. From here, RLHF was applied to a variety of tasks. Important examples include general summarization [2], recursive summarization of books [40], instruction following (InstructGPT) [3], browser-assisted question-answering (WebGPT) [4], supporting answers with citations (GopherCite) [41], and general dialogue (Sparrow) [42]. Aside from applications, a number of seminal papers defined key areas for the future of RLHF, including those on: 1. Reward model over-optimization [43]: The ability for RL optimizers to over-fit to models trained on preference data, 2. Language models as a general area of study for alignment [23], and 3.


rained on preference data, 2. Language models as a general area of study for alignment [23], and 3. Red teaming [44] – the process of assessing the safety of a language model. Work continued on refining RLHF for application to chat models. Anthropic continued to use it extensively for early versions of Claude [5] and early RLHF open-source tools emerged [45], [46], [47]. rlhfbook.com 19


2.3 2023 to the Present: The ChatGPT Era The announcement of ChatGPT was very clear about the role of RLHF in its training [48]: We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. Since then, RLHF has been used extensively in leading language models. It is well known to be used in Anthropic’s Constitutional AI for Claude [24], Meta’s Llama 2 [49] and Llama 3 [29], Nvidia’s Nemotron [30], Ai2’s Tülu 3 [6], and more. Today, RLHF is growing into a broader field of preference fine-tuning (PreFT), including new applications such as process rewards for intermediate reasoning steps [50], covered in Chapter 5; direct alignment algorithms inspired by Direct Preference Optimiz


steps [50], covered in Chapter 5; direct alignment algorithms inspired by Direct Preference Optimization (DPO) [25], covered in Chapter 8; learning from execution feedback from code or math [51], [52] and other online reasoning methods inspired by OpenAI’s o1 [53], covered in Chapter 7. rlhfbook.com 20


3 Training Overview In this chapter we provide a cursory overview of RLHF training, before getting into the specifics later in the book. RLHF, while optimizing a simple loss function, involves training multiple, different AI models in sequence and then linking them together in a complex, online optimization. Here, we introduce the core objective of RLHF, which is optimizing a proxy reward for human preferences with a distance-based regularizer (along with showing how it relates to classical RL problems). Then we showcase canonical recipes which use RLHF to create leading models to show how RLHF fits in with the rest of post-training methods. These example recipes will serve as references for later in the book, where we describe different optimization choices you have when doing RLHF, and w


later in the book, where we describe different optimization choices you have when doing RLHF, and we will point back to how different key models used different steps in training. 3.1 Problem Formulation The optimization of reinforcement learning from human feedback (RLHF) builds on top of the standard RL setup. In RL, an agent takes actions at sampled from a policy π(at | st) given the state of the environment st to maximize reward r(st, at) [54]. A policy is a function that maps each state to a probability distribution over actions. The early policies that evolved into modern literature on RLHF were in what is called deep reinforcement learning – when a neural network is used to learn said function.


what is called deep reinforcement learning – when a neural network is used to learn said function. Traditionally, the environment evolves according to transition (dynamics) p(st+1 | st, at) with an initial state distribution ρ0(s0). Together, the policy and dynamics induce a trajectory distribution. A trajectory’s overall probability is the product of the initial state probability, every action choice the policy makes, and every state transition the environment produces: pπ(τ) = ρ0(s0) T −1 Y t=0 π(at | st) p(st+1 | st, at). (1) Across a finite episode with horizon T, the goal of an RL agent is to solve the following optimization, where γ is a discount factor from 0 to 1 that balances the desirability of near-term versus future rewards: max π Eτ∼pπ "T −1 X t=0 γtr(st, at) # .


lances the desirability of near-term versus future rewards: max π Eτ∼pπ "T −1 X t=0 γtr(st, at) # . (2) The expected return for a given policy is often denoted J(π), with the optimal value written J∗= maxπ J(π). For continuing tasks, one often takes T →∞and relies on discounting (γ < 1) to keep the objective well-defined. Multiple methods for optimizing this expression are discussed in Chapter 6. A standard illustration of the RL loop is shown in fig. 4 (compare this to the RLHF loop in fig. 7). rlhfbook.com 21


Figure 4: Standard RL loop 3.1.1 A Simple Example: The Thermostat To build a basic intuition for what RL does, consider a thermostat trying to keep a room at a target temperature of 70◦F. In RL, the agent starts with no knowledge of the task and must discover a good policy through trial and error. The thermostat example has the following components (see fig. 5 for how each maps to the trajectory distribution in eq. 1): • State (st): the current room temperature, e.g. 65◦F. • Action (at): turn the heater on or off. • Reward (r): +1 when the temperature is within 2◦of the target, 0 otherwise. • Policy (π): the rule that decides whether to turn the heater on or off given the current temperature.


licy (π): the rule that decides whether to turn the heater on or off given the current temperature. Here is one policy the thermostat might learn, which may not be optimal depending on the exact transition dynamics of the environment: π(at = on | st) = ( 1 if st < 70◦F 0 otherwise (3) • Transition: the room warms when the heater is on and cools when it is off. The agent influences these dynamics through its actions, but the underlying physics – how fast the room heats or cools – are outside its control. Figure 5: Each term in the trajectory distribution (eq. 1) mapped to the thermostat RL example. Initially, the thermostat’s policy is essentially random – it flips the heater on and off with no regard for the current temperature, and the room’s temperature swings wildly.


er on and off with no regard for the current temperature, and the room’s temperature swings wildly. Over many rlhfbook.com 22


episodes of trial and error, the agent discovers that turning the heater on when the room is cold and off when it is warm leads to more reward, and gradually converges on a sensible policy. This is the core RL loop: observe a state, choose an action, receive a reward, and update the policy to get more reward over time. 3.1.2 Classic RL Example: CartPole For a richer example with continuous dynamics, consider the classic CartPole (inverted pendulum) control task, which appears in many RL textbooks, courses, and even research papers. Whereas the thermostat had a single state variable and a binary action, CartPole involves four continuous state variables and physics-based transitions – making it a standard benchmark for RL algorithms.


s state variables and physics-based transitions – making it a standard benchmark for RL algorithms. Figure 6: CartPole environment showing state variables (x, ˙x, θ, ˙θ) and actions (±F). • State (st): the cart position/velocity and pole angle/angular velocity: st = (xt, ˙xt, θt, ˙θt). (4) • Action (at): apply a left/right horizontal force to the cart, e.g. at ∈{−F, +F}. • Reward (r): a simple reward is rt = 1 each step the pole remains balanced and the cart stays on the track (e.g. |xt| ≤2.4 and |θt| ≤12◦), and the episode terminates when either bound is violated. • Dynamics / transition (p(st+1 | st, at)): in many environments the dynamics are deterministic (so p is a point mass) and can be written as st+1 = f(st, at) via Euler rlhfbook.com 23


integration with step size ∆t. A standard simplified CartPole update uses the constants cart mass mc, pole mass mp, pole half-length l, and gravity g (α is a mass-normalized intermediate with acceleration units): α = at + mpl ˙θ2 t sin θt mc + mp (5) ¨θt = g sin θt −cos θt α l  4 3 −mp cos2 θt mc+mp  (6) ¨xt = α −mpl ¨θt cos θt mc + mp (7) xt+1 = xt + ∆t ˙xt, ˙xt+1 = ˙xt + ∆t ¨xt, (8) θt+1 = θt + ∆t ˙θt, ˙θt+1 = ˙θt + ∆t ¨θt. (9) This is a concrete instance of the general setup above: the policy chooses at, the transition function advances the state, and the reward is accumulated over the episode. 3.1.3 Manipulating the Standard RL Setup The RL formulation for RLHF is seen as a less open-ended problem, where a few key pieces of RL are set to specific definitions in order to accommodate l


nded problem, where a few key pieces of RL are set to specific definitions in order to accommodate language models. There are multiple core changes from the standard RL setup to that of RLHF: Table tbl. 1 summarizes these differences between standard RL and the RLHF setup used for language models. 1. Switching from a reward function to a reward model. In RLHF, a learned model of human preferences, rθ(st, at) (or any other classification model) is used instead of an environmental reward function. This gives the designer a substantial increase in the flexibility of the approach and control over the final results, but at the cost of implementation complexity. In standard RL, the reward is seen as a static piece of the environment that cannot be changed or manipulated by the person designing t


as a static piece of the environment that cannot be changed or manipulated by the person designing the learning agent. 2. No state transitions exist. In RLHF, the initial states for the domain are prompts sampled from a training dataset and the “action” is the completion to said prompt (in the standard RLHF setup, the prompt is fixed and the model’s completion does not define the next prompt). The combination of one prompt and one completion constitutes a complete episode or rollout, which would be many repeated state-action, state-action chains in classical RL problems. 3. Response-level rewards and no discounting. RLHF attribution of reward is done for an entire sequence of actions, composed of multiple generated tokens, rather than in a fine-grained manner (this single-step structure is


ed of multiple generated tokens, rather than in a fine-grained manner (this single-step structure is sometimes called a bandit problem in the RL literature). To help the RL algorithms for RLHF see every token as part of the same action, implementations usually use a discount factor of γ = 1 (no discounting), unlike standard RL where γ < 1 balances short-term and long-term reward across many sequential decisions. rlhfbook.com 24


Table 1: Key differences between standard RL and RLHF for language models. Aspect Standard RL RLHF (language models) Policy Learned from scratch (random init) Fine-tuned from a pretrained language model Reward signal Environment reward function r(st, at) Learned reward / preference model rθ(x, y) (prompt x, completion y) State transition Yes: dynamics p(st+1 | st, at) Typically no: prompts x sampled from a dataset; the completion does not define the next prompt Action Single environment action at A completion y (a sequence of tokens) sampled from πθ(· | x) Reward granularity Often per-step / fine-grained Usually response-level (bandit-style) over the full completion, usually no discounting (γ = 1) Horizon Multi-step episode (T > 1) Often single-step (T = 1), though multi-turn can be modele


γ = 1) Horizon Multi-step episode (T > 1) Often single-step (T = 1), though multi-turn can be modeled as longer-horizon Given the single-turn nature of the problem, the optimization can be re-written without the time horizon and discount factor (and with an explicit reward model): max π Eτ∼π [rθ(st, at)] . (10) In many ways, the result is that while RLHF is heavily inspired by RL optimizers and problem formulations, the actual implementation is very distinct from traditional RL. 3.1.4 Fine-Tuning and Regularization In traditional RL problems, the agent must learn from a randomly initialized policy, but with RLHF, we start from a strong pretrained base model with many initial capabilities. This strong prior for RLHF induces a need to prevent the optimization from drifting too far from the i


is strong prior for RLHF induces a need to prevent the optimization from drifting too far from the initial policy. In order to succeed in a fine-tuning regime, RLHF techniques employ multiple types of regularization to control the optimization. The goal is to allow the reward maximization to still occur without the model succumbing to over-optimization, as discussed in Chapter 14. The most common change to the optimization function is to add a KL divergence penalty on the distance between the current RLHF policy and the starting point of the optimization. The β hyperparameter set when training the model controls the strength of this constraint – a larger β keeps the model closer to its starting point, while a smaller β gives the optimizer more freedom to chase reward: max π Eτ∼π [rθ(st, at


ing point, while a smaller β gives the optimizer more freedom to chase reward: max π Eτ∼π [rθ(st, at)] −βDKL(π(·|st)∥πref(·|st)). (11) rlhfbook.com 25


Figure 7: Standard RLHF loop rlhfbook.com 26


Within this formulation, a lot of study into RLHF training goes into understanding how to spend a certain “KL budget” as measured by a distance from the initial model. For more details, see Chapter 15 on Regularization. 3.1.5 Optimization Tools In this book, we detail many popular techniques for solving this optimization problem. The popular tools of post-training include: • Reward modeling (Chapter 5): A model is trained to capture the signal from collected preference data and can then output a scalar reward indicating the quality of future text. • Instruction fine-tuning (Chapter 4): A prerequisite to RLHF where models are taught the question-answer format used in the majority of language modeling interactions today by imitating preselected examples.


mat used in the majority of language modeling interactions today by imitating preselected examples. • Rejection sampling (Chapter 9): The most basic RLHF technique where candidate completions for instruction fine-tuning are filtered by a reward model imitating human preferences. • Policy gradients (Chapter 6): The reinforcement learning algorithms used in the seminal examples of RLHF to update parameters of a language model with respect to the signal from a reward model. • Direct alignment algorithms (Chapter 8): Algorithms that directly optimize a policy from pairwise preference data, rather than learning an intermediate reward model to then optimize later. Modern RLHF-trained models always utilize instruction fine-tuning followed by a mixture of the other optimization options.


els always utilize instruction fine-tuning followed by a mixture of the other optimization options. 3.1.6 Subtle Advantages of RL in Post-Training Language Models In the following chapters, we cover many optimization tools for post-training. Plenty of them, such as rejection sampling (Chapter 9) and direct alignment algorithms like DPO (Chapter 8), are far simpler than getting RL working. Still, despite the simplicity of alternatives, RL-based methods continue to win out. Some trends, such as the inference-time scaling with reinforcement learning with verifiable rewards (RLVR), are obvious, but RL has turned out to be a well-suited optimization tool for language models. Implementing RL requires a far larger infrastructure investment relative to instruction tuning or DPO-like algorithms, bu


res a far larger infrastructure investment relative to instruction tuning or DPO-like algorithms, but, at the risk of being overly colloquial, the gradient updates it provides “generally help the model a lot.” This is hard to quantify, but comes in a few recurring forms: • RL stages can “fix” rough edges on the model, making the model easier to chat with or more robust (this could come by training it to have numerical stability with inference tools like vLLM). The exact reason for this is not well-known in the literature, but its truth is reflected in the growing presence of RL today. • RL can be done surgically — the model does a good job of learning where the prompt distribution lies, and RL tends to not “squash” the general capabilities of the model.


e the prompt distribution lies, and RL tends to not “squash” the general capabilities of the model. A good example of this is Tülu 3 being trained with RL only on math prompts, while maintaining capabilities across a broad task suite [6]. Overall, RL losses on language models are robust, scalable, effective, and flexible, which opened large new fields of experimentation. The original method that started us down this rlhfbook.com 27


path was RLHF work. 3.2 Canonical Training Recipes Over time various models have been identified as canonical recipes for RLHF specifically or post-training generally. These recipes reflect data practices and model abilities at the time. As the recipes age, training models with the same characteristics becomes easier and requires less data. There is a general trend of post-training involving more optimization steps with more training algorithms across more diverse training datasets and evaluations. 3.2.1 InstructGPT Around the time ChatGPT first came out, the widely accepted (“canonical”) method for post-training an LM had three major steps, with RLHF being the central piece [55] [3] [5]. The three steps taken on top of a “base” language model (the next-token prediction model trained on la


e three steps taken on top of a “base” language model (the next-token prediction model trained on large-scale web text) are summarized below in fig. 8: 1. Instruction tuning on ~10K examples: This teaches the model to follow the question-answer format and teaches some basic skills from primarily human-written data. 2. Training a reward model on ~100K pairwise prompts (paper used 33K prompts): This model is trained from the instruction-tuned checkpoint and captures the diverse values one wishes to model in their final training. The reward model is the optimization target for RLHF. 3. Training the instruction-tuned model with RLHF on a separate ~100K prompts (paper used exactly 31K and does not document whether prompts were reused from other stages): The model is optimized against the reward


t document whether prompts were reused from other stages): The model is optimized against the reward model with a likely separate set of prompts, where it generates responses before receiving ratings. Once RLHF was done, the model was ready to be deployed to users. This recipe is the foundation of modern RLHF, but recipes have evolved substantially to include more stages and more data. Figure 8: A rendition of the early, three stage RLHF process with SFT, a reward model, and then optimization. rlhfbook.com 28


3.2.2 Tülu 3 Modern versions of post-training involve many, many more model versions and training stages (i.e. well more than the 5 RLHF steps documented for Llama 2 [49]). An example is shown below in fig. 9 where the model undergoes numerous training iterations before convergence. Figure 9: A rendition of modern post-training with many rounds. The most complex models trained in this era and onwards have not released full details of their training process. Leading models such as ChatGPT or Claude by 2026 involve many iterative rounds of training. This can even include techniques that train specialized models and then merge the weights together to get a final model capable of many subtasks [56] (e.g. Cohere’s Command A [57]).


weights together to get a final model capable of many subtasks [56] (e.g. Cohere’s Command A [57]). Figure 10: A summary of the Tülu 3 recipe with target skills and multi-step training recipe. Lambert et al. 2024, License CC-BY. A fully open example of this multi-stage approach to post-training where RLHF plays a major role is Tülu 3. The Tülu 3 recipe consists of three stages: 1. Instruction tuning on ~1M examples: This primarily synthetic dataset, drawn from a mix of frontier models such as GPT-4o and Llama 3.1 405B, teaches the model general instruction following and serves as the foundation for capabilities such as mathematics and coding. 2. On-policy preference data on ~1M preference pairs: This stage substantially rlhfbook.com 29


boosts the chattiness (e.g. Arena, formerly ChatBotArena, or AlpacaEval 2) of the model while also improving skills mentioned above in the instruction tuning stage. 3. Reinforcement Learning with Verifiable Rewards on ~10K prompts: This stage is a small-scale reinforcement learning run to boost core skills such as mathematics while maintaining overall performance (and is now seen as a precursor to modern reasoning models such as DeepSeek R1). The recipe has been successfully applied to Llama 3.1 [6], OLMo 2 [58], and SmolLM models [59]. 3.2.3 DeepSeek R1 With the rise of reasoning language models, such as OpenAI’s o1, the best practices in post-training evolved again to re-order and redistribute compute across training stages.


actices in post-training evolved again to re-order and redistribute compute across training stages. The clearest documentation of a reasoning model post-training recipe is DeepSeek R1 [15], which has been mirrored by Alibaba’s larger Qwen 3 models (i.e. only the 32B and 225B MoE models) [60] or Xiaomi’s MiMo 7B [61]. The DeepSeek recipe follows: 1. “Cold-start” with 100K+ on-policy reasoning samples: This data is sampled from an earlier RL checkpoint, R1-Zero, and heavily filtered to instill a specific reasoning process on DeepSeek-V3-Base. DeepSeek uses the term cold-start to describe how RL is learned from little supervised data. 2. Large-scale reinforcement learning training: This stage repeatedly covers reasoning problems with the model, running RLVR “until convergence” on a variety of


epeatedly covers reasoning problems with the model, running RLVR “until convergence” on a variety of benchmarks. 3. Rejection sampling and SFT: Near convergence, they apply rejection sampling to the RL checkpoint to build an SFT dataset of ~800K samples, then fine-tune the model on a filtered mix of roughly 3/4 reasoning problems and 1/4 general queries to produce a general-purpose model. 4. Mixed reinforcement learning training on reasoning problems (verifiable rewards) with general preference tuning reward models to polish the model. As above, there are evolutions of the recipe, particularly with steps 3 and 4 to finalize the model before exposing it to users. Many models start with tailored instruction datasets with chain-of-thought sequences that are heavily filtered and polished from


ed instruction datasets with chain-of-thought sequences that are heavily filtered and polished from existing models, providing a fast step to strong behaviors with SFT alone before moving onto RL [62]. rlhfbook.com 30


4 Instruction Fine-Tuning Early large pretrained language models were trained with a next-token prediction objective and, by default, did not come with an explicit interface for following instructions. Around the release of GPT-3 [63], prompting and in-context learning became a widely used way to adapt a single model to many tasks (though task-specific fine-tuning remained common), by showing examples in-context and asking the model to complete a similar task. A practical next step was instruction fine-tuning, which teaches the model to respond in an instructionresponse format rather than just continuing text. For example, given the prompt “What is the capital of France?”, a base model might continue with “What is the capital of Germany? What is the capital of Italy?. . .


base model might continue with “What is the capital of Germany? What is the capital of Italy?. . . ” — simply extending the pattern of questions — while an instruction-tuned model would respond with “The capital of France is Paris.” Instruction fine-tuning took off when two lines of work converged. First, NLP shifted from bespoke fine-tuning task setups to a unified “text-to-text” or instruction framing, which made it straightforward to standardize diverse datasets and train a single model across many tasks. Prominent examples of unifying the framework for tasks include Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5 models) [64], Finetuned Language Models Are Zero-Shot Learners (FLAN dataset) [65], Multitask Prompted Training Enables Zero-Shot Task G


els Are Zero-Shot Learners (FLAN dataset) [65], Multitask Prompted Training Enables Zero-Shot Task Generalization (T0 models) [66], and Cross-Task Generalization via Natural Language Crowdsourcing Instructions (Natural Instructions dataset) [67]. Second, scaling pretrained LMs and the rise of prompting/in-context learning showed that a single model could generalize across tasks, but that generalization becomes far more reliable when the model is explicitly trained on instruction-response examples. Together, these trends led to an era of fine-tuning pretrained language models on large collections of instructions—what is now commonly called instruction fine-tuning (IFT), or supervised fine-tuning (SFT), in which training general models became accessible to wider audiences.


upervised fine-tuning (SFT), in which training general models became accessible to wider audiences. Since its discovery, instruction fine-tuning, also called colloquially just instruction tuning, has matured and is standard practice across many language modeling pipelines. At its core, IFT is the simplest method for adapting language models to a desired task distribution. It serves as the foundation for RLHF by preparing the model for a format of instructions that is known as question-answering, and it is the first tool used by those attempting to apply modern techniques to new domains. Without a basic level of instruction-following abilities, most of the pipelines we discuss in this book—from preference data collection to online RLHF optimization—cannot be performed.


scuss in this book—from preference data collection to online RLHF optimization—cannot be performed. Instruction fine-tuning generally is covered extensively elsewhere and is supervised learning at its core, so this chapter focuses on the practical details that matter most for RLHF practitioners: how training data is formatted and structured. Decisions on data and formatting are directly leveraged in the later training stages to create a common language for the model to absorb post-training data. 4.1 Chat Templates and the Structure of Instructions The post-training process begins with defining a pattern to format user queries so that they are easily readable by a language model that processes information through a tokenizer.


o that they are easily readable by a language model that processes information through a tokenizer. When using a pretrained language model, the prompting is quite simple. The model only knows a few tokens: a beginning-of-sequence token (e.g., <bos_token>), an end-of-sequence token (e.g., <eos_token>), and a padding token (to manage training on batches with empty rlhfbook.com 31


components). This means, to prompt a base model, the user inputs a sequence of tokens for the model to continue from, such as: <bos_token> The capital of the United States is Then, the model would generate tokens until it runs out of its context window, or it generates the end-of-sequence token. All post-training stages, from instruction tuning to RLHF and other methods, rely on this formatting to train the model. The tool that handles the structure of the interaction with the user is called the chat template. An example which we will break down is below: {% if messages[0]['role'] == 'system' %} {# If the conversation begins with a system message, treat it as a special first turn. We set an offset so the user/assistant alternation check lines up correctly.


a special first turn. We set an offset so the user/assistant alternation check lines up correctly. #} {% set offset = 1 %} {% else %} {# No system message: user should be the first non-empty turn. #} {% set offset = 0 %} {% endif %} {# Emit the beginning-of-sequence token (model-specific). #} {{ bos_token }} {# Serialize each message into the model's chat-markup tokens. #} {% for message in messages %} {# Enforce role alternation: (system), user, assistant, user, assistant, ... The boolean expression compares "is this a user message?" against whether the current index (plus offset) is expected to be user or assistant. #} {% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %} {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }} {% endif


raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }} {% endif %} {# Wrap each message with special tokens: - <|im_start|><role>\n - message content (trimmed) - <|im_end|>\n This produces a single flat token sequence the LM can train on. #} {{ '<|im_start|>' + message['role'] + '\n' + message['content'] | trim + '<|im_end|>\n' }} {% endfor %} {# Optionally append an "assistant" start tag with no content. This cues generation to continue from the assistant role. #} {% if add_generation_prompt %} {{ '<|im_start|>assistant\n' }} {% endif %} This is the raw code for transforming a list of dictionaries in Python containing messages and roles into tokens that a language model can predict from. All information passed into models is assigned a role.


kens that a language model can predict from. All information passed into models is assigned a role. The traditional three roles are system, user, and assistant. rlhfbook.com 32


The system tag is only used for the first message of the conversation; it holds instructions for the agent in text that will not be received from or exposed to the user. These system prompts are used to provide additional context to the models, such as the date and time, or to patch behaviors. As a fun example, models can be told things such as “You are a friendly chatbot who always responds in the style of a pirate.” Next, the two other roles are straightforward: user holds the messages from the person using the AI, and assistant holds the responses from the model (that is, engaging as an AI assistant). In order to translate all this information into tokens, we use the code listing above that we started with.


to translate all this information into tokens, we use the code listing above that we started with. The model has a series of special tokens that separate the various messages from each other. If we run the above code with the example query “How many helicopters can a human eat in one sitting?”, the token sequence passed into the model would look as follows: <|im_start|>system You are a friendly chatbot who always responds in the style of a pirate<|im_end|> <|im_start|>user How many helicopters can a human eat in one sitting?<|im_end|> <|im_start|>assistant Notice how the final tokens in the sequence are <|im_start|>assistant. This is how the model knows to continue generating tokens until it finally generates its end-of-sequence token, which in this case is <|im_end|>.


ting tokens until it finally generates its end-of-sequence token, which in this case is <|im_end|>. By packing all question-answer pair data (and downstream preference tuning data) into this format, modern language models follow it with perfect consistency. This is the language that instruction-tuned models use to exchange information with users and the models running on GPUs or other computing devices. The behavior can be extended naively to multiple turns, as shown below: <|im_start|>system You are a friendly chatbot who always responds in the style of a pirate<|im_end|> <|im_start|>user How many helicopters can a human eat in one sitting?<|im_end|> <|im_start|>assistant Oh just 6.<|im_end|> <|im_start|>user Are you sure about that?<|im_end|> <|im_start|>assistant In the open ecosystem,


|> <|im_start|>user Are you sure about that?<|im_end|> <|im_start|>assistant In the open ecosystem, the standard method for applying the chat template to a list of messages uses a Jinja snippet stored in the tokenizer configuration, as apply_chat_template. The above chat template is a derivative of OpenAI’s Chat Markup Language (ChatML), which was an early attempt to standardize message formatting. Now, OpenAI and other model providers use a hierarchical system where the user can configure a system message, yet there are higher-level instructions that may or may not be revealed to the user [68]. Many other chat templates exist. Some other examples include Zephyr’s [26]: rlhfbook.com 33


<|system|> You are a friendly chatbot who always responds in the style of a pirate</s> <|user|> How many helicopters can a human eat in one sitting?</s> <|assistant|> Or Tülu’s: <|user|> How are you doing? <|assistant|> I'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|> Beyond this, many chat templates include formatting and other tokens for tasks such as tool-use. 4.2 Best Practices for Instruction Tuning Instruction tuning as the foundation of post-training and creating helpful language models is well-established. There are many ways to achieve successful instruction tuning. For example, efficient fine-tuning with quantization of some model parameters makes training very accessible [69].


ficient fine-tuning with quantization of some model parameters makes training very accessible [69]. Also, in narrow domains such as chat alignment, i.e., without harder skills such as math or code, small, focused datasets can achieve strong performance [14]. Soon after the release of ChatGPT, human datasets with as few as 10K samples such as No Robots were state-of-the-art [70]. Years later, large-scale synthetic datasets work best [6] on most tasks. A few principles remain: • High-quality data is key to performance. The completions are what the model actually learns from (in many cases the prompts are not predicted over so the model does not learn to predict prompts). • Around 1M prompts can be used to create a model capable of excellent RLHF and post-training.


ts). • Around 1M prompts can be used to create a model capable of excellent RLHF and post-training. Further scaling can still help, but returns diminish quickly. • The best prompts are those in a similar distribution to downstream tasks of interest. • If multiple stages of training are done after instruction tuning, the models can recover from some noise in the instruction-tuning data. Crafting the overall optimization is more important than fixating on each individual stage. 4.3 Implementation Details While the loss function is the same as that used in pretraining, there are a few key implementation details that differ from the setting used for pretraining. Many practices, such as deciding on the types of parallelism used to shard models across many GPUs, are the same as pretraining, but


on the types of parallelism used to shard models across many GPUs, are the same as pretraining, but the total number of machines used is often lower (for the first technical change listed below): • Smaller batch sizes: Compared to pretraining, instruction tuning (and other posttraining techniques such as preference fine-tuning) use substantially smaller batch sizes to optimize well on a narrower data distribution while preserving the model’s rlhfbook.com 34


generalization from pretraining. For example, OLMo 2 uses a batch size of 1024 packed-rows for the 7B and 2048 for the 13B pretraining, where these models have a total context length of 4096 tokens, and each row in the batch is a combination of documents that fills the sequence length. For post-training, both these models only use a batch size of 256 prompts [58], without filling to the full sequence length (for far fewer non-masked tokens per batch). The smaller batch sizes mean that these training jobs cannot be sharded across as many devices as during pretraining – in practice, distributed training setups have minimum per-device batch sizes, so if you’re trying to retain a smaller global batch size for SFT you can use cumulatively fewer GPUs.


if you’re trying to retain a smaller global batch size for SFT you can use cumulatively fewer GPUs. In practice the batch size forcing a smaller concurrent GPU allotment per training job is not a limiting factor because the training token counts for SFT are much smaller than in pretraining, and training for multiple seeds is needed in post-training to obtain the best final performance. • Prompt masking: When pretraining, every token in the batch is predicted autoregressively and the loss is then applied to them. For instruction tuning, the prompt tokens are masked out so the model isn’t learning to accurately predict user queries – just responses. The same applies to other post-training algorithms. • Multi-turn masking: For multi-turn conversations, there are two common masking choices.


gorithms. • Multi-turn masking: For multi-turn conversations, there are two common masking choices. (1) Final-turn only: only the tokens in the final assistant turn are included in the loss, while all earlier context (including earlier assistant turns) is masked. Long conversations can still be “unrolled” into multiple training samples: for a conversation of N turns, each example predicts one assistant response while masking all prior context and excluding any future turns. (2) Mask user turns only: all user turns are masked, but every assistant turn is included in the loss. You can still unroll in this setting if you want more (shorter) training examples, but the key difference is that intermediate assistant replies are trained on directly.


ng examples, but the key difference is that intermediate assistant replies are trained on directly. • Same loss function as pretraining: Instruction tuning uses the same autoregressive loss function used in pretraining language models, but with substantially different data and masking (training only on full sequences, whereas pretraining documents can be split across batches), etc. • Learning rate: SFT typically uses a learning rate one to two orders of magnitude smaller than pretraining to best manage the different optimization dynamics (smaller datasets, smaller batches, and a strong pretrained initialization all favor more conservative updates). For example, OLMo 2 uses a peak learning rate of 3 × 10−4 for pretraining but 1 × 10−5 for SFT [58].


or example, OLMo 2 uses a peak learning rate of 3 × 10−4 for pretraining but 1 × 10−5 for SFT [58]. OLMo 3 uses a higher SFT learning rate of 5-8 × 10−5 [18], in part because its training infrastructure uses sequence packing, which fits multiple examples into each training sequence and increases the effective batch size measured in useful tokens. Larger batches produce lower-variance gradient estimates, which in turn supports a higher learning rate without destabilizing training – a relationship known as the linear scaling rule. The learning rate is commonly warmed up over a small fraction of training steps before decaying linearly. In practice, teams often sweep over multiple learning rates and select the best checkpoint on a held-out evaluation suite [18]. rlhfbook.com 35


5 Reward Modeling Reward models are core to the modern approach to RLHF by being where the complex human preferences are learned. They are what enable our models to learn from hard-to-specify signals. They compress complex features in the data into a representation that can be used in downstream training – a sort of magic that once again shows the complex capacity of modern deep learning. These models act as proxy objectives for the core optimization, as studied in the following chapters. As shown in fig. 11, the reward model plays a role like the standard RL environment, providing the learning signal for the agent, but unlike a fixed environment, we get to learn it from human preferences. Reward models have historically been used extensively in reinforcement learning research as a proxy f


eward models have historically been used extensively in reinforcement learning research as a proxy for environment rewards [54]. Reward models were proposed, in their modern form, as a tool for studying the value alignment problem [38]. These models tend to take in some sort of input and output a single scalar value of reward. This reward can take multiple forms – in traditional RL problems it was attempting to approximate the exact environment reward for the problem, but we will see in RLHF that reward models actually output a probability of a certain input being “of high quality” (i.e. the chosen answer among a pairwise preference relation). The practice of reward modeling for RLHF is closely related to inverse reinforcement learning, where the problem is to approximate an agent’s reward


ely related to inverse reinforcement learning, where the problem is to approximate an agent’s reward function given trajectories of behavior [71], and other areas of deep reinforcement learning. The high-level problem statement is the same, but the implementation and focus areas are entirely different, so they’re often considered as totally separate areas of study. The most common reward model, often called a Bradley-Terry reward model and the primary focus of this chapter, predicts the probability that a piece of text was close to a “preferred” piece of text from the training comparisons. Later in this section we also compare these to Outcome Reward Models (ORMs), Process Reward Models (PRMs), and other types of reward models.


se to Outcome Reward Models (ORMs), Process Reward Models (PRMs), and other types of reward models. Throughout this chapter, we use x to denote prompts and y to denote completions. This notation is common in the language model literature, where methods operate on full promptcompletion pairs rather than individual tokens. 5.1 Training a Bradley-Terry Reward Model The canonical implementation of a reward model is derived from the Bradley-Terry model of preference [72]. There are two popular expressions for how to train a standard reward model for RLHF – they are mathematically equivalent. To start, a Bradley-Terry model of preferences defines the probability that, in a pairwise comparison between two items i and j, a judge prefers i over j: P(i > j) = pi pi + pj .


a pairwise comparison between two items i and j, a judge prefers i over j: P(i > j) = pi pi + pj . (12) The Bradley-Terry model assumes that each item has a latent strength pi > 0, and that observed preferences are a noisy reflection of these underlying strengths. It is common to reparametrize the Bradley-Terry model with unbounded scores, where pi = eri, which results in the following form: rlhfbook.com 36


Figure 11: The reward model in RLHF plays the role of the environment component that returns rewards in standard RL. The key difference is that in RLHF, we get to control and learn this reward function from human preferences, rather than having it fixed by the environment. rlhfbook.com 37


P(i > j) = eri eri + erj = σ(ri −rj). (13) Only differences in scores matter: adding the same constant c to every rk leaves P(i > j) unchanged. These forms are not a law of nature, but a useful approximation of human preferences that often works well in RLHF. To train a reward model, we must formulate a loss function that satisfies the above relation. In practice, this is done by converting a language model into a model that outputs a scalar score, often via a small linear head that produces a single reward value from the model’s final hidden state. Given a prompt x and two sampled completions y1 and y2, we score both with a reward model rθ and write the conditional scores as rθ(yi | x). The probability that the reward model assigns to y1 being preferred to y2 becomes: P(y1 > y2 | x) = exp


probability that the reward model assigns to y1 being preferred to y2 becomes: P(y1 > y2 | x) = exp (rθ(y1 | x)) exp (rθ(y1 | x)) + exp (rθ(y2 | x)). (14) We denote the preferred completion as yc (chosen) and the rejected completion as yr. The resulting loss encourages the reward model to assign a higher score to the humanpreferred completion than the rejected one, using a sigmoid to convert the score difference into a probability. The preference likelihood in eq. 14 is the starting point. We first rewrite that likelihood into sigmoid form, and only in the last step convert it into the equivalent negative log-likelihood loss used to train the reward model: θ∗= arg max θ P(yc > yr | x) = arg max θ exp (rθ(yc | x)) exp (rθ(yc | x)) + exp (rθ(yr | x)) = arg max θ exp (rθ(yc | x)) exp (rθ(yc


max θ exp (rθ(yc | x)) exp (rθ(yc | x)) + exp (rθ(yr | x)) = arg max θ exp (rθ(yc | x)) exp (rθ(yc | x))  1 + exp(rθ(yr|x)) exp(rθ(yc|x))  = arg max θ 1 1 + exp(rθ(yr|x)) exp(rθ(yc|x)) = arg max θ 1 1 + exp (−(rθ(yc | x) −rθ(yr | x))) = arg max θ σ (rθ(yc | x) −rθ(yr | x)) = arg min θ −log (σ (rθ(yc | x) −rθ(yr | x))) (15) The first form is the log-sigmoid expression derived above, as in [3] and other works: L(θ) = −log (σ (rθ(yc | x) −rθ(yr | x))) (16) The second is a mathematically equivalent form expressed using the softplus function log(1 + ex), as in [23] and other works: L(θ) = log  1 + erθ(yr|x)−rθ(yc|x) (17) rlhfbook.com 38


These are equivalent by letting ∆= rθ(yc | x) −rθ(yr | x) and using σ(∆) = 1 1+e−∆, which implies −log σ(∆) = log(1 + e−∆) = log 1 + erθ(yr|x)−rθ(yc|x) . They both appear in the RLHF literature. Figure 12: Training a preference reward model requires pairs of chosen and rejected completions. The model computes a scalar score for each completion from a sequence-level representation, often the end-of-sequence (EOS) token’s hidden state, and the contrastive loss depends only on the score difference between the two. 5.2 The Default Reward Model Architecture The most common way reward models are implemented is through an abstraction similar to Transformers’ AutoModelForSequenceClassification, which appends a small linear head to the language model and produces a scalar reward score for a promp


ich appends a small linear head to the language model and produces a scalar reward score for a prompt-completion pair at training or inference. At inference time, the model outputs the relative likelihood that the piece of text is chosen as a single logit from the model. Other implementation options exist, such as just taking a linear layer directly from the final embeddings, but they are less common in open tooling. 5.3 Implementation Example Implementing the reward modeling loss is quite simple. More of the implementation challenge is on setting up a separate data loader and inference pipeline. Given the correct dataloader with tokenized, chosen and rejected prompts with completions, the loss is implemented as: import torch.nn as nn # inputs_chosen / inputs_rejected include the prompt to


oss is implemented as: import torch.nn as nn # inputs_chosen / inputs_rejected include the prompt tokens x and the respective # completion tokens (y_c or y_r) that the reward model scores jointly. rewards_chosen = model(**inputs_chosen) rewards_rejected = model(**inputs_rejected) loss = -nn.functional.logsigmoid(rewards_chosen - rewards_rejected).mean() As for the bigger picture, this is often within a causal language model (a model that generates tokens left-to-right, predicting each token conditioned on all previous ones) that has an additional head added (and learned with the above loss) that transitions from the final hidden state to the score of the inputs. The code takes in standard transformer inputs – input_ids (tokenized text) and attention_mask (which marks real tokens vs.


ard transformer inputs – input_ids (tokenized text) and attention_mask (which marks real tokens vs. padding) – and extracts the hidden state (the model’s internal representation of the input) at the last rlhfbook.com 39


real token, which is then passed through a linear layer to produce a scalar reward. This model will have a structure as follows: import torch import torch.nn as nn import torch.nn.functional as F class BradleyTerryRewardModel(nn.Module): """ Standard scalar reward model for Bradley-Terry preference learning. Usage (pairwise BT loss): rewards_chosen = model(**inputs_chosen) # (batch,) rewards_rejected = model(**inputs_rejected) # (batch,) loss = -F.logsigmoid(rewards_chosen - rewards_rejected).mean() """ def __init__(self, base_lm): super().__init__() self.lm = base_lm # e.g., AutoModelForCausalLM self.head = nn.Linear(self.lm.config.hidden_size, 1) def _sequence_rep(self, hidden, attention_mask): """ Get a single vector per sequence to score.


1) def _sequence_rep(self, hidden, attention_mask): """ Get a single vector per sequence to score. Default: last non-padding token (EOS token); if no mask, last token. hidden: (batch, seq_len, hidden_size) attention_mask: (batch, seq_len) """ # Index of last non-pad token in each sequence # attention_mask is 1 for real tokens, 0 for padding lengths = attention_mask.sum(dim=1) - 1 # (batch,) batch_idx = torch.arange(hidden.size(0), device=hidden.device) return hidden[batch_idx, lengths] # (batch, hidden_size) def forward(self, input_ids, attention_mask): """ A forward pass designed to show inference structure of a standard reward model. To train one, this function will need to be modified to compute rewards from both chosen and rejected inputs, applying the loss above.


ed to be modified to compute rewards from both chosen and rejected inputs, applying the loss above. """ outputs = self.lm( input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=True, return_dict=True, ) # Final hidden states: (batch, seq_len, hidden_size) hidden = outputs.hidden_states[-1] # One scalar reward per sequence: (batch,) seq_repr = self._sequence_rep(hidden, attention_mask) rewards = self.head(seq_repr).squeeze(-1) return rewards In this section and what follows, most of the implementation complexity for reward models (and much of post-training) is around constructing the data-loaders correctly and distributed learning systems. Note, when training reward models, the most common practice is to train rlhfbook.com 40


for only 1 epoch to avoid overfitting. 5.4 Reward Model Variants Reward modeling is a relatively under-explored area of RLHF. The traditional reward modeling loss has been modified in many popular works, but the modifications have not solidified into a single best practice. 5.4.1 Preference Margin Loss In the case where annotators are providing either scores or rankings on a Likert Scale (a rating scale with ordered categories indicating magnitude of preference, e.g. 1–5), the magnitude of the relational quantities can be used in training. The most common practice is to binarize the data along the preference direction, reducing the mixed information of relative ratings or the strength of the ranking to just chosen and rejected completions.


rmation of relative ratings or the strength of the ranking to just chosen and rejected completions. The additional information, such as the magnitude of the preference, has been used to improve model training, but it has not converged as a standard practice. Llama 2 proposes using the margin between two data points, m(yc, yr), to distinguish the magnitude of preference: L(θ) = −log (σ (rθ(yc | x) −rθ(yr | x) −m(yc, yr))) (18) For example, each completion is often given a ranking from 1 to 5 in terms of quality. In the case where the chosen sample was assigned a score of 5 and rejected a score of 2, the margin m(yc, yr) = 5 −2 = 3. Other functions for computing margins can be explored. Note that in Llama 3 the margin term was removed as the team observed diminishing improvements after scali


hat in Llama 3 the margin term was removed as the team observed diminishing improvements after scaling. 5.4.2 Balancing Multiple Comparisons Per Prompt InstructGPT studies the impact of using K = 4 to 9 completions per prompt to rank, producing K 2  pairwise comparisons from each prompt [3]. Because these comparisons are highly correlated (they share the same prompt), shuffling them into the dataset naively causes the reward model to overfit. To address this, they weight the loss updates per comparison per prompt – without reweighting, prompts with more completions would contribute more total loss simply because they generate more pairs. In practice, all K 2  comparisons from a single prompt are typically included in the same training batch and averaged together, so each prompt contrib


ompt are typically included in the same training batch and averaged together, so each prompt contributes one grouped update rather than appearing across many separate batches. This reduces overfitting to individual prompts and prevents prompts with more sampled completions from dominating the loss. The loss function becomes: L(θ) = −1 K 2 E(x,yc,yr)∼D log (σ (rθ(yc | x) −rθ(yr | x))) (19) 5.4.3 K-Wise Loss Function There are many other formulations that can create suitable models of human preferences for RLHF. One such example, used in the popular, early RLHF’d models Starling 7B and 34B [73], is a K-wise loss function based on the Plackett-Luce model [74]. rlhfbook.com 41


Zhu et al. 2023 [75] formalize the setup as follows. With a prompt, or state, si, K actions (ai 0, ai 1, · · · , ai K−1) are sampled from P(a0, · · · , aK−1|si). Then, labelers rank the K actions by preference, producing a permutation σi : [K] 7→[K], where σi(0) is the most preferred action. This yields a Plackett-Luce probability over the complete ranking of all K items: P(σi|si, ai 0, ai 1, . . . , ai K−1) = K−1 Y k=0 exp(rθ⋆(si, ai σi(k))) PK−1 j=k exp(rθ⋆(si, ai σi(j))) (20) When K = 2, this reduces to the Bradley-Terry (BT) model for pairwise comparisons. Regardless, once trained, these models are used similarly to other reward models during RLHF training. 5.5 Outcome Reward Models The majority of preference tuning for language models and other AI systems is done with the Bradley-Terr


majority of preference tuning for language models and other AI systems is done with the Bradley-Terry models discussed above. For reasoning-heavy tasks, one can use an Outcome Reward Model (ORM). The training data for an ORM is constructed in a similar manner to standard preference tuning. Here, we have a problem statement or prompt, x and two completions y1 and y2. The inductive bias used here is that one completion should be a correct solution to the problem and one incorrect, resulting in (yc, yic). The architecture of the models used is very similar to a standard reward model, with a linear layer appended to a model that can output a single logit (in the case of an RM) – with an ORM, the training objective that follows is slightly different [76]: [We] train verifiers with a joint objec


training objective that follows is slightly different [76]: [We] train verifiers with a joint objective where the model learns to label a model completion as correct or incorrect, in addition to the original language modeling objective. Architecturally, this means our verifiers are language models, with a small scalar head that outputs predictions on a per-token basis. We implement this scalar head as a single bias parameter and single gain parameter that operate on the logits outputted by the language model’s final unembedding layer. To translate, this is implemented as a language modeling head that can predict two classes per token (1 for correct, 0 for incorrect), rather than a classification head of a traditional RM that outputs one logit for the entire sequence.


ther than a classification head of a traditional RM that outputs one logit for the entire sequence. Formally, following [77] this is a per-token binary cross-entropy loss: LCE(θ) = −E(s,r)∼D[r log pθ(s) + (1 −r) log(1 −pθ(s))] (21) where r ∈{0, 1} is a binary label where 1 applies to a correct answer to a given prompt and 0 applies to an incorrect answer, and pθ(s) is the scalar proportional to the predicted probability of correctness from the model being trained. In code, this outcome label is copied onto every completion token, while prompt tokens are masked with -100 so they do not contribute to the loss. Implementing an outcome reward model (and other types, as we’ll see with the Process Reward Model) involves applying the cross-entropy loss per-token based on whether the completion is


Reward Model) involves applying the cross-entropy loss per-token based on whether the completion is a correct sample. This is far closer to the language modeling loss, where it does not need the structured chosen-rejected nature of standard Bradley-Terry reward models. rlhfbook.com 42


In the simplified ORM training setup below, we are not sampling new tokens or training an LLM on next-token prediction; we feed a fixed prompt-completion sequence through the backbone and train the ORM head to predict correctness labels. The model structure could follow as: import torch.nn as nn import torch.nn.functional as F class OutcomeRewardModel(nn.Module): def __init__(self, base_lm): super().__init__() self.lm = base_lm # e.g., AutoModelForCausalLM self.head = nn.Linear(self.lm.config.hidden_size, 1) def forward(self, input_ids, attention_mask=None, labels=None): """ input_ids contains a full prompt+completion sequence. labels is token-aligned: prompt tokens are -100, and each completion token repeats the sequence outcome label (1=correct, 0=incorrect).


ns are -100, and each completion token repeats the sequence outcome label (1=correct, 0=incorrect). If labels=None, this is an inference-only forward pass and the loss is returned as None. """ outputs = self.lm( input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=True, return_dict=True, ) # Final hidden states: (batch, seq_len, hidden_size) hidden = outputs.hidden_states[-1] # One scalar logit per token: (batch, seq_len) logits = self.head(hidden).squeeze(-1) # Inference-only forward pass: no loss is computed. if labels is None: return None, logits # Only compute loss on completion tokens (labels 0 or 1) # Prompt tokens have labels = -100 mask = labels != -100 loss = None if mask.any(): loss = F.binary_cross_entropy_with_logits( logits[mask], labels[mask].float() ) els


e if mask.any(): loss = F.binary_cross_entropy_with_logits( logits[mask], labels[mask].float() ) else: loss = logits.sum() * 0 return loss, logits A simplified version of the loss follows: # Feed the full prompt+completion sequence once; no token sampling happens here. # Assume model already has: model.lm (backbone) + model.head hidden = model.lm(**inputs, output_hidden_states=True).hidden_states[-1] logits_per_token = model.head(hidden).squeeze(-1) # (batch, seq_len) # This will sometimes be compressed as model.forward() in other implementations # Binary labels: 1=correct, 0=incorrect (prompt tokens masked as -100) mask = labels != -100 rlhfbook.com 43


loss = F.binary_cross_entropy_with_logits( logits_per_token[mask], labels[mask].float() ) The important intuition here is that an ORM will output a probability of correctness at every token in the sequence (judged only by the final answer – reasoning errors are not captured in the ORM training process). This can be a noisy process, as the updates and loss propagate per token depending on outcomes and attention mappings. Figure 13: At inference time, an outcome reward model outputs per-token correctness probabilities. Prompt tokens are masked (e.g., label=-100), while completion tokens each receive a probability indicating whether the model believes the response leads to a correct answer. Figure 14: Training an outcome reward model uses offline labels from a verifier or dataset (e.g., all 1


ure 14: Training an outcome reward model uses offline labels from a verifier or dataset (e.g., all 1s for correct completions). Each completion token is trained with binary crossentropy against the outcome label, and per-token probabilities are aggregated into a final score for verification, filtering, or reranking. These models have continued to be used, but are less supported in open-source RLHF tools. For example, the same type of ORM was used in the seminal work Let’s Verify Step by Step [50], but without the language modeling prediction piece of the loss. Then, the final loss is a cross-entropy loss on every token, predicting whether the final answer is correct. Given the lack of support, the term outcome reward model (ORM) has been used in multiple ways. Some literature, e.g.


support, the term outcome reward model (ORM) has been used in multiple ways. Some literature, e.g. [77], continues to use the original definition from Cobbe et al. 2021; others use it more broadly for any verifier trained to predict whether a completion is correct. rlhfbook.com 44


5.6 Process Reward Models Process Reward Models (PRMs), originally called process-supervised reward models, are reward models trained to output scores at every step in a chain-of-thought reasoning process. These differ from a standard RM that outputs a score only at an EOS token or an ORM that outputs a score at every token. Process Reward Models require supervision at the end of each reasoning step, and then are trained similarly where the tokens in the step are trained to their relevant target – the target is the step in PRMs and the entire response for ORMs. Following [50], a binary-labeled PRM is commonly optimized with a per-step cross-entropy loss: LPRM(θ) = −E(x,s)∼D " K X i=1 ysi log rθ(si | x, s<i) + (1 −ysi) log (1 −rθ(si | x, s<i)) # (22) where s is a sampled chain-of-thought wi


rθ(si | x, s<i) + (1 −ysi) log (1 −rθ(si | x, s<i)) # (22) where s is a sampled chain-of-thought with K annotated steps, ysi ∈{0, 1} denotes whether the i-th step is correct, and rθ(si | x, s<i) is the PRM’s predicted probability that step si is valid conditioned on the original prompt x and all previous steps s<i. Here’s an example of how this per-step label can be packaged in a trainer, from HuggingFace’s TRL (Transformer Reinforcement Learning) [47]: # Get the ID of the separator token and add it to the completions separator_ids = tokenizer.encode(step_separator, add_special_tokens=False) completions_ids = [completion + separator_ids for completion in completions_ids] # Create the label labels = [[-100] * (len(completion) - 1) + [label] for completion, label in zip(completions_ids, lab


labels = [[-100] * (len(completion) - 1) + [label] for completion, label in zip(completions_ids, labels)] Traditionally PRMs are trained with a language modeling head that outputs a token only at the end of a reasoning step, e.g. at the token corresponding to a double new line or other special token. These predictions tend to be -1 for incorrect, 0 for neutral, and 1 for correct. These labels do not necessarily tie to whether or not the model is on the right path, but rather to whether the step is correct. Figure 15: Process reward models provide supervision only at step boundaries (e.g., newline tokens). Each step receives a 3-class label: correct (+1), neutral (0), or incorrect (-1). All other tokens are masked during training. An example construction of a PRM is shown below.


(-1). All other tokens are masked during training. An example construction of a PRM is shown below. rlhfbook.com 45


import torch.nn as nn import torch.nn.functional as F class ProcessRewardModel(nn.Module): def __init__(self, base_lm, num_classes=3): super().__init__() self.lm = base_lm # e.g., AutoModelForCausalLM self.head = nn.Linear(self.lm.config.hidden_size, num_classes) def forward(self, input_ids, attention_mask=None, labels=None): """ The inputs are tokenized prompts and completions, where the end of a "reasoning step" is denoted by a designated separator token such as a newline or other special marker rather than batch padding. labels will be a list of labels, True, False, and Neutral (3 labels) which will be predicted by the model. If labels=None, this is an inference-only forward pass and the loss is returned as None.


the model. If labels=None, this is an inference-only forward pass and the loss is returned as None. """ outputs = self.lm( input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=True, return_dict=True, ) # Final hidden states: (batch, seq_len, hidden_size) hidden = outputs.hidden_states[-1] # One logit vector per token: (batch, seq_len, num_classes) logits = self.head(hidden) # Inference-only forward pass: no loss is computed. if labels is None: return None, logits # Only compute loss at step boundaries (where labels != -100) # Labels map: -1 -> 0, 0 -> 1, 1 -> 2 (class indices) mask = labels != -100 loss = None if mask.any(): loss = F.cross_entropy( logits[mask], labels[mask] ) else: loss = logits.sum() * 0 return loss, logits The core loss function looks very similar to