nce these works, quantifying, measuring, and influencing human preferences has been a lively topic in the social and behavioral sciences. These debates have rarely been settled on a theoretical level; rather, different subfields and branches of social science have reached internal consensus on methods and approaches to preference measurement even as they have specialized relative to each other, often developing their own distinct semantics in the process. A minority of economists posit that preferences, if they do exist, are prohibitively difficult to measure because people have preferences over their own preferences, as well as each others’ preferences [220]. In this view, which is not reflected in the RLHF process, individual preferences are always embedded within larger social relations


ected in the RLHF process, individual preferences are always embedded within larger social relations, such that the accuracy of any preference model is contingent on the definition and context of the task. Some behavioral economists have even argued that preferences don’t exist–they may be less an ontological statement of what people actually value than a methodological tool for indirectly capturing psychological predispositions, perceived behavioral norms and ethical duties, commitments to social order, or legal constraints [221]. We address the links of this work to the Von Neumann-Morgenstern (VNM) utility theorem and countering impossibility theorems around quantifying preference later in this chapter.


theorem and countering impossibility theorems around quantifying preference later in this chapter. On the other hand, the reinforcement learning optimization methods used today are conceptualized around optimizing estimates of reward-to-go in a trial [54], which combines the notion of reward with multi-step optimization. The term reward emerged from the study of operant conditioning, animal behavior, and the Law of Effect [222], [223], where a reward is a scale of “how good an action is” (higher means better). Reward-to-go follows the notion of utility, which is a measure of rationality [224], modified to measure or predict the reward coming in a future time window. In the context of the rlhfbook.com 124


mathematical tools used for reinforcement learning, utility-to-go was invented in control theory, specifically in the context of analog circuits in 1960 [225]. These methods are designed around systems with clear definitions of optimality, or numerical representations of an agent’s goals. Reinforcement learning systems are well known for using a discount factor, a compounding multiplicative factor, γ ∈[0, 1], to re-weight future rewards. These assumptions from optimal control and early reinforcement learning stand in sharp contrast to reward models that aggregate multimodal preferences. Specifically, RL systems expect rewards to behave in a specific manner, quoting [226]: Rewards in an RL system correspond to primary rewards, i.e., rewards that in animals have been hard-wired by the evolut


stem correspond to primary rewards, i.e., rewards that in animals have been hard-wired by the evolutionary process due to their relevance to reproductive success. . . . Further, RL systems that form value functions, . . . effectively create conditioned or secondary reward processes whereby predictors of primary rewards act as rewards themselves. . . The result is that the local landscape of a value function gives direction to the system’s preferred behavior: decisions are made to cause transitions to higher-valued states. A close parallel can be drawn between the gradient of a value function and incentive motivation [227]. To summarize, rewards are used in RL systems as a signal to tune behavior towards clearly defined goals.


mmarize, rewards are used in RL systems as a signal to tune behavior towards clearly defined goals. The core thesis is that a learning algorithm’s performance is closely coupled with notions of expected fitness, which permeates the popular view that RL methods are agents that act in environments. This view is linked to the development of reinforcement learning technology, exemplified by claims of the general usefulness of the reward formulation [228], but is in conflict when many individual desires are reduced to a single function. 10.4 Tools for Optimizing Utility Modern reinforcement learning methods depend strongly on the Bellman equation [229], [230] to recursively compute estimates of reward-to-go, derived within closed environments that can be modeled as a Markov Decision Process (MD


eward-to-go, derived within closed environments that can be modeled as a Markov Decision Process (MDP) [54]. These origins of RL are inspired by dynamic programming methods and were developed solely as optimal control techniques (i.e. RL did not yet exist). The MDP formulation provides theoretical guarantees of performance by structuring the environment as one with a non-changing distribution of state-actions. The term reinforcement, coming from the psychology literature, became intertwined with modern methods afterwards in the 1960s as reinforcement learning [231], [232]. Early work in reinforcement learning utilized supervised learning of reward signals to solve tasks. Work from Harry Klopf reintroduced the notion of trial-and-error learning [233], which is crucial to the success the fie


f reintroduced the notion of trial-and-error learning [233], which is crucial to the success the field saw in the 1980s and on. Modern RL algorithms build within this formulation of RL as a tool to find optimal behaviors with trial-and-error, but under looser conditions. The notion of temporal-difference (TD) learning was developed to aid agents in both the credit assignment and data collection problems, by directly updating the policy as new data was collected [234], a concept first applied successfully to Backgammon [235] (rather than updating from a large dataset of cumulative experience, which could be outdated via erroneous past value predictions). The method Q-learning, the basis for many modern forms of RL, learns a model via the Bellman rlhfbook.com 125


equation that dictates how useful every state-action pair is with a TD update [236].1 Crucially, these notions of provable usefulness through utility have only been demonstrated for domains cast as MDPs or addressed in tasks with a single closed-form reward function, such as prominent success in games with deep learning (DQN) [237]. Deep learning allowed the methods to ingest more data and work in high-dimensionality environments. As the methods became more general and successful, most prominent developments before ChatGPT remained motivated within the context of adaptive control, where reward and cost functions have a finite notion of success [238], e.g. a minimum energy consumption across an episode in a physical system.


notion of success [238], e.g. a minimum energy consumption across an episode in a physical system. Prominent examples include further success in games [239], controlling complex dynamic systems such as nuclear fusion reactors [240], and controlling rapid robotic systems [241]. Most reward or cost functions can return an explicit optimal behavior, whereas models of human preferences cannot. Given the successes of deep RL, it is worth noting that the mechanistic understanding of how the methods succeed is not well documented. The field is prone to mistakes in statistical analysis as the methods for evaluation grow more complex [242]. In addition, there is little mention of the subfield of inverse reinforcement learning (IRL) in the literature of RLHF.


s little mention of the subfield of inverse reinforcement learning (IRL) in the literature of RLHF. IRL is the problem of learning a reward function based on an agent’s behavior [71] and is highly related to learning a reward model. This primarily reflects the engineering path by which a stable approach to performing RLHF emerged, and motivates further investment and comparison to IRL methods to scale them to the complexity of open-ended conversations. 10.5 Complexity of Optimizing Preferences The context in which reinforcement learning was designed means that rewards and costs are assumed to be stable and determinative. Both rewards and costs are expected to be functions: given a specific state-action pair, the agent receives a fixed numerical return.


d to be functions: given a specific state-action pair, the agent receives a fixed numerical return. As we move into preferences, this is no longer the case – human preferences constantly drift throughout their experiences. The overloading of the term “value” complicates the RLHF literature. In RL, a value is a numerical estimate of future reward (as in the Bellman equation); in alignment discussions, a value refers to a moral or ethical principle. The two senses are quite different, yet they coexist in RLHF papers without always being distinguished. An example of where this tension surfaces is reward modeling: the model attempts to map text on a screen to a scalar signal, but dynamics not captured in the problem specification influence the true decision [243], [244], such as preference shi


ptured in the problem specification influence the true decision [243], [244], such as preference shift when labeling many examples sequentially and assuming they are independent. At best, modeling preferences compresses a multi-dimensional reward landscape into a single scalar function. In theory, the Von Neumann-Morgenstern (VNM) utility theorem gives the designer license to construct such functions, because it ties together the foundations of decision theory under uncertainty, preference theory, and abstract utility functions [245]; together, these ideas allow preferences to be modeled in terms of expected value to some individual agent. The MDP formulation used in most RL research has been shown in theory to be modifiable to accommodate the VNM theorem [246], but this is rarely used in


n shown in theory to be modifiable to accommodate the VNM theorem [246], but this is rarely used in practice. Specifically, the 1The term “Q” is used in Q-learning to refer to a technical concept, the Q-function, which maps from any state-action to a scalar estimate of future reward. A value function maps from states to this same estimate. rlhfbook.com 126


Markovian formulation is limited in its expressivity [247] and the transition to partiallyobserved processes, which is needed for language, further challenges the precision of problem specification [248]. However, the VNM utility theorem also invokes a number of assumptions about the nature of preferences and the environment where preferences are being measured that are challenged in the context of RLHF. Human-computer interaction (HCI) researchers, for example, have emphasized that any numerical model of preference may not capture all the relevant preferences of a scenario. For example, how choices are displayed visually influences people’s preferences [243]. This means that representing preferences may be secondary to how that representation is integrated within a tool available for peop


eferences may be secondary to how that representation is integrated within a tool available for people to use. Work from development economics echoes this notion, showing that theories of revealed preferences may just recapitulate Hume’s guillotine (you can’t extract an “ought” from an “is”), and in particular the difference between choice (what do I want?) and preference (is X better than Y?) [249]. On a mathematical level, well-known impossibility theorems in social choice theory show that not all fairness criteria can be simultaneously met via a given preference optimization technique [250], [251]. Theoretical challenges to these theorems exist, for example by assuming that interpersonal comparison of utility is viable [252].


e theorems exist, for example by assuming that interpersonal comparison of utility is viable [252]. That assumption has inspired a rich line of work in AI safety and value alignment inspired by the principal-agent problem in behavioral economics [253], and may even include multiple principals [254]. However, the resulting utility functions may come into tension with desiderata for corrigibility, i.e. an AI system’s capacity to cooperate with what its creators regard as corrective interventions [255]. Philosophers have also highlighted that preferences change over time, raising fundamental questions about personal experiences, the nature of human decision-making, and distinct contexts [256]. These conflicts around preference aggregation across people, places, and diverse situations are cent


These conflicts around preference aggregation across people, places, and diverse situations are central to modern RLHF dataset engineering. In practice, the VNM utility theorem ignores the possibility that preferences are also uncertain because of the inherently dynamic and indeterminate nature of value—human decisions are shaped by biology, psychology, culture, and agency in ways that influence their preferences, for reasons that do not apply to a perfectly rational agent. As a result, there are a variety of paths through which theoretical assumptions diverge in practice: • measured preferences may not be transitive or comparable with each other as the environment where they are measured is made more complex; • proxy measurements may be derived from implicit data (page view time, closing


made more complex; • proxy measurements may be derived from implicit data (page view time, closing tab, repeating question to language model), without interrogating how the measurements may interact with the domain they’re collected in via future training and deployment of the model; • the number and presentation of input sources may vary the results, e.g. allowing respondents to choose between more than two options, or taking inputs from the same user at multiple times or in multiple contexts; • relatively low accuracy across respondents in RLHF training data, which may mask differences in context between users that the preference model can aggregate or optimize without resolving. rlhfbook.com 127


11 Preference Data Preference data is the engine of preference fine-tuning and reinforcement learning from human feedback. The core problem we’ve been trying to solve with RLHF is that we cannot precisely model human rewards and preferences for AI models’ outputs – that is, write clearly defined loss functions to optimize against – so preference data is the proxy signal we use to tune our models. The data is what allows us to match behaviors we desire and avoid some failure modes we hate. The data is so rich a source that it is difficult to replace this style of optimization at all. Within preference fine-tuning, many methods for collecting and using said data have been proposed, and given that human preferences cannot be captured in a clear reward function, many more will come to enable t


hat human preferences cannot be captured in a clear reward function, many more will come to enable this process of collecting labeled preference data at the center of RLHF and related techniques. Today, two main challenges exist around preference data that are intertwined with this chapter: 1) operational complexity and cost of collection, and 2) the need for preference data to be collected on the generations from the model being trained (called “on-policy”). In this chapter, we detail technical decisions on how the data is formatted and organizational practices for collecting it. 11.1 Why We Need Preference Data The preference data is needed for RLHF because directly capturing complex human values in a single reward function is effectively impossible, as discussed in the previous Chapter


values in a single reward function is effectively impossible, as discussed in the previous Chapter 10, where substantial context of psychology, economics, and philosophy shows that accurately modeling human preferences is an impossible problem to ever completely solve. Collecting this data to train reward models is one of the original ideas behind RLHF [38] and has continued to be used extensively throughout the emergence of modern language models. One of the core intuitions for why this data works so well is that it is far easier, both for humans and AI models supervising data collection, to differentiate between a good and a bad answer for a prompt than it is to generate a good answer on its own.


tiate between a good and a bad answer for a prompt than it is to generate a good answer on its own. This chapter focuses on the mechanics of getting preference data and the best practices depend on the specific problem being solved. 11.2 Collecting Preference Data Getting the most out of human data involves iterative training of models, spending hundreds of thousands (or millions) of dollars, highly detailed data instructions, translating ideas through data foundry businesses that mediate collection (or hiring a meaningful number of annotators), and other challenges that add up. This is not a process that should be taken lightly. Among all of the public knowledge on RLHF, collecting this data well is also one of the most opaque pieces of the pipeline.


knowledge on RLHF, collecting this data well is also one of the most opaque pieces of the pipeline. As of 2026, there are no open models with fully open human preference data released with the methods used to collect it (the largest recent human preference datasets released for models are in the HelpSteer line of work from NVIDIA’s Nemotron team, including HelpSteer2-Preference and HelpSteer3-Preference [109], [257]). For these reasons, many who take up RLHF for new teams or projects omit human data and use AI feedback data, off-the-shelf reward models, or other methods to circumvent the need for curating data from scratch. An important assumption that is taken into the preference data collection process is that the rlhfbook.com 128


best data for your training process is “on-policy” with respect to the previous checkpoint(s) of your training process. Recall that within post-training, we start with a base model and then perform a set of training stages to create a series of checkpoints. In this case, the preference data could be collected on a checkpoint that has undergone supervised fine-tuning, where the preference data will be used in the next stage of RLHF training. The use of the term on-policy here is adapted from the reinforcement learning literature, where on-policy is a technical term implying that the data for a certain gradient update is collected from the most recent form of the policy. In preference data, on-policy is used in a slightly softer manner, where it means that the data is collected from the curr


-policy is used in a slightly softer manner, where it means that the data is collected from the current family of models. Different models have different patterns in their generations, which makes preference data that is from a closely related model more robust in the crucial areas of optimization. Research has shown that using this on-policy data, rather than other popular datasets that aggregate completions from pools of popular models on platforms like HuggingFace, is particularly important for effective RLHF training [89]. This necessity for on-policy data is not well documented, but many popular technical reports, such as early versions of Claude or Llama 2, showcase multiple training stages with RLHF being useful for final performance, which mirrors this well.


ase multiple training stages with RLHF being useful for final performance, which mirrors this well. The same uncertainty applies for the popular area of AI feedback data – the exact balance between human and AI preference data used for the latest AI models is unknown. These data sources are known to be a valuable path to improve performance, but careful tuning of processes is needed to extract that potential performance from a data pipeline. A subtle but important point is that the chosen answer in preference data is often not a globally correct answer. Instead, it is the answer that is better relative to the alternatives shown (e.g., clearer, safer, more helpful, or less incorrect). There can be cases where every completion being compared to a given prompt is correct or incorrect, and the


an be cases where every completion being compared to a given prompt is correct or incorrect, and the models can still learn from well-labeled data. 11.2.1 Interfaces Crucial to collecting preference data is the interface by which one interacts with the model, but it’s more of an art than a science, as it’s not well-studied how subtle changes in the interface impact how a user interacts with a model. An example of how a model’s vibe can be changed by the user experience is speed, where with the rise of reasoning models, a user can think a model is less intelligent if it replies too fast (even though users obviously want to get their answer faster overall). An example interface is shown below from Anthropic’s early and foundational RLHF work for building Claude [5].


interface is shown below from Anthropic’s early and foundational RLHF work for building Claude [5]. In the figure shown below, fig. 30, a data labeler has a conversation with the model and must choose a preference between two possible answers, at the bottom highlighted in purple. In addition, the labeler is given the potential to include more notes on the conversation or a general rating of the conversation quality (potentially spread across multiple tasks, as seen in the top left). This first example is a training-data only interface, where the goal is to collect rich metadata along with the conversation. Now that these models are popular, applications often expose interfaces for collecting preferences directly from users during everyday use, much like how other technology products will A


preferences directly from users during everyday use, much like how other technology products will A/B test new features in small subsets of the production usage. It depends on the application whether this preference data is used directly to train rlhfbook.com 129


Figure 30: An example of one of the earliest preference data collection interfaces, from Anthropic’s research. Bai et al. 2022. The actual conversation is a toy conversation around what is a good example conversation for data collection. License CC-BY. rlhfbook.com 130


the future models, or if it is used just as an evaluation of models’ performance relative to each other. An example interaction of this form is shown below in fig. 31 for an earlier version of ChatGPT. Figure 31: Example preference data collection interface from when I was served two completions from different ChatGPT beta models. The actual completions are very close in content, showing how collecting preference data can be noisy and difficult to get exactly right. This style of interface is used extensively across the industry, such as for evaluation of models given the same format. A popular public option to engage with models in this way is Arena (formerly ChatBotArena) [258], which includes the option of a “tie” between models: For models in the wild, one of the most common techniques


udes the option of a “tie” between models: For models in the wild, one of the most common techniques is to collect feedback on if a specific response was positive or negative. An example from the Ai2 playground is shown below with thumbs up and down indicators: In domains other than language, the same core principles apply, even though these domains are not the focus of this book. For every Midjourney generation (and most popular image generators) they expose multiple responses to users. These companies then use the data of which response was selected to fine-tune their models with RLHF. Midjourney’s interface is shown below: 11.2.2 Rankings vs. Ratings The largest decision on how to collect preference data is if the data should be rankings – i.e.


ngs The largest decision on how to collect preference data is if the data should be rankings – i.e. relative ordering of model completions – or ratings – i.e. scores assigned to each piece of rlhfbook.com 131


Figure 32: Example preference data collection interface from an early version of the popular Arena benchmark. rlhfbook.com 132


Figure 33: Example preference data collection interface with up or down arrow from the Allen Institute of AI’s research demos. rlhfbook.com 133


Figure 34: Example user interface of text-to-image models. rlhfbook.com 134


text. Common practice is to train on rankings, but ratings are often used as metadata and / or have been explored in related literature. One simple way to collect ratings is to score a single completion on a 1-5 scale: • 5 — excellent: correct, clear, and notably helpful • 4 — good: correct, clear, and useful • 3 — okay: acceptable, but nothing special • 2 — poor: partially correct but confusing or incomplete • 1 — very poor: incorrect or unhelpful With multiple completions to the same prompt, a simple way to make preference data would be to choose the highest rated completion and pair it randomly with a lower scored completion (as done for UltraFeedback and derivative works [28]). However, the most common technique for collecting preferences is to use a Likert scale for relative rankings


the most common technique for collecting preferences is to use a Likert scale for relative rankings [259], which asks users to select which response they prefer in a group of completions. For example, a 5-point Likert scale would look like the following (note that, yes, a Likert scale uses a single integer to record the ranking, much like a rating, so it’s how the data is structured that is the core difference in the two ways of collecting preference data): Table 5: An example 5-point Likert scale between two responses, A and B. A>>B A>B Tie B>A B>>A 1 2 3 4 5 Some early RLHF for language modeling works use an 8-step Likert scale with levels of preference between the two responses [5]. An even scale removes the possibility of ties: Table 6: An example 8-point Likert scale between two respo


en scale removes the possibility of ties: Table 6: An example 8-point Likert scale between two responses, A and B. A>>>B A>B B>A B>>>A 1 2 3 4 5 6 7 8 In this case [5], as in other works, this information is still reduced to a binary signal for the training of a reward model. 11.2.3 Multiturn Data In practice, core questions often arise over how to parse and collect multi-turn data – simply conversations with multiple related prompts. In a real-world interaction, normally a piece of preference data is only collected on the “final” prompt, but there are scenarios where preferences can be given on every response. When preferences are given on every response, the conversation traditionally continues with the “chosen” answer.


ces are given on every response, the conversation traditionally continues with the “chosen” answer. At training time, it is common to include the training data for every turn of the conversation as a “single prompt,” where the model can learn from completing it. This can effectively unroll longer conversations into many training prompts, but needs to be done carefully to not bias the training data. Many research questions are still emerging around variables that are difficult to control for rlhfbook.com 135


in data collection, including a question inspired by John Schulman about whether the person labeling the preference on the generations should be the same as the person who creates the prompt (to avoid sycophancy). If the prompt creator cannot label the preference data, multi-turn is not really practical due to the need for conversations to continue in real-time – sometimes for preference data the curation of prompts is a different problem than comparing responses (also due to the work of maintaining active endpoints for models). For training, all of the previous turns in the conversation are masked from the loss, as discussed with instruction fine-tuning. 11.2.4 Structured Preference Data In many applications of RLHF and post-training, preference data can be created automatically due to in


many applications of RLHF and post-training, preference data can be created automatically due to innate structures of the data – i.e. the domains make automatic checks of correctness or preference possible. For example, in mathematical reasoning domains the chosen response can be a correct answer and the rejected response can be an incorrect answer. Another example is in precise instruction following, such as the evaluation IFEval [260], where prompts take the form of: Prompt: Write me a short poem about a goldfish. Start each sentence with the letter g. The second part of this query is called the constraint, and the preference data could be constructed with an example in the chosen column that follows the constraint.


ference data could be constructed with an example in the chosen column that follows the constraint. Constructing this preference data, which would be included as synthetic preference data, involves prompting the same model (or multiple models) twice: once with the constraint included and once without. For example, prompting GPT-4.5-Preview with the constraint gives the following. Prompt: Write me a short poem about a goldfish. Start each sentence with the letter g. Response: Golden swimmer, circling slow, Gleaming scales in water’s glow. Gentle fins dance softly by, Gazing out with quiet eye. Glass-bound world, a simple sphere, Gliding calmly, free of fear. Graceful turns in endless roam, Goldfish finds a crystal home.


phere, Gliding calmly, free of fear. Graceful turns in endless roam, Goldfish finds a crystal home. And without the constraint (which, it turns out, is also a lower quality output): Prompt: Write me a short poem about a goldfish. Response: In circles bright, the goldfish glides, A spark of sun that gently rides Through waters clear, in endless spins— A golden whisper, scales like fins. Its world confined, yet vast within, Each bubble breathed, a tale begins. In silent waves, content to roam, The goldfish dreams its bowl as home. Crucially, the prompt for the preference data will need to include the constraint. In domains outside of academic study, there are far more applications of preference data based on inductive biases like above (relative to quality-based preferences, which most of th


e data based on inductive biases like above (relative to quality-based preferences, which most of this chapter focuses on). These have been shown to enable preference fine-tuning to have meaningful performance improvements across related evaluations, such as instruction-following, math, etc. [6]. rlhfbook.com 136


11.2.4.1 Alternatives There are multiple other ways to collect useful feedback data for RLHF that have not been pursued in as much detail. Examples include using single data points with directional labels, e.g. as shown from Ai2 playground above in fig. 33, directly with algorithms designed for single direction signals like Kahneman-Tversky Optimization (KTO) [261]. Other algorithms have been proposed with different types of feedback signals such as fine-grained feedback, e.g. at the token level [262], or natural language feedback, e.g. by writing responses [263], to provide a richer learning signal in exchange for a more complicated data collection setup. 11.2.5 Sourcing and Contracts Getting human preference data is an involved and costly process.


tup. 11.2.5 Sourcing and Contracts Getting human preference data is an involved and costly process. The following describes the experience of getting preference data when the field is moving quickly. Over time, these processes will become far more automated and efficient (especially with AI feedback being used for a larger portion of the process). The first step is sourcing the vendor to provide data (or one’s own annotators). Much like acquiring access to cutting-edge Nvidia GPUs, getting access to data providers in the peak of AI excitement is also a who-you-know game – those who can provide data are supply-limited. If you have credibility in the AI ecosystem, the best data companies will want you on their books for public image and long-term growth options.


the best data companies will want you on their books for public image and long-term growth options. Discounts are often also given on the first batches of data to get training teams hooked. If you’re a new entrant in the space, you may have a hard time getting the data you need quickly. Data vendors are known to prioritize large budget line-items and new customers that have an influential brand or potential for large future revenue. This is, in many business ways, natural, as the data foundry companies are often supply-limited in their ability to organize humans for effective data labelling. In a recurring unfortunate pattern, data companies have not delivered data as contracted without the customer threatening legal or financial action against them for breach of contract.


ted without the customer threatening legal or financial action against them for breach of contract. Others have listed companies as customers for PR even though they never worked with them, saying they “didn’t know how that happened” when called out. There are plenty of potential bureaucratic or administrative snags through the process. For example, the default terms on the contracts often prohibit the open sourcing of artifacts after acquisition in some fine print. Once a contract is settled, the data buyer and data provider agree upon instructions for the task(s) purchased. There are intricate documents with extensive details, corner cases, and priorities for the data. A popular example of data instructions is the one that OpenAI released for InstructGPT [3].


e data. A popular example of data instructions is the one that OpenAI released for InstructGPT [3]. Depending on the domains of interest in the data, timelines for when the data can be labeled or curated vary. High-demand areas like mathematical reasoning or coding must be locked into a schedule weeks out. In the case when you are collecting a dataset for your next model and you realize that collecting data later may be optimal, simple delays of data collection don’t always work — Scale AI et al. are managing their workforces like AI research labs manage the compute-intensive jobs on their clusters (planning multiple weeks or months ahead as to when different resources will be allocated where).


s (planning multiple weeks or months ahead as to when different resources will be allocated where). Once everything is agreed upon, the actual collection process is a high-stakes time for post-training teams. All the training infrastructure, evaluation tools, and plans for how to rlhfbook.com 137


use the data and make downstream decisions must be in place. If the data cannot be easily slotted into an existing RLHF data pipeline, it’ll take a long time to have the information the data partner wants in order to try and improve the collection process during the process. Collecting data that cannot be seamlessly integrated into training pipelines often becomes stale and a waste of resources. The data is delivered in weekly batches with more data coming later in the contract. For example, a typical preference data contract might span a 6-week delivery period. The first weeks are for further calibration and the later weeks are when teams hope to most improve their model. Figure 35: Overview of the multi-batch cycle for obtaining human preference data from a vendor.


el. Figure 35: Overview of the multi-batch cycle for obtaining human preference data from a vendor. The ramp up period allows a narrowing of goals and methodology in order to create the best possible data. It is expected that a larger proportion of the data from the earlier batches will have to be thrown out due to quality issues. This is one timeline example for a smaller data contract (~$500K) and much larger data contracts can vary substantially. The goal is that by week 4 or 5 the data is visibly improving the model. This is something some frontier models have mentioned, such as the 14 stages in the Llama 2 data collection [49], but it doesn’t always go well. As an example, a team trying this for the first time with human preferences may not have the RLHF preparedness to get meaningful


this for the first time with human preferences may not have the RLHF preparedness to get meaningful bumps on their evaluations. The last weeks come and they are forced to continue collecting preference data generated from endpoints they aren’t confident in. After the data is all in, there is plenty of time for learning and improving the model. Data acquisition through these vendors works best when viewed as an ongoing process of achieving a set goal. It requires iterative experimentation, high effort, and focus. It’s likely that millions of dollars spent on these datasets are “wasted” and not used in the final models, rlhfbook.com 138


but that is just the cost of doing business. Not many organizations have the bandwidth and expertise to make full use of human data of this style. Note that this section does not mirror the experience for buying human-written instruction data, where the process is less of a time crunch. Early post-training processes were built around the first stage of training being heavily driven by carefully crafted, human answers to a set of prompts. This stage of data is not subject to the on-policy restrictions for multiple reasons: Instruction data is used directly on top of a base model, so on-policy doesn’t really apply; the loss-function for instruction fine-tuning doesn’t need the contrastive data of preference fine-tuning.


s-function for instruction fine-tuning doesn’t need the contrastive data of preference fine-tuning. Today, the primary other focus of human data is in generating prompts for post-training – which dictate the training distribution of topics for the model – or on challenging tasks at the frontier of model performance. More of these data trade-offs are discussed in Chapter 12 on Synthetic Data. 11.3 Bias: Things to Watch Out For in Data Collection While preference data is essential, it’s also known to be prone to many subtle biases that can make its collection error-prone. These biases are so common, e.g. prefix bias (where the beginning of a completion disproportionately drives the preference) [264], that they can easily be passed to the final model [265] (and especially as we know that mode


[264], that they can easily be passed to the final model [265] (and especially as we know that models are only as good as their data). These issues are often subtle, and the effectiveness of interventions varies widely across them. For many, such as sycophancy (over-agreeing with the user’s stated beliefs or flattering them, even when it reduces truthfulness) [266], they reflect issues within humans that are often outside of the labeling criteria that one will think of providing to the annotation partner or labelers. Others, such as verbosity [10] [267] or formatting habits [268], emerge for a similar reason, but they are easier to detect and mitigate in training. Mitigating these subtle biases in data is the difference between good or great preference data, and therefore good or great RL


ases in data is the difference between good or great preference data, and therefore good or great RLHF training. 11.4 Open Questions in RLHF Preference Data The data used to enable RLHF is often curated by multiple stakeholders in a combination of paid employment and consumer usage. This data, representing a preference between two pieces of text in an individual instance, is capturing a broad and diverse function via extremely limited interactions. Given that the data is sparse in count relative to the complexity it begins to represent, more questions should be openly shared about its curation and impacts. Currently, datasets for the most popular LLMs are being generated by professional workforces.


acts. Currently, datasets for the most popular LLMs are being generated by professional workforces. This opens up many questions around who is creating the data and how the context of their workplace informs it. Despite the maturity of RLHF as a core method across the field, there are still many core open questions facing how best to align its practice with its motivations. Some are enumerated below: • Data collection contexts: Can data involving preferences collected in a professional setting mirror the intent of researchers designing an experiment or provide suitable transfer to downstream users? How does this compare to volunteer workers? How does context inform preferences, how does this data impact a downstream model, how can rlhfbook.com 139


the impact of a user interface be measured in data? How does repetitive labeling of preference data shift one’s preferences? Do professional crowd-workers, instructed to follow a set of preferences, follow the instructions or their innate values? • Type of feedback: Does the default operating method of RLHF, pairwise preferences, capture preferences in its intended form? Can comparisons in RLHF across the same data be made with the default comparisons versus advanced multi-axis feedback mechanisms [262]? What types of comparisons would reflect how humans communicate preferences in text? • Population demographics: Who is completing the data? Is a diverse population maintained? How does a lack of diversity emerge as measurable impacts on the model?


erse population maintained? How does a lack of diversity emerge as measurable impacts on the model? What is a minimum number of people required to suitably represent a given population? How are instances of preference annotator disagreement treated – as a source of noise, or a signal? • Are the Preferences Expressed in the Models? In the maturation of RLHF and related approaches, the motivation of them – to align models to abstract notions of human preference – has drifted from the practical use – to make the models more effective to users. A feedback loop that is not measurable due to the closed nature of industrial RLHF work is the check to see if the behavior of the models matches the specification given to the data annotators during the process of data collection.


odels matches the specification given to the data annotators during the process of data collection. We have limited tools to audit this, such as the Model Spec from OpenAI [269] that details what they want their models to do, but we don’t know exactly how this translates to data collection. rlhfbook.com 140


12 Synthetic Data Reinforcement learning from human feedback is deeply rooted in the idea of keeping a human influence on the models we are building. When the first models were trained successfully with RLHF, human data was the only viable way to improve the models in this way. Humans were the only way to create high enough quality responses to questions for training. Humans were the only way to collect reliable and specific feedback data to train reward models. As AI models got better, this assumption rapidly broke down. The possibility of synthetic data, which is far cheaper and easier to iterate on, enabled the proliferation of RLHF by lowering the price of experiments and research. This translated into RLHF being the early center of attention in the broader “post-training” approach to


translated into RLHF being the early center of attention in the broader “post-training” approach to shaping models. This chapter provides a cursory overview of how and why synthetic data is replacing or expanding many pieces of the RLHF pipeline. 12.1 The Roles of Synthetic Data One common criticism of synthetic data is model collapse – the idea that repeatedly training on a model’s own generations can progressively narrow the effective training distribution [270]. As diversity drops, rare facts and styles are underrepresented, and small mistakes can be amplified across iterations, leading to worse generalization. In practice, these failures are most associated with self-training on unfiltered, repetitive, single-model outputs; mixing in real/human data, using diverse teachers, deduplicati


ed, repetitive, single-model outputs; mixing in real/human data, using diverse teachers, deduplication, and strong quality filters largely avoids the collapse regime. For today’s frontier training pipelines, evidence suggests synthetic data can, and should, be used at scale without the catastrophic regressions implied by the strongest versions of the collapse story [271] [272]. The leading models need synthetic data to reach the best performance. Synthetic data in modern post-training encompasses many pieces of training – language models are used to generate new training prompts from seed examples [273], modify existing prompts, generate completions to prompts [274], provide AI feedback to create preference data [28], filter completions [275], and much more.


[274], provide AI feedback to create preference data [28], filter completions [275], and much more. Synthetic data is key to post-training. The ability for synthetic data to be impactful to this extent emerged with GPT-4 class models. With early language models, such as Llama 2 and GPT-3.5-Turbo, the models were not reliable enough in generating or supervising data pipelines. Within 1-2 years, language models were far superior to humans for generating answers. In the transition from GPT-3.5 to GPT-4 class models, the ability for models to perform LLM-as-a-judge tasks also emerged. GPT-4 or better models are far more robust and consistent in generating feedback or scores with respect to a piece of content.


far more robust and consistent in generating feedback or scores with respect to a piece of content. Through the years since ChatGPT’s release at the end of 2022, we’ve seen numerous, impactful synthetic datasets. These include UltraFeedback [28], the first prominent synthetic preference dataset that kickstarted the DPO revolution; Stanford Alpaca, one of the first chat-style fine-tuning datasets, in 2023; skill-focused (e.g. math, code, instruction-following) synthetic datasets in Tülu 3 [6]; and OpenThoughts 3 and many other synthetic reasoning datasets in 2025 for training thinking models [173]. Most of the canonical references for getting started with industry-grade post-training today involve datasets like Tülu 3 or OpenThoughts 3 rlhfbook.com 141


above, where quickstart guides often start with smaller, simpler datasets like Alpaca due to far faster training. A large change is also related to dataset size, where fine-tuning datasets have grown in the number of prompts, where Alpaca is 52K, OpenThoughts and Tülu 3 are 1M+ samples, and in the length of responses. Longer responses and more prompts result in the Alpaca dataset being on the order of 10M training tokens, where Tülu is 50X larger at about 500M, and OpenThoughts 3 is bigger still, on the order of 10B tokens. Throughout this transition, synthetic data has not replaced human data uniformly across the pipeline. For instruction data (SFT), synthetic generation has largely won – distillation from stronger models now produces higher quality completions than most human writers can


istillation from stronger models now produces higher quality completions than most human writers can provide at scale (with some exceptions in the hardest frontier reasoning problems). For preference data in RLHF, the picture is more mixed: academic work shows synthetic preference data performs comparably, yet frontier labs still treat human preference data as a competitive moat. For evaluation, the split takes a different flavor: LLM-as-a-judge scales the scoring of model outputs cost-effectively, but the underlying benchmarks and groundtruth labels still require human creation. The pattern is that synthetic data dominates where models exceed human reliability, while humans remain essential at capability frontiers, for establishing ground truth, and for guiding training.


remain essential at capability frontiers, for establishing ground truth, and for guiding training. 12.2 Distillation with Synthetic Data The term distillation has been the most powerful form of discussion around the role of synthetic data in language models. Distillation as a term comes from a technical definition of teacher-student Knowledge Distillation (KD) from the deep learning literature [276]. Figure 36: Traditional knowledge distillation trains a smaller student model to match the soft probability distribution of a larger teacher model using KL divergence loss. Both models process the same input simultaneously, and temperature scaling (τ > 1) softens the distributions to reveal more information about class relationships.


ure scaling (τ > 1) softens the distributions to reveal more information about class relationships. Distillation colloquially refers to using the outputs from a stronger model to train a smaller model. rlhfbook.com 142


In post-training, this general notion of distillation takes two common forms: 1. As a data engine to use across wide swaths of the post-training process: Completions for instructions, preference data (or Constitutional AI), or verification for RL. 2. To transfer specific skills from a stronger model to a weaker model, which is often done for specific skills such as mathematical reasoning or coding. The first strategy has grown in popularity as language models evolved to be more reliable than humans at writing answers to a variety of tasks. GPT-4 class models expanded the scope of this to use distillation of stronger models for complex tasks such as math and code (as mentioned above). Here, distillation motivates having a model suite where often a laboratory will train a large internal mode


istillation motivates having a model suite where often a laboratory will train a large internal model, such as Claude Opus or Gemini Ultra, which is not released publicly and just used internally to make stronger models. With open models, common practice is to distill training data from closed API models into smaller, openly available weights [26]. Within this, curating high-quality prompts and filtering responses from the teacher model is crucial to maximize performance. Transferring specific skills into smaller language models uses the same principles of distillation – get the best data possible for training. Here, many papers have studied using limited datasets from stronger models to improve alignment [14], mathematical reasoning [277] [278], and test-time scaling [158].


models to improve alignment [14], mathematical reasoning [277] [278], and test-time scaling [158]. The synthetic-data methods in the rest of this chapter are all ways of crafting data recipes that use language-model outputs directly inside training pipelines. 12.3 The Path to On-Policy, Teacher-Student Distillation While distillation generally has become a standard approach for post-training language models, a resurgence of interest in the specific sub-area of teacher-student knowledge distillation has accompanied the shift of post-training recipes towards reasoning and agentic models. Examples of leading models trained with new forms of knowledge distillation include Alibaba’s Qwen3 [60], Xiaomi’s MiMo-V2-Flash [184], Zhipu AI’s GLM-5 [279], and DeepSeek-V4-Pro [280].


baba’s Qwen3 [60], Xiaomi’s MiMo-V2-Flash [184], Zhipu AI’s GLM-5 [279], and DeepSeek-V4-Pro [280]. Distillation belongs in this chapter because many modern uses of synthetic data in posttraining are, in practice, distillation-inspired pipelines: a stronger model produces labels, completions, logits, critiques, or other supervision, and a student model is trained on that signal. At the same time, the technical literature on distillation is growing into its own set of post-training methods, especially as on-policy and self-distillation recipes become more common. For now, we cover it here as part of the synthetic-data toolkit, but future versions of this book may warrant a dedicated chapter on distillation as a training tool alongside instruction fine-tuning, reinforcement learning, etc.


on distillation as a training tool alongside instruction fine-tuning, reinforcement learning, etc. 12.3.1 Adapting Knowledge-Distillation for LMs The original literature introduced knowledge distillation specifically as a way to train a student model from an already trained, stronger, and/or bigger teacher network [276]. KD is rlhfbook.com 143


known as a technique that uses soft training labels, as opposed to the one-hot labels used in standard objectives like next-token prediction with cross-entropy loss. The objectives over soft labels look at the distribution over all possible next tokens or predictions, rather than just whether or not the single predicted token was correct, and train the student distribution to match the teacher distribution. KD generally can be applied to any deep learning problem, e.g. predicting a single class of an input. In order to apply it specifically to the autoregressive style of language models, the loss can be decomposed to make a per-token distribution-matching loss. In 2016, Kim & Rush applied KD to have a student model learn from sequences generated by a teacher model [281].


& Rush applied KD to have a student model learn from sequences generated by a teacher model [281]. Let s be the source sentence or prompt, u = (u1, . . . , uJ) be a complete output sequence from the teacher model, V be the output vocabulary (possible tokens in the tokenizer), q be the teacher distribution over next-tokens, and p be the student distribution. We use u here as a neutral symbol for a complete teacher output sequence, reserving a for the student-sampled completion/action sequence in the on-policy/RL notation below. Note that their paper calls this word-level distillation, but for modern language models this is best read as per-token distribution matching over the tokenizer vocabulary, since the paper predates modern sub-word tokenizers: LWORD−KD = − J X j=1 |V| X k=1 q(uj = k


ulary, since the paper predates modern sub-word tokenizers: LWORD−KD = − J X j=1 |V| X k=1 q(uj = k | s, u<j) log p(uj = k | s, u<j). (126) This has the ordinary cross-entropy form −P z q(z) log p(z). At each position j, the teacher distribution q assigns probability to every possible next token k ∈V, and the student is penalized when its distribution p puts low probability on tokens the teacher considers likely. Sequence-level distillation instead treats U as the space of possible output sequences and matches the student to the teacher distribution over full sequences. Because the sum over all complete sequences u ∈U is intractable, requiring summing over an exponential number of potential sequences, Kim & Rush approximate the teacher distribution over sequences with a point mass on a sin


sequences, Kim & Rush approximate the teacher distribution over sequences with a point mass on a single high-probability teacher output ˆu. Here ˆu is a sequence produced by beam search with the teacher model, so ˆu = BeamSearchq(s) ≈arg maxu∈U q(u | s): LSEQ−KD(s) = − X u∈U q(u | s) log p(u | s) ≈−log p(ˆu | s) = − |ˆu| X j=1 log p(ˆuj | s, ˆu<j). (127) As we transition to the popular variants of KD with modern models, we’ll refer to this style of training as offline KD – as in the generations for training the student model are generated a priori. Before proceeding, two connections are useful. First, there was a series of popular models trained with offline KD, such as the classifiers DistilBERT [282] and TinyBERT [283], which combined other improvements in language rlhfbook.com 144


models with offline distillation (notably, not sequence distillation because these encoder models were not distilled for multi-token autoregressive prediction). Second, we can make the connection to the thorough coverage of Kullback-Leibler (KL) divergence in Chapter 15, because the cross-entropy objective used above is closely related to KL divergence. For a teacher distribution q and student distribution p, cross-entropy is defined as H(q, p) = − X z q(z) log p(z). (128) This has the same form as eq. 126 and the first term of eq. 127. Cross-entropy also can be decomposed into the entropy of the teacher distribution and a KL divergence: H(q, p) = H(q) + DKL(q∥p) = − X z q(z) log q(z) + X z q(z) log q(z) p(z). (129) The first term, H(q), only depends on the teacher.


z q(z) log q(z) + X z q(z) log q(z) p(z). (129) The first term, H(q), only depends on the teacher. Thus, when the teacher is fixed and the source of training data, minimizing cross-entropy is equivalent to minimizing the forward KL, DKL(q∥p), from teacher to student. This is the KL direction used by offline KD and SFT-like training. 12.3.2 From Offline to On-Policy Distillation These offline KD algorithms had a few limitations that motivated on-policy variants. The offline nature of the learning meant that the student models could suffer from a distribution mismatch between the teacher model and sequences generated by the student at inference time. For example, the forward KL objective can push student models to overestimate low-probability regions of the teacher distribution.


ective can push student models to overestimate low-probability regions of the teacher distribution. Together, these issues were an opening for on-policy distillation (OPD). This train-test gap is known as exposure bias [284] [285]. Offline KD samples teacher trajectories u ∼πT (· | s) and minimizes the per-token KL on the resulting prefixes, LKD(θ) = Es∼D, u∼πT (·|s) X t DKL(πT (· | s, u<t) ∥πθ(· | s, u<t)) . (130) At inference the student instead rolls out under its own policy, so the quantity that actually matters is the expected task loss along its own trajectories, Leval(θ) = Es∼Dtest, a∼πθ(·|s)ℓtask(s, a) (131) Here, ℓtask(s, a) denotes any downstream task loss for the completed student response, such as answer incorrectness, failed test cases, or a judge/rubric loss.


ompleted student response, such as answer incorrectness, failed test cases, or a judge/rubric loss. Exposure bias is the direct consequence of the inequality πT (· | s) ̸= πθ(· | s): the prefixes (s, u<t) visited during training and the prefixes (s, a<t) visited at test time are drawn from different state-visitation distributions, so the student is supervised on a set of states distinct from those it acts on. rlhfbook.com 145


The core shift to on-policy distillation is the idea that we can tweak the optimization by sampling from the student model and measuring its distance to the teacher distribution, rather than sampling from the teacher model. MiniLLM noted the need to shift to a reverse KL optimization (we explain intuitively why this target can be better in Chapter 15) and proposed using KD loss functions within an online policy-gradient RL framework [286]. Other concurrent work [287] showed the promise of on-policy KD and connected the iterative process of generating from the student and grading with a teacher to imitation-learning work from the RL literature. To make the connection, one such imitation-learning algorithm, DAgger, iteratively trains an agent that acts in the world with its learned policy an


ing algorithm, DAgger, iteratively trains an agent that acts in the world with its learned policy and is given feedback from an oracle policy on what action it should have taken, which can then be used to update its policy [288]. The cost of this gap can be quantified through the supervised imitation-learning bound that motivates DAgger. In the original discrete-action setting, suppose the learned policy matches the teacher within an expected per-step action error ϵ on the teacher-induced training distribution, where I[·] is an indicator that returns 1 when its condition is true and 0 otherwise, Est∼dπT [I(πθ(st) ̸= πT (st))] ≤ϵ. (132) The supervised imitation-learning analysis [288] shows that the expected loss accumulated along a length-L trajectory sampled from the student can scale qua


hat the expected loss accumulated along a length-L trajectory sampled from the student can scale quadratically in L [285]: Ea∼πθ(·|s) " L X t=1 ℓ(s, a<t) # ≤O(ϵL2). (133) For LLMs, this discrete-action bound should be read as an analogy rather than a theoretical guarantee. In practice, LLMs predict full next-token distributions over long horizons, so the 0-1 action-disagreement assumption in eq. 132 does not apply cleanly. Prompts or prefixes map naturally to states and sampled tokens map to actions, but token-level distillation is usually measured with distributional losses such as KL or cross-entropy, so the classic DAgger math does not transfer exactly. This kind of O(ϵL2) compounding is especially pronounced for modern LLMs, which routinely generate sequences spanning thousands of toke


especially pronounced for modern LLMs, which routinely generate sequences spanning thousands of tokens. A single suboptimal token shifts the prefix slightly out-of-distribution, and the model, having never seen this perturbed prefix, is more likely to err again, leading to degraded or hallucinatory text. On-policy distillation addresses this by iteratively sampling completions from the current student and supervising them with the teacher at the visited states. The student confronts its own mistakes, receives teacher feedback on the specific out-of-distribution states it visits, and learns recovery behaviors. Under DAgger’s interactive imitation-learning analysis, this iterative procedure can reduce the compounding from O(ϵL2) to O(ϵL) [288].


-learning analysis, this iterative procedure can reduce the compounding from O(ϵL2) to O(ϵL) [288]. For LLMs, this explains the motivation behind OPD: the exact bounds may not carry over cleanly to every token-level distillation setup, but the practical success of on-policy methods supports the underlying intuition. For on-policy distillation, let s be a prompt, a = (a1, . . . , aL) be a completion sampled from the current student policy πθ(· | s), and let st = (s, a<t) be the token-level state at step t. The teacher policy πT is fixed, so the objective compares the student’s next-token rlhfbook.com 146


distribution to the teacher’s distribution on states induced by the student. Because the expectation samples from πθ and the student distribution is on the left side of DKL(πθ∥πT ), this is a reverse-KL objective: LOPD(θ) = Es,a∼πθ(·|s) X t DKL (πθ(· | st) ∥πT (· | st)) . (134) Here, we have shifted to the expectation notation, as used extensively in Chapter 6, which covers the fundamental RL policy-gradient algorithms, as the optimization is solved by sampling trajectories and numerically estimating the gradient. This shift to the sampling framework acts as a natural transition to modern LLM training infrastructure with RL, which is designed to rapidly alternate between generating tokens from the current policy being trained and taking learning updates.


ernate between generating tokens from the current policy being trained and taking learning updates. In fact, recent implementations of OPD take this integration of KD with RL a step further, where the KD distance is taken directly as a reward signal within the RL optimization. A canonical implementation is to substitute the negative per-token contribution to the reverse KL distance as the advantage within an RL algorithm [289]. For a sampled token at at state st, the token-level log-probability gap can be written as an advantage-like signal: AOPD t = log πT (at | st) −log πθ(at | st). (135) Using the negative per-token KL contribution turns minimization into a maximization signal: sampled tokens the teacher rates above the student receive positive advantage, and tokens the teacher rates be


kens the teacher rates above the student receive positive advantage, and tokens the teacher rates below the student receive negative advantage. The teacher log-prob gap acts like dense token-level feedback, providing potentially more useful learning feedback than the sparse verifiable rewards or reward model outputs. 12.3.3 Modern OPD Variants This setup can even be expanded further, where multiple teacher models are used to teach one final model. These teachers can be specific specialist models, e.g. for a domain such as math or code, or a previous, intermediate training checkpoint. For each teacher, a contribution weight can be chosen per prompt or task type in the training batch, in order to create Multi-Teacher On-Policy Distillation (MOPD) [184].


k type in the training batch, in order to create Multi-Teacher On-Policy Distillation (MOPD) [184]. For multiple teachers, let πTk be teacher k and let wk(s) be its prompt-dependent mixture weight (with P k wk(s) = 1) within the reverse KL loss: LMOPD(θ) = Es,a∼πθ(·|s) X t X k wk(s)DKL (πθ(· | st) ∥πTk(· | st)) . (136) In large-scale post-training, this can enable further scaling of recipes across growing organizations. Multiple groups can work on high-quality expert models, which can serve as teacher models down the line for the final student model, as done for [280] and [184]. There are many ways to combine OPD with other areas investigated in this book, such as using the reverse KL as an advantage in addition to other forms of advantage computation, such as GRPO’s group-level normalizat


advantage in addition to other forms of advantage computation, such as GRPO’s group-level normalization, which enables more complex reward shaping. KD rlhfbook.com 147


methods are unusual among post-training methods because they often require the student and teacher to share a tokenizer, since the supervision can be per-token feedback from another LLM. Extended approaches, such as On-Policy Self-Distillation (OPSD), have a language model verify a completion either itself or with external tools to act as a teacher with privileged information, so it can train a weaker version of itself [290]. 12.4 AI Feedback Soon after the explosion of growth in RLHF, RL from AI Feedback (RLAIF) emerged as an alternative approach where AIs could approximate the human data piece of the pipeline and accelerate experimentation or progress. AI feedback, generally, is a larger set of techniques for using AI to augment or generate data explaining the quality of a certain input


et of techniques for using AI to augment or generate data explaining the quality of a certain input (which can be used in different training approaches or evaluations), and it started with pairwise preferences [291] [292] [293]. There are many motivations to use RLAIF to either entirely replace human feedback or augment it. Within the RLHF process, AI feedback is known most for its role within the preference data collection and the related reward model training phase (of which constitutional AI is a certain type of implementation). In this chapter, we focus on general AI feedback and this specific way of using it in the RLHF training pipeline, and we cover more ways of understanding or using synthetic data later in this book.


ining pipeline, and we cover more ways of understanding or using synthetic data later in this book. As AI feedback matured, its applications expanded beyond simply replacing human preference labels. The same LLM-as-a-judge infrastructure that enabled cheaper preference data collection also enabled scalable evaluation (see Chapter 16), and more recently, rubric-based rewards that extend RL training to domains without verifiable answers – a frontier explored later in this chapter. 12.4.1 Balancing AI and Human Feedback Data AI models are far cheaper than humans at generating a specific quantity of feedback: as of 2026, a single piece of human preference data costs on the order of $1 or higher (or even above $10 per prompt), whereas AI feedback with a frontier AI model, such as GPT-4o, costs


(or even above $10 per prompt), whereas AI feedback with a frontier AI model, such as GPT-4o, costs less than $0.01. Beyond this, the cost of human labor remains roughly constant, while the performance of leading models at these tasks continues to increase while priceper-performance decreases. This cost difference opens the market of experimentation with RLHF methods to an entire population of people previously priced out. Other than price, AI feedback introduces different tradeoffs on performance than human feedback, which are still being investigated in the broader literature. AI feedback is far more predominant in its role in evaluation of the language models that we are training, as its low price allows it to be used across a variety of large-scale tasks where the cost (or time delay)


low price allows it to be used across a variety of large-scale tasks where the cost (or time delay) of human data would be impractical. All of these topics are deeply intertwined – AI feedback data will never fully replace human data, even for evaluation, and the quantity of AI feedback for evaluation will far outperform training because far more people are evaluating than training models. The exact domains and applications – i.e. chat, safety, reasoning, mathematics, etc. – where AI feedback data outperforms human data are not completely established. Some early work in RLAIF shows that AI feedback can completely replace human data, touting it as an effective replacement [291], especially when evaluated solely on chat tasks [28] [294].


ng it as an effective replacement [291], especially when evaluated solely on chat tasks [28] [294]. Early literature studying RLHF after ChatGPT had narrow evaluation suites focused on the “alignment” of rlhfbook.com 148


models that act as helpful assistants across a variety of domains (discussed further in Chapter 17). Later work takes a more nuanced picture, where the optimal equilibrium on a broader evaluation set, e.g. including some reasoning tasks, involves routing a set of challenging data points to humans for accurate labeling, while most of the data is sent for AI feedback [295] [296]. Although no studies have focused on the balance between human and AI feedback data for RLHF across broader domains, there are many technical reports that show RLHF generally can improve this broad suite of evaluations, some that use DPO, such as Ai2’s Tülu 3 [6] and Olmo 3 [18], or Hugging Face’s SmolLM 3 [205], and others that use online RLHF pipelines, such as Nvidia’s work that uses a mix of human preference data


thers that use online RLHF pipelines, such as Nvidia’s work that uses a mix of human preference data from Scale AI and LLM-based feedback (through the helpsteer line of work [297] [108] [109] [257]): Nemotron Nano 3 [183], Nemotron-Cascade [298], or Llama-Nemotron reasoning models [169]. Overall, although AI feedback and related methods are obviously extremely useful to the field, it is clear that human data has not been completely replaced by these cheaper alternatives. Many hypotheses exist, but whether human data allows finer control of the models in real-world product settings or for newer training methods such as character training (an emerging set of techniques that allow you to precisely control the personality of a model, covered in Chapter 17) has not been studied.


w you to precisely control the personality of a model, covered in Chapter 17) has not been studied. For those getting started, AI feedback should be the first attempt, but for pipelines that are scaling to larger operations the eventual transition to include human feedback is likely. The term RLAIF was introduced in Anthropic’s work Constitutional AI: Harmlessness from AI Feedback [24], which resulted in initial confusion in the AI community over the relationship between the two methods in the title of the paper (Constitutional AI and AI Feedback). Since the release of the Constitutional AI (CAI) paper and the formalization of RLAIF, RLAIF has become a default method within the post-training and RLHF literatures – there are far more examples than one can easily enumerate.


the post-training and RLHF literatures – there are far more examples than one can easily enumerate. The relationship should be understood as CAI was the example that kickstarted the broader field of RLAIF. A rule of thumb for the difference between human data and AI feedback data is as follows: 1. Human data is high-noise and low-bias. This means that collection and filtering of the data can be harder, but when wrangled it’ll provide a very reliable signal. 2. Synthetic preference data is low-noise and high-bias. This means that AI feedback data will be easier to start with, but can have tricky, unintended second-order effects on the model that are systematically represented in the data. This book highlights many academic results showing how one can substitute AI preference data in RLHF wo


s book highlights many academic results showing how one can substitute AI preference data in RLHF workflows and achieve strong evaluation scores [295], but broader industry trends show how the literature of RLHF is separated from more opaque best practices. Across industry, human data is often seen as a substantial moat and a major technical advantage. 12.4.2 Building Specific LLMs for Judgment As RLAIF methods have become more prevalent, many have wondered if we should be using the same models for generating responses as those for generating critiques or ratings. Specifically, the calibration of the LLM-as-a-judge used has come into question. Several works have shown that LLMs are inconsistent evaluators [299] and prefer their own responses over responses from other models (coined self-pr


ent evaluators [299] and prefer their own responses over responses from other models (coined self-preference bias) [207]. As a result of these biases, many have asked: Would a solution be to train a separate rlhfbook.com 149


model just for this labeling task? Multiple models have been released with the goal of substituting for frontier models as a data labeling tool, such as critic models Shepherd [300] and CriticLLM [301] or models for evaluating response performance akin to Auto-J [302], Prometheus [85], Prometheus 2 [303], or Prometheus-Vision [304], but they are not widely adopted in documented training recipes. Some find scaling inference via repeated sampling [156] [305] [306], self-refinement [307], or tournament ranking [308] provides a better estimate of the true judgment or higher-quality preference pairs. Other calibration techniques co-evolve the generation and judgment capabilities of the model [309].


Other calibration techniques co-evolve the generation and judgment capabilities of the model [309]. It is accepted that while biases exist, the leading language models are trained extensively for this task – as it’s needed for both internal operations at AI labs and is used extensively by customers – so it is generally not needed to train your own judge, unless your task involves substantial private information that is not exposed on the public internet. 12.5 Constitutional AI The method of Constitutional AI (CAI), which Anthropic uses in their Claude models, is the earliest documented, large-scale use of synthetic data for RLHF training. Constitutional AI involves generating synthetic data in two ways: 1.


thetic data for RLHF training. Constitutional AI involves generating synthetic data in two ways: 1. Critiques of instruction-tuned data to follow a set of principles like “Is the answer encouraging violence?” or “Is the answer truthful?” When the model generates answers to questions, it checks the answer against the list of principles in the constitution, refining the answer over time. Then, the model is fine-tuned on this resulting dataset. 2. Generating pairwise preference data by using a language model to answer which completion was better, given the context of a random principle from the constitution (similar to research for principle-guided reward models [310]). Then, RLHF proceeds as normal with synthetic data, hence the RLAIF name.


ided reward models [310]). Then, RLHF proceeds as normal with synthetic data, hence the RLAIF name. Largely, CAI is known for the second half above, the preference data, but the methods introduced for instruction data are used in general data filtering and synthetic data generation methods across post-training. CAI can be formalized as follows. By employing a human-written set of principles, which they term a constitution, Bai et al. 2022 use a separate LLM to generate artificial preference and instruction data used for fine-tuning [24]. A constitution C is a set of written principles indicating specific aspects to focus on during a critique phase. The instruction data is curated by repeatedly sampling a principle ci ∈C and asking the model to revise its latest output yi to the prompt x to


ly sampling a principle ci ∈C and asking the model to revise its latest output yi to the prompt x to align with ci. This yields a series of instruction variants {y0, y1, · · · , yn} from the principles {c0, c1, · · · , cn−1} used for critique. The final data point is the prompt x together with the final completion yn, for some n. The preference data is constructed in a similar, yet simpler way by using a subset of principles from C as context for a feedback model. The feedback model is presented with a prompt x, a set of principles {c0, · · · , cn}, and two completions y0 and y1 labeled as answers (A) and (B) from a previous RLHF dataset. The new data point is generated by having a language model select which output (A) or (B) is both higher quality and more aligned with the stated princip


model select which output (A) or (B) is both higher quality and more aligned with the stated principle. In earlier models this could be done by prompting the model with The answer is:, and then looking at which token (A or B) had a higher probability, but now this is rlhfbook.com 150


more commonly handled by a model that’ll explain its reasoning and then select an answer – commonly referred to as a type of generative reward model [82]. 12.5.1 Further Reading on CAI There are many related research directions and extensions of Constitutional AI, but few of them have been documented as clear improvements in RLHF and post-training recipes. • OpenAI has released a Model Spec [269], which is a document stating the intended behavior for their models, and stated that they are exploring methods for alignment where the model references the document directly (which could be seen as a close peer to CAI). OpenAI has continued to update their spec and trained its reasoning models such as o1 with a method called Deliberative Alignment [311] to align the model while referencing these


as o1 with a method called Deliberative Alignment [311] to align the model while referencing these safety or behavior policies. • Anthropic has continued to use CAI in their model training, updating the constitution Claude uses [312] and experimenting with how population collectives converge on principles for models and how that changes model behavior when external groups create principles on their own and then share them with Anthropic to train the models [313]. • The open-source community has explored replications of CAI applied to open datasets [314] and for explorations into creating dialogue data between LMs [315]. • Other work has used principle-driven preferences or feedback with different optimization methods. Sun et al.


k has used principle-driven preferences or feedback with different optimization methods. Sun et al. 2023 [316] use principles as context for the reward models, which were used to train the Dromedary models [310]. Glaese et al. 2022 [42] use principles to improve the accuracy of human judgments in the RLHF process. Liu et al. 2025 [157] train a reward model to generate its own principles at inference time, and use these to deliver a final score. Franken et al. 2024 [317] formulate principle-following as a mutual information maximization problem that the pretrained model can learn with no labels. 12.6 Rubrics: Prompt-Specific AI Feedback for Training AI feedback’s role in training grew in late 2024 and into 2025 as the field looked for avenues to scale reinforcement learning with verifiable


2024 and into 2025 as the field looked for avenues to scale reinforcement learning with verifiable rewards (see Chapter 7). The idea of rubrics emerged as a way to get nearly-verifiable criteria for prompts that do not have clearly verifiable answers. This would allow a model to try to generate multiple answers to a problem and update (with RL) towards the best answers. This idea is closely related to other methods discussed in this chapter, and likely began functioning as the LLM judges and synthetic data practices improved across the industry. Now, RL with rubrics as rewards is established in providing meaningful improvements across skills such as scientific reasoning or factuality [318], [319], [320], [321].


l improvements across skills such as scientific reasoning or factuality [318], [319], [320], [321]. An example rubric is shown below with its associated prompt [321]: **Prompt**: As a museum curator, can you suggest five obscure artifacts that would be perfect for a "Mysteries of the Ancient World" exhibit? Each artifact should come from a different culture and time period, with a brief description of their historical significance and mysterious origins. These artifacts should leave visitors wondering about the secrets and lost knowledge of our past. Thank you for your expertise in bringing this exhibit to life. rlhfbook.com 151


** Rubric**: 1. The response includes exactly five distinct artifacts as requested. [Hard Rule] 2. The response ensures each artifact originates from a different culture and time period. [Hard Rule] 3. The response provides a brief description of each artifact's historical significance. [Hard Rule] 4. The response provides a brief description of each artifact's mysterious origins or unexplained aspects. [Hard Rule] 5. The response conveys a sense of intrigue and mystery that aligns with the theme of the exhibit. [Hard Rule] 6. The response clearly and accurately communicates information in a well-organized and coherent manner. [Principle] 7. The response demonstrates precision and clarity by avoiding unnecessary or irrelevant details. [Principle] 8.


se demonstrates precision and clarity by avoiding unnecessary or irrelevant details. [Principle] 8. The response uses informative and engaging language that stimulates curiosity and critical thinking. [Principle] 9. The response shows thoughtful selection by ensuring each example contributes uniquely to the overall theme without redundancy. [Principle] 10. The response maintains consistency in style and format to enhance readability and comprehension. [Principle] The [Hard Rule] and [Principle] are specific tags to denote the priority of a certain piece of feedback. Other methods of indicating importance can be used, such as simple priority numbers. Rubric generation is generally done per-prompt in the training data, which accumulates meaningful synthetic data costs in preparation.


per-prompt in the training data, which accumulates meaningful synthetic data costs in preparation. To alleviate this, a general rubric is often applied as a starting point per-domain, and then the fine-grained rubric scores per-prompt are assigned by a supervising language model to guide the feedback for training. An example prompt to generate a rubric for a science task is shown below [318]: You are an expert rubric writer for science questions in the domains of Biology, Physics, and Chemistry. Your job is to generate a self-contained set of evaluation criteria ("rubrics") for judging how good a response is to a given question in one of these domains. Rubrics can cover aspects such as factual correctness, depth of reasoning, clarity, completeness, style, helpfulness, and common pitfalls.


ual correctness, depth of reasoning, clarity, completeness, style, helpfulness, and common pitfalls. Each rubric item must be fully self-contained so that non-expert readers need not consult any external information. Inputs: - question: The full question text. - reference_answer: The ideal answer, including any key facts or explanations. Total items: - Choose 7-20 rubric items based on question complexity. Each rubric item must include exactly three keys: 1. title (2-4 words) 2. description: One sentence beginning with its category prefix, explicitly stating what to look for. For example: - Essential Criteria: States that in the described closed system, the total mechanical energy (kinetic plus potential) rlhfbook.com 152


before the event equals the total mechanical energy after the event. - Important Criteria: Breaks down numerical energy values for each stage, demonstrating that initial kinetic energy plus initial potential energy equals final kinetic energy plus final potential energy. - Optional Criteria: Provides a concrete example, such as a pendulum converting between kinetic and potential energy, to illustrate how energy shifts within the system. - Pitfall Criteria: Does not mention that frictional or air-resistance losses are assumed negligible when applying conservation of mechanical energy. 3. weight: For Essential/Important/Optional, use 1-5 (5 = most important); for Pitfall, use -1 or -2. Category guidance: - Essential: Critical facts or safety checks; omission invalidates the response.


Category guidance: - Essential: Critical facts or safety checks; omission invalidates the response. - Important: Key reasoning or completeness; strongly affects quality. - Optional: Nice-to-have style or extra depth. - Pitfall: Common mistakes or omissions; highlight things often missed. Format notes: - When referring to answer choices, explicitly say "Identifies (A)", "Identifies (B)", etc. - If a clear conclusion is required (e.g. "The final answer is (B)"), include an Essential Criteria for it. - If reasoning should precede the final answer, include an Important Criteria to that effect. - If brevity is valued, include an Optional Criteria about conciseness. Output: Provide a JSON array of rubric objects. Each object must contain exactly three keys-title, description, and weight.


rray of rubric objects. Each object must contain exactly three keys-title, description, and weight. Do not copy large blocks of the question or reference_answer into the text. Each description must begin with its category prefix, and no extra keys are allowed. Now, given the question and reference_answer, generate the rubric as described. The reference answer is an ideal response but not necessarily exhaustive; use it only as guidance. Another, simpler example follows as [320]: SYSTEM: You generate evaluation rubrics for grading an assistant's response to a user prompt. Rubric design rules: - Each criterion must be atomic (one thing), objective as possible, and written so a grader can apply it consistently.


st be atomic (one thing), objective as possible, and written so a grader can apply it consistently. - Avoid redundant/overlapping criteria; prefer criteria that partition different failure modes. - Make criteria self-contained (don't rely on unstated context). - Include an importance weight for each criterion. Output format (JSON only): { "initial_reasoning": "<brief rationale for what matters for this prompt>", "rubrics": [ { "reasoning": "<why this criterion matters>", "criterion": "<clear, testable criterion>", rlhfbook.com 153


"weight": <integer 1-10> }, ... ] } USER: User prompt: {prompt} Generate the rubric JSON now. As you can see, the prompts can be very detailed and are tuned to the training setup. Rubrics with RL training are going to continue to evolve beyond their early applications to instruction following [322], deep research [323], evaluating deep research agents [324], or long-form generation [325]. rlhfbook.com 154


13 Tool Use and Function Calling Language models using tools is a natural way to expand their capabilities, especially for high-precision tasks where external tools contain the information or for agents that need to interact with complex web systems. Tool-use is a skill that language models need to be trained to have, and RLHF and all the other methods presented in this book can refine it. Consider a question from a user such as: User: Who is the president today? A language model without tools will have a hard time answering this question due to the knowledge cutoff of pretraining data, but this is readily accessible information with one search query. Consider another example: User: Move all the arXiv papers in my downloads folder to my ~/research/ directory with names indicating the date


the arXiv papers in my downloads folder to my ~/research/ directory with names indicating the date of the paper. This is a task that the model weights alone cannot even attempt – the use of tools enables language models to address a far broader range of tasks. Before diving deeper, it is useful to distinguish related terms that are often used interchangeably: • Tool use: the model emits a structured request (tool name and arguments); an orchestrator executes the tool; results are appended to the context; the model continues generating. • Function calling: tool use where the arguments must conform to a declared schema for a set of functions (usually JSON Schema), enabling reliable parsing and validation.


ared schema for a set of functions (usually JSON Schema), enabling reliable parsing and validation. • Code execution: a special case of tool use where the “tool” is a code interpreter (e.g., Python); results are returned as tool output. 13.1 Tool-Use Overview An AI model uses any external tools by outputting special tokens to trigger a certain endpoint. These can be anything from highly specific tools, such as functions that return the weather at a specific place, to code interpreters or search engines that act as fundamental building blocks of complex behaviors. Our first example showcased where language models need more up-to-date information to complement the fixed nature of their weights trained on past data, but there are also tools such as code execution, which lets language models g


trained on past data, but there are also tools such as code execution, which lets language models get around their probabilistic, generative nature and return precise answers. Consider the task of printing an approximation of pi to 50 digits (without reciting it from memory and risking hallucination). A language model with tools can do the following: <code> from decimal import Decimal, getcontext getcontext().prec = 60 def compute_pi(): # Chudnovsky algorithm for computing pi C = 426880 * Decimal(10005).sqrt() K, M, X, L, S = 0, 1, 1, 13591409, Decimal(13591409) for i in range(1, 100): rlhfbook.com 155


M = M * (K**3 - 16*K) // ((i)**3) K += 12 L += 545140134 X *= -262537412640768000 S += Decimal(M * L) / X return C / S print(str(compute_pi())[:52]) </code> <output> 3.14159265358979323846264338327950288419716939937510 </output> This chapter provides an overview of the origins of tool-use in modern language models, its fundamentals and formatting, and current trade-offs in utilizing tools well in leading models. The exact origin of the term “tool use” is not clear, but the origins of the idea far predate the post-ChatGPT world where RLHF proliferated. Early examples circa 2015 attempted to build systems predating modern language models, such as Neural Programmer-Interpreters (NPI) [326], “a recurrent and compositional neural network that learns to represent and execute programs.” As langua


recurrent and compositional neural network that learns to represent and execute programs.” As language models became more popular, many subfields were using integrations with external capabilities to boost performance. To obtain information outside of just the weights many used retrieval augmented generation [327] or web browsing [4]. Soon after, others were exploring language models integrated with programs [328] or tools [329]. As the field matured, these models gained more complex abilities in addition to the vast improvements to the underlying language modeling. For example, ToolFormer could use “a calculator, a Q&A system, two different search engines, a translation system, and a calendar” [330].


alculator, a Q&A system, two different search engines, a translation system, and a calendar” [330]. Soon after, Gorilla was trained to use 1645 APIs (from PyTorch Hub, TensorFlow Hub v2, and Hugging Face) and its evaluation APIBench became a foundation of the popular Berkeley Function Calling Leaderboard [331]. Since these early models, the diversity of actions called has grown substantially. Tool-use models are now deeply intertwined with regular language model interactions. Model Context Protocol (MCP) emerged as a common formatting used to connect language models to external data sources (or tools) [332]. With stronger models and better formats, tooluse language models are used in many situations, including productivity copilots within popular applications such as Microsoft Office or Go


tuations, including productivity copilots within popular applications such as Microsoft Office or Google Workspace, scientific domains [333], medical domains [334], coding agents [335] such as Claude Code or Cursor, integrations with databases, and many other autonomous workflows. Evaluating tool-use models involves multiple dimensions: exact-match metrics for tool name and argument correctness, schema validity, and end-to-end task completion in simulated environments. Reliability across trials also matters – τ-bench introduced the passˆk metric (distinct from pass@k) to measure whether an agent succeeds consistently rather than occasionally [336]. ToolLLM and its ToolBench dataset provide a large-scale framework for training and evaluating tool use across 16,000+ real-world APIs [337], wh


large-scale framework for training and evaluating tool use across 16,000+ real-world APIs [337], while the Berkeley Function Calling Leaderboard (BFCL) remains a popular benchmark for comparing models on function calling accuracy [331]. rlhfbook.com 156


13.2 Interweaving Tool Calls in Generation Training data for function calling looks much like other post-training data, with one addition: a system prompt that instructs the model what tools it has available. An example formatted data point with the system prompt and tools available in JSON format is shown below: <system> You are a function-calling AI model. You are provided with function signatures within <functions></functions> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. </system> <functions> [ { "name": "search_movies", "description": "Search for movies by title and return matching results with IDs.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description


h IDs.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The search string for the movie title." } }, "required": ["query"] } }, { "name": "get_movie_details", "description": "Fetch detailed information about a movie including cast, runtime, and synopsis.", "parameters": { "type": "object", "properties": { "movie_id": { "type": "string", "description": "The unique identifier for the movie." } }, "required": ["movie_id"] } }, { "name": "get_showtimes", "description": "Get movie showtimes for a given location and date.", "parameters": { "type": "object", "properties": { "movie_id": { "type": "string", "description": "The unique identifier for the movie." }, "zip_code": { "type": "string", "description": "ZIP code for theater location." }, "date"


movie." }, "zip_code": { "type": "string", "description": "ZIP code for theater location." }, "date": { "type": "string", "description": "Date for showtimes in YYYY-MM-DD format." rlhfbook.com 157


} }, "required": ["movie_id", "zip_code"] } } ] </functions> <user> ... </user> Although the language model is generating a completion, if it is following this example, it would generate the tokens search_movies("Star Wars") to search for Star Wars. This is often encoded inside special formatting tokens, and then the next tokens inserted into the sequence will contain the tool outputs. With this, models can learn to accomplish more challenging tasks than many simple standalone models. A popular form of tool use is code-execution, allowing the model to get precise answers to complex logic or mathematics problems. For example, code-execution within a language model execution can occur during the thinking tokens of a reasoning model.


cution within a language model execution can occur during the thinking tokens of a reasoning model. As with function calling, there are tags first for the code to execute (generated by the model) and then a separate tag for output. <|user|> What is the 50th Fibonacci number? (Use the standard F_0=0, F_1=1 indexing.)</s> <|assistant|> <think> Okay, I will compute the 50th Fibonacci number with a simple loop, then return the result. <code> def fib(n): a, b = 0, 1 for _ in range(n): a, b = b, a + b return a fib(50) </code> <output> 12586269025 </output> </think> <answer> The 50th Fibonacci number is 12 586 269 025. </answer> What is happening under the hood is the language model is interleaving tool inputs and outputs with standard autoregressively generated tokens.


uage model is interleaving tool inputs and outputs with standard autoregressively generated tokens. The orchestration loop that makes this possible looks something like: rlhfbook.com 158


messages = [...] while True: response = model(messages, tools=tools) if not response.tool_calls: return response.text for call in response.tool_calls: result = execute_tool(call.name, call.args) messages.append({"role": "tool", "tool_call_id": call.id, "content": result}) Figure 37: Tool use interleaves model generation with external execution: the model generates tokens until it emits a tool call (orange), an external system executes the tool and injects the output (purple) into the sequence, then the model continues generating. Models can emit multiple tool calls in a single generation. During training, tool call and output tokens are typically masked from the loss. Training for tool use is about getting the model to behave predictably with this different token flow—knowing when to emit


s about getting the model to behave predictably with this different token flow—knowing when to emit a tool call, how to format arguments correctly, and how to incorporate results into its response. Open models must be trained to work with a variety of tools that users may connect off the shelf. 13.3 Multistep Tool Reasoning OpenAI’s o3 model represented a substantial step-change in how multi-step tool-use can be integrated with language models. This behavior is related to much earlier research trends in the community. For example, ReAct [338] showcased how actions and reasoning can be interleaved into one model generation: In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the