he term distillation is now overloaded with a specific technical definition from the ML literature. • In-context Learning (ICL): In-context here refers to any information within the context window of the language model. Usually, this is information added to the prompt. The simplest form of in-context learning is adding examples of a similar form before the prompt. Advanced versions can learn which information to include for a specific use-case. • (Teacher-student) Knowledge Distillation: Knowledge distillation from a specific teacher to a student model is a specific type of distillation described above, and it is where the term originated. It is a specific deep learning method where a neural network loss is modified to learn from the log-probabilities of the teacher model over multiple pot


network loss is modified to learn from the log-probabilities of the teacher model over multiple potential tokens/logits, instead of learning directly from a chosen output [276]. An example of a modern series of models trained with Knowledge Distillation is Gemma 2 [403] or Gemma 3. For a language modeling setup, the next-token loss function can be modified as follows [287], where the student model Pθ learns from the teacher distribution Pϕ: LKD(θ) = −Ex∼D " T X t=1 Pϕ(xt | x<t) log Pθ(xt | x<t) # . (157) • Synthetic Data: This is any training data for an AI model that is the output from another AI system. This could be anything from text generated from an open-ended prompt of a model to a model rewriting existing content. rlhfbook.com 225


B Beyond “Just Style” Early developments in RLHF gave it a reputation for being “just style transfer”; other harsh critiques focused on how RLHF manipulates the way information is presented in outputs. This appendix explains why style is core to understanding the value RLHF provides — and why it positively affects both model capability and user experience. The idea of RLHF being solely about style transfer has held back the RLHF narrative for two reasons. The first is how RLHF became associated with small, unimportant changes to the model. When people discuss style transfer, they don’t describe this as being important or exciting – they think of it as superficial. Yet, style is a never-ending source of human value; it’s why retelling stories can result in new bestselling books (such as Sap


g source of human value; it’s why retelling stories can result in new bestselling books (such as Sapiens), and it is a fundamental part of continuing to progress our intellectual ecosystem. Style is intertwined with what the information is. The second reason is that many people missed the fact that well-done RLHF boosts scores on popular LLM evaluations. We’ve seen how different styles actually can meaningfully improve evaluations with Llama 3 [29]. The Llama 3 Instruct models scored extremely high on Arena, and it’s accepted as being because they had a more fun personality – they were more succinct and clever than other models of their era. Regardless of the benchmark scores that many LLM users are obsessed with, if RLHF is going to make language models simply more fun, that is delivered


sers are obsessed with, if RLHF is going to make language models simply more fun, that is delivered value. Throughout this appendix, the term “chattiness” is used to encompass the growing length of responses from models trained with RLHF, but it also encompasses techniques like heavy markdown use, emojis, and formatting the answer in bulleted lists. To ground the discussion in this appendix, an example completion for two models in the same Tülu 3 family are shown below – one with only instruction fine-tuning and the second with preference fine-tuning applied via length-controlled DPO. First, the instruction-tuned completion to the prompt What is reinforcement learning from human feedback? (the exact model is Llama 3.1 Tulu 3 70B SFT) [6]: Reinforcement learning from human feedback, also kn


he exact model is Llama 3.1 Tulu 3 70B SFT) [6]: Reinforcement learning from human feedback, also known as human-in-the-loop reinforcement learning, is a subfield of machine learning where the primary goal is to train reinforcement learning agents to perform tasks by incorporating feedback from humans. This feedback can come in various forms, such as binary rewards (e.g., right or wrong), scalar rewards, preferences between actions, demonstrations, natural language instructions, or corrections to the agent's behavior. The idea behind reinforcement learning from human feedback is to leverage human intuition and decision-making abilities to guide the learning process of the agent. In traditional reinforcement learning, an agent learns to make decisions by interacting with an environment and


al reinforcement learning, an agent learns to make decisions by interacting with an environment and receiving rewards or punishments based on the outcomes of those interactions. However, defining a suitable reward function can be challenging and time-consuming, often leading to reward shaping or reward hacking, where the agent finds unexpected and often unintended ways to maximize the reward. By using human feedback, reinforcement learning can address some of these issues by allowing humans to directly influence the learning process. Human trainers can provide real-time feedback, scale their feedback to match the complexity of the task, and update their feedback based on the agent's progress.


dback to match the complexity of the task, and update their feedback based on the agent's progress. This approach can lead to more robust and generalized agents, as they are guided not by handcrafted reward functions but by the nuanced understanding and experience of human trainers. rlhfbook.com 226


However, reinforcement learning from human feedback faces several challenges, such as the high cost of obtaining human feedback, subjective and inconsistent feedback, scaling issues, and methods for efficient and effective integration of human feedback into learning algorithms. Despite these challenges, the research in this area has shown promising results in various applications, including robotics, video games, prosthetics control, natural language processing, and many more. And the post-RLHF completion from Llama 3.1 Tulu 3 70B DPO: Reinforcement learning from human feedback (RLHF) is a subset of reinforcement learning (RL) where the rewards that guide the learning process are provided by humans, rather than a pre-specified numerical reward function.


the learning process are provided by humans, rather than a pre-specified numerical reward function. This is particularly useful in situations where it is hard or infeasible to design a suitable reward function, or where the desired performance involves subjective judgments that humans are uniquely equipped to provide. In traditional reinforcement learning, an agent interacts with an environment. With each interaction, the agent performs an action, receives an observation, and gets a numerical reward that tells it how well it performed in that situation. Through many such interactions, the agent learns to maximize the cumulative reward. In contrast, in reinforcement learning from human feedback: 1.


s to maximize the cumulative reward. In contrast, in reinforcement learning from human feedback: 1. **Human Input:** Instead of a predefined reward function, the feedback comes from human evaluators, who provide qualitative, subjective judgments about the agent's performance. This feedback can be in various forms, such as numerical ratings, binary preference signals, natural language feedback, or even demonstrations showing how to improve. 2. **Feedback Loop:** Human evaluators observe the agent's actions or outcomes and provide feedback in real-time (or close to real-time). The agent learns through this iterative feedback loop, gradually adapting its behavior to align more closely with human preferences. 3.


ive feedback loop, gradually adapting its behavior to align more closely with human preferences. 3. **Reward Shaping:** Because raw human feedback is not always directly usable in learning algorithms due to variability and potential bias, a reward shaping process may be involved, whereby human feedback is converted into a more stable signal that guides learning. 4. **Preference-based RL:** One common approach within RLHF is preference-based reinforcement learning, where the agent is trained by comparing pairs of actions (or outcomes) and receiving human feedback on which one is better. From these pairwise comparisons, the agent infers a reward function that it can use to guide its actions. 5.


e pairwise comparisons, the agent infers a reward function that it can use to guide its actions. 5. **Inverse Reinforcement Learning (IRL):** Another technique used in RLHF is IRL, which aims to infer the underlying reward function that could explain human actions or decisions, and then use that reward function to train the agent. **Applications:** Reinforcement Learning from human feedback has found applications in: - **Robotics:** Teaching robots to perform tasks that are difficult to specify with predefined reward functions. - **Game playing:** Human feedback can guide AI agents to better emulate human-like strategies or behaviors. rlhfbook.com 227


- **Language and content generation:** For example, suggestively guiding AI models to generate more coherent, creative, or helpful content. - **Recommendation systems:** Human feedback can help adjust and refine the recommendations provided to users. **Challenges:** Using human feedback introduces some specific challenges: 1. **Scalability:** Human feedback is time-consuming and expensive to collect, and it's hard to scale to complex or high-dimensional tasks. 2. **Bias and Noise:** Human feedback can be noisy, inconsistent, and biased, leading to suboptimal learning outcomes. 3. **Alignment:** Ensuring that the reinforcement learning agent aligns with the true preferences of the human evaluator(s) and not just the superficial feedback is difficult. 4.


e true preferences of the human evaluator(s) and not just the superficial feedback is difficult. 4. **Safety and Misuse:** There can be concerns about the safety and ethical implications of using potentially flawed human feedback, especially if the feedback process is corrupted or gamed. Researchers are actively working on techniques to mitigate these challenges and improve RLHF approaches. One such approach is leveraging machine learning models to amplify the efficacy of the human feedback, for example, by learning to generalize from sparse human input or by synthesizing plausible feedback in situations where direct human input is not available. The response from the DPO model has similar information, as both models are trained on the same pretrained base model, but the preference fine-tu


nformation, as both models are trained on the same pretrained base model, but the preference fine-tuning adds more structure, formatting, and of course length, which has been shown repeatedly to be associated with general notions of human preference for language models. For more examples like this, see the completions library associated with this book at https://www.rlhfbook.com/library. B.1 The Chattiness Balance RLHF or preference fine-tuning methods are being used by countless people to boost scores like AlpacaEval and other automatic chat leaderboards (which use LLM-as-a-judge to approximate how helpful, harmless, and honest an agent is across simple conversational tasks), but the massive gains RLHF confers here come without shifting scores proportionally on harder-to-game evaluations


e gains RLHF confers here come without shifting scores proportionally on harder-to-game evaluations like Arena. The tension is that while RLHF methods give a measurable improvement on these models, that training doesn’t always transfer into performance that people care about. Through the establishment of the RLHF literature, a large swath of models have been released with related methods to boost the “alignment” of a model with RLHF, but they often took it way too far and published evaluation scores that were anywhere from misleading to meaningless. These RLHF methods motivated by alignment, when done right, make the models easier to work with and more enjoyable. This often comes with clear improvements on evaluation tools like MT Bench or AlpacaEval.


njoyable. This often comes with clear improvements on evaluation tools like MT Bench or AlpacaEval. In the fall of 2023, there was a peak in the debate over direct preference optimization (DPO) and its role relative to proximal policy optimization (PPO) and other RL-based methods for preference fine-tuning – the balance of chat evaluations to real-world performance was at the center of this (For more technical discussion on the trade-offs, see Chapter 8, Ivison et al. 2024 [125], or this talk). The problem is that you can also use techniques like DPO and rlhfbook.com 228


PPO in feedback loops or in an abundance of data to actually severely harm the model on other tasks like mathematics or coding in a trade for this chat performance. During the proliferation of the DPO versus PPO debate there were many papers that came out with incredible benchmarks but no meaningful adoption. If these papers released model weights, they weren’t popular in public usage because these models were not robust in general usage. When applying RLHF in the fall of 2023 or soon after, there is no way to make an aligned version of a 7 billion parameter model actually beat GPT-4 across comprehensive benchmarks (this sort of comparison will hold, where small models of the day cannot robustly beat the best, large frontier models).


son will hold, where small models of the day cannot robustly beat the best, large frontier models). It seems obvious, but there are always papers claiming these sorts of results. fig. 45 is from a paper called Direct Nash Optimization (DNO), which makes the case that their model is state-of-the-art or so on AlpacaEval for 7B models in April 2024 [201]. For context, DNO is a batched, on-policy iterative alternative to reward-model+PPO (classic RLHF) or one-shot DPO that directly optimizes pairwise preferences (win-rate gaps) by framing alignment as finding a Nash equilibrium against a preference oracle. These challenges emerge when academic incentives interface with technologies becoming of extreme interest to the broader society.


cademic incentives interface with technologies becoming of extreme interest to the broader society. Figure 45: Results from the paper on Direct Nash Optimization (DNO) highlighting their small model outperforming the likes of GPT-4. Rosset et al. 2024. License CC-BY. Even the pioneering paper Self Rewarding Language Models from January of 2024 [294] disclosed unrealistically strong scores on Llama 2 70B. At the time, of course, a 70B model can get closer to GPT-4 than a 7B model can (as we saw with the impressive Llama 3 releases in 2024), but it’s important to separate the reality of models from the claims in modern RLHF papers. These models are tuned to narrow test sets and do not hold up well in real use versus the far larger models they claim to beat.


rrow test sets and do not hold up well in real use versus the far larger models they claim to beat. Many more methods have come and gone similar to this, sharing valuable insights and oversold results, which make RLHF harder to understand. A symptom of models that have “funky RLHF” applied to them has often been a length bias. This got so common that multiple evaluation systems like AlpacaEval and WildBench both have linear length correction mechanisms in them. This patches the incentives for doping on chattiness to ‘beat GPT-4’ or the leading frontier model of the day, and creates a less gamified dynamic where shorter, useful models can actually win. rlhfbook.com 229


Regardless, aligning chat models only for chattiness now has a bit of a reputational tax associated with it in the literature, where it’s acknowledged that these narrow methods can harm a model in other ways. This note from the original Alibaba Qwen models in 2023 is something that has been observed multiple times in early alignment experiments, exaggerating a trade-off between chattiness and performance [404]. We pretrained the models with a large amount of data, and we post-trained the models with both supervised fine-tuning and direct preference optimization. However, DPO leads to improvements in human preference evaluation but degradation in benchmark evaluation. An early, good example of this tradeoff done right is a model like Starling Beta from March of 2024 [73].


ly, good example of this tradeoff done right is a model like Starling Beta from March of 2024 [73]. It’s a model that was fine-tuned from another chat model, OpenChat [405] (which was in fact trained by an entire other organization). Its training entirely focuses on k-wise reward model training and PPO optimization, and moves it up 10 places in Arena. The average response length of the model increases, but in a way that’s good enough to actually help the human raters. Later examples, such as Olmo 3, are documented as undergoing substantial chat training, but with the authors preferring a final model checkpoint with higher math, coding, and reasoning scores instead of potential checkpoints that are highest on LLM-as-a-judge-based chat benchmarks [18].


res instead of potential checkpoints that are highest on LLM-as-a-judge-based chat benchmarks [18]. A natural question is: Why does RLHF make model responses longer? Fundamentally, evaluations like Arena have shown us that average users of models often like longer, complete answers when compared with terse responses. Longer answers can feel more thorough, helpful, or even trustworthy to users evaluating them quickly. This does not represent the preference of every user, but these models are trained to match the average preferences of many data labelers, so RLHF tends to make models more verbose. rlhfbook.com 230


C Practical Issues This appendix covers practical considerations for running post-training experiments at scale. This takes the form of a list of lessons, rather than a coherent narrative. C.1 Compute Costs of Post-Training There are two different ways of scoping costs for post-training runs. The largest cost is in developing the recipe, which can easily be 10X to 100X the compute of the final few training runs. The secondary costs, which are easier to measure, are the costs of thoroughly applying a recipe, which entails multiple seeds, careful evaluation, potential engineering headaches, etc. For the first cost, to develop a post-training recipe like Tülu 3 [6], the team ran on the order of thousands of experiments/evaluations at the 7B scale before having the final model.


on the order of thousands of experiments/evaluations at the 7B scale before having the final model. For final runs, the Olmo 3 report has a detailed accounting of what is involved in training the final 32B Think model [18]: Post-training follows a different operational pattern in which we run each stage multiple times, sweeping over learning rates and other hyperparameters. The theory for post-training, particularly, RL, is less developed, so we have to run multiple experiments to identify the optimal hyperparameters for a given base model. We hope to address this in future work. During post-training, checkpoint evaluation consumes a larger proportion of compute resources, in part due to long generations from reasoning models on core benchmarks.


ion of compute resources, in part due to long generations from reasoning models on core benchmarks. For SFT, we swept over four candidate learning rates, on 256 GPUs each, in parallel for 36 hours. Then approximately 12 hours was spent on evaluation, merging, and checkpoint confirmation, totaling approximately two days. DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues. After the initial release of Olmo 3, we continued our best RL run for another 21 days on 224 GPUs to produce Olmo 3.1 Think 32B.


Olmo 3, we continued our best RL run for another 21 days on 224 GPUs to produce Olmo 3.1 Think 32B. As scaling reinforcement learning becomes more standard practice, this will shift yet again [17]. Continuing the above example, where the original Olmo 3 32B Think post-training took only a couple of weeks, to release the improved Olmo 3.1 32B Think model the team needed to train it for an additional 3.5 weeks with RLVR. This is a substantial cost in time more than in total compute. C.2 Evaluation Variance One underappreciated challenge in post-training is evaluation variance, especially with the rise of reasoning models that need to use sampling with temperatures above 0 to get the best evaluation scores. With any sampling from models, the outputs become more variable.


to get the best evaluation scores. With any sampling from models, the outputs become more variable. Different benchmarks have vastly different stability characteristics, due to the variance in difficulty of the prompts, the number of prompts in the evaluation set, the brittleness of the models being trained, etc. rlhfbook.com 231


During Olmo 3, the team tracked the variance of different evaluations used to evaluate reasoning models. The table below shows the standard deviation of each evaluation, computed as the mean of the standard deviation from 3 runs of 14 models (take the variance of each model, then average per evaluation): Table 9: Standard deviation of evaluation benchmarks across multiple inference runs, categorized by stability (data from Olmo 3). Category Benchmark Std. Dev. High Variance GPQA 1.48 AlpacaEval 3 1.24 IFEval 0.88 Stable ZebraLogic 0.56 Omega 0.56 AIME 24 (Avg@32) 0.54 HumanEvalPlus 0.46 AgiEval 0.43 BigBenchHard 0.39 Very Stable LiveCodeBench (Avg@10) 0.29 MBPPPlus 0.27 MATH 0.25 MMLU 0.22 PopQA 0.16 Some evaluations, such as LiveCodeBench, were both noisy and cheap (via few prompts in the


opQA 0.16 Some evaluations, such as LiveCodeBench, were both noisy and cheap (via few prompts in the set), so by re-running the evaluation 10 times per model, the evaluation could move from the high-variance set to a stable setting. This could be done for every evaluation, but it can easily balloon costs. We also see sources of variance in evaluation settings like batch size, tensor parallel settings within vLLM (e.g., TP=2 for baselines), and other sensitive numerics for sampling long generations across infrastructure. Variance is everywhere with reasoners. C.3 Managing Training Performance Variance Throughout all the post-training recipes and tools discussed in this book, the final model is subject to meaningful variance in performance.


and tools discussed in this book, the final model is subject to meaningful variance in performance. Understanding the distribution of this variance, its sources, and its effects is crucial to creating strong models. The goal of training a final model is to sample many points, by varying training parameters and random seeds, in order to get the strongest model possible. Note that this is a balance between the model actually being better, and not just the benefit of re-rolling from evaluation noise. Where the previous section focuses on evaluation noise, the trickier source of noise is training uncertainty. Where evaluation noise can be managed by running more tests on a given checkpoint (uniformly reducing noise), models are trained once and can benefit from a positive outlier.


kpoint (uniformly reducing noise), models are trained once and can benefit from a positive outlier. In practice, training teams take many steps to capture the maximum possible value out of their training recipe: rlhfbook.com 232


1. Sweep core optimization values like learning rate, batch size, etc. for every final model run. For example, with a new base model, I’d recommend running 10 learning rates over a wide region to be sure you’re in the optimal range, then re-run in the tighter, optimal window. 2. Run multiple seeds on the best few settings. Random seed can have meaningful effects on the final model, and it’s worth spending compute on. 3. Model merging is established as a key tool used to create strong models. Merging can be done in many ways, from merging different checkpoints on the same data to merging specialized models for specific domains. Generally, merging is seen as a strong and simple tool in final recipes, but clear best practices aren’t established for preparing a model for later merging in a rec


ecipes, but clear best practices aren’t established for preparing a model for later merging in a recipe [406]. C.4 Identifying Bad Training Jobs A simple intuition that’s important to establish when training models is the different types of model issues. You want most of your time to be spent on issues where the current data, algorithm, or recipe just isn’t good enough. On the other hand, there are plenty of times when, while setting up a new recipe, certain methods are just broken. The best way to understand this is to evaluate many models on a largely static evaluation suite. Then you develop an intuition for which tests are hard to move with post-training interventions (often knowledge-heavy evaluations such as MMLU).


are hard to move with post-training interventions (often knowledge-heavy evaluations such as MMLU). When something is very, very broken in a post-training setup, these largely stable evaluations can often drop by 10-20 points in a training job. This is one of the most useful signals there are when developing tooling! rlhfbook.com 233


ii


Adaptive Computation and Machine Learning Francis Bach, series editor A complete list of books published in the Adaptive Computation and Machine Learning series appears at the back of this book.


Reinforcement Learning: An Introduction second edition Richard S. Sutton and Andrew G. Barto The MIT Press Cambridge, Massachusetts London, England


© 2018, 2020 Richard S. Sutton and Andrew G. Barto All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the copyright holder. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/2.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. This book was set in 10/12, CMR by Westchester Publishing Services. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Names: Sutton, Richard S., author. | Barto, Andrew G., author.


gress Cataloging-in-Publication Data Names: Sutton, Richard S., author. | Barto, Andrew G., author. Title: Reinforcement learning : an introduction / Richard S. Sutton and Andrew G. Barto. Description: Second edition. | Cambridge, MA : The MIT Press, [2018] | Series: Adaptive computation and machine learning series | Includes bibliographical references and index. Identiﬁers: LCCN 2018023826 | ISBN 9780262039246 (hardcover : alk. paper) Subjects: LCSH: Reinforcement learning. Classiﬁcation: LCC Q325.6 .R45 2018 | DDC 006.3/1--dc23 LC record available at https://lccn.loc.gov/2018023826 10 9 8 7 6 5 4 3 2 1


In memory of A. Harry Klopf


Contents Preface to the Second Edition xiii Preface to the First Edition xvii Summary of Notation xix 1 Introduction 1 1.1 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Elements of Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . 6 1.4 Limitations and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5 An Extended Example: Tic-Tac-Toe . . . . . . . . . . . . . . . . . . . . . 8 1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.7 Early History of Reinforcement Learning . . . . . . . . . . . . . . . . . . . 13 I Tabular Solution Methods 23 2 Multi-armed Bandits 25 2.1 A k-armed Bandit Problem .


. . . . . 13 I Tabular Solution Methods 23 2 Multi-armed Bandits 25 2.1 A k-armed Bandit Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.2 Action-value Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.3 The 10-armed Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.4 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.5 Tracking a Nonstationary Problem . . . . . . . . . . . . . . . . . . . . . . 32 2.6 Optimistic Initial Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.7 Upper-Conﬁdence-Bound Action Selection . . . . . . . . . . . . . . . . . . 35 2.8 Gradient Bandit Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.9 Associative Search (Contextual Bandits) . .


. . . . . . . . . . . . . . . . . . . . . . . . 37 2.9 Associative Search (Contextual Bandits) . . . . . . . . . . . . . . . . . . . 41 2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42


viii Contents 3 Finite Markov Decision Processes 47 3.1 The Agent–Environment Interface . . . . . . . . . . . . . . . . . . . . . . 47 3.2 Goals and Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.3 Returns and Episodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.4 Uniﬁed Notation for Episodic and Continuing Tasks . . . . . . . . . . . . 57 3.5 Policies and Value Functions . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.6 Optimal Policies and Optimal Value Functions . . . . . . . . . . . . . . . 62 3.7 Optimality and Approximation . . . . . . . . . . . . . . . . . . . . . . . . 67 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4 Dynamic Programming 73 4.1 Policy Evaluation (Prediction) . . . . . .


. . . . . . . . . . . . 68 4 Dynamic Programming 73 4.1 Policy Evaluation (Prediction) . . . . . . . . . . . . . . . . . . . . . . . . 74 4.2 Policy Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.3 Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.4 Value Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.5 Asynchronous Dynamic Programming . . . . . . . . . . . . . . . . . . . . 85 4.6 Generalized Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.7 Eﬃciency of Dynamic Programming . . . . . . . . . . . . . . . . . . . . . 87 4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5 Monte Carlo Methods 91 5.1 Monte Carlo Prediction . . . .


. . . . . . . . . . . . . . . . . . 88 5 Monte Carlo Methods 91 5.1 Monte Carlo Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.2 Monte Carlo Estimation of Action Values . . . . . . . . . . . . . . . . . . 96 5.3 Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.4 Monte Carlo Control without Exploring Starts . . . . . . . . . . . . . . . 100 5.5 O↵-policy Prediction via Importance Sampling . . . . . . . . . . . . . . . 103 5.6 Incremental Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5.7 O↵-policy Monte Carlo Control . . . . . . . . . . . . . . . . . . . . . . . . 110 5.8 *Discounting-aware Importance Sampling . . . . . . . . . . . . . . . . . . 112 5.9 *Per-decision Importance Sampling . . . . . . . . .


ing . . . . . . . . . . . . . . . . . . 112 5.9 *Per-decision Importance Sampling . . . . . . . . . . . . . . . . . . . . . . 114 5.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 6 Temporal-Di↵erence Learning 119 6.1 TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.2 Advantages of TD Prediction Methods . . . . . . . . . . . . . . . . . . . . 124 6.3 Optimality of TD(0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.4 Sarsa: On-policy TD Control . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.5 Q-learning: O↵-policy TD Control . . . . . . . . . . . . . . . . . . . . . . 131 6.6 Expected Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . 131 6.6 Expected Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.7 Maximization Bias and Double Learning . . . . . . . . . . . . . . . . . . . 134 6.8 Games, Afterstates, and Other Special Cases . . . . . . . . . . . . . . . . 136 6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138


Contents ix 7 n-step Bootstrapping 141 7.1 n-step TD Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 7.2 n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 7.3 n-step O↵-policy Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 7.4 *Per-decision Methods with Control Variates . . . . . . . . . . . . . . . . 150 7.5 O↵-policy Learning Without Importance Sampling: The n-step Tree Backup Algorithm . . . . . . . . . . . . . . . . . . . . . . 152 7.6 *A Unifying Algorithm: n-step Q(σ) . . . . . . . . . . . . . . . . . . . . . 154 7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 8 Planning and Learning with Tabular Methods 159 8.1 Models and Planning . . . . . . . . . . . . . . . .


nning and Learning with Tabular Methods 159 8.1 Models and Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 8.2 Dyna: Integrated Planning, Acting, and Learning . . . . . . . . . . . . . . 161 8.3 When the Model Is Wrong . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 8.4 Prioritized Sweeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 8.5 Expected vs. Sample Updates . . . . . . . . . . . . . . . . . . . . . . . . . 172 8.6 Trajectory Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 8.7 Real-time Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . . 177 8.8 Planning at Decision Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 8.9 Heuristic Search . . . . . . . . . . . . . . . . . . . . .


. . . . . . . . . . . . . . . . 180 8.9 Heuristic Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 8.10 Rollout Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 8.11 Monte Carlo Tree Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 8.12 Summary of the Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 8.13 Summary of Part I: Dimensions . . . . . . . . . . . . . . . . . . . . . . . . 189 II Approximate Solution Methods 195 9 On-policy Prediction with Approximation 197 9.1 Value-function Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 198 9.2 The Prediction Objective (VE) . . . . . . . . . . . . . . . . . . . . . . . . 199 9.3 Stochastic-gradient and Semi-gradient Methods . . . . . . .


. . . . . . . . . . . . . . . . 199 9.3 Stochastic-gradient and Semi-gradient Methods . . . . . . . . . . . . . . . 200 9.4 Linear Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 9.5 Feature Construction for Linear Methods . . . . . . . . . . . . . . . . . . 210 9.5.1 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 9.5.2 Fourier Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 9.5.3 Coarse Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 9.5.4 Tile Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 9.5.5 Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . . 221 9.6 Selecting Step-Size Parameters Manually . . . . . . . . . . . . . . . . . . .


. . . . . . . 221 9.6 Selecting Step-Size Parameters Manually . . . . . . . . . . . . . . . . . . . 222 9.7 Nonlinear Function Approximation: Artiﬁcial Neural Networks . . . . . . 223 9.8 Least-Squares TD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228


x Contents 9.9 Memory-based Function Approximation . . . . . . . . . . . . . . . . . . . 230 9.10 Kernel-based Function Approximation . . . . . . . . . . . . . . . . . . . . 232 9.11 Looking Deeper at On-policy Learning: Interest and Emphasis . . . . . . 234 9.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 10 On-policy Control with Approximation 243 10.1 Episodic Semi-gradient Control . . . . . . . . . . . . . . . . . . . . . . . . 243 10.2 Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . . . . . . . 247 10.3 Average Reward: A New Problem Setting for Continuing Tasks . . . . . . 249 10.4 Deprecating the Discounted Setting . . . . . . . . . . . . . . . . . . . . . . 253 10.5 Di↵erential Semi-gradient n-step Sarsa . . . . . . . . . .


. . . . . . . . . . . . . . . . 253 10.5 Di↵erential Semi-gradient n-step Sarsa . . . . . . . . . . . . . . . . . . . . 255 10.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 11 *O↵-policy Methods with Approximation 257 11.1 Semi-gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 11.2 Examples of O↵-policy Divergence . . . . . . . . . . . . . . . . . . . . . . 260 11.3 The Deadly Triad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 11.4 Linear Value-function Geometry . . . . . . . . . . . . . . . . . . . . . . . 266 11.5 Gradient Descent in the Bellman Error . . . . . . . . . . . . . . . . . . . . 269 11.6 The Bellman Error is Not Learnable . . . . . . . . . . . . . . . . . . . . .


. . . . . . . 269 11.6 The Bellman Error is Not Learnable . . . . . . . . . . . . . . . . . . . . . 274 11.7 Gradient-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 11.8 Emphatic-TD Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 11.9 Reducing Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 11.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 12 Eligibility Traces 287 12.1 The λ-return . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 12.2 TD(λ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 12.3 n-step Truncated λ-return Methods . . . . . . . . . . . . . . . . . . . . . 295 12.4 Redoing Updates: Online λ-return Algorithm .


ods . . . . . . . . . . . . . . . . . . . . . 295 12.4 Redoing Updates: Online λ-return Algorithm . . . . . . . . . . . . . . . . 297 12.5 True Online TD(λ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 12.6 *Dutch Traces in Monte Carlo Learning . . . . . . . . . . . . . . . . . . . 301 12.7 Sarsa(λ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 12.8 Variable λ and γ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 12.9 O↵-policy Traces with Control Variates . . . . . . . . . . . . . . . . . . . 309 12.10 Watkins’s Q(λ) to Tree-Backup(λ) . . . . . . . . . . . . . . . . . . . . . . 312 12.11 Stable O↵-policy Methods with Traces . . . . . . . . . . . . . . . . . . . 314 12.12 Implementation Issues . . . . . . . . . . . .


races . . . . . . . . . . . . . . . . . . . 314 12.12 Implementation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 12.13 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317


Contents xi 13 Policy Gradient Methods 321 13.1 Policy Approximation and its Advantages . . . . . . . . . . . . . . . . . . 322 13.2 The Policy Gradient Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 324 13.3 REINFORCE: Monte Carlo Policy Gradient . . . . . . . . . . . . . . . . . 326 13.4 REINFORCE with Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . 329 13.5 Actor–Critic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 13.6 Policy Gradient for Continuing Problems . . . . . . . . . . . . . . . . . . 333 13.7 Policy Parameterization for Continuous Actions . . . . . . . . . . . . . . . 335 13.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . 335 13.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 III Looking Deeper 339 14 Psychology 341 14.1 Prediction and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 14.2 Classical Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 14.2.1 Blocking and Higher-order Conditioning . . . . . . . . . . . . . . . 345 14.2.2 The Rescorla–Wagner Model . . . . . . . . . . . . . . . . . . . . . 346 14.2.3 The TD Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 14.2.4 TD Model Simulations . . . . . . . . . . . . . . . . . . . . . . . . . 350 14.3 Instrumental Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 14.4 Delayed Reinforcement . . . . . . . . . . . . . . . . . .


. . . . . . . . . . . . . . . . 357 14.4 Delayed Reinforcement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 14.5 Cognitive Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 14.6 Habitual and Goal-directed Behavior . . . . . . . . . . . . . . . . . . . . . 364 14.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 15 Neuroscience 377 15.1 Neuroscience Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378 15.2 Reward Signals, Reinforcement Signals, Values, and Prediction Errors . . 380 15.3 The Reward Prediction Error Hypothesis . . . . . . . . . . . . . . . . . . 381 15.4 Dopamine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . 381 15.4 Dopamine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 15.5 Experimental Support for the Reward Prediction Error Hypothesis . . . . 387 15.6 TD Error/Dopamine Correspondence . . . . . . . . . . . . . . . . . . . . . 390 15.7 Neural Actor–Critic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 15.8 Actor and Critic Learning Rules . . . . . . . . . . . . . . . . . . . . . . . 398 15.9 Hedonistic Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 15.10 Collective Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . 404 15.11 Model-based Methods in the Brain . . . . . . . . . . . . . . . . . . . . . . 407 15.12 Addiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . 407 15.12 Addiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 15.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410


xii Contents 16 Applications and Case Studies 421 16.1 TD-Gammon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421 16.2 Samuel’s Checkers Player . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 16.3 Watson’s Daily-Double Wagering . . . . . . . . . . . . . . . . . . . . . . . 429 16.4 Optimizing Memory Control . . . . . . . . . . . . . . . . . . . . . . . . . . 432 16.5 Human-level Video Game Play . . . . . . . . . . . . . . . . . . . . . . . . 436 16.6 Mastering the Game of Go . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 16.6.1 AlphaGo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444 16.6.2 AlphaGo Zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 16.7 Personalized Web Services . . . . . . . . . . . . . . .


. . . . . . . . . . . . . . . . . 447 16.7 Personalized Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 16.8 Thermal Soaring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 17 Frontiers 459 17.1 General Value Functions and Auxiliary Tasks . . . . . . . . . . . . . . . . 459 17.2 Temporal Abstraction via Options . . . . . . . . . . . . . . . . . . . . . . 461 17.3 Observations and State . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464 17.4 Designing Reward Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 17.5 Remaining Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472 17.6 Reinforcement Learning and the Future of Artiﬁcial Intelligence . . . . . . 475 References 481 Index 519


Preface to the Second Edition The twenty years since the publication of the ﬁrst edition of this book have seen tremendous progress in artiﬁcial intelligence, propelled in large part by advances in machine learning, including advances in reinforcement learning. Although the impressive computational power that became available is responsible for some of these advances, new developments in theory and algorithms have been driving forces as well. In the face of this progress, a second edition of our 1998 book was long overdue, and we ﬁnally began the project in 2012. Our goal for the second edition was the same as our goal for the ﬁrst: to provide a clear and simple account of the key ideas and algorithms of reinforcement learning that is accessible to readers in all the related disciplines.


algorithms of reinforcement learning that is accessible to readers in all the related disciplines. The edition remains an introduction, and we retain a focus on core, online learning algorithms. This edition includes some new topics that rose to importance over the intervening years, and we expanded coverage of topics that we now understand better. But we made no attempt to provide comprehensive coverage of the ﬁeld, which has exploded in many di↵erent directions. We apologize for having to leave out all but a handful of these contributions. As in the ﬁrst edition, we chose not to produce a rigorous formal treatment of reinforcement learning, or to formulate it in the most general terms. However, our deeper understanding of some topics since the ﬁrst edition required a bit more mathematic


wever, our deeper understanding of some topics since the ﬁrst edition required a bit more mathematics to explain; we have set o↵the more mathematical parts in shaded boxes that the nonmathematically-inclined may choose to skip. We also use a slightly di↵erent notation than was used in the ﬁrst edition. In teaching, we have found that the new notation helps to address some common points of confusion. It emphasizes the di↵erence between random variables, denoted with capital letters, and their instantiations, denoted in lower case. For example, the state, action, and reward at time step t are denoted St, At, and Rt, while their possible values might be denoted s, a, and r. Along with this, it is natural to use lower case for value functions (e.g., v⇡) and restrict capitals to their tabular e


is natural to use lower case for value functions (e.g., v⇡) and restrict capitals to their tabular estimates (e.g., Qt(s, a)). Approximate value functions are deterministic functions of random parameters and are thus also in lower case (e.g., ˆv(s,wt) ⇡v⇡(s)). Vectors, such as the weight vector wt (formerly ✓t) and the feature vector xt (formerly φt), are bold and written in lowercase even if they are random variables. Uppercase bold is reserved for matrices. In the ﬁrst edition we used special notations, Pa ss0 and Ra ss0, for the transition probabilities and expected rewards. One weakness of that notation is that it still did not fully characterize the dynamics of the rewards, giving only their expectations, which is suﬃcient for dynamic programming but not for reinforcement learning.


y their expectations, which is suﬃcient for dynamic programming but not for reinforcement learning. Another weakness


xiv Preface to the Second Edition is the excess of subscripts and superscripts. In this edition we use the explicit notation of p(s0, r|s, a) for the joint probability for the next state and reward given the current state and action. All the changes in notation are summarized in a table on page xix. The second edition is signiﬁcantly expanded, and its top-level organization has been changed. After the introductory ﬁrst chapter, the second edition is divided into three new parts. The ﬁrst part (Chapters 2–8) treats as much of reinforcement learning as possible without going beyond the tabular case for which exact solutions can be found. We cover both learning and planning methods for the tabular case, as well as their uniﬁcation in n-step methods and in Dyna.


d planning methods for the tabular case, as well as their uniﬁcation in n-step methods and in Dyna. Many algorithms presented in this part are new to the second edition, including UCB, Expected Sarsa, Double learning, tree-backup, Q(σ), RTDP, and MCTS. Doing the tabular case ﬁrst, and thoroughly, enables core ideas to be developed in the simplest possible setting. The second part of the book (Chapters 9–13) is then devoted to extending the ideas to function approximation. It has new sections on artiﬁcial neural networks, the fourier basis, LSTD, kernel-based methods, Gradient-TD and Emphatic-TD methods, average-reward methods, true online TD(λ), and policygradient methods. The second edition signiﬁcantly expands the treatment of o↵-policy learning, ﬁrst for the tabular case in Chapters 5–7


signiﬁcantly expands the treatment of o↵-policy learning, ﬁrst for the tabular case in Chapters 5–7, then with function approximation in Chapters 11 and 12. Another change is that the second edition separates the forward-view idea of n-step bootstrapping (now treated more fully in Chapter 7) from the backwardview idea of eligibility traces (now treated independently in Chapter 12). The third part of the book has large new chapters on reinforcement learning’s relationships to psychology (Chapter 14) and neuroscience (Chapter 15), as well as an updated case-studies chapter including Atari game playing, Watson’s wagering strategy, and the Go playing programs AlphaGo and AlphaGo Zero (Chapter 16).


ing, Watson’s wagering strategy, and the Go playing programs AlphaGo and AlphaGo Zero (Chapter 16). Still, out of necessity we have included only a small subset of all that has been done in the ﬁeld. Our choices reﬂect our long-standing interests in inexpensive model-free methods that should scale well to large applications. The ﬁnal chapter now includes a discussion of the future societal impacts of reinforcement learning. For better or worse, the second edition is about twice as large as the ﬁrst. This book is designed to be used as the primary text for a one- or two-semester course on reinforcement learning. For a one-semester course, the ﬁrst ten chapters should be covered in order and form a good core, to which can be added material from the other chapters, from other books such as Be


orm a good core, to which can be added material from the other chapters, from other books such as Bertsekas and Tsitsiklis (1996), Wiering and van Otterlo (2012), and Szepesv´ari (2010), or from the literature, according to taste. Depending of the students’ background, some additional material on online supervised learning may be helpful. The ideas of options and option models are a natural addition (Sutton, Precup and Singh, 1999). A two-semester course can cover all the chapters as well as supplementary material. The book can also be used as part of broader courses on machine learning, artiﬁcial intelligence, or neural networks. In this case, it may be desirable to cover only a subset of the material.


ence, or neural networks. In this case, it may be desirable to cover only a subset of the material. We recommend covering Chapter 1 for a brief overview, Chapter 2 through Section 2.4, Chapter 3, and then selecting sections from the remaining chapters according to time and interests. Chapter 6 is the most important for the subject and for the rest of the book. A course focusing on machine learning or neural networks should cover Chapters 9 and 10, and a course focusing on artiﬁcial intelligence or planning should cover Chapter 8. Throughout the book, sections and chapters that are more diﬃcult and not essential to the rest of the book are marked


Preface to the Second Edition xv with a ⇤. These can be omitted on ﬁrst reading without creating problems later on. Some exercises are also marked with a ⇤to indicate that they are more advanced and not essential to understanding the basic material of the chapter. Most chapters end with a section entitled “Bibliographical and Historical Remarks,” wherein we credit the sources of the ideas presented in that chapter, provide pointers to further reading and ongoing research, and describe relevant historical background. Despite our attempts to make these sections authoritative and complete, we have undoubtedly left out some important prior work. For that we again apologize, and we welcome corrections and extensions for incorporation into the electronic version of the book.


nd we welcome corrections and extensions for incorporation into the electronic version of the book. Like the ﬁrst edition, this edition of the book is dedicated to the memory of A. Harry Klopf. It was Harry who introduced us to each other, and it was his ideas about the brain and artiﬁcial intelligence that launched our long excursion into reinforcement learning. Trained in neurophysiology and long interested in machine intelligence, Harry was a senior scientist aﬃliated with the Avionics Directorate of the Air Force Oﬃce of Scientiﬁc Research (AFOSR) at Wright-Patterson Air Force Base, Ohio. He was dissatisﬁed with the great importance attributed to equilibrium-seeking processes, including homeostasis and error-correcting pattern classiﬁcation methods, in explaining natural intelligence a


homeostasis and error-correcting pattern classiﬁcation methods, in explaining natural intelligence and in providing a basis for machine intelligence. He noted that systems that try to maximize something (whatever that might be) are qualitatively di↵erent from equilibriumseeking systems, and he argued that maximizing systems hold the key to understanding important aspects of natural intelligence and for building artiﬁcial intelligences. Harry was instrumental in obtaining funding from AFOSR for a project to assess the scientiﬁc merit of these and related ideas. This project was conducted in the late 1970s at the University of Massachusetts Amherst (UMass Amherst), initially under the direction of Michael Arbib, William Kilmer, and Nico Spinelli, professors in the Department of Computer and


n of Michael Arbib, William Kilmer, and Nico Spinelli, professors in the Department of Computer and Information Science at UMass Amherst, and founding members of the Cybernetics Center for Systems Neuroscience at the University, a farsighted group focusing on the intersection of neuroscience and artiﬁcial intelligence. Barto, a recent PhD from the University of Michigan, was hired as post doctoral researcher on the project. Meanwhile, Sutton, an undergraduate studying computer science and psychology at Stanford, had been corresponding with Harry regarding their mutual interest in the role of stimulus timing in classical conditioning. Harry suggested to the UMass group that Sutton would be a great addition to the project.


ditioning. Harry suggested to the UMass group that Sutton would be a great addition to the project. Thus, Sutton became a UMass graduate student, whose PhD was directed by Barto, who had become an Associate Professor. The study of reinforcement learning as presented in this book is rightfully an outcome of that project instigated by Harry and inspired by his ideas. Further, Harry was responsible for bringing us, the authors, together in what has been a long and enjoyable interaction. By dedicating this book to Harry we honor his essential contributions, not only to the ﬁeld of reinforcement learning, but also to our collaboration. We also thank Professors Arbib, Kilmer, and Spinelli for the opportunity they provided to us to begin exploring these ideas.


Arbib, Kilmer, and Spinelli for the opportunity they provided to us to begin exploring these ideas. Finally, we thank AFOSR for generous support over the early years of our research, and the NSF for its generous support over many of the following years. We have very many people to thank for their inspiration and help with this second edition. Everyone we acknowledged for their inspiration and help with the ﬁrst edition


xvi Preface to the Second Edition deserve our deepest gratitude for this edition as well, which would not exist were it not for their contributions to edition number one. To that long list we must add many others who contributed speciﬁcally to the second edition. Our students over the many years that we have taught this material contributed in countless ways: exposing errors, o↵ering ﬁxes, and—not the least—being confused in places where we could have explained things better. We especially thank Martha Steenstrup for reading and providing detailed comments throughout. The chapters on psychology and neuroscience could not have been written without the help of many experts in those ﬁelds. We thank John Moore for his patient tutoring over many many years on animal learning experiments, theory


hank John Moore for his patient tutoring over many many years on animal learning experiments, theory, and neuroscience, and for his careful reading of multiple drafts of Chapters 14 and 15. We also thank Matt Botvinick, Nathaniel Daw, Peter Dayan, and Yael Niv for their penetrating comments on drafts of these chapters, their essential guidance through the massive literature, and their interception of many of our errors in early drafts. Of course, the remaining errors in these chapters—and there must still be some—are totally our own. We thank Phil Thomas for helping us make these chapters accessible to non-psychologists and non-neuroscientists, and we thank Peter Sterling for helping us improve the exposition.


logists and non-neuroscientists, and we thank Peter Sterling for helping us improve the exposition. We are grateful to Jim Houk for introducing us to the subject of information processing in the basal ganglia and for alerting us to other relevant aspects of neuroscience. Jos´e Mart´ınez, Terry Sejnowski, David Silver, Gerry Tesauro, Georgios Theocharous, and Phil Thomas generously helped us understand details of their reinforcement learning applications for inclusion in the case-studies chapter, and they provided helpful comments on drafts of these sections. Special thanks are owed to David Silver for helping us better understand Monte Carlo Tree Search and the DeepMind Go-playing programs. We thank George Konidaris for his help with the section on the Fourier basis.


Go-playing programs. We thank George Konidaris for his help with the section on the Fourier basis. Emilio Cartoni, Thomas Cederborg, Stefan Dernbach, Clemens Rosenbaum, Patrick Taylor, Thomas Colin, and Pierre-Luc Bacon helped us in a number important ways for which we are most grateful. Sutton would also like to thank the members of the Reinforcement Learning and Artiﬁcial Intelligence laboratory at the University of Alberta for contributions to the second edition. He owes a particular debt to Rupam Mahmood for essential contributions to the treatment of o↵-policy Monte Carlo methods in Chapter 5, to Hamid Maei for helping develop the perspective on o↵-policy learning presented in Chapter 11, to Eric Graves for conducting the experiments in Chapter 13, to Shangtong Zhang for replicating


11, to Eric Graves for conducting the experiments in Chapter 13, to Shangtong Zhang for replicating and thus verifying almost all the experimental results, to Kris De Asis for improving the new technical content of Chapters 7 and 12, and to Harm van Seijen for insights that led to the separation of n-step methods from eligibility traces and (along with Hado van Hasselt) for the ideas involving exact equivalence of forward and backward views of eligibility traces presented in Chapter 12. Sutton also gratefully acknowledges the support and freedom he was granted by the Government of Alberta and the National Science and Engineering Research Council of Canada throughout the period during which the second edition was conceived and written.


Council of Canada throughout the period during which the second edition was conceived and written. In particular, he would like to thank Randy Goebel for creating a supportive and far-sighted environment for research in Alberta. He would also like to thank DeepMind their support in the last six months of writing the book. Finally, we owe thanks to the many careful readers of drafts of the second edition that we posted on the internet. They found many errors that we had missed and alerted us to potential points of confusion.


Preface to the First Edition We ﬁrst came to focus on what is now known as reinforcement learning in late 1979. We were both at the University of Massachusetts, working on one of the earliest projects to revive the idea that networks of neuronlike adaptive elements might prove to be a promising approach to artiﬁcial adaptive intelligence. The project explored the “heterostatic theory of adaptive systems” developed by A. Harry Klopf. Harry’s work was a rich source of ideas, and we were permitted to explore them critically and compare them with the long history of prior work in adaptive systems. Our task became one of teasing the ideas apart and understanding their relationships and relative importance.


ecame one of teasing the ideas apart and understanding their relationships and relative importance. This continues today, but in 1979 we came to realize that perhaps the simplest of the ideas, which had long been taken for granted, had received surprisingly little attention from a computational perspective. This was simply the idea of a learning system that wants something, that adapts its behavior in order to maximize a special signal from its environment. This was the idea of a “hedonistic” learning system, or, as we would say now, the idea of reinforcement learning. Like others, we had a sense that reinforcement learning had been thoroughly explored in the early days of cybernetics and artiﬁcial intelligence.


learning had been thoroughly explored in the early days of cybernetics and artiﬁcial intelligence. On closer inspection, though, we found that it had been explored only slightly. While reinforcement learning had clearly motivated some of the earliest computational studies of learning, most of these researchers had gone on to other things, such as pattern classiﬁcation, supervised learning, and adaptive control, or they had abandoned the study of learning altogether. As a result, the special issues involved in learning how to get something from the environment received relatively little attention. In retrospect, focusing on this idea was the critical step that set this branch of research in motion.


retrospect, focusing on this idea was the critical step that set this branch of research in motion. Little progress could be made in the computational study of reinforcement learning until it was recognized that such a fundamental idea had not yet been thoroughly explored. The ﬁeld has come a long way since then, evolving and maturing in several directions. Reinforcement learning has gradually become one of the most active research areas in machine learning, artiﬁcial intelligence, and neural network research. The ﬁeld has developed strong mathematical foundations and impressive applications. The computational study of reinforcement learning is now a large ﬁeld, with hundreds of active researchers around the world in diverse disciplines such as psychology, control theory, artiﬁcial intelli


rchers around the world in diverse disciplines such as psychology, control theory, artiﬁcial intelligence, and neuroscience. Particularly important have been the contributions establishing and developing the relationships to the theory of optimal control and dynamic programming.


xviii Preface to the First Edition The overall problem of learning from interaction to achieve goals is still far from being solved, but our understanding of it has improved signiﬁcantly. We can now place component ideas, such as temporal-di↵erence learning, dynamic programming, and function approximation, within a coherent perspective with respect to the overall problem. Our goal in writing this book was to provide a clear and simple account of the key ideas and algorithms of reinforcement learning. We wanted our treatment to be accessible to readers in all of the related disciplines, but we could not cover all of these perspectives in detail. For the most part, our treatment takes the point of view of artiﬁcial intelligence and engineering.


For the most part, our treatment takes the point of view of artiﬁcial intelligence and engineering. Coverage of connections to other ﬁelds we leave to others or to another time. We also chose not to produce a rigorous formal treatment of reinforcement learning. We did not reach for the highest possible level of mathematical abstraction and did not rely on a theorem–proof format. We tried to choose a level of mathematical detail that points the mathematically inclined in the right directions without distracting from the simplicity and potential generality of the underlying ideas. In some sense we have been working toward this book for thirty years, and we have lots of people to thank. First, we thank those who have personally helped us develop the overall view presented in this book: Harry


we thank those who have personally helped us develop the overall view presented in this book: Harry Klopf, for helping us recognize that reinforcement learning needed to be revived; Chris Watkins, Dimitri Bertsekas, John Tsitsiklis, and Paul Werbos, for helping us see the value of the relationships to dynamic programming; John Moore and Jim Kehoe, for insights and inspirations from animal learning theory; Oliver Selfridge, for emphasizing the breadth and importance of adaptation; and, more generally, our colleagues and students who have contributed in countless ways: Ron Williams, Charles Anderson, Satinder Singh, Sridhar Mahadevan, Steve Bradtke, Bob Crites, Peter Dayan, and Leemon Baird. Our view of reinforcement learning has been signiﬁcantly enriched by discussions with Paul Cohen, Pau


ur view of reinforcement learning has been signiﬁcantly enriched by discussions with Paul Cohen, Paul Utgo↵, Martha Steenstrup, Gerry Tesauro, Mike Jordan, Leslie Kaelbling, Andrew Moore, Chris Atkeson, Tom Mitchell, Nils Nilsson, Stuart Russell, Tom Dietterich, Tom Dean, and Bob Narendra. We thank Michael Littman, Gerry Tesauro, Bob Crites, Satinder Singh, and Wei Zhang for providing speciﬁcs of Sections 4.7, 15.1, 15.4, 15.5, and 15.6 respectively. We thank the Air Force Oﬃce of Scientiﬁc Research, the National Science Foundation, and GTE Laboratories for their long and farsighted support. We also wish to thank the many people who have read drafts of this book and provided valuable comments, including Tom Kalt, John Tsitsiklis, Pawel Cichosz, Olle G¨allmo, Chuck Anderson, Stuart Russell,


s, including Tom Kalt, John Tsitsiklis, Pawel Cichosz, Olle G¨allmo, Chuck Anderson, Stuart Russell, Ben Van Roy, Paul Steenstrup, Paul Cohen, Sridhar Mahadevan, Jette Randlov, Brian Sheppard, Thomas O’Connell, Richard Coggins, Cristina Versino, John H. Hiett, Andreas Badelt, Jay Ponte, Joe Beck, Justus Piater, Martha Steenstrup, Satinder Singh, Tommi Jaakkola, Dimitri Bertsekas, Torbj¨orn Ekman, Christina Bj¨orkman, Jakob Carlstr¨om, and Olle Palmgren. Finally, we thank Gwyn Mitchell for helping in many ways, and Harry Stanton and Bob Prior for being our champions at MIT Press.


Summary of Notation Capital letters are used for random variables, whereas lower case letters are used for the values of random variables and for scalar functions. Quantities that are required to be real-valued vectors are written in bold and in lower case (even if random variables). Matrices are bold capitals. .= equality relationship that is true by deﬁnition ⇡ approximately equal / proportional to Pr{X =x} probability that a random variable X takes on the value x X ⇠p random variable X selected from distribution p(x) .= Pr{X =x} E[X] expectation of a random variable X, i.e., E[X] .= P x p(x)x argmaxa f(a) a value of a at which f(a) takes its maximal value ln x natural logarithm of x ex, exp(x) the base of the natural logarithm, e ⇡2.71828, carried to power x; eln x = x R set of real num


xp(x) the base of the natural logarithm, e ⇡2.71828, carried to power x; eln x = x R set of real numbers f : X ! Y function f from elements of set X to elements of set Y assignment (a, b] the real interval between a and b including b but not including a " probability of taking a random action in an "-greedy policy ↵, β step-size parameters γ discount-rate parameter λ decay-rate parameter for eligibility traces predicate indicator function ( predicate .= 1 if the predicate is true, else 0) In a multi-arm bandit problem: k number of actions (arms) t discrete time step or play number q⇤(a) true value (expected reward) of action a Qt(a) estimate at time t of q⇤(a) Nt(a) number of times action a has been selected up prior to time t Ht(a) learned preference for selecting action a at time t ⇡t(a)


has been selected up prior to time t Ht(a) learned preference for selecting action a at time t ⇡t(a) probability of selecting action a at time t ¯Rt estimate at time t of the expected reward given ⇡t


xx Summary of Notation In a Markov Decision Process: s, s0 states a an action r a reward S set of all nonterminal states S+ set of all states, including the terminal state A(s) set of all actions available in state s R set of all possible rewards, a ﬁnite subset of R ⇢ subset of (e.g., R ⇢R) 2 is an element of; e.g. (s 2 S, r 2 R) |S| number of elements in set S t discrete time step T, T(t) ﬁnal time step of an episode, or of the episode including time step t At action at time t St state at time t, typically due, stochastically, to St−1 and At−1 Rt reward at time t, typically due, stochastically, to St−1 and At−1 ⇡ policy (decision-making rule) ⇡(s) action taken in state s under deterministic policy ⇡ ⇡(a|s) probability of taking action a in state s under stochastic policy ⇡ Gt return foll


c policy ⇡ ⇡(a|s) probability of taking action a in state s under stochastic policy ⇡ Gt return following time t h horizon, the time step one looks up to in a forward view Gt:t+n, Gt:h n-step return from t + 1 to t + n, or to h (discounted and corrected) ¯Gt:h ﬂat return (undiscounted and uncorrected) from t + 1 to h (Section 5.8) Gλ t λ-return (Section 12.1) Gλ t:h truncated, corrected λ-return (Section 12.3) Gλs t , Gλa t λ-return, corrected by estimated state, or action, values (Section 12.8) p(s0, r|s, a) probability of transition to state s0 with reward r, from state s and action a p(s0|s, a) probability of transition to state s0, from state s taking action a r(s, a) expected immediate reward from state s after action a r(s, a, s0) expected immediate reward on transition from s to s0


reward from state s after action a r(s, a, s0) expected immediate reward on transition from s to s0 under action a v⇡(s) value of state s under policy ⇡(expected return) v⇤(s) value of state s under the optimal policy q⇡(s, a) value of taking action a in state s under policy ⇡ q⇤(s, a) value of taking action a in state s under the optimal policy V, Vt array estimates of state-value function v⇡or v⇤ Q, Qt array estimates of action-value function q⇡or q⇤ ¯Vt(s) expected approximate action value; for example, ¯Vt(s) .= P a ⇡(a|s)Qt(s, a) Ut target for estimate at time t


Summary of Notation xxi δt temporal-di↵erence (TD) error at t (a random variable) (Section 6.1) δs t , δa t state- and action-speciﬁc forms of the TD error (Section 12.9) n in n-step methods, n is the number of steps of bootstrapping d dimensionality—the number of components of w d0 alternate dimensionality—the number of components of ✓ w, wt d-vector of weights underlying an approximate value function wi, wt,i ith component of learnable weight vector ˆv(s,w) approximate value of state s given weight vector w vw(s) alternate notation for ˆv(s,w) ˆq(s, a, w) approximate value of state–action pair s, a given weight vector w rˆv(s,w) column vector of partial derivatives of ˆv(s,w) with respect to w rˆq(s, a, w) column vector of partial derivatives of ˆq(s, a, w) with respect to w x(s) vector


to w rˆq(s, a, w) column vector of partial derivatives of ˆq(s, a, w) with respect to w x(s) vector of features visible when in state s x(s, a) vector of features visible when in state s taking action a xi(s), xi(s, a) ith component of vector x(s) or x(s, a) xt shorthand for x(St) or x(St, At) w>x inner product of vectors, w>x .= P i wixi; for example, ˆv(s,w) .= w>x(s) v, vt secondary d-vector of weights, used to learn w (Chapter 11) zt d-vector of eligibility traces at time t (Chapter 12) ✓, ✓t parameter vector of target policy (Chapter 13) ⇡(a|s, ✓) probability of taking action a in state s given parameter vector ✓ ⇡✓ policy corresponding to parameter ✓ r⇡(a|s, ✓) column vector of partial derivatives of ⇡(a|s, ✓) with respect to ✓ J(✓) performance measure for the policy ⇡✓ rJ(✓) column


derivatives of ⇡(a|s, ✓) with respect to ✓ J(✓) performance measure for the policy ⇡✓ rJ(✓) column vector of partial derivatives of J(✓) with respect to ✓ h(s, a, ✓) preference for selecting action a in state s based on ✓ b(a|s) behavior policy used to select actions while learning about target policy ⇡ b(s) a baseline function b : S 7! R for policy-gradient methods b branching factor for an MDP or search tree ⇢t:h importance sampling ratio for time t through time h (Section 5.5) ⇢t importance sampling ratio for time t alone, ⇢t .= ⇢t:t r(⇡) average reward (reward rate) for policy ⇡(Section 10.3) ¯Rt estimate of r(⇡) at time t µ(s) on-policy distribution over states (Section 9.2) µ |S|-vector of the µ(s) for all s 2 S kvk2 µ µ-weighted squared norm of value function v, i.e., kvk2 µ .= P s


or of the µ(s) for all s 2 S kvk2 µ µ-weighted squared norm of value function v, i.e., kvk2 µ .= P s2S µ(s)v(s)2 ⌘(s) expected number of visits to state s per episode (page 199) ⇧ projection operator for value functions (page 268) B⇡ Bellman operator for value functions (Section 11.4)


xxii Summary of Notation A d ⇥d matrix A .= E h xt # xt −γxt+1 $>i b d-dimensional vector b .= E[Rt+1xt] wTD TD ﬁxed point wTD .= A−1b (a d-vector, Section 9.4) I identity matrix P |S| ⇥|S| matrix of state-transition probabilities under ⇡ D |S| ⇥|S| diagonal matrix with µ on its diagonal X |S| ⇥d matrix with the x(s) as its rows ¯δw(s) Bellman error (expected TD error) for vw at state s (Section 11.4) ¯δw, BE Bellman error vector, with components ¯δw(s) VE(w) mean square value error VE(w) .= kvw −v⇡k2 µ (Section 9.2) BE(w) mean square Bellman error BE(w) .= &&¯δw &&2 µ PBE(w) mean square projected Bellman error PBE(w) .= &&⇧¯δw &&2 µ TDE(w) mean square temporal-di↵erence error TDE(w) .= Eb ⇥ ⇢tδ2 t ⇤ (Section 11.5) RE(w) mean square return error (Section 11.6)


Chapter 1 Introduction The idea that we learn by interacting with our environment is probably the ﬁrst to occur to us when we think about the nature of learning. When an infant plays, waves its arms, or looks about, it has no explicit teacher, but it does have a direct sensorimotor connection to its environment. Exercising this connection produces a wealth of information about cause and e↵ect, about the consequences of actions, and about what to do in order to achieve goals. Throughout our lives, such interactions are undoubtedly a major source of knowledge about our environment and ourselves. Whether we are learning to drive a car or to hold a conversation, we are acutely aware of how our environment responds to what we do, and we seek to inﬂuence what happens through our behavior.


our environment responds to what we do, and we seek to inﬂuence what happens through our behavior. Learning from interaction is a foundational idea underlying nearly all theories of learning and intelligence. In this book we explore a computational approach to learning from interaction. Rather than directly theorizing about how people or animals learn, we primarily explore idealized learning situations and evaluate the e↵ectiveness of various learning methods. That is, we adopt the perspective of an artiﬁcial intelligence researcher or engineer. We explore designs for machines that are e↵ective in solving learning problems of scientiﬁc or economic interest, evaluating the designs through mathematical analysis or computational experiments.


onomic interest, evaluating the designs through mathematical analysis or computational experiments. The approach we explore, called reinforcement learning, is much more focused on goal-directed learning from interaction than are other approaches to machine learning. 1.1 Reinforcement Learning Reinforcement learning is learning what to do—how to map situations to actions—so as to maximize a numerical reward signal. The learner is not told which actions to take, but instead must discover which actions yield the most reward by trying them. In the most interesting and challenging cases, actions may a↵ect not only the immediate reward but also the next situation and, through that, all subsequent rewards.


ot only the immediate reward but also the next situation and, through that, all subsequent rewards. These two characteristics—trial-and-error search and delayed reward—are the two most important distinguishing features of reinforcement learning.


2 Chapter 1: Introduction Reinforcement learning, like many topics whose names end with “ing,” such as machine learning and mountaineering, is simultaneously a problem, a class of solution methods that work well on the problem, and the ﬁeld that studies this problem and its solution methods. It is convenient to use a single name for all three things, but at the same time essential to keep the three conceptually separate. In particular, the distinction between problems and solution methods is very important in reinforcement learning; failing to make this distinction is the source of many confusions. We formalize the problem of reinforcement learning using ideas from dynamical systems theory, speciﬁcally, as the optimal control of incompletely-known Markov decision processes.


ystems theory, speciﬁcally, as the optimal control of incompletely-known Markov decision processes. The details of this formalization must wait until Chapter 3, but the basic idea is simply to capture the most important aspects of the real problem facing a learning agent interacting over time with its environment to achieve a goal. A learning agent must be able to sense the state of its environment to some extent and must be able to take actions that a↵ect the state. The agent also must have a goal or goals relating to the state of the environment. Markov decision processes are intended to include just these three aspects—sensation, action, and goal—in their simplest possible forms without trivializing any of them.


ects—sensation, action, and goal—in their simplest possible forms without trivializing any of them. Any method that is well suited to solving such problems we consider to be a reinforcement learning method. Reinforcement learning is di↵erent from supervised learning, the kind of learning studied in most current research in the ﬁeld of machine learning. Supervised learning is learning from a training set of labeled examples provided by a knowledgable external supervisor. Each example is a description of a situation together with a speciﬁcation—the label—of the correct action the system should take in that situation, which is often to identify a category to which the situation belongs. The object of this kind of learning is for the system to extrapolate, or generalize, its responses so that


ect of this kind of learning is for the system to extrapolate, or generalize, its responses so that it acts correctly in situations not present in the training set. This is an important kind of learning, but alone it is not adequate for learning from interaction. In interactive problems it is often impractical to obtain examples of desired behavior that are both correct and representative of all the situations in which the agent has to act. In uncharted territory—where one would expect learning to be most beneﬁcial—an agent must be able to learn from its own experience. Reinforcement learning is also di↵erent from what machine learning researchers call unsupervised learning, which is typically about ﬁnding structure hidden in collections of unlabeled data.


rvised learning, which is typically about ﬁnding structure hidden in collections of unlabeled data. The terms supervised learning and unsupervised learning would seem to exhaustively classify machine learning paradigms, but they do not. Although one might be tempted to think of reinforcement learning as a kind of unsupervised learning because it does not rely on examples of correct behavior, reinforcement learning is trying to maximize a reward signal instead of trying to ﬁnd hidden structure. Uncovering structure in an agent’s experience can certainly be useful in reinforcement learning, but by itself does not address the reinforcement learning problem of maximizing a reward signal. We therefore consider reinforcement learning to be a third machine learning paradigm, alongside supervised


efore consider reinforcement learning to be a third machine learning paradigm, alongside supervised learning and unsupervised learning and perhaps other paradigms.


1.1. Reinforcement Learning 3 One of the challenges that arise in reinforcement learning, and not in other kinds of learning, is the trade-o↵between exploration and exploitation. To obtain a lot of reward, a reinforcement learning agent must prefer actions that it has tried in the past and found to be e↵ective in producing reward. But to discover such actions, it has to try actions that it has not selected before. The agent has to exploit what it has already experienced in order to obtain reward, but it also has to explore in order to make better action selections in the future. The dilemma is that neither exploration nor exploitation can be pursued exclusively without failing at the task. The agent must try a variety of actions and progressively favor those that appear to be best.


task. The agent must try a variety of actions and progressively favor those that appear to be best. On a stochastic task, each action must be tried many times to gain a reliable estimate of its expected reward. The exploration–exploitation dilemma has been intensively studied by mathematicians for many decades, yet remains unresolved. For now, we simply note that the entire issue of balancing exploration and exploitation does not even arise in supervised and unsupervised learning, at least in the purest forms of these paradigms. Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment. This is in contrast to many approaches that consider subproblems without addressing how they might ﬁt


his is in contrast to many approaches that consider subproblems without addressing how they might ﬁt into a larger picture. For example, we have mentioned that many machine learning researchers have studied supervised learning without specifying how such an ability would ultimately be useful. Other researchers have developed theories of planning with general goals, but without considering planning’s role in real-time decision making, or the question of where the predictive models necessary for planning would come from. Although these approaches have yielded many useful results, their focus on isolated subproblems is a signiﬁcant limitation. Reinforcement learning takes the opposite tack, starting with a complete, interactive, goal-seeking agent.


cement learning takes the opposite tack, starting with a complete, interactive, goal-seeking agent. All reinforcement learning agents have explicit goals, can sense aspects of their environments, and can choose actions to inﬂuence their environments. Moreover, it is usually assumed from the beginning that the agent has to operate despite signiﬁcant uncertainty about the environment it faces. When reinforcement learning involves planning, it has to address the interplay between planning and real-time action selection, as well as the question of how environment models are acquired and improved. When reinforcement learning involves supervised learning, it does so for speciﬁc reasons that determine which capabilities are critical and which are not.


g, it does so for speciﬁc reasons that determine which capabilities are critical and which are not. For learning research to make progress, important subproblems have to be isolated and studied, but they should be subproblems that play clear roles in complete, interactive, goal-seeking agents, even if all the details of the complete agent cannot yet be ﬁlled in. By a complete, interactive, goal-seeking agent we do not always mean something like a complete organism or robot. These are clearly examples, but a complete, interactive, goal-seeking agent can also be a component of a larger behaving system. In this case, the agent directly interacts with the rest of the larger system and indirectly interacts with the larger system’s environment.


s with the rest of the larger system and indirectly interacts with the larger system’s environment. A simple example is an agent that monitors the charge level of robot’s battery and sends commands to the robot’s control architecture. This agent’s environment is the rest of the robot together with the robot’s environment. It is


4 Chapter 1: Introduction important to look beyond the most obvious examples of agents and their environments to appreciate the generality of the reinforcement learning framework. One of the most exciting aspects of modern reinforcement learning is its substantive and fruitful interactions with other engineering and scientiﬁc disciplines. Reinforcement learning is part of a decades-long trend within artiﬁcial intelligence and machine learning toward greater integration with statistics, optimization, and other mathematical subjects. For example, the ability of some reinforcement learning methods to learn with parameterized approximators addresses the classical “curse of dimensionality” in operations research and control theory.


mators addresses the classical “curse of dimensionality” in operations research and control theory. More distinctively, reinforcement learning has also interacted strongly with psychology and neuroscience, with substantial beneﬁts going both ways. Of all the forms of machine learning, reinforcement learning is the closest to the kind of learning that humans and other animals do, and many of the core algorithms of reinforcement learning were originally inspired by biological learning systems. Reinforcement learning has also given back, both through a psychological model of animal learning that better matches some of the empirical data, and through an inﬂuential model of parts of the brain’s reward system.


some of the empirical data, and through an inﬂuential model of parts of the brain’s reward system. The body of this book develops the ideas of reinforcement learning that pertain to engineering and artiﬁcial intelligence, with connections to psychology and neuroscience summarized in Chapters 14 and 15. Finally, reinforcement learning is also part of a larger trend in artiﬁcial intelligence back toward simple general principles. Since the late 1960s, many artiﬁcial intelligence researchers presumed that there are no general principles to be discovered, that intelligence is instead due to the possession of a vast number of special purpose tricks, procedures, and heuristics. It was sometimes said that if we could just get enough relevant facts into a machine, say one million, or one billion,


aid that if we could just get enough relevant facts into a machine, say one million, or one billion, then it would become intelligent. Methods based on general principles, such as search or learning, were characterized as “weak methods,” whereas those based on speciﬁc knowledge were called “strong methods.” This view is uncommon today. From our point of view, it was premature: too little e↵ort had been put into the search for general principles to conclude that there were none. Modern artiﬁcial intelligence now includes much research looking for general principles of learning, search, and decision making. It is not clear how far back the pendulum will swing, but reinforcement learning research is certainly part of the swing back toward simpler and fewer general principles of artiﬁcial inte


ch is certainly part of the swing back toward simpler and fewer general principles of artiﬁcial intelligence. 1.2 Examples A good way to understand reinforcement learning is to consider some of the examples and possible applications that have guided its development. • A master chess player makes a move. The choice is informed both by planning— anticipating possible replies and counterreplies—and by immediate, intuitive judgments of the desirability of particular positions and moves. • An adaptive controller adjusts parameters of a petroleum reﬁnery’s operation in real time. The controller optimizes the yield/cost/quality trade-o↵on the basis


1.2. Examples 5 of speciﬁed marginal costs without sticking strictly to the set points originally suggested by engineers. • A gazelle calf struggles to its feet minutes after being born. Half an hour later it is running at 20 miles per hour. • A mobile robot decides whether it should enter a new room in search of more trash to collect or start trying to ﬁnd its way back to its battery recharging station. It makes its decision based on the current charge level of its battery and how quickly and easily it has been able to ﬁnd the recharger in the past. • Phil prepares his breakfast. Closely examined, even this apparently mundane activity reveals a complex web of conditional behavior and interlocking goal–subgoal relationships: walking to the cupboard, opening it, selecting a cereal box, then


ocking goal–subgoal relationships: walking to the cupboard, opening it, selecting a cereal box, then reaching for, grasping, and retrieving the box. Other complex, tuned, interactive sequences of behavior are required to obtain a bowl, spoon, and milk carton. Each step involves a series of eye movements to obtain information and to guide reaching and locomotion. Rapid judgments are continually made about how to carry the objects or whether it is better to ferry some of them to the dining table before obtaining others. Each step is guided by goals, such as grasping a spoon or getting to the refrigerator, and is in service of other goals, such as having the spoon to eat with once the cereal is prepared and ultimately obtaining nourishment.


h as having the spoon to eat with once the cereal is prepared and ultimately obtaining nourishment. Whether he is aware of it or not, Phil is accessing information about the state of his body that determines his nutritional needs, level of hunger, and food preferences. These examples share features that are so basic that they are easy to overlook. All involve interaction between an active decision-making agent and its environment, within which the agent seeks to achieve a goal despite uncertainty about its environment. The agent’s actions are permitted to a↵ect the future state of the environment (e.g., the next chess position, the level of reservoirs of the reﬁnery, the robot’s next location and the future charge level of its battery), thereby a↵ecting the actions and opportunities availa


n and the future charge level of its battery), thereby a↵ecting the actions and opportunities available to the agent at later times. Correct choice requires taking into account indirect, delayed consequences of actions, and thus may require foresight or planning. At the same time, in all of these examples the e↵ects of actions cannot be fully predicted; thus the agent must monitor its environment frequently and react appropriately. For example, Phil must watch the milk he pours into his cereal bowl to keep it from overﬂowing. All these examples involve goals that are explicit in the sense that the agent can judge progress toward its goal based on what it can sense directly. The chess player knows whether or not he wins, the reﬁnery controller knows how much petroleum is being produced, the


knows whether or not he wins, the reﬁnery controller knows how much petroleum is being produced, the gazelle calf knows when it falls, the mobile robot knows when its batteries run down, and Phil knows whether or not he is enjoying his breakfast. In all of these examples the agent can use its experience to improve its performance over time. The chess player reﬁnes the intuition he uses to evaluate positions, thereby improving his play; the gazelle calf improves the eﬃciency with which it can run; Phil learns to streamline making his breakfast. The knowledge the agent brings to the task at the start—either from previous experience with related tasks or built into it by design or


6 Chapter 1: Introduction evolution—inﬂuences what is useful or easy to learn, but interaction with the environment is essential for adjusting behavior to exploit speciﬁc features of the task. 1.3 Elements of Reinforcement Learning Beyond the agent and the environment, one can identify four main subelements of a reinforcement learning system: a policy, a reward signal, a value function, and, optionally, a model of the environment. A policy deﬁnes the learning agent’s way of behaving at a given time. Roughly speaking, a policy is a mapping from perceived states of the environment to actions to be taken when in those states. It corresponds to what in psychology would be called a set of stimulus–response rules or associations.


corresponds to what in psychology would be called a set of stimulus–response rules or associations. In some cases the policy may be a simple function or lookup table, whereas in others it may involve extensive computation such as a search process. The policy is the core of a reinforcement learning agent in the sense that it alone is suﬃcient to determine behavior. In general, policies may be stochastic, specifying probabilities for each action. A reward signal deﬁnes the goal of a reinforcement learning problem. On each time step, the environment sends to the reinforcement learning agent a single number called the reward. The agent’s sole objective is to maximize the total reward it receives over the long run. The reward signal thus deﬁnes what are the good and bad events for the agent.


es over the long run. The reward signal thus deﬁnes what are the good and bad events for the agent. In a biological system, we might think of rewards as analogous to the experiences of pleasure or pain. They are the immediate and deﬁning features of the problem faced by the agent. The reward signal is the primary basis for altering the policy; if an action selected by the policy is followed by low reward, then the policy may be changed to select some other action in that situation in the future. In general, reward signals may be stochastic functions of the state of the environment and the actions taken. Whereas the reward signal indicates what is good in an immediate sense, a value function speciﬁes what is good in the long run.


dicates what is good in an immediate sense, a value function speciﬁes what is good in the long run. Roughly speaking, the value of a state is the total amount of reward an agent can expect to accumulate over the future, starting from that state. Whereas rewards determine the immediate, intrinsic desirability of environmental states, values indicate the long-term desirability of states after taking into account the states that are likely to follow and the rewards available in those states. For example, a state might always yield a low immediate reward but still have a high value because it is regularly followed by other states that yield high rewards. Or the reverse could be true. To make a human analogy, rewards are somewhat like pleasure (if high) and pain (if low), whereas values corresp


uman analogy, rewards are somewhat like pleasure (if high) and pain (if low), whereas values correspond to a more reﬁned and farsighted judgment of how pleased or displeased we are that our environment is in a particular state. Rewards are in a sense primary, whereas values, as predictions of rewards, are secondary. Without rewards there could be no values, and the only purpose of estimating values is to achieve more reward. Nevertheless, it is values with which we are most concerned when making and evaluating decisions. Action choices are made based on value judgments. We seek actions that bring about states of highest value, not highest reward, because these actions obtain the greatest amount of reward for us over the long run.


ighest reward, because these actions obtain the greatest amount of reward for us over the long run. Unfortunately, it is much harder to determine values than it is to determine rewards. Rewards are basically given directly by the environment, but values must be estimated and re-estimated from


1.4. Limitations and Scope 7 the sequences of observations an agent makes over its entire lifetime. In fact, the most important component of almost all reinforcement learning algorithms we consider is a method for eﬃciently estimating values. The central role of value estimation is arguably the most important thing that has been learned about reinforcement learning over the last six decades. The fourth and ﬁnal element of some reinforcement learning systems is a model of the environment. This is something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave. For example, given a state and action, the model might predict the resultant next state and next reward.


xample, given a state and action, the model might predict the resultant next state and next reward. Models are used for planning, by which we mean any way of deciding on a course of action by considering possible future situations before they are actually experienced. Methods for solving reinforcement learning problems that use models and planning are called model-based methods, as opposed to simpler model-free methods that are explicitly trial-and-error learners—viewed as almost the opposite of planning. In Chapter 8 we explore reinforcement learning systems that simultaneously learn by trial and error, learn a model of the environment, and use the model for planning. Modern reinforcement learning spans the spectrum from low-level, trial-and-error learning to high-level, deliberative plan