earning spans the spectrum from low-level, trial-and-error learning to high-level, deliberative planning. 1.4 Limitations and Scope Reinforcement learning relies heavily on the concept of state—as input to the policy and value function, and as both input to and output from the model. Informally, we can think of the state as a signal conveying to the agent some sense of “how the environment is” at a particular time. The formal deﬁnition of state as we use it here is given by the framework of Markov decision processes presented in Chapter 3. More generally, however, we encourage the reader to follow the informal meaning and think of the state as whatever information is available to the agent about its environment.


ing and think of the state as whatever information is available to the agent about its environment. In e↵ect, we assume that the state signal is produced by some preprocessing system that is nominally part of the agent’s environment. We do not address the issues of constructing, changing, or learning the state signal in this book (other than brieﬂy in Section 17.3). We take this approach not because we consider state representation to be unimportant, but in order to focus fully on the decision-making issues. In other words, our concern in this book is not with designing the state signal, but with deciding what action to take as a function of whatever state signal is available. Most of the reinforcement learning methods we consider in this book are structured around estimating value functio


inforcement learning methods we consider in this book are structured around estimating value functions, but it is not strictly necessary to do this to solve reinforcement learning problems. For example, solution methods such as genetic algorithms, genetic programming, simulated annealing, and other optimization methods never estimate value functions. These methods apply multiple static policies each interacting over an extended period of time with a separate instance of the environment. The policies that obtain the most reward, and random variations of them, are carried over to the next generation of policies, and the process repeats. We call these evolutionary methods because their operation is analogous to the way biological evolution produces organisms


8 Chapter 1: Introduction with skilled behavior even if they do not learn during their individual lifetimes. If the space of policies is suﬃciently small, or can be structured so that good policies are common or easy to ﬁnd—or if a lot of time is available for the search—then evolutionary methods can be e↵ective. In addition, evolutionary methods have advantages on problems in which the learning agent cannot sense the complete state of its environment. Our focus is on reinforcement learning methods that learn while interacting with the environment, which evolutionary methods do not do. Methods able to take advantage of the details of individual behavioral interactions can be much more eﬃcient than evolutionary methods in many cases.


ndividual behavioral interactions can be much more eﬃcient than evolutionary methods in many cases. Evolutionary methods ignore much of the useful structure of the reinforcement learning problem: they do not use the fact that the policy they are searching for is a function from states to actions; they do not notice which states an individual passes through during its lifetime, or which actions it selects. In some cases such information can be misleading (e.g., when states are misperceived), but more often it should enable more eﬃcient search. Although evolution and learning share many features and naturally work together, we do not consider evolutionary methods by themselves to be especially well suited to reinforcement learning problems and, accordingly, we do not cover them in this book.


well suited to reinforcement learning problems and, accordingly, we do not cover them in this book. 1.5 An Extended Example: Tic-Tac-Toe To illustrate the general idea of reinforcement learning and contrast it with other approaches, we next consider a single example in more detail. X X X O O X O Consider the familiar child’s game of tic-tac-toe. Two players take turns playing on a three-by-three board. One player plays Xs and the other Os until one player wins by placing three marks in a row, horizontally, vertically, or diagonally, as the X player has in the game shown to the right. If the board ﬁlls up with neither player getting three in a row, then the game is a draw. Because a skilled player can play so as never to lose, let us assume that we are playing against an imperfect player,


player can play so as never to lose, let us assume that we are playing against an imperfect player, one whose play is sometimes incorrect and allows us to win. For the moment, in fact, let us consider draws and losses to be equally bad for us. How might we construct a player that will ﬁnd the imperfections in its opponent’s play and learn to maximize its chances of winning? Although this is a simple problem, it cannot readily be solved in a satisfactory way through classical techniques. For example, the classical “minimax” solution from game theory is not correct here because it assumes a particular way of playing by the opponent. For example, a minimax player would never reach a game state from which it could lose, even if in fact it always won from that state because of incorrect play by


from which it could lose, even if in fact it always won from that state because of incorrect play by the opponent. Classical optimization methods for sequential decision problems, such as dynamic programming, can compute an optimal solution for any opponent, but require as input a complete speciﬁcation of that opponent, including the probabilities with which the opponent makes each move in each board state. Let us assume that this information is not available a priori for this problem, as it is not for the vast majority of problems of


1.5. An Extended Example: Tic-Tac-Toe 9 practical interest. On the other hand, such information can be estimated from experience, in this case by playing many games against the opponent. About the best one can do on this problem is ﬁrst to learn a model of the opponent’s behavior, up to some level of conﬁdence, and then apply dynamic programming to compute an optimal solution given the approximate opponent model. In the end, this is not that di↵erent from some of the reinforcement learning methods we examine later in this book. An evolutionary method applied to this problem would directly search the space of possible policies for one with a high probability of winning against the opponent. Here, a policy is a rule that tells the player what move to make for every state of the game—every po


ere, a policy is a rule that tells the player what move to make for every state of the game—every possible conﬁguration of Xs and Os on the three-by-three board. For each policy considered, an estimate of its winning probability would be obtained by playing some number of games against the opponent. This evaluation would then direct which policy or policies were considered next. A typical evolutionary method would hill-climb in policy space, successively generating and evaluating policies in an attempt to obtain incremental improvements. Or, perhaps, a genetic-style algorithm could be used that would maintain and evaluate a population of policies. Literally hundreds of di↵erent optimization methods could be applied.


ate a population of policies. Literally hundreds of di↵erent optimization methods could be applied. Here is how the tic-tac-toe problem would be approached with a method making use of a value function. First we would set up a table of numbers, one for each possible state of the game. Each number will be the latest estimate of the probability of our winning from that state. We treat this estimate as the state’s value, and the whole table is the learned value function. State A has higher value than state B, or is considered “better” than state B, if the current estimate of the probability of our winning from A is higher than it is from B. Assuming we always play Xs, then for all states with three Xs in a row the probability of winning is 1, because we have already won.


for all states with three Xs in a row the probability of winning is 1, because we have already won. Similarly, for all states with three Os in a row, or that are ﬁlled up, the correct probability is 0, as we cannot win from them. We set the initial values of all the other states to 0.5, representing a guess that we have a 50% chance of winning. We then play many games against the opponent. To select our moves we examine the states that would result from each of our possible moves (one for each blank space on the board) and look up their current values in the table. Most of the time we move greedily, selecting the move that leads to the state with greatest value, that is, with the highest estimated probability of winning.


leads to the state with greatest value, that is, with the highest estimated probability of winning. Occasionally, however, we select randomly from among the other moves instead. These are called exploratory moves because they cause us to experience states that we might otherwise never see. A sequence of moves made and considered during a game can be diagrammed as in Figure 1.1. While we are playing, we change the values of the states in which we ﬁnd ourselves during the game. We attempt to make them more accurate estimates of the probabilities of winning. To do this, we “back up” the value of the state after each greedy move to the state before the move, as suggested by the arrows in Figure 1.1.


tate after each greedy move to the state before the move, as suggested by the arrows in Figure 1.1. More precisely, the current value of the earlier state is updated to be closer to the value of the later state. This can be done by moving the earlier state’s value a fraction of the way toward the value of the later state. If we let St denote the state before the greedy move, and St+1 the state after that move, then the update to the estimated value of St, denoted V (St),


10 Chapter 1: Introduction .. • our move{ opponent's move{ our move{ starting position • • • a b c* d e e* opponent's move{ c •f •g* g opponent's move{ our move{ . • a g* c starting position b c* d e* e f g … Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken during a game; the dashed lines represent moves that we (our reinforcement learning player) considered but did not make. The * indicates the move currently estimated to be the best. Our second move was an exploratory move, meaning that it was taken even though another sibling move, the one leading to e⇤, was ranked higher. Exploratory moves do not result in any learning, but each of our other moves does, causing updates as suggested by the red arrows in which estimated values are moved up the


ves does, causing updates as suggested by the red arrows in which estimated values are moved up the tree from later nodes to earlier nodes as detailed in the text. can be written as V (St) V (St) + ↵ h V (St+1) −V (St) i , where ↵is a small positive fraction called the step-size parameter, which inﬂuences the rate of learning. This update rule is an example of a temporal-di↵erence learning method, so called because its changes are based on a di↵erence, V (St+1)−V (St), between estimates at two successive times. The method described above performs quite well on this task. For example, if the step-size parameter is reduced properly over time, then this method converges, for any ﬁxed opponent, to the true probabilities of winning from each state given optimal play by our player.


ed opponent, to the true probabilities of winning from each state given optimal play by our player. Furthermore, the moves then taken (except on exploratory moves) are in fact the optimal moves against this (imperfect) opponent. In other words, the method converges to an optimal policy for playing the game against this opponent. If the step-size parameter is not reduced all the way to zero over time, then this player also plays well against opponents that slowly change their way of playing.


1.5. An Extended Example: Tic-Tac-Toe 11 This example illustrates the di↵erences between evolutionary methods and methods that learn value functions. To evaluate a policy, an evolutionary method holds the policy ﬁxed and plays many games against the opponent or simulates many games using a model of the opponent. The frequency of wins gives an unbiased estimate of the probability of winning with that policy, and can be used to direct the next policy selection. But each policy change is made only after many games, and only the ﬁnal outcome of each game is used: what happens during the games is ignored. For example, if the player wins, then all of its behavior in the game is given credit, independently of how speciﬁc moves might have been critical to the win.


n the game is given credit, independently of how speciﬁc moves might have been critical to the win. Credit is even given to moves that never occurred! Value function methods, in contrast, allow individual states to be evaluated. In the end, evolutionary and value function methods both search the space of policies, but learning a value function takes advantage of information available during the course of play. This simple example illustrates some of the key features of reinforcement learning methods. First, there is the emphasis on learning while interacting with an environment, in this case with an opponent player. Second, there is a clear goal, and correct behavior requires planning or foresight that takes into account delayed e↵ects of one’s choices.


ct behavior requires planning or foresight that takes into account delayed e↵ects of one’s choices. For example, the simple reinforcement learning player would learn to set up multi-move traps for a shortsighted opponent. It is a striking feature of the reinforcement learning solution that it can achieve the e↵ects of planning and lookahead without using a model of the opponent and without conducting an explicit search over possible sequences of future states and actions. While this example illustrates some of the key features of reinforcement learning, it is so simple that it might give the impression that reinforcement learning is more limited than it really is. Although tic-tac-toe is a two-person game, reinforcement learning also applies in the case in which there is no external advers


wo-person game, reinforcement learning also applies in the case in which there is no external adversary, that is, in the case of a “game against nature.” Reinforcement learning also is not restricted to problems in which behavior breaks down into separate episodes, like the separate games of tic-tac-toe, with reward only at the end of each episode. It is just as applicable when behavior continues indeﬁnitely and when rewards of various magnitudes can be received at any time. Reinforcement learning is also applicable to problems that do not even break down into discrete time steps like the plays of tic-tac-toe. The general principles apply to continuous-time problems as well, although the theory gets more complicated and we omit it from this introductory treatment.


as well, although the theory gets more complicated and we omit it from this introductory treatment. Tic-tac-toe has a relatively small, ﬁnite state set, whereas reinforcement learning can be used when the state set is very large, or even inﬁnite. For example, Gerry Tesauro (1992, 1995) combined the algorithm described above with an artiﬁcial neural network to learn to play backgammon, which has approximately 1020 states. With this many states it is impossible ever to experience more than a small fraction of them. Tesauro’s program learned to play far better than any previous program and eventually better than the world’s best human players (see Section 16.1). The artiﬁcial neural network provides the program with the ability to generalize from its experience, so that in new states it selec


vides the program with the ability to generalize from its experience, so that in new states it selects moves based on information saved from similar states faced in the past, as determined by the network. How well a reinforcement learning system can work in problems with such large state sets is intimately tied to how appropriately it can generalize from past


12 Chapter 1: Introduction experience. It is in this role that we have the greatest need for supervised learning methods within reinforcement learning. Artiﬁcial neural networks and deep learning (Section 9.7) are not the only, or necessarily the best, way to do this. In this tic-tac-toe example, learning started with no prior knowledge beyond the rules of the game, but reinforcement learning by no means entails a tabula rasa view of learning and intelligence. On the contrary, prior information can be incorporated into reinforcement learning in a variety of ways that can be critical for eﬃcient learning (e.g., see Sections 9.5, 17.4, and 13.1). We also have access to the true state in the tic-tac-toe example, whereas reinforcement learning can also be applied when part of the state is hidd


c-tac-toe example, whereas reinforcement learning can also be applied when part of the state is hidden, or when di↵erent states appear to the learner to be the same. Finally, the tic-tac-toe player was able to look ahead and know the states that would result from each of its possible moves. To do this, it had to have a model of the game that allowed it to foresee how its environment would change in response to moves that it might never make. Many problems are like this, but in others even a short-term model of the e↵ects of actions is lacking. Reinforcement learning can be applied in either case. A model is not required, but models can easily be used if they are available or can be learned (Chapter 8).


is not required, but models can easily be used if they are available or can be learned (Chapter 8). On the other hand, there are reinforcement learning methods that do not need any kind of environment model at all. Model-free systems cannot even think about how their environments will change in response to a single action. The tic-tac-toe player is model-free in this sense with respect to its opponent: it has no model of its opponent of any kind. Because models have to be reasonably accurate to be useful, model-free methods can have advantages over more complex methods when the real bottleneck in solving a problem is the diﬃculty of constructing a suﬃciently accurate environment model. Model-free methods are also important building blocks for model-based methods.


e environment model. Model-free methods are also important building blocks for model-based methods. In this book we devote several chapters to model-free methods before we discuss how they can be used as components of more complex model-based methods. Reinforcement learning can be used at both high and low levels in a system. Although the tic-tac-toe player learned only about the basic moves of the game, nothing prevents reinforcement learning from working at higher levels where each of the “actions” may itself be the application of a possibly elaborate problem-solving method. In hierarchical learning systems, reinforcement learning can work simultaneously on several levels. Exercise 1.1: Self-Play Suppose, instead of playing against a random opponent, the reinforcement learning algorithm


lf-Play Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself, with both sides learning. What do you think would happen in this case? Would it learn a di↵erent policy for selecting moves? ⇤ Exercise 1.2: Symmetries Many tic-tac-toe positions appear di↵erent but are really the same because of symmetries. How might we amend the learning process described above to take advantage of this? In what ways would this change improve the learning process? Now think again. Suppose the opponent did not take advantage of symmetries. In that case, should we? Is it true, then, that symmetrically equivalent positions should necessarily have the same value?


? Is it true, then, that symmetrically equivalent positions should necessarily have the same value? ⇤ Exercise 1.3: Greedy Play Suppose the reinforcement learning player was greedy, that is, it always played the move that brought it to the position that it rated the best. Might it


1.7. Early History of Reinforcement Learning 13 learn to play better, or worse, than a nongreedy player? What problems might occur? ⇤ Exercise 1.4: Learning from Exploration Suppose learning updates occurred after all moves, including exploratory moves. If the step-size parameter is appropriately reduced over time (but not the tendency to explore), then the state values would converge to a di↵erent set of probabilities. What (conceptually) are the two sets of probabilities computed when we do, and when we do not, learn from exploratory moves? Assuming that we do continue to make exploratory moves, which set of probabilities might be better to learn? Which would result in more wins? ⇤ Exercise 1.5: Other Improvements Can you think of other ways to improve the reinforcement learning player?


e 1.5: Other Improvements Can you think of other ways to improve the reinforcement learning player? Can you think of any better way to solve the tic-tac-toe problem as posed? ⇤ 1.6 Summary Reinforcement learning is a computational approach to understanding and automating goal-directed learning and decision making. It is distinguished from other computational approaches by its emphasis on learning by an agent from direct interaction with its environment, without requiring exemplary supervision or complete models of the environment. In our opinion, reinforcement learning is the ﬁrst ﬁeld to seriously address the computational issues that arise when learning from interaction with an environment in order to achieve long-term goals.


that arise when learning from interaction with an environment in order to achieve long-term goals. Reinforcement learning uses the formal framework of Markov decision processes to deﬁne the interaction between a learning agent and its environment in terms of states, actions, and rewards. This framework is intended to be a simple way of representing essential features of the artiﬁcial intelligence problem. These features include a sense of cause and e↵ect, a sense of uncertainty and nondeterminism, and the existence of explicit goals. The concepts of value and value function are key to most of the reinforcement learning methods that we consider in this book. We take the position that value functions are important for eﬃcient search in the space of policies.


e take the position that value functions are important for eﬃcient search in the space of policies. The use of value functions distinguishes reinforcement learning methods from evolutionary methods that search directly in policy space guided by evaluations of entire policies. 1.7 Early History of Reinforcement Learning The early history of reinforcement learning has two main threads, both long and rich, that were pursued independently before intertwining in modern reinforcement learning. One thread concerns learning by trial and error, and originated in the psychology of animal learning. This thread runs through some of the earliest work in artiﬁcial intelligence and led to the revival of reinforcement learning in the early 1980s.


work in artiﬁcial intelligence and led to the revival of reinforcement learning in the early 1980s. The second thread concerns the problem of optimal control and its solution using value functions and dynamic programming. For the most part, this thread did not involve learning. The two threads were mostly independent, but became interrelated to some extent around a


14 Chapter 1: Introduction third, less distinct thread concerning temporal-di↵erence methods such as that used in the tic-tac-toe example in this chapter. All three threads came together in the late 1980s to produce the modern ﬁeld of reinforcement learning as we present it in this book. The thread focusing on trial-and-error learning is the one with which we are most familiar and about which we have the most to say in this brief history. Before doing that, however, we brieﬂy discuss the optimal control thread. The term “optimal control” came into use in the late 1950s to describe the problem of designing a controller to minimize or maximize a measure of a dynamical system’s behavior over time.


esigning a controller to minimize or maximize a measure of a dynamical system’s behavior over time. One of the approaches to this problem was developed in the mid-1950s by Richard Bellman and others through extending a nineteenth century theory of Hamilton and Jacobi. This approach uses the concepts of a dynamical system’s state and of a value function, or “optimal return function,” to deﬁne a functional equation, now often called the Bellman equation. The class of methods for solving optimal control problems by solving this equation came to be known as dynamic programming (Bellman, 1957a). Bellman (1957b) also introduced the discrete stochastic version of the optimal control problem known as Markov decision processes (MDPs).


screte stochastic version of the optimal control problem known as Markov decision processes (MDPs). Ronald Howard (1960) devised the policy iteration method for MDPs. All of these are essential elements underlying the theory and algorithms of modern reinforcement learning. Dynamic programming is widely considered the only feasible way of solving general stochastic optimal control problems. It su↵ers from what Bellman called “the curse of dimensionality,” meaning that its computational requirements grow exponentially with the number of state variables, but it is still far more eﬃcient and more widely applicable than any other general method. Dynamic programming has been extensively developed since the late 1950s, including extensions to partially observable MDPs (surveyed by Lovejoy, 1991),


since the late 1950s, including extensions to partially observable MDPs (surveyed by Lovejoy, 1991), many applications (surveyed by White, 1985, 1988, 1993), approximation methods (surveyed by Rust, 1996), and asynchronous methods (Bertsekas, 1982, 1983). Many excellent modern treatments of dynamic programming are available (e.g., Bertsekas, 2005, 2012; Puterman, 1994; Ross, 1983; Whittle, 1982, 1983). Bryson (1996) provides an authoritative history of optimal control. Connections between optimal control and dynamic programming, on the one hand, and learning, on the other, were slow to be recognized. We cannot be sure about what accounted for this separation, but its main cause was likely the separation between the disciplines involved and their di↵erent goals.


its main cause was likely the separation between the disciplines involved and their di↵erent goals. Also contributing may have been the prevalent view of dynamic programming as an o↵-line computation depending essentially on accurate system models and analytic solutions to the Bellman equation. Further, the simplest form of dynamic programming is a computation that proceeds backwards in time, making it diﬃcult to see how it could be involved in a learning process that must proceed in a forward direction. Some of the earliest work in dynamic programming, such as that by Bellman and Dreyfus (1959), might now be classiﬁed as following a learning approach. Witten’s (1977) work (discussed below) certainly qualiﬁes as a combination of learning and dynamic-programming ideas.


rk (discussed below) certainly qualiﬁes as a combination of learning and dynamic-programming ideas. Werbos (1987) argued explicitly for greater interrelation of dynamic programming and learning methods and for dynamic programming’s relevance to understanding neural and cognitive mechanisms. For us the full integration of dynamic programming methods with online learning did not occur


1.7. Early History of Reinforcement Learning 15 until the work of Chris Watkins in 1989, whose treatment of reinforcement learning using the MDP formalism has been widely adopted. Since then these relationships have been extensively developed by many researchers, most particularly by Dimitri Bertsekas and John Tsitsiklis (1996), who coined the term “neurodynamic programming” to refer to the combination of dynamic programming and artiﬁcial neural networks. Another term currently in use is “approximate dynamic programming.” These various approaches emphasize di↵erent aspects of the subject, but they all share with reinforcement learning an interest in circumventing the classical shortcomings of dynamic programming.


nforcement learning an interest in circumventing the classical shortcomings of dynamic programming. We consider all of the work in optimal control also to be, in a sense, work in reinforcement learning. We deﬁne a reinforcement learning method as any e↵ective way of solving reinforcement learning problems, and it is now clear that these problems are closely related to optimal control problems, particularly stochastic optimal control problems such as those formulated as MDPs. Accordingly, we must consider the solution methods of optimal control, such as dynamic programming, also to be reinforcement learning methods. Because almost all of the conventional methods require complete knowledge of the system to be controlled, it feels a little unnatural to say that they are part of reinforcement


the system to be controlled, it feels a little unnatural to say that they are part of reinforcement learning. On the other hand, many dynamic programming algorithms are incremental and iterative. Like learning methods, they gradually reach the correct answer through successive approximations. As we show in the rest of this book, these similarities are far more than superﬁcial. The theories and solution methods for the cases of complete and incomplete knowledge are so closely related that we feel they must be considered together as part of the same subject matter. Let us return now to the other major thread leading to the modern ﬁeld of reinforcement learning, the thread centered on the idea of trial-and-error learning.


modern ﬁeld of reinforcement learning, the thread centered on the idea of trial-and-error learning. We only touch on the major points of contact here, taking up this topic in more detail in Section 14.3. According to American psychologist R. S. Woodworth (1938) the idea of trial-and-error learning goes as far back as the 1850s to Alexander Bain’s discussion of learning by “groping and experiment” and more explicitly to the British ethologist and psychologist Conway Lloyd Morgan’s 1894 use of the term to describe his observations of animal behavior. Perhaps the ﬁrst to succinctly express the essence of trial-and-error learning as a principle of learning was Edward Thorndike: Of several responses made to the same situation, those which are accompanied or closely followed by satisfaction to t


ses made to the same situation, those which are accompanied or closely followed by satisfaction to the animal will, other things being equal, be more ﬁrmly connected with the situation, so that, when it recurs, they will be more likely to recur; those which are accompanied or closely followed by discomfort to the animal will, other things being equal, have their connections with that situation weakened, so that, when it recurs, they will be less likely to occur. The greater the satisfaction or discomfort, the greater the strengthening or weakening of the bond. (Thorndike, 1911, p. 244) Thorndike called this the “Law of E↵ect” because it describes the e↵ect of reinforcing events on the tendency to select actions.


w of E↵ect” because it describes the e↵ect of reinforcing events on the tendency to select actions. Thorndike later modiﬁed the law to better account for subsequent data on animal learning (such as di↵erences between the e↵ects of reward and punishment), and the law in its various forms has generated considerable controversy among learning theorists (e.g., see Gallistel, 2005; Herrnstein, 1970; Kimble,


16 Chapter 1: Introduction 1961, 1967; Mazur, 1994). Despite this, the Law of E↵ect—in one form or another—is widely regarded as a basic principle underlying much behavior (e.g., Hilgard and Bower, 1975; Dennett, 1978; Campbell, 1960; Cziko, 1995). It is the basis of the inﬂuential learning theories of Clark Hull (1943, 1952) and the inﬂuential experimental methods of B. F. Skinner (1938). The term “reinforcement” in the context of animal learning came into use well after Thorndike’s expression of the Law of E↵ect, ﬁrst appearing in this context (to the best of our knowledge) in the 1927 English translation of Pavlov’s monograph on conditioned reﬂexes. Pavlov described reinforcement as the strengthening of a pattern of behavior due to an animal receiving a stimulus—a reinforcer—in an appro


trengthening of a pattern of behavior due to an animal receiving a stimulus—a reinforcer—in an appropriate temporal relationship with another stimulus or with a response. Some psychologists extended the idea of reinforcement to include weakening as well as strengthening of behavior, and extended the idea of a reinforcer to include possibly the omission or termination of stimulus. To be considered a reinforcer, the strengthening or weakening must persist after the reinforcer is withdrawn; a stimulus that merely attracts an animal’s attention or that energizes its behavior without producing lasting changes would not be considered a reinforcer. The idea of implementing trial-and-error learning in a computer appeared among the earliest thoughts about the possibility of artiﬁcial intelligence.


in a computer appeared among the earliest thoughts about the possibility of artiﬁcial intelligence. In a 1948 report, Alan Turing described a design for a “pleasure-pain system” that worked along the lines of the Law of E↵ect: When a conﬁguration is reached for which the action is undetermined, a random choice for the missing data is made and the appropriate entry is made in the description, tentatively, and is applied. When a pain stimulus occurs all tentative entries are cancelled, and when a pleasure stimulus occurs they are all made permanent. (Turing, 1948) Many ingenious electro-mechanical machines were constructed that demonstrated trialand-error learning. The earliest may have been a machine built by Thomas Ross (1933) that was able to ﬁnd its way through a simple maze and remember


machine built by Thomas Ross (1933) that was able to ﬁnd its way through a simple maze and remember the path through the settings of switches. In 1951 W. Grey Walter built a version of his “mechanical tortoise” (Walter, 1950) capable of a simple form of learning. In 1952 Claude Shannon demonstrated a maze-running mouse named Theseus that used trial and error to ﬁnd its way through a maze, with the maze itself remembering the successful directions via magnets and relays under its ﬂoor (see also Shannon, 1951). J. A. Deutsch (1954) described a maze-solving machine based on his behavior theory (Deutsch, 1953) that has some properties in common with model-based reinforcement learning (Chapter 8).


tsch, 1953) that has some properties in common with model-based reinforcement learning (Chapter 8). In his PhD dissertation, Marvin Minsky (1954) discussed computational models of reinforcement learning and described his construction of an analog machine composed of components he called SNARCs (Stochastic Neural-Analog Reinforcement Calculators) meant to resemble modiﬁable synaptic connections in the brain (Chapter 15). The web site cyberneticzoo.com contains a wealth of information on these and many other electro-mechanical learning machines. Building electro-mechanical learning machines gave way to programming digital computers to perform various types of learning, some of which implemented trial-and-error learning.


computers to perform various types of learning, some of which implemented trial-and-error learning. Farley and Clark (1954) described a digital simulation of a neural-network


1.7. Early History of Reinforcement Learning 17 learning machine that learned by trial and error. But their interests soon shifted from trial-and-error learning to generalization and pattern recognition, that is, from reinforcement learning to supervised learning (Clark and Farley, 1955). This began a pattern of confusion about the relationship between these types of learning. Many researchers seemed to believe that they were studying reinforcement learning when they were actually studying supervised learning. For example, artiﬁcial neural network pioneers such as Rosenblatt (1962) and Widrow and Ho↵(1960) were clearly motivated by reinforcement learning—they used the language of rewards and punishments—but the systems they studied were supervised learning systems suitable for pattern reco


punishments—but the systems they studied were supervised learning systems suitable for pattern recognition and perceptual learning. Even today, some researchers and textbooks minimize or blur the distinction between these types of learning. For example, some textbooks have used the term “trialand-error” to describe artiﬁcial neural networks that learn from training examples. This is an understandable confusion because these networks use error information to update connection weights, but this misses the essential character of trial-and-error learning as selecting actions on the basis of evaluative feedback that does not rely on knowledge of what the correct action should be. Partly as a result of these confusions, research into genuine trial-and-error learning became rare in the 1960s and


ult of these confusions, research into genuine trial-and-error learning became rare in the 1960s and 1970s, although there were notable exceptions. In the 1960s the terms “reinforcement” and “reinforcement learning” were used in the engineering literature for the ﬁrst time to describe engineering uses of trial-and-error learning (e.g., Waltz and Fu, 1965; Mendel, 1966; Fu, 1970; Mendel and McClaren, 1970). Particularly inﬂuential was Minsky’s paper “Steps Toward Artiﬁcial Intelligence” (Minsky, 1961), which discussed several issues relevant to trial-and-error learning, including prediction, expectation, and what he called the basic credit-assignment problem for complex reinforcement learning systems: How do you distribute credit for success among the many decisions that may have been invol


systems: How do you distribute credit for success among the many decisions that may have been involved in producing it? All of the methods we discuss in this book are, in a sense, directed toward solving this problem. Minsky’s paper is well worth reading today. In the next few paragraphs we discuss some of the other exceptions and partial exceptions to the relative neglect of computational and theoretical study of genuine trial-and-error learning in the 1960s and 1970s. One exception was the work of the New Zealand researcher John Andreae, who developed a system called STeLLA that learned by trial and error in interaction with its environment. This system included an internal model of the world and, later, an “internal monologue” to deal with problems of hidden state (Andreae, 1963, 1969;


orld and, later, an “internal monologue” to deal with problems of hidden state (Andreae, 1963, 1969; Andreae and Cashin, 1969). Andreae’s later work (1977) placed more emphasis on learning from a teacher, but still included learning by trial and error, with the generation of novel events being one of the system’s goals. A feature of this work was a “leakback process,” elaborated more fully in Andreae (1998), that implemented a credit-assignment mechanism similar to the backing-up update operations that we describe. Unfortunately, his pioneering research was not well known and did not greatly impact subsequent reinforcement learning research. Recent summaries are available (Andreae, 2017a,b). More inﬂuential was the work of Donald Michie.


. Recent summaries are available (Andreae, 2017a,b). More inﬂuential was the work of Donald Michie. In 1961 and 1963 he described a simple trial-and-error learning system for learning how to play tic-tac-toe (or naughts


18 Chapter 1: Introduction and crosses) called MENACE (for Matchbox Educable Naughts and Crosses Engine). It consisted of a matchbox for each possible game position, each matchbox containing a number of colored beads, a di↵erent color for each possible move from that position. By drawing a bead at random from the matchbox corresponding to the current game position, one could determine MENACE’s move. When a game was over, beads were added to or removed from the boxes used during play to reward or punish MENACE’s decisions. Michie and Chambers (1968) described another tic-tac-toe reinforcement learner called GLEE (Game Learning Expectimaxing Engine) and a reinforcement learning controller called BOXES.


led GLEE (Game Learning Expectimaxing Engine) and a reinforcement learning controller called BOXES. They applied BOXES to the task of learning to balance a pole hinged to a movable cart on the basis of a failure signal occurring only when the pole fell or the cart reached the end of a track. This task was adapted from the earlier work of Widrow and Smith (1964), who used supervised learning methods, assuming instruction from a teacher already able to balance the pole. Michie and Chambers’s version of pole-balancing is one of the best early examples of a reinforcement learning task under conditions of incomplete knowledge. It inﬂuenced much later work in reinforcement learning, beginning with some of our own studies (Barto, Sutton, and Anderson, 1983; Sutton, 1984).


learning, beginning with some of our own studies (Barto, Sutton, and Anderson, 1983; Sutton, 1984). Michie (1974) consistently emphasized trial and error and learning as essential aspects of artiﬁcial intelligence. Widrow, Gupta, and Maitra (1973) modiﬁed the Least-Mean-Square (LMS) algorithm of Widrow and Ho↵(1960) to produce a reinforcement learning rule that could learn from success and failure signals instead of from training examples. They called this form of learning “selective bootstrap adaptation” and described it as “learning with a critic” instead of “learning with a teacher.” They analyzed this rule and showed how it could learn to play blackjack. This was an isolated foray into reinforcement learning by Widrow, whose contributions to supervised learning were much more inﬂuentia


einforcement learning by Widrow, whose contributions to supervised learning were much more inﬂuential. Our use of the term “critic” is derived from Widrow, Gupta, and Maitra’s paper. Buchanan, Mitchell, Smith, and Johnson (1978) independently used the term critic in the context of machine learning (see also Dietterich and Buchanan, 1984), but for them a critic was an expert system able to do more than evaluate performance. Research on learning automata had a more direct inﬂuence on the trial-and-error thread leading to modern reinforcement learning research. These are methods for solving a nonassociative, purely selectional learning problem known as the k-armed bandit by analogy to a slot machine, or “one-armed bandit,” except with k levers (see Chapter 2).


d bandit by analogy to a slot machine, or “one-armed bandit,” except with k levers (see Chapter 2). Learning automata are simple, low-memory machines for improving the probability of reward in these problems. Learning automata originated with work in the 1960s of the Russian mathematician and physicist M. L. Tsetlin and colleagues (published posthumously in Tsetlin, 1973) and has been extensively developed since then within engineering (see Narendra and Thathachar, 1974, 1989). These developments included the study of stochastic learning automata, which are methods for updating action probabilities on the basis of reward signals. Although not developed in the tradition of stochastic learning automata, Harth and Tzanakou’s (1974) Alopex algorithm (for Algorithm of pattern extraction) is a s


automata, Harth and Tzanakou’s (1974) Alopex algorithm (for Algorithm of pattern extraction) is a stochastic method for detecting correlations between actions and reinforcement that inﬂuenced some of our early research (Barto, Sutton, and Brouwer, 1981). Stochastic learning automata were foreshadowed by earlier work in psychology, beginning with William Estes’ (1950) e↵ort toward a statistical theory of learning and further developed by others (e.g., Bush and Mosteller, 1955; Sternberg, 1963).


1.7. Early History of Reinforcement Learning 19 The statistical learning theories developed in psychology were adopted by researchers in economics, leading to a thread of research in that ﬁeld devoted to reinforcement learning. This work began in 1973 with the application of Bush and Mosteller’s learning theory to a collection of classical economic models (Cross, 1973). One goal of this research was to study artiﬁcial agents that act more like real people than do traditional idealized economic agents (Arthur, 1991). This approach expanded to the study of reinforcement learning in the context of game theory. Reinforcement learning in economics developed largely independently of the early work in reinforcement learning in artiﬁcial intelligence, though game theory remains a topic of interest


in reinforcement learning in artiﬁcial intelligence, though game theory remains a topic of interest in both ﬁelds (beyond the scope of this book). Camerer (2011) discusses the reinforcement learning tradition in economics, and Now´e, Vrancx, and De Hauwere (2012) provide an overview of the subject from the point of view of multi-agent extensions to the approach that we introduce in this book. Reinforcement learning in the context of game theory is a much di↵erent subject than reinforcement learning used in programs to play tic-tac-toe, checkers, and other recreational games. See, for example, Szita (2012) for an overview of this aspect of reinforcement learning and games. John Holland (1975) outlined a general theory of adaptive systems based on selectional principles.


John Holland (1975) outlined a general theory of adaptive systems based on selectional principles. His early work concerned trial and error primarily in its nonassociative form, as in evolutionary methods and the k-armed bandit. In 1976 and more fully in 1986, he introduced classiﬁer systems, true reinforcement learning systems including association and value functions. A key component of Holland’s classiﬁer systems was the “bucket-brigade algorithm” for credit assignment, which is closely related to the temporal di↵erence algorithm used in our tic-tac-toe example and discussed in Chapter 6. Another key component was a genetic algorithm, an evolutionary method whose role was to evolve useful representations.


nt was a genetic algorithm, an evolutionary method whose role was to evolve useful representations. Classiﬁer systems have been extensively developed by many researchers to form a major branch of reinforcement learning research (reviewed by Urbanowicz and Moore, 2009), but genetic algorithms—which we do not consider to be reinforcement learning systems by themselves—have received much more attention, as have other approaches to evolutionary computation (e.g., Fogel, Owens and Walsh, 1966; Koza, 1992). The individual most responsible for reviving the trial-and-error thread of reinforcement learning within artiﬁcial intelligence was Harry Klopf (1972, 1975, 1982). Klopf recognized that essential aspects of adaptive behavior were being lost as learning researchers came to focus almost exclusi


al aspects of adaptive behavior were being lost as learning researchers came to focus almost exclusively on supervised learning. What was missing, according to Klopf, were the hedonic aspects of behavior: the drive to achieve some result from the environment, to control the environment toward desired ends and away from undesired ends (see Section 15.9). This is the essential idea of trial-and-error learning. Klopf’s ideas were especially inﬂuential on the authors because our assessment of them (Barto and Sutton, 1981a) led to our appreciation of the distinction between supervised and reinforcement learning, and to our eventual focus on reinforcement learning. Much of the early work that we and colleagues accomplished was directed toward showing that reinforcement learning and supervised le


nd colleagues accomplished was directed toward showing that reinforcement learning and supervised learning were indeed di↵erent (Barto, Sutton, and Brouwer, 1981; Barto and Sutton, 1981b; Barto and Anandan, 1985). Other studies showed how reinforcement learning could address important problems in artiﬁcial neural


20 Chapter 1: Introduction network learning, in particular, how it could produce learning algorithms for multilayer networks (Barto, Anderson, and Sutton, 1982; Barto and Anderson, 1985; Barto, 1985, 1986; Barto and Jordan, 1987; see Section 15.10). We turn now to the third thread to the history of reinforcement learning, that concerning temporal-di↵erence learning. Temporal-di↵erence learning methods are distinctive in being driven by the di↵erence between temporally successive estimates of the same quantity—for example, of the probability of winning in the tic-tac-toe example. This thread is smaller and less distinct than the other two, but it has played a particularly important role in the ﬁeld, in part because temporal-di↵erence methods seem to be new and unique to reinforcement learni


e ﬁeld, in part because temporal-di↵erence methods seem to be new and unique to reinforcement learning. The origins of temporal-di↵erence learning are in part in animal learning psychology, in particular, in the notion of secondary reinforcers. A secondary reinforcer is a stimulus that has been paired with a primary reinforcer such as food or pain and, as a result, has come to take on similar reinforcing properties. Minsky (1954) may have been the ﬁrst to realize that this psychological principle could be important for artiﬁcial learning systems. Arthur Samuel (1959) was the ﬁrst to propose and implement a learning method that included temporal-di↵erence ideas, as part of his celebrated checkers-playing program (Section 16.2).


cluded temporal-di↵erence ideas, as part of his celebrated checkers-playing program (Section 16.2). Samuel made no reference to Minsky’s work or to possible connections to animal learning. His inspiration apparently came from Claude Shannon’s (1950) suggestion that a computer could be programmed to use an evaluation function to play chess, and that it might be able to improve its play by modifying this function online. (It is possible that these ideas of Shannon’s also inﬂuenced Bellman, but we know of no evidence for this.) Minsky (1961) extensively discussed Samuel’s work in his “Steps” paper, suggesting the connection to secondary reinforcement theories, both natural and artiﬁcial. As we have discussed, in the decade following the work of Minsky and Samuel, little computational work was


have discussed, in the decade following the work of Minsky and Samuel, little computational work was done on trial-and-error learning, and apparently no computational work at all was done on temporal-di↵erence learning. In 1972, Klopf brought trial-anderror learning together with an important component of temporal-di↵erence learning. Klopf was interested in principles that would scale to learning in large systems, and thus was intrigued by notions of local reinforcement, whereby subcomponents of an overall learning system could reinforce one another. He developed the idea of “generalized reinforcement,” whereby every component (nominally, every neuron) views all of its inputs in reinforcement terms: excitatory inputs as rewards and inhibitory inputs as punishments.


s inputs in reinforcement terms: excitatory inputs as rewards and inhibitory inputs as punishments. This is not the same idea as what we now know as temporal-di↵erence learning, and in retrospect it is farther from it than was Samuel’s work. On the other hand, Klopf linked the idea with trial-and-error learning and related it to the massive empirical database of animal learning psychology. Sutton (1978a,b,c) developed Klopf’s ideas further, particularly the links to animal learning theories, describing learning rules driven by changes in temporally successive predictions. He and Barto reﬁned these ideas and developed a psychological model of classical conditioning based on temporal-di↵erence learning (Sutton and Barto, 1981a; Barto and Sutton, 1982).


onditioning based on temporal-di↵erence learning (Sutton and Barto, 1981a; Barto and Sutton, 1982). There followed several other inﬂuential psychological models of classical conditioning based on temporal-di↵erence learning (e.g., Klopf, 1988; Moore et al.,


1.7. Early History of Reinforcement Learning 21 1986; Sutton and Barto, 1987, 1990). Some neuroscience models developed at this time are well interpreted in terms of temporal-di↵erence learning (Hawkins and Kandel, 1984; Byrne, Gingrich, and Baxter, 1990; Gelperin, Hopﬁeld, and Tank, 1985; Tesauro, 1986; Friston et al., 1994), although in most cases there was no historical connection. Our early work on temporal-di↵erence learning was strongly inﬂuenced by animal learning theories and by Klopf’s work. Relationships to Minsky’s “Steps” paper and to Samuel’s checkers players were recognized only afterward. By 1981, however, we were fully aware of all the prior work mentioned above as part of the temporal-di↵erence and trial-and-error threads.


f all the prior work mentioned above as part of the temporal-di↵erence and trial-and-error threads. At this time we developed a method for using temporal-di↵erence learning combined with trial-and-error learning, known as the actor–critic architecture, and applied this method to Michie and Chambers’s pole-balancing problem (Barto, Sutton, and Anderson, 1983). This method was extensively studied in Sutton’s (1984) PhD dissertation and extended to use backpropagation neural networks in Anderson’s (1986) PhD dissertation. Around this time, Holland (1986) incorporated temporaldi↵erence ideas explicitly into his classiﬁer systems in the form of his bucket-brigade algorithm. A key step was taken by Sutton (1988) by separating temporal-di↵erence learning from control, treating it as a general pre


y Sutton (1988) by separating temporal-di↵erence learning from control, treating it as a general prediction method. That paper also introduced the TD(λ) algorithm and proved some of its convergence properties. As we were ﬁnalizing our work on the actor–critic architecture in 1981, we discovered a paper by Ian Witten (1977, 1976a) which appears to be the earliest publication of a temporal-di↵erence learning rule. He proposed the method that we now call tabular TD(0) for use as part of an adaptive controller for solving MDPs. This work was ﬁrst submitted for journal publication in 1974 and also appeared in Witten’s 1976 PhD dissertation. Witten’s work was a descendant of Andreae’s early experiments with STeLLA and other trial-and-error learning systems.


a descendant of Andreae’s early experiments with STeLLA and other trial-and-error learning systems. Thus, Witten’s 1977 paper spanned both major threads of reinforcement learning research—trial-and-error learning and optimal control—while making a distinct early contribution to temporal-di↵erence learning. The temporal-di↵erence and optimal control threads were fully brought together in 1989 with Chris Watkins’s development of Q-learning. This work extended and integrated prior work in all three threads of reinforcement learning research. Paul Werbos (1987) contributed to this integration by arguing for the convergence of trial-and-error learning and dynamic programming since 1977. By the time of Watkins’s work there had been tremendous growth in reinforcement learning research, primarily


me of Watkins’s work there had been tremendous growth in reinforcement learning research, primarily in the machine learning subﬁeld of artiﬁcial intelligence, but also in artiﬁcial neural networks and artiﬁcial intelligence more broadly. In 1992, the remarkable success of Gerry Tesauro’s backgammon playing program, TD-Gammon, brought additional attention to the ﬁeld. In the time since publication of the ﬁrst edition of this book, a ﬂourishing subﬁeld of neuroscience developed that focuses on the relationship between reinforcement learning algorithms and reinforcement learning in the nervous system. Most responsible for this is an uncanny similarity between the behavior of temporal-di↵erence algorithms and the activity of dopamine producing neurons in the brain, as pointed out by a number o


algorithms and the activity of dopamine producing neurons in the brain, as pointed out by a number of researchers (Friston et al., 1994; Barto, 1995a; Houk, Adams, and Barto, 1995; Montague, Dayan, and Sejnowski, 1996; and Schultz, Dayan, and Montague, 1997). Chapter 15 provides an introduction to this exciting aspect of reinforcement learning. Other important


22 Chapter 1: Introduction contributions made in the recent history of reinforcement learning are too numerous to mention in this brief account; we cite many of these at the end of the individual chapters in which they arise. Bibliographical Remarks For additional general coverage of reinforcement learning, we refer the reader to the books by Szepesv´ari (2010), Bertsekas and Tsitsiklis (1996), Kaelbling (1993a), and Sugiyama, Hachiya, and Morimura (2013). Books that take a control or operations research perspective include those of Si, Barto, Powell, and Wunsch (2004), Powell (2011), Lewis and Liu (2012), and Bertsekas (2012). Cao’s (2009) review places reinforcement learning in the context of other approaches to learning and optimization of stochastic dynamic systems.


ning in the context of other approaches to learning and optimization of stochastic dynamic systems. Three special issues of the journal Machine Learning focus on reinforcement learning: Sutton (1992a), Kaelbling (1996), and Singh (2002). Useful surveys are provided by Barto (1995b); Kaelbling, Littman, and Moore (1996); and Keerthi and Ravindran (1997). The volume edited by Weiring and van Otterlo (2012) provides an excellent overview of recent developments. 1.2 The example of Phil’s breakfast in this chapter was inspired by Agre (1988). 1.5 The temporal-di↵erence method used in the tic-tac-toe example is developed in Chapter 6.


Part I: Tabular Solution Methods In this part of the book we describe almost all the core ideas of reinforcement learning algorithms in their simplest forms: that in which the state and action spaces are small enough for the approximate value functions to be represented as arrays, or tables. In this case, the methods can often ﬁnd exact solutions, that is, they can often ﬁnd exactly the optimal value function and the optimal policy. This contrasts with the approximate methods described in the next part of the book, which only ﬁnd approximate solutions, but which in return can be applied e↵ectively to much larger problems. The ﬁrst chapter of this part of the book describes solution methods for the special case of the reinforcement learning problem in which there is only a single state, cal


or the special case of the reinforcement learning problem in which there is only a single state, called bandit problems. The second chapter describes the general problem formulation that we treat throughout the rest of the book—ﬁnite Markov decision processes—and its main ideas including Bellman equations and value functions. The next three chapters describe three fundamental classes of methods for solving ﬁnite Markov decision problems: dynamic programming, Monte Carlo methods, and temporaldi↵erence learning. Each class of methods has its strengths and weaknesses. Dynamic programming methods are well developed mathematically, but require a complete and accurate model of the environment. Monte Carlo methods don’t require a model and are conceptually simple, but are not well suited for step


te Carlo methods don’t require a model and are conceptually simple, but are not well suited for step-by-step incremental computation. Finally, temporal-di↵erence methods require no model and are fully incremental, but are more complex to analyze. The methods also di↵er in several ways with respect to their eﬃciency and speed of convergence. The remaining two chapters describe how these three classes of methods can be combined to obtain the best features of each of them. In one chapter we describe how the strengths of Monte Carlo methods can be combined with the strengths of temporaldi↵erence methods via multi-step bootstrapping methods. In the ﬁnal chapter of this part of the book we show how temporal-di↵erence learning methods can be combined with model learning and planning methods (such


w temporal-di↵erence learning methods can be combined with model learning and planning methods (such as dynamic programming) for a complete and uniﬁed solution to the tabular reinforcement learning problem.


Chapter 2 Multi-armed Bandits The most important feature distinguishing reinforcement learning from other types of learning is that it uses training information that evaluates the actions taken rather than instructs by giving correct actions. This is what creates the need for active exploration, for an explicit search for good behavior. Purely evaluative feedback indicates how good the action taken was, but not whether it was the best or the worst action possible. Purely instructive feedback, on the other hand, indicates the correct action to take, independently of the action actually taken. This kind of feedback is the basis of supervised learning, which includes large parts of pattern classiﬁcation, artiﬁcial neural networks, and system identiﬁcation.


includes large parts of pattern classiﬁcation, artiﬁcial neural networks, and system identiﬁcation. In their pure forms, these two kinds of feedback are quite distinct: evaluative feedback depends entirely on the action taken, whereas instructive feedback is independent of the action taken. In this chapter we study the evaluative aspect of reinforcement learning in a simpliﬁed setting, one that does not involve learning to act in more than one situation. This nonassociative setting is the one in which most prior work involving evaluative feedback has been done, and it avoids much of the complexity of the full reinforcement learning problem. Studying this case enables us to see most clearly how evaluative feedback di↵ers from, and yet can be combined with, instructive feedback.


st clearly how evaluative feedback di↵ers from, and yet can be combined with, instructive feedback. The particular nonassociative, evaluative feedback problem that we explore is a simple version of the k-armed bandit problem. We use this problem to introduce a number of basic learning methods which we extend in later chapters to apply to the full reinforcement learning problem. At the end of this chapter, we take a step closer to the full reinforcement learning problem by discussing what happens when the bandit problem becomes associative, that is, when the best action depends on the situation. 2.1 A k-armed Bandit Problem Consider the following learning problem. You are faced repeatedly with a choice among k di↵erent options, or actions.


wing learning problem. You are faced repeatedly with a choice among k di↵erent options, or actions. After each choice you receive a numerical reward chosen from a stationary probability distribution that depends on the action you selected. Your


26 Chapter 2: Multi-armed Bandits objective is to maximize the expected total reward over some time period, for example, over 1000 action selections, or time steps. This is the original form of the k-armed bandit problem, so named by analogy to a slot machine, or “one-armed bandit,” except that it has k levers instead of one. Each action selection is like a play of one of the slot machine’s levers, and the rewards are the payo↵s for hitting the jackpot. Through repeated action selections you are to maximize your winnings by concentrating your actions on the best levers. Another analogy is that of a doctor choosing between experimental treatments for a series of seriously ill patients. Each action is the selection of a treatment, and each reward is the survival or well-being of the patient.


ction is the selection of a treatment, and each reward is the survival or well-being of the patient. Today the term “bandit problem” is sometimes used for a generalization of the problem described above, but in this book we use it to refer just to this simple case. In our k-armed bandit problem, each of the k actions has an expected or mean reward given that that action is selected; let us call this the value of that action. We denote the action selected on time step t as At, and the corresponding reward as Rt. The value then of an arbitrary action a, denoted q⇤(a), is the expected reward given that a is selected: q⇤(a) .= E[Rt | At =a] . If you knew the value of each action, then it would be trivial to solve the k-armed bandit problem: you would always select the action with highest value


e trivial to solve the k-armed bandit problem: you would always select the action with highest value. We assume that you do not know the action values with certainty, although you may have estimates. We denote the estimated value of action a at time step t as Qt(a). We would like Qt(a) to be close to q⇤(a). If you maintain estimates of the action values, then at any time step there is at least one action whose estimated value is greatest. We call these the greedy actions. When you select one of these actions, we say that you are exploiting your current knowledge of the values of the actions. If instead you select one of the nongreedy actions, then we say you are exploring, because this enables you to improve your estimate of the nongreedy action’s value.


u are exploring, because this enables you to improve your estimate of the nongreedy action’s value. Exploitation is the right thing to do to maximize the expected reward on the one step, but exploration may produce the greater total reward in the long run. For example, suppose a greedy action’s value is known with certainty, while several other actions are estimated to be nearly as good but with substantial uncertainty. The uncertainty is such that at least one of these other actions probably is actually better than the greedy action, but you don’t know which one. If you have many time steps ahead on which to make action selections, then it may be better to explore the nongreedy actions and discover which of them are better than the greedy action.


tter to explore the nongreedy actions and discover which of them are better than the greedy action. Reward is lower in the short run, during exploration, but higher in the long run because after you have discovered the better actions, you can exploit them many times. Because it is not possible both to explore and to exploit with any single action selection, one often refers to the “conﬂict” between exploration and exploitation. In any speciﬁc case, whether it is better to explore or exploit depends in a complex way on the precise values of the estimates, uncertainties, and the number of remaining steps. There are many sophisticated methods for balancing exploration and exploitation for particular mathematical formulations of the k-armed bandit and related problems.


2.2. Action-value Methods 27 However, most of these methods make strong assumptions about stationarity and prior knowledge that are either violated or impossible to verify in most applications and in the full reinforcement learning problem that we consider in subsequent chapters. The guarantees of optimality or bounded loss for these methods are of little comfort when the assumptions of their theory do not apply. In this book we do not worry about balancing exploration and exploitation in a sophisticated way; we worry only about balancing them at all. In this chapter we present several simple balancing methods for the k-armed bandit problem and show that they work much better than methods that always exploit.


or the k-armed bandit problem and show that they work much better than methods that always exploit. The need to balance exploration and exploitation is a distinctive challenge that arises in reinforcement learning; the simplicity of our version of the k-armed bandit problem enables us to show this in a particularly clear form. 2.2 Action-value Methods We begin by looking more closely at methods for estimating the values of actions and for using the estimates to make action selection decisions, which we collectively call action-value methods. Recall that the true value of an action is the mean reward when that action is selected. One natural way to estimate this is by averaging the rewards actually received: Qt(a) .= sum of rewards when a taken prior to t number of times a taken prior to t


tually received: Qt(a) .= sum of rewards when a taken prior to t number of times a taken prior to t = Pt−1 i=1 Ri · Ai=a Pt−1 i=1 Ai=a , (2.1) where predicate denotes the random variable that is 1 if predicate is true and 0 if it is not. If the denominator is zero, then we instead deﬁne Qt(a) as some default value, such as 0. As the denominator goes to inﬁnity, by the law of large numbers, Qt(a) converges to q⇤(a). We call this the sample-average method for estimating action values because each estimate is an average of the sample of relevant rewards. Of course this is just one way to estimate action values, and not necessarily the best one. Nevertheless, for now let us stay with this simple estimation method and turn to the question of how the estimates might be used to select actions.


le estimation method and turn to the question of how the estimates might be used to select actions. The simplest action selection rule is to select one of the actions with the highest estimated value, that is, one of the greedy actions as deﬁned in the previous section. If there is more than one greedy action, then a selection is made among them in some arbitrary way, perhaps randomly. We write this greedy action selection method as At .= argmax a Qt(a), (2.2) where argmaxa denotes the action a for which the expression that follows is maximized (with ties broken arbitrarily). Greedy action selection always exploits current knowledge to maximize immediate reward; it spends no time at all sampling apparently inferior actions to see if they might really be better.


t spends no time at all sampling apparently inferior actions to see if they might really be better. A simple alternative is to behave greedily most of the time, but every once in a while, say with small probability ", instead select randomly


28 Chapter 2: Multi-armed Bandits from among all the actions with equal probability, independently of the action-value estimates. We call methods using this near-greedy action selection rule "-greedy methods. An advantage of these methods is that, in the limit as the number of steps increases, every action will be sampled an inﬁnite number of times, thus ensuring that all the Qt(a) converge to their respective q⇤(a). This of course implies that the probability of selecting the optimal action converges to greater than 1 −", that is, to near certainty. These are just asymptotic guarantees, however, and say little about the practical e↵ectiveness of the methods. Exercise 2.1 In "-greedy action selection, for the case of two actions and " = 0.5, what is the probability that the greedy action i


selection, for the case of two actions and " = 0.5, what is the probability that the greedy action is selected? ⇤ 2.3 The 10-armed Testbed To roughly assess the relative e↵ectiveness of the greedy and "-greedy action-value methods, we compared them numerically on a suite of test problems. This was a set of 2000 randomly generated k-armed bandit problems with k = 10. For each bandit problem, such as the one shown in Figure 2.1, the action values, q⇤(a), a = 1, . . . , 10, 0 1 2 3 -3 -2 -1 q⇤(1) q⇤(2) q⇤(3) q⇤(4) q⇤(5) q⇤(6) q⇤(7) q⇤(8) q⇤(9) q⇤(10) Reward distribution 1 2 6 3 5 4 7 8 9 10 Action Figure 2.1: An example bandit problem from the 10-armed testbed. The true value q⇤(a) of each of the ten actions was selected according to a normal distribution with mean zero and unit variance, and


he ten actions was selected according to a normal distribution with mean zero and unit variance, and then the actual rewards were selected according to a mean q⇤(a), unit-variance normal distribution, as suggested by these gray distributions.


2.3. The 10-armed Testbed 29 were selected according to a normal (Gaussian) distribution with mean 0 and variance 1. Then, when a learning method applied to that problem selected action At at time step t, the actual reward, Rt, was selected from a normal distribution with mean q⇤(At) and variance 1. These distributions are shown in gray in Figure 2.1. We call this suite of test tasks the 10-armed testbed. For any learning method, we can measure its performance and behavior as it improves with experience over 1000 time steps when applied to one of the bandit problems. This makes up one run. Repeating this for 2000 independent runs, each with a di↵erent bandit problem, we obtained measures of the learning algorithm’s average behavior.


with a di↵erent bandit problem, we obtained measures of the learning algorithm’s average behavior. Figure 2.2 compares a greedy method with two "-greedy methods ("=0.01 and "=0.1), as described above, on the 10-armed testbed. All the methods formed their action-value estimates using the sample-average technique (with an initial estimate of 0). The upper graph shows the increase in expected reward with experience. The greedy method improved slightly faster than the other methods at the very beginning, but then leveled o↵at a lower level. It achieved a reward-per-step of only about 1, compared with the best possible of about 1.54 on this testbed. The greedy method performed signiﬁcantly worse in the long run because it often got stuck performing suboptimal actions.


formed signiﬁcantly worse in the long run because it often got stuck performing suboptimal actions. The lower graph (greedy) 0 0.5 1 1.5 Average reward 0 250 500 750 1000 Steps 0% 20% 40% 60% 80% 100% % Optimal action 0 250 500 750 1000 Steps 1 1 "=0.1 "=0.01 "=0.1 "=0.01 "=0 (greedy) "=0 Figure 2.2: Average performance of "-greedy action-value methods on the 10-armed testbed. These data are averages over 2000 runs with di↵erent bandit problems. All methods used sample averages as their action-value estimates.


30 Chapter 2: Multi-armed Bandits shows that the greedy method found the optimal action in only approximately one-third of the tasks. In the other two-thirds, its initial samples of the optimal action were disappointing, and it never returned to it. The "-greedy methods eventually performed better because they continued to explore and to improve their chances of recognizing the optimal action. The " = 0.1 method explored more, and usually found the optimal action earlier, but it never selected that action more than 91% of the time. The " = 0.01 method improved more slowly, but eventually would perform better than the " = 0.1 method on both performance measures shown in the ﬁgure. It is also possible to reduce " over time to try to get the best of both high and low values.


gure. It is also possible to reduce " over time to try to get the best of both high and low values. The advantage of "-greedy over greedy methods depends on the task. For example, suppose the reward variance had been larger, say 10 instead of 1. With noisier rewards it takes more exploration to ﬁnd the optimal action, and "-greedy methods should fare even better relative to the greedy method. On the other hand, if the reward variances were zero, then the greedy method would know the true value of each action after trying it once. In this case the greedy method might actually perform best because it would soon ﬁnd the optimal action and then never explore. But even in the deterministic case there is a large advantage to exploring if we weaken some of the other assumptions.


erministic case there is a large advantage to exploring if we weaken some of the other assumptions. For example, suppose the bandit task were nonstationary, that is, the true values of the actions changed over time. In this case exploration is needed even in the deterministic case to make sure one of the nongreedy actions has not changed to become better than the greedy one. As we shall see in the next few chapters, nonstationarity is the case most commonly encountered in reinforcement learning. Even if the underlying task is stationary and deterministic, the learner faces a set of banditlike decision tasks each of which changes over time as learning proceeds and the agent’s decision-making policy changes. Reinforcement learning requires a balance between exploration and exploitation.


ing policy changes. Reinforcement learning requires a balance between exploration and exploitation. Exercise 2.2: Bandit example Consider a k-armed bandit problem with k = 4 actions, denoted 1, 2, 3, and 4. Consider applying to this problem a bandit algorithm using "-greedy action selection, sample-average action-value estimates, and initial estimates of Q1(a) = 0, for all a. Suppose the initial sequence of actions and rewards is A1 = 1, R1 = −1, A2 = 2, R2 = 1, A3 = 2, R3 = −2, A4 = 2, R4 = 2, A5 = 3, R5 = 0. On some of these time steps the " case may have occurred, causing an action to be selected at random. On which time steps did this deﬁnitely occur? On which time steps could this possibly have occurred?


n which time steps did this deﬁnitely occur? On which time steps could this possibly have occurred? ⇤ Exercise 2.3 In the comparison shown in Figure 2.2, which method will perform best in the long run in terms of cumulative reward and probability of selecting the best action? How much better will it be? Express your answer quantitatively. ⇤ 2.4 Incremental Implementation The action-value methods we have discussed so far all estimate action values as sample averages of observed rewards. We now turn to the question of how these averages can be computed in a computationally eﬃcient manner, in particular, with constant memory and constant per-time-step computation.


2.4. Incremental Implementation 31 To simplify notation we concentrate on a single action. Let Ri now denote the reward received after the ith selection of this action, and let Qn denote the estimate of its action value after it has been selected n −1 times, which we can now write simply as Qn .= R1 + R2 + · · · + Rn−1 n −1 . The obvious implementation would be to maintain a record of all the rewards and then perform this computation whenever the estimated value was needed. However, if this is done, then the memory and computational requirements would grow over time as more rewards are seen. Each additional reward would require additional memory to store it and additional computation to compute the sum in the numerator. As you might suspect, this is not really necessary.


omputation to compute the sum in the numerator. As you might suspect, this is not really necessary. It is easy to devise incremental formulas for updating averages with small, constant computation required to process each new reward. Given Qn and the nth reward, Rn, the new average of all n rewards can be computed by Qn+1 = 1 n n X i=1 Ri = 1 n Rn + n−1 X i=1 Ri ! = 1 n Rn + (n −1) 1 n −1 n−1 X i=1 Ri ! = 1 n ⇣ Rn + (n −1)Qn ⌘ = 1 n ⇣ Rn + nQn −Qn ⌘ = Qn + 1 n h Rn −Qn i , (2.3) which holds even for n = 1, obtaining Q2 = R1 for arbitrary Q1. This implementation requires memory only for Qn and n, and only the small computation (2.3) for each new reward. This update rule (2.3) is of a form that occurs frequently throughout this book.


r each new reward. This update rule (2.3) is of a form that occurs frequently throughout this book. The general form is NewEstimate OldEstimate + StepSize h Target −OldEstimate i . (2.4) The expression ⇥ Target−OldEstimate ⇤ is an error in the estimate. It is reduced by taking a step toward the “Target.” The target is presumed to indicate a desirable direction in which to move, though it may be noisy. In the case above, for example, the target is the nth reward. Note that the step-size parameter (StepSize) used in the incremental method (2.3) changes from time step to time step. In processing the nth reward for action a, the


32 Chapter 2: Multi-armed Bandits method uses the step-size parameter 1 n. In this book we denote the step-size parameter by ↵or, more generally, by ↵t(a). Pseudocode for a complete bandit algorithm using incrementally computed sample averages and "-greedy action selection is shown in the box below. The function bandit(a) is assumed to take an action and return a corresponding reward. A simple bandit algorithm Initialize, for a = 1 to k: Q(a) 0 N(a) 0 Loop forever: A ⇢argmaxa Q(a) with probability 1 −" (breaking ties randomly) a random action with probability " R bandit(A) N(A) N(A) + 1 Q(A) Q(A) + 1 N(A) ⇥ R −Q(A) ⇤ 2.5 Tracking a Nonstationary Problem The averaging methods discussed so far are appropriate for stationary bandit problems, that is, for bandit problems in which the reward pr


are appropriate for stationary bandit problems, that is, for bandit problems in which the reward probabilities do not change over time. As noted earlier, we often encounter reinforcement learning problems that are e↵ectively nonstationary. In such cases it makes sense to give more weight to recent rewards than to long-past rewards. One of the most popular ways of doing this is to use a constant step-size parameter. For example, the incremental update rule (2.3) for updating an average Qn of the n −1 past rewards is modiﬁed to be Qn+1 .= Qn + ↵ h Rn −Qn i , (2.5) where the step-size parameter ↵2 (0, 1] is constant. This results in Qn+1 being a weighted average of past rewards and the initial estimate Q1: Qn+1 = Qn + ↵ h Rn −Qn i = ↵Rn + (1 −↵)Qn = ↵Rn + (1 −↵) [↵Rn−1 + (1 −↵)Qn−1] = ↵Rn +


estimate Q1: Qn+1 = Qn + ↵ h Rn −Qn i = ↵Rn + (1 −↵)Qn = ↵Rn + (1 −↵) [↵Rn−1 + (1 −↵)Qn−1] = ↵Rn + (1 −↵)↵Rn−1 + (1 −↵)2Qn−1 = ↵Rn + (1 −↵)↵Rn−1 + (1 −↵)2↵Rn−2 + · · · + (1 −↵)n−1↵R1 + (1 −↵)nQ1 = (1 −↵)nQ1 + n X i=1 ↵(1 −↵)n−iRi. (2.6)


2.5. Tracking a Nonstationary Problem 33 We call this a weighted average because the sum of the weights is (1 −↵)n + Pn i=1 ↵(1 − ↵)n−i = 1, as you can check for yourself. Note that the weight, ↵(1 −↵)n−i, given to the reward Ri depends on how many rewards ago, n −i, it was observed. The quantity 1 −↵ is less than 1, and thus the weight given to Ri decreases as the number of intervening rewards increases. In fact, the weight decays exponentially according to the exponent on 1 −↵. (If 1 −↵= 0, then all the weight goes on the very last reward, Rn, because of the convention that 00 = 1.) Accordingly, this is sometimes called an exponential recency-weighted average. Sometimes it is convenient to vary the step-size parameter from step to step.


ncy-weighted average. Sometimes it is convenient to vary the step-size parameter from step to step. Let ↵n(a) denote the step-size parameter used to process the reward received after the nth selection of action a. As we have noted, the choice ↵n(a) = 1 n results in the sample-average method, which is guaranteed to converge to the true action values by the law of large numbers. But of course convergence is not guaranteed for all choices of the sequence {↵n(a)}. A well-known result in stochastic approximation theory gives us the conditions required to assure convergence with probability 1: 1 X n=1 ↵n(a) = 1 and 1 X n=1 ↵2 n(a) < 1. (2.7) The ﬁrst condition is required to guarantee that the steps are large enough to eventually overcome any initial conditions or random ﬂuctuations.


hat the steps are large enough to eventually overcome any initial conditions or random ﬂuctuations. The second condition guarantees that eventually the steps become small enough to assure convergence. Note that both convergence conditions are met for the sample-average case, ↵n(a) = 1 n, but not for the case of constant step-size parameter, ↵n(a) = ↵. In the latter case, the second condition is not met, indicating that the estimates never completely converge but continue to vary in response to the most recently received rewards. As we mentioned above, this is actually desirable in a nonstationary environment, and problems that are e↵ectively nonstationary are the most common in reinforcement learning.


ment, and problems that are e↵ectively nonstationary are the most common in reinforcement learning. In addition, sequences of step-size parameters that meet the conditions (2.7) often converge very slowly or need considerable tuning in order to obtain a satisfactory convergence rate. Although sequences of step-size parameters that meet these convergence conditions are often used in theoretical work, they are seldom used in applications and empirical research. Exercise 2.4 If the step-size parameters, ↵n, are not constant, then the estimate Qn is a weighted average of previously received rewards with a weighting di↵erent from that given by (2.6). What is the weighting on each prior reward for the general case, analogous to (2.6), in terms of the sequence of step-size parameters?


reward for the general case, analogous to (2.6), in terms of the sequence of step-size parameters? ⇤ Exercise 2.5 (programming) Design and conduct an experiment to demonstrate the diﬃculties that sample-average methods have for nonstationary problems. Use a modiﬁed version of the 10-armed testbed in which all the q⇤(a) start out equal and then take independent random walks (say by adding a normally distributed increment with mean 0 and standard deviation 0.01 to all the q⇤(a) on each step). Prepare plots like Figure 2.2 for an action-value method using sample averages, incrementally computed, and another action-value method using a constant step-size parameter, ↵= 0.1. Use " = 0.1 and longer runs, say of 10,000 steps. ⇤


34 Chapter 2: Multi-armed Bandits 2.6 Optimistic Initial Values All the methods we have discussed so far are dependent to some extent on the initial action-value estimates, Q1(a). In the language of statistics, these methods are biased by their initial estimates. For the sample-average methods, the bias disappears once all actions have been selected at least once, but for methods with constant ↵, the bias is permanent, though decreasing over time as given by (2.6). In practice, this kind of bias is usually not a problem and can sometimes be very helpful. The downside is that the initial estimates become, in e↵ect, a set of parameters that must be picked by the user, if only to set them all to zero.


me, in e↵ect, a set of parameters that must be picked by the user, if only to set them all to zero. The upside is that they provide an easy way to supply some prior knowledge about what level of rewards can be expected. Initial action values can also be used as a simple way to encourage exploration. Suppose that instead of setting the initial action values to zero, as we did in the 10-armed testbed, we set them all to +5. Recall that the q⇤(a) in this problem are selected from a normal distribution with mean 0 and variance 1. An initial estimate of +5 is thus wildly optimistic. But this optimism encourages action-value methods to explore. Whichever actions are initially selected, the reward is less than the starting estimates; the learner switches to other actions, being “disappointed” wit


is less than the starting estimates; the learner switches to other actions, being “disappointed” with the rewards it is receiving. The result is that all actions are tried several times before the value estimates converge. The system does a fair amount of exploration even if greedy actions are selected all the time. Figure 2.3 shows the performance on the 10-armed bandit testbed of a greedy method using Q1(a) = +5, for all a. For comparison, also shown is an "-greedy method with Q1(a) = 0. Initially, the optimistic method performs worse because it explores more, but eventually it performs better because its exploration decreases with time. We call this technique for encouraging exploration optimistic initial values.


decreases with time. We call this technique for encouraging exploration optimistic initial values. We regard it as a simple trick that can be quite e↵ective on stationary problems, but it is far from being a generally useful approach to encouraging exploration. For example, it is not well suited to nonstationary problems because its drive for exploration is inherently 0% 20% 40% 60% 80% 100% % Optimal action 0 200 400 600 800 1000 Plays optimistic, greedy Q0 = 5, = 0 realistic, -greedy Q0 = 0, = 0.1 1 1 Steps 1 Optimistic, greedy Q1 =5, "=0 Realistic, -greedy " Q1 =0, "=0.1 Figure 2.3: The e↵ect of optimistic initial action-value estimates on the 10-armed testbed. Both methods used a constant step-size parameter, ↵= 0.1.


2.7. Upper-Conﬁdence-Bound Action Selection 35 temporary. If the task changes, creating a renewed need for exploration, this method cannot help. Indeed, any method that focuses on the initial conditions in any special way is unlikely to help with the general nonstationary case. The beginning of time occurs only once, and thus we should not focus on it too much. This criticism applies as well to the sample-average methods, which also treat the beginning of time as a special event, averaging all subsequent rewards with equal weights. Nevertheless, all of these methods are very simple, and one of them—or some simple combination of them—is often adequate in practice. In the rest of this book we make frequent use of several of these simple exploration techniques.


e. In the rest of this book we make frequent use of several of these simple exploration techniques. Exercise 2.6: Mysterious Spikes The results shown in Figure 2.3 should be quite reliable because they are averages over 2000 individual, randomly chosen 10-armed bandit tasks. Why, then, are there oscillations and spikes in the early part of the curve for the optimistic method? In other words, what might make this method perform particularly better or worse, on average, on particular early steps? ⇤ Exercise 2.7: Unbiased Constant-Step-Size Trick In most of this chapter we have used sample averages to estimate action values because sample averages do not produce the initial bias that constant step sizes do (see the analysis leading to (2.6)).


es do not produce the initial bias that constant step sizes do (see the analysis leading to (2.6)). However, sample averages are not a completely satisfactory solution because they may perform poorly on nonstationary problems. Is it possible to avoid the bias of constant step sizes while retaining their advantages on nonstationary problems? One way is to use a step size of βn .= ↵/¯on, (2.8) to process the nth reward for a particular action, where ↵> 0 is a conventional constant step size, and ¯on is a trace of one that starts at 0: ¯on .= ¯on−1 + ↵(1 −¯on−1), for n > 0, with ¯o0 .= 0. (2.9) Carry out an analysis like that in (2.6) to show that Qn is an exponential recency-weighted average without initial bias.


like that in (2.6) to show that Qn is an exponential recency-weighted average without initial bias. ⇤ 2.7 Upper-Conﬁdence-Bound Action Selection Exploration is needed because there is always uncertainty about the accuracy of the action-value estimates. The greedy actions are those that look best at present, but some of the other actions may actually be better. "-greedy action selection forces the non-greedy actions to be tried, but indiscriminately, with no preference for those that are nearly greedy or particularly uncertain. It would be better to select among the non-greedy actions according to their potential for actually being optimal, taking into account both how close their estimates are to being maximal and the uncertainties in those estimates.


count both how close their estimates are to being maximal and the uncertainties in those estimates. One e↵ective way of doing this is to select actions according to At .= argmax a " Qt(a) + c s ln t Nt(a) # , (2.10) where ln t denotes the natural logarithm of t (the number that e ⇡2.71828 would have to be raised to in order to equal t), Nt(a) denotes the number of times that action a has


36 Chapter 2: Multi-armed Bandits been selected prior to time t (the denominator in (2.1)), and the number c > 0 controls the degree of exploration. If Nt(a) = 0, then a is considered to be a maximizing action. The idea of this upper conﬁdence bound (UCB) action selection is that the square-root term is a measure of the uncertainty or variance in the estimate of a’s value. The quantity being max’ed over is thus a sort of upper bound on the possible true value of action a, with c determining the conﬁdence level. Each time a is selected the uncertainty is presumably reduced: Nt(a) increments, and, as it appears in the denominator, the uncertainty term decreases. On the other hand, each time an action other than a is selected, t increases but Nt(a) does not; because t appears in the numerator


action other than a is selected, t increases but Nt(a) does not; because t appears in the numerator, the uncertainty estimate increases. The use of the natural logarithm means that the increases get smaller over time, but are unbounded; all actions will eventually be selected, but actions with lower value estimates, or that have already been selected frequently, will be selected with decreasing frequency over time. Results with UCB on the 10-armed testbed are shown in Figure 2.4. UCB often performs well, as shown here, but is more diﬃcult than "-greedy to extend beyond bandits to the more general reinforcement learning settings considered in the rest of this book. One diﬃculty is in dealing with nonstationary problems; methods more complex than those presented in Section 2.5 would be need


with nonstationary problems; methods more complex than those presented in Section 2.5 would be needed. Another diﬃculty is dealing with large state spaces, particularly when using function approximation as developed in Part II of this book. In these more advanced settings the idea of UCB action selection is usually not practical. 1 250 500 750 1000 0 0.5 1 1.5 휀-greedy 휀 = 0.1 UCB c = 2 Average reward Steps Figure 2.4: Average performance of UCB action selection on the 10-armed testbed. As shown, UCB generally performs better than "-greedy action selection, except in the ﬁrst k steps, when it selects randomly among the as-yet-untried actions. Exercise 2.8: UCB Spikes In Figure 2.4 the UCB algorithm shows a distinct spike in performance on the 11th step.


UCB Spikes In Figure 2.4 the UCB algorithm shows a distinct spike in performance on the 11th step. Why is this? Note that for your answer to be fully satisfactory it must explain both why the reward increases on the 11th step and why it decreases on the subsequent steps. Hint: If c = 1, then the spike is less prominent. ⇤


2.8. Gradient Bandit Algorithms 37 2.8 Gradient Bandit Algorithms So far in this chapter we have considered methods that estimate action values and use those estimates to select actions. This is often a good approach, but it is not the only one possible. In this section we consider learning a numerical preference for each action a, which we denote Ht(a) 2 R. The larger the preference, the more often that action is taken, but the preference has no interpretation in terms of reward. Only the relative preference of one action over another is important; if we add 1000 to all the action preferences there is no e↵ect on the action probabilities, which are determined according to a soft-max distribution (i.e., Gibbs or Boltzmann distribution) as follows: Pr{At =a} .= eHt(a) Pk b=1 eHt(b) .= ⇡t(a)


ution (i.e., Gibbs or Boltzmann distribution) as follows: Pr{At =a} .= eHt(a) Pk b=1 eHt(b) .= ⇡t(a), (2.11) where here we have also introduced a useful new notation, ⇡t(a), for the probability of taking action a at time t. Initially all action preferences are the same (e.g., H1(a) = 0, for all a) so that all actions have an equal probability of being selected. Exercise 2.9 Show that in the case of two actions, the soft-max distribution is the same as that given by the logistic, or sigmoid, function often used in statistics and artiﬁcial neural networks. ⇤ There is a natural learning algorithm for soft-max action preferences based on the idea of stochastic gradient ascent. On each step, after selecting action At and receiving the reward Rt, the action preferences are updated by: Ht+1(At) .


r selecting action At and receiving the reward Rt, the action preferences are updated by: Ht+1(At) .= Ht(At) + ↵ # Rt −¯Rt $# 1 −⇡t(At) $ , and Ht+1(a) .= Ht(a) −↵ # Rt −¯Rt $ ⇡t(a), for all a 6= At, (2.12) where ↵> 0 is a step-size parameter, and ¯Rt 2 R is the average of the rewards up to but not including time t (with ¯R1 .= R1), which can be computed incrementally as described in Section 2.4 (or Section 2.5 if the problem is nonstationary).1 The ¯Rt term serves as a baseline with which the reward is compared. If the reward is higher than the baseline, then the probability of taking At in the future is increased, and if the reward is below baseline, then the probability is decreased. The non-selected actions move in the opposite direction.


seline, then the probability is decreased. The non-selected actions move in the opposite direction. Figure 2.5 shows results with the gradient bandit algorithm on a variant of the 10armed testbed in which the true expected rewards were selected according to a normal distribution with a mean of +4 instead of zero (and with unit variance as before). This shifting up of all the rewards has absolutely no e↵ect on the gradient bandit algorithm because of the reward baseline term, which instantaneously adapts to the new level. But if the baseline were omitted (that is, if ¯Rt was taken to be constant zero in (2.12)), then performance would be signiﬁcantly degraded, as shown in the ﬁgure. 1In the empirical results in this chapter, the baseline ¯Rt also included Rt.