38 Chapter 2: Multi-armed Bandits % Optimal action Steps α = 0.1 100% 80% 60% 40% 20% 0% α = 0.4 α = 0.1 α = 0.4 without baseline with baseline 1 250 500 750 1000 Figure 2.5: Average performance of the gradient bandit algorithm with and without a reward baseline on the 10-armed testbed when the q⇤(a) are chosen to be near +4 rather than near zero. The Bandit Gradient Algorithm as Stochastic Gradient Ascent One can gain a deeper insight into the gradient bandit algorithm by understanding it as a stochastic approximation to gradient ascent. In exact gradient ascent, each action preference Ht(a) would be incremented in proportion to the increment’s e↵ect on performance: Ht+1(a) .= Ht(a) + ↵@E[Rt] @Ht(a) , (2.13) where the measure of performance here is the expected reward: E[Rt] = X x ⇡t(x)q⇤


t] @Ht(a) , (2.13) where the measure of performance here is the expected reward: E[Rt] = X x ⇡t(x)q⇤(x), and the measure of the increment’s e↵ect is the partial derivative of this performance measure with respect to the action preference. Of course, it is not possible to implement gradient ascent exactly in our case because by assumption we do not know the q⇤(x), but in fact the updates of our algorithm (2.12) are equal to (2.13) in expected value, making the algorithm an instance of stochastic gradient ascent. The calculations showing this require only beginning calculus, but take several


2.8. Gradient Bandit Algorithms 39 steps. First we take a closer look at the exact performance gradient: @E[Rt] @Ht(a) = @ @Ht(a) "X x ⇡t(x)q⇤(x) # = X x q⇤(x) @⇡t(x) @Ht(a) = X x # q⇤(x) −Bt $ @⇡t(x) @Ht(a), where Bt, called the baseline, can be any scalar that does not depend on x. We can include a baseline here without changing the equality because the gradient sums to zero over all the actions, P x @ ⇡t(x) @Ht(a) = 0. As Ht(a) is changed, some actions’ probabilities go up and some go down, but the sum of the changes must be zero because the sum of the probabilities is always one. Next we multiply each term of the sum by ⇡t(x)/⇡t(x): @E[Rt] @Ht(a) = X x ⇡t(x) # q⇤(x) −Bt $ @⇡t(x) @Ht(a)/⇡t(x).


y each term of the sum by ⇡t(x)/⇡t(x): @E[Rt] @Ht(a) = X x ⇡t(x) # q⇤(x) −Bt $ @⇡t(x) @Ht(a)/⇡t(x). The equation is now in the form of an expectation, summing over all possible values x of the random variable At, then multiplying by the probability of taking those values. Thus: = E # q⇤(At) −Bt $@⇡t(At) @Ht(a) /⇡t(At) 3 = E # Rt −¯Rt $@⇡t(At) @Ht(a) /⇡t(At) 3 , where here we have chosen the baseline Bt = ¯Rt and substituted Rt for q⇤(At), which is permitted because E[Rt|At] = q⇤(At). Shortly we will establish that @ ⇡t(x) @Ht(a) = ⇡t(x) # a=x −⇡t(a) $ , where a=x is deﬁned to be 1 if a = x, else 0. Assuming that for now, we have = E ⇥# Rt −¯Rt $ ⇡t(At) # a=At −⇡t(a) $ /⇡t(At) ⇤ = E ⇥# Rt −¯Rt $# a=At −⇡t(a) $⇤ .


r now, we have = E ⇥# Rt −¯Rt $ ⇡t(At) # a=At −⇡t(a) $ /⇡t(At) ⇤ = E ⇥# Rt −¯Rt $# a=At −⇡t(a) $⇤ . Recall that our plan has been to write the performance gradient as an expectation of something that we can sample on each step, as we have just done, and then update on each step in proportion to the sample. Substituting a sample of the expectation above for the performance gradient in (2.13) yields: Ht+1(a) = Ht(a) + ↵ # Rt −¯Rt $# a=At −⇡t(a) $ , for all a, which you may recognize as being equivalent to our original algorithm (2.12).


40 Chapter 2: Multi-armed Bandits Thus it remains only to show that @ ⇡t(x) @Ht(a) = ⇡t(x) # a=x −⇡t(a) $ , as we assumed. Recall the standard quotient rule for derivatives: @ @x f(x) g(x) 3 = @f(x) @x g(x) −f(x) @g(x) @x g(x)2 . Using this, we can write @⇡t(x) @Ht(a) = @ @Ht(a) ⇡t(x) = @ @Ht(a) " eHt(x) Pk y=1 eHt(y) # = @eHt(x) @Ht(a) Pk y=1 eHt(y) −eHt(x) @ Pk y=1 eHt(y) @Ht(a) ⇣Pk y=1 eHt(y) ⌘2 (by the quotient rule) = a=xeHt(x) Pk y=1 eHt(y) −eHt(x)eHt(a) ⇣Pk y=1 eHt(y) ⌘2 (because @ex @x = ex) = a=xeHt(x) Pk y=1 eHt(y) − eHt(x)eHt(a) ⇣Pk y=1 eHt(y) ⌘2 = a=x⇡t(x) −⇡t(x)⇡t(a) = ⇡t(x) # a=x −⇡t(a) $ . Q.E.D. We have just shown that the expected update of the gradient bandit algorithm is equal to the gradient of expected reward, and thus that the algorithm is an instance of stochastic g


equal to the gradient of expected reward, and thus that the algorithm is an instance of stochastic gradient ascent. This assures us that the algorithm has robust convergence properties. Note that we did not require any properties of the reward baseline other than that it does not depend on the selected action. For example, we could have set it to zero, or to 1000, and the algorithm would still be an instance of stochastic gradient ascent. The choice of the baseline does not a↵ect the expected update of the algorithm, but it does a↵ect the variance of the update and thus the rate of convergence (as shown, for example, in Figure 2.5). Choosing it as the average of the rewards may not be the very best, but it is simple and works well in practice.


2.9. Associative Search (Contextual Bandits) 41 2.9 Associative Search (Contextual Bandits) So far in this chapter we have considered only nonassociative tasks, that is, tasks in which there is no need to associate di↵erent actions with di↵erent situations. In these tasks the learner either tries to ﬁnd a single best action when the task is stationary, or tries to track the best action as it changes over time when the task is nonstationary. However, in a general reinforcement learning task there is more than one situation, and the goal is to learn a policy: a mapping from situations to the actions that are best in those situations. To set the stage for the full problem, we brieﬂy discuss the simplest way in which nonassociative tasks extend to the associative setting.


we brieﬂy discuss the simplest way in which nonassociative tasks extend to the associative setting. As an example, suppose there are several di↵erent k-armed bandit tasks, and that on each step you confront one of these chosen at random. Thus, the bandit task changes randomly from step to step. If the probabilities with which each task is selected for you do not change over time, this would appear as a single stationary k-armed bandit task, and you could use one of the methods described in this chapter. Now suppose, however, that when a bandit task is selected for you, you are given some distinctive clue about its identity (but not its action values). Maybe you are facing an actual slot machine that changes the color of its display as it changes its action values.


acing an actual slot machine that changes the color of its display as it changes its action values. Now you can learn a policy associating each task, signaled by the color you see, with the best action to take when facing that task—for instance, if red, select arm 1; if green, select arm 2. With the right policy you can usually do much better than you could in the absence of any information distinguishing one bandit task from another. This is an example of an associative search task, so called because it involves both trial-and-error learning to search for the best actions, and association of these actions with the situations in which they are best. Associative search tasks are often now called contextual bandits in the literature.


they are best. Associative search tasks are often now called contextual bandits in the literature. Associative search tasks are intermediate between the k-armed bandit problem and the full reinforcement learning problem. They are like the full reinforcement learning problem in that they involve learning a policy, but they are also like our version of the k-armed bandit problem in that each action a↵ects only the immediate reward. If actions are allowed to a↵ect the next situation as well as the reward, then we have the full reinforcement learning problem. We present this problem in the next chapter and consider its ramiﬁcations throughout the rest of the book. Exercise 2.10 Suppose you face a 2-armed bandit task whose true action values change randomly from time step to time step.


ou face a 2-armed bandit task whose true action values change randomly from time step to time step. Speciﬁcally, suppose that, for any time step, the true values of actions 1 and 2 are respectively 10 and 20 with probability 0.5 (case A), and 90 and 80 with probability 0.5 (case B). If you are not able to tell which case you face at any step, what is the best expected reward you can achieve and how should you behave to achieve it? Now suppose that on each step you are told whether you are facing case A or case B (although you still don’t know the true action values). This is an associative search task. What is the best expected reward you can achieve in this task, and how should you behave to achieve it? ⇤


42 Chapter 2: Multi-armed Bandits 2.10 Summary We have presented in this chapter several simple ways of balancing exploration and exploitation. The "-greedy methods choose randomly a small fraction of the time, whereas UCB methods choose deterministically but achieve exploration by subtly favoring at each step the actions that have so far received fewer samples. Gradient bandit algorithms estimate not action values, but action preferences, and favor the more preferred actions in a graded, probabilistic manner using a soft-max distribution. The simple expedient of initializing estimates optimistically causes even greedy methods to explore signiﬁcantly. It is natural to ask which of these methods is best.


s even greedy methods to explore signiﬁcantly. It is natural to ask which of these methods is best. Although this is a diﬃcult question to answer in general, we can certainly run them all on the 10-armed testbed that we have used throughout this chapter and compare their performances. A complication is that they all have a parameter; to get a meaningful comparison we have to consider their performance as a function of their parameter. Our graphs so far have shown the course of learning over time for each algorithm and parameter setting, to produce a learning curve for that algorithm and parameter setting. If we plotted learning curves for all algorithms and all parameter settings, then the graph would be too complex and crowded to make clear comparisons.


all parameter settings, then the graph would be too complex and crowded to make clear comparisons. Instead we summarize a complete learning curve by its average value over the 1000 steps; this value is proportional to the area under the learning curve. Figure 2.6 shows this measure for the various bandit algorithms from this chapter, each as a function of its own parameter shown on a single scale on the x-axis. This kind of graph is called a parameter study. Note that the parameter values are varied by factors of two and presented on a log scale. Note also the characteristic inverted-U shapes of each algorithm’s performance; all the algorithms perform best at an intermediate value of their parameter, neither too large nor too small.


gorithms perform best at an intermediate value of their parameter, neither too large nor too small. In assessing Average reward over ﬁrst 1000 steps 1.5 1.4 1.3 1.2 1.1 1 휀-greedy UCB gradient bandit greedy with optimistic initialization α = 0.1 1 2 4 1/2 1/4 1/8 1/16 1/32 1/64 1/128 " ↵c Q0 Figure 2.6: A parameter study of the various bandit algorithms presented in this chapter. Each point is the average reward obtained over 1000 steps with a particular algorithm at a particular setting of its parameter.


2.10. Summary 43 a method, we should attend not just to how well it does at its best parameter setting, but also to how sensitive it is to its parameter value. All of these algorithms are fairly insensitive, performing well over a range of parameter values varying by about an order of magnitude. Overall, on this problem, UCB seems to perform best. Despite their simplicity, in our opinion the methods presented in this chapter can fairly be considered the state of the art. There are more sophisticated methods, but their complexity and assumptions make them impractical for the full reinforcement learning problem that is our real focus. Starting in Chapter 5 we present learning methods for solving the full reinforcement learning problem that use in part the simple methods explored in this chap


ng the full reinforcement learning problem that use in part the simple methods explored in this chapter. Although the simple methods explored in this chapter may be the best we can do at present, they are far from a fully satisfactory solution to the problem of balancing exploration and exploitation. One well-studied approach to balancing exploration and exploitation in k-armed bandit problems is to compute a special kind of action value called a Gittins index. In certain important special cases, this computation is tractable and leads directly to optimal solutions, although it does require complete knowledge of the prior distribution of possible problems, which we generally assume is not available.


owledge of the prior distribution of possible problems, which we generally assume is not available. In addition, neither the theory nor the computational tractability of this approach appear to generalize to the full reinforcement learning problem that we consider in the rest of the book. The Gittins-index approach is an instance of Bayesian methods, which assume a known initial distribution over the action values and then update the distribution exactly after each step (assuming that the true action values are stationary). In general, the update computations can be very complex, but for certain special distributions (called conjugate priors) they are easy. One possibility is to then select actions at each step according to their posterior probability of being the best action.


then select actions at each step according to their posterior probability of being the best action. This method, sometimes called posterior sampling or Thompson sampling, often performs similarly to the best of the distribution-free methods we have presented in this chapter. In the Bayesian setting it is even conceivable to compute the optimal balance between exploration and exploitation. One can compute for any possible action the probability of each possible immediate reward and the resultant posterior distributions over action values. This evolving distribution becomes the information state of the problem. Given a horizon, say of 1000 steps, one can consider all possible actions, all possible resulting rewards, all possible next actions, all next rewards, and so on for all 1000 steps.


sible resulting rewards, all possible next actions, all next rewards, and so on for all 1000 steps. Given the assumptions, the rewards and probabilities of each possible chain of events can be determined; one need only pick the best. But the tree of possibilities grows extremely rapidly; even if there were only two actions and two rewards, the tree would have 22000 leaves. It is generally not feasible to perform this immense computation exactly, but perhaps it could be approximated eﬃciently. This approach would e↵ectively turn the bandit problem into an instance of the full reinforcement learning problem. In the end, we may be able to use approximate reinforcement learning methods such as those presented in Part II of this book to approach this optimal solution.


learning methods such as those presented in Part II of this book to approach this optimal solution. But that is a topic for research and beyond the scope of this introductory book.


44 Chapter 2: Multi-armed Bandits Exercise 2.11 (programming) Make a ﬁgure analogous to Figure 2.6 for the nonstationary case outlined in Exercise 2.5. Include the constant-step-size "-greedy algorithm with ↵=0.1. Use runs of 200,000 steps and, as a performance measure for each algorithm and parameter setting, use the average reward over the last 100,000 steps. ⇤ Bibliographical and Historical Remarks 2.1 Bandit problems have been studied in statistics, engineering, and psychology. In statistics, bandit problems fall under the heading “sequential design of experiments,” introduced by Thompson (1933, 1934) and Robbins (1952), and studied by Bellman (1956). Berry and Fristedt (1985) provide an extensive treatment of bandit problems from the perspective of statistics.


istedt (1985) provide an extensive treatment of bandit problems from the perspective of statistics. Narendra and Thathachar (1989) treat bandit problems from the engineering perspective, providing a good discussion of the various theoretical traditions that have focused on them. In psychology, bandit problems have played roles in statistical learning theory (e.g., Bush and Mosteller, 1955; Estes, 1950). The term greedy is often used in the heuristic search literature (e.g., Pearl, 1984). The conﬂict between exploration and exploitation is known in control engineering as the conﬂict between identiﬁcation (or estimation) and control (e.g., Witten, 1976b). Feldbaum (1965) called it the dual control problem, referring to the need to solve the two problems of identiﬁcation and control simultane


trol problem, referring to the need to solve the two problems of identiﬁcation and control simultaneously when trying to control a system under uncertainty. In discussing aspects of genetic algorithms, Holland (1975) emphasized the importance of this conﬂict, referring to it as the conﬂict between the need to exploit and the need for new information. 2.2 Action-value methods for our k-armed bandit problem were ﬁrst proposed by Thathachar and Sastry (1985). These are often called estimator algorithms in the learning automata literature. The term action value is due to Watkins (1989). The ﬁrst to use "-greedy methods may also have been Watkins (1989, p. 187), but the idea is so simple that some earlier use seems likely.


lso have been Watkins (1989, p. 187), but the idea is so simple that some earlier use seems likely. 2.4–5 This material falls under the general heading of stochastic iterative algorithms, which is well covered by Bertsekas and Tsitsiklis (1996). 2.6 Optimistic initialization was used in reinforcement learning by Sutton (1996). 2.7 Early work on using estimates of the upper conﬁdence bound to select actions was done by Lai and Robbins (1985), Kaelbling (1993b), and Agrawal (1995). The UCB algorithm we present here is called UCB1 in the literature and was ﬁrst developed by Auer, Cesa-Bianchi and Fischer (2002). 2.8 Gradient bandit algorithms are a special case of the gradient-based reinforcement learning algorithms introduced by Williams (1992) that later developed into the actor–critic and


nt learning algorithms introduced by Williams (1992) that later developed into the actor–critic and policy-gradient algorithms that we treat later in this book. Our development here was inﬂuenced by that by Balaraman Ravindran (personal


2.10. Summary 45 communication). Further discussion of the choice of baseline is provided by Greensmith, Bartlett, and Baxter (2002, 2004) and by Dick (2015). Early systematic studies of algorithms like this were done by Sutton (1984). The term soft-max for the action selection rule (2.11) is due to Bridle (1990). This rule appears to have been ﬁrst proposed by Luce (1959). 2.9 The term associative search and the corresponding problem were introduced by Barto, Sutton, and Brouwer (1981). The term associative reinforcement learning has also been used for associative search (Barto and Anandan, 1985), but we prefer to reserve that term as a synonym for the full reinforcement learning problem (as in Sutton, 1984).


to reserve that term as a synonym for the full reinforcement learning problem (as in Sutton, 1984). (And, as we noted, the modern literature also uses the term “contextual bandits” for this problem.) We note that Thorndike’s Law of E↵ect (quoted in Chapter 1) describes associative search by referring to the formation of associative links between situations (states) and actions. According to the terminology of operant, or instrumental, conditioning (e.g., Skinner, 1938), a discriminative stimulus is a stimulus that signals the presence of a particular reinforcement contingency. In our terms, di↵erent discriminative stimuli correspond to di↵erent states. 2.10 Bellman (1956) was the ﬁrst to show how dynamic programming could be used to compute the optimal balance between exploration and explo


w how dynamic programming could be used to compute the optimal balance between exploration and exploitation within a Bayesian formulation of the problem. The Gittins index approach is due to Gittins and Jones (1974). Du↵(1995) showed how it is possible to learn Gittins indices for bandit problems through reinforcement learning. The survey by Kumar (1985) provides a good discussion of Bayesian and non-Bayesian approaches to these problems. The term information state comes from the literature on partially observable MDPs; see, for example, Lovejoy (1991). Other theoretical research focuses on the eﬃciency of exploration, usually expressed as how quickly an algorithm can approach an optimal decision-making policy.


tion, usually expressed as how quickly an algorithm can approach an optimal decision-making policy. One way to formalize exploration eﬃciency is by adapting to reinforcement learning the notion of sample complexity for a supervised learning algorithm, which is the number of training examples the algorithm needs to attain a desired degree of accuracy in learning the target function. A deﬁnition of the sample complexity of exploration for a reinforcement learning algorithm is the number of time steps in which the algorithm does not select near-optimal actions (Kakade, 2003). Li (2012) discusses this and several other approaches in a survey of theoretical approaches to exploration eﬃciency in reinforcement learning. A thorough modern treatment of Thompson sampling is provided by Russo et al.


einforcement learning. A thorough modern treatment of Thompson sampling is provided by Russo et al. (2018).


Chapter 3 Finite Markov Decision Processes In this chapter we introduce the formal problem of ﬁnite Markov decision processes, or ﬁnite MDPs, which we try to solve in the rest of the book. This problem involves evaluative feedback, as in bandits, but also an associative aspect—choosing di↵erent actions in di↵erent situations. MDPs are a classical formalization of sequential decision making, where actions inﬂuence not just immediate rewards, but also subsequent situations, or states, and through those future rewards. Thus MDPs involve delayed reward and the need to trade o↵immediate and delayed reward. Whereas in bandit problems we estimated the value q⇤(a) of each action a, in MDPs we estimate the value q⇤(s, a) of each action a in each state s, or we estimate the value v⇤(s) of each state


te the value q⇤(s, a) of each action a in each state s, or we estimate the value v⇤(s) of each state given optimal action selections. These state-dependent quantities are essential to accurately assigning credit for long-term consequences to individual action selections. MDPs are a mathematically idealized form of the reinforcement learning problem for which precise theoretical statements can be made. We introduce key elements of the problem’s mathematical structure, such as returns, value functions, and Bellman equations. We try to convey the wide range of applications that can be formulated as ﬁnite MDPs. As in all of artiﬁcial intelligence, there is a tension between breadth of applicability and mathematical tractability.


al intelligence, there is a tension between breadth of applicability and mathematical tractability. In this chapter we introduce this tension and discuss some of the trade-o↵s and challenges that it implies. Some ways in which reinforcement learning can be taken beyond MDPs are treated in Chapter 17. 3.1 The Agent–Environment Interface MDPs are meant to be a straightforward framing of the problem of learning from interaction to achieve a goal. The learner and decision maker is called the agent. The thing it interacts with, comprising everything outside the agent, is called the environment. These interact continually, the agent selecting actions and the environment responding to


48 Chapter 3: Finite Markov Decision Processes these actions and presenting new situations to the agent.1 The environment also gives rise to rewards, special numerical values that the agent seeks to maximize over time through its choice of actions. Agent Environment action At reward Rt state St Rt+1 St+1 Figure 3.1: The agent–environment interaction in a Markov decision process. More speciﬁcally, the agent and environment interact at each of a sequence of discrete time steps, t = 0, 1, 2, 3, . . ..2 At each time step t, the agent receives some representation of the environment’s state, St 2 S, and on that basis selects an action, At 2 A(s).3 One time step later, in part as a consequence of its action, the agent receives a numerical reward, Rt+1 2 R ⇢R, and ﬁnds itself in a new state, St+1.


ts action, the agent receives a numerical reward, Rt+1 2 R ⇢R, and ﬁnds itself in a new state, St+1.4 The MDP and agent together thereby give rise to a sequence or trajectory that begins like this: S0, A0, R1, S1, A1, R2, S2, A2, R3, . . . (3.1) In a ﬁnite MDP, the sets of states, actions, and rewards (S, A, and R) all have a ﬁnite number of elements. In this case, the random variables Rt and St have well deﬁned discrete probability distributions dependent only on the preceding state and action. That is, for particular values of these random variables, s0 2 S and r 2 R, there is a probability of those values occurring at time t, given particular values of the preceding state and action: p(s0, r|s, a) .= Pr{St =s0, Rt =r | St−1 =s, At−1 =a}, (3.2) for all s0, s 2 S, r 2 R, and a 2 A(s).


(s0, r|s, a) .= Pr{St =s0, Rt =r | St−1 =s, At−1 =a}, (3.2) for all s0, s 2 S, r 2 R, and a 2 A(s). The function p deﬁnes the dynamics of the MDP. The dot over the equals sign in the equation reminds us that it is a deﬁnition (in this case of the function p) rather than a fact that follows from previous deﬁnitions. The dynamics function p : S ⇥R ⇥S ⇥A ! [0, 1] is an ordinary deterministic function of four arguments. The ‘|’ in the middle of it comes from the notation for conditional probability, 1We use the terms agent, environment, and action instead of the engineers’ terms controller, controlled system (or plant), and control signal because they are meaningful to a wider audience. 2We restrict attention to discrete time to keep things as simple as possible, even though many of the ideas


rict attention to discrete time to keep things as simple as possible, even though many of the ideas can be extended to the continuous-time case (e.g., see Bertsekas and Tsitsiklis, 1996; Doya, 1996). 3To simplify notation, we sometimes assume the special case in which the action set is the same in all states and write it simply as A. 4We use Rt+1 instead of Rt to denote the reward due to At because it emphasizes that the next reward and next state, Rt+1 and St+1, are jointly determined. Unfortunately, both conventions are widely used in the literature.


3.1. The Agent–Environment Interface 49 but here it just reminds us that p speciﬁes a probability distribution for each choice of s and a, that is, that X s02S X r2R p(s0, r|s, a) = 1, for all s 2 S, a 2 A(s). (3.3) In a Markov decision process, the probabilities given by p completely characterize the environment’s dynamics. That is, the probability of each possible value for St and Rt depends on the immediately preceding state and action, St−1 and At−1, and, given them, not at all on earlier states and actions. This is best viewed as a restriction not on the decision process, but on the state. The state must include information about all aspects of the past agent–environment interaction that make a di↵erence for the future. If it does, then the state is said to have the Markov property.


at make a di↵erence for the future. If it does, then the state is said to have the Markov property. We will assume the Markov property throughout this book, though starting in Part II we will consider approximation methods that do not rely on it, and in Chapter 17 we consider how a Markov state can be eﬃciently learned and constructed from non-Markov observations. From the four-argument dynamics function, p, one can compute anything else one might want to know about the environment, such as the state-transition probabilities (which we denote, with a slight abuse of notation, as a three-argument function p : S⇥S⇥A ! [0, 1]), p(s0|s, a) .= Pr{St =s0 | St−1 =s, At−1 =a} = X r2R p(s0, r|s, a). (3.4) We can also compute the expected rewards for state–action pairs as a two-argument function r :


3.4) We can also compute the expected rewards for state–action pairs as a two-argument function r : S ⇥A ! R: r(s, a) .= E[Rt | St−1 =s, At−1 =a] = X r2R r X s02S p(s0, r|s, a), (3.5) and the expected rewards for state–action–next-state triples as a three-argument function r : S ⇥A ⇥S ! R, r(s, a, s0) .= E[Rt | St−1 =s, At−1 =a, St = s0] = X r2R r p(s0, r|s, a) p(s0|s, a) . (3.6) In this book, we usually use the four-argument p function (3.2), but each of these other notations are also occasionally convenient. The MDP framework is abstract and ﬂexible and can be applied to many di↵erent problems in many di↵erent ways. For example, the time steps need not refer to ﬁxed intervals of real time; they can refer to arbitrary successive stages of decision making and acting.


ntervals of real time; they can refer to arbitrary successive stages of decision making and acting. The actions can be low-level controls, such as the voltages applied to the motors of a robot arm, or high-level decisions, such as whether or not to have lunch or to go to graduate school. Similarly, the states can take a wide variety of forms. They can be completely determined by low-level sensations, such as direct sensor readings, or they can be more high-level and abstract, such as symbolic descriptions of objects in a room. Some of what makes up a state could be based on memory of past sensations or


50 Chapter 3: Finite Markov Decision Processes even be entirely mental or subjective. For example, an agent could be in the state of not being sure where an object is, or of having just been surprised in some clearly deﬁned sense. Similarly, some actions might be totally mental or computational. For example, some actions might control what an agent chooses to think about, or where it focuses its attention. In general, actions can be any decisions we want to learn how to make, and states can be anything we can know that might be useful in making them. In particular, the boundary between agent and environment is typically not the same as the physical boundary of a robot’s or an animal’s body. Usually, the boundary is drawn closer to the agent than that.


ary of a robot’s or an animal’s body. Usually, the boundary is drawn closer to the agent than that. For example, the motors and mechanical linkages of a robot and its sensing hardware should usually be considered parts of the environment rather than parts of the agent. Similarly, if we apply the MDP framework to a person or animal, the muscles, skeleton, and sensory organs should be considered part of the environment. Rewards, too, presumably are computed inside the physical bodies of natural and artiﬁcial learning systems, but are considered external to the agent. The general rule we follow is that anything that cannot be changed arbitrarily by the agent is considered to be outside of it and thus part of its environment.


hanged arbitrarily by the agent is considered to be outside of it and thus part of its environment. We do not assume that everything in the environment is unknown to the agent. For example, the agent often knows quite a bit about how its rewards are computed as a function of its actions and the states in which they are taken. But we always consider the reward computation to be external to the agent because it deﬁnes the task facing the agent and thus must be beyond its ability to change arbitrarily. In fact, in some cases the agent may know everything about how its environment works and still face a diﬃcult reinforcement learning task, just as we may know exactly how a puzzle like Rubik’s cube works, but still be unable to solve it.


just as we may know exactly how a puzzle like Rubik’s cube works, but still be unable to solve it. The agent–environment boundary represents the limit of the agent’s absolute control, not of its knowledge. The agent–environment boundary can be located at di↵erent places for di↵erent purposes. In a complicated robot, many di↵erent agents may be operating at once, each with its own boundary. For example, one agent may make high-level decisions which form part of the states faced by a lower-level agent that implements the high-level decisions. In practice, the agent–environment boundary is determined once one has selected particular states, actions, and rewards, and thus has identiﬁed a speciﬁc decision-making task of interest.


ar states, actions, and rewards, and thus has identiﬁed a speciﬁc decision-making task of interest. The MDP framework is a considerable abstraction of the problem of goal-directed learning from interaction. It proposes that whatever the details of the sensory, memory, and control apparatus, and whatever objective one is trying to achieve, any problem of learning goal-directed behavior can be reduced to three signals passing back and forth between an agent and its environment: one signal to represent the choices made by the agent (the actions), one signal to represent the basis on which the choices are made (the states), and one signal to deﬁne the agent’s goal (the rewards). This framework may not be suﬃcient to represent all decision-learning problems usefully, but it has proved to be wid


ay not be suﬃcient to represent all decision-learning problems usefully, but it has proved to be widely useful and applicable. Of course, the particular states and actions vary greatly from task to task, and how they are represented can strongly a↵ect performance. In reinforcement learning, as in other kinds of learning, such representational choices are at present more art than science.


3.1. The Agent–Environment Interface 51 In this book we o↵er some advice and examples regarding good ways of representing states and actions, but our primary focus is on general principles for learning how to behave once the representations have been selected. Example 3.1: Bioreactor Suppose reinforcement learning is being applied to determine moment-by-moment temperatures and stirring rates for a bioreactor (a large vat of nutrients and bacteria used to produce useful chemicals). The actions in such an application might be target temperatures and target stirring rates that are passed to lower-level control systems that, in turn, directly activate heating elements and motors to attain the targets.


control systems that, in turn, directly activate heating elements and motors to attain the targets. The states are likely to be thermocouple and other sensory readings, perhaps ﬁltered and delayed, plus symbolic inputs representing the ingredients in the vat and the target chemical. The rewards might be moment-by-moment measures of the rate at which the useful chemical is produced by the bioreactor. Notice that here each state is a list, or vector, of sensor readings and symbolic inputs, and each action is a vector consisting of a target temperature and a stirring rate. It is typical of reinforcement learning tasks to have states and actions with such structured representations. Rewards, on the other hand, are always single numbers.


ctions with such structured representations. Rewards, on the other hand, are always single numbers. Example 3.2: Pick-and-Place Robot Consider using reinforcement learning to control the motion of a robot arm in a repetitive pick-and-place task. If we want to learn movements that are fast and smooth, the learning agent will have to control the motors directly and have low-latency information about the current positions and velocities of the mechanical linkages. The actions in this case might be the voltages applied to each motor at each joint, and the states might be the latest readings of joint angles and velocities. The reward might be +1 for each object successfully picked up and placed. To encourage smooth movements, on each time step a small, negative reward could be given as a functi


To encourage smooth movements, on each time step a small, negative reward could be given as a function of the moment-to-moment jerkiness of the motion. Exercise 3.1 Devise three example tasks of your own that ﬁt into the MDP framework, identifying for each its states, actions, and rewards. Make the three examples as di↵erent from each other as possible. The framework is abstract and ﬂexible and can be applied in many di↵erent ways. Stretch its limits in some way in at least one of your examples. ⇤ Exercise 3.2 Is the MDP framework adequate to usefully represent all goal-directed learning tasks? Can you think of any clear exceptions? ⇤ Exercise 3.3 Consider the problem of driving. You could deﬁne the actions in terms of the accelerator, steering wheel, and brake, that is, where your body me


eﬁne the actions in terms of the accelerator, steering wheel, and brake, that is, where your body meets the machine. Or you could deﬁne them farther out—say, where the rubber meets the road, considering your actions to be tire torques. Or you could deﬁne them farther in—say, where your brain meets your body, the actions being muscle twitches to control your limbs. Or you could go to a really high level and say that your actions are your choices of where to drive. What is the right level, the right place to draw the line between agent and environment? On what basis is one location of the line to be preferred over another? Is there any fundamental reason for preferring one location over another, or is it a free choice? ⇤


52 Chapter 3: Finite Markov Decision Processes Example 3.3 Recycling Robot A mobile robot has the job of collecting empty soda cans in an oﬃce environment. It has sensors for detecting cans, and an arm and gripper that can pick them up and place them in an onboard bin; it runs on a rechargeable battery. The robot’s control system has components for interpreting sensory information, for navigating, and for controlling the arm and gripper. High-level decisions about how to search for cans are made by a reinforcement learning agent based on the current charge level of the battery. To make a simple example, we assume that only two charge levels can be distinguished, comprising a small state set S = {high, low}.


ume that only two charge levels can be distinguished, comprising a small state set S = {high, low}. In each state, the agent can decide whether to (1) actively search for a can for a certain period of time, (2) remain stationary and wait for someone to bring it a can, or (3) head back to its home base to recharge its battery. When the energy level is high, recharging would always be foolish, so we do not include it in the action set for this state. The action sets are then A(high) = {search, wait} and A(low) = {search, wait, recharge}. The rewards are zero most of the time, but become positive when the robot secures an empty can, or large and negative if the battery runs all the way down. The best way to ﬁnd cans is to actively search for them, but this runs down the robot’s battery, where


e best way to ﬁnd cans is to actively search for them, but this runs down the robot’s battery, whereas waiting does not. Whenever the robot is searching, the possibility exists that its battery will become depleted. In this case the robot must shut down and wait to be rescued (producing a low reward). If the energy level is high, then a period of active search can always be completed without risk of depleting the battery. A period of searching that begins with a high energy level leaves the energy level high with probability ↵and reduces it to low with probability 1 −↵. On the other hand, a period of searching undertaken when the energy level is low leaves it low with probability β and depletes the battery with probability 1 −β.


nergy level is low leaves it low with probability β and depletes the battery with probability 1 −β. In the latter case, the robot must be rescued, and the battery is then recharged back to high. Each can collected by the robot counts as a unit reward, whereas a reward of −3 results whenever the robot has to be rescued. Let rsearch and rwait, with rsearch > rwait, denote the expected number of cans the robot will collect (and hence the expected reward) while searching and while waiting respectively. Finally, suppose that no cans can be collected during a run home for recharging, and that no cans can be collected on a step in which the battery is depleted. This system is then a ﬁnite MDP, and we can write down the transition probabilities and the expected rewards, with dynamics as indicated


we can write down the transition probabilities and the expected rewards, with dynamics as indicated in the table on the left: s a s0 p(s0 |s, a) r(s, a, s0) high search high ↵ rsearch high search low 1 −↵ rsearch low search high 1 −β −3 low search low β rsearch high wait high 1 rwait high wait low 0 low wait high 0 low wait low 1 rwait low recharge high 1 0 low recharge low 0 search high low 1, 0 search recharge wait wait β, rsearch ↵, rsearch 1−↵, rsearch 1−β, −3 1, rwait 1, rwait Note that there is a row in the table for each possible combination of current state, s, action, a 2 A(s), and next state, s0. Some transitions have zero probability of occurring, so no expected reward is speciﬁed for them. Shown on the right is another useful way of


3.2. Goals and Rewards 53 summarizing the dynamics of a ﬁnite MDP, as a transition graph. There are two kinds of nodes: state nodes and action nodes. There is a state node for each possible state (a large open circle labeled by the name of the state), and an action node for each state–action pair (a small solid circle labeled by the name of the action and connected by a line to the state node). Starting in state s and taking action a moves you along the line from state node s to action node (s, a). Then the environment responds with a transition to the next state’s node via one of the arrows leaving action node (s, a). Each arrow corresponds to a triple (s, s0, a), where s0 is the next state, and we label the arrow with the transition probability, p(s0|s, a), and the expected reward for th


, and we label the arrow with the transition probability, p(s0|s, a), and the expected reward for that transition, r(s, a, s0). Note that the transition probabilities labeling the arrows leaving an action node always sum to 1. Exercise 3.4 Give a table analogous to that in Example 3.3, but for p(s0, r|s, a). It should have columns for s, a, s0, r, and p(s0, r|s, a), and a row for every 4-tuple for which p(s0, r|s, a) > 0. ⇤ 3.2 Goals and Rewards In reinforcement learning, the purpose or goal of the agent is formalized in terms of a special signal, called the reward, passing from the environment to the agent. At each time step, the reward is a simple number, Rt 2 R. Informally, the agent’s goal is to maximize the total amount of reward it receives.


number, Rt 2 R. Informally, the agent’s goal is to maximize the total amount of reward it receives. This means maximizing not immediate reward, but cumulative reward in the long run. We can clearly state this informal idea as the reward hypothesis: That all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward). The use of a reward signal to formalize the idea of a goal is one of the most distinctive features of reinforcement learning. Although formulating goals in terms of reward signals might at ﬁrst appear limiting, in practice it has proved to be ﬂexible and widely applicable. The best way to see this is to consider examples of how it has been, or could be, used.


applicable. The best way to see this is to consider examples of how it has been, or could be, used. For example, to make a robot learn to walk, researchers have provided reward on each time step proportional to the robot’s forward motion. In making a robot learn how to escape from a maze, the reward is often −1 for every time step that passes prior to escape; this encourages the agent to escape as quickly as possible. To make a robot learn to ﬁnd and collect empty soda cans for recycling, one might give it a reward of zero most of the time, and then a reward of +1 for each can collected. One might also want to give the robot negative rewards when it bumps into things or when somebody yells at it.


lso want to give the robot negative rewards when it bumps into things or when somebody yells at it. For an agent to learn to play checkers or chess, the natural rewards are +1 for winning, −1 for losing, and 0 for drawing and for all nonterminal positions. You can see what is happening in all of these examples. The agent always learns to maximize its reward. If we want it to do something for us, we must provide rewards to it in such a way that in maximizing them the agent will also achieve our goals. It


54 Chapter 3: Finite Markov Decision Processes is thus critical that the rewards we set up truly indicate what we want accomplished. In particular, the reward signal is not the place to impart to the agent prior knowledge about how to achieve what we want it to do.5 For example, a chess-playing agent should be rewarded only for actually winning, not for achieving subgoals such as taking its opponent’s pieces or aining control of the center of the board. If achieving these sorts of subgoals were rewarded, then the agent might ﬁnd a way to achieve them without achieving the real goal. For example, it might ﬁnd a way to take the opponent’s pieces even at the cost of losing the game. The reward signal is your way of communicating to the agent what you want achieved, not how you want it achieve


signal is your way of communicating to the agent what you want achieved, not how you want it achieved.6 3.3 Returns and Episodes So far we have discussed informally the objective of learning. We have said that the agent’s goal is to maximize the cumulative reward it receives in the long run. How might this be deﬁned formally? If the sequence of rewards received after time step t is denoted Rt+1, Rt+2, Rt+3, . . ., then what precise aspect of this sequence do we wish to maximize? In general, we seek to maximize the expected return, where the return, denoted Gt, is deﬁned as some speciﬁc function of the reward sequence. In the simplest case the return is the sum of the rewards: Gt .= Rt+1 + Rt+2 + Rt+3 + · · · + RT , (3.7) where T is a ﬁnal time step.


the sum of the rewards: Gt .= Rt+1 + Rt+2 + Rt+3 + · · · + RT , (3.7) where T is a ﬁnal time step. This approach makes sense in applications in which there is a natural notion of ﬁnal time step, that is, when the agent–environment interaction breaks naturally into subsequences, which we call episodes,7 such as plays of a game, trips through a maze, or any sort of repeated interaction. Each episode ends in a special state called the terminal state, followed by a reset to a standard starting state or to a sample from a standard distribution of starting states. Even if you think of episodes as ending in di↵erent ways, such as winning and losing a game, the next episode begins independently of how the previous one ended.


as winning and losing a game, the next episode begins independently of how the previous one ended. Thus the episodes can all be considered to end in the same terminal state, with di↵erent rewards for the di↵erent outcomes. Tasks with episodes of this kind are called episodic tasks. In episodic tasks we sometimes need to distinguish the set of all nonterminal states, denoted S, from the set of all states plus the terminal state, denoted S+. The time of termination, T, is a random variable that normally varies from episode to episode. On the other hand, in many cases the agent–environment interaction does not break naturally into identiﬁable episodes, but goes on continually without limit. For example, this would be the natural way to formulate an on-going process-control task, or an applic


r example, this would be the natural way to formulate an on-going process-control task, or an application to a robot with a long life span. We call these continuing tasks. The return formulation (3.7) is problematic for continuing tasks because the ﬁnal time step would be T = 1, and the return, which is what we are trying to maximize, could easily be inﬁnite. 5Better places for imparting this kind of prior knowledge are the initial policy or initial value function. 6Section 17.4 delves further into the issue of designing e↵ective reward signals. 7Episodes are sometimes called “trials” in the literature.


3.3. Returns and Episodes 55 (For example, suppose the agent receives a reward of +1 at each time step.) Thus, in this book we usually use a deﬁnition of return that is slightly more complex conceptually but much simpler mathematically. The additional concept that we need is that of discounting. According to this approach, the agent tries to select actions so that the sum of the discounted rewards it receives over the future is maximized. In particular, it chooses At to maximize the expected discounted return: Gt .= Rt+1 + γRt+2 + γ2Rt+3 + · · · = 1 X k=0 γkRt+k+1, (3.8) where γ is a parameter, 0 γ 1, called the discount rate. The discount rate determines the present value of future rewards: a reward received k time steps in the future is worth only γk−1 times what it would be worth if i


s: a reward received k time steps in the future is worth only γk−1 times what it would be worth if it were received immediately. If γ < 1, the inﬁnite sum in (3.8) has a ﬁnite value as long as the reward sequence {Rk} is bounded. If γ = 0, the agent is “myopic” in being concerned only with maximizing immediate rewards: its objective in this case is to learn how to choose At so as to maximize only Rt+1. If each of the agent’s actions happened to inﬂuence only the immediate reward, not future rewards as well, then a myopic agent could maximize (3.8) by separately maximizing each immediate reward. But in general, acting to maximize immediate reward can reduce access to future rewards so that the return is reduced.


ing to maximize immediate reward can reduce access to future rewards so that the return is reduced. As γ approaches 1, the return objective takes future rewards into account more strongly; the agent becomes more farsighted. Returns at successive time steps are related to each other in a way that is important for the theory and algorithms of reinforcement learning: Gt .= Rt+1 + γRt+2 + γ2Rt+3 + γ3Rt+4 + · · · = Rt+1 + γ # Rt+2 + γRt+3 + γ2Rt+4 + · · · $ = Rt+1 + γGt+1 (3.9) Note that this works for all time steps t < T, even if termination occurs at t + 1, provided we deﬁne GT = 0. This often makes it easy to compute returns from reward sequences. Note that although the return (3.8) is a sum of an inﬁnite number of terms, it is still ﬁnite if the reward is nonzero and constant—if γ < 1.


um of an inﬁnite number of terms, it is still ﬁnite if the reward is nonzero and constant—if γ < 1. For example, if the reward is a constant +1, then the return is Gt = 1 X k=0 γk = 1 1 −γ . (3.10) Exercise 3.5 The equations in Section 3.1 are for the continuing case and need to be modiﬁed (very slightly) to apply to episodic tasks. Show that you know the modiﬁcations needed by giving the modiﬁed version of (3.3). ⇤


56 Chapter 3: Finite Markov Decision Processes Example 3.4: Pole-Balancing The objective in this task is to apply forces to a cart moving along a track so as to keep a pole hinged to the cart from falling over: A failure is said to occur if the pole falls past a given angle from vertical or if the cart runs o↵the track. The pole is reset to vertical after each failure. This task could be treated as episodic, where the natural episodes are the repeated attempts to balance the pole. The reward in this case could be +1 for every time step on which failure did not occur, so that the return at each time would be the number of steps until failure. In this case, successful balancing forever would mean a return of inﬁnity.


of steps until failure. In this case, successful balancing forever would mean a return of inﬁnity. Alternatively, we could treat pole-balancing as a continuing task, using discounting. In this case the reward would be −1 on each failure and zero at all other times. The return at each time would then be related to −γK−1, where K is the number of time steps before failure (as well as to the times of later failures). In either case, the return is maximized by keeping the pole balanced for as long as possible. Exercise 3.6 Suppose you treated pole-balancing as an episodic task but also used discounting, with all rewards zero except for −1 upon failure. What then would the return be at each time? How does this return di↵er from that in the discounted, continuing formulation of this task?


time? How does this return di↵er from that in the discounted, continuing formulation of this task? ⇤ Exercise 3.7 Imagine that you are designing a robot to run a maze. You decide to give it a reward of +1 for escaping from the maze and a reward of zero at all other times. The task seems to break down naturally into episodes—the successive runs through the maze—so you decide to treat it as an episodic task, where the goal is to maximize expected total reward (3.7). After running the learning agent for a while, you ﬁnd that it is showing no improvement in escaping from the maze. What is going wrong? Have you e↵ectively communicated to the agent what you want it to achieve? ⇤ Exercise 3.8 Suppose γ = 0.5 and the following sequence of rewards is received R1 = −1, R2 = 2, R3 = 6, R4 = 3, and R


ose γ = 0.5 and the following sequence of rewards is received R1 = −1, R2 = 2, R3 = 6, R4 = 3, and R5 = 2, with T = 5. What are G0, G1, . . ., G5? Hint: Work backwards. ⇤ Exercise 3.9 Suppose γ = 0.9 and the reward sequence is R1 = 2 followed by an inﬁnite sequence of 7s. What are G1 and G0? ⇤ Exercise 3.10 Prove the second equality in (3.10). ⇤


3.4. Uniﬁed Notation for Episodic and Continuing Tasks 57 3.4 Uniﬁed Notation for Episodic and Continuing Tasks In the preceding section we described two kinds of reinforcement learning tasks, one in which the agent–environment interaction naturally breaks down into a sequence of separate episodes (episodic tasks), and one in which it does not (continuing tasks). The former case is mathematically easier because each action a↵ects only the ﬁnite number of rewards subsequently received during the episode. In this book we consider sometimes one kind of problem and sometimes the other, but often both. It is therefore useful to establish one notation that enables us to talk precisely about both cases simultaneously. To be precise about episodic tasks requires some additional notation.


ut both cases simultaneously. To be precise about episodic tasks requires some additional notation. Rather than one long sequence of time steps, we need to consider a series of episodes, each of which consists of a ﬁnite sequence of time steps. We number the time steps of each episode starting anew from zero. Therefore, we have to refer not just to St, the state representation at time t, but to St,i, the state representation at time t of episode i (and similarly for At,i, Rt,i, ⇡t,i, Ti, etc.). However, it turns out that when we discuss episodic tasks we almost never have to distinguish between di↵erent episodes. We are almost always considering a particular episode, or stating something that is true for all episodes.


almost always considering a particular episode, or stating something that is true for all episodes. Accordingly, in practice we almost always abuse notation slightly by dropping the explicit reference to episode number. That is, we write St to refer to St,i, and so on. We need one other convention to obtain a single notation that covers both episodic and continuing tasks. We have deﬁned the return as a sum over a ﬁnite number of terms in one case (3.7) and as a sum over an inﬁnite number of terms in the other (3.8). These two can be uniﬁed by considering episode termination to be the entering of a special absorbing state that transitions only to itself and that generates only rewards of zero.


a special absorbing state that transitions only to itself and that generates only rewards of zero. For example, consider the state transition diagram: R1 = +1 S0 S1 R2 = +1 S2 R3 = +1 R4 = 0 R5 = 0 . . . Here the solid square represents the special absorbing state corresponding to the end of an episode. Starting from S0, we get the reward sequence +1, +1, +1, 0, 0, 0, . . .. Summing these, we get the same return whether we sum over the ﬁrst T rewards (here T = 3) or over the full inﬁnite sequence. This remains true even if we introduce discounting. Thus, we can deﬁne the return, in general, according to (3.8), using the convention of omitting episode numbers when they are not needed, and including the possibility that γ = 1 if the sum remains deﬁned (e.g., because all episodes terminate).


cluding the possibility that γ = 1 if the sum remains deﬁned (e.g., because all episodes terminate). Alternatively, we can write Gt .= T X k=t+1 γk−t−1Rk, (3.11) including the possibility that T = 1 or γ = 1 (but not both). We use these conventions throughout the rest of the book to simplify notation and to express the close parallels


58 Chapter 3: Finite Markov Decision Processes between episodic and continuing tasks. (Later, in Chapter 10, we will introduce a formulation that is both continuing and undiscounted.) 3.5 Policies and Value Functions Almost all reinforcement learning algorithms involve estimating value functions—functions of states (or of state–action pairs) that estimate how good it is for the agent to be in a given state (or how good it is to perform a given action in a given state). The notion of “how good” here is deﬁned in terms of future rewards that can be expected, or, to be precise, in terms of expected return. Of course the rewards the agent can expect to receive in the future depend on what actions it will take.


urse the rewards the agent can expect to receive in the future depend on what actions it will take. Accordingly, value functions are deﬁned with respect to particular ways of acting, called policies. Formally, a policy is a mapping from states to probabilities of selecting each possible action. If the agent is following policy ⇡at time t, then ⇡(a|s) is the probability that At = a if St = s. Like p, ⇡is an ordinary function; the “|” in the middle of ⇡(a|s) merely reminds us that it deﬁnes a probability distribution over a 2 A(s) for each s 2 S. Reinforcement learning methods specify how the agent’s policy is changed as a result of its experience. Exercise 3.11 If the current state is St, and actions are selected according to a stochastic policy ⇡, then what is the expectation of Rt+1 in te


actions are selected according to a stochastic policy ⇡, then what is the expectation of Rt+1 in terms of ⇡and the four-argument function p (3.2)? ⇤ The value function of a state s under a policy ⇡, denoted v⇡(s), is the expected return when starting in s and following ⇡thereafter. For MDPs, we can deﬁne v⇡formally by v⇡(s) .= E⇡[Gt | St =s] = E⇡ " 1 X k=0 γkRt+k+1 44444 St =s # , for all s 2 S, (3.12) where E⇡[·] denotes the expected value of a random variable given that the agent follows policy ⇡, and t is any time step. Note that the value of the terminal state, if any, is always zero. We call the function v⇡the state-value function for policy ⇡. Similarly, we deﬁne the value of taking action a in state s under a policy ⇡, denoted q⇡(s, a), as the expected return starting from s, takin


ction a in state s under a policy ⇡, denoted q⇡(s, a), as the expected return starting from s, taking the action a, and thereafter following policy ⇡: q⇡(s, a) .= E⇡[Gt | St =s, At = a] = E⇡ " 1 X k=0 γkRt+k+1 44444 St =s, At =a # . (3.13) We call q⇡the action-value function for policy ⇡. Exercise 3.12 Give an equation for v⇡in terms of q⇡and ⇡. ⇤ Exercise 3.13 Give an equation for q⇡in terms of v⇡and the four-argument p. ⇤ The value functions v⇡and q⇡can be estimated from experience. For example, if an agent follows policy ⇡and maintains an average, for each state encountered, of the actual returns that have followed that state, then the average will converge to the state’s value, v⇡(s), as the number of times that state is encountered approaches inﬁnity. If separate


3.5. Policies and Value Functions 59 averages are kept for each action taken in each state, then these averages will similarly converge to the action values, q⇡(s, a). We call estimation methods of this kind Monte Carlo methods because they involve averaging over many random samples of actual returns. These kinds of methods are presented in Chapter 5. Of course, if there are very many states, then it may not be practical to keep separate averages for each state individually. Instead, the agent would have to maintain v⇡and q⇡as parameterized functions (with fewer parameters than states) and adjust the parameters to better match the observed returns. This can also produce accurate estimates, although much depends on the nature of the parameterized function approximator.


accurate estimates, although much depends on the nature of the parameterized function approximator. These possibilities are discussed in Part II of the book. A fundamental property of value functions used throughout reinforcement learning and dynamic programming is that they satisfy recursive relationships similar to that which we have already established for the return (3.9). For any policy ⇡and any state s, the following consistency condition holds between the value of s and the value of its possible successor states: v⇡(s) .= E⇡[Gt | St =s] = E⇡[Rt+1 + γGt+1 | St =s] (by (3.9)) = X a ⇡(a|s) X s0 X r p(s0, r|s, a) h r + γE⇡[Gt+1|St+1 =s0] i = X a ⇡(a|s) X s0,r p(s0, r|s, a) h r + γv⇡(s0) i , for all s 2 S, (3.14) where it is implicit that the actions, a, are taken from the set A(s), that


, for all s 2 S, (3.14) where it is implicit that the actions, a, are taken from the set A(s), that the next states, s0, are taken from the set S (or from S+ in the case of an episodic problem), and that the rewards, r, are taken from the set R. Note also how in the last equation we have merged the two sums, one over all the values of s0 and the other over all the values of r, into one sum over all the possible values of both. We use this kind of merged sum often to simplify formulas. Note how the ﬁnal expression can be read easily as an expected value. It is really a sum over all values of the three variables, a, s0, and r. For each triple, we compute its probability, ⇡(a|s)p(s0, r|s, a), weight the quantity in brackets by that probability, then sum over all possibilities to get an expec


weight the quantity in brackets by that probability, then sum over all possibilities to get an expected value. ⇡ s s0 ⇡ r p a Backup diagram for v⇡ Equation (3.14) is the Bellman equation for v⇡. It expresses a relationship between the value of a state and the values of its successor states. Think of looking ahead from a state to its possible successor states, as suggested by the diagram to the right. Each open circle represents a state and each solid circle represents a state–action pair. Starting from state s, the root node at the top, the agent could take any of some set of actions— three are shown in the diagram—based on its policy ⇡. From each of these, the environment could respond with one of several next states, s0 (two are shown in the ﬁgure), along with a reward, r, depending on


h one of several next states, s0 (two are shown in the ﬁgure), along with a reward, r, depending on its dynamics given by the function p. The Bellman equation (3.14) averages over all the possibilities, weighting each by its probability of occurring. It states that the value of the start state must equal the (discounted) value of the expected next state, plus the reward expected along the way.


60 Chapter 3: Finite Markov Decision Processes The value function v⇡is the unique solution to its Bellman equation. We show in subsequent chapters how this Bellman equation forms the basis of a number of ways to compute, approximate, and learn v⇡. We call diagrams like that above backup diagrams because they diagram relationships that form the basis of the update or backup operations that are at the heart of reinforcement learning methods. These operations transfer value information back to a state (or a state–action pair) from its successor states (or state–action pairs). We use backup diagrams throughout the book to provide graphical summaries of the algorithms we discuss. (Note that, unlike transition graphs, the state nodes of backup diagrams do not necessarily represent distinct state


ke transition graphs, the state nodes of backup diagrams do not necessarily represent distinct states; for example, a state might be its own successor.) Example 3.5: Gridworld Figure 3.2 (left) shows a rectangular gridworld representation of a simple ﬁnite MDP. The cells of the grid correspond to the states of the environment. At each cell, four actions are possible: north, south, east, and west, which deterministically cause the agent to move one cell in the respective direction on the grid. Actions that would take the agent o↵the grid leave its location unchanged, but also result in a reward of −1. Other actions result in a reward of 0, except those that move the agent out of the special states A and B. From state A, all four actions yield a reward of +10 and take the agent to A0.


cial states A and B. From state A, all four actions yield a reward of +10 and take the agent to A0. From state B, all actions yield a reward of +5 and take the agent to B0. 3.3 8.8 4.4 5.3 1.5 1.5 3.0 2.3 1.9 0.5 0.1 0.7 0.7 0.4 -0.4 -1.0 -0.4 -0.4 -0.6 -1.2 -1.9 -1.3 -1.2 -1.4 -2.0 A B A' B' +10 +5 Actions Figure 3.2: Gridworld example: exceptional reward dynamics (left) and state-value function for the equiprobable random policy (right). Suppose the agent selects all four actions with equal probability in all states. Figure 3.2 (right) shows the value function, v⇡, for this policy, for the discounted reward case with γ = 0.9. This value function was computed by solving the system of linear equations (3.14).


se with γ = 0.9. This value function was computed by solving the system of linear equations (3.14). Notice the negative values near the lower edge; these are the result of the high probability of hitting the edge of the grid there under the random policy. State A is the best state to be in under this policy. Note that A’s expected return is less than its immediate reward of 10, because from A the agent is taken to state A0 from which it is likely to run into the edge of the grid. State B, on the other hand, is valued more than its immediate reward of 5, because from B the agent is taken to B0 which has a positive value. From B0 the expected penalty (negative reward) for possibly running into an edge is more than compensated for by the expected gain for possibly stumbling onto A or B.


into an edge is more than compensated for by the expected gain for possibly stumbling onto A or B. Exercise 3.14 The Bellman equation (3.14) must hold for each state for the value function v⇡shown in Figure 3.2 (right) of Example 3.5. Show numerically that this equation holds for the center state, valued at +0.7, with respect to its four neighboring states, valued at +2.3, +0.4, −0.4, and +0.7. (These numbers are accurate only to one decimal place.) ⇤


3.5. Policies and Value Functions 61 Exercise 3.15 In the gridworld example, rewards are positive for goals, negative for running into the edge of the world, and zero the rest of the time. Are the signs of these rewards important, or only the intervals between them? Prove, using (3.8), that adding a constant c to all the rewards adds a constant, vc, to the values of all states, and thus does not a↵ect the relative values of any states under any policies. What is vc in terms of c and γ? ⇤ Exercise 3.16 Now consider adding a constant c to all the rewards in an episodic task, such as maze running. Would this have any e↵ect, or would it leave the task unchanged as in the continuing task above? Why or why not? Give an example.


would it leave the task unchanged as in the continuing task above? Why or why not? Give an example. ⇤ Example 3.6: Golf To formulate playing a hole of golf as a reinforcement learning task, we count a penalty (negative reward) of −1 for each stroke until we hit the ball into the hole. The state is the location of the ball. The value of a state is the negative of the number of strokes to the hole from that location. Our actions are how we aim and swing at the ball, of course, and which club we select. Let us take the former as given and consider just the choice of club, which we assume is either a putter or a driver. The upper part of Figure 3.3 shows a possible state-value function, vputt(s), for the policy that Q*(s,driver) V putt sand green !1 s a n d !2 !2 !3 !4 !1 !5 !6 !4 !3 !3 !2 !4


, for the policy that Q*(s,driver) V putt sand green !1 s a n d !2 !2 !3 !4 !1 !5 !6 !4 !3 !3 !2 !4 sand green !1 s a n d !2 !3 !2 0 0 !" !" vputt q⇤(s, driver) Figure 3.3: A golf example: the state-value function for putting (upper) and the optimal actionvalue function for using the driver (lower). always uses the putter. The terminal state in-the-hole has a value of 0. From anywhere on the green we assume we can make a putt; these states have value −1. O↵the green we cannot reach the hole by putting, and the value is lower. If we can reach the green from a state by putting, then that state must have value one less than the green’s value, that is, −2. For simplicity, let us assume we can putt very precisely and deterministically, but with a limited range.


mplicity, let us assume we can putt very precisely and deterministically, but with a limited range. This gives us the sharp contour line labeled −2 in the ﬁgure; all locations between that line and the green require exactly two strokes to complete the hole. Similarly, any location within putting range of the −2 contour line must have a value of −3, and so on to get all the contour lines shown in the ﬁgure. Putting doesn’t get us out of sand traps, so they have a value of −1. Overall, it takes us six strokes to get from the tee to the hole by putting. r s0 s, a a0 ⇡ p q⇡backup diagram Exercise 3.17 What is the Bellman equation for action values, that is, for q⇡? It must give the action value q⇡(s, a) in terms of the action values, q⇡(s0, a0), of possible successors to the state–action pair


q⇡(s, a) in terms of the action values, q⇡(s0, a0), of possible successors to the state–action pair (s, a). Hint: The backup diagram to the right corresponds to this equation. Show the sequence of equations analogous to (3.14), but for action values. ⇤


62 Chapter 3: Finite Markov Decision Processes Exercise 3.18 The value of a state depends on the values of the actions possible in that state and on how likely each action is to be taken under the current policy. We can think of this in terms of a small backup diagram rooted at the state and considering each possible action: s taken with probability ⇡(a|s) v⇡(s) q⇡(s, a) a1 a2 a3 Give the equation corresponding to this intuition and diagram for the value at the root node, v⇡(s), in terms of the value at the expected leaf node, q⇡(s, a), given St = s. This equation should include an expectation conditioned on following the policy, ⇡. Then give a second equation in which the expected value is written out explicitly in terms of ⇡(a|s) such that no expected value notation appears in the equati


written out explicitly in terms of ⇡(a|s) such that no expected value notation appears in the equation. ⇤ Exercise 3.19 The value of an action, q⇡(s, a), depends on the expected next reward and the expected sum of the remaining rewards. Again we can think of this in terms of a small backup diagram, this one rooted at an action (state–action pair) and branching to the possible next states: s, a q⇡(s, a) s0 3 s0 2 s0 1 r1 r2 r3 v⇡(s0) expected rewards Give the equation corresponding to this intuition and diagram for the action value, q⇡(s, a), in terms of the expected next reward, Rt+1, and the expected next state value, v⇡(St+1), given that St =s and At =a. This equation should include an expectation but not one conditioned on following the policy.


At =a. This equation should include an expectation but not one conditioned on following the policy. Then give a second equation, writing out the expected value explicitly in terms of p(s0, r|s, a) deﬁned by (3.2), such that no expected value notation appears in the equation. ⇤ 3.6 Optimal Policies and Optimal Value Functions Solving a reinforcement learning task means, roughly, ﬁnding a policy that achieves a lot of reward over the long run. For ﬁnite MDPs, we can precisely deﬁne an optimal policy in the following way. Value functions deﬁne a partial ordering over policies. A policy ⇡is deﬁned to be better than or equal to a policy ⇡0 if its expected return is greater than or equal to that of ⇡0 for all states. In other words, ⇡≥⇡0 if and only if v⇡(s) ≥v⇡0(s) for all s 2 S.


qual to that of ⇡0 for all states. In other words, ⇡≥⇡0 if and only if v⇡(s) ≥v⇡0(s) for all s 2 S. There is always at least one policy that is better than or equal to all other policies. This is an optimal policy. Although there may be more than one, we denote all the optimal policies by ⇡⇤. They share the same state-value function, called the optimal state-value function, denoted v⇤, and deﬁned as v⇤(s) .= max ⇡ v⇡(s), (3.15) for all s 2 S.


3.6. Optimal Policies and Optimal Value Functions 63 Optimal policies also share the same optimal action-value function, denoted q⇤, and deﬁned as q⇤(s, a) .= max ⇡ q⇡(s, a), (3.16) for all s 2 S and a 2 A(s). For the state–action pair (s, a), this function gives the expected return for taking action a in state s and thereafter following an optimal policy. Thus, we can write q⇤in terms of v⇤as follows: q⇤(s, a) = E[Rt+1 + γv⇤(St+1) | St =s, At =a] . (3.17) Example 3.7: Optimal Value Functions for Golf The lower part of Figure 3.3 shows the contours of a possible optimal action-value function q⇤(s, driver). These are the values of each state if we ﬁrst play a stroke with the driver and afterward select either the driver or the putter, whichever is better.


a stroke with the driver and afterward select either the driver or the putter, whichever is better. The driver enables us to hit the ball farther, but with less accuracy. We can reach the hole in one shot using the driver only if we are already very close; thus the −1 contour for q⇤(s, driver) covers only a small portion of the green. If we have two strokes, however, then we can reach the hole from much farther away, as shown by the −2 contour. In this case we don’t have to drive all the way to within the small −1 contour, but only to anywhere on the green; from there we can use the putter. The optimal action-value function gives the values after committing to a particular ﬁrst action, in this case, to the driver, but afterward using whichever actions are best.


articular ﬁrst action, in this case, to the driver, but afterward using whichever actions are best. The −3 contour is still farther out and includes the starting tee. From the tee, the best sequence of actions is two drives and one putt, sinking the ball in three strokes. Because v⇤is the value function for a policy, it must satisfy the self-consistency condition given by the Bellman equation for state values (3.14). Because it is the optimal value function, however, v⇤’s consistency condition can be written in a special form without reference to any speciﬁc policy. This is the Bellman equation for v⇤, or the Bellman optimality equation. Intuitively, the Bellman optimality equation expresses the fact that the value of a state under an optimal policy must equal the expected return for the b


the fact that the value of a state under an optimal policy must equal the expected return for the best action from that state: v⇤(s) = max a2A(s) q⇡⇤(s, a) = max a E⇡⇤[Gt | St =s, At =a] = max a E⇡⇤[Rt+1 + γGt+1 | St =s, At =a] (by (3.9)) = max a E[Rt+1 + γv⇤(St+1) | St =s, At =a] (3.18) = max a X s0,r p(s0, r|s, a) ⇥ r + γv⇤(s0) ⇤ . (3.19)


64 Chapter 3: Finite Markov Decision Processes The last two equations are two forms of the Bellman optimality equation for v⇤. The Bellman optimality equation for q⇤is q⇤(s, a) = E h Rt+1 + γ max a0 q⇤(St+1, a0) 444 St = s, At = a i = X s0,r p(s0, r|s, a) h r + γ max a0 q⇤(s0, a0) i . (3.20) The backup diagrams in the ﬁgure below show graphically the spans of future states and actions considered in the Bellman optimality equations for v⇤and q⇤. These are the same as the backup diagrams for v⇡and q⇡presented earlier except that arcs have been added at the agent’s choice points to represent that the maximum over that choice is taken rather than the expected value given some policy. The backup diagram on the left graphically represents the Bellman optimality equation (3.19) and the backup dia


diagram on the left graphically represents the Bellman optimality equation (3.19) and the backup diagram on the right graphically represents (3.20). s s0 a r r s0 s, a a0 max max (v⇤) (q⇤) Figure 3.4: Backup diagrams for v⇤and q⇤ For ﬁnite MDPs, the Bellman optimality equation for v⇤(3.19) has a unique solution. The Bellman optimality equation is actually a system of equations, one for each state, so if there are n states, then there are n equations in n unknowns. If the dynamics p of the environment are known, then in principle one can solve this system of equations for v⇤ using any one of a variety of methods for solving systems of nonlinear equations. One can solve a related set of equations for q⇤. Once one has v⇤, it is relatively easy to determine an optimal policy.


ted set of equations for q⇤. Once one has v⇤, it is relatively easy to determine an optimal policy. For each state s, there will be one or more actions at which the maximum is obtained in the Bellman optimality equation. Any policy that assigns nonzero probability only to these actions is an optimal policy. You can think of this as a one-step search. If you have the optimal value function, v⇤, then the actions that appear best after a one-step search will be optimal actions. Another way of saying this is that any policy that is greedy with respect to the optimal evaluation function v⇤is an optimal policy. The term greedy is used in computer science to describe any search or decision procedure that selects alternatives based only on local or immediate considerations, without considering the


e that selects alternatives based only on local or immediate considerations, without considering the possibility that such a selection may prevent future access to even better alternatives. Consequently, it describes policies that select actions based only on their short-term consequences. The beauty of v⇤ is that if one uses it to evaluate the short-term consequences of actions—speciﬁcally, the one-step consequences—then a greedy policy is actually optimal in the long-term sense in which we are interested because v⇤already takes into account the reward consequences of all possible future behavior. By means of v⇤, the optimal expected long-term return is


3.6. Optimal Policies and Optimal Value Functions 65 turned into a quantity that is locally and immediately available for each state. Hence, a one-step-ahead search yields the long-term optimal actions. Having q⇤makes choosing optimal actions even easier. With q⇤, the agent does not even have to do a one-step-ahead search: for any state s, it can simply ﬁnd any action that maximizes q⇤(s, a). The action-value function e↵ectively caches the results of all one-step-ahead searches. It provides the optimal expected long-term return as a value that is locally and immediately available for each state–action pair. Hence, at the cost of representing a function of state–action pairs, instead of just of states, the optimal actionvalue function allows optimal actions to be selected without having to


of states, the optimal actionvalue function allows optimal actions to be selected without having to know anything about possible successor states and their values, that is, without having to know anything about the environment’s dynamics. Example 3.8: Solving the Gridworld Suppose we solve the Bellman equation for v⇤ for the simple grid task introduced in Example 3.5 and shown again in Figure 3.5 (left). Recall that state A is followed by a reward of +10 and transition to state A0, while state B is followed by a reward of +5 and transition to state B0. Figure 3.5 (middle) shows the optimal value function, and Figure 3.5 (right) shows the corresponding optimal policies. Where there are multiple arrows in a cell, all of the corresponding actions are optimal.


policies. Where there are multiple arrows in a cell, all of the corresponding actions are optimal. a) gridworld b) V* c) !* 22.0 24.4 22.0 19.4 17.5 19.8 22.0 19.8 17.8 16.0 17.8 19.8 17.8 16.0 14.4 16.0 17.8 16.0 14.4 13.0 14.4 16.0 14.4 13.0 11.7 A B A' B' +10 +5 v* π* Gridworld v⇤ ⇡⇤ Figure 3.5: Optimal solutions to the gridworld example. Example 3.9: Bellman Optimality Equations for the Recycling Robot Using (3.19), we can explicitly give the Bellman optimality equation for the recycling robot example. To make things more compact, we abbreviate the states high and low, and the actions search, wait, and recharge respectively by h, l, s, w, and re. Because there are only two states, the Bellman optimality equation consists of two equations.


d re. Because there are only two states, the Bellman optimality equation consists of two equations. The equation for v⇤(h) can be written as follows: v⇤(h) = max ⇢p(h|h, s)[r(h, s, h) + γv⇤(h)] + p(l|h, s)[r(h, s, l) + γv⇤(l)], p(h|h, w)[r(h, w, h) + γv⇤(h)] + p(l|h, w)[r(h, w, l) + γv⇤(l)] 5 = max ⇢ ↵[rs + γv⇤(h)] + (1 −↵)[rs + γv⇤(l)], 1[rw + γv⇤(h)] + 0[rw + γv⇤(l)] 5 = max ⇢rs + γ[↵v⇤(h) + (1 −↵)v⇤(l)], rw + γv⇤(h) 5 .


66 Chapter 3: Finite Markov Decision Processes Following the same procedure for v⇤(l) yields the equation v⇤(l) = max 8 < : βrs −3(1 −β) + γ[(1 −β)v⇤(h) + βv⇤(l)], rw + γv⇤(l), γv⇤(h) 9 = ; . For any choice of rs, rw, ↵, β, and γ, with 0 γ < 1, 0 ↵, β 1, there is exactly one pair of numbers, v⇤(h) and v⇤(l), that simultaneously satisfy these two nonlinear equations. Explicitly solving the Bellman optimality equation provides one route to ﬁnding an optimal policy, and thus to solving the reinforcement learning problem. However, this solution is rarely directly useful. It is akin to an exhaustive search, looking ahead at all possibilities, computing their probabilities of occurrence and their desirabilities in terms of expected rewards.


computing their probabilities of occurrence and their desirabilities in terms of expected rewards. This solution relies on at least three assumptions that are rarely true in practice: (1) the dynamics of the environment are accurately known; (2) computational resources are suﬃcient to complete the calculation; and (3) the states have the Markov property. For the kinds of tasks in which we are interested, one is generally not able to implement this solution exactly because various combinations of these assumptions are violated. For example, although the ﬁrst and third assumptions present no problems for the game of backgammon, the second is a major impediment. Because the game has about 1020 states, it would take thousands of years on today’s fastest computers to solve the Bellman equation


states, it would take thousands of years on today’s fastest computers to solve the Bellman equation for v⇤, and the same is true for ﬁnding q⇤. In reinforcement learning one typically has to settle for approximate solutions. Many di↵erent decision-making methods can be viewed as ways of approximately solving the Bellman optimality equation. For example, heuristic search methods can be viewed as expanding the right-hand side of (3.19) several times, up to some depth, forming a “tree” of possibilities, and then using a heuristic evaluation function to approximate v⇤at the “leaf” nodes. (Heuristic search methods such as A⇤are almost always based on the episodic case.) The methods of dynamic programming can be related even more closely to the Bellman optimality equation.


methods of dynamic programming can be related even more closely to the Bellman optimality equation. Many reinforcement learning methods can be clearly understood as approximately solving the Bellman optimality equation, using actual experienced transitions in place of knowledge of the expected transitions. We consider a variety of such methods in the following chapters. Exercise 3.20 Draw or describe the optimal state-value function for the golf example. ⇤ Exercise 3.21 Draw or describe the contours of the optimal action-value function for putting, q⇤(s, putter), for the golf example. ⇤ +2 0 0 +1 left right Exercise 3.22 Consider the continuing MDP shown to the right. The only decision to be made is that in the top state, where two actions are available, left and right.


only decision to be made is that in the top state, where two actions are available, left and right. The numbers show the rewards that are received deterministically after each action. There are exactly two deterministic policies, ⇡left and ⇡right. What policy is optimal if γ = 0? If γ = 0.9? If γ = 0.5? ⇤


3.7. Optimality and Approximation 67 Exercise 3.23 Give the Bellman equation for q⇤for the recycling robot. ⇤ Exercise 3.24 Figure 3.5 gives the optimal value of the best state of the gridworld as 24.4, to one decimal place. Use your knowledge of the optimal policy and (3.8) to express this value symbolically, and then to compute it to three decimal places. ⇤ Exercise 3.25 Give an equation for v⇤in terms of q⇤. ⇤ Exercise 3.26 Give an equation for q⇤in terms of v⇤and the four-argument p. ⇤ Exercise 3.27 Give an equation for ⇡⇤in terms of q⇤. ⇤ Exercise 3.28 Give an equation for ⇡⇤in terms of v⇤and the four-argument p. ⇤ Exercise 3.29 Rewrite the four Bellman equations for the four value functions (v⇡, v⇤, q⇡, and q⇤) in terms of the three argument function p (3.4) and the two-argument func


tions (v⇡, v⇤, q⇡, and q⇤) in terms of the three argument function p (3.4) and the two-argument function r (3.5). ⇤ 3.7 Optimality and Approximation We have deﬁned optimal value functions and optimal policies. Clearly, an agent that learns an optimal policy has done very well, but in practice this rarely happens. For the kinds of tasks in which we are interested, optimal policies can be generated only with extreme computational cost. A well-deﬁned notion of optimality organizes the approach to learning we describe in this book and provides a way to understand the theoretical properties of various learning algorithms, but it is an ideal that agents can only approximate. As we discussed above, even if we have a complete and accurate model of the environment’s dynamics, it is usually not poss


even if we have a complete and accurate model of the environment’s dynamics, it is usually not possible to simply compute an optimal policy by solving the Bellman optimality equation. For example, board games such as chess are a tiny fraction of human experience, yet large, custom-designed computers still cannot compute the optimal moves. A critical aspect of the problem facing the agent is always the computational power available to it, in particular, the amount of computation it can perform in a single time step. The memory available is also an important constraint. A large amount of memory is often required to build up approximations of value functions, policies, and models. In tasks with small, ﬁnite state sets, it is possible to form these approximations using arrays or tables with o


h small, ﬁnite state sets, it is possible to form these approximations using arrays or tables with one entry for each state (or state–action pair). This we call the tabular case, and the corresponding methods we call tabular methods. In many cases of practical interest, however, there are far more states than could possibly be entries in a table. In these cases the functions must be approximated, using some sort of more compact parameterized function representation. Our framing of the reinforcement learning problem forces us to settle for approximations. However, it also presents us with some unique opportunities for achieving useful approximations. For example, in approximating optimal behavior, there may be many states that the agent faces with such a low probability that selecting subop


vior, there may be many states that the agent faces with such a low probability that selecting suboptimal actions for them has little impact on the amount of reward the agent receives. Tesauro’s backgammon player, for example, plays with exceptional skill even though it might make


68 Chapter 3: Finite Markov Decision Processes very bad decisions on board conﬁgurations that never occur in games against experts. In fact, it is possible that TD-Gammon makes bad decisions for a large fraction of the game’s state set. The online nature of reinforcement learning makes it possible to approximate optimal policies in ways that put more e↵ort into learning to make good decisions for frequently encountered states, at the expense of less e↵ort for infrequently encountered states. This is one key property that distinguishes reinforcement learning from other approaches to approximately solving MDPs. 3.8 Summary Let us summarize the elements of the reinforcement learning problem that we have presented in this chapter.


ummarize the elements of the reinforcement learning problem that we have presented in this chapter. Reinforcement learning is about learning from interaction how to behave in order to achieve a goal. The reinforcement learning agent and its environment interact over a sequence of discrete time steps. The speciﬁcation of their interface deﬁnes a particular task: the actions are the choices made by the agent; the states are the basis for making the choices; and the rewards are the basis for evaluating the choices. Everything inside the agent is known and controllable. Its environment, on the other hand, is incompletely controllable and may or may not be completely known. A policy is a stochastic rule by which the agent selects actions as a function of states.


ly known. A policy is a stochastic rule by which the agent selects actions as a function of states. The agent’s objective is to maximize the amount of reward it receives over time. When the reinforcement learning setup described above is formulated with well deﬁned transition probabilities it constitutes a Markov decision process (MDP). A ﬁnite MDP is an MDP with ﬁnite state, action, and (as we formulate it here) reward sets. Much of the current theory of reinforcement learning is restricted to ﬁnite MDPs, but the methods and ideas apply more generally. The return is the function of future rewards that the agent seeks to maximize (in expected value). It has several di↵erent deﬁnitions depending upon the nature of the task and whether one wishes to discount delayed reward.


deﬁnitions depending upon the nature of the task and whether one wishes to discount delayed reward. The undiscounted formulation is appropriate for episodic tasks, in which the agent–environment interaction breaks naturally into episodes; the discounted formulation is appropriate for tabular continuing tasks, in which the interaction does not naturally break into episodes but continues without limit (but see Sections 10.3–4). We try to deﬁne the returns for the two kinds of tasks such that one set of equations can apply to both the episodic and continuing cases. A policy’s value functions (v⇡and q⇡) assign to each state, or state–action pair, the expected return from that state, or state–action pair, given that the agent uses the policy.


r, the expected return from that state, or state–action pair, given that the agent uses the policy. The optimal value functions (v⇤and q⇤) assign to each state, or state–action pair, the largest expected return achievable by any policy. A policy whose value functions are optimal is an optimal policy. Whereas the optimal value functions for states and state–action pairs are unique for a given MDP, there can be many optimal policies. Any policy that is greedy with respect to the optimal value functions must be an optimal policy. The Bellman optimality equations are special consistency conditions that the optimal value functions must satisfy and that can, in principle, be solved for the optimal value functions, from which an optimal policy can be determined with relative ease.


3.8. Summary 69 A reinforcement learning problem can be posed in a variety of di↵erent ways depending on assumptions about the level of knowledge initially available to the agent. In problems of complete knowledge, the agent has a complete and accurate model of the environment’s dynamics. If the environment is an MDP, then such a model consists of the complete fourargument dynamics function p (3.2). In problems of incomplete knowledge, a complete and perfect model of the environment is not available. Even if the agent had a complete and accurate environment model, the agent would typically be unable to fully use it because of limitations on its memory and computation per time step. In particular, extensive memory may be required to build up accurate approximations of value functions, polic


ular, extensive memory may be required to build up accurate approximations of value functions, policies, and models. In most cases of practical interest there are far more states than could possibly be entries in a table, and approximations must be made. A well-deﬁned notion of optimality organizes the approach to learning we describe in this book and provides a way to understand the theoretical properties of various learning algorithms, but it is an ideal that reinforcement learning agents can only approximate to varying degrees. In reinforcement learning we are very much concerned with cases in which optimal solutions cannot be found but must be approximated in some way. Bibliographical and Historical Remarks The reinforcement learning problem is deeply indebted to the idea of Markov dec


d Historical Remarks The reinforcement learning problem is deeply indebted to the idea of Markov decision processes (MDPs) from the ﬁeld of optimal control. These historical inﬂuences and other major inﬂuences from psychology are described in the brief history given in Chapter 1. Reinforcement learning adds to MDPs a focus on approximation and incomplete information for realistically large problems. MDPs and the reinforcement learning problem are only weakly linked to traditional learning and decision-making problems in artiﬁcial intelligence. However, artiﬁcial intelligence is now vigorously exploring MDP formulations for planning and decision making from a variety of perspectives. MDPs are more general than previous formulations used in artiﬁcial intelligence in that they permit more gen


more general than previous formulations used in artiﬁcial intelligence in that they permit more general kinds of goals and uncertainty. The theory of MDPs is treated by, for example, Bertsekas (2005), White (1969), Whittle (1982, 1983), and Puterman (1994). A particularly compact treatment of the ﬁnite case is given by Ross (1983). MDPs are also studied under the heading of stochastic optimal control, where adaptive optimal control methods are most closely related to reinforcement learning (e.g., Kumar, 1985; Kumar and Varaiya, 1986). The theory of MDPs evolved from e↵orts to understand the problem of making sequences of decisions under uncertainty, where each decision can depend on the previous decisions and their outcomes.


ons under uncertainty, where each decision can depend on the previous decisions and their outcomes. It is sometimes called the theory of multistage decision processes, or sequential decision processes, and has roots in the statistical literature on sequential sampling beginning with the papers by Thompson (1933, 1934) and Robbins (1952) that we cited in Chapter 2 in connection with bandit problems (which are prototypical MDPs if formulated as multiple-situation problems). The earliest instance (that we are aware of) in which reinforcement learning was discussed using the MDP formalism is Andreae’s (1969) description of a uniﬁed view of


70 Chapter 3: Finite Markov Decision Processes learning machines. Witten and Corbin (1973) experimented with a reinforcement learning system later analyzed by Witten (1977, 1976a) using the MDP formalism. Although he did not explicitly mention MDPs, Werbos (1977) suggested approximate solution methods for stochastic optimal control problems that are related to modern reinforcement learning methods (see also Werbos, 1982, 1987, 1988, 1989, 1992). Although Werbos’s ideas were not widely recognized at the time, they were prescient in emphasizing the importance of approximately solving optimal control problems in a variety of domains, including artiﬁcial intelligence. The most inﬂuential integration of reinforcement learning and MDPs is due to Watkins (1989).


gence. The most inﬂuential integration of reinforcement learning and MDPs is due to Watkins (1989). 3.1 Our characterization of the dynamics of an MDP in terms of p(s0, r|s, a) is slightly unusual. It is more common in the MDP literature to describe the dynamics in terms of the state transition probabilities p(s0|s, a) and expected next rewards r(s, a). In reinforcement learning, however, we more often have to refer to individual actual or sample rewards (rather than just their expected values). Our notation also makes it plainer that St and Rt are in general jointly determined, and thus must have the same time index. In teaching reinforcement learning, we have found our notation to be more straightforward conceptually and easier to understand.


rning, we have found our notation to be more straightforward conceptually and easier to understand. For a good intuitive discussion of the system-theoretic concept of state, see Minsky (1967). The bioreactor example is based on the work of Ungar (1990) and Miller and Williams (1992). The recycling robot example was inspired by the can-collecting robot built by Jonathan Connell (1989). Kober and Peters (2012) present a collection of robotics applications of reinforcement learning. 3.2 An explicit statement of the reward hypothesis was suggested by Michael Littman (personal communication). 3.3–4 The terminology of episodic and continuing tasks is di↵erent from that usually used in the MDP literature.