5.5. O↵-policy Prediction via Importance Sampling 105 When importance sampling is done as a simple average in this way it is called ordinary importance sampling. An important alternative is weighted importance sampling, which uses a weighted average, deﬁned as V (s) .= P t2T(s) ⇢t:T (t)−1Gt P t2T(s) ⇢t:T (t)−1 , (5.6) or zero if the denominator is zero. To understand these two varieties of importance sampling, consider the estimates of their ﬁrst-visit methods after observing a single return from state s. In the weighted-average estimate, the ratio ⇢t:T (t)−1 for the single return cancels in the numerator and denominator, so that the estimate is equal to the observed return independent of the ratio (assuming the ratio is nonzero).


estimate is equal to the observed return independent of the ratio (assuming the ratio is nonzero). Given that this return was the only one observed, this is a reasonable estimate, but its expectation is vb(s) rather than v⇡(s), and in this statistical sense it is biased. In contrast, the ﬁrst-visit version of the ordinary importance-sampling estimator (5.5) is always v⇡(s) in expectation (it is unbiased), but it can be extreme. Suppose the ratio were ten, indicating that the trajectory observed is ten times as likely under the target policy as under the behavior policy. In this case the ordinary importance-sampling estimate would be ten times the observed return. That is, it would be quite far from the observed return even though the episode’s trajectory is considered very representative


far from the observed return even though the episode’s trajectory is considered very representative of the target policy. Formally, the di↵erence between the ﬁrst-visit methods of the two kinds of importance sampling is expressed in their biases and variances. Ordinary importance sampling is unbiased whereas weighted importance sampling is biased (though the bias converges asymptotically to zero). On the other hand, the variance of ordinary importance sampling is in general unbounded because the variance of the ratios can be unbounded, whereas in the weighted estimator the largest weight on any single return is one. In fact, assuming bounded returns, the variance of the weighted importance-sampling estimator converges to zero even if the variance of the ratios themselves is inﬁnite (Precup


ampling estimator converges to zero even if the variance of the ratios themselves is inﬁnite (Precup, Sutton, and Dasgupta 2001). In practice, the weighted estimator usually has dramatically lower variance and is strongly preferred. Nevertheless, we will not totally abandon ordinary importance sampling as it is easier to extend to the approximate methods using function approximation that we explore in the second part of this book. The every-visit methods for ordinary and weighed importance sampling are both biased, though, again, the bias falls asymptotically to zero as the number of samples increases. In practice, every-visit methods are often preferred because they remove the need to keep track of which states have been visited and because they are much easier to extend to approximations


track of which states have been visited and because they are much easier to extend to approximations. A complete every-visit MC algorithm for o↵-policy policy evaluation using weighted importance sampling is given in the next section on page 110. Exercise 5.5 Consider an MDP with a single nonterminal state and a single action that transitions back to the nonterminal state with probability p and transitions to the terminal state with probability 1−p. Let the reward be +1 on all transitions, and let γ =1. Suppose you observe one episode that lasts 10 steps, with a return of 10. What are the ﬁrst-visit and every-visit estimators of the value of the nonterminal state? ⇤


106 Chapter 5: Monte Carlo Methods Example 5.4: O↵-policy Estimation of a Blackjack State Value We applied both ordinary and weighted importance-sampling methods to estimate the value of a single blackjack state (Example 5.1) from o↵-policy data. Recall that one of the advantages of Monte Carlo methods is that they can be used to evaluate a single state without forming estimates for any other states. In this example, we evaluated the state in which the dealer is showing a deuce, the sum of the player’s cards is 13, and the player has a usable ace (that is, the player holds an ace and a deuce, or equivalently three aces). The data was generated by starting in this state then choosing to hit or stick at random with equal probability (the behavior policy).


in this state then choosing to hit or stick at random with equal probability (the behavior policy). The target policy was to stick only on a sum of 20 or 21, as in Example 5.1. The value of this state under the target policy is approximately −0.27726 (this was determined by separately generating one-hundred million episodes using the target policy and averaging their returns). Both o↵-policy methods closely approximated this value after 1000 o↵-policy episodes using the random policy. To make sure they did this reliably, we performed 100 independent runs, each starting from estimates of zero and learning for 10,000 episodes. Figure 5.3 shows the resultant learning curves—the squared error of the estimates of each method as a function of number of episodes, averaged over the 100 runs.


or of the estimates of each method as a function of number of episodes, averaged over the 100 runs. The error approaches zero for both algorithms, but the weighted importance-sampling method has much lower error at the beginning, as is typical in practice. Ordinary importance sampling Weighted importance sampling Episodes (log scale) 0 10 100 1000 10,000 Mean square error (average over 100 runs) 0 5 Figure 5.3: Weighted importance sampling produces lower error estimates of the value of a single blackjack state from o↵-policy episodes. Example 5.5: Inﬁnite Variance The estimates of ordinary importance sampling will typically have inﬁnite variance, and thus unsatisfactory convergence properties, whenever the scaled returns have inﬁnite variance—and this can easily happen in o↵-policy learnin


s, whenever the scaled returns have inﬁnite variance—and this can easily happen in o↵-policy learning when trajectories contain loops. A simple example is shown inset in Figure 5.4. There is only one nonterminal state s and two actions, right and left. The right action causes a deterministic transition to termination, whereas the left action transitions, with probability 0.9, back to s or, with probability 0.1, on to termination. The rewards are +1 on the latter transition and otherwise zero. Consider the target policy that always selects left. All episodes under this policy consist of some number (possibly zero) of transitions back


5.5. O↵-policy Prediction via Importance Sampling 107 to s followed by termination with a reward and return of +1. Thus the value of s under the target policy is 1 (γ = 1). Suppose we are estimating this value from o↵-policy data using the behavior policy that selects right and left with equal probability. 1 100,000 1,000,000 10,000,000 100,000,000 2 0.1 0.9 R = +1 s ⇡(left|s) = 1 left right R = 0 R = 0 b(left|s) = 1 2 v⇡(s) Monte-Carlo estimate of with ordinary importance sampling (ten runs) Episodes (log scale) 1 10 100 1000 10,000 0 Figure 5.4: Ordinary importance sampling produces surprisingly unstable estimates on the one-state MDP shown inset (Example 5.5). The correct estimate here is 1 (γ = 1), and, even though this is the expected value of a sample return (after importance samplin


1 (γ = 1), and, even though this is the expected value of a sample return (after importance sampling), the variance of the samples is inﬁnite, and the estimates do not converge to this value. These results are for o↵-policy ﬁrst-visit MC. The lower part of Figure 5.4 shows ten independent runs of the ﬁrst-visit MC algorithm using ordinary importance sampling. Even after millions of episodes, the estimates fail to converge to the correct value of 1. In contrast, the weighted importance-sampling algorithm would give an estimate of exactly 1 forever after the ﬁrst episode that ended with the left action. All returns not equal to 1 (that is, ending with the right action) would be inconsistent with the target policy and thus would have a ⇢t:T (t)−1 of zero and contribute neither to the numerat


ith the target policy and thus would have a ⇢t:T (t)−1 of zero and contribute neither to the numerator nor denominator of (5.6). The weighted importancesampling algorithm produces a weighted average of only the returns consistent with the target policy, and all of these would be exactly 1. We can verify that the variance of the importance-sampling-scaled returns is inﬁnite in this example by a simple calculation. The variance of any random variable X is the expected value of the deviation from its mean ¯X, which can be written Var[X] .= E h# X −¯X $2i = E ⇥ X2 −2X ¯X + ¯X2⇤ = E ⇥ X2⇤ −¯X2. Thus, if the mean is ﬁnite, as it is in our case, the variance is inﬁnite if and only if the expectation of the square of the random variable is inﬁnite. Thus, we need only show


108 Chapter 5: Monte Carlo Methods that the expected square of the importance-sampling-scaled return is inﬁnite: E 2 4 T −1 Y t=0 ⇡(At|St) b(At|St) G0 !23 5 . To compute this expectation, we break it down into cases based on episode length and termination. First note that, for any episode ending with the right action, the importance sampling ratio is zero, because the target policy would never take this action; these episodes thus contribute nothing to the expectation (the quantity in parenthesis will be zero) and can be ignored. We need only consider episodes that involve some number (possibly zero) of left actions that transition back to the nonterminal state, followed by a left action transitioning to termination.


t transition back to the nonterminal state, followed by a left action transitioning to termination. All of these episodes have a return of 1, so the G0 factor can be ignored. To get the expected square we need only consider each length of episode, multiplying the probability of the episode’s occurrence by the square of its importance-sampling ratio, and add these up: = 1 2 · 0.1 ✓1 0.5 ◆2 (the length 1 episode) + 1 2 · 0.9 · 1 2 · 0.1 ✓1 0.5 1 0.5 ◆2 (the length 2 episode) + 1 2 · 0.9 · 1 2 · 0.9 · 1 2 · 0.1 ✓1 0.5 1 0.5 1 0.5 ◆2 (the length 3 episode) + · · · = 0.1 1 X k=0 0.9k · 2k · 2 = 0.2 1 X k=0 1.8k = 1. Exercise 5.6 What is the equation analogous to (5.6) for action values Q(s, a) instead of state values V (s), again given returns generated using b?


.6) for action values Q(s, a) instead of state values V (s), again given returns generated using b? ⇤ Exercise 5.7 In learning curves such as those shown in Figure 5.3 error generally decreases with training, as indeed happened for the ordinary importance-sampling method. But for the weighted importance-sampling method error ﬁrst increased and then decreased. Why do you think this happened? ⇤ Exercise 5.8 The results with Example 5.5 and shown in Figure 5.4 used a ﬁrst-visit MC method. Suppose that instead an every-visit MC method was used on the same problem. Would the variance of the estimator still be inﬁnite? Why or why not? ⇤


5.7. O↵-policy Monte Carlo Control 109 5.6 Incremental Implementation Monte Carlo prediction methods can be implemented incrementally, on an episode-byepisode basis, using extensions of the techniques described in Chapter 2 (Section 2.4). Whereas in Chapter 2 we averaged rewards, in Monte Carlo methods we average returns. In all other respects exactly the same methods as used in Chapter 2 can be used for onpolicy Monte Carlo methods. For o↵-policy Monte Carlo methods, we need to separately consider those that use ordinary importance sampling and those that use weighted importance sampling. In ordinary importance sampling, the returns are scaled by the importance sampling ratio ⇢t:T (t)−1 (5.3), then simply averaged, as in (5.5).


ns are scaled by the importance sampling ratio ⇢t:T (t)−1 (5.3), then simply averaged, as in (5.5). For these methods we can again use the incremental methods of Chapter 2, but using the scaled returns in place of the rewards of that chapter. This leaves the case of o↵-policy methods using weighted importance sampling. Here we have to form a weighted average of the returns, and a slightly di↵erent incremental algorithm is required. Suppose we have a sequence of returns G1, G2, . . . , Gn−1, all starting in the same state and each with a corresponding random weight Wi (e.g., Wi = ⇢ti:T (ti)−1). We wish to form the estimate Vn .= Pn−1 k=1 WkGk Pn−1 k=1 Wk , n ≥2, (5.7) and keep it up-to-date as we obtain a single additional return Gn.


1 WkGk Pn−1 k=1 Wk , n ≥2, (5.7) and keep it up-to-date as we obtain a single additional return Gn. In addition to keeping track of Vn, we must maintain for each state the cumulative sum Cn of the weights given to the ﬁrst n returns. The update rule for Vn is Vn+1 .= Vn + Wn Cn h Gn −Vn i , n ≥1, (5.8) and Cn+1 .= Cn + Wn+1, where C0 .= 0 (and V1 is arbitrary and thus need not be speciﬁed). The box on the next page contains a complete episode-by-episode incremental algorithm for Monte Carlo policy evaluation. The algorithm is nominally for the o↵-policy case, using weighted importance sampling, but applies as well to the on-policy case just by choosing the target and behavior policies as the same (in which case (⇡= b), W is always 1).


ust by choosing the target and behavior policies as the same (in which case (⇡= b), W is always 1). The approximation Q converges to q⇡(for all encountered state–action pairs) while actions are selected according to a potentially di↵erent policy, b. Exercise 5.9 Modify the algorithm for ﬁrst-visit MC policy evaluation (Section 5.1) to use the incremental implementation for sample averages described in Section 2.4. ⇤ Exercise 5.10 Derive the weighted-average update rule (5.8) from (5.7). Follow the pattern of the derivation of the unweighted rule (2.3). ⇤


110 Chapter 5: Monte Carlo Methods O↵-policy MC prediction (policy evaluation) for estimating Q ⇡q⇡ Input: an arbitrary target policy ⇡ Initialize, for all s 2 S, a 2 A(s): Q(s, a) 2 R (arbitrarily) C(s, a) 0 Loop forever (for each episode): b any policy with coverage of ⇡ Generate an episode following b: S0, A0, R1, . . . , ST −1, AT −1, RT G 0 W 1 Loop for each step of episode, t = T −1, T −2, . . . , 0, while W 6= 0: G γG + Rt+1 C(St, At) C(St, At) + W Q(St, At) Q(St, At) + W C(St,At) [G −Q(St, At)] W W ⇡(At|St) b(At|St) 5.7 O↵-policy Monte Carlo Control We are now ready to present an example of the second class of learning control methods we consider in this book: o↵-policy methods. Recall that the distinguishing feature of on-policy methods is that they estimate the value of a policy


ll that the distinguishing feature of on-policy methods is that they estimate the value of a policy while using it for control. In o↵-policy methods these two functions are separated. The policy used to generate behavior, called the behavior policy, may in fact be unrelated to the policy that is evaluated and improved, called the target policy. An advantage of this separation is that the target policy may be deterministic (e.g., greedy), while the behavior policy can continue to sample all possible actions. O↵-policy Monte Carlo control methods use one of the techniques presented in the preceding two sections. They follow the behavior policy while learning about and improving the target policy.


two sections. They follow the behavior policy while learning about and improving the target policy. These techniques require that the behavior policy has a nonzero probability of selecting all actions that might be selected by the target policy (coverage). To explore all possibilities, we require that the behavior policy be soft (i.e., that it select all actions in all states with nonzero probability). The box on the next page shows an o↵-policy Monte Carlo control method, based on GPI and weighted importance sampling, for estimating ⇡⇤and q⇤. The target policy ⇡⇡⇡⇤is the greedy policy with respect to Q, which is an estimate of q⇡. The behavior policy b can be anything, but in order to assure convergence of ⇡to the optimal policy, an inﬁnite number of returns must be obtained for each pair


e convergence of ⇡to the optimal policy, an inﬁnite number of returns must be obtained for each pair of state and action. This can be assured by choosing b to be "-soft. The policy ⇡converges to optimal at all encountered states even though actions are selected according to a di↵erent soft policy b, which may change between or even within episodes.


5.7. O↵-policy Monte Carlo Control 111 O↵-policy MC control, for estimating ⇡⇡⇡⇤ Initialize, for all s 2 S, a 2 A(s): Q(s, a) 2 R (arbitrarily) C(s, a) 0 ⇡(s) argmaxa Q(s, a) (with ties broken consistently) Loop forever (for each episode): b any soft policy Generate an episode using b: S0, A0, R1, . . . , ST −1, AT −1, RT G 0 W 1 Loop for each step of episode, t = T −1, T −2, . . . , 0: G γG + Rt+1 C(St, At) C(St, At) + W Q(St, At) Q(St, At) + W C(St,At) [G −Q(St, At)] ⇡(St) argmaxa Q(St, a) (with ties broken consistently) If At 6= ⇡(St) then exit inner Loop (proceed to next episode) W W 1 b(At|St) A potential problem is that this method learns only from the tails of episodes, when all of the remaining actions in the episode are greedy.


earns only from the tails of episodes, when all of the remaining actions in the episode are greedy. If nongreedy actions are common, then learning will be slow, particularly for states appearing in the early portions of long episodes. Potentially, this could greatly slow learning. There has been insuﬃcient experience with o↵-policy Monte Carlo methods to assess how serious this problem is. If it is serious, the most important way to address it is probably by incorporating temporaldi↵erence learning, the algorithmic idea developed in the next chapter. Alternatively, if γ is less than 1, then the idea developed in the next section may also help signiﬁcantly. Exercise 5.11 In the boxed algorithm for o↵-policy MC control, you may have been expecting the W update to have involved the importance


m for o↵-policy MC control, you may have been expecting the W update to have involved the importance-sampling ratio ⇡(At|St) b(At|St) , but instead it involves 1 b(At|St). Why is this nevertheless correct? ⇤ Exercise 5.12: Racetrack (programming) Consider driving a race car around a turn like those shown in Figure 5.5. You want to go as fast as possible, but not so fast as to run o↵the track. In our simpliﬁed racetrack, the car is at one of a discrete set of grid positions, the cells in the diagram. The velocity is also discrete, a number of grid cells moved horizontally and vertically per time step. The actions are increments to the velocity components. Each may be changed by +1, −1, or 0 in each step, for a total of nine (3 ⇥3) actions.


y components. Each may be changed by +1, −1, or 0 in each step, for a total of nine (3 ⇥3) actions. Both velocity components are restricted to be nonnegative and less than 5, and they cannot both be zero except at the starting line. Each episode begins in one of the randomly selected start states with both velocity components zero and ends when the car crosses the ﬁnish line. The rewards are −1 for each step until the car crosses the ﬁnish line. If the car hits the track boundary, it is moved back to a random position on the starting line, both velocity components are reduced to zero, and the


112 Chapter 5: Monte Carlo Methods Starting line Finish line Starting line Finish line Figure 5.5: A couple of right turns for the racetrack task. episode continues. Before updating the car’s location at each time step, check to see if the projected path of the car intersects the track boundary. If it intersects the ﬁnish line, the episode ends; if it intersects anywhere else, the car is considered to have hit the track boundary and is sent back to the starting line. To make the task more challenging, with probability 0.1 at each time step the velocity increments are both zero, independently of the intended increments. Apply a Monte Carlo control method to this task to compute the optimal policy from each starting state.


y a Monte Carlo control method to this task to compute the optimal policy from each starting state. Exhibit several trajectories following the optimal policy (but turn the noise o↵for these trajectories). ⇤ 5.8 *Discounting-aware Importance Sampling The o↵-policy methods that we have considered so far are based on forming importancesampling weights for returns considered as unitary wholes, without taking into account the returns’ internal structures as sums of discounted rewards. We now brieﬂy consider cutting-edge research ideas for using this structure to signiﬁcantly reduce the variance of o↵-policy estimators. For example, consider the case where episodes are long and γ is signiﬁcantly less than 1. For concreteness, say that episodes last 100 steps and that γ = 0.


d γ is signiﬁcantly less than 1. For concreteness, say that episodes last 100 steps and that γ = 0. The return from time 0 will then be just G0 = R1, but its importance sampling ratio will be a product of 100 factors, ⇡(A0|S0) b(A0|S0) ⇡(A1|S1) b(A1|S1) · · · ⇡(A99|S99) b(A99|S99) . In ordinary importance sampling, the return will be scaled by the entire product, but it is really only necessary to scale by the ﬁrst factor, by ⇡(A0|S0) b(A0|S0) . The other 99 factors ⇡(A1|S1) b(A1|S1) · · · ⇡(A99|S99) b(A99|S99) are irrelevant because after the ﬁrst reward the return has already been determined. These later factors are all independent of the return and of expected value 1; they do not change the expected update, but they add enormously to its variance.


expected value 1; they do not change the expected update, but they add enormously to its variance. In some cases they could even make the variance inﬁnite. Let us now consider an idea for avoiding this large extraneous variance.


5.9. Per-decision Importance Sampling 113 The essence of the idea is to think of discounting as determining a probability of termination or, equivalently, a degree of partial termination. For any γ 2 [0, 1), we can think of the return G0 as partly terminating in one step, to the degree 1 −γ, producing a return of just the ﬁrst reward, R1, and as partly terminating after two steps, to the degree (1 −γ)γ, producing a return of R1 + R2, and so on. The latter degree corresponds to terminating on the second step, 1 −γ, and not having already terminated on the ﬁrst step, γ. The degree of termination on the third step is thus (1 −γ)γ2, with the γ2 reﬂecting that termination did not occur on either of the ﬁrst two steps.


hus (1 −γ)γ2, with the γ2 reﬂecting that termination did not occur on either of the ﬁrst two steps. The partial returns here are called ﬂat partial returns: ¯Gt:h .= Rt+1 + Rt+2 + · · · + Rh, 0 t < h T, where “ﬂat” denotes the absence of discounting, and “partial” denotes that these returns do not extend all the way to termination but instead stop at h, called the horizon (and T is the time of termination of the episode). The conventional full return Gt can be viewed as a sum of ﬂat partial returns as suggested above as follows: Gt .= Rt+1 + γRt+2 + γ2Rt+3 + · · · + γT −t−1RT = (1 −γ)Rt+1 + (1 −γ)γ (Rt+1 + Rt+2) + (1 −γ)γ2 (Rt+1 + Rt+2 + Rt+3) ... + (1 −γ)γT −t−2 (Rt+1 + Rt+2 + · · · + RT −1) + γT −t−1 (Rt+1 + Rt+2 + · · · + RT ) = (1 −γ) T −1 X h=t+1 γh−t−1 ¯Gt:h + γT −t−1 ¯Gt:T .


+ RT −1) + γT −t−1 (Rt+1 + Rt+2 + · · · + RT ) = (1 −γ) T −1 X h=t+1 γh−t−1 ¯Gt:h + γT −t−1 ¯Gt:T . Now we need to scale the ﬂat partial returns by an importance sampling ratio that is similarly truncated. As ¯Gt:h only involves rewards up to a horizon h, we only need the ratio of the probabilities up to h −1. We deﬁne an ordinary importance-sampling estimator, analogous to (5.5), as V (s) .= P t2T(s) ⇣ (1 −γ) PT (t)−1 h=t+1 γh−t−1⇢t:h−1 ¯Gt:h + γT (t)−t−1⇢t:T (t)−1 ¯Gt:T (t) ⌘ |T(s)| , (5.9) and a weighted importance-sampling estimator, analogous to (5.6), as V (s) .= P t2T(s) ⇣ (1 −γ) PT (t)−1 h=t+1 γh−t−1⇢t:h−1 ¯Gt:h + γT (t)−t−1⇢t:T (t)−1 ¯Gt:T (t) ⌘ P t2T(s) ⇣ (1 −γ) PT (t)−1 h=t+1 γh−t−1⇢t:h−1 + γT (t)−t−1⇢t:T (t)−1 ⌘ .


−t−1⇢t:T (t)−1 ¯Gt:T (t) ⌘ P t2T(s) ⇣ (1 −γ) PT (t)−1 h=t+1 γh−t−1⇢t:h−1 + γT (t)−t−1⇢t:T (t)−1 ⌘ . (5.10) We call these two estimators discounting-aware importance sampling estimators. They take into account the discount rate but have no e↵ect (are the same as the o↵-policy estimators from Section 5.5) if γ = 1.


114 Chapter 5: Monte Carlo Methods 5.9 *Per-decision Importance Sampling There is one more way in which the structure of the return as a sum of rewards can be taken into account in o↵-policy importance sampling, a way that may be able to reduce variance even in the absence of discounting (that is, even if γ = 1). In the o↵-policy estimators (5.5) and (5.6), each term of the sum in the numerator is itself a sum: ⇢t:T −1Gt = ⇢t:T −1 # Rt+1 + γRt+2 + · · · + γT −t−1RT $ = ⇢t:T −1Rt+1 + γ⇢t:T −1Rt+2 + · · · + γT −t−1⇢t:T −1RT . (5.11) The o↵-policy estimators rely on the expected values of these terms, which can be written in a simpler way. Note that each sub-term of (5.11) is a product of a random reward and a random importance-sampling ratio.


hat each sub-term of (5.11) is a product of a random reward and a random importance-sampling ratio. For example, the ﬁrst sub-term can be written, using (5.3), as ⇢t:T −1Rt+1 = ⇡(At|St) b(At|St) ⇡(At+1|St+1) b(At+1|St+1) ⇡(At+2|St+2) b(At+2|St+2) · · · ⇡(AT −1|ST −1) b(AT −1|ST −1) Rt+1. (5.12) Of all these factors, one might suspect that only the ﬁrst and the last (the reward) are related; all the others are for events that occurred after the reward. Moreover, the expected value of all these other factors is one: E ⇡(Ak|Sk) b(Ak|Sk) 3 .= X a b(a|Sk)⇡(a|Sk) b(a|Sk) = X a ⇡(a|Sk) = 1. (5.13) With a few more steps, one can show that, as suspected, all of these other factors have no e↵ect in expectation, in other words, that E[⇢t:T −1Rt+1] = E[⇢t:tRt+1] .


ese other factors have no e↵ect in expectation, in other words, that E[⇢t:T −1Rt+1] = E[⇢t:tRt+1] . (5.14) If we repeat this process for the kth sub-term of (5.11), we get E[⇢t:T −1Rt+k] = E[⇢t:t+k−1Rt+k] . It follows then that the expectation of our original term (5.11) can be written E[⇢t:T −1Gt] = E h ˜Gt i , where ˜Gt = ⇢t:tRt+1 + γ⇢t:t+1Rt+2 + γ2⇢t:t+2Rt+3 + · · · + γT −t−1⇢t:T −1RT . We call this idea per-decision importance sampling.


5.10. Summary 115 It follows immediately that there is an alternate importance-sampling estimator, with the same unbiased expectation (in the ﬁrst-visit case) as the ordinary-importance-sampling estimator (5.5), using ˜Gt: V (s) .= P t2T(s) ˜Gt |T(s)| , (5.15) which we might expect to sometimes be of lower variance. Is there a per-decision version of weighted importance sampling? This is less clear. So far, all the estimators that have been proposed for this that we know of are not consistent (that is, they do not converge to the true value with inﬁnite data). ⇤Exercise 5.13 Show the steps to derive (5.14) from (5.12). ⇤ ⇤Exercise 5.14 Modify the algorithm for o↵-policy Monte Carlo control (page 111) to use the idea of the truncated weighted-average estimator (5.10).


Monte Carlo control (page 111) to use the idea of the truncated weighted-average estimator (5.10). Note that you will ﬁrst need to convert this equation to action values. ⇤ 5.10 Summary The Monte Carlo methods presented in this chapter learn value functions and optimal policies from experience in the form of sample episodes. This gives them at least three kinds of advantages over DP methods. First, they can be used to learn optimal behavior directly from interaction with the environment, with no model of the environment’s dynamics. Second, they can be used with simulation or sample models. For surprisingly many applications it is easy to simulate sample episodes even though it is diﬃcult to construct the kind of explicit model of transition probabilities required by DP methods.


diﬃcult to construct the kind of explicit model of transition probabilities required by DP methods. Third, it is easy and eﬃcient to focus Monte Carlo methods on a small subset of the states. A region of special interest can be accurately evaluated without going to the expense of accurately evaluating the rest of the state set (we explore this further in Chapter 8). A fourth advantage of Monte Carlo methods, which we discuss later in the book, is that they may be less harmed by violations of the Markov property. This is because they do not update their value estimates on the basis of the value estimates of successor states. In other words, it is because they do not bootstrap. In designing Monte Carlo control methods we have followed the overall schema of generalized policy iteration (GPI)


nte Carlo control methods we have followed the overall schema of generalized policy iteration (GPI) introduced in Chapter 4. GPI involves interacting processes of policy evaluation and policy improvement. Monte Carlo methods provide an alternative policy evaluation process. Rather than use a model to compute the value of each state, they simply average many returns that start in the state. Because a state’s value is the expected return, this average can become a good approximation to the value. In control methods we are particularly interested in approximating action-value functions, because these can be used to improve the policy without requiring a model of the environment’s transition dynamics.


n be used to improve the policy without requiring a model of the environment’s transition dynamics. Monte Carlo methods intermix policy evaluation and policy improvement steps on an episode-by-episode basis, and can be incrementally implemented on an episode-by-episode basis.


116 Chapter 5: Monte Carlo Methods Maintaining suﬃcient exploration is an issue in Monte Carlo control methods. It is not enough just to select the actions currently estimated to be best, because then no returns will be obtained for alternative actions, and it may never be learned that they are actually better. One approach is to ignore this problem by assuming that episodes begin with state–action pairs randomly selected to cover all possibilities. Such exploring starts can sometimes be arranged in applications with simulated episodes, but are unlikely in learning from real experience. In on-policy methods, the agent commits to always exploring and tries to ﬁnd the best policy that still explores.


ethods, the agent commits to always exploring and tries to ﬁnd the best policy that still explores. In o↵-policy methods, the agent also explores, but learns a deterministic optimal policy that may be unrelated to the policy followed. O↵-policy prediction refers to learning the value function of a target policy from data generated by a di↵erent behavior policy. Such learning methods are based on some form of importance sampling, that is, on weighting returns by the ratio of the probabilities of taking the observed actions under the two policies, thereby transforming their expectations from the behavior policy to the target policy. Ordinary importance sampling uses a simple average of the weighted returns, whereas weighted importance sampling uses a weighted average.


mple average of the weighted returns, whereas weighted importance sampling uses a weighted average. Ordinary importance sampling produces unbiased estimates, but has larger, possibly inﬁnite, variance, whereas weighted importance sampling always has ﬁnite variance and is preferred in practice. Despite their conceptual simplicity, o↵-policy Monte Carlo methods for both prediction and control remain unsettled and are a subject of ongoing research. The Monte Carlo methods treated in this chapter di↵er from the DP methods treated in the previous chapter in two major ways. First, they operate on sample experience, and thus can be used for direct learning without a model. Second, they do not bootstrap. That is, they do not update their value estimates on the basis of other value estimates.


bootstrap. That is, they do not update their value estimates on the basis of other value estimates. These two di↵erences are not tightly linked, and can be separated. In the next chapter we consider methods that learn from experience, like Monte Carlo methods, but also bootstrap, like DP methods. Bibliographical and Historical Remarks The term “Monte Carlo” dates from the 1940s, when physicists at Los Alamos devised games of chance that they could study to help understand complex physical phenomena relating to the atom bomb. Coverage of Monte Carlo methods in this sense can be found in several textbooks (e.g., Kalos and Whitlock, 1986; Rubinstein, 1981). 5.1–2 Singh and Sutton (1996) distinguished between every-visit and ﬁrst-visit MC methods and proved results relating these methods to re


uished between every-visit and ﬁrst-visit MC methods and proved results relating these methods to reinforcement learning algorithms. The blackjack example is based on an example used by Widrow, Gupta, and Maitra (1973). The soap bubble example is a classical Dirichlet problem whose Monte Carlo solution was ﬁrst proposed by Kakutani (1945; see Hersh and Griego, 1969; Doyle and Snell, 1984). Barto and Du↵(1994) discussed policy evaluation in the context of classical Monte Carlo algorithms for solving systems of linear equations. They used the


5.10. Summary 117 analysis of Curtiss (1954) to point out the computational advantages of Monte Carlo policy evaluation for large problems. 5.3–4 Monte Carlo ES was introduced in the 1998 edition of this book. That may have been the ﬁrst explicit connection between Monte Carlo estimation and control methods based on policy iteration. An early use of Monte Carlo methods to estimate action values in a reinforcement learning context was by Michie and Chambers (1968). In pole balancing (page 56), they used averages of episode durations to assess the worth (expected balancing “life”) of each possible action in each state, and then used these assessments to control action selections. Their method is similar in spirit to Monte Carlo ES with every-visit MC estimates.


tion selections. Their method is similar in spirit to Monte Carlo ES with every-visit MC estimates. Narendra and Wheeler (1986) studied a Monte Carlo method for ergodic ﬁnite Markov chains that used the return accumulated between successive visits to the same state as a reward for adjusting a learning automaton’s action probabilities. 5.5 Eﬃcient o↵-policy learning has become recognized as an important challenge that arises in several ﬁelds. For example, it is closely related to the idea of “interventions” and “counterfactuals” in probabilistic graphical (Bayesian) models (e.g., Pearl, 1995; Balke and Pearl, 1994). O↵-policy methods using importance sampling have a long history and yet still are not well understood.


policy methods using importance sampling have a long history and yet still are not well understood. Weighted importance sampling, which is also sometimes called normalized importance sampling (e.g., Koller and Friedman, 2009), is discussed by Rubinstein (1981), Hesterberg (1988), Shelton (2001), and Liu (2001) among others. The target policy in o↵-policy learning is sometimes referred to in the literature as the “estimation” policy, as it was in the ﬁrst edition of this book. 5.7 The racetrack exercise is adapted from Barto, Bradtke, and Singh (1995), and from Gardner (1973). 5.8 Our treatment of the idea of discounting-aware importance sampling is based on the analysis of Sutton, Mahmood, Precup, and van Hasselt (2014).


re importance sampling is based on the analysis of Sutton, Mahmood, Precup, and van Hasselt (2014). It has been worked out most fully to date by Mahmood (2017; Mahmood, van Hasselt, and Sutton, 2014). 5.9 Per-decision importance sampling was introduced by Precup, Sutton, and Singh (2000). They also combined o↵-policy learning with temporal-di↵erence learning, eligibility traces, and approximation methods, introducing subtle issues that we consider in later chapters. Exercise 5.15 Make new equations analogous to the importance-sampling Monte Carlo estimates (5.5) and (5.6), but for action value estimates Q(s, a). You will need new notation T(s, a) for the time steps on which the state–action pair s, a is visited on the episode.


notation T(s, a) for the time steps on which the state–action pair s, a is visited on the episode. Do these estimates involve more or less importance-sampling correction?


Chapter 6 Temporal-Di↵erence Learning If one had to identify one idea as central and novel to reinforcement learning, it would undoubtedly be temporal-di↵erence (TD) learning. TD learning is a combination of Monte Carlo ideas and dynamic programming (DP) ideas. Like Monte Carlo methods, TD methods can learn directly from raw experience without a model of the environment’s dynamics. Like DP, TD methods update estimates based in part on other learned estimates, without waiting for a ﬁnal outcome (they bootstrap). The relationship between TD, DP, and Monte Carlo methods is a recurring theme in the theory of reinforcement learning; this chapter is the beginning of our exploration of it. Before we are done, we will see that these ideas and methods blend into each other and can be combined in ma


e are done, we will see that these ideas and methods blend into each other and can be combined in many ways. In particular, in Chapter 7 we introduce n-step algorithms, which provide a bridge from TD to Monte Carlo methods, and in Chapter 12 we introduce the TD(λ) algorithm, which seamlessly uniﬁes them. As usual, we start by focusing on the policy evaluation or prediction problem, the problem of estimating the value function v⇡for a given policy ⇡. For the control problem (ﬁnding an optimal policy), DP, TD, and Monte Carlo methods all use some variation of generalized policy iteration (GPI). The di↵erences in the methods are primarily di↵erences in their approaches to the prediction problem. 6.1 TD Prediction Both TD and Monte Carlo methods use experience to solve the prediction problem.


. 6.1 TD Prediction Both TD and Monte Carlo methods use experience to solve the prediction problem. Given some experience following a policy ⇡, both methods update their estimate V of v⇡for the nonterminal states St occurring in that experience. Roughly speaking, Monte Carlo methods wait until the return following the visit is known, then use that return as a target for V (St). A simple every-visit Monte Carlo method suitable for nonstationary environments is V (St) V (St) + ↵ h Gt −V (St) i , (6.1)


120 Chapter 6: Temporal-Di↵erence Learning where Gt is the actual return following time t, and ↵is a constant step-size parameter (c.f., Equation 2.4). Let us call this method constant-↵MC. Whereas Monte Carlo methods must wait until the end of the episode to determine the increment to V (St) (only then is Gt known), TD methods need to wait only until the next time step. At time t + 1 they immediately form a target and make a useful update using the observed reward Rt+1 and the estimate V (St+1). The simplest TD method makes the update V (St) V (St) + ↵ h Rt+1 + γV (St+1) −V (St) i (6.2) immediately on transition to St+1 and receiving Rt+1. In e↵ect, the target for the Monte Carlo update is Gt, whereas the target for the TD update is Rt+1 + γV (St+1).


target for the Monte Carlo update is Gt, whereas the target for the TD update is Rt+1 + γV (St+1). This TD method is called TD(0), or one-step TD, because it is a special case of the TD(λ) and n-step TD methods developed in Chapter 12 and Chapter 7. The box below speciﬁes TD(0) completely in procedural form. Tabular TD(0) for estimating v⇡ Input: the policy ⇡to be evaluated Algorithm parameter: step size ↵2 (0, 1] Initialize V (s), for all s 2 S+, arbitrarily except that V (terminal) = 0 Loop for each episode: Initialize S Loop for each step of episode: A action given by ⇡for S Take action A, observe R, S0 V (S) V (S) + ↵ ⇥ R + γV (S0) −V (S) ⇤ S S0 until S is terminal Because TD(0) bases its update in part on an existing estimate, we say that it is a bootstrapping method, like DP.


ases its update in part on an existing estimate, we say that it is a bootstrapping method, like DP. We know from Chapter 3 that v⇡(s) .= E⇡[Gt | St =s] (6.3) = E⇡[Rt+1 + γGt+1 | St =s] (from (3.9)) = E⇡[Rt+1 + γv⇡(St+1) | St =s] . (6.4) Roughly speaking, Monte Carlo methods use an estimate of (6.3) as a target, whereas DP methods use an estimate of (6.4) as a target. The Monte Carlo target is an estimate because the expected value in (6.3) is not known; a sample return is used in place of the real expected return. The DP target is an estimate not because of the expected values, which are assumed to be completely provided by a model of the environment, but because v⇡(St+1) is not known and the current estimate, V (St+1), is used instead.


environment, but because v⇡(St+1) is not known and the current estimate, V (St+1), is used instead. The TD target is an estimate for both reasons: it samples the expected values in (6.4) and it uses the current estimate V instead of the true v⇡. Thus, TD methods combine the sampling of


6.1. TD Prediction 121 Monte Carlo with the bootstrapping of DP. As we shall see, with care and imagination this can take us a long way toward obtaining the advantages of both Monte Carlo and DP methods. TD(0) Shown to the right is the backup diagram for tabular TD(0). The value estimate for the state node at the top of the backup diagram is updated on the basis of the one sample transition from it to the immediately following state. We refer to TD and Monte Carlo updates as sample updates because they involve looking ahead to a sample successor state (or state–action pair), using the value of the successor and the reward along the way to compute a backed-up value, and then updating the value of the original state (or state– action pair) accordingly.


ed-up value, and then updating the value of the original state (or state– action pair) accordingly. Sample updates di↵er from the expected updates of DP methods in that they are based on a single sample successor rather than on a complete distribution of all possible successors. Finally, note that the quantity in brackets in the TD(0) update is a sort of error, measuring the di↵erence between the estimated value of St and the better estimate Rt+1 + γV (St+1). This quantity, called the TD error, arises in various forms throughout reinforcement learning: δt .= Rt+1 + γV (St+1) −V (St). (6.5) Notice that the TD error at each time is the error in the estimate made at that time. Because the TD error depends on the next state and next reward, it is not actually available until one time step late


ror depends on the next state and next reward, it is not actually available until one time step later. That is, δt is the error in V (St), available at time t + 1. Also note that if the array V does not change during the episode (as it does not in Monte Carlo methods), then the Monte Carlo error can be written as a sum of TD errors: Gt −V (St) = Rt+1 + γGt+1 −V (St) + γV (St+1) −γV (St+1) (from (3.9)) = δt + γ # Gt+1 −V (St+1) $ = δt + γδt+1 + γ2# Gt+2 −V (St+2) $ = δt + γδt+1 + γ2δt+2 + · · · + γT −t−1δT −1 + γT −t# GT −V (ST ) $ = δt + γδt+1 + γ2δt+2 + · · · + γT −t−1δT −1 + γT −t# 0 −0 $ = T −1 X k=t γk−tδk. (6.6) This identity is not exact if V is updated during the episode (as it is in TD(0)), but if the step size is small then it may still hold approximately.


he episode (as it is in TD(0)), but if the step size is small then it may still hold approximately. Generalizations of this identity play an important role in the theory and algorithms of temporal-di↵erence learning. Exercise 6.1 If V changes during the episode, then (6.6) only holds approximately; what would the di↵erence be between the two sides? Let Vt denote the array of state values used at time t in the TD error (6.5) and in the TD update (6.2). Redo the derivation above to determine the additional amount that must be added to the sum of TD errors in order to equal the Monte Carlo error. ⇤


122 Chapter 6: Temporal-Di↵erence Learning Example 6.1: Driving Home Each day as you drive home from work, you try to predict how long it will take to get home. When you leave your oﬃce, you note the time, the day of week, the weather, and anything else that might be relevant. Say on this Friday you are leaving at exactly 6 o’clock, and you estimate that it will take 30 minutes to get home. As you reach your car it is 6:05, and you notice it is starting to rain. Traﬃc is often slower in the rain, so you reestimate that it will take 35 minutes from then, or a total of 40 minutes. Fifteen minutes later you have completed the highway portion of your journey in good time. As you exit onto a secondary road you cut your estimate of total travel time to 35 minutes.


d time. As you exit onto a secondary road you cut your estimate of total travel time to 35 minutes. Unfortunately, at this point you get stuck behind a slow truck, and the road is too narrow to pass. You end up having to follow the truck until you turn onto the side street where you live at 6:40. Three minutes later you are home. The sequence of states, times, and predictions is thus as follows: Elapsed Time Predicted Predicted State (minutes) Time to Go Total Time leaving oﬃce, friday at 6 0 30 30 reach car, raining 5 35 40 exiting highway 20 15 35 2ndary road, behind truck 30 10 40 entering home street 40 3 43 arrive home 43 0 43 The rewards in this example are the elapsed times on each leg of the journey.1 We are not discounting (γ = 1), and thus the return for each state is the actual


g of the journey.1 We are not discounting (γ = 1), and thus the return for each state is the actual time to go from that state. The value of each state is the expected time to go. The second column of numbers gives the current estimated value for each state encountered. A simple way to view the operation of Monte Carlo methods is to plot the predicted total time (the last column) over the sequence, as in Figure 6.1 (left). The red arrows show the changes in predictions recommended by the constant-↵MC method (6.1), for ↵= 1. These are exactly the errors between the estimated value (predicted time to go) in each state and the actual return (actual time to go). For example, when you exited the highway you thought it would take only 15 minutes more to get home, but in fact it took 23 minutes.


highway you thought it would take only 15 minutes more to get home, but in fact it took 23 minutes. Equation 6.1 applies at this point and determines an increment in the estimate of time to go after exiting the highway. The error, Gt −V (St), at this time is eight minutes. Suppose the step-size parameter, ↵, is 1/2. Then the predicted time to go after exiting the highway would be revised upward by four minutes as a result of this experience. This is probably too large a change in this case; the truck was probably just an unlucky break. In any event, the change can only be made o↵-line, that is, after you have reached home. Only at this point do you know any of the actual returns. Is it necessary to wait until the ﬁnal outcome is known before learning can begin?


actual returns. Is it necessary to wait until the ﬁnal outcome is known before learning can begin? Suppose on another day you again estimate when leaving your oﬃce that it will take 30 minutes to drive home, but then you become stuck in a massive traﬃc jam. Twenty-ﬁve minutes after leaving the oﬃce you are still bumper-to-bumper on the highway. You now 1If this were a control problem with the objective of minimizing travel time, then we would of course make the rewards the negative of the elapsed time. But because we are concerned here only with prediction (policy evaluation), we can keep things simple by using positive numbers.


6.1. TD Prediction 123 road 30 35 40 45 Predicted total travel time leaving office exiting highway 2ndary home arrive Situation actual outcome reach car street home actual outcome Situation 30 35 40 45 road leaving office exiting highway 2ndary home arrive reach car street home Figure 6.1: Changes recommended in the driving home example by Monte Carlo methods (left) and TD methods (right). estimate that it will take another 25 minutes to get home, for a total of 50 minutes. As you wait in traﬃc, you already know that your initial estimate of 30 minutes was too optimistic. Must you wait until you get home before increasing your estimate for the initial state? According to the Monte Carlo approach you must, because you don’t yet know the true return.


state? According to the Monte Carlo approach you must, because you don’t yet know the true return. According to a TD approach, on the other hand, you would learn immediately, shifting your initial estimate from 30 minutes toward 50. In fact, each estimate would be shifted toward the estimate that immediately follows it. Returning to our ﬁrst day of driving, Figure 6.1 (right) shows the changes in the predictions recommended by the TD rule (6.2) (these are the changes made by the rule if ↵= 1). Each error is proportional to the change over time of the prediction, that is, to the temporal di↵erences in predictions. Besides giving you something to do while waiting in traﬃc, there are several computational reasons why it is advantageous to learn based on your current predictions rather than w


omputational reasons why it is advantageous to learn based on your current predictions rather than waiting until termination when you know the actual return. We brieﬂy discuss some of these in the next section. Exercise 6.2 This is an exercise to help develop your intuition about why TD methods are often more eﬃcient than Monte Carlo methods. Consider the driving home example and how it is addressed by TD and Monte Carlo methods. Can you imagine a scenario in which a TD update would be better on average than a Monte Carlo update? Give an example scenario—a description of past experience and a current state—in which you would expect the TD update to be better. Here’s a hint: Suppose you have lots of experience driving home from work.


TD update to be better. Here’s a hint: Suppose you have lots of experience driving home from work. Then you move to a new building and a new parking lot (but you still enter the highway at the same place). Now you are starting to learn predictions for the new building. Can you see why TD updates are likely to be much better, at least initially, in this case? Might the same sort of thing happen in the original scenario? ⇤


124 Chapter 6: Temporal-Di↵erence Learning 6.2 Advantages of TD Prediction Methods TD methods update their estimates based in part on other estimates. They learn a guess from a guess—they bootstrap. Is this a good thing to do? What advantages do TD methods have over Monte Carlo and DP methods? Developing and answering such questions will take the rest of this book and more. In this section we brieﬂy anticipate some of the answers. Obviously, TD methods have an advantage over DP methods in that they do not require a model of the environment, of its reward and next-state probability distributions. The next most obvious advantage of TD methods over Monte Carlo methods is that they are naturally implemented in an online, fully incremental fashion.


Monte Carlo methods is that they are naturally implemented in an online, fully incremental fashion. With Monte Carlo methods one must wait until the end of an episode, because only then is the return known, whereas with TD methods one need wait only one time step. Surprisingly often this turns out to be a critical consideration. Some applications have very long episodes, so that delaying all learning until the end of the episode is too slow. Other applications are continuing tasks and have no episodes at all. Finally, as we noted in the previous chapter, some Monte Carlo methods must ignore or discount episodes on which experimental actions are taken, which can greatly slow learning. TD methods are much less susceptible to these problems because they learn from each transition regardless o


ods are much less susceptible to these problems because they learn from each transition regardless of what subsequent actions are taken. But are TD methods sound? Certainly it is convenient to learn one guess from the next, without waiting for an actual outcome, but can we still guarantee convergence to the correct answer? Happily, the answer is yes. For any ﬁxed policy ⇡, TD(0) has been proved to converge to v⇡, in the mean for a constant step-size parameter if it is suﬃciently small, and with probability 1 if the step-size parameter decreases according to the usual stochastic approximation conditions (2.7). Most convergence proofs apply only to the table-based case of the algorithm presented above (6.2), but some also apply to the case of general linear function approximation.


hm presented above (6.2), but some also apply to the case of general linear function approximation. These results are discussed in a more general setting in Section 9.4. If both TD and Monte Carlo methods converge asymptotically to the correct predictions, then a natural next question is “Which gets there ﬁrst?” In other words, which method learns faster? Which makes the more eﬃcient use of limited data? At the current time this is an open question in the sense that no one has been able to prove mathematically that one method converges faster than the other. In fact, it is not even clear what is the most appropriate formal way to phrase this question! In practice, however, TD methods have usually been found to converge faster than constant-↵MC methods on stochastic tasks, as illustrated in


ually been found to converge faster than constant-↵MC methods on stochastic tasks, as illustrated in Example 6.2.


6.2. Advantages of TD Prediction Methods 125 Example 6.2 Random Walk In this example we empirically compare the prediction abilities of TD(0) and constant-↵MC when applied to the following Markov reward process: A B C D E 1 0 0 0 0 0 start A Markov reward process, or MRP, is a Markov decision process without actions. We will often use MRPs when focusing on the prediction problem, in which there is no need to distinguish the dynamics due to the environment from those due to the agent. In this MRP, all episodes start in the center state, C, then proceed either left or right by one state on each step, with equal probability. Episodes terminate either on the extreme left or the extreme right. When an episode terminates on the right, a reward of +1 occurs; all other rewards are zero.


right. When an episode terminates on the right, a reward of +1 occurs; all other rewards are zero. For example, a typical episode might consist of the following state-and-reward sequence: C, 0, B, 0, C, 0, D, 0, E, 1. Because this task is undiscounted, the true value of each state is the probability of terminating on the right if starting from that state. Thus, the true value of the center state is v⇡(C) = 0.5. The true values of all the states, A through E, are 1 6, 2 6, 3 6, 4 6, and 5 6. 0.8 0 0.2 0.4 0.6 A B C D E 0 10 1 100 State True values Estimated value 0 0.05 0.1 0.15 0.2 0.25 0 25 50 75 100 Walks / Episodes TD MC _  _  _  _  _  _  _  Empirical RMS error, averaged over states The left graph above shows the values learned after various numbers of episodes on a


ged over states The left graph above shows the values learned after various numbers of episodes on a single run of TD(0). The estimates after 100 episodes are about as close as they ever come to the true values—with a constant step-size parameter (↵= 0.1 in this example), the values ﬂuctuate indeﬁnitely in response to the outcomes of the most recent episodes. The right graph shows learning curves for the two methods for various values of ↵. The performance measure shown is the root mean square (RMS) error between the value function learned and the true value function, averaged over the ﬁve states, then averaged over 100 runs. In all cases the approximate value function was initialized to the intermediate value V (s) = 0.5, for all s.


es the approximate value function was initialized to the intermediate value V (s) = 0.5, for all s. The TD method was consistently better than the MC method on this task.


126 Chapter 6: Temporal-Di↵erence Learning Exercise 6.3 From the results shown in the left graph of the random walk example it appears that the ﬁrst episode results in a change in only V (A). What does this tell you about what happened on the ﬁrst episode? Why was only the estimate for this one state changed? By exactly how much was it changed? ⇤ Exercise 6.4 The speciﬁc results shown in the right graph of the random walk example are dependent on the value of the step-size parameter, ↵. Do you think the conclusions about which algorithm is better would be a↵ected if a wider range of ↵values were used? Is there a di↵erent, ﬁxed value of ↵at which either algorithm would have performed signiﬁcantly better than shown? Why or why not?


of ↵at which either algorithm would have performed signiﬁcantly better than shown? Why or why not? ⇤ ⇤Exercise 6.5 In the right graph of the random walk example, the RMS error of the TD method seems to go down and then up again, particularly at high ↵’s. What could have caused this? Do you think this always occurs, or might it be a function of how the approximate value function was initialized? ⇤ Exercise 6.6 In Example 6.2 we stated that the true values for the random walk example are 1 6, 2 6, 3 6, 4 6, and 5 6, for states A through E. Describe at least two di↵erent ways that these could have been computed. Which would you guess we actually used? Why? ⇤ 6.3 Optimality of TD(0) Suppose there is available only a ﬁnite amount of experience, say 10 episodes or 100 time steps.


0) Suppose there is available only a ﬁnite amount of experience, say 10 episodes or 100 time steps. In this case, a common approach with incremental learning methods is to present the experience repeatedly until the method converges upon an answer. Given an approximate value function, V , the increments speciﬁed by (6.1) or (6.2) are computed for every time step t at which a nonterminal state is visited, but the value function is changed only once, by the sum of all the increments. Then all the available experience is processed again with the new value function to produce a new overall increment, and so on, until the value function converges. We call this batch updating because updates are made only after processing each complete batch of training data.


batch updating because updates are made only after processing each complete batch of training data. Under batch updating, TD(0) converges deterministically to a single answer independent of the step-size parameter, ↵, as long as ↵is chosen to be suﬃciently small. The constant↵MC method also converges deterministically under the same conditions, but to a di↵erent answer. Understanding these two answers will help us understand the di↵erence between the two methods. Under normal updating the methods do not move all the way to their respective batch answers, but in some sense they take steps in these directions. Before trying to understand the two answers in general, for all possible tasks, we ﬁrst look at a few examples.


g to understand the two answers in general, for all possible tasks, we ﬁrst look at a few examples. Example 6.3: Random walk under batch updating Batch-updating versions of TD(0) and constant-↵MC were applied as follows to the random walk prediction example (Example 6.2). After each new episode, all episodes seen so far were treated as a batch. They were repeatedly presented to the algorithm, either TD(0) or constant-↵MC, with ↵suﬃciently small that the value function converged. The resulting value function was then compared with v⇡, and the average root mean square error across the ﬁve states (and across 100 independent repetitions of the whole experiment) was plotted to obtain


6.3. Optimality of TD(0) 127 the learning curves shown in Figure 6.2. Note that the batch TD method was consistently better than the batch Monte Carlo method. .0 .05 .1 .15 .2 .25 0 25 50 75 100 TD MC BATCH TRAINING Walks / Episodes RMS error, averaged over states Figure 6.2: Performance of TD(0) and constant-↵ MC under batch training on the random walk task. Under batch training, constant-↵ MC converges to values, V (s), that are sample averages of the actual returns experienced after visiting each state s. These are optimal estimates in the sense that they minimize the mean square error from the actual returns in the training set. In this sense it is surprising that the batch TD method was able to perform better according to the root mean square error measure shown in the ﬁgure to the ri


able to perform better according to the root mean square error measure shown in the ﬁgure to the right. How is it that batch TD was able to perform better than this optimal method? The answer is that the Monte Carlo method is optimal only in a limited way, and that TD is optimal in a way that is more relevant to predicting returns. Example 6.4: You are the Predictor Place yourself now in the role of the predictor of returns for an unknown Markov reward process. Suppose you observe the following eight episodes: A, 0, B, 0 B, 1 B, 1 B, 1 B, 1 B, 1 B, 1 B, 0 This means that the ﬁrst episode started in state A, transitioned to B with a reward of 0, and then terminated from B with a reward of 0. The other seven episodes were even shorter, starting from B and terminating immediately.


ward of 0. The other seven episodes were even shorter, starting from B and terminating immediately. Given this batch of data, what would you say are the optimal predictions, the best values for the estimates V (A) and V (B)? Everyone would probably agree that the optimal value for V (B) is 3 4, because six out of the eight times in state B the process terminated immediately with a return of 1, and the other two times in B the process terminated immediately with a return of 0. But what is the optimal value for the estimate V (A) given this data? Here there are A B r = 1 100% 75% 25% r = 0 r = 0 two reasonable answers. One is to observe that 100% of the times the process was in state A it traversed immediately to B (with a reward of 0); and because we have already decided that B has value 3


ersed immediately to B (with a reward of 0); and because we have already decided that B has value 3 4, therefore A must have value 3 4 as well. One way of viewing this answer is that it is based on ﬁrst modeling the Markov process, in this case as shown to the right, and then computing the correct estimates given the model, which indeed in this case gives V (A) = 3 4. This is also the answer that batch TD(0) gives.


128 Chapter 6: Temporal-Di↵erence Learning The other reasonable answer is simply to observe that we have seen A once and the return that followed it was 0; we therefore estimate V (A) as 0. This is the answer that batch Monte Carlo methods give. Notice that it is also the answer that gives minimum squared error on the training data. In fact, it gives zero error on the data. But still we expect the ﬁrst answer to be better. If the process is Markov, we expect that the ﬁrst answer will produce lower error on future data, even though the Monte Carlo answer is better on the existing data. Example 6.4 illustrates a general di↵erence between the estimates found by batch TD(0) and batch Monte Carlo methods.


rates a general di↵erence between the estimates found by batch TD(0) and batch Monte Carlo methods. Batch Monte Carlo methods always ﬁnd the estimates that minimize mean square error on the training set, whereas batch TD(0) always ﬁnds the estimates that would be exactly correct for the maximum-likelihood model of the Markov process. In general, the maximum-likelihood estimate of a parameter is the parameter value whose probability of generating the data is greatest. In this case, the maximum-likelihood estimate is the model of the Markov process formed in the obvious way from the observed episodes: the estimated transition probability from i to j is the fraction of observed transitions from i that went to j, and the associated expected reward is the average of the rewards observed on thos


i that went to j, and the associated expected reward is the average of the rewards observed on those transitions. Given this model, we can compute the estimate of the value function that would be exactly correct if the model were exactly correct. This is called the certainty-equivalence estimate because it is equivalent to assuming that the estimate of the underlying process was known with certainty rather than being approximated. In general, batch TD(0) converges to the certainty-equivalence estimate. This helps explain why TD methods converge more quickly than Monte Carlo methods. In batch form, TD(0) is faster than Monte Carlo methods because it computes the true certainty-equivalence estimate.


(0) is faster than Monte Carlo methods because it computes the true certainty-equivalence estimate. This explains the advantage of TD(0) shown in the batch results on the random walk task (Figure 6.2). The relationship to the certaintyequivalence estimate may also explain in part the speed advantage of nonbatch TD(0) (e.g., Example 6.2, page 125, right graph). Although the nonbatch methods do not achieve either the certainty-equivalence or the minimum squared error estimates, they can be understood as moving roughly in these directions. Nonbatch TD(0) may be faster than constant-↵MC because it is moving toward a better estimate, even though it is not getting all the way there. At the current time nothing more deﬁnite can be said about the relative eﬃciency of online TD and Monte Carlo meth


time nothing more deﬁnite can be said about the relative eﬃciency of online TD and Monte Carlo methods. Finally, it is worth noting that although the certainty-equivalence estimate is in some sense an optimal solution, it is almost never feasible to compute it directly. If n = |S| is the number of states, then just forming the maximum-likelihood estimate of the process may require on the order of n2 memory, and computing the corresponding value function requires on the order of n3 computational steps if done conventionally. In these terms it is indeed striking that TD methods can approximate the same solution using memory no more than order n and repeated computations over the training set. On tasks with large state spaces, TD methods may be the only feasible way of approximating the cert


On tasks with large state spaces, TD methods may be the only feasible way of approximating the certainty-equivalence solution. ⇤Exercise 6.7 Design an o↵-policy version of the TD(0) update that can be used with arbitrary target policy ⇡and covering behavior policy b, using at each step t the importance sampling ratio ⇢t:t (5.3). ⇤


6.4. Sarsa: On-policy TD Control 129 6.4 Sarsa: On-policy TD Control We turn now to the use of TD prediction methods for the control problem. As usual, we follow the pattern of generalized policy iteration (GPI), only this time using TD methods for the evaluation or prediction part. As with Monte Carlo methods, we face the need to trade o↵exploration and exploitation, and again approaches fall into two main classes: on-policy and o↵-policy. In this section we present an on-policy TD control method. The ﬁrst step is to learn an action-value function rather than a state-value function. In particular, for an on-policy method we must estimate q⇡(s, a) for the current behavior policy ⇡and for all states s and actions a.


thod we must estimate q⇡(s, a) for the current behavior policy ⇡and for all states s and actions a. This can be done using essentially the same TD method described above for learning v⇡. Recall that an episode consists of an alternating sequence of states and state–action pairs: At Rt+1 St At+1 Rt+2 St+1 At+2 Rt+3 St+2 At+3 St+3 . . . . . . In the previous section we considered transitions from state to state and learned the values of states. Now we consider transitions from state–action pair to state–action pair, and learn the values of state–action pairs. Formally these cases are identical: they are both Markov chains with a reward process. The theorems assuring the convergence of state values under TD(0) also apply to the corresponding algorithm for action values: Q(St, At) Q(St, At) +


lues under TD(0) also apply to the corresponding algorithm for action values: Q(St, At) Q(St, At) + ↵ h Rt+1 + γQ(St+1, At+1) −Q(St, At) i . (6.7) Sarsa This update is done after every transition from a nonterminal state St. If St+1 is terminal, then Q(St+1, At+1) is deﬁned as zero. This rule uses every element of the quintuple of events, (St, At, Rt+1, St+1, At+1), that make up a transition from one state–action pair to the next. This quintuple gives rise to the name Sarsa for the algorithm. The backup diagram for Sarsa is as shown to the right. It is straightforward to design an on-policy control algorithm based on the Sarsa prediction method. As in all on-policy methods, we continually estimate q⇡for the behavior policy ⇡, and at the same time change ⇡toward greediness with respect to q


timate q⇡for the behavior policy ⇡, and at the same time change ⇡toward greediness with respect to q⇡. The general form of the Sarsa control algorithm is given in the box on the next page. The convergence properties of the Sarsa algorithm depend on the nature of the policy’s dependence on Q. For example, one could use "-greedy or "-soft policies. Sarsa converges with probability 1 to an optimal policy and action-value function, under the usual conditions on the step sizes (2.7), as long as all state–action pairs are visited an inﬁnite number of times and the policy converges in the limit to the greedy policy (which can be arranged, for example, with "-greedy policies by setting " = 1/t). Exercise 6.8 Show that an action-value version of (6.6) holds for the action-value form of the TD error


rcise 6.8 Show that an action-value version of (6.6) holds for the action-value form of the TD error δt = Rt+1 + γQ(St+1, At+1) −Q(St, At), again assuming that the values don’t change from step to step. ⇤


130 Chapter 6: Temporal-Di↵erence Learning Sarsa (on-policy TD control) for estimating Q ⇡q⇤ Algorithm parameters: step size ↵2 (0, 1], small " > 0 Initialize Q(s, a), for all s 2 S+, a 2 A(s), arbitrarily except that Q(terminal, ·) = 0 Loop for each episode: Initialize S Choose A from S using policy derived from Q (e.g., "-greedy) Loop for each step of episode: Take action A, observe R, S0 Choose A0 from S0 using policy derived from Q (e.g., "-greedy) Q(S, A) Q(S, A) + ↵ ⇥ R + γQ(S0, A0) −Q(S, A) ⇤ S S0; A A0; until S is terminal Example 6.5: Windy Gridworld Shown inset below is a standard gridworld, with start and goal states, but with one di↵erence: there is a crosswind running upward through the middle of the grid.


states, but with one di↵erence: there is a crosswind running upward through the middle of the grid. The actions are the standard four—up, down, right, and left—but in the middle region the resultant next states are shifted upward by a “wind,” the strength of which varies from column to column. The strength of the wind 0 1000 2000 3000 4000 5000 6000 7000 8000 0 50 100 150 170 Time steps S G 0 0 0 0 1 1 1 1 2 2 Actions Episodes is given below each column, in number of cells shifted upward. For example, if you are one cell to the right of the goal, then the action left takes you to the cell just above the goal. This is an undiscounted episodic task, with constant rewards of −1 until the goal state is reached.


This is an undiscounted episodic task, with constant rewards of −1 until the goal state is reached. The graph to the right shows the results of applying "-greedy Sarsa to this task, with " = 0.1, ↵= 0.5, and the initial values Q(s, a) = 0 for all s, a. The increasing slope of the graph shows that the goal was reached more quickly over time. By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown inset); continued "-greedy exploration kept the average episode length at about 17 steps, two more than the minimum of 15. Note that Monte Carlo methods cannot easily be used here because termination is not guaranteed for all policies. If a policy was ever found that caused the agent to stay in the same state, then the next episode would never end.


ever found that caused the agent to stay in the same state, then the next episode would never end. Online learning methods such as Sarsa do not have this problem because they quickly learn during the episode that such policies are poor, and switch to something else.


6.5. Q-learning: O↵-policy TD Control 131 Exercise 6.9: Windy Gridworld with King’s Moves (programming) Re-solve the windy gridworld assuming eight possible actions, including the diagonal moves, rather than four. How much better can you do with the extra actions? Can you do even better by including a ninth action that causes no movement at all other than that caused by the wind? ⇤ Exercise 6.10: Stochastic Wind (programming) Re-solve the windy gridworld task with King’s moves, assuming that the e↵ect of the wind, if there is any, is stochastic, sometimes varying by 1 from the mean values given for each column. That is, a third of the time you move exactly according to these values, as in the previous exercise, but also a third of the time you move one cell above that, and another third of


e previous exercise, but also a third of the time you move one cell above that, and another third of the time you move one cell below that. For example, if you are one cell to the right of the goal and you move left, then one-third of the time you move one cell above the goal, one-third of the time you move two cells above the goal, and one-third of the time you move to the goal. ⇤ 6.5 Q-learning: O↵-policy TD Control One of the early breakthroughs in reinforcement learning was the development of an o↵-policy TD control algorithm known as Q-learning (Watkins, 1989), deﬁned by Q(St, At) Q(St, At) + ↵ h Rt+1 + γ max a Q(St+1, a) −Q(St, At) i . (6.8) In this case, the learned action-value function, Q, directly approximates q⇤, the optimal action-value function, independent of the policy being


ion, Q, directly approximates q⇤, the optimal action-value function, independent of the policy being followed. This dramatically simpliﬁes the analysis of the algorithm and enabled early convergence proofs. The policy still has an e↵ect in that it determines which state–action pairs are visited and updated. However, all that is required for correct convergence is that all pairs continue to be updated. As we observed in Chapter 5, this is a minimal requirement in the sense that any method guaranteed to ﬁnd optimal behavior in the general case must require it. Under this assumption and a variant of the usual stochastic approximation conditions on the sequence of step-size parameters, Q has been shown to converge with probability 1 to q⇤.


ons on the sequence of step-size parameters, Q has been shown to converge with probability 1 to q⇤. The Q-learning algorithm is shown below in procedural form. Q-learning (o↵-policy TD control) for estimating ⇡⇡⇡⇤ Algorithm parameters: step size ↵2 (0, 1], small " > 0 Initialize Q(s, a), for all s 2 S+, a 2 A(s), arbitrarily except that Q(terminal, ·) = 0 Loop for each episode: Initialize S Loop for each step of episode: Choose A from S using policy derived from Q (e.g., "-greedy) Take action A, observe R, S0 Q(S, A) Q(S, A) + ↵ ⇥ R + γ maxa Q(S0, a) −Q(S, A) ⇤ S S0 until S is terminal


132 Chapter 6: Temporal-Di↵erence Learning What is the backup diagram for Q-learning? The rule (6.8) updates a state–action pair, so the top node, the root of the update, must be a small, ﬁlled action node. The update is also from action nodes, maximizing over all those actions possible in the next state. Thus the bottom nodes of the backup diagram should be all these action nodes. Finally, remember that we indicate taking the maximum of these “next action” nodes with an arc across them (Figure 3.4-right). Can you guess now what the diagram is? If so, please do make a guess before turning to the answer in Figure 6.4 on page 134. Example 6.6: Cli↵Walking This gridworld example compares Sarsa and Q-learning, highlighting the di↵erence between on-policy (Sarsa) and o↵-policy (Q-learning) meth


and Q-learning, highlighting the di↵erence between on-policy (Sarsa) and o↵-policy (Q-learning) methods. S G T h e C l i f f R R = -1 Safer path Optimal path R = -100 Sarsa Q-learning Sum of rewards during episode Episodes -25 -50 -75 -100 0 100 200 300 400 500 Consider the gridworld shown to the right. This is a standard undiscounted, episodic task, with start and goal states, and the usual actions causing movement up, down, right, and left. Reward is −1 on all transitions except those into the region marked “The Cli↵.” Stepping into this region incurs a reward of −100 and sends the agent instantly back to the start. The graph to the right shows the performance of the Sarsa and Qlearning methods with "-greedy action selection, " = 0.1.


t shows the performance of the Sarsa and Qlearning methods with "-greedy action selection, " = 0.1. After an initial transient, Q-learning learns values for the optimal policy, that which travels right along the edge of the cli↵. Unfortunately, this results in its occasionally falling o↵ the cli↵because of the "-greedy action selection. Sarsa, on the other hand, takes the action selection into account and learns the longer but safer path through the upper part of the grid. Although Q-learning actually learns the values of the optimal policy, its online performance is worse than that of Sarsa, which learns the roundabout policy. Of course, if " were gradually reduced, then both methods would asymptotically converge to the optimal policy.


if " were gradually reduced, then both methods would asymptotically converge to the optimal policy. Exercise 6.11 Why is Q-learning considered an o↵-policy control method? ⇤ Exercise 6.12 Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa? Will they make exactly the same action selections and weight updates? ⇤


6.6. Expected Sarsa 133 6.6 Expected Sarsa Consider the learning algorithm that is just like Q-learning except that instead of the maximum over next state–action pairs it uses the expected value, taking into account how likely each action is under the current policy. That is, consider the algorithm with the update rule Q(St, At) Q(St, At) + ↵ h Rt+1 + γE⇡[Q(St+1, At+1) | St+1] −Q(St, At) i = Q(St, At) + ↵ h Rt+1 + γ X a ⇡(a|St+1)Q(St+1, a) −Q(St, At) i , (6.9) but that otherwise follows the schema of Q-learning. Given the next state, St+1, this algorithm moves deterministically in the same direction as Sarsa moves in expectation, and accordingly it is called Expected Sarsa. Its backup diagram is shown on the right in Figure 6.4.


nd accordingly it is called Expected Sarsa. Its backup diagram is shown on the right in Figure 6.4. Expected Sarsa is more complex computationally than Sarsa but, in return, it eliminates the variance due to the random selection of At+1. Given the same amount of experience we might expect it to perform slightly better than Sarsa, and indeed it generally does. Figure 6.3 shows summary results on the cli↵-walking task with Expected Sarsa compared to Sarsa and Q-learning. Expected Sarsa retains the signiﬁcant advantage of Sarsa over Q-learning on this problem. In addition, Expected Sarsa shows a signiﬁcant improvement 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −160 −140 −120 −100 −80 −60 −40 −20 0 n = 100, Sarsa n = 100, Q−learning n = 100, Expected Sarsa n = 1E5, Sarsa n = 1E5, Q−learning n = 1E5


= 100, Sarsa n = 100, Q−learning n = 100, Expected Sarsa n = 1E5, Sarsa n = 1E5, Q−learning n = 1E5, Expected Sarsa Expected Sarsa Sarsa Q-learning Asymptotic Performance Interim Performance Q-learning Sum of rewards per episode ↵ 1 0.1 0.2 0.4 0.6 0.8 0.3 0.5 0.7 0.9 0 -40 -80 -120 Figure 6.3: Interim and asymptotic performance of TD control methods on the cli↵-walking task as a function of ↵. All algorithms used an "-greedy policy with " = 0.1. Asymptotic performance is an average over 100,000 episodes whereas interim performance is an average over the ﬁrst 100 episodes. These data are averages of over 50,000 and 10 runs for the interim and asymptotic cases respectively. The solid circles mark the best interim performance of each method. Adapted from van Seijen et al. (2009).


134 Chapter 6: Temporal-Di↵erence Learning Q-learning Expected Sarsa Figure 6.4: The backup diagrams for Q-learning and Expected Sarsa. over Sarsa over a wide range of values for the step-size parameter ↵. In cli↵walking the state transitions are all deterministic and all randomness comes from the policy. In such cases, Expected Sarsa can safely set ↵= 1 without su↵ering any degradation of asymptotic performance, whereas Sarsa can only perform well in the long run at a small value of ↵, at which short-term performance is poor. In this and other examples there is a consistent empirical advantage of Expected Sarsa over Sarsa. In these cli↵walking results Expected Sarsa was used on-policy, but in general it might use a policy di↵erent from the target policy ⇡to generate behavior, in which cas


in general it might use a policy di↵erent from the target policy ⇡to generate behavior, in which case it becomes an o↵-policy algorithm. For example, suppose ⇡is the greedy policy while behavior is more exploratory; then Expected Sarsa is exactly Q-learning. In this sense Expected Sarsa subsumes and generalizes Q-learning while reliably improving over Sarsa. Except for the small additional computational cost, Expected Sarsa may completely dominate both of the other more-well-known TD control algorithms. 6.7 Maximization Bias and Double Learning All the control algorithms that we have discussed so far involve maximization in the construction of their target policies. For example, in Q-learning the target policy is the greedy policy given the current action values, which is deﬁned with a max


g the target policy is the greedy policy given the current action values, which is deﬁned with a max, and in Sarsa the policy is often "-greedy, which also involves a maximization operation. In these algorithms, a maximum over estimated values is used implicitly as an estimate of the maximum value, which can lead to a signiﬁcant positive bias. To see why, consider a single state s where there are many actions a whose true values, q(s, a), are all zero but whose estimated values, Q(s, a), are uncertain and thus distributed some above and some below zero. The maximum of the true values is zero, but the maximum of the estimates is positive, a positive bias. We call this maximization bias. Example 6.7: Maximization Bias Example The small MDP shown inset in Figure 6.5 provides a simple example


le 6.7: Maximization Bias Example The small MDP shown inset in Figure 6.5 provides a simple example of how maximization bias can harm the performance of TD control algorithms. The MDP has two non-terminal states A and B. Episodes always start in A with a choice between two actions, left and right. The right action transitions immediately to the terminal state with a reward and return of zero. The left action transitions to B, also with a reward of zero, from which there are many possible actions all of which cause immediate termination with a reward drawn from a normal distribution with mean −0.1 and variance 1.0. Thus, the expected return for any trajectory starting with left is −0.1, and thus taking left in state A is always a


6.7. Maximization Bias and Double Learning 135 B A right left 0 . . . N(−0.1, 1) 0 Q-learning Double Q-learning Episodes 100 1 200 300 % left actions from A 100% 75% 50% 25% 5% 0 optimal Figure 6.5: Comparison of Q-learning and Double Q-learning on a simple episodic MDP (shown inset). Q-learning initially learns to take the left action much more often than the right action, and always takes it signiﬁcantly more often than the 5% minimum probability enforced by "-greedy action selection with " = 0.1. In contrast, Double Q-learning is essentially una↵ected by maximization bias. These data are averaged over 10,000 runs. The initial action-value estimates were zero. Any ties in "-greedy action selection were broken randomly. mistake.


ion-value estimates were zero. Any ties in "-greedy action selection were broken randomly. mistake. Nevertheless, our control methods may favor left because of maximization bias making B appear to have a positive value. Figure 6.5 shows that Q-learning with "-greedy action selection initially learns to strongly favor the left action on this example. Even at asymptote, Q-learning takes the left action about 5% more often than is optimal at our parameter settings (" = 0.1, ↵= 0.1, and γ = 1). Are there algorithms that avoid maximization bias? To start, consider a bandit case in which we have noisy estimates of the value of each of many actions, obtained as sample averages of the rewards received on all the plays with each action.


any actions, obtained as sample averages of the rewards received on all the plays with each action. As we discussed above, there will be a positive maximization bias if we use the maximum of the estimates as an estimate of the maximum of the true values. One way to view the problem is that it is due to using the same samples (plays) both to determine the maximizing action and to estimate its value. Suppose we divided the plays in two sets and used them to learn two independent estimates, call them Q1(a) and Q2(a), each an estimate of the true value q(a), for all a 2 A. We could then use one estimate, say Q1, to determine the maximizing action A⇤= argmaxa Q1(a), and the other, Q2, to provide the estimate of its value, Q2(A⇤) = Q2(argmaxa Q1(a)).


argmaxa Q1(a), and the other, Q2, to provide the estimate of its value, Q2(A⇤) = Q2(argmaxa Q1(a)). This estimate will then be unbiased in the sense that E[Q2(A⇤)] = q(A⇤). We can also repeat the process with the role of the two estimates reversed to yield a second unbiased estimate Q1(argmaxa Q2(a)). This is the idea of double learning. Note that although we learn two estimates, only one estimate is updated on each play; double learning doubles the memory requirements, but does not increase the amount of computation per step. The idea of double learning extends naturally to algorithms for full MDPs. For example, the double learning algorithm analogous to Q-learning, called Double Q-learning, divides the time steps in two, perhaps by ﬂipping a coin on each step.


g, called Double Q-learning, divides the time steps in two, perhaps by ﬂipping a coin on each step. If the coin comes up heads, the update is


136 Chapter 6: Temporal-Di↵erence Learning Q1(St, At) Q1(St, At)+↵ h Rt+1+γQ2 # St+1, argmax a Q1(St+1, a) $ −Q1(St, At) i . (6.10) If the coin comes up tails, then the same update is done with Q1 and Q2 switched, so that Q2 is updated. The two approximate value functions are treated completely symmetrically. The behavior policy can use both action-value estimates. For example, an "-greedy policy for Double Q-learning could be based on the average (or sum) of the two action-value estimates. A complete algorithm for Double Q-learning is given in the box below. This is the algorithm used to produce the results in Figure 6.5. In that example, double learning seems to eliminate the harm caused by maximization bias. Of course there are also double versions of Sarsa and Expected Sarsa.


caused by maximization bias. Of course there are also double versions of Sarsa and Expected Sarsa. Double Q-learning, for estimating Q1 ⇡Q2 ⇡q⇤ Algorithm parameters: step size ↵2 (0, 1], small " > 0 Initialize Q1(s, a) and Q2(s, a), for all s 2 S+, a 2 A(s), such that Q(terminal, ·) = 0 Loop for each episode: Initialize S Loop for each step of episode: Choose A from S using the policy "-greedy in Q1 + Q2 Take action A, observe R, S0 With 0.5 probabilility: Q1(S, A) Q1(S, A) + ↵ ⇣ R + γQ2 # S0, argmaxa Q1(S0, a) $ −Q1(S, A) ⌘ else: Q2(S, A) Q2(S, A) + ↵ ⇣ R + γQ1 # S0, argmaxa Q2(S0, a) $ −Q2(S, A) ⌘ S S0 until S is terminal ⇤Exercise 6.13 What are the update equations for Double Expected Sarsa with an "-greedy target policy?


ercise 6.13 What are the update equations for Double Expected Sarsa with an "-greedy target policy? ⇤ 6.8 Games, Afterstates, and Other Special Cases In this book we try to present a uniform approach to a wide class of tasks, but of course there are always exceptional tasks that are better treated in a specialized way. For example, our general approach involves learning an action-value function, but in Chapter 1 we presented a TD method for learning to play tic-tac-toe that learned something much more like a state-value function. If we look closely at that example, it becomes apparent that the function learned there is neither an action-value function nor a state-value function in the usual sense.


on learned there is neither an action-value function nor a state-value function in the usual sense. A conventional state-value function evaluates states in which the agent has the option of selecting an action, but the state-value function used in


6.8. Games, Afterstates, and Other Special Cases 137 tic-tac-toe evaluates board positions after the agent has made its move. Let us call these afterstates, and value functions over these, afterstate value functions. Afterstates are useful when we have knowledge of an initial part of the environment’s dynamics but not necessarily of the full dynamics. For example, in games we typically know the immediate e↵ects of our moves. We know for each possible chess move what the resulting position will be, but not how our opponent will reply. Afterstate value functions are a natural way to take advantage of this kind of knowledge and thereby produce a more eﬃcient learning method. The reason it is more eﬃcient to design algorithms in terms of afterstates is apparent from the tic-tac-toe example.


more eﬃcient to design algorithms in terms of afterstates is apparent from the tic-tac-toe example. A conventional action-value function would map from positions and moves to an estimate of the value. But many position—move pairs produce the same resulting position, as in the example below: In such cases the X O X X O + X O + X X position–move pairs are di↵erent but produce the same “afterposition,” and thus must have the same value. A conventional action-value function would have to separately assess both pairs, whereas an afterstate value function would immediately assess both equally. Any learning about the position–move pair on the left would immediately transfer to the pair on the right. Afterstates arise in many tasks, not just games.


uld immediately transfer to the pair on the right. Afterstates arise in many tasks, not just games. For example, in queuing tasks there are actions such as assigning customers to servers, rejecting customers, or discarding information. In such cases the actions are in fact deﬁned in terms of their immediate e↵ects, which are completely known. It is impossible to describe all the possible kinds of specialized problems and corresponding specialized learning algorithms. However, the principles developed in this book should apply widely. For example, afterstate methods are still aptly described in terms of generalized policy iteration, with a policy and (afterstate) value function interacting in essentially the same way.


y iteration, with a policy and (afterstate) value function interacting in essentially the same way. In many cases one will still face the choice between on-policy and o↵-policy methods for managing the need for persistent exploration. Exercise 6.14 Describe how the task of Jack’s Car Rental (Example 4.2) could be reformulated in terms of afterstates. Why, in terms of this speciﬁc task, would such a reformulation be likely to speed convergence? ⇤


138 Chapter 6: Temporal-Di↵erence Learning 6.9 Summary In this chapter we introduced a new kind of learning method, temporal-di↵erence (TD) learning, and showed how it can be applied to the reinforcement learning problem. As usual, we divided the overall problem into a prediction problem and a control problem. TD methods are alternatives to Monte Carlo methods for solving the prediction problem. In both cases, the extension to the control problem is via the idea of generalized policy iteration (GPI) that we abstracted from dynamic programming. This is the idea that approximate policy and value functions should interact in such a way that they both move toward their optimal values. One of the two processes making up GPI drives the value function to accurately predict returns for the current


two processes making up GPI drives the value function to accurately predict returns for the current policy; this is the prediction problem. The other process drives the policy to improve locally (e.g., to be "-greedy) with respect to the current value function. When the ﬁrst process is based on experience, a complication arises concerning maintaining suﬃcient exploration. We can classify TD control methods according to whether they deal with this complication by using an on-policy or o↵-policy approach. Sarsa is an on-policy method, and Q-learning is an o↵-policy method. Expected Sarsa is also an o↵-policy method as we present it here. There is a third way in which TD methods can be extended to control which we did not include in this chapter, called actor–critic methods.


s can be extended to control which we did not include in this chapter, called actor–critic methods. These methods are covered in full in Chapter 13. The methods presented in this chapter are today the most widely used reinforcement learning methods. This is probably due to their great simplicity: they can be applied online, with a minimal amount of computation, to experience generated from interaction with an environment; they can be expressed nearly completely by single equations that can be implemented with small computer programs. In the next few chapters we extend these algorithms, making them slightly more complicated and signiﬁcantly more powerful. All the new algorithms will retain the essence of those introduced here: they will be able to process experience online, with relatively


e essence of those introduced here: they will be able to process experience online, with relatively little computation, and they will be driven by TD errors. The special cases of TD methods introduced in the present chapter should rightly be called one-step, tabular, model-free TD methods. In the next two chapters we extend them to n-step forms (a link to Monte Carlo methods) and forms that include a model of the environment (a link to planning and dynamic programming). Then, in the second part of the book we extend them to various forms of function approximation rather than tables (a link to deep learning and artiﬁcial neural networks). Finally, in this chapter we have discussed TD methods entirely within the context of reinforcement learning problems, but TD methods are actually more gen


entirely within the context of reinforcement learning problems, but TD methods are actually more general than this. They are general methods for learning to make long-term predictions about dynamical systems. For example, TD methods may be relevant to predicting ﬁnancial data, life spans, election outcomes, weather patterns, animal behavior, demands on power stations, or customer purchases. It was only when TD methods were analyzed as pure prediction methods, independent of their use in reinforcement learning, that their theoretical properties ﬁrst came to be well understood. Even so, these other potential applications of TD learning methods have not yet been extensively explored.


6.9. Summary 139 Bibliographical and Historical Remarks As we outlined in Chapter 1, the idea of TD learning has its early roots in animal learning psychology and artiﬁcial intelligence, most notably the work of Samuel (1959) and Klopf (1972). Samuel’s work is described as a case study in Section 16.2. Also related to TD learning are Holland’s (1975, 1976) early ideas about consistency among value predictions. These inﬂuenced one of the authors (Barto), who was a graduate student from 1970 to 1975 at the University of Michigan, where Holland was teaching. Holland’s ideas led to a number of TD-related systems, including the work of Booker (1982) and the bucket brigade of Holland (1986), which is related to Sarsa as discussed below.


oker (1982) and the bucket brigade of Holland (1986), which is related to Sarsa as discussed below. 6.1–2 Most of the speciﬁc material from these sections is from Sutton (1988), including the TD(0) algorithm, the random walk example, and the term “temporaldi↵erence learning.” The characterization of the relationship to dynamic programming and Monte Carlo methods was inﬂuenced by Watkins (1989), Werbos (1987), and others. The use of backup diagrams was new to the ﬁrst edition of this book. Tabular TD(0) was proved to converge in the mean by Sutton (1988) and with probability 1 by Dayan (1992), based on the work of Watkins and Dayan (1992). These results were extended and strengthened by Jaakkola, Jordan, and Singh (1994) and Tsitsiklis (1994) by using extensions of the powerful existing the


ola, Jordan, and Singh (1994) and Tsitsiklis (1994) by using extensions of the powerful existing theory of stochastic approximation. Other extensions and generalizations are covered in later chapters. 6.3 The optimality of the TD algorithm under batch training was established by Sutton (1988). Illuminating this result is Barnard’s (1993) derivation of the TD algorithm as a combination of one step of an incremental method for learning a model of the Markov chain and one step of a method for computing predictions from the model. The term certainty equivalence is from the adaptive control literature (e.g., Goodwin and Sin, 1984). 6.4 The Sarsa algorithm was introduced by Rummery and Niranjan (1994).


.g., Goodwin and Sin, 1984). 6.4 The Sarsa algorithm was introduced by Rummery and Niranjan (1994). They explored it in conjunction with artiﬁcial neural networks and called it “Modiﬁed Connectionist Q-learning”. The name “Sarsa” was introduced by Sutton (1996). The convergence of one-step tabular Sarsa (the form treated in this chapter) has been proved by Singh, Jaakkola, Littman, and Szepesv´ari (2000). The “windy gridworld” example was suggested by Tom Kalt. Holland’s (1986) bucket brigade idea evolved into an algorithm closely related to Sarsa. The original idea of the bucket brigade involved chains of rules triggering each other; it focused on passing credit back from the current rule to the rules that triggered it.