rameter of the function approximator has three components, one giving the value of state B, one giving the value of state C, and one giving the value of both states A1 and A2. Other than the selection of the initial state, the system is deterministic. If it starts in A1, then it transitions to B with a reward of 0 and then on to termination with a reward of 1. If it starts in A2, then it transitions to C, and then to termination, with both rewards zero. To a learning algorithm, seeing only the features, the system looks identical to the A-split example. The system seems to always start in A, followed by either B or C with equal probability, and then terminating with a 1 or a 0 depending deterministically on the previous state.


robability, and then terminating with a 1 or a 0 depending deterministically on the previous state. As in the A-split example, the true values of B and C are 1 and 0, and the best shared value of A1 and A2 is 1 2, by symmetry. Because this problem appears externally identical to the A-split example, we already know what values will be found by the algorithms. Semi-gradient TD converges to the ideal values just mentioned, while the naive residual-gradient algorithm converges to values of 3 4 and 1 4 for B and C respectively. All state transitions are deterministic, so the non-naive residual-gradient algorithm will also converge to these values (it is the same algorithm in this case). It follows then that this ‘naive’ solution must also be the one that minimizes the BE, and so it is.


t follows then that this ‘naive’ solution must also be the one that minimizes the BE, and so it is. On a deterministic problem, the Bellman errors and TD errors are all the same, so the BE is always the same as the TDE. Optimizing the BE on this example gives rise to the same failure mode as with the naive residual-gradient algorithm on the A-split example.


274 Chapter 11: O↵-policy Methods with Approximation equation is possible. But if we examine examples with genuine function approximation, then the residual-gradient algorithm, and indeed the BE objective, seem to ﬁnd the wrong value functions. One of the most telling such examples is the variation on the A-split example known as the A-presplit example, shown on the preceding page, in which the residual-gradient algorithm ﬁnds the same poor solution as its naive version. This example shows intuitively that minimizing the BE (which the residual-gradient algorithm surely does) may not be a desirable goal. The third way in which the convergence of the residual-gradient algorithm is not satisfactory is explained in the next section.


onvergence of the residual-gradient algorithm is not satisfactory is explained in the next section. Like the second way, the third way is also a problem with the BE objective itself rather than with any particular algorithm for achieving it. 11.6 The Bellman Error is Not Learnable The concept of learnability that we introduce in this section is di↵erent from that commonly used in machine learning. There, a hypothesis is said to be “learnable” if it is eﬃciently learnable, meaning that it can be learned within a polynomial rather than an exponential number of examples. Here we use the term in a more basic way, to mean learnable at all, with any amount of experience. It turns out many quantities of apparent interest in reinforcement learning cannot be learned even from an inﬁnite amount of e


es of apparent interest in reinforcement learning cannot be learned even from an inﬁnite amount of experiential data. These quantities are well deﬁned and can be computed given knowledge of the internal structure of the environment, but cannot be computed or estimated from the observed sequence of feature vectors, actions, and rewards.2 We say that they are not learnable. It will turn out that the Bellman error objective (BE) introduced in the last two sections is not learnable in this sense. That the Bellman error objective cannot be learned from the observable data is probably the strongest reason not to seek it. To make the concept of learnability clear, let’s start with some simple examples.


n not to seek it. To make the concept of learnability clear, let’s start with some simple examples. Consider the two Markov reward processes3 (MRPs) diagrammed below: 0 2 0 2 2 0 w w w Where two edges leave a state, both transitions are assumed to occur with equal probability, and the numbers indicate the reward received. All the states appear the same; they all produce the same single-component feature vector x = 1 and have approximated value w. Thus, the only varying part of the data trajectory is the reward sequence. The left MRP stays in the same state and emits an endless stream of 0s and 2s at random, each with 0.5 probability. The right MRP, on every step, either stays in its current state or 2They would of course be estimated if the state sequence were observed rather than only the


state or 2They would of course be estimated if the state sequence were observed rather than only the corresponding feature vectors. 3All MRPs can be considered MDPs with a single action in all states; what we conclude about MRPs here applies as well to MDPs.


11.6. The Bellman Error is Not Learnable 275 switches to the other, with equal probability. The reward is deterministic in this MRP, always a 0 from one state and always a 2 from the other, but because the each state is equally likely on each step, the observable data is again an endless stream of 0s and 2s at random, identical to that produced by the left MRP. (We can assume the right MRP starts in one of two states at random with equal probability.) Thus, even given an inﬁnite amount of data, it would not be possible to tell which of these two MRPs was generating it. In particular, we could not tell if the MRP has one state or two, is stochastic or deterministic. These things are not learnable. This pair of MRPs also illustrates that the VE objective (9.1) is not learnable.


are not learnable. This pair of MRPs also illustrates that the VE objective (9.1) is not learnable. If γ = 0, then the true values of the three states (in both MRPs), left to right, are 1, 0, and 2. Suppose w = 1. Then the VE is 0 for the left MRP and 1 for the right MRP. Because the VE is di↵erent in the two problems, yet the data generated has the same distribution, the VE cannot be learned. The VE is not a unique function of the data distribution. And if it cannot be learned, then how could the VE possibly be useful as an objective for learning? If an objective cannot be learned, it does indeed draw its utility into question. In the case of the VE, however, there is a way out. Note that the same solution, w = 1, is optimal for both MRPs above (assuming µ is the same for the two indistin


he same solution, w = 1, is optimal for both MRPs above (assuming µ is the same for the two indistinguishable states in the right MRP). Is this a coincidence, or could it be generally true that all MDPs with the same data distribution also have the same optimal parameter vector? If this is true—and we will show next that it is—then the VE remains a usable objective. The VE is not learnable, but the parameter that optimizes it is! To understand this, it is useful to bring in another natural objective function, this time one that is clearly learnable. One error that is always observable is that between the value estimate at each time and the return from that time. The mean square return error, denoted RE, is the expectation, under µ, of the square of this error.


The mean square return error, denoted RE, is the expectation, under µ, of the square of this error. In the on-policy case the RE can be written RE(w) = E h# Gt −ˆv(St,w) $2i = VE(w) + E h# Gt −v⇡(St) $2i . (11.24) Thus, the two objectives are the same except for a variance term that does not depend on the parameter vector. The two objectives must therefore have the same optimal parameter value w⇤. The overall relationships are summarized in the left side of Figure 11.4. ⇤Exercise 11.4 Prove (11.24). Hint: Write the RE as an expectation over possible states s of the expectation of the squared error given that St = s. Then add and subtract the true value of state s from the error (before squaring), grouping the subtracted true value with the return and the added true value with the estimated


ing), grouping the subtracted true value with the return and the added true value with the estimated value. Then, if you expand the square, the most complex term will end up being zero, leaving you with (11.24). ⇤ Now let us return to the BE. The BE is like the VE in that it can be computed from knowledge of the MDP but is not learnable from data. But it is not like the VE in that its minimum solution is not learnable. The box on the next page presents a counterexample— two MRPs that generate the same data distribution but whose minimizing parameter vector is di↵erent, proving that the optimal parameter vector is not a function of the


276 Chapter 11: O↵-policy Methods with Approximation Example 11.4: Counterexample to the learnability of the Bellman error To show the full range of possibilities we need a slightly more complex pair of Markov reward processes (MRPs) than those considered earlier. Consider the following two MRPs: B A 1 0 -1 B A 0 -1 B 0 1 -1 Where two edges leave a state, both transitions are assumed to occur with equal probability, and the numbers indicate the reward received. The MRP on the left has two states that are represented distinctly. The MRP on the right has three states, two of which, B and B0, appear the same and must be given the same approximate value. Speciﬁcally, w has two components and the value of state A is given by the ﬁrst component and the value of B and B0 is given by the second.


e value of state A is given by the ﬁrst component and the value of B and B0 is given by the second. The second MRP has been designed so that equal time is spent in all three states, so we can take µ(s) = 1 3, for all s. Note that the observable data distribution is identical for the two MRPs. In both cases the agent will see single occurrences of A followed by a 0, then some number of apparent Bs, each followed by a −1 except the last, which is followed by a 1, then we start all over again with a single A and a 0, etc. All the statistical details are the same as well; in both MRPs, the probability of a string of k Bs is 2−k. Now suppose w = 0. In the ﬁrst MRP, this is an exact solution, and the BE is zero.


of k Bs is 2−k. Now suppose w = 0. In the ﬁrst MRP, this is an exact solution, and the BE is zero. In the second MRP, this solution produces a squared error in both B and B0 of 1, such that BE = µ(B)1 + µ(B0)1 = 2 3. These two MRPs, which generate the same data distribution, have di↵erent BEs; the BE is not learnable. Moreover (and unlike the earlier example for the VE) the minimizing value of w is di↵erent for the two MRPs. For the ﬁrst MRP, w = 0 minimizes the BE for any γ. For the second MRP, the minimizing w is a complicated function of γ, but in the limit, as γ ! 1, it is (−1 2, 0)>. Thus the solution that minimizes BE cannot be estimated from data alone; knowledge of the MRP beyond what is revealed in the data is required.


be estimated from data alone; knowledge of the MRP beyond what is revealed in the data is required. In this sense, it is impossible in principle to pursue the BE as an objective for learning. It may be surprising that in the second MRP the BE-minimizing value of A is so far from zero. Recall that A has a dedicated weight and thus its value is unconstrained by function approximation. A is followed by a reward of 0 and transition to a state with a value of nearly 0, which suggests vw(A) should be 0; why is its optimal value substantially negative rather than 0? The answer is that making vw(A) negative reduces the error upon arriving in A from B. The reward on this deterministic transition is 1, which implies that B should have a value 1 more than A.


ward on this deterministic transition is 1, which implies that B should have a value 1 more than A. Because B’s value is approximately zero, A’s value is driven toward −1. The BE-minimizing value of ⇡−1 2 for A is a compromise between reducing the errors on leaving and on entering A.


11.6. The Bellman Error is Not Learnable 277 MDP1 MDP2 MSBE1 MSBE2 ✓∗ 1 ✓∗ 2 MSPBE ✓∗ 3 MSTDE ✓∗ 4 Data distribution MDP1 MDP2 MSVE1 MSVE2 MSRE Data distribution ✓∗ w⇤ w⇤ 1 w⇤ 2 w⇤ 3 w⇤ 4 VE1 VE2 RE BE2 BE1 PBE TDE Monte Carlo objectives Bootstrapping objectives Figure 11.4: Causal relationships among the data distribution, MDPs, and various objectives. Left, Monte Carlo objectives: Two di↵erent MDPs can produce the same data distribution yet also produce di↵erent VEs, proving that the VE objective cannot be determined from data and is not learnable. However, all such VEs must have the same optimal parameter vector, w⇤! Moreover, this same w⇤can be determined from another objective, the RE, which is uniquely determined from the data distribution.


determined from another objective, the RE, which is uniquely determined from the data distribution. Thus w⇤and the RE are learnable even though the VEs are not. Right, Bootstrapping objectives: Two di↵erent MDPs can produce the same data distribution yet also produce di↵erent BEs and have di↵erent minimizing parameter vectors; these are not learnable from the data distribution. The PBE and TDE objectives and their (di↵erent) minima can be directly determined from data and thus are learnable. data and thus cannot be learned from it. The other bootstrapping objectives that we have considered, the PBE and TDE, can be determined from data (are learnable) and determine optimal solutions that are in general di↵erent from each other and the BE minimums.


) and determine optimal solutions that are in general di↵erent from each other and the BE minimums. The general case is summarized in the right side of Figure 11.4. Thus, the BE is not learnable; it cannot be estimated from feature vectors and other observable data. This limits the BE to model-based settings. There can be no algorithm that minimizes the BE without access to the underlying MDP states beyond the feature vectors. The residual-gradient algorithm is only able to minimize BE because it is allowed to double sample from the same state—not a state that has the same feature vector, but one that is guaranteed to be the same underlying state. We can see now that there is no way around this. Minimizing the BE requires some such access to the nominal, underlying MDP.


is no way around this. Minimizing the BE requires some such access to the nominal, underlying MDP. This is an important limitation of the BE beyond that identiﬁed in the A-presplit example on page 273. All this directs more attention toward the PBE.


278 Chapter 11: O↵-policy Methods with Approximation 11.7 Gradient-TD Methods We now consider SGD methods for minimizing the PBE. As true SGD methods, these Gradient-TD methods have robust convergence properties even under o↵-policy training and nonlinear function approximation. Remember that in the linear case there is always an exact solution, the TD ﬁxed point wTD, at which the PBE is zero. This solution could be found by least-squares methods (Section 9.8), but only by methods of quadratic O(d2) complexity in the number of parameters. We seek instead an SGD method, which should be O(d) and have robust convergence properties. Gradient-TD methods come close to achieving these goals, at the cost of a rough doubling of computational complexity.


s come close to achieving these goals, at the cost of a rough doubling of computational complexity. To derive an SGD method for the PBE (assuming linear function approximation) we begin by expanding and rewriting the objective (11.22) in matrix terms: PBE(w) = &&⇧¯δw &&2 µ = (⇧¯δw)>D⇧¯δw (from (11.14)) = ¯δ> w⇧>D⇧¯δw = ¯δ> wDX # X>DX $−1X>D¯δw (11.25) (using (11.13) and the identity ⇧>D⇧= DX # X>DX $−1 X>D) = # X>D¯δw $># X>DX $−1# X>D¯δw $ . (11.26) The gradient with respect to w is rPBE(w) = 2r ⇥ X>D¯δw ⇤> # X>DX $−1# X>D¯δw $ . To turn this into an SGD method, we have to sample something on every time step that has this quantity as its expected value. Let us take µ to be the distribution of states visited under the behavior policy.


s expected value. Let us take µ to be the distribution of states visited under the behavior policy. All three of the factors above can then be written in terms of expectations under this distribution. For example, the last factor can be written X>D¯δw = X s µ(s)x(s)¯δw(s) = E[⇢tδtxt] , which is just the expectation of the semi-gradient TD(0) update (11.2). The ﬁrst factor is the transpose of the gradient of this update: rE[⇢tδtxt]> = E ⇥ ⇢trδ> t x> t ⇤ = E ⇥ ⇢tr(Rt+1 + γw>xt+1 −w>xt)>x> t ⇤ (using episodic δt) = E ⇥ ⇢t(γxt+1 −xt)x> t ⇤ . Finally, the middle factor is the inverse of the expected outer-product matrix of the feature vectors: X>DX = X s µ(s)x(s)x(s)> = E ⇥ xtx> t ⇤ .


11.7. Gradient-TD Methods 279 Substituting these expectations for the three factors in our expression for the gradient of the PBE, we get rPBE(w) = 2E ⇥ ⇢t(γxt+1 −xt)x> t ⇤ E ⇥ xtx> t ⇤−1 E[⇢tδtxt] . (11.27) It might not be obvious that we have made any progress by writing the gradient in this form. It is a product of three expressions and the ﬁrst and last are not independent. They both depend on the next feature vector xt+1; we cannot simply sample both of these expectations and then multiply the samples. This would give us a biased estimate of the gradient just as in the residual-gradient algorithm. Another idea would be to estimate the three expectations separately and then combine them to produce an unbiased estimate of the gradient.


hree expectations separately and then combine them to produce an unbiased estimate of the gradient. This would work, but would require a lot of computational resources, particularly to store the ﬁrst two expectations, which are d ⇥d matrices, and to compute the inverse of the second. This idea can be improved. If two of the three expectations are estimated and stored, then the third could be sampled and used in conjunction with the two stored quantities. For example, you could store estimates of the second two quantities (using the increment inverse-updating techniques in Section 9.8) and then sample the ﬁrst expression. Unfortunately, the overall algorithm would still be of quadratic complexity (of order O(d2)).


sion. Unfortunately, the overall algorithm would still be of quadratic complexity (of order O(d2)). The idea of storing some estimates separately and then combining them with samples is a good one and is also used in Gradient-TD methods. Gradient-TD methods estimate and store the product of the second two factors in (11.27). These factors are a d ⇥d matrix and a d-vector, so their product is just a d-vector, like w itself. We denote this second learned vector as v: v ⇡E ⇥ xtx> t ⇤−1 E[⇢tδtxt] . (11.28) This form is familiar to students of linear supervised learning. It is the solution to a linear least-squares problem that tries to approximate ⇢tδt from the features. The standard SGD method for incrementally ﬁnding the vector v that minimizes the expected squared error # v>xt −⇢tδt $2 is k


for incrementally ﬁnding the vector v that minimizes the expected squared error # v>xt −⇢tδt $2 is known as the Least Mean Square (LMS) rule (here augmented with an importance sampling ratio): vt+1 .= vt + β⇢t # δt −v> t xt $ xt, where β > 0 is another step-size parameter. We can use this method to e↵ectively achieve (11.28) with O(d) storage and per-step computation. Given a stored estimate vt approximating (11.28), we can update our main parameter vector wt using SGD methods based on (11.27). The simplest such rule is wt+1 = wt −1 2↵rPBE(wt) (the general SGD rule) = wt −1 2↵2E ⇥ ⇢t(γxt+1 −xt)x> t ⇤ E ⇥ xtx> t ⇤−1 E[⇢tδtxt] (from (11.27)) = wt + ↵E ⇥ ⇢t(xt −γxt+1)x> t ⇤ E ⇥ xtx> t ⇤−1 E[⇢tδtxt] (11.29) ⇡wt + ↵E ⇥ ⇢t(xt −γxt+1)x> t ⇤ vt (based on (11.28)) ⇡wt + ↵⇢t (xt −γxt+1) x> t vt.


⇢tδtxt] (11.29) ⇡wt + ↵E ⇥ ⇢t(xt −γxt+1)x> t ⇤ vt (based on (11.28)) ⇡wt + ↵⇢t (xt −γxt+1) x> t vt. (sampling)


280 Chapter 11: O↵-policy Methods with Approximation This algorithm is called GTD2. Note that if the ﬁnal inner product (x> t vt) is done ﬁrst, then the entire algorithm is of O(d) complexity. A slightly better algorithm can be derived by doing a few more analytic steps before substituting in vt. Continuing from (11.29): wt+1 = wt + ↵E ⇥ ⇢t(xt −γxt+1)x> t ⇤ E ⇥ xtx> t ⇤−1 E[⇢tδtxt] = wt + ↵ # E ⇥ ⇢txtx> t ⇤ −γE ⇥ ⇢txt+1x> t ⇤$ E ⇥ xtx> t ⇤−1 E[⇢tδtxt] = wt + ↵ # E ⇥ xtx> t ⇤ −γE ⇥ ⇢txt+1x> t ⇤$ E ⇥ xtx> t ⇤−1 E[⇢tδtxt] = wt + ↵ ⇣ E[⇢tδtxt] −γE ⇥ ⇢txt+1x> t ⇤ E ⇥ xtx> t ⇤−1 E[⇢tδtxt] ⌘ ⇡wt + ↵ # E[⇢tδtxt] −γE ⇥ ⇢txt+1x> t ⇤ vt $ (based on (11.28)) ⇡wt + ↵⇢t # δtxt −γxt+1x> t vt $ , (sampling) which again is O(d) if the ﬁnal product (x> t vt) is done ﬁrst.


# δtxt −γxt+1x> t vt $ , (sampling) which again is O(d) if the ﬁnal product (x> t vt) is done ﬁrst. This algorithm is known as either TD(0) with gradient correction (TDC) or, alternatively, as GTD(0). Figure 11.5 shows a sample and the expected behavior of TDC on Baird’s counterexample. As intended, the PBE falls to zero, but note that the individual components of the parameter vector do not approach zero. In fact, these values are still far from w1– w6 w8 w8 p VE p VE p PBE p PBE 10 5 2 0 -2.34 0 1000 0 1000 Steps w7 w1– w6 TDC Expected TDC Sweeps w7 Figure 11.5: The behavior of the TDC algorithm on Baird’s counterexample. On the left is shown a typical single run, and on the right is shown the expected behavior of this algorithm if the updates are done synchronously (analogous to (11.9),


the expected behavior of this algorithm if the updates are done synchronously (analogous to (11.9), except for the two TDC parameter vectors). The step sizes were ↵= 0.005 and β = 0.05.


11.8. Emphatic-TD Methods 281 an optimal solution, ˆv(s) = 0, for all s, for which w would have to be proportional to (1, 1, 1, 1, 1, 1, 4, −2)>. After 1000 iterations we are still far from an optimal solution, as we can see from the VE, which remains almost 2. The system is actually converging to an optimal solution, but progress is extremely slow because the PBE is already so close to zero. GTD2 and TDC both involve two learning processes, a primary one for w and a secondary one for v. The logic of the primary learning process relies on the secondary learning process having ﬁnished, at least approximately, whereas the secondary learning process proceeds without being inﬂuenced by the ﬁrst. We call this sort of asymmetrical dependence a cascade.


oceeds without being inﬂuenced by the ﬁrst. We call this sort of asymmetrical dependence a cascade. In cascades we often assume that the secondary learning process is proceeding faster and thus is always at its asymptotic value, ready and accurate to assist the primary learning process. The convergence proofs for these methods often make this assumption explicitly. These are called two-time-scale proofs. The fast time scale is that of the secondary learning process, and the slower time scale is that of the primary learning process. If ↵is the step size of the primary learning process, and β is the step size of the secondary learning process, then these convergence proofs will typically require that in the limit β ! 0 and ↵ β ! 0.


process, then these convergence proofs will typically require that in the limit β ! 0 and ↵ β ! 0. Gradient-TD methods are currently the most well understood and widely used stable o↵-policy methods. There are extensions to action values and control (GQ, Maei et al., 2010), to eligibility traces (GTD(λ) and GQ(λ), Maei, 2011; Maei and Sutton, 2010), and to nonlinear function approximation (Maei et al., 2009). There have also been proposed hybrid algorithms midway between semi-gradient TD and gradient TD (Hackman, 2012; White and White, 2016). Hybrid-TD algorithms behave like Gradient-TD algorithms in states where the target and behavior policies are very di↵erent, and behave like semigradient algorithms in states where the target and behavior policies are the same.


behave like semigradient algorithms in states where the target and behavior policies are the same. Finally, the Gradient-TD idea has been combined with the ideas of proximal methods and control variates to produce more eﬃcient methods (Mahadevan et al., 2014; Du et al., 2017). 11.8 Emphatic-TD Methods We turn now to the second major strategy that has been extensively explored for obtaining a cheap and eﬃcient o↵-policy learning method with function approximation. Recall that linear semi-gradient TD methods are eﬃcient and stable when trained under the on-policy distribution, and that we showed in Section 9.4 that this has to do with the positive deﬁniteness of the matrix A (9.11)4 and the match between the on-policy state distribution µ⇡and the state-transition probabilities p(s|s, a) und


atch between the on-policy state distribution µ⇡and the state-transition probabilities p(s|s, a) under the target policy. In o↵-policy learning, we reweight the state transitions using importance sampling so that they become appropriate for learning about the target policy, but the state distribution is still that of the behavior policy. There is a mismatch. A natural idea is to somehow reweight the states, emphasizing some and de-emphasizing others, so as to return the distribution of updates to the on-policy distribution. There would then be a match, and stability and convergence would follow from existing results. This is the idea of 4In the o↵-policy case, the matrix A is generally deﬁned as Es⇠b ⇥ x(s)E ⇥ x(St+1)>"" St =s, At ⇠⇡ ⇤⇤ .


282 Chapter 11: O↵-policy Methods with Approximation Emphatic-TD methods, ﬁrst introduced for on-policy training in Section 9.11. Actually, the notion of “the on-policy distribution” is not quite right, as there are many on-policy distributions, and any one of these is suﬃcient to guarantee stability. Consider an undiscounted episodic problem. The way episodes terminate is fully determined by the transition probabilities, but there may be several di↵erent ways the episodes might begin. However the episodes start, if all state transitions are due to the target policy, then the state distribution that results is an on-policy distribution. You might start close to the terminal state and visit only a few states with high probability before ending the episode.


to the terminal state and visit only a few states with high probability before ending the episode. Or you might start far away and pass through many states before terminating. Both are on-policy distributions, and training on both with a linear semi-gradient method would be guaranteed to be stable. However the process starts, an on-policy distribution results as long as all states encountered are updated up until termination. If there is discounting, it can be treated as partial or probabilistic termination for these purposes. If γ = 0.9, then we can consider that with probability 0.1 the process terminates on every time step and then immediately restarts in the state that is transitioned to.


s terminates on every time step and then immediately restarts in the state that is transitioned to. A discounted problem is one that is continually terminating and restarting with probability 1 −γ on every step. This way of thinking about discounting is an example of a more general notion of pseudo termination—termination that does not a↵ect the sequence of state transitions, but does a↵ect the learning process and the quantities being learned. This kind of pseudo termination is important to o↵-policy learning because the restarting is optional—remember we can start any way we want to—and the termination relieves the need to keep including encountered states within the on-policy distribution.


rmination relieves the need to keep including encountered states within the on-policy distribution. That is, if we don’t consider the new states as restarts, then discounting quickly gives us a limited on-policy distribution. The one-step Emphatic-TD algorithm for learning episodic state values is deﬁned by: δt = Rt+1 + γˆv(St+1,wt) −ˆv(St,wt), wt+1 = wt + ↵Mt⇢tδtrˆv(St,wt), Mt = γ⇢t−1Mt−1 + It, with It, the interest, being arbitrary and Mt, the emphasis, being initialized to M−1 = 0. How does this algorithm perform on Baird’s counterexample? Figure 11.6 shows the trajectory in expectation of the components of the parameter vector (for the case in which It =1, for all t). There are some oscillations but eventually everything converges and the VE goes to zero.


or all t). There are some oscillations but eventually everything converges and the VE goes to zero. These trajectories are obtained by iteratively computing the expectation of the parameter vector trajectory without any of the variance due to sampling of transitions and rewards. We do not show the results of applying the Emphatic-TD algorithm directly because its variance on Baird’s counterexample is so high that it is nigh impossible to get consistent results in computational experiments. The algorithm converges to the optimal solution in theory on this problem, but in practice it does not. We turn to the topic of reducing the variance of all these algorithms in the next section.


11.9. Reducing Variance 283 Sweeps w1– w6 w7 w8 p VE 10 5 2 0 -5 0 1000 Figure 11.6: The behavior of the one-step Emphatic-TD algorithm in expectation on Baird’s counterexample. The step size was ↵= 0.03. 11.9 Reducing Variance O↵-policy learning is inherently of greater variance than on-policy learning. This is not surprising; if you receive data less closely related to a policy, you should expect to learn less about the policy’s values. In the extreme, one may be able to learn nothing. You can’t expect to learn how to drive by cooking dinner, for example. Only if the target and behavior policies are related, if they visit similar states and take similar actions, should one be able to make signiﬁcant progress in o↵-policy training.


tes and take similar actions, should one be able to make signiﬁcant progress in o↵-policy training. On the other hand, any policy has many neighbors, many similar policies with considerable overlap in states visited and actions chosen, and yet which are not identical. The raison d’ˆetre of o↵-policy learning is to enable generalization to this vast number of related-but-not-identical policies. The problem remains of how to make the best use of the experience. Now that we have some methods that are stable in expected value (if the step sizes are set right), attention naturally turns to reducing the variance of the estimates. There are many possible ideas, and we can just touch on a few of them in this introductory text.


s. There are many possible ideas, and we can just touch on a few of them in this introductory text. Why is controlling variance especially critical in o↵-policy methods based on importance sampling? As we have seen, importance sampling often involves products of policy ratios. The ratios are always one in expectation (5.13), but their actual values may be very high or as low as zero. Successive ratios are uncorrelated, so their products are also always one in expected value, but they can be of very high variance. Recall that these ratios multiply the step size in SGD methods, so high variance means taking steps that vary greatly in their sizes. This is problematic for SGD because of the occasional very large steps.


ary greatly in their sizes. This is problematic for SGD because of the occasional very large steps. They must not be so large as to take the parameter to a part of the space with a very di↵erent gradient. SGD methods rely on averaging over multiple steps to get a good sense of the gradient, and if they make large moves from single samples they become unreliable. If the step-size parameter is set small enough to prevent this, then the expected step


284 Chapter 11: O↵-policy Methods with Approximation can end up being very small, resulting in very slow learning. The notions of momentum (Derthick, 1984), of Polyak-Ruppert averaging (Polyak, 1990; Ruppert, 1988; Polyak and Juditsky, 1992), or further extensions of these ideas may signiﬁcantly help. Methods for adaptively setting separate step sizes for di↵erent components of the parameter vector are also pertinent (e.g., Jacobs, 1988; Sutton, 1992b, c), as are the “importance weight aware” updates of Karampatziakis and Langford (2010). In Chapter 5 we saw how weighted importance sampling is signiﬁcantly better behaved, with lower variance updates, than ordinary importance sampling. However, adapting weighted importance sampling to function approximation is challenging and can probably o


r, adapting weighted importance sampling to function approximation is challenging and can probably only be done approximately with O(d) complexity (Mahmood and Sutton, 2015). The Tree Backup algorithm (Section 7.5) shows that it is possible to perform some o↵-policy learning without using importance sampling. This idea has been extended to the o↵-policy case to produce stable and more eﬃcient methods by Munos, Stepleton, Harutyunyan, and Bellemare (2016) and by Mahmood, Yu and Sutton (2017). Another, complementary strategy is to allow the target policy to be determined in part by the behavior policy, in such a way that it never can be so di↵erent from it to create large importance sampling ratios.


in such a way that it never can be so di↵erent from it to create large importance sampling ratios. For example, the target policy can be deﬁned by reference to the behavior policy, as in the “recognizers” proposed by Precup et al. (2006). 11.10 Summary O↵-policy learning is a tempting challenge, testing our ingenuity in designing stable and eﬃcient learning algorithms. Tabular Q-learning makes o↵-policy learning seem easy, and it has natural generalizations to Expected Sarsa and to the Tree Backup algorithm. But as we have seen in this chapter, the extension of these ideas to signiﬁcant function approximation, even linear function approximation, involves new challenges and forces us to deepen our understanding of reinforcement learning algorithms.


lves new challenges and forces us to deepen our understanding of reinforcement learning algorithms. Why go to such lengths? One reason to seek o↵-policy algorithms is to give ﬂexibility in dealing with the tradeo↵between exploration and exploitation. Another is to free behavior from learning, and avoid the tyranny of the target policy. TD learning appears to hold out the possibility of learning about multiple things in parallel, of using one stream of experience to solve many tasks simultaneously. We can certainly do this in special cases, just not in every case that we would like to or as eﬃciently as we would like to. In this chapter we divided the challenge of o↵-policy learning into two parts.


as we would like to. In this chapter we divided the challenge of o↵-policy learning into two parts. The ﬁrst part, correcting the targets of learning for the behavior policy, is straightforwardly dealt with using the techniques devised earlier for the tabular case, albeit at the cost of increasing the variance of the updates and thereby slowing learning. High variance will probably always remains a challenge for o↵-policy learning. The second part of the challenge of o↵-policy learning emerges as the instability of semi-gradient TD methods that involve bootstrapping. We seek powerful function approximation, o↵-policy learning, and the eﬃciency and ﬂexibility of bootstrapping


11.10. Summary 285 TD methods, but it is challenging to combine all three aspects of this deadly triad in one algorithm without introducing the potential for instability. There have been several attempts. The most popular has been to seek to perform true stochastic gradient descent (SGD) in the Bellman error (a.k.a. the Bellman residual). However, our analysis concludes that this is not an appealing goal in many cases, and that anyway it is impossible to achieve with a learning algorithm—the gradient of the BE is not learnable from experience that reveals only feature vectors and not underlying states. Another approach, GradientTD methods, performs SGD in the projected Bellman error. The gradient of the PBE is learnable with O(d) complexity, but at the cost of a second parameter vector wit


dient of the PBE is learnable with O(d) complexity, but at the cost of a second parameter vector with a second step size. The newest family of methods, Emphatic-TD methods, reﬁne an old idea for reweighting updates, emphasizing some and de-emphasizing others. In this way they restore the special properties that make on-policy learning stable with computationally simple semi-gradient methods. The whole area of o↵-policy learning is relatively new and unsettled. Which methods are best or even adequate is not yet clear. Are the complexities of the new methods introduced at the end of this chapter really necessary? Which of them can be combined e↵ectively with variance reduction methods? The potential for o↵-policy learning remains tantalizing, the best way to achieve it still a mystery.


e potential for o↵-policy learning remains tantalizing, the best way to achieve it still a mystery. Bibliographical and Historical Remarks 11.1 The ﬁrst semi-gradient method was linear TD(λ) (Sutton, 1988). The name “semi-gradient” is more recent (Sutton, 2015a). Semi-gradient o↵-policy TD(0) with general importance-sampling ratio may not have been explicitly stated until Sutton, Mahmood, and White (2016), but the action-value forms were introduced by Precup, Sutton, and Singh (2000), who also did eligibility trace forms of these algorithms (see Chapter 12). Their continuing, undiscounted forms have not been signiﬁcantly explored. The n-step forms given here are new. 11.2 The earliest w-to-2w example was given by Tsitsiklis and Van Roy (1996), who also introduced the speciﬁc counterexample


w example was given by Tsitsiklis and Van Roy (1996), who also introduced the speciﬁc counterexample in the box on page 263. Baird’s counterexample is due to Baird (1995), though the version we present here is slightly modiﬁed. Averaging methods for function approximation were developed by Gordon (1995, 1996b). Other examples of instability with o↵-policy DP methods and more complex methods of function approximation are given by Boyan and Moore (1995). Bradtke (1993) gives an example in which Q-learning using linear function approximation in a linear quadratic regulation problem converges to a destabilizing policy. 11.3 The deadly triad was ﬁrst identiﬁed by Sutton (1995b) and thoroughly analyzed by Tsitsiklis and Van Roy (1997). The name “deadly triad” is due to Sutton (2015a).


oughly analyzed by Tsitsiklis and Van Roy (1997). The name “deadly triad” is due to Sutton (2015a). 11.4 This kind of linear analysis was pioneered by Tsitsiklis and Van Roy (1996; 1997), including the dynamic programming operator. Diagrams like Figure 11.3 were


286 Chapter 11: O↵-policy Methods with Approximation introduced by Lagoudakis and Parr (2003). What we have called the Bellman operator, and denoted B⇡, is more commonly denoted T ⇡and called a “dynamic programming operator,” while a generalized form, denoted T (λ), is called the “TD(λ) operator” (Tsitsiklis and Van Roy, 1996, 1997). 11.5 The BE was ﬁrst proposed as an objective function for dynamic programming by Schweitzer and Seidmann (1985). Baird (1995, 1999) extended it to TD learning based on stochastic gradient descent. In the literature, BE minimization is often referred to as Bellman residual minimization. The earliest A-split example is due to Dayan (1992). The two forms given here were introduced by Sutton et al. (2009a). 11.6 The contents of this section are new to this text.


e were introduced by Sutton et al. (2009a). 11.6 The contents of this section are new to this text. 11.7 Gradient-TD methods were introduced by Sutton, Szepesv´ari, and Maei (2009b). The methods highlighted in this section were introduced by Sutton et al. (2009a) and Mahmood et al. (2014). A major extension to proximal TD methods was developed by Mahadevan et al. (2014). The most sensitive empirical investigations to date of Gradient-TD and related methods are given by Geist and Scherrer (2014), Dann, Neumann, and Peters (2014), White (2015), and Ghiassian, Patterson, White, Sutton, and White (2018). Recent developments in the theory of Gradient-TD methods are presented by Yu (2017). 11.8 Emphatic-TD methods were introduced by Sutton, Mahmood, and White (2016).


sented by Yu (2017). 11.8 Emphatic-TD methods were introduced by Sutton, Mahmood, and White (2016). Full convergence proofs and other theory were later established by Yu (2015; 2016; Yu, Mahmood, and Sutton, 2017), Hallak, Tamar, and Mannor (2015), and Hallak, Tamar, Munos, and Mannor (2016).


Chapter 12 Eligibility Traces Eligibility traces are one of the basic mechanisms of reinforcement learning. For example, in the popular TD(λ) algorithm, the λ refers to the use of an eligibility trace. Almost any temporal-di↵erence (TD) method, such as Q-learning or Sarsa, can be combined with eligibility traces to obtain a more general method that may learn more eﬃciently. Eligibility traces unify and generalize TD and Monte Carlo methods. When TD methods are augmented with eligibility traces, they produce a family of methods spanning a spectrum that has Monte Carlo methods at one end (λ=1) and one-step TD methods at the other (λ = 0). In between are intermediate methods that are often better than either extreme method.


ther (λ = 0). In between are intermediate methods that are often better than either extreme method. Eligibility traces also provide a way of implementing Monte Carlo methods online and on continuing problems without episodes. Of course, we have already seen one way of unifying TD and Monte Carlo methods: the n-step TD methods of Chapter 7. What eligibility traces o↵er beyond these is an elegant algorithmic mechanism with signiﬁcant computational advantages. The mechanism is a short-term memory vector, the eligibility trace zt 2 Rd, that parallels the long-term weight vector wt 2 Rd. The rough idea is that when a component of wt participates in producing an estimated value, then the corresponding component of zt is bumped up and then begins to fade away.


estimated value, then the corresponding component of zt is bumped up and then begins to fade away. Learning will then occur in that component of wt if a nonzero TD error occurs before the trace falls back to zero. The trace-decay parameter λ 2 [0, 1] determines the rate at which the trace falls. The primary computational advantage of eligibility traces over n-step methods is that only a single trace vector is required rather than a store of the last n feature vectors. Learning also occurs continually and uniformly in time rather than being delayed and then catching up at the end of the episode. In addition learning can occur and a↵ect behavior immediately after a state is encountered rather than being delayed n steps.


ccur and a↵ect behavior immediately after a state is encountered rather than being delayed n steps. Eligibility traces illustrate that a learning algorithm can sometimes be implemented in a di↵erent way to obtain computational advantages. Many algorithms are most naturally formulated and understood as an update of a state’s value based on events that follow that state over multiple future time steps. For example, Monte Carlo methods (Chapter 5) update a state based on all the future rewards, and n-step TD methods (Chapter 7)


288 Chapter 12: Eligibility Traces update based on the next n rewards and state n steps in the future. Such formulations, based on looking forward from the updated state, are called forward views. Forward views are always somewhat complex to implement because the update depends on later things that are not available at the time. However, as we show in this chapter it is often possible to achieve nearly the same updates—and sometimes exactly the same updates—with an algorithm that uses the current TD error, looking backward to recently visited states using an eligibility trace. These alternate ways of looking at and implementing learning algorithms are called backward views. Backward views, transformations between forward views and backward views, and equivalences between them, date back to


ransformations between forward views and backward views, and equivalences between them, date back to the introduction of temporal di↵erence learning but have become much more powerful and sophisticated since 2014. Here we present the basics of the modern view. As usual, ﬁrst we fully develop the ideas for state values and prediction, then extend them to action values and control. We develop them ﬁrst for the on-policy case then extend them to o↵-policy learning. Our treatment pays special attention to the case of linear function approximation, for which the results with eligibility traces are stronger. All these results apply also to the tabular and state aggregation cases because these are special cases of linear function approximation.


bular and state aggregation cases because these are special cases of linear function approximation. 12.1 The λ-return In Chapter 7 we deﬁned an n-step return as the sum of the ﬁrst n rewards plus the estimated value of the state reached in n steps, each appropriately discounted (7.1). The general form of that equation, for any parameterized function approximator, is Gt:t+n .= Rt+1 +γRt+2 +· · ·+γn−1Rt+n +γnˆv(St+n,wt+n−1), 0 t T −n, (12.1) where ˆv(s,w) is the approximate value of state s given weight vector w (Chapter 9), and T is the time of episode termination, if any. We noted in Chapter 7 that each n-step return, for n ≥1, is a valid update target for a tabular learning update, just as it is for an approximate SGD learning update such as (9.7).


for a tabular learning update, just as it is for an approximate SGD learning update such as (9.7). Now we note that a valid update can be done not just toward any n-step return, but toward any average of n-step returns for di↵erent ns. For example, an update can be done toward a target that is half of a two-step return and half of a four-step return: 1 2Gt:t+2 + 1 2Gt:t+4. Any set of n-step returns can be averaged in this way, even an inﬁnite set, as long as the weights on the component returns are positive and sum to 1. The composite return possesses an error reduction property similar to that of individual n-step returns (7.3) and thus can be used to construct updates with guaranteed convergence properties. Averaging produces a substantial new range of algorithms.


s with guaranteed convergence properties. Averaging produces a substantial new range of algorithms. For example, one could average one-step and inﬁnite-step returns to obtain another way of interrelating TD and Monte Carlo methods. In principle, one could even average experience-based updates with DP updates to get a simple combination of experience-based and model-based methods (cf. Chapter 8). An update that averages simpler component updates is called a compound update. The backup diagram for a compound update consists of the backup diagrams for each of the component updates with a horizontal line above them and the weighting fractions below.


12.1. The λ-return 289 1 2 1 2 For example, the compound update for the case mentioned at the start of this section, mixing half of a two-step return and half of a four-step return, has the diagram shown to the right. A compound update can only be done when the longest of its component updates is complete. The update at the right, for example, could only be done at time t+4 for the estimate formed at time t. In general one would like to limit the length of the longest component update because of the corresponding delay in the updates. The TD(λ) algorithm can be understood as one particular way of averaging n-step updates. This average contains all the n-step updates, each weighted proportionally to λn−1 (where λ 2 [0, 1)), and is normalized by a factor of 1−λ to ensure that the weights sum


ally to λn−1 (where λ 2 [0, 1)), and is normalized by a factor of 1−λ to ensure that the weights sum to 1 (Figure 12.1). The resulting update is toward a return, called the λ-return, deﬁned in its state-based form by Gλ t .= (1 −λ) 1 X n=1 λn−1Gt:t+n. (12.2) Figure 12.2 further illustrates the weighting on the sequence of n-step returns in the λ-return. The one-step return is given the largest weight, 1 −λ; the two-step return is given the next largest weight, (1−λ)λ; the three-step return is given the weight (1−λ)λ2; and so on. The weight fades by λ with each additional step. After a terminal state has been reached, all subsequent n-step returns are equal to the conventional return, Gt. If 1 −λ (1 −λ)λ (1 −λ)λ2 λT −t−1 · · · · · · St At At+1 AT −1 St+1 Rt+1 ST RT · · · St+2 Rt+2 At+2 TD(λ


1 −λ (1 −λ)λ (1 −λ)λ2 λT −t−1 · · · · · · St At At+1 AT −1 St+1 Rt+1 ST RT · · · St+2 Rt+2 At+2 TD(λ) X = 1 Figure 12.1: The backup diagram for TD(λ). If λ = 0, then the overall update reduces to its ﬁrst component, the one-step TD update, whereas if λ = 1, then the overall update reduces to its last component, the Monte Carlo update.


290 Chapter 12: Eligibility Traces 1!" weight given to the 3-step return decay by " weight given to actual, final return t T Time Weight total area = 1 is (1 −λ)λ2 is λT −t−1 Weighting Figure 12.2: Weighting given in the λ-return to each of the n-step returns. we want, we can separate these post-termination terms from the main sum, yielding Gλ t = (1 −λ) T −t−1 X n=1 λn−1Gt:t+n + λT −t−1Gt, (12.3) as indicated in the ﬁgures. This equation makes it clearer what happens when λ = 1. In this case the main sum goes to zero, and the remaining term reduces to the conventional return. Thus, for λ = 1, updating according to the λ-return is a Monte Carlo algorithm. On the other hand, if λ = 0, then the λ-return reduces to Gt:t+1, the one-step return.


o algorithm. On the other hand, if λ = 0, then the λ-return reduces to Gt:t+1, the one-step return. Thus, for λ = 0, updating according to the λ-return is a one-step TD method. Exercise 12.1 Just as the return can be written recursively in terms of the ﬁrst reward and itself one-step later (3.9), so can the λ-return. Derive the analogous recursive relationship from (12.2) and (12.1). ⇤ Exercise 12.2 The parameter λ characterizes how fast the exponential weighting in Figure 12.2 falls o↵, and thus how far into the future the λ-return algorithm looks in determining its update. But a rate factor such as λ is sometimes an awkward way of characterizing the speed of the decay. For some purposes it is better to specify a time constant, or half-life.


ng the speed of the decay. For some purposes it is better to specify a time constant, or half-life. What is the equation relating λ and the half-life, ⌧λ, the time by which the weighting sequence will have fallen to half of its initial value? ⇤ We are now ready to deﬁne our ﬁrst learning algorithm based on the λ-return: the o↵-line λ-return algorithm. As an o↵-line algorithm, it makes no changes to the weight vector during the episode. Then, at the end of the episode, a whole sequence of o↵-line updates are made according to our usual semi-gradient rule, using the λ-return as the target: wt+1 .= wt + ↵ h Gλ t −ˆv(St,wt) i rˆv(St,wt), t = 0, . . . , T −1. (12.4)


12.1. The λ-return 291 The λ-return gives us an alternative way of moving smoothly between Monte Carlo and one-step TD methods that can be compared with the n-step bootstrapping way developed in Chapter 7. There we assessed e↵ectiveness on a 19-state random walk task (Example 7.1, page 144). Figure 12.3 shows the performance of the o↵-line λ-return algorithm on this task alongside that of the n-step methods (repeated from Figure 7.2). The experiment was just as described earlier except that for the λ-return algorithm we varied λ instead of n. The performance measure used is the estimated root-mean-square error between the correct and estimated values of each state measured at the end of the episode, averaged over the ﬁrst 10 episodes and the 19 states.


ach state measured at the end of the episode, averaged over the ﬁrst 10 episodes and the 19 states. Note that overall performance of the o↵-line λ-return algorithms is comparable to that of the n-step algorithms. In both cases we get best performance with an intermediate value of the bootstrapping parameter, n for n-step methods and λ for the o↵-line λ-return algorithm. n-step TD methods (from Chapter 7) ↵ Average RMS error over 19 states and ﬁrst 10 episodes n=1 n=2 n=4 n=8 n=16 n=32 n=32 n=64 128 512 256 0.55 0.5 0.45 0.35 0.3 0.25 0.4 0.4 0.2 0 0.8 0.6 1 0.55 0.5 0.45 0.35 0.3 0.25 0.4 0.4 0.2 0 0.8 0.6 1 Offline λ-return algorithm ↵ RMS error at the end of the episode over the ﬁrst 10 episodes λ=0 λ=.4 λ=.8 λ=.9 λ=.95 λ=.975 λ=.99 λ=1 λ=.95 Figure 12.3: 19-state Random walk results (E


pisodes λ=0 λ=.4 λ=.8 λ=.9 λ=.95 λ=.975 λ=.99 λ=1 λ=.95 Figure 12.3: 19-state Random walk results (Example 7.1): Performance of the o↵-line λ-return algorithm alongside that of the n-step TD methods. In both case, intermediate values of the bootstrapping parameter (λ or n) performed best. The results with the o↵-line λ-return algorithm are slightly better at the best values of ↵and λ, and at high ↵. The approach that we have been taking so far is what we call the theoretical, or forward, view of a learning algorithm. For each state visited, we look forward in time to all the future rewards and decide how best to combine them. We might imagine ourselves riding the stream of states, looking forward from each state to determine its update, as suggested by Figure 12.4.


am of states, looking forward from each state to determine its update, as suggested by Figure 12.4. After looking forward from and updating one state, we move on to the next and never have to work with the preceding state again. Future states, on the other hand, are viewed and processed repeatedly, once from each vantage point preceding them.


292 Chapter 12: Eligibility Traces Time rt+3 rt+2 rt+1 rT st+1 st+2 st+3 st St+1 St St+2 St+3 Rt+3 Rt+2 Rt+1 RT Figure 12.4: The forward view. We decide how to update each state by looking forward to future rewards and states. 12.2 TD(λ) TD(λ) is one of the oldest and most widely used algorithms in reinforcement learning. It was the ﬁrst algorithm for which a formal relationship was shown between a more theoretical forward view and a more computationally-congenial backward view using eligibility traces. Here we will show empirically that it approximates the o↵-line λ-return algorithm presented in the previous section. TD(λ) improves over the o↵-line λ-return algorithm in three ways. First it updates the weight vector on every step of an episode rather than only at the end, and thus its est


updates the weight vector on every step of an episode rather than only at the end, and thus its estimates may be better sooner. Second, its computations are equally distributed in time rather than all at the end of the episode. And third, it can be applied to continuing problems rather than just to episodic problems. In this section we present the semi-gradient version of TD(λ) with function approximation. With function approximation, the eligibility trace is a vector zt 2 Rd with the same number of components as the weight vector wt. Whereas the weight vector is a long-term memory, accumulating over the lifetime of the system, the eligibility trace is a short-term memory, typically lasting less time than the length of an episode.


ligibility trace is a short-term memory, typically lasting less time than the length of an episode. Eligibility traces assist in the learning process; their only consequence is that they a↵ect the weight vector, and then the weight vector determines the estimated value. In TD(λ), the eligibility trace vector is initialized to zero at the beginning of the episode, is incremented on each time step by the value gradient, and then fades away by γλ: z−1 .= 0, zt .= γλzt−1 + rˆv(St,wt), 0 t T, (12.5) where γ is the discount rate and λ is the parameter introduced in the previous section, which we henceforth call the trace-decay parameter. The eligibility trace keeps track of which components of the weight vector have contributed, positively or negatively, to recent state valuations, where “rece


he weight vector have contributed, positively or negatively, to recent state valuations, where “recent” is deﬁned in terms of γλ. (Recall that in linear function approximation, rˆv(St,wt) is the feature vector, xt, in which case the eligibility trace vector is just a sum of past, fading, input vectors.) The trace is said to indicate the eligibility of each component of the weight vector for undergoing learning changes


12.2. TD(λ) 293 should a reinforcing event occur. The reinforcing events we are concerned with are the moment-by-moment one-step TD errors. The TD error for state-value prediction is δt .= Rt+1 + γˆv(St+1,wt) −ˆv(St,wt). (12.6) In TD(λ), the weight vector is updated on each step proportional to the scalar TD error and the vector eligibility trace: wt+1 .= wt + ↵δtzt. (12.7) Semi-gradient TD(λ) for estimating ˆv ⇡v⇡ Input: the policy ⇡to be evaluated Input: a di↵erentiable function ˆv : S+ ⇥Rd ! R such that ˆv(terminal,·) = 0 Algorithm parameters: step size ↵> 0, trace decay rate λ 2 [0, 1] Initialize value-function weights w arbitrarily (e.g., w = 0) Loop for each episode: Initialize S z 0 (a d-dimensional vector) Loop for each step of episode: | Choose A ⇠⇡(·|S) | Take action A, observe R


a d-dimensional vector) Loop for each step of episode: | Choose A ⇠⇡(·|S) | Take action A, observe R, S0 | z γλz + rˆv(S,w) | δ R + γˆv(S0,w) −ˆv(S,w) | w w + ↵δz | S S0 until S0 is terminal !t et et et et Time st st+1 st-1 st-2 st-3 St St+1 St-1 St-2 St-3 δt zt zt zt zt Figure 12.5: The backward or mechanistic view of TD(λ). Each update depends on the current TD error combined with the current eligibility traces of past events.


294 Chapter 12: Eligibility Traces TD(λ) is oriented backward in time. At each moment we look at the current TD error and assign it backward to each prior state according to how much that state contributed to the current eligibility trace at that time. We might imagine ourselves riding along the stream of states, computing TD errors, and shouting them back to the previously visited states, as suggested by Figure 12.5. Where the TD error and traces come together, we get the update given by (12.7), changing the values of those past states for when they occur again in the future. To better understand the backward view of TD(λ), consider what happens at various values of λ. If λ = 0, then by (12.5) the trace at t is exactly the value gradient corresponding to St.


es of λ. If λ = 0, then by (12.5) the trace at t is exactly the value gradient corresponding to St. Thus the TD(λ) update (12.7) reduces to the one-step semi-gradient TD update treated in Chapter 9 (and, in the tabular case, to the simple TD rule (6.2)). This is why that algorithm was called TD(0). In terms of Figure 12.5, TD(0) is the case in which only the one state preceding the current one is updated by the TD error (other states may have their value estimates changed by generalization due to function approximation). For larger values of λ, but still λ < 1, more of the preceding states are updated, but each more temporally distant state is updated less because the corresponding eligibility trace is smaller, as suggested by the ﬁgure.


is updated less because the corresponding eligibility trace is smaller, as suggested by the ﬁgure. We say that the earlier states are given less credit for the TD error. If λ = 1, then the credit given to earlier states falls only by γ per step. This turns out to be just the right thing to do to achieve Monte Carlo behavior. For example, remember that the TD error, δt, includes an undiscounted term of Rt+1. In passing this back k steps it needs to be discounted, like any reward in a return, by γk, which is just what the falling eligibility trace achieves. If λ = 1 and γ = 1, then the eligibility traces do not decay at all with time. In this case the method behaves like a Monte Carlo method for an undiscounted, episodic task. If λ = 1, the algorithm is also known as TD(1).


te Carlo method for an undiscounted, episodic task. If λ = 1, the algorithm is also known as TD(1). TD(1) is a way of implementing Monte Carlo algorithms that is more general than those presented earlier and that signiﬁcantly increases their range of applicability. Whereas the earlier Monte Carlo methods were limited to episodic tasks, TD(1) can be applied to discounted continuing tasks as well. Moreover, TD(1) can be performed incrementally and online. One disadvantage of Monte Carlo methods is that they learn nothing from an episode until it is over. For example, if a Monte Carlo control method takes an action that produces a very poor reward but does not end the episode, then the agent’s tendency to repeat the action will be undiminished during the episode.


he episode, then the agent’s tendency to repeat the action will be undiminished during the episode. Online TD(1), on the other hand, learns in an n-step TD way from the incomplete ongoing episode, where the n steps are all the way up to the current step. If something unusually good or bad happens during an episode, control methods based on TD(1) can learn immediately and alter their behavior on that same episode. It is revealing to revisit the 19-state random walk example (Example 7.1) to see how well TD(λ) does in approximating the o↵-line λ-return algorithm. The results for both algorithms are shown in Figure 12.6. For each λ value, if ↵is selected optimally for it (or smaller), then the two algorithms perform virtually identically.


↵is selected optimally for it (or smaller), then the two algorithms perform virtually identically. If ↵is chosen larger than is optimal, however, then the λ-return algorithm is only a little worse whereas TD(λ) is much worse and may even be unstable. This is not catastrophic for TD(λ) on this problem, as these higher parameter values are not what one would want to use anyway, but for other problems it can be a signiﬁcant weakness.


12.3. n-step Truncated λ-return Methods 295 Offline λ-return algorithm (from the previous section) ↵ λ=0 λ=.4 λ=.8 λ=.9 λ=.95 .975 .99 1 TD(λ) ↵ λ=.8 λ=.9 RMS error at the end of the episode over the ﬁrst 10 episodes 0.4 0.2 0 0.8 0.6 1 λ=0 λ=.4 λ=.8 λ=.9 λ=.95 λ=.975 λ=.99 λ=1 λ=.95 0.55 0.5 0.45 0.35 0.3 0.25 0.4 0.4 0.2 0 0.8 0.6 1 Figure 12.6: 19-state Random walk results (Example 7.1): Performance of TD(λ) alongside that of the o↵-line λ-return algorithm. The two algorithms performed virtually identically at low (less than optimal) ↵values, but TD(λ) was worse at high ↵values. Linear TD(λ) has been proved to converge in the on-policy case if the step-size parameter is reduced over time according to the usual conditions (2.7).


olicy case if the step-size parameter is reduced over time according to the usual conditions (2.7). Just as discussed in Section 9.4, convergence is not to the minimum-error weight vector, but to a nearby weight vector that depends on λ. The bound on solution quality presented in that section (9.14) can now be generalized to apply for any λ. For the continuing discounted case, VE(w1)  1 −γλ 1 −γ min w VE(w). (12.8) That is, the asymptotic error is no more than 1−γλ 1−γ times the smallest possible error. As λ approaches 1, the bound approaches the minimum error (and it is loosest at λ = 0). In practice, however, λ = 1 is often the poorest choice, as will be illustrated later in Figure 12.14. Exercise 12.3 Some insight into how TD(λ) can closely approximate the o↵-line λ-return algorithm ca


Exercise 12.3 Some insight into how TD(λ) can closely approximate the o↵-line λ-return algorithm can be gained by seeing that the latter’s error term (in brackets in (12.4)) can be written as the sum of TD errors (12.6) for a single ﬁxed w. Show this, following the pattern of (6.6), and using the recursive relationship for the λ-return you obtained in Exercise 12.1. ⇤ Exercise 12.4 Use your result from the preceding exercise to show that, if the weight updates over an episode were computed on each step but not actually used to change the weights (w remained ﬁxed), then the sum of TD(λ)’s weight updates would be the same as the sum of the o↵-line λ-return algorithm’s updates. ⇤ 12.3 n-step Truncated λ-return Methods The o↵-line λ-return algorithm is an important ideal, but it is of limited


runcated λ-return Methods The o↵-line λ-return algorithm is an important ideal, but it is of limited utility because it uses the λ-return (12.2), which is not known until the end of the episode. In the


296 Chapter 12: Eligibility Traces continuing case, the λ-return is technically never known, as it depends on n-step returns for arbitrarily large n, and thus on rewards arbitrarily far in the future. However, the dependence becomes weaker for longer-delayed rewards, falling by γλ for each step of delay. A natural approximation, then, would be to truncate the sequence after some number of steps. Our existing notion of n-step returns provides a natural way to do this in which the missing rewards are replaced with estimated values. In general, we deﬁne the truncated λ-return for time t, given data only up to some later horizon, h, as Gλ t:h .= (1 −λ) h−t−1 X n=1 λn−1Gt:t+n + λh−t−1Gt:h, 0 t < h T.


up to some later horizon, h, as Gλ t:h .= (1 −λ) h−t−1 X n=1 λn−1Gt:t+n + λh−t−1Gt:h, 0 t < h T. (12.9) If you compare this equation with the λ-return (12.3), it is clear that the horizon h is playing the same role as was previously played by T, the time of termination. Whereas in the λ-return there is a residual weight given to the conventional return Gt, here it is given to the longest available n-step return, Gt:h (Figure 12.2). The truncated λ-return immediately gives rise to a family of n-step λ-return algorithms similar to the n-step methods of Chapter 7. In all of these algorithms, updates are delayed by n steps and only take into account the ﬁrst n rewards, but now all the k-step returns are included for 1 k n (whereas the earlier n-step algorithms used only the n-step return)


returns are included for 1 k n (whereas the earlier n-step algorithms used only the n-step return), weighted geometrically as in Figure 12.2. In the state-value case, this family of algorithms is known as Truncated TD(λ), or TTD(λ). The compound backup diagram, shown in Figure 12.7, is similar to that for TD(λ) (Figure 12.1) except that the longest component update is at most n steps rather than always going all the way to the 1 −λ (1 −λ)λ (1 −λ)λ2 λT −t−1 or, if t + n≥T · · · · · · · · · λn−1 St At At+1 AT −1 St+n Rt+n St+1 Rt+1 ST RT At+n−1 n-step truncated TD(λ) Figure 12.7: The backup diagram for Truncated TD(λ).


12.4. Redoing Updates: Online λ-return Algorithm 297 end of the episode. TTD(λ) is deﬁned by (cf. (9.15)): wt+n .= wt+n−1 + ↵ ⇥ Gλ t:t+n −ˆv(St,wt+n−1) ⇤ rˆv(St,wt+n−1), 0 t < T. This algorithm can be implemented eﬃciently so that per-step computation does not scale with n (though of course memory must). Much as in n-step TD methods, no updates are made on the ﬁrst n −1 time steps of each episode, and n −1 additional updates are made upon termination. Eﬃcient implementation relies on the fact that the k-step λ-return can be written exactly as Gλ t:t+k = ˆv(St,wt−1) + t+k−1 X i=t (γλ)i−tδ0 i, (12.10) where δ0 t .= Rt+1 + γˆv(St+1,wt) −ˆv(St,wt−1). Exercise 12.5 Several times in this book (often in exercises) we have established that returns can be written as sums of TD errors if the value


ten in exercises) we have established that returns can be written as sums of TD errors if the value function is held constant. Why is (12.10) another instance of this? Prove (12.10). ⇤ 12.4 Redoing Updates: Online λ-return Algorithm Choosing the truncation parameter n in Truncated TD(λ) involves a tradeo↵. n should be large so that the method closely approximates the o↵-line λ-return algorithm, but it should also be small so that the updates can be made sooner and can inﬂuence behavior sooner. Can we get the best of both? Well, yes, in principle we can, albeit at the cost of computational complexity. The idea is that, on each time step as you gather a new increment of data, you go back and redo all the updates since the beginning of the current episode.


increment of data, you go back and redo all the updates since the beginning of the current episode. The new updates will be better than the ones you previously made because now they can take into account the time step’s new data. That is, the updates are always towards an n-step truncated λ-return target, but they always use the latest horizon. In each pass over that episode you can use a slightly longer horizon and obtain slightly better results. Recall that the truncated λ-return is deﬁned in (12.9) as Gλ t:h .= (1 −λ) h−t−1 X n=1 λn−1Gt:t+n + λh−t−1Gt:h. Let us step through how this target could ideally be used if computational complexity was not an issue. The episode begins with an estimate at time 0 using the weights w0 from the end of the previous episode.


pisode begins with an estimate at time 0 using the weights w0 from the end of the previous episode. Learning begins when the data horizon is extended to time step 1. The target for the estimate at step 0, given the data up to horizon 1, could only be the one-step return G0:1, which includes R1 and bootstraps from the estimate ˆv(S1,w0). Note that this is exactly what Gλ 0:1 is, with the sum in the ﬁrst term of the


298 Chapter 12: Eligibility Traces equation degenerating to zero. Using this update target, we construct w1. Then, after advancing the data horizon to step 2, what do we do? We have new data in the form of R2 and S2, as well as the new w1, so now we can construct a better update target Gλ 0:2 for the ﬁrst update from S0 as well as a better update target Gλ 1:2 for the second update from S1. Using these improved targets, we redo the updates at S1 and S2, starting again from w0, to produce w2. Now we advance the horizon to step 3 and repeat, going all the way back to produce three new targets, redoing all updates starting from the original w0 to produce w3, and so on. Each time the horizon is advanced, all the updates are redone starting from w0 using the weight vector from the preceding hor


advanced, all the updates are redone starting from w0 using the weight vector from the preceding horizon. This conceptual algorithm involves multiple passes over the episode, one at each horizon, each generating a di↵erent sequence of weight vectors. To describe it clearly we have to distinguish between the weight vectors computed at the di↵erent horizons. Let us use wh t to denote the weights used to generate the value at time t in the sequence up to horizon h. The ﬁrst weight vector wh 0 in each sequence is that inherited from the previous episode (so they are the same for all h), and the last weight vector wh h in each sequence deﬁnes the ultimate weight-vector sequence of the algorithm. At the ﬁnal horizon h = T we obtain the ﬁnal weights wT T which will be passed on to form the initia


At the ﬁnal horizon h = T we obtain the ﬁnal weights wT T which will be passed on to form the initial weights of the next episode. With these conventions, the three ﬁrst sequences described in the previous paragraph can be given explicitly: h = 1 : w1 1 .= w1 0 + ↵ ⇥ Gλ 0:1 −ˆv(S0,w1 0) ⇤ rˆv(S0,w1 0), h = 2 : w2 1 .= w2 0 + ↵ ⇥ Gλ 0:2 −ˆv(S0,w2 0) ⇤ rˆv(S0,w2 0), w2 2 .= w2 1 + ↵ ⇥ Gλ 1:2 −ˆv(S1,w2 1) ⇤ rˆv(S1,w2 1), h = 3 : w3 1 .= w3 0 + ↵ ⇥ Gλ 0:3 −ˆv(S0,w3 0) ⇤ rˆv(S0,w3 0), w3 2 .= w3 1 + ↵ ⇥ Gλ 1:3 −ˆv(S1,w3 1) ⇤ rˆv(S1,w3 1), w3 3 .= w3 2 + ↵ ⇥ Gλ 2:3 −ˆv(S2,w3 2) ⇤ rˆv(S2,w3 2). The general form for the update is wh t+1 .= wh t + ↵ ⇥ Gλ t:h −ˆv(St,wh t ) ⇤ rˆv(St,wh t ), 0 t < h T. This update, together with wt .= wt t deﬁnes the online λ-return algorithm.


St,wh t ), 0 t < h T. This update, together with wt .= wt t deﬁnes the online λ-return algorithm. The online λ-return algorithm is fully online, determining a new weight vector wt at each step t during an episode, using only information available at time t. Its main drawback is that it is computationally complex, passing over the portion of the episode experienced so far on every step. Note that it is strictly more complex than the o↵-line λ-return algorithm, which passes through all the steps at the time of termination but does not make any updates during the episode. In return, the online algorithm can be expected to perform better than the o↵-line one, not only during the episode when it makes an update while the o↵-line algorithm makes none, but also at the end of the episode because


t makes an update while the o↵-line algorithm makes none, but also at the end of the episode because the weight vector used in bootstrapping (in Gλ t:h) has had a larger number of informative


12.5. True Online TD(λ) 299 updates. This e↵ect can be seen if one looks carefully at Figure 12.8, which compares the two algorithms on the 19-state random walk task. Offline λ-return algorithm (from Section 12.1) ↵ ↵ RMS error at the end of the episode over the ﬁrst 10 episodes 0.4 0.2 0 0.8 0.6 1 λ=0 λ=.4 λ=.8 λ=.9 λ=.95 λ=.975 λ=.99 λ=1 λ=.95 0.55 0.5 0.45 0.35 0.3 0.25 0.4 0.4 0.2 0 0.8 0.6 1 On-line λ-return algorithm = true online TD(λ) λ=0 λ=.4 λ=.8 λ=.9 λ=.95 λ=.975 λ=.99 λ=1 λ=.95 Figure 12.8: 19-state Random walk results (Example 7.1): Performance of online and o↵-line λ-return algorithms. The performance measure here is the VE at the end of the episode, which should be the best case for the o↵-line algorithm. Nevertheless, the online algorithm performs subtly better.


the best case for the o↵-line algorithm. Nevertheless, the online algorithm performs subtly better. For comparison, the λ=0 line is the same for both methods. 12.5 True Online TD(λ) The online λ-return algorithm just presented is currently the best performing temporaldi↵erence algorithm. It is an ideal which online TD(λ) only approximates. As presented, however, the online λ-return algorithm is very complex. Is there a way to invert this forward-view algorithm to produce an eﬃcient backward-view algorithm using eligibility traces? It turns out that there is indeed an exact computationally congenial implementation of the online λ-return algorithm for the case of linear function approximation. This implementation is known as the true online TD(λ) algorithm because it is “truer” to the ideal


This implementation is known as the true online TD(λ) algorithm because it is “truer” to the ideal of the online λ-return algorithm than the TD(λ) algorithm is. The derivation of true online TD(λ) is a little too complex to present here (see the next section and the appendix to the paper by van Seijen et al., 2016) but its strategy is simple. The sequence of weight vectors produced by the online λ-return algorithm can be arranged in a triangle: w0 0 w1 0 w1 1 w2 0 w2 1 w2 2 w3 0 w3 1 w3 2 w3 3 ... ... ... ... ... wT 0 wT 1 wT 2 wT 3 · · · wT T One row of this triangle is produced on each time step. It turns out that the weight vectors on the diagonal, the wt t, are the only ones really needed. The ﬁrst, w0 0, is the initial weight


300 Chapter 12: Eligibility Traces vector of the episode, the last, wT T , is the ﬁnal weight vector, and each weight vector along the way, wt t, plays a role in bootstrapping in the n-step returns of the updates. In the ﬁnal algorithm the diagonal weight vectors are renamed without a superscript, wt .= wt t. The strategy then is to ﬁnd a compact, eﬃcient way of computing each wt t from the one before. If this is done, for the linear case in which ˆv(s,w) = w>x(s), then we arrive at the true online TD(λ) algorithm: wt+1 .= wt + ↵δtzt + ↵ # w> t xt −w> t−1xt $ (zt −xt), where we have used the shorthand xt .= x(St), δt is deﬁned as in TD(λ) (12.6), and zt is deﬁned by zt .= γλzt−1 + # 1 −↵γλz> t−1xt $ xt.


= x(St), δt is deﬁned as in TD(λ) (12.6), and zt is deﬁned by zt .= γλzt−1 + # 1 −↵γλz> t−1xt $ xt. (12.11) This algorithm has been proven to produce exactly the same sequence of weight vectors, wt, 0 t T, as the online λ-return algorithm (van Seijen et al. 2016). Thus the results on the random walk task on the left of Figure 12.8 are also its results on that task. Now, however, the algorithm is much less expensive. The memory requirements of true online TD(λ) are identical to those of conventional TD(λ), while the per-step computation is increased by about 50% (there is one more inner product in the eligibility-trace update). Overall, the per-step computational complexity remains of O(d), the same as TD(λ). Pseudocode for the complete algorithm is given in the box.


xity remains of O(d), the same as TD(λ). Pseudocode for the complete algorithm is given in the box. True online TD(λ) for estimating w>x ⇡v⇡ Input: the policy ⇡to be evaluated Input: a feature function x : S+ ! Rd such that x(terminal, ·) = 0 Algorithm parameters: step size ↵> 0, trace decay rate λ 2 [0, 1] Initialize value-function weights w 2 Rd (e.g., w = 0) Loop for each episode: Initialize state and obtain initial feature vector x z 0 (a d-dimensional vector) Vold 0 (a temporary scalar variable) Loop for each step of episode: | Choose A ⇠⇡ | Take action A, observe R, x0 (feature vector of the next state) | V w>x | V 0 w>x0 | δ R + γV 0 −V | z γλz + # 1 −↵γλz>x $ x | w w + ↵(δ + V −Vold)z −↵(V −Vold)x | Vold V 0 | x x0 until x0 = 0 (signaling arrival at a terminal state) The eligibilit


z −↵(V −Vold)x | Vold V 0 | x x0 until x0 = 0 (signaling arrival at a terminal state) The eligibility trace (12.11) used in true online TD(λ) is called a dutch trace to distinguish it from the trace (12.5) used in TD(λ), which is called an accumulating trace.


12.6. *Dutch Traces in Monte Carlo Learning 301 Earlier work often used a third kind of trace called the replacing trace, deﬁned only for the tabular case or for binary feature vectors such as those produced by tile coding. The replacing trace is deﬁned on a component-by-component basis depending on whether the component of the feature vector was 1 or 0: zi,t .= ⇢1 if xi,t = 1 γλzi,t−1 otherwise. (12.12) Nowadays, we see replacing traces as crude approximations to dutch traces, which largely supersede them. Dutch traces usually perform better than replacing traces and have a clearer theoretical basis. Accumulating traces remain of interest for nonlinear function approximations where dutch traces are not available.


aces remain of interest for nonlinear function approximations where dutch traces are not available. 12.6 *Dutch Traces in Monte Carlo Learning Although eligibility traces are closely associated historically with TD learning, in fact they have nothing to do with it. In fact, eligibility traces arise even in Monte Carlo learning, as we show in this section. We show that the linear MC algorithm (Chapter 9), taken as a forward view, can be used to derive an equivalent yet computationally cheaper backward-view algorithm using dutch traces. This is the only equivalence of forward- and backward-views that we explicitly demonstrate in this book. It gives some of the ﬂavor of the proof of equivalence of true online TD(λ) and the online λ-return algorithm, but is much simpler.


e proof of equivalence of true online TD(λ) and the online λ-return algorithm, but is much simpler. The linear version of the gradient Monte Carlo prediction algorithm (page 202) makes the following sequence of updates, one for each time step of the episode: wt+1 .= wt + ↵ ⇥ G −w> t xt ⇤ xt, 0 t < T. (12.13) To simplify the example, we assume here that the return G is a single reward received at the end of the episode (this is why G is not subscripted by time) and that there is no discounting. In this case the update is also known as the Least Mean Square (LMS) rule. As a Monte Carlo algorithm, all the updates depend on the ﬁnal reward/return, so none can be made until the end of the episode. The MC algorithm is an o↵-line algorithm and we do not seek to improve this aspect of it.


episode. The MC algorithm is an o↵-line algorithm and we do not seek to improve this aspect of it. Rather we seek merely an implementation of this algorithm with computational advantages. We will still update the weight vector only at the end of the episode, but we will do some computation during each step of the episode and less at its end. This will give a more equal distribution of computation—O(d) per step—and also remove the need to store the feature vectors at each step for use later at the end of each episode. Instead, we will introduce an additional vector memory, the eligibility trace, keeping in it a summary of all the feature vectors seen so far. This will be suﬃcient to eﬃciently recreate exactly the same overall update as the sequence of MC


302 Chapter 12: Eligibility Traces updates (12.13), by the end of the episode: wT = wT −1 + ↵ # G −w> T −1xT −1 $ xT −1 = wT −1 + ↵xT −1 # −x> T −1wT −1 $ + ↵GxT −1 = # I −↵xT −1x> T −1 $ wT −1 + ↵GxT −1 = FT −1wT −1 + ↵GxT −1 where Ft .= I −↵xtx> t is a forgetting, or fading, matrix. Now, recursing, = FT −1 (FT −2wT −2 + ↵GxT −2) + ↵GxT −1 = FT −1FT −2wT −2 + ↵G (FT −1xT −2 + xT −1) = FT −1FT −2 (FT −3wT −3 + ↵GxT −3) + ↵G (FT −1xT −2 + xT −1) = FT −1FT −2FT −3wT −3 + ↵G (FT −1FT −2xT −3 + FT −1xT −2 + xT −1) ... = FT −1FT −2 · · · F0w0 | {z } aT −1 + ↵G T −1 X k=0 FT −1FT −2 · · · Fk+1xk | {z } zT −1 = aT −1 + ↵GzT −1, (12.14) where aT −1 and zT −1 are the values at time T −1 of two auxiliary memory vectors that can be updated incrementally without knowledge of G and with O(d) complexity


ary memory vectors that can be updated incrementally without knowledge of G and with O(d) complexity per time step. The zt vector is in fact a dutch-style eligibility trace. It is initialized to z0 = x0 and then updated according to zt .= t X k=0 FtFt−1 · · · Fk+1xk, 1 t < T = t−1 X k=0 FtFt−1 · · · Fk+1xk + xt = Ft t−1 X k=0 Ft−1Ft−2 · · · Fk+1xk + xt = Ftzt−1 + xt = # I −↵xtx> t $ zt−1 + xt = zt−1 −↵xtx> t zt−1 + xt = zt−1 −↵ # z> t−1xt $ xt + xt = zt−1 + # 1 −↵z> t−1xt $ xt, which is the dutch trace for the case of γλ=1 (cf. Eq. 12.11). The at auxiliary vector is initialized to a0 = w0 and then updated according to at .= FtFt−1 · · · F0w0 = Ftat−1 = at−1 −↵xtx> t at−1, 1 t < T.


12.7. Sarsa(λ) 303 The auxiliary vectors, at and zt, are updated on each time step t < T and then, at time T when G is observed, they are used in (12.14) to compute wT . In this way we achieve exactly the same ﬁnal result as the MC/LMS algorithm that has poor computational properties (12.13), but now with an incremental algorithm whose time and memory complexity per step is O(d). This is surprising and intriguing because the notion of an eligibility trace (and the dutch trace in particular) has arisen in a setting without temporal-di↵erence (TD) learning (in contrast to van Seijen and Sutton, 2014). It seems eligibility traces are not speciﬁc to TD learning at all; they are more fundamental than that.


eems eligibility traces are not speciﬁc to TD learning at all; they are more fundamental than that. The need for eligibility traces seems to arise whenever one tries to learn long-term predictions in an eﬃcient manner. 12.7 Sarsa(λ) Very few changes in the ideas already presented in this chapter are required in order to extend eligibility-traces to action-value methods. To learn approximate action values, ˆq(s, a, w), rather than approximate state values, ˆv(s,w), we need to use the action-value form of the n-step return, from Chapter 10: Gt:t+n .= Rt+1 + · · · + γn−1Rt+n + γnˆq(St+n, At+n, wt+n−1), t + n < T, with Gt:t+n .= Gt if t + n ≥T. Using this, we can form the action-value form of the λ-return, which is otherwise identical to the state-value form (12.3).


the action-value form of the λ-return, which is otherwise identical to the state-value form (12.3). The action-value form of the o↵-line λ-return algorithm (12.4) simply uses ˆq rather than ˆv: wt+1 .= wt + ↵ h Gλ t −ˆq(St, At, wt) i rˆq(St, At, wt), t = 0, . . . , T −1, (12.15) where Gλ t .= Gλ t:1. The compound backup diagram for this forward view is shown in Figure 12.9. Notice the similarity to the diagram of the TD(λ) algorithm (Figure 12.1). The ﬁrst update looks ahead one full step, to the next state–action pair, the second looks ahead two steps, to the second state–action pair, and so on. A ﬁnal update is based on the complete return. The weighting of each n-step update in the λ-return is just as in TD(λ) and the λ-return algorithm (12.3).


ghting of each n-step update in the λ-return is just as in TD(λ) and the λ-return algorithm (12.3). The temporal-di↵erence method for action values, known as Sarsa(λ), approximates this forward view. It has the same update rule as given earlier for TD(λ): wt+1 .= wt + ↵δtzt, except, naturally, using the action-value form of the TD error: δt .= Rt+1 + γˆq(St+1, At+1, wt) −ˆq(St, At, wt), (12.16) and the action-value form of the eligibility trace: z−1 .= 0, zt .= γλzt−1 + rˆq(St, At, wt), 0 t T.


304 Chapter 12: Eligibility Traces 1 −λ (1 −λ)λ (1 −λ)λ2 λT −t−1 · · · · · · St At At+1 AT −1 St+1 Rt+1 ST RT · · · St+2 Rt+2 At+2 X = 1 Sarsa(λ) Figure 12.9: Sarsa(λ)’s backup diagram. Compare with Figure 12.1. Complete pseudocode for Sarsa(λ) with linear function approximation, binary features, and either accumulating or replacing traces is given in the box on the next page. This pseudocode highlights a few optimizations possible in the special case of binary features (features are either active (=1) or inactive (=0)). Example 12.1: Traces in Gridworld The use of eligibility traces can substantially increase the eﬃciency of control algorithms over one-step methods and even over n-step methods. The reason for this is illustrated by the gridworld example below.


ds and even over n-step methods. The reason for this is illustrated by the gridworld example below. Path taken Action values increased by one-step Sarsa Action values increased by Sarsa( ) with =0.9 G G G Path taken Action values increased by one-step Sarsa Action values increased by Sarsa( ) with  =0.9 by 10-step Sarsa G G G Path taken Action values increased by one-step Sarsa Action values increased by 10-step Sarsa G G G h h The ﬁrst panel shows the path taken by an agent in a single episode. The initial estimated values were zero, and all rewards were zero except for a positive reward at the goal location marked by G. The arrows in the other panels show, for various algorithms, which action-values would be increased, and by how much, upon reaching the goal.


arious algorithms, which action-values would be increased, and by how much, upon reaching the goal. A one-step method would increment only the last action value, whereas an n-step method would equally increment the last n actions’ values (assuming γ = 1), and an eligibility trace method would update all the action values up to the beginning of the episode, to di↵erent degrees, fading with recency. The fading strategy is often the best.


12.7. Sarsa(λ) 305 Sarsa(λ) with binary features and linear function approximation for estimating w>x ⇡q⇡or q⇤ Input: a function F(s, a) returning the set of (indices of) active features for s, a Input: a policy ⇡ Algorithm parameters: step size ↵> 0, trace decay rate λ 2 [0, 1], small " > 0 Initialize: w = (w1, . . . , wd)> 2 Rd (e.g., w = 0), z = (z1, . . . , zd)> 2 Rd Loop for each episode: Initialize S Choose A ⇠⇡(·|S) or "-greedy according to ˆq(S, ·, w) z 0 Loop for each step of episode: Take action A, observe R, S0 δ R Loop for i in F(S, A): δ δ −wi zi zi + 1 (accumulating traces) or zi 1 (replacing traces) If S0 is terminal then: w w + ↵δz Go to next episode Choose A0 ⇠⇡(·|S0) or "-greedy according to ˆq(S0, ·, w) Loop for i in F(S0, A0): δ δ + γwi w w + ↵δz z γλz S S0; A A0 Exerci


reedy according to ˆq(S0, ·, w) Loop for i in F(S0, A0): δ δ + γwi w w + ↵δz z γλz S S0; A A0 Exercise 12.6 Modify the pseudocode for Sarsa(λ) to use dutch traces (12.11) without the other distinctive features of a true online algorithm. Assume linear function approximation and binary features. ⇤ Example 12.2: Sarsa(λ) on Mountain Car Figure 12.10 (left) on the next page shows results with Sarsa(λ) on the Mountain Car task introduced in Example 10.1. The function approximation, action selection, and environmental details were exactly as in Chapter 10, and thus it is appropriate to numerically compare these results with the Chapter 10 results for n-step Sarsa (right side of the ﬁgure). The earlier results varied the update length n whereas here for Sarsa(λ) we vary the trace parameter λ, wh


rlier results varied the update length n whereas here for Sarsa(λ) we vary the trace parameter λ, which plays a similar role. The fading-trace bootstrapping strategy of Sarsa(λ) appears to result in more eﬃcient learning on this problem. There is also an action-value version of our ideal TD method, the online λ-return algorithm (Section 12.4) and its eﬃcient implementation as true online TD(λ) (Section 12.5). Everything in Section 12.4 goes through without change other than to use the action-value form of the n-step return given at the beginning of the current section. The analyses in Sections 12.5 and 12.6 also carry through for action values, the only change being the use


306 Chapter 12: Eligibility Traces 220 240 260 300 0 0.5 1 1.5 Mountain Car Steps per episode averaged over ﬁrst 50 episodes and 100 runs 280 200 180 × number of tilings (8) ↵ λ=.96 λ=.92 λ=.99 λ=.84 λ=.68 λ=0 220 240 260 300 0 0.5 1 1.5 280 0 0.5 1 1.5 n=1 n=2 n=4 n=8 n=16 n=8 n=4 n=2 n=16 n=1 λ=.98 × number of tilings (8) ↵ Sarsa(λ) with replacing traces n-step Sarsa λ=.92 λ=.84 Figure 12.10: Early performance on the Mountain Car task of Sarsa(λ) with replacing traces and n-step Sarsa (copied from Figure 10.4) as a function of the step size, ↵. of state–action feature vectors xt = x(St, At) instead of state feature vectors xt = x(St). Pseudocode for the resulting eﬃcient algorithm, called true online Sarsa(λ) is given in the box on the next page.


the resulting eﬃcient algorithm, called true online Sarsa(λ) is given in the box on the next page. The ﬁgure below compares the performance of various versions of Sarsa(λ) on the Mountain Car example. 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 −550 −500 −450 −400 −350 −300 −250 −200 −150 Sarsa(λ), replacing, clearing Sarsa(λ), replacing, no clearing Sarsa(λ), accumulating Mountain Car Reward per episode averaged over ﬁrst 20 episodes and 100 runs × number of tilings (8) ↵ 7UXHRQOLQH6DUVD λ Sarsa(λ) with replacing traces Sarsa(λ) with replacing traces and clearing the traces of other actions Sarsa(λ) with accumulating traces Figure 12.11: Summary comparison of Sarsa(λ) algorithms on the Mountain Car task.


cumulating traces Figure 12.11: Summary comparison of Sarsa(λ) algorithms on the Mountain Car task. True online Sarsa(λ) performed better than regular Sarsa(λ) with both accumulating and replacing traces. Also included is a version of Sarsa(λ) with replacing traces in which, on each time step, the traces for the state and the actions not selected were set to zero.


12.8. Variable λ and γ 307 True online Sarsa(λ) for estimating w>x ⇡q⇡or q⇤ Input: a feature function x : S+ ⇥A ! Rd such that x(terminal, ·) = 0 Input: a policy ⇡(if estimating q⇡) Algorithm parameters: step size ↵> 0, trace decay rate λ 2 [0, 1], small " > 0 Initialize: w 2 Rd (e.g., w = 0) Loop for each episode: Initialize S Choose A ⇠⇡(·|S) or "-greedy according to ˆq(S, ·, w) x x(S, A) z 0 Qold 0 Loop for each step of episode: | Take action A, observe R, S0 | Choose A0 ⇠⇡(·|S0) or "-greedy according to ˆq(S0, ·, w) | x0 x(S0, A0) | Q w>x | Q0 w>x0 | δ R + γQ0 −Q | z γλz + # 1 −↵γλz>x $ x | w w + ↵(δ + Q −Qold)z −↵(Q −Qold)x | Qold Q0 | x x0 | A A0 until S0 is terminal Finally, there is also a truncated version of Sarsa(λ), called forward Sarsa(λ) (van Seijen, 2016), which appears to b


also a truncated version of Sarsa(λ), called forward Sarsa(λ) (van Seijen, 2016), which appears to be a particularly e↵ective model-free control method for use in conjunction with multi-layer artiﬁcial neural networks. 12.8 Variable λ and γ We are starting now to reach the end of our development of fundamental TD learning algorithms. To present the ﬁnal algorithms in their most general forms, it is useful to generalize the degree of bootstrapping and discounting beyond constant parameters to functions potentially dependent on the state and action. That is, each time step will have a di↵erent λ and γ, denoted λt and γt. We change notation now so that λ : S⇥A ! [0, 1] is now a function from states and actions to the unit interval such that λt .= λ(St, At), and similarly, γ : S !


tion from states and actions to the unit interval such that λt .= λ(St, At), and similarly, γ : S ! [0, 1] is a function from states to the unit interval such that γt .= γ(St). Introducing the function γ, the termination function, is particularly signiﬁcant because it changes the return, the fundamental random variable whose expectation we seek to


308 Chapter 12: Eligibility Traces estimate. Now the return is deﬁned more generally as Gt .= Rt+1 + γt+1Gt+1 = Rt+1 + γt+1Rt+2 + γt+1γt+2Rt+3 + γt+1γt+2γt+3Rt+4 + · · · = 1 X k=t k Y i=t+1 γi ! Rk+1, (12.17) where, to assure the sums are ﬁnite, we require that Q1 k=t γk = 0 with probability one for all t. One convenient aspect of this deﬁnition is that it enables the episodic setting and its algorithms to be presented in terms of a single stream of experience, without special terminal states, start distributions, or termination times. An erstwhile terminal state becomes a state at which γ(s) = 0 and which transitions to the start distribution. In that way (and by choosing γ(·) as a constant in all other states) we can recover the classical episodic setting as a special case.


as a constant in all other states) we can recover the classical episodic setting as a special case. State dependent termination includes other prediction cases such as pseudo termination, in which we seek to predict a quantity without altering the ﬂow of the Markov process. Discounted returns can be thought of as such a quantity, in which case state-dependent termination uniﬁes the episodic and discounted-continuing cases. (The undiscounted-continuing case still needs some special treatment.) The generalization to variable bootstrapping is not a change in the problem, like discounting, but a change in the solution strategy. The generalization a↵ects the λreturns for states and actions. The new state-based λ-return can be written recursively as Gλs t .= Rt+1 + γt+1 # (1 −λt+1)ˆv(St+1,wt) +


ew state-based λ-return can be written recursively as Gλs t .= Rt+1 + γt+1 # (1 −λt+1)ˆv(St+1,wt) + λt+1Gλs t+1 $ , (12.18) where now we have added the “s” to the superscript λ to remind us that this is a return that bootstraps from state values, distinguishing it from returns that bootstrap from action values, which we present below with “a” in the superscript. This equation says that the λ-return is the ﬁrst reward, undiscounted and una↵ected by bootstrapping, plus possibly a second term to the extent that we are not discounting at the next state (that is, according to γt+1; recall that this is zero if the next state is terminal). To the extent that we aren’t terminating at the next state, we have a second term which is itself divided into two cases depending on the degree of bootstrappi


we have a second term which is itself divided into two cases depending on the degree of bootstrapping in the state. To the extent we are bootstrapping, this term is the estimated value at the state, whereas, to the extent that we are not bootstrapping, the term is the λ-return for the next time step. The action-based λ-return is either the Sarsa form Gλa t .= Rt+1 + γt+1 ⇣ (1 −λt+1)ˆq(St+1, At+1, wt) + λt+1Gλa t+1 ⌘ , (12.19) or the Expected Sarsa form, Gλa t .= Rt+1 + γt+1 ⇣ (1 −λt+1) ¯Vt(St+1) + λt+1Gλa t+1 ⌘ , (12.20) where (7.8) is generalized to function approximation as ¯Vt(s) .= X a ⇡(a|s)ˆq(s, a, wt). (12.21)


12.9. O↵-policy Traces with Control Variates 309 Exercise 12.7 Generalize the three recursive equations above to their truncated versions, deﬁning Gλs t:h and Gλa t:h. ⇤ 12.9 O↵-policy Traces with Control Variates The ﬁnal step is to incorporate importance sampling. For methods using non-truncated λ-returns, there is not a practical option in which the importance-sampling weighting is applied to the target return (as there is for n-step methods as explained in Section 7.3). Instead, we move directly to the bootstrapping generalization of per-decision importance sampling with control variates (Section 7.4). In the state case, our ﬁnal deﬁnition of the λ-return generalizes (12.18), after the model of (7.13), to Gλs t .= ⇢t ⇣ Rt+1 +γt+1 # (1−λt+1)ˆv(St+1,wt)+λt+1Gλs t+1 $⌘ +(1−⇢t)ˆv(St,wt), (


odel of (7.13), to Gλs t .= ⇢t ⇣ Rt+1 +γt+1 # (1−λt+1)ˆv(St+1,wt)+λt+1Gλs t+1 $⌘ +(1−⇢t)ˆv(St,wt), (12.22) where ⇢t = ⇡(At|St) b(At|St) is the usual single-step importance sampling ratio. Much like the other returns we have seen in this book, this ﬁnal λ-return can be approximated simply in terms of sums of the state-based TD error, δs t .= Rt+1 + γt+1ˆv(St+1,wt) −ˆv(St,wt), (12.23) as Gλs t ⇡ˆv(St,wt) + ⇢t 1 X k=t δs k k Y i=t+1 γiλi⇢i, (12.24) with the approximation becoming exact if the approximate value function does not change. Exercise 12.8 Prove that (12.24) becomes exact if the value function does not change. To save writing, consider the case of t = 0, and use the notation Vk .= ˆv(Sk,w). ⇤ Exercise 12.9 The truncated version of the general o↵-policy return is denoted Gλs t:h.


ˆv(Sk,w). ⇤ Exercise 12.9 The truncated version of the general o↵-policy return is denoted Gλs t:h. Guess the correct equation, based on (12.24). ⇤ The above form of the λ-return (12.24) is convenient to use in a forward-view update, wt+1 = wt + ↵ # Gλs t −ˆv(St,wt) $ rˆv(St,wt) ⇡wt + ↵⇢t 1 X k=t δs k k Y i=t+1 γiλi⇢i ! rˆv(St,wt), which to the experienced eye looks like an eligibility-based TD update—the product is like an eligibility trace and it is multiplied by TD errors. But this is just one time step of a forward view. The relationship that we are looking for is that the forward-view update, summed over time, is approximately equal to a backward-view update, summed over time (this relationship is only approximate because again we ignore changes in the value