310 Chapter 12: Eligibility Traces function). The sum of the forward-view update over time is 1 X t=0 (wt+1 −wt) ⇡ 1 X t=0 1 X k=t ↵⇢tδs krˆv(St,wt) k Y i=t+1 γiλi⇢i = 1 X k=0 k X t=0 ↵⇢trˆv(St,wt)δs k k Y i=t+1 γiλi⇢i (using the summation rule: Py t=x Py k=t = Py k=x Pk t=x) = 1 X k=0 ↵δs k k X t=0 ⇢trˆv(St,wt) k Y i=t+1 γiλi⇢i, which would be in the form of the sum of a backward-view TD update if the entire expression from the second sum on could be written and updated incrementally as an eligibility trace, which we now show can be done. That is, we show that if this expression was the trace at time k, then we could update it from its value at time k −1 by: zk = k X t=0 ⇢trˆv(St,wt) k Y i=t+1 γiλi⇢i = k−1 X t=0 ⇢trˆv(St,wt) k Y i=t+1 γiλi⇢i + ⇢krˆv(Sk,wk) = γkλk⇢k k−1 X t=0 ⇢trˆv(St,wt)


t+1 γiλi⇢i = k−1 X t=0 ⇢trˆv(St,wt) k Y i=t+1 γiλi⇢i + ⇢krˆv(Sk,wk) = γkλk⇢k k−1 X t=0 ⇢trˆv(St,wt) k−1 Y i=t+1 γiλi⇢i | {z } zk−1 + ⇢krˆv(Sk,wk) = ⇢k # γkλkzk−1 + rˆv(Sk,wk) $ , which, changing the index from k to t, is the general accumulating trace update for state values: zt .= ⇢t # γtλtzt−1 + rˆv(St,wt) $ , (12.25) This eligibility trace, together with the usual semi-gradient parameter-update rule for TD(λ) (12.7), forms a general TD(λ) algorithm that can be applied to either on-policy or o↵-policy data. In the on-policy case, the algorithm is exactly TD(λ) because ⇢t is alway 1 and (12.25) becomes the usual accumulating trace (12.5) (extended to variable λ and γ). In the o↵-policy case, the algorithm often works well but, as a semi-gradient method, is not guaranteed to be stable.


ase, the algorithm often works well but, as a semi-gradient method, is not guaranteed to be stable. In the next few sections we will consider extensions of it that do guarantee stability. A very similar series of steps can be followed to derive the o↵-policy eligibility traces for action-value methods and corresponding general Sarsa(λ) algorithms. One could start with either recursive form for the general action-based λ-return, (12.19) or (12.20), but the latter (the Expected Sarsa form) works out to be simpler. We extend (12.20) to the


12.9. O↵-policy Traces with Control Variates 311 o↵-policy case after the model of (7.14) to produce Gλa t .= Rt+1+γt+1 ⇣ (1−λt+1) ¯Vt(St+1)+λt+1 ⇥ ⇢t+1Gλa t+1+ ¯Vt(St+1)−⇢t+1ˆq(St+1, At+1, wt) ⇤⌘ = Rt+1 + γt+1 ⇣ ¯Vt(St+1) + λt+1⇢t+1 ⇥ Gλa t+1 −ˆq(St+1, At+1, wt) ⇤⌘ (12.26) where ¯Vt(St+1) is as given by (12.21). Again the λ-return can be written approximately as the sum of TD errors, Gλa t ⇡ˆq(St, At, wt) + 1 X k=t δa k k Y i=t+1 γiλi⇢i, (12.27) using the expectation form of the action-based TD error: δa t = Rt+1 + γt+1 ¯Vt(St+1) −ˆq(St, At, wt). (12.28) As before, the approximation becomes exact if the approximate value function does not change. Exercise 12.10 Prove that (12.27) becomes exact if the value function does not change.


not change. Exercise 12.10 Prove that (12.27) becomes exact if the value function does not change. To save writing, consider the case of t = 0, and use the notation Qk = ˆq(Sk, Ak, w). Hint: Start by writing out δa 0 and Gλa 0 , then Gλa 0 −Q0. ⇤ Exercise 12.11 The truncated version of the general o↵-policy return is denoted Gλa t:h. Guess the correct equation for it, based on (12.27). ⇤ Using steps entirely analogous to those for the state case, one can write a forward-view update based on (12.27), transform the sum of the updates using the summation rule, and ﬁnally derive the following form for the eligibility trace for action values: zt .= γtλt⇢tzt−1 + rˆq(St, At, wt). (12.29) This eligibility trace, together with the expectation-based TD error (12.28) and the usual semi-gradient para


ibility trace, together with the expectation-based TD error (12.28) and the usual semi-gradient parameter-update rule (12.7), forms an elegant, eﬃcient Expected Sarsa(λ) algorithm that can be applied to either on-policy or o↵-policy data. It is probably the best algorithm of this type at the current time (though of course it is not guaranteed to be stable until combined in some way with one of the methods presented in the following sections). In the on-policy case with constant λ and γ, and the usual state–action TD error (12.16), the algorithm would be identical to the Sarsa(λ) algorithm presented in Section 12.7. Exercise 12.12 Show in detail the steps outlined above for deriving (12.29) from (12.27).


ion 12.7. Exercise 12.12 Show in detail the steps outlined above for deriving (12.29) from (12.27). Start with the update (12.15), substitute Gλa t from (12.26) for Gλ t , then follow similar steps as led to (12.25). ⇤ At λ = 1, these algorithms become closely related to corresponding Monte Carlo algorithms. One might expect that an exact equivalence would hold for episodic problems and o↵-line updating, but in fact the relationship is subtler and slightly weaker than that. Under these most favorable conditions still there is not an episode by episode equivalence of updates, only of their expectations. This should not be surprising as these methods


312 Chapter 12: Eligibility Traces make irrevocable updates as a trajectory unfolds, whereas true Monte Carlo methods would make no update for a trajectory if any action within it has zero probability under the target policy. In particular, all of these methods, even at λ = 1, still bootstrap in the sense that their targets depend on the current value estimates—it’s just that the dependence cancels out in expected value. Whether this is a good or bad property in practice is another question. Recently, methods have been proposed that do achieve an exact equivalence (Sutton, Mahmood, Precup and van Hasselt, 2014). These methods require an additional vector of “provisional weights” that keep track of updates which have been made but may need to be retracted (or emphasized) depending on the ac


ack of updates which have been made but may need to be retracted (or emphasized) depending on the actions taken later. The state and state–action versions of these methods are called PTD(λ) and PQ(λ) respectively, where the ‘P’ stands for Provisional. The practical consequences of all these new o↵-policy methods have not yet been established. Undoubtedly, issues of high variance will arise as they do in all o↵-policy methods using importance sampling (Section 11.9). If λ < 1, then all these o↵-policy algorithms involve bootstrapping and the deadly triad applies (Section 11.3), meaning that they can be guaranteed stable only for the tabular case, for state aggregation, and for other limited forms of function approximation.


for the tabular case, for state aggregation, and for other limited forms of function approximation. For linear and more-general forms of function approximation the parameter vector may diverge to inﬁnity as in the examples in Chapter 11. As we discussed there, the challenge of o↵-policy learning has two parts. O↵-policy eligibility traces deal e↵ectively with the ﬁrst part of the challenge, correcting for the expected value of the targets, but not at all with the second part of the challenge, having to do with the distribution of updates. Algorithmic strategies for meeting the second part of the challenge of o↵-policy learning with eligibility traces are summarized in Section 12.11. Exercise 12.13 What are the dutch-trace and replacing-trace versions of o↵-policy eligibility traces for sta


12.13 What are the dutch-trace and replacing-trace versions of o↵-policy eligibility traces for state-value and action-value methods? ⇤ 12.10 Watkins’s Q(λ) to Tree-Backup(λ) Several methods have been proposed over the years to extend Q-learning to eligibility traces. The original is Watkins’s Q(λ), which decays its eligibility traces in the usual way as long as a greedy action was taken, then cuts the traces to zero after the ﬁrst non-greedy action. The backup diagram for Watkins’s Q(λ) is shown in Figure 12.12. In Chapter 6, we uniﬁed Q-learning and Expected Sarsa in the o↵-policy version of the latter, which includes Q-learning as a special case, and generalizes it to arbitrary target policies, and in the previous section of this chapter we completed our treatment of Expected Sarsa by


licies, and in the previous section of this chapter we completed our treatment of Expected Sarsa by generalizing it to o↵-policy eligibility traces. In Chapter 7, however, we distinguished n-step Expected Sarsa from n-step Tree Backup, where the latter retained the property of not using importance sampling. It remains then to present the eligibility trace version of Tree Backup, which we will call Tree-Backup(λ), or TB(λ) for short. This is arguably the true successor to Q-learning because it retains its appealing absence of importance sampling even though it can be applied to o↵-policy data.


12.10. Watkins’s Q(λ) to Tree-Backup(λ) 313 1 −λ (1 −λ)λ (1 −λ)λ2 λT −t−1 · · · St At At+1 St+1 Rt+1 ST RT · · · St+2 Rt+2 At+2 X = 1 OR · · · · · · St+n Rt+n First non-greedy action λn−1 Watkins’s Q(λ) Figure 12.12: The backup diagram for Watkins’s Q(λ). The series of component updates ends either with the end of the episode or with the ﬁrst nongreedy action, whichever comes ﬁrst. The concept of TB(λ) is straightforward. As shown in its backup diagram in Figure 12.13, the tree-backup updates of each length (from Section 7.5) are weighted in the usual way dependent on the bootstrapping parameter λ. To get the detailed equations, with the right indices on the general bootstrapping and discounting parameters, it is best to start with a recursive form (12.20) for the λ-return using action val


ting parameters, it is best to start with a recursive form (12.20) for the λ-return using action values, and then expand the bootstrapping case of the target after the model of (7.16): Gλa t .= Rt+1+γt+1 ✓ (1−λt+1) ¯Vt(St+1)+λt+1 h X a6=At+1 ⇡(a|St+1)ˆq(St+1, a, wt)+⇡(At+1|St+1)Gλa t+1 i◆ = Rt+1 + γt+1 ✓ ¯Vt(St+1) + λt+1⇡(At+1|St+1) ⇣ Gλa t+1 −ˆq(St+1, At+1, wt) ⌘◆ As per the usual pattern, it can also be written approximately (ignoring changes in the approximate value function) as a sum of TD errors, Gλa t ⇡ˆq(St, At, wt) + 1 X k=t δa k k Y i=t+1 γiλi⇡(Ai|Si), using the expectation form of the action-based TD error (12.28). Following the same steps as in the previous section, we arrive at a special eligibility trace update involving the target-policy probabilities of the selected actions,


special eligibility trace update involving the target-policy probabilities of the selected actions, zt .= γtλt⇡(At|St)zt−1 + rˆq(St, At, wt).


314 Chapter 12: Eligibility Traces 1 −λ (1 −λ)λ (1 −λ)λ2 λT −t−1 · · · St At At+1 AT −1 St+1 Rt+1 ST RT · · · St+2 Rt+2 At+2 X = 1 · · · Tree Backup(λ) ST −1 Figure 12.13: The backup diagram for the λ version of the Tree Backup algorithm. This, together with the usual parameter-update rule (12.7), deﬁnes the TB(λ) algorithm. Like all semi-gradient algorithms, TB(λ) is not guaranteed to be stable when used with o↵-policy data and with a powerful function approximator. To obtain those assurances, TB(λ) would have to be combined with one of the methods presented in the next section. ⇤Exercise 12.14 How might Double Expected Sarsa be extended to eligibility traces? ⇤ 12.11 Stable O↵-policy Methods with Traces Several methods using eligibility traces have been proposed that achieve guarantees o


ds with Traces Several methods using eligibility traces have been proposed that achieve guarantees of stability under o↵-policy training, and here we present four of the most important using this book’s standard notation, including general bootstrapping and discounting functions. All are based on either the Gradient-TD or the Emphatic-TD ideas presented in Sections 11.7 and 11.8. All the algorithms assume linear function approximation, though extensions to nonlinear function approximation can also be found in the literature. GTD(λ) is the eligibility-trace algorithm analogous to TDC, the better of the two state-value Gradient-TD prediction algorithms discussed in Section 11.7. Its goal is to learn a parameter wt such that ˆv(s,w) .= w> t x(s) ⇡v⇡(s), even from data that is due to following


learn a parameter wt such that ˆv(s,w) .= w> t x(s) ⇡v⇡(s), even from data that is due to following another policy b. Its update is wt+1 .= wt + ↵δs t zt −↵γt+1(1 −λt+1) # z> t vt $ xt+1,


12.11. Stable O↵-policy Methods with Traces 315 with δs t , zt, and ⇢t deﬁned in the usual ways for state values (12.23) (12.25) (11.1), and vt+1 .= vt + βδs t zt −β # v> t xt $ xt, (12.30) where, as in Section 11.7, v 2 Rd is a vector of the same dimension as w, initialized to v0 = 0, and β > 0 is a second step-size parameter. GQ(λ) is the Gradient-TD algorithm for action values with eligibility traces. Its goal is to learn a parameter wt such that ˆq(s, a, wt) .= w> t x(s, a) ⇡q⇡(s, a) from o↵-policy data. If the target policy is "-greedy, or otherwise biased toward the greedy policy for ˆq, then GQ(λ) can be used as a control algorithm. Its update is wt+1 .= wt + ↵δa t zt −↵γt+1(1 −λt+1) # z> t vt $ ¯xt+1, where ¯xt is the average feature vector for St under the target policy, ¯xt .= X


# z> t vt $ ¯xt+1, where ¯xt is the average feature vector for St under the target policy, ¯xt .= X a ⇡(a|St)x(St, a), δa t is the expectation form of the TD error, which can be written δa t .= Rt+1 + γt+1w> t ¯xt+1 −w> t xt, zt is deﬁned in the usual way for action values (12.29), and the rest is as in GTD(λ), including the update for vt (12.30). HTD(λ) is a hybrid state-value algorithm combining aspects of GTD(λ) and TD(λ). Its most appealing feature is that it is a strict generalization of TD(λ) to o↵-policy learning, meaning that if the behavior policy happens to be the same as the target policy, then HTD(λ) becomes the same as TD(λ), which is not true for GTD(λ). This is appealing because TD(λ) is often faster than GTD(λ) when both algorithms converge, and TD(λ) requires setting only


se TD(λ) is often faster than GTD(λ) when both algorithms converge, and TD(λ) requires setting only a single step size. HTD(λ) is deﬁned by wt+1 .= wt + ↵δs t zt + ↵ # (zt −zb t)>vt $ (xt −γt+1xt+1), vt+1 .= vt + βδs t zt −β ⇣ zb t >vt ⌘ (xt −γt+1xt+1), with v0 .= 0, zt .= ⇢t # γtλtzt−1 + xt $ , with z−1 .= 0, zb t .= γtλtzb t−1 + xt, with zb −1 .= 0, where β > 0 again is a second step-size parameter. In addition to the second set of weights, vt, HTD(λ) also has a second set of eligibility traces, zb t. These are conventional accumulating eligibility traces for the behavior policy and become equal to zt if all the ⇢t are 1, which causes the last term in the wt update to be zero and the overall update to reduce to TD(λ).


, which causes the last term in the wt update to be zero and the overall update to reduce to TD(λ). Emphatic TD(λ) is the extension of the one-step Emphatic-TD algorithm (Sections 9.11 and 11.8) to eligibility traces. The resultant algorithm retains strong o↵-policy convergence guarantees while enabling any degree of bootstrapping, albeit at the cost of


316 Chapter 12: Eligibility Traces high variance and potentially slow convergence. Emphatic TD(λ) is deﬁned by wt+1 .= wt + ↵δtzt δt .= Rt+1 + γt+1w> t xt+1 −w> t xt zt .= ⇢t # γtλtzt−1 + Mtxt $ , with z−1 .= 0, Mt .= λtIt + (1 −λt)Ft Ft .= ⇢t−1γtFt−1 + It, with F0 .= i(S0), where Mt ≥0 is the general form of emphasis, Ft ≥0 is termed the followon trace, and It ≥0 is the interest, as described in Section 11.8. Note that Mt, like δt, is not really an additional memory variable. It can be removed from the algorithm by substituting its deﬁnition into the eligibility-trace equation. Pseudocode and software for the true online version of Emphatic-TD(λ) are available on the web (Sutton, 2015b). In the on-policy case (⇢t = 1, for all t), Emphatic-TD(λ) is similar to conventional TD(λ), but still


the on-policy case (⇢t = 1, for all t), Emphatic-TD(λ) is similar to conventional TD(λ), but still signiﬁcantly di↵erent. In fact, whereas Emphatic-TD(λ) is guaranteed to converge for all state-dependent λ functions, TD(λ) is not. TD(λ) is guaranteed convergent only for all constant λ. See Yu’s counterexample (Ghiassian, Raﬁee, and Sutton, 2016). 12.12 Implementation Issues It might at ﬁrst appear that tabular methods using eligibility traces are much more complex than one-step methods. A naive implementation would require every state (or state–action pair) to update both its value estimate and its eligibility trace on every time step. This would not be a problem for implementations on single-instruction, multipledata, parallel computers or in plausible artiﬁcial neural network (ANN) impl


le-instruction, multipledata, parallel computers or in plausible artiﬁcial neural network (ANN) implementations, but it is a problem for implementations on conventional serial computers. Fortunately, for typical values of λ and γ the eligibility traces of almost all states are almost always nearly zero; only those states that have recently been visited will have traces signiﬁcantly greater than zero and only these few states need to be updated to closely approximate these algorithms. In practice, then, implementations on conventional computers may keep track of and update only the few traces that are signiﬁcantly greater than zero. Using this trick, the computational expense of using traces in tabular methods is typically just a few times that of a one-step method.


expense of using traces in tabular methods is typically just a few times that of a one-step method. The exact multiple of course depends on λ and γ and on the expense of the other computations. Note that the tabular case is in some sense the worst case for the computational complexity of eligibility traces. When function approximation is used, the computational advantages of not using traces generally decrease. For example, if ANNs and backpropagation are used, then eligibility traces generally cause only a doubling of the required memory and computation per step. Truncated λ-return methods (Section 12.3) can be computationally eﬃcient on conventional computers though they always require some additional memory.


12.13. Conclusions 317 12.13 Conclusions Eligibility traces in conjunction with TD errors provide an eﬃcient, incremental way of shifting and choosing between Monte Carlo and TD methods. The n-step methods of Chapter 7 also enabled this, but eligibility trace methods are more general, often faster to learn, and o↵er di↵erent computational complexity tradeo↵s. This chapter has o↵ered an introduction to the elegant, emerging theoretical understanding of eligibility traces for onand o↵-policy learning and for variable bootstrapping and discounting. One aspect of this elegant theory is true online methods, which exactly reproduce the behavior of expensive ideal methods while retaining the computational congeniality of conventional TD methods.


expensive ideal methods while retaining the computational congeniality of conventional TD methods. Another aspect is the possibility of derivations that automatically convert from intuitive forward-view methods to more eﬃcient incremental backward-view algorithms. We illustrated this general idea in a derivation that started with a classical, expensive Monte Carlo algorithm and ended with a cheap incremental non-TD implementation using the same novel eligibility trace used in true online TD methods. As we mentioned in Chapter 5, Monte Carlo methods may have advantages in nonMarkov tasks because they do not bootstrap. Because eligibility traces make TD methods more like Monte Carlo methods, they also can have advantages in these cases.


traces make TD methods more like Monte Carlo methods, they also can have advantages in these cases. If one wants to use TD methods because of their other advantages, but the task is at least partially non-Markov, then the use of an eligibility trace method is indicated. Eligibility traces are the ﬁrst line of defense against both long-delayed rewards and non-Markov tasks. By adjusting λ, we can place eligibility trace methods anywhere along a continuum from Monte Carlo to one-step TD methods. Where shall we place them? We do not yet have a good theoretical answer to this question, but a clear empirical answer appears to be emerging. On tasks with many steps per episode, or many steps within the half-life of discounting, it appears signiﬁcantly better to use eligibility traces than not to (


the half-life of discounting, it appears signiﬁcantly better to use eligibility traces than not to (e.g., see Figure 12.14). On the other hand, if the traces are so long as to produce a pure Monte Carlo method, or nearly so, then performance degrades sharply. An intermediate mixture appears to be the best choice. Eligibility traces should be used to bring us toward Monte Carlo methods, but not all the way there. In the future it may be possible to more ﬁnely vary the trade-o↵between TD and Monte Carlo methods by using variable λ, but at present it is not clear how this can be done reliably and usefully. Methods using eligibility traces require more computation than one-step methods, but in return they o↵er signiﬁcantly faster learning, particularly when rewards are delayed by many steps.


return they o↵er signiﬁcantly faster learning, particularly when rewards are delayed by many steps. Thus it often makes sense to use eligibility traces when data are scarce and cannot be repeatedly processed, as is often the case in online applications. On the other hand, in o↵-line applications in which data can be generated cheaply, perhaps from an inexpensive simulation, then it often does not pay to use eligibility traces. In these cases the objective is not to get more out of a limited amount of data, but simply to process as much data as possible as quickly as possible. In these cases the speedup per datum due to traces is typically not worth their computational cost, and one-step methods are favored.


318 Chapter 12: Eligibility Traces accumulating traces 0.2 0.3 0.4 0.5 0 0.2 0.4 0.6 0.8 1 ! RANDOM WALK 50 100 150 200 250 300 Failures per 100,000 steps 0 0.2 0.4 0.6 0.8 1 ! CART AND POLE 400 450 500 550 600 650 700 Steps per episode 0 0.2 0.4 0.6 0.8 1 ! MOUNTAIN CAR replacing traces 150 160 170 180 190 200 210 220 230 240 Cost per episode 0 0.2 0.4 0.6 0.8 1 ! PUDDLE WORLD replacing traces accumulating traces replacing traces accumulating traces RMS error Figure 12.14: The e↵ect of λ on reinforcement learning performance in four di↵erent test problems. In all cases, performance is generally best (a lower number in the graph) at an intermediate value of λ. The two left panels are applications to simple continuous-state control tasks using the Sarsa(λ) algorithm and tile coding, with ei


tions to simple continuous-state control tasks using the Sarsa(λ) algorithm and tile coding, with either replacing or accumulating traces (Sutton, 1996). The upper-right panel is for policy evaluation on a random walk task using TD(λ) (Singh and Sutton, 1996). The lower right panel is unpublished data for the pole-balancing task (Example 3.4) from an earlier study (Sutton, 1984).


12.13. Conclusions 319 Bibliographical and Historical Remarks Eligibility traces came into reinforcement learning via the fecund ideas of Klopf (1972). Our use of eligibility traces is based on Klopf’s work (Sutton, 1978a, 1978b, 1978c; Barto and Sutton, 1981a, 1981b; Sutton and Barto, 1981a; Barto, Sutton, and Anderson, 1983; Sutton, 1984). We may have been the ﬁrst to use the term “eligibility trace” (Sutton and Barto, 1981a). The idea that stimuli produce after e↵ects in the nervous system that are important for learning is very old (see Chapter 14). Some of the earliest uses of eligibility traces were in the actor–critic methods discussed in Chapter 13 (Barto, Sutton, and Anderson, 1983; Sutton, 1984).


the actor–critic methods discussed in Chapter 13 (Barto, Sutton, and Anderson, 1983; Sutton, 1984). 12.1 Compound updates were called “complex backups” in the ﬁrst edition of this book. The λ-return and its error-reduction properties were introduced by Watkins (1989) and further developed by Jaakkola, Jordan, and Singh (1994). The random walk results in this and subsequent sections are new to this text, as are the terms “forward view” and “backward view.” The notion of a λ-return algorithm was introduced in the ﬁrst edition of this text. The more reﬁned treatment presented here was developed in conjunction with Harm van Seijen (e.g., van Seijen and Sutton, 2014). 12.2 TD(λ) with accumulating traces was introduced by Sutton (1988, 1984).


eijen and Sutton, 2014). 12.2 TD(λ) with accumulating traces was introduced by Sutton (1988, 1984). Convergence in the mean was proved by Dayan (1992), and with probability 1 by many researchers, including Peng (1993), Dayan and Sejnowski (1994), Tsitsiklis (1994), and Gurvits, Lin, and Hanson (1994). The bound on the error of the asymptotic λ-dependent solution of linear TD(λ) is due to Tsitsiklis and Van Roy (1997). 12.3 Truncated TD methods were developed by Cichosz (1995) and van Seijen (2016). 12.4 The idea of redoing updates was extensively developed by van Seijen, originally under the name “best-match learning” (van Seijen, 2011; van Seijen, Whiteson, van Hasselt, and Weiring, 2011). 12.5 True online TD(λ) is primarily due to Harm van Seijen (van Seijen and Sutton, 2014; van Seijen


12.5 True online TD(λ) is primarily due to Harm van Seijen (van Seijen and Sutton, 2014; van Seijen et al., 2016) though some of its key ideas were discovered independently by Hado van Hasselt (personal communication). The name “dutch traces” is in recognition of the contributions of both scientists. Replacing traces are due to Singh and Sutton (1996). 12.6 The material in this section is from van Hasselt and Sutton (2015). 12.7 Sarsa(λ) with accumulating traces was ﬁrst explored as a control method by Rummery and Niranjan (1994; Rummery, 1995). True Online Sarsa(λ) was


320 Chapter 12: Eligibility Traces introduced by van Seijen and Sutton (2014). The algorithm on page 307 was adapted from van Seijen et al. (2016). The Mountain Car results were made for this text, except for Figure 12.11 which is adapted from van Seijen and Sutton (2014). 12.8 Perhaps the ﬁrst published discussion of variable λ was by Watkins (1989), who pointed out that the cutting o↵of the update sequence (Figure 12.12) in his Q(λ) when a nongreedy action was selected could be implemented by temporarily setting λ to 0. Variable λ was introduced in the ﬁrst edition of this text. The roots of variable γ are in the work on options (Sutton, Precup, and Singh, 1999) and its precursors (Sutton, 1995a), becoming explicit in the GQ(λ) paper (Maei and Sutton, 2010), which also introduced some of


1995a), becoming explicit in the GQ(λ) paper (Maei and Sutton, 2010), which also introduced some of these recursive forms for the λ-returns. A di↵erent notion of variable λ has been developed by Yu (2012). 12.9 O↵-policy eligibility traces were introduced by Precup et al. (2000, 2001), then further developed by Bertsekas and Yu (2009), Maei (2011; Maei and Sutton, 2010), Yu (2012), and by Sutton, Mahmood, Precup, and van Hasselt (2014). The last reference in particular gives a powerful forward view for o↵-policy TD methods with general state-dependent λ and γ. The presentation here seems to be new. This section ends with an elegant Expected Sarsa(λ) algorithm. Although it is a natural algorithm, to our knowledge it has not previously been described or tested in the literature.


tural algorithm, to our knowledge it has not previously been described or tested in the literature. 12.10 Watkins’s Q(λ) is due to Watkins (1989). The tabular, episodic, o↵-line version has been proven convergent by Munos, Stepleton, Harutyunyan, and Bellemare (2016). Alternative Q(λ) algorithms were proposed by Peng and Williams (1994, 1996) and by Sutton, Mahmood, Precup, and van Hasselt (2014). Tree Backup(λ) is due to Precup, Sutton, and Singh (2000). 12.11 GTD(λ) is due to Maei (2011). GQ(λ) is due to Maei and Sutton (2010). HTD(λ) is due to White and White (2016) based on the one-step HTD algorithm introduced by Hackman (2012). The latest developments in the theory of Gradient-TD methods are by Yu (2017).


d by Hackman (2012). The latest developments in the theory of Gradient-TD methods are by Yu (2017). Emphatic TD(λ) was introduced by Sutton, Mahmood, and White (2016), who proved its stability. Yu (2015, 2016) proved its convergence, and the algorithm was developed further by Hallak et al. (2015, 2016).


Chapter 13 Policy Gradient Methods In this chapter we consider something new. So far in this book almost all the methods have been action-value methods; they learned the values of actions and then selected actions based on their estimated action values1; their policies would not even exist without the action-value estimates. In this chapter we consider methods that instead learn a parameterized policy that can select actions without consulting a value function. A value function may still be used to learn the policy parameter, but is not required for action selection. We use the notation ✓2 Rd0 for the policy’s parameter vector. Thus we write ⇡(a|s, ✓) = Pr{At =a | St =s, ✓t =✓} for the probability that action a is taken at time t given that the environment is in state s at time t with para


bility that action a is taken at time t given that the environment is in state s at time t with parameter ✓. If a method uses a learned value function as well, then the value function’s weight vector is denoted w 2 Rd as usual, as in ˆv(s,w). In this chapter we consider methods for learning the policy parameter based on the gradient of some scalar performance measure J(✓) with respect to the policy parameter. These methods seek to maximize performance, so their updates approximate gradient ascent in J: ✓t+1 = ✓t + ↵\ rJ(✓t), (13.1) where \ rJ(✓t) 2 Rd0 is a stochastic estimate whose expectation approximates the gradient of the performance measure with respect to its argument ✓t. All methods that follow this general schema we call policy gradient methods, whether or not they also learn an a


that follow this general schema we call policy gradient methods, whether or not they also learn an approximate value function. Methods that learn approximations to both policy and value functions are often called actor–critic methods, where ‘actor’ is a reference to the learned policy, and ‘critic’ refers to the learned value function, usually a state-value function. First we treat the episodic case, in which performance is deﬁned as the value of the start state under the parameterized policy, before going on to consider the continuing case, in 1The lone exception is the gradient bandit algorithms of Section 2.8. In fact, that section goes through many of the same steps, in the single-state bandit case, as we go through here for full MDPs.


rough many of the same steps, in the single-state bandit case, as we go through here for full MDPs. Reviewing that section would be good preparation for fully understanding this chapter.


322 Chapter 13: Policy Gradient Methods which performance is deﬁned as the average reward rate, as in Section 10.3. In the end, we are able to express the algorithms for both cases in very similar terms. 13.1 Policy Approximation and its Advantages In policy gradient methods, the policy can be parameterized in any way, as long as ⇡(a|s, ✓) is di↵erentiable with respect to its parameters, that is, as long as r⇡(a|s, ✓) (the column vector of partial derivatives of ⇡(a|s, ✓) with respect to the components of ✓) exists and is ﬁnite for all s 2 S, a 2 A(s), and ✓2 Rd0. In practice, to ensure exploration we generally require that the policy never becomes deterministic (i.e., that ⇡(a|s, ✓) 2 (0, 1), for all s, a, ✓).


quire that the policy never becomes deterministic (i.e., that ⇡(a|s, ✓) 2 (0, 1), for all s, a, ✓). In this section we introduce the most common parameterization for discrete action spaces and point out the advantages it o↵ers over action-value methods. Policy-based methods also o↵er useful ways of dealing with continuous action spaces, as we describe later in Section 13.7. If the action space is discrete and not too large, then a natural and common kind of parameterization is to form parameterized numerical preferences h(s, a, ✓) 2 R for each state–action pair. The actions with the highest preferences in each state are given the highest probabilities of being selected, for example, according to an exponential soft-max distribution: ⇡(a|s, ✓) .= eh(s,a,✓) P b eh(s,b,✓) , (13.2) where e ⇡2.


g to an exponential soft-max distribution: ⇡(a|s, ✓) .= eh(s,a,✓) P b eh(s,b,✓) , (13.2) where e ⇡2.71828 is the base of the natural logarithm. Note that the denominator here is just what is required so that the action probabilities in each state sum to one. We call this kind of policy parameterization soft-max in action preferences. The action preferences themselves can be parameterized arbitrarily. For example, they might be computed by a deep artiﬁcial neural network (ANN), where ✓is the vector of all the connection weights of the network (as in the AlphaGo system described in Section 16.6). Or the preferences could simply be linear in features, h(s, a, ✓) = ✓>x(s, a), (13.3) using feature vectors x(s, a) 2 Rd0 constructed by any of the methods described in Section 9.5.


.3) using feature vectors x(s, a) 2 Rd0 constructed by any of the methods described in Section 9.5. One advantage of parameterizing policies according to the soft-max in action preferences is that the approximate policy can approach a deterministic policy, whereas with "-greedy action selection over action values there is always an " probability of selecting a random action. Of course, one could select according to a soft-max distribution based on action values, but this alone would not allow the policy to approach a deterministic policy. Instead, the action-value estimates would converge to their corresponding true values, which would di↵er by a ﬁnite amount, translating to speciﬁc probabilities other than 0 and 1.


lues, which would di↵er by a ﬁnite amount, translating to speciﬁc probabilities other than 0 and 1. If the soft-max distribution included a temperature parameter, then the temperature could be reduced over time to approach determinism, but in practice it would be diﬃcult to choose the reduction schedule, or even the initial temperature, without more prior knowledge of the true action values than we would like to assume. Action preferences


13.1. Policy Approximation and its Advantages 323 are di↵erent because they do not approach speciﬁc values; instead they are driven to produce the optimal stochastic policy. If the optimal policy is deterministic, then the preferences of the optimal actions will be driven inﬁnitely higher than all suboptimal actions (if permitted by the parameterization). A second advantage of parameterizing policies according to the soft-max in action preferences is that it enables the selection of actions with arbitrary probabilities. In problems with signiﬁcant function approximation, the best approximate policy may be stochastic. For example, in card games with imperfect information the optimal play is often to do two di↵erent things with speciﬁc probabilities, such as when bluﬃng in Poker.


l play is often to do two di↵erent things with speciﬁc probabilities, such as when bluﬃng in Poker. Action-value methods have no natural way of ﬁnding stochastic optimal policies, whereas policy approximating methods can, as shown in Example 13.1. Example 13.1 Short corridor with switched actions Consider the small corridor gridworld shown inset in the graph below. The reward is −1 per step, as usual. In each of the three nonterminal states there are only two actions, right and left. These actions have their usual consequences in the ﬁrst and third states (left causes no movement in the ﬁrst state), but in the second state they are reversed, so that right moves to the left and left moves to the right.


in the second state they are reversed, so that right moves to the left and left moves to the right. The problem is diﬃcult because all the states appear identical under the function approximation. In particular, we deﬁne x(s, right) = [1, 0]> and x(s, left) = [0, 1]>, for all s. An action-value method with "-greedy action selection is forced to choose between just two policies: choosing right with high probability 1 −"/2 on all steps or choosing left with the same high probability on all time steps. If " = 0.1, then these two policies achieve a value (at the start state) of less than −44 and −82, respectively, as shown in the graph. A method can do signiﬁcantly better if it can learn a speciﬁc probability with which to select right.


method can do signiﬁcantly better if it can learn a speciﬁc probability with which to select right. The best probability is about 0.59, which achieves a value of about −11.6. probability of right action -11.6 0.1 0.2 -20 -40 -60 -80 -100 0.3 0.4 0 0.6 0.7 0.8 0.9 0.5 1 휀-greedy left 휀-greedy right optimal stochastic policy J(✓) = v⇡✓(S) G S


324 Chapter 13: Policy Gradient Methods Perhaps the simplest advantage that policy parameterization may have over actionvalue parameterization is that the policy may be a simpler function to approximate. Problems vary in the complexity of their policies and action-value functions. For some, the action-value function is simpler and thus easier to approximate. For others, the policy is simpler. In the latter case a policy-based method will typically learn faster and yield a superior asymptotic policy (as in Tetris; see S¸im¸sek, Alg´orta, and Kothiyal, 2016). Finally, we note that the choice of policy parameterization is sometimes a good way of injecting prior knowledge about the desired form of the policy into the reinforcement learning system.


ecting prior knowledge about the desired form of the policy into the reinforcement learning system. This is often the most important reason for using a policy-based learning method. Exercise 13.1 Use your knowledge of the gridworld and its dynamics to determine an exact symbolic expression for the optimal probability of selecting the right action in Example 13.1. ⇤ 13.2 The Policy Gradient Theorem In addition to the practical advantages of policy parameterization over "-greedy action selection, there is also an important theoretical advantage. With continuous policy parameterization the action probabilities change smoothly as a function of the learned parameter, whereas in "-greedy selection the action probabilities may change dramatically for an arbitrarily small change in the estimated a


the action probabilities may change dramatically for an arbitrarily small change in the estimated action values, if that change results in a di↵erent action having the maximal value. Largely because of this, stronger convergence guarantees are available for policy-gradient methods than for action-value methods. In particular, it is the continuity of the policy dependence on the parameters that enables policy-gradient methods to approximate gradient ascent (13.1). The episodic and continuing cases deﬁne the performance measure, J(✓), di↵erently and thus have to be treated separately to some extent. Nevertheless, we will try to present both cases uniformly, and we develop a notation so that the major theoretical results can be described with a single set of equations.


p a notation so that the major theoretical results can be described with a single set of equations. In this section we treat the episodic case, for which we deﬁne the performance measure as the value of the start state of the episode. We can simplify the notation without losing any meaningful generality by assuming that every episode starts in some particular (non-random) state s0. Then, in the episodic case we deﬁne performance as J(✓) .= v⇡✓(s0), (13.4) where v⇡✓is the true value function for ⇡✓, the policy determined by ✓. From here on in our discussion we will assume no discounting (γ = 1) for the episodic case, although for completeness we do include the possibility of discounting in the boxed algorithms.


se, although for completeness we do include the possibility of discounting in the boxed algorithms. With function approximation it may seem challenging to change the policy parameter in a way that ensures improvement. The problem is that performance depends on both the action selections and the distribution of states in which those selections are made, and that both of these are a↵ected by the policy parameter. Given a state, the e↵ect of the policy parameter on the actions, and thus on reward, can be computed in a relatively straightforward way from knowledge of the parameterization. But the e↵ect of the policy


13.2. The Policy Gradient Theorem 325 Proof of the Policy Gradient Theorem (episodic case) With just elementary calculus and re-arranging of terms, we can prove the policy gradient theorem from ﬁrst principles. To keep the notation simple, we leave it implicit in all cases that ⇡is a function of ✓, and all gradients are also implicitly with respect to ✓. First note that the gradient of the state-value function can be written in terms of the action-value function as rv⇡(s) = r "X a ⇡(a|s)q⇡(s, a) # , for all s 2 S (Exercise 3.18) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s)rq⇡(s, a) i (product rule of calculus) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s)r X s0,r p(s0, r|s, a) # r + v⇡(s0) $i (Exercise 3.19 and Equation 3.2) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s) X s0 p(s0|s, a)rv⇡(s0) i (Eq.


$i (Exercise 3.19 and Equation 3.2) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s) X s0 p(s0|s, a)rv⇡(s0) i (Eq. 3.4) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s) X s0 p(s0|s, a) (unrolling) X a0 ⇥ r⇡(a0|s0)q⇡(s0, a0) + ⇡(a0|s0) X s00 p(s00|s0, a0)rv⇡(s00) ⇤i = X x2S 1 X k=0 Pr(s!x, k, ⇡) X a r⇡(a|x)q⇡(x, a), after repeated unrolling, where Pr(s ! x, k, ⇡) is the probability of transitioning from state s to state x in k steps under policy ⇡. It is then immediate that rJ(✓) = rv⇡(s0) = X s 1 X k=0 Pr(s0 !s, k, ⇡) ! X a r⇡(a|s)q⇡(s, a) = X s ⌘(s) X a r⇡(a|s)q⇡(s, a) (box page 199) = X s0 ⌘(s0) X s ⌘(s) P s0 ⌘(s0) X a r⇡(a|s)q⇡(s, a) = X s0 ⌘(s0) X s µ(s) X a r⇡(a|s)q⇡(s, a) (Eq. 9.3) / X s µ(s) X a r⇡(a|s)q⇡(s, a) (Q.E.D.)


326 Chapter 13: Policy Gradient Methods on the state distribution is a function of the environment and is typically unknown. How can we estimate the performance gradient with respect to the policy parameter when the gradient depends on the unknown e↵ect of policy changes on the state distribution? Fortunately, there is an excellent theoretical answer to this challenge in the form of the policy gradient theorem, which provides an analytic expression for the gradient of performance with respect to the policy parameter (which is what we need to approximate for gradient ascent (13.1)) that does not involve the derivative of the state distribution. The policy gradient theorem for the episodic case establishes that rJ(✓) / X s µ(s) X a q⇡(s, a)r⇡(a|s, ✓), (13.5) where the gradients are column ve


e establishes that rJ(✓) / X s µ(s) X a q⇡(s, a)r⇡(a|s, ✓), (13.5) where the gradients are column vectors of partial derivatives with respect to the components of ✓, and ⇡denotes the policy corresponding to parameter vector ✓. The symbol / here means “proportional to”. In the episodic case, the constant of proportionality is the average length of an episode, and in the continuing case it is 1, so that the relationship is actually an equality. The distribution µ here (as in Chapters 9 and 10) is the on-policy distribution under ⇡(see page 199). The policy gradient theorem is proved for the episodic case in the box on the previous page. 13.3 REINFORCE: Monte Carlo Policy Gradient We are now ready to derive our ﬁrst policy-gradient learning algorithm.


Monte Carlo Policy Gradient We are now ready to derive our ﬁrst policy-gradient learning algorithm. Recall our overall strategy of stochastic gradient ascent (13.1), which requires a way to obtain samples such that the expectation of the sample gradient is proportional to the actual gradient of the performance measure as a function of the parameter. The sample gradients need only be proportional to the gradient because any constant of proportionality can be absorbed into the step size ↵, which is otherwise arbitrary. The policy gradient theorem gives an exact expression proportional to the gradient; all that is needed is some way of sampling whose expectation equals or approximates this expression.


ll that is needed is some way of sampling whose expectation equals or approximates this expression. Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy ⇡; if ⇡is followed, then states will be encountered in these proportions. Thus rJ(✓) / X s µ(s) X a q⇡(s, a)r⇡(a|s, ✓) = E⇡ "X a q⇡(St, a)r⇡(a|St, ✓) # . (13.6) We could stop here and instantiate our stochastic gradient-ascent algorithm (13.1) as ✓t+1 .= ✓t + ↵ X a ˆq(St, a, w)r⇡(a|St, ✓), (13.7) where ˆq is some learned approximation to q⇡. This algorithm, which has been called an all-actions method because its update involves all of the actions, is promising and


13.3. REINFORCE: Monte Carlo Policy Gradient 327 deserving of further study, but our current interest is the classical REINFORCE algorithm (Willams, 1992) whose update at time t involves just At, the one action actually taken at time t. We continue our derivation of REINFORCE by introducing At in the same way as we introduced St in (13.6)—by replacing a sum over the random variable’s possible values by an expectation under ⇡, and then sampling the expectation. Equation (13.6) involves an appropriate sum over actions, but each term is not weighted by ⇡(a|St, ✓) as is needed for an expectation under ⇡. So we introduce such a weighting, without changing the equality, by multiplying and then dividing the summed terms by ⇡(a|St, ✓).


ng, without changing the equality, by multiplying and then dividing the summed terms by ⇡(a|St, ✓). Continuing from (13.6), we have rJ(✓) / E⇡ "X a ⇡(a|St, ✓)q⇡(St, a)r⇡(a|St, ✓) ⇡(a|St, ✓) # = E⇡  q⇡(St, At)r⇡(At|St, ✓) ⇡(At|St, ✓) 3 (replacing a by the sample At ⇠⇡) = E⇡  Gt r⇡(At|St, ✓) ⇡(At|St, ✓) 3 , (because E⇡[Gt|St, At] = q⇡(St, At)) where Gt is the return as usual. The ﬁnal expression in brackets is exactly what is needed, a quantity that can be sampled on each time step whose expectation is proportional to the gradient. Using this sample to instantiate our generic stochastic gradient ascent algorithm (13.1) yields the REINFORCE update: ✓t+1 .= ✓t + ↵Gt r⇡(At|St, ✓t) ⇡(At|St, ✓t) . (13.8) This update has an intuitive appeal.


E update: ✓t+1 .= ✓t + ↵Gt r⇡(At|St, ✓t) ⇡(At|St, ✓t) . (13.8) This update has an intuitive appeal. Each increment is proportional to the product of a return Gt and a vector, the gradient of the probability of taking the action actually taken divided by the probability of taking that action. The vector is the direction in parameter space that most increases the probability of repeating the action At on future visits to state St. The update increases the parameter vector in this direction proportional to the return, and inversely proportional to the action probability. The former makes sense because it causes the parameter to move most in the directions that favor actions that yield the highest return.


uses the parameter to move most in the directions that favor actions that yield the highest return. The latter makes sense because otherwise actions that are selected frequently are at an advantage (the updates will be more often in their direction) and might win out even if they do not yield the highest return. Note that REINFORCE uses the complete return from time t, which includes all future rewards up until the end of the episode. In this sense REINFORCE is a Monte Carlo algorithm and is well deﬁned only for the episodic case with all updates made in retrospect after the episode is completed (like the Monte Carlo algorithms in Chapter 5). This is shown explicitly in the boxed algorithm on the next page.


e Carlo algorithms in Chapter 5). This is shown explicitly in the boxed algorithm on the next page. Notice that the update in the last line of pseudocode appears rather di↵erent from the REINFORCE update rule (13.8). One di↵erence is that the pseudocode uses the compact expression r ln ⇡(At|St, ✓t) for the fractional vector r⇡(At|St,✓t) ⇡(At|St,✓t) in (13.8). That these two expressions for the vector are equivalent follows from the identity r ln x = rx x .


328 Chapter 13: Policy Gradient Methods This vector has been given several names and notations in the literature; we will refer to it simply as the eligibility vector. Note that it is the only place that the policy parameterization appears in the algorithm. REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for ⇡⇤ Input: a di↵erentiable policy parameterization ⇡(a|s, ✓) Algorithm parameter: step size ↵> 0 Initialize policy parameter ✓2 Rd0 (e.g., to 0) Loop forever (for each episode): Generate an episode S0, A0, R1, . . . , ST −1, AT −1, RT , following ⇡(·|·, ✓) Loop for each step of the episode t = 0, 1, . . . , T −1: G PT k=t+1 γk−t−1Rk (Gt) ✓ ✓+ ↵γtGr ln ⇡(At|St, ✓) The second di↵erence between the pseudocode update and the REINFORCE update equation (13.8) is that the former incl


rence between the pseudocode update and the REINFORCE update equation (13.8) is that the former includes a factor of γt. This is because, as mentioned earlier, in the text we are treating the non-discounted case (γ =1) while in the boxed algorithms we are giving the algorithms for the general discounted case. All of the ideas go through in the discounted case with appropriate adjustments (including to the box on page 199) but involve additional complexity that distracts from the main ideas. ⇤Exercise 13.2 Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy gradient theorem (page 325), and the steps leading to the REINFORCE update equation (13.8), so that (13.8) ends up with a factor of γt and thus aligns with the general algorithm given in the pseudo


at (13.8) ends up with a factor of γt and thus aligns with the general algorithm given in the pseudocode. ⇤ Figure 13.1 shows the performance of REINFORCE on the short-corridor gridworld from Example 13.1. ↵= 2−13 ↵= 2−12 Episode 1000 800 600 400 200 1 -80 -90 -60 -40 -20 -10 Total reward on episode averaged over 100 runs G0 v⇤(s0) ↵= 2−14 Figure 13.1: REINFORCE on the short-corridor gridworld (Example 13.1). With a good step size, the total reward per episode approaches the optimal value of the start state.


13.4. REINFORCE with Baseline 329 As a stochastic gradient method, REINFORCE has good theoretical convergence properties. By construction, the expected update over an episode is in the same direction as the performance gradient. This assures an improvement in expected performance for suﬃciently small ↵, and convergence to a local optimum under standard stochastic approximation conditions for decreasing ↵. However, as a Monte Carlo method REINFORCE may be of high variance and thus produce slow learning. Exercise 13.3 In Section 13.1 we considered policy parameterizations using the soft-max in action preferences (13.2) with linear action preferences (13.3). For this parameterization, prove that the eligibility vector is r ln ⇡(a|s, ✓) = x(s, a) − X b ⇡(b|s, ✓)x(s, b), (13.9) using the deﬁnit


t the eligibility vector is r ln ⇡(a|s, ✓) = x(s, a) − X b ⇡(b|s, ✓)x(s, b), (13.9) using the deﬁnitions and elementary calculus. ⇤ 13.4 REINFORCE with Baseline The policy gradient theorem (13.5) can be generalized to include a comparison of the action value to an arbitrary baseline b(s): rJ(✓) / X s µ(s) X a ⇣ q⇡(s, a) −b(s) ⌘ r⇡(a|s, ✓). (13.10) The baseline can be any function, even a random variable, as long as it does not vary with a; the equation remains valid because the subtracted quantity is zero: X a b(s)r⇡(a|s, ✓) = b(s)r X a ⇡(a|s, ✓) = b(s)r1 = 0. The policy gradient theorem with baseline (13.10) can be used to derive an update rule using similar steps as in the previous section.


seline (13.10) can be used to derive an update rule using similar steps as in the previous section. The update rule that we end up with is a new version of REINFORCE that includes a general baseline: ✓t+1 .= ✓t + ↵ ⇣ Gt −b(St) ⌘r⇡(At|St, ✓t) ⇡(At|St, ✓t) . (13.11) Because the baseline could be uniformly zero, this update is a strict generalization of REINFORCE. In general, the baseline leaves the expected value of the update unchanged, but it can have a large e↵ect on its variance. For example, we saw in Section 2.8 that an analogous baseline can signiﬁcantly reduce the variance (and thus speed the learning) of gradient bandit algorithms. In the bandit algorithms the baseline was just a number (the average of the rewards seen so far), but for MDPs the baseline should vary with state.


number (the average of the rewards seen so far), but for MDPs the baseline should vary with state. In some states all actions have high values and we need a high baseline to di↵erentiate the higher valued actions from the less highly valued ones; in other states all actions will have low values and a low baseline is appropriate. One natural choice for the baseline is an estimate of the state value, ˆv(St,w), where w 2 Rd is a weight vector learned by one of the methods presented in previous chapters.


330 Chapter 13: Policy Gradient Methods Because REINFORCE is a Monte Carlo method for learning the policy parameter, ✓, it seems natural to also use a Monte Carlo method to learn the state-value weights, w. A complete pseudocode algorithm for REINFORCE with baseline using such a learned state-value function as the baseline is given in the box below. REINFORCE with Baseline (episodic), for estimating ⇡✓⇡⇡⇤ Input: a di↵erentiable policy parameterization ⇡(a|s, ✓) Input: a di↵erentiable state-value function parameterization ˆv(s,w) Algorithm parameters: step sizes ↵✓> 0, ↵w > 0 Initialize policy parameter ✓2 Rd0 and state-value weights w 2 Rd (e.g., to 0) Loop forever (for each episode): Generate an episode S0, A0, R1, . . .


weights w 2 Rd (e.g., to 0) Loop forever (for each episode): Generate an episode S0, A0, R1, . . . , ST −1, AT −1, RT , following ⇡(·|·, ✓) Loop for each step of the episode t = 0, 1, . . . , T −1: G PT k=t+1 γk−t−1Rk (Gt) δ G −ˆv(St,w) w w + ↵wδrˆv(St,w) ✓ ✓+ ↵✓γtδr ln ⇡(At|St, ✓) This algorithm has two step sizes, denoted ↵✓and ↵w (where ↵✓is the ↵in (13.11)). Choosing the step size for values (here ↵w) is relatively easy; in the linear case we have rules of thumb for setting it, such as ↵w = 0.1/E ⇥ krˆv(St,w)k2 µ ⇤ (see Section 9.6). It is much less clear how to set the step size for the policy parameters, ↵✓, whose best value depends on the range of variation of the rewards and on the policy parameterization.


ose best value depends on the range of variation of the rewards and on the policy parameterization. ↵= 2−13 Episode 1000 800 600 400 200 1 -80 -90 -60 -40 -20 -10 v⇤(s0) REINFORCE REINFORCE with baseline ↵= 2−9 ↵✓= 2−9, ↵w = 2−6 Total reward on episode averaged over 100 runs G0 Figure 13.2: Adding a baseline to REINFORCE can make it learn much faster, as illustrated here on the short-corridor gridworld (Example 13.1). The step size used here for plain REINFORCE is that at which it performs best (to the nearest power of two; see Figure 13.1).


13.5. Actor–Critic Methods 331 Figure 13.2 compares the behavior of REINFORCE with and without a baseline on the short-corridor gridword (Example 13.1). Here the approximate state-value function used in the baseline is ˆv(s,w) = w. That is, w is a single component, w. 13.5 Actor–Critic Methods In REINFORCE with baseline, the learned state-value function estimates the value of the ﬁrst state of each state transition. This estimate sets a baseline for the subsequent return, but is made prior to the transition’s action and thus cannot be used to assess that action. In actor–critic methods, on the other hand, the state-value function is applied also to the second state of the transition. The estimated value of the second state, when discounted and added to the reward, constitutes the one-step


imated value of the second state, when discounted and added to the reward, constitutes the one-step return, Gt:t+1, which is a useful estimate of the actual return and thus is a way of assessing the action. As we have seen in the TD learning of value functions throughout this book, the one-step return is often superior to the actual return in terms of its variance and computational congeniality, even though it introduces bias. We also know how we can ﬂexibly modulate the extent of the bias with n-step returns and eligibility traces (Chapters 7 and 12). When the state-value function is used to assess actions in this way it is called a critic, and the overall policy-gradient method is termed an actor–critic method.


way it is called a critic, and the overall policy-gradient method is termed an actor–critic method. Note that the bias in the gradient estimate is not due to bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method. First consider one-step actor–critic methods, the analog of the TD methods introduced in Chapter 6 such as TD(0), Sarsa(0), and Q-learning. The main appeal of one-step methods is that they are fully online and incremental, yet avoid the complexities of eligibility traces. They are a special case of the eligibility trace methods, but easier to understand. One-step actor–critic methods replace the full return of REINFORCE (13.11) with the one-step return (and use a learned state-value function as the baseline) as follows: ✓t+1 .= ✓t


the one-step return (and use a learned state-value function as the baseline) as follows: ✓t+1 .= ✓t + ↵ ⇣ Gt:t+1 −ˆv(St,w) ⌘r⇡(At|St, ✓t) ⇡(At|St, ✓t) (13.12) = ✓t + ↵ ⇣ Rt+1 + γˆv(St+1,w) −ˆv(St,w) ⌘r⇡(At|St, ✓t) ⇡(At|St, ✓t) (13.13) = ✓t + ↵δt r⇡(At|St, ✓t) ⇡(At|St, ✓t) . (13.14) The natural state-value-function learning method to pair with this is semi-gradient TD(0). Pseudocode for the complete algorithm is given in the box at the top of the next page. Note that it is now a fully online, incremental algorithm, with states, actions, and rewards processed as they occur and then never revisited.


332 Chapter 13: Policy Gradient Methods One-step Actor–Critic (episodic), for estimating ⇡✓⇡⇡⇤ Input: a di↵erentiable policy parameterization ⇡(a|s, ✓) Input: a di↵erentiable state-value function parameterization ˆv(s,w) Parameters: step sizes ↵✓> 0, ↵w > 0 Initialize policy parameter ✓2 Rd0 and state-value weights w 2 Rd (e.g., to 0) Loop forever (for each episode): Initialize S (ﬁrst state of episode) I 1 Loop while S is not terminal (for each time step): A ⇠⇡(·|S, ✓) Take action A, observe S0, R δ R + γ ˆv(S0,w) −ˆv(S,w) (if S0 is terminal, then ˆv(S0,w) .= 0) w w + ↵wδrˆv(S,w) ✓ ✓+ ↵✓I δr ln ⇡(A|S, ✓) I γI S S0 The generalizations to the forward view of n-step methods and then to a λ-return algorithm are straightforward.


zations to the forward view of n-step methods and then to a λ-return algorithm are straightforward. The one-step return in (13.12) is merely replaced by Gt:t+n or Gλ t respectively. The backward view of the λ-return algorithm is also straightforward, using separate eligibility traces for the actor and critic, each after the patterns in Chapter 12. Pseudocode for the complete algorithm is given in the box below. Actor–Critic with Eligibility Traces (episodic), for estimating ⇡✓⇡⇡⇤ Input: a di↵erentiable policy parameterization ⇡(a|s, ✓) Input: a di↵erentiable state-value function parameterization ˆv(s,w) Parameters: trace-decay rates λ✓2 [0, 1], λw 2 [0, 1]; step sizes ↵✓> 0, ↵w > 0 Initialize policy parameter ✓2 Rd0 and state-value weights w 2 Rd (e.g., to 0) Loop forever (for each episode


e policy parameter ✓2 Rd0 and state-value weights w 2 Rd (e.g., to 0) Loop forever (for each episode): Initialize S (ﬁrst state of episode) z✓ 0 (d0-component eligibility trace vector) zw 0 (d-component eligibility trace vector) I 1 Loop while S is not terminal (for each time step): A ⇠⇡(·|S, ✓) Take action A, observe S0, R δ R + γ ˆv(S0,w) −ˆv(S,w) (if S0 is terminal, then ˆv(S0,w) .= 0) zw γλwzw + rˆv(S,w) z✓ γλ✓z✓+ I r ln ⇡(A|S, ✓) w w + ↵wδzw ✓ ✓+ ↵✓δz✓ I γI S S0


13.6. Policy Gradient for Continuing Problems 333 13.6 Policy Gradient for Continuing Problems As discussed in Section 10.3, for continuing problems without episode boundaries we need to deﬁne performance in terms of the average rate of reward per time step: J(✓) .= r(⇡) .= lim h!1 1 h h X t=1 E[Rt | S0, A0:t−1 ⇠⇡] (13.15) = lim t!1 E[Rt | S0, A0:t−1 ⇠⇡] = X s µ(s) X a ⇡(a|s) X s0,r p(s0, r|s, a)r, where µ is the steady-state distribution under ⇡, µ(s) .= limt!1 Pr{St =s|A0:t ⇠⇡}, which is assumed to exist and to be independent of S0 (an ergodicity assumption). Remember that this is the special distribution under which, if you select actions according to ⇡, you remain in the same distribution: X s µ(s) X a ⇡(a|s, ✓)p(s0|s, a) = µ(s0), for all s0 2 S.


o ⇡, you remain in the same distribution: X s µ(s) X a ⇡(a|s, ✓)p(s0|s, a) = µ(s0), for all s0 2 S. (13.16) Complete pseudocode for the actor–critic algorithm in the continuing case (backward view) is given in the box below. Actor–Critic with Eligibility Traces (continuing), for estimating ⇡✓⇡⇡⇤ Input: a di↵erentiable policy parameterization ⇡(a|s, ✓) Input: a di↵erentiable state-value function parameterization ˆv(s,w) Algorithm parameters: λw 2 [0, 1], λ✓2 [0, 1], ↵w > 0, ↵✓> 0, ↵¯ R > 0 Initialize ¯R 2 R (e.g., to 0) Initialize state-value weights w 2 Rd and policy parameter ✓2 Rd0 (e.g., to 0) Initialize S 2 S (e.g., to s0) zw 0 (d-component eligibility trace vector) z✓ 0 (d0-component eligibility trace vector) Loop forever (for each time step): A ⇠⇡(·|S, ✓) Take action A, observe S0, R


igibility trace vector) Loop forever (for each time step): A ⇠⇡(·|S, ✓) Take action A, observe S0, R δ R −¯R + ˆv(S0,w) −ˆv(S,w) ¯R ¯R + ↵¯ Rδ zw λwzw + rˆv(S,w) z✓ λ✓z✓+ r ln ⇡(A|S, ✓) w w + ↵wδzw ✓ ✓+ ↵✓δz✓ S S0


334 Chapter 13: Policy Gradient Methods Naturally, in the continuing case, we deﬁne values, v⇡(s) .= E⇡[Gt|St =s] and q⇡(s, a) .= E⇡[Gt|St =s, At =a], with respect to the di↵erential return: Gt .= Rt+1 −r(⇡) + Rt+2 −r(⇡) + Rt+3 −r(⇡) + · · · . (13.17) With these alternate deﬁnitions, the policy gradient theorem as given for the episodic case (13.5) remains true for the continuing case. A proof is given in the box on the next page. The forward and backward view equations also remain the same. Proof of the Policy Gradient Theorem (continuing case) The proof of the policy gradient theorem for the continuing case begins similarly to the episodic case. Again we leave it implicit in all cases that ⇡is a function of ✓and that the gradients are with respect to ✓.


eave it implicit in all cases that ⇡is a function of ✓and that the gradients are with respect to ✓. Recall that in the continuing case J(✓) = r(⇡) (13.15) and that v⇡and q⇡denote values with respect to the di↵erential return (13.17). The gradient of the state-value function can be written, for any s 2 S, as rv⇡(s) = r "X a ⇡(a|s)q⇡(s, a) # , for all s 2 S (Exercise 3.18) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s)rq⇡(s, a) i (product rule of calculus) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s)r X s0,r p(s0, r|s, a) # r −r(✓) + v⇡(s0) $i = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s) ⇥ −rr(✓) + X s0 p(s0|s, a)rv⇡(s0) ⇤i . After re-arranging terms, we obtain rr(✓) = X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s) X s0 p(s0|s, a)rv⇡(s0) i −rv⇡(s). Notice that the left-hand side can be written rJ(✓), and that it does not depend on s.


) i −rv⇡(s). Notice that the left-hand side can be written rJ(✓), and that it does not depend on s. Thus the right-hand side does not depend on s either, and we can safely sum it over all s 2 S, weighted by µ(s), without changing it (because P s µ(s) = 1): rJ(✓) = X s µ(s) X a h r⇡(a|s)q⇡(s, a) + ⇡(a|s) X s0 p(s0|s, a)rv⇡(s0) i −rv⇡(s) ! = X s µ(s) X a r⇡(a|s)q⇡(s, a) + X s µ(s) X a ⇡(a|s) X s0 p(s0|s, a)rv⇡(s0) − X s µ(s)rv⇡(s)


13.7. Policy Parameterization for Continuous Actions 335 = X s µ(s) X a r⇡(a|s)q⇡(s, a) + X s0 X s µ(s) X a ⇡(a|s)p(s0|s, a) | {z } µ(s0) (13.16) rv⇡(s0) − X s µ(s)rv⇡(s) = X s µ(s) X a r⇡(a|s)q⇡(s, a) + X s0 µ(s0)rv⇡(s0) − X s µ(s)rv⇡(s) = X s µ(s) X a r⇡(a|s)q⇡(s, a). Q.E.D. 13.7 Policy Parameterization for Continuous Actions Policy-based methods o↵er practical ways of dealing with large action spaces, even continuous spaces with an inﬁnite number of actions. Instead of computing learned probabilities for each of the many actions, we instead learn statistics of the probability distribution. For example, the action set might be the real numbers, with actions chosen from a normal (Gaussian) distribution.


he action set might be the real numbers, with actions chosen from a normal (Gaussian) distribution. The probability density function for the normal distribution is conventionally written p(x) .= 1 σ p 2⇡exp ✓ −(x −µ)2 2σ2 ◆ , (13.18) xp ✓ −(x −µ)2 2σ2 ◆ 0.8 0.6 0.4 0.2 0.0 −5 −3 1 3 5 x 1.0 −1 0 2 4 −2 −4 0, µ= 0, µ= 0, µ= −2, µ= 2 0.2, σ = 2 1.0, σ = 2 5.0, σ = 2 0.5, σ = where µ and σ here are the mean and standard deviation of the normal distribution, and of course ⇡here is just the number ⇡⇡3.14159. The probability density functions for several di↵erent means and standard deviations are shown to the right. The value p(x) is the density of the probability at x, not the probability. It can be greater than 1; it is the total area under p(x) that must sum to 1.


not the probability. It can be greater than 1; it is the total area under p(x) that must sum to 1. In general, one can take the integral under p(x) for any range of x values to get the probability of x falling within that range. To produce a policy parameterization, the policy can be deﬁned as the normal probability density over a real-valued scalar action, with mean and standard deviation given by parametric function approximators that depend on the state. That is, ⇡(a|s, ✓) .= 1 σ(s, ✓) p 2⇡exp ✓ −(a −µ(s, ✓))2 2σ(s, ✓)2 ◆ , (13.19) where µ : S⇥Rd0 ! R and σ : S⇥Rd0 ! R+ are two parameterized function approximators.


336 Chapter 13: Policy Gradient Methods To complete the example we need only give a form for these approximators. For this we divide the policy’s parameter vector into two parts, ✓= [✓µ, ✓σ]>, one part to be used for the approximation of the mean and one part for the approximation of the standard deviation. The mean can be approximated as a linear function. The standard deviation must always be positive and is better approximated as the exponential of a linear function. Thus µ(s, ✓) .= ✓µ >xµ(s) and σ(s, ✓) .= exp ⇣ ✓σ >xσ(s) ⌘ , (13.20) where xµ(s) and xσ(s) are state feature vectors perhaps constructed by one of the methods described in Section 9.5. With these deﬁnitions, all the algorithms described in the rest of this chapter can be applied to learn to select real-valued actions.


rithms described in the rest of this chapter can be applied to learn to select real-valued actions. Exercise 13.4 Show that for the Gaussian policy parameterization (Equations 13.19 and 13.20) the eligibility vector has the following two parts: r ln ⇡(a|s, ✓µ) = r⇡(a|s, ✓µ) ⇡(a|s, ✓) = 1 σ(s, ✓)2 # a −µ(s, ✓) $ xµ(s), and r ln ⇡(a|s, ✓σ) = r⇡(a|s, ✓σ) ⇡(a|s, ✓) = # a −µ(s, ✓) $2 σ(s, ✓)2 −1 ! xσ(s). ⇤ Exercise 13.5 A Bernoulli-logistic unit is a stochastic neuron-like unit used in some ANNs (Section 9.7). Its input at time t is a feature vector x(St); its output, At, is a random variable having two values, 0 and 1, with Pr{At = 1} = Pt and Pr{At = 0} = 1 −Pt (the Bernoulli distribution). Let h(s, 0, ✓) and h(s, 1, ✓) be the preferences in state s for the unit’s two actions given policy par


h(s, 0, ✓) and h(s, 1, ✓) be the preferences in state s for the unit’s two actions given policy parameter ✓. Assume that the di↵erence between the action preferences is given by a weighted sum of the unit’s input vector, that is, assume that h(s, 1, ✓) −h(s, 0, ✓) = ✓>x(s), where ✓is the unit’s weight vector. (a) Show that if the exponential soft-max distribution (13.2) is used to convert action preferences to policies, then Pt = ⇡(1|St, ✓t) = 1/(1 + exp(−✓> t x(St))) (the logistic function). (b) What is the Monte-Carlo REINFORCE update of ✓t to ✓t+1 upon receipt of return Gt? (c) Express the eligibility r ln ⇡(a|s, ✓) for a Bernoulli-logistic unit, in terms of a, x(s), and ⇡(a|s, ✓) by calculating the gradient.


, ✓) for a Bernoulli-logistic unit, in terms of a, x(s), and ⇡(a|s, ✓) by calculating the gradient. Hint for part (c): Deﬁne P = ⇡(1|s, ✓) and compute the derivative of the logarithm, for each action, using the chain rule on P. Combine the two results into one expression that depends on a and P, and then use the chain rule again, this time on ✓>x(s), noting that the derivative of the logistic function f(x) = 1/(1 + e−x) is f(x)(1 −f(x)). ⇤


13.8. Summary 337 13.8 Summary Prior to this chapter, this book focused on action-value methods—meaning methods that learn action values and then use them to determine action selections. In this chapter, on the other hand, we considered methods that learn a parameterized policy that enables actions to be taken without consulting action-value estimates. In particular, we have considered policy-gradient methods—meaning methods that update the policy parameter on each step in the direction of an estimate of the gradient of performance with respect to the policy parameter. Methods that learn and store a policy parameter have many advantages. They can learn speciﬁc probabilities for taking the actions.


policy parameter have many advantages. They can learn speciﬁc probabilities for taking the actions. They can learn appropriate levels of exploration and approach deterministic policies asymptotically. They can naturally handle continuous action spaces. All these things are easy for policy-based methods but awkward or impossible for "-greedy methods and for action-value methods in general. In addition, on some problems the policy is just simpler to represent parametrically than the value function; these problems are more suited to parameterized policy methods. Parameterized policy methods also have an important theoretical advantage over action-value methods in the form of the policy gradient theorem, which gives an exact formula for how performance is a↵ected by the policy parameter that d


theorem, which gives an exact formula for how performance is a↵ected by the policy parameter that does not involve derivatives of the state distribution. This theorem provides a theoretical foundation for all policy gradient methods. The REINFORCE method follows directly from the policy gradient theorem. Adding a state-value function as a baseline reduces REINFORCE’s variance without introducing bias. If the state-value function is also used to assess—or criticize—the policy’s action selections, then the value function is called a critic and the policy is called an actor; the overall method is called an actor–critic method. The critic introduces bias into the actor’s gradient estimates, but is often desirable for the same reason that bootstrapping TD methods are often superior to Monte Ca


is often desirable for the same reason that bootstrapping TD methods are often superior to Monte Carlo methods (substantially reduced variance). Overall, policy-gradient methods provide a signiﬁcantly di↵erent set of strengths and weaknesses than action-value methods. Today they are less well understood in some respects, but a subject of excitement and ongoing research. Bibliographical and Historical Remarks Methods that we now see as related to policy gradients were actually some of the earliest to be studied in reinforcement learning (Witten, 1977; Barto, Sutton, and Anderson, 1983; Sutton, 1984; Williams, 1987, 1992) and in predecessor ﬁelds (see Phansalkar and Thathachar, 1995). They were largely supplanted in the 1990s by the action-value methods that are the focus of the other chapt


re largely supplanted in the 1990s by the action-value methods that are the focus of the other chapters of this book. In recent years, however, attention has returned to actor–critic methods and to policy-gradient methods in general. Among the further developments beyond what we cover here are natural-gradient methods (Amari, 1998; Kakade, 2002, Peters, Vijayakumar and Schaal, 2005; Peters and Schaal, 2008; Park, Kim and Kang, 2005; Bhatnagar, Sutton, Ghavamzadeh and Lee, 2009; see Grondman, Busoniu, Lopes and Babuska, 2012), deterministic policy-gradient methods


338 Chapter 13: Policy Gradient Methods (Silver et al., 2014), o↵-policy policy-gradient methods (Degris, White, and Sutton, 2012; Maei, 2018), and entropy regularization (see Schulman, Chen, and Abbeel, 2017). Major applications include acrobatic helicopter autopilots and AlphaGo (Section 16.6). Our presentation in this chapter is based primarily on that by Sutton, McAllester, Singh, and Mansour (2000), who introduced the term “policy gradient methods.” A useful overview is provided by Bhatnagar et al. (2009). One of the earliest related works is by Aleksandrov, Sysoyev, and Shemeneva (1968). Thomas (2014) ﬁrst realized that the factor of γt, as speciﬁed in the boxed algorithms of this chapter, was needed in the case of discounted episodic problems.


ed in the boxed algorithms of this chapter, was needed in the case of discounted episodic problems. 13.1 Example 13.1 and the results with it in this chapter were developed with Eric Graves. 13.2 The policy gradient theorem here and on page 334 was ﬁrst obtained by Marbach and Tsitsiklis (1998, 2001) and then independently by Sutton et al. (2000). A similar expression was obtained by Cao and Chen (1997). Other early results are due to Konda and Tsitsiklis (2000, 2003), Baxter and Bartlett (2001), and Baxter, Bartlett, and Weaver (2001). Some additional results are developed by Sutton, Singh, and McAllester (2000). 13.3 REINFORCE is due to Williams (1987, 1992). Phansalkar and Thathachar (1995) proved both local and global convergence theorems for modiﬁed versions of REINFORCE algorithms.


95) proved both local and global convergence theorems for modiﬁed versions of REINFORCE algorithms. The all-actions algorithm was ﬁrst presented in an unpublished but widely circulated incomplete paper (Sutton, Singh, and McAllester, 2000) and then developed further by Ciosek and Whiteson (2017, 2018), who termed it “expected policy gradients,” and by Asadi, Allen, Roderick, Mohamed, Konidaris, and Littman (2017), who called it “mean actor critic.” 13.4 The baseline was introduced in Williams’s (1987, 1992) original work. Greensmith, Bartlett, and Baxter (2004) analyzed an arguably better baseline (see Dick, 2015). Thomas and Brunskill (2017) argue that an action-dependent baseline can be used without incurring bias.


as and Brunskill (2017) argue that an action-dependent baseline can be used without incurring bias. 13.5–6 Actor–critic methods were among the earliest to be investigated in reinforcement learning (Witten, 1977; Barto, Sutton, and Anderson, 1983; Sutton, 1984). The algorithms presented here are based on the work of Degris, White, and Sutton (2012). Actor–critic methods are sometimes referred to as advantage actor–critic (“A2C”) methods in the literature. 13.7 The ﬁrst to show how continuous actions could be handled this way appears to have been Williams (1987, 1992). The ﬁgure on page 335 is adapted from Wikipedia.


Part III: Looking Deeper In this last part of the book we look beyond the standard reinforcement learning ideas presented in the ﬁrst two parts of the book to brieﬂy survey their relationships with psychology and neuroscience, a sampling of reinforcement learning applications, and some of the active frontiers for future reinforcement learning research.


Chapter 14 Psychology In previous chapters we developed ideas for algorithms based on computational considerations alone. In this chapter we look at some of these algorithms from another perspective: the perspective of psychology and its study of how animals learn. The goals of this chapter are, ﬁrst, to discuss ways that reinforcement learning ideas and algorithms correspond to what psychologists have discovered about animal learning, and second, to explain the inﬂuence reinforcement learning is having on the study of animal learning. The clear formalism provided by reinforcement learning that systemizes tasks, returns, and algorithms is proving to be enormously useful in making sense of experimental data, in suggesting new kinds of experiments, and in pointing to factors that may be crit


perimental data, in suggesting new kinds of experiments, and in pointing to factors that may be critical to manipulate and to measure. The idea of optimizing return over the long term that is at the core of reinforcement learning is contributing to our understanding of otherwise puzzling features of animal learning and behavior. Some of the correspondences between reinforcement learning and psychological theories are not surprising because the development of reinforcement learning drew inspiration from psychological learning theories. However, as developed in this book, reinforcement learning explores idealized situations from the perspective of an artiﬁcial intelligence researcher or engineer, with the goal of solving computational problems with eﬃcient algorithms, rather than to replicat


er, with the goal of solving computational problems with eﬃcient algorithms, rather than to replicate or explain in detail how animals learn. As a result, some of the correspondences we describe connect ideas that arose independently in their respective ﬁelds. We believe these points of contact are specially meaningful because they expose computational principles important to learning, whether it is learning by artiﬁcial or by natural systems. For the most part, we describe correspondences between reinforcement learning and learning theories developed to explain how animals like rats, pigeons, and rabbits learn in controlled laboratory experiments. Thousands of these experiments were conducted throughout the 20th century, and many are still being conducted today.


e experiments were conducted throughout the 20th century, and many are still being conducted today. Although sometimes dismissed as irrelevant to wider issues in psychology, these experiments probe subtle properties of animal learning, often motivated by precise theoretical questions. As psychology shifted its focus to more cognitive aspects of behavior, that is, to mental


342 Chapter 14: Psychology processes such as thought and reasoning, animal learning experiments came to play less of a role in psychology than they once did. But this experimentation led to the discovery of learning principles that are elemental and widespread throughout the animal kingdom, principles that should not be neglected in designing artiﬁcial learning systems. In addition, as we shall see, some aspects of cognitive processing connect naturally to the computational perspective provided by reinforcement learning. This chapter’s ﬁnal section includes references relevant to the connections we discuss as well as to connections we neglect. We hope this chapter encourages readers to probe all of these connections more deeply.


we neglect. We hope this chapter encourages readers to probe all of these connections more deeply. Also included in this ﬁnal section is a discussion of how the terminology used in reinforcement learning relates to that of psychology. Many of the terms and phrases used in reinforcement learning are borrowed from animal learning theories, but the computational/engineering meanings of these terms and phrases do not always coincide with their meanings in psychology. 14.1 Prediction and Control The algorithms we describe in this book fall into two broad categories: algorithms for prediction and algorithms for control.1 These categories arise naturally in solution methods for the reinforcement learning problem presented in Chapter 3.


arise naturally in solution methods for the reinforcement learning problem presented in Chapter 3. In many ways these categories respectively correspond to categories of learning extensively studied by psychologists: classical, or Pavlovian, conditioning and instrumental, or operant, conditioning. These correspondences are not completely accidental because of psychology’s inﬂuence on reinforcement learning, but they are nevertheless striking because they connect ideas arising from di↵erent objectives. The prediction algorithms presented in this book estimate quantities that depend on how features of an agent’s environment are expected to unfold over the future. We speciﬁcally focus on estimating the amount of reward an agent can expect to receive over the future while it interacts with it


ating the amount of reward an agent can expect to receive over the future while it interacts with its environment. In this role, prediction algorithms are policy evaluation algorithms, which are integral components of algorithms for improving policies. But prediction algorithms are not limited to predicting future reward; they can predict any feature of the environment (see, for example, Modayil, White, and Sutton, 2014). The correspondence between prediction algorithms and classical conditioning rests on their common property of predicting upcoming stimuli, whether or not those stimuli are rewarding (or punishing). The situation in an instrumental, or operant, conditioning experiment is di↵erent.


(or punishing). The situation in an instrumental, or operant, conditioning experiment is di↵erent. Here, the experimental apparatus is set up so that an animal is given something it likes (a reward) or something it dislikes (a penalty) depending on what the animal did. The animal learns to increase its tendency to produce rewarded behavior and to decrease its tendency to produce penalized behavior. The reinforcing stimulus is said to be contingent on the animal’s behavior, whereas in classical conditioning it is not (although it is diﬃcult to remove all behavior contingencies in a classical conditioning experiment). Instrumental 1What control means for us is di↵erent from what it typically means in animal learning theories; there the environment controls the agent instead of the other way


means in animal learning theories; there the environment controls the agent instead of the other way around. See our comments on terminology at the end of this chapter.


14.2. Classical Conditioning 343 conditioning experiments are like those that inspired Thorndike’s Law of E↵ect that we brieﬂy discuss in Chapter 1. Control is at the core of this form of learning, which corresponds to the operation of reinforcement learning’s policy-improvement algorithms. Thinking of classical conditioning in terms of prediction, and instrumental conditioning in terms of control, is a starting point for connecting our computational view of reinforcement learning to animal learning, but in reality, the situation is more complicated than this. There is more to classical conditioning than prediction; it also involves action, and so is a mode of control, sometimes called Pavlovian control.


ediction; it also involves action, and so is a mode of control, sometimes called Pavlovian control. Further, classical and instrumental conditioning interact in interesting ways, with both sorts of learning likely being engaged in most experimental situations. Despite these complications, aligning the classical/instrumental distinction with the prediction/control distinction is a convenient ﬁrst approximation in connecting reinforcement learning to animal learning. In psychology, the term reinforcement is used to describe learning in both classical and instrumental conditioning. Originally referring only to the strengthening of a pattern of behavior, it is frequently also used for the weakening of a pattern of behavior.


ng of a pattern of behavior, it is frequently also used for the weakening of a pattern of behavior. A stimulus considered to be the cause of the change in behavior is called a reinforcer, whether or not it is contingent on the animal’s previous behavior. At the end of this chapter we discuss this terminology in more detail and how it relates to terminology used in machine learning. 14.2 Classical Conditioning While studying the activity of the digestive system, the celebrated Russian physiologist Ivan Pavlov found that an animal’s innate responses to certain triggering stimuli can come to be triggered by other stimuli that are quite unrelated to the inborn triggers. His experimental subjects were dogs that had undergone minor surgery to allow the intensity of their salivary reﬂex to be acc


were dogs that had undergone minor surgery to allow the intensity of their salivary reﬂex to be accurately measured. In one case he describes, the dog did not salivate under most circumstances, but about 5 seconds after being presented with food it produced about six drops of saliva over the next several seconds. After several repetitions of presenting another stimulus, one not related to food, in this case the sound of a metronome, shortly before the introduction of food, the dog salivated in response to the sound of the metronome in the same way it did to the food. “The activity of the salivary gland has thus been called into play by impulses of sound—a stimulus quite alien to food” (Pavlov, 1927, p. 22).


s been called into play by impulses of sound—a stimulus quite alien to food” (Pavlov, 1927, p. 22). Summarizing the signiﬁcance of this ﬁnding, Pavlov wrote: It is pretty evident that under natural conditions the normal animal must respond not only to stimuli which themselves bring immediate beneﬁt or harm, but also to other physical or chemical agencies—waves of sound, light, and the like—which in themselves only signal the approach of these stimuli; though it is not the sight and sound of the beast of prey which is in itself harmful to the smaller animal, but its teeth and claws. (Pavlov, 1927, p. 14) Connecting new stimuli to innate reﬂexes in this way is now called classical, or Pavlovian, conditioning. Pavlov (or more exactly, his translators) called inborn responses (e.g.,


344 Chapter 14: Psychology salivation in his demonstration described above) “unconditioned responses” (URs), their natural triggering stimuli (e.g., food) “unconditioned stimuli” (USs), and new responses triggered by predictive stimuli (e.g., here also salivation) “conditioned responses” (CRs). A stimulus that is initially neutral, meaning that it does not normally elicit strong responses (e.g., the metronome sound), becomes a “conditioned stimulus” (CS) as the animal learns that it predicts the US and so comes to produce a CR in response to the CS. These terms are still used in describing classical conditioning experiments (though better translations would have been “conditional” and “unconditional” instead of conditioned and unconditioned).


ations would have been “conditional” and “unconditional” instead of conditioned and unconditioned). The US is called a reinforcer because it reinforces producing a CR in response to the CS. t Trace Conditioning Delay Conditioning CS US CS US ISI The arrangement of stimuli in two common types of classical conditioning experiments is shown to the right. In delay conditioning, the CS extends throughout the interstimulus interval, or ISI, which is the time interval between the CS onset and the US onset (with the CS ending when the US ends in a common version shown here). In trace conditioning, the US begins after the CS ends, and the time interval between CS o↵set and US onset is called the trace interval.


fter the CS ends, and the time interval between CS o↵set and US onset is called the trace interval. The salivation of Pavlov’s dogs to the sound of a metronome is just one example of classical conditioning, which has been intensively studied across many response systems of many species of animals. URs are often preparatory in some way, like the salivation of Pavlov’s dog, or protective in some way, like an eye blink in response to something irritating to the eye, or freezing in response to seeing a predator. Experiencing the CS-US predictive relationship over a series of trials causes the animal to learn that the CS predicts the US so that the animal can respond to the CS with a CR that prepares the animal for, or protects it from, the predicted US.


an respond to the CS with a CR that prepares the animal for, or protects it from, the predicted US. Some CRs are similar to the UR but begin earlier and di↵er in ways that increase their e↵ectiveness. In one intensively studied type of experiment, for example, a tone CS reliably predicts a pu↵of air (the US) to a rabbit’s eye, triggering a UR consisting of the closure of a protective inner eyelid called the nictitating membrane. After one or more trials, the tone comes to trigger a CR consisting of membrane closure that begins before the air pu↵and eventually becomes timed so that peak closure occurs just when the air pu↵is likely to occur. This CR, being initiated in anticipation of the air pu↵and appropriately timed, o↵ers better protection than simply initiating closure as a reaction to


pu↵and appropriately timed, o↵ers better protection than simply initiating closure as a reaction to the irritating US. The ability to act in anticipation of important events by learning about predictive relationships among stimuli is so beneﬁcial that it is widely present across the animal kingdom.


14.2. Classical Conditioning 345 14.2.1 Blocking and Higher-order Conditioning Many interesting properties of classical conditioning have been observed in experiments. Beyond the anticipatory nature of CRs, two widely observed properties ﬁgured prominently in the development of classical conditioning models: blocking and higher-order conditioning. Blocking occurs when an animal fails to learn a CR when a potential CS is presented along with another CS that had been used previously to condition the animal to produce that CR. For example, in the ﬁrst stage of a blocking experiment involving rabbit nictitating membrane conditioning, a rabbit is ﬁrst conditioned with a tone CS and an air pu↵US to produce the CR of closing its nictitating membrane in anticipation of the air pu↵.


an air pu↵US to produce the CR of closing its nictitating membrane in anticipation of the air pu↵. The experiment’s second stage consists of additional trials in which a second stimulus, say a light, is added to the tone to form a compound tone/light CS followed by the same air pu↵US. In the experiment’s third phase, the second stimulus alone—the light—is presented to the rabbit to see if the rabbit has learned to respond to it with a CR. It turns out that the rabbit produces very few, or no, CRs in response to the light: learning to the light had been blocked by the previous learning to the tone.2 Blocking results like this challenged the idea that conditioning depends only on simple temporal contiguity, that is, that a necessary and suﬃcient condition for conditioning is that a US frequ


ral contiguity, that is, that a necessary and suﬃcient condition for conditioning is that a US frequently follows a CS closely in time. In the next section we describe the Rescorla–Wagner model (Rescorla and Wagner, 1972) that o↵ered an inﬂuential explanation for blocking. Higher-order conditioning occurs when a previously-conditioned CS acts as a US in conditioning another initially neutral stimulus. Pavlov described an experiment in which his assistant ﬁrst conditioned a dog to salivate to the sound of a metronome that predicted a food US, as described above. After this stage of conditioning, a number of trials were conducted in which a black square, to which the dog was initially indi↵erent, was placed in the dog’s line of vision followed by the sound of the metronome—and this was not f


nt, was placed in the dog’s line of vision followed by the sound of the metronome—and this was not followed by food. In just ten trials, the dog began to salivate merely upon seeing the black square, despite the fact that the sight of it had never been followed by food. The sound of the metronome itself acted as a US in conditioning a salivation CR to the black square CS. This was second-order conditioning. If the black square had been used as a US to establish salivation CRs to another otherwise neutral CS, it would have been third-order conditioning, and so on. Higher-order conditioning is diﬃcult to demonstrate, especially above the second order, in part because a higher-order reinforcer loses its reinforcing value due to not being repeatedly followed by the original US during higher-or


loses its reinforcing value due to not being repeatedly followed by the original US during higher-order conditioning trials. But under the right conditions, such as intermixing ﬁrst-order trials with higher-order trials or by providing a general energizing stimulus, higher-order conditioning beyond the second order can be demonstrated. As we describe below, the TD model of classical conditioning uses the bootstrapping idea that is central to our approach to extend the Rescorla–Wagner model’s account of blocking to include both the anticipatory nature of CRs and higher-order conditioning. 2Comparison with a control group is necessary to show that the previous conditioning to the tone is responsible for blocking learning to the light.


show that the previous conditioning to the tone is responsible for blocking learning to the light. This is done by trials with the tone/light CS but with no prior conditioning to the tone. Learning to the light in this case is unimpaired. Moore and Schmajuk (2008) give a full account of this procedure.


346 Chapter 14: Psychology Higher-order instrumental conditioning occurs as well. In this case, a stimulus that consistently predicts primary reinforcement becomes a reinforcer itself, where reinforcement is primary if its rewarding or penalizing quality has been built into the animal by evolution. The predicting stimulus becomes a secondary reinforcer, or more generally, a higher-order or conditioned reinforcer—the latter being a better term when the predicted reinforcing stimulus is itself a secondary, or an even higher-order, reinforcer. A conditioned reinforcer delivers conditioned reinforcement: conditioned reward or conditioned penalty. Conditioned reinforcement acts like primary reinforcement in increasing an animal’s tendency to produce behavior that leads to conditioned reward, an


orcement in increasing an animal’s tendency to produce behavior that leads to conditioned reward, and to decrease an animal’s tendency to produce behavior that leads to conditioned penalty. (See our comments at the end of this chapter that explain how our terminology sometimes di↵ers, as it does here, from terminology used in psychology.) Conditioned reinforcement is a key phenomenon that explains, for instance, why we work for the conditioned reinforcer money, whose worth derives solely from what is predicted by having it. In actor–critic methods described in Section 13.5 (and discussed in the context of neuroscience in Sections 15.7 and 15.8), the critic uses a TD method to evaluate the actor’s policy, and its value estimates provide conditioned reinforcement to the actor, allowing the a


tor’s policy, and its value estimates provide conditioned reinforcement to the actor, allowing the actor to improve its policy. This analog of higher-order instrumental conditioning helps address the credit-assignment problem mentioned in Section 1.7 because the critic gives moment-by-moment reinforcement to the actor when the primary reward signal is delayed. We discuss this more below in Section 14.4. 14.2.2 The Rescorla–Wagner Model Rescorla and Wagner created their model mainly to account for blocking. The core idea of the Rescorla–Wagner model is that an animal only learns when events violate its expectations, in other words, only when the animal is surprised (although without necessarily implying any conscious expectation or emotion).


e animal is surprised (although without necessarily implying any conscious expectation or emotion). We ﬁrst present Rescorla and Wagner’s model using their terminology and notation before shifting to the terminology and notation we use to describe the TD model. Here is how Rescorla and Wagner described their model. The model adjusts the “associative strength” of each component stimulus of a compound CS, which is a number representing how strongly or reliably that component is predictive of a US. When a compound CS consisting of several component stimuli is presented in a classical conditioning trial, the associative strength of each component stimulus changes in a way that depends on an associative strength associated with the entire stimulus compound, called the “aggregate associative str


ciative strength associated with the entire stimulus compound, called the “aggregate associative strength,” and not just on the associative strength of each component itself. Rescorla and Wagner considered a compound CS AX, consisting of component stimuli A and X, where the animal may have already experienced stimulus A, and stimulus X might be new to the animal. Let VA, VX, and VAX respectively denote the associative strengths of stimuli A, X, and the compound AX. Suppose that on a trial the compound CS AX is followed by a US, which we label stimulus Y. Then the associative strengths of


14.2. Classical Conditioning 347 the stimulus components change according to these expressions: ∆VA = ↵AβY(RY −VAX) ∆VX = ↵XβY(RY −VAX), where ↵AβY and ↵XβY are the step-size parameters, which depend on the identities of the CS components and the US, and RY is the asymptotic level of associative strength that the US Y can support. (Rescorla and Wagner used λ here instead of R, but we use R to avoid confusion with our use of λ and because we usually think of this as the magnitude of a reward signal, with the caveat that the US in classical conditioning is not necessarily rewarding or penalizing.) A key assumption of the model is that the aggregate associative strength VAX is equal to VA + VX. The associative strengths as changed by these ∆s become the associative strengths at the beginning


The associative strengths as changed by these ∆s become the associative strengths at the beginning of the next trial. To be complete, the model needs a response-generation mechanism, which is a way of mapping values of V s to CRs. Because this mapping would depend on details of the experimental situation, Rescorla and Wagner did not specify a mapping but simply assumed that larger V s would produce stronger or more likely CRs, and that negative V s would mean that there would be no CRs. The Rescorla–Wagner model accounts for the acquisition of CRs in a way that explains blocking. As long as the aggregate associative strength, VAX, of the stimulus compound is below the asymptotic level of associative strength, RY, that the US Y can support, the prediction error RY−VAX is positive.