of the TD method is no more than 1 1−γ times the smallest possible error, that attained in the limit by the Monte Carlo method. Because γ is often near one, this expansion factor can be quite large, so there is substantial potential loss in asymptotic performance with the TD method. On the other hand, recall that the TD methods are often of vastly reduced variance compared to Monte Carlo methods, and thus faster, as we saw in Chapters 6 and 7. Which method will be best depends on the nature of the approximation and problem, and on how long learning continues.


208 Chapter 9: On-policy Prediction with Approximation A bound analogous to (9.14) applies to other on-policy bootstrapping methods as well. For example, linear semi-gradient DP (Eq. 9.7 with Ut .= P a ⇡(a|St) P s0,r p(s0, r|St, a)[r+ γˆv(s0,wt)]) with updates according to the on-policy distribution will also converge to the TD ﬁxed point. One-step semi-gradient action-value methods, such as semi-gradient Sarsa(0) covered in the next chapter converge to an analogous ﬁxed point and an analogous bound. For episodic tasks, there is a slightly di↵erent but related bound (see Bertsekas and Tsitsiklis, 1996). There are also a few technical conditions on the rewards, features, and decrease in the step-size parameter, which we have omitted here.


ions on the rewards, features, and decrease in the step-size parameter, which we have omitted here. The full details can be found in the original paper (Tsitsiklis and Van Roy, 1997). Critical to these convergence results is that states are updated according to the onpolicy distribution. For other update distributions, bootstrapping methods using function approximation may actually diverge to inﬁnity. Examples of this and a discussion of possible solution methods are given in Chapter 11. Example 9.2: Bootstrapping on the 1000-state Random Walk State aggregation is a special case of linear function approximation, so let’s return to the 1000-state random walk to illustrate some of the observations made in this chapter.


s return to the 1000-state random walk to illustrate some of the observations made in this chapter. The left panel of Figure 9.2 shows the ﬁnal value function learned by the semi-gradient TD(0) algorithm (page 203) using the same state aggregation as in Example 9.1. We see that the nearasymptotic TD approximation is indeed farther from the true values than the Monte Carlo approximation shown in Figure 9.1. Nevertheless, TD methods retain large potential advantages in learning rate, and generalize Monte Carlo methods, as we investigated fully with n-step TD methods in Chapter 7. The right panel of Figure 9.2 shows results with an n-step semi-gradient TD method using state aggregation on the 1000-state random walk that are strikingly similar to those we obtained earlier with tabular methods


000-state random walk that are strikingly similar to those we obtained earlier with tabular methods and the 19-state random walk (Figure 7.2). To obtain such quantitatively similar results we switched the state aggregation to 20 groups of 50 states each. The 20 groups were then quantitatively close 0.55 0.5 0.45 0.35 0.3 0.25 0.4 0.4 0.2 0 0.8 0.6 1 ↵ Average RMS error over 1000 states and ﬁrst 10 episodes n=1 n=2 n=4 n=8 n=16 n=32 n=64 128 512 256 State True value v⇡ Approximate TD value 1 0 -1 1 1000 ˆv Figure 9.2: Bootstrapping with state aggregation on the 1000-state random walk task. Left: Asymptotic values of semi-gradient TD are worse than the asymptotic Monte Carlo values in Figure 9.1.


ymptotic values of semi-gradient TD are worse than the asymptotic Monte Carlo values in Figure 9.1. Right: Performance of n-step methods with state-aggregation are strikingly similar to those with tabular representations (cf. Figure 7.2). These data are averages over 100 runs.


9.4. Linear Methods 209 to the 19 states of the tabular problem. In particular, recall that state transitions were up to 100 states to the left or right. A typical transition would then be of 50 states to the right or left, which is quantitatively analogous to the single-state state transitions of the 19-state tabular system. To complete the match, we use here the same performance measure—an unweighted average of the RMS error over all states and over the ﬁrst 10 episodes—rather than a VE objective as is otherwise more appropriate when using function approximation. The semi-gradient n-step TD algorithm used in the example above is the natural extension of the tabular n-step TD algorithm presented in Chapter 7 to semi-gradient function approximation. Pseudocode is given in the box below.


esented in Chapter 7 to semi-gradient function approximation. Pseudocode is given in the box below. n-step semi-gradient TD for estimating ˆv ⇡v⇡ Input: the policy ⇡to be evaluated Input: a di↵erentiable function ˆv : S+ ⇥Rd ! R such that ˆv(terminal,·) = 0 Algorithm parameters: step size ↵> 0, a positive integer n Initialize value-function weights w arbitrarily (e.g., w = 0) All store and access operations (St and Rt) can take their index mod n + 1 Loop for each episode: Initialize and store S0 6= terminal T 1 Loop for t = 0, 1, 2, . . . : | If t < T, then: | Take an action according to ⇡(·|St) | Observe and store the next reward as Rt+1 and the next state as St+1 | If St+1 is terminal, then T t + 1 | ⌧ t −n + 1 (⌧is the time whose state’s estimate is being updated) | If ⌧≥0: | G Pmin(⌧+n


n T t + 1 | ⌧ t −n + 1 (⌧is the time whose state’s estimate is being updated) | If ⌧≥0: | G Pmin(⌧+n,T ) i=⌧+1 γi−⌧−1Ri | If ⌧+ n < T, then: G G + γnˆv(S⌧+n,w) (G⌧:⌧+n) | w w + ↵[G −ˆv(S⌧,w)] rˆv(S⌧,w) Until ⌧= T −1 The key equation of this algorithm, analogous to (7.2), is wt+n .= wt+n−1 + ↵[Gt:t+n −ˆv(St,wt+n−1)] rˆv(St,wt+n−1), 0 t < T, (9.15) where the n-step return is generalized from (7.1) to Gt:t+n .= Rt+1 +γRt+2 +· · ·+γn−1Rt+n +γnˆv(St+n,wt+n−1), 0 t T −n. (9.16) Exercise 9.1 Show that tabular methods such as presented in Part I of this book are a special case of linear function approximation. What would the feature vectors be? ⇤


210 Chapter 9: On-policy Prediction with Approximation 9.5 Feature Construction for Linear Methods Linear methods are interesting because of their convergence guarantees, but also because in practice they can be very eﬃcient in terms of both data and computation. Whether or not this is so depends critically on how the states are represented in terms of features, which we investigate in this large section. Choosing features appropriate to the task is an important way of adding prior domain knowledge to reinforcement learning systems. Intuitively, the features should correspond to the aspects of the state space along which generalization may be appropriate. If we are valuing geometric objects, for example, we might want to have features for each possible shape, color, size, or function.


cts, for example, we might want to have features for each possible shape, color, size, or function. If we are valuing states of a mobile robot, then we might want to have features for locations, degrees of remaining battery power, recent sonar readings, and so on. A limitation of the linear form is that it cannot take into account any interactions between features, such as the presence of feature i being good only in the absence of feature j. For example, in the pole-balancing task (Example 3.4), high angular velocity can be either good or bad depending on the angle. If the angle is high, then high angular velocity means an imminent danger of falling—a bad state—whereas if the angle is low, then high angular velocity means the pole is righting itself—a good state.


eas if the angle is low, then high angular velocity means the pole is righting itself—a good state. A linear value function could not represent this if its features coded separately for the angle and the angular velocity. It needs instead, or in addition, features for combinations of these two underlying state dimensions. In the following subsections we consider a variety of general ways of doing this. 9.5.1 Polynomials The states of many problems are initially expressed as numbers, such as positions and velocities in the pole-balancing task (Example 3.4), the number of cars in each lot in the Jack’s car rental problem (Example 4.2), or the gambler’s capital in the gambler problem (Example 4.3).


’s car rental problem (Example 4.2), or the gambler’s capital in the gambler problem (Example 4.3). In these types of problems, function approximation for reinforcement learning has much in common with the familiar tasks of interpolation and regression. Various families of features commonly used for interpolation and regression can also be used in reinforcement learning. Polynomials make up one of the simplest families of features used for interpolation and regression. While the basic polynomial features we discuss here do not work as well as other types of features in reinforcement learning, they serve as a good introduction because they are simple and familiar. As an example, suppose a reinforcement learning problem has states with two numerical dimensions.


. As an example, suppose a reinforcement learning problem has states with two numerical dimensions. For a single representative state s, let its two numbers be s1 2 R and s2 2 R. You might choose to represent s simply by its two state dimensions, so that x(s) = (s1, s2)>, but then you would not be able to take into account any interactions between these dimensions. In addition, if both s1 and s2 were zero, then the approximate value would have to also be zero. Both limitations can be overcome by instead representing s by the four-dimensional feature vector x(s) = (1, s1, s2, s1s2)>. The initial 1 feature allows the representation of aﬃne functions in the original state numbers, and the ﬁnal product feature, s1s2, enables interactions to be taken into account.


l state numbers, and the ﬁnal product feature, s1s2, enables interactions to be taken into account. Or you might choose to use higher-dimensional feature vectors like x(s) = (1, s1, s2, s1s2, s2 1, s2 2, s1s2 2, s2 1s2, s2 1s2 2)> to


9.5. Feature Construction for Linear Methods 211 take more complex interactions into account. Such feature vectors enable approximations as arbitrary quadratic functions of the state numbers—even though the approximation is still linear in the weights that have to be learned. Generalizing this example from two to k numbers, we can represent highly-complex interactions among a problem’s state dimensions: Suppose each state s corresponds to k numbers, s1, s2, ..., sk, with each si 2 R. For this k-dimensional state space, each order-n polynomial-basis feature xi can be written as xi(s) = ⇧k j=1sci,j j , (9.17) where each ci,j is an integer in the set {0, 1, . . . , n} for an integer n ≥0. These features make up the order-n polynomial basis for dimension k, which contains (n + 1)k di↵erent fea


features make up the order-n polynomial basis for dimension k, which contains (n + 1)k di↵erent features. Higher-order polynomial bases allow for more accurate approximations of more complicated functions. But because the number of features in an order-n polynomial basis grows exponentially with the dimension k of the natural state space (if n>0), it is generally necessary to select a subset of them for function approximation. This can be done using prior beliefs about the nature of the function to be approximated, and some automated selection methods developed for polynomial regression can be adapted to deal with the incremental and nonstationary nature of reinforcement learning. Exercise 9.2 Why does (9.17) deﬁne (n + 1)k distinct features for dimension k?


nforcement learning. Exercise 9.2 Why does (9.17) deﬁne (n + 1)k distinct features for dimension k? ⇤ Exercise 9.3 What n and ci,j produce the feature vectors x(s) = (1, s1, s2, s1s2, s2 1, s2 2, s1s2 2, s2 1s2, s2 1s2 2)>? ⇤ 9.5.2 Fourier Basis Another linear function approximation method is based on the time-honored Fourier series, which expresses periodic functions as weighted sums of sine and cosine basis functions (features) of di↵erent frequencies. (A function f is periodic if f(x) = f(x + ⌧) for all x and some period ⌧.) The Fourier series and the more general Fourier transform are widely used in applied sciences in part because if a function to be approximated is known, then the basis function weights are given by simple formulae and, further, with enough basis functions essentiall


s function weights are given by simple formulae and, further, with enough basis functions essentially any function can be approximated as accurately as desired. In reinforcement learning, where the functions to be approximated are unknown, Fourier basis functions are of interest because they are easy to use and can perform well in a range of reinforcement learning problems. First consider the one-dimensional case. The usual Fourier series representation of a function of one dimension having period ⌧represents the function as a linear combination of sine and cosine functions that are each periodic with periods that evenly divide ⌧(in other words, whose frequencies are integer multiples of a fundamental frequency 1/⌧).


y divide ⌧(in other words, whose frequencies are integer multiples of a fundamental frequency 1/⌧). But if you are interested in approximating an aperiodic function deﬁned over a bounded


212 Chapter 9: On-policy Prediction with Approximation interval, then you can use these Fourier basis features with ⌧set to the length of the interval. The function of interest is then just one period of the periodic linear combination of the sine and cosine features. Furthermore, if you set ⌧to twice the length of the interval of interest and restrict attention to the approximation over the half interval [0, ⌧/2], then you can use just the cosine features. This is possible because you can represent any even function, that is, any function that is symmetric about the origin, with just the cosine basis. So any function over the half-period [0, ⌧/2] can be approximated as closely as desired with enough cosine features.


ver the half-period [0, ⌧/2] can be approximated as closely as desired with enough cosine features. (Saying “any function” is not exactly correct because the function has to be mathematically well-behaved, but we skip this technicality here.) Alternatively, it is possible to use just sine features, linear combinations of which are always odd functions, that is functions that are anti-symmetric about the origin. But it is generally better to keep just the cosine features because “half-even” functions tend to be easier to approximate than “half-odd” functions because the latter are often discontinuous at the origin. Of course, this does not rule out using both sine and cosine features to approximate over the interval [0, ⌧/2], which might be advantageous in some circumstances.


tures to approximate over the interval [0, ⌧/2], which might be advantageous in some circumstances. Following this logic and letting ⌧= 2 so that the features are deﬁned over the half-⌧ interval [0, 1], the one-dimensional order-n Fourier cosine basis consists of the n + 1 features xi(s) = cos(i⇡s), s 2 [0, 1], for i = 0, . . . , n. Figure 9.3 shows one-dimensional Fourier cosine features xi, for i = 1, 2, 3, 4; x0 is a constant function. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1 -1 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1 -1 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1 -1 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8


0.6 0.8 1 1 -1 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1 -1 1 0 Figure 9.3: One-dimensional Fourier cosine-basis features xi, i = 1, 2, 3, 4, for approximating functions over the interval [0, 1]. After Konidaris et al. (2011). This same reasoning applies to the Fourier cosine series approximation in the multidimensional case as described in the box below. Suppose each state s corresponds to a vector of k numbers, s = (s1, s2, ..., sk)>, with each si 2 [0, 1]. The ith feature in the order-n Fourier cosine basis can then be written xi(s) = cos # ⇡s>ci$ , (9.18) where ci = (ci 1, . . . , ci k)>, with ci j 2 {0, . . . , n} for j = 1, . . . , k and i = 1, . . . , (n+1)k. This deﬁnes a feature for each of the (n + 1)k possible integer vectors ci.


i = 1, . . . , (n+1)k. This deﬁnes a feature for each of the (n + 1)k possible integer vectors ci. The inner


9.5. Feature Construction for Linear Methods 213 product s>ci has the e↵ect of assigning an integer in {0, . . . , n} to each dimension of s. As in the one-dimensional case, this integer determines the feature’s frequency along that dimension. The features can of course be shifted and scaled to suit the bounded state space of a particular application. As an example, consider the k = 2 case in which s = (s1, s2)>, where each ci = (ci 1, ci 2)>. Figure 9.4 shows a selection of six Fourier cosine features, each labeled by the vector ci that deﬁnes it (s1 is the horizontal axis and ci is shown as a row vector with the index i omitted). Any zero in c means the feature is constant along that state dimension.


with the index i omitted). Any zero in c means the feature is constant along that state dimension. So if c = (0, 0)>, the feature is constant over both dimensions; if c = (c1, 0)> the feature is constant over the second dimension and varies over the ﬁrst with frequency depending on c1; and similarly, for c = (0, c2)>. When c = (c1, c2)> with neither cj = 0, the feature varies along both dimensions and represents an interaction between the two state variables. The values of c1 and c2 determine the frequency along each dimension, and their ratio gives the direction of the interaction. c = (0, 1) 1 1 0 0 1 1 0 0 c = (0, 1)> c = (1, 0) 1 1 0 0 1 1 0 0 c = (1, 0)> c = (1, 1) 1 1 0 0 1 1 0 0 c = (1, 1)> c = (1, 5) 1 1 0 0 1 1 0 0 c = (0, 5)> c = (2, 5) 1 1 0 0 1 1 0 0 c = (2, 5)> 1 1 0 0 c = (5


(1, 1)> c = (1, 5) 1 1 0 0 1 1 0 0 c = (0, 5)> c = (2, 5) 1 1 0 0 1 1 0 0 c = (2, 5)> 1 1 0 0 c = (5, 2)> Figure 9.4: A selection of six two-dimensional Fourier cosine features, each labeled by the vector ci that deﬁnes it (s1 is the horizontal axis, and ci is shown with the index i omitted). After Konidaris et al. (2011). When using Fourier cosine features with a learning algorithm such as (9.7), semigradient TD(0), or semi-gradient Sarsa, it may be helpful to use a di↵erent step-size parameter for each feature. If ↵is the basic step-size parameter, then Konidaris, Osentoski, and Thomas (2011) suggest setting the step-size parameter for feature xi to ↵i = ↵/ p (ci 1)2 + · · · + (ci k)2 (except when each ci j = 0, in which case ↵i = ↵).


214 Chapter 9: On-policy Prediction with Approximation Fourier cosine features with Sarsa can produce good performance compared to several other collections of basis functions, including polynomial and radial basis functions. Not surprisingly, however, Fourier features have trouble with discontinuities because it is diﬃcult to avoid “ringing” around points of discontinuity unless very high frequency basis functions are included. The number of features in the order-n Fourier basis grows exponentially with the dimension of the state space, but if that dimension is small enough (e.g., k 5), then one can select n so that all of the order-n Fourier features can be used. This makes the selection of features more-or-less automatic.


order-n Fourier features can be used. This makes the selection of features more-or-less automatic. For high dimension state spaces, however, it is necessary to select a subset of these features. This can be done using prior beliefs about the nature of the function to be approximated, and some automated selection methods can be adapted to deal with the incremental and nonstationary nature of reinforcement learning. An advantage of Fourier basis features in this regard is that it is easy to select features by setting the ci vectors to account for suspected interactions among the state variables and by limiting the values in the cj vectors so that the approximation can ﬁlter out high frequency components considered to be noise.


j vectors so that the approximation can ﬁlter out high frequency components considered to be noise. On the other hand, because Fourier features are non-zero over the entire state space (with the few zeros excepted), they represent global properties of states, which can make it diﬃcult to ﬁnd good ways to represent local properties. Figure 9.5 shows learning curves comparing the Fourier and polynomial bases on the 1000-state random walk example. In general, we do not recommend using polynomials for online learning.2 .4 .3 .2 .1 0 0 5000 Episodes Polynomial basis Fourier basis p VE averaged over 30 runs Figure 9.5: Fourier basis vs polynomials on the 1000-state random walk. Shown are learning curves for the gradient Monte Carlo method with Fourier and polynomial bases of order 5, 10, and 20.


curves for the gradient Monte Carlo method with Fourier and polynomial bases of order 5, 10, and 20. The step-size parameters were roughly optimized for each case: ↵= 0.0001 for the polynomial basis and ↵= 0.00005 for the Fourier basis. The performance measure (y-axis) is the root mean square value error (9.1). 2There are families of polynomials more complicated than those we have discussed, for example, di↵erent families of orthogonal polynomials, and these might work better, but at present there is little experience with them in reinforcement learning.


9.5. Feature Construction for Linear Methods 215 9.5.3 Coarse Coding s0 s Figure 9.6: Coarse coding. Generalization from state s to state s0 depends on the number of their features whose receptive ﬁelds (in this case, circles) overlap. These states have one feature in common, so there will be slight generalization between them. Consider a task in which the natural representation of the state set is a continuous twodimensional space. One kind of representation for this case is made up of features corresponding to circles in state space, as shown to the right. If the state is inside a circle, then the corresponding feature has the value 1 and is said to be present; otherwise the feature is 0 and is said to be absent. This kind of 1–0-valued feature is called a binary feature.


feature is 0 and is said to be absent. This kind of 1–0-valued feature is called a binary feature. Given a state, which binary features are present indicate within which circles the state lies, and thus coarsely code for its location. Representing a state with features that overlap in this way (although they need not be circles or binary) is known as coarse coding. Assuming linear gradient-descent function approximation, consider the e↵ect of the size and density of the circles. Corresponding to each circle is a single weight (a component of w) that is a↵ected by learning. If we train at one state, a point in the space, then the weights of all circles intersecting that state will be a↵ected.


ate, a point in the space, then the weights of all circles intersecting that state will be a↵ected. Thus, by (9.8), the approximate value function will be a↵ected at all states within the union of the circles, with a greater e↵ect the more circles a point has “in common” with the state, as shown in Figure 9.6. If the circles are small, then the generalization will be over a short distance, as in Figure 9.7 (left), whereas if they are large, it will be over a large distance, as in Figure 9.7 (middle). Moreover, a) Narrow generalization b) Broad generalization c) Asymmetric generalization Figure 9.7: Generalization in linear function approximation methods is determined by the sizes and shapes of the features’ receptive ﬁelds.


ction approximation methods is determined by the sizes and shapes of the features’ receptive ﬁelds. All three of these cases have roughly the same number and density of features.


216 Chapter 9: On-policy Prediction with Approximation the shape of the features will determine the nature of the generalization. For example, if they are not strictly circular, but are elongated in one direction, then generalization will be similarly a↵ected, as in Figure 9.7 (right). Features with large receptive ﬁelds give broad generalization, but might also seem to limit the learned function to a coarse approximation, unable to make discriminations much ﬁner than the width of the receptive ﬁelds. Happily, this is not the case. Initial generalization from one point to another is indeed controlled by the size and shape of the receptive ﬁelds, but acuity, the ﬁnest discrimination ultimately possible, is controlled more by the total number of features.


, the ﬁnest discrimination ultimately possible, is controlled more by the total number of features. Example 9.3: Coarseness of Coarse Coding This example illustrates the e↵ect on learning of the size of the receptive ﬁelds in coarse coding. Linear function approximation based on coarse coding and (9.7) was used to learn a one-dimensional square-wave function (shown at the top of Figure 9.8). The values of this function were used as the targets, Ut. With just one dimension, the receptive ﬁelds were intervals rather than circles. Learning was repeated with three di↵erent sizes of the intervals: narrow, medium, and broad, as shown at the bottom of the ﬁgure. All three cases had the same density of features, about 50 over the extent of the function being learned.


ree cases had the same density of features, about 50 over the extent of the function being learned. Training examples were generated uniformly at random over this extent. The step-size parameter was ↵= 0.2 n , where n is the number of features that were present at one time. Figure 9.8 shows the functions learned in all three cases over the course of learning. Note that the width of the features had a strong e↵ect early in learning. With broad features, the generalization tended to be broad; with narrow features, only the close neighbors of each trained point were changed, causing the function learned to be more bumpy. However, the ﬁnal function learned was a↵ected only slightly by the width of the features.


e bumpy. However, the ﬁnal function learned was a↵ected only slightly by the width of the features. Receptive ﬁeld shape tends to have a strong e↵ect on generalization but little e↵ect on asymptotic solution quality. 10 40 160 640 2560 10240 Narrow features desired function Medium features Broad features #Examples approximation feature width Figure 9.8: Example of feature width’s strong e↵ect on initial generalization (ﬁrst row) and weak e↵ect on asymptotic accuracy (last row).


9.5. Feature Construction for Linear Methods 217 9.5.4 Tile Coding Tile coding is a form of coarse coding for multi-dimensional continuous spaces that is ﬂexible and computationally eﬃcient. It may be the most practical feature representation for modern sequential digital computers. In tile coding the receptive ﬁelds of the features are grouped into partitions of the state space. Each such partition is called a tiling, and each element of the partition is called a tile. For example, the simplest tiling of a two-dimensional state space is a uniform grid such as that shown on the left side of Figure 9.9. The tiles or receptive ﬁeld here are squares rather than the circles in Figure 9.6. If just this single tiling were used, then the state indicated by the white spot would be represented by t


t this single tiling were used, then the state indicated by the white spot would be represented by the single feature whose tile it falls within; generalization would be complete to all states within the same tile and nonexistent to states outside it. With just one tiling, we would not have coarse coding but just a case of state aggregation. Point in state space to be represented Tiling 1 Tiling 2 Tiling 3 Tiling 4 Continuous 2D state space Four active tiles/features overlap the point and are used to represent it Figure 9.9: Multiple, overlapping grid-tilings on a limited two-dimensional space. These tilings are o↵set from one another by a uniform amount in each dimension. To get the strengths of coarse coding requires overlapping receptive ﬁelds, and by deﬁnition the tiles of a partition


hs of coarse coding requires overlapping receptive ﬁelds, and by deﬁnition the tiles of a partition do not overlap. To get true coarse coding with tile coding, multiple tilings are used, each o↵set by a fraction of a tile width. A simple case with four tilings is shown on the right side of Figure 9.9. Every state, such as that indicated by the white spot, falls in exactly one tile in each of the four tilings. These four tiles correspond to four features that become active when the state occurs. Speciﬁcally, the feature vector x(s) has one component for each tile in each tiling. In this example there are 4 ⇥4 ⇥4 = 64 components, all of which will be 0 except for the four corresponding to the tiles that s falls within.


ponents, all of which will be 0 except for the four corresponding to the tiles that s falls within. Figure 9.10 shows the advantage of multiple o↵set tilings (coarse coding) over a single tiling on the 1000-state random walk example. An immediate practical advantage of tile coding is that, because it works with partitions, the overall number of features that are active at one time is the same for any state. Exactly one feature is present in each tiling, so the total number of features present is always the same as the number of tilings. This allows the step-size parameter, ↵, to


218 Chapter 9: On-policy Prediction with Approximation .4 .3 .2 .1 0 averaged over 30 runs 0 5000 Episodes State aggregation (one tiling) Tile coding (50 tilings) p VE Figure 9.10: Why we use coarse coding. Shown are learning curves on the 1000-state random walk example for the gradient Monte Carlo algorithm with a single tiling and with multiple tilings. The space of 1000 states was treated as a single continuous dimension, covered with tiles each 200 states wide. The multiple tilings were o↵set from each other by 4 states. The step-size parameter was set so that the initial learning rate in the two cases was the same, ↵= 0.0001 for the single tiling and ↵= 0.0001/50 for the 50 tilings. be set in an easy, intuitive way.


0.0001 for the single tiling and ↵= 0.0001/50 for the 50 tilings. be set in an easy, intuitive way. For example, choosing ↵= 1 n, where n is the number of tilings, results in exact one-trial learning. If the example s 7! v is trained on, then whatever the prior estimate, ˆv(s,wt), the new estimate will be ˆv(s,wt+1) = v. Usually one wishes to change more slowly than this, to allow for generalization and stochastic variation in target outputs. For example, one might choose ↵= 1 10n, in which case the estimate for the trained state would move one-tenth of the way to the target in one update, and neighboring states will be moved less, proportional to the number of tiles they have in common. Tile coding also gains computational advantages from its use of binary feature vectors.


in common. Tile coding also gains computational advantages from its use of binary feature vectors. Because each component is either 0 or 1, the weighted sum making up the approximate value function (9.8) is almost trivial to compute. Rather than performing d multiplications and additions, one simply computes the indices of the n ⌧d active features and then adds up the n corresponding components of the weight vector. Generalization occurs to states other than the one trained if those states fall within any of the same tiles, proportional to the number of tiles in common. Even the choice of how to o↵set the tilings from each other a↵ects generalization. If they are o↵set uniformly in each dimension, as they were in Figure 9.9, then di↵erent states can generalize in qualitatively di↵erent wa


ension, as they were in Figure 9.9, then di↵erent states can generalize in qualitatively di↵erent ways, as shown in the upper half of Figure 9.11. Each of the eight subﬁgures show the pattern of generalization from a trained state to nearby points. In this example there are eight tilings, thus 64 subregions within a tile that generalize distinctly, but all according to one of these eight patterns. Note how uniform o↵sets result in a strong e↵ect along the diagonal in many patterns. These artifacts can be avoided if the tilings are o↵set asymmetrically, as shown in the lower half of the ﬁgure. These lower generalization patterns are better because they are all well centered on the trained state with no obvious asymmetries.


9.5. Feature Construction for Linear Methods 219 Possible generalizations for uniformly oﬀset tilings Possible generalizations for asymmetrically oﬀset tilings Figure 9.11: Why tile asymmetrical o↵sets are preferred in tile coding. Shown is the strength of generalization from a trained state, indicated by the small black plus, to nearby states, for the case of eight tilings. If the tilings are uniformly o↵set (above), then there are diagonal artifacts and substantial variations in the generalization, whereas with asymmetrically o↵set tilings the generalization is more spherical and homogeneous. Tilings in all cases are o↵set from each other by a fraction of a tile width in each dimension. If w denotes the tile width and n the number of tilings, then w n is a fundamental unit.


dimension. If w denotes the tile width and n the number of tilings, then w n is a fundamental unit. Within small squares w n on a side, all states activate the same tiles, have the same feature representation, and the same approximated value. If a state is moved by w n in any cartesian direction, the feature representation changes by one component/tile. Uniformly o↵set tilings are o↵set from each other by exactly this unit distance. For a two-dimensional space, we say that each tiling is o↵set by the displacement vector (1, 1), meaning that it is o↵set from the previous tiling by w n times this vector. In these terms, the asymmetrically o↵set tilings shown in the lower part of Figure 9.11 are o↵set by a displacement vector of (1, 3).


o↵set tilings shown in the lower part of Figure 9.11 are o↵set by a displacement vector of (1, 3). Extensive studies have been made of the e↵ect of di↵erent displacement vectors on the generalization of tile coding (Parks and Militzer, 1991; An, 1991; An, Miller and Parks,


220 Chapter 9: On-policy Prediction with Approximation 1991; Miller, An, Glanz and Carter, 1990), assessing their homegeneity and tendency toward diagonal artifacts like those seen for the (1, 1) displacement vectors. Based on this work, Miller and Glanz (1996) recommend using displacement vectors consisting of the ﬁrst odd integers. In particular, for a continuous space of dimension k, a good choice is to use the ﬁrst odd integers (1, 3, 5, 7, . . . , 2k −1), with n (the number of tilings) set to an integer power of 2 greater than or equal to 4k. This is what we have done to produce the tilings in the lower half of Figure 9.11, in which k = 2, n = 23 ≥4k, and the displacement vector is (1, 3).


n the lower half of Figure 9.11, in which k = 2, n = 23 ≥4k, and the displacement vector is (1, 3). In a three-dimensional case, the ﬁrst four tilings would be o↵set in total from a base position by (0, 0, 0), (1, 3, 5), (2, 6, 10), and (3, 9, 15). Open-source software that can eﬃciently make tilings like this for any k is readily available. In choosing a tiling strategy, one has to pick the number of the tilings and the shape of the tiles. The number of tilings, along with the size of the tiles, determines the resolution or ﬁneness of the asymptotic approximation, as in general coarse coding and illustrated in Figure 9.8. The shape of the tiles will determine the nature of generalization as in Figure 9.7.


in Figure 9.8. The shape of the tiles will determine the nature of generalization as in Figure 9.7. Square tiles will generalize roughly equally in each dimension as indicated in Figure 9.11 (lower). Tiles that are elongated along one dimension, such as the stripe tilings in Figure 9.12 (middle), will promote generalization along that dimension. The tilings in Figure 9.12 (middle) are also denser and thinner on the left, promoting discrimination along the horizontal dimension at lower values along that dimension. The diagonal stripe tiling in Figure 9.12 (right) will promote generalization along one diagonal. In higher dimensions, axis-aligned stripes correspond to ignoring some of the dimensions in some of the tilings, that is, to hyperplanar slices.


rrespond to ignoring some of the dimensions in some of the tilings, that is, to hyperplanar slices. Irregular tilings such as shown in Figure 9.12 (left) are also possible, though rare in practice and beyond the standard software. a) Irregular b) Log stripes c) Diagonal stripes Figure 9.12: Tilings need not be grids. They can be arbitrarily shaped and non-uniform, while still in many cases being computationally eﬃcient to compute. In practice, it is often desirable to use di↵erent shaped tiles in di↵erent tilings. For example, one might use some vertical stripe tilings and some horizontal stripe tilings. This would encourage generalization along either dimension. However, with stripe tilings alone it is not possible to learn that a particular conjunction of horizontal and vertical coordina


alone it is not possible to learn that a particular conjunction of horizontal and vertical coordinates has a distinctive value (whatever is learned for it will bleed into states with the same horizontal and vertical coordinates). For this one needs the conjunctive rectangular tiles such as originally shown in Figure 9.9. With multiple tilings—some horizontal, some vertical, and some conjunctive—one can get everything: a preference for generalizing along each dimension, yet the ability to learn speciﬁc values for conjunctions (see Sutton,


9.5. Feature Construction for Linear Methods 221 1996 for examples). The choice of tilings determines generalization, and until this choice can be e↵ectively automated, it is important that tile coding enables the choice to be made ﬂexibly and in a way that makes sense to people. Another useful trick for reducing memory requirements is hashing—a consistent pseudorandom collapsing of a large tiling into a much smaller set of tiles. Hashing produces tiles consisting of noncontiguous, disjoint regions randomly spread throughout the state one tile space, but that still form an exhaustive partition. For example, one tile might consist of the four subtiles shown to the right. Through hashing, memory requirements are often reduced by large factors with little loss of performance.


gh hashing, memory requirements are often reduced by large factors with little loss of performance. This is possible because high resolution is needed in only a small fraction of the state space. Hashing frees us from the curse of dimensionality in the sense that memory requirements need not be exponential in the number of dimensions, but need merely match the real demands of the task. Open-source implementations of tile coding commonly include eﬃcient hashing. Exercise 9.4 Suppose we believe that one of two state dimensions is more likely to have an e↵ect on the value function than is the other, that generalization should be primarily across this dimension rather than along it. What kind of tilings could be used to take advantage of this prior knowledge?


rather than along it. What kind of tilings could be used to take advantage of this prior knowledge? ⇤ 9.5.5 Radial Basis Functions Radial basis functions (RBFs) are the natural generalization of coarse coding to continuousvalued features. Rather than each feature being either 0 or 1, it can be anything in the interval [0, 1], reﬂecting various degrees to which the feature is present. A typical RBF feature, xi, has a Gaussian (bell-shaped) response xi(s) dependent only on the distance between the state, s, and the feature’s prototypical or center state, ci, and relative to the feature’s width, σi: xi(s) .= exp ✓ −||s −ci||2 2σ2 i ◆ . The norm or distance metric of course can be chosen in whatever way seems most appropriate to the states and task at hand.


tric of course can be chosen in whatever way seems most appropriate to the states and task at hand. The ﬁgure below shows a one-dimensional example with a Euclidean distance metric. ci !i ci+1 ci-1 Figure 9.13: One-dimensional radial basis functions.


222 Chapter 9: On-policy Prediction with Approximation The primary advantage of RBFs over binary features is that they produce approximate functions that vary smoothly and are di↵erentiable. Although this is appealing, in most cases it has no practical signiﬁcance. Nevertheless, extensive studies have been made of graded response functions such as RBFs in the context of tile coding (An, 1991; Miller et al., 1991; An et al., 1991; Lane, Handelman and Gelfand, 1992). All of these methods require substantial additional computational complexity (over tile coding) and often reduce performance when there are more than two state dimensions. In high dimensions the edges of tiles are much more important, and it has proven diﬃcult to obtain well controlled graded tile activations near the edges.


ortant, and it has proven diﬃcult to obtain well controlled graded tile activations near the edges. An RBF network is a linear function approximator using RBFs for its features. Learning is deﬁned by equations (9.7) and (9.8), exactly as in other linear function approximators. In addition, some learning methods for RBF networks change the centers and widths of the features as well, bringing them into the realm of nonlinear function approximators. Nonlinear methods may be able to ﬁt target functions much more precisely. The downside to RBF networks, and to nonlinear RBF networks especially, is greater computational complexity and, often, more manual tuning before learning is robust and eﬃcient.


ater computational complexity and, often, more manual tuning before learning is robust and eﬃcient. 9.6 Selecting Step-Size Parameters Manually Most SGD methods require the designer to select an appropriate step-size parameter ↵. Ideally this selection would be automated, and in some cases it has been, but for most cases it is still common practice to set it manually. To do this, and to better understand the algorithms, it is useful to develop some intuitive sense of the role of the step-size parameter. Can we say in general how it should be set? Theoretical considerations are unfortunately of little help. The theory of stochastic approximation gives us conditions (2.7) on a slowly decreasing step-size sequence that are suﬃcient to guarantee convergence, but these tend to result in learnin


g step-size sequence that are suﬃcient to guarantee convergence, but these tend to result in learning that is too slow. The classical choice ↵t = 1/t, which produces sample averages in tabular MC methods, is not appropriate for TD methods, for nonstationary problems, or for any method using function approximation. For linear methods, there are recursive least-squares methods that set an optimal matrix step size, and these methods can be extended to temporaldi↵erence learning as in the LSTD method described in Section 9.8, but these require O(d2) step-size parameters, or d times more parameters than we are learning. For this reason we rule them out for use on large problems where function approximation is most needed.


this reason we rule them out for use on large problems where function approximation is most needed. To get some intuitive feel for how to set the step-size parameter manually, it is best to go back momentarily to the tabular case. There we can understand that a step size of ↵= 1 will result in a complete elimination of the sample error after one target (see (2.4) with a step size of one). As discussed on page 201, we usually want to learn slower than this. In the tabular case, a step size of ↵= 1 10 would take about 10 experiences to converge approximately to their mean target, and if we wanted to learn in 100 experiences we would use ↵= 1 100. In general, if ↵= 1 ⌧, then the tabular estimate for a state will approach the mean of its targets, with the most recent targets having the greates


e for a state will approach the mean of its targets, with the most recent targets having the greatest e↵ect, after about ⌧experiences with the state.


9.7. Nonlinear Function Approximation: Artiﬁcial Neural Networks 223 With general function approximation there is not such a clear notion of number of experiences with a state, as each state may be similar to and dissimilar from all the others to various degrees. However, there is a similar rule that gives similar behavior in the case of linear function approximation. Suppose you wanted to learn in about ⌧experiences with substantially the same feature vector. A good rule of thumb for setting the step-size parameter of linear SGD methods is then ↵.= # ⌧E ⇥ x>x ⇤$−1 , (9.19) where x is a random feature vector chosen from the same distribution as input vectors will be in the SGD. This method works best if the feature vectors do not vary greatly in length; ideally x>x is a constant.


method works best if the feature vectors do not vary greatly in length; ideally x>x is a constant. Exercise 9.5 Suppose you are using tile coding to transform a seven-dimensional continuous state space into binary feature vectors to estimate a state value function ˆv(s,w) ⇡v⇡(s). You believe that the dimensions do not interact strongly, so you decide to use eight tilings of each dimension separately (stripe tilings), for 7 ⇥8 = 56 tilings. In addition, in case there are some pairwise interactions between the dimensions, you also take all #7 2 $ = 21 pairs of dimensions and tile each pair conjunctively with rectangular tiles. You make two tilings for each pair of dimensions, making a grand total of 21 ⇥2 + 56 = 98 tilings.


You make two tilings for each pair of dimensions, making a grand total of 21 ⇥2 + 56 = 98 tilings. Given these feature vectors, you suspect that you still have to average out some noise, so you decide that you want learning to be gradual, taking about 10 presentations with the same feature vector before learning nears its asymptote. What step-size parameter ↵ should you use? Why? ⇤ Exercise 9.6 If ⌧= 1 and x(St)>x(St) = E ⇥ x>x ⇤ , prove that (9.19) together with (9.7) and linear function approximation results in the error being reduced to zero in one update. 9.7 Nonlinear Function Approximation: Artiﬁcial Neural Networks Artiﬁcial neural networks (ANNs) are widely used for nonlinear function approximation.


ral Networks Artiﬁcial neural networks (ANNs) are widely used for nonlinear function approximation. An ANN is a network of interconnected units that have some of the properties of neurons, the main components of nervous systems. ANNs have a long history, with the latest advances in training deeply-layered ANNs (deep learning) being responsible for some of the most impressive abilities of machine learning systems, including reinforcement learning systems. In Chapter 16 we describe several impressive examples of reinforcement learning systems that use ANN function approximation. Figure 9.14 shows a generic feedforward ANN, meaning that there are no loops in the network, that is, there are no paths within the network by which a unit’s output can inﬂuence its input.


rk, that is, there are no paths within the network by which a unit’s output can inﬂuence its input. The network in the ﬁgure has an output layer consisting of two output units, an input layer with four input units, and two “hidden layers”: layers that are neither input nor output layers. A real-valued weight is associated with each link. A weight roughly corresponds to the eﬃcacy of a synaptic connection in a real neural network (see Section 15.1). If an ANN has at least one loop in its connections, it is a recurrent rather


224 Chapter 9: On-policy Prediction with Approximation Figure 9.14: A generic feedforward ANN with four input units, two output units, and two hidden layers. than a feedforward ANN. Although both feedforward and recurrent ANNs have been used in reinforcement learning, here we look only at the simpler feedforward case. The units (the circles in Figure 9.14) are typically semi-linear units, meaning that they compute a weighted sum of their input signals and then apply to the result a nonlinear function, called the activation function, to produce the unit’s output, or activation. Di↵erent activation functions are used, but they are typically S-shaped, or sigmoid, functions such as the logistic function f(x) = 1/(1 + e−x), though sometimes the rectiﬁer nonlinearity f(x) = max(0, x) is used.


c function f(x) = 1/(1 + e−x), though sometimes the rectiﬁer nonlinearity f(x) = max(0, x) is used. A step function like f(x) = 1 if x ≥✓, and 0 otherwise, results in a binary unit with threshold ✓. The units in a network’s input layer are somewhat di↵erent in having their activations set to externally-supplied values that are the inputs to the function the network is approximating. The activation of each output unit of a feedforward ANN is a nonlinear function of the activation patterns over the network’s input units. The functions are parameterized by the network’s connection weights. An ANN with no hidden layers can represent only a very small fraction of the possible input-output functions.


h no hidden layers can represent only a very small fraction of the possible input-output functions. However an ANN with a single hidden layer containing a large enough ﬁnite number of sigmoid units can approximate any continuous function on a compact region of the network’s input space to any degree of accuracy (Cybenko, 1989). This is also true for other nonlinear activation functions that satisfy mild conditions, but nonlinearity is essential: if all the units in a multi-layer feedforward ANN have linear activation functions, the entire network is equivalent to a network with no hidden layers (because linear functions of linear functions are themselves linear). Despite this “universal approximation” property of one-hidden-layer ANNs, both experience and theory show that approximating the


oximation” property of one-hidden-layer ANNs, both experience and theory show that approximating the complex functions needed for many artiﬁcial intelligence tasks is made easier—indeed may require—abstractions that are hierarchical compositions of many layers of lower-level abstractions, that is, abstractions


9.7. Nonlinear Function Approximation: Artiﬁcial Neural Networks 225 produced by deep architectures such as ANNs with many hidden layers. (See Bengio, 2009, for a thorough review.) The successive layers of a deep ANN compute increasingly abstract representations of the network’s “raw” input, with each unit providing a feature contributing to a hierarchical representation of the overall input-output function of the network. Training the hidden layers of an ANN is therefore a way to automatically create features appropriate for a given problem so that hierarchical representations can be produced without relying exclusively on hand-crafted features. This has been an enduring challenge for artiﬁcial intelligence and explains why learning algorithms for ANNs with hidden layers have received so


cial intelligence and explains why learning algorithms for ANNs with hidden layers have received so much attention over the years. ANNs typically learn by a stochastic gradient method (Section 9.3). Each weight is adjusted in a direction aimed at improving the network’s overall performance as measured by an objective function to be either minimized or maximized. In the most common supervised learning case, the objective function is the expected error, or loss, over a set of labeled training examples. In reinforcement learning, ANNs can use TD errors to learn value functions, or they can aim to maximize expected reward as in a gradient bandit (Section 2.8) or a policy-gradient algorithm (Chapter 13).


expected reward as in a gradient bandit (Section 2.8) or a policy-gradient algorithm (Chapter 13). In all of these cases it is necessary to estimate how a change in each connection weight would inﬂuence the network’s overall performance, in other words, to estimate the partial derivative of an objective function with respect to each weight, given the current values of all the network’s weights. The gradient is the vector of these partial derivatives. The most successful way to do this for ANNs with hidden layers (provided the units have di↵erentiable activation functions) is the backpropagation algorithm, which consists of alternating forward and backward passes through the network. Each forward pass computes the activation of each unit given the current activations of the network’s input


rward pass computes the activation of each unit given the current activations of the network’s input units. After each forward pass, a backward pass eﬃciently computes a partial derivative for each weight. (As in other stochastic gradient learning algorithms, the vector of these partial derivatives is an estimate of the true gradient.) In Section 15.10 we discuss methods for training ANNs with hidden layers that use reinforcement learning principles instead of backpropagation. These methods are less eﬃcient than the backpropagation algorithm, but they may be closer to how real neural networks learn. The backpropagation algorithm can produce good results for shallow networks having 1 or 2 hidden layers, but it may not work well for deeper ANNs.


results for shallow networks having 1 or 2 hidden layers, but it may not work well for deeper ANNs. In fact, training a network with k + 1 hidden layers can actually result in poorer performance than training a network with k hidden layers, even though the deeper network can represent all the functions that the shallower network can (Bengio, 2009). Explaining results like these is not easy, but several factors are important. First, the large number of weights in a typical deep ANN makes it diﬃcult to avoid the problem of overﬁtting, that is, the problem of failing to generalize correctly to cases on which the network has not been trained. Second, backpropagation does not work well for deep ANNs because the partial derivatives computed by its backward passes either decay rapidly toward the


NNs because the partial derivatives computed by its backward passes either decay rapidly toward the input side of the network, making learning by deep layers extremely slow, or the partial derivatives grow rapidly toward the input side of the network, making learning unstable. Methods for dealing with these problems are largely responsible for many impressive recent results


226 Chapter 9: On-policy Prediction with Approximation achieved by systems that use deep ANNs. Overﬁtting is a problem for any function approximation method that adjusts functions with many degrees of freedom on the basis of limited training data. It is less of a problem for online reinforcement learning that does not rely on limited training sets, but generalizing e↵ectively is still an important issue. Overﬁtting is a problem for ANNs in general, but especially so for deep ANNs because they tend to have very large numbers of weights. Many methods have been developed for reducing overﬁtting. These include stopping training when performance begins to decrease on validation data di↵erent from the training data (cross validation), modifying the objective function to discourage complexity of


the training data (cross validation), modifying the objective function to discourage complexity of the approximation (regularization), and introducing dependencies among the weights to reduce the number of degrees of freedom (e.g., weight sharing). A particularly e↵ective method for reducing overﬁtting by deep ANNs is the dropout method introduced by Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov (2014). During training, units are randomly removed from the network (dropped out) along with their connections. This can be thought of as training a large number of “thinned” networks. Combining the results of these thinned networks at test time is a way to improve generalization performance.


the results of these thinned networks at test time is a way to improve generalization performance. The dropout method eﬃciently approximates this combination by multiplying each outgoing weight of a unit by the probability that that unit was retained during training. Srivastava et al. found that this method signiﬁcantly improves generalization performance. It encourages individual hidden units to learn features that work well with random collections of other features. This increases the versatility of the features formed by the hidden units so that the network does not overly specialize to rarely-occurring cases. Hinton, Osindero, and Teh (2006) took a major step toward solving the problem of training the deep layers of a deep ANN in their work with deep belief networks, layered networks


of training the deep layers of a deep ANN in their work with deep belief networks, layered networks closely related to the deep ANNs discussed here. In their method, the deepest layers are trained one at a time using an unsupervised learning algorithm. Without relying on the overall objective function, unsupervised learning can extract features that capture statistical regularities of the input stream. The deepest layer is trained ﬁrst, then with input provided by this trained layer, the next deepest layer is trained, and so on, until the weights in all, or many, of the network’s layers are set to values that now act as initial values for supervised learning. The network is then ﬁne-tuned by backpropagation with respect to the overall objective function.


g. The network is then ﬁne-tuned by backpropagation with respect to the overall objective function. Studies show that this approach generally works much better than backpropagation with weights initialized with random values. The better performance of networks trained with weights initialized this way could be due to many factors, but one idea is that this method places the network in a region of weight space from which a gradient-based algorithm can make good progress. Batch normalization (Io↵e and Szegedy, 2015) is another technique that makes it easier to train deep ANNs. It has long been known that ANN learning is easier if the network input is normalized, for example, by adjusting each input variable to have zero mean and unit variance.


t is normalized, for example, by adjusting each input variable to have zero mean and unit variance. Batch normalization for training deep ANNs normalizes the output of deep layers before they feed into the following layer. Io↵e and Szegedy (2015) used statistics from subsets, or “mini-batches,” of training examples to normalize these between-layer signals to improve the learning rate of deep ANNs.


9.7. Nonlinear Function Approximation: Artiﬁcial Neural Networks 227 Another technique useful for training deep ANNs is deep residual learning (He, Zhang, Ren, and Sun, 2016). Sometimes it is easier to learn how a function di↵ers from the identity function than to learn the function itself. Then adding this di↵erence, or residual function, to the input produces the desired function. In deep ANNs, a block of layers can be made to learn a residual function simply by adding shortcut, or skip, connections around the block. These connections add the input to the block to its output, and no additional weights are needed. He et al. (2016) evaluated this method using deep convolutional networks with skip connections around every pair of adjacent layers, ﬁnding substantial improvement over networks


skip connections around every pair of adjacent layers, ﬁnding substantial improvement over networks without the skip connections on benchmark image classiﬁcation tasks. Both batch normalization and deep residual learning were used in the reinforcement learning application to the game of Go that we describe in Chapter 16. A type of deep ANN that has proven to be very successful in applications, including impressive reinforcement learning applications (Chapter 16), is the deep convolutional network. This type of network is specialized for processing high-dimensional data arranged in spatial arrays, such as images. It was inspired by how early visual processing works in the brain (LeCun, Bottou, Bengio and Ha↵ner, 1998).


nspired by how early visual processing works in the brain (LeCun, Bottou, Bengio and Ha↵ner, 1998). Because of its special architecture, a deep convolutional network can be trained by backpropagation without resorting to methods like those described above to train the deep layers. Figure 9.15 illustrates the architecture of a deep convolutional network. This instance, from LeCun et al. (1998), was designed to recognize hand-written characters. It consists of alternating convolutional and subsampling layers, followed by several fully connected ﬁnal layers. Each convolutional layer produces a number of feature maps. A feature map is a pattern of activity over an array of units, where each unit performs the same operation on data in its receptive ﬁeld, which is the part of the data it “sees”


performs the same operation on data in its receptive ﬁeld, which is the part of the data it “sees” from the preceding layer (or from the external input in the case of the ﬁrst convolutional layer). The units of a feature map are identical to one another except that their receptive ﬁelds, which are all the same size and shape, are shifted to di↵erent locations on the arrays of incoming data. Units in the same feature map share the same weights. This means that a feature map detects the same feature no matter where it is located in the input Figure 9.15: Deep Convolutional Network. Republished with permission of Proceedings of the IEEE, from Gradient-based learning applied to document recognition, LeCun, Bottou, Bengio, and Ha↵ner, volume 86, 1998; permission conveyed through Copyright Clea


tion, LeCun, Bottou, Bengio, and Ha↵ner, volume 86, 1998; permission conveyed through Copyright Clearance Center, Inc.


228 Chapter 9: On-policy Prediction with Approximation array. In the network in Figure 9.15, for example, the ﬁrst convolutional layer produces 6 feature maps, each consisting of 28 ⇥28 units. Each unit in each feature map has a 5 ⇥5 receptive ﬁeld, and these receptive ﬁelds overlap (in this case by four columns and four rows). Consequently, each of the 6 feature maps is speciﬁed by just 25 adjustable weights. The subsampling layers of a deep convolutional network reduce the spatial resolution of the feature maps. Each feature map in a subsampling layer consists of units that average over a receptive ﬁeld of units in the feature maps of the preceding convolutional layer. For example, each unit in each of the 6 feature maps in the ﬁrst subsampling layer of the network of Figure 9.15 average


it in each of the 6 feature maps in the ﬁrst subsampling layer of the network of Figure 9.15 averages over a 2 ⇥2 non-overlapping receptive ﬁeld over one of the feature maps produced by the ﬁrst convolutional layer, resulting in six 14 ⇥14 feature maps. Subsampling layers reduce the network’s sensitivity to the spatial locations of the features detected, that is, they help make the network’s responses spatially invariant. This is useful because a feature detected at one place in an image is likely to be useful at other places as well. Advances in the design and training of ANNs—of which we have only mentioned a few—all contribute to reinforcement learning. Although current reinforcement learning theory is mostly limited to methods using tabular or linear function approximation methods, the


ning theory is mostly limited to methods using tabular or linear function approximation methods, the impressive performances of notable reinforcement learning applications owe much of their success to nonlinear function approximation by multi-layer ANNs. We discuss several of these applications in Chapter 16. 9.8 Least-Squares TD All the methods we have discussed so far in this chapter have required computation per time step proportional to the number of parameters. With more computation, however, one can do better. In this section we present a method for linear function approximation that is arguably the best that can be done for this case. As we established in Section 9.4 TD(0) with linear function approximation converges asymptotically (for appropriately decreasing step sizes) to the TD


function approximation converges asymptotically (for appropriately decreasing step sizes) to the TD ﬁxed point: wTD = A−1b, where A .= E ⇥ xt(xt −γxt+1)>⇤ and b .= E[Rt+1xt] . Why, one might ask, must we compute this solution iteratively? This is wasteful of data! Could one not do better by computing estimates of A and b, and then directly computing the TD ﬁxed point? The Least-Squares TD algorithm, commonly known as LSTD, does exactly this. It forms the natural estimates bAt .= t−1 X k=0 xk(xk −γxk+1)> + "I and bbt .= t−1 X k=0 Rk+1xk, (9.20)


9.8. Least-Squares TD 229 where I is the identity matrix, and "I, for some small " > 0, ensures that bAt is always invertible. It might seem that these estimates should both be divided by t, and indeed they should; as deﬁned here, these are really estimates of t times A and t times b. However, the extra t factors cancel out when LSTD uses these estimates to estimate the TD ﬁxed point as wt .= bA−1 t bbt. (9.21) This algorithm is the most data eﬃcient form of linear TD(0), but it is also more expensive computationally. Recall that semi-gradient TD(0) requires memory and perstep computation that is only O(d). How complex is LSTD? As it is written above the complexity seems to increase with t, but the two approximations in (9.20) could be implemented incrementally using the techniques we have


but the two approximations in (9.20) could be implemented incrementally using the techniques we have covered earlier (e.g., in Chapter 2) so that they can be done in constant time per step. Even so, the update for bAt would involve an outer product (a column vector times a row vector) and thus would be a matrix update; its computational complexity would be O(d2), and of course the memory required to hold the bAt matrix would be O(d2). A potentially greater problem is that our ﬁnal computation (9.21) uses the inverse of bAt, and the computational complexity of a general inverse computation is O(d3). Fortunately, an inverse of a matrix of our special form—a sum of outer products—can also be updated incrementally with only O(d2) computations, as bA−1 t = ⇣ bAt−1 + xt−1(xt−1 −γxt)>⌘−1 (from (9


dated incrementally with only O(d2) computations, as bA−1 t = ⇣ bAt−1 + xt−1(xt−1 −γxt)>⌘−1 (from (9.20)) = bA−1 t−1 − bA−1 t−1xt−1(xt−1 −γxt)> bA−1 t−1 1 + (xt−1 −γxt)> bA−1 t−1xt−1 , (9.22) for t > 0, with bA0 .= "I. Although the identity (9.22), known as the Sherman-Morrison formula, is superﬁcially complicated, it involves only vector-matrix and vector-vector multiplications and thus is only O(d2). Thus we can store the inverse matrix bA−1 t , maintain it with (9.22), and then use it in (9.21), all with only O(d2) memory and per-step computation. The complete algorithm is given in the box on the next page. Of course, O(d2) is still signiﬁcantly more expensive than the O(d) of semi-gradient TD.


next page. Of course, O(d2) is still signiﬁcantly more expensive than the O(d) of semi-gradient TD. Whether the greater data eﬃciency of LSTD is worth this computational expense depends on how large d is, how important it is to learn quickly, and the expense of other parts of the system. The fact that LSTD requires no step-size parameter is sometimes also touted, but the advantage of this is probably overstated. LSTD does not require a step size, but it does requires "; if " is chosen too small the sequence of inverses can vary wildly, and if " is chosen too large then learning is slowed. In addition, LSTD’s lack of a step-size parameter means that it never forgets. This is sometimes desirable, but it is problematic if the target policy ⇡changes as it does in reinforcement learning and GPI


le, but it is problematic if the target policy ⇡changes as it does in reinforcement learning and GPI. In control applications, LSTD typically has to be combined with some other mechanism to induce forgetting, mooting any initial advantage of not requiring a step-size parameter.


230 Chapter 9: On-policy Prediction with Approximation LSTD for estimating ˆv = w>x(·) ⇡v⇡(O(d2) version) Input: feature representation x : S+ ! Rd such that x(terminal) = 0 Algorithm parameter: small " > 0 d A−1 "−1I A d ⇥d matrix bb 0 A d-dimensional vector Loop for each episode: Initialize S; x x(S) Loop for each step of episode: Choose and take action A ⇠⇡(·|S), observe R, S0; x0 x(S0) v d A−1> (x −γx0) d A−1 d A−1 − # d A−1x $ v>/ # 1 + v>x $ bb bb + Rx w d A−1bb S S0; x x0 until S0 is terminal 9.9 Memory-based Function Approximation So far we have discussed the parametric approach to approximating value functions. In this approach, a learning algorithm adjusts the parameters of a functional form intended to approximate the value function over a problem’s entire state space.


f a functional form intended to approximate the value function over a problem’s entire state space. Each update, s 7! g, is a training example used by the learning algorithm to change the parameters with the aim of reducing the approximation error. After the update, the training example can be discarded (although it might be saved to be used again). When an approximate value of a state (which we will call the query state) is needed, the function is simply evaluated at that state using the latest parameters produced by the learning algorithm. Memory-based function approximation methods are very di↵erent. They simply save training examples in memory as they arrive (or at least save a subset of the examples) without updating any parameters.


memory as they arrive (or at least save a subset of the examples) without updating any parameters. Then, whenever a query state’s value estimate is needed, a set of examples is retrieved from memory and used to compute a value estimate for the query state. This approach is sometimes called lazy learning because processing training examples is postponed until the system is queried to provide an output. Memory-based function approximation methods are prime examples of nonparametric methods. Unlike parametric methods, the approximating function’s form is not limited to a ﬁxed parameterized class of functions, such as linear functions or polynomials, but is instead determined by the training examples themselves, together with some means for combining them to output estimated values for query


amples themselves, together with some means for combining them to output estimated values for query states. As more training examples accumulate in memory, one expects nonparametric methods to produce increasingly accurate approximations of any target function.


9.9. Memory-based Function Approximation 231 There are many di↵erent memory-based methods depending on how the stored training examples are selected and how they are used to respond to a query. Here, we focus on local-learning methods that approximate a value function only locally in the neighborhood of the current query state. These methods retrieve a set of training examples from memory whose states are judged to be the most relevant to the query state, where relevance usually depends on the distance between states: the closer a training example’s state is to the query state, the more relevant it is considered to be, where distance can be deﬁned in many di↵erent ways. After the query state is given a value, the local approximation is discarded.


n many di↵erent ways. After the query state is given a value, the local approximation is discarded. The simplest example of the memory-based approach is the nearest neighbor method, which simply ﬁnds the example in memory whose state is closest to the query state and returns that example’s value as the approximate value of the query state. In other words, if the query state is s, and s0 7! g is the example in memory in which s0 is the closest state to s, then g is returned as the approximate value of s. Slightly more complicated are weighted average methods that retrieve a set of nearest neighbor examples and return a weighted average of their target values, where the weights generally decrease with increasing distance between their states and the query state.


e the weights generally decrease with increasing distance between their states and the query state. Locally weighted regression is similar, but it ﬁts a surface to the values of a set of nearest states by means of a parametric approximation method that minimizes a weighted error measure like (9.1), where the weights depend on distances from the query state. The value returned is the evaluation of the locally-ﬁtted surface at the query state, after which the local approximation surface is discarded. Being nonparametric, memory-based methods have the advantage over parametric methods of not limiting approximations to pre-speciﬁed functional forms. This allows accuracy to improve as more data accumulates.


mations to pre-speciﬁed functional forms. This allows accuracy to improve as more data accumulates. Memory-based local approximation methods have other properties that make them well suited for reinforcement learning. Because trajectory sampling is of such importance in reinforcement learning, as discussed in Section 8.6, memory-based local methods can focus function approximation on local neighborhoods of states (or state–action pairs) visited in real or simulated trajectories. There may be no need for global approximation because many areas of the state space will never (or almost never) be reached. In addition, memory-based methods allow an agent’s experience to have a relatively immediate a↵ect on value estimates in the neighborhood of the current state, in contrast with a parametric m


e a↵ect on value estimates in the neighborhood of the current state, in contrast with a parametric method’s need to incrementally adjust parameters of a global approximation. Avoiding global approximation is also a way to address the curse of dimensionality. For example, for a state space with k dimensions, a tabular method storing a global approximation requires memory exponential in k. On the other hand, in storing examples for a memory-based method, each example requires memory proportional to k, and the memory required to store, say, n examples is linear in n. Nothing is exponential in k or n. Of course, the critical remaining issue is whether a memory-based method can answer queries quickly enough to be useful to an agent.


issue is whether a memory-based method can answer queries quickly enough to be useful to an agent. A related concern is how speed degrades as the size of the memory grows. Finding nearest neighbors in a large database can take too long to be practical in many applications.


232 Chapter 9: On-policy Prediction with Approximation Proponents of memory-based methods have developed ways to accelerate the nearest neighbor search. Using parallel computers or special purpose hardware is one approach; another is the use of special multi-dimensional data structures to store the training data. One data structure studied for this application is the k-d tree (short for k-dimensional tree), which recursively splits a k-dimensional space into regions arranged as nodes of a binary tree. Depending on the amount of data and how it is distributed over the state space, nearest-neighbor search using k-d trees can quickly eliminate large regions of the space in the search for neighbors, making the searches feasible in some problems where naive searches would take too long.


neighbors, making the searches feasible in some problems where naive searches would take too long. Locally weighted regression additionally requires fast ways to do the local regression computations which have to be repeated to answer each query. Researchers have developed many ways to address these problems, including methods for forgetting entries in order to keep the size of the database within bounds. The Bibliographic and Historical Comments section at the end of this chapter points to some of the relevant literature, including a selection of papers describing applications of memory-based learning to reinforcement learning. 9.10 Kernel-based Function Approximation Memory-based methods such as the weighted average and locally weighted regression methods described above depend on assig


such as the weighted average and locally weighted regression methods described above depend on assigning weights to examples s0 7! g in the database depending on the distance between s0 and a query states s. The function that assigns these weights is called a kernel function, or simply a kernel. In the weighted average and locally weighted regressions methods, for example, a kernel function k : R ! R assigns weights to distances between states. More generally, weights do not have to depend on distances; they can depend on some other measure of similarity between states. In this case, k : S ⇥S ! R, so that k(s, s0) is the weight given to data about s0 in its inﬂuence on answering queries about s.


so that k(s, s0) is the weight given to data about s0 in its inﬂuence on answering queries about s. Viewed slightly di↵erently, k(s, s0) is a measure of the strength of generalization from s0 to s. Kernel functions numerically express how relevant knowledge about any state is to any other state. As an example, the strengths of generalization for tile coding shown in Figure 9.11 correspond to di↵erent kernel functions resulting from uniform and asymmetrical tile o↵sets. Although tile coding does not explicitly use a kernel function in its operation, it generalizes according to one. In fact, as we discuss more below, the strength of generalization resulting from linear parametric function approximation can always be described by a kernel function.


sulting from linear parametric function approximation can always be described by a kernel function. Kernel regression is the memory-based method that computes a kernel weighted average of the targets of all examples stored in memory, assigning the result to the query state. If D is the set of stored examples, and g(s0) denotes the target for state s0 in a stored example, then kernel regression approximates the target function, in this case a value function depending on D, as ˆv(s,D) = X s02D k(s, s0)g(s0). (9.23)


9.10. Kernel-based Function Approximation 233 The weighted average method described above is a special case in which k(s, s0) is non-zero only when s and s0 are close to one another so that the sum need not be computed over all of D. A common kernel is the Gaussian radial basis function (RBF) used in RBF function approximation as described in Section 9.5.5. In the method described there, RBFs are features whose centers and widths are either ﬁxed from the start, with centers presumably concentrated in areas where many examples are expected to fall, or are adjusted in some way during learning. Barring methods that adjust centers and widths, this is a linear parametric method whose parameters are the weights of each RBF, which are typically learned by stochastic gradient, or semi-gradient, de


re the weights of each RBF, which are typically learned by stochastic gradient, or semi-gradient, descent. The form of the approximation is a linear combination of the pre-determined RBFs. Kernel regression with an RBF kernel di↵ers from this in two ways. First, it is memory-based: the RBFs are centered on the states of the stored examples. Second, it is nonparametric: there are no parameters to learn; the response to a query is given by (9.23). Of course, many issues have to be addressed for practical implementation of kernel regression, issues that are beyond the scope of our brief discussion. However, it turns out that any linear parametric regression method like those we described in Section 9.4, with states represented by feature vectors x(s) = (x1(s), x2(s), . . .


we described in Section 9.4, with states represented by feature vectors x(s) = (x1(s), x2(s), . . . , xd(s))>, can be recast as kernel regression where k(s, s0) is the inner product of the feature vector representations of s and s0; that is k(s, s0) = x(s)>x(s0). (9.24) Kernel regression with this kernel function produces the same approximation that a linear parametric method would if it used these feature vectors and learned with the same training data. We skip the mathematical justiﬁcation for this, which can be found in any modern machine learning text, such as Bishop (2006), and simply point out an important implication. Instead of constructing features for linear parametric function approximators, one can instead construct kernel functions directly without referring at all to feature


roximators, one can instead construct kernel functions directly without referring at all to feature vectors. Not all kernel functions can be expressed as inner products of feature vectors as in (9.24), but a kernel function that can be expressed like this can o↵er signiﬁcant advantages over the equivalent parametric method. For many sets of feature vectors, (9.24) has a compact functional form that can be evaluated without any computation taking place in the d-dimensional feature space. In these cases, kernel regression is much less complex than directly using a linear parametric method with states represented by these feature vectors. This is the so-called “kernel trick” that allows e↵ectively working in the high-dimension of an expansive feature space while actually working only with the


ely working in the high-dimension of an expansive feature space while actually working only with the set of stored training examples. The kernel trick is the basis of many machine learning methods, and researchers have shown how it can sometimes beneﬁt reinforcement learning.


234 Chapter 9: On-policy Prediction with Approximation 9.11 Looking Deeper at On-policy Learning: Interest and Emphasis The algorithms we have considered so far in this chapter have treated all the states encountered equally, as if they were all equally important. In some cases, however, we are more interested in some states than others. In discounted episodic problems, for example, we may be more interested in accurately valuing early states in the episode than in later states where discounting may have made the rewards much less important to the value of the start state. Or, if an action-value function is being learned, it may be less important to accurately value poor actions whose value is much less than the greedy action.


be less important to accurately value poor actions whose value is much less than the greedy action. Function approximation resources are always limited, and if they were used in a more targeted way, then performance could be improved. One reason we have treated all states encountered equally is that then we are updating according to the on-policy distribution, for which stronger theoretical results are available for semi-gradient methods. Recall that the on-policy distribution was deﬁned as the distribution of states encountered in an MDP while following the target policy. Now we will generalize this concept signiﬁcantly. Rather than having one on-policy distribution for the MDP, we will have many.


concept signiﬁcantly. Rather than having one on-policy distribution for the MDP, we will have many. All of them will have in common that they are a distribution of states encountered in trajectories while following the target policy, but they will vary in how the trajectories are, in a sense, initiated. We now introduce some new concepts. First we introduce a non-negative scalar measure, a random variable It called interest, indicating the degree to which we are interested in accurately valuing the state (or state–action pair) at time t. If we don’t care at all about the state, then the interest should be zero; if we fully care, it might be one, though it is formally allowed to take any non-negative value.


o; if we fully care, it might be one, though it is formally allowed to take any non-negative value. The interest can be set in any causal way; for example, it may depend on the trajectory up to time t or the learned parameters at time t. The distribution µ in the VE (9.1) is then deﬁned as the distribution of states encountered while following the target policy, weighted by the interest. Second, we introduce another non-negative scalar random variable, the emphasis Mt. This scalar multiplies the learning update and thus emphasizes or de-emphasizes the learning done at time t. The general n-step learning rule, replacing (9.15), is wt+n .= wt+n−1 +↵Mt [Gt:t+n −ˆv(St,wt+n−1)] rˆv(St,wt+n−1), 0 t < T, (9.25) with the n-step return given by (9.16) and the emphasis determined recursively from t


t < T, (9.25) with the n-step return given by (9.16) and the emphasis determined recursively from the interest by: Mt = It + γnMt−n, 0 t < T, (9.26) with Mt .= 0, for all t < 0. These equations are taken to include the Monte Carlo case, for which Gt:t+n = Gt, all the updates are made at end of the episode, n = T −t, and Mt = It.


9.11. Looking Deeper at On-policy Learning: Interest and Emphasis 235 Example 9.4 illustrates how interest and emphasis can result in more accurate value estimates. Example 9.4: Interest and Emphasis To see the potential beneﬁts of using interest and emphasis, consider the four-state Markov reward process shown below: +1 +1 +1 +1 v⇡= 4 v⇡= 3 v⇡= 2 v⇡= 1 i = 1 i = 0 i = 0 i = 0 w1 w1 w2 w2 Episodes start in the leftmost state, then transition one state to the right, with a reward of +1, on each step until the terminal state is reached. The true value of the ﬁrst state is thus 4, of the second state 3, and so on as shown below each state. These are the true values; the estimated values can only approximate these because they are constrained by the parameterization.


e estimated values can only approximate these because they are constrained by the parameterization. There are two components to the parameter vector w = (w1, w2)>, and the parameterization is as written inside each state. The estimated values of the ﬁrst two states are given by w1 alone and thus must be the same even though their true values are di↵erent. Similarly, the estimated values of the third and fourth states are given by w2 alone and must be the same even though their true values are di↵erent. Suppose that we are interested in accurately valuing only the leftmost state; we assign it an interest of 1 while all the other states are assigned an interest of 0, as indicated above the states. First consider applying gradient Monte Carlo algorithms to this problem.


ndicated above the states. First consider applying gradient Monte Carlo algorithms to this problem. The algorithms presented earlier in this chapter that do not take into account interest and emphasis (in (9.7) and the box on page 202) will converge (for decreasing step sizes) to the parameter vector w1 = (3.5, 1.5), which gives the ﬁrst state—the only one we are interested in—a value of 3.5 (i.e., intermediate between the true values of the ﬁrst and second states). The methods presented in this section that do use interest and emphasis, on the other hand, will learn the value of the ﬁrst state exactly correctly; w1 will converge to 4 while w2 will never be updated because the emphasis is zero in all states save the leftmost. Now consider applying two-step semi-gradient TD methods.


s is zero in all states save the leftmost. Now consider applying two-step semi-gradient TD methods. The methods from earlier in this chapter without interest and emphasis (in (9.15) and (9.16) and the box on page 209) will again converge to w1 = (3.5, 1.5), while the methods with interest and emphasis converge to w1 = (4, 2). The latter produces the exactly correct values for the ﬁrst state and for the third state (which the ﬁrst state bootstraps from) while never making any updates corresponding to the second or fourth states.


236 Chapter 9: On-policy Prediction with Approximation 9.12 Summary Reinforcement learning systems must be capable of generalization if they are to be applicable to artiﬁcial intelligence or to large engineering applications. To achieve this, any of a broad range of existing methods for supervised-learning function approximation can be used simply by treating each update as a training example. Perhaps the most suitable supervised learning methods are those using parameterized function approximation, in which the policy is parameterized by a weight vector w. Although the weight vector has many components, the state space is much larger still, and we must settle for an approximate solution. We deﬁned the mean square value error, VE(w), as a measure of the error in the values v⇡w(s) for a wei


deﬁned the mean square value error, VE(w), as a measure of the error in the values v⇡w(s) for a weight vector w under the on-policy distribution, µ. The VE gives us a clear way to rank di↵erent value-function approximations in the on-policy case. To ﬁnd a good weight vector, the most popular methods are variations of stochastic gradient descent (SGD). In this chapter we have focused on the on-policy case with a ﬁxed policy, also known as policy evaluation or prediction; a natural learning algorithm for this case is n-step semi-gradient TD, which includes gradient Monte Carlo and semi-gradient TD(0) algorithms as the special cases when n=1 and n=1 respectively. Semi-gradient TD methods are not true gradient methods.


pecial cases when n=1 and n=1 respectively. Semi-gradient TD methods are not true gradient methods. In such bootstrapping methods (including DP), the weight vector appears in the update target, yet this is not taken into account in computing the gradient—thus they are semi-gradient methods. As such, they cannot rely on classical SGD results. Nevertheless, good results can be obtained for semi-gradient methods in the special case of linear function approximation, in which the value estimates are sums of features times corresponding weights. The linear case is the most well understood theoretically and works well in practice when provided with appropriate features. Choosing the features is one of the most important ways of adding prior domain knowledge to reinforcement learning systems.


one of the most important ways of adding prior domain knowledge to reinforcement learning systems. They can be chosen as polynomials, but this case generalizes poorly in the online learning setting typically considered in reinforcement learning. Better is to choose features according the Fourier basis, or according to some form of coarse coding with sparse overlapping receptive ﬁelds. Tile coding is a form of coarse coding that is particularly computationally eﬃcient and ﬂexible. Radial basis functions are useful for one- or two-dimensional tasks in which a smoothly varying response is important. LSTD is the most data-eﬃcient linear TD prediction method, but requires computation proportional to the square of the number of weights, whereas all the other methods are of complexity linear in


l to the square of the number of weights, whereas all the other methods are of complexity linear in the number of weights. Nonlinear methods include artiﬁcial neural networks trained by backpropagation and variations of SGD; these methods have become very popular in recent years under the name deep reinforcement learning. Linear semi-gradient n-step TD is guaranteed to converge under standard conditions, for all n, to a VE that is within a bound of the optimal error (achieved asymptotically by Monte Carlo methods). This bound is always tighter for higher n and approaches zero as n ! 1. However, in practice very high n results in very slow learning, and some degree of bootstrapping (n < 1) is usually preferable, just as we saw in comparisons of tabular n-step methods in Chapter 7 and in com


usually preferable, just as we saw in comparisons of tabular n-step methods in Chapter 7 and in comparisons of tabular TD and Monte Carlo methods in Chapter 6.


9.12. Summary 237 Exercise 9.7 One of the simplest artiﬁcial neural networks consists of a single semi-linear unit with a logistic nonlinearity. The need to handle approximate value functions of this form is common in games that end with either a win or a loss, in which case the value of a state can be interpreted as the probability of winning. Derive the learning algorithm for this case, from (9.7), such that no gradient notation appears. ⇤Exercise 9.8 Arguably, the squared error used to derive (9.7) is inappropriate for the case treated in the preceding exercise, and the right error measure is the cross-entropy loss (which you can ﬁnd on Wikipedia). Repeat the derivation in Section 9.3, using the cross-entropy loss instead of the squared error in (9.4), all the way to an explicit form wi


ing the cross-entropy loss instead of the squared error in (9.4), all the way to an explicit form with no gradient or logarithm notation in it. Is your ﬁnal form more complex, or simpler, than that you obtained in the preceding exercise? Bibliographical and Historical Remarks Generalization and function approximation have always been an integral part of reinforcement learning. Bertsekas and Tsitsiklis (1996), Bertsekas (2012), and Sugiyama et al. (2013) present the state of the art in function approximation in reinforcement learning. Some of the early work with function approximation in reinforcement learning is discussed at the end of this section. 9.3 Gradient-descent methods for minimizing mean square error in supervised learning are well known.


.3 Gradient-descent methods for minimizing mean square error in supervised learning are well known. Widrow and Ho↵(1960) introduced the least-mean-square (LMS) algorithm, which is the prototypical incremental gradient-descent algorithm. Details of this and related algorithms are provided in many texts (e.g., Widrow and Stearns, 1985; Bishop, 1995; Duda and Hart, 1973). Semi-gradient TD(0) was ﬁrst explored by Sutton (1984, 1988), as part of the linear TD(λ) algorithm that we will treat in Chapter 12. The term “semi-gradient” to describe these bootstrapping methods is new to the second edition of this book. The earliest use of state aggregation in reinforcement learning may have been Michie and Chambers’s BOXES system (1968).


tate aggregation in reinforcement learning may have been Michie and Chambers’s BOXES system (1968). The theory of state aggregation in reinforcement learning has been developed by Singh, Jaakkola, and Jordan (1995) and Tsitsiklis and Van Roy (1996). State aggregation has been used in dynamic programming from its earliest days (e.g., Bellman, 1957a). 9.4 Sutton (1988) proved convergence of linear TD(0) in the mean to the minimal VE solution for the case in which the feature vectors, {x(s) : s 2 S}, are linearly independent. Convergence with probability 1 was proved by several researchers at about the same time (Peng, 1993; Dayan and Sejnowski, 1994; Tsitsiklis, 1994; Gurvits, Lin, and Hanson, 1994). In addition, Jaakkola, Jordan, and Singh (1994) proved convergence under online updating.


n, 1994). In addition, Jaakkola, Jordan, and Singh (1994) proved convergence under online updating. All of these results assumed linearly independent feature vectors, which implies at least as many components to wt as there are states. Convergence for the more important case of general (dependent) feature vectors was ﬁrst shown by Dayan (1992). A signiﬁcant


238 Chapter 9: On-policy Prediction with Approximation generalization and strengthening of Dayan’s result was proved by Tsitsiklis and Van Roy (1997). They proved the main result presented in this section, the bound on the asymptotic error of linear bootstrapping methods. 9.5 Our presentation of the range of possibilities for linear function approximation is based on that by Barto (1990). 9.5.2 Konidaris, Osentoski, and Thomas (2011) introduced the Fourier basis in a simple form suitable for reinforcement learning problems with multi-dimensional continuous state spaces and functions that do not have to be periodic. 9.5.3 The term coarse coding is due to Hinton (1984), and our Figure 9.6 is based on one of his ﬁgures.


3 The term coarse coding is due to Hinton (1984), and our Figure 9.6 is based on one of his ﬁgures. Waltz and Fu (1965) provide an early example of this type of function approximation in a reinforcement learning system. 9.5.4 Tile coding, including hashing, was introduced by Albus (1971, 1981). He described it in terms of his “cerebellar model articulator controller,” or CMAC, as tile coding is sometimes known in the literature. The term “tile coding” was new to the ﬁrst edition of this book, though the idea of describing CMAC in these terms is taken from Watkins (1989). Tile coding has been used in many reinforcement learning systems (e.g., Shewchuk and Dean, 1990; Lin and Kim, 1991; Miller, Scalera, and Kim, 1994; Sofge and White, 1992; Tham, 1994; Sutton, 1996; Watkins, 1989) as well as


, Scalera, and Kim, 1994; Sofge and White, 1992; Tham, 1994; Sutton, 1996; Watkins, 1989) as well as in other types of learning control systems (e.g., Kraft and Campagna, 1990; Kraft, Miller, and Dietz, 1992). This section draws heavily on the work of Miller and Glanz (1996). General software for tile coding is available in several languages (e.g., see http://incompleteideas.net/tiles/tiles3.html). 9.5.5 Function approximation using radial basis functions has received wide attention ever since being related to ANNs by Broomhead and Lowe (1988). Powell (1987) reviewed earlier uses of RBFs, and Poggio and Girosi (1989, 1990) extensively developed and applied this approach. 9.6 Automatic methods for adapting the step-size parameter include RMSprop (Tieleman and Hinton, 2012), Adam (Kingma and


s for adapting the step-size parameter include RMSprop (Tieleman and Hinton, 2012), Adam (Kingma and Ba, 2015), stochastic meta-descent methods such as Delta-Bar-Delta (Jacobs, 1988), its incremental generalization (Sutton, 1992b, c; Mahmood et al., 2012), and nonlinear generalizations (Schraudolph, 1999, 2002). Methods explicitly designed for reinforcement learning include AlphaBound (Dabney and Barto, 2012), SID and NOSID (Dabney, 2014), TIDBD (Kearney et al., in preparation) and the application of stochastic meta-descent to policy gradient learning (Schraudolph, Yu, and Aberdeen, 2006). 9.7 The introduction of the threshold logic unit as an abstract model neuron by McCulloch and Pitts (1943) was the beginning of ANNs.