tic spike arriving shortly after the postsynaptic neuron ﬁres, then the strength of the synapse decreases. STDP is a type of Hebbian plasticity that takes the activation time of a neuron into account, which is one of the ingredients needed for actor-like learning.


402 Chapter 15: Neuroscience The discovery of STDP has led neuroscientists to investigate the possibility of a threefactor form of STDP in which neuromodulatory input must follow appropriately-timed pre- and postsynaptic spikes. This form of synaptic plasticity, called reward-modulated STDP, is much like the actor learning rule discussed here. Synaptic changes that would be produced by regular STDP only occur if there is neuromodulatory input within a time window after a presynaptic spike is closely followed by a postsynaptic spike. Evidence is accumulating that reward-modulated STDP occurs at the spines of medium spiny neurons of the dorsal striatum, with dopamine providing the neuromodulatory factor— the sites where actor learning takes place in the hypothetical neural implementation of


ory factor— the sites where actor learning takes place in the hypothetical neural implementation of an actor–critic algorithm illustrated in Figure 15.5b. Experiments have demonstrated reward-modulated STDP in which lasting changes in the eﬃcacies of corticostriatal synapses occur only if a neuromodulatory pulse arrives within a time window that can last up to 10 seconds after a presynaptic spike is closely followed by a postsynaptic spike (Yagishita et al. 2014). Although the evidence is indirect, these experiments point to the existence of contingent eligibility traces having prolonged time courses. The molecular mechanisms producing these traces, as well as the much shorter traces that likely underly STDP, are not yet understood, but research focusing on time-dependent and neuromodulato


kely underly STDP, are not yet understood, but research focusing on time-dependent and neuromodulator-dependent synaptic plasticity is continuing. The neuron-like actor unit that we have described here, with its Law-of-E↵ect-style learning rule, appeared in somewhat simpler form in the actor–critic network of Barto et al. (1983). That network was inspired by the “hedonistic neuron” hypothesis proposed by physiologist A. H. Klopf (1972, 1982). Not all the details of Klopf’s hypothesis are consistent with what has been learned about synaptic plasticity, but the discovery of STDP and the growing evidence for a reward-modulated form of STDP suggest that Klopf’s ideas may not have been far o↵the mark. We discuss Klopf’s hedonistic neuron hypothesis next.


opf’s ideas may not have been far o↵the mark. We discuss Klopf’s hedonistic neuron hypothesis next. 15.9 Hedonistic Neurons In his hedonistic neuron hypothesis, Klopf (1972, 1982) conjectured that individual neurons seek to maximize the di↵erence between synaptic input treated as rewarding and synaptic input treated as punishing by adjusting the eﬃcacies of their synapses on the basis of rewarding or punishing consequences of their own action potentials. In other words, individual neurons can be trained with response-contingent reinforcement like an animal can be trained in an instrumental conditioning task. His hypothesis included the idea that rewards and punishments are conveyed to a neuron via the same synaptic input that excites or inhibits the neuron’s spike-generating activity.


neuron via the same synaptic input that excites or inhibits the neuron’s spike-generating activity. (Had Klopf known what we know today about neuromodulatory systems, he might have assigned the reinforcing role to neuromodulatory input, but he wanted to avoid any centralized source of training information.) Synaptically-local traces of past pre- and postsynaptic activity had the key function in Klopf’s hypothesis of making synapses eligible—the term he introduced—for modiﬁcation by later reward or punishment. He conjectured that these traces are implemented by molecular mechanisms local to each synapse and therefore di↵erent from the electrical activity of both the pre- and the postsynaptic neurons. In


15.9. Hedonistic Neurons 403 the Bibliographical and Historical Remarks section of this chapter we bring attention to some similar proposals made by others. Klopf speciﬁcally conjectured that synaptic eﬃcacies change in the following way. When a neuron ﬁres an action potential, all of its synapses that were active in contributing to that action potential become eligible to undergo changes in their eﬃcacies. If the action potential is followed within an appropriate time period by an increase of reward, the eﬃcacies of all the eligible synapses increase. Symmetrically, if the action potential is followed within an appropriate time period by an increase of punishment, the eﬃcacies of eligible synapses decrease.


n appropriate time period by an increase of punishment, the eﬃcacies of eligible synapses decrease. This is implemented by triggering an eligibility trace at a synapse upon a coincidence of presynaptic and postsynaptic activity (or more exactly, upon pairing of presynaptic activity with the postsynaptic activity that that presynaptic activity participates in causing)—what we call a contingent eligibility trace. This is essentially the three-factor learning rule of an actor unit described in the previous section. The shape and time course of an eligibility trace in Klopf’s theory reﬂects the durations of the many feedback loops in which the neuron is embedded, some of which lie entirely within the brain and body of the organism, while others extend out through the organism’s external enviro


n the brain and body of the organism, while others extend out through the organism’s external environment as mediated by its motor and sensory systems. His idea was that the shape of a synaptic eligibility trace is like a histogram of the durations of the feedback loops in which the neuron is embedded. The peak of an eligibility trace would then occur at the duration of the most prevalent feedback loops in which that neuron participates. The eligibility traces used by algorithms described in this book are simpliﬁed versions of Klopf’s original idea, being exponentially (or geometrically) decreasing functions controlled by the parameters λ and γ. This simpliﬁes simulations as well as theory, but we regard these simple eligibility traces as a placeholders for traces closer to Klopf’s origina


but we regard these simple eligibility traces as a placeholders for traces closer to Klopf’s original conception, which would have computational advantages in complex reinforcement learning systems by reﬁning the credit-assignment process. Klopf’s hedonistic neuron hypothesis is not as implausible as it may at ﬁrst appear. A well-studied example of a single cell that seeks some stimuli and avoids others is the bacterium Escherichia coli. The movement of this single-cell organism is inﬂuenced by chemical stimuli in its environment, behavior known as chemotaxis. It swims in its liquid environment by rotating hairlike structures called ﬂagella attached to its surface. (Yes, it rotates them!) Molecules in the bacterium’s environment bind to receptors on its surface.


(Yes, it rotates them!) Molecules in the bacterium’s environment bind to receptors on its surface. Binding events modulate the frequency with which the bacterium reverses ﬂagellar rotation. Each reversal causes the bacterium to tumble in place and then head o↵in a random new direction. A little chemical memory and computation causes the frequency of ﬂagellar reversal to decrease when the bacterium swims toward higher concentrations of molecules it needs to survive (attractants) and increase when the bacterium swims toward higher concentrations of molecules that are harmful (repellants). The result is that the bacterium tends to persist in swimming up attractant gradients and tends to avoid swimming up repellant gradients. The chemotactic behavior just described is called klinokinesis.


id swimming up repellant gradients. The chemotactic behavior just described is called klinokinesis. It is a kind of trialand-error behavior, although it is unlikely that learning is involved: the bacterium needs a modicum of short-term memory to detect molecular concentration gradients, but it probably does not maintain long-term memories. Artiﬁcial intelligence pioneer Oliver


404 Chapter 15: Neuroscience Selfridge called this strategy “run and twiddle,” pointing out its utility as a basic adaptive strategy: “keep going in the same way if things are getting better, and otherwise move around” (Selfridge, 1978, 1984). Similarly, one might think of a neuron “swimming” (not literally of course) in a medium composed of the complex collection of feedback loops in which it is embedded, acting to obtain one type of input signal and to avoid others. Unlike the bacterium, however, the neuron’s synaptic strengths retain information about its past trial-and-error behavior. If this view of the behavior of a neuron (or just one type of neuron) is plausible, then the closed-loop nature of how the neuron interacts with its environment is important for understanding its behavior


nature of how the neuron interacts with its environment is important for understanding its behavior, where the neuron’s environment consists of the rest of the animal together with the environment with which the animal as a whole interacts. Klopf’s hedonistic neuron hypothesis extended beyond the idea that individual neurons are reinforcement learning agents. He argued that many aspects of intelligent behavior can be understood as the result of the collective behavior of a population of self-interested hedonistic neurons interacting with one another in an immense society or economic system making up an animal’s nervous system. Whether or not this view of nervous systems is useful, the collective behavior of reinforcement learning agents has implications for neuroscience.


useful, the collective behavior of reinforcement learning agents has implications for neuroscience. We take up this subject next. 15.10 Collective Reinforcement Learning The behavior of populations of reinforcement learning agents is deeply relevant to the study of social and economic systems, and if anything like Klopf’s hedonistic neuron hypothesis is correct, to neuroscience as well. The hypothesis described above about how an actor–critic algorithm might be implemented in the brain only narrowly addresses the implications of the fact that the dorsal and ventral subdivisions of the striatum, the respective locations of the actor and the critic according to the hypothesis, each contain millions of medium spiny neurons whose synapses undergo change modulated by phasic bursts of dopamine n


lions of medium spiny neurons whose synapses undergo change modulated by phasic bursts of dopamine neuron activity. The actor in Figure 15.5a is a single-layer network of k actor units. The actions produced by this network are vectors (A1, A2, · · · , Ak)> presumed to drive the animal’s behavior. Changes in the eﬃcacies of the synapses of all of these units depend on the reinforcement signal δ. Because actor units attempt to make δ as large as possible, δ e↵ectively acts as a reward signal for them (so in this case reinforcement is the same as reward). Thus, each actor unit is itself a reinforcement learning agent—a hedonistic neuron if you will. Now, to make the situation as simple as possible, assume that each of these units receives the same reward signal at the same time (although, as


ble, assume that each of these units receives the same reward signal at the same time (although, as indicated above, the assumption that dopamine is released at all the corticostriatal synapses under the same conditions and at the same times is likely an oversimpliﬁcation). What can reinforcement learning theory tell us about what happens when all members of a population of reinforcement learning agents learn according to a common reward signal? The ﬁeld of multi-agent reinforcement learning considers many aspects of learning by populations of reinforcement learning agents. Although this ﬁeld is beyond the scope of this book, we believe that some of its basic concepts and results are relevant to thinking


15.10. Collective Reinforcement Learning 405 about the brain’s di↵use neuromodulatory systems. In multi-agent reinforcement learning (and in game theory), the scenario in which all the agents try to maximize a common reward signal that they simultaneously receive is known as a cooperative game or a team problem. What makes a team problem interesting and challenging is that the common reward signal sent to each agent evaluates the pattern of activity produced by the entire population, that is, it evaluates the collective action of the team members. This means that any individual agent has only limited ability to a↵ect the reward signal because any single agent contributes just one component of the collective action evaluated by the common reward signal.


gent contributes just one component of the collective action evaluated by the common reward signal. E↵ective learning in this scenario requires addressing a structural credit assignment problem: which team members, or groups of team members, deserve credit for a favorable reward signal, or blame for an unfavorable reward signal? It is a cooperative game, or a team problem, because the agents are united in seeking to increase the same reward signal: there are no conﬂicts of interest among the agents. The scenario would be a competitive game if di↵erent agents receive di↵erent reward signals, where each reward signal again evaluates the collective action of the population, and the objective of each agent is to increase its own reward signal.


ive action of the population, and the objective of each agent is to increase its own reward signal. In this case there might be conﬂicts of interest among the agents, meaning that actions that are good for some agents are bad for others. Even deciding what the best collective action should be is a non-trivial aspect of game theory. This competitive setting might be relevant to neuroscience too (for example, to account for heterogeneity of dopamine neuron activity), but here we focus only on the cooperative, or team, case. How can each reinforcement learning agent in a team learn to “do the right thing” so that the collective action of the team is highly rewarded? An interesting result is that if each agent can learn e↵ectively despite its reward signal being corrupted by a large amount of


t if each agent can learn e↵ectively despite its reward signal being corrupted by a large amount of noise, and despite its lack of access to complete state information, then the population as a whole will learn to produce collective actions that improve as evaluated by the common reward signal, even when the agents cannot communicate with one another. Each agent faces its own reinforcement learning task in which its inﬂuence on the reward signal is deeply buried in the noise created by the inﬂuences of other agents. In fact, for any agent, all the other agents are part of its environment because its input, both the part conveying state information and the reward part, depends on how all the other agents are behaving.


conveying state information and the reward part, depends on how all the other agents are behaving. Furthermore, lacking access to the actions of the other agents, indeed lacking access to the parameters determining their policies, each agent can only partially observe the state of its environment. This makes each team member’s learning task very diﬃcult, but if each uses a reinforcement learning algorithm able to increase a reward signal even under these diﬃcult conditions, teams of reinforcement learning agents can learn to produce collective actions that improve over time as evaluated by the team’s common reward signal. If the team members are neuron-like units, then each unit has to have the goal of increasing the amount of reward it receives over time, as the actor unit does that we d


the goal of increasing the amount of reward it receives over time, as the actor unit does that we described in Section 15.8. Each unit’s learning algorithm has to have two essential features. First, it has to use contingent eligibility traces. Recall that a contingent eligibility trace, in neural terms, is initiated (or increased) at a synapse when its presynaptic input


406 Chapter 15: Neuroscience participates in causing the postsynaptic neuron to ﬁre. A non-contingent eligibility trace, in contrast, is initiated or increased by presynaptic input independently of what the postsynaptic neuron does. As explained in Section 15.8, by keeping information about what actions were taken in what states, contingent eligibility traces allow credit for reward, or blame for punishment, to be apportioned to an agent’s policy parameters according to the contribution the values of these parameters made in determining the agent’s action. By similar reasoning, a team member must remember its recent action so that it can either increase or decrease the likelihood of producing that action according to the reward signal that is subsequently received.


e likelihood of producing that action according to the reward signal that is subsequently received. The action component of a contingent eligibility trace implements this action memory. Because of the complexity of the learning task, however, contingent eligibility is merely a preliminary step in the credit assignment process: the relationship between a single team member’s action and changes in the team’s reward signal is a statistical correlation that has to be estimated over many trials. Contingent eligibility is an essential but preliminary step in this process. Learning with non-contingent eligibility traces does not work at all in the team setting because it does not provide a way to correlate actions with consequent changes in the reward signal.


ecause it does not provide a way to correlate actions with consequent changes in the reward signal. Non-contingent eligibility traces are adequate for learning to predict, as the critic component of the actor–critic algorithm does, but they do not support learning to control, as the actor component must do. The members of a population of critic-like agents may still receive a common reinforcement signal, but they would all learn to predict the same quantity (which in the case of an actor–critic method, would be the expected return for the current policy). How successful each member of the population would be in learning to predict the expected return would depend on the information it receives, which could be very di↵erent for di↵erent members of the population.


n the information it receives, which could be very di↵erent for di↵erent members of the population. There would be no need for the population to produce di↵erentiated patterns of activity. This is not a team problem as deﬁned here. A second requirement for collective learning in a team problem is that there has to be variability in the actions of the team members in order for the team to explore the space of collective actions. The simplest way for a team of reinforcement learning agents to do this is for each member to independently explore its own action space through persistent variability in its output. This will cause the team as a whole to vary its collective actions. For example, a team of the actor units described in Section 15.8 explores the space of collective actions because the


am of the actor units described in Section 15.8 explores the space of collective actions because the output of each unit, being a Bernoulli-logistic unit, probabilistically depends on the weighted sum of its input vector’s components. The weighted sum biases ﬁring probability up or down, but there is always variability. Because each unit uses a REINFORCE policy gradient algorithm (Chapter 13), each unit adjusts its weights with the goal of maximizing the average reward rate it experiences while stochastically exploring its own action space. One can show, as Williams (1992) did, that a team of Bernoulli-logistic REINFORCE units implements a policy gradient algorithm as a whole with respect to average rate of the team’s common reward signal, where the actions are the collective actions of th


average rate of the team’s common reward signal, where the actions are the collective actions of the team. Further, Williams (1992) showed that a team of Bernoulli-logistic units using REINFORCE ascends the average reward gradient when the units in the team are interconnected to form a multilayer ANN. In this case, the reward signal is broadcast to all the units in


15.11. Model-based Methods in the Brain 407 the network, though reward may depend only on the collective actions of the network’s output units. This means that a multilayer team of Bernoulli-logistic REINFORCE units learns like a multilayer network trained by the widely-used error backpropagation method, but in this case the backpropagation process is replaced by the broadcasted reward signal. In practice, the error backpropagation method is considerably faster, but the reinforcement learning team method is more plausible as a neural mechanism, especially in light of what is being learned about reward-modulated STDP as discussed in Section 15.8. Exploration through independent exploration by team members is only the simplest way for a team to explore; more sophisticated methods are possibl


team members is only the simplest way for a team to explore; more sophisticated methods are possible if the team members coordinate their actions to focus on particular parts of the collective action space, either by communicating with one another or by responding to common inputs. There are also mechanisms more sophisticated than contingent eligibility traces for addressing structural credit assignment, which is easier in a team problem when the set of possible collective actions is restricted in some way. An extreme case is a winner-take-all arrangement (for example, the result of lateral inhibition in the brain) that restricts collective actions to those to which only one, or a few, team members contribute.


n) that restricts collective actions to those to which only one, or a few, team members contribute. In this case the winners get the credit or blame for resulting reward or punishment. Details of learning in cooperative games (or team problems) and non-cooperative game problems are beyond the scope of this book. The Bibliographical and Historical Remarks section at the end of this chapter cites a selection of the relevant publications, including extensive references to research on implications for neuroscience of collective reinforcement learning. 15.11 Model-based Methods in the Brain Reinforcement learning’s distinction between model-free and model-based algorithms is proving to be useful for thinking about animal learning and decision processes.


based algorithms is proving to be useful for thinking about animal learning and decision processes. Section 14.6 discusses how this distinction aligns with that between habitual and goal-directed animal behavior. The hypothesis discussed above about how the brain might implement an actor–critic algorithm is relevant only to an animal’s habitual mode of behavior because the basic actor–critic method is model-free. What neural mechanisms are responsible for producing goal-directed behavior, and how do they interact with those underlying habitual behavior? One way to investigate questions about the brain structures involved in these modes of behavior is to inactivate an area of a rat’s brain and then observe what the rat does in an outcome-devaluation experiment (Section 14.6).


rat’s brain and then observe what the rat does in an outcome-devaluation experiment (Section 14.6). Results from experiments like these indicate that the actor–critic hypothesis described above is too simple in placing the actor in the dorsal striatum. Inactivating one part of the dorsal striatum, the dorsolateral striatum (DLS), impairs habit learning, causing the animal to rely more on goal-directed processes. On the other hand, inactivating the dorsomedial striatum (DMS) impairs goal-directed processes, requiring the animal to rely more on habit learning. Results like these support the view that the DLS in rodents is more involved in model-free processes, whereas their DMS is more involved in model-based processes. Results of


408 Chapter 15: Neuroscience studies with human subjects in similar experiments using functional neuroimaging, and with non-human primates, support the view that the analogous structures in the primate brain are di↵erentially involved in habitual and goal-directed modes of behavior. Other studies identify activity associated with model-based processes in the prefrontal cortex of the human brain, the front-most part of the frontal cortex implicated in executive function, including planning and decision making. Speciﬁcally implicated is the orbitofrontal cortex (OFC), the part of the prefrontal cortex immediately above the eyes. Functional neuroimaging in humans, and also recordings of the activities of single neurons in monkeys, reveals strong activity in the OFC related to the subjective r


ivities of single neurons in monkeys, reveals strong activity in the OFC related to the subjective reward value of biologically signiﬁcant stimuli, as well as activity related to the reward expected as a consequence of actions. Although not free of controversy, these results suggest signiﬁcant involvement of the OFC in goal-directed choice. It may be critical for the reward part of an animal’s environment model. Another structure involved in model-based behavior is the hippocampus, a structure critical for memory and spatial navigation. A rat’s hippocampus plays a critical role in the rat’s ability to navigate a maze in the goal-directed manner that led Tolman to the idea that animals use models, or cognitive maps, in selecting actions (Section 14.5).


Tolman to the idea that animals use models, or cognitive maps, in selecting actions (Section 14.5). The hippocampus may also be a critical component of our human ability to imagine new experiences (Hassabis and Maguire, 2007; ´Olafsd´ottir, Barry, Saleem, Hassabis, and Spiers, 2015). The ﬁndings that most directly implicate the hippocampus in planning—the process needed to enlist an environment model in making decisions—come from experiments that decode the activity of neurons in the hippocampus to determine what part of space hippocampal activity is representing on a moment-to-moment basis. When a rat pauses at a choice point in a maze, the representation of space in the hippocampus sweeps forward (and not backwards) along the possible paths the animal can take from that point (Johnson an


forward (and not backwards) along the possible paths the animal can take from that point (Johnson and Redish, 2007). Furthermore, the spatial trajectories represented by these sweeps closely correspond to the rat’s subsequent navigational behavior (Pfei↵er and Foster, 2013). These results suggest that the hippocampus is critical for the statetransition part of an animal’s environment model, and that it is part of a system that uses the model to simulate possible future state sequences to assess the consequences of possible courses of action: a form of planning. The results described above add to a voluminous literature on neural mechanisms underlying goal-directed, or model-based, learning and decision making, but many questions remain unanswered.


goal-directed, or model-based, learning and decision making, but many questions remain unanswered. For example, how can areas as structurally similar as the DLS and DMS be essential components of modes of learning and behavior that are as di↵erent as model-free and model-based algorithms? Are separate structures responsible for (what we call) the transition and reward components of an environment model? Is all planning conducted at decision time via simulations of possible future courses of action as the forward sweeping activity in the hippocampus suggests? In other words, is all planning something like a rollout algorithm (Section 8.10)? Or are models sometimes engaged in the background to reﬁne or recompute value information as illustrated by the Dyna architecture (Section 8.2)?


ound to reﬁne or recompute value information as illustrated by the Dyna architecture (Section 8.2)? How does the brain arbitrate between the use of the habit and goal-directed systems? Is there, in fact, a clear separation between the neural substrates of these systems?


15.12. Addiction 409 The evidence is not pointing to a positive answer to this last question. Summarizing the situation, Doll, Simon, and Daw (2012) wrote that “model-based inﬂuences appear ubiquitous more or less wherever the brain processes reward information,” and this is true even in the regions thought to be critical for model-free learning. This includes the dopamine signals themselves, which can exhibit the inﬂuence of model-based information in addition to the reward prediction errors thought to be the basis of model-free processes. Continuing neuroscience research informed by reinforcement learning’s model-free and model-based distinction has the potential to sharpen our understanding of habitual and goal-directed processes in the brain.


as the potential to sharpen our understanding of habitual and goal-directed processes in the brain. A better grasp of these neural mechanisms may lead to algorithms combining model-free and model-based methods in ways that have not yet been explored in computational reinforcement learning. 15.12 Addiction Understanding the neural basis of drug abuse is a high-priority goal of neuroscience with the potential to produce new treatments for this serious public health problem. One view is that drug craving is the result of the same motivation and learning processes that lead us to seek natural rewarding experiences that serve our biological needs. Addictive substances, by being intensely reinforcing, e↵ectively co-opt our natural mechanisms of learning and decision making.


ng intensely reinforcing, e↵ectively co-opt our natural mechanisms of learning and decision making. This is plausible given that many—though not all— drugs of abuse increase levels of dopamine either directly or indirectly in regions around terminals of dopamine neuron axons in the striatum, a brain structure ﬁrmly implicated in normal reward-based learning (Section 15.7). But the self-destructive behavior associated with drug addiction is not characteristic of normal learning. What is di↵erent about dopamine-mediated learning when the reward is the result of an addictive drug? Is addiction the result of normal learning in response to substances that were largely unavailable throughout our evolutionary history, so that evolution could not select against their damaging e↵ects?


oughout our evolutionary history, so that evolution could not select against their damaging e↵ects? Or do addictive substances somehow interfere with normal dopamine-mediated learning? The reward prediction error hypothesis of dopamine neuron activity and its connection to TD learning are the basis of a model due to Redish (2004) of some—but certainly not all—features of addiction. The model is based on the observation that administration of cocaine and some other addictive drugs produces a transient increase in dopamine. In the model, this dopamine surge is assumed to increase the TD error, δ, in a way that cannot be cancelled out by changes in the value function. In other words, whereas δ is reduced to the degree that a normal reward is predicted by antecedent events (Section 15.6), the


is reduced to the degree that a normal reward is predicted by antecedent events (Section 15.6), the contribution to δ due to an addictive stimulus does not decrease as the reward signal becomes predicted: drug rewards cannot be “predicted away.” The model does this by preventing δ from ever becoming negative when the reward signal is due to an addictive drug, thus eliminating the error-correcting feature of TD learning for states associated with administration of the drug. The result is that the values of these states increase without bound, making actions leading to these states preferred above all others.


410 Chapter 15: Neuroscience Addictive behavior is much more complicated than this result from Redish’s model, but the model’s main idea may be a piece of the puzzle. Or the model might be misleading. Dopamine appears not to play a critical role in all forms of addiction, and not everyone is equally susceptible to developing addictive behavior. Moreover, the model does not include the changes in many circuits and brain regions that accompany chronic drug taking, for example, changes that lead to a drug’s diminishing e↵ect with repeated use. It is also likely that addiction involves model-based processes. Still, Redish’s model illustrates how reinforcement learning theory can be enlisted in the e↵ort to understand a major health problem.


ow reinforcement learning theory can be enlisted in the e↵ort to understand a major health problem. In a similar manner, reinforcement learning theory has been inﬂuential in the development of the new ﬁeld of computational psychiatry, which aims to improve understanding of mental disorders through mathematical and computational methods. 15.13 Summary The neural pathways involved in the brain’s reward system are complex and incompletely understood, but neuroscience research directed toward understanding these pathways and their roles in behavior is progressing rapidly. This research is revealing striking correspondences between the brain’s reward system and the theory of reinforcement learning as presented in this book.


tween the brain’s reward system and the theory of reinforcement learning as presented in this book. The reward prediction error hypothesis of dopamine neuron activity was proposed by scientists who recognized striking parallels between the behavior of TD errors and the activity of neurons that produce dopamine, a neurotransmitter essential in mammals for reward-related learning and behavior. Experiments conducted in the late 1980s and 1990s in the laboratory of neuroscientist Wolfram Schultz showed that dopamine neurons respond to rewarding events with substantial bursts of activity, called phasic responses, only if the animal does not expect those events, suggesting that dopamine neurons are signaling reward prediction errors instead of reward itself.


, suggesting that dopamine neurons are signaling reward prediction errors instead of reward itself. Further, these experiments showed that as an animal learns to predict a rewarding event on the basis of preceding sensory cues, the phasic activity of dopamine neurons shifts to earlier predictive cues while decreasing to later predictive cues. This parallels the backing-up e↵ect of the TD error as a reinforcement learning agent learns to predict reward. Other experimental results ﬁrmly establish that the phasic activity of dopamine neurons is a reinforcement signal for learning that reaches multiple areas of the brain by means of profusely branching axons of dopamine producing neurons. These results are consistent with the distinction we make between a reward signal, Rt, and a reinforcement


results are consistent with the distinction we make between a reward signal, Rt, and a reinforcement signal, which is the TD error δt in most of the algorithms we present. Phasic responses of dopamine neurons are reinforcement signals, not reward signals. A prominent hypothesis is that the brain implements something like an actor–critic algorithm. Two structures in the brain (the dorsal and ventral subdivisions of the striatum), both of which play critical roles in reward-based learning, may function respectively like an actor and a critic. That the TD error is the reinforcement signal for both the actor and the critic ﬁts well with the facts that dopamine neuron axons target both the dorsal and ventral subdivisions of the striatum; that dopamine appears to be


15.13. Summary 411 critical for modulating synaptic plasticity in both structures; and that the e↵ect on a target structure of a neuromodulator such as dopamine depends on properties of the target structure and not just on properties of the neuromodulator. The actor and the critic can be implemented by ANNs consisting of neuron-like units having learning rules based on the policy-gradient actor–critic method described in Section 13.5. Each connection in these networks is like a synapse between neurons in the brain, and the learning rules correspond to rules governing how synaptic eﬃcacies change as functions of the activities of the presynaptic and the postsynaptic neurons, together with neuromodulatory input corresponding to input from dopamine neurons.


synaptic neurons, together with neuromodulatory input corresponding to input from dopamine neurons. In this setting, each synapse has its own eligibility trace that records past activity involving that synapse. The only di↵erence between the actor and critic learning rules is that they use di↵erent kinds of eligibility traces: the critic unit’s traces are non-contingent because they do not involve the critic unit’s output, whereas the actor unit’s traces are contingent because in addition to the actor unit’s input, they depend on the actor unit’s output. In the hypothetical implementation of an actor–critic system in the brain, these learning rules respectively correspond to rules governing plasticity of corticostriatal synapses that convey signals from the cortex to the principal neurons


plasticity of corticostriatal synapses that convey signals from the cortex to the principal neurons in the dorsal and ventral striatal subdivisions, synapses that also receive inputs from dopamine neurons. The learning rule of an actor unit in the actor–critic network closely corresponds to reward-modulated spike-timing-dependent plasticity. In spike-timing-dependent plasticity (STDP), the relative timing of pre- and postsynaptic activity determines the direction of synaptic change. In reward-modulated STDP, changes in synapses in addition depend on a neuromodulator, such as dopamine, arriving within a time window that can last up to 10 seconds after the conditions for STDP are met. Evidence is accumulating that reward-modulated STDP occurs at corticostriatal synapses, where the actor’s le


is accumulating that reward-modulated STDP occurs at corticostriatal synapses, where the actor’s learning takes place in the hypothetical neural implementation of an actor–critic system, adds to the plausibility of the hypothesis that something like an actor–critic system exists in the brains of some animals. The idea of synaptic eligibility and basic features of the actor learning rule derive from Klopf’s hypothesis of the “hedonistic neuron” (Klopf, 1972, 1981). He conjectured that individual neurons seek to obtain reward and to avoid punishment by adjusting the eﬃcacies of their synapses on the basis of rewarding or punishing consequences of their action potentials. A neuron’s activity can a↵ect its later input because the neuron is embedded in many feedback loops, some within the anim


an a↵ect its later input because the neuron is embedded in many feedback loops, some within the animal’s nervous system and body and others passing through the animal’s external environment. Klopf’s idea of eligibility is that synapses are temporarily marked as eligible for modiﬁcation if they participated in the neuron’s ﬁring (making this the contingent form of eligibility trace). A synapse’s eﬃcacy is modiﬁed if a reinforcing signal arrives while the synapse is eligible. We alluded to the chemotactic behavior of a bacterium as an example of a single cell that directs its movements in order to seek some molecules and to avoid others. A conspicuous feature of the dopamine system is that ﬁbers releasing dopamine project widely to multiple parts of the brain.


the dopamine system is that ﬁbers releasing dopamine project widely to multiple parts of the brain. Although it is likely that only some populations of dopamine neurons broadcast the same reinforcement signal, if this signal reaches the synapses of many neurons involved in actor-type learning, then the situation can


412 Chapter 15: Neuroscience be modeled as a team problem. In this type of problem, each agent in a collection of reinforcement learning agents receives the same reinforcement signal, where that signal depends on the activities of all members of the collection, or team. If each team member uses a suﬃciently capable learning algorithm, the team can learn collectively to improve performance of the entire team as evaluated by the globally-broadcast reinforcement signal, even if the team members do not directly communicate with one another. This is consistent with the wide dispersion of dopamine signals in the brain and provides a neurally plausible alternative to the widely-used error-backpropagation method for training multilayer networks.


sible alternative to the widely-used error-backpropagation method for training multilayer networks. The distinction between model-free and model-based reinforcement learning is helping neuroscientists investigate the neural bases of habitual and goal-directed learning and decision making. Research so far points to their being some brain regions more involved in one type of process than the other, but the picture remains unclear because model-free and model-based processes do not appear to be neatly separated in the brain. Many questions remain unanswered. Perhaps most intriguing is evidence that the hippocampus, a structure traditionally associated with spatial navigation and memory, appears to be involved in simulating possible future courses of action as part of an animal’s decisionmakin


to be involved in simulating possible future courses of action as part of an animal’s decisionmaking process. This suggests that it is part of a system that uses an environment model for planning. Reinforcement learning theory is also inﬂuencing thinking about neural processes underlying drug abuse. A model of some features of drug addiction is based on the reward prediction error hypothesis. It proposes that an addicting stimulant, such as cocaine, destabilizes TD learning to produce unbounded growth in the values of actions associated with drug intake. This is far from a complete model of addiction, but it illustrates how a computational perspective suggests theories that can be tested with further research.


strates how a computational perspective suggests theories that can be tested with further research. The new ﬁeld of computational psychiatry similarly focuses on the use of computational models, some derived from reinforcement learning, to better understand mental disorders. This chapter only touched the surface of how the neuroscience of reinforcement learning and the development of reinforcement learning in computer science and engineering have inﬂuenced one another. Most features of reinforcement learning algorithms owe their design to purely computational considerations, but some have been inﬂuenced by hypotheses about neural learning mechanisms. Remarkably, as experimental data has accumulated about the brain’s reward processes, many of the purely computationallymotivated features of


mulated about the brain’s reward processes, many of the purely computationallymotivated features of reinforcement learning algorithms are turning out to be consistent with neuroscience data. Other features of computational reinforcement learning, such as eligibility traces and the ability of teams of reinforcement learning agents to learn to act collectively under the inﬂuence of a globally-broadcast reinforcement signal, may also turn out to parallel experimental data as neuroscientists continue to unravel the neural basis of reward-based animal learning and behavior.


15.13. Summary 413 Bibliographical and Historical Remarks The number of publications treating parallels between the neuroscience of learning and decision making and the approach to reinforcement learning presented in this book is enormous. We can cite only a small selection. Niv (2009), Dayan and Niv (2008), Gimcher (2011), Ludvig, Bellemare, and Pearson (2011), and Shah (2012) are good places to start. Together with economics, evolutionary biology, and mathematical psychology, reinforcement learning theory is helping to formulate quantitative models of the neural mechanisms of choice in humans and non-human primates. With its focus on learning, this chapter only lightly touches upon the neuroscience of decision making.


its focus on learning, this chapter only lightly touches upon the neuroscience of decision making. Glimcher (2003) introduced the ﬁeld of “neuroeconomics,” in which reinforcement learning contributes to the study of the neural basis of decision making from an economics perspective. See also Glimcher and Fehr (2013). The text on computational and mathematical modeling in neuroscience by Dayan and Abbott (2001) includes reinforcement learning’s role in these approaches. Sterling and Laughlin (2015) examined the neural basis of learning in terms of general design principles that enable eﬃcient adaptive behavior. 15.1 There are many good expositions of basic neuroscience. Kandel, Schwartz, Jessell, Siegelbaum, and Hudspeth (2013) is an authoritative and very comprehensive source.


hwartz, Jessell, Siegelbaum, and Hudspeth (2013) is an authoritative and very comprehensive source. 15.2 Berridge and Kringelbach (2008) reviewed the neural basis of reward and pleasure, pointing out that reward processing has many dimensions and involves many neural systems. Space prevents discussion of the inﬂuential research of Berridge and Robinson (1998), who distinguish between the hedonic impact of a stimulus, which they call “liking,” and the motivational e↵ect, which they call “wanting.” Hare, O’Doherty, Camerer, Schultz, and Rangel (2008) examined the neural basis of value-related signals from an economic perspective, distinguishing between goal values, decision values, and prediction errors. Decision value is goal value minus action cost.


oal values, decision values, and prediction errors. Decision value is goal value minus action cost. See also Rangel, Camerer, and Montague (2008), Rangel and Hare (2010), and Peters and B¨uchel (2010). 15.3 The reward prediction error hypothesis of dopamine neuron activity is most prominently discussed by Schultz, Dayan, and Montague (1997). The hypothesis was ﬁrst explicitly put forward by Montague, Dayan, and Sejnowski (1996). As they stated the hypothesis, it referred to reward prediction errors (RPEs) but not speciﬁcally to TD errors; however, their development of the hypothesis made it clear that they were referring to TD errors. The earliest recognition of the TD-error/dopamine connection of which we are aware is that of Montague, Dayan, Nowlan, Pouget, and Sejnowski (1993), who prop


ion of which we are aware is that of Montague, Dayan, Nowlan, Pouget, and Sejnowski (1993), who proposed a TD-error-modulated Hebbian learning rule motivated by results on dopamine signaling from Schultz’s group. The connection was also pointed out in an abstract by Quartz, Dayan, Montague, and Sejnowski (1992). Montague and Sejnowski (1994) emphasized the importance of prediction in the brain and outlined how predictive Hebbian learning modulated by TD errors could be implemented via a di↵use neuromodulatory system, such as the dopamine system. Friston, Tononi, Reeke, Sporns,


414 Chapter 15: Neuroscience and Edelman (1994) presented a model of value-dependent learning in the brain in which synaptic changes are mediated by a TD-like error provided by a global neuromodulatory signal (although they did not single out dopamine). Montague, Dayan, Person, and Sejnowski (1995) presented a model of honeybee foraging using the TD error. The model is based on research by Hammer, Menzel, and colleagues (Hammer and Menzel, 1995; Hammer, 1997) showing that the neuromodulator octopamine acts as a reinforcement signal in the honeybee. Montague et al. (1995) pointed out that dopamine likely plays a similar role in the vertebrate brain. Barto (1995a) related the actor–critic architecture to basal-ganglionic circuits and discussed the relationship between TD learning and the mai


itecture to basal-ganglionic circuits and discussed the relationship between TD learning and the main results from Schultz’s group. Houk, Adams, and Barto (1995) suggested how TD learning and the actor–critic architecture might map onto the anatomy, physiology, and molecular mechanism of the basal ganglia. Doya and Sejnowski (1998) extended their earlier paper on a model of birdsong learning (Doya and Sejnowski, 1995) by including a TD-like error identiﬁed with dopamine to reinforce the selection of auditory input to be memorized. O’Reilly and Frank (2006) and O’Reilly, Frank, Hazy, and Watz (2007) argued that phasic dopamine signals are RPEs but not TD errors. In support of their theory they cited results with variable interstimulus intervals that do not match predictions of a simple TD m


y cited results with variable interstimulus intervals that do not match predictions of a simple TD model, as well as the observation that higher-order conditioning beyond second-order conditioning is rarely observed, while TD learning is not so limited. Dayan and Niv (2008) discussed “the good, the bad, and the ugly” of how reinforcement learning theory and the reward prediction error hypothesis align with experimental data. Glimcher (2011) reviewed the empirical ﬁndings that support the reward prediction error hypothesis and emphasized the signiﬁcance of the hypothesis for contemporary neuroscience. 15.4 Graybiel (2000) is a brief primer on the basal ganglia. The experiments mentioned that involve optogenetic activation of dopamine neurons were conducted by Tsai, Zhang, Adamantidis, Stube


involve optogenetic activation of dopamine neurons were conducted by Tsai, Zhang, Adamantidis, Stuber, Bonci, de Lecea, and Deisseroth (2009), Steinberg, Keiﬂin, Boivin, Witten, Deisseroth, and Janak (2013), and Claridge-Chang, Roorda, Vrontou, Sjulson, Li, Hirsh, and Miesenb¨ock (2009). Fiorillo, Yun, and Song (2013), Lammel, Lim, and Malenka (2014), and Saddoris, Cacciapaglia, Wightmman, and Carelli (2015) are among studies showing that the signaling properties of dopamine neurons are specialized for di↵erent target regions. RPEsignaling neurons may belong to one among multiple populations of dopamine neurons having di↵erent targets and subserving di↵erent functions. Eshel, Tian, Bukwich, and Uchida (2016) found homogeneity of reward prediction error responses of dopamine neurons in the


and Uchida (2016) found homogeneity of reward prediction error responses of dopamine neurons in the lateral VTA during classical conditioning in mice, though their results do not rule out response diversity across wider areas. Gershman, Pesaran, and Daw (2009) studied reinforcement learning tasks that can be decomposed into independent components with separate reward signals, ﬁnding evidence in human neuroimaging data suggesting that the brain exploits this kind of structure.


15.13. Summary 415 15.5 Schultz’s 1998 survey article is a good entr´ee into the very extensive literature on reward predicting signaling of dopamine neurons. Berns, McClure, Pagnoni, and Montague (2001), Breiter, Aharon, Kahneman, Dale, and Shizgal (2001), Pagnoni, Zink, Montague, and Berns (2002), and O’Doherty, Dayan, Friston, Critchley, and Dolan (2003) described functional brain imaging studies supporting the existence of signals like TD errors in the human brain. 15.6 This section roughly follows Barto (1995a) in explaining how TD errors mimic the main results from Schultz’s group on the phasic responses of dopamine neurons. 15.7 This section is largely based on Takahashi, Schoenbaum, and Niv (2008) and Niv (2009).


eurons. 15.7 This section is largely based on Takahashi, Schoenbaum, and Niv (2008) and Niv (2009). To the best of our knowledge, Barto (1995a) and Houk, Adams, and Barto (1995) ﬁrst speculated about possible implementations of actor–critic algorithms in the basal ganglia. On the basis of functional magnetic resonance imaging of human subjects while engaged in instrumental conditioning, O’Doherty, Dayan, Schultz, Deichmann, Friston, and Dolan (2004) suggested that the actor and the critic are most likely located respectively in the dorsal and ventral striatum. Gershman, Moustafa, and Ludvig (2014) focused on how time is represented in reinforcement learning models of the basal ganglia, discussing evidence for, and implications of, various computational approaches to time representation.


cussing evidence for, and implications of, various computational approaches to time representation. The hypothetical neural implementation of the actor–critic architecture described in this section includes very little detail about known basal ganglia anatomy and physiology. In addition to the more detailed hypothesis of Houk, Adams, and Barto (1995), a number of other hypotheses include more speciﬁc connections to anatomy and physiology and are claimed to explain additional data. These include hypotheses proposed by Suri and Schultz (1998, 1999), Brown, Bullock, and Grossberg (1999), Contreras-Vidal and Schultz (1999), Suri, Bargas, and Arbib (2001), O’Reilly and Frank (2006), and O’Reilly, Frank, Hazy, and Watz (2007).


i, Bargas, and Arbib (2001), O’Reilly and Frank (2006), and O’Reilly, Frank, Hazy, and Watz (2007). Joel, Niv, and Ruppin (2002) critically evaluated the anatomical plausibility of several of these models and present an alternative intended to accommodate some neglected features of basal ganglionic circuitry. 15.8 The actor learning rule discussed here is more complicated than the one in the early actor–critic network of Barto et al. (1983). Actor-unit eligibility traces in that network were traces of just At⇥x(St) instead of the full (At−⇡(1|St, ✓))x(St). That work did not beneﬁt from the policy-gradient theory presented in Chapter 13 or the contributions of Williams (1986, 1992), who showed how an ANN of Bernoulli-logistic units could implement a policy-gradient method.


1992), who showed how an ANN of Bernoulli-logistic units could implement a policy-gradient method. Reynolds and Wickens (2002) proposed a three-factor rule for synaptic plasticity in the corticostriatal pathway in which dopamine modulates changes in corticostriatal synaptic eﬃcacy. They discussed the experimental support for this kind of learning rule and its possible molecular basis. The deﬁnitive demonstration of spike-timing-dependent plasticity (STDP) is attributed to Markram, L¨ubke, Frotscher, and Sakmann (1997), with evidence from earlier experiments


416 Chapter 15: Neuroscience by Levy and Steward (1983) and others that the relative timing of pre- and postsynaptic spikes is critical for inducing changes in synaptic eﬃcacy. Rao and Sejnowski (2001) suggested how STDP could be the result of a TD-like mechanism at synapses with non-contingent eligibility traces lasting about 10 milliseconds. Dayan (2002) commented that this would require an error as in Sutton and Barto’s (1981a) early model of classical conditioning and not a true TD error. Representative publications from the extensive literature on reward-modulated STDP are Wickens (1990), Reynolds and Wickens (2002), and Calabresi, Picconi, Tozzi and Di Filippo (2007). Pawlak and Kerr (2008) showed that dopamine is necessary to induce STDP at the corticostriatal synapses of medium spi


2008) showed that dopamine is necessary to induce STDP at the corticostriatal synapses of medium spiny neurons. See also Pawlak, Wickens, Kirkwood, and Kerr (2010). Yagishita, Hayashi-Takagi, Ellis-Davies, Urakubo, Ishii, and Kasai (2014) found that dopamine promotes spine enlargement of the medium spiny neurons of mice only during a time window of from 0.3 to 2 seconds after STDP stimulation. Izhikevich (2007) proposed and explored the idea of using STDP timing conditions to trigger contingent eligibility traces. Fr´emaux, Sprekeler, and Gerstner (2010) proposed theoretical conditions for successful learning by rules based on reward-modulated STDP. 15.9 Klopf’s hedonistic neuron hypothesis (Klopf 1972, 1982) inspired our actor–critic algorithm implemented as an ANN with a single neuron-li


(Klopf 1972, 1982) inspired our actor–critic algorithm implemented as an ANN with a single neuron-like unit, called the actor unit, implementing a Law-of-E↵ect-like learning rule (Barto, Sutton, and Anderson, 1983). Ideas related to Klopf’s synaptically-local eligibility have been proposed by others. Crow (1968) proposed that changes in the synapses of cortical neurons are sensitive to the consequences of neural activity. Emphasizing the need to address the time delay between neural activity and its consequences in a reward-modulated form of synaptic plasticity, he proposed a contingent form of eligibility, but associated with entire neurons instead of individual synapses. According to his hypothesis, a wave of neuronal activity leads to a short-term change in the cells involved in the wav


ypothesis, a wave of neuronal activity leads to a short-term change in the cells involved in the wave such that they are picked out from a background of cells not so activated. ... such cells are rendered sensitive by the short-term change to a reward signal ... in such a way that if such a signal occurs before the end of the decay time of the change the synaptic connexions between the cells are made more e↵ective. (Crow, 1968) Crow argued against previous proposals that reverberating neural circuits play this role by pointing out that the e↵ect of a reward signal on such a circuit would “...establish the synaptic connexions leading to the reverberation (that is to say, those involved in activity at the time of the reward signal) and not those on the path which led to the adaptive motor ou


ivity at the time of the reward signal) and not those on the path which led to the adaptive motor output.” Crow further postulated that reward signals are delivered via a “distinct neural ﬁber system,” presumably the one into which Olds and Milner (1954) tapped, that would transform synaptic connections “from a short into a long-term form.”


15.13. Summary 417 In another farsighted hypothesis, Miller proposed a Law-of-E↵ect-like learning rule that includes synaptically-local contingent eligibility traces: ... it is envisaged that in a particular sensory situation neurone B, by chance, ﬁres a ‘meaningful burst’ of activity, which is then translated into motor acts, which then change the situation. It must be supposed that the meaningful burst has an inﬂuence, at the neuronal level, on all of its own synapses which are active at the time ... thereby making a preliminary selection of the synapses to be strengthened, though not yet actually strengthening them. ...The strengthening signal ... makes the ﬁnal selection ... and accomplishes the deﬁnitive change in the appropriate synapses. (Miller, 1981, p.


selection ... and accomplishes the deﬁnitive change in the appropriate synapses. (Miller, 1981, p. 81) Miller’s hypothesis also included a critic-like mechanism, which he called a “sensory analyzer unit,” that worked according to classical conditioning principles to provide reinforcement signals to neurons so that they would learn to move from lower- to higher-valued states, thus anticipating the use of the TD error as a reinforcement signal in the actor–critic architecture. Miller’s idea not only parallels Klopf’s (with the exception of its explicit invocation of a distinct “strengthening signal”), it also anticipated the general features of reward-modulated STDP. A related though di↵erent idea, which Seung (2003) called the “hedonistic synapse,” is that synapses individually adjust the


idea, which Seung (2003) called the “hedonistic synapse,” is that synapses individually adjust the probability that they release neurotransmitter in the manner of the Law of E↵ect: if reward follows release, the release probability increases, and decreases if reward follows failure to release. This is essentially the same as the learning scheme Minsky used in his 1954 Princeton PhD dissertation, where he called the synapse-like learning element a SNARC (Stochastic Neural-Analog Reinforcement Calculator). Contingent eligibility is involved in these ideas too, although it is contingent on the activity of an individual synapse instead of the postsynaptic neuron. Also related is the proposal of Unnikrishnan and Venugopal (1994) that uses the correlation-based method of Harth and Tzanakou (197


Unnikrishnan and Venugopal (1994) that uses the correlation-based method of Harth and Tzanakou (1974) to adjust ANN weights. Frey and Morris (1997) proposed the idea of a “synaptic tag” for the induction of long-lasting strengthening of synaptic eﬃcacy. Though not unlike Klopf’s eligibility, their tag was hypothesized to consist of a temporary strengthening of a synapse that could be transformed into a long-lasting strengthening by subsequent neuron activation. The model of O’Reilly and Frank (2006) and O’Reilly, Frank, Hazy, and Watz (2007) uses working memory to bridge temporal intervals instead of eligibility traces. Wickens and Kotter (1995) discuss possible mechanisms for synaptic eligibility.


eligibility traces. Wickens and Kotter (1995) discuss possible mechanisms for synaptic eligibility. He, Huertas, Hong, Tie, Hell, Shouval, Kirkwood (2015) provide evidence supporting the existence of contingent eligibility traces in synapses of cortical neurons with time courses like those of the eligibility traces Klopf postulated. The metaphor of a neuron using a learning rule related to bacterial chemotaxis was discussed by Barto (1989). Koshland’s extensive study of bacterial chemotaxis was in part motivated by similarities between features of bacteria and features of neurons (Koshland, 1980). See also Berg (1975). Shimansky (2009) proposed a


418 Chapter 15: Neuroscience synaptic learning rule somewhat similar to Seung’s mentioned above in which each synapse individually acts like a chemotactic bacterium. In this case a collection of synapses “swims” toward attractants in the high-dimensional space of synaptic weight values. Montague, Dayan, Person, and Sejnowski (1995) proposed a chemotactic-like model of the bee’s foraging behavior involving the neuromodulator octopamine. 15.10 Research on the behavior of reinforcement learning agents in team and game problems has a long history roughly occurring in three phases. To the best of our knowledge, the ﬁrst phase began with investigations by the Russian mathematician and physicist M. L. Tsetlin. A collection of his work was published as Tsetlin (1973) after his death in 1966.


st M. L. Tsetlin. A collection of his work was published as Tsetlin (1973) after his death in 1966. Our Sections 1.7 and 4.8 refer to his study of learning automata in connection to bandit problems. The Tsetlin collection also includes studies of learning automata in team and game problems, which led to later work in this area using stochastic learning automata as described by Narendra and Thathachar (1974, 1989), Viswanathan and Narendra (1974), Lakshmivarahan and Narendra (1982), Narendra and Wheeler (1983), and Thathachar and Sastry (2002). Thathachar and Sastry (2011) is a more recent comprehensive account. These studies were mostly restricted to non-associative learning automata, meaning that they did not address associative, or contextual, bandit problems (Section 2.9).


omata, meaning that they did not address associative, or contextual, bandit problems (Section 2.9). The second phase began with the extension of learning automata to the associative, or contextual, case. Barto, Sutton, and Brouwer (1981) and Barto and Sutton (1981b) experimented with associative stochastic learning automata in singlelayer ANNs to which a global reinforcement signal was broadcast. The learning algorithm was an associative extension of the Alopex algorithm of Harth and Tzanakou (1974). Barto et al. called neuron-like elements implementing this kind of learning associative search elements (ASEs). Barto and Anandan (1985) introduced an associative reinforcement learning algorithm called the associative reward-penalty (AR−P) algorithm.


ssociative reinforcement learning algorithm called the associative reward-penalty (AR−P) algorithm. They proved a convergence result by combining theory of stochastic learning automata with theory of pattern classiﬁcation. Barto (1985, 1986) and Barto and Jordan (1987) described results with teams of AR−P units connected into multi-layer ANNs, showing that they could learn nonlinear functions, such as XOR and others, with a globally-broadcast reinforcement signal. Barto (1985) extensively discussed this approach to ANNs and how this type of learning rule is related to others in the literature at that time. Williams (1992) mathematically analyzed and broadened this class of learning rules and related their use to the error backpropagation method for training multilayer ANNs.


rning rules and related their use to the error backpropagation method for training multilayer ANNs. Williams (1988) described several ways that backpropagation and reinforcement learning can be combined for training ANNs. Williams (1992) showed that a special case of the AR−P algorithm is a REINFORCE algorithm, although better results were obtained with the general AR−P algorithm (Barto, 1985).


15.13. Summary 419 The third phase of interest in teams of reinforcement learning agents was inﬂuenced by increased understanding of the role of dopamine as a widely broadcast neuromodulator and speculation about the existence of reward-modulated STDP. Much more so than earlier research, this research considers details of synaptic plasticity and other constraints from neuroscience. Publications include the following (chronologically and alphabetically): Bartlett and Baxter (1999, 2000), Xie and Seung (2004), Baras and Meir (2007), Farries and Fairhall (2007), Florian (2007), Izhikevich (2007), Pecevski, Maass, and Legenstein (2008), Legenstein, Pecevski, and Maass (2008), Kolodziejski, Porr, and W¨org¨otter (2009), Urbanczik and Senn (2009), and Vasilaki, Fr´emaux, Urbanczik, Senn, and Ger


and W¨org¨otter (2009), Urbanczik and Senn (2009), and Vasilaki, Fr´emaux, Urbanczik, Senn, and Gerstner (2009). Now´e, Vrancx, and De Hauwere (2012) reviewed more recent developments in the wider ﬁeld of multi-agent reinforcement learning 15.11 Yin and Knowlton (2006) reviewed ﬁndings from outcome-devaluation experiments with rodents supporting the view that habitual and goal-directed behavior (as psychologists use the phrase) are respectively most associated with processing in the dorsolateral striatum (DLS) and the dorsomedial striatum (DMS). Results of functional imaging experiments with human subjects in the outcomedevaluation setting by Valentin, Dickinson, and O’Doherty (2007) suggest that the orbitofrontal cortex (OFC) is an important component of goal-directed choice.


007) suggest that the orbitofrontal cortex (OFC) is an important component of goal-directed choice. Single unit recordings in monkeys by Padoa-Schioppa and Assad (2006) support the role of the OFC in encoding values guiding choice behavior. Rangel, Camerer, and Montague (2008) and Rangel and Hare (2010) reviewed ﬁndings from the perspective of neuroeconomics about how the brain makes goal-directed decisions. Pezzulo, van der Meer, Lansink, and Pennartz (2014) reviewed the neuroscience of internally generated sequences and presented a model of how these mechanisms might be components of model-based planning. Daw and Shohamy (2008) proposed that while dopamine signaling connects well to habitual, or model-free, behavior, other processes are involved in goal-directed, or model-based, behavior


al, or model-free, behavior, other processes are involved in goal-directed, or model-based, behavior. Data from experiments by Bromberg-Martin, Matsumoto, Hong, and Hikosaka (2010) indicate that dopamine signals contain information pertinent to both habitual and goal-directed behavior. Doll, Simon, and Daw (2012) argued that there may not a clear separation in the brain between mechanisms that subserve habitual and goal-directed learning and choice. 15.12 Keiﬂin and Janak (2015) reviewed connections between TD errors and addiction. Nutt, Lingford-Hughes, Erritzoe, and Stokes (2015) critically evaluated the hypothesis that addiction is due to a disorder of the dopamine system. Montague, Dolan, Friston, and Dayan (2012) outlined the goals and early e↵orts in the ﬁeld of computational psychia


, Friston, and Dayan (2012) outlined the goals and early e↵orts in the ﬁeld of computational psychiatry, and Adams, Huys, and Roiser (2015) reviewed more recent progress.


Chapter 16 Applications and Case Studies In this chapter we present a few case studies of reinforcement learning. Several of these are substantial applications of potential economic signiﬁcance. One, Samuel’s checkers player, is primarily of historical interest. Our presentations are intended to illustrate some of the trade-o↵s and issues that arise in real applications. For example, we emphasize how domain knowledge is incorporated into the formulation and solution of the problem. We also highlight the representation issues that are so often critical to successful applications. The algorithms used in some of these case studies are substantially more complex than those we have presented in the rest of the book.


e case studies are substantially more complex than those we have presented in the rest of the book. Applications of reinforcement learning are still far from routine and typically require as much art as science. Making applications easier and more straightforward is one of the goals of current research in reinforcement learning. 16.1 TD-Gammon One of the most impressive applications of reinforcement learning to date is that by Gerald Tesauro to the game of backgammon (Tesauro, 1992, 1994, 1995, 2002). Tesauro’s program, TD-Gammon, required little backgammon knowledge, yet learned to play extremely well, near the level of the world’s strongest grandmasters. The learning algorithm in TD-Gammon was a straightforward combination of the TD(λ) algorithm and nonlinear function approximation using


was a straightforward combination of the TD(λ) algorithm and nonlinear function approximation using a multilayer artiﬁcial neural network (ANN) trained by backpropagating TD errors. Backgammon is a major game in the sense that it is played throughout the world, with numerous tournaments and regular world championship matches. It is in part a game of chance, and it is a popular vehicle for waging signiﬁcant sums of money. There are probably more professional backgammon players than there are professional chess players. The game is played with 15 white and 15 black pieces on a board of 24 locations, called points. To the right on the next page is shown a typical position early in the game, seen from the perspective of the white player.


page is shown a typical position early in the game, seen from the perspective of the white player. White here has just rolled the dice and obtained a 5 and a 2. This means that he can move one of his pieces 5 steps and one


422 Chapter 16: Applications and Case Studies white pieces move counterclockwise 1 2 3 4 5 6 7 8 9 10 11 12 18 17 16 15 14 13 19 20 21 22 23 24 black pieces move clockwise A backgammon position (possibly the same piece) 2 steps. For example, he could move two pieces from the 12 point, one to the 17 point, and one to the 14 point. White’s objective is to advance all of his pieces into the last quadrant (points 19–24) and then o↵ the board. The ﬁrst player to remove all his pieces wins. One complication is that the pieces interact as they pass each other going in di↵erent directions. For example, if it were black’s move, he could use the dice roll of 2 to move a piece from the 24 point to the 22 point, “hitting” the white piece there.


dice roll of 2 to move a piece from the 24 point to the 22 point, “hitting” the white piece there. Pieces that have been hit are placed on the “bar” in the middle of the board (where we already see one previously hit black piece), from whence they reenter the race from the start. However, if there are two pieces on a point, then the opponent cannot move to that point; the pieces are protected from being hit. Thus, white cannot use his 5–2 dice roll to move either of his pieces on the 1 point, because their possible resulting points are occupied by groups of black pieces. Forming contiguous blocks of occupied points to block the opponent is one of the elementary strategies of the game. Backgammon involves several further complications, but the above description gives the basic idea.


Backgammon involves several further complications, but the above description gives the basic idea. With 30 pieces and 24 possible locations (26, counting the bar and o↵-the-board) it should be clear that the number of possible backgammon positions is enormous, far more than the number of memory elements one could have in any physically realizable computer. The number of moves possible from each position is also large. For a typical dice roll there might be 20 di↵erent ways of playing. In considering future moves, such as the response of the opponent, one must consider the possible dice rolls as well. The result is that the game tree has an e↵ective branching factor of about 400. This is far too large to permit e↵ective use of the conventional heuristic search methods that have proved so e


too large to permit e↵ective use of the conventional heuristic search methods that have proved so e↵ective in games like chess and checkers. On the other hand, the game is a good match to the capabilities of TD learning methods. Although the game is highly stochastic, a complete description of the game’s state is available at all times. The game evolves over a sequence of moves and positions until ﬁnally ending in a win for one player or the other, ending the game. The outcome can be interpreted as a ﬁnal reward to be predicted. On the other hand, the theoretical results we have described so far cannot be usefully applied to this task. The number of states is so large that a lookup table cannot be used, and the opponent is a source of uncertainty and time variation.


that a lookup table cannot be used, and the opponent is a source of uncertainty and time variation. TD-Gammon used a nonlinear form of TD(λ). The estimated value, ˆv(s,w), of any state (board position) s was meant to estimate the probability of winning starting from state s. To achieve this, rewards were deﬁned as zero for all time steps except those on which the game is won. To implement the value function, TD-Gammon used a standard multilayer ANN, much like that shown to the right on the next page. (The real


16.1. TD-Gammon 423 Vt+1! Vt hidden unit backgammon position (198 input units) predicted probability of winning, Vt TD error, . . . . . . . . . . . . . . . . . . ˆv(St+1,w) −ˆv(St,w) TD error ˆv(St,w) hidden units (40-80) Figure 16.1: The TD-Gammon ANN network had two additional units in its ﬁnal layer to estimate the probability of each player’s winning in a special way called a “gammon” or “backgammon.”) The network consisted of a layer of input units, a layer of hidden units, and a ﬁnal output unit. The input to the network was a representation of a backgammon position, and the output was an estimate of the value of that position. In the ﬁrst version of TD-Gammon, TD-Gammon 0.0, backgammon positions were represented to the network in a relatively direct way that involved little backgamm


n positions were represented to the network in a relatively direct way that involved little backgammon knowledge. It did, however, involve substantial knowledge of how ANNs work and how information is best presented to them. It is instructive to note the exact representation Tesauro chose. There were a total of 198 input units to the network. For each point on the backgammon board, four units indicated the number of white pieces on the point. If there were no white pieces, then all four units took on the value zero. If there was one piece, then the ﬁrst unit took on the value 1. This encoded the elementary concept of a “blot,” i.e., a piece that can be hit by the opponent. If there were two or more pieces, then the second unit was set to 1.


at can be hit by the opponent. If there were two or more pieces, then the second unit was set to 1. This encoded the basic concept of a “made point” on which the opponent cannot land. If there were exactly three pieces on the point, then the third unit was set to 1. This encoded the basic concept of a “single spare,” i.e., an extra piece in addition to the two pieces that made the point. Finally, if there were more than three pieces, the fourth unit was set to a value proportionate to the number of additional pieces beyond three. Letting n denote the total number of pieces on the point, if n > 3, then the fourth unit took on the value (n−3)/2. This encoded a linear representation of “multiple spares” at the given point.


on the value (n−3)/2. This encoded a linear representation of “multiple spares” at the given point. With four units for white and four for black at each of the 24 points, that made a total of 192 units. Two additional units encoded the number of white and black pieces on the bar (each took the value n/2, where n is the number of pieces on the bar), and two more encoded the number of black and white pieces already successfully removed from the board (these took the value n/15, where n is the number of pieces already borne o↵). Finally, two units indicated in a binary fashion whether it was white’s or black’s turn to move. The general logic behind these choices should be clear. Basically, Tesauro tried to represent the position in a straightforward way, while keeping the number of units rela


uro tried to represent the position in a straightforward way, while keeping the number of units relatively small. He provided one unit for each conceptually distinct possibility that seemed likely to be relevant, and he scaled them to roughly the same range, in this case between 0 and 1. Given a representation of a backgammon position, the network computed its estimated value in the standard way. Corresponding to each connection from an input unit to a hidden unit was a real-valued weight. Signals from each input unit were multiplied by


424 Chapter 16: Applications and Case Studies their corresponding weights and summed at the hidden unit. The output, h(j), of hidden unit j was a nonlinear sigmoid function of the weighted sum: h(j) = σ X i wijxi ! = 1 1 + e−P i wijxi , where xi is the value of the ith input unit and wij is the weight of its connection to the jth hidden unit (all the weights in the network together make up the parameter vector w). The output of the sigmoid is always between 0 and 1, and has a natural interpretation as a probability based on a summation of evidence. The computation from hidden units to the output unit was entirely analogous. Each connection from a hidden unit to the output unit had a separate weight.


as entirely analogous. Each connection from a hidden unit to the output unit had a separate weight. The output unit formed the weighted sum and then passed it through the same sigmoid nonlinearity. TD-Gammon used the semi-gradient form of the TD(λ) algorithm described in Section 12.2, with the gradients computed by the error backpropagation algorithm (Rumelhart, Hinton, and Williams, 1986). Recall that the general update rule for this case is wt+1 .= wt + ↵ h Rt+1 + γˆv(St+1,wt) −ˆv(St,wt) i zt, (16.1) where wt is the vector of all modiﬁable parameters (in this case, the weights of the network) and zt is a vector of eligibility traces, one for each component of wt, updated by zt .= γλzt−1 + rˆv(St,wt), with z0 .= 0.


igibility traces, one for each component of wt, updated by zt .= γλzt−1 + rˆv(St,wt), with z0 .= 0. The gradient in this equation can be computed eﬃciently by the backpropagation procedure. For the backgammon application, in which γ = 1 and the reward is always zero except upon winning, the TD error portion of the learning rule is usually just ˆv(St+1,w) −ˆv(St,w), as suggested in Figure 16.1. To apply the learning rule we need a source of backgammon games. Tesauro obtained an unending sequence of games by playing his learning backgammon player against itself. To choose its moves, TD-Gammon considered each of the 20 or so ways it could play its dice roll and the corresponding positions that would result. The resulting positions are afterstates as discussed in Section 6.8.


g positions that would result. The resulting positions are afterstates as discussed in Section 6.8. The network was consulted to estimate each of their values. The move was then selected that would lead to the position with the highest estimated value. Continuing in this way, with TD-Gammon making the moves for both sides, it was possible to easily generate large numbers of backgammon games. Each game was treated as an episode, with the sequence of positions acting as the states, S0, S1, S2, . . .. Tesauro applied the nonlinear TD rule (16.1) fully incrementally, that is, after each individual move. The weights of the network were set initially to small random values. The initial evaluations were thus entirely arbitrary.


rk were set initially to small random values. The initial evaluations were thus entirely arbitrary. Because the moves were selected on the basis of these evaluations, the initial moves were inevitably poor, and the initial games often lasted hundreds or thousands of moves before one side or the other won, almost by accident. After a few dozen games however, performance improved rapidly. After playing about 300,000 games against itself, TD-Gammon 0.0 as described above learned to play approximately as well as the best previous backgammon computer


16.1. TD-Gammon 425 programs. This was a striking result because all the previous high-performance computer programs had used extensive backgammon knowledge. For example, the reigning champion program at the time was, arguably, Neurogammon, another program written by Tesauro that used an ANN but not TD learning. Neurogammon’s network was trained on a large training corpus of exemplary moves provided by backgammon experts, and, in addition, started with a set of features specially crafted for backgammon. Neurogammon was a highly tuned, highly e↵ective backgammon program that decisively won the World Backgammon Olympiad in 1989. TD-Gammon 0.0, on the other hand, was constructed with essentially zero backgammon knowledge.


1989. TD-Gammon 0.0, on the other hand, was constructed with essentially zero backgammon knowledge. That it was able to do as well as Neurogammon and all other approaches is striking testimony to the potential of self-play learning methods. The tournament success of TD-Gammon 0.0 with zero expert backgammon knowledge suggested an obvious modiﬁcation: add the specialized backgammon features but keep the self-play TD learning method. This produced TD-Gammon 1.0. TD-Gammon 1.0 was clearly substantially better than all previous backgammon programs and found serious competition only among human experts. Later versions of the program, TD-Gammon 2.0 (40 hidden units) and TD-Gammon 2.1 (80 hidden units), were augmented with a selective two-ply search procedure.


its) and TD-Gammon 2.1 (80 hidden units), were augmented with a selective two-ply search procedure. To select moves, these programs looked ahead not just to the positions that would immediately result, but also to the opponent’s possible dice rolls and moves. Assuming the opponent always took the move that appeared immediately best for him, the expected value of each candidate move was computed and the best was selected. To save computer time, the second ply of search was conducted only for candidate moves that were ranked highly after the ﬁrst ply, about four or ﬁve moves on average. Two-ply search a↵ected only the moves selected; the learning process proceeded exactly as before. The ﬁnal versions of the program, TD-Gammon 3.0 and 3.1, used 160 hidden units and a selective three-ply searc


ersions of the program, TD-Gammon 3.0 and 3.1, used 160 hidden units and a selective three-ply search. TD-Gammon illustrates the combination of learned value functions and decision-time search as in heuristic search and MCTS methods. In follow-on work, Tesauro and Galperin (1997) explored trajectory sampling methods as an alternative to full-width search, which reduced the error rate of live play by large numerical factors (4x–6x) while keeping the think time reasonable at ⇠5–10 seconds per move. During the 1990s, Tesauro was able to play his programs in a signiﬁcant number of games against world-class human players. A summary of the results is given in Table 16.1. Program Hidden Training Opponents Results Units Games TD-Gammon 0.0 40 300,000 other programs tied for best TD-Gammon 1.0 80 3


ponents Results Units Games TD-Gammon 0.0 40 300,000 other programs tied for best TD-Gammon 1.0 80 300,000 Robertie, Magriel, ... −13 pts / 51 games TD-Gammon 2.0 40 800,000 various Grandmasters −7 pts / 38 games TD-Gammon 2.1 80 1,500,000 Robertie −1 pt / 40 games TD-Gammon 3.0 80 1,500,000 Kazaros +6 pts / 20 games Table 16.1: Summary of TD-Gammon Results


426 Chapter 16: Applications and Case Studies Based on these results and analyses by backgammon grandmasters (Robertie, 1992; see Tesauro, 1995), TD-Gammon 3.0 appeared to play at close to, or possibly better than, the playing strength of the best human players in the world. Tesauro reported in a subsequent article (Tesauro, 2002) the results of an extensive rollout analysis of the move decisions and doubling decisions of TD-Gammon relative to top human players. The conclusion was that TD-Gammon 3.1 had a “lopsided advantage” in piece-movement decisions, and a “slight edge” in doubling decisions, over top humans. TD-Gammon had a signiﬁcant impact on the way the best human players play the game.


over top humans. TD-Gammon had a signiﬁcant impact on the way the best human players play the game. For example, it learned to play certain opening positions di↵erently than was the convention among the best human players. Based on TD-Gammon’s success and further analysis, the best human players now play these positions as TD-Gammon does (Tesauro, 1995). The impact on human play was greatly accelerated when several other self-teaching ANN backgammon programs inspired by TD-Gammon, such as Jellyﬁsh, Snowie, and GNUBackgammon, became widely available. These programs enabled wide dissemination of new knowledge generated by the ANNs, resulting in great improvements in the overall caliber of human tournament play (Tesauro, 2002).


s, resulting in great improvements in the overall caliber of human tournament play (Tesauro, 2002). 16.2 Samuel’s Checkers Player An important precursor to Tesauro’s TD-Gammon was the seminal work of Arthur Samuel (1959, 1967) in constructing programs for learning to play checkers. Samuel was one of the ﬁrst to make e↵ective use of heuristic search methods and of what we would now call temporal-di↵erence learning. His checkers players are instructive case studies in addition to being of historical interest. We emphasize the relationship of Samuel’s methods to modern reinforcement learning methods and try to convey some of Samuel’s motivation for using them. Samuel ﬁrst wrote a checkers-playing program for the IBM 701 in 1952.


’s motivation for using them. Samuel ﬁrst wrote a checkers-playing program for the IBM 701 in 1952. His ﬁrst learning program was completed in 1955 and was demonstrated on television in 1956. Later versions of the program achieved good, though not expert, playing skill. Samuel was attracted to game-playing as a domain for studying machine learning because games are less complicated than problems “taken from life” while still allowing fruitful study of how heuristic procedures and learning can be used together. He chose to study checkers instead of chess because its relative simplicity made it possible to focus more strongly on learning. Samuel’s programs played by performing a lookahead search from each current position.


on learning. Samuel’s programs played by performing a lookahead search from each current position. They used what we now call heuristic search methods to determine how to expand the search tree and when to stop searching. The terminal board positions of each search were evaluated, or “scored,” by a value function, or “scoring polynomial,” using linear function approximation. In this and other respects Samuel’s work seems to have been inspired by the suggestions of Shannon (1950). In particular, Samuel’s program was based on Shannon’s minimax procedure to ﬁnd the best move from the current position. Working backward through the search tree from the scored terminal positions, each position was given the score of the position that would result from the best move, assuming that the machine wo


s given the score of the position that would result from the best move, assuming that the machine would always try to maximize the score, while the opponent would always try to


16.2. Samuel’s Checkers Player 427 minimize it. Samuel called this the “backed-up score” of the position. When the minimax procedure reached the search tree’s root—the current position—it yielded the best move under the assumption that the opponent would be using the same evaluation criterion, shifted to its point of view. Some versions of Samuel’s programs used sophisticated search control methods analogous to what are known as “alpha-beta” cuto↵s (e.g., see Pearl, 1984). Samuel used two main learning methods, the simplest of which he called rote learning. It consisted simply of saving a description of each board position encountered during play together with its backed-up value determined by the minimax procedure.


tion encountered during play together with its backed-up value determined by the minimax procedure. The result was that if a position that had already been encountered were to occur again as a terminal position of a search tree, the depth of the search was e↵ectively ampliﬁed because this position’s stored value cached the results of one or more searches conducted earlier. One initial problem was that the program was not encouraged to move along the most direct path to a win. Samuel gave it a “a sense of direction” by decreasing a position’s value a small amount each time it was backed up a level (called a ply) during the minimax analysis. “If the program is now faced with a choice of board positions whose scores di↵er only by the ply number, it will automatically make the most advantageou


positions whose scores di↵er only by the ply number, it will automatically make the most advantageous choice, choosing a low-ply alternative if winning and a high-ply alternative if losing” (Samuel, 1959, p. 80). Samuel found this discounting-like technique essential to successful learning. Rote learning produced slow but continual improvement that was most e↵ective for opening and endgame play. His program became a “better-than-average novice” after learning from many games against itself, a variety of human opponents, and from book games in a supervised learning mode. Rote learning and other aspects of Samuel’s work strongly suggest the essential idea of temporal-di↵erence learning—that the value of a state should equal the value of likely following states.


ral-di↵erence learning—that the value of a state should equal the value of likely following states. Samuel came closest to this idea in his second learning method, his “learning by generalization” procedure for modifying the parameters of the value function. Samuel’s method was the same in concept as that used much later by Tesauro in TD-Gammon. He played his program many games against another version of itself and performed an update after each move. The idea of Samuel’s update is suggested by the backup diagram in Figure 16.2. Each open circle represents a position where the program moves next, an on-move position, and each solid circle represents a position where the opponent moves next. An update was made to the value of each on-move position after a move by each side, resulting in a s


An update was made to the value of each on-move position after a move by each side, resulting in a second on-move position. The update was toward the minimax value of a search launched from the second on-move position. Thus, the overall e↵ect was that of a backing-up over one full move of real events and then a search over possible events, as suggested by Figure 16.2. Samuel’s actual algorithm was signiﬁcantly more complex than this for computational reasons, but this was the basic idea. Samuel did not include explicit rewards. Instead, he ﬁxed the weight of the most important feature, the piece advantage feature, which measured the number of pieces the program had relative to how many its opponent had, giving higher weight to kings, and including reﬁnements so that it was better to trade


pponent had, giving higher weight to kings, and including reﬁnements so that it was better to trade pieces when winning than when losing. Thus, the goal of Samuel’s program was to improve its piece advantage, which in checkers is highly correlated with winning.


428 Chapter 16: Applications and Case Studies hypothetical events actual events backup Figure 16.2: The backup diagram for Samuel’s checkers player. However, Samuel’s learning method may have been missing an essential part of a sound temporal-di↵erence algorithm. Temporal-di↵erence learning can be viewed as a way of making a value function consistent with itself, and this we can clearly see in Samuel’s method. But also needed is a way of tying the value function to the true value of the states. We have enforced this via rewards and by discounting or giving a ﬁxed value to the terminal state. But Samuel’s method included no rewards and no special treatment of the terminal positions of games. As Samuel himself pointed out, his value function could have become consistent merely by giving a co


As Samuel himself pointed out, his value function could have become consistent merely by giving a constant value to all positions. He hoped to discourage such solutions by giving his piece-advantage term a large, nonmodiﬁable weight. But although this may decrease the likelihood of ﬁnding useless evaluation functions, it does not prohibit them. For example, a constant function could still be attained by setting the modiﬁable weights so as to cancel the e↵ect of the nonmodiﬁable one. Because Samuel’s learning procedure was not constrained to ﬁnd useful evaluation functions, it should have been possible for it to become worse with experience. In fact, Samuel reported observing this during extensive self-play training sessions.


h experience. In fact, Samuel reported observing this during extensive self-play training sessions. To get the program improving again, Samuel had to intervene and set the weight with the largest absolute value back to zero. His interpretation was that this drastic intervention jarred the program out of local optima, but another possibility is that it jarred the program out of evaluation functions that were consistent but had little to do with winning or losing the game. Despite these potential problems, Samuel’s checkers player using the generalization learning method approached “better-than-average” play. Fairly good amateur opponents characterized it as “tricky but beatable” (Samuel, 1959).


rage” play. Fairly good amateur opponents characterized it as “tricky but beatable” (Samuel, 1959). In contrast to the rote-learning version, this version was able to develop a good middle game but remained weak in opening and endgame play. This program also included an ability to search through sets of features to ﬁnd those that were most useful in forming the value function. A later version (Samuel, 1967) included reﬁnements in its search procedure, such as alpha-beta pruning,


16.3. Watson’s Daily-Double Wagering 429 extensive use of a supervised learning mode called “book learning,” and hierarchical lookup tables called signature tables (Griﬃth, 1966) to represent the value function instead of linear function approximation. This version learned to play much better than the 1959 program, though still not at a master level. Samuel’s checkers-playing program was widely recognized as a signiﬁcant achievement in artiﬁcial intelligence and machine learning. 16.3 Watson’s Daily-Double Wagering IBM Watson1 is the system developed by a team of IBM researchers to play the popular TV quiz show Jeopardy!.2 It gained fame in 2011 by winning ﬁrst prize in an exhibition match against human champions.


rdy!.2 It gained fame in 2011 by winning ﬁrst prize in an exhibition match against human champions. Although the main technical achievement demonstrated by Watson was its ability to quickly and accurately answer natural language questions over broad areas of general knowledge, its winning Jeopardy! performance also relied on sophisticated decision-making strategies for critical parts of the game. Tesauro, Gondek, Lechner, Fan, and Prager (2012, 2013) adapted Tesauro’s TD-Gammon system described above to create the strategy used by Watson in “Daily-Double” (DD) wagering in its celebrated winning performance against human champions. These authors report that the e↵ectiveness of this wagering strategy went well beyond what human players are able to do in live game play, and that it, along wit


trategy went well beyond what human players are able to do in live game play, and that it, along with other advanced strategies, was an important contributor to Watson’s impressive winning performance. Here we focus only on DD wagering because it is the component of Watson that owes the most to reinforcement learning. Jeopardy! is played by three contestants who face a board showing 30 squares, each of which hides a clue and has a dollar value. The squares are arranged in six columns, each corresponding to a di↵erent category. A contestant selects a square, the host reads the square’s clue, and each contestant may choose to respond to the clue by sounding a buzzer (“buzzing in”). The ﬁrst contestant to buzz in gets to try responding to the clue.