ible over its lifetime (and thereby, through the prospective “wisdom” of evolution, improve its chances of solving its real problem, which is to pass its genes on to future generations). However, as we suggest in Chapter 15, it is unlikely that there is a single “master” reward signal like Rt in an animal’s brain. Not all reinforcers are rewards or penalties. Sometimes reinforcement is not the result of an animal receiving a stimulus that evaluates its behavior by labeling the behavior good or bad. A behavior pattern can be reinforced by a stimulus that arrives to an animal no matter how the animal behaved. As described in Section 14.1, whether the delivery of reinforcement depends, or does not depend, on preceding behavior is the deﬁning di↵erence between instrumental, or operant, conditi


not depend, on preceding behavior is the deﬁning di↵erence between instrumental, or operant, conditioning experiments and classical, or Pavlovian, conditioning experiments. Reinforcement is at work in both types of experiments, but only in the former is it feedback that evaluates past behavior. (Though it has often been pointed out that even when the reinforcing US in a classical conditioning experiment is not contingent on the subject’s preceding behavior, its reinforcing value can be inﬂuenced by this behavior, an example being that a closed eye makes an air pu↵to the eye less aversive.) The distinction between reward signals and reinforcement signals is a crucial point when we discuss neural correlates of these signals in the next chapter.


signals is a crucial point when we discuss neural correlates of these signals in the next chapter. Like a reward signal, for us, the reinforcement signal at any speciﬁc time is a positive or negative number, or zero. A reinforcement signal is the major factor directing changes a learning algorithm


14.7. Summary 375 makes in an agent’s policy, value estimates, or environment models. The deﬁnition that makes the most sense to us is that a reinforcement signal at any time is a number that multiplies (possibly along with some constants) a vector to determine parameter updates in some learning algorithm. For some algorithms, the reward signal alone is the critical multiplier in the parameterupdate equation. For these algorithms the reinforcement signal is the same as the reward signal. But for most of the algorithms we discuss in this book, reinforcement signals include terms in addition to the reward signal, an example being a TD error δt = Rt+1 + γV (St+1) −V (St), which is the reinforcement signal for TD state-value learning (and analogous TD errors for action-value learning).


inforcement signal for TD state-value learning (and analogous TD errors for action-value learning). In this reinforcement signal, Rt+1 is the primary reinforcement contribution, and the temporal di↵erence in predicted values, γV (St+1) −V (St) (or an analogous temporal di↵erence for action values), is the conditioned reinforcement contribution. Thus, whenever γV (St+1) −V (St) = 0, δt signals ‘pure’ primary reinforcement; and whenever Rt+1 = 0, it signals ‘pure’ conditioned reinforcement, but it often signals a mixture of these. Note as we mentioned in Section 6.1, this δt is not available until time t + 1. We therefore think of δt as the reinforcement signal at time t + 1, which is ﬁtting because it reinforces predictions and/or actions made earlier at step t.


ime t + 1, which is ﬁtting because it reinforces predictions and/or actions made earlier at step t. A possible source of confusion is the terminology used by the famous psychologist B. F. Skinner and his followers. For Skinner, positive reinforcement occurs when the consequences of an animal’s behavior increase the frequency of that behavior; punishment occurs when the behavior’s consequences decrease that behavior’s frequency. Negative reinforcement occurs when behavior leads to the removal of an aversive stimulus (that is, a stimulus the animal does not like), thereby increasing the frequency of that behavior. Negative punishment, on the other hand, occurs when behavior leads to the removal of an appetitive stimulus (that is, a stimulus the animal likes), thereby decreasing the frequency


l of an appetitive stimulus (that is, a stimulus the animal likes), thereby decreasing the frequency of that behavior. We ﬁnd no critical need for these distinctions because our approach is more abstract than this, with both reward and reinforcement signals allowed to take on both positive and negative values. (But note especially that when our reinforcement signal is negative, it is not the same as Skinner’s negative reinforcement.) On the other hand, it has often been pointed out that using a single number as a reward or a penalty signal, depending only on its sign, is at odds with the fact that animals’ appetitive and aversive systems have qualitatively di↵erent properties and involve di↵erent brain mechanisms.


and aversive systems have qualitatively di↵erent properties and involve di↵erent brain mechanisms. This points to a direction in which the reinforcement learning framework might be developed in the future to exploit computational advantages of separate appetitive and aversive systems, but for now we are passing over these possibilities. Another discrepancy in terminology is how we use the word action. To many cognitive scientists, an action is purposeful in the sense of being the result of an animal’s knowledge about the relationship between the behavior in question and the consequences of that behavior. An action is goal-directed and the result of a decision, whereas a response is triggered by a stimulus and is the result of a reﬂex or a habit.


a decision, whereas a response is triggered by a stimulus and is the result of a reﬂex or a habit. We use the word action without di↵erentiating among what others call actions, decisions, and responses. These are important distinctions, but for us they are encompassed by di↵erences between


376 Chapter 14: Psychology model-free and model-based reinforcement learning algorithms, which we discussed above in relation to habitual and goal-directed behavior in Section 14.6. Dickinson (1985) discusses the distinction between responses and actions. A term used a lot in this book is control. What we mean by control is entirely di↵erent from what it means to animal learning psychologists. By control we mean that an agent inﬂuences its environment to bring about states or events that the agent prefers: the agent exerts control over its environment. This is the sense of control used by control engineers. In psychology, on the other hand, control typically means that an animal’s behavior is inﬂuenced by—is controlled by—the stimuli the animal receives (stimulus control) or the reinforcem


s inﬂuenced by—is controlled by—the stimuli the animal receives (stimulus control) or the reinforcement schedule it experiences. Here the environment is controlling the agent. Control in this sense is the basis of behavior modiﬁcation therapy. Of course, both of these directions of control are at play when an agent interacts with its environment, but our focus is on the agent as controller, not the environment as controller. A view equivalent to ours, and perhaps more illuminating, is that the agent is actually controlling the input it receives from its environment (Powers, 1973). This is not what psychologists mean by stimulus control. Sometimes reinforcement learning is understood to refer solely to learning policies directly from rewards (and penalties) without the involvement of value


solely to learning policies directly from rewards (and penalties) without the involvement of value functions or environment models. This is what psychologists call stimulus-response, or S-R, learning. But for us, along with most of today’s psychologists, reinforcement learning is much broader than this, including in addition to S-R learning, methods involving value functions, environment models, planning, and other processes that are commonly thought to belong to the more cognitive side of mental functioning.


Chapter 15 Neuroscience Neuroscience is the multidisciplinary study of nervous systems: how they regulate bodily functions; control behavior; change over time as a result of development, learning, and aging; and how cellular and molecular mechanisms make these functions possible. One of the most exciting aspects of reinforcement learning is the mounting evidence from neuroscience that the nervous systems of humans and many other animals implement algorithms that correspond in striking ways to reinforcement learning algorithms. The main objective of this chapter is to explain these parallels and what they suggest about the neural basis of reward-related learning in animals. The most remarkable point of contact between reinforcement learning and neuroscience involves dopamine, a chemical dee


e point of contact between reinforcement learning and neuroscience involves dopamine, a chemical deeply involved in reward processing in the brains of mammals. Dopamine appears to convey temporal-di↵erence (TD) errors to brain structures where learning and decision making take place. This parallel is expressed by the reward prediction error hypothesis of dopamine neuron activity, a hypothesis that resulted from the convergence of computational reinforcement learning and results of neuroscience experiments. In this chapter we discuss this hypothesis, the neuroscience ﬁndings that led to it, and why it is a signiﬁcant contribution to understanding brain reward systems. We also discuss parallels between reinforcement learning and neuroscience that are less striking than this dopamine/TD-error


s between reinforcement learning and neuroscience that are less striking than this dopamine/TD-error parallel but that provide useful conceptual tools for thinking about reward-based learning in animals. Other elements of reinforcement learning have the potential to impact the study of nervous systems, but their connections to neuroscience are still relatively undeveloped. We discuss several of these evolving connections that we think will grow in importance over time. As we outlined in the history section of this book’s introductory chapter (Section 1.7), many aspects of reinforcement learning were inﬂuenced by neuroscience. A second objective of this chapter is to acquaint readers with ideas about brain function that have contributed to our approach to reinforcement learning.


rs with ideas about brain function that have contributed to our approach to reinforcement learning. Some elements of reinforcement learning are easier to understand when seen in light of theories of brain function. This is particularly true for the idea of the eligibility trace, one of the basic mechanisms of reinforcement learning, that originated as a conjectured property of synapses, the structures by which nerve cells—neurons—communicate with one another.


378 Chapter 15: Neuroscience In this chapter we do not delve very deeply into the enormous complexity of the neural systems underlying reward-based learning in animals: this chapter is too short, and we are not neuroscientists. We do not try to describe—or even to name—the very many brain structures and pathways, or any of the molecular mechanisms, believed to be involved in these processes. We also do not do justice to hypotheses and models that are alternatives to those that align so well with reinforcement learning. It should not be surprising that there are di↵ering views among experts in the ﬁeld. We can only provide a glimpse into this fascinating and developing story. We hope, though, that this chapter convinces you that a very fruitful channel has emerged connecting reinforcement l


that this chapter convinces you that a very fruitful channel has emerged connecting reinforcement learning and its theoretical underpinnings to the neuroscience of reward-based learning in animals. Many excellent publications cover links between reinforcement learning and neuroscience, some of which we cite in this chapter’s ﬁnal section. Our treatment di↵ers from most of these because we assume familiarity with reinforcement learning as presented in the earlier chapters of this book, but we do not assume knowledge of neuroscience. We begin with a brief introduction to the neuroscience concepts needed for a basic understanding of what is to follow. 15.1 Neuroscience Basics Some basic information about nervous systems is helpful for following what we cover in this chapter.


ome basic information about nervous systems is helpful for following what we cover in this chapter. Terms that we refer to later are italicized. Skipping this section will not be a problem if you already have an elementary knowledge of neuroscience. Neurons, the main components of nervous systems, are cells specialized for processing and transmitting information using electrical and chemical signals. They come in many forms, but a neuron typically has a cell body, dendrites, and a single axon. Dendrites are structures that branch from the cell body to receive input from other neurons (or to also receive external signals in the case of sensory neurons). A neuron’s axon is a ﬁber that carries the neuron’s output to other neurons (or to muscles or glands).


euron’s axon is a ﬁber that carries the neuron’s output to other neurons (or to muscles or glands). A neuron’s output consists of sequences of electrical pulses called action potentials that travel along the axon. Action potentials are also called spikes, and a neuron is said to ﬁre when it generates a spike. In models of neural networks it is common to use real numbers to represent a neuron’s ﬁring rate, the average number of spikes per some unit of time. A neuron’s axon can branch widely so that the neuron’s action potentials reach many targets. The branching structure of a neuron’s axon is called the neuron’s axonal arbor. Because the conduction of an action potential is an active process, not unlike the burning of a fuse, when an action potential reaches an axonal branch point it “ligh


, not unlike the burning of a fuse, when an action potential reaches an axonal branch point it “lights up” action potentials on all of the outgoing branches (although propagation to a branch can sometimes fail). As a result, the activity of a neuron with a large axonal arbor can inﬂuence many target sites.


15.1. Neuroscience Basics 379 A synapse is a structure generally at the termination of an axon branch that mediates the communication of one neuron to another. A synapse transmits information from the presynaptic neuron’s axon to a dendrite or cell body of the postsynaptic neuron. With a few exceptions, synapses release a chemical neurotransmitter upon the arrival of an action potential from the presynaptic neuron. (The exceptions are cases of direct electric coupling between neurons, but these will not concern us here.) Neurotransmitter molecules released from the presynaptic side of the synapse di↵use across the synaptic cleft, the very small space between the presynaptic ending and the postsynaptic neuron, and then bind to receptors on the surface of the postsynaptic neuron to excite or


stsynaptic neuron, and then bind to receptors on the surface of the postsynaptic neuron to excite or inhibit its spike-generating activity, or to modulate its behavior in other ways. A particular neurotransmitter may bind to several di↵erent types of receptors, with each producing a di↵erent e↵ect on the postsynaptic neuron. For example, there are at least ﬁve di↵erent receptor types by which the neurotransmitter dopamine can a↵ect a postsynaptic neuron. Many di↵erent chemicals have been identiﬁed as neurotransmitters in animal nervous systems. A neuron’s background activity is its level of activity, usually its ﬁring rate, when the neuron does not appear to be driven by synaptic input related to the task of interest to the experimenter, for example, when the neuron’s activity is not corre


ed to the task of interest to the experimenter, for example, when the neuron’s activity is not correlated with a stimulus delivered to a subject as part of an experiment. Background activity can be irregular due to input from the wider network, or due to noise within the neuron or its synapses. Sometimes background activity is the result of dynamic processes intrinsic to the neuron. A neuron’s phasic activity, in contrast to its background activity, consists of bursts of spiking activity usually caused by synaptic input. Activity that varies slowly and often in a graded manner, whether as background activity or not, is called a neuron’s tonic activity. The strength or e↵ectiveness by which the neurotransmitter released at a synapse inﬂuences the postsynaptic neuron is the synapse’s eﬃcacy.


he neurotransmitter released at a synapse inﬂuences the postsynaptic neuron is the synapse’s eﬃcacy. One way a nervous system can change through experience is through changes in synaptic eﬃcacies as a result of combinations of the activities of the presynaptic and postsynaptic neurons, and sometimes by the presence of a neuromodulator, which is a neurotransmitter having e↵ects other than, or in addition to, direct fast excitation or inhibition. Brains contain several di↵erent neuromodulation systems consisting of clusters of neurons with widely branching axonal arbors, with each system using a di↵erent neurotransmitter. Neuromodulation can alter the function of neural circuits, mediate motivation, arousal, attention, memory, mood, emotion, sleep, and body temperature.


rcuits, mediate motivation, arousal, attention, memory, mood, emotion, sleep, and body temperature. Important here is that a neuromodulatory system can distribute something like a scalar signal, such as a reinforcement signal, to alter the operation of synapses in widely distributed sites critical for learning. The ability of synaptic eﬃcacies to change is called synaptic plasticity. It is one of the primary mechanisms responsible for learning. The parameters, or weights, adjusted by learning algorithms correspond to synaptic eﬃcacies. As we detail below, modulation of synaptic plasticity via the neuromodulator dopamine is a plausible mechanism for how the brain might implement learning algorithms like many of those described in this book.


380 Chapter 15: Neuroscience 15.2 Reward Signals, Reinforcement Signals, Values, and Prediction Errors Links between neuroscience and computational reinforcement learning begin as parallels between signals in the brain and signals playing prominent roles in reinforcement learning theory and algorithms. In Chapter 3 we said that any problem of learning goal-directed behavior can be reduced to the three signals representing actions, states, and rewards. However, to explain links that have been made between neuroscience and reinforcement learning, we have to be less abstract than this and consider other reinforcement learning signals that correspond, in certain ways, to signals in the brain. In addition to reward signals, these include reinforcement signals (which we argue are di↵erent from r


addition to reward signals, these include reinforcement signals (which we argue are di↵erent from reward signals), value signals, and signals conveying prediction errors. When we label a signal by its function in this way, we are doing it in the context of reinforcement learning theory in which the signal corresponds to a term in an equation or an algorithm. On the other hand, when we refer to a signal in the brain, we mean a physiological event such as a burst of action potentials or the secretion of a neurotransmitter. Labeling a neural signal by its function, for example calling the phasic activity of a dopamine neuron a reinforcement signal, means that the neural signal behaves like, and is conjectured to function like, the corresponding theoretical signal.


ral signal behaves like, and is conjectured to function like, the corresponding theoretical signal. Uncovering evidence for these correspondences involves many challenges. Neural activity related to reward processing can be found in nearly every part of the brain, and it is diﬃcult to interpret results unambiguously because representations of di↵erent reward-related signals tend to be highly correlated with one another. Experiments need to be carefully designed to allow one type of reward-related signal to be distinguished with any degree of certainty from others—or from an abundance of other signals not related to reward processing. Despite these diﬃculties, many experiments have been conducted with the aim of reconciling aspects of reinforcement learning theory and algorithms with neural


cted with the aim of reconciling aspects of reinforcement learning theory and algorithms with neural signals, and some compelling links have been established. To prepare for examining these links, in the rest of this section we remind the reader of what various reward-related signals mean according to reinforcement learning theory. In our Comments on Terminology at the end of the previous chapter, we said that Rt is like a reward signal in an animal’s brain and not an object or event in the animal’s environment. In reinforcement learning, the reward signal (along with an agent’s environment) deﬁnes the problem a reinforcement learning agent is trying to solve. In this respect, Rt is like a signal in an animal’s brain that distributes primary reward to sites throughout the brain.


s like a signal in an animal’s brain that distributes primary reward to sites throughout the brain. But it is unlikely that a unitary master reward signal like Rt exists in an animal’s brain. It is best to think of Rt as an abstraction summarizing the overall e↵ect of a multitude of neural signals generated by many systems in the brain that assess the rewarding or punishing qualities of sensations and states. Reinforcement signals in reinforcement learning are di↵erent from reward signals. The function of a reinforcement signal is to direct the changes a learning algorithm makes in an agent’s policy, value estimates, or environment models. For a TD method, for instance, the reinforcement signal at time t is the TD error δt−1 = Rt + γV (St) −V (St−1).1 The 1As we mentioned in Section 6.1, δ


nal at time t is the TD error δt−1 = Rt + γV (St) −V (St−1).1 The 1As we mentioned in Section 6.1, δt in our notation is deﬁned to be Rt+1 + γV (St+1) −V (St), so δt


15.3. The Reward Prediction Error Hypothesis 381 reinforcement signal for some algorithms could be just the reward signal, but for most of the algorithms we consider the reinforcement signal is the reward signal adjusted by other information, such as the value estimates in TD errors. Estimates of state values or of action values, that is, V or Q, specify what is good or bad for the agent over the long run. They are predictions of the total reward an agent can expect to accumulate over the future. Agents make good decisions by selecting actions leading to states with the largest estimated state values, or by selecting actions with the largest estimated action values. Prediction errors measure discrepancies between expected and actual signals or sensations.


values. Prediction errors measure discrepancies between expected and actual signals or sensations. Reward prediction errors (RPEs) speciﬁcally measure discrepancies between the expected and the received reward signal, being positive when the reward signal is greater than expected, and negative otherwise. TD errors like (6.5) are special kinds of RPEs that signal discrepancies between current and earlier expectations of reward over the long-term. When neuroscientists refer to RPEs they generally (though not always) mean TD RPEs, which we simply call TD errors throughout this chapter. Also in this chapter, a TD error is generally one that does not depend on actions, as opposed to TD errors used in learning action-values by algorithms like Sarsa and Q-learning.


ns, as opposed to TD errors used in learning action-values by algorithms like Sarsa and Q-learning. This is because the most well-known links to neuroscience are stated in terms of action-free TD errors, but we do not mean to rule out possible similar links involving action-dependent TD errors. (TD errors for predicting signals other than rewards are useful too, but that case will not concern us here. See, for example, Modayil, White, and Sutton, 2014.) One can ask many questions about links between neuroscience data and these theoreticallydeﬁned signals. Is an observed signal more like a reward signal, a value signal, a prediction error, a reinforcement signal, or something altogether di↵erent?


gnal, a value signal, a prediction error, a reinforcement signal, or something altogether di↵erent? And if it is an error signal, is it an RPE, a TD error, or a simpler error like the Rescorla–Wagner error (14.3)? And if it is a TD error, does it depend on actions like the TD error of Q-learning or Sarsa? As indicated above, probing the brain to answer questions like these is extremely diﬃcult. But experimental evidence suggests that one neurotransmitter, speciﬁcally the neurotransmitter dopamine, signals RPEs, and further, that the phasic activity of dopamine-producing neurons in fact conveys TD errors (see Section 15.1 for a deﬁnition of phasic activity). This evidence led to the reward prediction error hypothesis of dopamine neuron activity, which we describe next.


led to the reward prediction error hypothesis of dopamine neuron activity, which we describe next. 15.3 The Reward Prediction Error Hypothesis The reward prediction error hypothesis of dopamine neuron activity proposes that one of the functions of the phasic activity of dopamine-producing neurons in mammals is to deliver an error between an old and a new estimate of expected future reward to target areas throughout the brain. This hypothesis (though not in these exact words) was ﬁrst explicitly stated by Montague, Dayan, and Sejnowski (1996), who showed how the TD error concept from reinforcement learning accounts for many features of the phasic is not available until time t + 1. The TD error available at t is actually δt−1 = Rt + γV (St) −V (St−1).


available until time t + 1. The TD error available at t is actually δt−1 = Rt + γV (St) −V (St−1). Because we are thinking of time steps as very small, or even inﬁnitesimal, time intervals, one should not attribute undue importance to this one-step time shift.


382 Chapter 15: Neuroscience activity of dopamine neurons in mammals. The experiments that led to this hypothesis were performed in the 1980s and early 1990s in the laboratory of neuroscientist Wolfram Schultz. Section 15.5 describes these inﬂuential experiments, Section 15.6 explains how the results of these experiments align with TD errors, and the Bibliographical and Historical Remarks section at the end of this chapter includes a guide to the literature surrounding the development of this inﬂuential hypothesis. Montague et al. (1996) compared the TD errors of the TD model of classical conditioning with the phasic activity of dopamine-producing neurons during classical conditioning experiments.


g with the phasic activity of dopamine-producing neurons during classical conditioning experiments. Recall from Section 14.2 that the TD model of classical conditioning is basically the semi-gradient-descent TD(λ) algorithm with linear function approximation. Montague et al. made several assumptions to set up this comparison. First, because a TD error can be negative but neurons cannot have a negative ﬁring rate, they assumed that the quantity corresponding to dopamine neuron activity is δt−1 + bt, where bt is the background ﬁring rate of the neuron. A negative TD error corresponds to a drop in a dopamine neuron’s ﬁring rate below its background rate.2 A second assumption was needed about the states visited in each classical conditioning trial and how they are represented as inputs to the


e states visited in each classical conditioning trial and how they are represented as inputs to the learning algorithm. This is the same issue we discussed in Section 14.2.4 for the TD model. Montague et al. chose a complete serial compound (CSC) representation as shown in the left column of Figure 14.1, but where the sequence of short-duration internal signals continues until the onset of the US, which here is the arrival of a non-zero reward signal. This representation allows the TD error to mimic the fact that dopamine neuron activity not only predicts a future reward, but that it is also sensitive to when after a predictive cue that reward is expected to arrive. There has to be some way to keep track of the time between sensory cues and the arrival of reward.


There has to be some way to keep track of the time between sensory cues and the arrival of reward. If a stimulus initiates a sequence of internal signals that continues after the stimulus ends, and if there is a di↵erent signal for each time step following the stimulus, then each time step after the stimulus is represented by a distinct state. Thus, the TD error, being state-dependent, can be sensitive to the timing of events within a trial. In simulated trials with these assumptions about background ﬁring rate and input representation, TD errors of the TD model are remarkably similar to dopamine neuron phasic activity. Previewing our description of details about these similarities in Section 15.5 below, the TD errors parallel the following features of dopamine neuron activity: (1) the ph


on 15.5 below, the TD errors parallel the following features of dopamine neuron activity: (1) the phasic response of a dopamine neuron only occurs when a rewarding event is unpredicted; (2) early in learning, neutral cues that precede a reward do not cause substantial phasic dopamine responses, but with continued learning these cues gain predictive value and come to elicit phasic dopamine responses; (3) if an even earlier cue reliably precedes a cue that has already acquired predictive value, the phasic dopamine response shifts to the earlier cue, ceasing for the later cue; and (4) if after learning, the predicted rewarding event is omitted, a dopamine neuron’s response decreases below its baseline level shortly after the expected time of the rewarding event.


response decreases below its baseline level shortly after the expected time of the rewarding event. 2In the literature relating TD errors to the activity of dopamine neurons, their δt is the same as our δt−1 = Rt + γV (St) −V (St−1).


15.4. Dopamine 383 Although not every dopamine neuron monitored in the experiments of Schultz and colleagues behaved in all of these ways, the striking correspondence between the activities of most of the monitored neurons and TD errors lends strong support to the reward prediction error hypothesis. There are situations, however, in which predictions based on the hypothesis do not match what is observed in experiments. The choice of input representation is critical to how closely TD errors match some of the details of dopamine neuron activity, particularly details about the timing of dopamine neuron responses. Di↵erent ideas, some of which we discuss below, have been proposed about input representations and other features of TD learning to make the TD errors ﬁt the data better, though the


resentations and other features of TD learning to make the TD errors ﬁt the data better, though the main parallels appear with the CSC representation that Montague et al. used. Overall, the reward prediction error hypothesis has received wide acceptance among neuroscientists studying reward-based learning, and it has proven to be remarkably resilient in the face of accumulating results from neuroscience experiments. To prepare for our description of the neuroscience experiments supporting the reward prediction error hypothesis, and to provide some context so that the signiﬁcance of the hypothesis can be appreciated, we next present some of what is known about dopamine, the brain structures it inﬂuences, and how it is involved in reward-based learning.


about dopamine, the brain structures it inﬂuences, and how it is involved in reward-based learning. 15.4 Dopamine Dopamine is produced as a neurotransmitter by neurons whose cell bodies lie mainly in two clusters of neurons in the midbrain of mammals: the substantia nigra pars compacta (SNpc) and the ventral tegmental area (VTA). Dopamine plays essential roles in many processes in the mammalian brain. Prominent among these are motivation, learning, action-selection, most forms of addiction, and the disorders schizophrenia and Parkinson’s disease. Dopamine is called a neuromodulator because it performs many functions other than direct fast excitation or inhibition of targeted neurons. Although much remains unknown about dopamine’s functions and details of its cellular e↵ects, it is clear th


h much remains unknown about dopamine’s functions and details of its cellular e↵ects, it is clear that it is fundamental to reward processing in the mammalian brain. Dopamine is not the only neuromodulator involved in reward processing, and its role in aversive situations—punishment—remains controversial. Dopamine also can function di↵erently in non-mammals. But no one doubts that dopamine is essential for reward-related processes in mammals, including humans. An early, traditional view is that dopamine neurons broadcast a reward signal to multiple brain regions implicated in learning and motivation. This view followed from a famous 1954 paper by James Olds and Peter Milner that described the e↵ects of electrical stimulation on certain areas of a rat’s brain.


Peter Milner that described the e↵ects of electrical stimulation on certain areas of a rat’s brain. They found that electrical stimulation to particular regions acted as a very powerful reward in controlling the rat’s behavior: “...the control exercised over the animal’s behavior by means of this reward is extreme, possibly exceeding that exercised by any other reward previously used in animal experimentation” (Olds and Milner, 1954). Later research revealed that the sites at which stimulation was most e↵ective in producing this rewarding e↵ect excited dopamine pathways, either directly or indirectly, that ordinarily are excited by natural rewarding stimuli. E↵ects similar to these were also observed with human subjects.


cited by natural rewarding stimuli. E↵ects similar to these were also observed with human subjects. These observations strongly suggested that dopamine neuron activity signals reward.


384 Chapter 15: Neuroscience But if the reward prediction error hypothesis is correct—even if it accounts for only some features of a dopamine neuron’s activity—this traditional view of dopamine neuron activity is not entirely correct: phasic responses of dopamine neurons signal reward prediction errors, not reward itself. In reinforcement learning’s terms, a dopamine neuron’s phasic response at a time t corresponds to δt−1 = Rt + γV (St) −V (St−1), not to Rt. Reinforcement learning theory and algorithms help reconcile the reward-predictionerror view with the conventional notion that dopamine signals reward. In many of the algorithms we discuss in this book, δ functions as a reinforcement signal, meaning that it is the main driver of learning.


n this book, δ functions as a reinforcement signal, meaning that it is the main driver of learning. For example, δ is the critical factor in the TD model of classical conditioning, and δ is the reinforcement signal for learning both a value function and a policy in an actor–critic architecture (Sections 13.5 and 15.7). Action-dependent forms of δ are reinforcement signals for Q-learning and Sarsa. The reward signal Rt is a crucial component of δt−1, but it is not the complete determinant of its reinforcing e↵ect in these algorithms. The additional term γV (St) −V (St−1) is the higher-order reinforcement part of δt−1, and even if reward occurs (Rt 6= 0), the TD error can be silent if the reward is fully predicted (which is fully explained in Section 15.6 below).


or can be silent if the reward is fully predicted (which is fully explained in Section 15.6 below). A closer look at Olds’ and Milner’s 1954 paper, in fact, reveals that it is mainly about the reinforcing e↵ect of electrical stimulation in an instrumental conditioning task. Electrical stimulation not only energized the rats’ behavior—through dopamine’s e↵ect on motivation—it also led to the rats quickly learning to stimulate themselves by pressing a lever, which they would do frequently for long periods of time. The activity of dopamine neurons triggered by electrical stimulation reinforced the rats’ lever pressing. More recent experiments using optogenetic methods clinch the role of phasic responses of dopamine neurons as reinforcement signals.


togenetic methods clinch the role of phasic responses of dopamine neurons as reinforcement signals. These methods allow neuroscientists to precisely control the activity of selected neuron types at a millisecond timescale in awake behaving animals. Optogenetic methods introduce light-sensitive proteins into selected neuron types so that these neurons can be activated or silenced by means of ﬂashes of laser light. The ﬁrst experiment using optogenetic methods to study dopamine neurons showed that optogenetic stimulation producing phasic activation of dopamine neurons in mice was enough to condition the mice to prefer the side of a chamber where they received this stimulation as compared to the chamber’s other side where they received no, or lower-frequency, stimulation (Tsai et al. 2009).


he chamber’s other side where they received no, or lower-frequency, stimulation (Tsai et al. 2009). In another example, Steinberg et al. (2013) used optogenetic activation of dopamine neurons to create artiﬁcial bursts of dopamine neuron activity in rats at the times when rewarding stimuli were expected but omitted—times when dopamine neuron activity normally pauses. With these pauses replaced by artiﬁcial bursts, responding was sustained when it would ordinarily decrease due to lack of reinforcement (in extinction trials), and learning was enabled when it would ordinarily be blocked due to the reward being already predicted (the blocking paradigm; Section 14.2.1). Additional evidence for the reinforcing function of dopamine comes from optogenetic experiments with fruit ﬂies, except in the


e reinforcing function of dopamine comes from optogenetic experiments with fruit ﬂies, except in these animals dopamine’s e↵ect is the opposite of its e↵ect in mammals: optically triggered bursts of dopamine neuron activity act just like electric foot shock in reinforcing avoidance behavior, at least for the population


15.4. Dopamine 385 of dopamine neurons activated (Claridge-Chang et al. 2009). Although none of these optogenetic experiments showed that phasic dopamine neuron activity is speciﬁcally like a TD error, they convincingly demonstrated that phasic dopamine neuron activity acts just like δ acts (or perhaps like minus δ acts in fruit ﬂies) as the reinforcement signal in algorithms for both prediction (classical conditioning) and control (instrumental conditioning). Axonal arbor of a single neuron producing dopamine as a neurotransmitter. These axons make synaptic contacts with a huge number of dendrites of neurons in targeted brain areas. Adapted from The Journal of Neuroscience, Matsuda, Furuta, Nakamura, Hioki, Fujiyama, Arai, and Kaneko, volume 29, 2009, page 451.


roscience, Matsuda, Furuta, Nakamura, Hioki, Fujiyama, Arai, and Kaneko, volume 29, 2009, page 451. Dopamine neurons are particularly well suited to broadcasting a reinforcement signal to many areas of the brain. These neurons have huge axonal arbors, each releasing dopamine at 100 to 1,000 times more synaptic sites than reached by the axons of typical neurons. Shown to the right is the axonal arbor of a single dopamine neuron whose cell body is in the SNpc of a rat’s brain. Each axon of a SNpc or VTA dopamine neuron makes roughly 500,000 synaptic contacts on the dendrites of neurons in targeted brain areas. If dopamine neurons broadcast a reinforcement signal like reinforcement learning’s δ, then because this is a scalar signal, i.e., a single number, all dopamine neurons in both the SNpc


, then because this is a scalar signal, i.e., a single number, all dopamine neurons in both the SNpc and VTA would be expected to activate moreor-less identically so that they would act in near synchrony to send the same signal to all of the sites their axons target. Although it has been a common belief that dopamine neurons do act together like this, modern evidence is pointing to the more complicated picture that di↵erent subpopulations of dopamine neurons respond to input di↵erently depending on the structures to which they send their signals and the di↵erent ways these signals act on their target structures. Dopamine has functions other than signaling RPEs, and even for dopamine neurons that do signal RPEs, it can make sense to send di↵erent RPEs to di↵erent structures depending on the


that do signal RPEs, it can make sense to send di↵erent RPEs to di↵erent structures depending on the roles these structures play in producing reinforced behavior. This is beyond what we treat in any detail in this book, but vector-valued RPE signals make sense from the perspective of reinforcement learning when decisions can be decomposed into separate sub-decisions, or more generally, as a way to address the structural version of the credit assignment problem: How do you distribute credit for success (or blame for failure) of a decision among the many component structures that could have been involved in producing it? We say a bit more about this in Section 15.10 below. The axons of most dopamine neurons make synaptic contact with neurons in the frontal cortex and the basal ganglia, areas


pamine neurons make synaptic contact with neurons in the frontal cortex and the basal ganglia, areas of the brain involved in voluntary movement, decision making, learning, and cognitive functions such as planning. Because most ideas relating


386 Chapter 15: Neuroscience dopamine to reinforcement learning focus on the basal ganglia, and the connections from dopamine neurons are particularly dense there, we focus on the basal ganglia here. The basal ganglia are a collection of neuron groups, or nuclei, lying at the base of the forebrain. The main input structure of the basal ganglia is called the striatum. Essentially all of the cerebral cortex, among other structures, provides input to the striatum. The activity of cortical neurons conveys a wealth of information about sensory input, internal states, and motor activity. The axons of cortical neurons make synaptic contacts on the dendrites of the main input/output neurons of the striatum, called medium spiny neurons.


cts on the dendrites of the main input/output neurons of the striatum, called medium spiny neurons. Output from the striatum loops back via other basal ganglia nuclei and the thalamus to frontal areas of cortex, and to motor areas, making it possible for the striatum to inﬂuence movement, abstract decision processes, and reward processing. Two main subdivisions of the striatum are important for reinforcement learning: the dorsal striatum, primarily implicated in inﬂuencing action selection, and the ventral striatum, thought to be critical for di↵erent aspects of reward processing, including the assignment of a↵ective value to sensations. The dendrites of medium spiny neurons are covered with spines on whose tips the axons of neurons in the cortex make synaptic contact.


ons are covered with spines on whose tips the axons of neurons in the cortex make synaptic contact. Also making synaptic contact with these spines—in this case contacting the spine stems—are axons of dopamine neurons (Figure 15.1). This arrangement brings together presynaptic activity of cortical neurons, Figure 15.1: Spine of a striatal neuron showing input from both cortical and dopamine neurons. Axons of cortical neurons inﬂuence striatal neurons via corticostriatal synapses releasing the neurotransmitter glutamate at the tips of spines covering the dendrites of striatal neurons. An axon of a VTA or SNpc dopamine neuron is shown passing by the spine (from the lower right). “Dopamine varicosities” on this axon release dopamine at or near the spine stem, in an arrangement that brings toge


osities” on this axon release dopamine at or near the spine stem, in an arrangement that brings together presynaptic input from cortex, postsynaptic activity of the striatal neuron, and dopamine, making it possible that several types of learning rules govern the plasticity of corticostriatal synapses. Each axon of a dopamine neuron makes synaptic contact with the stems of roughly 500,000 spines. Some of the complexity omitted from our discussion is shown here by other neurotransmitter pathways and multiple receptor types, such as D1 an D2 dopamine receptors by which dopamine can produce di↵erent e↵ects at spines and other postsynaptic sites. From Journal of Neurophysiology, W. Schultz, vol. 80, 1998, page 10.


15.5. Experimental Support for the Reward Prediction Error Hypothesis 387 postsynaptic activity of medium spiny neurons, and input from dopamine neurons. What actually occurs at these spines is complex and not completely understood. Figure 15.1 hints at the complexity by showing two types of receptors for dopamine, receptors for glutamate—the neurotransmitter of the cortical inputs—and multiple ways that the various signals can interact. But evidence is mounting that changes in the eﬃcacies of the synapses on the pathway from the cortex to the striatum, which neuroscientists call corticostriatal synapses, depend critically on appropriately-timed dopamine signals. 15.5 Experimental Support for the Reward Prediction Error Hypothesis Dopamine neurons respond with bursts of activity to intense


r the Reward Prediction Error Hypothesis Dopamine neurons respond with bursts of activity to intense, novel, or unexpected visual and auditory stimuli that trigger eye and body movements, but very little of their activity is related to the movements themselves. This is surprising because degeneration of dopamine neurons is a cause of Parkinson’s disease, whose symptoms include motor disorders, particularly deﬁcits in self-initiated movement. Motivated by the weak relationship between dopamine neuron activity and stimulus-triggered eye and body movements, Romo and Schultz (1990) and Schultz and Romo (1990) took the ﬁrst steps toward the reward prediction error hypothesis by recording the activity of dopamine neurons and muscle activity while monkeys moved their arms.


s by recording the activity of dopamine neurons and muscle activity while monkeys moved their arms. They trained two monkeys to reach from a resting hand position into a bin containing a bit of apple, a piece of cookie, or a raisin, when the monkey saw and heard the bin’s door open. The monkey could then grab and bring the food to its mouth. After a monkey became good at this, it was trained on two additional tasks. The purpose of the ﬁrst task was to see what dopamine neurons do when movements are self-initiated. The bin was left open but covered from above so that the monkey could not see inside but could reach in from below. No triggering stimuli were presented, and after the monkey reached for and ate the food morsel, the experimenter usually (though not always), silently and unseen by


ed for and ate the food morsel, the experimenter usually (though not always), silently and unseen by the monkey, replaced food in the bin by sticking it onto a rigid wire. Here too, the activity of the dopamine neurons Romo and Schultz monitored was not related to the monkey’s movements, but a large percentage of these neurons produced phasic responses whenever the monkey ﬁrst touched a food morsel. These neurons did not respond when the monkey touched just the wire or explored the bin when no food was there. This was good evidence that the neurons were responding to the food and not to other aspects of the task. The purpose of Romo and Schultz’s second task was to see what happens when movements are triggered by stimuli. This task used a di↵erent bin with a movable cover.


appens when movements are triggered by stimuli. This task used a di↵erent bin with a movable cover. The sight and sound of the bin opening triggered reaching movements to the bin. In this case, Romo and Schultz found that after some period of training, the dopamine neurons no longer responded to the touch of the food but instead responded to the sight and sound of the opening cover of the food bin. The phasic responses of these neurons had shifted from the reward itself to stimuli predicting the availability of the reward. In a followup study, Romo and Schultz found that most of the dopamine neurons whose activity they


388 Chapter 15: Neuroscience monitored did not respond to the sight and sound of the bin opening outside the context of the behavioral task. These observations suggested that the dopamine neurons were responding neither to the initiation of a movement nor to the sensory properties of the stimuli, but were rather signaling an expectation of reward. Schultz’s group conducted many additional studies involving both SNpc and VTA dopamine neurons. A particular series of experiments was inﬂuential in suggesting that the phasic responses of dopamine neurons correspond to TD errors and not to simpler errors like those in the Rescorla–Wagner model (14.3). In the ﬁrst of these experiments (Ljungberg, Apicella, and Schultz, 1992), monkeys were trained to depress a lever after a light was illuminated a


picella, and Schultz, 1992), monkeys were trained to depress a lever after a light was illuminated as a ‘trigger cue’ to obtain a drop of apple juice. As Romo and Schultz had observed earlier, many dopamine neurons initially responded to the reward—the drop of juice (Figure 15.2, top panel). But many of these neurons lost that reward response as training continued and developed responses instead to the illumination of the light that predicted the reward (Figure 15.2, middle panel). With continued training, lever pressing became faster while the number of dopamine neurons responding to the trigger cue decreased. Following this study, the same monkeys were trained on a new task (Schultz, Apicella, and Ljungberg, 1993). Here the monkeys faced two levers, each with a light above it.


ltz, Apicella, and Ljungberg, 1993). Here the monkeys faced two levers, each with a light above it. Illuminating one of these lights was an ‘instruction cue’ indicating which of the two levers Figure 15.2: The response of dopamine neurons shifts from initial responses to primary reward to earlier predictive stimuli. These are plots of the number of action potentials produced by monitored dopamine neurons within small time intervals, averaged over all the monitored dopamine neurons (ranging from 23 to 44 neurons for these data). Top: dopamine neurons are activated by the unpredicted delivery of drop of apple juice. Middle: with learning, dopamine neurons developed responses to the reward-predicting trigger cue and lost responsiveness to the delivery of reward.


d responses to the reward-predicting trigger cue and lost responsiveness to the delivery of reward. Bottom: with the addition of an instruction cue preceding the trigger cue by 1 second, dopamine neurons shifted their responses from the trigger cue to the earlier instruction cue. From Schultz et al. (1995), MIT Press.


15.5. Experimental Support for the Reward Prediction Error Hypothesis 389 Figure 15.3: The response of dopamine neurons drops below baseline shortly after the time when an expected reward fails to occur. Top: dopamine neurons are activated by the unpredicted delivery of a drop of apple juice. Middle: dopamine neurons respond to a conditioned stimulus (CS) that predicts reward and do not respond to the reward itself. Bottom: when the reward predicted by the CS fails to occur, the activity of dopamine neurons drops below baseline shortly after the time the reward is expected to occur. At the top of each of these panels is shown the average number of action potentials produced by monitored dopamine neurons within small time intervals around the indicated times.


ials produced by monitored dopamine neurons within small time intervals around the indicated times. The raster plots below show the activity patterns of the individual dopamine neurons that were monitored; each dot represents an action potential. From Schultz, Dayan, and Montague, A Neural Substrate of Prediction and Reward, Science, vol. 275, issue 5306, pages 1593-1598, March 14, 1997. Reprinted with permission from AAAS. would produce a drop of apple juice. In this task, the instruction cue preceded the trigger cue of the previous task by a ﬁxed interval of 1 second. The monkeys learned to withhold reaching until seeing the trigger cue, and dopamine neuron activity increased, but now the responses of the monitored dopamine neurons occurred almost exclusively to the earlier instruction c


responses of the monitored dopamine neurons occurred almost exclusively to the earlier instruction cue and not to the trigger cue (Figure 15.2, bottom panel). Here again the number of dopamine neurons responding to the instruction cue was much reduced when the task was well learned. During learning across these tasks, dopamine neuron activity shifted from initially responding to the reward to responding to the earlier predictive stimuli, ﬁrst progressing to the trigger stimulus then to the still earlier instruction cue. As responding moved earlier in time it disappeared from the later stimuli. This shifting of responses to earlier reward predictors, while losing responses to later predictors is a hallmark of TD learning (see, for example, Figure 14.2).


losing responses to later predictors is a hallmark of TD learning (see, for example, Figure 14.2). The task just described revealed another property of dopamine neuron activity shared with TD learning. The monkeys sometimes pressed the wrong key, that is, the key other than the instructed one, and consequently received no reward. In these trials, many of the dopamine neurons showed a sharp decrease in their ﬁring rates below baseline shortly after the reward’s usual time of delivery, and this happened without the availability of any external cue to mark the usual time of reward delivery (Figure 15.3). Somehow the


390 Chapter 15: Neuroscience monkeys were internally keeping track of the timing of the reward. (Response timing is one area where the simplest version of TD learning needs to be modiﬁed to account for some of the details of the timing of dopamine neuron responses. We consider this issue in the following section.) The observations from the studies described above led Schultz and his group to conclude that dopamine neurons respond to unpredicted rewards, to the earliest predictors of reward, and that dopamine neuron activity decreases below baseline if a reward, or a predictor of reward, does not occur at its expected time. Researchers familiar with reinforcement learning were quick to recognize that these results are strikingly similar to how the TD error behaves as the reinforcement signa


ize that these results are strikingly similar to how the TD error behaves as the reinforcement signal in a TD algorithm. The next section explores this similarity by working through a speciﬁc example in detail. 15.6 TD Error/Dopamine Correspondence This section explains the correspondence between the TD error δ and the phasic responses of dopamine neurons observed in the experiments just described. We examine how δ changes over the course of learning in a task something like the one described above where a monkey ﬁrst sees an instruction cue and then a ﬁxed time later has to respond correctly to a trigger cue in order to obtain reward. We use a simple idealized version of this task, but we go into a lot more detail than is usual because we want to emphasize the theoretical basis of the par


o into a lot more detail than is usual because we want to emphasize the theoretical basis of the parallel between TD errors and dopamine neuron activity. The ﬁrst simplifying assumption is that the agent has already learned the actions required to obtain reward. Then its task is just to learn accurate predictions of future reward for the sequence of states it experiences. This is then a prediction task, or more technically, a policy-evaluation task: learning the value function for a ﬁxed policy (Sections 4.1 and 6.1). The value function to be learned assigns to each state a value that predicts the return that will follow that state if the agent selects actions according to the given policy, where the return is the (possibly discounted) sum of all the future rewards.


g to the given policy, where the return is the (possibly discounted) sum of all the future rewards. This is unrealistic as a model of the monkey’s situation because the monkey would likely learn these predictions at the same time that it is learning to act correctly (as would a reinforcement learning algorithm that learns policies as well as value functions, such as an actor–critic algorithm), but this scenario is simpler to describe than one in which a policy and a value function are learned simultaneously. Now imagine that the agent’s experience divides into multiple trials, in each of which the same sequence of states repeats, with a distinct state occurring on each time step during the trial.


ame sequence of states repeats, with a distinct state occurring on each time step during the trial. Further imagine that the return being predicted is limited to the return over a trial, which makes a trial analogous to a reinforcement learning episode as we have deﬁned it. In reality, of course, the returns being predicted are not conﬁned to single trials, and the time interval between trials is an important factor in determining what an animal learns. This is true for TD learning as well, but here we assume that returns do not accumulate over multiple trials. Given this, then, a trial in experiments like those conducted by Schultz and colleagues is equivalent to an episode of reinforcement learning.


ke those conducted by Schultz and colleagues is equivalent to an episode of reinforcement learning. (Though in this discussion, we will use the term trial instead of episode to relate better to the experiments.)


15.6. TD Error/Dopamine Correspondence 391 As usual, we also need to make an assumption about how states are represented as inputs to the learning algorithm, an assumption that inﬂuences how closely the TD error corresponds to dopamine neuron activity. We discuss this issue later, but for now we assume the same CSC representation used by Montague et al. (1996) in which there is a separate internal stimulus for each state visited at each time step in a trial. This reduces the process to the tabular case covered in the ﬁrst part of this book. Finally, we assume that the agent uses TD(0) to learn a value function, V , stored in a lookup table initialized to be zero for all the states. We also assume that this is a deterministic task and that the discount factor, γ, is very nearly one so that


ssume that this is a deterministic task and that the discount factor, γ, is very nearly one so that we can ignore it. Figure 15.4 shows the time courses of R, V , and δ at several stages of learning in this policy-evaluation task. The time axes represent the time interval over which a sequence of states is visited in a trial (where for clarity we omit showing individual states). The reward signal is zero throughout each trial except when the agent reaches the rewarding state, shown near the right end of the time line, when the reward signal becomes some positive number, say R?. The goal of TD learning is to predict the return for each state visited in a trial, which in this undiscounted case and given our assumption that predictions are conﬁned to individual trials, is simply R?


unted case and given our assumption that predictions are conﬁned to individual trials, is simply R? for each state. R RR R t R R V V δ δ δ early in learning learning complete omitted R regular predictors of over this interval R R? Figure 15.4: The behavior of the TD error δ during TD learning is consistent with features of the phasic activation of dopamine neurons. (Here δ is the TD error available at time t, i.e., δt−1). Top: a sequence of states, shown as an interval of regular predictors, is followed by a non-zero reward R?. Early in learning: the initial value function, V , and initial δ, which at ﬁrst is equal to R?. Learning complete: the value function accurately predicts future reward, δ is positive at the earliest predictive state, and δ = 0 at the time of the non-zero reward.


ward, δ is positive at the earliest predictive state, and δ = 0 at the time of the non-zero reward. R? omitted: at the time the predicted reward is omitted, δ becomes negative. See text for a complete explanation of why this happens.


392 Chapter 15: Neuroscience Preceding the rewarding state is a sequence of reward-predicting states, with the earliest reward-predicting state shown near the left end of the time line. This is like the state near the start of a trial, for example like the state marked by the instruction cue in a trial of the monkey experiment of Schultz et al. (1993) described above. It is the ﬁrst state in a trial that reliably predicts that trial’s reward. (Of course, in reality states visited on preceding trials are even earlier reward-predicting states, but because we are conﬁning predictions to individual trials, these do not qualify as predictors of this trial’s reward. Below we give a more satisfactory, though more abstract, description of an earliest reward-predicting state.) The latest reward-pre


ry, though more abstract, description of an earliest reward-predicting state.) The latest reward-predicting state in a trial is the state immediately preceding the trial’s rewarding state. This is the state near the far right end of the time line in Figure 15.4. Note that the rewarding state of a trial does not predict the return for that trial: the value of this state would come to predict the return over all the following trials, which here we are assuming to be zero in this episodic formulation. Figure 15.4 shows the ﬁrst-trial time courses of V and δ as the graphs labeled ‘early in learning.’ Because the reward signal is zero throughout the trial except when the rewarding state is reached, and all the V -values are zero, the TD error is also zero until it becomes R?


ing state is reached, and all the V -values are zero, the TD error is also zero until it becomes R? at the rewarding state. This follows because δt−1 = Rt + Vt −Vt−1 = Rt + 0 −0 = Rt, which is zero until it equals R? when the reward occurs. Here Vt and Vt−1 are respectively the estimated values of the states visited at times t and t −1 in a trial. The TD error at this stage of learning is analogous to a dopamine neuron responding to an unpredicted reward (e.g., a drop of apple juice) at the start of training. Throughout this ﬁrst trial and all successive trials, TD(0) updates occur at each state transition as described in Chapter 6. This successively increases the values of the reward-predicting states, with the increases spreading backwards from the rewarding state, until the values conve


ting states, with the increases spreading backwards from the rewarding state, until the values converge to the correct return predictions. In this case (because we are assuming no discounting) the correct predictions are equal to R? for all the reward-predicting states. This can be seen in Figure 15.4 as the graph of V labeled ‘learning complete’ where the values of all the states from the earliest to the latest reward-predicting states all equal R?. The values of the states preceding the earliest reward-predicting state remain low (which Figure 15.4 shows as zero) because they are not reliable predictors of reward. When learning is complete, that is, when V attains its correct values, the TD errors associated with transitions from any reward-predicting state are zero because the predictio


D errors associated with transitions from any reward-predicting state are zero because the predictions are now accurate. This is because for a transition from a reward-predicting state to another reward-predicting state, we have δt−1 = Rt + Vt −Vt−1 = 0 + R? −R? = 0, and for the transition from the latest reward-predicting state to the rewarding state, we have δt−1 = Rt +Vt −Vt−1 = R? +0−R? = 0. On the other hand, the TD error on a transition from any state to the earliest reward-predicting state is positive because of the mismatch between this state’s low value and the larger value of the following reward-predicting state. Indeed, if the value of a state preceding the earliest reward-predicting state were zero, then after the transition to the earliest reward-predicting state, we would ha


ting state were zero, then after the transition to the earliest reward-predicting state, we would have that δt−1 = Rt + Vt −Vt−1 = 0 + R? −0 = R?. The ‘learning complete’ graph of δ in Figure 15.4 shows this positive value at the earliest reward-predicting state, and zeros everywhere else.


15.6. TD Error/Dopamine Correspondence 393 The positive TD error upon transitioning to the earliest reward-predicting state is analogous to the persistence of dopamine responses to the earliest stimuli predicting reward. By the same token, when learning is complete, a transition from the latest reward-predicting state to the rewarding state produces a zero TD error because the latest reward-predicting state’s value, being correct, cancels the reward. This parallels the observation that fewer dopamine neurons generate a phasic response to a fully predicted reward than to an unpredicted reward. After learning, if the reward is suddenly omitted, the TD error goes negative at the usual time of reward because the value of the latest reward-predicting state is then too high: δt−1 = Rt + Vt −Vt−1


eward because the value of the latest reward-predicting state is then too high: δt−1 = Rt + Vt −Vt−1 = 0 + 0 −R? = −R?, as shown at the right end of the ‘R omitted’ graph of δ in Figure 15.4. This is like dopamine neuron activity decreasing below baseline at the time an expected reward is omitted as seen in the experiment of Schultz et al. (1993) described above and shown in Figure 15.3. The idea of an earliest reward-predicting state deserves more attention. In the scenario described above, because experience is divided into trials, and we assumed that predictions are conﬁned to individual trials, the earliest reward-predicting state is always the ﬁrst state of a trial. Clearly this is artiﬁcial.


he earliest reward-predicting state is always the ﬁrst state of a trial. Clearly this is artiﬁcial. A more general way to think of an earliest reward-predicting state is that it is an unpredicted predictor of reward, and there can be many such states. In an animal’s life, many di↵erent states may precede an earliest reward-predicting state. However, because these states are more often followed by other states that do not predict reward, their reward-predicting powers, that is, their values, remain low. A TD algorithm, if operating throughout the animal’s life, would update the values of these states too, but the updates would not consistently accumulate because, by assumption, none of these states reliably precedes an earliest reward-predicting state.


because, by assumption, none of these states reliably precedes an earliest reward-predicting state. If any of them did, they would be reward-predicting states as well. This might explain why with overtraining, dopamine responses decrease to even the earliest reward-predicting stimulus in a trial. With overtraining one would expect that even a formerly-unpredicted predictor state would become predicted by stimuli associated with earlier states: the animal’s interaction with its environment both inside and outside of an experimental task would become commonplace. Upon breaking this routine with the introduction of a new task, however, one would see TD errors reappear, as indeed is observed in dopamine neuron activity.


task, however, one would see TD errors reappear, as indeed is observed in dopamine neuron activity. The example described above explains why the TD error shares key features with the phasic activity of dopamine neurons when the animal is learning in a task similar to the idealized task of our example. But not every property of the phasic activity of dopamine neurons coincides so neatly with properties of δ. One of the most troubling discrepancies involves what happens when a reward occurs earlier than expected. We have seen that the omission of an expected reward produces a negative prediction error at the reward’s expected time, which corresponds to the activity of dopamine neurons decreasing below baseline when this happens.


which corresponds to the activity of dopamine neurons decreasing below baseline when this happens. If the reward arrives later than expected, it is then an unexpected reward and generates a positive prediction error. This happens with both TD errors and dopamine neuron responses. But when reward arrives earlier than expected, dopamine neurons do not do what the TD error does—at least with the CSC representation used by Montague et al. (1996) and by us in our example. Dopamine


394 Chapter 15: Neuroscience neurons do respond to the early reward, which is consistent with a positive TD error because the reward is not predicted to occur then. However, at the later time when the reward is expected but omitted, the TD error is negative whereas, in contrast to this prediction, dopamine neuron activity does not drop below baseline in the way the TD model predicts (Hollerman and Schultz, 1998). Something more complicated is going on in the animal’s brain than simply TD learning with a CSC representation. Some of the mismatches between the TD error and dopamine neuron activity can be addressed by selecting suitable parameter values for the TD algorithm and by using stimulus representations other than the CSC representation.


alues for the TD algorithm and by using stimulus representations other than the CSC representation. For instance, to address the early-reward mismatch just described, Suri and Schultz (1999) proposed a CSC representation in which the sequences of internal signals initiated by earlier stimuli are cancelled by the occurrence of a reward. Another proposal by Daw, Courville, and Touretzky (2006) is that the brain’s TD system uses representations produced by statistical modeling carried out in sensory cortex rather than simpler representations based on raw sensory input. Ludvig, Sutton, and Kehoe (2008) found that TD learning with a microstimulus (MS) representation (Figure 14.1) ﬁts the activity of dopamine neurons in the early-reward and other situations better than when a CSC representation


of dopamine neurons in the early-reward and other situations better than when a CSC representation is used. Pan, Schmidt, Wickens, and Hyland (2005) found that even with the CSC representation, prolonged eligibility traces improve the ﬁt of the TD error to some aspects of dopamine neuron activity. In general, many ﬁne details of TD-error behavior depend on subtle interactions between eligibility traces, discounting, and stimulus representations. Findings like these elaborate and reﬁne the reward prediction error hypothesis without refuting its core claim that the phasic activity of dopamine neurons is well characterized as signaling TD errors. On the other hand, there are other discrepancies between the TD theory and experimental data that are not so easily accommodated by selecting param


s between the TD theory and experimental data that are not so easily accommodated by selecting parameter values and stimulus representations (we mention some of these discrepancies in the Bibliographical and Historical Remarks section at the end of this chapter), and more mismatches are likely to be discovered as neuroscientists conduct ever more reﬁned experiments. But the reward prediction error hypothesis has been functioning very e↵ectively as a catalyst for improving our understanding of how the brain’s reward system works. Intricate experiments have been designed to validate or refute predictions derived from the hypothesis, and experimental results have, in turn, led to reﬁnement and elaboration of the TD error/dopamine hypothesis.


mental results have, in turn, led to reﬁnement and elaboration of the TD error/dopamine hypothesis. A remarkable aspect of these developments is that the reinforcement learning algorithms and theory that connect so well with properties of the dopamine system were developed from a computational perspective in total absence of any knowledge about the relevant properties of dopamine neurons—remember, TD learning and its connections to optimal control and dynamic programming were developed many years before any of the experiments were conducted that revealed the TD-like nature of dopamine neuron activity. This unplanned correspondence, despite not being perfect, suggests that the TD error/dopamine parallel captures something signiﬁcant about brain reward processes.


sts that the TD error/dopamine parallel captures something signiﬁcant about brain reward processes. In addition to accounting for many features of the phasic activity of dopamine neurons, the reward prediction error hypothesis links neuroscience to other aspects of reinforcement


15.7. Neural Actor–Critic 395 learning, in particular, to learning algorithms that use TD errors as reinforcement signals. Neuroscience is still far from reaching complete understanding of the circuits, molecular mechanisms, and functions of the phasic activity of dopamine neurons, but evidence supporting the reward prediction error hypothesis, along with evidence that phasic dopamine responses are reinforcement signals for learning, suggest that the brain might implement something like an actor–critic algorithm in which TD errors play critical roles. Other reinforcement learning algorithms are plausible candidates too, but actor–critic algorithms ﬁt the anatomy and physiology of the mammalian brain particularly well, as we describe in the following two sections.


physiology of the mammalian brain particularly well, as we describe in the following two sections. 15.7 Neural Actor–Critic Actor–critic algorithms learn both policies and value functions. The ‘actor’ is the component that learns policies, and the ‘critic’ is the component that learns about whatever policy is currently being followed by the actor in order to ‘criticize’ the actor’s action choices. The critic uses a TD algorithm to learn the state-value function for the actor’s current policy. The value function allows the critic to critique the actor’s action choices by sending TD errors, δ, to the actor. A positive δ means that the action was ‘good’ because it led to a state with a better-than-expected value; a negative δ means that the action was ‘bad’ because it led to a state with a w


than-expected value; a negative δ means that the action was ‘bad’ because it led to a state with a worse-than-expected value. Based on these critiques, the actor continually updates its policy. Two distinctive features of actor–critic algorithms are responsible for thinking that the brain might implement an algorithm like this. First, the two components of an actor–critic algorithm—the actor and the critic—suggest that two parts of the striatum—the dorsal and ventral subdivisions (Section 15.4), both critical for reward-based learning—may function respectively something like an actor and a critic. A second property of actor– critic algorithms that suggests a brain implementation is that the TD error has the dual role of being the reinforcement signal for both the actor and the critic, thou


TD error has the dual role of being the reinforcement signal for both the actor and the critic, though it has a di↵erent inﬂuence on learning in each of these components. This ﬁts well with several properties of the neural circuitry: axons of dopamine neurons target both the dorsal and ventral subdivisions of the striatum; dopamine appears to be critical for modulating synaptic plasticity in both structures; and how a neuromodulator such as dopamine acts on a target structure depends on properties of the target structure and not just on properties of the neuromodulator. Section 13.5 presents actor–critic algorithms as policy gradient methods, but the actor– critic algorithm of Barto, Sutton, and Anderson (1983) was simpler and was presented as an artiﬁcial neural network (ANN).


to, Sutton, and Anderson (1983) was simpler and was presented as an artiﬁcial neural network (ANN). Here we describe an ANN implementation something like that of Barto et al., and we follow Takahashi, Schoenbaum, and Niv (2008) in giving a schematic proposal for how this ANN might be implemented by real neural networks in the brain. We postpone discussion of the actor and critic learning rules until Section 15.8, where we present them as special cases of the policy-gradient formulation and discuss what they suggest about how dopamine might modulate synaptic plasticity.


396 Chapter 15: Neuroscience Figure 15.5a shows an implementation of an actor–critic algorithm as an ANN with component networks implementing the actor and the critic. The critic consists of a single neuron-like unit, V , whose output activity represents state values, and a component shown as the diamond labeled TD that computes TD errors by combining V ’s output with reward signals and with previous state values (as suggested by the loop from the TD diamond to itself). The actor network has a single layer of k actor units labeled Ai, i = 1, . . . , k. The output of each actor unit is a component of a k-dimensional action vector. An alternative is that there are k separate actions, one commanded by each actor unit, that compete with one another to be executed, but here we will think of the


nded by each actor unit, that compete with one another to be executed, but here we will think of the entire A-vector as an action. Reward Actor TD . . . . . . Critic . . . δ Actions States/Stimuli TD error Environment Actor VTA SNc S2 S1 Sn . . . . . . Critic . . . S1 Sn S2 Actions States/Stimuli Dopamine striatum Ventral Dorsal striatum Cortex (multiple areas) Environment Reward φ1 φ2 φn φ1 φ2 φn A1 A2 A3 Ak V δ (a) (b) x1 x2 xn x1 x2 xn Figure 15.5: Actor–critic ANN and a hypothetical neural implementation. a) Actor–critic algorithm as an ANN. The actor adjusts a policy based on the TD error δ it receives from the critic; the critic adjusts state-value parameters using the same δ. The critic produces a TD error from the reward signal, R, and the current change in its estimate of state va


ic produces a TD error from the reward signal, R, and the current change in its estimate of state values. The actor does not have direct access to the reward signal, and the critic does not have direct access to the action. b) Hypothetical neural implementation of an actor–critic algorithm. The actor and the value-learning part of the critic are respectively placed in the dorsal and ventral subdivisions of the striatum. The TD error is transmitted by dopamine neurons located in the VTA and SNpc to modulate changes in synaptic eﬃcacies of input from cortical areas to the ventral and dorsal striatum. Adapted from Frontiers in Neuroscience, vol. 2(1), 2008, Y. Takahashi, G. Schoenbaum, and Y. Niv, Silencing the critics: Understanding the e↵ects of cocaine sensitization on dorsolateral and ven


iv, Silencing the critics: Understanding the e↵ects of cocaine sensitization on dorsolateral and ventral striatum in the context of an Actor/Critic model.


15.7. Neural Actor–Critic 397 Both the critic and actor networks receive input consisting of multiple features representing the state of the agent’s environment. (Recall from Chapter 1 that the environment of a reinforcement learning agent includes components both inside and outside of the ‘organism’ containing the agent.) The ﬁgure shows these features as the circles labeled x1, x2, . . . , xn, shown twice just to keep the ﬁgure simple. A weight representing the eﬃcacy of a synapse is associated with each connection from each feature xi to the critic unit, V , and to each of the action units, Ai. The weights in the critic network parameterize the value function, and the weights in the actor network parameterize the policy.


work parameterize the value function, and the weights in the actor network parameterize the policy. The networks learn as these weights change according to the critic and actor learning rules that we describe in the following section. The TD error produced by circuitry in the critic is the reinforcement signal for changing the weights in both the critic and the actor networks. This is shown in Figure 15.5a by the line labeled ‘TD error δ’ extending across all of the connections in the critic and actor networks. This aspect of the network implementation, together with the reward prediction error hypothesis and the fact that the activity of dopamine neurons is so widely distributed by the extensive axonal arbors of these neurons, suggests that an actor–critic network something like this may


nsive axonal arbors of these neurons, suggests that an actor–critic network something like this may not be too farfetched as a hypothesis about how reward-related learning might happen in the brain. Figure 15.5b suggests—very schematically—how the ANN on the ﬁgure’s left might map onto structures in the brain according to the hypothesis of Takahashi et al. (2008). The hypothesis puts the actor and the value-learning part of the critic respectively in the dorsal and ventral subdivisions of the striatum, the input structure of the basal ganglia. Recall from Section 15.4 that the dorsal striatum is primarily implicated in inﬂuencing action selection, and the ventral striatum is thought to be critical for di↵erent aspects of reward processing, including the assignment of a↵ective value to sens


itical for di↵erent aspects of reward processing, including the assignment of a↵ective value to sensations. The cerebral cortex, along with other structures, sends input to the striatum conveying information about stimuli, internal states, and motor activity. In this hypothetical actor–critic brain implementation, the ventral striatum sends value information to the VTA and SNpc, where dopamine neurons in these nuclei combine it with information about reward to generate activity corresponding to TD errors (though exactly how dopaminergic neurons calculate these errors is not yet understood). The ‘TD error δ’ line in Figure 15.5a becomes the line labeled ‘Dopamine’ in Figure 15.5b, which represents the widely branching axons of dopamine neurons whose cell bodies are in the VTA and SNpc.


epresents the widely branching axons of dopamine neurons whose cell bodies are in the VTA and SNpc. Referring back to Figure 15.1, these axons make synaptic contact with the spines on the dendrites of medium spiny neurons, the main input/output neurons of both the dorsal and ventral divisions of the striatum. Axons of the cortical neurons that send input to the striatum make synaptic contact on the tips of these spines. According to the hypothesis, it is at these spines where changes in the eﬃcacies of the synapses from cortical regions to the striatum are governed by learning rules that critically depend on a reinforcement signal supplied by dopamine. An important implication of the hypothesis illustrated in Figure 15.5b is that the dopamine signal is not the ‘master’ reward signal like t


sis illustrated in Figure 15.5b is that the dopamine signal is not the ‘master’ reward signal like the scalar Rt of reinforcement learning. In fact, the hypothesis implies that one should not necessarily be able to probe the brain and record any signal like Rt in the activity of any single neuron.


398 Chapter 15: Neuroscience Many interconnected neural systems generate reward-related information, with di↵erent structures being recruited depending on di↵erent types of rewards. Dopamine neurons receive information from many di↵erent brain areas, so the input to the SNpc and VTA labeled ‘Reward’ in Figure 15.5b should be thought of as vector of reward-related information arriving to neurons in these nuclei along multiple input channels. What the theoretical scalar reward signal Rt might correspond to, then, is the net contribution of all reward-related information to dopamine neuron activity. It is the result of a pattern of activity across many neurons in di↵erent areas of the brain. Although the actor–critic neural implementation illustrated in Figure 15.5b may be correct on some cou


though the actor–critic neural implementation illustrated in Figure 15.5b may be correct on some counts, it clearly needs to be reﬁned, extended, and modiﬁed to qualify as a full-ﬂedged model of the function of the phasic activity of dopamine neurons. The Historical and Bibliographic Remarks section at the end of this chapter cites publications that discuss in more detail both empirical support for this hypothesis and places where it falls short. We now look in detail at what the actor and critic learning algorithms suggest about the rules governing changes in synaptic eﬃcacies of corticostriatal synapses. 15.8 Actor and Critic Learning Rules If the brain does implement something like the actor–critic algorithm—and assuming populations of dopamine neurons broadcast a common reinforcement s


tor–critic algorithm—and assuming populations of dopamine neurons broadcast a common reinforcement signal to the corticostriatal synapses of both the dorsal and ventral striatum as illustrated in Figure 15.5b (which is likely an oversimpliﬁcation as we mentioned above)—then this reinforcement signal a↵ects the synapses of these two structures in di↵erent ways. The learning rules for the critic and the actor use the same reinforcement signal, the TD error δ, but its e↵ect on learning is di↵erent for these two components. The TD error (combined with eligibility traces) tells the actor how to update action probabilities in order to reach higher-valued states. Learning by the actor is like instrumental conditioning using a Law-of-E↵ect-type learning rule (Section 1.7): the actor works to keep


umental conditioning using a Law-of-E↵ect-type learning rule (Section 1.7): the actor works to keep δ as positive as possible. On the other hand, the TD error (when combined with eligibility traces) tells the critic the direction and magnitude in which to change the parameters of the value function in order to improve its predictive accuracy. The critic works to reduce δ’s magnitude to be as close to zero as possible using a learning rule like the TD model of classical conditioning (Section 14.2). The di↵erence between the critic and actor learning rules is relatively simple, but this di↵erence has a profound e↵ect on learning and is essential to how the actor–critic algorithm works. The di↵erence lies solely in the eligibility traces each type of learning rule uses.


gorithm works. The di↵erence lies solely in the eligibility traces each type of learning rule uses. More than one set of learning rules can be used in actor–critic neural networks like those in Figure 15.5b but, to be speciﬁc, here we focus on the actor–critic algorithm for continuing problems with eligibility traces presented in Section 13.6. On each transition from state St to state St+1, taking action At and receiving reward Rt+1, that algorithm computes the TD error (δ) and then updates the eligibility trace vectors (zw t and z✓ t ) and


15.8. Actor and Critic Learning Rules 399 the parameters for the critic and actor (w and ✓), according to δt = Rt+1 + γ ˆv(St+1,w) −ˆv(St,w), zw t = γλwzw t−1 + rˆv(St,w), z✓ t = γλ✓z✓ t−1 + r ln ⇡(At|St, ✓), w w + ↵wδtzw t , ✓ ✓+ ↵✓δz✓ t , where γ 2 [0, 1) is a discount-rate parameter, λw 2 [0, 1] and λ✓2 [0, 1] are bootstrapping parameters for the critic and the actor respectively, and ↵w > 0 and ↵✓> 0 are analogous step-size parameters. Think of the approximate value function ˆv as the output of a single linear neuron-like unit, called the critic unit and labeled V in Figure 15.5a. Then the value function is a linear function of the feature-vector representation of state s, x(s) = (x1(s), . . . , xn(s))>, parameterized by a weight vector w = (w1, . . . , wn)>: ˆv(s,w) = w>x(s).


(x1(s), . . . , xn(s))>, parameterized by a weight vector w = (w1, . . . , wn)>: ˆv(s,w) = w>x(s). (15.1) Each xi(s) is like the presynaptic signal to a neuron’s synapse whose eﬃcacy is wi. The weights of the critic are incremented according to the rule above by ↵wδtzw t , where the reinforcement signal, δt, corresponds to a dopamine signal being broadcast to all of the critic unit’s synapses. The eligibility trace vector, zw t , for the critic unit is a trace (average of recent values) of rˆv(St,w). Because ˆv(s,w) is linear in the weights, rˆv(St,w) = x(St). In neural terms, this means that each synapse has its own eligibility trace, which is one component of the vector zw t . A synapse’s eligibility trace accumulates according to the level of activity arriving at that synapse, that is,


eligibility trace accumulates according to the level of activity arriving at that synapse, that is, the level of presynaptic activity, represented here by the component of the feature vector x(St) arriving at that synapse. The trace otherwise decays toward zero at a rate governed by the fraction λw. A synapse is eligible for modiﬁcation as long as its eligibility trace is non-zero. How the synapse’s eﬃcacy is actually modiﬁed depends on the reinforcement signals that arrive while the synapse is eligible. We call eligibility traces like these of the critic unit’s synapses noncontingent eligibility traces because they only depend on presynaptic activity and are not contingent in any way on postsynaptic activity.


hey only depend on presynaptic activity and are not contingent in any way on postsynaptic activity. The non-contingent eligibility traces of the critic unit’s synapses mean that the critic unit’s learning rule is essentially the TD model of classical conditioning described in Section 14.2. With the deﬁnition we have given above of the critic unit and its learning rule, the critic in Figure 15.5a is the same as the critic in the ANN actor–critic of Barto et al. (1983). Clearly, a critic like this consisting of just one linear neuron-like unit is the simplest starting point; this critic unit is a proxy for a more complicated neural network able to learn value functions of greater complexity. The actor in Figure 15.5a is a one-layer network of k neuron-like actor units, each receiving at time


he actor in Figure 15.5a is a one-layer network of k neuron-like actor units, each receiving at time t the same feature vector, x(St), that the critic unit receives. Each actor unit j, j = 1, . . . , k, has its own weight vector, ✓j, but because the actor units are all identical, we describe just one of the units and omit the subscript. One way for these


400 Chapter 15: Neuroscience units to follow the actor–critic algorithm given in the equations above is for each to be a Bernoulli-logistic unit. This means that the output of each actor unit at each time is a random variable, At, taking value 0 or 1. Think of value 1 as the neuron ﬁring, that is, emitting an action potential. The weighted sum, ✓>x(St), of a unit’s input vector determines the unit’s action probabilities via the exponential soft-max distribution (13.2), which for two actions is the logistic function: ⇡(1|s, ✓) = 1 −⇡(0|s, ✓) = 1 1 + exp(−✓>x(s)). (15.2) The weights of each actor unit are incremented, as above, by: ✓ ✓+ ↵✓δtz✓ t , where δ again corresponds to the dopamine signal: the same reinforcement signal that is sent to all the critic unit’s synapses.


the dopamine signal: the same reinforcement signal that is sent to all the critic unit’s synapses. Figure 15.5a shows δt being broadcast to all the synapses of all the actor units (which makes this actor network a team of reinforcement learning agents, something we discuss in Section 15.10 below). The actor eligibility trace vector z✓ t is a trace (average of recent values) of r ln ⇡(At|St, ✓). To understand this eligibility trace refer to Exercise 13.5, which deﬁnes this kind of unit and asks you to give a learning rule for it. That exercise asked you to express r ln ⇡(a|s, ✓) in terms of a, x(s), and ⇡(a|s, ✓) (for arbitrary state s and action a) by calculating the gradient. For the action and state actually occurring at time t, the answer is r ln ⇡(At|St, ✓) = # At −⇡(1|St, ✓) $ x(St).


n and state actually occurring at time t, the answer is r ln ⇡(At|St, ✓) = # At −⇡(1|St, ✓) $ x(St). (15.3) Unlike the non-contingent eligibility trace of a critic synapse that only accumulates the presynaptic activity x(St), the eligibility trace of an actor unit’s synapse in addition depends on the activity of the actor unit itself. We call this a contingent eligibility trace because it is contingent on this postsynaptic activity. The eligibility trace at each synapse continually decays, but increments or decrements depending on the activity of the presynaptic neuron and whether or not the postsynaptic neuron ﬁres. The factor At −⇡(1|St, ✓) in (15.3) is positive when At = 1 and negative otherwise.


ic neuron ﬁres. The factor At −⇡(1|St, ✓) in (15.3) is positive when At = 1 and negative otherwise. The postsynaptic contingency in the eligibility traces of actor units is the only di↵erence between the critic and actor learning rules. By keeping information about what actions were taken in what states, contingent eligibility traces allow credit for reward (positive δ), or blame for punishment (negative δ), to be apportioned among the policy parameters (the eﬃcacies of the actor units’ synapses) according to the contributions these parameters made to the units’ outputs that could have inﬂuenced later values of δ. Contingent eligibility traces mark the synapses as to how they should be modiﬁed to alter the units’ future responses to favor positive values of δ.


s to how they should be modiﬁed to alter the units’ future responses to favor positive values of δ. What do the critic and actor learning rules suggest about how eﬃcacies of corticostriatal synapses change? Both learning rules are related to Donald Hebb’s classic proposal that whenever a presynaptic signal participates in activating the postsynaptic neuron, the synapse’s eﬃcacy increases (Hebb, 1949). The critic and actor learning rules share with Hebb’s proposal the idea that changes in a synapse’s eﬃcacy depend on the interaction of several factors. In the critic learning rule the interaction is between the reinforcement signal δ and eligibility traces that depend only on presynaptic signals.


between the reinforcement signal δ and eligibility traces that depend only on presynaptic signals. Neuroscientists call this a two-factor learning rule because the interaction is between two signals or


15.8. Actor and Critic Learning Rules 401 quantities. The actor learning rule, on the other hand, is a three-factor learning rule because, in addition to depending on δ, its eligibility traces depend on both presynaptic and postsynaptic activity. Unlike Hebb’s proposal, however, the relative timing of the factors is critical to how synaptic eﬃcacies change, with eligibility traces intervening to allow the reinforcement signal to a↵ect synapses that were active in the recent past. Some subtleties about signal timing for the actor and critic learning rules deserve closer attention. In deﬁning the neuron-like actor and critic units, we ignored the small amount of time it takes synaptic input to e↵ect the ﬁring of a real neuron.


s, we ignored the small amount of time it takes synaptic input to e↵ect the ﬁring of a real neuron. When an action potential from the presynaptic neuron arrives at a synapse, neurotransmitter molecules are released that di↵use across the synaptic cleft to the postsynaptic neuron, where they bind to receptors on the postsynaptic neuron’s surface; this activates molecular machinery that causes the postsynaptic neuron to ﬁre (or to inhibit its ﬁring in the case of inhibitory synaptic input). This process can take several tens of milliseconds. According to (15.1) and (15.2), though, the input to a critic and actor unit instantaneously produces the unit’s output. Ignoring activation time like this is common in abstract models of Hebbian-style plasticity in which synaptic eﬃcacies change accordi


s is common in abstract models of Hebbian-style plasticity in which synaptic eﬃcacies change according to a simple product of simultaneous pre- and postsynaptic activity. More realistic models must take activation time into account. Activation time is especially important for a more realistic actor unit because it inﬂuences how contingent eligibility traces have to work in order to properly apportion credit for reinforcement to the appropriate synapses. The expression # At−⇡(1|St, ✓) $ x(St) deﬁning contingent eligibility traces for the actor unit’s learning rule given above includes the postsynaptic factor # At −⇡(1|St, ✓) $ and the presynaptic factor x(St). This works because by ignoring activation time, the presynaptic activity x(St) participates in causing the postsynaptic activity app


ctivation time, the presynaptic activity x(St) participates in causing the postsynaptic activity appearing in # At−⇡(1|St, ✓) $ . To assign credit for reinforcement correctly, the presynaptic factor deﬁning the eligibility trace must be a cause of the postsynaptic factor that also deﬁnes the trace. Contingent eligibility traces for a more realistic actor unit would have to take activation time into account. (Activation time should not be confused with the time required for a neuron to receive a reinforcement signal inﬂuenced by that neuron’s activity. The function of eligibility traces is to span this time interval which is generally much longer than the activation time. We discuss this further in the following section.) There are hints from neuroscience for how this process might work in


her in the following section.) There are hints from neuroscience for how this process might work in the brain. Neuroscientists have discovered a form of Hebbian plasticity called spike-timing-dependent plasticity (STDP) that lends plausibility to the existence of actor-like synaptic plasticity in the brain. STDP is a Hebbian-style plasticity, but changes in a synapse’s eﬃcacy depend on the relative timing of presynaptic and postsynaptic action potentials. The dependence can take di↵erent forms, but in the one most studied, a synapse increases in strength if spikes incoming via that synapse arrive shortly before the postsynaptic neuron ﬁres. If the timing relation is reversed, with a presynaptic spike arriving shortly after the postsynaptic neuron ﬁres, then the strength of the synapse decr