nits respectively (≈130,000 parameters). Actions were not included until the 2nd hidden layer of Q. When learning from pixels we used 3 convolutional layers (no pooling) with 32 ﬁlters at each layer. This was followed by two fully connected layers with 200 units (≈430,000 parameters). The ﬁnal layer weights and biases of both the actor and critic were initialized from a uniform distribution [−3 × 10−3, 3 × 10−3] and [3 × 10−4, 3 × 10−4] for the low dimensional and pixel cases respectively. This was to ensure the initial outputs for the policy and value estimates were near zero. The other layers were initialized from uniform distributions [−1 √f , 1 √f ] where f is the fan-in of the layer. The actions were not included until the fully-connected layers.


where f is the fan-in of the layer. The actions were not included until the fully-connected layers. We trained with minibatch sizes of 64 for the low dimensional problems and 16 on pixels. We used a replay buffer size of 106. For the exploration noise process we used temporally correlated noise in order to explore well in physical environments that have momentum. We used an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930) with θ = 0.15 and σ = 0.2. The Ornstein-Uhlenbeck process models the velocity of a Brownian particle with friction, which results in temporally correlated values centered around 0. 8 PLANNING ALGORITHM Our planner is implemented as a model-predictive controller (Tassa et al., 2012): at every time step we run a single iteration of trajectory optimization (using iLQG


a et al., 2012): at every time step we run a single iteration of trajectory optimization (using iLQG, (Todorov & Li, 2005)), starting from the true state of the system. Every single trajectory optimization is planned over a horizon between 250ms and 600ms, and this planning horizon recedes as the simulation of the world unfolds, as is the case in model-predictive control. The iLQG iteration begins with an initial rollout of the previous policy, which determines the nominal trajectory. We use repeated samples of simulated dynamics to approximate a linear expansion of the dynamics around every step of the trajectory, as well as a quadratic expansion of the cost function. We use this sequence of locally-linear-quadratic models to integrate the value function backwards in time along the nomina


f locally-linear-quadratic models to integrate the value function backwards in time along the nominal trajectory. This back-pass results in a putative modiﬁcation to the action sequence that will decrease the total cost. We perform a derivative-free line-search over this direction in the space of action sequences by integrating the dynamics forward (the forwardpass), and choose the best trajectory. We store this action sequence in order to warm-start the next iLQG iteration, and execute the ﬁrst action in the simulator. This results in a new state, which is used as the initial state in the next iteration of trajectory optimization. 9 ENVIRONMENT DETAILS 9.1 TORCS ENVIRONMENT For the Torcs environment we used a reward function which provides a positive reward at each step for the velocity o


vironment we used a reward function which provides a positive reward at each step for the velocity of the car projected along the track direction and a penalty of −1 for collisions. Episodes were terminated if progress was not made along the track after 500 frames. 11


Published as a conference paper at ICLR 2016 9.2 MUJOCO ENVIRONMENTS For physical control tasks we used reward functions which provide feedback at every step. In all tasks, the reward contained a small action cost. For all tasks that have a static goal state (e.g. pendulum swingup and reaching) we provide a smoothly varying reward based on distance to a goal state, and in some cases an additional positive reward when within a small radius of the target state. For grasping and manipulation tasks we used a reward with a term which encourages movement towards the payload and a second component which encourages moving the payload to the target. In locomotion tasks we reward forward action and penalize hard impacts to encourage smooth rather than hopping gaits (Schulman et al., 2015b).


n and penalize hard impacts to encourage smooth rather than hopping gaits (Schulman et al., 2015b). In addition, we used a negative reward and early termination for falls which were determined by simple threshholds on the height and torso angle (in the case of walker2d). Table 2 states the dimensionality of the problems and below is a summary of all the physics environments. task name dim(s) dim(a) dim(o) blockworld1 18 5 43 blockworld3da 31 9 102 canada 22 7 62 canada2d 14 3 29 cart 2 1 3 cartpole 4 1 14 cartpoleBalance 4 1 14 cartpoleParallelDouble 6 1 16 cartpoleParallelTriple 8 1 23 cartpoleSerialDouble 6 1 14 cartpoleSerialTriple 8 1 23 cheetah 18 6 17 ﬁxedReacher 10 3 23 ﬁxedReacherDouble 8 2 18 ﬁxedReacherSingle 6 1 13 gripper 18 5 43 gripperRandom 18 5 43 hardCheetah 18 6 17 hardCh


ble 8 2 18 ﬁxedReacherSingle 6 1 13 gripper 18 5 43 gripperRandom 18 5 43 hardCheetah 18 6 17 hardCheetahNice 18 6 17 hopper 14 4 14 hyq 37 12 37 hyqKick 37 12 37 movingGripper 22 7 49 movingGripperRandom 22 7 49 pendulum 2 1 3 reacher 10 3 23 reacher3daFixedTarget 20 7 61 reacher3daRandomTarget 20 7 61 reacherDouble 6 1 13 reacherObstacle 18 5 38 reacherSingle 6 1 13 walker2d 18 6 41 Table 2: Dimensionality of the MuJoCo tasks: the dimensionality of the underlying physics model dim(s), number of action dimensions dim(a) and observation dimensions dim(o). task name Brief Description blockworld1 Agent is required to use an arm with gripper constrained to the 2D plane to grab a falling block and lift it against gravity to a ﬁxed target position. 12


Published as a conference paper at ICLR 2016 blockworld3da Agent is required to use a human-like arm with 7-DOF and a simple gripper to grab a block and lift it against gravity to a ﬁxed target position. canada Agent is required to use a 7-DOF arm with hockey-stick like appendage to hit a ball to a target. canada2d Agent is required to use an arm with hockey-stick like appendage to hit a ball initialzed to a random start location to a random target location. cart Agent must move a simple mass to rest at 0. The mass begins each trial in random positions and with random velocities. cartpole The classic cart-pole swing-up task. Agent must balance a pole attached to a cart by applying forces to the cart alone. The pole starts each episode hanging upside-down.


d to a cart by applying forces to the cart alone. The pole starts each episode hanging upside-down. cartpoleBalance The classic cart-pole balance task. Agent must balance a pole attached to a cart by applying forces to the cart alone. The pole starts in the upright positions at the beginning of each episode. cartpoleParallelDouble Variant on the classic cart-pole. Two poles, both attached to the cart, should be kept upright as much as possible. cartpoleSerialDouble Variant on the classic cart-pole. Two poles, one attached to the cart and the second attached to the end of the ﬁrst, should be kept upright as much as possible. cartpoleSerialTriple Variant on the classic cart-pole. Three poles, one attached to the cart, the second attached to the end of the ﬁrst, and the third attached to the


one attached to the cart, the second attached to the end of the ﬁrst, and the third attached to the end of the second, should be kept upright as much as possible. cheetah The agent should move forward as quickly as possible with a cheetahlike body that is constrained to the plane. This environment is based very closely on the one introduced by Wawrzy´nski (2009); Wawrzy´nski & Tanwani (2013). ﬁxedReacher Agent is required to move a 3-DOF arm to a ﬁxed target position. ﬁxedReacherDouble Agent is required to move a 2-DOF arm to a ﬁxed target position. ﬁxedReacherSingle Agent is required to move a simple 1-DOF arm to a ﬁxed target position. gripper Agent must use an arm with gripper appendage to grasp an object and manuver the object to a ﬁxed target.


must use an arm with gripper appendage to grasp an object and manuver the object to a ﬁxed target. gripperRandom The same task as gripper except that the arm object and target position are initialized in random locations. hardCheetah The agent should move forward as quickly as possible with a cheetahlike body that is constrained to the plane. This environment is based very closely on the one introduced by Wawrzy´nski (2009); Wawrzy´nski & Tanwani (2013), but has been made much more difﬁcult by removing the stabalizing joint stiffness from the model. hopper Agent must balance a multiple degree of freedom monoped to keep it from falling. hyq Agent is required to keep a quadroped model based on the hyq robot from falling. 13


Published as a conference paper at ICLR 2016 movingGripper Agent must use an arm with gripper attached to a moveable platform to grasp an object and move it to a ﬁxed target. movingGripperRandom The same as the movingGripper environment except that the object position, target position, and arm state are initialized randomly. pendulum The classic pendulum swing-up problem. The pendulum should be brought to the upright position and balanced. Torque limits prevent the agent from swinging the pendulum up directly. reacher3daFixedTarget Agent is required to move a 7-DOF human-like arm to a ﬁxed target position. reacher3daRandomTarget Agent is required to move a 7-DOF human-like arm from random starting locations to random target positions.


required to move a 7-DOF human-like arm from random starting locations to random target positions. reacher Agent is required to move a 3-DOF arm from random starting locations to random target positions. reacherSingle Agent is required to move a simple 1-DOF arm from random starting locations to random target positions. reacherObstacle Agent is required to move a 5-DOF arm around an obstacle to a randomized target position. walker2d Agent should move forward as quickly as possible with a bipedal walker constrained to the plane without falling down or pitching the torso too far forward or backward. 14


Deep Reinforcement Learning with Double Q-learning Hado van Hasselt and Arthur Guez and David Silver Google DeepMind Abstract The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions afﬁrmatively. In particular, we ﬁrst show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function a


m, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a speciﬁc adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games. The goal of reinforcement learning (Sutton and Barto, 1998) is to learn good policies for sequential decision problems, by optimizing a cumulative future reward signal. Q-learning (Watkins, 1989) is one of the most popular reinforcement learning algorithms, but it is known to sometimes learn unrealistically high action values because it includes a maximization step over estimated action values, which tends to prefer overestimated to underestimated values.


on step over estimated action values, which tends to prefer overestimated to underestimated values. In previous work, overestimations have been attributed to insufﬁciently ﬂexible function approximation (Thrun and Schwartz, 1993) and noise (van Hasselt, 2010, 2011). In this paper, we unify these views and show overestimations can occur when the action values are inaccurate, irrespective of the source of approximation error. Of course, imprecise value estimates are the norm during learning, which indicates that overestimations may be much more common than previously appreciated. It is an open question whether, if the overestimations do occur, this negatively affects performance in practice. Overoptimistic value estimates are not necessarily a problem in and of themselves.


nce in practice. Overoptimistic value estimates are not necessarily a problem in and of themselves. If all values would be uniformly higher then the relative action preferences are preserved and we would not expect the resulting policy to be any worse. Furthermore, it is known that sometimes it is good to be optimistic: optimism in the face of uncertainty is a well-known Copyright c⃝2016, Association for the Advancement of Artiﬁcial Intelligence (www.aaai.org). All rights reserved. exploration technique (Kaelbling et al., 1996). If, however, the overestimations are not uniform and not concentrated at states about which we wish to learn more, then they might negatively affect the quality of the resulting policy.


which we wish to learn more, then they might negatively affect the quality of the resulting policy. Thrun and Schwartz (1993) give speciﬁc examples in which this leads to suboptimal policies, even asymptotically. To test whether overestimations occur in practice and at scale, we investigate the performance of the recent DQN algorithm (Mnih et al., 2015). DQN combines Q-learning with a ﬂexible deep neural network and was tested on a varied and large set of deterministic Atari 2600 games, reaching human-level performance on many games. In some ways, this setting is a best-case scenario for Q-learning, because the deep neural network provides ﬂexible function approximation with the potential for a low asymptotic approximation error, and the determinism of the environments prevents the harmful


r a low asymptotic approximation error, and the determinism of the environments prevents the harmful effects of noise. Perhaps surprisingly, we show that even in this comparatively favorable setting DQN sometimes substantially overestimates the values of the actions. We show that the idea behind the Double Q-learning algorithm (van Hasselt, 2010), which was ﬁrst proposed in a tabular setting, can be generalized to work with arbitrary function approximation, including deep neural networks. We use this to construct a new algorithm we call Double DQN. We then show that this algorithm not only yields more accurate value estimates, but leads to much higher scores on several games. This demonstrates that the overestimations of DQN were indeed leading to poorer policies and that it is beneﬁcial t


es that the overestimations of DQN were indeed leading to poorer policies and that it is beneﬁcial to reduce them. In addition, by improving upon DQN we obtain state-of-the-art results on the Atari domain. Background To solve sequential decision problems we can learn estimates for the optimal value of each action, deﬁned as the expected sum of future rewards when taking that action and following the optimal policy thereafter. Under a given policy π, the true value of an action a in a state s is Qπ(s, a) ≡E [R1 + γR2 + . . . | S0 = s, A0 = a, π] , where γ ∈[0, 1] is a discount factor that trades off the importance of immediate and later rewards. The optimal value is then Q∗(s, a) = maxπ Qπ(s, a).


the importance of immediate and later rewards. The optimal value is then Q∗(s, a) = maxπ Qπ(s, a). An optimal policy is easily derived from the optimal values by selecting the highestvalued action in each state. arXiv:1509.06461v3 [cs.LG] 8 Dec 2015


Estimates for the optimal action values can be learned using Q-learning (Watkins, 1989), a form of temporal difference learning (Sutton, 1988). Most interesting problems are too large to learn all action values in all states separately. Instead, we can learn a parameterized value function Q(s, a; θt). The standard Q-learning update for the parameters after taking action At in state St and observing the immediate reward Rt+1 and resulting state St+1 is then θt+1 = θt+α(Y Q t −Q(St, At; θt))∇θtQ(St, At; θt) . (1) where α is a scalar step size and the target Y Q t is deﬁned as Y Q t ≡Rt+1 + γ max a Q(St+1, a; θt) . (2) This update resembles stochastic gradient descent, updating the current value Q(St, At; θt) towards a target value Y Q t .


tochastic gradient descent, updating the current value Q(St, At; θt) towards a target value Y Q t . Deep Q Networks A deep Q network (DQN) is a multi-layered neural network that for a given state s outputs a vector of action values Q(s, · ; θ), where θ are the parameters of the network. For an n-dimensional state space and an action space containing m actions, the neural network is a function from Rn to Rm. Two important ingredients of the DQN algorithm as proposed by Mnih et al. (2015) are the use of a target network, and the use of experience replay. The target network, with parameters θ−, is the same as the online network except that its parameters are copied every τ steps from the online network, so that then θ− t = θt, and kept ﬁxed on all other steps.


ed every τ steps from the online network, so that then θ− t = θt, and kept ﬁxed on all other steps. The target used by DQN is then Y DQN t ≡Rt+1 + γ max a Q(St+1, a; θ− t ) . (3) For the experience replay (Lin, 1992), observed transitions are stored for some time and sampled uniformly from this memory bank to update the network. Both the target network and the experience replay dramatically improve the performance of the algorithm (Mnih et al., 2015). Double Q-learning The max operator in standard Q-learning and DQN, in (2) and (3), uses the same values both to select and to evaluate an action. This makes it more likely to select overestimated values, resulting in overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation.


overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation. This is the idea behind Double Q-learning (van Hasselt, 2010). In the original Double Q-learning algorithm, two value functions are learned by assigning each experience randomly to update one of the two value functions, such that there are two sets of weights, θ and θ′. For each update, one set of weights is used to determine the greedy policy and the other to determine its value. For a clear comparison, we can ﬁrst untangle the selection and evaluation in Q-learning and rewrite its target (2) as Y Q t = Rt+1 + γQ(St+1, argmax a Q(St+1, a; θt); θt) . The Double Q-learning error can then be written as Y DoubleQ t ≡Rt+1 + γQ(St+1, argmax a Q(St+1, a; θt); θ′ t) .


learning error can then be written as Y DoubleQ t ≡Rt+1 + γQ(St+1, argmax a Q(St+1, a; θt); θ′ t) . (4) Notice that the selection of the action, in the argmax, is still due to the online weights θt. This means that, as in Qlearning, we are still estimating the value of the greedy policy according to the current values, as deﬁned by θt. However, we use the second set of weights θ′ t to fairly evaluate the value of this policy. This second set of weights can be updated symmetrically by switching the roles of θ and θ′. Overoptimism due to estimation errors Q-learning’s overestimations were ﬁrst investigated by Thrun and Schwartz (1993), who showed that if the action values contain random errors uniformly distributed in an interval [−ϵ, ϵ] then each target is overestimated up to γϵ m−1 m+1, wh


uniformly distributed in an interval [−ϵ, ϵ] then each target is overestimated up to γϵ m−1 m+1, where m is the number of actions. In addition, Thrun and Schwartz give a concrete example in which these overestimations even asymptotically lead to sub-optimal policies, and show the overestimations manifest themselves in a small toy problem when using function approximation. Later van Hasselt (2010) argued that noise in the environment can lead to overestimations even when using tabular representation, and proposed Double Q-learning as a solution. In this section we demonstrate more generally that estimation errors of any kind can induce an upward bias, regardless of whether these errors are due to environmental noise, function approximation, non-stationarity, or any other source.


rors are due to environmental noise, function approximation, non-stationarity, or any other source. This is important, because in practice any method will incur some inaccuracies during learning, simply due to the fact that the true values are initially unknown. The result by Thrun and Schwartz (1993) cited above gives an upper bound to the overestimation for a speciﬁc setup, but it is also possible, and potentially more interesting, to derive a lower bound. Theorem 1. Consider a state s in which all the true optimal action values are equal at Q∗(s, a) = V∗(s) for some V∗(s). Let Qt be arbitrary value estimates that are on the whole unbiased in the sense that P a(Qt(s, a) −V∗(s)) = 0, but that are not all correct, such that 1 m P a(Qt(s, a)−V∗(s))2 = C for some C > 0, where m ≥2 is the num


at are not all correct, such that 1 m P a(Qt(s, a)−V∗(s))2 = C for some C > 0, where m ≥2 is the number of actions in s. Under these conditions, maxa Qt(s, a) ≥V∗(s) + q C m−1. This lower bound is tight. Under the same conditions, the lower bound on the absolute error of the Double Q-learning estimate is zero. (Proof in appendix.) Note that we did not need to assume that estimation errors for different actions are independent. This theorem shows that even if the value estimates are on average correct, estimation errors of any source can drive the estimates up and away from the true optimal values. The lower bound in Theorem 1 decreases with the number of actions. This is an artifact of considering the lower bound, which requires very speciﬁc values to be attained.


s is an artifact of considering the lower bound, which requires very speciﬁc values to be attained. More typically, the overoptimism increases with the number of actions as shown in Figure 1. Q-learning’s overestimations there indeed increase with the number of actions,


2 4 8 16 32 64 128 256 512 1024 number of actions 0.0 0.5 1.0 1.5 error maxa Q(s, a) −V∗(s) Q′(s, argmaxaQ(s, a)) −V∗(s) Figure 1: The orange bars show the bias in a single Qlearning update when the action values are Q(s, a) = V∗(s) + ϵa and the errors {ϵa}m a=1 are independent standard normal random variables. The second set of action values Q′, used for the blue bars, was generated identically and independently. All bars are the average of 100 repetitions. while Double Q-learning is unbiased. As another example, if for all actions Q∗(s, a) = V∗(s) and the estimation errors Qt(s, a) −V∗(s) are uniformly random in [−1, 1], then the overoptimism is m−1 m+1. (Proof in appendix.) We now turn to function approximation and consider a real-valued continuous state space with 10 discrete actions i


function approximation and consider a real-valued continuous state space with 10 discrete actions in each state. For simplicity, the true optimal action values in this example depend only on state so that in each state all actions have the same true value. These true values are shown in the left column of plots in Figure 2 (purple lines) and are deﬁned as either Q∗(s, a) = sin(s) (top row) or Q∗(s, a) = 2 exp(−s2) (middle and bottom rows). The left plots also show an approximation for a single action (green lines) as a function of state as well as the samples the estimate is based on (green dots). The estimate is a d-degree polynomial that is ﬁt to the true values at sampled states, where d = 6 (top and middle rows) or d = 9 (bottom row).


s ﬁt to the true values at sampled states, where d = 6 (top and middle rows) or d = 9 (bottom row). The samples match the true function exactly: there is no noise and we assume we have ground truth for the action value on these sampled states. The approximation is inexact even on the sampled states for the top two rows because the function approximation is insufﬁciently ﬂexible. In the bottom row, the function is ﬂexible enough to ﬁt the green dots, but this reduces the accuracy in unsampled states. Notice that the sampled states are spaced further apart near the left side of the left plots, resulting in larger estimation errors. In many ways this is a typical learning setting, where at each point in time we only have limited data.


any ways this is a typical learning setting, where at each point in time we only have limited data. The middle column of plots in Figure 2 shows estimated action value functions for all 10 actions (green lines), as functions of state, along with the maximum action value in each state (black dashed line). Although the true value function is the same for all actions, the approximations differ because we have supplied different sets of sampled states.1 The maximum is often higher than the ground truth shown in purple on the left. This is conﬁrmed in the right plots, which shows the difference between the black and purple curves in orange. The orange line is almost always positive, 1Each action-value function is ﬁt with a different subset of integer states.


lmost always positive, 1Each action-value function is ﬁt with a different subset of integer states. States −6 and 6 are always included to avoid extrapolations, and for each action two adjacent integers are missing: for action a1 states −5 and −4 are not sampled, for a2 states −4 and −3 are not sampled, and so on. This causes the estimated values to differ. indicating an upward bias. The right plots also show the estimates from Double Q-learning in blue2, which are on average much closer to zero. This demonstrates that Double Qlearning indeed can successfully reduce the overoptimism of Q-learning. The different rows in Figure 2 show variations of the same experiment. The difference between the top and middle rows is the true value function, demonstrating that overestimations are not an art


he top and middle rows is the true value function, demonstrating that overestimations are not an artifact of a speciﬁc true value function. The difference between the middle and bottom rows is the ﬂexibility of the function approximation. In the left-middle plot, the estimates are even incorrect for some of the sampled states because the function is insufﬁciently ﬂexible. The function in the bottom-left plot is more ﬂexible but this causes higher estimation errors for unseen states, resulting in higher overestimations. This is important because ﬂexible parametric function approximators are often employed in reinforcement learning (see, e.g., Tesauro 1995; Sallans and Hinton 2004; Riedmiller 2005; Mnih et al. 2015).


ent learning (see, e.g., Tesauro 1995; Sallans and Hinton 2004; Riedmiller 2005; Mnih et al. 2015). In contrast to van Hasselt (2010) we did not use a statistical argument to ﬁnd overestimations, the process to obtain Figure 2 is fully deterministic. In contrast to Thrun and Schwartz (1993), we did not rely on inﬂexible function approximation with irreducible asymptotic errors; the bottom row shows that a function that is ﬂexible enough to cover all samples leads to high overestimations. This indicates that the overestimations can occur quite generally. In the examples above, overestimations occur even when assuming we have samples of the true action value at certain states. The value estimates can further deteriorate if we bootstrap off of action values that are already overoptimistic, si


tes can further deteriorate if we bootstrap off of action values that are already overoptimistic, since this causes overestimations to propagate throughout our estimates. Although uniformly overestimating values might not hurt the resulting policy, in practice overestimation errors will differ for different states and actions. Overestimation combined with bootstrapping then has the pernicious effect of propagating the wrong relative information about which states are more valuable than others, directly affecting the quality of the learned policies. The overestimations should not be confused with optimism in the face of uncertainty (Sutton, 1990; Agrawal, 1995; Kaelbling et al., 1996; Auer et al., 2002; Brafman and Tennenholtz, 2003; Szita and L˝orincz, 2008; Strehl et al., 2009), where an


al., 2002; Brafman and Tennenholtz, 2003; Szita and L˝orincz, 2008; Strehl et al., 2009), where an exploration bonus is given to states or actions with uncertain values. Conversely, the overestimations discussed here occur only after updating, resulting in overoptimism in the face of apparent certainty. This was already observed by Thrun and Schwartz (1993), who noted that, in contrast to optimism in the face of uncertainty, these overestimations actually can impede learning an optimal policy. We will see this negative effect on policy quality conﬁrmed later in the experiments as well: when we reduce the overestimations using Double Q-learning, the policies improve. 2We arbitrarily used the samples of action ai+5 (for i ≤5) or ai−5 (for i > 5) as the second set of samples for the double e


samples of action ai+5 (for i ≤5) or ai−5 (for i > 5) as the second set of samples for the double estimator of action ai.


−2 0 2 Qt(s, a) Q∗(s, a) True value and an estimate −2 0 2 maxa Qt(s, a) All estimates and max −1 0 1 maxa Qt(s, a) −maxa Q∗(s, a) Double-Q estimate +0.61 −0.02 Average error Bias as function of state 0 2 Qt(s, a) Q∗(s, a) 0 2 maxa Qt(s, a) −1 0 1 maxa Qt(s, a) −maxa Q∗(s, a) Double-Q estimate +0.47 +0.02 −6 −4 −2 0 2 4 6 state 0 2 4 Qt(s, a) Q∗(s, a) −6 −4 −2 0 2 4 6 state 0 2 4 maxa Qt(s, a) −6 −4 −2 0 2 4 6 state 0 2 4 maxa Qt(s, a)− maxa Q∗(s, a) Double-Q estimate +3.35 −0.02 Figure 2: Illustration of overestimations during learning. In each state (x-axis), there are 10 actions. The left column shows the true values V∗(s) (purple line). All true action values are deﬁned by Q∗(s, a) = V∗(s).


n shows the true values V∗(s) (purple line). All true action values are deﬁned by Q∗(s, a) = V∗(s). The green line shows estimated values Q(s, a) for one action as a function of state, ﬁtted to the true value at several sampled states (green dots). The middle column plots show all the estimated values (green), and the maximum of these values (dashed black). The maximum is higher than the true value (purple, left plot) almost everywhere. The right column plots shows the difference in orange. The blue line in the right plots is the estimate used by Double Q-learning with a second set of samples for each state. The blue line is much closer to zero, indicating less bias. The three rows correspond to different true functions (left, purple) or capacities of the ﬁtted function (left, green).


spond to different true functions (left, purple) or capacities of the ﬁtted function (left, green). (Details in the text) Double DQN The idea of Double Q-learning is to reduce overestimations by decomposing the max operation in the target into action selection and action evaluation. Although not fully decoupled, the target network in the DQN architecture provides a natural candidate for the second value function, without having to introduce additional networks. We therefore propose to evaluate the greedy policy according to the online network, but using the target network to estimate its value. In reference to both Double Q-learning and DQN, we refer to the resulting algorithm as Double DQN. Its update is the same as for DQN, but replacing the target Y DQN t with Y DoubleDQN t ≡Rt+1+γQ(St+


Its update is the same as for DQN, but replacing the target Y DQN t with Y DoubleDQN t ≡Rt+1+γQ(St+1, argmax a Q(St+1, a; θt), θ− t ) . In comparison to Double Q-learning (4), the weights of the second network θ′ t are replaced with the weights of the target network θ− t for the evaluation of the current greedy policy. The update to the target network stays unchanged from DQN, and remains a periodic copy of the online network. This version of Double DQN is perhaps the minimal possible change to DQN towards Double Q-learning. The goal is to get most of the beneﬁt of Double Q-learning, while keeping the rest of the DQN algorithm intact for a fair comparison, and with minimal computational overhead.


he rest of the DQN algorithm intact for a fair comparison, and with minimal computational overhead. Empirical results In this section, we analyze the overestimations of DQN and show that Double DQN improves over DQN both in terms of value accuracy and in terms of policy quality. To further test the robustness of the approach we additionally evaluate the algorithms with random starts generated from expert human trajectories, as proposed by Nair et al. (2015). Our testbed consists of Atari 2600 games, using the Arcade Learning Environment (Bellemare et al., 2013). The goal is for a single algorithm, with a ﬁxed set of hyperparameters, to learn to play each of the games separately from interaction given only the screen pixels as input.


learn to play each of the games separately from interaction given only the screen pixels as input. This is a demanding testbed: not only are the inputs high-dimensional, the game visuals and game mechanics vary substantially between games. Good solutions must therefore rely heavily on the learning algorithm — it is not practically feasible to overﬁt the domain by relying only on tuning. We closely follow the experimental setting and network architecture outlined by Mnih et al. (2015). Brieﬂy, the network architecture is a convolutional neural network (Fukushima, 1988; LeCun et al., 1998) with 3 convolution layers and a fully-connected hidden layer (approximately 1.5M parameters in total). The network takes the last four frames as input and outputs the action value of each action.


otal). The network takes the last four frames as input and outputs the action value of each action. On each game, the network is trained on a single GPU for 200M frames, or approximately 1 week. Results on overoptimism Figure 3 shows examples of DQN’s overestimations in six Atari games. DQN and Double DQN were both trained under the exact conditions described by Mnih et al. (2015). DQN is consistently and sometimes vastly overoptimistic about the value of the current greedy policy, as can be seen by comparing the orange learning curves in the top row of plots to the straight orange lines, which represent the actual discounted value of the best learned policy. More precisely, the (averaged) value estimates are computed regularly during training with full evaluation phases of length T = 125,


alue estimates are computed regularly during training with full evaluation phases of length T = 125, 000 steps as 1 T T X t=1 argmax a Q(St, a; θ) .


0 50 100 150 200 10 15 20 Value estimates Alien 0 50 100 150 200 4 6 8 Space Invaders 0 50 100 150 200 1.0 1.5 2.0 2.5 Time Pilot 0 50 100 150 200 Training steps (in millions) 0 2 4 6 8 DQN estimate Double DQN estimate DQN true value Double DQN true value Zaxxon 0 50 100 150 200 1 10 100 Value estimates (log scale) DQN Double DQN Wizard of Wor 0 50 100 150 200 5 10 20 40 80 DQN Double DQN Asterix 0 50 100 150 200 Training steps (in millions) 0 1000 2000 3000 4000 Score DQN Double DQN Wizard of Wor 0 50 100 150 200 Training steps (in millions) 0 2000 4000 6000 DQN Double DQN Asterix Figure 3: The top and middle rows show value estimates by DQN (orange) and Double DQN (blue) on six Atari games.


top and middle rows show value estimates by DQN (orange) and Double DQN (blue) on six Atari games. The results are obtained by running DQN and Double DQN with 6 different random seeds with the hyper-parameters employed by Mnih et al. (2015). The darker line shows the median over seeds and we average the two extreme values to obtain the shaded area (i.e., 10% and 90% quantiles with linear interpolation). The straight horizontal orange (for DQN) and blue (for Double DQN) lines in the top row are computed by running the corresponding agents after learning concluded, and averaging the actual discounted return obtained from each visited state. These straight lines would match the learning curves at the right side of the plots if there is no bias.


straight lines would match the learning curves at the right side of the plots if there is no bias. The middle row shows the value estimates (in log scale) for two games in which DQN’s overoptimism is quite extreme. The bottom row shows the detrimental effect of this on the score achieved by the agent as it is evaluated during training: the scores drop when the overestimations begin. Learning with Double DQN is much more stable. The ground truth averaged values are obtained by running the best learned policies for several episodes and computing the actual cumulative rewards. Without overestimations we would expect these quantities to match up (i.e., the curve to match the straight line at the right of each plot).


hese quantities to match up (i.e., the curve to match the straight line at the right of each plot). Instead, the learning curves of DQN consistently end up much higher than the true values. The learning curves for Double DQN, shown in blue, are much closer to the blue straight line representing the true value of the ﬁnal policy. Note that the blue straight line is often higher than the orange straight line. This indicates that Double DQN does not just produce more accurate value estimates but also better policies. More extreme overestimations are shown in the middle two plots, where DQN is highly unstable on the games Asterix and Wizard of Wor. Notice the log scale for the values on the y-axis. The bottom two plots shows the corresponding scores for these two games.


the values on the y-axis. The bottom two plots shows the corresponding scores for these two games. Notice that the increases in value estimates for DQN in the middle plots coincide with decreasing scores in bottom plots. Again, this indicates that the overestimations are harming the quality of the resulting policies. If seen in isolation, one might perhaps be tempted to think the observed instability is related to inherent instability problems of off-policy learning with function approximation (Baird, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2008; Maei, 2011; Sutton et al., 2015). However, we see that learning is much more stable with Double DQN, DQN Double DQN Median 93.5% 114.7% Mean 241.1% 330.3% Table 1: Summary of normalized performance up to 5 minutes of play on 49 games.


Mean 241.1% 330.3% Table 1: Summary of normalized performance up to 5 minutes of play on 49 games. Results for DQN are from Mnih et al. (2015) suggesting that the cause for these instabilities is in fact Qlearning’s overoptimism. Figure 3 only shows a few examples, but overestimations were observed for DQN in all 49 tested Atari games, albeit in varying amounts. Quality of the learned policies Overoptimism does not always adversely affect the quality of the learned policy. For example, DQN achieves optimal behavior in Pong despite slightly overestimating the policy value. Nevertheless, reducing overestimations can signiﬁcantly beneﬁt the stability of learning; we see clear examples of this in Figure 3.


tions can signiﬁcantly beneﬁt the stability of learning; we see clear examples of this in Figure 3. We now assess more generally how much Double DQN helps in terms of policy quality by evaluating on all 49 games that DQN was tested on. As described by Mnih et al. (2015) each evaluation episode starts by executing a special no-op action that does not affect the environment up to 30 times, to provide different starting points for the agent. Some exploration during evaluation provides additional randomization. For Double DQN we used the exact same hyper-parameters as for DQN,


DQN Double DQN Double DQN (tuned) Median 47.5% 88.4% 116.7% Mean 122.0% 273.1% 475.2% Table 2: Summary of normalized performance up to 30 minutes of play on 49 games with human starts. Results for DQN are from Nair et al. (2015). to allow for a controlled experiment focused just on reducing overestimations. The learned policies are evaluated for 5 mins of emulator time (18,000 frames) with an ϵgreedy policy where ϵ = 0.05. The scores are averaged over 100 episodes. The only difference between Double DQN and DQN is the target, using Y DoubleDQN t rather than Y DQN. This evaluation is somewhat adversarial, as the used hyperparameters were tuned for DQN but not for Double DQN. To obtain summary statistics across games, we normalize the score for each game as follows: scorenormalized = scoreag


statistics across games, we normalize the score for each game as follows: scorenormalized = scoreagent −scorerandom scorehuman −scorerandom . (5) The ‘random’ and ‘human’ scores are the same as used by Mnih et al. (2015), and are given in the appendix. Table 1, under no ops, shows that on the whole Double DQN clearly improves over DQN. A detailed comparison (in appendix) shows that there are several games in which Double DQN greatly improves upon DQN. Noteworthy examples include Road Runner (from 233% to 617%), Asterix (from 70% to 180%), Zaxxon (from 54% to 111%), and Double Dunk (from 17% to 397%). The Gorila algorithm (Nair et al., 2015), which is a massively distributed version of DQN, is not included in the table because the architecture and infrastructure is sufﬁciently different to


is not included in the table because the architecture and infrastructure is sufﬁciently different to make a direct comparison unclear. For completeness, we note that Gorila obtained median and mean normalized scores of 96% and 495%, respectively. Robustness to Human starts One concern with the previous evaluation is that in deterministic games with a unique starting point the learner could potentially learn to remember sequences of actions without much need to generalize. While successful, the solution would not be particularly robust. By testing the agents from various starting points, we can test whether the found solutions generalize well, and as such provide a challenging testbed for the learned polices (Nair et al., 2015).


ralize well, and as such provide a challenging testbed for the learned polices (Nair et al., 2015). We obtained 100 starting points sampled for each game from a human expert’s trajectory, as proposed by Nair et al. (2015). We start an evaluation episode from each of these starting points and run the emulator for up to 108,000 frames (30 mins at 60Hz including the trajectory before the starting point). Each agent is only evaluated on the rewards accumulated after the starting point. For this evaluation we include a tuned version of Double DQN. Some tuning is appropriate because the hyperparameters were tuned for DQN, which is a different algorithm. For the tuned version of Double DQN, we increased the number of frames between each two copies of the target network from 10,000 to 30,000, to r


eased the number of frames between each two copies of the target network from 10,000 to 30,000, to reduce overestimations further because immediately after each switch DQN and Double DQN 0% 100% 200% 300% 400% 500% 1000% 1500% 2000% 2500% 5000% 7500% Normalized score Human ∗∗Solaris∗∗ Private Eye Gravitar Venture Montezuma’s Revenge Asteroids ∗∗Pitfall∗∗ Ms. Pacman Amidar ∗∗Yars Revenge∗∗ Alien Centipede Bowling ∗∗Skiing∗∗ Frostbite Chopper Command Seaquest ∗∗Berzerk∗∗ H.E.R.O. Tutankham Ice Hockey Battle Zone River Raid ∗∗Surround∗∗ Q*Bert Tennis Fishing Derby Zaxxon Pong Freeway Beam Rider Bank Heist Time Pilot Name This Game Wizard of Wor Kung-Fu Master Enduro James Bond Space Invaders Up and Down ∗∗Phoenix∗∗ ∗∗Defender∗∗ Asterix Kangaroo Crazy Climber Krull Road Runner Star Gunner Boxi


and Down ∗∗Phoenix∗∗ ∗∗Defender∗∗ Asterix Kangaroo Crazy Climber Krull Road Runner Star Gunner Boxing Gopher Robotank Double Dunk Assault Breakout Demon Attack Atlantis Video Pinball Double DQN (tuned) Double DQN DQN Figure 4: Normalized scores on 57 Atari games, tested for 100 episodes per game with human starts. Compared to Mnih et al. (2015), eight games additional games were tested. These are indicated with stars and a bold font. both revert to Q-learning. In addition, we reduced the exploration during learning from ϵ = 0.1 to ϵ = 0.01, and then used ϵ = 0.001 during evaluation. Finally, the tuned version uses a single shared bias for all action values in the top layer of the network. Each of these changes improved performance and together they result in clearly better results.3 Table


ach of these changes improved performance and together they result in clearly better results.3 Table 2 reports summary statistics for this evaluation on the 49 games from Mnih et al. (2015). Double DQN obtains clearly higher median and mean scores. Again Gorila DQN (Nair et al., 2015) is not included in the table, but for completeness note it obtained a median of 78% and a mean of 259%. Detailed results, plus results for an additional 8 games, are available in Figure 4 and in the appendix. On several games the improvements from DQN to Double DQN are striking, in some cases bringing scores much closer to 3Except for Tennis, where the lower ϵ during training seemed to hurt rather than help.


human, or even surpassing these. Double DQN appears more robust to this more challenging evaluation, suggesting that appropriate generalizations occur and that the found solutions do not exploit the determinism of the environments. This is appealing, as it indicates progress towards ﬁnding general solutions rather than a deterministic sequence of steps that would be less robust. Discussion This paper has ﬁve contributions. First, we have shown why Q-learning can be overoptimistic in large-scale problems, even if these are deterministic, due to the inherent estimation errors of learning. Second, by analyzing the value estimates on Atari games we have shown that these overestimations are more common and severe in practice than previously acknowledged.


own that these overestimations are more common and severe in practice than previously acknowledged. Third, we have shown that Double Q-learning can be used at scale to successfully reduce this overoptimism, resulting in more stable and reliable learning. Fourth, we have proposed a speciﬁc implementation called Double DQN, that uses the existing architecture and deep neural network of the DQN algorithm without requiring additional networks or parameters. Finally, we have shown that Double DQN ﬁnds better policies, obtaining new state-ofthe-art results on the Atari 2600 domain. Acknowledgments We would like to thank Tom Schaul, Volodymyr Mnih, Marc Bellemare, Thomas Degris, Georg Ostrovski, and Richard Sutton for helpful comments, and everyone at Google DeepMind for a constructive research e


d Richard Sutton for helpful comments, and everyone at Google DeepMind for a constructive research environment. References R. Agrawal. Sample mean based index policies with O(log n) regret for the multi-armed bandit problem. Advances in Applied Probability, pages 1054–1078, 1995. P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235– 256, 2002. L. Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning: Proceedings of the Twelfth International Conference, pages 30–37, 1995. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res. (JAIR), 47:253–279, 2013. R. I. Brafman and M.


platform for general agents. J. Artif. Intell. Res. (JAIR), 47:253–279, 2013. R. I. Brafman and M. Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. The Journal of Machine Learning Research, 3:213–231, 2003. K. Fukushima. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks, 1(2):119– 130, 1988. L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of Artiﬁcial Intelligence Research, 4:237–285, 1996. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. L. Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching.


998. L. Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3):293– 321, 1992. H. R. Maei. Gradient temporal-difference learning algorithms. PhD thesis, University of Alberta, 2011. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Humanlevel control through deep reinforcement learning. Nature, 518 (7540):529–533, 2015. A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. D. Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen, S. Legg, V. Mnih, K. Kavukcuoglu, and D. Silver.


neershelvam, M. Suleyman, C. Beattie, S. Petersen, S. Legg, V. Mnih, K. Kavukcuoglu, and D. Silver. Massively parallel methods for deep reinforcement learning. In Deep Learning Workshop, ICML, 2015. M. Riedmiller. Neural ﬁtted Q iteration - ﬁrst experiences with a data efﬁcient neural reinforcement learning method. In J. Gama, R. Camacho, P. Brazdil, A. Jorge, and L. Torgo, editors, Proceedings of the 16th European Conference on Machine Learning (ECML’05), pages 317–328. Springer, 2005. B. Sallans and G. E. Hinton. Reinforcement learning with factored states and actions. The Journal of Machine Learning Research, 5:1063–1088, 2004. A. L. Strehl, L. Li, and M. L. Littman. Reinforcement learning in ﬁnite MDPs: PAC analysis. The Journal of Machine Learning Research, 10:2413–2444, 2009. R. S.


ng in ﬁnite MDPs: PAC analysis. The Journal of Machine Learning Research, 10:2413–2444, 2009. R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9–44, 1988. R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the seventh international conference on machine learning, pages 216–224, 1990. R. S. Sutton and A. G. Barto. Introduction to reinforcement learning. MIT Press, 1998. R. S. Sutton, C. Szepesv´ari, and H. R. Maei. A convergent O(n) algorithm for off-policy temporal-difference learning with linear function approximation. Advances in Neural Information Processing Systems 21 (NIPS-08), 21:1609–1616, 2008. R. S. Sutton, A. R. Mahmood, and M. White.


ion Processing Systems 21 (NIPS-08), 21:1609–1616, 2008. R. S. Sutton, A. R. Mahmood, and M. White. An emphatic approach to the problem of off-policy temporal-difference learning. arXiv preprint arXiv:1503.04269, 2015. I. Szita and A. L˝orincz. The many faces of optimism: a unifying approach. In Proceedings of the 25th international conference on Machine learning, pages 1048–1055. ACM, 2008. G. Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58–68, 1995. S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In M. Mozer, P. Smolensky, D. Touretzky, J. Elman, and A. Weigend, editors, Proceedings of the 1993 Connectionist Models Summer School, Hillsdale, NJ, 1993. Lawrence Erlbaum. J. N. Tsitsiklis and B. Van Roy.


onist Models Summer School, Hillsdale, NJ, 1993. Lawrence Erlbaum. J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690, 1997. H. van Hasselt. Double Q-learning. Advances in Neural Information Processing Systems, 23:2613–2621, 2010. H. van Hasselt. Insights in Reinforcement Learning. PhD thesis, Utrecht University, 2011. C. J. C. H. Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge England, 1989.


Appendix Theorem 1. Consider a state s in which all the true optimal action values are equal at Q∗(s, a) = V∗(s) for some V∗(s). Let Qt be arbitrary value estimates that are on the whole unbiased in the sense that P a(Qt(s, a) −V∗(s)) = 0, but that are not all zero, such that 1 m P a(Qt(s, a) −V∗(s))2 = C for some C > 0, where m ≥2 is the number of actions in s. Under these conditions, maxa Qt(s, a) ≥V∗(s) + q C m−1. This lower bound is tight. Under the same conditions, the lower bound on the absolute error of the Double Q-learning estimate is zero. Proof of Theorem 1. Deﬁne the errors for each action a as ϵa = Qt(s, a) −V∗(s). Suppose that there exists a setting of {ϵa} such that maxa ϵa < q C m−1.


a as ϵa = Qt(s, a) −V∗(s). Suppose that there exists a setting of {ϵa} such that maxa ϵa < q C m−1. Let {ϵ+ i } be the set of positive ϵ of size n, and {ϵ− j } the set of strictly negative ϵ of size m −n (such that {ϵ} = {ϵ+ i } ∪{ϵ− j }). If n = m, then P a ϵa = 0 =⇒ ϵa = 0 ∀a, which contradicts P a ϵ2 a = mC. Hence, it must be that n ≤m −1. Then, Pn i=1 ϵ+ i ≤n maxi ϵ+ i < n q C m−1, and therefore (using the constraint P a ϵa = 0) we also have that Pm−n j=1 |ϵ− j | < n q C m−1. This implies maxj |ϵ− j | < n q C m−1. By H¨older’s inequality, then m−n X j=1 (ϵ− j )2 ≤ m−n X j=1 |ϵ− j | · max j |ϵ− j | < n r C m −1n r C m −1 . We can now combine these relations to compute an upper-bound on the sum of squares for all ϵa: m X a=1 (ϵa)2 = n X i=1 (ϵ+ i )2 + m−n X j=1 (ϵ− j )2 < n C m −1 + n r


e sum of squares for all ϵa: m X a=1 (ϵa)2 = n X i=1 (ϵ+ i )2 + m−n X j=1 (ϵ− j )2 < n C m −1 + n r C m −1n r C m −1 = C n(n + 1) m −1 ≤mC. This contradicts the assumption that Pm a=1 ϵ2 a < mC, and therefore maxa ϵa ≥ q C m−1 for all settings of ϵ that satisfy the constraints. We can check that the lower-bound is tight by setting ϵa = q C m−1 for a = 1, . . . , m −1 and ϵm = − p (m −1)C. This veriﬁes P a ϵ2 a = mC and P a ϵa = 0. The only tight lower bound on the absolute error for Double Qlearning |Q′ t(s, argmaxa Qt(s, a)) −V∗(s)| is zero. This can be seen by because we can have Qt(s, a1) = V∗(s) + r C m −1 m , and Qt(s, ai) = V∗(s) − s C 1 m(m −1) , for i > 1. Then the conditions of the theorem hold. If then, furthermore, we have Q′ t(s, a1) = V∗(s) then the error is zero.


ions of the theorem hold. If then, furthermore, we have Q′ t(s, a1) = V∗(s) then the error is zero. The remaining action values Q′ t(s, ai), for i > 1, are arbitrary. Theorem 2. Consider a state s in which all the true optimal action values are equal at Q∗(s, a) = V∗(s). Suppose that the estimation errors Qt(s, a)−Q∗(s, a) are independently distributed uniformly randomly in [−1, 1]. Then, E h max a Qt(s, a) −V∗(s) i = m −1 m + 1 Proof. Deﬁne ϵa = Qt(s, a) −Q∗(s, a); this is a uniform random variable in [−1, 1]. The probability that maxa Qt(s, a) ≤x for some x is equal to the probability that ϵa ≤x for all a simultaneously. Because the estimation errors are independent, we can derive P(max a ϵa ≤x) = P(X1 ≤x ∧X2 ≤x ∧. . . ∧Xm ≤x) = m Y a=1 P(ϵa ≤x) .


are independent, we can derive P(max a ϵa ≤x) = P(X1 ≤x ∧X2 ≤x ∧. . . ∧Xm ≤x) = m Y a=1 P(ϵa ≤x) . The function P(ϵa ≤x) is the cumulative distribution function (CDF) of ϵa, which here is simply deﬁned as P(ϵa ≤x) =    0 if x ≤−1 1+x 2 if x ∈(−1, 1) 1 if x ≥1 This implies that P(max a ϵa ≤x) = m Y a=1 P(ϵa ≤x) =    0 if x ≤−1  1+x 2 m if x ∈(−1, 1) 1 if x ≥1 This gives us the CDF of the random variable maxa ϵa. Its expectation can be written as an integral E h max a ϵa i = Z 1 −1 xfmax(x) dx , where fmax is the probability density function of this variable, deﬁned as the derivative of the CDF: fmax(x) = d dxP(maxa ϵa ≤ x), so that for x ∈[−1, 1] we have fmax(x) = m 2  1+x 2 m−1. Evaluating the integral yields E h max a ϵa i = Z 1 −1 xfmax(x) dx = x + 1 2 m mx −1 m + 1 1 −1 =


valuating the integral yields E h max a ϵa i = Z 1 −1 xfmax(x) dx = x + 1 2 m mx −1 m + 1 1 −1 = m −1 m + 1 . Experimental Details for the Atari 2600 Domain We selected the 49 games to match the list used by Mnih et al. (2015), see Tables below for the full list. Each agent step is composed of four frames (the last selected action is repeated during these frames) and reward values (obtained from the Arcade Learning Environment (Bellemare et al., 2013)) are clipped between -1 and 1.


Network Architecture The convolution network used in the experiment is exactly the one proposed by proposed by Mnih et al. (2015), we only provide details here for completeness. The input to the network is a 84x84x4 tensor containing a rescaled, and gray-scale, version of the last four frames. The ﬁrst convolution layer convolves the input with 32 ﬁlters of size 8 (stride 4), the second layer has 64 layers of size 4 (stride 2), the ﬁnal convolution layer has 64 ﬁlters of size 3 (stride 1). This is followed by a fully-connected hidden layer of 512 units. All these layers are separated by Rectiﬁer Linear Units (ReLu). Finally, a fully-connected linear layer projects to the output of the network, i.e., the Q-values.


Finally, a fully-connected linear layer projects to the output of the network, i.e., the Q-values. The optimization employed to train the network is RMSProp (with momentum parameter 0.95). Hyper-parameters In all experiments, the discount was set to γ = 0.99, and the learning rate to α = 0.00025. The number of steps between target network updates was τ = 10, 000. Training is done over 50M steps (i.e., 200M frames). The agent is evaluated every 1M steps, and the best policy across these evaluations is kept as the output of the learning process. The size of the experience replay memory is 1M tuples. The memory gets sampled to update the network every 4 steps with minibatches of size 32. The simple exploration policy used is an ϵ-greedy policy with the ϵ decreasing linearly from 1 to 0.1 ove


imple exploration policy used is an ϵ-greedy policy with the ϵ decreasing linearly from 1 to 0.1 over 1M steps. Supplementary Results in the Atari 2600 Domain The Tables below provide further detailed results for our experiments in the Atari domain.


Game Random Human DQN Double DQN Alien 227.80 6875.40 3069.33 2907.30 Amidar 5.80 1675.80 739.50 702.10 Assault 222.40 1496.40 3358.63 5022.90 Asterix 210.00 8503.30 6011.67 15150.00 Asteroids 719.10 13156.70 1629.33 930.60 Atlantis 12850.00 29028.10 85950.00 64758.00 Bank Heist 14.20 734.40 429.67 728.30 Battle Zone 2360.00 37800.00 26300.00 25730.00 Beam Rider 363.90 5774.70 6845.93 7654.00 Bowling 23.10 154.80 42.40 70.50 Boxing 0.10 4.30 71.83 81.70 Breakout 1.70 31.80 401.20 375.00 Centipede 2090.90 11963.20 8309.40 4139.40 Chopper Command 811.00 9881.80 6686.67 4653.00 Crazy Climber 10780.50 35410.50 114103.33 101874.00 Demon Attack 152.10 3401.30 9711.17 9711.90 Double Dunk -18.60 -15.50 -18.07 -6.30 Enduro 0.00 309.60 301.77 319.50 Fishing Derby -91.70 5.50 -0.80 20.30 Freeway 0.00


.50 -18.07 -6.30 Enduro 0.00 309.60 301.77 319.50 Fishing Derby -91.70 5.50 -0.80 20.30 Freeway 0.00 29.60 30.30 31.80 Frostbite 65.20 4334.70 328.33 241.50 Gopher 257.60 2321.00 8520.00 8215.40 Gravitar 173.00 2672.00 306.67 170.50 H.E.R.O. 1027.00 25762.50 19950.33 20357.00 Ice Hockey -11.20 0.90 -1.60 -2.40 James Bond 29.00 406.70 576.67 438.00 Kangaroo 52.00 3035.00 6740.00 13651.00 Krull 1598.00 2394.60 3804.67 4396.70 Kung-Fu Master 258.50 22736.20 23270.00 29486.00 Montezuma’s Revenge 0.00 4366.70 0.00 0.00 Ms. Pacman 307.30 15693.40 2311.00 3210.00 Name This Game 2292.30 4076.20 7256.67 6997.10 Pong -20.70 9.30 18.90 21.00 Private Eye 24.90 69571.30 1787.57 670.10 Q*Bert 163.90 13455.00 10595.83 14875.00 River Raid 1338.50 13513.30 8315.67 12015.30 Road Runner 11.50 7845.00 18256.6


.00 10595.83 14875.00 River Raid 1338.50 13513.30 8315.67 12015.30 Road Runner 11.50 7845.00 18256.67 48377.00 Robotank 2.20 11.90 51.57 46.70 Seaquest 68.40 20181.80 5286.00 7995.00 Space Invaders 148.00 1652.30 1975.50 3154.60 Star Gunner 664.00 10250.00 57996.67 65188.00 Tennis -23.80 -8.90 -2.47 1.70 Time Pilot 3568.00 5925.00 5946.67 7964.00 Tutankham 11.40 167.60 186.70 190.60 Up and Down 533.40 9082.00 8456.33 16769.90 Venture 0.00 1187.50 380.00 93.00 Video Pinball 16256.90 17297.60 42684.07 70009.00 Wizard of Wor 563.50 4756.50 3393.33 5204.00 Zaxxon 32.50 9173.30 4976.67 10182.00 Table 3: Raw scores for the no-op evaluation condition (5 minutes emulator time). DQN as given by Mnih et al. (2015).


Game DQN Double DQN Alien 42.75 % 40.31 % Amidar 43.93 % 41.69 % Assault 246.17 % 376.81 % Asterix 69.96 % 180.15 % Asteroids 7.32 % 1.70 % Atlantis 451.85 % 320.85 % Bank Heist 57.69 % 99.15 % Battle Zone 67.55 % 65.94 % Beam Rider 119.80 % 134.73 % Bowling 14.65 % 35.99 % Boxing 1707.86 % 1942.86 % Breakout 1327.24 % 1240.20 % Centipede 62.99 % 20.75 % Chopper Command 64.78 % 42.36 % Crazy Climber 419.50 % 369.85 % Demon Attack 294.20 % 294.22 % Double Dunk 17.10 % 396.77 % Enduro 97.47 % 103.20 % Fishing Derby 93.52 % 115.23 % Freeway 102.36 % 107.43 % Frostbite 6.16 % 4.13 % Gopher 400.43 % 385.66 % Gravitar 5.35 % -0.10 % H.E.R.O. 76.50 % 78.15 % Ice Hockey 79.34 % 72.73 % James Bond 145.00 % 108.29 % Kangaroo 224.20 % 455.88 % Krull 277.01 % 351.33 % Kung-Fu Master 102.37 % 130.03 %


5.00 % 108.29 % Kangaroo 224.20 % 455.88 % Krull 277.01 % 351.33 % Kung-Fu Master 102.37 % 130.03 % Montezuma’s Revenge 0.00 % 0.00 % Ms. Pacman 13.02 % 18.87 % Name This Game 278.29 % 263.74 % Pong 132.00 % 139.00 % Private Eye 2.53 % 0.93 % Q*Bert 78.49 % 110.68 % River Raid 57.31 % 87.70 % Road Runner 232.91 % 617.42 % Robotank 508.97 % 458.76 % Seaquest 25.94 % 39.41 % Space Invaders 121.49 % 199.87 % Star Gunner 598.09 % 673.11 % Tennis 143.15 % 171.14 % Time Pilot 100.92 % 186.51 % Tutankham 112.23 % 114.72 % Up and Down 92.68 % 189.93 % Venture 32.00 % 7.83 % Video Pinball 2539.36 % 5164.99 % Wizard of Wor 67.49 % 110.67 % Zaxxon 54.09 % 111.04 % Table 4: Normalized results for no-op evaluation condition (5 minutes emulator time).


Game Random Human DQN Double DQN Double DQN (tuned) Alien 128.30 6371.30 570.2 621.6 1033.4 Amidar 11.80 1540.40 133.4 188.2 169.1 Assault 166.90 628.90 3332.3 2774.3 6060.8 Asterix 164.50 7536.00 124.5 5285.0 16837.0 Asteroids 871.30 36517.30 697.1 1219.0 1193.2 Atlantis 13463.00 26575.00 76108.0 260556.0 319688.0 Bank Heist 21.70 644.50 176.3 469.8 886.0 Battle Zone 3560.00 33030.00 17560.0 25240.0 24740.0 Beam Rider 254.60 14961.00 8672.4 9107.9 17417.2 Berzerk 196.10 2237.50 635.8 1011.1 Bowling 35.20 146.50 41.2 62.3 69.6 Boxing -1.50 9.60 25.8 52.1 73.5 Breakout 1.60 27.90 303.9 338.7 368.9 Centipede 1925.50 10321.90 3773.1 5166.6 3853.5 Chopper Command 644.00 8930.00 3046.0 2483.0 3495.0 Crazy Climber 9337.00 32667.00 50992.0 94315.0 113782.0 Defender 1965.50 14296.00 8531.0 27510.0


5.0 Crazy Climber 9337.00 32667.00 50992.0 94315.0 113782.0 Defender 1965.50 14296.00 8531.0 27510.0 Demon Attack 208.30 3442.80 12835.2 13943.5 69803.4 Double Dunk -16.00 -14.40 -21.6 -6.4 -0.3 Enduro -81.80 740.20 475.6 475.9 1216.6 Fishing Derby -77.10 5.10 -2.3 -3.4 3.2 Freeway 0.10 25.60 25.8 26.3 28.8 Frostbite 66.40 4202.80 157.4 258.3 1448.1 Gopher 250.00 2311.00 2731.8 8742.8 15253.0 Gravitar 245.50 3116.00 216.5 170.0 200.5 H.E.R.O. 1580.30 25839.40 12952.5 15341.4 14892.5 Ice Hockey -9.70 0.50 -3.8 -3.6 -2.5 James Bond 33.50 368.50 348.5 416.0 573.0 Kangaroo 100.00 2739.00 2696.0 6138.0 11204.0 Krull 1151.90 2109.10 3864.0 6130.4 6796.1 Kung-Fu Master 304.00 20786.80 11875.0 22771.0 30207.0 Montezuma’s Revenge 25.00 4182.00 50.0 30.0 42.0 Ms.


Master 304.00 20786.80 11875.0 22771.0 30207.0 Montezuma’s Revenge 25.00 4182.00 50.0 30.0 42.0 Ms. Pacman 197.80 15375.00 763.5 1401.8 1241.3 Name This Game 1747.80 6796.00 5439.9 7871.5 8960.3 Phoenix 1134.40 6686.20 10364.0 12366.5 Pit Fall -348.80 5998.90 -432.9 -186.7 Pong -18.00 15.50 16.2 17.7 19.1 Private Eye 662.80 64169.10 298.2 346.3 -575.5 Q*Bert 183.00 12085.00 4589.8 10713.3 11020.8 River Raid 588.30 14382.20 4065.3 6579.0 10838.4 Road Runner 200.00 6878.00 9264.0 43884.0 43156.0 Robotank 2.40 8.90 58.5 52.0 59.1 Seaquest 215.50 40425.80 2793.9 4199.4 14498.0 Skiing -15287.40 -3686.60 -29404.3 -11490.4 Solaris 2047.20 11032.60 2166.8 810.0 Space Invaders 182.60 1464.90 1449.7 1495.7 2628.7 Star Gunner 697.00 9528.00 34081.0 53052.0 58365.0 Surround -9.70 5.40 -7.6 1.9 Tennis


495.7 2628.7 Star Gunner 697.00 9528.00 34081.0 53052.0 58365.0 Surround -9.70 5.40 -7.6 1.9 Tennis -21.40 -6.70 -2.3 11.0 -7.8 Time Pilot 3273.00 5650.00 5640.0 5375.0 6608.0 Tutankham 12.70 138.30 32.4 63.6 92.2 Up and Down 707.20 9896.10 3311.3 4721.1 19086.9 Venture 18.00 1039.00 54.0 75.0 21.0 Video Pinball 20452.0 15641.10 20228.1 148883.6 367823.7 Wizard of Wor 804.00 4556.00 246.0 155.0 6201.0 Yars Revenge 1476.90 47135.20 5439.5 6270.6 Zaxxon 475.00 8443.00 831.0 7874.0 8593.0 Table 5: Raw scores for the human start condition (30 minutes emulator time). DQN as given by Nair et al. (2015).


Game DQN Double DQN Double DQN (tuned) Alien 7.08% 7.90% 14.50% Amidar 7.95% 11.54% 10.29% Assault 685.15% 564.37% 1275.74% Asterix -0.54% 69.46% 226.18% Asteroids -0.49% 0.98% 0.90% Atlantis 477.77% 1884.48% 2335.46% Bank Heist 24.82% 71.95% 138.78% Battle Zone 47.51% 73.57% 71.87% Beam Rider 57.24% 60.20% 116.70% Berzerk 21.54% 39.92% Bowling 5.39% 24.35% 30.91% Boxing 245.95% 482.88% 675.68% Breakout 1149.43% 1281.75% 1396.58% Centipede 22.00% 38.60% 22.96% Chopper Command 28.99% 22.19% 34.41% Crazy Climber 178.55% 364.24% 447.69% Defender 53.25% 207.17% Demon Attack 390.38% 424.65% 2151.65% Double Dunk -350.00% 600.00% 981.25% Enduro 67.81% 67.85% 157.96% Fishing Derby 91.00% 89.66% 97.69% Freeway 100.78% 102.75% 112.55% Frostbite 2.20% 4.64% 33.40% Gopher 120.42% 412.07% 727.95% Gravi


9% Freeway 100.78% 102.75% 112.55% Frostbite 2.20% 4.64% 33.40% Gopher 120.42% 412.07% 727.95% Gravitar -1.01% -2.63% -1.57% H.E.R.O. 46.88% 56.73% 54.88% Ice Hockey 57.84% 59.80% 70.59% James Bond 94.03% 114.18% 161.04% Kangaroo 98.37% 228.80% 420.77% Krull 283.34% 520.11% 589.66% Kung-Fu Master 56.49% 109.69% 145.99% Montezuma’s Revenge 0.60% 0.12% 0.41% Ms. Pacman 3.73% 7.93% 6.88% Name This Game 73.14% 121.30% 142.87% Phoenix 166.25% 202.31% Pit Fall -1.32% 2.55% Pong 102.09% 106.57% 110.75% Private Eye -0.57% -0.50% -1.95% Q*Bert 37.03% 88.48% 91.06% River Raid 25.21% 43.43% 74.31% Road Runner 135.73% 654.15% 643.25% Robotank 863.08% 763.08% 872.31% Seaquest 6.41% 9.91% 35.52% Skiing -121.69% 32.73% Solaris 1.33% -13.77% Space Invaders 98.81% 102.40% 190.76% Star Gunner 378.03% 592.85


21.69% 32.73% Solaris 1.33% -13.77% Space Invaders 98.81% 102.40% 190.76% Star Gunner 378.03% 592.85% 653.02% Surround 13.91% 76.82% Tennis 129.93% 220.41% 92.52% Time Pilot 99.58% 88.43% 140.30% Tutankham 15.68% 40.53% 63.30% Up and Down 28.34% 43.68% 200.02% Venture 3.53% 5.58% 0.29% Video Pinball -4.65% 2669.60% 7220.51% Wizard of Wor -14.87% -17.30% 143.84% Yars Revenge 8.68% 10.50% Zaxxon 4.47% 92.86% 101.88% Table 6: Normalized scores for the human start condition (30 minutes emulator time).


Published as a conference paper at ICLR 2016 PRIORITIZED EXPERIENCE REPLAY Tom Schaul, John Quan, Ioannis Antonoglou and David Silver Google DeepMind {schaul,johnquan,ioannisa,davidsilver}@google.com ABSTRACT Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their signiﬁcance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efﬁciently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm th


We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new stateof-the-art, outperforming DQN with uniform replay on 41 out of 49 games. 1 INTRODUCTION Online reinforcement learning (RL) agents incrementally update their parameters (of the policy, value function or model) while they observe a stream of experience. In their simplest form, they discard incoming data immediately, after a single update. Two issues with this are (a) strongly correlated updates that break the i.i.d. assumption of many popular stochastic gradient-based algorithms, and (b) the rapid forgetting of possibly rare experiences that would be useful later on.


lgorithms, and (b) the rapid forgetting of possibly rare experiences that would be useful later on. Experience replay (Lin, 1992) addresses both of these issues: with experience stored in a replay memory, it becomes possible to break the temporal correlations by mixing more and less recent experience for the updates, and rare experience will be used for more than just a single update. This was demonstrated in the Deep Q-Network (DQN) algorithm (Mnih et al., 2013; 2015), which stabilized the training of a value function, represented by a deep neural network, by using experience replay. Speciﬁcally, DQN used a large sliding window replay memory, sampled from it uniformly at random, and revisited each transition1 eight times on average.


memory, sampled from it uniformly at random, and revisited each transition1 eight times on average. In general, experience replay can reduce the amount of experience required to learn, and replace it with more computation and more memory – which are often cheaper resources than the RL agent’s interactions with its environment. In this paper, we investigate how prioritizing which transitions are replayed can make experience replay more efﬁcient and effective than if all transitions are replayed uniformly. The key idea is that an RL agent can learn more effectively from some transitions than from others. Transitions may be more or less surprising, redundant, or task-relevant. Some transitions may not be immediately useful to the agent, but might become so when the agent competence increases


may not be immediately useful to the agent, but might become so when the agent competence increases (Schmidhuber, 1991). Experience replay liberates online learning agents from processing transitions in the exact order they are experienced. Prioritized replay further liberates agents from considering transitions with the same frequency that they are experienced. In particular, we propose to more frequently replay transitions with high expected learning progress, as measured by the magnitude of their temporal-difference (TD) error. This prioritization can lead to a loss of diversity, which we alleviate with stochastic prioritization, and introduce bias, which we correct with importance sampling.


iate with stochastic prioritization, and introduce bias, which we correct with importance sampling. Our resulting algorithms are robust and scalable, which we demonstrate on the Atari 2600 benchmark suite, where we obtain faster learning and state-of-the-art performance. 1 A transition is the atomic unit of interaction in RL, in our case a tuple of (state St−1, action At−1, reward Rt, discount γt, next state St). We choose this for simplicity, but most of the arguments in this paper also hold for a coarser ways of chunking experience, e.g. into sequences or episodes. 1 arXiv:1511.05952v4 [cs.LG] 25 Feb 2016


Published as a conference paper at ICLR 2016 2 BACKGROUND Numerous neuroscience studies have identiﬁed evidence of experience replay in the hippocampus of rodents, suggesting that sequences of prior experience are replayed, either during awake resting or sleep. Sequences associated with rewards appear to be replayed more frequently (Atherton et al., 2015; ´Olafsd´ottir et al., 2015; Foster & Wilson, 2006). Experiences with high magnitude TD error also appear to be replayed more often (Singer & Frank, 2009; McNamara et al., 2014). It is well-known that planning algorithms such as value iteration can be made more efﬁcient by prioritizing updates in an appropriate order. Prioritized sweeping (Moore & Atkeson, 1993; Andre et al., 1998) selects which state to update next, prioritized according


oore & Atkeson, 1993; Andre et al., 1998) selects which state to update next, prioritized according to the change in value, if that update was executed. The TD error provides one way to measure these priorities (van Seijen & Sutton, 2013). Our approach uses a similar prioritization method, but for model-free RL rather than model-based planning. Furthermore, we use a stochastic prioritization that is more robust when learning a function approximator from samples. TD-errors have also been used as a prioritization mechanism for determining where to focus resources, for example when choosing where to explore (White et al., 2014) or which features to select (Geramifard et al., 2011; Sun et al., 2011).


plore (White et al., 2014) or which features to select (Geramifard et al., 2011; Sun et al., 2011). In supervised learning, there are numerous techniques to deal with imbalanced datasets when class identities are known, including re-sampling, under-sampling and over-sampling techniques, possibly combined with ensemble methods (for a review, see Galar et al., 2012). A recent paper introduced a form of re-sampling in the context of deep RL with experience replay (Narasimhan et al., 2015); the method separates experience into two buckets, one for positive and one for negative rewards, and then picks a ﬁxed fraction from each to replay. This is only applicable in domains that (unlike ours) have a natural notion of ‘positive/negative’ experience.


y applicable in domains that (unlike ours) have a natural notion of ‘positive/negative’ experience. Furthermore, Hinton (2007) introduced a form of non-uniform sampling based on error, with an importance sampling correction, which led to a 3x speed-up on MNIST digit classiﬁcation. There have been several proposed methods for playing Atari with deep reinforcement learning, including deep Q-networks (DQN) (Mnih et al., 2013; 2015; Guo et al., 2014; Stadie et al., 2015; Nair et al., 2015; Bellemare et al., 2016), and the Double DQN algorithm (van Hasselt et al., 2016), which is the current published state-of-the-art. Simultaneously with our work, an architectural innovation that separates advantages from the value function (see the co-submission by Wang et al., 2015) has also led to substanti


tages from the value function (see the co-submission by Wang et al., 2015) has also led to substantial improvements on the Atari benchmark. 3 PRIORITIZED REPLAY Using a replay memory leads to design choices at two levels: which experiences to store, and which experiences to replay (and how to do so). This paper addresses only the latter: making the most effective use of the replay memory for learning, assuming that its contents are outside of our control (but see also Section 6). 3.1 A MOTIVATING EXAMPLE To understand the potential gains of prioritization, we introduce an artiﬁcial ‘Blind Cliffwalk’ environment (described in Figure 1, left) that exempliﬁes the challenge of exploration when rewards are rare.


t (described in Figure 1, left) that exempliﬁes the challenge of exploration when rewards are rare. With only n states, the environment requires an exponential number of random steps until the ﬁrst non-zero reward; to be precise, the chance that a random sequence of actions will lead to the reward is 2−n. Furthermore, the most relevant transitions (from rare successes) are hidden in a mass of highly redundant failure cases (similar to a bipedal robot falling over repeatedly, before it discovers how to walk). We use this example to highlight the difference between the learning times of two agents. Both agents perform Q-learning updates on transitions drawn from the same replay memory. The ﬁrst agent replays transitions uniformly at random, while the second agent invokes an oracle to priorit


t agent replays transitions uniformly at random, while the second agent invokes an oracle to prioritize transitions. This oracle greedily selects the transition that maximally reduces the global loss in its current state (in hindsight, after the parameter update). For the details of the setup, see Appendix B.1. Figure 1 (right) shows that picking the transitions in a good order can lead to exponential speed-ups 2


Published as a conference paper at ICLR 2016 1 2 n-1 … n R=1 Figure 1: Left: Illustration of the ‘Blind Cliffwalk’ example domain: there are two actions, a ‘right’ and a ‘wrong’ one, and the episode is terminated whenever the agent takes the ‘wrong’ action (dashed red arrows). Taking the ‘right’ action progresses through a sequence of n states (black arrows), at the end of which lies a ﬁnal reward of 1 (green arrow); reward is 0 elsewhere. We chose a representation such that generalizing over what action is ‘right’ is not possible. Right: Median number of learning steps required to learn the value function as a function of the size of the total number of transitions in the replay memory. Note the log-log scale, which highlights the exponential speed-up from replaying with an oracle (bright


e the log-log scale, which highlights the exponential speed-up from replaying with an oracle (bright blue), compared to uniform replay (black); faint lines are min/max values from 10 independent runs. over uniform choice. Such an oracle is of course not realistic, yet the large gap motivates our search for a practical approach that improves on uniform random replay. 3.2 PRIORITIZING WITH TD-ERROR The central component of prioritized replay is the criterion by which the importance of each transition is measured. One idealised criterion would be the amount the RL agent can learn from a transition in its current state (expected learning progress). While this measure is not directly accessible, a reasonable proxy is the magnitude of a transition’s TD error δ, which indicates how ‘surprising’ o


a reasonable proxy is the magnitude of a transition’s TD error δ, which indicates how ‘surprising’ or unexpected the transition is: speciﬁcally, how far the value is from its next-step bootstrap estimate (Andre et al., 1998). This is particularly suitable for incremental, online RL algorithms, such as SARSA or Q-learning, that already compute the TD-error and update the parameters in proportion to δ. The TD-error can be a poor estimate in some circumstances as well, e.g. when rewards are noisy; see Appendix A for a discussion of alternatives. To demonstrate the potential effectiveness of prioritizing replay by TD error, we compare the uniform and oracle baselines in the Blind Cliffwalk to a ‘greedy TD-error prioritization’ algorithm.


niform and oracle baselines in the Blind Cliffwalk to a ‘greedy TD-error prioritization’ algorithm. This algorithm stores the last encountered TD error along with each transition in the replay memory. The transition with the largest absolute TD error is replayed from the memory. A Q-learning update is applied to this transition, which updates the weights in proportion to the TD error. New transitions arrive without a known TD-error, so we put them at maximal priority in order to guarantee that all experience is seen at least once. Figure 2 (left), shows that this algorithm results in a substantial reduction in the effort required to solve the Blind Cliffwalk task.2 Implementation: To scale to large memory sizes N, we use a binary heap data structure for the priority queue, for which ﬁnding


o large memory sizes N, we use a binary heap data structure for the priority queue, for which ﬁnding the maximum priority transition when sampling is O(1) and updating priorities (with the new TD-error after a learning step) is O(log N). See Appendix B.2.1 for more details. 3.3 STOCHASTIC PRIORITIZATION However, greedy TD-error prioritization has several issues. First, to avoid expensive sweeps over the entire replay memory, TD errors are only updated for the transitions that are replayed. One consequence is that transitions that have a low TD error on ﬁrst visit may not be replayed for a long time (which means effectively never with a sliding window replay memory). Further, it is sensitive to noise spikes (e.g.


ectively never with a sliding window replay memory). Further, it is sensitive to noise spikes (e.g. when rewards are stochastic), which can be exacerbated by bootstrapping, where 2 Note that a random (or optimistic) initialization of the Q-values is necessary with greedy prioritization. If initializing with zero instead, unrewarded transitions would appear to have zero error initially, be placed at the bottom of the queue, and not be revisited until the error on other transitions drops below numerical precision. 3


Published as a conference paper at ICLR 2016 Figure 2: Median number of updates required for Q-learning to learn the value function on the Blind Cliffwalk example, as a function of the total number of transitions (only a single one of which was successful and saw the non-zero reward). Faint lines are min/max values from 10 random initializations. Black is uniform random replay, cyan uses the hindsight-oracle to select transitions, red and blue use prioritized replay (rank-based and proportional respectively). The results differ by multiple orders of magnitude, thus the need for a log-log plot. In both subplots it is evident that replaying experience in the right order makes an enormous difference to the number of updates required. See Appendix B.1 for details.


order makes an enormous difference to the number of updates required. See Appendix B.1 for details. Left: Tabular representation, greedy prioritization. Right: Linear function approximation, both variants of stochastic prioritization. approximation errors appear as another source of noise. Finally, greedy prioritization focuses on a small subset of the experience: errors shrink slowly, especially when using function approximation, meaning that the initially high error transitions get replayed frequently. This lack of diversity that makes the system prone to over-ﬁtting. To overcome these issues, we introduce a stochastic sampling method that interpolates between pure greedy prioritization and uniform random sampling.


c sampling method that interpolates between pure greedy prioritization and uniform random sampling. We ensure that the probability of being sampled is monotonic in a transition’s priority, while guaranteeing a non-zero probability even for the lowest-priority transition. Concretely, we deﬁne the probability of sampling transition i as P(i) = pα i P k pα k (1) where pi > 0 is the priority of transition i. The exponent α determines how much prioritization is used, with α = 0 corresponding to the uniform case. The ﬁrst variant we consider is the direct, proportional prioritization where pi = |δi| + ϵ, where ϵ is a small positive constant that prevents the edge-case of transitions not being revisited once their error is zero.


e constant that prevents the edge-case of transitions not being revisited once their error is zero. The second variant is an indirect, rank-based prioritization where pi = 1 rank(i), where rank(i) is the rank of transition i when the replay memory is sorted according to |δi|. In this case, P becomes a power-law distribution with exponent α. Both distributions are monotonic in |δ|, but the latter is likely to be more robust, as it is insensitive to outliers. Both variants of stochastic prioritization lead to large speed-ups over the uniform baseline on the Cliffwalk task, as shown on Figure 2 (right). Implementation: To efﬁciently sample from distribution (1), the complexity cannot depend on N.


ht). Implementation: To efﬁciently sample from distribution (1), the complexity cannot depend on N. For the rank-based variant, we can approximate the cumulative density function with a piecewise linear function with k segments of equal probability. The segment boundaries can be precomputed (they change only when N or α change). At runtime, we sample a segment, and then sample uniformly among the transitions within it. This works particularly well in conjunction with a minibatchbased learning algorithm: choose k to be the size of the minibatch, and sample exactly one transition from each segment – this is a form of stratiﬁed sampling that has the added advantage of balancing out the minibatch (there will always be exactly one transition with high magnitude δ, one with medium magnitude, etc


h (there will always be exactly one transition with high magnitude δ, one with medium magnitude, etc). The proportional variant is different, also admits an efﬁcient implementation based on a ‘sum-tree’ data structure (where every node is the sum of its children, with the priorities as the leaf nodes), which can be efﬁciently updated and sampled from. See Appendix B.2.1 for more additional details. 4


Published as a conference paper at ICLR 2016 Algorithm 1 Double DQN with proportional prioritization 1: Input: minibatch k, step-size η, replay period K and size N, exponents α and β, budget T. 2: Initialize replay memory H = ∅, ∆= 0, p1 = 1 3: Observe S0 and choose A0 ∼πθ(S0) 4: for t = 1 to T do 5: Observe St, Rt, γt 6: Store transition (St−1, At−1, Rt, γt, St) in H with maximal priority pt = maxi<t pi 7: if t ≡0 mod K then 8: for j = 1 to k do 9: Sample transition j ∼P(j) = pα j / P i pα i 10: Compute importance-sampling weight wj = (N · P(j))−β / maxi wi 11: Compute TD-error δj = Rj + γjQtarget (Sj, arg maxa Q(Sj, a)) −Q(Sj−1, Aj−1) 12: Update transition priority pj ←|δj| 13: Accumulate weight-change ∆←∆+ wj · δj · ∇θQ(Sj−1, Aj−1) 14: end for 15: Update weights θ ←θ + η · ∆, reset ∆= 0


weight-change ∆←∆+ wj · δj · ∇θQ(Sj−1, Aj−1) 14: end for 15: Update weights θ ←θ + η · ∆, reset ∆= 0 16: From time to time copy weights into target network θtarget ←θ 17: end if 18: Choose action At ∼πθ(St) 19: end for 3.4 ANNEALING THE BIAS The estimation of the expected value with stochastic updates relies on those updates corresponding to the same distribution as its expectation. Prioritized replay introduces bias because it changes this distribution in an uncontrolled fashion, and therefore changes the solution that the estimates will converge to (even if the policy and state distribution are ﬁxed). We can correct this bias by using importance-sampling (IS) weights wi =  1 N · 1 P(i) β that fully compensates for the non-uniform probabilities P(i) if β = 1.


ghts wi =  1 N · 1 P(i) β that fully compensates for the non-uniform probabilities P(i) if β = 1. These weights can be folded into the Q-learning update by using wiδi instead of δi (this is thus weighted IS, not ordinary IS, see e.g. Mahmood et al., 2014). For stability reasons, we always normalize weights by 1/ maxi wi so that they only scale the update downwards. In typical reinforcement learning scenarios, the unbiased nature of the updates is most important near convergence at the end of training, as the process is highly non-stationary anyway, due to changing policies, state distributions and bootstrap targets; we hypothesize that a small bias can be ignored in this context (see also Figure 12 in the appendix for a case study of full IS correction on Atari).


this context (see also Figure 12 in the appendix for a case study of full IS correction on Atari). We therefore exploit the ﬂexibility of annealing the amount of importance-sampling correction over time, by deﬁning a schedule on the exponent β that reaches 1 only at the end of learning. In practice, we linearly anneal β from its initial value β0 to 1. Note that the choice of this hyperparameter interacts with choice of prioritization exponent α; increasing both simultaneously prioritizes sampling more aggressively at the same time as correcting for it more strongly. Importance sampling has another beneﬁt when combined with prioritized replay in the context of non-linear function approximation (e.g.


eﬁt when combined with prioritized replay in the context of non-linear function approximation (e.g. deep neural networks): here large steps can be very disruptive, because the ﬁrst-order approximation of the gradient is only reliable locally, and have to be prevented with a smaller global step-size. In our approach instead, prioritization makes sure high-error transitions are seen many times, while the IS correction reduces the gradient magnitudes (and thus the effective step size in parameter space), and allowing the algorithm follow the curvature of highly non-linear optimization landscapes because the Taylor expansion is constantly re-approximated. We combine our prioritized replay algorithm into a full-scale reinforcement learning agent, based on the state-of-the-art Double DQN algorit


thm into a full-scale reinforcement learning agent, based on the state-of-the-art Double DQN algorithm. Our principal modiﬁcation is to replace the uniform random sampling used by Double DQN with our stochastic prioritization and importance sampling methods (see Algorithm 1). 5


Published as a conference paper at ICLR 2016 -100% 0% 100% 200% normalized scoreagent normalized scoreDouble DQN Video Pinball Demon Attack Robotank Breakout Defender Tutankham Boxing Stargunner Bowling Bank Heist Centipede Q*bert Yars' Revenge Freeway Pong Pitfall! Montezuma's Revenge Berzerk Amidar Kangaroo Alien Asteroids Gravitar Private Eye Surround Ms. Pac-Man Time Pilot Kung-Fu Master Fishing Derby Skiing Battlezone Wizard of Wor Zaxxon Venture Solaris Ice Hockey Up'n Down H.E.R.O. Chopper Command Tennis River Raid Krull Frostbite Name This Game Seaquest Crazy Climber Beam Rider Enduro Asterix Road Runner Phoenix Assault Space Invaders Double Dunk James Bond 007 Atlantis Gopher rank-based proportional Figure 3: Difference in normalized score (the gap between random and human is 100%


ased proportional Figure 3: Difference in normalized score (the gap between random and human is 100%) on 57 games with human starts, comparing Double DQN with and without prioritized replay (rank-based variant in red, proportional in blue), showing substantial improvements in most games. Exact scores are in Table 6. See also Figure 9 where regular DQN is the baseline. 4 ATARI EXPERIMENTS With all these concepts in place, we now investigate to what extent replay with such prioritized sampling can improve performance in realistic problem domains. For this, we chose the collection of Atari benchmarks (Bellemare et al., 2012) with their end-to-end RL from vision setup, because they are popular and contain diverse sets of challenges, including delayed credit assignment, partial observability, a


nd contain diverse sets of challenges, including delayed credit assignment, partial observability, and difﬁcult function approximation (Mnih et al., 2015; van Hasselt et al., 2016). Our hypothesis is that prioritized replay is generally useful, so that it will make learning with experience replay more efﬁcient without requiring careful problem-speciﬁc hyperparameter tuning. We consider two baseline algorithms that use uniform experience replay, namely the version of the DQN algorithm from the Nature paper (Mnih et al., 2015), and its recent extension Double DQN (van Hasselt et al., 2016) that substantially improved the state-of-the-art by reducing the overestimation bias with Double Q-learning (van Hasselt, 2010). Throughout this paper we use the tuned version of the Double DQN algorithm.


ng (van Hasselt, 2010). Throughout this paper we use the tuned version of the Double DQN algorithm. For this paper, the most relevant component of these baselines is the replay mechanism: all experienced transitions are stored in a sliding window memory that retains the last 106 transitions. The algorithm processes minibatches of 32 transitions sampled uniformly from the memory. One minibatch update is done for each 4 new transitions entering the memory, so all experience is replayed 8 times on average. Rewards and TD-errors are clipped to fall within [−1, 1] for stability reasons. We use the identical neural network architecture, learning algorithm, replay memory and evaluation setup as for the baselines (see Appendix B.2).


re, learning algorithm, replay memory and evaluation setup as for the baselines (see Appendix B.2). The only difference is the mechanism for sampling transitions from the replay memory, with is now done according to Algorithm 1 instead of uniformly. We compare the baselines to both variants of prioritized replay (rank-based and proportional). Only a single hyperparameter adjustment was necessary compared to the baseline: Given that prioritized replay picks high-error transitions more often, the typical gradient magnitudes are larger, so we reduced the step-size η by a factor 4 compared to the (Double) DQN setup. For the α and β0 hyperparameters that are introduced by prioritization, we did a coarse grid search (evaluated on a subset of 8 games), and found the sweet spot to be α = 0.7, β0 =


coarse grid search (evaluated on a subset of 8 games), and found the sweet spot to be α = 0.7, β0 = 0.5 for the rank-based variant and α = 0.6, β0 = 0.4 for the proportional variant. These choices are trading off aggressiveness with robustness, but it is easy to revert to a behavior closer to the baseline by reducing α and/or increasing β. We produce the results by running each variant with a single hyperparameter setting across all games, as was done for the baselines. Our main evaluation metric is the quality of the best policy, 6


Published as a conference paper at ICLR 2016 0 50 100 150 200 training step (1e6) 0% 20% 40% 60% 80% 100% 120% 140% normalized max score 0 50 100 150 200 training step (1e6) 0% 20% 40% 60% 80% 100% 120% 140% normalized mean score uniform rank-based proportional uniform DQN Figure 4: Summary plots of learning speed. Left: median over 57 games of the maximum baselinenormalized score achieved so far. The baseline-normalized score is calculated as in Equation 4 but using the maximum Double DQN score seen across training is used instead of the human score. The equivalence points are highlighted with dashed lines; those are the steps at which the curves reach 100%, (i.e., when the algorithm performs equivalently to Double DQN in terms of median over games).


100%, (i.e., when the algorithm performs equivalently to Double DQN in terms of median over games). For rank-based and proportional prioritization these are at 47% and 38% of total training time. Right: Similar to the left, but using the mean instead of maximum, which captures cumulative performance rather than peak performance. Here rank-based and proportional prioritization reach the equivalence points at 41% and 43% of total training time, respectively. For the detailed learning curves that these plots summarize, see Figure 7. in terms of average score per episode, given start states sampled from human traces (as introduced in Nair et al., 2015 and used in van Hasselt et al., 2016, which requires more robustness and generalization as the agent cannot rely on repeating a single memorized


requires more robustness and generalization as the agent cannot rely on repeating a single memorized trajectory). These results are summarized in Table 1 and Figure 3, but full results and raw scores can be found in Tables 7 and 6 in the Appendix. A secondary metric is the learning speed, which we summarize on Figure 4, with more detailed learning curves on Figures 7 and 8. DQN Double DQN (tuned) baseline rank-based baseline rank-based proportional Median 48% 106% 111% 113% 128% Mean 122% 355% 418% 454% 551% > baseline – 41 – 38 42 > human 15 25 30 33 33 # games 49 49 57 57 57 Table 1: Summary of normalized scores. See Table 6 in the appendix for full results. We ﬁnd that adding prioritized replay to DQN leads to a substantial improvement in score on 41 out of 49 games (compare columns 2 a


replay to DQN leads to a substantial improvement in score on 41 out of 49 games (compare columns 2 and 3 of Table 6 or Figure 9 in the appendix), with the median normalized performance across 49 games increasing from 48% to 106%. Furthermore, we ﬁnd that the boost from prioritized experience replay is complementary to the one from introducing Double Q-learning into DQN: performance increases another notch, leading to the current state-of-the-art on the Atari benchmark (see Figure 3). Compared to Double DQN, the median performance across 57 games increased from 111% to 128%, and the mean performance from 418% to 551% bringing additional games such as River Raid, Seaquest and Surround to a human level for the ﬁrst time, and making large jumps on others (e.g.


d, Seaquest and Surround to a human level for the ﬁrst time, and making large jumps on others (e.g. Gopher, James Bond 007 or Space Invaders). Note that mean performance is not a very reliable metric because a single game (Video Pinball) has a dominant contribution. Prioritizing replay gives a performance boost on almost all games, and on aggregate, learning is twice as fast (see Figures 4 and 8). The learning curves on Figure 7 illustrate that while the two variants of prioritization usually lead to similar results, there are games where one of them remains close to the Double DQN baseline while the other one leads to a big boost, for example Double Dunk or Surround for the rank-based variant, and Alien, Asterix, Enduro, Phoenix or Space Invaders for the proportional variant.


-based variant, and Alien, Asterix, Enduro, Phoenix or Space Invaders for the proportional variant. Another observation from the learning curves is that compared to the uniform baseline, prioritization is effective at reducing the delay until performance gets off the ground in games that otherwise suffer from such a delay, such as Battlezone, Zaxxon or Frostbite. 7


Published as a conference paper at ICLR 2016 5 DISCUSSION In the head-to-head comparison between rank-based prioritization and proportional prioritization, we expected the rank-based variant to be more robust because it is not affected by outliers nor error magnitudes. Furthermore, its heavy-tail property also guarantees that samples will be diverse, and the stratiﬁed sampling from partitions of different errors will keep the total minibatch gradient at a stable magnitude throughout training. On the other hand, the ranks make the algorithm blind to the relative error scales, which could incur a performance drop when there is structure in the distribution of errors to be exploited, such as in sparse reward scenarios.


ere is structure in the distribution of errors to be exploited, such as in sparse reward scenarios. Perhaps surprisingly, both variants perform similarly in practice; we suspect this is due to the heavy use of clipping (of rewards and TD-errors) in the DQN algorithm, which removes outliers. Monitoring the distribution of TDerrors as a function of time for a number of games (see Figure 10 in the appendix), and found that it becomes close to a heavy-tailed distribution as learning progresses, while still differing substantially across games; this empirically validates the form of Equation 1. Figure 11, in the appendix, shows how this distribution interacts with Equation 1 to produce the effective replay probabilities.