hows how this distribution interacts with Equation 1 to produce the effective replay probabilities. While doing this analysis, we stumbled upon another phenomenon (obvious in retrospect), namely that some fraction of the visited transitions are never replayed before they drop out of the sliding window memory, and many more are replayed for the ﬁrst time only long after they are encountered. Also, uniform sampling is implicitly biased toward out-of-date transitions that were generated by a policy that has typically seen hundreds of thousands of updates since. Prioritized replay with its bonus for unseen transitions directly corrects the ﬁrst of these issues, and also tends to help with the second one, as more recent transitions tend to have larger error – this is because old transitions wil


cond one, as more recent transitions tend to have larger error – this is because old transitions will have had more opportunities to have them corrected, and because novel data tends to be less well predicted by the value function. We hypothesize that deep neural networks interact with prioritized replay in another interesting way. When we distinguish learning the value given a representation (i.e., the top layers) from learning an improved representation (i.e., the bottom layers), then transitions for which the representation is good will quickly reduce their error and then be replayed much less, increasing the learning focus on others where the representation is poor, thus putting more resources into distinguishing aliased states – if the observations and network capacity allow for it.


sources into distinguishing aliased states – if the observations and network capacity allow for it. 6 EXTENSIONS Prioritized Supervised Learning: The analogous approach to prioritized replay in the context of supervised learning is to sample non-uniformly from the dataset, each sample using a priority based on its last-seen error. This can help focus the learning on those samples that can still be learned from, devoting additional resources to the (hard) boundary cases, somewhat similarly to boosting (Galar et al., 2012). Furthermore, if the dataset is imbalanced, we hypothesize that samples from the rare classes will be sampled disproportionately often, because their errors shrink less fast, and the chosen samples from the common classes will be those nearest to the decision boundaries, l


, and the chosen samples from the common classes will be those nearest to the decision boundaries, leading to an effect similar to hard negative mining (Felzenszwalb et al., 2008). To check whether these intuitions hold, we conducted a preliminary experiment on a class-imbalanced variant of the classical MNIST digit classiﬁcation problem (LeCun et al., 1998), where we removed 99% of the samples for digits 0, 1, 2, 3, 4 in the training set, while leaving the test/validation sets untouched (i.e., those retain class balance). We compare two scenarios: in the informed case, we reweight the errors of the impoverished classes artiﬁcially (by a factor 100), in the uninformed scenario, we provide no hint that the test distribution will differ from the training distribution.


scenario, we provide no hint that the test distribution will differ from the training distribution. See Appendix B.3 for the details of the convolutional neural network training setup. Prioritized sampling (uninformed, with α = 1, β = 0) outperforms the uninformed uniform baseline, and approaches the performance of the informed uniform baseline in terms of generalization (see Figure 5); again, prioritized training is also faster in terms of learning speed. Off-policy Replay: Two standard approaches to off-policy RL are rejection sampling and using importance sampling ratios ρ to correct for how likely a transition would have been on-policy. Our approach contains analogues to both these approaches, the replay probability P and the IS-correction w.


oach contains analogues to both these approaches, the replay probability P and the IS-correction w. It appears therefore natural to apply it to off-policy RL, if transitions are available in a replay memory. In particular, we recover weighted IS with w = ρ, α = 0, β = 1 and rejection sampling 8


Published as a conference paper at ICLR 2016 Figure 5: Classiﬁcation errors as a function of supervised learning updates on severely classimbalanced MNIST. Prioritized sampling improves performance, leading to comparable errors on the test set, and approaching the imbalance-informed performance (median of 3 random initializations). Left: Number of misclassiﬁed test set samples. Right: Test set loss, highlighting overﬁtting. with p = min(1; ρ), α = 1, β = 0, in the proportional variant. Our experiments indicate that intermediate variants, possibly with annealing or ranking, could be more useful in practice – especially when IS ratios introduce high variance, i.e., when the policy of interest differs substantially from the behavior policy in some states.


e, i.e., when the policy of interest differs substantially from the behavior policy in some states. Of course, off-policy correction is complementary to our prioritization based on expected learning progress, and the same framework can be used for a hybrid prioritization by deﬁning p = ρ · |δ|, or some other sensible trade-off based on both ρ and δ. Feedback for Exploration: An interesting side-effect of prioritized replay is that the total number Mi that a transition will end up being replayed varies widely, and this gives a rough indication of how useful it was to the agent. This potentially valuable signal can be fed back to the exploration strategy that generates the transitions. For example, we could sample exploration hyperparameters (such as the fraction of random actions ϵ, the Bol


mple, we could sample exploration hyperparameters (such as the fraction of random actions ϵ, the Boltzmann temperature, or the amount of of intrinsic reward to mix in) from a parametrized distribution at the beginning of each episode, monitor the usefulness of the experience via Mi, and update the distribution toward generating more useful experience. Or, in a parallel system like the Gorila agent (Nair et al., 2015), it could guide resource allocation between a collection of concurrent but heterogeneous ‘actors’, each with different exploration hyperparameters. Prioritized Memories: Considerations that help determine which transitions to replay are likely to also be relevant for determining which memories to store and when to erase them (e.g.


are likely to also be relevant for determining which memories to store and when to erase them (e.g. when it becomes likely that they will never be replayed anymore). An explicit control over which memories to keep or erase can help reduce the required total memory size, because it reduces redundancy (frequently visited transitions will have low error, so many of them will be dropped), while automatically adjusting for what has been learned already (dropping many of the ‘easy’ transitions) and biasing the contents of the memory to where the errors remain high. This is a non-trivial aspect, because memory requirements for DQN are currently dominated by the size of the replay memory, no longer by the size of the neural network.


currently dominated by the size of the replay memory, no longer by the size of the neural network. Erasing is a more ﬁnal decision than reducing the replay probability, thus an even stronger emphasis of diversity may be necessary, for example by tracking the age of each transitions and using it to modulate the priority in such a way as to preserve sufﬁcient old experience to prevent cycles (related to ‘hall of fame’ ideas in multi-agent literature, Rosin & Belew, 1997). The priority mechanism is also ﬂexible enough to permit integrating experience from other sources, such as from a planner or from human expert trajectories (Guo et al., 2014), since knowing the source can be used to modulate each transition’s priority, e.g.


uo et al., 2014), since knowing the source can be used to modulate each transition’s priority, e.g. in such a way as to preserve a sufﬁcient fraction of external experience in memory. 7 CONCLUSION This paper introduced prioritized replay, a method that can make learning from experience replay more efﬁcient. We studied a couple of variants, devised implementations that scale to large replay memories, and found that prioritized replay speeds up learning by a factor 2 and leads to a new state-of-the-art of performance on the Atari benchmark. We laid out further variants and extensions that hold promise, namely for class-imbalanced supervised learning. 9


Published as a conference paper at ICLR 2016 ACKNOWLEDGMENTS We thank our Deepmind colleagues, in particular Hado van Hasselt, Joseph Modayil, Nicolas Heess, Marc Bellemare, Razvan Pascanu, Dharshan Kumaran, Daan Wierstra, Arthur Guez, Charles Blundell, Alex Pritzel, Alex Graves, Balaji Lakshminarayanan, Ziyu Wang, Nando de Freitas, Remi Munos and Geoff Hinton for insightful discussions and feedback. REFERENCES Andre, David, Friedman, Nir, and Parr, Ronald. Generalized prioritized sweeping. In Advances in Neural Information Processing Systems. Citeseer, 1998. Atherton, Laura A, Dupret, David, and Mellor, Jack R. Memory trace replay: the shaping of memory consolidation by neuromodulation. Trends in neurosciences, 38(9):560–570, 2015.


e shaping of memory consolidation by neuromodulation. Trends in neurosciences, 38(9):560–570, 2015. Bellemare, Marc G, Naddaf, Yavar, Veness, Joel, and Bowling, Michael. The arcade learning environment: An evaluation platform for general agents. arXiv preprint arXiv:1207.4708, 2012. Bellemare, Marc G., Ostrovski, Georg, Guez, Arthur, Thomas, Philip S., and Munos, R´emi. Increasing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence, 2016. URL http://arxiv.org/abs/1512.04860. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Cl´ement. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011. Diamond, Jared. Zebras and the Anna Karenina principle.


PS Workshop, number EPFL-CONF-192376, 2011. Diamond, Jared. Zebras and the Anna Karenina principle. Natural History, 103:4–4, 1994. Felzenszwalb, Pedro, McAllester, David, and Ramanan, Deva. A discriminatively trained, multiscale, deformable part model. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8. IEEE, 2008. Foster, David J and Wilson, Matthew A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440(7084):680–683, 2006. Galar, Mikel, Fernandez, Alberto, Barrenechea, Edurne, Bustince, Humberto, and Herrera, Francisco. A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybridbased approaches.


view on ensembles for the class imbalance problem: bagging-, boosting-, and hybridbased approaches. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 42(4):463–484, 2012. Geramifard, Alborz, Doshi, Finale, Redding, Joshua, Roy, Nicholas, and How, Jonathan. Online discovery of feature dependencies. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 881–888, 2011. Guo, Xiaoxiao, Singh, Satinder, Lee, Honglak, Lewis, Richard L, and Wang, Xiaoshi. Deep Learning for Real-Time Atari Game Play Using Ofﬂine Monte-Carlo Tree Search Planning. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 27, pp. 3338–3346.


, and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 27, pp. 3338–3346. Curran Associates, Inc., 2014. Hinton, Geoffrey E. To recognize shapes, ﬁrst learn to generate images. Progress in brain research, 165:535–547, 2007. Kingma, Diederik P. and Ba, Jimmy. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, Nov 1998. ISSN 0018-9219. doi: 10.1109/5.726791. LeCun, Yann, Cortes, Corinna, and Burges, Christopher JC. The MNIST database of handwritten digits, 1998. Lin, Long-Ji. Self-improving reactive agents based on reinforcement learning, planning and teaching.


in, Long-Ji. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293–321, 1992. Mahmood, A Rupam, van Hasselt, Hado P, and Sutton, Richard S. Weighted importance sampling for off-policy learning with linear function approximation. In Advances in Neural Information Processing Systems, pp. 3014–3022, 2014. McNamara, Colin G, Tejero-Cantero, ´Alvaro, Trouche, St´ephanie, Campo-Urriza, Natalia, and Dupret, David. Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence. Nature neuroscience, 2014. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. arXiv preprint 10


Published as a conference paper at ICLR 2016 arXiv:1312.5602, 2013. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. Moore, Andrew W and Atkeson, Christopher G. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 13(1):103–130, 1993. Nair, Arun, Srinivasan, Praveen, Blackwell, Sam, Alcicek, Cagdas, Fearon, Rory, Maria, Alessandro De, Panneershelvam, Vedavyas, Suleyman, Mustafa, Beattie, Cha


agdas, Fearon, Rory, Maria, Alessandro De, Panneershelvam, Vedavyas, Suleyman, Mustafa, Beattie, Charles, Petersen, Stig, Legg, Shane, Mnih, Volodymyr, Kavukcuoglu, Koray, and Silver, David. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296, 2015. Narasimhan, Karthik, Kulkarni, Tejas, and Barzilay, Regina. Language understanding for textbased games using deep reinforcement learning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015. ´Olafsd´ottir, H Freyja, Barry, Caswell, Saleem, Aman B, Hassabis, Demis, and Spiers, Hugo J. Hippocampal place cells construct reward related sequences through unexplored space. Elife, 4: e06063, 2015. Riedmiller, Martin. Rprop-description and implementation details. 1994.


ce. Elife, 4: e06063, 2015. Riedmiller, Martin. Rprop-description and implementation details. 1994. Rosin, Christopher D and Belew, Richard K. New methods for competitive coevolution. Evolutionary Computation, 5(1):1–29, 1997. Schaul, Tom, Zhang, Sixin, and Lecun, Yann. No more pesky learning rates. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 343–351, 2013. Schmidhuber, J¨urgen. Curious model-building control systems. In Neural Networks, 1991. 1991 IEEE International Joint Conference on, pp. 1458–1463. IEEE, 1991. Singer, Annabelle C and Frank, Loren M. Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron, 64(6):910–921, 2009. Stadie, Bradly C, Levine, Sergey, and Abbeel, Pieter.


the hippocampus. Neuron, 64(6):910–921, 2009. Stadie, Bradly C, Levine, Sergey, and Abbeel, Pieter. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814, 2015. Sun, Yi, Ring, Mark, Schmidhuber, J¨urgen, and Gomez, Faustino J. Incremental basis construction from temporal difference error. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 481–488, 2011. van Hasselt, Hado. Double Q-learning. In Advances in Neural Information Processing Systems, pp. 2613–2621, 2010. van Hasselt, Hado, Guez, Arthur, and Silver, David. Deep Reinforcement Learning with Double Qlearning. In Proceedings of the Thirtieth AAAI Conference on Artiﬁcial Intelligence, 2016. URL http://arxiv.org/abs/1509.06461.


the Thirtieth AAAI Conference on Artiﬁcial Intelligence, 2016. URL http://arxiv.org/abs/1509.06461. van Seijen, Harm and Sutton, Richard. Planning by prioritized sweeping with small backups. In Proceedings of The 30th International Conference on Machine Learning, pp. 361–369, 2013. Wang, Z., de Freitas, N., and Lanctot, M. Dueling network architectures for deep reinforcement learning. Technical report, 2015. URL http://arxiv.org/abs/1511.06581. Watkins, Christopher JCH and Dayan, Peter. Q-learning. Machine learning, 8(3-4):279–292, 1992. White, Adam, Modayil, Joseph, and Sutton, Richard S. Surprise and curiosity for big data robotics. In Workshops at the Twenty-Eighth AAAI Conference on Artiﬁcial Intelligence, 2014. 11


Published as a conference paper at ICLR 2016 A PRIORITIZATION VARIANTS The absolute TD-error is only one possible proxy for the ideal priority measure of expected learning progress. While it captures the scale of potential improvement, it ignores inherent stochasticity in rewards or transitions, as well as possible limitations from partial observability or FA capacity; in other words, it is problematic when there are unlearnable transitions. In that case, its derivative – which could be approximated by the difference between a transition’s current |δ| and the |δ| when it was last replayed3 – may be more useful. This measure is less immediately available, however, and is inﬂuenced by whatever was replayed in the meanwhile, which increases its variance.


however, and is inﬂuenced by whatever was replayed in the meanwhile, which increases its variance. In preliminary experiments, we found it did not outperform |δ|, but this may say more about the class of (near-deterministic) environments we investigated, than about the measure itself. An orthogonal variant is to consider the norm of the weight-change induced by replaying a transition – this can be effective if the underlying optimizer employs adaptive step-sizes that reduce gradients in high-noise directions (Schaul et al., 2013; Kingma & Ba, 2014), thus placing the burden of distinguishing between learnable and unlearnable transitions on the optimizer. It is possible to modulate prioritization by not treating positive TD-errors the same than negative ones; we can for example invoke the A


tion by not treating positive TD-errors the same than negative ones; we can for example invoke the Anna Karenina principle (Diamond, 1994), interpreted to mean that there are many ways in which a transition can be less good than expected, but only one in which can be better, to introduce an asymmetry and prioritize positive TD-errors more than negative ones of equal magnitude, because the former are more likely to be informative. Such an asymmetry in replay frequency was also observed in rat studies (Singer & Frank, 2009). Again, our preliminary experiments with such variants were inconclusive. The evidence from neuroscience suggest that a prioritization based on episodic return rather than expected learning progress may be useful too Atherton et al. (2015); ´Olafsd´ottir et al.


ther than expected learning progress may be useful too Atherton et al. (2015); ´Olafsd´ottir et al. (2015); Foster & Wilson (2006). For this case, we could boost the replay probabilities of entire episodes, instead of transitions, or boost individual transitions by their observed return-to-go (or even their value estimates). For the issue of preserving sufﬁcient diversity (to prevent overﬁtting, premature convergence or impoverished representations), there are alternative solutions to our choice of introducing stochasticity, for example, the priorities could be modulated by a novelty measure in observation space. Nothing prevents a hybrid approach where some fraction of the elements (of each minibatch) are sampled according to one priority measure and the rest according to another one, int


minibatch) are sampled according to one priority measure and the rest according to another one, introducing additional diversity. An orthogonal idea is to increase priorities of transitions that have not been replayed for a while, by introducing an explicit staleness bonus that guarantees that every transition is revisited from time to time, with that chance increasing at the same rate as its last-seen TD-error becomes stale. In the simple case where this bonus grows linearly with time, this can be implemented at no additional cost by subtracting a quantity proportional to the global step-count from the new priority on any update.4 In the particular case of RL with bootstrapping from value functions, it is possible to exploit the sequential structure of the replay memory using the followi


functions, it is possible to exploit the sequential structure of the replay memory using the following intuition: a transition that led to a large amount of learning (about its outgoing state) has the potential to change the bootstrapping target for all transitions leading into that state, and thus there is more to be learned about these. Of course we know at least one of these, namely the historic predecessor transition, and so boosting its priority makes it more likely to be revisited soon. Similarly to eligibility traces, this lets information trickle backward from a future outcome to the value estimates of the actions and states that led to it. In practice, we add |δ| of the current transition to predecessor transition’s priority, but only if the predecessor transition is not a termina


sition to predecessor transition’s priority, but only if the predecessor transition is not a terminal one. This idea is related to ‘reverse replay’ observed in rodents Foster & Wilson (2006), and to a recent extension of prioritized sweeping (van Seijen & Sutton, 2013). 3Of course, more robust approximations would be a function of the history of all encountered δ values. In particular, one could imagine an RProp-style update (Riedmiller, 1994) to priorities that increase the priority while the signs match, and reduce it whenever consecutive errors (for the same transition) differ in sign. 4 If bootstrapping is used with policy iteration, such that the target values come from separate network (as is the case for DQN), then there is a large increase in uncertainty about the priorities when t


(as is the case for DQN), then there is a large increase in uncertainty about the priorities when the target network is updated in the outer iteration. At these points, the staleness bonus is increased in proportion to the number of individual (low-level) updates that happened in-between. 12


Published as a conference paper at ICLR 2016 B EXPERIMENTAL DETAILS B.1 BLIND CLIFFWALK For the Blind Cliffwalk experiments (Section 3.1 and following), we use a straight-forward Qlearning (Watkins & Dayan, 1992) setup. The Q-values are represented using either a tabular look-up table, or a linear function approximator, in both cases represented Q(s, a) := θ⊤φ(s, a). For each transition, we compute its TD-error using: δt := Rt + γt max a Q(St, a) −Q(St−1, At−1) (2) and update the parameters using stochastic gradient ascent: θ ← θ + η · δt · ∇θQ St−1,At−1 = θ + η · δt · φ(St−1, At−1) (3) For the linear FA case we use a very simple encoding of state as a 1-hot vector (as for tabular), but concatenated with a constant bias feature of value 1.


state as a 1-hot vector (as for tabular), but concatenated with a constant bias feature of value 1. To make generalization across actions impossible, we alternate which action is ‘right’ and which one is ‘wrong’ for each state. All elements are initialized with small values near zero, θi ∼N(0, 0.1). We vary the size of the problem (number of states n) from 2 to 16. The discount factor is set to γ = 1 −1 n which keeps values on approximately the same scale independently of n. This allows us to used a ﬁxed step-size of η = 1 4 in all experiments. The replay memory is ﬁlled by exhaustively executing all 2n possible sequences of actions until termination (in random order). This guarantees that exactly one sequence will succeed and hit the ﬁnal reward, and all others will fail with zero reward.


xactly one sequence will succeed and hit the ﬁnal reward, and all others will fail with zero reward. The replay memory contains all the relevant experience (the total number of transitions is 2n+1 −2), at the frequency that it would be encountered when acting online with a random behavior policy. Given this, we can in principle learn until convergence by just increasing the amount of computation; here, convergence is deﬁned as a mean-squared error (MSE) between the Q-value estimates and the ground truth below 10−3. B.2 ATARI EXPERIMENTS B.2.1 IMPLEMENTATION DETAILS Prioritizing using a replay memory with N = 106 transitions introduced some performance challenges. Here we describe a number of things we did to minimize additional run-time and memory overhead, extending the discussion in Sect


things we did to minimize additional run-time and memory overhead, extending the discussion in Section 3. Rank-based prioritization Early experiments with Atari showed that maintaining a sorted data structure of 106 transitions with constantly changing TD-errors dominated running time. Our ﬁnal solution was to store transitions in a priority queue implemented with an array-based binary heap. The heap array was then directly used as an approximation of a sorted array, which is infrequently sorted once every 106 steps to prevent the heap becoming too unbalanced. This is an unconventional use of a binary heap, however our tests on smaller environments showed learning was unaffected compared to using a perfectly sorted array.


on smaller environments showed learning was unaffected compared to using a perfectly sorted array. This is likely due to the last-seen TD-error only being a proxy for the usefulness of a transition and our use of stochastic prioritized sampling. A small improvement in running time came from avoiding excessive recalculation of partitions for the sampling distribution. We reused the same partition for values of N that are close together and by updating α and β infrequently. Our ﬁnal implementation for rank-based prioritization produced an additional 2%-4% increase in running time and negligible additional memory usage. This could be reduced further in a number of ways, e.g. with a more efﬁcient heap implementation, but it was good enough for our experiments.


of ways, e.g. with a more efﬁcient heap implementation, but it was good enough for our experiments. Proportional prioritization The ‘sum-tree’ data structure used here is very similar in spirit to the array representation of a binary heap. However, instead of the usual heap property, the value of a parent node is the sum of its children. Leaf nodes store the transition priorities and the internal nodes are intermediate sums, with the parent node containing the sum over all priorities, ptotal. This provides a efﬁcient way of calculating the cumulative sum of priorities, allowing O(log N) updates and sampling. To sample a minibatch of size k, the range [0, ptotal] is divided equally into k ranges. 13


Published as a conference paper at ICLR 2016 Next, a value is uniformly sampled from each range. Finally the transitions that correspond to each of these sampled values are retrieved from the tree. Overhead is similar to rank-based prioritization. As mentioned in Section 3.4, whenever importance sampling is used, all weights wi were scaled so that maxi wi = 1. We found that this worked better in practice as it kept all weights within a reasonable range, avoiding the possibility of extremely large updates. It is worth mentioning that this normalization interacts with annealing on β: as β approaches 1, the normalization constant grows, which reduces the effective average update in a similar way to annealing the step-size η.


nt grows, which reduces the effective average update in a similar way to annealing the step-size η. B.2.2 HYPERPARAMETERS Throughout this paper our baseline was DQN and the tuned version of Double DQN. We tuned hyperparameters over a subset of Atari games: Breakout, Pong, Ms. Pac-Man, Q*bert, Alien, Battlezone, Asterix. Table 2 lists the values tried and Table 3 lists the chosen parameters. Hyperparameter Range of values α 0, 0.4, 0.5, 0.6, 0.7, 0.8 β 0, 0.4, 0.5, 0.6, 1 η ηbaseline, ηbaseline/2, ηbaseline/4, ηbaseline/8 Table 2: Hyperparameters considered in experiments. Here ηbaseline = 0.00025. DQN Double DQN (tuned) Hyperparameter baseline rank-based baseline rank-based proportional α (priority) 0 0.5 →0 0 0.7 0.6 β (IS) 0 0 0 0.5 →1 0.4 →1 η (step-size) 0.00025 ηbaseline/4 0.00025 ηba


riority) 0 0.5 →0 0 0.7 0.6 β (IS) 0 0 0 0.5 →1 0.4 →1 η (step-size) 0.00025 ηbaseline/4 0.00025 ηbaseline/4 ηbaseline/4 Table 3: Chosen hyperparameters for prioritized variants of DQN. Arrows indicate linear annealing, where the limiting value is reached at the end of training. Note the rank-based variant with DQN as the baseline is an early version without IS. Here, the bias introduced by prioritized replay was instead corrected by annealing α to zero. B.2.3 EVALUATION We evaluate our agents using the human starts evaluation method described in (van Hasselt et al., 2016). Human starts evaluation uses start states sampled randomly from human traces. The test evaluation that agents periodically undergo during training uses start states that are randomized by doing a random number of no-ops


lly undergo during training uses start states that are randomized by doing a random number of no-ops at the beginning of each episode. Human starts evaluation averages the score over 100 evaluations of 30 minutes of game time. All learning curve plots show scores under the test evaluation and were generated using the same code base, with the same random seed initializations. Table 4 and Table 5 show evaluation method differences and the ϵ used in the ϵ-greedy policy for each agent during evaluation. The agent evaluated with the human starts evaluation is the best agent found during training as in (van Hasselt et al., 2016). Evaluation method Frames Emulator time Number of evaluations Agent start point Human starts 108,000 30 mins 100 human starts Test 500,000 139 mins 1 up to 30 random no-


tart point Human starts 108,000 30 mins 100 human starts Test 500,000 139 mins 1 up to 30 random no-ops Table 4: Evaluation method comparison. Normalized score is calculated as in (van Hasselt et al., 2016): scorenormalized = scoreagent −scorerandom |scorehuman −scorerandom| (4) Note the absolute value of the denominator is taken. This only affects Video Pinball where the random score is higher than the human score. Combined with a high agent score, Video Pinball has 14


Published as a conference paper at ICLR 2016 DQN Double DQN (tuned) Evaluation method baseline rank-based baseline rank-based proportional Human starts 0.05 0.01 0.001 0.001 0.001 Test 0.05 0.05 0.01 0.01 0.01 Table 5: The ϵ used in the ϵ-greedy policy for each agent, for each evaluation method. a large effect on the mean normalized score. We continue to use this so our normalized scores are comparable. B.3 CLASS-IMBALANCED MNIST B.3.1 DATASET SETUP In our supervised learning setting we modiﬁed MNIST to obtain a new training dataset with a significant label imbalance. This new dataset was obtained by considering a small subset of the samples that correspond to the ﬁrst 5 digits (0, 1, 2, 3, 4) and all of the samples that correspond to the remaining 5 labels (5, 6, 7, 8, 9).


s (0, 1, 2, 3, 4) and all of the samples that correspond to the remaining 5 labels (5, 6, 7, 8, 9). For each of the ﬁrst 5 digits we randomly sampled 1% of the available examples, i.e., 1% of the available 0s, 1% of the available 1s etc. In the resulting dataset there are examples of all 10 different classes but it is highly imbalanced since there are 100 times more examples that correspond to the 5, 6, 7, 8, 9 classes than to the 0, 1, 2, 3, 4 ones. In all our experiments we used the original MNIST test dataset without removing any samples. Figure 6: The architecture of the feed-forward network used in the Prioritized Supervised Learning experiments. B.3.2 TRAINING SETUP In our experiments we used a 4 layer feed-forward neural network with an architecture similar to that of LeNet5 (Lecun


we used a 4 layer feed-forward neural network with an architecture similar to that of LeNet5 (Lecun et al., 1998). This is a 2 layer convolutional neural network followed by 2 fully connected layers at the top. Each convolutional layer is comprised of a pure convolution followed by a rectiﬁer non-linearity and a subsampling max pooling operation. The two fully-connected layers in the network are also separated by a rectiﬁer non-linearity. The last layer is a softmax which is used to obtain a normalized distribution over the possible labels. The complete architecture is shown in Figure 6, and is implemented using Torch7 (Collobert et al., 2011). The model was trained using stochastic gradient descent with no momentum and a minibatch of size 60.


he model was trained using stochastic gradient descent with no momentum and a minibatch of size 60. In all our experiments we considered 6 different step-sizes (0.3, 0.1, 0.03, 0.01, 0.003 and 0.001) and for each case presented in this work, we selected the step-size that lead to the best (balanced) validation performance. We used the negative log-likelihood loss criterion and we ran experiments with both the weighted and unweighted version of the loss. In the weighted case the loss of the examples that correspond to the ﬁrst 5 digits (0, 1, 2, 3, 4) was scaled by a factor of a 100 to accommodate the label imbalance in the training set described above. 15


Published as a conference paper at ICLR 2016 0 1500 3000 4500 6000 Alien 0 400 800 1200 1600 2000 Amidar 0 1000 2000 3000 4000 Assault 0 6000 12000 18000 24000 30000 Asterix 400 800 1200 1600 2000 Asteroids 0 150000 300000 450000 600000 Atlantis 0 250 500 750 1000 Bank Heist 0 8000 16000 24000 32000 Battlezone 0 5000 10000 15000 20000 25000 Beam Rider 0 300 600 900 1200 Berzerk 0 15 30 45 60 Bowling 100 50 0 50 100 Boxing 0 80 160 240 320 400 Breakout 1000 2000 3000 4000 Centipede 0 1500 3000 4500 6000 Chopper Command 0 30000 60000 90000 120000 Crazy Climber 0 5000 10000 15000 20000 25000 Defender 0 15000 30000 45000 60000 Demon Attack 20 10 0 10 Double Dunk 0 500 1000 1500 2000 2500 Enduro 80 40 0 40 Fishing Derby 0 8 16 24 32 Freeway 0 800 1600 2400 3200 4000 Frostbite 0 5000 10000 15000


80 40 0 40 Fishing Derby 0 8 16 24 32 Freeway 0 800 1600 2400 3200 4000 Frostbite 0 5000 10000 15000 20000 25000 Gopher 0 100 200 300 400 Gravitar 0 5000 10000 15000 20000 25000 H.E.R.O. 24 16 8 0 Ice Hockey 0 800 1600 2400 3200 4000 James Bond 007 0 4000 8000 12000 16000 Kangaroo 0 2000 4000 6000 8000 10000 Krull 0 8000 16000 24000 32000 Kung-Fu Master 0.0 0.2 0.4 0.6 0.8 1.0 Montezuma's Revenge 0 800 1600 2400 3200 4000 Ms. Pac-Man 0 3000 6000 9000 12000 Name This Game 0 5000 10000 15000 20000 25000 Phoenix 1000 750 500 250 0 Pitfall! 30 15 0 15 30 Pong 600 400 200 0 200 Private Eye 0 4000 8000 12000 16000 Q*bert 0 4000 8000 12000 16000 20000 River Raid 0 15000 30000 45000 60000 Road Runner 0 15 30 45 60 Robotank 0 6000 12000 18000 24000 30000 Seaquest 30000 25000 20000 15000 10000 Skiin


r 0 15 30 45 60 Robotank 0 6000 12000 18000 24000 30000 Seaquest 30000 25000 20000 15000 10000 Skiing 0 600 1200 1800 2400 3000 Solaris 0 1000 2000 3000 4000 Space Invaders 0 15000 30000 45000 60000 Stargunner 8 4 0 4 Surround 25 20 15 10 5 0 Tennis 0 2000 4000 6000 8000 10000 Time Pilot 0 40 80 120 160 Tutankham 0 5000 10000 15000 20000 25000 Up'n Down 0 50 100 150 200 250 Venture 0 80000 160000 240000 320000 400000 Video Pinball 0 2000 4000 6000 8000 10000 Wizard of Wor 0 3000 6000 9000 12000 Yars' Revenge 0 2500 5000 7500 10000 Zaxxon uniform rank-based proportional uniform DQN Figure 7: Learning curves (in raw score) for Double DQN (uniform baseline, in black), with rank-based prioritized replay (red), proportional prioritization (blue), for all 57 games of the Atari benchmark suite.


ed replay (red), proportional prioritization (blue), for all 57 games of the Atari benchmark suite. Each curve corresponds to a single training run over 200 million unique frames, using test evaluation (see Section B.2.3), with a moving average smoothed over 10 points. Learning curves for the original DQN are in gray. See Figure 8 for a more detailed view on a subset of these. 16


Published as a conference paper at ICLR 2016 0 1000 2000 3000 4000 5000 6000 7000 8000 reward Alien 0 5000 10000 15000 20000 25000 30000 35000 40000 reward Battlezone 0 20 40 60 80 100 120 140 160 180 200 training step (1e6) 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 reward Asterix 0 20 40 60 80 100 120 140 160 180 200 training step (1e6) 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 reward Q*bert human uniform rank-based proportional Figure 8: Detailed learning curves for rank-based (red) and proportional (blue) prioritization, as compared to the uniform Double DQN baseline (black) on a selection of games. The solid lines are the median scores, and the shaded area denotes the interquartile range across 8 random initializations. The dashed green lines are human scores.


s the interquartile range across 8 random initializations. The dashed green lines are human scores. While the variability between runs is substantial, there are signiﬁcant differences in ﬁnal achieved score, and also in learning speed. -100% 0% 100% 200% normalized scoreagent normalized scoreDQN Robotank Assault Time Pilot Centipede Amidar Montezuma's Revenge Ice Hockey Tennis Ms. Pac-Man Asteroids Private Eye Frostbite Gravitar Venture Fishing Derby Pong H.E.R.O. Alien Freeway Bowling Battlezone Space Invaders Seaquest Beam Rider Kung-Fu Master Chopper Command River Raid Tutankham Zaxxon Name This Game Wizard of Wor Q*bert Enduro Bank Heist Asterix Up'n Down Demon Attack James Bond 007 Kangaroo Crazy Climber Stargunner Boxing Breakout Gopher Road Runner Krull Atlantis Double Dunk Video Pi


aroo Crazy Climber Stargunner Boxing Breakout Gopher Road Runner Krull Atlantis Double Dunk Video Pinball Figure 9: Difference in normalized score (the gap between random and human is 100%) on 49 games with human starts, comparing DQN with and without rank-based prioritized replay, showing substantial improvements in many games. Exact scores are in Table 6. See also Figure 3 where Double DQN is the baseline. 17


Published as a conference paper at ICLR 2016 2 6 21 65 200 training step 0.0 0.2 0.4 0.6 0.8 1.0 rank-based prioritized replay error Alien Asterix Battlezone Q*bert 0.0 0.2 0.4 0.6 0.8 1.0 transition (1e6) 0.0 0.2 0.4 0.6 0.8 1.0 uniform replay error 0.0 0.2 0.4 0.6 0.8 1.0 transition (1e6) 0.0 0.2 0.4 0.6 0.8 1.0 transition (1e6) 0.0 0.2 0.4 0.6 0.8 1.0 transition (1e6) Figure 10: Visualization of the last-seen absolute TD-error for all transitions in the replay memory, sorted, for a selection of Atari games. The lines are color-coded by the time during learning, at a resolution of 106 frames, with the coldest colors in the beginning and the warmest toward the end of training. We observe that in some games it starts quite peaked but quickly becomes spread out, following approximately a he


at in some games it starts quite peaked but quickly becomes spread out, following approximately a heavy-tailed distribution. This phenomenon happens for both rank-based prioritized replay (top) and uniform replay (bottom) but is faster for prioritized replay. 0.00 0.25 0.50 0.75 1.00 transition error 0.0 0.5 1.0 1.5 2.0 2.5 sample probability Alien 0.00 0.25 0.50 0.75 1.00 transition error Asterix 0.00 0.25 0.50 0.75 1.00 transition error Battlezone 0.00 0.25 0.50 0.75 1.00 transition error Q*bert Figure 11: Effective replay probability, as a function of absolute TD-error, for the rank-based prioritized replay variant near the start of training. This shows the effect of Equation 1 with α = 0.7 in practice, compared to the uniform baseline (dashed horizontal line).


of Equation 1 with α = 0.7 in practice, compared to the uniform baseline (dashed horizontal line). The effect is irregular, but qualitatively similar for a selection of games. 18


Published as a conference paper at ICLR 2016 0 1000 2000 3000 4000 5000 6000 7000 reward Alien 0 5000 10000 15000 20000 25000 30000 35000 40000 reward Battlezone 0 20 40 60 80 100 120 140 160 180 200 training step (1e6) 0 2000 4000 6000 8000 10000 12000 14000 16000 reward Asterix 0 20 40 60 80 100 120 140 160 180 200 training step (1e6) 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 reward Q*bert uniform rank-based, no IS rank-based, full IS Figure 12: Effect of importance sampling: These learning curves (as in Figure 8) show how rank-based prioritization is affected by full importance-sampling correction (i.e., β = 1, in orange), as compared to the uniform baseline (black, α = 0) and pure, uncorrected prioritized replay (violet, β = 0), on a few selected games.


e (black, α = 0) and pure, uncorrected prioritized replay (violet, β = 0), on a few selected games. The shaded area corresponds to the interquartile range. The step-size for full IS correction is the same as for uniform replay. For uncorrected prioritized replay, the step-size is reduced by a factor of 4. Compared to uncorrected prioritized replay, importance sampling makes learning less aggressive, leading on the one hand to slower initial learning, but on the other hand to a smaller risk of premature convergence and sometimes better ultimate results. Compared to uniform replay, fully corrected prioritization is on average better. 19


Published as a conference paper at ICLR 2016 DQN Double DQN (tuned) Game baseline rank-based baseline rank-based proportional Alien 7% 17% 14% 19% 12% Amidar 8% 6% 10% 8% 14% Assault 685% 631% 1276% 1381% 1641% Asterix -1% 123% 226% 303% 431% Asteroids -0% 2% 1% 2% 2% Atlantis 478% 1480% 2335% 2419% 4425% Bank Heist 25% 129% 139% 137% 128% Battlezone 48% 63% 72% 75% 87% Beam Rider 57% 80% 117% 210% 176% Berzerk 22% 40% 33% 47% Bowling 5% 20% 31% 15% 27% Boxing 246% 641% 676% 665% 632% Breakout 1149% 1823% 1397% 1298% 1407% Centipede 22% 12% 23% 19% 18% Chopper Command 29% 73% 34% 48% 72% Crazy Climber 179% 429% 448% 507% 522% Defender 151% 207% 176% 155% Demon Attack 390% 596% 2152% 1888% 2256% Double Dunk -350% 669% 981% 2000% 1169% Enduro 68% 164% 158% 233% 239% Fishing Derby 91% 98% 98%


56% Double Dunk -350% 669% 981% 2000% 1169% Enduro 68% 164% 158% 233% 239% Fishing Derby 91% 98% 98% 106% 105% Freeway 101% 111% 113% 113% 109% Frostbite 2% 5% 33% 83% 69% Gopher 120% 836% 728% 1679% 2792% Gravitar -1% 4% -2% 1% -1% H.E.R.O. 47% 56% 55% 80% 78% Ice Hockey 58% 58% 71% 93% 85% James Bond 007 94% 311% 161% 1172% 1038% Kangaroo 98% 339% 421% 458% 384% Krull 283% 1051% 590% 598% 653% Kung-Fu Master 56% 97% 146% 153% 151% Montezuma’s Revenge 1% 0% 0% 1% -0% Ms. Pac-Man 4% 5% 7% 11% 11% Name This Game 73% 138% 143% 173% 200% Phoenix 270% 202% 284% 474% Pitfall! 2% 3% -1% 5% Pong 102% 110% 111% 110% 110% Private Eye -1% 2% -2% 0% -1% Q*bert 37% 106% 91% 82% 93% River Raid 25% 70% 74% 81% 128% Road Runner 136% 854% 643% 780% 850% Robotank 863% 752% 872% 828% 815% Seaquest 6% 29% 36


% 81% 128% Road Runner 136% 854% 643% 780% 850% Robotank 863% 752% 872% 828% 815% Seaquest 6% 29% 36% 63% 97% Skiing -122% 33% 44% 38% Solaris -21% -14% 3% 2% Space Invaders 99% 118% 191% 291% 693% Stargunner 378% 660% 653% 689% 580% Surround 29% 77% 103% 58% Tennis 130% 130% 93% 110% 132% Time Pilot 100% 89% 140% 113% 176% Tutankham 16% 67% 63% 35% 17% Up’n Down 28% 173% 200% 125% 313% Venture 4% 9% 0% 7% 22% Video Pinball -5% 4042% 7221% 5727% 7367% Wizard of Wor -15% 52% 144% 131% 177% Yars’ Revenge 11% 10% 7% 10% Zaxxon 4% 68% 102% 113% 113% Table 6: Normalized scores on 57 Atari games (random is 0%, human is 100%), from a single training run each, using human starts evaluation (see Section B.2.3). Baselines are from van Hasselt et al.


run each, using human starts evaluation (see Section B.2.3). Baselines are from van Hasselt et al. (2016), see Equation 4 for how normalized scores are calculated. 20


Published as a conference paper at ICLR 2016 DQN Double DQN (tuned) Game random human baseline Gorila rank-b. baseline rank-b. prop. Alien 128.3 6371.3 570.2 813.5 1191.0 1033.4 1334.7 900.5 Amidar 11.8 1540.4 133.4 189.2 98.9 169.1 129.1 218.4 Assault 166.9 628.9 3332.3 1195.8 3081.3 6060.8 6548.9 7748.5 Asterix 164.5 7536.0 124.5 3324.7 9199.5 16837.0 22484.5 31907.5 Asteroids 871.3 36517.3 697.1 933.6 1677.2 1193.2 1745.1 1654.0 Atlantis 13463.0 26575.0 76108.0 629166.5 207526.0 319688.0 330647.0 593642.0 Bank Heist 21.7 644.5 176.3 399.4 823.7 886.0 876.6 816.8 Battlezone 3560.0 33030.0 17560.0 19938.0 22250.0 24740.0 25520.0 29100.0 Beam Rider 254.6 14961.0 8672.4 3822.1 12041.9 17417.2 31181.3 26172.7 Berzerk 196.1 2237.5 644.0 1011.1 865.9 1165.6 Bowling 35.2 146.5 41.2 54.0 58.0 69


31181.3 26172.7 Berzerk 196.1 2237.5 644.0 1011.1 865.9 1165.6 Bowling 35.2 146.5 41.2 54.0 58.0 69.6 52.0 65.8 Boxing -1.5 9.6 25.8 74.2 69.6 73.5 72.3 68.6 Breakout 1.6 27.9 303.9 313.0 481.1 368.9 343.0 371.6 Centipede 1925.5 10321.9 3773.1 6296.9 2959.4 3853.5 3489.1 3421.9 Chopper Command 644.0 8930.0 3046.0 3191.8 6685.0 3495.0 4635.0 6604.0 Crazy Climber 9337.0 32667.0 50992.0 65451.0 109337.0 113782.0 127512.0 131086.0 Defender 1965.5 14296.0 20634.0 27510.0 23666.5 21093.5 Demon Attack 208.3 3442.8 12835.2 14880.1 19478.8 69803.4 61277.5 73185.8 Double Dunk -16.0 -14.4 -21.6 -11.3 -5.3 -0.3 16.0 2.7 Enduro -81.8 740.2 475.6 71.0 1265.6 1216.6 1831.0 1884.4 Fishing Derby -77.1 5.1 -2.3 4.6 3.5 3.2 9.8 9.2 Freeway 0.1 25.6 25.8 10.2 28.4 28.8 28.9 27.9 Frostbite 66.4 4202.8 157.4 4


2.3 4.6 3.5 3.2 9.8 9.2 Freeway 0.1 25.6 25.8 10.2 28.4 28.8 28.9 27.9 Frostbite 66.4 4202.8 157.4 426.6 288.7 1448.1 3510.0 2930.2 Gopher 250.0 2311.0 2731.8 4373.0 17478.2 15253.0 34858.8 57783.8 Gravitar 245.5 3116.0 216.5 538.4 351.0 200.5 269.5 218.0 H.E.R.O. 1580.3 25839.4 12952.5 8963.4 15150.9 14892.5 20889.9 20506.4 Ice Hockey -9.7 0.5 -3.8 -1.7 -3.8 -2.5 -0.2 -1.0 James Bond 007 33.5 368.5 348.5 444.0 1074.5 573.0 3961.0 3511.5 Kangaroo 100.0 2739.0 2696.0 1431.0 9053.0 11204.0 12185.0 10241.0 Krull 1151.9 2109.1 3864.0 6363.1 11209.5 6796.1 6872.8 7406.5 Kung-Fu Master 304.0 20786.8 11875.0 20620.0 20181.0 30207.0 31676.0 31244.0 Montezuma’s Revenge 25.0 4182.0 50.0 84.0 44.0 42.0 51.0 13.0 Ms.


0 20181.0 30207.0 31676.0 31244.0 Montezuma’s Revenge 25.0 4182.0 50.0 84.0 44.0 42.0 51.0 13.0 Ms. Pac-Man 197.8 15375.0 763.5 1263.0 964.7 1241.3 1865.9 1824.6 Name This Game 1747.8 6796.0 5439.9 9238.5 8738.5 8960.3 10497.6 11836.1 Phoenix 1134.4 6686.2 16107.8 12366.5 16903.6 27430.1 Pitfall! -348.8 5998.9 -193.7 -186.7 -427.0 -14.8 Pong -18.0 15.5 16.2 16.7 18.7 19.1 18.9 18.9 Private Eye 662.8 64169.1 298.2 2598.6 2202.3 -575.5 670.7 179.0 Q*bert 183.0 12085.0 4589.8 7089.8 12740.5 11020.8 9944.0 11277.0 River Raid 588.3 14382.2 4065.3 5310.3 10205.5 10838.4 11807.2 18184.4 Road Runner 200.0 6878.0 9264.0 43079.8 57207.0 43156.0 52264.0 56990.0 Robotank 2.4 8.9 58.5 61.8 51.3 59.1 56.2 55.4 Seaquest 215.5 40425.8 2793.9 10145.9 11848.8 14498.0 25463.7 39096.7 Skiing -15287.4 -3686.6


55.4 Seaquest 215.5 40425.8 2793.9 10145.9 11848.8 14498.0 25463.7 39096.7 Skiing -15287.4 -3686.6 -29404.3 -11490.4 -10169.1 -10852.8 Solaris 2047.2 11032.6 134.6 810.0 2272.8 2238.2 Space Invaders 182.6 1464.9 1449.7 1183.3 1696.9 2628.7 3912.1 9063.0 Stargunner 697.0 9528.0 34081.0 14919.2 58946.0 58365.0 61582.0 51959.0 Surround -9.7 5.4 -5.3 1.9 5.9 -0.9 Tennis -21.4 -6.7 -2.3 -0.7 -2.3 -7.8 -5.3 -2.0 Time Pilot 3273.0 5650.0 5640.0 8267.8 5391.0 6608.0 5963.0 7448.0 Tutankham 12.7 138.3 32.4 118.5 96.5 92.2 56.9 33.6 Up’n Down 707.2 9896.1 3311.3 8747.7 16626.5 19086.9 12157.4 29443.7 Venture 18.0 1039.0 54.0 523.4 110.0 21.0 94.0 244.0 Video Pinball 20452.0 15641.1 20228.1 112093.4 214925.3 367823.7 295972.8 374886.9 Wizard of Wor 804.0 4556.0 246.0 10431.0 2755.0 6201.0 5727.0 745


14925.3 367823.7 295972.8 374886.9 Wizard of Wor 804.0 4556.0 246.0 10431.0 2755.0 6201.0 5727.0 7451.0 Yars’ Revenge 1476.9 47135.2 6626.7 6270.6 4687.4 5965.1 Zaxxon 475.0 8443.0 831.0 6159.4 5901.0 8593.0 9474.0 9501.0 Table 7: Raw scores obtained on the original 49 Atari games plus 8 additional games where available, evaluated on human starts. Human, random, DQN and tuned Double DQN scores are from van Hasselt et al. (2016). Note that the Gorila results from Nair et al. (2015) used much more data and computation, but the other methods are more directly comparable to each other in this respect. 21


Dueling Network Architectures for Deep Reinforcement Learning Ziyu Wang ZIYU@GOOGLE.COM Tom Schaul SCHAUL@GOOGLE.COM Matteo Hessel MTTHSS@GOOGLE.COM Hado van Hasselt HADO@GOOGLE.COM Marc Lanctot LANCTOT@GOOGLE.COM Nando de Freitas NANDODEFREITAS@GMAIL.COM Google DeepMind, London, UK Abstract In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function.


mators: one for the state value function and one for the state-dependent action advantage function. The main beneﬁt of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain. 1. Introduction Over the past years, deep learning has contributed to dramatic advances in scalability and performance of machine learning (LeCun et al., 2015). One exciting application is the sequential decision-making setting of reinforcement learning (RL) and control.


g application is the sequential decision-making setting of reinforcement learning (RL) and control. Notable examples include deep Q-learning (Mnih et al., 2015), deep visuomotor policies (Levine et al., 2015), attention with recurrent networks (Ba et al., 2015), and model predictive control with embeddings (Watter et al., 2015). Other recent successes include massively parallel frameworks (Nair et al., 2015) and expert move prediction in the game of Go (Maddison et al., 2015), which produced policies matching those of Monte Carlo tree search programs, and squarely beaten a professional player when combined with search (Silver et al., 2016). In spite of this, most of the approaches for RL use standard neural networks, such as convolutional networks, MLPs, LSTMs and autoencoders.


for RL use standard neural networks, such as convolutional networks, MLPs, LSTMs and autoencoders. The focus in these recent advances has been on designing improved control and RL algorithms, or simply on incorporating existing neural network architectures into RL methods. Here, we take an alternative but complementary approach of focusing primarily on innovating a neural network architecture that is better suited for model-free RL. This approach has the beneﬁt that the new network can be easily combined with existing and future algorithms for RL. That is, this paper advances a new network (Figure 1), but uses already published algorithms. The proposed network architecture, which we name the dueling architecture, explicitly separates the representation of state values and (state-dependent


e dueling architecture, explicitly separates the representation of state values and (state-dependent) action advantages. The dueling architecture consists of two streams that represent the value and advantage functions, while sharing a common Figure 1. A popular single stream Q-network (top) and the dueling Q-network (bottom). The dueling network has two streams to separately estimate (scalar) state-value and the advantages for each action; the green output module implements equation (9) to combine them. Both networks output Q-values for each action. arXiv:1511.06581v3 [cs.LG] 5 Apr 2016


Dueling Network Architectures for Deep Reinforcement Learning convolutional feature learning module. The two streams are combined via a special aggregating layer to produce an estimate of the state-action value function Q as shown in Figure 1. This dueling network should be understood as a single Q network with two streams that replaces the popular single-stream Q network in existing algorithms such as Deep Q-Networks (DQN; Mnih et al., 2015). The dueling network automatically produces separate estimates of the state value function and advantage function, without any extra supervision. Intuitively, the dueling architecture can learn which states are (or are not) valuable, without having to learn the effect of each action for each state.


states are (or are not) valuable, without having to learn the effect of each action for each state. This is particularly useful in states where its actions do not affect the environment in any relevant way. To illustrate this, consider the saliency maps shown in Figure 21. These maps were generated by computing the Jacobians of the trained value and advantage streams with respect to the input video, following the method proposed by Simonyan et al. (2013). (The experimental section describes this methodology in more detail.) The ﬁgure shows the value and advantage saliency maps for two different time steps. In one time step (leftmost pair of images), we see that the value network stream pays attention to the road and in particular to the horizon, where new cars appear.


network stream pays attention to the road and in particular to the horizon, where new cars appear. It also pays attention to the score. The advantage stream on the other hand does not pay much attention to the visual input because its action choice is practically irrelevant when there are no cars in front. However, in the second time step (rightmost pair of images) the advantage stream pays attention as there is a car immediately in front, making its choice of action very relevant. In the experiments, we demonstrate that the dueling architecture can more quickly identify the correct action during policy evaluation as redundant or similar actions are added to the learning problem. We also evaluate the gains brought in by the dueling architecture on the challenging Atari 2600 testbed.


so evaluate the gains brought in by the dueling architecture on the challenging Atari 2600 testbed. Here, an RL agent with the same structure and hyper-parameters must be able to play 57 different games by observing image pixels and game scores only. The results illustrate vast improvements over the single-stream baselines of Mnih et al. (2015) and van Hasselt et al. (2015). The combination of prioritized replay (Schaul et al., 2016) with the proposed dueling network results in the new state-of-the-art for this popular domain. 1.1. Related Work The notion of maintaining separate value and advantage functions goes back to Baird (1993). In Baird’s original 1https://www.youtube.com/playlist?list= PLVFXyCSfS2Pau0gBh0mwTxDmutywWyFBP VALUE ADVANTAGE VALUE ADVANTAGE Figure 2.


ube.com/playlist?list= PLVFXyCSfS2Pau0gBh0mwTxDmutywWyFBP VALUE ADVANTAGE VALUE ADVANTAGE Figure 2. See, attend and drive: Value and advantage saliency maps (red-tinted overlay) on the Atari game Enduro, for a trained dueling architecture. The value stream learns to pay attention to the road. The advantage stream learns to pay attention only when there are cars immediately in front, so as to avoid collisions. advantage updating algorithm, the shared Bellman residual update equation is decomposed into two updates: one for a state value function, and one for its associated advantage function. Advantage updating was shown to converge faster than Q-learning in simple continuous time domains in (Harmon et al., 1995).


hown to converge faster than Q-learning in simple continuous time domains in (Harmon et al., 1995). Its successor, the advantage learning algorithm, represents only a single advantage function (Harmon & Baird, 1996). The dueling architecture represents both the value V (s) and advantage A(s, a) functions with a single deep model whose output combines the two to produce a state-action value Q(s, a). Unlike in advantage updating, the representation and algorithm are decoupled by construction. Consequently, the dueling architecture can be used in combination with a myriad of model free RL algorithms. There is a long history of advantage functions in policy gradients, starting with (Sutton et al., 2000). As a recent example of this line of work, Schulman et al.


nts, starting with (Sutton et al., 2000). As a recent example of this line of work, Schulman et al. (2015) estimate advantage values online to reduce the variance of policy gradient algorithms. There have been several attempts at playing Atari with deep reinforcement learning, including Mnih et al. (2015); Guo et al. (2014); Stadie et al. (2015); Nair et al. (2015); van Hasselt et al. (2015); Bellemare et al. (2016) and Schaul


Dueling Network Architectures for Deep Reinforcement Learning et al. (2016). The results of Schaul et al. (2016) are the current published state-of-the-art. 2. Background We consider a sequential decision making setup, in which an agent interacts with an environment E over discrete time steps, see Sutton & Barto (1998) for an introduction. In the Atari domain, for example, the agent perceives a video st consisting of M image frames: st = (xt−M+1, . . . , xt) ∈ S at time step t. The agent then chooses an action from a discrete set at ∈A = {1, . . . , |A|} and observes a reward signal rt produced by the game emulator. The agent seeks maximize the expected discounted return, where we deﬁne the discounted return as Rt = P∞ τ=t γτ−trτ.


aximize the expected discounted return, where we deﬁne the discounted return as Rt = P∞ τ=t γτ−trτ. In this formulation, γ ∈[0, 1] is a discount factor that trades-off the importance of immediate and future rewards. For an agent behaving according to a stochastic policy π, the values of the state-action pair (s, a) and the state s are deﬁned as follows Qπ(s, a) = E [Rt| st = s, at = a, π] , and V π(s) = Ea∼π(s) [Qπ(s, a)] . (1) The preceding state-action value function (Q function for short) can be computed recursively with dynamic programming: Qπ(s, a) = Es′  r + γEa′∼π(s′) [Qπ(s′, a′)] | s, a, π  . We deﬁne the optimal Q∗(s, a) = maxπ Qπ(s, a). Under the deterministic policy a = arg maxa′∈A Q∗(s, a′), it follows that V ∗(s) = maxa Q∗(s, a).


. Under the deterministic policy a = arg maxa′∈A Q∗(s, a′), it follows that V ∗(s) = maxa Q∗(s, a). From this, it also follows that the optimal Q function satisﬁes the Bellman equation: Q∗(s, a) = Es′ h r + γ max a′ Q∗(s′, a′) | s, a i . (2) We deﬁne another important quantity, the advantage function, relating the value and Q functions: Aπ(s, a) = Qπ(s, a) −V π(s). (3) Note that Ea∼π(s) [Aπ(s, a)] = 0. Intuitively, the value function V measures the how good it is to be in a particular state s. The Q function, however, measures the the value of choosing a particular action when in this state. The advantage function subtracts the value of the state from the Q function to obtain a relative measure of the importance of each action. 2.1.


f the state from the Q function to obtain a relative measure of the importance of each action. 2.1. Deep Q-networks The value functions as described in the preceding section are high dimensional objects. To approximate them, we can use a deep Q-network: Q(s, a; θ) with parameters θ. To estimate this network, we optimize the following sequence of loss functions at iteration i: Li(θi) = Es,a,r,s′  yDQN i −Q(s, a; θi) 2 , (4) with yDQN i = r + γ max a′ Q(s′, a′; θ−), (5) where θ−represents the parameters of a ﬁxed and separate target network. We could attempt to use standard Qlearning to learn the parameters of the network Q(s, a; θ) online. However, this estimator performs poorly in practice.


e parameters of the network Q(s, a; θ) online. However, this estimator performs poorly in practice. A key innovation in (Mnih et al., 2015) was to freeze the parameters of the target network Q(s′, a′; θ−) for a ﬁxed number of iterations while updating the online network Q(s, a; θi) by gradient descent. (This greatly improves the stability of the algorithm.) The speciﬁc gradient update is ∇θiLi(θi) = Es,a,r,s′ h yDQN i −Q(s, a; θi)  ∇θiQ(s, a; θi) i This approach is model free in the sense that the states and rewards are produced by the environment. It is also offpolicy because these states and rewards are obtained with a behavior policy (epsilon greedy in DQN) different from the online policy that is being learned.


h a behavior policy (epsilon greedy in DQN) different from the online policy that is being learned. Another key ingredient behind the success of DQN is experience replay (Lin, 1993; Mnih et al., 2015). During learning, the agent accumulates a dataset Dt = {e1, e2, . . . , et} of experiences et = (st, at, rt, st+1) from many episodes. When training the Q-network, instead only using the current experience as prescribed by standard temporaldifference learning, the network is trained by sampling mini-batches of experiences from D uniformly at random. The sequence of losses thus takes the form Li(θi) = E(s,a,r,s′)∼U(D)  yDQN i −Q(s, a; θi) 2 . Experience replay increases data efﬁciency through re-use of experience samples in multiple updates and, importantly, it reduces variance as uniform


h re-use of experience samples in multiple updates and, importantly, it reduces variance as uniform sampling from the replay buffer reduces the correlation among the samples used in the update. 2.2. Double Deep Q-networks The previous section described the main components of DQN as presented in (Mnih et al., 2015). In this paper,


Dueling Network Architectures for Deep Reinforcement Learning we use the improved Double DQN (DDQN) learning algorithm of van Hasselt et al. (2015). In Q-learning and DQN, the max operator uses the same values to both select and evaluate an action. This can therefore lead to overoptimistic value estimates (van Hasselt, 2010). To mitigate this problem, DDQN uses the following target: yDDQN i = r + γQ(s′, arg max a′ Q(s′, a′; θi); θ−). (6) DDQN is the same as for DQN (see Mnih et al. (2015)), but with the target yDQN i replaced by yDDQN i . The pseudocode for DDQN is presented in Appendix A. 2.3. Prioritized Replay A recent innovation in prioritized experience replay (Schaul et al., 2016) built on top of DDQN and further improved the state-of-the-art.


rience replay (Schaul et al., 2016) built on top of DDQN and further improved the state-of-the-art. Their key idea was to increase the replay probability of experience tuples that have a high expected learning progress (as measured via the proxy of absolute TD-error). This led to both faster learning and to better ﬁnal policy quality across most games of the Atari benchmark suite, as compared to uniform experience replay. To strengthen the claim that our dueling architecture is complementary to algorithmic innovations, we show that it improves performance for both the uniform and the prioritized replay baselines (for which we picked the easier to implement rank-based variant), with the resulting prioritized dueling variant holding the new state-of-the-art. 3.


based variant), with the resulting prioritized dueling variant holding the new state-of-the-art. 3. The Dueling Network Architecture The key insight behind our new architecture, as illustrated in Figure 2, is that for many states, it is unnecessary to estimate the value of each action choice. For example, in the Enduro game setting, knowing whether to move left or right only matters when a collision is eminent. In some states, it is of paramount importance to know which action to take, but in many other states the choice of action has no repercussion on what happens. For bootstrapping based algorithms, however, the estimation of state values is of great importance for every state. To bring this insight to fruition, we design a single Qnetwork architecture, as illustrated in Figure 1, which


his insight to fruition, we design a single Qnetwork architecture, as illustrated in Figure 1, which we refer to as the dueling network. The lower layers of the dueling network are convolutional as in the original DQNs (Mnih et al., 2015). However, instead of following the convolutional layers with a single sequence of fully connected layers, we instead use two sequences (or streams) of fully connected layers. The streams are constructed such that they have they have the capability of providing separate estimates of the value and advantage functions. Finally, the two streams are combined to produce a single output Q function. As in (Mnih et al., 2015), the output of the network is a set of Q values, one for each action.


on. As in (Mnih et al., 2015), the output of the network is a set of Q values, one for each action. Since the output of the dueling network is a Q function, it can be trained with the many existing algorithms, such as DDQN and SARSA. In addition, it can take advantage of any improvements to these algorithms, including better replay memories, better exploration policies, intrinsic motivation, and so on. The module that combines the two streams of fullyconnected layers to output a Q estimate requires very thoughtful design. From the expressions for advantage Qπ(s, a) = V π(s) + Aπ(s, a) and state-value V π(s) = Ea∼π(s) [Qπ(s, a)], it follows that Ea∼π(s) [Aπ(s, a)] = 0. Moreover, for a deterministic policy, a∗= arg maxa′∈A Q(s, a′), it follows that Q(s, a∗) = V (s) and hence A(s, a∗) = 0.


ministic policy, a∗= arg maxa′∈A Q(s, a′), it follows that Q(s, a∗) = V (s) and hence A(s, a∗) = 0. Let us consider the dueling network shown in Figure 1, where we make one stream of fully-connected layers output a scalar V (s; θ, β), and the other stream output an |A|dimensional vector A(s, a; θ, α). Here, θ denotes the parameters of the convolutional layers, while α and β are the parameters of the two streams of fully-connected layers. Using the deﬁnition of advantage, we might be tempted to construct the aggregating module as follows: Q(s, a; θ, α, β) = V (s; θ, β) + A(s, a; θ, α), (7) Note that this expression applies to all (s, a) instances; that is, to express equation (7) in matrix form we need to replicate the scalar, V (s; θ, β), |A| times.


is, to express equation (7) in matrix form we need to replicate the scalar, V (s; θ, β), |A| times. However, we need to keep in mind that Q(s, a; θ, α, β) is only a parameterized estimate of the true Q-function. Moreover, it would be wrong to conclude that V (s; θ, β) is a good estimator of the state-value function, or likewise that A(s, a; θ, α) provides a reasonable estimate of the advantage function. Equation (7) is unidentiﬁable in the sense that given Q we cannot recover V and A uniquely. To see this, add a constant to V (s; θ, β) and subtract the same constant from A(s, a; θ, α). This constant cancels out resulting in the same Q value. This lack of identiﬁability is mirrored by poor practical performance when this equation is used directly.


ck of identiﬁability is mirrored by poor practical performance when this equation is used directly. To address this issue of identiﬁability, we can force the advantage function estimator to have zero advantage at the chosen action. That is, we let the last module of the network implement the forward mapping Q(s, a; θ, α, β) = V (s; θ, β) +  A(s, a; θ, α) −max a′∈|A| A(s, a′; θ, α)  . (8)


Dueling Network Architectures for Deep Reinforcement Learning Now, for a∗ = arg maxa′∈A Q(s, a′; θ, α, β) = arg maxa′∈A A(s, a′; θ, α), we obtain Q(s, a∗; θ, α, β) = V (s; θ, β). Hence, the stream V (s; θ, β) provides an estimate of the value function, while the other stream produces an estimate of the advantage function. An alternative module replaces the max operator with an average: Q(s, a; θ, α, β) = V (s; θ, β) + A(s, a; θ, α) −1 |A| X a′ A(s, a′; θ, α) ! . (9) On the one hand this loses the original semantics of V and A because they are now off-target by a constant, but on the other hand it increases the stability of the optimization: with (9) the advantages only need to change as fast as the mean, instead of having to compensate any change to the optimal action’s advantage in (8).


t as the mean, instead of having to compensate any change to the optimal action’s advantage in (8). We also experimented with a softmax version of equation (8), but found it to deliver similar results to the simpler module of equation (9). Hence, all the experiments reported in this paper use the module of equation (9). Note that while subtracting the mean in equation (9) helps with identiﬁability, it does not change the relative rank of the A (and hence Q) values, preserving any greedy or ϵgreedy policy based on Q values from equation (7). When acting, it sufﬁces to evaluate the advantage stream to make decisions. It is important to note that equation (9) is viewed and implemented as part of the network and not as a separate algorithmic step.


uation (9) is viewed and implemented as part of the network and not as a separate algorithmic step. Training of the dueling architectures, as with standard Q networks (e.g. the deep Q-network of Mnih et al. (2015)), requires only back-propagation. The estimates V (s; θ, β) and A(s, a; θ, α) are computed automatically without any extra supervision or algorithmic modiﬁcations. As the dueling architecture shares the same input-output interface with standard Q networks, we can recycle all learning algorithms with Q networks (e.g., DDQN and SARSA) to train the dueling architecture. 4. Experiments We now show the practical performance of the dueling network. We start with a simple policy evaluation task and then show larger scale results for learning policies for general Atari game-playing.


ation task and then show larger scale results for learning policies for general Atari game-playing. 4.1. Policy evaluation We start by measuring the performance of the dueling architecture on a policy evaluation task. We choose this particular task because it is very useful for evaluating network architectures, as it is devoid of confounding factors such as the choice of exploration strategy, and the interaction between policy improvement and policy evaluation. In this experiment, we employ temporal difference learning (without eligibility traces, i.e., λ = 0) to learn Q values. More speciﬁcally, given a behavior policy π, we seek to estimate the state-action value Qπ(·, ·) by optimizing the sequence of costs of equation (4), with target yi = r + γEa′∼π(s′) [Q(s′, a′; θi)] .


optimizing the sequence of costs of equation (4), with target yi = r + γEa′∼π(s′) [Q(s′, a′; θi)] . The above update rule is the same as that of Expected SARSA (van Seijen et al., 2009). We, however, do not modify the behavior policy as in Expected SARSA. To evaluate the learned Q values, we choose a simple environment where the exact Qπ(s, a) values can be computed separately for all (s, a) ∈S × A. This environment, which we call the corridor is composed of three connected corridors. A schematic drawing of the corridor environment is shown in Figure 3, The agent starts from the bottom left corner of the environment and must move to the top right to get the largest reward. A total of 5 actions are available: go up, down, left, right and no-op.


to get the largest reward. A total of 5 actions are available: go up, down, left, right and no-op. We also have the freedom of adding an arbitrary number of no-op actions. In our setup, the two vertical sections both have 10 states while the horizontal section has 50. We use an ϵ-greedy policy as the behavior policy π, which chooses a random action with probability ϵ or an action according to the optimal Q function arg maxa∈A Q∗(s, a) with probability 1 −ϵ. In our experiments, ϵ is chosen to be 0.001. We compare a single-stream Q architecture with the dueling architecture on three variants of the corridor environment with 5, 10 and 20 actions respectively. The 10 and 20 action variants are formed by adding no-ops to the original environment.


espectively. The 10 and 20 action variants are formed by adding no-ops to the original environment. We measure performance by Squared Error (SE) against the true state values: P s∈S,a∈A(Q(s, a; θ) − Qπ(s, a))2. The single-stream architecture is a three layer MLP with 50 units on each hidden layer. The dueling architecture is also composed of three layers. After the ﬁrst hidden layer of 50 units, however, the network branches off into two streams each of them a two layer MLP with 25 hidden units. The results of the comparison are summarized in Figure 3. The results show that with 5 actions, both architectures converge at about the same speed. However, when we increase the number of actions, the dueling architecture performs better than the traditional Q-network.


ase the number of actions, the dueling architecture performs better than the traditional Q-network. In the dueling network, the stream V (s; θ, β) learns a general value that is shared across many similar actions at s, hence leading to faster convergence. This is a very promising result be


Dueling Network Architectures for Deep Reinforcement Learning CORRIDOR ENVIRONMENT 5 ACTIONS 10 ACTIONS 20 ACTIONS 103 104 No. Iterations 100 101 102 103 SE Single Duel 103 104 No. Iterations 100 101 102 103 103 104 No. Iterations 100 101 102 103 (a) (b) (c) (d) Figure 3. (a) The corridor environment. The star marks the starting state. The redness of a state signiﬁes the reward the agent receives upon arrival. The game terminates upon reaching either reward state. The agent’s actions are going up, down, left, right and no action. Plots (b), (c) and (d) shows squared error for policy evaluation with 5, 10, and 20 actions on a log-log scale. The dueling network (Duel) consistently outperforms a conventional single-stream network (Single), with the performance gap increasing with the number o


a conventional single-stream network (Single), with the performance gap increasing with the number of actions. cause many control tasks with large action spaces have this property, and consequently we should expect that the dueling network will often lead to much faster convergence than a traditional single stream network. In the following section, we will indeed see that the dueling network results in substantial gains in performance in a wide-range of Atari games. 4.2. General Atari Game-Playing We perform a comprehensive evaluation of our proposed method on the Arcade Learning Environment (Bellemare et al., 2013), which is composed of 57 Atari games. The challenge is to deploy a single algorithm and architecture, with a ﬁxed set of hyper-parameters, to learn to play all the games given


gorithm and architecture, with a ﬁxed set of hyper-parameters, to learn to play all the games given only raw pixel observations and game rewards. This environment is very demanding because it is both comprised of a large number of highly diverse games and the observations are high-dimensional. We follow closely the setup of van Hasselt et al. (2015) and compare to their results using single-stream Q-networks. We train the dueling network with the DDQN algorithm as presented in Appendix A. At the end of this section, we incorporate prioritized experience replay (Schaul et al., 2016). Our network architecture has the same low-level convolutional structure of DQN (Mnih et al., 2015; van Hasselt et al., 2015). There are 3 convolutional layers followed by 2 fully-connected layers.


; van Hasselt et al., 2015). There are 3 convolutional layers followed by 2 fully-connected layers. The ﬁrst convolutional layer has 32 8×8 ﬁlters with stride 4, the second 64 4×4 ﬁlters with stride 2, and the third and ﬁnal convolutional layer consists 64 3 × 3 ﬁlters with stride 1. As shown in Figure 1, the dueling network splits into two streams of fully connected layers. The value and advantage streams both have a fullyconnected layer with 512 units. The ﬁnal hidden layers of the value and advantage streams are both fully-connected with the value stream having one output and the advantage as many outputs as there are valid actions2. We combine the value and advantage streams using the module described by Equation (9).


id actions2. We combine the value and advantage streams using the module described by Equation (9). Rectiﬁer non-linearities (Fukushima, 1980) are inserted between all adjacent layers. We adopt the optimizers and hyper-parameters of van Hasselt et al. (2015), with the exception of the learning rate which we chose to be slightly lower (we do not do this for double DQN as it can deteriorate its performance). Since both the advantage and the value stream propagate gradients to the last convolutional layer in the backward pass, we rescale the combined gradient entering the last convolutional layer by 1/ √ 2. This simple heuristic mildly increases stability. In addition, we clip the gradients to have their norm less than or equal to 10.


ncreases stability. In addition, we clip the gradients to have their norm less than or equal to 10. This clipping is not standard practice in deep RL, but common in recurrent network training (Bengio et al., 2013). To isolate the contributions of the dueling architecture, we re-train DDQN with a single stream network using exactly the same procedure as described above. Speciﬁcally, we apply gradient clipping, and use 1024 hidden units for the ﬁrst fully-connected layer of the network so that both architectures (dueling and single) have roughly the same number of parameters. We refer to this re-trained model as Single Clip, while the original trained model of van Hasselt et al. (2015) is referred to as Single.


ingle Clip, while the original trained model of van Hasselt et al. (2015) is referred to as Single. As in (van Hasselt et al., 2015), we start the game with up to 30 no-op actions to provide random starting positions for the agent. To evaluate our approach, we measure improvement in percentage (positive or negative) in score over the better of human and baseline agent scores: ScoreAgent −ScoreBaseline max{ScoreHuman, ScoreBaseline} −ScoreRandom . (10) We took the maximum over human and baseline agent scores as it prevents insigniﬁcant changes to appear as 2The number of actions ranges between 3-18 actions in the ALE environment.


Dueling Network Architectures for Deep Reinforcement Learning Freeway Video Pinball Breakout Assault Beam Rider Solaris James Bond Tutankham Bowling Private Eye Montezuma's Revenge Pong Robotank Pitfall! Skiing H.E.R.O. Asteroids Demon Attack Wizard Of Wor Gravitar Gopher Boxing Berzerk Alien Kangaroo Kung-Fu Master Battle Zone Name This Game Defender Centipede Crazy Climber Ice Hockey Zaxxon Q*Bert Fishing Derby Amidar Venture River Raid Double Dunk Surround Star Gunner Ms. Pac-Man Krull Bank Heist Road Runner Asterix Time Pilot Frostbite Yars' Revenge Seaquest Chopper Command Enduro Phoenix Up and Down Space Invaders Tennis Atlantis -100.00% -68.31% -17.56% -14.93% -9.71% -7.37% -3.42% -3.38% -1.89% -0.04% 0.00% 0.24% 0.32% 0.45% 1.29% 2.31% 4.51% 4.78% 5.24% 5.54% 6.02% 8.52% 9.86% 10.3


38% -1.89% -0.04% 0.00% 0.24% 0.32% 0.45% 1.29% 2.31% 4.51% 4.78% 5.24% 5.54% 6.02% 8.52% 9.86% 10.34% 14.39% 15.56% 15.65% 16.28% 21.18% 21.68% 24.68% 26.45% 27.45% 27.68% 28.82% 31.40% 33.60% 39.79% 42.75% 44.24% 48.92% 53.76% 55.85% 57.19% 57.57% 63.17% 69.73% 70.02% 73.63% 80.51% 82.20% 86.35% 94.33% 97.90% 164.11% 180.00% 296.67% Figure 4. Improvements of dueling architecture over the baseline Single network of van Hasselt et al. (2015), using the metric described in Equation (10). Bars to the right indicate by how much the dueling network outperforms the single-stream network. large improvements when neither the agent in question nor the baseline are doing well. For example, an agent that achieves 2% human performance should not be interpreted as two times better when the baseline ag


hat achieves 2% human performance should not be interpreted as two times better when the baseline agent achieves 1% human performance. We also chose not to measure performance in terms of percentage of human performance alone because a tiny difference relative to the baseline on some games can translate into hundreds of percent in human performance difference. The results for the wide suite of 57 games are summarized in Table 1. Detailed results are presented in the Appendix. Using this 30 no-ops performance measure, it is clear that the dueling network (Duel Clip) does substantially better than the Single Clip network of similar capacity. It also does considerably better than the baseline (Single) of van Hasselt et al. (2015).


capacity. It also does considerably better than the baseline (Single) of van Hasselt et al. (2015). For comparison we also show results for the deep Q-network of Mnih et al. (2015), referred to as Nature DQN. Figure 4 shows the improvement of the dueling network over the baseline Single network of van Hasselt et al. (2015). Again, we seen that the improvements are often very dramatic. As shown in Table 1, Single Clip performs better than Single. We veriﬁed that this gain was mostly brought in by gradient clipping. For this reason, we incorporate gradient clipping in all the new approaches. Table 1. Mean and median scores across all 57 Atari games, measured in percentages of human performance. 30 no-ops Human Starts Mean Median Mean Median Prior.


measured in percentages of human performance. 30 no-ops Human Starts Mean Median Mean Median Prior. Duel Clip 591.9% 172.1% 567.0% 115.3% Prior. Single 434.6% 123.7% 386.7% 112.9% Duel Clip 373.1% 151.5% 343.8% 117.1% Single Clip 341.2% 132.6% 302.8% 114.1% Single 307.3% 117.8% 332.9% 110.9% Nature DQN 227.9% 79.1% 219.6% 68.5% Duel Clip does better than Single Clip on 75.4% of the games (43 out of 57). It also achieves higher scores compared to the Single baseline on 80.7% (46 out of 57) of the games. Of all the games with 18 actions, Duel Clip is better 86.6% of the time (26 out of 30). This is consistent with the ﬁndings of the previous section. Overall, our agent (Duel Clip) achieves human level performance on 42 out of 57 games.


ous section. Overall, our agent (Duel Clip) achieves human level performance on 42 out of 57 games. Raw scores for all the games, as well as measurements in human performance percentage, are presented in the Appendix. Robustness to human starts. One shortcoming of the 30 no-ops metric is that an agent does not necessarily have to generalize well to play the Atari games. Due to the deterministic nature of the Atari environment, from an unique starting point, an agent could learn to achieve good performance by simply remembering sequences of actions. To obtain a more robust measure, we adopt the methodology of Nair et al. (2015). Speciﬁcally, for each game, we use 100 starting points sampled from a human expert’s trajectory.


). Speciﬁcally, for each game, we use 100 starting points sampled from a human expert’s trajectory. From each of these points, an evaluation episode is launched for up to 108,000 frames. The agents are evaluated only on rewards accrued after the starting point. We refer to this metric as Human Starts. As shown in Table 1, under the Human Starts metric, Duel Clip once again outperforms the single stream variants. In particular, our agent does better than the Single baseline on 70.2% (40 out of 57) games and on games of 18 actions, Duel Clip is 83.3% better (25 out of 30). Combining with Prioritized Experience Replay. The dueling architecture can be easily combined with other algorithmic improvements.


rience Replay. The dueling architecture can be easily combined with other algorithmic improvements. In particular, prioritization of the experience replay has been shown to signiﬁcantly improve performance of Atari games (Schaul et al., 2016). Furthermore, as prioritization and the dueling architecture address very different aspects of the learning process, their combination is promising. So in our ﬁnal experiment, we investigate the integration of the dueling architecture with prioritized experience replay. We use the prioritized variant of DDQN (Prior. Single) as the new baseline algorithm, which replaces with the uniform sampling of the experi


Dueling Network Architectures for Deep Reinforcement Learning Kangaroo James Bond Double Dunk Skiing Seaquest Robotank Ms. Pac-Man Surround Solaris Time Pilot Ice Hockey Gravitar H.E.R.O. Alien Asteroids Breakout Freeway Bowling Venture Tennis Montezuma's Revenge Private Eye Pong Fishing Derby Demon Attack Boxing Pitfall! Road Runner Krull Enduro Atlantis Battle Zone Q*Bert Crazy Climber Tutankham Kung-Fu Master Amidar Beam Rider Centipede Zaxxon Name This Game Defender River Raid Bank Heist Assault Chopper Command Video Pinball Frostbite Berzerk Star Gunner Yars' Revenge Up and Down Wizard Of Wor Gopher Phoenix Space Invaders Asterix 1097.02% -89.22% -84.70% -83.56% -77.99% -60.56% -58.11% -48.03% -40.74% -37.65% -29.21% -13.60% -9.77% -6.72% -3.81% -3.13% -2.12% -2.08% -0.87% -0.51% 0.00


-48.03% -40.74% -37.65% -29.21% -13.60% -9.77% -6.72% -3.81% -3.13% -2.12% -2.08% -0.87% -0.51% 0.00% 0.00% 0.01% 0.73% 1.37% 1.44% 3.46% 5.33% 7.89% 7.95% 10.20% 11.16% 11.46% 15.56% 16.16% 21.38% 22.36% 24.98% 29.94% 32.48% 32.74% 33.09% 35.33% 38.56% 43.11% 51.07% 58.87% 69.92% 70.29% 83.91% 98.69% 113.16% 113.47% 178.13% 223.03% 281.56% 457.93% Figure 5. Improvements of dueling architecture over Prioritized DDQN baseline, using the same metric as Figure 4. Again, the dueling architecture leads to signiﬁcant improvements over the single-stream baseline on the majority of games. ence tuples by rank-based prioritized sampling. We keep all the parameters of the prioritized replay as described in (Schaul et al., 2016), namely a priority exponent of 0.7, and an annealing schedule on the impo


d in (Schaul et al., 2016), namely a priority exponent of 0.7, and an annealing schedule on the importance sampling exponent from 0.5 to 1. We combine this baseline with our dueling architecture (as above), and again use gradient clipping (Prior. Duel Clip). Note that, although orthogonal in their objectives, these extensions (prioritization, dueling and gradient clipping) interact in subtle ways. For example, prioritization interacts with gradient clipping, as sampling transitions with high absolute TD-errors more often leads to gradients with higher norms. To avoid adverse interactions, we roughly re-tuned the learning rate and the gradient clipping norm on a subset of 9 games. As a result of rough tuning, we settled on 6.25×10−5 for the learning rate and 10 for the gradient clipping nor


of rough tuning, we settled on 6.25×10−5 for the learning rate and 10 for the gradient clipping norm (the same as in the previous section). When evaluated on all 57 Atari games, our prioritized dueling agent performs signiﬁcantly better than both the prioritized baseline agent and the dueling agent alone. The full mean and median performance against the human performance percentage is shown in Table 1. When initializing the games using up to 30 no-ops action, we observe mean and median scores of 591% and 172% respectively. The direct comparison between the prioritized baseline and prioritized dueling versions, using the metric described in Equation 10, is presented in Figure 5. The combination of prioritized replay and the dueling network results in vast improvements over the previous sta


ion of prioritized replay and the dueling network results in vast improvements over the previous stateof-the-art in the popular ALE benchmark. Saliency maps. To better understand the roles of the value and the advantage streams, we compute saliency maps (Simonyan et al., 2013). More speciﬁcally, to visualize the salient part of the image as seen by the value stream, we compute the absolute value of the Jacobian of bV with respect to the input frames: |∇s bV (s; θ)|. Similarly, to visualize the salient part of the image as seen by the advantage stream, we compute |∇s bA(s, arg maxa′ bA(s, a′); θ)|. Both quantities are of the same dimensionality as the input frames and therefore can be visualized easily alongside the input frames.


mensionality as the input frames and therefore can be visualized easily alongside the input frames. Here, we place the gray scale input frames in the green and blue channel and the saliency maps in the red channel. All three channels together form an RGB image. Figure 2 depicts the value and advantage saliency maps on the Enduro game for two different time steps. As observed in the introduction, the value stream pays attention to the horizon where the appearance of a car could affect future performance. The value stream also pays attention to the score. The advantage stream, on the other hand, cares more about cars that are on an immediate collision course. 5. Discussion The advantage of the dueling architecture lies partly in its ability to learn the state-value function efﬁciently.


f the dueling architecture lies partly in its ability to learn the state-value function efﬁciently. With every update of the Q values in the dueling architecture, the value stream V is updated – this contrasts with the updates in a single-stream architecture where only the value for one of the actions is updated, the values for all other actions remain untouched. This more frequent updating of the value stream in our approach allocates more resources to V , and thus allows for better approximation of the state values, which in turn need to be accurate for temporaldifference-based methods like Q-learning to work (Sutton & Barto, 1998). This phenomenon is reﬂected in the experiments, where the advantage of the dueling architecture over single-stream Q networks grows when the number of action


advantage of the dueling architecture over single-stream Q networks grows when the number of actions is large. Furthermore, the differences between Q-values for a given state are often very small relative to the magnitude of Q. For example, after training with DDQN on the game of Seaquest, the average action gap (the gap between the Q values of the best and the second best action in a given state) across visited states is roughly 0.04, whereas the average state value across those states is about 15. This difference in scales can lead to small amounts of noise in the updates can lead to reorderings of the actions, and thus make the nearly greedy policy switch abruptly. The dueling ar


Dueling Network Architectures for Deep Reinforcement Learning chitecture with its separate advantage stream is robust to such effects. 6. Conclusions We introduced a new neural network architecture that decouples value and advantage in deep Q-networks, while sharing a common feature learning module. The new dueling architecture, in combination with some algorithmic improvements, leads to dramatic improvements over existing approaches for deep RL in the challenging Atari domain. The results presented in this paper are the new state-of-theart in this popular domain. References Ba, J., Mnih, V., and Kavukcuoglu, K. Multiple object recognition with visual attention. In ICLR, 2015. Baird, L.C. Advantage updating. Technical Report WLTR-93-1146, Wright-Patterson Air Force Base, 1993.


ird, L.C. Advantage updating. Technical Report WLTR-93-1146, Wright-Patterson Air Force Base, 1993. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 47:253–279, 2013. Bellemare, M. G., Ostrovski, G., Guez, A., Thomas, P. S., and Munos, R. Increasing the action gap: New operators for reinforcement learning. In AAAI, 2016. To appear. Bengio, Y., Boulanger-Lewandowski, N., and Pascanu, R. Advances in optimizing recurrent networks. In ICASSP, pp. 8624–8628, 2013. Fukushima, K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36:193–202, 1980.


m of pattern recognition unaffected by shift in position. Biological Cybernetics, 36:193–202, 1980. Guo, X., Singh, S., Lee, H., Lewis, R. L., and Wang, X. Deep learning for real-time Atari game play using ofﬂine Monte-Carlo tree search planning. In NIPS, pp. 3338– 3346. 2014. Harmon, M.E. and Baird, L.C. Multi-player residual advantage learning with general function approximation. Technical Report WL-TR-1065, Wright-Patterson Air Force Base, 1996. Harmon, M.E., Baird, L.C., and Klopf, A.H. Advantage updating applied to a differential game. In G. Tesauro, D.S. Touretzky and Leen, T.K. (eds.), NIPS, 1995. LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature, 521(7553):436–444, 2015. Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-toend training of deep visuomotor policies.


. Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-toend training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015. Lin, L.J. Reinforcement learning for robots using neural networks. PhD thesis, School of Computer Science, Carnegie Mellon University, 1993. Maddison, C. J., Huang, A., Sutskever, I., and Silver, D. Move Evaluation in Go Using Deep Convolutional Neural Networks. In ICLR, 2015. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518 (7540):529–533, 2015.


abis, D. Human-level control through deep reinforcement learning. Nature, 518 (7540):529–533, 2015. Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., Maria, A. De, Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu, K., and Silver, D. Massively parallel methods for deep reinforcement learning. In Deep Learning Workshop, ICML, 2015. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In ICLR, 2016. Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneers


dison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 01 2016. Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classiﬁcation models and saliency maps. arXiv preprint arXiv:1312.6034, 2013. Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814, 2015. Sutton, R. S. and Barto, A. G. Introduction to reinforcement learning. MIT Press, 1998.


0814, 2015. Sutton, R. S. and Barto, A. G. Introduction to reinforcement learning. MIT Press, 1998. Sutton, R. S., Mcallester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pp. 1057–1063, 2000. van Hasselt, H. Double Q-learning. NIPS, 23:2613–2621, 2010. van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double Q-learning. arXiv preprint


Dueling Network Architectures for Deep Reinforcement Learning arXiv:1509.06461, 2015. van Seijen, H., van Hasselt, H., Whiteson, S., and Wiering, M. A theoretical and empirical analysis of Expected Sarsa. In IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pp. 177–184. 2009. Watter, M., Springenberg, J. T., Boedecker, J., and Riedmiller, M. A. Embed to control: A locally linear latent dynamics model for control from raw images. In NIPS, 2015.


Dueling Network Architectures for Deep Reinforcement Learning A. Double DQN Algorithm Algorithm 1: Double DQN Algorithm. input : D – empty replay buffer; θ – initial network parameters, θ−– copy of θ input : Nr – replay buffer maximum size; Nb – training batch size; N −– target network replacement freq. for episode e ∈{1, 2, . . . , M } do Initialize frame sequence x ←() for t ∈{0, 1, . . .} do Set state s ←x, sample action a ∼πB Sample next frame xt from environment E given (s, a) and receive reward r, and append xt to x if |x| > Nf then delete oldest frame xtmin from x end Set s′ ←x, and add transition tuple (s, a, r, s′) to D, replacing the oldest tuple if |D| ≥Nr Sample a minibatch of Nb tuples (s, a, r, s′) ∼Unif(D) Construct target values, one for each of the Nb tuples: Deﬁne amax (s


tuples (s, a, r, s′) ∼Unif(D) Construct target values, one for each of the Nb tuples: Deﬁne amax (s′; θ) = arg maxa′ Q(s′, a′; θ) yj =  r if s′ is terminal r + γQ(s′, amax (s′; θ); θ−), otherwise. Do a gradient descent step with loss ∥yj −Q(s, a; θ)∥2 Replace target parameters θ−←θ every N −steps end end


Dueling Network Architectures for Deep Reinforcement Learning Table 2. Raw scores across all games. Starting with 30 no-op actions. GAMES NO. ACTIONS RANDOM HUMAN DQN DDQN DUEL PRIOR. PRIOR. DUEL. Alien 18 227.8 7,127.7 1,620.0 3,747.7 4,461.4 4,203.8 3,941.0 Amidar 10 5.8 1,719.5 978.0 1,793.3 2,354.5 1,838.9 2,296.8 Assault 7 222.4 742.0 4,280.4 5,393.2 4,621.0 7,672.1 11,477.0 Asterix 9 210.0 8,503.3 4,359.0 17,356.5 28,188.0 31,527.0 375,080.0 Asteroids 14 719.1 47,388.7 1,364.5 734.7 2,837.7 2,654.3 1,192.7 Atlantis 4 12,850.0 29,028.1 279,987.0 106,056.0 382,572.0 357,324.0 395,762.0 Bank Heist 18 14.2 753.1 455.0 1,030.6 1,611.9 1,054.6 1,503.1 Battle Zone 18 2,360.0 37,187.5 29,900.0 31,700.0 37,150.0 31,530.0 35,520.0 Beam Rider 9 363.9 16,926.5 8,627.5 13,772.8 12,164.0 23,384.2