θold(· | st), πθ(· | st)]] ≤δ. (4) Here, θold is the vector of policy parameters before the update. This problem can eﬃciently be approximately solved using the conjugate gradient algorithm, after making a linear approximation to the objective and a quadratic approximation to the constraint. The theory justifying TRPO actually suggests using a penalty instead of a constraint, i.e., solving the unconstrained optimization problem maximize θ ˆEt  πθ(at | st) πθold(at | st) ˆAt −β KL[πθold(· | st), πθ(· | st)]  (5) for some coeﬃcient β. This follows from the fact that a certain surrogate objective (which computes the max KL over states instead of the mean) forms a lower bound (i.e., a pessimistic bound) on the performance of the policy π.


ad of the mean) forms a lower bound (i.e., a pessimistic bound) on the performance of the policy π. TRPO uses a hard constraint rather than a penalty because it is hard to choose a single value of β that performs well across diﬀerent problems—or even within a single problem, where the the characteristics change over the course of learning. Hence, to achieve our goal of a ﬁrst-order algorithm that emulates the monotonic improvement of TRPO, experiments show that it is not suﬃcient to simply choose a ﬁxed penalty coeﬃcient β and optimize the penalized objective Equation (5) with SGD; additional modiﬁcations are required. 2


3 Clipped Surrogate Objective Let rt(θ) denote the probability ratio rt(θ) = πθ(at | st) πθold(at | st), so r(θold) = 1. TRPO maximizes a “surrogate” objective LCPI(θ) = ˆEt  πθ(at | st) πθold(at | st) ˆAt  = ˆEt h rt(θ) ˆAt i . (6) The superscript CPI refers to conservative policy iteration [KL02], where this objective was proposed. Without a constraint, maximization of LCPI would lead to an excessively large policy update; hence, we now consider how to modify the objective, to penalize changes to the policy that move rt(θ) away from 1. The main objective we propose is the following: LCLIP (θ) = ˆEt h min(rt(θ) ˆAt, clip(rt(θ), 1 −ϵ, 1 + ϵ) ˆAt) i (7) where epsilon is a hyperparameter, say, ϵ = 0.2. The motivation for this objective is as follows. The ﬁrst term inside the min is LCPI.


ay, ϵ = 0.2. The motivation for this objective is as follows. The ﬁrst term inside the min is LCPI. The second term, clip(rt(θ), 1−ϵ, 1+ϵ) ˆAt, modiﬁes the surrogate objective by clipping the probability ratio, which removes the incentive for moving rt outside of the interval [1 −ϵ, 1 + ϵ]. Finally, we take the minimum of the clipped and unclipped objective, so the ﬁnal objective is a lower bound (i.e., a pessimistic bound) on the unclipped objective. With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse. Note that LCLIP (θ) = LCPI(θ) to ﬁrst order around θold (i.e., where r = 1), however, they become diﬀerent as θ moves away from θold.


rst order around θold (i.e., where r = 1), however, they become diﬀerent as θ moves away from θold. Figure 1 plots a single term (i.e., a single t) in LCLIP ; note that the probability ratio r is clipped at 1 −ϵ or 1 + ϵ depending on whether the advantage is positive or negative. r LCLIP 0 1 1 + ϵ A > 0 r LCLIP 0 1 1 −ϵ A < 0 Figure 1: Plots showing one term (i.e., a single timestep) of the surrogate function LCLIP as a function of the probability ratio r, for positive advantages (left) and negative advantages (right). The red circle on each plot shows the starting point for the optimization, i.e., r = 1. Note that LCLIP sums many of these terms. Figure 2 provides another source of intuition about the surrogate objective LCLIP .


of these terms. Figure 2 provides another source of intuition about the surrogate objective LCLIP . It shows how several objectives vary as we interpolate along the policy update direction, obtained by proximal policy optimization (the algorithm we will introduce shortly) on a continuous control problem. We can see that LCLIP is a lower bound on LCPI, with a penalty for having too large of a policy update. 3


0 1 Linear interpolation factor 0.02 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Et[KLt] LCPI = Et[rtAt] Et[clip(rt, 1 , 1 + )At] LCLIP = Et[min(rtAt, clip(rt, 1 , 1 + )At)] Figure 2: Surrogate objectives, as we interpolate between the initial policy parameter θold, and the updated policy parameter, which we compute after one iteration of PPO. The updated policy has a KL divergence of about 0.02 from the initial policy, and this is the point at which LCLIP is maximal. This plot corresponds to the ﬁrst policy update on the Hopper-v1 problem, using hyperparameters provided in Section 6.1. 4 Adaptive KL Penalty Coeﬃcient Another approach, which can be used as an alternative to the clipped surrogate objective, or in addition to it, is to use a penalty on KL divergence, and to adapt the penalty coeﬃcien


ctive, or in addition to it, is to use a penalty on KL divergence, and to adapt the penalty coeﬃcient so that we achieve some target value of the KL divergence dtarg each policy update. In our experiments, we found that the KL penalty performed worse than the clipped surrogate objective, however, we’ve included it here because it’s an important baseline. In the simplest instantiation of this algorithm, we perform the following steps in each policy update: • Using several epochs of minibatch SGD, optimize the KL-penalized objective LKLPEN(θ) = ˆEt  πθ(at | st) πθold(at | st) ˆAt −β KL[πθold(· | st), πθ(· | st)]  (8) • Compute d = ˆEt[KL[πθold(· | st), πθ(· | st)]] – If d < dtarg/1.5, β ←β/2 – If d > dtarg × 1.5, β ←β × 2 The updated β is used for the next policy update.


dtarg/1.5, β ←β/2 – If d > dtarg × 1.5, β ←β × 2 The updated β is used for the next policy update. With this scheme, we occasionally see policy updates where the KL divergence is signiﬁcantly diﬀerent from dtarg, however, these are rare, and β quickly adjusts. The parameters 1.5 and 2 above are chosen heuristically, but the algorithm is not very sensitive to them. The initial value of β is a another hyperparameter but is not important in practice because the algorithm quickly adjusts it. 5 Algorithm The surrogate losses from the previous sections can be computed and diﬀerentiated with a minor change to a typical policy gradient implementation. For implementations that use automatic differentation, one simply constructs the loss LCLIP or LKLPEN instead of LPG, and one performs multiple ste


tation, one simply constructs the loss LCLIP or LKLPEN instead of LPG, and one performs multiple steps of stochastic gradient ascent on this objective. Most techniques for computing variance-reduced advantage-function estimators make use a learned state-value function V (s); for example, generalized advantage estimation [Sch+15a], or the 4


ﬁnite-horizon estimators in [Mni+16]. If using a neural network architecture that shares parameters between the policy and value function, we must use a loss function that combines the policy surrogate and a value function error term. This objective can further be augmented by adding an entropy bonus to ensure suﬃcient exploration, as suggested in past work [Wil92; Mni+16]. Combining these terms, we obtain the following objective, which is (approximately) maximized each iteration: LCLIP+V F+S t (θ) = ˆEt  LCLIP t (θ) −c1LV F t (θ) + c2S[πθ](st)  , (9) where c1, c2 are coeﬃcients, and S denotes an entropy bonus, and LV F t is a squared-error loss (Vθ(st) −V targ t )2. One style of policy gradient implementation, popularized in [Mni+16] and well-suited for use with recurrent neural network


adient implementation, popularized in [Mni+16] and well-suited for use with recurrent neural networks, runs the policy for T timesteps (where T is much less than the episode length), and uses the collected samples for an update. This style requires an advantage estimator that does not look beyond timestep T. The estimator used by [Mni+16] is ˆAt = −V (st) + rt + γrt+1 + · · · + γT−t+1rT−1 + γT−tV (sT ) (10) where t speciﬁes the time index in [0, T], within a given length-T trajectory segment. Generalizing this choice, we can use a truncated version of generalized advantage estimation, which reduces to Equation (10) when λ = 1: ˆAt = δt + (γλ)δt+1 + · · · + · · · + (γλ)T−t+1δT−1, (11) where δt = rt + γV (st+1) −V (st) (12) A proximal policy optimization (PPO) algorithm that uses ﬁxed-length


t = rt + γV (st+1) −V (st) (12) A proximal policy optimization (PPO) algorithm that uses ﬁxed-length trajectory segments is shown below. Each iteration, each of N (parallel) actors collect T timesteps of data. Then we construct the surrogate loss on these NT timesteps of data, and optimize it with minibatch SGD (or usually for better performance, Adam [KB14]), for K epochs. Algorithm 1 PPO, Actor-Critic Style for iteration=1, 2, . . . do for actor=1, 2, . . . , N do Run policy πθold in environment for T timesteps Compute advantage estimates ˆA1, . . . , ˆAT end for Optimize surrogate L wrt θ, with K epochs and minibatch size M ≤NT θold ←θ end for 6 Experiments 6.1 Comparison of Surrogate Objectives First, we compare several diﬀerent surrogate objectives under diﬀerent hyperparameters.


Objectives First, we compare several diﬀerent surrogate objectives under diﬀerent hyperparameters. Here, we compare the surrogate objective LCLIP to several natural variations and ablated versions. No clipping or penalty: Lt(θ) = rt(θ) ˆAt Clipping: Lt(θ) = min(rt(θ) ˆAt, clip(rt(θ)), 1 −ϵ, 1 + ϵ) ˆAt KL penalty (ﬁxed or adaptive) Lt(θ) = rt(θ) ˆAt −β KL[πθold, πθ] 5


For the KL penalty, one can either use a ﬁxed penalty coeﬃcient β or an adaptive coeﬃcient as described in Section 4 using target KL value dtarg. Note that we also tried clipping in log space, but found the performance to be no better. Because we are searching over hyperparameters for each algorithm variant, we chose a computationally cheap benchmark to test the algorithms on. Namely, we used 7 simulated robotics tasks2 implemented in OpenAI Gym [Bro+16], which use the MuJoCo [TET12] physics engine. We do one million timesteps of training on each one. Besides the hyperparameters used for clipping (ϵ) and the KL penalty (β, dtarg), which we search over, the other hyperparameters are provided in in Table 3.


KL penalty (β, dtarg), which we search over, the other hyperparameters are provided in in Table 3. To represent the policy, we used a fully-connected MLP with two hidden layers of 64 units, and tanh nonlinearities, outputting the mean of a Gaussian distribution, with variable standard deviations, following [Sch+15b; Dua+16]. We don’t share parameters between the policy and value function (so coeﬃcient c1 is irrelevant), and we don’t use an entropy bonus. Each algorithm was run on all 7 environments, with 3 random seeds on each. We scored each run of the algorithm by computing the average total reward of the last 100 episodes. We shifted and scaled the scores for each environment so that the random policy gave a score of 0 and the best result was set to 1, and averaged over 21 runs to prod


random policy gave a score of 0 and the best result was set to 1, and averaged over 21 runs to produce a single scalar for each algorithm setting. The results are shown in Table 1. Note that the score is negative for the setting without clipping or penalties, because for one environment (half cheetah) it leads to a very negative score, which is worse than the initial random policy. algorithm avg. normalized score No clipping or penalty -0.39 Clipping, ϵ = 0.1 0.76 Clipping, ϵ = 0.2 0.82 Clipping, ϵ = 0.3 0.70 Adaptive KL dtarg = 0.003 0.68 Adaptive KL dtarg = 0.01 0.74 Adaptive KL dtarg = 0.03 0.71 Fixed KL, β = 0.3 0.62 Fixed KL, β = 1. 0.71 Fixed KL, β = 3. 0.72 Fixed KL, β = 10. 0.69 Table 1: Results from continuous control benchmark.


71 Fixed KL, β = 3. 0.72 Fixed KL, β = 10. 0.69 Table 1: Results from continuous control benchmark. Average normalized scores (over 21 runs of the algorithm, on 7 environments) for each algorithm / hyperparameter setting . β was initialized at 1. 6.2 Comparison to Other Algorithms in the Continuous Domain Next, we compare PPO (with the “clipped” surrogate objective from Section 3) to several other methods from the literature, which are considered to be eﬀective for continuous problems. We compared against tuned implementations of the following algorithms: trust region policy optimization [Sch+15b], cross-entropy method (CEM) [SL06], vanilla policy gradient with adaptive stepsize3, 2HalfCheetah, Hopper, InvertedDoublePendulum, InvertedPendulum, Reacher, Swimmer, and Walker2d, all “-v1” 3Aft


ah, Hopper, InvertedDoublePendulum, InvertedPendulum, Reacher, Swimmer, and Walker2d, all “-v1” 3After each batch of data, the Adam stepsize is adjusted based on the KL divergence of the original and updated policy, using a rule similar to the one shown in Section 4. An implementation is available at https://github.com/ berkeleydeeprlcourse/homework/tree/master/hw4. 6


A2C [Mni+16], A2C with trust region [Wan+16]. A2C stands for advantage actor critic, and is a synchronous version of A3C, which we found to have the same or better performance than the asynchronous version. For PPO, we used the hyperparameters from the previous section, with ϵ = 0.2. We see that PPO outperforms the previous methods on almost all the continuous control environments. 0 1000000 500 0 500 1000 1500 2000 HalfCheetah-v1 0 1000000 0 500 1000 1500 2000 2500 Hopper-v1 0 1000000 0 2000 4000 6000 8000 InvertedDoublePendulum-v1 0 1000000 0 200 400 600 800 1000 InvertedPendulum-v1 0 1000000 120 100 80 60 40 20 Reacher-v1 0 1000000 0 20 40 60 80 100 120 Swimmer-v1 0 1000000 0 1000 2000 3000 Walker2d-v1 A2C A2C + Trust Region CEM PPO (Clip) Vanilla PG, Adaptive TRPO Figure 3: Comparison


00 Walker2d-v1 A2C A2C + Trust Region CEM PPO (Clip) Vanilla PG, Adaptive TRPO Figure 3: Comparison of several algorithms on several MuJoCo environments, training for one million timesteps. 6.3 Showcase in the Continuous Domain: Humanoid Running and Steering To showcase the performance of PPO on high-dimensional continuous control problems, we train on a set of problems involving a 3D humanoid, where the robot must run, steer, and get up oﬀthe ground, possibly while being pelted by cubes. The three tasks we test on are (1) RoboschoolHumanoid: forward locomotion only, (2) RoboschoolHumanoidFlagrun: position of target is randomly varied every 200 timesteps or whenever the goal is reached, (3) RoboschoolHumanoidFlagrunHarder, where the robot is pelted by cubes and needs to get up oﬀthe ground


RoboschoolHumanoidFlagrunHarder, where the robot is pelted by cubes and needs to get up oﬀthe ground. See Figure 5 for still frames of a learned policy, and Figure 4 for learning curves on the three tasks. Hyperparameters are provided in Table 4. In concurrent work, Heess et al. [Hee+17] used the adaptive KL variant of PPO (Section 4) to learn locomotion policies for 3D robots. 0 50M Timestep 0 1000 2000 3000 4000 RoboschoolHumanoid-v0 0 100M Timestep 0 500 1000 1500 2000 2500 RoboschoolHumanoidFlagrun-v0 0 100M Timestep 0 1000 2000 3000 RoboschoolHumanoidFlagrunHarder-v0 Figure 4: Learning curves from PPO on 3D humanoid control tasks, using Roboschool. 7


Figure 5: Still frames of the policy learned from RoboschoolHumanoidFlagrun. In the ﬁrst six frames, the robot runs towards a target. Then the position is randomly changed, and the robot turns and runs toward the new target. 6.4 Comparison to Other Algorithms on the Atari Domain We also ran PPO on the Arcade Learning Environment [Bel+15] benchmark and compared against well-tuned implementations of A2C [Mni+16] and ACER [Wan+16]. For all three algorithms, we used the same policy network architechture as used in [Mni+16]. The hyperparameters for PPO are provided in Table 5. For the other two algorithms, we used hyperparameters that were tuned to maximize performance on this benchmark. A table of results and learning curves for all 49 games is provided in Appendix B.


this benchmark. A table of results and learning curves for all 49 games is provided in Appendix B. We consider the following two scoring metrics: (1) average reward per episode over entire training period (which favors fast learning), and (2) average reward per episode over last 100 episodes of training (which favors ﬁnal performance). Table 2 shows the number of games “won” by each algorithm, where we compute the victor by averaging the scoring metric across three trials. A2C ACER PPO Tie (1) avg. episode reward over all of training 1 18 30 0 (2) avg. episode reward over last 100 episodes 1 28 19 1 Table 2: Number of games “won” by each algorithm, where the scoring metric is averaged across three trials.


Number of games “won” by each algorithm, where the scoring metric is averaged across three trials. 7 Conclusion We have introduced proximal policy optimization, a family of policy optimization methods that use multiple epochs of stochastic gradient ascent to perform each policy update. These methods have the stability and reliability of trust-region methods but are much simpler to implement, requiring only few lines of code change to a vanilla policy gradient implementation, applicable in more general settings (for example, when using a joint architecture for the policy and value function), and have better overall performance. 8 Acknowledgements Thanks to Rocky Duan, Peter Chen, and others at OpenAI for insightful comments. 8


References [Bel+15] M. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. “The arcade learning environment: An evaluation platform for general agents”. In: Twenty-Fourth International Joint Conference on Artiﬁcial Intelligence. 2015. [Bro+16] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. “OpenAI Gym”. In: arXiv preprint arXiv:1606.01540 (2016). [Dua+16] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. “Benchmarking Deep Reinforcement Learning for Continuous Control”. In: arXiv preprint arXiv:1604.06778 (2016). [Hee+17] N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, A. Eslami, M. Riedmiller, et al. “Emergence of Locomotion Behaviours in Rich Environments”. In: arXiv preprint arXiv:1707.02286 (2017).


ergence of Locomotion Behaviours in Rich Environments”. In: arXiv preprint arXiv:1707.02286 (2017). [KL02] S. Kakade and J. Langford. “Approximately optimal approximate reinforcement learning”. In: ICML. Vol. 2. 2002, pp. 267–274. [KB14] D. Kingma and J. Ba. “Adam: A method for stochastic optimization”. In: arXiv preprint arXiv:1412.6980 (2014). [Mni+15] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. “Human-level control through deep reinforcement learning”. In: Nature 518.7540 (2015), pp. 529–533. [Mni+16] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. “Asynchronous methods for deep reinforcement learning”.


, T. Harley, D. Silver, and K. Kavukcuoglu. “Asynchronous methods for deep reinforcement learning”. In: arXiv preprint arXiv:1602.01783 (2016). [Sch+15a] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. “High-dimensional continuous control using generalized advantage estimation”. In: arXiv preprint arXiv:1506.02438 (2015). [Sch+15b] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. “Trust region policy optimization”. In: CoRR, abs/1502.05477 (2015). [SL06] I. Szita and A. L¨orincz. “Learning Tetris using the noisy cross-entropy method”. In: Neural computation 18.12 (2006), pp. 2936–2941. [TET12] E. Todorov, T. Erez, and Y. Tassa. “MuJoCo: A physics engine for model-based control”.


941. [TET12] E. Todorov, T. Erez, and Y. Tassa. “MuJoCo: A physics engine for model-based control”. In: Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE. 2012, pp. 5026–5033. [Wan+16] Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas. “Sample Eﬃcient Actor-Critic with Experience Replay”. In: arXiv preprint arXiv:1611.01224 (2016). [Wil92] R. J. Williams. “Simple statistical gradient-following algorithms for connectionist reinforcement learning”. In: Machine learning 8.3-4 (1992), pp. 229–256. 9


A Hyperparameters Hyperparameter Value Horizon (T) 2048 Adam stepsize 3 × 10−4 Num. epochs 10 Minibatch size 64 Discount (γ) 0.99 GAE parameter (λ) 0.95 Table 3: PPO hyperparameters used for the Mujoco 1 million timestep benchmark. Hyperparameter Value Horizon (T) 512 Adam stepsize ∗ Num. epochs 15 Minibatch size 4096 Discount (γ) 0.99 GAE parameter (λ) 0.95 Number of actors 32 (locomotion), 128 (ﬂagrun) Log stdev. of action distribution LinearAnneal(−0.7, −1.6) Table 4: PPO hyperparameters used for the Roboschool experiments. Adam stepsize was adjusted based on the target value of the KL divergence. Hyperparameter Value Horizon (T) 128 Adam stepsize 2.5 × 10−4 × α Num. epochs 3 Minibatch size 32 × 8 Discount (γ) 0.99 GAE parameter (λ) 0.95 Number of actors 8 Clipping parameter ϵ 0.1 × α V


ze 32 × 8 Discount (γ) 0.99 GAE parameter (λ) 0.95 Number of actors 8 Clipping parameter ϵ 0.1 × α VF coeﬀ. c1 (9) 1 Entropy coeﬀ. c2 (9) 0.01 Table 5: PPO hyperparameters used in Atari experiments. α is linearly annealed from 1 to 0 over the course of learning. B Performance on More Atari Games Here we include a comparison of PPO against A2C on a larger collection of 49 Atari games. Figure 6 shows the learning curves of each of three random seeds, while Table 6 shows the mean performance. 10


1000 2000 Alien 0 250 500 750 Amidar 0 2000 4000 Assault 0 2500 5000 7500 Asterix 1500 2000 2500 Asteroids 0 1000000 2000000 3000000 Atlantis 0 500 1000 BankHeist 5000 10000 15000 20000 BattleZone 1000 2000 3000 4000 BeamRider 30 40 50 Bowling 0 50 100 Boxing 0 200 400 Breakout 5000 10000 Centipede 2000 4000 6000 ChopperCommand 50000 100000 CrazyClimber 0 20000 40000 DemonAttack 17.5 15.0 12.5 10.0 DoubleDunk 0 250 500 750 Enduro 100 50 0 FishingDerby 0 10 20 30 Freeway 100 200 300 Frostbite 0 20000 40000 Gopher 250 500 750 Gravitar 10 8 6 4 IceHockey 0 200 400 600 Jamesbond 0 5000 10000 Kangaroo 2000 4000 6000 8000 Krull 0 20000 40000 KungFuMaster 0 50 100 MontezumaRevenge 1000 2000 3000 MsPacman 2500 5000 7500 10000 NameThisGame 100 0 Pitfall 20 0 20 Pong 0 500 PrivateEye 0 5000 10000 15


Pacman 2500 5000 7500 10000 NameThisGame 100 0 Pitfall 20 0 20 Pong 0 500 PrivateEye 0 5000 10000 15000 Qbert 2500 5000 7500 10000 Riverraid 0 20000 40000 RoadRunner 2 4 6 Robotank 0 500 1000 1500 Seaquest 500 1000 SpaceInvaders 0 20000 40000 StarGunner 20 15 10 Tennis 3000 4000 TimePilot 0 100 200 300 Tutankham 0 100000 200000 UpNDown 0 40M Frames 0 5 10 Venture 0 40M Frames 50000 100000 150000 VideoPinball 0 40M Frames 2000 4000 WizardOfWor 0 40M Frames 0 2000 4000 6000 Zaxxon A2C ACER PPO Figure 6: Comparison of PPO and A2C on all 49 ATARI games included in OpenAI Gym at the time of publication. 11


A2C ACER PPO Alien 1141.7 1655.4 1850.3 Amidar 380.8 827.6 674.6 Assault 1562.9 4653.8 4971.9 Asterix 3176.3 6801.2 4532.5 Asteroids 1653.3 2389.3 2097.5 Atlantis 729265.3 1841376.0 2311815.0 BankHeist 1095.3 1177.5 1280.6 BattleZone 3080.0 8983.3 17366.7 BeamRider 3031.7 3863.3 1590.0 Bowling 30.1 33.3 40.1 Boxing 17.7 98.9 94.6 Breakout 303.0 456.4 274.8 Centipede 3496.5 8904.8 4386.4 ChopperCommand 1171.7 5287.7 3516.3 CrazyClimber 107770.0 132461.0 110202.0 DemonAttack 6639.1 38808.3 11378.4 DoubleDunk -16.2 -13.2 -14.9 Enduro 0.0 0.0 758.3 FishingDerby 20.6 34.7 17.8 Freeway 0.0 0.0 32.5 Frostbite 261.8 285.6 314.2 Gopher 1500.9 37802.3 2932.9 Gravitar 194.0 225.3 737.2 IceHockey -6.4 -5.9 -4.2 Jamesbond 52.3 261.8 560.7 Kangaroo 45.3 50.0 9928.7 Krull 8367.4 7268.4 7942.3 KungFuMaste


5.9 -4.2 Jamesbond 52.3 261.8 560.7 Kangaroo 45.3 50.0 9928.7 Krull 8367.4 7268.4 7942.3 KungFuMaster 24900.3 27599.3 23310.3 MontezumaRevenge 0.0 0.3 42.0 MsPacman 1626.9 2718.5 2096.5 NameThisGame 5961.2 8488.0 6254.9 Pitfall -55.0 -16.9 -32.9 Pong 19.7 20.7 20.7 PrivateEye 91.3 182.0 69.5 Qbert 10065.7 15316.6 14293.3 Riverraid 7653.5 9125.1 8393.6 RoadRunner 32810.0 35466.0 25076.0 Robotank 2.2 2.5 5.5 Seaquest 1714.3 1739.5 1204.5 SpaceInvaders 744.5 1213.9 942.5 StarGunner 26204.0 49817.7 32689.0 Tennis -22.2 -17.6 -14.8 TimePilot 2898.0 4175.7 4342.0 Tutankham 206.8 280.8 254.4 UpNDown 17369.8 145051.4 95445.0 Venture 0.0 0.0 0.0 VideoPinball 19735.9 156225.6 37389.0 WizardOfWor 859.0 2308.3 4185.3 Zaxxon 16.3 29.0 5008.7 Table 6: Mean ﬁnal scores (last 100 episodes) of PPO and A2C


2308.3 4185.3 Zaxxon 16.3 29.0 5008.7 Table 6: Mean ﬁnal scores (last 100 episodes) of PPO and A2C on Atari games after 40M game frames (10M timesteps). 12


Rainbow: Combining Improvements in Deep Reinforcement Learning Matteo Hessel DeepMind Joseph Modayil DeepMind Hado van Hasselt DeepMind Tom Schaul DeepMind Georg Ostrovski DeepMind Will Dabney DeepMind Dan Horgan DeepMind Bilal Piot DeepMind Mohammad Azar DeepMind David Silver DeepMind Abstract The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efﬁciency and ﬁnal performance.


-art performance on the Atari 2600 benchmark, both in terms of data efﬁciency and ﬁnal performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance. Introduction The many recent successes in scaling reinforcement learning (RL) to complex sequential decision-making problems were kick-started by the Deep Q-Networks algorithm (DQN; Mnih et al. 2013, 2015). Its combination of Q-learning with convolutional neural networks and experience replay enabled it to learn, from raw pixels, how to play many Atari games at human-level performance. Since then, many extensions have been proposed that enhance its speed or stability. Double DQN (DDQN; van Hasselt, Guez, and Silver 2016) addresses an overestimation bias of Q-learning (


uble DQN (DDQN; van Hasselt, Guez, and Silver 2016) addresses an overestimation bias of Q-learning (van Hasselt 2010), by decoupling selection and evaluation of the bootstrap action. Prioritized experience replay (Schaul et al. 2015) improves data efﬁciency, by replaying more often transitions from which there is more to learn. The dueling network architecture (Wang et al. 2016) helps to generalize across actions by separately representing state values and action advantages. Learning from multi-step bootstrap targets (Sutton 1988; Sutton and Barto 1998), as used in A3C (Mnih et al. 2016), shifts the bias-variance tradeoff and helps to propagate newly observed rewards faster to earlier visited states.


s-variance tradeoff and helps to propagate newly observed rewards faster to earlier visited states. Distributional Q-learning (Bellemare, Dabney, and Munos 2017) learns a categorical distribution of discounted returns, instead of estimating the mean. Noisy DQN (Fortunato et al. 2017) uses stochastic network layers for exploration. This list is, of course, far from exhaustive. Each of these algorithms enables substantial performance improvements in isolation. Since they do so by addressing Copyright c⃝2018, Association for the Advancement of Artiﬁcial Intelligence (www.aaai.org). All rights reserved. 7 44 100 200 Millions of frames 0% 100% 200% Median human-normalized score DQN DDQN Prioritized DDQN Dueling DDQN A3C Distributional DQN Noisy DQN Rainbow Figure 1: Median human-normalized perf


ed DDQN Dueling DDQN A3C Distributional DQN Noisy DQN Rainbow Figure 1: Median human-normalized performance across 57 Atari games. We compare our integrated agent (rainbowcolored) to DQN (grey) and six published baselines. Note that we match DQN’s best performance after 7M frames, surpass any baseline within 44M frames, and reach substantially improved ﬁnal performance. Curves are smoothed with a moving average over 5 points. radically different issues, and since they build on a shared framework, they could plausibly be combined. In some cases this has been done: Prioritized DDQN and Dueling DDQN both use double Q-learning, and Dueling DDQN was also combined with prioritized experience replay. In this paper we propose to study an agent that combines all the aforementioned ingredients.


eplay. In this paper we propose to study an agent that combines all the aforementioned ingredients. We show how these different ideas can be integrated, and that they are indeed largely complementary. In fact, their combination results in new stateof-the-art results on the benchmark suite of 57 Atari 2600 games from the Arcade Learning Environment (Bellemare et al. 2013), both in terms of data efﬁciency and of ﬁnal performance. Finally we show results from ablation studies to help understand the contributions of the different components. arXiv:1710.02298v1 [cs.AI] 6 Oct 2017


Background Reinforcement learning addresses the problem of an agent learning to act in an environment in order to maximize a scalar reward signal. No direct supervision is provided to the agent, for instance it is never directly told the best action. Agents and environments. At each discrete time step t = 0, 1, 2 . . ., the environment provides the agent with an observation St, the agent responds by selecting an action At, and then the environment provides the next reward Rt+1, discount γt+1, and state St+1. This interaction is formalized as a Markov Decision Process, or MDP, which is a tuple ⟨S, A, T, r, γ⟩, where S is a ﬁnite set of states, A is a ﬁnite set of actions, T(s, a, s′) = P[St+1 = s′ | St = s, At = a] is the (stochastic) transition function, r(s, a) = E[Rt+1 | St = s, At = a]


= s′ | St = s, At = a] is the (stochastic) transition function, r(s, a) = E[Rt+1 | St = s, At = a] is the reward function, and γ ∈[0, 1] is a discount factor. In our experiments MDPs will be episodic with a constant γt = γ, except on episode termination where γt = 0, but the algorithms are expressed in the general form. On the agent side, action selection is given by a policy π that deﬁnes a probability distribution over actions for each state. From the state St encountered at time t, we deﬁne the discounted return Gt = P∞ k=0 γ(k) t Rt+k+1 as the discounted sum of future rewards collected by the agent, where the discount for a reward k steps in the future is given by the product of discounts before that time, γ(k) t = Qk i=1 γt+i.


k steps in the future is given by the product of discounts before that time, γ(k) t = Qk i=1 γt+i. An agent aims to maximize the expected discounted return by ﬁnding a good policy. The policy may be learned directly, or it may be constructed as a function of some other learned quantities. In value-based reinforcement learning, the agent learns an estimate of the expected discounted return, or value, when following a policy π starting from a given state, vπ(s) = Eπ[Gt|St = s], or state-action pair, qπ(s, a) = Eπ[Gt|St = s, At = a]. A common way of deriving a new policy from a state-action value function is to act ϵ-greedily with respect to the action values. This corresponds to taking the action with the highest value (the greedy action) with probability (1−ϵ), and to otherwise act uniform


tion with the highest value (the greedy action) with probability (1−ϵ), and to otherwise act uniformly at random with probability ϵ. Policies of this kind are used to introduce a form of exploration: by randomly selecting actions that are sub-optimal according to its current estimates, the agent can discover and correct its estimates when appropriate. The main limitation is that it is difﬁcult to discover alternative courses of action that extend far into the future; this has motivated research on more directed forms of exploration. Deep reinforcement learning and DQN. Large state and/or action spaces make it intractable to learn Q value estimates for each state and action pair independently.


spaces make it intractable to learn Q value estimates for each state and action pair independently. In deep reinforcement learning, we represent the various components of agents, such as policies π(s, a) or values q(s, a), with deep (i.e., multi-layer) neural networks. The parameters of these networks are trained by gradient descent to minimize some suitable loss function. In DQN (Mnih et al. 2015) deep networks and reinforcement learning were successfully combined by using a convolutional neural net to approximate the action values for a given state St (which is fed as input to the network in the form of a stack of raw pixel frames). At each step, based on the current state, the agent selects an action ϵ-greedily with respect to the action values, and adds a transition (St, At, Rt+1, γt+1


ts an action ϵ-greedily with respect to the action values, and adds a transition (St, At, Rt+1, γt+1, St+1) to a replay memory buffer (Lin 1992), that holds the last million transitions. The parameters of the neural network are optimized by using stochastic gradient descent to minimize the loss (Rt+1 + γt+1 max a′ qθ(St+1, a′) −qθ(St, At))2 , (1) where t is a time step randomly picked from the replay memory. The gradient of the loss is back-propagated only into the parameters θ of the online network (which is also used to select actions); the term θ represents the parameters of a target network; a periodic copy of the online network which is not directly optimized. The optimization is performed using RMSprop (Tieleman and Hinton 2012), a variant of stochastic gradient descent, on mini-batc


med using RMSprop (Tieleman and Hinton 2012), a variant of stochastic gradient descent, on mini-batches sampled uniformly from the experience replay. This means that in the loss above, the time index t will be a random time index from the last million transitions, rather than the current time. The use of experience replay and target networks enables relatively stable learning of Q values, and led to superhuman performance on several Atari games. Extensions to DQN DQN has been an important milestone, but several limitations of this algorithm are now known, and many extensions have been proposed. We propose a selection of six extensions that each have addressed a limitation and improved overall performance.


selection of six extensions that each have addressed a limitation and improved overall performance. To keep the size of the selection manageable, we picked a set of extensions that address distinct concerns (e.g., just one of the many addressing exploration). Double Q-learning. Conventional Q-learning is affected by an overestimation bias, due to the maximization step in Equation 1, and this can harm learning. Double Q-learning (van Hasselt 2010), addresses this overestimation by decoupling, in the maximization performed for the bootstrap target, the selection of the action from its evaluation. It is possible to effectively combine this with DQN (van Hasselt, Guez, and Silver 2016), using the loss (Rt+1+γt+1qθ(St+1, argmax a′ qθ(St+1, a′))−qθ(St, At))2.


lt, Guez, and Silver 2016), using the loss (Rt+1+γt+1qθ(St+1, argmax a′ qθ(St+1, a′))−qθ(St, At))2. This change was shown to reduce harmful overestimations that were present for DQN, thereby improving performance. Prioritized replay. DQN samples uniformly from the replay buffer. Ideally, we want to sample more frequently those transitions from which there is much to learn. As a proxy for learning potential, prioritized experience replay (Schaul et al. 2015) samples transitions with probability pt relative to the last encountered absolute TD error: pt ∝ Rt+1 + γt+1 max a′ qθ(St+1, a′) −qθ(St, At) ω , where ω is a hyper-parameter that determines the shape of the distribution. New transitions are inserted into the replay


buffer with maximum priority, providing a bias towards recent transitions. Note that stochastic transitions might also be favoured, even when there is little left to learn about them. Dueling networks. The dueling network is a neural network architecture designed for value based RL. It features two streams of computation, the value and advantage streams, sharing a convolutional encoder, and merged by a special aggregator (Wang et al. 2016). This corresponds to the following factorization of action values: qθ(s, a) = vη(fξ(s)) + aψ(fξ(s), a) − P a′ aψ(fξ(s), a′) Nactions , where ξ, η, and ψ are, respectively, the parameters of the shared encoder fξ, of the value stream vη, and of the advantage stream aψ; and θ = {ξ, η, ψ} is their concatenation. Multi-step learning.


vη, and of the advantage stream aψ; and θ = {ξ, η, ψ} is their concatenation. Multi-step learning. Q-learning accumulates a single reward and then uses the greedy action at the next step to bootstrap. Alternatively, forward-view multi-step targets can be used (Sutton 1988). We deﬁne the truncated n-step return from a given state St as R(n) t ≡ n−1 X k=0 γ(k) t Rt+k+1 . (2) A multi-step variant of DQN is then deﬁned by minimizing the alternative loss, (R(n) t + γ(n) t max a′ qθ(St+n, a′) −qθ(St, At))2. Multi-step targets with suitably tuned n often lead to faster learning (Sutton and Barto 1998). Distributional RL. We can learn to approximate the distribution of returns instead of the expected return.


utional RL. We can learn to approximate the distribution of returns instead of the expected return. Recently Bellemare, Dabney, and Munos (2017) proposed to model such distributions with probability masses placed on a discrete support z, where z is a vector with Natoms ∈N+ atoms, deﬁned by zi = vmin + (i −1) vmax−vmin Natoms−1 for i ∈{1, . . . , Natoms}. The approximating distribution dt at time t is deﬁned on this support, with the probability mass pi θ(St, At) on each atom i, such that dt = (z, pθ(St, At)). The goal is to update θ such that this distribution closely matches the actual distribution of returns. To learn the probability masses, the key insight is that return distributions satisfy a variant of Bellman’s equation.


ility masses, the key insight is that return distributions satisfy a variant of Bellman’s equation. For a given state St and action At, the distribution of the returns under the optimal policy π∗should match a target distribution deﬁned by taking the distribution for the next state St+1 and action a∗ t+1 = π∗(St+1), contracting it towards zero according to the discount, and shifting it by the reward (or distribution of rewards, in the stochastic case). A distributional variant of Q-learning is then derived by ﬁrst constructing a new support for the target distribution, and then minimizing the Kullbeck-Leibler divergence between the distribution dt and the target distribution d′ t ≡(Rt+1 + γt+1z, pθ(St+1, a∗ t+1)), DKL(Φzd′ t||dt) .


stribution dt and the target distribution d′ t ≡(Rt+1 + γt+1z, pθ(St+1, a∗ t+1)), DKL(Φzd′ t||dt) . (3) Here Φz is a L2-projection of the target distribution onto the ﬁxed support z, and a∗ t+1 = argmaxa qθ(St+1, a) is the greedy action with respect to the mean action values qθ(St+1, a) = z⊤pθ(St+1, a) in state St+1. As in the non-distributional case, we can use a frozen copy of the parameters θ to construct the target distribution. The parametrized distribution can be represented by a neural network, as in DQN, but with Natoms ×Nactions outputs. A softmax is applied independently for each action dimension of the output to ensure that the distribution for each action is appropriately normalized. Noisy Nets.


the output to ensure that the distribution for each action is appropriately normalized. Noisy Nets. The limitations of exploring using ϵ-greedy policies are clear in games such as Montezuma’s Revenge, where many actions must be executed to collect the ﬁrst reward. Noisy Nets (Fortunato et al. 2017) propose a noisy linear layer that combines a deterministic and noisy stream, y = (b + Wx) + (bnoisy ⊙ϵb + (Wnoisy ⊙ϵw)x), (4) where ϵb and ϵw are random variables, and ⊙denotes the element-wise product. This transformation can then be used in place of the standard linear y = b + Wx. Over time, the network can learn to ignore the noisy stream, but will do so at different rates in different parts of the state space, allowing state-conditional exploration with a form of self-annealing.


ent parts of the state space, allowing state-conditional exploration with a form of self-annealing. The Integrated Agent In this paper we integrate all the aforementioned components into a single integrated agent, which we call Rainbow. First, we replace the 1-step distributional loss (3) with a multi-step variant. We construct the target distribution by contracting the value distribution in St+n according to the cumulative discount, and shifting it by the truncated n-step discounted return. This corresponds to deﬁning the target distribution as d(n) t = (R(n) t + γ(n) t z, pθ(St+n, a∗ t+n)). The resulting loss is DKL(Φzd(n) t ||dt) , where, again, Φz is the projection onto z. We combine the multi-step distributional loss with double Q-learning by using the greedy action in St+n selected a


multi-step distributional loss with double Q-learning by using the greedy action in St+n selected according to the online network as the bootstrap action a∗ t+n, and evaluating such action using the target network. In standard proportional prioritized replay (Schaul et al. 2015) the absolute TD error is used to prioritize the transitions. This can be computed in the distributional setting, using the mean action values. However, in our experiments all distributional Rainbow variants prioritize transitions by the KL loss, since this is what the algorithm is minimizing: pt ∝  DKL(Φzd(n) t ||dt) ω . The KL loss as priority might be more robust to noisy stochastic environments because the loss can continue to decrease even when the returns are not deterministic.


environments because the loss can continue to decrease even when the returns are not deterministic. The network architecture is a dueling network architecture adapted for use with return distributions. The network


has a shared representation fξ(s), which is then fed into a value stream vη with Natoms outputs, and into an advantage stream aξ with Natoms × Nactions outputs, where ai ξ(fξ(s), a) will denote the output corresponding to atom i and action a. For each atom zi, the value and advantage streams are aggregated, as in dueling DQN, and then passed through a softmax layer to obtain the normalised parametric distributions used to estimate the returns’ distributions: pi θ(s, a) = exp(vi η(φ) + ai ψ(φ, a) −ai ψ(s)) P j exp(vj η(φ) + aj ψ(φ, a) −aj ψ(s)) , where φ = fξ(s) and ai ψ(s) = 1 Nactions P a′ ai ψ(φ, a′). We then replace all linear layers with their noisy equivalent described in Equation (4). Within these noisy linear layers we use factorised Gaussian noise (Fortunato et al.


n Equation (4). Within these noisy linear layers we use factorised Gaussian noise (Fortunato et al. 2017) to reduce the number of independent noise variables. Experimental Methods We now describe the methods and setup used for conﬁguring and evaluating the learning agents. Evaluation Methodology. We evaluated all agents on 57 Atari 2600 games from the arcade learning environment (Bellemare et al. 2013). We follow the training and evaluation procedures of Mnih et al. (2015) and van Hasselt et al. (2016). The average scores of the agent are evaluated during training, every 1M steps in the environment, by suspending learning and evaluating the latest agent for 500K frames. Episodes are truncated at 108K frames (or 30 minutes of simulated play), as in van Hasselt et al. (2016).


es are truncated at 108K frames (or 30 minutes of simulated play), as in van Hasselt et al. (2016). Agents’ scores are normalized, per game, so that 0% corresponds to a random agent and 100% to the average score of a human expert. Normalized scores can be aggregated across all Atari levels to compare the performance of different agents. It is common to track the median human normalized performance across all games. We also consider the number of games where the agent’s performance is above some fraction of human performance, to disentangle where improvements in the median come from. The mean human normalized performance is potentially less informative, as it is dominated by a few games (e.g., Atlantis) where agents achieve scores orders of magnitude higher than humans do.


a few games (e.g., Atlantis) where agents achieve scores orders of magnitude higher than humans do. Besides tracking the median performance as a function of environment steps, at the end of training we re-evaluate the best agent snapshot using two different testing regimes. In the no-ops starts regime, we insert a random number (up to 30) of no-op actions at the beginning of each episode (as we do also in training). In the human starts regime, episodes are initialized with points randomly sampled from the initial portion of human expert trajectories (Nair et al. 2015); the difference between the two regimes indicates the extent to which the agent has over-ﬁt to its own trajectories. Due to space constraints, we focus on aggregate results across games.


r-ﬁt to its own trajectories. Due to space constraints, we focus on aggregate results across games. However, in the appendix we provide full learning curves for all games and all agents, as well as detailed comparison tables of raw and normalized scores, in both the no-op and human starts testing regimes. Hyper-parameter tuning. All Rainbow’s components have a number of hyper-parameters. The combinatorial space of hyper-parameters is too large for an exhaustive search, therefore we have performed limited tuning. For each component, we started with the values used in the paper that introduced this component, and tuned the most sensitive among hyper-parameters by manual coordinate descent. DQN and its variants do not perform learning updates during the ﬁrst 200K frames, to ensure sufﬁciently


and its variants do not perform learning updates during the ﬁrst 200K frames, to ensure sufﬁciently uncorrelated updates. We have found that, with prioritized replay, it is possible to start learning sooner, after only 80K frames. DQN starts with an exploration ϵ of 1, corresponding to acting uniformly at random; it anneals the amount of exploration over the ﬁrst 4M frames, to a ﬁnal value of 0.1 (lowered to 0.01 in later variants). Whenever using Noisy Nets, we acted fully greedily (ϵ = 0), with a value of 0.5 for the σ0 hyper-parameter used to initialize the weights in the noisy stream1. For agents without Noisy Nets, we used ϵ-greedy but decreased the exploration rate faster than was previously used, annealing ϵ to 0.01 in the ﬁrst 250K frames.


the exploration rate faster than was previously used, annealing ϵ to 0.01 in the ﬁrst 250K frames. We used the Adam optimizer (Kingma and Ba 2014), which we found less sensitive to the choice of the learning rate than RMSProp. DQN uses a learning rate of α = 0.00025 In all Rainbow’s variants we used a learning rate of α/4, selected among {α/2, α/4, α/6}, and a value of 1.5 × 10−4 for Adam’s ϵ hyper-parameter. For replay prioritization we used the recommended proportional variant, with priority exponent ω of 0.5, and linearly increased the importance sampling exponent β from 0.4 to 1 over the course of training. The priority exponent ω was tuned comparing values of {0.4, 0.5, 0.7}. Using the KL loss of distributional DQN as priority, we have observed that performance is very robust to the


KL loss of distributional DQN as priority, we have observed that performance is very robust to the choice of ω. The value of n in multi-step learning is a sensitive hyper-parameter of Rainbow. We compared values of n = 1, 3, and 5. We observed that both n = 3 and 5 did well initially, but overall n = 3 performed the best by the end. The hyper-parameters (see Table 1) are identical across all 57 games, i.e., the Rainbow agent really is a single agent setup that performs well across all the games. 1The noise was generated on the GPU. Tensorﬂow noise generation can be unreliable on GPU. If generating the noise on the CPU, lowering σ0 to 0.1 may be helpful. Parameter Value Min history to start learning 80K frames Adam learning rate 0.0000625 Exploration ϵ 0.0 Noisy Nets σ0 0.5 Target Network


learning 80K frames Adam learning rate 0.0000625 Exploration ϵ 0.0 Noisy Nets σ0 0.5 Target Network Period 32K frames Adam ϵ 1.5 × 10−4 Prioritization type proportional Prioritization exponent ω 0.5 Prioritization importance sampling β 0.4 →1.0 Multi-step returns n 3 Distributional atoms 51 Distributional min/max values [−10, 10] Table 1: Rainbow hyper-parameters


10 25 40 57 number of games #games > 20% human #games > 50% human #games > 100% human #games > 200% human #games > 500% human DQN DDQN Prioritized DDQN Dueling DDQN A3C Distributional DQN Noisy DQN Rainbow 0 50 100 150 200 Millions of frames 10 25 40 57 number of games 0 50 100 150 200 Millions of frames 0 50 100 150 200 Millions of frames 0 50 100 150 200 Millions of frames 0 50 100 150 200 Millions of frames DQN no double no priority no dueling no multi-step no distribution no noisy Rainbow Figure 2: Each plot shows, for several agents, the number of games where they have achieved at least a given fraction of human performance, as a function of time. From left to right we consider the 20%, 50%, 100%, 200% and 500% thresholds. On the ﬁrst row we compare Rainbow to the baselines.


the 20%, 50%, 100%, 200% and 500% thresholds. On the ﬁrst row we compare Rainbow to the baselines. On the second row we compare Rainbow to its ablations. Analysis In this section we analyse the main experimental results. First, we show that Rainbow compares favorably to several published agents. Then we perform ablation studies, comparing several variants of the agent, each corresponding to removing a single component from Rainbow. Comparison to published baselines. In Figure 1 we compare the Rainbow’s performance (measured in terms of the median human normalized score across games) to the corresponding curves for A3C, DQN, DDQN, Prioritized DDQN, Dueling DDQN, Distributional DQN, and Noisy DQN.


nding curves for A3C, DQN, DDQN, Prioritized DDQN, Dueling DDQN, Distributional DQN, and Noisy DQN. We thank the authors of the Dueling and Prioritized agents for providing the learning curves of these, and report our own re-runs for DQN, A3C, DDQN, Distributional DQN and Noisy DQN. The performance of Rainbow is signiﬁcantly better than any of the baselines, both in data efﬁciency, as well as in ﬁnal performance. Note that we match ﬁnal performance of DQN after 7M frames, surpass the best ﬁnal performance of these baselines in 44M frames, and reach substantially improved ﬁnal performance. In the ﬁnal evaluations of the agent, after the end of training, Rainbow achieves a median score of 223% in the no-ops regime; in the human starts regime we measured a median score of 153%.


score of 223% in the no-ops regime; in the human starts regime we measured a median score of 153%. In Table 2 we compare these scores to the published median scores of the individual baselines. In Figure 2 (top row) we plot the number of games where an agent has reached some speciﬁed level of human normalized performance. From left to right, the subplots show on how many games the different agents have achieved 20%, 50%, 100%, 200% and 500% human normalized performance. This allows us to identify where the overall improvements in performance come from. Note that the gap in performance between Rainbow and other agents is apparent at all levels of performance: the Rainbow agent is improving scores on games where the baseline agents were already good, as well as improving in games where base


cores on games where the baseline agents were already good, as well as improving in games where baseline agents are still far from human performance. Learning speed. As in the original DQN setup, we ran each agent on a single GPU. The 7M frames required to match DQN’s ﬁnal performance correspond to less than 10 hours of wall-clock time. A full run of 200M frames corresponds to approximately 10 days, and this varies by less than 20% between all of the discussed variants. The literaAgent no-ops human starts DQN 79% 68% DDQN (*) 117% 110% Prioritized DDQN (*) 140% 128% Dueling DDQN (*) 151% 117% A3C (*) 116% Noisy DQN 118% 102% Distributional DQN 164% 125% Rainbow 223% 153% Table 2: Median normalized scores of the best agent snapshots for Rainbow and baselines.


223% 153% Table 2: Median normalized scores of the best agent snapshots for Rainbow and baselines. For methods marked with an asterisk, the scores come from the corresponding publication. DQN’s scores comes from the dueling networks paper, since DQN’s paper did not report scores for all 57 games. The others scores come from our own implementations.


50 100 150 200 Millions of frames 0% 100% 200% Median normalized score DQN no double no priority no dueling no multi-step no distribution no noisy Rainbow Figure 3: Median human-normalized performance across 57 Atari games, as a function of time. We compare our integrated agent (rainbow-colored) to DQN (gray) and to six different ablations (dashed lines). Curves are smoothed with a moving average over 5 points. ture contains many alternative training setups that improve performance as a function of wall-clock time by exploiting parallelism, e.g., Nair et al. (2015), Salimans et al. (2017), and Mnih et al. (2016). Properly relating the performance across such very different hardware/compute resources is non-trivial, so we focused exclusively on algorithmic variations, allowing apples-to-app


esources is non-trivial, so we focused exclusively on algorithmic variations, allowing apples-to-apples comparisons. While we consider them to be important and complementary, we leave questions of scalability and parallelism to future work. Ablation studies. Since Rainbow integrates several different ideas into a single agent, we conducted additional experiments to understand the contribution of the various components, in the context of this speciﬁc combination. To gain a better understanding of the contribution of each component to the Rainbow agent, we performed ablation studies. In each ablation, we removed one component from the full Rainbow combination. Figure 3 shows a comparison for median normalized score of the full Rainbow to six ablated variants.


igure 3 shows a comparison for median normalized score of the full Rainbow to six ablated variants. Figure 2 (bottom row) shows a more detailed breakdown of how these ablations perform relative to different thresholds of human normalized performance, and Figure 4 shows the gain or loss from each ablation for every game, averaged over the full learning run. Prioritized replay and multi-step learning were the two most crucial components of Rainbow, in that removing either component caused a large drop in median performance. Unsurprisingly, the removal of either of these hurt early performance. Perhaps more surprisingly, the removal of multistep learning also hurt ﬁnal performance. Zooming in on individual games (Figure 4), we see both components helped almost uniformly across games (the full


n individual games (Figure 4), we see both components helped almost uniformly across games (the full Rainbow performed better than either ablation in 53 games out of 57). Distributional Q-learning ranked immediately below the previous techniques for relevance to the agent’s performance. Notably, in early learning no difference is apparent, as shown in Figure 3, where for the ﬁrst 40 million frames the distributional-ablation performed as well as the full agent. However, without distributions, the performance of the agent then started lagging behind. When the results are separated relatively to human performance in Figure 2, we see that the distributional-ablation primarily seems to lags on games that are above human level or near it.


the distributional-ablation primarily seems to lags on games that are above human level or near it. In terms of median performance, the agent performed better when Noisy Nets were included; when these are removed and exploration is delegated to the traditional ϵgreedy mechanism, performance was worse in aggregate (red line in Figure 3). While the removal of Noisy Nets produced a large drop in performance for several games, it also provided small increases in other games (Figure 4). In aggregate, we did not observe a signiﬁcant difference when removing the dueling network from the full Rainbow. The median score, however, hides the fact that the impact of Dueling differed between games, as shown by Figure 4.


e, however, hides the fact that the impact of Dueling differed between games, as shown by Figure 4. Figure 2 shows that Dueling perhaps provided some improvement on games with above-human performance levels (# games > 200%), and some degradation on games with sub-human performance (# games > 20%). Also in the case of double Q-learning, the observed difference in median performance (Figure 3) is limited, with the component sometimes harming or helping depending on the game (Figure 4). To further investigate the role of double Qlearning, we compared the predictions of our trained agents to the actual discounted returns computed from clipped rewards. Comparing Rainbow to the agent where double Qlearning was ablated, we observed that the actual returns are often higher than 10 and therefore fa


Qlearning was ablated, we observed that the actual returns are often higher than 10 and therefore fall outside the support of the distribution, spanning from −10 to +10. This leads to underestimated returns, rather than overestimations. We hypothesize that clipping the values to this constrained range counteracts the overestimation bias of Q-learning. Note, however, that the importance of double Q-learning may increase if the support of the distributions is expanded. In the appendix, for each game we show ﬁnal performance and learning curves for Rainbow, its ablations, and baselines. Discussion We have demonstrated that several improvements to DQN can be successfully integrated into a single learning algorithm that achieves state-of-the-art performance.


uccessfully integrated into a single learning algorithm that achieves state-of-the-art performance. Moreover, we have shown that within the integrated algorithm, all but one of the components provided clear performance beneﬁts. There are many more algorithmic components that we were not able to include, which would be promising candidates for further experiments on integrated agents. Among the many possible candidates, we discuss several below. We have focused here on value-based methods in the Q-learning family. We have not considered purely policybased RL algorithms such as trust-region policy optimisa


DQN Rainbow no noisy DQN Rainbow no distribution DQN Rainbow no multi-step DQN Rainbow no dueling DQN Rainbow no priority alien amidar assault asterix asteroids atlantis bank_heist battle_zone beam_rider berzerk boxing breakout centipede chopper_command crazy_climber defender demon_attack double_dunk enduro fishing_derby freeway frostbite gopher gravitar hero ice_hockey jamesbond kangaroo krull kung_fu_master montezuma_revenge ms_pacman name_this_game phoenix pitfall pong private_eye qbert riverraid road_runner robotank seaquest skiing solaris space_invaders star_gunner surround tennis time_pilot tutankham up_n_down video_pinball wizard_of_wor yars_revenge zaxxon DQN Rainbow no double Figure 4: Performance drops of ablation agents on all 57 Atari games.


zaxxon DQN Rainbow no double Figure 4: Performance drops of ablation agents on all 57 Atari games. Performance is the area under the learning curve, normalized relative to the Rainbow agent and DQN. Two games where DQN outperforms Rainbow are omitted. The ablation leading to the strongest drop is highlighted for each game. The removal of either prioritization or multi-step learning reduces performance across most games, but the contribution of each component varies substantially per game. tion (Schulman et al. 2015), nor actor-critic methods (Mnih et al. 2016; O’Donoghue et al. 2016). A number of algorithms exploit a sequence of data to achieve improved learning efﬁciency. Optimality tightening (He et al.


exploit a sequence of data to achieve improved learning efﬁciency. Optimality tightening (He et al. 2016) uses multi-step returns to construct additional inequality bounds, instead of using them to replace the 1-step targets used in Q-learning. Eligibility traces allow a soft combination over n-step returns (Sutton 1988). However, sequential methods all leverage more computation per gradient than the multi-step targets used in Rainbow. Furthermore, introducing prioritized sequence replay raises questions of how to store, replay and prioritise sequences. Episodic control (Blundell et al. 2016) also focuses on data efﬁciency, and was shown to be very effective in some domains. It improves early learning by using episodic memory as a complementary learning system, capable of immediately re-en


y learning by using episodic memory as a complementary learning system, capable of immediately re-enacting successful action sequences. Besides Noisy Nets, numerous other exploration methods could also be useful algorithmic ingredients: among these Bootstrapped DQN (Osband et al. 2016), intrinsic motivation (Stadie, Levine, and Abbeel 2015) and count-based exploration (Bellemare et al. 2016). Integration of these alternative components is fruitful subject for further research. In this paper we have focused on the core learning updates, without exploring alternative computational architectures. Asynchronous learning from parallel copies of the environment, as in A3C (Mnih et al. 2016), Gorila (Nair et al. 2015), or Evolution Strategies (Salimans et al.


, as in A3C (Mnih et al. 2016), Gorila (Nair et al. 2015), or Evolution Strategies (Salimans et al. 2017), can be effective in speeding up learning, at least in terms of wallclock time. Note, however, they can be less data efﬁcient. Hierarchical RL has also been applied with success to several complex Atari games. Among successful applications of HRL we highlight h-DQN (Kulkarni et al. 2016a) and Feudal Networks (Vezhnevets et al. 2017). The state representation could also be made more efﬁcient by exploiting auxiliary tasks such as pixel control or feature control (Jaderberg et al. 2016), supervised predictions (Dosovitskiy and Koltun 2016) or successor features (Kulkarni et al. 2016b). To evaluate Rainbow fairly against the baselines, we have followed the common domain modiﬁcations of cli


valuate Rainbow fairly against the baselines, we have followed the common domain modiﬁcations of clipping rewards, ﬁxed action-repetition, and frame-stacking, but these might be removed by other learning algorithm improvements. Pop-Art normalization (van Hasselt et al. 2016) allows reward clipping to be removed, while preserving a similar level of performance. Fine-grained action repetition (Sharma, Lakshminarayanan, and Ravindran 2017) enabled to learn how to repeat actions. A recurrent state network


(Hausknecht and Stone 2015) can learn a temporal state representation, replacing the ﬁxed stack of observation frames. In general, we believe that exposing the real game to the agent is a promising direction for future research. References Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res. (JAIR) 47:253– 279. Bellemare, M. G.; Srinivasan, S.; Ostrovski, G.; Schaul, T.; Saxton, D.; and Munos, R. 2016. Unifying count-based exploration and intrinsic motivation. In NIPS. Bellemare, M. G.; Dabney, W.; and Munos, R. 2017. A distributional perspective on reinforcement learning. In ICML. Blundell, C.; Uria, B.; Pritzel, A.; Li, Y.; Ruderman, A.; Leibo, J.


inforcement learning. In ICML. Blundell, C.; Uria, B.; Pritzel, A.; Li, Y.; Ruderman, A.; Leibo, J. Z.; Rae, J.; Wierstra, D.; and Hassabis, D. 2016. Model-Free Episodic Control. ArXiv e-prints. Dosovitskiy, A., and Koltun, V. 2016. Learning to act by predicting the future. CoRR abs/1611.01779. Fortunato, M.; Azar, M. G.; Piot, B.; Menick, J.; Osband, I.; Graves, A.; Mnih, V.; Munos, R.; Hassabis, D.; Pietquin, O.; Blundell, C.; and Legg, S. 2017. Noisy networks for exploration. CoRR abs/1706.10295. Hausknecht, M., and Stone, P. 2015. Deep recurrent Qlearning for partially observable MDPs. arXiv preprint arXiv:1507.06527. He, F. S.; Liu, Y.; Schwing, A. G.; and Peng, J. 2016. Learning to play in a day: Faster deep reinforcement learning by optimality tightening. CoRR abs/1611.01606.


to play in a day: Faster deep reinforcement learning by optimality tightening. CoRR abs/1611.01606. Jaderberg, M.; Mnih, V.; Czarnecki, W. M.; Schaul, T.; Leibo, J. Z.; Silver, D.; and Kavukcuoglu, K. 2016. Reinforcement learning with unsupervised auxiliary tasks. CoRR abs/1611.05397. Kingma, D. P., and Ba, J. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR). Kulkarni, T. D.; Narasimhan, K.; Saeedi, A.; and Tenenbaum, J. B. 2016a. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. CoRR abs/1604.06057. Kulkarni, T. D.; Saeedi, A.; Gautam, S.; and Gershman, S. J. 2016b. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396. Lin, L.-J.


an, S. J. 2016b. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396. Lin, L.-J. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning 8(3):293–321. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. A. 2013. Playing atari with deep reinforcement learning. CoRR abs/1312.5602. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529–533. Mnih, V.; Badia, A.


an-level control through deep reinforcement learning. Nature 518(7540):529–533. Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning. Nair, A.; Srinivasan, P.; Blackwell, S.; Alcicek, C.; Fearon, R.; De Maria, A.; Panneershelvam, V.; Suleyman, M.; Beattie, C.; Petersen, S.; Legg, S.; Mnih, V.; Kavukcuoglu, K.; and Silver, D. 2015. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296. O’Donoghue, B.; Munos, R.; Kavukcuoglu, K.; and Mnih, V. 2016. Pgq: Combining policy gradient and q-learning. CoRR abs/1611.01626. Osband, I.; Blundell, C.; Pritzel, A.; and Roy, B. V. 2016.


nt and q-learning. CoRR abs/1611.01626. Osband, I.; Blundell, C.; Pritzel, A.; and Roy, B. V. 2016. Deep exploration via bootstrapped dqn. In NIPS. Salimans, T.; Ho, J.; Chen, X.; and Sutskever, I. 2017. Evolution strategies as a scalable alternative to reinforcement learning. CoRR abs/1703.03864. Schaul, T.; Quan, J.; Antonoglou, I.; and Silver, D. 2015. Prioritized experience replay. In Proc. of ICLR. Schulman, J.; Levine, S.; Moritz, P.; Jordan, M.; and Abbeel, P. 2015. Trust region policy optimization. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, 1889– 1897. JMLR.org. Sharma, S.; Lakshminarayanan, A. S.; and Ravindran, B. 2017.


ume 37, ICML’15, 1889– 1897. JMLR.org. Sharma, S.; Lakshminarayanan, A. S.; and Ravindran, B. 2017. Learning to repeat: Fine grained action repetition for deep reinforcement learning. arXiv preprint arXiv:1702.06054. Stadie, B. C.; Levine, S.; and Abbeel, P. 2015. Incentivizing exploration in reinforcement learning with deep predictive models. CoRR abs/1507.00814. Sutton, R. S., and Barto, A. G. 1998. Reinforcement Learning: An Introduction. The MIT press, Cambridge MA. Sutton, R. S. 1988. Learning to predict by the methods of temporal differences. Machine learning 3(1):9–44. Tieleman, T., and Hinton, G. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26–31.


running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26–31. van Hasselt, H.; Guez, A.; Guez, A.; Hessel, M.; Mnih, V.; and Silver, D. 2016. Learning values across many orders of magnitude. In Advances in Neural Information Processing Systems 29, 4287–4295. van Hasselt, H.; Guez, A.; and Silver, D. 2016. Deep reinforcement learning with double Q-learning. In Proc. of AAAI, 2094–2100. van Hasselt, H. 2010. Double Q-learning. In Advances in Neural Information Processing Systems 23, 2613–2621. Vezhnevets, A. S.; Osindero, S.; Schaul, T.; Heess, N.; Jaderberg, M.; Silver, D.; and Kavukcuoglu, K. 2017. Feudal networks for hierarchical reinforcement learning. CoRR abs/1703.01161.


Wang, Z.; Schaul, T.; Hessel, M.; van Hasselt, H.; Lanctot, M.; and de Freitas, N. 2016. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, 1995– 2003.


Appendix Table 3 lists the preprocessing of environment frames, rewards and discounts introduced by DQN. Table 4 lists the additional hyper-parameters that Rainbow inherits from DQN and the other baselines considered in this paper. The hyper-parameters for which Rainbow uses non standard settings are instead listed in the main text. In the subsequent pages, we list the tables showing, for each game, the score achieved by Rainbow and several baselines in both the no-ops regime (Table 6) and the human-starts regime (Table 5). In Figures 5 and 6 we also plot, for each game, the learning curves of Rainbow, several baselines, and all ablation experiments. These learning curves are smoothed with a moving average over a window of 10.


ablation experiments. These learning curves are smoothed with a moving average over a window of 10. Hyper-parameter value Grey-scaling True Observation down-sampling (84, 84) Frames stacked 4 Action repetitions 4 Reward clipping [-1, 1] Terminal on loss of life True Max frames per episode 108K Table 3: Preprocessing: the values of these hyper-parameters are the same used by DQN and its variants. They are here listed for completeness. Observations are grey-scaled and rescaled to 84 × 84 pixels. 4 consecutive frames are concatenated as each state’s representation. Each action selected by the agent is repeated for 4 times. Rewards are clipped between −1, +1. In games where the player has multiple lives, transitions associated to the loss of a life are considered terminal.


he player has multiple lives, transitions associated to the loss of a life are considered terminal. All episodes are capped after 108K frames. Hyper-parameter value Q network: channels 32, 64, 64 Q network: ﬁlter size 8 × 8, 4 × 4, 3 × 3 Q network: stride 4, 2, 1 Q network: hidden units 512 Q network: output units Number of actions Discount factor 0.99 Memory size 1M transitions Replay period every 4 agent steps Minibatch size 32 Table 4: Additional hyper-parameters: the values of these hyper-parameters are the same used by DQN and it’s variants. The network has 3 convolutional layers: with 32, 64 and 64 channels. The layers use 8 × 8, 4 × 4, 3 × 3 ﬁlters with strides of 4, 2, 1, respectively.


4 and 64 channels. The layers use 8 × 8, 4 × 4, 3 × 3 ﬁlters with strides of 4, 2, 1, respectively. The value and advantage streams of the dueling architecture have both a hidden layer with 512 units. The output layer of the network has a number of units equal to the number of actions available in the game. We use a discount factor of 0.99, which is set to 0 on terminal transitions. We perform a learning update every 4 agent steps, using mini-batches of 32 transitions.


Game DQN A3C DDQN Prior. DDQN Duel. DDQN Distrib. DQN Noisy DQN Rainbow alien 634.0 518.4 1033.4 900.5 1,486.5 1,997.5 533.3 6,022.9 amidar 178.4 263.9 169.1 218.4 172.7 237.7 148.0 202.8 assault 3489.3 5474.9 6060.8 7,748.5 3,994.8 5,101.3 5,124.3 14,491.7 asterix 3170.5 22140.5 16837.0 31,907.5 15,840.0 395,599.5 8,277.3 280,114.0 asteroids 1458.7 4474.5 1193.2 1,654.0 2,035.4 2,071.7 4,078.1 2,249.4 atlantis 292491.0 911,091.0 319688.0 593,642.0 445,360.0 289,803.0 303,666.5 814,684.0 bank heist 312.7 970.1 886.0 816.8 1,129.3 835.6 955.0 826.0 battle zone 23750.0 12950.0 24740.0 29,100.0 31,320.0 32,250.0 26,985.0 52,040.0 beam rider 9743.2 22707.9 17417.2 26,172.7 14,591.3 15,002.4 15,241.5 21,768.5 berzerk 493.4 817.9 1011.1 1,165.6 910.6 1,000.0 670.8 1,793.4 bowling 56.5 35.1 69.6


41.5 21,768.5 berzerk 493.4 817.9 1011.1 1,165.6 910.6 1,000.0 670.8 1,793.4 bowling 56.5 35.1 69.6 65.8 65.7 76.8 79.3 39.4 boxing 70.3 59.8 73.5 68.6 77.3 62.1 66.3 54.9 breakout 354.5 681.9 368.9 371.6 411.6 548.7 423.3 379.5 centipede 3973.9 3755.8 3853.5 3,421.9 4,881.0 7,476.9 4,214.4 7,160.9 chopper command 5017.0 7021.0 3495.0 6,604.0 3,784.0 9,600.5 8,778.5 10,916.0 crazy climber 98128.0 112646.0 113782.0 131,086.0 124,566.0 154,416.5 98,576.5 143,962.0 defender 15917.5 56533.0 27510.0 21,093.5 33,996.0 32,246.0 18,037.5 47,671.3 demon attack 12550.7 113,308.4 69803.4 73,185.8 56,322.8 109,856.6 25,207.8 109,670.7 double dunk -6.0 -0.1 -0.3 2.7 -0.8 -3.7 -1.0 -0.6 enduro 626.7 -82.5 1216.6 1,884.4 2,077.4 2,133.4 1,021.5 2,061.1 ﬁshing derby -1.6 18.8 3.2 9.2 -4.1 -4.9 -3.7 22.6 f


1216.6 1,884.4 2,077.4 2,133.4 1,021.5 2,061.1 ﬁshing derby -1.6 18.8 3.2 9.2 -4.1 -4.9 -3.7 22.6 freeway 26.9 0.1 28.8 27.9 0.2 28.8 27.1 29.1 frostbite 496.1 190.5 1448.1 2,930.2 2,332.4 2,813.9 418.8 4,141.1 gopher 8190.4 10022.8 15253.0 57,783.8 20,051.4 27,778.3 13,131.0 72,595.7 gravitar 298.0 303.5 200.5 218.0 297.0 422.0 250.5 567.5 hero 14992.9 32464.1 14892.5 20,506.4 15,207.9 28,554.2 2,454.2 50,496.8 ice hockey -1.6 -2.8 -2.5 -1.0 -1.3 -0.1 -2.4 -0.7 kangaroo 4496.0 94.0 11204.0 10,241.0 10,334.0 9,555.5 7,465.0 10,841.0 krull 6206.0 5560.0 6796.1 7,406.5 8,051.6 6,757.8 6,833.5 6,715.5 kung fu master 20882.0 28819.0 30207.0 31,244.0 24,288.0 33,890.0 27,921.0 28,999.8 montezuma revenge 47.0 67.0 42.0 13.0 22.0 130.0 55.0 154.0 ms pacman 1092.3 653.7 1241.3 1,824.6 2,250.6 2,0


revenge 47.0 67.0 42.0 13.0 22.0 130.0 55.0 154.0 ms pacman 1092.3 653.7 1241.3 1,824.6 2,250.6 2,064.1 1,012.1 2,570.2 name this game 6738.8 10476.1 8960.3 11,836.1 11,185.1 11,382.3 7,186.4 11,686.5 phoenix 7484.8 52894.1 12366.5 27,430.1 20,410.5 31,358.3 15,505.0 103,061.6 pitfall -113.2 -78.5 -186.7 -14.8 -46.9 -342.8 -154.4 -37.6 pong 18.0 5.6 19.1 18.9 18.8 18.9 18.0 19.0 private eye 207.9 206.9 -575.5 179.0 292.6 5,717.5 5,955.4 1,704.4 qbert 9271.5 15148.8 11020.8 11,277.0 14,175.8 15,035.9 9,176.6 18,397.6 road runner 35215.0 34216.0 43156.0 56,990.0 58,549.0 56,086.0 35,376.5 54,261.0 robotank 58.7 32.8 59.1 55.4 62.0 49.8 50.9 55.2 seaquest 4216.7 2355.4 14498.0 39,096.7 37,361.6 3,275.4 2,353.1 19,176.0 skiing -12142.1 -10911.1 -11490.4 -10,852.8 -11,928.0 -13,247.7 -13,905.9


6 3,275.4 2,353.1 19,176.0 skiing -12142.1 -10911.1 -11490.4 -10,852.8 -11,928.0 -13,247.7 -13,905.9 -11,685.8 solaris 1295.4 1956.0 810.0 2,238.2 1,768.4 2,530.2 2,608.2 2,860.7 space invaders 1293.8 15,730.5 2628.7 9,063.0 5,993.1 6,368.6 1,697.2 12,629.0 star gunner 52970.0 138218.0 58365.0 51,959.0 90,804.0 67,054.5 31,864.5 123,853.0 surround -6.0 -9.7 1.9 -0.9 4.0 4.5 -3.1 7.0 tennis 11.1 -6.3 -7.8 -2.0 4.4 22.6 -2.1 -2.2 time pilot 4786.0 12,679.0 6608.0 7,448.0 6,601.0 7,684.5 5,311.0 11,190.5 tutankham 45.6 156.3 92.2 33.6 48.0 124.3 123.3 126.9 venture 136.0 23.0 21.0 244.0 200.0 462.0 10.5 45.0 video pinball 154414.1 331628.1 367823.7 374,886.9 110,976.2 455,052.7 241,851.7 506,817.2 wizard of wor 1609.0 17,244.0 6201.0 7,451.0 7,054.0 11,824.5 4,796.5 14,631.5 yars revenge 4577


7.2 wizard of wor 1609.0 17,244.0 6201.0 7,451.0 7,054.0 11,824.5 4,796.5 14,631.5 yars revenge 4577.5 7157.5 6270.6 5,965.1 25,976.5 8,267.7 5,487.3 93,007.9 zaxxon 4412.0 24,622.0 8593.0 9,501.0 10,164.0 15,130.0 7,650.5 19,658.0 Table 5: Human Starts evaluation regime: Raw scores across all games, averaged over 200 testing episodes, from the agent snapshot that obtained the highest score during training. We report the published scores for DQN, A3C, DDQN, Dueling DDQN, and Prioritized DDQN. For Distributional DQN and Rainbow we report our own evaluations of the agents.


Game DQN DDQN Prior. DDQN Duel. DDQN Distrib. DQN Noisy DQN Rainbow alien 1620.0 3747.7 6,648.6 4,461.4 4,055.8 2,394.9 9,491.7 amidar 978.0 1793.3 2,051.8 2,354.5 1,267.9 1,608.0 5,131.2 assault 4280.0 5393.2 7,965.7 4,621.0 5,909.0 5,198.6 14,198.5 asterix 4359.0 17356.5 41,268.0 28,188.0 400,529.5 12,403.8 428,200.3 asteroids 1364.5 734.7 1,699.3 2,837.7 2,354.7 4,814.1 2,712.8 atlantis 279987.0 106056.0 427,658.0 382,572.0 273,895.0 329,010.0 826,659.5 bank heist 455.0 1030.6 1,126.8 1,611.9 1,056.7 1,323.0 1,358.0 battle zone 29900.0 31700.0 38,130.0 37,150.0 41,145.0 32,050.0 62,010.0 beam rider 8627.5 13772.8 22,430.7 12,164.0 13,213.4 12,534.0 16,850.2 berzerk 585.6 1225.4 1,614.2 1,472.6 1,421.8 837.3 2,545.6 bowling 50.4 68.1 62.6 65.5 74.1 77.3 30.0 boxing 88.0 91.6 98.8 99.4 98


472.6 1,421.8 837.3 2,545.6 bowling 50.4 68.1 62.6 65.5 74.1 77.3 30.0 boxing 88.0 91.6 98.8 99.4 98.1 83.3 99.6 breakout 385.5 418.5 381.5 345.3 612.5 459.1 417.5 centipede 4657.7 5409.4 5,175.4 7,561.4 9,015.5 4,355.8 8,167.3 chopper command 6126.0 5809.0 5,135.0 11,215.0 13,136.0 9,519.0 16,654.0 crazy climber 110763.0 117282.0 183,137.0 143,570.0 178,355.0 118,768.0 168,788.5 defender 23633.0 35338.5 24,162.5 42,214.0 37,896.8 23,083.0 55,105.0 demon attack 12149.4 58044.2 70,171.8 60,813.3 110,626.5 24,950.1 111,185.2 double dunk -6.6 -5.5 4.8 0.1 -3.8 -1.8 -0.3 enduro 729.0 1211.8 2,155.0 2,258.2 2,259.3 1,129.2 2,125.9 ﬁshing derby -4.9 15.5 30.2 46.4 9.1 7.7 31.3 freeway 30.8 33.3 32.9 0.0 33.6 32.0 34.0 frostbite 797.4 1683.3 3,421.6 4,672.8 3,938.2 583.6 9,590.5 gopher 8777.4 148


.9 0.0 33.6 32.0 34.0 frostbite 797.4 1683.3 3,421.6 4,672.8 3,938.2 583.6 9,590.5 gopher 8777.4 14840.8 49,097.4 15,718.4 28,841.0 15,107.9 70,354.6 gravitar 473.0 412.0 330.5 588.0 681.0 443.5 1,419.3 hero 20437.8 20130.2 27,153.9 20,818.2 33,860.9 5,053.1 55,887.4 ice hockey -1.9 -2.7 0.3 0.5 1.3 -2.1 1.1 kangaroo 7259.0 12992.0 14,492.0 14,854.0 12,909.0 12,117.0 14,637.5 krull 8422.3 7920.5 10,263.1 11,451.9 9,885.9 9,061.9 8,741.5 kung fu master 26059.0 29710.0 43,470.0 34,294.0 43,009.0 34,099.0 52,181.0 montezuma revenge 0.0 0.0 0.0 0.0 367.0 0.0 384.0 ms pacman 3085.6 2711.4 4,751.2 6,283.5 3,769.2 2,501.6 5,380.4 name this game 8207.8 10616.0 13,439.4 11,971.1 12,983.6 8,332.4 13,136.0 phoenix 8485.2 12252.5 32,808.3 23,092.2 34,775.0 16,974.3 108,528.6 pitfall -286.1 -29.9 0.0 0


36.0 phoenix 8485.2 12252.5 32,808.3 23,092.2 34,775.0 16,974.3 108,528.6 pitfall -286.1 -29.9 0.0 0.0 -2.1 -18.2 0.0 pong 19.5 20.9 20.7 21.0 20.8 21.0 20.9 private eye 146.7 129.7 200.0 103.0 15,172.9 3,966.0 4,234.0 qbert 13117.3 15088.5 18,802.8 19,220.3 16,956.0 15,276.3 33,817.5 road runner 39544.0 44127.0 62,785.0 69,524.0 63,366.0 41,681.0 62,041.0 robotank 63.9 65.1 58.6 65.3 54.2 53.5 61.4 seaquest 5860.6 16452.7 44,417.4 50,254.2 4,754.4 2,495.4 15,898.9 skiing -13062.3 -9021.8 -9,900.5 -8,857.4 -14,959.8 -16,307.3 -12,957.8 solaris 3482.8 3067.8 1,710.8 2,250.8 5,643.1 3,204.5 3,560.3 space invaders 1692.3 2525.5 7,696.9 6,427.3 6,869.1 2,145.5 18,789.0 star gunner 54282.0 60142.0 56,641.0 89,238.0 69,306.5 34,504.5 127,029.0 surround -5.6 -2.9 2.1 4.4 6.2 -3.3 9.7 tennis 12.2


0 56,641.0 89,238.0 69,306.5 34,504.5 127,029.0 surround -5.6 -2.9 2.1 4.4 6.2 -3.3 9.7 tennis 12.2 -22.8 0.0 5.1 23.6 0.0 -0.0 time pilot 4870.0 8339.0 11,448.0 11,666.0 7,875.0 6,157.0 12,926.0 tutankham 68.1 218.4 87.2 211.4 249.4 231.6 241.0 venture 163.0 98.0 863.0 497.0 1,107.0 0.0 5.5 video pinball 196760.4 309941.9 406,420.4 98,209.5 478,646.7 270,444.6 533,936.5 wizard of wor 2704.0 7492.0 10,373.0 7,855.0 15,994.5 5,432.0 17,862.5 yars revenge 18089.9 11712.6 16,451.7 49,622.1 16,608.6 9,570.1 102,557.0 zaxxon 5363.0 10163.0 13,490.0 12,944.0 18,347.5 9,390.0 22,209.5 Table 6: No-op starts evaluation regime: Raw scores across all games, averaged over 200 testing episodes, from the agent snapshot that obtained the highest score during training.


over 200 testing episodes, from the agent snapshot that obtained the highest score during training. We report the published scores for DQN, DDQN, Dueling DDQN, and Prioritized DDQN. For Distributional DQN and Rainbow we report our own evaluations of the agents. A3C is not listed since the paper did not report the scores for the no-ops regime.


0.0 0.5 1.0 1e4 alien 0 2 4 6 1e3 amidar 0.0 0.5 1.0 1.5 2.0 1e4 assault 0 2 4 6 8 1e5 asterix 0 2 4 1e3 asteroids 0.0 0.5 1.0 1.5 1e6 atlantis 0.0 0.5 1.0 1.5 1e3 bank_heist 0 2 4 6 8 1e4 battle_zone 0.0 0.5 1.0 1.5 2.0 1e4 beam_rider 0 1 2 3 1e3 berzerk 0 20 40 60 80 bowling 100 50 0 50 100 boxing 0 200 400 600 800 breakout 0.0 0.5 1.0 1e4 centipede 0.0 0.5 1.0 1.5 2.0 1e4chopper_command 0.0 0.5 1.0 1.5 2.0 1e5 crazy_climber 0 2 4 6 1e4 defender 0.0 0.5 1.0 1.5 1e5 demon_attack 30 20 10 0 10 double_dunk 0 1 2 3 4 1e3 enduro 100 50 0 50 fishing_derby 0 10 20 30 40 freeway 0.0 0.5 1.0 1e4 frostbite 0 2 4 6 8 1e4 gopher 0 1 2 3 1e3 gravitar 0 2 4 6 1e4 hero 30 20 10 0 10 ice_hockey 0 1 2 3 1e4 jamesbond 0.0 0.5 1.0 1.5 1e4 kangaroo 0.0 0.5 1.0 1e4 krull 0 2 4 6 1e4 kung_fu_master 0 100 200


4 jamesbond 0.0 0.5 1.0 1.5 1e4 kangaroo 0.0 0.5 1.0 1e4 krull 0 2 4 6 1e4 kung_fu_master 0 100 200 300 400 montezuma_revenge 0 2 4 6 1e3 ms_pacman 0.0 0.5 1.0 1.5 1e4 name_this_game 0 1 2 3 1e5 phoenix 1.5 1.0 0.5 0.0 1e3 pitfall 40 20 0 20 40 pong 1 0 1 2 1e4 private_eye 0 1 2 3 4 1e4 qbert 0 1 2 3 1e4 riverraid 0 2 4 6 8 1e4 road_runner 0 20 40 60 80 robotank 0 1 2 3 4 1e4 seaquest 4 3 2 1 0 1e4 skiing 0 1 2 3 4 1e3 solaris 0 1 2 3 1e4 space_invaders 0.0 0.5 1.0 1.5 1e5 star_gunner 10 5 0 5 10 surround 40 20 0 20 40 tennis 0.0 0.5 1.0 1.5 2.0 1e4 time_pilot 0 100 200 300 400 tutankham 0.0 0.5 1.0 1.5 1e5 up_n_down 0 50 100 150 200 0 200 400 600 800 venture 0 50 100 150 200 0 2 4 6 8 1e5 video_pinball 0 50 100 150 200 0.0 0.5 1.0 1.5 2.0 1e4 wizard_of_wor 0 50 100 150 200 0.0 0.5 1.0 1.5


ideo_pinball 0 50 100 150 200 0.0 0.5 1.0 1.5 2.0 1e4 wizard_of_wor 0 50 100 150 200 0.0 0.5 1.0 1.5 1e5 yars_revenge 0 50 100 150 200 0 1 2 3 1e4 zaxxon DQN DDQN Prioritized DDQN Dueling DDQN A3C Distributional DQN Noisy DQN Rainbow Figure 5: Learning curves for Rainbow and the baselines discussed in the paper, for each individual game. Every curve is smoothed with a moving average of 10 to improve readability.


0.0 0.5 1.0 1.5 1e4 alien 0 2 4 6 1e3 amidar 0.0 0.5 1.0 1.5 2.0 1e4 assault 0 2 4 6 1e5 asterix 0 2 4 6 8 1e3 asteroids 0.0 0.5 1.0 1.5 1e6 atlantis 0.0 0.5 1.0 1.5 2.0 1e3 bank_heist 0 2 4 6 8 1e4 battle_zone 0 1 2 3 1e4 beam_rider 0 2 4 6 1e3 berzerk 0 50 100 bowling 50 0 50 100 boxing 0 200 400 600 breakout 0.0 0.5 1.0 1e4 centipede 0.0 0.5 1.0 1.5 2.0 1e4chopper_command 0 1 2 3 1e5 crazy_climber 0 1 2 3 1e5 defender 0.0 0.5 1.0 1.5 1e5 demon_attack 4 2 0 2 1e1 double_dunk 0 1 2 3 1e3 enduro 100 50 0 50 100 fishing_derby 0 1 2 3 4 1e1 freeway 0.0 0.5 1.0 1e4 frostbite 0 2 4 6 8 1e4 gopher 0 1 2 3 1e3 gravitar 0 2 4 6 1e4 hero 2 1 0 1 1e1 ice_hockey 0 1 2 3 1e4 jamesbond 0.0 0.5 1.0 1.5 1e4 kangaroo 0.0 0.5 1.0 1.5 1e4 krull 0 2 4 6 1e4 kung_fu_master 0 200 400 600 montezuma_revenge 0 2


e4 kangaroo 0.0 0.5 1.0 1.5 1e4 krull 0 2 4 6 1e4 kung_fu_master 0 200 400 600 montezuma_revenge 0 2 4 6 1e3 ms_pacman 0.0 0.5 1.0 1.5 2.0 1e4 name_this_game 0 1 2 3 4 1e5 phoenix 1.5 1.0 0.5 0.0 1e3 pitfall 4 2 0 2 4 1e1 pong 1 0 1 2 1e3 private_eye 0 1 2 3 4 1e4 qbert 0 1 2 3 1e4 riverraid 0 2 4 6 8 1e4 road_runner 0 2 4 6 8 1e1 robotank 0 1 2 3 1e5 seaquest 4 3 2 1 0 1e4 skiing 0 1 2 3 4 1e3 solaris 0 1 2 3 1e4 space_invaders 0.0 0.5 1.0 1.5 1e5 star_gunner 1.0 0.5 0.0 0.5 1.0 1e1 surround 3 2 1 0 1 1e1 tennis 0.0 0.5 1.0 1.5 2.0 1e4 time_pilot 0 100 200 300 400 tutankham 0.0 0.5 1.0 1.5 2.0 1e5 up_n_down 0 50 100 150 200 0.0 0.5 1.0 1.5 1e3 venture 0 50 100 150 200 0 2 4 6 8 1e5 video_pinball 0 50 100 150 200 0.0 0.5 1.0 1.5 2.0 1e4 wizard_of_wor 0 50 100 150 200 0.0 0.5 1.0 1.5 1e5 ya


nball 0 50 100 150 200 0.0 0.5 1.0 1.5 2.0 1e4 wizard_of_wor 0 50 100 150 200 0.0 0.5 1.0 1.5 1e5 yars_revenge 0 50 100 150 200 0 1 2 3 1e4 zaxxon DQN no double no prior no duel no n-steps no distrib no noisy Rainbow Figure 6: Learning curves for Rainbow and its ablations, for each individual game. Every curve is smoothed with a moving average of 10 to improve readability.


Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Tuomas Haarnoja 1 Aurick Zhou 1 Pieter Abbeel 1 Sergey Levine 1 Abstract Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an offpolicy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework.


olicy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds. 1.


, our approach is very stable, achieving very similar performance across different random seeds. 1. Introduction Model-free deep reinforcement learning (RL) algorithms have been applied in a range of challenging domains, from games (Mnih et al., 2013; Silver et al., 2016) to robotic control (Schulman et al., 2015). The combination of RL and high-capacity function approximators such as neural networks holds the promise of automating a wide range of decision making and control tasks, but widespread adoption 1Berkeley Artiﬁcial Intelligence Research, University of California, Berkeley, USA. Correspondence to: Tuomas Haarnoja <haarnoja@berkeley.edu>. of these methods in real-world domains has been hampered by two major challenges.


ja@berkeley.edu>. of these methods in real-world domains has been hampered by two major challenges. First, model-free deep RL methods are notoriously expensive in terms of their sample complexity. Even relatively simple tasks can require millions of steps of data collection, and complex behaviors with highdimensional observations might need substantially more. Second, these methods are often brittle with respect to their hyperparameters: learning rates, exploration constants, and other settings must be set carefully for different problem settings to achieve good results. Both of these challenges severely limit the applicability of model-free deep RL to real-world tasks. One cause for the poor sample efﬁciency of deep RL methods is on-policy learning: some of the most commonly used deep RL


r sample efﬁciency of deep RL methods is on-policy learning: some of the most commonly used deep RL algorithms, such as TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017b) or A3C (Mnih et al., 2016), require new samples to be collected for each gradient step. This quickly becomes extravagantly expensive, as the number of gradient steps and samples per step needed to learn an effective policy increases with task complexity. Offpolicy algorithms aim to reuse past experience. This is not directly feasible with conventional policy gradient formulations, but is relatively straightforward for Q-learning based methods (Mnih et al., 2015). Unfortunately, the combination of off-policy learning and high-dimensional, nonlinear function approximation with neural networks presents a major challe


and high-dimensional, nonlinear function approximation with neural networks presents a major challenge for stability and convergence (Bhatnagar et al., 2009). This challenge is further exacerbated in continuous state and action spaces, where a separate actor network is often used to perform the maximization in Q-learning. A commonly used algorithm in such settings, deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), provides for sample-efﬁcient learning but is notoriously challenging to use due to its extreme brittleness and hyperparameter sensitivity (Duan et al., 2016; Henderson et al., 2017). We explore how to design an efﬁcient and stable modelfree deep RL algorithm for continuous state and action spaces.


o design an efﬁcient and stable modelfree deep RL algorithm for continuous state and action spaces. To that end, we draw on the maximum entropy framework, which augments the standard maximum reward reinforcement learning objective with an entropy maximization term (Ziebart et al., 2008; Toussaint, 2009; Rawlik et al., arXiv:1801.01290v2 [cs.LG] 8 Aug 2018


Soft Actor-Critic 2012; Fox et al., 2016; Haarnoja et al., 2017). Maximum entropy reinforcement learning alters the RL objective, though the original objective can be recovered using a temperature parameter (Haarnoja et al., 2017). More importantly, the maximum entropy formulation provides a substantial improvement in exploration and robustness: as discussed by Ziebart (2010), maximum entropy policies are robust in the face of model and estimation errors, and as demonstrated by (Haarnoja et al., 2017), they improve exploration by acquiring diverse behaviors. Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a;


6), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a; Nachum et al., 2017a; Haarnoja et al., 2017). However, the on-policy variants suffer from poor sample complexity for the reasons discussed above, while the off-policy variants require complex approximate inference procedures in continuous action spaces. In this paper, we demonstrate that we can devise an offpolicy maximum entropy actor-critic algorithm, which we call soft actor-critic (SAC), which provides for both sampleefﬁcient learning and stability. This algorithm extends readily to very complex, high-dimensional tasks, such as the Humanoid benchmark (Duan et al., 2016) with 21 action dimensions, where off-policy methods such as DDPG typically struggle to obtain good results (Gu et al.,


ensions, where off-policy methods such as DDPG typically struggle to obtain good results (Gu et al., 2016). SAC also avoids the complexity and potential instability associated with approximate inference in prior off-policy maximum entropy algorithms based on soft Q-learning (Haarnoja et al., 2017). We present a convergence proof for policy iteration in the maximum entropy framework, and then introduce a new algorithm based on an approximation to this procedure that can be practically implemented with deep neural networks, which we call soft actor-critic. We present empirical results that show that soft actor-critic attains a substantial improvement in both performance and sample efﬁciency over both off-policy and on-policy prior methods.


rovement in both performance and sample efﬁciency over both off-policy and on-policy prior methods. We also compare to twin delayed deep deterministic (TD3) policy gradient algorithm (Fujimoto et al., 2018), which is a concurrent work that proposes a deterministic algorithm that substantially improves on DDPG. 2. Related Work Our soft actor-critic algorithm incorporates three key ingredients: an actor-critic architecture with separate policy and value function networks, an off-policy formulation that enables reuse of previously collected data for efﬁciency, and entropy maximization to enable stability and exploration. We review prior works that draw on some of these ideas in this section. Actor-critic algorithms are typically derived starting from policy iteration, which alternates between


tor-critic algorithms are typically derived starting from policy iteration, which alternates between policy evaluation—computing the value function for a policy— and policy improvement—using the value function to obtain a better policy (Barto et al., 1983; Sutton & Barto, 1998). In large-scale reinforcement learning problems, it is typically impractical to run either of these steps to convergence, and instead the value function and policy are optimized jointly. In this case, the policy is referred to as the actor, and the value function as the critic. Many actor-critic algorithms build on the standard, on-policy policy gradient formulation to update the actor (Peters & Schaal, 2008), and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use i


y of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer (Schulman et al., 2017b; 2015; Mnih et al., 2016; Gruslys et al., 2017). On-policy training tends to improve stability but results in poor sample complexity. There have been efforts to increase the sample efﬁciency while retaining robustness by incorporating off-policy samples and by using higher order variance reduction techniques (O’Donoghue et al., 2016; Gu et al., 2016). However, fully off-policy algorithms still attain better efﬁciency. A particularly popular off-policy actor-critic method, DDPG (Lillicrap et al., 2015), which is a deep variant of the deterministic policy gradient (Silver et al., 2014) algorithm, uses a Q-function estimator to enable off-policy learn


icy gradient (Silver et al., 2014) algorithm, uses a Q-function estimator to enable off-policy learning, and a deterministic actor that maximizes this Q-function. As such, this method can be viewed both as a deterministic actor-critic algorithm and an approximate Q-learning algorithm. Unfortunately, the interplay between the deterministic actor network and the Q-function typically makes DDPG extremely difﬁcult to stabilize and brittle to hyperparameter settings (Duan et al., 2016; Henderson et al., 2017). As a consequence, it is difﬁcult to extend DDPG to complex, high-dimensional tasks, and on-policy policy gradient methods still tend to produce the best results in such settings (Gu et al., 2016).


policy gradient methods still tend to produce the best results in such settings (Gu et al., 2016). Our method instead combines off-policy actorcritic training with a stochastic actor, and further aims to maximize the entropy of this actor with an entropy maximization objective. We ﬁnd that this actually results in a considerably more stable and scalable algorithm that, in practice, exceeds both the efﬁciency and ﬁnal performance of DDPG. A similar method can be derived as a zero-step special case of stochastic value gradients (SVG(0)) (Heess et al., 2015). However, SVG(0) differs from our method in that it optimizes the standard maximum expected return objective, and it does not make use of a separate value network, which we found to make training more stable.


and it does not make use of a separate value network, which we found to make training more stable. Maximum entropy reinforcement learning optimizes policies to maximize both the expected return and the expected entropy of the policy. This framework has been used in many contexts, from inverse reinforcement learning (Ziebart et al., 2008) to optimal control (Todorov, 2008; Toussaint, 2009; Rawlik et al., 2012). In guided policy search (Levine & Koltun, 2013; Levine et al., 2016), the maximum entropy distribution is used to guide policy learn


Soft Actor-Critic ing towards high-reward regions. More recently, several papers have noted the connection between Q-learning and policy gradient methods in the framework of maximum entropy learning (O’Donoghue et al., 2016; Haarnoja et al., 2017; Nachum et al., 2017a; Schulman et al., 2017a). While most of the prior model-free works assume a discrete action space, Nachum et al. (2017b) approximate the maximum entropy distribution with a Gaussian and Haarnoja et al. (2017) with a sampling network trained to draw samples from the optimal policy. Although the soft Q-learning algorithm proposed by Haarnoja et al. (2017) has a value function and actor network, it is not a true actor-critic algorithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect th


ithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, Haarnoja et al. (2017) motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior. In contrast, we prove that our method converges to the optimal policy from a given policy class, regardless of the policy parameterization. Furthermore, these prior maximum entropy methods generally do not exceed the performance of state-of-the-art off-policy algorithms, such as DDPG, when learning from scratch, though they may have other beneﬁts, such as improved exploration and ease of ﬁne-tuning.


m scratch, though they may have other beneﬁts, such as improved exploration and ease of ﬁne-tuning. In our experiments, we demonstrate that our soft actor-critic algorithm does in fact exceed the performance of prior state-of-the-art off-policy deep RL methods by a wide margin. 3. Preliminaries We ﬁrst introduce notation and summarize the standard and maximum entropy reinforcement learning frameworks. 3.1. Notation We address policy learning in continuous action spaces. We consider an inﬁnite-horizon Markov decision process (MDP), deﬁned by the tuple (S, A, p, r), where the state space S and the action space A are continuous, and the unknown state transition probability p : S × S × A → [0, ∞) represents the probability density of the next state st+1 ∈S given the current state st ∈S and act