) represents the probability density of the next state st+1 ∈S given the current state st ∈S and action at ∈A. The environment emits a bounded reward r : S × A → [rmin, rmax] on each transition. We will use ρπ(st) and ρπ(st, at) to denote the state and state-action marginals of the trajectory distribution induced by a policy π(at|st). 3.2. Maximum Entropy Reinforcement Learning Standard RL maximizes the expected sum of rewards P t E(st,at)∼ρπ [r(st, at)]. We will consider a more general maximum entropy objective (see e.g. Ziebart (2010)), which favors stochastic policies by augmenting the objective with the expected entropy of the policy over ρπ(st): J(π) = T X t=0 E(st,at)∼ρπ [r(st, at) + αH(π( · |st))] .


xpected entropy of the policy over ρπ(st): J(π) = T X t=0 E(st,at)∼ρπ [r(st, at) + αH(π( · |st))] . (1) The temperature parameter α determines the relative importance of the entropy term against the reward, and thus controls the stochasticity of the optimal policy. The maximum entropy objective differs from the standard maximum expected reward objective used in conventional reinforcement learning, though the conventional objective can be recovered in the limit as α →0. For the rest of this paper, we will omit writing the temperature explicitly, as it can always be subsumed into the reward by scaling it by α−1. This objective has a number of conceptual and practical advantages. First, the policy is incentivized to explore more widely, while giving up on clearly unpromising avenues.


the policy is incentivized to explore more widely, while giving up on clearly unpromising avenues. Second, the policy can capture multiple modes of nearoptimal behavior. In problem settings where multiple actions seem equally attractive, the policy will commit equal probability mass to those actions. Lastly, prior work has observed improved exploration with this objective (Haarnoja et al., 2017; Schulman et al., 2017a), and in our experiments, we observe that it considerably improves learning speed over state-of-art methods that optimize the conventional RL objective function. We can extend the objective to inﬁnite horizon problems by introducing a discount factor γ to ensure that the sum of expected rewards and entropies is ﬁnite.


introducing a discount factor γ to ensure that the sum of expected rewards and entropies is ﬁnite. Writing down the maximum entropy objective for the inﬁnite horizon discounted case is more involved (Thomas, 2014) and is deferred to Appendix A. Prior methods have proposed directly solving for the optimal Q-function, from which the optimal policy can be recovered (Ziebart et al., 2008; Fox et al., 2016; Haarnoja et al., 2017). We will discuss how we can devise a soft actor-critic algorithm through a policy iteration formulation, where we instead evaluate the Q-function of the current policy and update the policy through an off-policy gradient update. Though such algorithms have previously been proposed for conventional reinforcement learning, our method is, to our knowledge, the ﬁrst off-p


en proposed for conventional reinforcement learning, our method is, to our knowledge, the ﬁrst off-policy actor-critic method in the maximum entropy reinforcement learning framework. 4. From Soft Policy Iteration to Soft Actor-Critic Our off-policy soft actor-critic algorithm can be derived starting from a maximum entropy variant of the policy iteration method. We will ﬁrst present this derivation, verify that the corresponding algorithm converges to the optimal policy from its density class, and then present a practical deep reinforcement learning algorithm based on this theory.


Soft Actor-Critic 4.1. Derivation of Soft Policy Iteration We will begin by deriving soft policy iteration, a general algorithm for learning optimal maximum entropy policies that alternates between policy evaluation and policy improvement in the maximum entropy framework. Our derivation is based on a tabular setting, to enable theoretical analysis and convergence guarantees, and we extend this method into the general continuous setting in the next section. We will show that soft policy iteration converges to the optimal policy within a set of policies which might correspond, for instance, to a set of parameterized densities. In the policy evaluation step of soft policy iteration, we wish to compute the value of a policy π according to the maximum entropy objective in Equation 1.


e wish to compute the value of a policy π according to the maximum entropy objective in Equation 1. For a ﬁxed policy, the soft Q-value can be computed iteratively, starting from any function Q : S × A →R and repeatedly applying a modiﬁed Bellman backup operator T π given by T πQ(st, at) ≜r(st, at) + γ Est+1∼p [V (st+1)] , (2) where V (st) = Eat∼π [Q(st, at) −log π(at|st)] (3) is the soft state value function. We can obtain the soft value function for any policy π by repeatedly applying T π as formalized below. Lemma 1 (Soft Policy Evaluation). Consider the soft Bellman backup operator T π in Equation 2 and a mapping Q0 : S×A →R with |A| < ∞, and deﬁne Qk+1 = T πQk. Then the sequence Qk will converge to the soft Q-value of π as k →∞. Proof. See Appendix B.1.


πQk. Then the sequence Qk will converge to the soft Q-value of π as k →∞. Proof. See Appendix B.1. In the policy improvement step, we update the policy towards the exponential of the new Q-function. This particular choice of update can be guaranteed to result in an improved policy in terms of its soft value. Since in practice we prefer policies that are tractable, we will additionally restrict the policy to some set of policies Π, which can correspond, for example, to a parameterized family of distributions such as Gaussians. To account for the constraint that π ∈Π, we project the improved policy into the desired set of policies. While in principle we could choose any projection, it will turn out to be convenient to use the information projection deﬁned in terms of the Kullback-Leibler di


rn out to be convenient to use the information projection deﬁned in terms of the Kullback-Leibler divergence. In the other words, in the policy improvement step, for each state, we update the policy according to πnew = arg min π′∈ΠDKL  π′( · |st) exp (Qπold(st, · )) Zπold(st)  . (4) The partition function Zπold(st) normalizes the distribution, and while it is intractable in general, it does not contribute to the gradient with respect to the new policy and can thus be ignored, as noted in the next section. For this projection, we can show that the new, projected policy has a higher value than the old policy with respect to the objective in Equation 1. We formalize this result in Lemma 2. Lemma 2 (Soft Policy Improvement).


he objective in Equation 1. We formalize this result in Lemma 2. Lemma 2 (Soft Policy Improvement). Let πold ∈Π and let πnew be the optimizer of the minimization problem deﬁned in Equation 4. Then Qπnew(st, at) ≥Qπold(st, at) for all (st, at) ∈S × A with |A| < ∞. Proof. See Appendix B.2. The full soft policy iteration algorithm alternates between the soft policy evaluation and the soft policy improvement steps, and it will provably converge to the optimal maximum entropy policy among the policies in Π (Theorem 1). Although this algorithm will provably ﬁnd the optimal solution, we can perform it in its exact form only in the tabular case. Therefore, we will next approximate the algorithm for continuous domains, where we need to rely on a function approximator to represent the Q-values, and


continuous domains, where we need to rely on a function approximator to represent the Q-values, and running the two steps until convergence would be computationally too expensive. The approximation gives rise to a new practical algorithm, called soft actor-critic. Theorem 1 (Soft Policy Iteration). Repeated application of soft policy evaluation and soft policy improvement from any π ∈Π converges to a policy π∗such that Qπ∗(st, at) ≥ Qπ(st, at) for all π ∈Π and (st, at) ∈S × A, assuming |A| < ∞. Proof. See Appendix B.3. 4.2. Soft Actor-Critic As discussed above, large continuous domains require us to derive a practical approximation to soft policy iteration. To that end, we will use function approximators for both the Q-function and the policy, and instead of running evaluation and improvem


approximators for both the Q-function and the policy, and instead of running evaluation and improvement to convergence, alternate between optimizing both networks with stochastic gradient descent. We will consider a parameterized state value function Vψ(st), soft Q-function Qθ(st, at), and a tractable policy πφ(at|st). The parameters of these networks are ψ, θ, and φ. For example, the value functions can be modeled as expressive neural networks, and the policy as a Gaussian with mean and covariance given by neural networks. We will next derive update rules for these parameter vectors. The state value function approximates the soft value. There is no need in principle to include a separate function approximator for the state value, since it is related to the Q-function and policy according


nction approximator for the state value, since it is related to the Q-function and policy according to Equation 3. This quantity can be


Soft Actor-Critic estimated from a single action sample from the current policy without introducing a bias, but in practice, including a separate function approximator for the soft value can stabilize training and is convenient to train simultaneously with the other networks. The soft value function is trained to minimize the squared residual error JV (ψ) = Est∼D h 1 2 Vψ(st) −Eat∼πφ [Qθ(st, at) −log πφ(at|st)] 2i (5) where D is the distribution of previously sampled states and actions, or a replay buffer. The gradient of Equation 5 can be estimated with an unbiased estimator ˆ∇ψJV (ψ) = ∇ψVψ(st) (Vψ(st) −Qθ(st, at) + log πφ(at|st)) , (6) where the actions are sampled according to the current policy, instead of the replay buffer.


, (6) where the actions are sampled according to the current policy, instead of the replay buffer. The soft Q-function parameters can be trained to minimize the soft Bellman residual JQ(θ) = E(st,at)∼D 1 2  Qθ(st, at) −ˆQ(st, at) 2 , (7) with ˆQ(st, at) = r(st, at) + γ Est+1∼p  V ¯ ψ(st+1)  , (8) which again can be optimized with stochastic gradients ˆ∇θJQ(θ) = ∇θQθ(at, st) Qθ(st, at) −r(st, at) −γV ¯ ψ(st+1) . (9) The update makes use of a target value network V ¯ ψ, where ¯ψ can be an exponentially moving average of the value network weights, which has been shown to stabilize training (Mnih et al., 2015). Alternatively, we can update the target weights to match the current value function weights periodically (see Appendix E).


pdate the target weights to match the current value function weights periodically (see Appendix E). Finally, the policy parameters can be learned by directly minimizing the expected KL-divergence in Equation 4: Jπ(φ) = Est∼D  DKL  πφ( · |st) exp (Qθ(st, · )) Zθ(st)  . (10) There are several options for minimizing Jπ. A typical solution for policy gradient methods is to use the likelihood ratio gradient estimator (Williams, 1992), which does not require backpropagating the gradient through the policy and the target density networks. However, in our case, the target density is the Q-function, which is represented by a neural network an can be differentiated, and it is thus convenient to apply the reparameterization trick instead, resulting in a lower variance estimator.


convenient to apply the reparameterization trick instead, resulting in a lower variance estimator. To that end, we reparameterize the policy using a neural network transformation at = fφ(ϵt; st), (11) Algorithm 1 Soft Actor-Critic Initialize parameter vectors ψ, ¯ψ, θ, φ. for each iteration do for each environment step do at ∼πφ(at|st) st+1 ∼p(st+1|st, at) D ←D ∪{(st, at, r(st, at), st+1)} end for for each gradient step do ψ ←ψ −λV ˆ∇ψJV (ψ) θi ←θi −λQ ˆ∇θiJQ(θi) for i ∈{1, 2} φ ←φ −λπ ˆ∇φJπ(φ) ¯ψ ←τψ + (1 −τ) ¯ψ end for end for where ϵt is an input noise vector, sampled from some ﬁxed distribution, such as a spherical Gaussian. We can now rewrite the objective in Equation 10 as Jπ(φ) = Est∼D,ϵt∼N [log πφ(fφ(ϵt; st)|st) −Qθ(st, fφ(ϵt; st))] , (12) where πφ is deﬁned implicitly in terms of


∼D,ϵt∼N [log πφ(fφ(ϵt; st)|st) −Qθ(st, fφ(ϵt; st))] , (12) where πφ is deﬁned implicitly in terms of fφ, and we have noted that the partition function is independent of φ and can thus be omitted. We can approximate the gradient of Equation 12 with ˆ∇φJπ(φ) = ∇φ log πφ(at|st) + (∇at log πφ(at|st) −∇atQ(st, at))∇φfφ(ϵt; st), (13) where at is evaluated at fφ(ϵt; st). This unbiased gradient estimator extends the DDPG style policy gradients (Lillicrap et al., 2015) to any tractable stochastic policy. Our algorithm also makes use of two Q-functions to mitigate positive bias in the policy improvement step that is known to degrade performance of value based methods (Hasselt, 2010; Fujimoto et al., 2018).


that is known to degrade performance of value based methods (Hasselt, 2010; Fujimoto et al., 2018). In particular, we parameterize two Q-functions, with parameters θi, and train them independently to optimize JQ(θi). We then use the minimum of the Q-functions for the value gradient in Equation 6 and policy gradient in Equation 13, as proposed by Fujimoto et al. (2018). Although our algorithm can learn challenging tasks, including a 21-dimensional Humanoid, using just a single Q-function, we found two Q-functions signiﬁcantly speed up training, especially on harder tasks. The complete algorithm is described in Algorithm 1. The method alternates between collecting experience from the environment with the current policy and updating the function approximators using the stochastic gradients fr


nt with the current policy and updating the function approximators using the stochastic gradients from batches sampled from a replay buffer. In practice, we take a single environment step followed by one or several gradient steps (see Appendix D


Soft Actor-Critic 0.0 0.2 0.4 0.6 0.8 1.0 million steps 0 1000 2000 3000 4000 average return (a) Hopper-v1 0.0 0.2 0.4 0.6 0.8 1.0 million steps 0 1000 2000 3000 4000 5000 6000 average return (b) Walker2d-v1 0.0 0.5 1.0 1.5 2.0 2.5 3.0 million steps 0 5000 10000 15000 average return (c) HalfCheetah-v1 0.0 0.5 1.0 1.5 2.0 2.5 3.0 million steps 0 2000 4000 6000 average return (d) Ant-v1 0 2 4 6 8 10 million steps 0 2000 4000 6000 8000 average return (e) Humanoid-v1 0 2 4 6 8 10 million steps 0 2000 4000 6000 average return ( ) SAC DDPG PPO SQL TD3 (concurrent) (f) Humanoid (rllab) Figure 1. Training curves on continuous control benchmarks. Soft actor-critic (yellow) performs consistently across all tasks and outperforming both on-policy and off-policy methods in the most challenging tasks.


ss all tasks and outperforming both on-policy and off-policy methods in the most challenging tasks. for all hyperparameter). Using off-policy data from a replay buffer is feasible because both value estimators and the policy can be trained entirely on off-policy data. The algorithm is agnostic to the parameterization of the policy, as long as it can be evaluated for any arbitrary state-action tuple. 5. Experiments The goal of our experimental evaluation is to understand how the sample complexity and stability of our method compares with prior off-policy and on-policy deep reinforcement learning algorithms. We compare our method to prior techniques on a range of challenging continuous control tasks from the OpenAI gym benchmark suite (Brockman et al., 2016) and also on the rllab implementat


tasks from the OpenAI gym benchmark suite (Brockman et al., 2016) and also on the rllab implementation of the Humanoid task (Duan et al., 2016). Although the easier tasks can be solved by a wide range of different algorithms, the more complex benchmarks, such as the 21-dimensional Humanoid (rllab), are exceptionally difﬁcult to solve with off-policy algorithms (Duan et al., 2016). The stability of the algorithm also plays a large role in performance: easier tasks make it more practical to tune hyperparameters to achieve good results, while the already narrow basins of effective hyperparameters become prohibitively small for the more sensitive algorithms on the hardest benchmarks, leading to poor performance (Gu et al., 2016).


more sensitive algorithms on the hardest benchmarks, leading to poor performance (Gu et al., 2016). We compare our method to deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015), an algorithm that is regarded as one of the more efﬁcient off-policy deep RL methods (Duan et al., 2016); proximal policy optimization (PPO) (Schulman et al., 2017b), a stable and effective on-policy policy gradient algorithm; and soft Q-learning (SQL) (Haarnoja et al., 2017), a recent off-policy algorithm for learning maximum entropy policies. Our SQL implementation also includes two Q-functions, which we found to improve its performance in most environments. We additionally compare to twin delayed deep deterministic policy gradient algorithm (TD3) (Fujimoto et al., 2018), using the author-provided


ep deterministic policy gradient algorithm (TD3) (Fujimoto et al., 2018), using the author-provided implementation. This is an extension to DDPG, proposed concurrently to our method, that ﬁrst applied the double Q-learning trick to continuous control along with other improvements. We have included trust region path consistency learning (Trust-PCL) (Nachum et al., 2017b) and two other variants of SAC in Appendix E. We turned off the exploration noise for evaluation for DDPG and PPO. For maximum entropy algorithms, which do not explicitly inject exploration noise, we either evaluated with the exploration noise (SQL) or use the mean action (SAC). The source code of our SAC implementation1 and videos2 are available online. 1github.com/haarnoja/sac 2sites.google.com/view/soft-actor-critic


Soft Actor-Critic 5.1. Comparative Evaluation Figure 1 shows the total average return of evaluation rollouts during training for DDPG, PPO, and TD3. We train ﬁve different instances of each algorithm with different random seeds, with each performing one evaluation rollout every 1000 environment steps. The solid curves corresponds to the mean and the shaded region to the minimum and maximum returns over the ﬁve trials. The results show that, overall, SAC performs comparably to the baseline methods on the easier tasks and outperforms them on the harder tasks with a large margin, both in terms of learning speed and the ﬁnal performance. For example, DDPG fails to make any progress on Ant-v1, Humanoidv1, and Humanoid (rllab), a result that is corroborated by prior work (Gu et al., 2016; Duan e


manoidv1, and Humanoid (rllab), a result that is corroborated by prior work (Gu et al., 2016; Duan et al., 2016). SAC also learns considerably faster than PPO as a consequence of the large batch sizes PPO needs to learn stably on more high-dimensional and complex tasks. Another maximum entropy RL algorithm, SQL, can also learn all tasks, but it is slower than SAC and has worse asymptotic performance. The quantitative results attained by SAC in our experiments also compare very favorably to results reported by other methods in prior work (Duan et al., 2016; Gu et al., 2016; Henderson et al., 2017), indicating that both the sample efﬁciency and ﬁnal performance of SAC on these benchmark tasks exceeds the state of the art.


sample efﬁciency and ﬁnal performance of SAC on these benchmark tasks exceeds the state of the art. All hyperparameters used in this experiment for SAC are listed in Appendix D. 5.2. Ablation Study The results in the previous section suggest that algorithms based on the maximum entropy principle can outperform conventional RL methods on challenging tasks such as the humanoid tasks. In this section, we further examine which particular components of SAC are important for good performance. We also examine how sensitive SAC is to some of the most important hyperparameters, namely reward scaling and target value update smoothing constant. Stochastic vs. deterministic policy. Soft actor-critic learns stochastic policies via a maximum entropy objective.


deterministic policy. Soft actor-critic learns stochastic policies via a maximum entropy objective. The entropy appears in both the policy and value function. In the policy, it prevents premature convergence of the policy variance (Equation 10). In the value function, it encourages exploration by increasing the value of regions of state space that lead to high-entropy behavior (Equation 5). To compare how the stochasticity of the policy and entropy maximization affects the performance, we compare to a deterministic variant of SAC that does not maximize the entropy and that closely resembles DDPG, with the exception of having two Q-functions, using hard target updates, not having a separate target actor, and using ﬁxed rather than learned exploration noise.


updates, not having a separate target actor, and using ﬁxed rather than learned exploration noise. Figure 2 compares ﬁve individual runs with both variants, initialized with different random 0 2 4 6 8 10 million steps 0 2000 4000 6000 average return Humanoid (rllab) stochastic policy deterministic policy Figure 2. Comparison of SAC (blue) and a deterministic variant of SAC (red) in terms of the stability of individual random seeds on the Humanoid (rllab) benchmark. The comparison indicates that stochasticity can stabilize training as the variability between the seeds becomes much higher with a deterministic policy. seeds. Soft actor-critic performs much more consistently, while the deterministic variant exhibits very high variability across seeds, indicating substantially worse stability.


istic variant exhibits very high variability across seeds, indicating substantially worse stability. As evident from the ﬁgure, learning a stochastic policy with entropy maximization can drastically stabilize training. This becomes especially important with harder tasks, where tuning hyperparameters is challenging. In this comparison, we updated the target value network weights with hard updates, by periodically overwriting the target network parameters to match the current value network (see Appendix E for a comparison of average performance on all benchmark tasks). Policy evaluation. Since SAC converges to stochastic policies, it is often beneﬁcial to make the ﬁnal policy deterministic at the end for best performance.


icies, it is often beneﬁcial to make the ﬁnal policy deterministic at the end for best performance. For evaluation, we approximate the maximum a posteriori action by choosing the mean of the policy distribution. Figure 3(a) compares training returns to evaluation returns obtained with this strategy indicating that deterministic evaluation can yield better performance. It should be noted that all of the training curves depict the sum of rewards, which is different from the objective optimized by SAC and other maximum entropy RL algorithms, including SQL and Trust-PCL, which maximize also the entropy of the policy. Reward scale. Soft actor-critic is particularly sensitive to the scaling of the reward signal, because it serves the role of the temperature of the energy-based optimal policy and


reward signal, because it serves the role of the temperature of the energy-based optimal policy and thus controls its stochasticity. Larger reward magnitudes correspond to lower entries. Figure 3(b) shows how learning performance changes when the reward scale is varied: For small reward magnitudes, the policy becomes nearly uniform, and consequently fails to exploit the reward signal, resulting in substantial degradation of performance. For large reward magnitudes, the model learns quickly at ﬁrst,


Soft Actor-Critic 0.0 0.5 1.0 1.5 2.0 2.5 3.0 million steps 0 2000 4000 6000 average return deterministic evaluation stochastic evaluation (a) Evaluation 0.0 0.5 1.0 1.5 2.0 2.5 3.0 million steps 0 2000 4000 6000 average return 1 3 10 30 100 (b) Reward Scale 0.0 0.5 1.0 1.5 2.0 2.5 3.0 million steps −2000 0 2000 4000 6000 average return 0.0001 0.001 0.01 0.1 (c) Target Smoothing Coefﬁcient (τ) Figure 3. Sensitivity of soft actor-critic to selected hyperparameters on Ant-v1 task. (a) Evaluating the policy using the mean action generally results in a higher return. Note that the policy is trained to maximize also the entropy, and the mean action does not, in general, correspond the optimal action for the maximum return objective.


e mean action does not, in general, correspond the optimal action for the maximum return objective. (b) Soft actor-critic is sensitive to reward scaling since it is related to the temperature of the optimal policy. The optimal reward scale varies between environments, and should be tuned for each task separately. (c) Target value smoothing coefﬁcient τ is used to stabilize training. Fast moving target (large τ) can result in instabilities (red), whereas slow moving target (small τ) makes training slower (blue). but the policy then becomes nearly deterministic, leading to poor local minima due to lack of adequate exploration. With the right reward scaling, the model balances exploration and exploitation, leading to faster learning and better asymptotic performance.


alances exploration and exploitation, leading to faster learning and better asymptotic performance. In practice, we found reward scale to be the only hyperparameter that requires tuning, and its natural interpretation as the inverse of the temperature in the maximum entropy framework provides good intuition for how to adjust this parameter. Target network update. It is common to use a separate target value network that slowly tracks the actual value function to improve stability. We use an exponentially moving average, with a smoothing constant τ, to update the target value network weights as common in the prior work (Lillicrap et al., 2015; Mnih et al., 2015). A value of one corresponds to a hard update where the weights are copied directly at every iteration and zero to not updating the


a hard update where the weights are copied directly at every iteration and zero to not updating the target at all. In Figure 3(c), we compare the performance of SAC when τ varies. Large τ can lead to instabilities while small τ can make training slower. However, we found the range of suitable values of τ to be relatively wide and we used the same value (0.005) across all of the tasks. In Figure 4 (Appendix E) we also compare to another variant of SAC, where instead of using exponentially moving average, we copy over the current network weights directly into the target network every 1000 gradient steps. We found this variant to beneﬁt from taking more than one gradient step between the environment steps, which can improve performance but also increases the computational cost. 6.


the environment steps, which can improve performance but also increases the computational cost. 6. Conclusion We present soft actor-critic (SAC), an off-policy maximum entropy deep reinforcement learning algorithm that provides sample-efﬁcient learning while retaining the beneﬁts of entropy maximization and stability. Our theoretical results derive soft policy iteration, which we show to converge to the optimal policy. From this result, we can formulate a soft actor-critic algorithm, and we empirically show that it outperforms state-of-the-art model-free deep RL methods, including the off-policy DDPG algorithm and the on-policy PPO algorithm. In fact, the sample efﬁciency of this approach actually exceeds that of DDPG by a substantial margin.


fact, the sample efﬁciency of this approach actually exceeds that of DDPG by a substantial margin. Our results suggest that stochastic, entropy maximizing reinforcement learning algorithms can provide a promising avenue for improved robustness and stability, and further exploration of maximum entropy methods, including methods that incorporate second order information (e.g., trust regions (Schulman et al., 2015)) or more expressive policy classes is an exciting avenue for future work. Acknowledgments We would like to thank Vitchyr Pong for insightful discussions and help in implementing our algorithm as well as providing the DDPG baseline code; Oﬁr Nachum for offering support in running Trust-PCL experiments; and George Tucker for his valuable feedback on an early version of this paper.


ust-PCL experiments; and George Tucker for his valuable feedback on an early version of this paper. This work was supported by Siemens and Berkeley DeepDrive.


Soft Actor-Critic References Barto, A. G., Sutton, R. S., and Anderson, C. W. Neuronlike adaptive elements that can solve difﬁcult learning control problems. IEEE transactions on systems, man, and cybernetics, pp. 834–846, 1983. Bhatnagar, S., Precup, D., Silver, D., Sutton, R. S., Maei, H. R., and Szepesv´ari, C. Convergent temporal-difference learning with arbitrary smooth function approximation. In Advances in Neural Information Processing Systems (NIPS), pp. 1204–1212, 2009. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016. Duan, Y., Chen, X. Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control.


, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning (ICML), 2016. Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. In Conference on Uncertainty in Artiﬁcial Intelligence (UAI), 2016. Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018. Gruslys, A., Azar, M. G., Bellemare, M. G., and Munos, R. The reactor: A sample-efﬁcient actor-critic architecture. arXiv preprint arXiv:1704.04651, 2017. Gu, S., Lillicrap, T., Ghahramani, Z., Turner, R. E., and Levine, S. Q-prop: Sample-efﬁcient policy gradient with an off-policy critic.


., Turner, R. E., and Levine, S. Q-prop: Sample-efﬁcient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.02247, 2016. Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (ICML), pp. 1352–1361, 2017. Hasselt, H. V. Double Q-learning. In Advances in Neural Information Processing Systems (NIPS), pp. 2613–2621, 2010. Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., and Tassa, Y. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (NIPS), pp. 2944–2952, 2015. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters.


m, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560, 2017. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference for Learning Presentations (ICLR), 2015. Levine, S. and Koltun, V. Guided policy search. In International Conference on Machine Learning (ICML), pp. 1–9, 2013. Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.


tra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529–533, 2015. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016.


hods for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016. Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems (NIPS), pp. 2772–2782, 2017a. Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. Trust-PCL: An off-policy trust region method for continuous control. arXiv preprint arXiv:1707.01891, 2017b. O’Donoghue, B., Munos, R., Kavukcuoglu, K., and Mnih, V. PGQ: Combining policy gradient and Q-learning. arXiv preprint arXiv:1611.01626, 2016. Peters, J. and Schaal, S. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682– 697, 2008. Rawlik, K., Toussaint, M., and Vijayakumar, S.


cy gradients. Neural networks, 21(4):682– 697, 2008. Rawlik, K., Toussaint, M., and Vijayakumar, S. On stochastic optimal control and reinforcement learning by approximate inference. Robotics: Science and Systems (RSS), 2012. Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning (ICML), pp. 1889–1897, 2015.


Soft Actor-Critic Schulman, J., Abbeel, P., and Chen, X. Equivalence between policy gradients and soft Q-learning. arXiv preprint arXiv:1704.06440, 2017a. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b. Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML), 2014. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D.


renner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, Jan 2016. ISSN 0028-0836. Article. Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. Thomas, P. Bias in natural actor-critic algorithms. In International Conference on Machine Learning (ICML), pp. 441–448, 2014. Todorov, E. General duality between optimal control and estimation. In IEEE Conference on Decision and Control (CDC), pp. 4286–4292. IEEE, 2008. Toussaint, M. Robot trajectory optimization using approximate inference. In International Conference on Machine Learning (ICML), pp. 1049–1056. ACM, 2009. Williams, R. J.


. In International Conference on Machine Learning (ICML), pp. 1049–1056. ACM, 2009. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010. Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artiﬁcial Intelligence (AAAI), pp. 1433–1438, 2008.


Soft Actor-Critic A. Maximum Entropy Objective The exact deﬁnition of the discounted maximum entropy objective is complicated by the fact that, when using a discount factor for policy gradient methods, we typically do not discount the state distribution, only the rewards. In that sense, discounted policy gradients typically do not optimize the true discounted objective. Instead, they optimize average reward, with the discount serving to reduce variance, as discussed by Thomas (2014). However, we can deﬁne the objective that is optimized under a discount factor as J(π) = ∞ X t=0 E(st,at)∼ρπ " ∞ X l=t γl−t Esl∼p,al∼π [r(st, at) + αH(π( · |st))|st, at] # . (14) This objective corresponds to maximizing the discounted expected reward and entropy for future states originating from every state-ac


imizing the discounted expected reward and entropy for future states originating from every state-action tuple (st, at) weighted by its probability ρπ under the current policy. B. Proofs B.1. Lemma 1 Lemma 1 (Soft Policy Evaluation). Consider the soft Bellman backup operator T π in Equation 2 and a mapping Q0 : S × A →R with |A| < ∞, and deﬁne Qk+1 = T πQk. Then the sequence Qk will converge to the soft Q-value of π as k →∞. Proof. Deﬁne the entropy augmented reward as rπ(st, at) ≜r(st, at) + Est+1∼p [H (π( · |st+1))] and rewrite the update rule as Q(st, at) ←rπ(st, at) + γ Est+1∼p,at+1∼π [Q(st+1, at+1)] (15) and apply the standard convergence results for policy evaluation (Sutton & Barto, 1998). The assumption |A| < ∞is required to guarantee that the entropy augmented reward is bounded.


1998). The assumption |A| < ∞is required to guarantee that the entropy augmented reward is bounded. B.2. Lemma 2 Lemma 2 (Soft Policy Improvement). Let πold ∈Π and let πnew be the optimizer of the minimization problem deﬁned in Equation 4. Then Qπnew(st, at) ≥Qπold(st, at) for all (st, at) ∈S × A with |A| < ∞. Proof. Let πold ∈Π and let Qπold and V πold be the corresponding soft state-action value and soft state value, and let πnew be deﬁned as πnew( · |st) = arg min π′∈Π DKL (π′( · |st) ∥exp (Qπold(st, · ) −log Zπold(st))) = arg min π′∈Π Jπold(π′( · |st)). (16) It must be the case that Jπold(πnew( · |st)) ≤Jπold(πold( · |st)), since we can always choose πnew = πold ∈Π. Hence Eat∼πnew [log πnew(at|st) −Qπold(st, at) + log Zπold(st)] ≤Eat∼πold [log πold(at|st) −Qπold(st, at) + log Zπold(st)


new(at|st) −Qπold(st, at) + log Zπold(st)] ≤Eat∼πold [log πold(at|st) −Qπold(st, at) + log Zπold(st)], (17) and since partition function Zπold depends only on the state, the inequality reduces to Eat∼πnew [Qπold(st, at) −log πnew(at|st)] ≥V πold(st). (18) Next, consider the soft Bellman equation: Qπold(st, at) = r(st, at) + γ Est+1∼p [V πold(st+1)] ≤r(st, at) + γ Est+1∼p  Eat+1∼πnew [Qπold(st+1, at+1) −log πnew(at+1|st+1)]  ... ≤Qπnew(st, at), (19) where we have repeatedly expanded Qπold on the RHS by applying the soft Bellman equation and the bound in Equation 18. Convergence to Qπnew follows from Lemma 1.


Soft Actor-Critic B.3. Theorem 1 Theorem 1 (Soft Policy Iteration). Repeated application of soft policy evaluation and soft policy improvement to any π ∈Π converges to a policy π∗such that Qπ∗(st, at) ≥Qπ(st, at) for all π ∈Π and (st, at) ∈S × A, assuming |A| < ∞. Proof. Let πi be the policy at iteration i. By Lemma 2, the sequence Qπi is monotonically increasing. Since Qπ is bounded above for π ∈Π (both the reward and entropy are bounded), the sequence converges to some π∗. We will still need to show that π∗is indeed optimal. At convergence, it must be case that Jπ∗(π∗( · |st)) < Jπ∗(π( · |st)) for all π ∈Π, π ̸= π∗. Using the same iterative argument as in the proof of Lemma 2, we get Qπ∗(st, at) > Qπ(st, at) for all (st, at) ∈S × A, that is, the soft value of any other policy in Π is low


t, at) > Qπ(st, at) for all (st, at) ∈S × A, that is, the soft value of any other policy in Π is lower than that of the converged policy. Hence π∗is optimal in Π. C. Enforcing Action Bounds We use an unbounded Gaussian as the action distribution. However, in practice, the actions needs to be bounded to a ﬁnite interval. To that end, we apply an invertible squashing function (tanh) to the Gaussian samples, and employ the change of variables formula to compute the likelihoods of the bounded actions. In the other words, let u ∈RD be a random variable and µ(u|s) the corresponding density with inﬁnite support. Then a = tanh(u), where tanh is applied elementwise, is a random variable with support in (−1, 1) with a density given by π(a|s) = µ(u|s) det  da du  −1 .


random variable with support in (−1, 1) with a density given by π(a|s) = µ(u|s) det  da du  −1 . (20) Since the Jacobian da/du = diag(1 −tanh2(u)) is diagonal, the log-likelihood has a simple form log π(a|s) = log µ(u|s) − D X i=1 log 1 −tanh2(ui)  , (21) where ui is the ith element of u.


Soft Actor-Critic D. Hyperparameters Table 1 lists the common SAC parameters used in the comparative evaluation in Figure 1 and Figure 4. Table 2 lists the reward scale parameter that was tuned for each environment. Table 1. SAC Hyperparameters Parameter Value Shared optimizer Adam (Kingma & Ba, 2015) learning rate 3 · 10−4 discount (γ) 0.99 replay buffer size 106 number of hidden layers (all networks) 2 number of hidden units per layer 256 number of samples per minibatch 256 nonlinearity ReLU SAC target smoothing coefﬁcient (τ) 0.005 target update interval 1 gradient steps 1 SAC (hard target update) target smoothing coefﬁcient (τ) 1 target update interval 1000 gradient steps (except humanoids) 4 gradient steps (humanoids) 1 Table 2.


rget update interval 1000 gradient steps (except humanoids) 4 gradient steps (humanoids) 1 Table 2. SAC Environment Speciﬁc Parameters Environment Action Dimensions Reward Scale Hopper-v1 3 5 Walker2d-v1 6 5 HalfCheetah-v1 6 5 Ant-v1 8 5 Humanoid-v1 17 20 Humanoid (rllab) 21 10


Soft Actor-Critic E. Additional Baseline Results Figure 4 compares SAC to Trust-PCL (Figure 4. Trust-PC fails to solve most of the task within the given number of environment steps, although it can eventually solve the easier tasks (Nachum et al., 2017b) if ran longer. The ﬁgure also includes two variants of SAC: a variant that periodically copies the target value network weights directly instead of using exponentially moving average, and a deterministic ablation which assumes a deterministic policy in the value update (Equation 6) and the policy update (Equation 13), and thus strongly resembles DDPG with the exception of having two Q-functions, using hard target updates, not having a separate target actor, and using ﬁxed exploration noise rather than learned.


updates, not having a separate target actor, and using ﬁxed exploration noise rather than learned. Both of these methods can learn all of the tasks and they perform comparably to SAC on all but Humanoid (rllab) task, on which SAC is the fastest. 0.0 0.2 0.4 0.6 0.8 1.0 million steps 0 1000 2000 3000 4000 average return (a) Hopper-v1 0.0 0.2 0.4 0.6 0.8 1.0 million steps 0 1000 2000 3000 4000 5000 6000 average return (b) Walker2d-v1 0.0 0.5 1.0 1.5 2.0 2.5 3.0 million steps 0 5000 10000 15000 average return (c) HalfCheetah-v1 0.0 0.5 1.0 1.5 2.0 2.5 3.0 million steps 0 2000 4000 6000 average return (d) Ant-v1 0 2 4 6 8 10 million steps 0 2000 4000 6000 8000 average return (e) Humanoid-v1 0 2 4 6 8 10 million steps 0 2000 4000 6000 average return ( ) SAC SAC (hard target update) SAC (hard t


4 6 8 10 million steps 0 2000 4000 6000 average return ( ) SAC SAC (hard target update) SAC (hard target update, deterministic) Trust-PCL (f) Humanoid (rllab) Figure 4. Training curves for additional baseline (Trust-PCL) and for two SAC variants. Soft actor-critic with hard target update (blue) differs from standard SAC in that it copies the value function network weights directly every 1000 iterations, instead of using exponentially smoothed average of the weights. The deterministic ablation (red) uses a deterministic policy with ﬁxed Gaussian exploration noise, does not use a value function, drops the entropy terms in the actor and critic function updates, and uses hard target updates for the target Q-functions.


in the actor and critic function updates, and uses hard target updates for the target Q-functions. It is equivalent to DDPG that uses two Q-functions, hard target updates, and removes the target actor.


IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures Lasse Espeholt * 1 Hubert Soyer * 1 Remi Munos * 1 Karen Simonyan 1 Volodymyr Mnih 1 Tom Ward 1 Yotam Doron 1 Vlad Firoiu 1 Tim Harley 1 Iain Dunning 1 Shane Legg 1 Koray Kavukcuoglu 1 Abstract In this work we aim to solve a large collection of tasks using a single reinforcement learning agent with a single set of parameters. A key challenge is to handle the increased amount of data and extended training time. We have developed a new distributed agent IMPALA (Importance Weighted Actor-Learner Architecture) that not only uses resources more efﬁciently in singlemachine training but also scales to thousands of machines without sacriﬁcing data efﬁciency or resource utilisation.


but also scales to thousands of machines without sacriﬁcing data efﬁciency or resource utilisation. We achieve stable learning at high throughput by combining decoupled acting and learning with a novel off-policy correction method called V-trace. We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari-57 (all available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show that IMPALA is able to achieve better performance than previous agents with less data, and crucially exhibits positive transfer between tasks as a result of its multi-task approach. The source code is publicly available at github.com/deepmind/scalable agent. 1.


ulti-task approach. The source code is publicly available at github.com/deepmind/scalable agent. 1. Introduction Deep reinforcement learning methods have recently mastered a wide variety of domains through trial and error learning (Mnih et al., 2015; Silver et al., 2017; 2016; Zoph et al., 2017; Lillicrap et al., 2015; Barth-Maron et al., 2018). While the improvements on tasks like the game of Go (Silver et al., 2017) and Atari games (Horgan et al., 2018) have been dramatic, the progress has been primarily in single task performance, where an agent is trained on each task *Equal contribution 1DeepMind Technologies, London, United Kingdom. Correspondence to: Lasse Espeholt <lespeholt@google.com>. separately.


gies, London, United Kingdom. Correspondence to: Lasse Espeholt <lespeholt@google.com>. separately. We are interested in developing new methods capable of mastering a diverse set of tasks simultaneously as well as environments suitable for evaluating such methods. One of the main challenges in training a single agent on many tasks at once is scalability. Since the current state-ofthe-art methods like A3C (Mnih et al., 2016) or UNREAL (Jaderberg et al., 2017b) can require as much as a billion frames and multiple days to master a single domain, training them on tens of domains at once is too slow to be practical. We propose the Importance Weighted Actor-Learner Architecture (IMPALA) shown in Figure 1.


ractical. We propose the Importance Weighted Actor-Learner Architecture (IMPALA) shown in Figure 1. IMPALA is capable of scaling to thousands of machines without sacriﬁcing training stability or data efﬁciency. Unlike the popular A3C-based agents, in which workers communicate gradients with respect to the parameters of the policy to a central parameter server, IMPALA actors communicate trajectories of experience (sequences of states, actions, and rewards) to a centralised learner. Since the learner in IMPALA has access to full trajectories of experience we use a GPU to perform updates on mini-batches of trajectories while aggressively parallelising all time independent operations. This type of decoupled architecture can achieve very high throughput.


time independent operations. This type of decoupled architecture can achieve very high throughput. However, because the policy used to generate a trajectory can lag behind the policy on the learner by several updates at the time of gradient calculation, learning becomes off-policy. Therefore, we introduce the V-trace off-policy actor-critic algorithm to correct for this harmful discrepancy. With the scalable architecture and V-trace combined, IMPALA achieves exceptionally high data throughput rates of 250,000 frames per second, making it over 30 times faster than single-machine A3C. Crucially, IMPALA is also more data efﬁcient than A3C based agents and more robust to hyperparameter values and network architectures, allowing it to make better use of deeper neural networks.


rameter values and network architectures, allowing it to make better use of deeper neural networks. We demonstrate the effectiveness of IMPALA by training a single agent on multi-task problems using DMLab-30, a new challenge set which consists of 30 diverse cognitive tasks in the 3D DeepMind Lab (Beattie et al., 2016) environment and by training a single agent on all games in the Atari-57 set of tasks. arXiv:1802.01561v3 [cs.LG] 28 Jun 2018


IMPALA: Importance Weighted Actor-Learner Architectures Actor Actor Actor Actor Actor Actor Learner Observations Parameters Actor Actor Observations Observations Parameters Gradients Learner Worker Master Learner Actor Actor Actor Figure 1. Left: Single Learner. Each actor generates trajectories and sends them via a queue to the learner. Before starting the next trajectory, actor retrieves the latest policy parameters from learner. Right: Multiple Synchronous Learners. Policy parameters are distributed across multiple learners that work synchronously. 2. Related Work The earliest attempts to scale up deep reinforcement learning relied on distributed asynchronous SGD (Dean et al., 2012) with multiple workers.


orcement learning relied on distributed asynchronous SGD (Dean et al., 2012) with multiple workers. Examples include distributed A3C (Mnih et al., 2016) and Gorila (Nair et al., 2015), a distributed version of Deep Q-Networks (Mnih et al., 2015). Recent alternatives to asynchronous SGD for RL include using evolutionary processes (Salimans et al., 2017), distributed BA3C (Adamski et al., 2018) and Ape-X (Horgan et al., 2018) which has a distributed replay but a synchronous learner. There have also been multiple efforts that scale up reinforcement learning by utilising GPUs. One of the simplest of such methods is batched A2C (Clemente et al., 2017). At every step, batched A2C produces a batch of actions and applies them to a batch of environments.


At every step, batched A2C produces a batch of actions and applies them to a batch of environments. Therefore, the slowest environment in each batch determines the time it takes to perform the entire batch step (see Figure 2a and 2b). In other words, high variance in environment speed can severely limit performance. Batched A2C works particularly well on Atari environments, because rendering and game logic are computationally very cheap in comparison to the expensive tensor operations performed by reinforcement learning agents. However, more visually or physically complex environments can be slower to simulate and can have high variance in the time required for each step. Environments may also have variable length (sub)episodes causing a slowdown when initialising an episode.


nments may also have variable length (sub)episodes causing a slowdown when initialising an episode. The most similar architecture to IMPALA is GA3C (Babaeizadeh et al., 2016), which also uses asynchronous data collection to more effectively utilise GPUs. It decouples the acting/forward pass from the gradient calculation/backward pass by using dynamic batching. The actor/learner asynchrony in GA3C leads to instabilities during learning, which (Babaeizadeh et al., 2016) only partially mitigates by adding a small constant to action probabilities Environment steps Forward pass Backward pass Actor 2 Actor 3 Actor 1 Actor 0 4 time steps (a) Batched A2C (sync step.) Actor 2 Actor 3 Actor 1 Actor 0 4 time steps (b) Batched A2C (sync traj.) … … Actor 2 Actor 3 Actor 1 Actor 0 Actor 4 Actor 5 Actor


4 time steps (b) Batched A2C (sync traj.) … … Actor 2 Actor 3 Actor 1 Actor 0 Actor 4 Actor 5 Actor 6 Actor 7 ...next unroll (c) IMPALA Figure 2. Timeline for one unroll with 4 steps using different architectures. Strategies shown in (a) and (b) can lead to low GPU utilisation due to rendering time variance within a batch. In (a), the actors are synchronised after every step. In (b) after every n steps. IMPALA (c) decouples acting from learning. during the estimation of the policy gradient. In contrast, IMPALA uses the more principled V-trace algorithm. Related previous work on off-policy RL include (Precup et al., 2000; 2001; Wawrzynski, 2009; Geist & Scherrer, 2014; O’Donoghue et al., 2017) and (Harutyunyan et al., 2016).


Wawrzynski, 2009; Geist & Scherrer, 2014; O’Donoghue et al., 2017) and (Harutyunyan et al., 2016). The closest work to ours is the Retrace algorithm (Munos et al., 2016) which introduced an off-policy correction for multi-step RL, and has been used in several agent architectures (Wang et al., 2017; Gruslys et al., 2018). Retrace requires learning state-action-value functions Q in order to make the off-policy correction. However, many actor-critic methods such as A3C learn a state-value function V instead of a state-action-value function Q. V-trace is based on the state-value function. 3. IMPALA IMPALA (Figure 1) uses an actor-critic setup to learn a policy π and a baseline function V π. The process of generating experiences is decoupled from learning the parameters of π and V π.


V π. The process of generating experiences is decoupled from learning the parameters of π and V π. The architecture consists of a set of actors, repeatedly generating trajectories of experience, and one or more learners that use the experiences sent from actors to learn π off-policy. At the beginning of each trajectory, an actor updates its own local policy µ to the latest learner policy π and runs it for n steps in its environment. After n steps, the actor sends the trajectory of states, actions and rewards x1, a1, r1, . . . , xn, an, rn together with the corresponding policy distributions µ(at|xt) and initial LSTM state to the learner through a queue. The learner then continuously updates its policy π on batches of trajectories, each collected from many actors.


then continuously updates its policy π on batches of trajectories, each collected from many actors. This simple architecture enables the learner(s) to be accelerated using GPUs and actors to be easily distributed across many machines. However, the learner policy π is potentially several updates ahead of the actor’s policy µ at the time of update, therefore there is a policy-lag between the actors and learner(s). V-trace cor


IMPALA: Importance Weighted Actor-Learner Architectures rects for this lag to achieve extremely high data throughput while maintaining data efﬁciency. Using an actor-learner architecture, provides fault tolerance like distributed A3C but often has lower communication overhead since the actors send observations rather than parameters/gradients. With the introduction of very deep model architectures, the speed of a single GPU is often the limiting factor during training. IMPALA can be used with distributed set of learners to train large neural networks efﬁciently as shown in Figure 1. Parameters are distributed across the learners and actors retrieve the parameters from all the learners in parallel while only sending observations to a single learner.


e parameters from all the learners in parallel while only sending observations to a single learner. IMPALA use synchronised parameter update which is vital to maintain data efﬁciency when scaling to many machines (Chen et al., 2016). 3.1. Efﬁciency Optimisations GPUs and many-core CPUs beneﬁt greatly from running few large, parallelisable operations instead of many small operations. Since the learner in IMPALA performs updates on entire batches of trajectories, it is able to parallelise more of its computations than an online agent like A3C. As an example, a typical deep RL agent features a convolutional network followed by a Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and a fully connected output layer after the LSTM.


m Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and a fully connected output layer after the LSTM. An IMPALA learner applies the convolutional network to all inputs in parallel by folding the time dimension into the batch dimension. Similarly, it also applies the output layer to all time steps in parallel once all LSTM states are computed. This optimisation increases the effective batch size to thousands. LSTM-based agents also obtain signiﬁcant speedups on the learner by exploiting the network structure dependencies and operation fusion (Appleyard et al., 2016). Finally, we also make use of several off the shelf optimisations available in TensorFlow (Abadi et al., 2017) such as preparing the next batch of data for the learner while still performing computation, compiling parts of the com


he next batch of data for the learner while still performing computation, compiling parts of the computational graph with XLA (a TensorFlow Just-In-Time compiler) and optimising the data format to get the maximum performance from the cuDNN framework (Chetlur et al., 2014). 4. V-trace Off-policy learning is important in the decoupled distributed actor-learner architecture because of the lag between when actions are generated by the actors and when the learner estimates the gradient. To this end, we introduce a novel offpolicy actor-critic algorithm for the learner, called V-trace. First, let us introduce some notations. We consider the problem of discounted inﬁnite-horizon RL in Markov Decision Processes (MDP), see (Puterman, 1994; Sutton & Barto, 1998) where the goal is to ﬁnd a policy π t


on Processes (MDP), see (Puterman, 1994; Sutton & Barto, 1998) where the goal is to ﬁnd a policy π that maximises the expected sum of future discounted rewards: V π(x) def = Eπ  P t≥0 γtrt  , where γ ∈[0, 1) is the discount factor, rt = r(xt, at) is the reward at time t, xt is the state at time t (initialised in x0 = x) and at ∼π(·|xt) is the action generated by following some policy π. The goal of an off-policy RL algorithm is to use trajectories generated by some policy µ, called the behaviour policy, to learn the value function V π of another policy π (possibly different from µ), called the target policy. 4.1. V-trace target Consider a trajectory (xt, at, rt)t=s+n t=s generated by the actor following some policy µ.


target Consider a trajectory (xt, at, rt)t=s+n t=s generated by the actor following some policy µ. We deﬁne the n-steps V-trace target for V (xs), our value approximation at state xs, as: vs def = V (xs) + Ps+n−1 t=s γt−s Qt−1 i=s ci  δtV , (1) where δtV def = ρt rt + γV (xt+1) −V (xt)  is a temporal difference for V , and ρt def = min ¯ρ, π(at|xt) µ(at|xt)  and ci def = min ¯c, π(ai|xi) µ(ai|xi)  are truncated importance sampling (IS) weights (we make use of the notation Qt−1 i=s ci = 1 for s = t). In addition we assume that the truncation levels are such that ¯ρ ≥¯c. Notice that in the on-policy case (when π = µ), and assuming that ¯c ≥1, then all ci = 1 and ρt = 1, thus (1) rewrites vs = V (xs) + Ps+n−1 t=s γt−srt + γV (xt+1) −V (xt)  = Ps+n−1 t=s γt−srt + γnV (xs+n), (2) whi


tes vs = V (xs) + Ps+n−1 t=s γt−srt + γV (xt+1) −V (xt)  = Ps+n−1 t=s γt−srt + γnV (xs+n), (2) which is the on-policy n-steps Bellman target. Thus in the on-policy case, V-trace reduces to the on-policy n-steps Bellman update. This property (which Retrace (Munos et al., 2016) does not have) allows one to use the same algorithm for off- and on-policy data. Notice that the (truncated) IS weights ci and ρt play different roles. The weight ρt appears in the deﬁnition of the temporal difference δtV and deﬁnes the ﬁxed point of this update rule. In a tabular case, where functions can be perfectly represented, the ﬁxed point of this update (i.e., when V (xs) = vs for all states), characterised by δtV being equal to zero in expectation (under µ), is the value function V π¯ ρ of some policy π¯ρ,


δtV being equal to zero in expectation (under µ), is the value function V π¯ ρ of some policy π¯ρ, deﬁned by π¯ρ(a|x) def = min ¯ρµ(a|x), π(a|x)  P b∈A min ¯ρµ(b|x), π(b|x) , (3) (see the analysis in Appendix A ). So when ¯ρ is inﬁnite (i.e. no truncation of ρt), then this is the value function V π of the target policy. However if we choose a truncation


IMPALA: Importance Weighted Actor-Learner Architectures level ¯ρ < ∞, our ﬁxed point is the value function V π¯ ρ of a policy π¯ρ which is somewhere between µ and π. At the limit when ¯ρ is close to zero, we obtain the value function of the behaviour policy V µ. In Appendix A we prove the contraction of a related V-trace operator and the convergence of the corresponding online V-trace algorithm. The weights ci are similar to the “trace cutting” coefﬁcients in Retrace. Their product cs . . . ct−1 measures how much a temporal difference δtV observed at time t impacts the update of the value function at a previous time s. The more dissimilar π and µ are (the more off-policy we are), the larger the variance of this product. We use the truncation level ¯c as a variance reduction technique.


ger the variance of this product. We use the truncation level ¯c as a variance reduction technique. However notice that this truncation does not impact the solution to which we converge (which is characterised by ¯ρ only). Thus we see that the truncation levels ¯c and ¯ρ represent different features of the algorithm: ¯ρ impacts the nature of the value function we converge to, whereas ¯c impacts the speed at which we converge to this function. Remark 1. V-trace targets can be computed recursively: vs = V (xs) + δsV + γcs vs+1 −V (xs+1)  . Remark 2. Like in Retrace(λ), we can also consider an additional discounting parameter λ ∈[0, 1] in the deﬁnition of V-trace by setting ci = λ min ¯c, π(ai|xi) µ(ai|xi)  . In the onpolicy case, when n = ∞, V-trace then reduces to TD(λ). 4.2.


in ¯c, π(ai|xi) µ(ai|xi)  . In the onpolicy case, when n = ∞, V-trace then reduces to TD(λ). 4.2. Actor-Critic algorithm POLICY GRADIENT In the on-policy case, the gradient of the value function V µ(x0) with respect to some parameter of the policy µ is ∇V µ(x0) = Eµ hP s≥0γs∇log µ(as|xs)Qµ(xs, as) i , where Qµ(xs, as) def = Eµ  P t≥s γt−srt|xs, as  is the state-action value of policy µ at (xs, as). This is usually implemented by a stochastic gradient ascent that updates the policy parameters in the direction of Eas∼µ(·|xs) h ∇log µ(as|xs)qs xs i , where qs is an estimate of Qµ(xs, as), and averaged over the set of states xs that are visited under some behaviour policy µ. Now in the off-policy setting that we consider, we can use an IS weight between the policy being evaluated π¯ρ and t


olicy setting that we consider, we can use an IS weight between the policy being evaluated π¯ρ and the behaviour policy µ, to update our policy parameter in the direction of Eas∼µ(·|xs) hπ¯ρ(as|xs) µ(as|xs) ∇log π¯ρ(as|xs)qs xs i (4) where qs def = rs + γvs+1 is an estimate of Qπ¯ ρ(xs, as) built from the V-trace estimate vs+1 at the next state xs+1. The reason why we use qs instead of vs as the target for our Q-value Qπ¯ ρ(xs, as) is that, assuming our value estimate is correct at all states, i.e. V = V π¯ ρ, then we have E[qs|xs, as] = Qπ¯ ρ(xs, as) (whereas we do not have this property if we choose qt = vt). See Appendix A for analysis and Appendix E.3 for a comparison of different ways to estimate qs.


t). See Appendix A for analysis and Appendix E.3 for a comparison of different ways to estimate qs. In order to reduce the variance of the policy gradient estimate (4), we usually subtract from qs a state-dependent baseline, such as the current value approximation V (xs). Finally notice that (4) estimates the policy gradient for π¯ρ which is the policy evaluated by the V-trace algorithm when using a truncation level ¯ρ. However assuming the bias V π¯ ρ −V π is small (e.g. if ¯ρ is large enough) then we can expect qs to provide us with a good estimate of Qπ(xs, as). Taking into account these remarks, we derive the following canonical V-trace actor-critic algorithm. V-TRACE ACTOR-CRITIC ALGORITHM Consider a parametric representation Vθ of the value function and the current policy πω.


ALGORITHM Consider a parametric representation Vθ of the value function and the current policy πω. Trajectories have been generated by actors following some behaviour policy µ. The V-trace targets vs are deﬁned by (1). At training time s, the value parameters θ are updated by gradient descent on the l2 loss to the target vs, i.e., in the direction of vs −Vθ(xs)  ∇θVθ(xs), and the policy parameters ω in the direction of the policy gradient: ρs∇ω log πω(as|xs) rs + γvs+1 −Vθ(xs)  . In order to prevent premature convergence we may add an entropy bonus, like in A3C, along the direction −∇ω X a πω(a|xs) log πω(a|xs). The overall update is obtained by summing these three gradients rescaled by appropriate coefﬁcients, which are hyperparameters of the algorithm. 5.


three gradients rescaled by appropriate coefﬁcients, which are hyperparameters of the algorithm. 5. Experiments We investigate the performance of IMPALA under multiple settings. For data efﬁciency, computational performance and effectiveness of the off-policy correction we look at the learning behaviour of IMPALA agents trained on individual tasks. For multi-task learning we train agents—each with one set of weights for all tasks—on a newly introduced collection of 30 DeepMind Lab tasks and on all 57 games of the Atari Learning Environment (Bellemare et al., 2013a). For all the experiments we have used two different model architectures: a shallow model similar to (Mnih et al., 2016)


IMPALA: Importance Weighted Actor-Learner Architectures /255 Conv. 8 ⇥8, stride 4 ReLU Conv. 4 ⇥4, stride 2 ReLU FC 256 ReLU Vt rt−1 at−1 32 16 3 96 ⇥72 LSTM 256 ht−1 ⇡(at) LSTM 64 Embedding 20 blue ladder + Conv. 3 ⇥3, stride 1 ReLU Conv. 3 ⇥3, stride 1 /255 Conv. 3 ⇥3, stride 1 FC 256 ReLU Max 3 ⇥3, stride 2 Residual Block Residual Block ⇥3 ReLU [16, 32, 32] ch. LSTM 256 Vt rt−1 at−1 LSTM 256 ht−1 ⇡(at) 96 ⇥72 3 ReLU LSTM 64 Embedding 20 blue ladder Figure 3. Model Architectures. Left: Small architecture, 2 convolutional layers and 1.2 million parameters. Right: Large architecture, 15 convolutional layers and 1.6 million parameters. Architecture CPUs GPUs1 FPS2 Single-Machine Task 1 Task 2 A3C 32 workers 64 0 6.5K 9K Batched A2C (sync step) 48 0 9K 5K Batched A2C (sync step) 48 1 13K 5.5


A3C 32 workers 64 0 6.5K 9K Batched A2C (sync step) 48 0 9K 5K Batched A2C (sync step) 48 1 13K 5.5K Batched A2C (sync traj.) 48 0 16K 17.5K Batched A2C (dyn. batch) 48 1 16K 13K IMPALA 48 actors 48 0 17K 20.5K IMPALA (dyn. batch) 48 actors3 48 1 21K 24K Distributed A3C 200 0 46K 50K IMPALA 150 1 80K IMPALA (optimised) 375 1 200K IMPALA (optimised) batch 128 500 1 250K 1 Nvidia P100 2 In frames/sec (4 times the agent steps due to action repeat). 3 Limited by amount of rendering possible on a single machine. Table 1. Throughput on seekavoid arena 01 (task 1) and rooms keys doors puzzle (task 2) with the shallow model in Figure 3. The latter has variable length episodes and slow restarts. Batched A2C and IMPALA use batch size 32 if not otherwise mentioned.


th episodes and slow restarts. Batched A2C and IMPALA use batch size 32 if not otherwise mentioned. with an LSTM before the policy and value (shown in Figure 3 (left)) and a deeper residual model (He et al., 2016) (shown in Figure 3 (right)). For tasks with a language channel we used an LSTM with text embeddings as input. 5.1. Computational Performance High throughput, computational efﬁciency and scalability are among the main design goals of IMPALA. To demonstrate that IMPALA outperforms current algorithms in these metrics we compare A3C (Mnih et al., 2016), batched A2C variations and IMPALA variants with various optimisations. For single-machine experiments using GPUs, we use dynamic batching in the forward pass to avoid several batch size 1 forward passes.


ing GPUs, we use dynamic batching in the forward pass to avoid several batch size 1 forward passes. Our dynamic batching module is implemented by specialised TensorFlow operations but is conceptually similar to the queues used in GA3C. Table 1 details the results for single-machine and multi-machine versions with the shallow model from Figure 3. In the singlemachine case, IMPALA achieves the highest performance on both tasks, ahead of all batched A2C variants and ahead of A3C. However, the distributed, multi-machine setup is where IMPALA can really demonstrate its scalability. With the optimisations from Section 3.1 to speed up the GPUbased learner, the IMPALA agent achieves a throughput rate of 250,000 frames/sec or 21 billion frames/day.


earner, the IMPALA agent achieves a throughput rate of 250,000 frames/sec or 21 billion frames/day. Note, to reduce the number of actors needed per learner, one can use auxiliary losses, data from experience replay or other expensive learner-only computation. 5.2. Single-Task Training To investigate IMPALA’s learning dynamics, we employ the single-task scenario where we train agents individually on 5 different DeepMind Lab tasks. The task set consists of a planning task, two maze navigation tasks, a laser tag task with scripted bots and a simple fruit collection task. We perform hyperparameter sweeps over the weighting of entropy regularisation, the learning rate and the RMSProp epsilon. For each experiment we use an identical set of 24 pre-sampled hyperparameter combinations from the rang


each experiment we use an identical set of 24 pre-sampled hyperparameter combinations from the ranges in Appendix D.1 . The other hyperparameters were ﬁxed to values speciﬁed in Appendix D.3 . 5.2.1. CONVERGENCE AND STABILITY Figure 4 shows a comparison between IMPALA, A3C and batched A2C with the shallow model in Figure 3. In all of the 5 tasks, either batched A2C or IMPALA reach the best ﬁnal average return and in all tasks but seekavoid arena 01 they are ahead of A3C throughout the entire course of training. IMPALA outperforms the synchronous batched A2C on 2 out of 5 tasks while achieving much higher throughput (see Table 1). We hypothesise that this behaviour could stem from the V-trace off-policy correction acting similarly to generalised advantage estimation (Schulman et al., 2016)


e off-policy correction acting similarly to generalised advantage estimation (Schulman et al., 2016) and asynchronous data collection yielding more diverse batches of experience. In addition to reaching better ﬁnal performance, IMPALA is also more robust to the choice of hyperparameters than A3C. Figure 4 compares the ﬁnal performance of the aforementioned methods across different hyperparameter combinations, sorted by average ﬁnal return from high to low. Note that IMPALA achieves higher scores over a larger number of combinations than A3C. 5.2.2. V-TRACE ANALYSIS To analyse V-trace we investigate four different algorithms: 1. No-correction - No off-policy correction.


IMPALA: Importance Weighted Actor-Learner Architectures IMPALA - 1 GPU - 200 actors Batched A2C - Single Machine - 32 workers A3C - Single Machine - 32 workers A3C - Distributed - 200 workers 0.0 0.2 0.4 0.6 0.8 1.0 Environment Frames 1e9 10 15 20 25 30 35 40 45 50 55 Return rooms_watermaze 0.0 0.2 0.4 0.6 0.8 1.0 Environment Frames 1e9 0 5 10 15 20 25 30 rooms_keys_doors_puzzle 0.0 0.2 0.4 0.6 0.8 1.0 Environment Frames 1e9 −5 0 5 10 15 20 25 30 35 lasertag_three_opponents_small 0.0 0.2 0.4 0.6 0.8 1.0 Environment Frames 1e9 0 50 100 150 200 250 explore_goal_locations_small 0.0 0.2 0.4 0.6 0.8 1.0 Environment Frames 1e9 5 10 15 20 25 30 35 40 45 seekavoid_arena_01 1 5 9 13 17 21 24 Hyperparameter Combination 0 10 20 30 40 50 60 Final Return rooms_watermaze 1 5 9 13 17 21 24 Hyperparameter


ameter Combination 0 10 20 30 40 50 60 Final Return rooms_watermaze 1 5 9 13 17 21 24 Hyperparameter Combination 0 5 10 15 20 25 30 35 40 rooms_keys_doors_puzzle 1 5 9 13 17 21 24 Hyperparameter Combination −5 0 5 10 15 20 25 30 35 40 lasertag_three_opponents_small 1 5 9 13 17 21 24 Hyperparameter Combination 0 50 100 150 200 250 300 explore_goal_locations_small 1 5 9 13 17 21 24 Hyperparameter Combination 0 10 20 30 40 50 seekavoid_arena_01 Figure 4. Top Row: Single task training on 5 DeepMind Lab tasks. Each curve is the mean of the best 3 runs based on ﬁnal return. IMPALA achieves better performance than A3C. Bottom Row: Stability across hyperparameter combinations sorted by the ﬁnal performance across different hyperparameter combinations. IMPALA is consistently more stable than A3C.


formance across different hyperparameter combinations. IMPALA is consistently more stable than A3C. Task 1 Task 2 Task 3 Task 4 Task 5 Without Replay V-trace 46.8 32.9 31.3 229.2 43.8 1-Step 51.8 35.9 25.4 215.8 43.7 ε-correction 44.2 27.3 4.3 107.7 41.5 No-correction 40.3 29.1 5.0 94.9 16.1 With Replay V-trace 47.1 35.8 34.5 250.8 46.9 1-Step 54.7 34.4 26.4 204.8 41.6 ε-correction 30.4 30.2 3.9 101.5 37.6 No-correction 35.0 21.1 2.8 85.0 11.2 Tasks: rooms watermaze, rooms keys doors puzzle, lasertag three opponents small, explore goal locations small, seekavoid arena 01 Table 2. Average ﬁnal return over 3 best hyperparameters for different off-policy correction methods on 5 DeepMind Lab tasks.


rn over 3 best hyperparameters for different off-policy correction methods on 5 DeepMind Lab tasks. When the lag in policy is negligible both V-trace and 1-step importance sampling perform similarly well and better than ε-correction/Nocorrection. However, when the lag increases due to use of experience replay, V-trace performs better than all other methods in 4 out 5 tasks. 2. ε-correction - Add a small value (ε = 1e-6) during gradient calculation to prevent log π(a) from becoming very small and leading to numerical instabilities, similar to (Babaeizadeh et al., 2016). 3. 1-step importance sampling - No off-policy correction when optimising V (x). For the policy gradient, multiply the advantage at each time step by the corresponding importance weight.


e policy gradient, multiply the advantage at each time step by the corresponding importance weight. This variant is similar to V-trace without “traces” and is included to investigate the importance of “traces” in V-trace. 4. V-trace as described in Section 4. For V-trace and 1-step importance sampling we clip each importance weight ρt and ct at 1 (i.e. ¯c = ¯ρ = 1). This reduces the variance of the gradient estimate but introduces a bias. Out of ¯ρ ∈[1, 10, 100] we found that ¯ρ = 1 worked best. We evaluate all algorithms on the set of 5 DeepMind Lab tasks from the previous section. We also add an experience replay buffer on the learner to increase the off-policy gap between π and µ. In the experience replay experiments we draw 50% of the items in each batch uniformly at random from the re


experience replay experiments we draw 50% of the items in each batch uniformly at random from the replay buffer. Table 2 shows the ﬁnal performance for each algorithm with and without replay respectively. In the no replay setting, V-trace performs best on 3 out of 5 tasks, followed by 1-step importance sampling, ε-correction and No-correction. Although 1-step importance sampling performs similarly to V-trace in the no-replay setting, the gap widens on 4 out 5 tasks when using experience replay. This suggests that the cruder 1-step importance sampling approximation becomes insufﬁcient as the target and behaviour policies deviate from each other more strongly. Also note that V-trace is the only variant that consistently beneﬁts from adding experience replay.


Also note that V-trace is the only variant that consistently beneﬁts from adding experience replay. ε-correction improves signiﬁcantly over No-correction on two tasks but lies far behind the importance-sampling based methods, particularly in the more off-policy setting with experience replay. Figure E.1 shows results of a more detailed analysis. Figure E.2 shows that the importance-sampling based methods also perform better across all hyperparameters and are typically more robust. 5.3. Multi-Task Training IMPALA’s high data throughput and data efﬁciency allow us to train not only on one task but on multiple tasks in parallel with only a minimal change to the training setup. Instead of running the same task on all actors, we allocate a ﬁxed number of actors to each task in the multi-task su


g the same task on all actors, we allocate a ﬁxed number of actors to each task in the multi-task suite. Note, the model does not know which task it is being trained or evaluated on.


IMPALA: Importance Weighted Actor-Learner Architectures Model Test score A3C, deep 23.8% IMPALA, shallow 37.1% IMPALA-Experts, deep 44.5% IMPALA, deep 46.5% IMPALA, deep, PBT 49.4% IMPALA, deep, PBT, 8 learners 49.1% Table 3. Mean capped human normalised scores on DMLab-30. All models were evaluated on the test tasks with 500 episodes per task. The table shows the best score for each architecture. 5.3.1. DMLAB-30 To test IMPALA’s performance in a multi-task setting we use DMLab-30, a set of 30 diverse tasks built on DeepMind Lab. Among the many task types in the suite are visually complex environments with natural-looking terrain, instruction-based tasks with grounded language (Hermann et al., 2017), navigation tasks, cognitive (Leibo et al., 2018) and ﬁrst-person tagging tasks featuring s


., 2017), navigation tasks, cognitive (Leibo et al., 2018) and ﬁrst-person tagging tasks featuring scripted bots as opponents. A detailed description of DMLab-30 and the tasks are available at github.com/deepmind/lab and deepmind.com/dm-lab-30. We compare multiple variants of IMPALA with a distributed A3C implementation. Except for agents using populationbased training (PBT) (Jaderberg et al., 2017a), all agents are trained with hyperparameter sweeps across the same range given in Appendix D.1 . We report mean capped human normalised score where the score for each task is capped at 100% (see Appendix B ). Using mean capped human normalised score emphasises the need to solve multiple tasks instead of focusing on becoming super human on a single task.


ises the need to solve multiple tasks instead of focusing on becoming super human on a single task. For PBT we use the mean capped human normalised score as ﬁtness function and tune entropy cost, learning rate and RMSProp ε. See Appendix F for the speciﬁcs of the PBT setup. In particular, we compare the following agent variants. A3C, deep, a distributed implementation with 210 workers (7 per task) featuring the deep residual network architecture (Figure 3 (Right)). IMPALA, shallow with 210 actors and IMPALA, deep with 150 actors both with a single learner. IMPALA, deep, PBT, the same as IMPALA, deep, but additionally using the PBT (Jaderberg et al., 2017a) for hyperparameter optimisation. Finally IMPALA, deep, PBT, 8 learners, which utilises 8 learner GPUs to maximise learning speed.


n. Finally IMPALA, deep, PBT, 8 learners, which utilises 8 learner GPUs to maximise learning speed. We also train IMPALA agents in an expert setting, IMPALA-Experts, deep, where a separate agent is trained per task. In this case we did not optimise hyperparameters for each task separately but instead across all tasks on which the 30 expert agents were trained. Table 3 and Figure 5 show all variants of IMPALA performing much better than the deep distributed A3C. Moreover, the deep variant of IMPALA performs better than the shallow network version not only in terms of ﬁnal performance but throughout the entire training. Note in Table 3 that IMPALA, deep, PBT, 8 learners, although providing much higher throughput, reaches the same ﬁnal performance as the 1 GPU IMPALA, deep, PBT in the same nu


h higher throughput, reaches the same ﬁnal performance as the 1 GPU IMPALA, deep, PBT in the same number of steps. Of particular importance is the gap between the IMPALAExperts which were trained on each task individually and IMPALA, deep, PBT which was trained on all tasks at once. As Figure 5 shows, the multi-task version is outperforms IMPALA-Experts throughout training and the breakdown into individual scores in Appendix B shows positive transfer on tasks such as language tasks and laser tag tasks. Comparing A3C to IMPALA with respect to wall clock time (Figure 6) further highlights the scalability gap between the two approaches. IMPALA with 1 learner takes only around 10 hours to reach the same performance that A3C approaches after 7.5 days.


earner takes only around 10 hours to reach the same performance that A3C approaches after 7.5 days. Using 8 learner GPUs instead of 1 further speeds up training of the deep model by a factor of 7 to 210K frames/sec, up from 30K frames/sec. 5.3.2. ATARI The Atari Learning Environment (ALE) (Bellemare et al., 2013b) has been the testing ground of most recent deep reinforcement agents. Its 57 tasks pose challenging reinforcement learning problems including exploration, planning, reactive play and complex visual input. Most games feature very different visuals and game mechanics which makes this domain particularly challenging for multi-task learning. We train IMPALA and A3C agents on each game individually and compare their performance using the deep network (without the LSTM) introduced in S


individually and compare their performance using the deep network (without the LSTM) introduced in Section 5. We also provide results using a shallow network that is equivalent to the feed forward network used in (Mnih et al., 2016) which features three convolutional layers. The network is provided with a short term history by stacking the 4 most recent observations at each step. For details on pre-processing and hyperparameter setup please refer to Appendix G . In addition to individual per-game experts, trained for 200 million frames with a ﬁxed set of hyperparameters, we train an IMPALA Atari-57 agent—one agent, one set of weights— on all 57 Atari games at once for 200 million frames per game or a total of 11.4 billion frames.


s— on all 57 Atari games at once for 200 million frames per game or a total of 11.4 billion frames. For the Atari-57 agent, we use population based training with a population size of 24 to adapt entropy regularisation, learning rate, RMSProp ε and the global gradient norm clipping threshold throughout training. We compare all algorithms in terms of median human normalised score across all 57 Atari games. Evaluation follows a standard protocol, each game-score is the mean over 200 evaluation episodes, each episode was started with a random


IMPALA: Importance Weighted Actor-Learner Architectures Figure 5. Performance of best agent in each sweep/population during training on the DMLab-30 task-set wrt. data consumed across all environments. IMPALA with multi-task training is not only faster, it also converges at higher accuracy with better data efﬁciency across all 30 tasks. The x-axis is data consumed by one agent out of a hyperparameter sweep/PBT population of 24 agents, total data consumed across the whole population/sweep can be obtained by multiplying with the population/sweep size. 0.0 0.2 0.4 0.6 0.8 1.0 Environment Frames 1e10 0 10 20 30 40 50 60 Mean Capped Normalized Score 0 20 40 60 80 100 120 140 160 180 Wall Clock Time (hours) 0 10 20 30 40 50 60 Mean Capped Normalized Score IMPALA, deep, PBT - 8 GPUs IMPALA, deep,


me (hours) 0 10 20 30 40 50 60 Mean Capped Normalized Score IMPALA, deep, PBT - 8 GPUs IMPALA, deep, PBT IMPALA, deep IMPALA, shallow IMPALA-Experts, deep A3C, deep Figure 6. Performance on DMLab-30 wrt. wall-clock time. All models used the deep architecture (Figure 3). The high throughput of IMPALA results in orders of magnitude faster learning. number of no-op actions (uniformly chosen from [1, 30]) to combat the determinism of the ALE environment. As table 4 shows, IMPALA experts provide both better ﬁnal performance and data efﬁciency than their A3C counterparts in the deep and the shallow conﬁguration. As in our DeepMind Lab experiments, the deep residual network leads to higher scores than the shallow network, irrespective of the reinforcement learning algorithm used.


higher scores than the shallow network, irrespective of the reinforcement learning algorithm used. Note that the shallow IMPALA experiment completes training over 200 million frames in less than one hour. We want to particularly emphasise that IMPALA, deep, multitask, a single agent trained on all 57 ALE games at once, reaches 59.7% median human normalised score. Despite Human Normalised Return Median Mean A3C, shallow, experts 54.9% 285.9% A3C, deep, experts 117.9% 503.6% Reactor, experts 187% N/A IMPALA, shallow, experts 93.2% 466.4% IMPALA, deep, experts 191.8% 957.6% IMPALA, deep, multi-task 59.7% 176.9% Table 4. Human normalised scores on Atari-57. Up to 30 no-ops at the beginning of each episode.


.9% Table 4. Human normalised scores on Atari-57. Up to 30 no-ops at the beginning of each episode. For a level-by-level comparison to ACKTR (Wu et al., 2017) and Reactor see Appendix C.1 . the high diversity in visual appearance and game mechanics within the ALE suite, IMPALA multi-task still manages to stay competitive to A3C, shallow, experts, commonly used as a baseline in related work. ALE is typically considered a hard multi-task environment, often accompanied by negative transfer between tasks (Rusu et al., 2016). To our knowledge, IMPALA is the ﬁrst agent to be trained in a multi-task setting on all 57 games of ALE that is competitive with a standard expert baseline. 6. Conclusion We have introduced a new highly scalable distributed agent, IMPALA, and a new off-policy learning algo


have introduced a new highly scalable distributed agent, IMPALA, and a new off-policy learning algorithm, V-trace. With its simple but scalable distributed architecture, IMPALA can make efﬁcient use of available compute at small and large scale. This directly translates to very quick turnaround for investigating new ideas and opens up unexplored opportunities. V-trace is a general off-policy learning algorithm that is more stable and robust compared to other off-policy correction methods for actor critic agents. We have demonstrated that IMPALA achieves better performance compared to A3C variants in terms of data efﬁciency, stability and ﬁnal performance. We have further evaluated IMPALA on the new DMLab-30 set and the Atari-57 set.


nd ﬁnal performance. We have further evaluated IMPALA on the new DMLab-30 set and the Atari-57 set. To the best of our knowledge, IMPALA is the ﬁrst Deep-RL agent that has been successfully tested in such large-scale multi-task settings and it has shown superior performance compared to A3C based agents (49.4% vs. 23.8% human normalised score on DMLab-30). Most importantly, our experiments on DMLab-30 show that, in the multi-task setting, positive transfer between individual tasks lead IMPALA to achieve better performance compared to the expert training setting. We believe that IMPALA provides a simple yet scalable and robust framework for building better Deep-RL agents and has the potential to enable research on new challenges.


IMPALA: Importance Weighted Actor-Learner Architectures Acknowledgements We would like to thank Denis Teplyashin, Ricardo Barreira, Manuel Sanchez for their work improving the performance on DMLab-30 environments and Matteo Hessel, Jony Hudson, Igor Babuschkin, Max Jaderberg, Ivo Danihelka, Jacob Menick and David Silver for their comments and insightful discussions. References Abadi, M., Isard, M., and Murray, D. G. A computational model for tensorﬂow: An introduction. In Proceedings of the 1st ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2017, 2017. ISBN 978-1-4503-5071-6. Adamski, I., Adamski, R., Grel, T., Jedrych, A., Kaczmarek, K., and Michalewski, H. Distributed deep reinforcement learning: Learn how to play atari games in 21 minutes.


chalewski, H. Distributed deep reinforcement learning: Learn how to play atari games in 21 minutes. CoRR, abs/1801.02852, 2018. Appleyard, J., Kocisk´y, T., and Blunsom, P. Optimizing performance of recurrent neural networks on gpus. CoRR, abs/1604.01946, 2016. Babaeizadeh, M., Frosio, I., Tyree, S., Clemons, J., and Kautz, J. GA3C: GPU-based A3C for deep reinforcement learning. NIPS Workshop, 2016. Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Tirumala, D., Muldal, A., Heess, N., and Lillicrap, T. Distributional policy gradients. ICLR, 2018. Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Kuttler, H., Lefrancq, A., Green, S., Valdes, V., Sadik, A., Schrittwieser, J., Anderson, K., York, S., Cant, M., Cain, A., Bolton, A., Gaffney, S., King, H.,


, Schrittwieser, J., Anderson, K., York, S., Cant, M., Cain, A., Bolton, A., Gaffney, S., King, H., Hassabis, D., Legg, S., and Petersen, S. Deepmind lab. CoRR, abs/1612.03801, 2016. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 47:253–279, June 2013a. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res.(JAIR), 47:253–279, 2013b. Chen, J., Monga, R., Bengio, S., and J´ozefowicz, R. Revisiting distributed synchronous SGD. CoRR, abs/1604.00981, 2016. Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., and Shelhamer, E.


. Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., and Shelhamer, E. cudnn: Efﬁcient primitives for deep learning. CoRR, abs/1410.0759, 2014. Clemente, A. V., Mart´ınez, H. N. C., and Chandra, A. Efﬁcient parallel methods for deep reinforcement learning. CoRR, abs/1705.04862, 2017. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., and Ng, A. Y. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, pp. 1223–1231, 2012. Geist, M. and Scherrer, B. Off-policy learning with eligibility traces: A survey. The Journal of Machine Learning Research, 15(1):289–333, 2014. Gruslys, A., Dabney, W., Azar, M. G., Piot, B., Bellemare, M. G., and Munos, R.


5(1):289–333, 2014. Gruslys, A., Dabney, W., Azar, M. G., Piot, B., Bellemare, M. G., and Munos, R. The Reactor: A fast and sample-efﬁcient actor-critic agent for reinforcement learning. ICLR, 2018. Harutyunyan, A., Bellemare, M. G., Stepleton, T., and Munos, R. Q(λ) with Off-Policy Corrections, pp. 305– 320. Springer International Publishing, Cham, 2016. He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In European Conference on Computer Vision, pp. 630–645. Springer, 2016. Hermann, K. M., Hill, F., Green, S., Wang, F., Faulkner, R., Soyer, H., Szepesvari, D., Czarnecki, W., Jaderberg, M., Teplyashin, D., et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017. Hochreiter, S. and Schmidhuber, J.


in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D. Distributed prioritized experience replay. ICLR, 2018. Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K. Population based training of neural networks. CoRR, abs/1711.09846, 2017a. Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. ICLR, 2017b. Leibo, J. Z., d’Autume, C. d.


einforcement learning with unsupervised auxiliary tasks. ICLR, 2017b. Leibo, J. Z., d’Autume, C. d. M., Zoran, D., Amos, D., Beattie, C., Anderson, K., Casta˜neda, A. G., Sanchez, M., Green, S., Gruslys, A., et al. Psychlab: A psychology laboratory for deep reinforcement learning agents. arXiv preprint arXiv:1801.08116, 2018.


IMPALA: Importance Weighted Actor-Learner Architectures Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529–533, 2015. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. ICML, 2016. Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efﬁcient off-policy reinforcement learning.


pleton, T., Harutyunyan, A., and Bellemare, M. Safe and efﬁcient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1046–1054, 2016. Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., Maria, A. D., Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu, K., and Silver, D. Massively parallel methods for deep reinforcement learning. CoRR, abs/1507.04296, 2015. O’Donoghue, B., Munos, R., Kavukcuoglu, K., and Mnih, V. Combining policy gradient and Q-learning. In ICLR, 2017. Precup, D., Sutton, R. S., and Singh, S. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, 2000. Precup, D., Sutton, R. S., and Dasgupta, S.


nth International Conference on Machine Learning, 2000. Precup, D., Sutton, R. S., and Dasgupta, S. Off-policy temporal-difference learning with function approximation. In Proceedings of the 18th International Conference on Machine Laerning, pp. 417–424, 2001. Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1994. ISBN 0471619779. Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. Salimans, T., Ho, J., Chen, X., and Sutskever, I. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.


ategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of go with deep neural networks and tree search. Nature, 529:484–503, 2016. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui,


I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., Driessche, G. v. d., Graepel, T., and Hassabis, D. Mastering the game of go without human knowledge. Nature, 550(7676):354–359, 10 2017. ISSN 0028-0836. doi: 10.1038/nature24270. Sutton, R. and Barto, A. Reinforcement learning: An introduction, volume 116. Cambridge Univ Press, 1998. Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., and de Freitas, N. Sample efﬁcient actor-critic with experience replay. In ICLR, 2017. Wawrzynski, P. Real-time reinforcement learning by sequential actor-critics and experience replay. Neural Networks, 22(10):1484–1497, 2009. Wu, Y., Mansimov, E., Liao, S., Grosse, R. B., and Ba, J.


Neural Networks, 22(10):1484–1497, 2009. Wu, Y., Mansimov, E., Liao, S., Grosse, R. B., and Ba, J. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. CoRR, abs/1708.05144, 2017. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning transferable architectures for scalable image recognition. arXiv preprint arXiv:1707.07012, 2017.


Supplementary Material A. Analysis of V-trace A.1. V-trace operator Deﬁne the V-trace operator R: RV (x) def = V (x) + Eµ h X t≥0 γtc0 . . . ct−1  ρt rt + γV (xt+1) −V (xt)  x0 = x, µ i , (5) where the expectation Eµ is with respect to the policy µ which has generated the trajectory (xt)t≥0, i.e., x0 = x, xt+1 ∼p(·|xt, at), at ∼µ(·|xt). Here we consider the inﬁnite-horizon operator but very similar results hold for the n-step truncated operator. Theorem 1. Let ρt = min ¯ρ, π(at|xt) µ(at|xt)  and ct = min ¯c, π(at|xt) µ(at|xt)  be truncated importance sampling weights, with ¯ρ ≥¯c. Assume that there exists β ∈(0, 1] such that Eµρ0 ≥β. Then the operator R deﬁned by (5) has a unique ﬁxed point V π¯ ρ, which is the value function of the policy π¯ρ deﬁned by π¯ρ(a|x) def = min ¯ρµ(a|x)


d point V π¯ ρ, which is the value function of the policy π¯ρ deﬁned by π¯ρ(a|x) def = min ¯ρµ(a|x), π(a|x)  P b∈A min ¯ρµ(b|x), π(b|x) , (6) Furthermore, R is a η-contraction mapping in sup-norm, with η def = γ−1 −(γ−1 −1)Eµ  X t≥0 γt t−2 Y i=0 ci  ρt−1  ≤1 −(1 −γ)β < 1. Remark 3. The truncation levels ¯c and ¯ρ play different roles in this operator: • ¯ρ impacts the ﬁxed-point of the operator, thus the policy π¯ρ which is evaluated. For ¯ρ = ∞(untruncated ρt) we get the value function of the target policy V π, whereas for ﬁnite ¯ρ, we evaluate a policy which is in between µ and π (and when ρ is close to 0, then we evaluate V µ). So the larger ¯ρ the smaller the bias in off-policy learning. The variance naturally grows with ¯ρ.


So the larger ¯ρ the smaller the bias in off-policy learning. The variance naturally grows with ¯ρ. However notice that we do not take the product of those ρt coefﬁcients (in contrast to the cs coefﬁcients) so the variance does not explode with the time horizon. • ¯c impacts the contraction modulus η of R (thus the speed at which an online-algorithm like V-trace will converge to its ﬁxed point V π¯ ρ). In terms of variance reduction, here is it really important to truncate the importance sampling ratios in ct because we take the product of those. Fortunately, our result says that for any level of truncation ¯c, the ﬁxed point (the value function V π¯ ρ we converge to) is the same: it does not depend on ¯c but on ¯ρ only. Proof.


value function V π¯ ρ we converge to) is the same: it does not depend on ¯c but on ¯ρ only. Proof. First notice that we can rewrite R as RV (x) = (1 −Eµρ0)V (x) + Eµ  X t≥0 γt t−1 Y s=0 cs  ρtrt + γ[ρt −ctρt+1]V (xt+1)   . Thus RV1(x) −RV2(x) = (1 −Eµρ0)  V1(x) −V2(x)  + Eµ  X t≥0 γt+1 t−1 Y s=0 cs  [ρt −ctρt+1]  V1(xt+1) −V2(xt+1)   . = Eµ   X t≥0 γt t−2 Y s=0 cs  [ρt−1 −ct−1ρt | {z } αt ]  V1(xt) −V2(xt)   ,