Playing Atari with Deep Reinforcement Learning Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra Martin Riedmiller DeepMind Technologies {vlad,koray,david,alex.graves,ioannis,daan,martin.riedmiller} @ deepmind.com Abstract We present the ﬁrst deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm.


from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We ﬁnd that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them. 1 Introduction Learning to control agents directly from high-dimensional sensory inputs like vision and speech is one of the long-standing challenges of reinforcement learning (RL). Most successful RL applications that operate on these domains have relied on hand-crafted features combined with linear value functions or policy representations. Clearly, the performance of such systems heavily relies on the quality of the feature representation. Recent advances in deep learning have made it possible to extract high-level features from raw sensory data, leading to breakthroughs i


ve made it possible to extract high-level features from raw sensory data, leading to breakthroughs in computer vision [11, 22, 16] and speech recognition [6, 7]. These methods utilise a range of neural network architectures, including convolutional networks, multilayer perceptrons, restricted Boltzmann machines and recurrent neural networks, and have exploited both supervised and unsupervised learning. It seems natural to ask whether similar techniques could also be beneﬁcial for RL with sensory data. However reinforcement learning presents several challenges from a deep learning perspective. Firstly, most successful deep learning applications to date have required large amounts of handlabelled training data.


ssful deep learning applications to date have required large amounts of handlabelled training data. RL algorithms, on the other hand, must be able to learn from a scalar reward signal that is frequently sparse, noisy and delayed. The delay between actions and resulting rewards, which can be thousands of timesteps long, seems particularly daunting when compared to the direct association between inputs and targets found in supervised learning. Another issue is that most deep learning algorithms assume the data samples to be independent, while in reinforcement learning one typically encounters sequences of highly correlated states. Furthermore, in RL the data distribution changes as the algorithm learns new behaviours, which can be problematic for deep learning methods that assume a ﬁxed unde


hm learns new behaviours, which can be problematic for deep learning methods that assume a ﬁxed underlying distribution. This paper demonstrates that a convolutional neural network can overcome these challenges to learn successful control policies from raw video data in complex RL environments. The network is trained with a variant of the Q-learning [26] algorithm, with stochastic gradient descent to update the weights. To alleviate the problems of correlated data and non-stationary distributions, we use 1 arXiv:1312.5602v1 [cs.LG] 19 Dec 2013


Figure 1: Screen shots from ﬁve Atari 2600 Games: (Left-to-right) Pong, Breakout, Space Invaders, Seaquest, Beam Rider an experience replay mechanism [13] which randomly samples previous transitions, and thereby smooths the training distribution over many past behaviors. We apply our approach to a range of Atari 2600 games implemented in The Arcade Learning Environment (ALE) [3]. Atari 2600 is a challenging RL testbed that presents agents with a high dimensional visual input (210 × 160 RGB video at 60Hz) and a diverse and interesting set of tasks that were designed to be difﬁcult for humans players. Our goal is to create a single neural network agent that is able to successfully learn to play as many of the games as possible.


e neural network agent that is able to successfully learn to play as many of the games as possible. The network was not provided with any game-speciﬁc information or hand-designed visual features, and was not privy to the internal state of the emulator; it learned from nothing but the video input, the reward and terminal signals, and the set of possible actions—just as a human player would. Furthermore the network architecture and all hyperparameters used for training were kept constant across the games. So far the network has outperformed all previous RL algorithms on six of the seven games we have attempted and surpassed an expert human player on three of them. Figure 1 provides sample screenshots from ﬁve of the games used for training.


yer on three of them. Figure 1 provides sample screenshots from ﬁve of the games used for training. 2 Background We consider tasks in which an agent interacts with an environment E, in this case the Atari emulator, in a sequence of actions, observations and rewards. At each time-step the agent selects an action at from the set of legal game actions, A = {1, . . . , K}. The action is passed to the emulator and modiﬁes its internal state and the game score. In general E may be stochastic. The emulator’s internal state is not observed by the agent; instead it observes an image xt ∈Rd from the emulator, which is a vector of raw pixel values representing the current screen. In addition it receives a reward rt representing the change in game score.


ting the current screen. In addition it receives a reward rt representing the change in game score. Note that in general the game score may depend on the whole prior sequence of actions and observations; feedback about an action may only be received after many thousands of time-steps have elapsed. Since the agent only observes images of the current screen, the task is partially observed and many emulator states are perceptually aliased, i.e. it is impossible to fully understand the current situation from only the current screen xt. We therefore consider sequences of actions and observations, st = x1, a1, x2, ..., at−1, xt, and learn game strategies that depend upon these sequences. All sequences in the emulator are assumed to terminate in a ﬁnite number of time-steps.


sequences. All sequences in the emulator are assumed to terminate in a ﬁnite number of time-steps. This formalism gives rise to a large but ﬁnite Markov decision process (MDP) in which each sequence is a distinct state. As a result, we can apply standard reinforcement learning methods for MDPs, simply by using the complete sequence st as the state representation at time t. The goal of the agent is to interact with the emulator by selecting actions in a way that maximises future rewards. We make the standard assumption that future rewards are discounted by a factor of γ per time-step, and deﬁne the future discounted return at time t as Rt = PT t′=t γt′−trt′, where T is the time-step at which the game terminates.


d return at time t as Rt = PT t′=t γt′−trt′, where T is the time-step at which the game terminates. We deﬁne the optimal action-value function Q∗(s, a) as the maximum expected return achievable by following any strategy, after seeing some sequence s and then taking some action a, Q∗(s, a) = maxπ E [Rt|st = s, at = a, π], where π is a policy mapping sequences to actions (or distributions over actions). The optimal action-value function obeys an important identity known as the Bellman equation. This is based on the following intuition: if the optimal value Q∗(s′, a′) of the sequence s′ at the next time-step was known for all possible actions a′, then the optimal strategy is to select the action a′ 2


maximising the expected value of r + γQ∗(s′, a′), Q∗(s, a) = Es′∼E h r + γ max a′ Q∗(s′, a′) s, a i (1) The basic idea behind many reinforcement learning algorithms is to estimate the actionvalue function, by using the Bellman equation as an iterative update, Qi+1(s, a) = E [r + γ maxa′ Qi(s′, a′)|s, a]. Such value iteration algorithms converge to the optimal actionvalue function, Qi →Q∗as i →∞[23]. In practice, this basic approach is totally impractical, because the action-value function is estimated separately for each sequence, without any generalisation. Instead, it is common to use a function approximator to estimate the action-value function, Q(s, a; θ) ≈Q∗(s, a). In the reinforcement learning community this is typically a linear function approximator, but sometimes a non-linear func


learning community this is typically a linear function approximator, but sometimes a non-linear function approximator is used instead, such as a neural network. We refer to a neural network function approximator with weights θ as a Q-network. A Q-network can be trained by minimising a sequence of loss functions Li(θi) that changes at each iteration i, Li (θi) = Es,a∼ρ(·) h (yi −Q (s, a; θi))2i , (2) where yi = Es′∼E [r + γ maxa′ Q(s′, a′; θi−1)|s, a] is the target for iteration i and ρ(s, a) is a probability distribution over sequences s and actions a that we refer to as the behaviour distribution. The parameters from the previous iteration θi−1 are held ﬁxed when optimising the loss function Li (θi).


arameters from the previous iteration θi−1 are held ﬁxed when optimising the loss function Li (θi). Note that the targets depend on the network weights; this is in contrast with the targets used for supervised learning, which are ﬁxed before learning begins. Differentiating the loss function with respect to the weights we arrive at the following gradient, ∇θiLi (θi) = Es,a∼ρ(·);s′∼E h r + γ max a′ Q(s′, a′; θi−1) −Q(s, a; θi)  ∇θiQ(s, a; θi) i . (3) Rather than computing the full expectations in the above gradient, it is often computationally expedient to optimise the loss function by stochastic gradient descent. If the weights are updated after every time-step, and the expectations are replaced by single samples from the behaviour distribution ρ and the emulator E respectively, then we


placed by single samples from the behaviour distribution ρ and the emulator E respectively, then we arrive at the familiar Q-learning algorithm [26]. Note that this algorithm is model-free: it solves the reinforcement learning task directly using samples from the emulator E, without explicitly constructing an estimate of E. It is also off-policy: it learns about the greedy strategy a = maxa Q(s, a; θ), while following a behaviour distribution that ensures adequate exploration of the state space. In practice, the behaviour distribution is often selected by an ϵ-greedy strategy that follows the greedy strategy with probability 1 −ϵ and selects a random action with probability ϵ. 3 Related Work Perhaps the best-known success story of reinforcement learning is TD-gammon, a backgammonplaying pr


Perhaps the best-known success story of reinforcement learning is TD-gammon, a backgammonplaying program which learnt entirely by reinforcement learning and self-play, and achieved a superhuman level of play [24]. TD-gammon used a model-free reinforcement learning algorithm similar to Q-learning, and approximated the value function using a multi-layer perceptron with one hidden layer1. However, early attempts to follow up on TD-gammon, including applications of the same method to chess, Go and checkers were less successful. This led to a widespread belief that the TD-gammon approach was a special case that only worked in backgammon, perhaps because the stochasticity in the dice rolls helps explore the state space and also makes the value function particularly smooth [19].


ice rolls helps explore the state space and also makes the value function particularly smooth [19]. Furthermore, it was shown that combining model-free reinforcement learning algorithms such as Qlearning with non-linear function approximators [25], or indeed with off-policy learning [1] could cause the Q-network to diverge. Subsequently, the majority of work in reinforcement learning focused on linear function approximators with better convergence guarantees [25]. 1In fact TD-Gammon approximated the state value function V (s) rather than the action-value function Q(s, a), and learnt on-policy directly from the self-play games 3


More recently, there has been a revival of interest in combining deep learning with reinforcement learning. Deep neural networks have been used to estimate the environment E; restricted Boltzmann machines have been used to estimate the value function [21]; or the policy [9]. In addition, the divergence issues with Q-learning have been partially addressed by gradient temporal-difference methods. These methods are proven to converge when evaluating a ﬁxed policy with a nonlinear function approximator [14]; or when learning a control policy with linear function approximation using a restricted variant of Q-learning [15]. However, these methods have not yet been extended to nonlinear control. Perhaps the most similar prior work to our own approach is neural ﬁtted Q-learning (NFQ) [20].


rol. Perhaps the most similar prior work to our own approach is neural ﬁtted Q-learning (NFQ) [20]. NFQ optimises the sequence of loss functions in Equation 2, using the RPROP algorithm to update the parameters of the Q-network. However, it uses a batch update that has a computational cost per iteration that is proportional to the size of the data set, whereas we consider stochastic gradient updates that have a low constant cost per iteration and scale to large data-sets. NFQ has also been successfully applied to simple real-world control tasks using purely visual input, by ﬁrst using deep autoencoders to learn a low dimensional representation of the task, and then applying NFQ to this representation [12].


rn a low dimensional representation of the task, and then applying NFQ to this representation [12]. In contrast our approach applies reinforcement learning end-to-end, directly from the visual inputs; as a result it may learn features that are directly relevant to discriminating action-values. Q-learning has also previously been combined with experience replay and a simple neural network [13], but again starting with a low-dimensional state rather than raw visual inputs. The use of the Atari 2600 emulator as a reinforcement learning platform was introduced by [3], who applied standard reinforcement learning algorithms with linear function approximation and generic visual features. Subsequently, results were improved by using a larger number of features, and using tug-of-war hashing to rand


ly, results were improved by using a larger number of features, and using tug-of-war hashing to randomly project the features into a lower-dimensional space [2]. The HyperNEAT evolutionary architecture [8] has also been applied to the Atari platform, where it was used to evolve (separately, for each distinct game) a neural network representing a strategy for that game. When trained repeatedly against deterministic sequences using the emulator’s reset facility, these strategies were able to exploit design ﬂaws in several Atari games. 4 Deep Reinforcement Learning Recent breakthroughs in computer vision and speech recognition have relied on efﬁciently training deep neural networks on very large training sets.


ch recognition have relied on efﬁciently training deep neural networks on very large training sets. The most successful approaches are trained directly from the raw inputs, using lightweight updates based on stochastic gradient descent. By feeding sufﬁcient data into deep neural networks, it is often possible to learn better representations than handcrafted features [11]. These successes motivate our approach to reinforcement learning. Our goal is to connect a reinforcement learning algorithm to a deep neural network which operates directly on RGB images and efﬁciently process training data by using stochastic gradient updates. Tesauro’s TD-Gammon architecture provides a starting point for such an approach.


gradient updates. Tesauro’s TD-Gammon architecture provides a starting point for such an approach. This architecture updates the parameters of a network that estimates the value function, directly from on-policy samples of experience, st, at, rt, st+1, at+1, drawn from the algorithm’s interactions with the environment (or by self-play, in the case of backgammon). Since this approach was able to outperform the best human backgammon players 20 years ago, it is natural to wonder whether two decades of hardware improvements, coupled with modern deep neural network architectures and scalable RL algorithms might produce signiﬁcant progress. In contrast to TD-Gammon and similar online approaches, we utilize a technique known as experience replay [13] where we store the agent’s experiences at eac


we utilize a technique known as experience replay [13] where we store the agent’s experiences at each time-step, et = (st, at, rt, st+1) in a data-set D = e1, ..., eN, pooled over many episodes into a replay memory. During the inner loop of the algorithm, we apply Q-learning updates, or minibatch updates, to samples of experience, e ∼D, drawn at random from the pool of stored samples. After performing experience replay, the agent selects and executes an action according to an ϵ-greedy policy. Since using histories of arbitrary length as inputs to a neural network can be difﬁcult, our Q-function instead works on ﬁxed length representation of histories produced by a function φ. The full algorithm, which we call deep Q-learning, is presented in Algorithm 1.


ed by a function φ. The full algorithm, which we call deep Q-learning, is presented in Algorithm 1. This approach has several advantages over standard online Q-learning [23]. First, each step of experience is potentially used in many weight updates, which allows for greater data efﬁciency. 4


Algorithm 1 Deep Q-learning with Experience Replay Initialize replay memory D to capacity N Initialize action-value function Q with random weights for episode = 1, M do Initialise sequence s1 = {x1} and preprocessed sequenced φ1 = φ(s1) for t = 1, T do With probability ϵ select a random action at otherwise select at = maxa Q∗(φ(st), a; θ) Execute action at in emulator and observe reward rt and image xt+1 Set st+1 = st, at, xt+1 and preprocess φt+1 = φ(st+1) Store transition (φt, at, rt, φt+1) in D Sample random minibatch of transitions (φj, aj, rj, φj+1) from D Set yj =  rj for terminal φj+1 rj + γ maxa′ Q(φj+1, a′; θ) for non-terminal φj+1 Perform a gradient descent step on (yj −Q(φj, aj; θ))2 according to equation 3 end for end for Second, learning directly from consecutive samples is i


θ))2 according to equation 3 end for end for Second, learning directly from consecutive samples is inefﬁcient, due to the strong correlations between the samples; randomizing the samples breaks these correlations and therefore reduces the variance of the updates. Third, when learning on-policy the current parameters determine the next data sample that the parameters are trained on. For example, if the maximizing action is to move left then the training samples will be dominated by samples from the left-hand side; if the maximizing action then switches to the right then the training distribution will also switch. It is easy to see how unwanted feedback loops may arise and the parameters could get stuck in a poor local minimum, or even diverge catastrophically [25].


and the parameters could get stuck in a poor local minimum, or even diverge catastrophically [25]. By using experience replay the behavior distribution is averaged over many of its previous states, smoothing out learning and avoiding oscillations or divergence in the parameters. Note that when learning by experience replay, it is necessary to learn off-policy (because our current parameters are different to those used to generate the sample), which motivates the choice of Q-learning. In practice, our algorithm only stores the last N experience tuples in the replay memory, and samples uniformly at random from D when performing updates. This approach is in some respects limited since the memory buffer does not differentiate important transitions and always overwrites with recent transitions


ry buffer does not differentiate important transitions and always overwrites with recent transitions due to the ﬁnite memory size N. Similarly, the uniform sampling gives equal importance to all transitions in the replay memory. A more sophisticated sampling strategy might emphasize transitions from which we can learn the most, similar to prioritized sweeping [17]. 4.1 Preprocessing and Model Architecture Working directly with raw Atari frames, which are 210 × 160 pixel images with a 128 color palette, can be computationally demanding, so we apply a basic preprocessing step aimed at reducing the input dimensionality. The raw frames are preprocessed by ﬁrst converting their RGB representation to gray-scale and down-sampling it to a 110×84 image.


d by ﬁrst converting their RGB representation to gray-scale and down-sampling it to a 110×84 image. The ﬁnal input representation is obtained by cropping an 84 × 84 region of the image that roughly captures the playing area. The ﬁnal cropping stage is only required because we use the GPU implementation of 2D convolutions from [11], which expects square inputs. For the experiments in this paper, the function φ from algorithm 1 applies this preprocessing to the last 4 frames of a history and stacks them to produce the input to the Q-function. There are several possible ways of parameterizing Q using a neural network. Since Q maps historyaction pairs to scalar estimates of their Q-value, the history and the action have been used as inputs to the neural network by some previous approaches [20,


story and the action have been used as inputs to the neural network by some previous approaches [20, 12]. The main drawback of this type of architecture is that a separate forward pass is required to compute the Q-value of each action, resulting in a cost that scales linearly with the number of actions. We instead use an architecture in which there is a separate output unit for each possible action, and only the state representation is an input to the neural network. The outputs correspond to the predicted Q-values of the individual action for the input state. The main advantage of this type of architecture is the ability to compute Q-values for all possible actions in a given state with only a single forward pass through the network. 5


We now describe the exact architecture used for all seven Atari games. The input to the neural network consists is an 84 × 84 × 4 image produced by φ. The ﬁrst hidden layer convolves 16 8 × 8 ﬁlters with stride 4 with the input image and applies a rectiﬁer nonlinearity [10, 18]. The second hidden layer convolves 32 4 × 4 ﬁlters with stride 2, again followed by a rectiﬁer nonlinearity. The ﬁnal hidden layer is fully-connected and consists of 256 rectiﬁer units. The output layer is a fullyconnected linear layer with a single output for each valid action. The number of valid actions varied between 4 and 18 on the games we considered. We refer to convolutional networks trained with our approach as Deep Q-Networks (DQN).


considered. We refer to convolutional networks trained with our approach as Deep Q-Networks (DQN). 5 Experiments So far, we have performed experiments on seven popular ATARI games – Beam Rider, Breakout, Enduro, Pong, Q*bert, Seaquest, Space Invaders. We use the same network architecture, learning algorithm and hyperparameters settings across all seven games, showing that our approach is robust enough to work on a variety of games without incorporating game-speciﬁc information. While we evaluated our agents on the real and unmodiﬁed games, we made one change to the reward structure of the games during training only. Since the scale of scores varies greatly from game to game, we ﬁxed all positive rewards to be 1 and all negative rewards to be −1, leaving 0 rewards unchanged.


e ﬁxed all positive rewards to be 1 and all negative rewards to be −1, leaving 0 rewards unchanged. Clipping the rewards in this manner limits the scale of the error derivatives and makes it easier to use the same learning rate across multiple games. At the same time, it could affect the performance of our agent since it cannot differentiate between rewards of different magnitude. In these experiments, we used the RMSProp algorithm with minibatches of size 32. The behavior policy during training was ϵ-greedy with ϵ annealed linearly from 1 to 0.1 over the ﬁrst million frames, and ﬁxed at 0.1 thereafter. We trained for a total of 10 million frames and used a replay memory of one million most recent frames.


rained for a total of 10 million frames and used a replay memory of one million most recent frames. Following previous approaches to playing Atari games, we also use a simple frame-skipping technique [3]. More precisely, the agent sees and selects actions on every kth frame instead of every frame, and its last action is repeated on skipped frames. Since running the emulator forward for one step requires much less computation than having the agent select an action, this technique allows the agent to play roughly k times more games without signiﬁcantly increasing the runtime. We use k = 4 for all games except Space Invaders where we noticed that using k = 4 makes the lasers invisible because of the period at which they blink.


e we noticed that using k = 4 makes the lasers invisible because of the period at which they blink. We used k = 3 to make the lasers visible and this change was the only difference in hyperparameter values between any of the games. 5.1 Training and Stability In supervised learning, one can easily track the performance of a model during training by evaluating it on the training and validation sets. In reinforcement learning, however, accurately evaluating the progress of an agent during training can be challenging. Since our evaluation metric, as suggested by [3], is the total reward the agent collects in an episode or game averaged over a number of games, we periodically compute it during training.


in an episode or game averaged over a number of games, we periodically compute it during training. The average total reward metric tends to be very noisy because small changes to the weights of a policy can lead to large changes in the distribution of states the policy visits . The leftmost two plots in ﬁgure 2 show how the average total reward evolves during training on the games Seaquest and Breakout. Both averaged reward plots are indeed quite noisy, giving one the impression that the learning algorithm is not making steady progress. Another, more stable, metric is the policy’s estimated action-value function Q, which provides an estimate of how much discounted reward the agent can obtain by following its policy from any given state.


te of how much discounted reward the agent can obtain by following its policy from any given state. We collect a ﬁxed set of states by running a random policy before training starts and track the average of the maximum2 predicted Q for these states. The two rightmost plots in ﬁgure 2 show that average predicted Q increases much more smoothly than the average total reward obtained by the agent and plotting the same metrics on the other ﬁve games produces similarly smooth curves. In addition to seeing relatively smooth improvement to predicted Q during training we did not experience any divergence issues in any of our experiments. This suggests that, despite lacking any theoretical convergence guarantees, our method is able to train large neural networks using a reinforcement learning signal


guarantees, our method is able to train large neural networks using a reinforcement learning signal and stochastic gradient descent in a stable manner. 2The maximum for each state is taken over the possible actions. 6


0 50 100 150 200 250 0 10 20 30 40 50 60 70 80 90 100 Average Reward per Episode Training Epochs Average Reward on Breakout 0 200 400 600 800 1000 1200 1400 1600 1800 0 10 20 30 40 50 60 70 80 90 100 Average Reward per Episode Training Epochs Average Reward on Seaquest 0 0.5 1 1.5 2 2.5 3 3.5 4 0 10 20 30 40 50 60 70 80 90 100 Average Action Value (Q) Training Epochs Average Q on Breakout 0 1 2 3 4 5 6 7 8 9 0 10 20 30 40 50 60 70 80 90 100 Average Action Value (Q) Training Epochs Average Q on Seaquest Figure 2: The two plots on the left show average reward per episode on Breakout and Seaquest respectively during training. The statistics were computed by running an ϵ-greedy policy with ϵ = 0.05 for 10000 steps.


training. The statistics were computed by running an ϵ-greedy policy with ϵ = 0.05 for 10000 steps. The two plots on the right show the average maximum predicted action-value of a held out set of states on Breakout and Seaquest respectively. One epoch corresponds to 50000 minibatch weight updates or roughly 30 minutes of training time. Figure 3: The leftmost plot shows the predicted value function for a 30 frame segment of the game Seaquest. The three screenshots correspond to the frames labeled by A, B, and C respectively. 5.2 Visualizing the Value Function Figure 3 shows a visualization of the learned value function on the game Seaquest. The ﬁgure shows that the predicted value jumps after an enemy appears on the left of the screen (point A).


re shows that the predicted value jumps after an enemy appears on the left of the screen (point A). The agent then ﬁres a torpedo at the enemy and the predicted value peaks as the torpedo is about to hit the enemy (point B). Finally, the value falls to roughly its original value after the enemy disappears (point C). Figure 3 demonstrates that our method is able to learn how the value function evolves for a reasonably complex sequence of events. 5.3 Main Evaluation We compare our results with the best performing methods from the RL literature [3, 4]. The method labeled Sarsa used the Sarsa algorithm to learn linear policies on several different feature sets handengineered for the Atari task and we report the score for the best performing feature set [3].


handengineered for the Atari task and we report the score for the best performing feature set [3]. Contingency used the same basic approach as Sarsa but augmented the feature sets with a learned representation of the parts of the screen that are under the agent’s control [4]. Note that both of these methods incorporate signiﬁcant prior knowledge about the visual problem by using background subtraction and treating each of the 128 colors as a separate channel. Since many of the Atari games use one distinct color for each type of object, treating each color as a separate channel can be similar to producing a separate binary map encoding the presence of each object type. In contrast, our agents only receive the raw RGB screenshots as input and must learn to detect objects on their own.


agents only receive the raw RGB screenshots as input and must learn to detect objects on their own. In addition to the learned agents, we also report scores for an expert human game player and a policy that selects actions uniformly at random. The human performance is the median reward achieved after around two hours of playing each game. Note that our reported human scores are much higher than the ones in Bellemare et al. [3]. For the learned methods, we follow the evaluation strategy used in Bellemare et al. [3, 5] and report the average score obtained by running an ϵ-greedy policy with ϵ = 0.05 for a ﬁxed number of steps. The ﬁrst ﬁve rows of table 1 show the per-game average scores on all games.


a ﬁxed number of steps. The ﬁrst ﬁve rows of table 1 show the per-game average scores on all games. Our approach (labeled DQN) outperforms the other learning methods by a substantial margin on all seven games despite incorporating almost no prior knowledge about the inputs. We also include a comparison to the evolutionary policy search approach from [8] in the last three rows of table 1. We report two sets of results for this method. The HNeat Best score reﬂects the results obtained by using a hand-engineered object detector algorithm that outputs the locations and 7


B. Rider Breakout Enduro Pong Q*bert Seaquest S. Invaders Random 354 1.2 0 −20.4 157 110 179 Sarsa [3] 996 5.2 129 −19 614 665 271 Contingency [4] 1743 6 159 −17 960 723 268 DQN 4092 168 470 20 1952 1705 581 Human 7456 31 368 −3 18900 28010 3690 HNeat Best [8] 3616 52 106 19 1800 920 1720 HNeat Pixel [8] 1332 4 91 −16 1325 800 1145 DQN Best 5184 225 661 21 4500 1740 1075 Table 1: The upper table compares average total reward for various learning methods by running an ϵ-greedy policy with ϵ = 0.05 for a ﬁxed number of steps. The lower table reports results of the single best performing episode for HNeat and DQN. HNeat produces deterministic policies that always get the same score while DQN used an ϵ-greedy policy with ϵ = 0.05. types of objects on the Atari screen.


e same score while DQN used an ϵ-greedy policy with ϵ = 0.05. types of objects on the Atari screen. The HNeat Pixel score is obtained by using the special 8 color channel representation of the Atari emulator that represents an object label map at each channel. This method relies heavily on ﬁnding a deterministic sequence of states that represents a successful exploit. It is unlikely that strategies learnt in this way will generalize to random perturbations; therefore the algorithm was only evaluated on the highest scoring single episode. In contrast, our algorithm is evaluated on ϵ-greedy control sequences, and must therefore generalize across a wide variety of possible situations. Nevertheless, we show that on all the games, except Space Invaders, not only our max evaluation results (row


ess, we show that on all the games, except Space Invaders, not only our max evaluation results (row 8), but also our average results (row 4) achieve better performance. Finally, we show that our method achieves better performance than an expert human player on Breakout, Enduro and Pong and it achieves close to human performance on Beam Rider. The games Q*bert, Seaquest, Space Invaders, on which we are far from human performance, are more challenging because they require the network to ﬁnd a strategy that extends over long time scales. 6 Conclusion This paper introduced a new deep learning model for reinforcement learning, and demonstrated its ability to master difﬁcult control policies for Atari 2600 computer games, using only raw pixels as input.


to master difﬁcult control policies for Atari 2600 computer games, using only raw pixels as input. We also presented a variant of online Q-learning that combines stochastic minibatch updates with experience replay memory to ease the training of deep networks for RL. Our approach gave state-of-the-art results in six of the seven games it was tested on, with no adjustment of the architecture or hyperparameters. References [1] Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Machine Learning (ICML 1995), pages 30–37. Morgan Kaufmann, 1995. [2] Marc Bellemare, Joel Veness, and Michael Bowling. Sketch-based linear value function approximation.


Marc Bellemare, Joel Veness, and Michael Bowling. Sketch-based linear value function approximation. In Advances in Neural Information Processing Systems 25, pages 2222–2230, 2012. [3] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 47:253–279, 2013. [4] Marc G Bellemare, Joel Veness, and Michael Bowling. Investigating contingency awareness using atari 2600 games. In AAAI, 2012. [5] Marc G. Bellemare, Joel Veness, and Michael Bowling. Bayesian learning of recursively factored environments. In Proceedings of the Thirtieth International Conference on Machine Learning (ICML 2013), pages 1211–1219, 2013. 8


[6] George E. Dahl, Dong Yu, Li Deng, and Alex Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 20(1):30 –42, January 2012. [7] Alex Graves, Abdel-rahman Mohamed, and Geoffrey E. Hinton. Speech recognition with deep recurrent neural networks. In Proc. ICASSP, 2013. [8] Matthew Hausknecht, Risto Miikkulainen, and Peter Stone. A neuro-evolution approach to general atari game playing. 2013. [9] Nicolas Heess, David Silver, and Yee Whye Teh. Actor-critic reinforcement learning with energy-based policies. In European Workshop on Reinforcement Learning, page 43, 2012. [10] Kevin Jarrett, Koray Kavukcuoglu, MarcAurelio Ranzato, and Yann LeCun.


earning, page 43, 2012. [10] Kevin Jarrett, Koray Kavukcuoglu, MarcAurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision and Pattern Recognition (CVPR 2009), pages 2146–2153. IEEE, 2009. [11] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classiﬁcation with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pages 1106–1114, 2012. [12] Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In Neural Networks (IJCNN), The 2010 International Joint Conference on, pages 1–8. IEEE, 2010. [13] Long-Ji Lin. Reinforcement learning for robots using neural networks. Technical report, DTIC Document, 1993.


in. Reinforcement learning for robots using neural networks. Technical report, DTIC Document, 1993. [14] Hamid Maei, Csaba Szepesvari, Shalabh Bhatnagar, Doina Precup, David Silver, and Rich Sutton. Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation. In Advances in Neural Information Processing Systems 22, pages 1204–1212, 2009. [15] Hamid Maei, Csaba Szepesv´ari, Shalabh Bhatnagar, and Richard S. Sutton. Toward off-policy learning control with function approximation. In Proceedings of the 27th International Conference on Machine Learning (ICML 2010), pages 719–726, 2010. [16] Volodymyr Mnih. Machine Learning for Aerial Image Labeling. PhD thesis, University of Toronto, 2013. [17] Andrew Moore and Chris Atkeson.


erial Image Labeling. PhD thesis, University of Toronto, 2013. [17] Andrew Moore and Chris Atkeson. Prioritized sweeping: Reinforcement learning with less data and less real time. Machine Learning, 13:103–130, 1993. [18] Vinod Nair and Geoffrey E Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML 2010), pages 807–814, 2010. [19] Jordan B. Pollack and Alan D. Blair. Why did td-gammon work. In Advances in Neural Information Processing Systems 9, pages 10–16, 1996. [20] Martin Riedmiller. Neural ﬁtted q iteration–ﬁrst experiences with a data efﬁcient neural reinforcement learning method. In Machine Learning: ECML 2005, pages 317–328. Springer, 2005. [21] Brian Sallans and Geoffrey E. Hinton.


hine Learning: ECML 2005, pages 317–328. Springer, 2005. [21] Brian Sallans and Geoffrey E. Hinton. Reinforcement learning with factored states and actions. Journal of Machine Learning Research, 5:1063–1088, 2004. [22] Pierre Sermanet, Koray Kavukcuoglu, Soumith Chintala, and Yann LeCun. Pedestrian detection with unsupervised multi-stage feature learning. In Proc. International Conference on Computer Vision and Pattern Recognition (CVPR 2013). IEEE, 2013. [23] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 1998. [24] Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58–68, 1995. [25] John N Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation.


klis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. Automatic Control, IEEE Transactions on, 42(5):674–690, 1997. [26] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992. 9


Trust Region Policy Optimization John Schulman JOSCHU@EECS.BERKELEY.EDU Sergey Levine SLEVINE@EECS.BERKELEY.EDU Philipp Moritz PCMORITZ@EECS.BERKELEY.EDU Michael Jordan JORDAN@CS.BERKELEY.EDU Pieter Abbeel PABBEEL@CS.BERKELEY.EDU University of California, Berkeley, Department of Electrical Engineering and Computer Sciences Abstract We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justiﬁed procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks.


gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters. 1 Introduction Most algorithms for policy optimization can be classiﬁed into three broad categories: (1) policy iteration methods, which alternate between estimating the value function under the current policy and improving the policy (Bertsekas, 2005); (2) policy gradient methods, which use an estimator of the gradient of the expected return (total rewar


policy gradient methods, which use an estimator of the gradient of the expected return (total reward) obtained from sample trajectories (Peters & Schaal, 2008a) (and which, as we later discuss, have a close connection to policy iteration); and (3) derivative-free optimization methods, such as the cross-entropy method (CEM) and covariance matrix adaptation (CMA), which treat the return as a black box function to be optimized in terms of the policy parameters (Szita & L¨orincz, 2006). General derivative-free stochastic optimization methods such as CEM and CMA are preferred on many problems, because they achieve good results while being simple to understand and implement. For example, while Proceedings of the 31 st International Conference on Machine Learning, Lille, France, 2015.


, while Proceedings of the 31 st International Conference on Machine Learning, Lille, France, 2015. JMLR: W&CP volume 37. Copyright 2015 by the author(s). Tetris is a classic benchmark problem for approximate dynamic programming (ADP) methods, stochastic optimization methods are difﬁcult to beat on this task (Gabillon et al., 2013). For continuous control problems, methods like CMA have been successful at learning control policies for challenging tasks like locomotion when provided with hand-engineered policy classes with low-dimensional parameterizations (Wampler & Popovi´c, 2009). The inability of ADP and gradient-based methods to consistently beat gradient-free random search is unsatisfying, since gradient-based optimization algorithms enjoy much better sample complexity guarantees than


ng, since gradient-based optimization algorithms enjoy much better sample complexity guarantees than gradient-free methods (Nemirovski, 2005). Continuous gradient-based optimization has been very successful at learning function approximators for supervised learning tasks with huge numbers of parameters, and extending their success to reinforcement learning would allow for efﬁcient training of complex and powerful policies. In this article, we ﬁrst prove that minimizing a certain surrogate objective function guarantees policy improvement with non-trivial step sizes. Then we make a series of approximations to the theoretically-justiﬁed algorithm, yielding a practical algorithm, which we call trust region policy optimization (TRPO).


d algorithm, yielding a practical algorithm, which we call trust region policy optimization (TRPO). We describe two variants of this algorithm: ﬁrst, the single-path method, which can be applied in the model-free setting; second, the vine method, which requires the system to be restored to particular states, which is typically only possible in simulation. These algorithms are scalable and can optimize nonlinear policies with tens of thousands of parameters, which have previously posed a major challenge for model-free policy search (Deisenroth et al., 2013). In our experiments, we show that the same TRPO methods can learn complex policies for swimming, hopping, and walking, as well as playing Atari games directly from raw images.


licies for swimming, hopping, and walking, as well as playing Atari games directly from raw images. 2 Preliminaries Consider an inﬁnite-horizon discounted Markov decision process (MDP), deﬁned by the tuple (S, A, P, r, ρ0, γ), where S is a ﬁnite set of states, A is a ﬁnite set of actions, P : S × A × S →R is the transition probability distriarXiv:1502.05477v5 [cs.LG] 20 Apr 2017


Trust Region Policy Optimization bution, r : S →R is the reward function, ρ0 : S →R is the distribution of the initial state s0, and γ ∈(0, 1) is the discount factor. Let π denote a stochastic policy π : S × A →[0, 1], and let η(π) denote its expected discounted reward: η(π) = Es0,a0,... " ∞ X t=0 γtr(st) # , where s0 ∼ρ0(s0), at ∼π(at|st), st+1 ∼P(st+1|st, at). We will use the following standard deﬁnitions of the stateaction value function Qπ, the value function Vπ, and the advantage function Aπ: Qπ(st, at) = Est+1,at+1,... " ∞ X l=0 γlr(st+l) # , Vπ(st) = Eat,st+1,... " ∞ X l=0 γlr(st+l) # , Aπ(s, a) = Qπ(s, a) −Vπ(s), where at ∼π(at|st), st+1 ∼P(st+1|st, at) for t ≥0. The following useful identity expresses the expected return of another policy ˜π in terms of the advantage over π, accum


identity expresses the expected return of another policy ˜π in terms of the advantage over π, accumulated over timesteps (see Kakade & Langford (2002) or Appendix A for proof): η(˜π) = η(π) + Es0,a0,···∼˜π " ∞ X t=0 γtAπ(st, at) # (1) where the notation Es0,a0,···∼˜π [. . . ] indicates that actions are sampled at ∼˜π(·|st). Let ρπ be the (unnormalized) discounted visitation frequencies ρπ(s)=P(s0 = s)+γP(s1 = s)+γ2P(s2 = s)+. . . , where s0 ∼ρ0 and the actions are chosen according to π. We can rewrite Equation (1) with a sum over states instead of timesteps: η(˜π) = η(π) + ∞ X t=0 X s P(st = s|˜π) X a ˜π(a|s)γtAπ(s, a) = η(π) + X s ∞ X t=0 γtP(st = s|˜π) X a ˜π(a|s)Aπ(s, a) = η(π) + X s ρ˜π(s) X a ˜π(a|s)Aπ(s, a).


a) = η(π) + X s ∞ X t=0 γtP(st = s|˜π) X a ˜π(a|s)Aπ(s, a) = η(π) + X s ρ˜π(s) X a ˜π(a|s)Aπ(s, a). (2) This equation implies that any policy update π →˜π that has a nonnegative expected advantage at every state s, i.e., P a ˜π(a|s)Aπ(s, a) ≥0, is guaranteed to increase the policy performance η, or leave it constant in the case that the expected advantage is zero everywhere. This implies the classic result that the update performed by exact policy iteration, which uses the deterministic policy ˜π(s) = arg maxa Aπ(s, a), improves the policy if there is at least one state-action pair with a positive advantage value and nonzero state visitation probability, otherwise the algorithm has converged to the optimal policy.


nonzero state visitation probability, otherwise the algorithm has converged to the optimal policy. However, in the approximate setting, it will typically be unavoidable, due to estimation and approximation error, that there will be some states s for which the expected advantage is negative, that is, P a ˜π(a|s)Aπ(s, a) < 0. The complex dependency of ρ˜π(s) on ˜π makes Equation (2) difﬁcult to optimize directly. Instead, we introduce the following local approximation to η: Lπ(˜π) = η(π) + X s ρπ(s) X a ˜π(a|s)Aπ(s, a). (3) Note that Lπ uses the visitation frequency ρπ rather than ρ˜π, ignoring changes in state visitation density due to changes in the policy. However, if we have a parameterized policy πθ, where πθ(a|s) is a differentiable function of the parameter vector θ, then Lπ matches


ed policy πθ, where πθ(a|s) is a differentiable function of the parameter vector θ, then Lπ matches η to ﬁrst order (see Kakade & Langford (2002)). That is, for any parameter value θ0, Lπθ0(πθ0) = η(πθ0), ∇θLπθ0(πθ) θ=θ0 = ∇θη(πθ) θ=θ0. (4) Equation (4) implies that a sufﬁciently small step πθ0 →˜π that improves Lπθold will also improve η, but does not give us any guidance on how big of a step to take. To address this issue, Kakade & Langford (2002) proposed a policy updating scheme called conservative policy iteration, for which they could provide explicit lower bounds on the improvement of η. To deﬁne the conservative policy iteration update, let πold denote the current policy, and let π′ = arg maxπ′ Lπold(π′).


tive policy iteration update, let πold denote the current policy, and let π′ = arg maxπ′ Lπold(π′). The new policy πnew was deﬁned to be the following mixture: πnew(a|s) = (1 −α)πold(a|s) + απ′(a|s). (5) Kakade and Langford derived the following lower bound: η(πnew)≥Lπold(πnew) − 2ϵγ (1 −γ)2 α2 where ϵ = max s Ea∼π′(a|s) [Aπ(s, a)] . (6) (We have modiﬁed it to make it slightly weaker but simpler.) Note, however, that so far this bound only applies to mixture policies generated by Equation (5). This policy class is unwieldy and restrictive in practice, and it is desirable for a practical policy update scheme to be applicable to all general stochastic policy classes. 3 Monotonic Improvement Guarantee for General Stochastic Policies Equation (6), which applies to conservative policy iteration


arantee for General Stochastic Policies Equation (6), which applies to conservative policy iteration, implies that a policy update that improves the right-hand


Trust Region Policy Optimization side is guaranteed to improve the true performance η. Our principal theoretical result is that the policy improvement bound in Equation (6) can be extended to general stochastic policies, rather than just mixture polices, by replacing α with a distance measure between π and ˜π, and changing the constant ϵ appropriately. Since mixture policies are rarely used in practice, this result is crucial for extending the improvement guarantee to practical problems. The particular distance measure we use is the total variation divergence, which is deﬁned by DT V (p ∥q) = 1 2 P i|pi −qi| for discrete probability distributions p, q.1 Deﬁne Dmax TV (π, ˜π) as Dmax TV (π, ˜π) = max s DT V (π(·|s) ∥˜π(·|s)). (7) Theorem 1. Let α = Dmax TV (πold, πnew).


˜π) as Dmax TV (π, ˜π) = max s DT V (π(·|s) ∥˜π(·|s)). (7) Theorem 1. Let α = Dmax TV (πold, πnew). Then the following bound holds: η(πnew) ≥Lπold(πnew) − 4ϵγ (1 −γ)2 α2 where ϵ = max s,a |Aπ(s, a)| (8) We provide two proofs in the appendix. The ﬁrst proof extends Kakade and Langford’s result using the fact that the random variables from two distributions with total variation divergence less than α can be coupled, so that they are equal with probability 1 −α. The second proof uses perturbation theory. Next, we note the following relationship between the total variation divergence and the KL divergence (Pollard (2000), Ch. 3): DT V (p ∥q)2 ≤DKL(p ∥q). Let Dmax KL (π, ˜π) = maxs DKL(π(·|s) ∥˜π(·|s)).


(Pollard (2000), Ch. 3): DT V (p ∥q)2 ≤DKL(p ∥q). Let Dmax KL (π, ˜π) = maxs DKL(π(·|s) ∥˜π(·|s)). The following bound then follows directly from Theorem 1: η(˜π) ≥Lπ(˜π) −CDmax KL (π, ˜π), where C = 4ϵγ (1 −γ)2 . (9) Algorithm 1 describes an approximate policy iteration scheme based on the policy improvement bound in Equation (9). Note that for now, we assume exact evaluation of the advantage values Aπ. It follows from Equation (9) that Algorithm 1 is guaranteed to generate a monotonically improving sequence of policies η(π0) ≤η(π1) ≤η(π2) ≤. . . . To see this, let Mi(π) = Lπi(π) −CDmax KL (πi, π). Then η(πi+1) ≥Mi(πi+1) by Equation (9) η(πi) = Mi(πi), therefore, η(πi+1) −η(πi) ≥Mi(πi+1) −M(πi).


Then η(πi+1) ≥Mi(πi+1) by Equation (9) η(πi) = Mi(πi), therefore, η(πi+1) −η(πi) ≥Mi(πi+1) −M(πi). (10) Thus, by maximizing Mi at each iteration, we guarantee that the true objective η is non-decreasing. This algorithm 1Our result is straightforward to extend to continuous states and actions by replacing the sums with integrals. Algorithm 1 Policy iteration algorithm guaranteeing nondecreasing expected return η Initialize π0. for i = 0, 1, 2, . . . until convergence do Compute all advantage values Aπi(s, a). Solve the constrained optimization problem πi+1 = arg max π [Lπi(π) −CDmax KL (πi, π)] where C = 4ϵγ/(1 −γ)2 and Lπi(π)=η(πi)+ X s ρπi(s) X a π(a|s)Aπi(s, a) end for is a type of minorization-maximization (MM) algorithm (Hunter & Lange, 2004), which is a class of methods that also inc


zation-maximization (MM) algorithm (Hunter & Lange, 2004), which is a class of methods that also includes expectation maximization. In the terminology of MM algorithms, Mi is the surrogate function that minorizes η with equality at πi. This algorithm is also reminiscent of proximal gradient methods and mirror descent. Trust region policy optimization, which we propose in the following section, is an approximation to Algorithm 1, which uses a constraint on the KL divergence rather than a penalty to robustly allow large updates. 4 Optimization of Parameterized Policies In the previous section, we considered the policy optimization problem independently of the parameterization of π and under the assumption that the policy can be evaluated at all states.


the parameterization of π and under the assumption that the policy can be evaluated at all states. We now describe how to derive a practical algorithm from these theoretical foundations, under ﬁnite sample counts and arbitrary parameterizations. Since we consider parameterized policies πθ(a|s) with parameter vector θ, we will overload our previous notation to use functions of θ rather than π, e.g. η(θ) := η(πθ), Lθ(˜θ) := Lπθ(π˜θ), and DKL(θ ∥˜θ) := DKL(πθ ∥π˜θ). We will use θold to denote the previous policy parameters that we want to improve upon. The preceding section showed that η(θ) ≥Lθold(θ) − CDmax KL (θold, θ), with equality at θ = θold. Thus, by performing the following maximization, we are guaranteed to improve the true objective η: maximize θ [Lθold(θ) −CDmax KL (θold, θ)] .


ion, we are guaranteed to improve the true objective η: maximize θ [Lθold(θ) −CDmax KL (θold, θ)] . In practice, if we used the penalty coefﬁcient C recommended by the theory above, the step sizes would be very small. One way to take larger steps in a robust way is to use a constraint on the KL divergence between the new policy and the old policy, i.e., a trust region constraint: maximize θ Lθold(θ) (11) subject to Dmax KL (θold, θ) ≤δ.


Trust Region Policy Optimization This problem imposes a constraint that the KL divergence is bounded at every point in the state space. While it is motivated by the theory, this problem is impractical to solve due to the large number of constraints. Instead, we can use a heuristic approximation which considers the average KL divergence: D ρ KL(θ1, θ2) := Es∼ρ [DKL(πθ1(·|s) ∥πθ2(·|s))] . We therefore propose solving the following optimization problem to generate a policy update: maximize θ Lθold(θ) (12) subject to D ρθold KL (θold, θ) ≤δ. Similar policy updates have been proposed in prior work (Bagnell & Schneider, 2003; Peters & Schaal, 2008b; Peters et al., 2010), and we compare our approach to prior methods in Section 7 and in the experiments in Section 8.


10), and we compare our approach to prior methods in Section 7 and in the experiments in Section 8. Our experiments also show that this type of constrained update has similar empirical performance to the maximum KL divergence constraint in Equation (11). 5 Sample-Based Estimation of the Objective and Constraint The previous section proposed a constrained optimization problem on the policy parameters (Equation (12)), which optimizes an estimate of the expected total reward η subject to a constraint on the change in the policy at each update. This section describes how the objective and constraint functions can be approximated using Monte Carlo simulation. We seek to solve the following optimization problem, obtained by expanding Lθold in Equation (12): maximize θ X s ρθold(s) X a πθ(a|s)Aθo


zation problem, obtained by expanding Lθold in Equation (12): maximize θ X s ρθold(s) X a πθ(a|s)Aθold(s, a) subject to D ρθold KL (θold, θ) ≤δ. (13) We ﬁrst replace P s ρθold(s) [. . . ] in the objective by the expectation 1 1−γ Es∼ρθold [. . . ]. Next, we replace the advantage values Aθold by the Q-values Qθold in Equation (13), which only changes the objective by a constant. Last, we replace the sum over the actions by an importance sampling estimator. Using q to denote the sampling distribution, the contribution of a single sn to the loss function is X a πθ(a|sn)Aθold(sn, a) = Ea∼q πθ(a|sn) q(a|sn) Aθold(sn, a)  . Our optimization problem in Equation (13) is exactly equivalent to the following one, written in terms of expectations: maximize θ Es∼ρθold,a∼q πθ(a|s) q(a|s) Qθold(s, a)


ollowing one, written in terms of expectations: maximize θ Es∼ρθold,a∼q πθ(a|s) q(a|s) Qθold(s, a)  (14) subject to Es∼ρθold [DKL(πθold(·|s) ∥πθ(·|s))] ≤δ. all state-action pairs used in objective trajectories s an n ρ0 1 a2 sn rollout set two rollouts using CRN sampling trajectories ρ0 Figure 1. Left: illustration of single path procedure. Here, we generate a set of trajectories via simulation of the policy and incorporate all state-action pairs (sn, an) into the objective. Right: illustration of vine procedure. We generate a set of “trunk” trajectories, and then generate “branch” rollouts from a subset of the reached states. For each of these states sn, we perform multiple actions (a1 and a2 here) and perform a rollout after each action, using common random numbers (CRN) to reduce the


d a2 here) and perform a rollout after each action, using common random numbers (CRN) to reduce the variance. All that remains is to replace the expectations by sample averages and replace the Q value by an empirical estimate. The following sections describe two different schemes for performing this estimation. The ﬁrst sampling scheme, which we call single path, is the one that is typically used for policy gradient estimation (Bartlett & Baxter, 2011), and is based on sampling individual trajectories. The second scheme, which we call vine, involves constructing a rollout set and then performing multiple actions from each state in the rollout set. This method has mostly been explored in the context of policy iteration methods (Lagoudakis & Parr, 2003; Gabillon et al., 2013).


plored in the context of policy iteration methods (Lagoudakis & Parr, 2003; Gabillon et al., 2013). 5.1 Single Path In this estimation procedure, we collect a sequence of states by sampling s0 ∼ρ0 and then simulating the policy πθold for some number of timesteps to generate a trajectory s0, a0, s1, a1, . . . , sT −1, aT −1, sT . Hence, q(a|s) = πθold(a|s). Qθold(s, a) is computed at each state-action pair (st, at) by taking the discounted sum of future rewards along the trajectory. 5.2 Vine In this estimation procedure, we ﬁrst sample s0 ∼ρ0 and simulate the policy πθi to generate a number of trajectories. We then choose a subset of N states along these trajectories, denoted s1, s2, . . . , sN, which we call the “rollout set”.


of N states along these trajectories, denoted s1, s2, . . . , sN, which we call the “rollout set”. For each state sn in the rollout set, we sample K actions according to an,k ∼q(·|sn). Any choice of q(·|sn) with a support that includes the support of πθi(·|sn) will produce a consistent estimator. In practice, we found that q(·|sn) = πθi(·|sn) works well on continuous problems, such as robotic locomotion, while the uniform distribution works well on discrete tasks, such as the Atari games, where it can sometimes achieve better exploration. For each action an,k sampled at each state sn, we esti


Trust Region Policy Optimization mate ˆQθi(sn, an,k) by performing a rollout (i.e., a short trajectory) starting with state sn and action an,k. We can greatly reduce the variance of the Q-value differences between rollouts by using the same random number sequence for the noise in each of the K rollouts, i.e., common random numbers. See (Bertsekas, 2005) for additional discussion on Monte Carlo estimation of Q-values and (Ng & Jordan, 2000) for a discussion of common random numbers in reinforcement learning. In small, ﬁnite action spaces, we can generate a rollout for every possible action from a given state. The contribution to Lθold from a single state sn is as follows: Ln(θ) = K X k=1 πθ(ak|sn) ˆQ(sn, ak), (15) where the action space is A = {a1, a2, . . . , aK}.


ows: Ln(θ) = K X k=1 πθ(ak|sn) ˆQ(sn, ak), (15) where the action space is A = {a1, a2, . . . , aK}. In large or continuous state spaces, we can construct an estimator of the surrogate objective using importance sampling. The self-normalized estimator (Owen (2013), Chapter 9) of Lθold obtained at a single state sn is Ln(θ) = PK k=1 πθ(an,k|sn) πθold(an,k|sn) ˆQ(sn, an,k) PK k=1 πθ(an,k|sn) πθold(an,k|sn) , (16) assuming that we performed K actions an,1, an,2, . . . , an,K from state sn. This self-normalized estimator removes the need to use a baseline for the Q-values (note that the gradient is unchanged by adding a constant to the Q-values). Averaging over sn ∼ρ(π), we obtain an estimator for Lθold, as well as its gradient. The vine and single path methods are illustrated in Figure 1.


r for Lθold, as well as its gradient. The vine and single path methods are illustrated in Figure 1. We use the term vine, since the trajectories used for sampling can be likened to the stems of vines, which branch at various points (the rollout set) into several short offshoots (the rollout trajectories). The beneﬁt of the vine method over the single path method that is our local estimate of the objective has much lower variance given the same number of Q-value samples in the surrogate objective. That is, the vine method gives much better estimates of the advantage values. The downside of the vine method is that we must perform far more calls to the simulator for each of these advantage estimates.


thod is that we must perform far more calls to the simulator for each of these advantage estimates. Furthermore, the vine method requires us to generate multiple trajectories from each state in the rollout set, which limits this algorithm to settings where the system can be reset to an arbitrary state. In contrast, the single path algorithm requires no state resets and can be directly implemented on a physical system (Peters & Schaal, 2008b). 6 Practical Algorithm Here we present two practical policy optimization algorithm based on the ideas above, which use either the single path or vine sampling scheme from the preceding section. The algorithms repeatedly perform the following steps: 1. Use the single path or vine procedures to collect a set of state-action pairs along with Monte Carlo e


e the single path or vine procedures to collect a set of state-action pairs along with Monte Carlo estimates of their Q-values. 2. By averaging over samples, construct the estimated objective and constraint in Equation (14). 3. Approximately solve this constrained optimization problem to update the policy’s parameter vector θ. We use the conjugate gradient algorithm followed by a line search, which is altogether only slightly more expensive than computing the gradient itself. See Appendix C for details. With regard to (3), we construct the Fisher information matrix (FIM) by analytically computing the Hessian of the KL divergence, rather than using the covariance matrix of the gradients. That is, we estimate Aij as 1 N PN n=1 ∂2 ∂θi∂θj DKL(πθold(·|sn) ∥πθ(·|sn)), rather than 1 N PN n=1 ∂ ∂θ


is, we estimate Aij as 1 N PN n=1 ∂2 ∂θi∂θj DKL(πθold(·|sn) ∥πθ(·|sn)), rather than 1 N PN n=1 ∂ ∂θi log πθ(an|sn) ∂ ∂θj log πθ(an|sn). The analytic estimator integrates over the action at each state sn, and does not depend on the action an that was sampled. As described in Appendix C, this analytic estimator has computational beneﬁts in the large-scale setting, since it removes the need to store a dense Hessian or all policy gradients from a batch of trajectories. The rate of improvement in the policy is similar to the empirical FIM, as shown in the experiments. Let us brieﬂy summarize the relationship between the theory from Section 3 and the practical algorithm we have described: • The theory justiﬁes optimizing a surrogate objective with a penalty on KL divergence.


described: • The theory justiﬁes optimizing a surrogate objective with a penalty on KL divergence. However, the large penalty coefﬁcient C leads to prohibitively small steps, so we would like to decrease this coefﬁcient. Empirically, it is hard to robustly choose the penalty coefﬁcient, so we use a hard constraint instead of a penalty, with parameter δ (the bound on KL divergence). • The constraint on Dmax KL (θold, θ) is hard for numerical optimization and estimation, so instead we constrain DKL(θold, θ). • Our theory ignores estimation error for the advantage function. Kakade & Langford (2002) consider this error in their derivation, and the same arguments would hold in the setting of this paper, but we omit them for simplicity.


n, and the same arguments would hold in the setting of this paper, but we omit them for simplicity. 7 Connections with Prior Work As mentioned in Section 4, our derivation results in a policy update that is related to several prior methods, providing a unifying perspective on a number of policy update


Trust Region Policy Optimization schemes. The natural policy gradient (Kakade, 2002) can be obtained as a special case of the update in Equation (12) by using a linear approximation to L and a quadratic approximation to the DKL constraint, resulting in the following problem: maximize θ h ∇θLθold(θ) θ=θold · (θ −θold) i (17) subject to 1 2(θold −θ)T A(θold)(θold −θ) ≤δ, where A(θold)ij = ∂ ∂θi ∂ ∂θj Es∼ρπ [DKL(π(·|s, θold) ∥π(·|s, θ))] θ=θold. The update is θnew = θold + 1 λA(θold)−1∇θL(θ) θ=θold, where the stepsize 1 λ is typically treated as an algorithm parameter. This differs from our approach, which enforces the constraint at each update. Though this difference might seem subtle, our experiments demonstrate that it signiﬁcantly improves the algorithm’s performance on larger problems.


periments demonstrate that it signiﬁcantly improves the algorithm’s performance on larger problems. We can also obtain the standard policy gradient update by using an ℓ2 constraint or penalty: maximize θ h ∇θLθold(θ) θ=θold · (θ −θold) i (18) subject to 1 2∥θ −θold∥2 ≤δ. The policy iteration update can also be obtained by solving the unconstrained problem maximizeπ Lπold(π), using L as deﬁned in Equation (3). Several other methods employ an update similar to Equation (12). Relative entropy policy search (REPS) (Peters et al., 2010) constrains the state-action marginals p(s, a), while TRPO constrains the conditionals p(a|s). Unlike REPS, our approach does not require a costly nonlinear optimization in the inner loop.


a|s). Unlike REPS, our approach does not require a costly nonlinear optimization in the inner loop. Levine and Abbeel (2014) also use a KL divergence constraint, but its purpose is to encourage the policy not to stray from regions where the estimated dynamics model is valid, while we do not attempt to estimate the system dynamics explicitly. Pirotta et al. (2013) also build on and generalize Kakade and Langford’s results, and they derive different algorithms from the ones here. 8 Experiments We designed our experiments to investigate the following questions: 1. What are the performance characteristics of the single path and vine sampling procedures? 2. TRPO is related to prior methods (e.g. natural policy gradient) but makes several changes, most notably by using a ﬁxed KL divergence rathe


natural policy gradient) but makes several changes, most notably by using a ﬁxed KL divergence rather than a ﬁxed penalty coefﬁcient. How does this affect the performance of the algorithm? Figure 2. 2D robot models used for locomotion experiments. From left to right: swimmer, hopper, walker. The hopper and walker present a particular challenge, due to underactuation and contact discontinuities. Joint angles and kinematics Control Standard deviations Fully connected layer 30 units Input layer Mean parameters Sampling Screen input 4×4 4×4 4×4 4×4 4×4 4×4 4×4 4×4 Control Hidden layer 20 units Conv. layer Conv. layer Input layer 16 ﬁlters 16 ﬁlters Action probabilities Sampling Figure 3. Neural networks used for the locomotion task (top) and for playing Atari games (bottom). 3.


gure 3. Neural networks used for the locomotion task (top) and for playing Atari games (bottom). 3. Can TRPO be used to solve challenging large-scale problems? How does TRPO compare with other methods when applied to large-scale problems, with regard to ﬁnal performance, computation time, and sample complexity? To answer (1) and (2), we compare the performance of the single path and vine variants of TRPO, several ablated variants, and a number of prior policy optimization algorithms. With regard to (3), we show that both the single path and vine algorithm can obtain high-quality locomotion controllers from scratch, which is considered to be a hard problem. We also show that these algorithms produce competitive results when learning policies for playing Atari games from images using convolu


produce competitive results when learning policies for playing Atari games from images using convolutional neural networks with tens of thousands of parameters. 8.1 Simulated Robotic Locomotion We conducted the robotic locomotion experiments using the MuJoCo simulator (Todorov et al., 2012). The three simulated robots are shown in Figure 2. The states of the robots are their generalized positions and velocities, and the controls are joint torques. Underactuation, high dimensionality, and non-smooth dynamics due to contacts make these


Trust Region Policy Optimization tasks very challenging. The following models are included in our evaluation: 1. Swimmer. 10-dimensional state space, linear reward for forward progress and a quadratic penalty on joint effort to produce the reward r(x, u) = vx−10−5∥u∥2. The swimmer can propel itself forward by making an undulating motion. 2. Hopper. 12-dimensional state space, same reward as the swimmer, with a bonus of +1 for being in a nonterminal state. We ended the episodes when the hopper fell over, which was deﬁned by thresholds on the torso height and angle. 3. Walker. 18-dimensional state space. For the walker, we added a penalty for strong impacts of the feet against the ground to encourage a smooth walk rather than a hopping gait. We used δ = 0.01 for all experiments.


ground to encourage a smooth walk rather than a hopping gait. We used δ = 0.01 for all experiments. See Table 2 in the Appendix for more details on the experimental setup and parameters used. We used neural networks to represent the policy, with the architecture shown in Figure 3, and further details provided in Appendix D. To establish a standard baseline, we also included the classic cart-pole balancing problem, based on the formulation from Barto et al. (1983), using a linear policy with six parameters that is easy to optimize with derivative-free black-box optimization methods. The following algorithms were considered in the comparison: single path TRPO; vine TRPO; cross-entropy method (CEM), a gradient-free method (Szita & L¨orincz, 2006); covariance matrix adaption (CMA), another gra


CEM), a gradient-free method (Szita & L¨orincz, 2006); covariance matrix adaption (CMA), another gradient-free method (Hansen & Ostermeier, 1996); natural gradient, the classic natural policy gradient algorithm (Kakade, 2002), which differs from single path by the use of a ﬁxed penalty coefﬁcient (Lagrange multiplier) instead of the KL divergence constraint; empirical FIM, identical to single path, except that the FIM is estimated using the covariance matrix of the gradients rather than the analytic estimate; max KL, which was only tractable on the cart-pole problem, and uses the maximum KL divergence in Equation (11), rather than the average divergence, allowing us to evaluate the quality of this approximation. The parameters used in the experiments are provided in Appendix E.


e quality of this approximation. The parameters used in the experiments are provided in Appendix E. For the natural gradient method, we swept through the possible values of the stepsize in factors of three, and took the best value according to the ﬁnal performance. Learning curves showing the total reward averaged across ﬁve runs of each algorithm are shown in Figure 4. Single path and vine TRPO solved all of the problems, yielding the best solutions. Natural gradient performed well on the two easier problems, but was unable to generate hopping and walking gaits that made forward progress. These results provide empirical evidence that constraining the KL divergence is a more robust way to choose step sizes and make fast, consistent progress, compared to using a ﬁxed 0 10 20 30 40 50 number


oose step sizes and make fast, consistent progress, compared to using a ﬁxed 0 10 20 30 40 50 number of policy iterations 0 2 4 6 8 10 reward Cartpole Vine Single Path Natural Gradient Max KL Empirical FIM CEM CMA RWR 0 10 20 30 40 50 number of policy iterations 0.10 0.05 0.00 0.05 0.10 0.15 cost (-velocity + ctrl) Swimmer Vine Single Path Natural Gradient Empirical FIM CEM CMA RWR 0 50 100 150 200 number of policy iterations 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 reward Hopper Vine Single Path Natural Gradient CEM RWR 0 50 100 150 200 number of policy iterations 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 reward Walker Vine Single Path Natural Gradient CEM RWR Figure 4. Learning curves for locomotion tasks, averaged across ﬁve runs of each algorithm with random initializations.


urves for locomotion tasks, averaged across ﬁve runs of each algorithm with random initializations. Note that for the hopper and walker, a score of −1 is achievable without any forward velocity, indicating a policy that simply learned balanced standing, but not walking. penalty. CEM and CMA are derivative-free algorithms, hence their sample complexity scales unfavorably with the number of parameters, and they performed poorly on the larger problems. The max KL method learned somewhat more slowly than our ﬁnal method, due to the more restrictive form of the constraint, but overall the result suggests that the average KL divergence constraint has a similar effect as the theorecally justiﬁed maximum KL divergence.


ge KL divergence constraint has a similar effect as the theorecally justiﬁed maximum KL divergence. Videos of the policies learned by TRPO may be viewed on the project website: http://sites.google.com/ site/trpopaper/. Note that TRPO learned all of the gaits with generalpurpose policies and simple reward functions, using minimal prior knowledge. This is in contrast with most prior methods for learning locomotion, which typically rely on hand-architected policy classes that explicitly encode notions of balance and stepping (Tedrake et al., 2004; Geng et al., 2006; Wampler & Popovi´c, 2009). 8.2 Playing Games from Images To evaluate TRPO on a partially observed task with complex observations, we trained policies for playing Atari games, using raw images as input.


with complex observations, we trained policies for playing Atari games, using raw images as input. The games require learning a variety of behaviors, such as dodging bullets and hitting balls with paddles. Aside from the high dimensionality, challenging elements of these games include delayed rewards (no immediate penalty is incurred when a life is lost in Breakout or Space Invaders); complex sequences of behavior (Q*bert requires a character to hop on 21 different platforms); and non-stationary image statistics (Enduro involves a changing and ﬂickering background). We tested our algorithms on the same seven games reported on in (Mnih et al., 2013) and (Guo et al., 2014), which are


Trust Region Policy Optimization B. Rider Breakout Enduro Pong Q*bert Seaquest S. Invaders Random 354 1.2 0 −20.4 157 110 179 Human (Mnih et al., 2013) 7456 31.0 368 −3.0 18900 28010 3690 Deep Q Learning (Mnih et al., 2013) 4092 168.0 470 20.0 1952 1705 581 UCC-I (Guo et al., 2014) 5702 380 741 21 20025 2995 692 TRPO - single path 1425.2 10.8 534.6 20.9 1973.5 1908.6 568.4 TRPO - vine 859.5 34.2 430.8 20.9 7732.5 788.4 450.2 Table 1. Performance comparison for vision-based RL algorithms on the Atari domain. Our algorithms (bottom rows) were run once on each task, with the same architecture and parameters. Performance varies substantially from run to run (with different random initializations of the policy), but we could not obtain error statistics due to time constraints.


m initializations of the policy), but we could not obtain error statistics due to time constraints. made available through the Arcade Learning Environment (Bellemare et al., 2013) The images were preprocessed following the protocol in Mnih et al (2013), and the policy was represented by the convolutional neural network shown in Figure 3, with two convolutional layers with 16 channels and stride 2, followed by one fully-connected layer with 20 units, yielding 33,500 parameters. The results of the vine and single path algorithms are summarized in Table 1, which also includes an expert human performance and two recent methods: deep Q-learning (Mnih et al., 2013), and a combination of Monte-Carlo Tree Search with supervised training (Guo et al., 2014), called UCC-I.


a combination of Monte-Carlo Tree Search with supervised training (Guo et al., 2014), called UCC-I. The 500 iterations of our algorithm took about 30 hours (with slight variation between games) on a 16-core computer. While our method only outperformed the prior methods on some of the games, it consistently achieved reasonable scores. Unlike the prior methods, our approach was not designed speciﬁcally for this task. The ability to apply the same policy search method to methods as diverse as robotic locomotion and image-based game playing demonstrates the generality of TRPO. 9 Discussion We proposed and analyzed trust region methods for optimizing stochastic control policies. We proved monotonic improvement for an algorithm that repeatedly optimizes a local approximation to the expected retu


ic improvement for an algorithm that repeatedly optimizes a local approximation to the expected return of the policy with a KL divergence penalty, and we showed that an approximation to this method that incorporates a KL divergence constraint achieves good empirical results on a range of challenging policy learning tasks, outperforming prior methods. Our analysis also provides a perspective that uniﬁes policy gradient and policy iteration methods, and shows them to be special limiting cases of an algorithm that optimizes a certain objective subject to a trust region constraint. In the domain of robotic locomotion, we successfully learned controllers for swimming, walking and hopping in a physics simulator, using general purpose neural networks and minimally informative rewards.


ng in a physics simulator, using general purpose neural networks and minimally informative rewards. To our knowledge, no prior work has learned controllers from scratch for all of these tasks, using a generic policy search method and non-engineered, general-purpose policy representations. In the game-playing domain, we learned convolutional neural network policies that used raw images as inputs. This requires optimizing extremely high-dimensional policies, and only two prior methods report successful results on this task. Since the method we proposed is scalable and has strong theoretical foundations, we hope that it will serve as a jumping-off point for future work on training large, rich function approximators for a range of challenging problems.


for future work on training large, rich function approximators for a range of challenging problems. At the intersection of the two experimental domains we explored, there is the possibility of learning robotic control policies that use vision and raw sensory data as input, providing a uniﬁed scheme for training robotic controllers that perform both perception and control. The use of more sophisticated policies, including recurrent policies with hidden state, could further make it possible to roll state estimation and control into the same policy in the partiallyobserved setting. By combining our method with model learning, it would also be possible to substantially reduce its sample complexity, making it applicable to real-world settings where samples are expensive.


uce its sample complexity, making it applicable to real-world settings where samples are expensive. Acknowledgements We thank Emo Todorov and Yuval Tassa for providing the MuJoCo simulator; Bruno Scherrer, Tom Erez, Greg Wayne, and the anonymous ICML reviewers for insightful comments, and Vitchyr Pong and Shane Gu for pointing our errors in a previous version of the manuscript. This research was funded in part by the Ofﬁce of Naval Research through a Young Investigator Award and under grant number N00014-11-1-0688, DARPA through a Young Faculty Award, by the Army Research Ofﬁce through the MAST program. References Bagnell, J. A. and Schneider, J. Covariant policy search. IJCAI, 2003. Bartlett, P. L. and Baxter, J. Inﬁnite-horizon policy-gradient estimation.


icy search. IJCAI, 2003. Bartlett, P. L. and Baxter, J. Inﬁnite-horizon policy-gradient estimation. arXiv preprint arXiv:1106.0665, 2011. Barto, A., Sutton, R., and Anderson, C. Neuronlike adaptive elements that can solve difﬁcult learning control problems. IEEE Transactions on Systems, Man and Cybernetics, (5):834–846, 1983.


Trust Region Policy Optimization Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 47:253– 279, jun 2013. Bertsekas, D. Dynamic programming and optimal control, volume 1. 2005. Deisenroth, M., Neumann, G., and Peters, J. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(12):1–142, 2013. Gabillon, Victor, Ghavamzadeh, Mohammad, and Scherrer, Bruno. Approximate dynamic programming ﬁnally performs well in the game of Tetris. In Advances in Neural Information Processing Systems, 2013. Geng, T., Porr, B., and W¨org¨otter, F. Fast biped walking with a reﬂexive controller and realtime policy searching.


., and W¨org¨otter, F. Fast biped walking with a reﬂexive controller and realtime policy searching. In Advances in Neural Information Processing Systems (NIPS), 2006. Guo, X., Singh, S., Lee, H., Lewis, R. L., and Wang, X. Deep learning for real-time atari game play using ofﬂine MonteCarlo tree search planning. In Advances in Neural Information Processing Systems, pp. 3338–3346, 2014. Hansen, Nikolaus and Ostermeier, Andreas. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Evolutionary Computation, 1996., Proceedings of IEEE International Conference on, pp. 312–317. IEEE, 1996. Hunter, David R and Lange, Kenneth. A tutorial on MM algorithms. The American Statistician, 58(1):30–37, 2004. Kakade, Sham. A natural policy gradient.


algorithms. The American Statistician, 58(1):30–37, 2004. Kakade, Sham. A natural policy gradient. In Advances in Neural Information Processing Systems, pp. 1057–1063. MIT Press, 2002. Kakade, Sham and Langford, John. Approximately optimal approximate reinforcement learning. In ICML, volume 2, pp. 267–274, 2002. Lagoudakis, Michail G and Parr, Ronald. Reinforcement learning as classiﬁcation: Leveraging modern classiﬁers. In ICML, volume 3, pp. 424–431, 2003. Levin, D. A., Peres, Y., and Wilmer, E. L. Markov chains and mixing times. American Mathematical Society, 2009. Levine, Sergey and Abbeel, Pieter. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071–1079, 2014. Martens, J. and Sutskever, I.


vances in Neural Information Processing Systems, pp. 1071–1079, 2014. Martens, J. and Sutskever, I. Training deep and recurrent networks with hessian-free optimization. In Neural Networks: Tricks of the Trade, pp. 479–535. Springer, 2012. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. Nemirovski, Arkadi. Efﬁcient methods in convex programming. 2005. Ng, A. Y. and Jordan, M. PEGASUS: A policy search method for large mdps and pomdps. In Uncertainty in artiﬁcial intelligence (UAI), 2000. Owen, Art B. Monte Carlo theory, methods and examples. 2013. Pascanu, Razvan and Bengio, Yoshua. Revisiting natural gradient for deep networks.


examples. 2013. Pascanu, Razvan and Bengio, Yoshua. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013. Peters, J. and Schaal, S. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21(4):682–697, 2008a. Peters, J., M¨ulling, K., and Alt¨un, Y. Relative entropy policy search. In AAAI Conference on Artiﬁcial Intelligence, 2010. Peters, Jan and Schaal, Stefan. Natural actor-critic. Neurocomputing, 71(7):1180–1190, 2008b. Pirotta, Matteo, Restelli, Marcello, Pecorino, Alessio, and Calandriello, Daniele. Safe policy iteration. In Proceedings of The 30th International Conference on Machine Learning, pp. 307– 315, 2013. Pollard, David. Asymptopia: an exposition of statistical asymptotic theory. 2000.


. 307– 315, 2013. Pollard, David. Asymptopia: an exposition of statistical asymptotic theory. 2000. URL http://www.stat.yale.edu/ ˜pollard/Books/Asymptopia. Szita, Istv´an and L¨orincz, Andr´as. Learning tetris using the noisy cross-entropy method. Neural computation, 18(12): 2936–2941, 2006. Tedrake, R., Zhang, T., and Seung, H. Stochastic policy gradient reinforcement learning on a simple 3d biped. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2004. Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. MuJoCo: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026–5033. IEEE, 2012. Wampler, Kevin and Popovi´c, Zoran. Optimal gait and form for animal locomotion.


–5033. IEEE, 2012. Wampler, Kevin and Popovi´c, Zoran. Optimal gait and form for animal locomotion. In ACM Transactions on Graphics (TOG), volume 28, pp. 60. ACM, 2009. Wright, Stephen J and Nocedal, Jorge. Numerical optimization, volume 2. Springer New York, 1999.


Trust Region Policy Optimization A Proof of Policy Improvement Bound This proof (of Theorem 1) uses techniques from the proof of Theorem 4.1 in (Kakade & Langford, 2002), adapting them to the more general setting considered in this paper. An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly deﬁne the policies π and π′ so that they choose the same action with high probability = (1 −α). Surrogate loss Lπ(˜π) accounts for the the advantage of ˜π the ﬁrst time that it disagrees with π, but not subsequent disagreements. Hence, the error in Lπ is due to two or more disagreements between π and ˜π, hence, we get an O(α2) correction term, where α is the probability of disagreement.


tween π and ˜π, hence, we get an O(α2) correction term, where α is the probability of disagreement. We start out with a lemma from Kakade & Langford (2002) that shows that the difference in policy performance η(˜π)−η(π) can be decomposed as a sum of per-timestep advantages. Lemma 1. Given two policies π, ˜π, η(˜π) = η(π)+Eτ∼˜π " ∞ X t=0 γtAπ(st, at) # (19) This expectation is taken over trajectories τ := (s0, a0, s1, a0, . . . ), and the notation Eτ∼˜π [. . . ] indicates that actions are sampled from ˜π to generate τ. Proof. First note that Aπ(s, a) = Es′∼P (s′|s,a) [r(s) + γVπ(s′) −Vπ(s)]. Therefore, Eτ|˜π " ∞ X t=0 γtAπ(st, at) # (20) = Eτ|˜π " ∞ X t=0 γt(r(st) + γVπ(st+1) −Vπ(st)) # (21) = Eτ|˜π " −Vπ(s0) + ∞ X t=0 γtr(st) # (22) = −Es0 [Vπ(s0)] + Eτ|˜π " ∞ X t=0 γtr(st) # (23) = −η(π)


= Eτ|˜π " −Vπ(s0) + ∞ X t=0 γtr(st) # (22) = −Es0 [Vπ(s0)] + Eτ|˜π " ∞ X t=0 γtr(st) # (23) = −η(π) + η(˜π) (24) Rearranging, the result follows. Deﬁne ¯A(s) to be the expected advantage of ˜π over π at state s: ¯A(s) = Ea∼˜π(·|s) [Aπ(s, a)] . (25) Now Lemma 1 can be written as follows: η(˜π) = η(π) + Eτ∼˜π " ∞ X t=0 γt ¯A(st) # (26) Note that Lπ can be written as Lπ(˜π) = η(π) + Eτ∼π " ∞ X t=0 γt ¯A(st) # (27) The difference in these equations is whether the states are sampled using π or ˜π. To bound the difference between η(˜π) and Lπ(˜π), we will bound the difference arising from each timestep. To do this, we ﬁrst need to introduce a measure of how much π and ˜π agree. Speciﬁcally, we’ll couple the policies, so that they deﬁne a joint distribution over pairs of actions. Deﬁnition 1.


ll couple the policies, so that they deﬁne a joint distribution over pairs of actions. Deﬁnition 1. (π, ˜π) is an α-coupled policy pair if it deﬁnes a joint distribution (a, ˜a)|s, such that P(a ̸= ˜a|s) ≤α for all s. π and ˜π will denote the marginal distributions of a and ˜a, respectively.


Trust Region Policy Optimization Computationally, α-coupling means that if we randomly choose a seed for our random number generator, and then we sample from each of π and ˜π after setting that seed, the results will agree for at least fraction 1 −α of seeds. Lemma 2. Given that π, ˜π are α-coupled policies, for all s, ¯A(s) ≤2α max s,a |Aπ(s, a)| (28) Proof. ¯A(s) = E˜a∼˜π [Aπ(s, ˜a)] = E(a,˜a)∼(π,˜π) [Aπ(s, ˜a) −Aπ(s, a)] since Ea∼π [Aπ(s, a)] = 0 (29) = P(a ̸= ˜a|s)E(a,˜a)∼(π,˜π)|a̸=˜a [Aπ(s, ˜a) −Aπ(s, a)] (30) | ¯A(s)| ≤α · 2 max s,a |Aπ(s, a)| (31) Lemma 3. Let (π, ˜π) be an α-coupled policy pair. Then Est∼˜π  ¯A(st)  −Est∼π  ¯A(st)  ≤2α max s ¯A(s) ≤4α(1 −(1 −α)t) max s |Aπ(s, a)| (32) Proof.


en Est∼˜π  ¯A(st)  −Est∼π  ¯A(st)  ≤2α max s ¯A(s) ≤4α(1 −(1 −α)t) max s |Aπ(s, a)| (32) Proof. Given the coupled policy pair (π, ˜π), we can also obtain a coupling over the trajectory distributions produced by π and ˜π, respectively. Namely, we have pairs of trajectories τ, ˜τ, where τ is obtained by taking actions from π, and ˜τ is obtained by taking actions from ˜π, where the same random seed is used to generate both trajectories. We will consider the advantage of ˜π over π at timestep t, and decompose this expectation based on whether π agrees with ˜π at all timesteps i < t. Let nt denote the number of times that ai ̸= ˜ai for i < t, i.e., the number of times that π and ˜π disagree before timestep t.


times that ai ̸= ˜ai for i < t, i.e., the number of times that π and ˜π disagree before timestep t. Est∼˜π  ¯A(st)  = P(nt = 0)Est∼˜π|nt=0  ¯A(st)  + P(nt > 0)Est∼˜π|nt>0  ¯A(st)  (33) The expectation decomposes similarly for actions are sampled using π: Est∼π  ¯A(st)  = P(nt = 0)Est∼π|nt=0  ¯A(st)  + P(nt > 0)Est∼π|nt>0  ¯A(st)  (34) Note that the nt = 0 terms are equal: Est∼˜π|nt=0  ¯A(st)  = Est∼π|nt=0  ¯A(st)  , (35) because nt = 0 indicates that π and ˜π agreed on all timesteps less than t. Subtracting Equations (33) and (34), we get Est∼˜π  ¯A(st)  −Est∼π  ¯A(st)  = P(nt > 0) Est∼˜π|nt>0  ¯A(st)  −Est∼π|nt>0  ¯A(st)  (36) By deﬁnition of α, P(π, ˜π agree at timestep i) ≥1 −α, so P(nt = 0) ≥(1 −α)t, and P(nt > 0) ≤1 −(1 −α)t (37) Next, note that Est∼˜π|nt>0


mestep i) ≥1 −α, so P(nt = 0) ≥(1 −α)t, and P(nt > 0) ≤1 −(1 −α)t (37) Next, note that Est∼˜π|nt>0  ¯A(st)  −Est∼π|nt>0  ¯A(st)  ≤ Est∼˜π|nt>0  ¯A(st)  + Est∼π|nt>0  ¯A(st)  (38) ≤4α max s,a |Aπ(s, a)| (39) Where the second inequality follows from Lemma 3. Plugging Equation (37) and Equation (39) into Equation (36), we get Est∼˜π  ¯A(st)  −Est∼π  ¯A(st)  ≤4α(1 −(1 −α)t) max s,a |Aπ(s, a)| (40)


Trust Region Policy Optimization The preceding Lemma bounds the difference in expected advantage at each timestep t. We can sum over time to bound the difference between η(˜π) and Lπ(˜π). Subtracting Equation (26) and Equation (27), and deﬁning ϵ = maxs,a |Aπ(s, a)|, |η(˜π) −Lπ(˜π)| = ∞ X t=0 γt Eτ∼˜π  ¯A(st)  −Eτ∼π  ¯A(st)  (41) ≤ ∞ X t=0 γt · 4ϵα(1 −(1 −α)t) (42) = 4ϵα  1 1 −γ − 1 1 −γ(1 −α)  (43) = 4α2γϵ (1 −γ)(1 −γ(1 −α)) (44) ≤ 4α2γϵ (1 −γ)2 (45) Last, to replace α by the total variation divergence, we need to use the correspondence between TV divergence and coupled random variables: Suppose pX and pY are distributions with DT V (pX ∥pY ) = α. Then there exists a joint distribution (X, Y ) whose marginals are pX, pY , for which X = Y with probability 1 −α.


ts a joint distribution (X, Y ) whose marginals are pX, pY , for which X = Y with probability 1 −α. See (Levin et al., 2009), Proposition 4.7. It follows that if we have two policies π and ˜π such that maxs DT V (π(·|s) ∥˜π(·|s)) ≤α, then we can deﬁne an α-coupled policy pair (π, ˜π) with appropriate marginals. Taking α = maxs DT V (π(·|s) ∥˜π(·|s)) ≤α in Equation (45), Theorem 1 follows. B Perturbation Theory Proof of Policy Improvement Bound We also provide an alternative proof of Theorem 1 using perturbation theory. Proof. Let G = (1+γPπ+(γPπ)2+. . . ) = (1−γPπ)−1, and similarly Let ˜G = (1+γP˜π+(γP˜π)2+. . . ) = (1−γP˜π)−1. We will use the convention that ρ (a density on state space) is a vector and r (a reward function on state space) is a dual vector (i.e., linear functional on vecto


a vector and r (a reward function on state space) is a dual vector (i.e., linear functional on vectors), thus rρ is a scalar meaning the expected reward under density ρ. Note that η(π) = rGρ0, and η(˜π) = c ˜Gρ0. Let ∆= P˜π −Pπ. We want to bound η(˜π) −η(π) = r( ˜G −G)ρ0. We start with some standard perturbation theory manipulations. G−1 −˜G−1 = (1 −γPπ) −(1 −γP˜π) = γ∆. (46) Left multiply by G and right multiply by ˜G. ˜G −G = γG∆˜G ˜G = G + γG∆˜G (47) Substituting the right-hand side into ˜G gives ˜G = G + γG∆G + γ2G∆G∆˜G (48) So we have η(˜π) −η(π) = r( ˜G −G)ρ = γrG∆Gρ0 + γ2rG∆G∆˜Gρ0 (49) Let us ﬁrst consider the leading term γrG∆Gρ0. Note that rG = v, i.e., the inﬁnite-horizon state-value function. Also note that Gρ0 = ρπ. Thus we can write γcG∆Gρ0 = γv∆ρπ.


e inﬁnite-horizon state-value function. Also note that Gρ0 = ρπ. Thus we can write γcG∆Gρ0 = γv∆ρπ. We will show that this expression equals the expected


Trust Region Policy Optimization advantage Lπ(˜π) −Lπ(π). Lπ(˜π) −Lπ(π) = X s ρπ(s) X a (˜π(a|s) −π(a|s))Aπ(s, a) = X s ρπ(s) X a πθ(a|s) −π˜θ(a|s)  " r(s) + X s′ p(s′|s, a)γv(s′) −v(s) # = X s ρπ(s) X s′ X a (π(a|s) −˜π(a|s)) p(s′|s, a)γv(s′) = X s ρπ(s) X s′ (pπ(s′|s) −p˜π(s′|s))γv(s′) = γv∆ρπ (50) Next let us bound the O(∆2) term γ2rG∆G∆˜Gρ. First we consider the product γrG∆= γv∆. Consider the component s of this dual vector. |(γv∆)s| = X a (˜π(s, a) −π(s, a))Qπ(s, a) = X a (˜π(s, a) −π(s, a))Aπ(s, a) ≤ X a |˜π(s, a) −π(s, a)| · max a |Aπ(s, a)| ≤2αϵ (51) where the last line used the deﬁnition of the total-variation divergence, and the deﬁnition of ϵ = maxs,a |Aπ(s, a)|. We bound the other portion G∆˜Gρ using the ℓ1 operator norm ∥A∥1 = sup ρ ∥Aρ∥1 ∥ρ∥1  (52) where we have that ∥G∥


ther portion G∆˜Gρ using the ℓ1 operator norm ∥A∥1 = sup ρ ∥Aρ∥1 ∥ρ∥1  (52) where we have that ∥G∥1 = ∥˜G∥1 = 1/(1 −γ) and ∥∆∥1 = 2α. That gives ∥G∆˜Gρ∥1 ≤∥G∥1∥∆∥1∥˜G∥1∥ρ∥1 = 1 1 −γ · 2α · 1 1 −γ · 1 (53) So we have that γ2 rG∆G∆˜Gρ ≤γ∥γrG∆∥∞∥G∆˜Gρ∥1 ≤γ∥v∆∥∞∥G∆˜Gρ∥1 ≤γ · 2αϵ · 2α (1 −γ)2 = 4γϵ (1 −γ)2 α2 (54) C Efﬁciently Solving the Trust-Region Constrained Optimization Problem This section describes how to efﬁciently approximately solve the following constrained optimization problem, which we must solve at each iteration of TRPO: maximize L(θ) subject to DKL(θold, θ) ≤δ. (55)


Trust Region Policy Optimization The method we will describe involves two steps: (1) compute a search direction, using a linear approximation to objective and quadratic approximation to the constraint; and (2) perform a line search in that direction, ensuring that we improve the nonlinear objective while satisfying the nonlinear constraint. The search direction is computed by approximately solving the equation Ax = g, where A is the Fisher information matrix, i.e., the quadratic approximation to the KL divergence constraint: DKL(θold, θ) ≈1 2(θ −θold)T A(θ −θold), where Aij = ∂ ∂θi ∂ ∂θj DKL(θold, θ). In large-scale problems, it is prohibitively costly (with respect to computation and memory) to form the full matrix A (or A−1).


s prohibitively costly (with respect to computation and memory) to form the full matrix A (or A−1). However, the conjugate gradient algorithm allows us to approximately solve the equation Ax = b without forming this full matrix, when we merely have access to a function that computes matrix-vector products y →Ay. Appendix C.1 describes the most efﬁcient way to compute matrix-vector products with the Fisher information matrix. For additional exposition on the use of Hessian-vector products for optimizing neural network objectives, see (Martens & Sutskever, 2012) and (Pascanu & Bengio, 2013). Having computed the search direction s ≈A−1g, we next need to compute the maximal step length β such that θ + βs will satisfy the KL divergence constraint.


ed to compute the maximal step length β such that θ + βs will satisfy the KL divergence constraint. To do this, let δ = DKL ≈1 2(βs)T A(βs) = 1 2β2sT As. From this, we obtain β = p 2δ/sT As, where δ is the desired KL divergence. The term sT As can be computed through a single Hessian vector product, and it is also an intermediate result produced by the conjugate gradient algorithm. Last, we use a line search to ensure improvement of the surrogate objective and satisfaction of the KL divergence constraint, both of which are nonlinear in the parameter vector θ (and thus depart from the linear and quadratic approximations used to compute the step). We perform the line search on the objective Lθold(θ) −X[DKL(θold, θ) ≤δ], where X[. . .


the step). We perform the line search on the objective Lθold(θ) −X[DKL(θold, θ) ≤δ], where X[. . . ] equals zero when its argument is true and +∞when it is false. Starting with the maximal value of the step length β computed in the previous paragraph, we shrink β exponentially until the objective improves. Without this line search, the algorithm occasionally computes large steps that cause a catastrophic degradation of performance. C.1 Computing the Fisher-Vector Product Here we will describe how to compute the matrix-vector product between the averaged Fisher information matrix and arbitrary vectors. This matrix-vector product enables us to perform the conjugate gradient algorithm. Suppose that the parameterized policy maps from the input x to “distribution parameter” vector µθ(x), which