. . . 8 5.3 What is the beneﬁt of using a longer context length? . . . . . . . . . . . . . . . . . 9 5.4 Does Decision Transformer perform effective long-term credit assignment? . . . . 9 5.5 Can transformers be accurate critics in sparse reward settings? . . . . . . . . . . . 10 5.6 Does Decision Transformer perform well in sparse reward settings? . . . . . . . . 10 5.7 Why does Decision Transformer avoid the need for value pessimism or behavior regularization? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.8 How can Decision Transformer beneﬁt online RL regimes? . . . . . . . . . . . . . 11 6 Related Work 11 6.1 Ofﬂine reinforcement learning . . . . . . . . . . . . . . . . . . . . . . . . . . . .


d Work 11 6.1 Ofﬂine reinforcement learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 6.2 Supervised learning in reinforcement learning settings . . . . . . . . . . . . . . . . 11 6.3 Credit assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 6.4 Conditional language generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 6.5 Attention and transformer models . . . . . . . . . . . . . . . . . . . . . . . . . . 12 7 Conclusion 12 A Experimental Details 18 A.1 Atari . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.2 OpenAI Gym . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.2.1 Decision Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . . . 18 A.2.1 Decision Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A.2.2 Behavior Cloning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.3 Graph Shortest Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 B Atari Task Scores 20 2


1 Introduction Recent work has shown transformers [1] can model high-dimensional distributions of semantic concepts at scale, including effective zero-shot generalization in language [2] and out-of-distribution image generation [3]. Given the diversity of successful applications of such models, we seek to examine their application to sequential decision making problems formalized as reinforcement learning (RL). In contrast to prior work using transformers as an architectural choice for components within traditional RL algorithms [4, 5], we seek to study if generative trajectory modeling – i.e. modeling the joint distribution of the sequence of states, actions, and rewards – can serve as a replacement for conventional RL algorithms.


quence of states, actions, and rewards – can serve as a replacement for conventional RL algorithms. We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning [6], we will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping for long term credit assignment – thereby avoiding one of the “deadly triad” [6] known to destabilize RL. It also avoids the need for discounting future rewards, as typically done in TD learning, which can induce undesirable short-sighted behaviors. Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy to scale, utilizing a large body


former frameworks widely used in language and vision that are easy to scale, utilizing a large body of work studying stable training of transformer models. In addition to their demonstrated ability to model long sequences, transformers also have other advantages. Transformers can perform credit assignment directly via self-attention, in contrast to Bellman backups which slowly propagate rewards and are prone to “distractor” signals [7]. This can enable transformers to still work effectively in the presence of sparse or distracting rewards. Finally, empirical evidence suggest that a transformer modeling approach can model a wide distribution of behaviors, enabling better generalization and transfer [3].


proach can model a wide distribution of behaviors, enabling better generalization and transfer [3]. We explore our hypothesis by considering ofﬂine RL, where we will task agents with learning policies from suboptimal data – producing maximally effective behavior from ﬁxed, limited experience. This task is traditionally challenging due to error propagation and value overestimation [8]. However, it is a natural task when training with a sequence modeling objective. By training an autoregressive model on sequences of states, actions, and returns, we reduce policy sampling to autoregressive generative modeling. We can specify the expertise of the policy – which “skill” to query – by selecting the desired return tokens, acting as a prompt for generation. Illustrative example.


– by selecting the desired return tokens, acting as a prompt for generation. Illustrative example. To get an intuition for our proposal, consider the task of ﬁnding the shortest path on a directed graph, which can be posed as an RL problem. The reward is 0 when the agent is at the goal node and −1 otherwise. We train a GPT [9] model to predict next token in a sequence of returns-to-go (sum of future rewards), states, and actions. Training only on random walk data – with no expert demonstrations – we can generate optimal trajectories at test time by adding a prior to generate highest possible returns (see more details and empirical results in the Appendix) and subsequently generate the corresponding sequence of actions via conditioning.


in the Appendix) and subsequently generate the corresponding sequence of actions via conditioning. Thus, by combining the tools of sequence modeling with hindsight return information, we achieve policy improvement without the need for dynamic programming. ... goal -2 -1 -3 -4 -1 -3 -4 -∞ -∞ -∞ -∞ graph training dataset (random walks) goal start -3 -1 generation -2 0 0 -∞ Figure 2: Illustrative example of ﬁnding shortest path for a ﬁxed graph (left) posed as reinforcement learning. Training dataset consists of random walk trajectories and their per-node returns-to-go (middle). Conditioned on a starting state and generating largest possible return at each node, Decision Transformer sequences optimal paths. 3


Motivated by this observation, we propose Decision Transformer, where we use the GPT architecture to autoregressively model trajectories (shown in Figure 1). We study whether sequence modeling can perform policy optimization by evaluating Decision Transformer on ofﬂine RL benchmarks in Atari [10], OpenAI Gym [11], and Key-to-Door [12] environments. We show that – without using dynamic programming – Decision Transformer matches or exceeds the performance of state-of-the-art model-free ofﬂine RL algorithms [13, 14]. Furthermore, in tasks where long-term credit assignment is required, Decision Transformer capably outperforms the RL baselines. With this work, we aim to bridge sequence modeling and transformers with RL, and hope that sequence modeling serves as a strong algorithmic paradigm for


nd transformers with RL, and hope that sequence modeling serves as a strong algorithmic paradigm for RL. 2 Preliminaries 2.1 Ofﬂine reinforcement learning We consider learning in a Markov decision process (MDP) described by the tuple (S, A, P, R). The MDP tuple consists of states s ∈S, actions a ∈A, transition dynamics P(s′|s, a), and a reward function r = R(s, a). We use st, at, and rt = R(st, at) to denote the state, action, and reward at timestep t, respectively. A trajectory is made up of a sequence of states, actions, and rewards: τ = (s0, a0, r0, s1, a1, r1, . . . , sT , aT , rT ). The return of a trajectory at timestep t, Rt = PT t′=t rt′, is the sum of future rewards from that timestep.


n of a trajectory at timestep t, Rt = PT t′=t rt′, is the sum of future rewards from that timestep. The goal in reinforcement learning is to learn a policy which maximizes the expected return E hPT t=1 rt i in an MDP. In ofﬂine reinforcement learning, instead of obtaining data via environment interactions, we only have access to some ﬁxed limited dataset consisting of trajectory rollouts of arbitrary policies. This setting is harder as it removes the ability for agents to explore the environment and collect additional feedback. 2.2 Transformers Transformers were proposed by Vaswani et al. [1] as an architecture to efﬁciently model sequential data. These models consist of stacked self-attention layers with residual connections.


l sequential data. These models consist of stacked self-attention layers with residual connections. Each selfattention layer receives n embeddings {xi}n i=1 corresponding to unique input tokens, and outputs n embeddings {zi}n i=1, preserving the input dimensions. The i-th token is mapped via linear transformations to a key ki, query qi, and value vi. The i-th output of the self-attention layer is given by weighting the values vj by the normalized dot product between the query qi and other keys kj: zi = n X j=1 softmax({⟨qi, kj′⟩}n j′=1)j · vj. (1) As we shall see later, this allows the layer to assign “credit” by implicitly forming state-return associations via similarity of the query and key vectors (maximizing the dot product).


state-return associations via similarity of the query and key vectors (maximizing the dot product). In this work, we use the GPT architecture [9], which modiﬁes the transformer architecture with a causal selfattention mask to enable autoregressive generation, replacing the summation/softmax over the n tokens with only the previous tokens in the sequence (j ∈[1, i]). We defer the other architecture details to the original papers. 3 Method In this section, we present Decision Transformer, which models trajectories autoregressively with minimal modiﬁcation to the transformer architecture, as summarized in Figure 1 and Algorithm 1. Trajectory representation. The key desiderata in our choice of trajectory representation are that it should enable transformers to learn meaningful patterns and we


rajectory representation are that it should enable transformers to learn meaningful patterns and we should be able to conditionally generate actions at test time. It is nontrivial to model rewards since we would like the model to generate actions based on future desired returns, rather than past rewards. As a result, instead of feeding the rewards directly, we feed the model with the returns-to-go bRt = PT t′=t rt′. This leads to the following trajectory representation which is amenable to autoregressive training and generation: τ =  bR1, s1, a1, bR2, s2, a2, . . . , bRT , sT , aT  . (2) 4


At test time, we can specify the desired performance (e.g. 1 for success or 0 for failure), as well as the environment starting state, as the conditioning information to initiate generation. After executing the generated action for the current state, we decrement the target return by the achieved reward and repeat until episode termination. Architecture. We feed the last K timesteps into Decision Transformer, for a total of 3K tokens (one for each modality: return-to-go, state, or action). To obtain token embeddings, we learn a linear layer for each modality, which projects raw inputs to the embedding dimension, followed by layer normalization [15]. For environments with visual inputs, the state is fed into a convolutional encoder instead of a linear layer.


nments with visual inputs, the state is fed into a convolutional encoder instead of a linear layer. Additionally, an embedding for each timestep is learned and added to each token – note this is different than the standard positional embedding used by transformers, as one timestep corresponds to three tokens. The tokens are then processed by a GPT [9] model, which predicts future action tokens via autoregressive modeling. Training. We are given a dataset of ofﬂine trajectories. We sample minibatches of sequence length K from the dataset. The prediction head corresponding to the input token st is trained to predict at – either with cross-entropy loss for discrete actions or mean-squared error for continuous actions – and the losses for each timestep are averaged.


tions or mean-squared error for continuous actions – and the losses for each timestep are averaged. We did not ﬁnd predicting the states or returns-to-go to improve performance, although it is easily permissible within our framework (as shown in Section 5.4) and would be an interesting study for future work. Algorithm 1 Decision Transformer Pseudocode (for continuous actions) # R, s, a, t: returns -to -go , states , actions , or timesteps # transformer: transformer with causal masking (GPT) # embed_s , embed_a , embed_R: linear embedding layers # embed_t: learned episode positional embedding # pred_a: linear action prediction layer # main model def DecisionTransformer (R, s, a, t): # compute embeddings for tokens pos_embedding = embed_t(t) # per -timestep (note: not per -token) s_embedding


embeddings for tokens pos_embedding = embed_t(t) # per -timestep (note: not per -token) s_embedding = embed_s(s) + pos_embedding a_embedding = embed_a(a) + pos_embedding R_embedding = embed_R(R) + pos_embedding # interleave tokens as (R_1 , s_1 , a_1 , ..., R_K , s_K) input_embeds = stack(R_embedding , s_embedding , a_embedding) # use transformer to get hidden states hidden_states = transformer( input_embeds =input_embeds) # select hidden states for action prediction tokens a_hidden = unstack(hidden_states ). actions # predict action return pred_a(a_hidden) # training loop for (R, s, a, t) in dataloader: # dims: (batch_size , K, dim) a_preds = DecisionTransformer (R, s, a, t) loss = mean (( a_preds - a)**2) # L2 loss for continuous actions optimizer.zero_grad (); loss.backward (); optimiz


a_preds - a)**2) # L2 loss for continuous actions optimizer.zero_grad (); loss.backward (); optimizer.step () # evaluation loop target_return = 1 # for instance , expert -level return R, s, a, t, done = [target_return ], [env.reset ()], [], [1], False while not done: # autoregressive generation/sampling # sample next action action = DecisionTransformer (R, s, a, t)[-1] # for cts actions new_s , r, done , _ = env.step(action) # append new tokens to sequence R = R + [R[-1] - r] # decrement returns -to -go with reward s, a, t = s + [new_s], a + [action], t + [len(R)] R, s, a, t = R[-K:], ... # only keep context length of K 5


Atari OpenAI Gym Key-To-Door 50 100 Performance Decision Transformer (Ours) TD Learning Behavior Cloning Figure 3: Results comparing Decision Transformer (ours) to TD learning (CQL) and behavior cloning across Atari, OpenAI Gym, and Minigrid. On a diverse set of tasks, Decision Transformer performs comparably or better than traditional approaches. Performance is measured by normalized episode return (see text for details). 4 Evaluations on Ofﬂine RL Benchmarks In this section, we investigate the performance of Decision Transformer relative to dedicated ofﬂine RL and imitation learning algorithms. In particular, our primary points of comparison are modelfree ofﬂine RL algorithms based on TD-learning, since our Decision Transformer architecture is fundamentally model-free in nature as well.


earning, since our Decision Transformer architecture is fundamentally model-free in nature as well. Furthermore, TD-learning is the dominant paradigm in RL for sample efﬁciency, and also features prominently as a sub-routine in many model-based RL algorithms [16, 17]. We also compare with behavior cloning and variants, since it also involves a likelihood based policy learning formulation similar to ours. The exact algorithms depend on the environment but our motivations are as follows: • TD learning: most of these methods use an action-space constraint or value pessimism, and will be the most faithful comparison to Decision Transformer, representing standard RL methods. A state-of-the-art model-free method is Conservative Q-Learning (CQL) [14] which serves as our primary comparison.


art model-free method is Conservative Q-Learning (CQL) [14] which serves as our primary comparison. In addition, we also compare against other prior model-free RL algorithms like BEAR [18] and BRAC [19]. • Imitation learning: this regime similarly uses supervised losses for training, rather than Bellman backups. We use behavior cloning here, and include a more detailed discussion in Section 5.1. We evaluate on both discrete (Atari [10]) and continuous (OpenAI Gym [11]) control tasks. The former involves high-dimensional observation spaces and requires long-term credit assignment, while the latter requires ﬁne-grained continuous control, representing a diverse set of tasks. Our main results are summarized in Figure 3, where we show averaged normalized performance for each domain.


results are summarized in Figure 3, where we show averaged normalized performance for each domain. 4.1 Atari The Atari benchmark [10] is challenging due to its high-dimensional visual inputs and difﬁculty of credit assignment arising from the delay between actions and resulting rewards. We evaluate our method on 1% of all samples in the DQN-replay dataset as per Agarwal et al. [13], representing 500 thousand of the 50 million transitions observed by an online DQN agent [20] during training; we report the mean and standard deviation of 3 seeds. We normalize scores based on a professional gamer, following the protocol of Hafner et al. [21], where 100 represents the professional gamer score and 0 represents a random policy.


er et al. [21], where 100 represents the professional gamer score and 0 represents a random policy. We compare to CQL [14], REM [13], and QR-DQN [22] on four Atari tasks (Breakout, Qbert, Pong, and Seaquest) that are evaluated in Agarwal et al. [13]. We use context lengths of K = 30 for Decision Transformer (except K = 50 for Pong). We also report the performance of behavior cloning (BC), which utilizes the same network architecture and hyperparameters as Decision Transformer but does not have return-to-go conditioning2. For CQL, REM, and QR-DQN baselines, we report numbers directly from the CQL and REM papers. We show results in Table 1. Our method is competitive with CQL in 3 out of 4 games and outperforms or matches REM, QR-DQN, and BC on all 4 games.


etitive with CQL in 3 out of 4 games and outperforms or matches REM, QR-DQN, and BC on all 4 games. 2We also tried using an MLP with K = 1 as in prior work, but found this was worse than the transformer. 6


Game DT (Ours) CQL QR-DQN REM BC Breakout 267.5 ± 97.5 211.1 17.1 8.9 138.9 ± 61.7 Qbert 15.4 ± 11.4 104.2 0.0 0.0 17.3 ± 14.7 Pong 106.1 ± 8.1 111.9 18.0 0.5 85.2 ± 20.0 Seaquest 2.5 ± 0.4 1.7 0.4 0.7 2.1 ± 0.3 Table 1: Gamer-normalized scores for the 1% DQN-replay Atari dataset. We report the mean and variance across 3 seeds. Best mean scores are highlighted in bold. Decision Transformer (DT) performs comparably to CQL on 3 out of 4 games, and outperforms other baselines in most games. 4.2 OpenAI Gym In this section, we consider the continuous control tasks from the D4RL benchmark [23]. We also consider a 2D reacher environment that is not part of the benchmark, and generate the datasets using a similar methodology to the D4RL benchmark.


part of the benchmark, and generate the datasets using a similar methodology to the D4RL benchmark. Reacher is a goal-conditioned task and has sparse rewards, so it represents a different setting than the standard locomotion environments (HalfCheetah, Hopper, and Walker). The different dataset settings are described below. 1. Medium: 1 million timesteps generated by a “medium” policy that achieves approximately one-third the score of an expert policy. 2. Medium-Replay: the replay buffer of an agent trained to the performance of a medium policy (approximately 25k-400k timesteps in our environments). 3. Medium-Expert: 1 million timesteps generated by the medium policy concatenated with 1 million timesteps generated by an expert policy.


generated by the medium policy concatenated with 1 million timesteps generated by an expert policy. We compare to CQL [14], BEAR [18], BRAC [19], and AWR [24]. CQL represents the state-ofthe-art in model-free ofﬂine RL, an instantiation of TD learning with value pessimism. Score are normalized so that 100 represents an expert policy, as per Fu et al. [23]. CQL numbers are reported from the original paper; BC numbers are run by us; and the other methods are reported from the D4RL paper. Our results are shown in Table 2. Decision Transformer achieves the highest scores in a majority of the tasks and is competitive with the state of the art in the remaining tasks. Dataset Environment DT (Ours) CQL BEAR BRAC-v AWR BC Medium-Expert HalfCheetah 86.8 ± 1.3 62.4 53.4 41.9 52.7 59.9 Medium-Expert H


CQL BEAR BRAC-v AWR BC Medium-Expert HalfCheetah 86.8 ± 1.3 62.4 53.4 41.9 52.7 59.9 Medium-Expert Hopper 107.6 ± 1.8 111.0 96.3 0.8 27.1 79.6 Medium-Expert Walker 108.1 ± 0.2 98.7 40.1 81.6 53.8 36.6 Medium-Expert Reacher 89.1 ± 1.3 30.6 73.3 Medium HalfCheetah 42.6 ± 0.1 44.4 41.7 46.3 37.4 43.1 Medium Hopper 67.6 ± 1.0 58.0 52.1 31.1 35.9 63.9 Medium Walker 74.0 ± 1.4 79.2 59.1 81.1 17.4 77.3 Medium Reacher 51.2 ± 3.4 26.0 48.9 Medium-Replay HalfCheetah 36.6 ± 0.8 46.2 38.6 47.7 40.3 4.3 Medium-Replay Hopper 82.7 ± 7.0 48.6 33.7 0.6 28.4 27.6 Medium-Replay Walker 66.6 ± 3.0 26.7 19.2 0.9 15.5 36.9 Medium-Replay Reacher 18.0 ± 2.4 19.0 5.4 Average (Without Reacher) 74.7 63.9 48.2 36.9 34.3 46.4 Average (All Settings) 69.2 54.2 47.7 Table 2: Results for D4RL datasets3.


63.9 48.2 36.9 34.3 46.4 Average (All Settings) 69.2 54.2 47.7 Table 2: Results for D4RL datasets3. We report the mean and variance for three seeds. Decision Transformer (DT) outperforms conventional RL algorithms on almost all tasks. 3Given that CQL is generally the strongest TD learning method, for Reacher we only run the CQL baseline. 7


5 Discussion 5.1 Does Decision Transformer perform behavior cloning on a subset of the data? In this section, we seek to gain insight into whether Decision Transformer can be thought of as performing imitation learning on a subset of the data with a certain return. To investigate this, we propose a new method, Percentile Behavior Cloning (%BC), where we run behavior cloning on only the top X% of timesteps in the dataset, ordered by episode returns. The percentile X% interpolates between standard BC (X = 100%) that trains on the entire dataset and only cloning the best observed trajectory (X →0%), trading off between better generalization by training on more data with training a specialized model that focuses on a desirable subset of the data.


ning on more data with training a specialized model that focuses on a desirable subset of the data. We show full results comparing %BC to Decision Transformer and CQL in Table 3, sweeping over X ∈[10%, 25%, 40%, 100%]. Note that the only way to choose the optimal subset for cloning is to evaluate using rollouts from the environment, so %BC is not a realistic approach; rather, it serves to provide insight into the behavior of Decision Transformer. When data is plentiful – as in the D4RL regime – we ﬁnd %BC can match or beat other ofﬂine RL methods. On most environments, Decision Transformer is competitive with the performance of the best %BC, indicating it can hone in on a particular subset after training on the entire dataset distribution.


indicating it can hone in on a particular subset after training on the entire dataset distribution. Dataset Environment DT (Ours) 10%BC 25%BC 40%BC 100%BC CQL Medium HalfCheetah 42.6 ± 0.1 42.9 43.0 43.1 43.1 44.4 Medium Hopper 67.6 ± 1.0 65.9 65.2 65.3 63.9 58.0 Medium Walker 74.0 ± 1.4 78.8 80.9 78.8 77.3 79.2 Medium Reacher 51.2 ± 3.4 51.0 48.9 58.2 58.4 26.0 Medium-Replay HalfCheetah 36.6 ± 0.8 40.8 40.9 41.1 4.3 46.2 Medium-Replay Hopper 82.7 ± 7.0 70.6 58.6 31.0 27.6 48.6 Medium-Replay Walker 66.6 ± 3.0 70.4 67.8 67.2 36.9 26.7 Medium-Replay Reacher 18.0 ± 2.4 33.1 16.2 10.7 5.4 19.0 Average 56.1 56.7 52.7 49.4 39.5 43.5 Table 3: Comparison between Decision Transformer (DT) and Percentile Behavior Cloning (%BC).


5 43.5 Table 3: Comparison between Decision Transformer (DT) and Percentile Behavior Cloning (%BC). In contrast, when we study low data regimes – such as Atari, where we use 1% of a replay buffer as the dataset – %BC is weak (shown in Table 4). This suggests that in scenarios with relatively low amounts of data, Decision Transformer can outperform %BC by using all trajectories in the dataset to improve generalization, even if those trajectories are dissimilar from the return conditioning target. Our results indicate that Decision Transformer can be more effective than simply performing imitation learning on a subset of the dataset. On the tasks we considered, Decision Transformer either outperforms or is competitive to %BC, without the confound of having to select the optimal subset.


outperforms or is competitive to %BC, without the confound of having to select the optimal subset. Game DT (Ours) 10%BC 25%BC 40%BC 100%BC Breakout 267.5 ± 97.5 28.5 ± 8.2 73.5 ± 6.4 108.2 ± 67.5 138.9 ± 61.7 Qbert 15.4 ± 11.4 6.6 ± 1.7 16.0 ± 13.8 11.8 ± 5.8 17.3 ± 14.7 Pong 106.1 ± 8.1 2.5 ± 0.2 13.3 ± 2.7 72.7 ± 13.3 85.2 ± 20.0 Seaquest 2.5 ± 0.4 1.1 ± 0.2 1.1 ± 0.2 1.6 ± 0.4 2.1 ± 0.3 Table 4: %BC scores for Atari. We report the mean and variance across 3 seeds. Decision Transformer (DT) outperforms all versions of %BC in most games. 5.2 How well does Decision Transformer model the distribution of returns? We evaluate the ability of Decision Transformer to understand return-to-go tokens by varying the desired target return over a wide range – evaluating the multi-task distribution mo


s by varying the desired target return over a wide range – evaluating the multi-task distribution modeling capability of transformers. Figure 4 shows the average sampled return accumulated by the agent over the course of the evaluation episode for varying values of target return. On every task, the desired target returns and the true observed returns are highly correlated. On some tasks like Pong, HalfCheetah and Walker, Decision Transformer generates trajectories that almost perfectly match the desired returns 8


(as indicated by the overlap with the oracle line). Furthermore, on some Atari tasks like Seaquest, we can prompt the Decision Transformer with higher returns than the maximum episode return available in the dataset, demonstrating that Decision Transformer is sometimes capable of extrapolation. 0 100 200 300 0 100 200 300 Performance Breakout 0 20 40 60 80 100 0 50 100 Qbert 0 50 100 150 0 50 100 150 Pong 0 1 2 3 0 1 2 3 Seaquest 0 10 20 30 40 50 Target Return (Normalized) 0 20 40 Performance HalfCheetah 0 25 50 75 100 Target Return (Normalized) 0 50 100 Hopper 0 25 50 75 100 Target Return (Normalized) 0 50 100 Walker 0 5 10 15 20 25 Target Return (Normalized) 0 10 20 Reacher Decision Transformer Oracle Best Trajectory in Dataset Figure 4: Sampled (evaluation) returns accumulated by Decisi


ormer Oracle Best Trajectory in Dataset Figure 4: Sampled (evaluation) returns accumulated by Decision Transformer when conditioned on the speciﬁed target (desired) returns. Top: Atari. Bottom: D4RL medium-replay datasets. 5.3 What is the beneﬁt of using a longer context length? To assess the importance of access to previous states, actions, and returns, we ablate on the context length K. This is interesting since it is generally considered that the previous state (i.e. K = 1) is enough for reinforcement learning algorithms when frame stacking is used, as we do. Table 5 shows that performance of Decision Transformer is signiﬁcantly worse when K = 1, indicating that past information is useful for Atari games.


ormer is signiﬁcantly worse when K = 1, indicating that past information is useful for Atari games. One hypothesis is that when we are representing a distribution of policies – like with sequence modeling – the context allows the transformer to identify which policy generated the actions, enabling better learning and/or improving the training dynamics. Game DT (Ours) DT with no context (K = 1) Breakout 267.5 ± 97.5 73.9 ± 10 Qbert 15.1 ± 11.4 13.6 ± 11.3 Pong 106.1 ± 8.1 2.5 ± 0.2 Seaquest 2.5 ± 0.4 0.6 ± 0.1 Table 5: Ablation on context length. Decision Transformer (DT) performs better when using a longer context length (K = 50 for Pong, K = 30 for others). 5.4 Does Decision Transformer perform effective long-term credit assignment?


g, K = 30 for others). 5.4 Does Decision Transformer perform effective long-term credit assignment? To evaluate long-term credit assignment capabilities of our model, we consider a variant of the Key-to-Door environment proposed in Mesnard et al. [12]. This is a grid-based environment with a sequence of three phases: (1) in the ﬁrst phase, the agent is placed in a room with a key; (2) then, the agent is placed in an empty room; (3) and ﬁnally, the agent is placed in a room with a door. The agent receives a binary reward when reaching the door in the third phase, but only if it picked up the key in the ﬁrst phase. This problem is difﬁcult for credit assignment because credit must be propagated from the beginning to the end of the episode, skipping over actions taken in the middle.


propagated from the beginning to the end of the episode, skipping over actions taken in the middle. We train on datasets of trajectories generated by applying random actions and report success rates in Table 6. Furthermore, for the Key-to-Door environment we use the entire episode length as the context, rather than having a ﬁxed content window as in the other environments. Methods that use highsight return information: our Decision Transformer model and %BC (trained only on successful episodes) are able to learn effective policies – producing near-optimal paths, despite only training on random walks. TD learning (CQL) cannot effectively propagate Q-values over the long horizons involved and gets poor performance. 9


Dataset DT (Ours) CQL BC %BC Random 1K Random Trajectories 71.8% 13.1% 1.4% 69.9% 3.1% 10K Random Trajectories 94.6% 13.3% 1.6% 95.1% 3.1% Table 6: Success rate for Key-to-Door environment. Methods using hindsight (Decision Transformer, %BC) can learn successful policies, while TD learning struggles to perform credit assignment. 5.5 Can transformers be accurate critics in sparse reward settings? In previous sections, we established that decision transformer can produce effective policies (actors). We now evaluate whether transformer models can also be effective critics. We modify Decision Transformer to output return tokens in addition to action tokens on the Key-to-Door environment. Additionally, the ﬁrst return token is not given, but it is predicted instead (i.e.


or environment. Additionally, the ﬁrst return token is not given, but it is predicted instead (i.e. the model learns the initial distribution p( ˆR1)), similar to standard autoregressive generative models. We ﬁnd that the transformer continuously updates reward probability based on events during the episode, shown in Figure 5 (Left). Furthermore, we ﬁnd the transformer attends to critical events in the episode (picking up the key or reaching the door), shown in Figure 5 (Right), indicating formation of state-reward associations as discussed in Raposo et al. [25] and enabling accurate value prediction. key room distractor room door room Episode time 0.0 0.2 0.4 0.6 0.8 1.0 Predicted reward probability not pick up key pick up key and reach door pick up key and not reach door pick up key reac


obability not pick up key pick up key and reach door pick up key and not reach door pick up key reach door Episode time 0.0 0.2 0.4 0.6 0.8 1.0 Attention weight Figure 5: Left: Averages of running return probabilities predicted by the transformer model for three types of episode outcomes. Right: Transformer attention weights from all timesteps superimposed for a particular successful episode. The model attends to steps near pivotal events in the episode, such as picking up the key and reaching the door. 5.6 Does Decision Transformer perform well in sparse reward settings? A known weakness of TD learning algorithms is that they require densely populated rewards in order to perform well, which can be unrealistic and/or expensive.


uire densely populated rewards in order to perform well, which can be unrealistic and/or expensive. In contrast, Decision Transformer can improve robustness in these settings since it makes minimal assumptions on the density of the reward. To evaluate this, we consider a delayed return version of the D4RL benchmarks where the agent does not receive any rewards along the trajectory, and instead receives the cumulative reward of the trajectory in the ﬁnal timestep. Our results for delayed returns are shown in Table 7. Delayed returns minimally affect Decision Transformer; and due to the nature of the training process, while imitation learning methods are reward agnostic. While TD learning collapses, Decision Transformer and %BC still perform well, indicating that Decision Transformer can be


apses, Decision Transformer and %BC still perform well, indicating that Decision Transformer can be more robust to delayed rewards. Delayed (Sparse) Agnostic Original (Dense) Dataset Environment DT (Ours) CQL BC %BC DT (Ours) CQL Medium-Expert Hopper 107.3 ± 3.5 9.0 59.9 102.6 107.6 111.0 Medium Hopper 60.7 ± 4.5 5.2 63.9 65.9 67.6 58.0 Medium-Replay Hopper 78.5 ± 3.7 2.0 27.6 70.6 82.7 48.6 Table 7: Results for D4RL datasets with delayed (sparse) reward. Decision Transformer (DT) and imitation learning are minimally affected by the removal of dense rewards, while CQL fails. 10


5.7 Why does Decision Transformer avoid the need for value pessimism or behavior regularization? One key difference between Decision Transformer and prior ofﬂine RL algorithms is that we do not require policy regularization or conservatism to achieve good performance. Our conjecture is that TD-learning based algorithms learn an approximate value function and improve the policy by optimizing this value function. This act of optimizing a learned function can exacerbate and exploit any inaccuracies in the value function approximation, causing failures in policy improvement. Since Decision Transformer does not require explicit optimization using learned functions as objectives, it avoids the need for regularization or conservatism.


ation using learned functions as objectives, it avoids the need for regularization or conservatism. 5.8 How can Decision Transformer beneﬁt online RL regimes? Ofﬂine RL and the ability to model behaviors has the potential to enable sample-efﬁcient online RL for downstream tasks. Works studying the transition from ofﬂine to online generally ﬁnd that likelihood-based approaches, like our sequence modeling objective, are more successful [26, 27]. As a result, although we studied ofﬂine RL in this work, we believe Decision Transformer can meaningfully improve online RL methods by serving as a strong model for behavior generation. For instance, Decision Transformer can serve as a powerful “memorization engine” and in conjunction with powerful exploration algorithms like Go-Explore [28], has the


zation engine” and in conjunction with powerful exploration algorithms like Go-Explore [28], has the potential to simultaneously model and generative a diverse set of behaviors. 6 Related Work 6.1 Ofﬂine reinforcement learning To mitigate the impact of distribution shift in ofﬂine RL, prior algorithms either (a) constrain the policy action space [29, 30, 31] or (b) incorporate value pessimism [29, 14], or (c) incorporate pessimism into learned dynamics models [32, 33]. Since we do not use Decision Transformers to explicitly learn the dynamics model, we primarily compare against model-free algorithms in our work; in particular, adding a dynamics model tends to improve the performance of model-free algorithms.


; in particular, adding a dynamics model tends to improve the performance of model-free algorithms. Another line of work explores learning wide behavior distribution from an ofﬂine dataset by learning a task-agnostic set of skills, either with likelihood-based approaches [34, 35, 36, 37] or by maximizing mutual information [38, 39, 40]. Our work is similar to the likelihood-based approaches, which do not use iterative Bellman updates – although we use a simpler sequence modeling objective instead of a variational method, and use rewards for conditional generation of behaviors. 6.2 Supervised learning in reinforcement learning settings Some prior methods for reinforcement learning bear more resemblance to static supervised learning, such as Q-learning [41, 42], which still uses iterative ba


esemblance to static supervised learning, such as Q-learning [41, 42], which still uses iterative backups, or likelihood-based methods such as behavior cloning, which do not (discussed in previous section). Recent work [43, 44, 45] studies “upside-down” reinforcement learning (UDRL), which are similar to our method in seeking to model behaviors with a supervised loss conditioned on the target return. A key difference in our work is the shift of motivation to sequence modeling rather than supervised learning: while the practical methods differ primarily in the context length and architecture, sequence modeling enables behavior modeling even without access to the reward, in a similar style to language [9] or images [46], and is known to scale well [2]. The method proposed by Kumar et al.


to language [9] or images [46], and is known to scale well [2]. The method proposed by Kumar et al. [44] is most similar to our method with K = 1, which we ﬁnd sequence modeling/long contexts to outperform (see Section 5.3). Ghosh et al. [47] extends prior UDRL methods to use state goal conditioning, rather than rewards, and Paster et al. [48] further use an LSTM with state goal conditioning for goal-conditoned online RL settings. Concurrent to our work, Janner et al. [49] propose Trajectory Transformer, which is similar to Decision Transformer but additionally uses state and return prediction, as well as discretization, which incorporates model-based components. We believe that their experiments, in addition to our results, highlight the potential for sequence modeling to be a generally a


ments, in addition to our results, highlight the potential for sequence modeling to be a generally applicable idea for reinforcement learning. 11


6.3 Credit assignment Many works have studied better credit assignment via state-association, learning an architecture which decomposes the reward function such that certain “important” states comprise most of the credit [50, 51, 12]. They use the learned reward function to change the reward of an actor-critic algorithm to help propagate signal over long horizons. In particular, similar to our long-term setting, some works have speciﬁcally shown such state-associative architectures can perform better in delayed reward settings [52, 7, 53, 25]. In contrast, we allow these properties to naturally emerge in a transformer architecture, without having to explicitly learn a reward function or a critic.


ge in a transformer architecture, without having to explicitly learn a reward function or a critic. 6.4 Conditional language generation Various works have studied guided generation for images [54] and language [55, 56]. Several works [57, 58, 59, 60, 61, 62] have explored training or ﬁne-tuning of models for controllable text generation. Class-conditional language models can also be used to learn disciminators to guide generation [63, 55, 64, 65]. However, these approaches mostly assume constant “classes”, while in reinforcement learning the reward signal is time-varying. Furthermore, it is more natural to prompt the model desired target return and continuously decrease it by the observed rewards over time, since the transformer model and environment jointly generate the trajectory.


ved rewards over time, since the transformer model and environment jointly generate the trajectory. 6.5 Attention and transformer models Transformers [1] have been applied successfully to many tasks in natural language processing [66, 9] and computer vision [67, 68]. However, transformers are relatively unstudied in RL, mostly due to differing nature of the problem, such as higher variance in training. Zambaldi et al. [5] showed that augmenting transformers with relational reasoning improve performance in combinatorial environments and Ritter et al. [69] showed iterative self-attention allowed for RL agents to better utilize episodic memories. Parisotto et al. [4] discussed design decisions for more stable training of transformers in the high-variance RL setting.


iscussed design decisions for more stable training of transformers in the high-variance RL setting. Unlike our work, these still use actor-critic algorithms for optimization, focusing on novelty in architecture. Additionally, in imitation learning, some works have studied transformers as a replacement for LSTMs: Dasari and Gupta [70] study oneshot imitation learning, and Abramson et al. [71] combine language and image modalities for text-conditioned behavior generation. 7 Conclusion We proposed Decision Transformer, seeking to unify ideas in language/sequence modeling and reinforcement learning. On standard ofﬂine RL benchmarks, we showed Decision Transformer can match or outperform strong algorithms designed explicitly for ofﬂine RL with minimal modiﬁcations from standard language modelin


lgorithms designed explicitly for ofﬂine RL with minimal modiﬁcations from standard language modeling architectures. We hope this work inspires more investigation into using large transformer models for RL. We used a simple supervised loss that was effective in our experiments, but applications to large-scale datasets could beneﬁt from self-supervised pretraining tasks. In addition, one could consider more sophisticated embeddings for returns, states, and actions – for instance, conditioning on return distributions to model stochastic settings instead of deterministic returns. Transformer models can also be used to model the state evolution of trajectory, potentially serving as an alternative to model-based RL, and we hope to explore this in future work.


otentially serving as an alternative to model-based RL, and we hope to explore this in future work. For real-world applications, it is important to understand the types of errors transformers make in MDP settings and possible negative consequences, which are underexplored. It will also be important to consider the datasets we train models on, which can potentially add destructive biases, particularly as we consider studying augmenting RL agents with more data which may come from questionable sources. For instance, reward design by nefarious actors can potentially generate unintended behaviors as our model generates behaviors by conditioning on desired returns. 12


Acknowledgements This research was supported by Berkeley Deep Drive, Open Philanthropy, and the National Science Foundation under NSF:NRI #2024675. Part of this work was completed when Aravind Rajeswaran was a PhD student at the University of Washington, where he was supported by the J.P. Morgan PhD Fellowship in AI (2020-21). We also thank Luke Metz and Daniel Freeman for valuable feedback and discussions, as well as Justin Fu for assistance in setting up D4RL benchmarks, and Aviral Kumar for assistance with the CQL baselines and hyperparameters. 13


References [1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. [2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. [3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021. [4] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman,


Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, 2020. [5] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018. [6] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018. [7] Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne.


othy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature communications, 10(1):1–12, 2019. [8] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Ofﬂine reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. [9] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [10] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 47:253–279, 2013.


aluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 47:253–279, 2013. [11] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. [12] Thomas Mesnard, Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, et al. Counterfactual credit assignment in model-free reinforcement learning. arXiv preprint arXiv:2011.09464, 2020. [13] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on ofﬂine reinforcement learning. In International Conference on Machine Learning, 2020. [14] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.


erence on Machine Learning, 2020. [14] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for ofﬂine reinforcement learning. In Advances in Neural Information Processing Systems, 2020. [15] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [16] Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In ICML, 1990. [17] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pages 12498–12509, 2019. 14


[18] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949, 2019. [19] Yifan Wu, George Tucker, and Oﬁr Nachum. Behavior regularized ofﬂine reinforcement learning. arXiv preprint arXiv:1911.11361, 2019. [20] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. [21] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. [22] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos.


iv preprint arXiv:2010.02193, 2020. [22] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Conference on Artiﬁcial Intelligence, 2018. [23] Justin Fu, Aviral Kumar, Oﬁr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. [24] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. [25] David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425, 2021.


cis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425, 2021. [26] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313, 2018. [27] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with ofﬂine datasets. arXiv preprint arXiv:2006.09359, 2020. [28] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019. [29] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019.


reinforcement learning without exploration. In International Conference on Machine Learning, 2019. [30] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019. [31] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for ofﬂine reinforcement learning. In International Conference on Learning Representations, 2020. [32] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based ofﬂine reinforcement learning.


aran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based ofﬂine reinforcement learning. In Advances in Neural Information Processing Systems, 2020. [33] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based ofﬂine policy optimization. In Advances in Neural Information Processing Systems, 2020. [34] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Oﬁr Nachum. Opal: Ofﬂine primitive discovery for accelerating ofﬂine reinforcement learning. arXiv preprint arXiv:2010.13611, 2020. [35] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills.


to, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, 2020. [36] Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.11944, 2020. 15


[37] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021. [38] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019. [39] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill-space planning. arXiv preprint arXiv:2012.03548, 2020. [40] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamicsaware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.


re unsupervised discovery of skills. In International Conference on Learning Representations, 2020. [41] Christopher Watkins. Learning from delayed rewards. 01 1989. [42] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [43] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Ja´skowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877, 2019. [44] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465, 2019. [45] Acting without rewards. 2019. URL https://ogma.ai/2019/08/ acting-without-rewards/.


465, 2019. [45] Acting without rewards. 2019. URL https://ogma.ai/2019/08/ acting-without-rewards/. [46] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691–1703. PMLR, 2020. [47] Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. arXiv preprint arXiv:1912.06088, 2019. [48] Keiran Paster, Sheila A McIlraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419, 2020. [49] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem.


Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106.02039, 2021. [50] Johan Ferret, Raphaël Marinier, Matthieu Geist, and Olivier Pietquin. Self-attentional credit assignment for transfer in reinforcement learning. arXiv preprint arXiv:1907.08027, 2019. [51] Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Azar, Bilal Piot, Nicolas Heess, Hado van Hasselt, Greg Wayne, Satinder Singh, Doina Precup, et al. Hindsight credit assignment. arXiv preprint arXiv:1912.02503, 2019. [52] Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. arXiv preprint arXiv:1806.07857, 2018.


ochreiter. Rudder: Return decomposition for delayed rewards. arXiv preprint arXiv:1806.07857, 2018. [53] Yang Liu, Yunan Luo, Yuanyi Zhong, Xi Chen, Qiang Liu, and Jian Peng. Sequence modeling of temporal credit assignment for episodic reinforcement learning. arXiv preprint arXiv:1905.13420, 2019. [54] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition, 2019. [55] Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACL, System Demonstrations, 2017. 16


[56] Lilian Weng. Controllable neural text generation. lilianweng.github.io/lillog, 2021. URL https://lilianweng.github.io/lil-log/2021/01/02/ controllable-neural-text-generation.html. [57] Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02633, 2017. [58] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In International Conference on Machine Learning, 2017. [59] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361, 2019. [60] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.


soning. arXiv preprint arXiv:1906.02361, 2019. [60] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI conference on artiﬁcial intelligence, 2017. [61] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [62] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019. [63] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.


, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164, 2019. [64] Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.06087, 2018. [65] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shaﬁq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367, 2020. [66] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.


istina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [67] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020. [68] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. [69] Sam Ritter, Ryan Faulkner, Laurent Sartran, Adam Santoro, Matt Botvinick, and David Raposo. Rapid task-solving in novel environments.


Sartran, Adam Santoro, Matt Botvinick, and David Raposo. Rapid task-solving in novel environments. arXiv preprint arXiv:2006.03662, 2020. [70] Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. arXiv preprint arXiv:2011.05970, 2020. [71] Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020. [72] Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-ofthe-art natural language processing.


owicz, Joe Davison, Sam Shleifer, et al. Transformers: State-ofthe-art natural language processing. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020. [73] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 17


A Experimental Details Code for experiments can be found in the supplementary material. A.1 Atari We build our Decision Transformer implementation for Atari games off of minGPT (https:// github.com/karpathy/minGPT), a publicly available re-implementation of GPT. We use most of the default hyperparameters from their character-level GPT example (https://github.com/ karpathy/minGPT/blob/master/play_char.ipynb). We reduce the batch size (except in Pong), block size, number of layers, attention heads, and embedding dimension for faster training. For processing the observations, we use the DQN encoder from Mnih et al. [20] with an additional linear layer to project to the embedding dimension. For return-to-go conditioning, we use either 1× or 5× the maximum return in the dataset, but more possib


return-to-go conditioning, we use either 1× or 5× the maximum return in the dataset, but more possibilities exist for principled return-to-go conditioning. In Atari experiments, we use Tanh instead of LayerNorm (as described in Section 3) after embedding each modality, but did this does not make a signiﬁcant difference in performance. The full list of hyperparameters can be found in Table 8. Table 8: Hyperparameters of DT (and %BC) for Atari experiments. Hyperparameter Value Number of layers 6 Number of attention heads 8 Embedding dimension 128 Batch size 512 Pong 128 Breakout, Qbert, Seaquest Context length K 50 Pong 30 Breakout, Qbert, Seaquest Return-to-go conditioning 90 Breakout (≈1× max in dataset) 2500 Qbert (≈5× max in dataset) 20 Pong (≈1× max in dataset) 1450 Seaquest (≈5× max in


x in dataset) 2500 Qbert (≈5× max in dataset) 20 Pong (≈1× max in dataset) 1450 Seaquest (≈5× max in dataset) Nonlinearity ReLU, encoder GeLU, otherwise Encoder channels 32, 64, 64 Encoder ﬁlter sizes 8 × 8, 4 × 4, 3 × 3 Encoder strides 4, 2, 1 Max epochs 5 Dropout 0.1 Learning rate 6 ∗10−4 Adam betas (0.9, 0.95) Grad norm clip 1.0 Weight decay 0.1 Learning rate decay Linear warmup and cosine decay (see code for details) Warmup tokens 512 ∗20 Final tokens 2 ∗500000 ∗K A.2 OpenAI Gym A.2.1 Decision Transformer Our code is based on the Huggingface Transformers library [72]. Our hyperparameters on all OpenAI Gym tasks are shown below in Table 9. Heuristically, we ﬁnd using larger models helps to model the distribution of returns, compared to standard RL model sizes (which learn one policy).


to model the distribution of returns, compared to standard RL model sizes (which learn one policy). For reacher we use a smaller context length than the other environments, which we ﬁnd to be helpful as the environment is goal-conditioned and the episodes are shorter. We choose return targets based on expert performance for each environment, except for HalfCheetah where we ﬁnd 50% performance to be better due to the datasets containing lower relative returns to the other environments. Models were trained for 105 gradient steps using the AdamW optimizer [73] following PyTorch defaults. 18


Table 9: Hyperparameters of Decision Transformer for OpenAI Gym experiments. Hyperparameter Value Number of layers 3 Number of attention heads 1 Embedding dimension 128 Nonlinearity function ReLU Batch size 64 Context length K 20 HalfCheetah, Hopper, Walker 5 Reacher Return-to-go conditioning 6000 HalfCheetah 3600 Hopper 5000 Walker 50 Reacher Dropout 0.1 Learning rate 10−4 Grad norm clip 0.25 Weight decay 10−4 Learning rate decay Linear warmup for ﬁrst 105 training steps A.2.2 Behavior Cloning As brieﬂy mentioned in Section 4.2, we found previously reported behavior cloning baselines to be weak, and so run them ourselves using a similar setup as Decision Transformer. We tried using a transformer architecture, but found using an MLP (as in previous work) to be stronger.


ried using a transformer architecture, but found using an MLP (as in previous work) to be stronger. We train for 2.5 × 104 gradient steps; training more did not improve performance. Other hyperparameters are shown in Table 10. The percentile behavior cloning experiments use the same hyperparameters. Table 10: Hyperparameters of Behavior Cloning for OpenAI Gym experiments. Hyperparameter Value Number of layers 3 Embedding dimension 256 Nonlinearity function ReLU Batch size 64 Dropout 0.1 Learning rate 10−4 Weight decay 10−4 Learning rate decay Linear warmup for ﬁrst 105 training steps A.3 Graph Shortest Path We give details of the illustrative example discussed in the introduction. The task is to ﬁnd the shortest path on a ﬁxed directed graph, which can be formulated as an MDP where reward


s to ﬁnd the shortest path on a ﬁxed directed graph, which can be formulated as an MDP where reward is 0 when the agent is at the goal node and −1 otherwise. The observation is the integer index of the graph node the agent is in. The action is the integer index of the graph node to move to next. The transition dynamics transport the agent to the action’s node index if there is an edge in the graph, while the agent remains at the past node otherwise. The returns-to-go in this problem correspond to negative path lengths and maximizing them corresponds to generating shortest paths. In this environment, we use the GPT model as described in Section 3 to generate both actions and return-to-go tokens. This makes it possible for the model it generate its own (realizable) returns-to-go ˆR.


-go tokens. This makes it possible for the model it generate its own (realizable) returns-to-go ˆR. Since we require a return prompt to generate actions and we do assume knowledge of the optimal path length upfront, we use a simple prior over returns that favors shorter paths: Pprior( ˆR = k) ∝T + 1 −k, where T is the maximum trajectory length. Then, it is combined with the return probabilities generated by the GPT model: P( ˆRt|s0:t, a0:t−1, ˆR0:t−1) = 19


1 2 3 4 5 6 7 8 9 # of steps to goal 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 proportion of paths shortest path transformer random walk Figure 6: Histogram of steps to reach the goal node for random walks on the graph, shortest possible paths to the goal, and attempted shortest paths generated by the transformer model. ∞indicates the goal was not reached during the trajectory. PGPT( ˆRt|s0:t, a0:t−1, ˆR0:t−1) × Pprior( ˆRt)10. Note that the prior and return-to-go predictions are entirely computable by the model, and thus avoids the need for any external or oracle information like the optimal path length. Adjustment of generation by a prior has also been used for similar purposes in controllable text generation in prior work [65].


a prior has also been used for similar purposes in controllable text generation in prior work [65]. We train on a dataset of 1, 000 graph random walk trajectories of T = 10 steps each with a random graph of 20 nodes and edge sparsity coefﬁcient of 0.1. We report the results in Figure 6, where we ﬁnd that transformer model is able to signiﬁcantly improve upon the number of steps required to reach the goal, closely matching performance of optimal paths. There are two reasons for the favorable performance on this task. In one case, the training dataset of random walk trajectories may contain a segment that directly corresponds to the desired shortest path, in which case it will be generated by the model.


directly corresponds to the desired shortest path, in which case it will be generated by the model. In the second case, generated paths are entirely original and are not subsets of trajectories in the training dataset - they are generated from stitching sub-optimal segments. We ﬁnd this case accounts for 15.8% of generated paths in the experiment. While this is a simple example and uses a prior on generation that we do not use in other experiments for simplicity, it illustrates how hindsight return information can be used with generation priors to avoid the need for explicit dynamic programming. B Atari Task Scores Table 11 shows the normalized scores used for normalization used in Hafner et al. [21]. Tables 12 and 13 show the raw scores corresponding to Tables 1 and 4, respectively.


er et al. [21]. Tables 12 and 13 show the raw scores corresponding to Tables 1 and 4, respectively. For %BC scores, we use the same hyperparameters as Decision Transformer for fair comparison. For REM and QR-DQN, there is a slight discrepancy between Agarwal et al. [13] and Kumar et al. [14]; we report raw data provided to us by REM authors. Game Random Gamer Breakout 2 30 Qbert 164 13455 Pong −21 15 Seaquest 68 42055 Table 11: Atari baseline scores used for normalization. 20


Game DT (Ours) CQL QR-DQN REM BC Breakout 76.9 ± 27.3 61.1 6.8 4.5 40.9 ± 17.3 Qbert 2215.8 ± 1523.7 14012.0 156.0 160.1 2464.1 ± 1948.2 Pong 17.1 ± 2.9 19.3 −14.5 −20.8 9.7 ± 7.2 Seaquest 1129.3 ± 189.0 779.4 250.1 370.5 968.6 ± 133.8 Table 12: Raw scores for the 1% DQN-replay Atari dataset. We report the mean and variance across 3 seeds. Best mean scores are highlighted in bold. Decision Transformer performs comparably to CQL on 3 out of 4 games, and usually outperforms other baselines. Game DT (Ours) 10%BC 25%BC 40%BC 100%BC Breakout 76.9 ± 27.3 10.0 ± 2.3 22.6 ± 1.8 32.3 ± 18.9 40.9 ± 17.3 Qbert 2215.8 ± 1523.7 1045 ± 232.0 2302.5 ± 1844.1 1674.1 ± 776.0 2464.1 ± 1948.2 Pong 17.1 ± 2.9 −20.3 ± 0.1 −16.2 ± 1.0 5.2 ± 4.8 9.7 ± 7.2 Seaquest 1129.3 ± 189.0 521.3 ± 103.0 549.3 ± 96.2 758 ±


20.3 ± 0.1 −16.2 ± 1.0 5.2 ± 4.8 9.7 ± 7.2 Seaquest 1129.3 ± 189.0 521.3 ± 103.0 549.3 ± 96.2 758 ± 169.1 968.6 ± 133.8 Table 13: %BC scores for Atari. We report the mean and variance across 3 seeds. Decision Transformer usually outperforms %BC. 21


Training language models to follow instructions with human feedback Long Ouyang∗ Jeff Wu∗ Xu Jiang∗ Diogo Almeida∗ Carroll L. Wainwright∗ Pamela Mishkin∗ Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell† Peter Welinder Paul Christiano∗† Jan Leike∗ Ryan Lowe∗ OpenAI Abstract Making language models bigger does not inherently make them better at following a user’s intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by ﬁne-tuning with human feedback.


igning language models with user intent on a wide range of tasks by ﬁne-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to ﬁne-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further ﬁne-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.


structGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that ﬁne-tuning with human feedback is a promising direction for aligning language models with human intent. 1 Introduction Large language models (LMs) can be “prompted” to perform a range of natural language processing (NLP) tasks, given some examples of the task as input. However, these models often express unintended behaviors such as making up facts, generating biased or toxic text, or simply not following user instructions (Bender et al., 2021; Bo


generating biased or toxic text, or simply not following user instructions (Bender et al., 2021; Bommasani et al., 2021; Kenton et al., 2021; Weidinger et al., 2021; Tamkin et al., 2021; Gehman et al., 2020). This is because the language modeling objective ∗Primary authors. This was a joint project of the OpenAI Alignment team. RL and JL are the team leads. Corresponding author: lowe@openai.com. †Work done while at OpenAI. Current afﬁliations: AA: Anthropic; PC: Alignment Research Center. arXiv:2203.02155v1 [cs.CL] 4 Mar 2022


1.3B 6B 175B Model size 0.2 0.4 0.6 Win rate against SFT 175B Model PPO-ptx PPO SFT GPT (prompted) GPT Figure 1: Human evaluations of various models on our API prompt distribution, evaluated by how often outputs from each model were preferred to those from the 175B SFT model. Our InstructGPT models (PPO-ptx) as well as its variant trained without pretraining mix (PPO) signiﬁcantly outperform the GPT-3 baselines (GPT, GPT prompted); outputs from our 1.3B PPO-ptx model are preferred to those from the 175B GPT-3. Error bars throughout the paper are 95% conﬁdence intervals. used for many recent large LMs—predicting the next token on a webpage from the internet—is different from the objective “follow the user’s instructions helpfully and safely” (Radford et al., 2019; Brown et al., 2020; Fedus


llow the user’s instructions helpfully and safely” (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022). Thus, we say that the language modeling objective is misaligned. Averting these unintended behaviors is especially important for language models that are deployed and used in hundreds of applications. We make progress on aligning language models by training them to act in accordance with the user’s intention (Leike et al., 2018). This encompasses both explicit intentions such as following instructions and implicit intentions such as staying truthful, and not being biased, toxic, or otherwise harmful. Using the language of Askell et al. (2021), we want language models to be helpful (they should help the user solve their task), honest (t


2021), we want language models to be helpful (they should help the user solve their task), honest (they shouldn’t fabricate information or mislead the user), and harmless (they should not cause physical, psychological, or social harm to people or the environment). We elaborate on the evaluation of these criteria in Section 3.6. We focus on ﬁne-tuning approaches to aligning language models. Speciﬁcally, we use reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Stiennon et al., 2020) to ﬁne-tune GPT-3 to follow a broad class of written instructions (see Figure 2). This technique uses human preferences as a reward signal to ﬁne-tune our models. We ﬁrst hire a team of 40 contractors to label our data, based on their performance on a screening test (see Section 3.4 and A


contractors to label our data, based on their performance on a screening test (see Section 3.4 and Appendix B.1 for more details). We then collect a dataset of human-written demonstrations of the desired output behavior on (mostly English) prompts submitted to the OpenAI API3 and some labeler-written prompts, and use this to train our supervised learning baselines. Next, we collect a dataset of human-labeled comparisons between outputs from our models on a larger set of API prompts. We then train a reward model (RM) on this dataset to predict which model output our labelers would prefer. Finally, we use this RM as a reward function and ﬁne-tune our supervised learning baseline to maximize this reward using the PPO algorithm (Schulman et al., 2017). We illustrate this process in Figure 2.


his reward using the PPO algorithm (Schulman et al., 2017). We illustrate this process in Figure 2. This procedure aligns the behavior of GPT-3 to the stated preferences of a speciﬁc group of people (mostly our labelers and researchers), rather than any broader notion of “human values”; we discuss this further in Section 5.2. We call the resulting models InstructGPT. We mainly evaluate our models by having our labelers rate the quality of model outputs on our test set, consisting of prompts from held-out customers (who are not represented in the training data). We also conduct automatic evaluations on a range of public NLP datasets. We train three model 3Speciﬁcally, we train on prompts submitted to earlier versions of the InstructGPT models on the OpenAI API Playground, which were trained


itted to earlier versions of the InstructGPT models on the OpenAI API Playground, which were trained only using demonstration data. We ﬁlter out prompts containing PII. 2


Figure 2: A diagram illustrating the three steps of our method: (1) supervised ﬁne-tuning (SFT), (2) reward model (RM) training, and (3) reinforcement learning via proximal policy optimization (PPO) on this reward model. Blue arrows indicate that this data is used to train one of our models. In Step 2, boxes A-D are samples from our models that get ranked by labelers. See Section 3 for more details on our method. sizes (1.3B, 6B, and 175B parameters), and all of our models use the GPT-3 architecture. Our main ﬁndings are as follows: Labelers signiﬁcantly prefer InstructGPT outputs over outputs from GPT-3. On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having over 100x fewer parameters.


tGPT model are preferred to outputs from the 175B GPT-3, despite having over 100x fewer parameters. These models have the same architecture, and differ only by the fact that InstructGPT is ﬁne-tuned on our human data. This result holds true even when we add a few-shot prompt to GPT-3 to make it better at following instructions. Outputs from our 175B InstructGPT are preferred to 175B GPT-3 outputs 85 ± 3% of the time, and preferred 71 ± 4% of the time to few-shot 175B GPT-3. InstructGPT models also generate more appropriate outputs according to our labelers, and more reliably follow explicit constraints in the instruction. InstructGPT models show improvements in truthfulness over GPT-3. On the TruthfulQA benchmark, InstructGPT generates truthful and informative answers about twice as often


e TruthfulQA benchmark, InstructGPT generates truthful and informative answers about twice as often as GPT-3. Our results are equally strong on the subset of questions that were not adversarially selected against GPT-3. On “closed-domain” tasks from our API prompt distribution, where the output should not contain information that is not present in the input (e.g. summarization and closed-domain QA), InstructGPT models make up information not present in the input about half as often as GPT-3 (a 21% vs. 41% hallucination rate, respectively). InstructGPT shows small improvements in toxicity over GPT-3, but not bias. To measure toxicity, we use the RealToxicityPrompts dataset (Gehman et al., 2020) and conduct both automatic and human evaluations.


RealToxicityPrompts dataset (Gehman et al., 2020) and conduct both automatic and human evaluations. InstructGPT models generate about 25% fewer toxic outputs than GPT-3 when prompted to be respectful. InstructGPT does not signiﬁcantly improve over GPT-3 on the Winogender (Rudinger et al., 2018) and CrowSPairs (Nangia et al., 2020) datasets. We can minimize performance regressions on public NLP datasets by modifying our RLHF ﬁne-tuning procedure. During RLHF ﬁne-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).


HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015). This is an example of an “alignment tax” since our alignment procedure comes at the cost of 3


lower performance on certain tasks that we may care about. We can greatly reduce the performance regressions on these datasets by mixing PPO updates with updates that increase the log likelihood of the pretraining distribution (PPO-ptx), without compromising labeler preference scores. Our models generalize to the preferences of “held-out” labelers that did not produce any training data. To test the generalization of our models, we conduct a preliminary experiment with held-out labelers, and ﬁnd that they prefer InstructGPT outputs to outputs from GPT-3 at about the same rate as our training labelers. However, more work is needed to study how these models perform on broader groups of users, and how they perform on inputs where humans disagree about the desired behavior.


r groups of users, and how they perform on inputs where humans disagree about the desired behavior. Public NLP datasets are not reﬂective of how our language models are used. We compare GPT-3 ﬁne-tuned on our human preference data (i.e. InstructGPT) to GPT-3 ﬁne-tuned on two different compilations of public NLP tasks: the FLAN (Wei et al., 2021) and T0 (Sanh et al., 2021) (in particular, the T0++ variant). These datasets consist of a variety of NLP tasks, combined with natural language instructions for each task. On our API prompt distribution, our FLAN and T0 models perform slightly worse than our SFT baseline, and labelers signiﬁcantly prefer InstructGPT to these models (InstructGPT has a 73.4 ±2% winrate vs.


nd labelers signiﬁcantly prefer InstructGPT to these models (InstructGPT has a 73.4 ±2% winrate vs. our baseline, compared to 26.8 ±2% and 29.8 ±2% for our version of T0 and FLAN, respectively). InstructGPT models show promising generalization to instructions outside of the RLHF ﬁnetuning distribution. We qualitatively probe InstructGPT’s capabilities, and ﬁnd that it is able to follow instructions for summarizing code, answer questions about code, and sometimes follows instructions in different languages, despite these instructions being very rare in the ﬁne-tuning distribution. In contrast, GPT-3 can perform these tasks but requires more careful prompting, and does not usually follow instructions in these domains.


sks but requires more careful prompting, and does not usually follow instructions in these domains. This result is exciting because it suggests that our models are able to generalize the notion of “following instructions.” They retain some alignment even on tasks for which they get very little direct supervision signal. InstructGPT still makes simple mistakes. For example, InstructGPT can still fail to follow instructions, make up facts, give long hedging answers to simple questions, or fail to detect instructions with false premises. Overall, our results indicate that ﬁne-tuning large language models using human preferences signiﬁcantly improves their behavior on a wide range of tasks, though much work remains to be done to improve their safety and reliability.


a wide range of tasks, though much work remains to be done to improve their safety and reliability. The rest of this paper is structured as follows: We ﬁrst detail related work in Section 2, before diving into our method and experiment details in Section 3, including our high-level methodology (3.1), task and dataset details (3.3 and 3.2), human data collection (3.4), how we trained our models (3.5), and our evaluation procedure (3.6). We then present our results in Section 4, divided into three parts: results on the API prompt distribution (4.1), results on public NLP datasets (4.2), and qualitative results (4.3). Finally we give an extended discussion of our work in Section 5, including implications for alignment research (5.1), what we are aligning to (5.2), limitations (5.3), open ques


plications for alignment research (5.1), what we are aligning to (5.2), limitations (5.3), open questions (5.4), and broader impacts of this work (5.5). 2 Related work Research on alignment and learning from human feedback. We build on previous techniques to align models with human intentions, particularly reinforcement learning from human feedback (RLHF). Originally developed for training simple robots in simulated environments and Atari games (Christiano et al., 2017; Ibarz et al., 2018), it has recently been applied to ﬁne-tuning language models to summarize text (Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Wu et al., 2021). This work is in turn inﬂuenced by similar work using human feedback as a reward in domains such as dialogue (Jaques et al., 2019; Yi et al., 201


rk using human feedback as a reward in domains such as dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer et al., 2018; Bahdanau et al., 2016), semantic parsing (Lawrence and Riezler, 2018), story generation (Zhou and Xu, 2020), review generation (Cho et al., 2018), and evidence extraction (Perez et al., 2019). Madaan et al. (2022) use written human feedback to augment prompts and improve the performance of GPT-3. There has also been work on aligning agents in text-based environments using RL with 4


a normative prior (Nahian et al., 2021). Our work can be seen as a direct application of RLHF to aligning language models on a broad distribution of language tasks. The question of what it means for language models to be aligned has also received attention recently (Gabriel, 2020). Kenton et al. (2021) catalog behavioral issues in LMs that result from misalignment, including producing harmful content and gaming misspeciﬁed objectives. In concurrent work, Askell et al. (2021) propose language assistants as a testbed for alignment research, study some simple baselines, and their scaling properties. Training language models to follow instructions. Our work is also related to research on crosstask generalization in language models, where LMs are ﬁne-tuned on a broad range of public NLP dataset


sk generalization in language models, where LMs are ﬁne-tuned on a broad range of public NLP datasets (usually preﬁxed with an appropriate instruction) and evaluated on a different set of NLP tasks. There has been a range of work in this domain (Yi et al., 2019; Mishra et al., 2021; Wei et al., 2021; Khashabi et al., 2020; Sanh et al., 2021; Aribandi et al., 2021), which differ in training and evaluation data, formatting of instructions, size of pretrained models, and other experimental details. A consistent ﬁnding across studies is that ﬁne-tuning LMs on a range of NLP tasks, with instructions, improves their downstream performance on held-out tasks, both in the zero-shot and few-shot settings.


proves their downstream performance on held-out tasks, both in the zero-shot and few-shot settings. There is also a related line of work on instruction following for navigation, where models are trained to follow natural language instructions to navigate in a simulated environment (Bahdanau et al., 2018; Abramson et al., 2020; Zhao et al., 2021). Evaluating the harms of language models. A goal of modifying the behavior of language models is to mitigate the harms of these models when they’re deployed in the real world. These risks have been extensively documented (Bender et al., 2021; Bommasani et al., 2021; Kenton et al., 2021; Weidinger et al., 2021; Tamkin et al., 2021). Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et


n produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously; for a thorough review we direct the reader to Weidinger et al. (2021). Deploying language models in speciﬁc domains gives rise to new risks and challenges, for example in dialog systems (Henderson et al., 2018; Xu et al., 2020; Dinan et al., 2019b). There is a nascent but growing ﬁeld that aims to build benchmarks to concretely evaluate these harms, particularly around toxicity (Gehman et al., 2020), stereotypes (Nadeem et al., 2020), and social bias (Dhamala et al., 2021; Nangia et al., 2020; Rudinger et al., 2018).


et al., 2020), and social bias (Dhamala et al., 2021; Nangia et al., 2020; Rudinger et al., 2018). Making signiﬁcant progress on these problems is hard since well-intentioned interventions on LM behavior can have side-effects (Welbl et al., 2021; Blodgett et al., 2020); for instance, efforts to reduce the toxicity of LMs can reduce their ability to model text from under-represented groups, due to prejudicial correlations in the training data (Xu et al., 2021). Modifying the behavior of language models to mitigate harms. There are many ways to change the generation behavior of language models. Solaiman and Dennison (2021) ﬁne-tune LMs on a small, value-targeted dataset, which improves the models’ ability to adhere to these values on a question answering task. Ngo et al.


ich improves the models’ ability to adhere to these values on a question answering task. Ngo et al. (2021) ﬁlter the pretraining dataset by removing documents on which a language model has a high conditional likelihood of generating a set of researcher-written trigger phrases. When trained on this ﬁltered dataset, their LMs generate less harmful text, at the cost of a slight decrease in language modeling performance. Xu et al. (2020) use a variety of approaches to improve the safety of chatbots, including data ﬁltering, blocking certain words or n-grams during generation, safety-speciﬁc control tokens (Keskar et al., 2019; Dinan et al., 2019a), and human-in-theloop data collection (Dinan et al., 2019b).


kar et al., 2019; Dinan et al., 2019a), and human-in-theloop data collection (Dinan et al., 2019b). Other approaches for mitigating the generated bias by LMs use word embedding regularization (Liu et al., 2019; Huang et al., 2019), data augmentation (Liu et al., 2019; Dinan et al., 2019a; Sheng et al., 2019), null space projection to make the distribution over sensitive tokens more uniform (Liang et al., 2021), different objective functions (Qian et al., 2019), or causal mediation analysis (Vig et al., 2020). There is also work on steering the generation of language models using a second (usually smaller) language model (Dathathri et al., 2019; Krause et al., 2020), and variants of this idea have been applied to reducing language model toxicity (Schick et al., 2021). 5


Table 1: Distribution of use case categories from our API prompt dataset. Use-case (%) Generation 45.6% Open QA 12.4% Brainstorming 11.2% Chat 8.4% Rewrite 6.6% Summarization 4.2% Classiﬁcation 3.5% Other 3.5% Closed QA 2.6% Extract 1.9% Table 2: Illustrative prompts from our API prompt dataset. These are ﬁctional examples inspired by real usage—see more examples in Appendix A.2.1. Use-case Prompt Brainstorming List ﬁve ideas for how to regain enthusiasm for my career Generation Write a short story where a bear goes to the beach, makes friends with a seal, and then returns home. Rewrite This is the summary of a Broadway play: """ {summary} """ This is the outline of the commercial for that play: """ 3 Methods and experimental details 3.1 High-level methodology Our methodology follows that


lay: """ 3 Methods and experimental details 3.1 High-level methodology Our methodology follows that of Ziegler et al. (2019) and Stiennon et al. (2020), who applied it in the stylistic continuation and summarization domains. We start with a pretrained language model (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022), a distribution of prompts on which we want our model to produce aligned outputs, and a team of trained human labelers (see Sections 3.4 for details). We then apply the following three steps (Figure 2). Step 1: Collect demonstration data, and train a supervised policy. Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution).


sired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then ﬁne-tune a pretrained GPT-3 model on this data using supervised learning. Step 2: Collect comparison data, and train a reward model. We collect a dataset of comparisons between model outputs, where labelers indicate which output they prefer for a given input. We then train a reward model to predict the human-preferred output. Step 3: Optimize a policy against the reward model using PPO. We use the output of the RM as a scalar reward. We ﬁne-tune the supervised policy to optimize this reward using the PPO algorithm (Schulman et al., 2017). Steps 2 and 3 can be iterated continuously; more comparison data is collected on the current best policy, which is used to train a new RM and then


e comparison data is collected on the current best policy, which is used to train a new RM and then a new policy. In practice, most of our comparison data comes from our supervised policies, with some coming from our PPO policies. 3.2 Dataset Our prompt dataset consists primarily of text prompts submitted to the OpenAI API, speciﬁcally those using an earlier version of the InstructGPT models (trained via supervised learning on a subset of our demonstration data) on the Playground interface.4 Customers using the Playground were informed that their data could be used to train further models via a recurring notiﬁcation any time InstructGPT models were used. In this paper we do not use data from customers using the API in production.


tGPT models were used. In this paper we do not use data from customers using the API in production. We heuristically deduplicate prompts by checking for prompts that share a long common preﬁx, and we limit the number of prompts to 200 per user ID. We also create our train, validation, and test splits based on user ID, so that the validation and test sets contain no data from users whose data is in the training set. To avoid the models learning potentially sensitive customer details, we ﬁlter all prompts in the training split for personally identiﬁable information (PII). 4This is an interface hosted by OpenAI to interact directly with models on our API; see https://beta. openai.com/playground. 6


To train the very ﬁrst InstructGPT models, we asked labelers to write prompts themselves. This is because we needed an initial source of instruction-like prompts to bootstrap the process, and these kinds of prompts weren’t often submitted to the regular GPT-3 models on the API. We asked labelers to write three kinds of prompts: • Plain: We simply ask the labelers to come up with an arbitrary task, while ensuring the tasks had sufﬁcient diversity. • Few-shot: We ask the labelers to come up with an instruction, and multiple query/response pairs for that instruction. • User-based: We had a number of use-cases stated in waitlist applications to the OpenAI API. We asked labelers to come up with prompts corresponding to these use cases.


ions to the OpenAI API. We asked labelers to come up with prompts corresponding to these use cases. From these prompts, we produce three different datasets used in our ﬁne-tuning procedure: (1) our SFT dataset, with labeler demonstrations used to train our SFT models, (2) our RM dataset, with labeler rankings of model outputs used to train our RMs, and (3) our PPO dataset, without any human labels, which are used as inputs for RLHF ﬁne-tuning. The SFT dataset contains about 13k training prompts (from the API and labeler-written), the RM dataset has 33k training prompts (from the API and labeler-written), and the PPO dataset has 31k training prompts (only from the API). More details on dataset sizes are provided in Table 6.


as 31k training prompts (only from the API). More details on dataset sizes are provided in Table 6. To give a sense of the composition of our dataset, in Table 1 we show the distribution of use-case categories for our API prompts (speciﬁcally the RM dataset) as labeled by our contractors. Most of the use-cases have are generative, rather than classiﬁcation or QA. We also show some illustrative prompts (written by researchers to mimic the kinds of prompts submitted to InstructGPT models) in Table 2; more prompts submitted to InstructGPT models are shown in Appendix A.2.1, and prompts submitted to GPT-3 models are shown in Appendix A.2.2. We provide more details about our dataset in Appendix A.


GPT-3 models are shown in Appendix A.2.2. We provide more details about our dataset in Appendix A. 3.3 Tasks Our training tasks are from two sources: (1) a dataset of prompts written by our labelers and (2) a dataset of prompts submitted to early InstructGPT models on our API (see Table 6). These prompts are very diverse and include generation, question answering, dialog, summarization, extractions, and other natural language tasks (see Table 1). Our dataset is over 96% English, however in Section 4.3 we also probe our model’s ability to respond to instructions in other languages and complete coding tasks. For each natural language prompt, the task is most often speciﬁed directly through a natural language instruction (e.g.


guage prompt, the task is most often speciﬁed directly through a natural language instruction (e.g. “Write a story about a wise frog”), but could also be indirectly through either few-shot examples (e.g. giving two examples of frog stories, and prompting the model to generate a new one) or implicit continuation (e.g. providing the start of a story about a frog). In each case, we ask our labelers to do their best to infer the intent of the user who wrote the prompt, and ask them to skip inputs where the task is very unclear. Moreover, our labelers also take into account the implicit intentions such as truthfulness of the response, and potentially harmful outputs such as biased or toxic language, guided by the instructions we provide them (see Appendix B) and their best judgment.


oxic language, guided by the instructions we provide them (see Appendix B) and their best judgment. 3.4 Human data collection To produce our demonstration and comparison data, and to conduct our main evaluations, we hired a team of about 40 contractors on Upwork and through ScaleAI. Compared to earlier work that collects human preference data on the task of summarization (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021), our inputs span a much broader range of tasks, and can occasionally include controversial and sensitive topics. Our aim was to select a group of labelers who were sensitive to the preferences of different demographic groups, and who were good at identifying outputs that were potentially harmful.


fferent demographic groups, and who were good at identifying outputs that were potentially harmful. Thus, we conducted a screening test designed to measure labeler performance on these axes. We selected labelers who performed well on this test; for more information about our selection procedure and labeler demographics, see Appendix B.1. During training and evaluation, our alignment criteria may come into conﬂict: for example, when a user requests a potentially harmful response. During training we prioritize helpfulness to the user (not 7


doing so requires making some difﬁcult design decisions that we leave to future work; see Section 5.4 for more discussion). However, in our ﬁnal evaluations we asked labelers prioritize truthfulness and harmlessness (since this is what we really care about). As in Stiennon et al. (2020), we collaborate closely with labelers over the course of the project. We have an onboarding process to train labelers on the project, write detailed instructions for each task (see Appendix B.2), and answer labeler questions in a shared chat room. As an initial study to see how well our model generalizes to the preferences of other labelers, we hire a separate set of labelers who do not produce any of the training data. These labelers are sourced from the same vendors, but do not undergo a screening test.