0M 50M 100M Env steps 0 25 50 75 100 Agents with item (%) Iron Ingot 0M 50M 100M Env steps 0 25 50 75 100 Iron Pickaxe 0M 50M 100M Env steps 0 25 50 75 100 Diamond Dreamer IMPALA Rainbow PPO Figure 5: Fraction of trained agents that discover each of the three latest items in the Minecraft Diamond task. Although previous algorithms progress up to the iron pickaxe, Dreamer is the only compared algorithm that manages to discover a diamond, and does so reliably. Results We evaluate the generality of Dreamer across 8 domains—with over 150 tasks—under fixed hyperparameters. We designed the experiments to compare Dreamer to the best methods in the literature, which are often specifically designed and tuned for the benchmark at hand.


thods in the literature, which are often specifically designed and tuned for the benchmark at hand. We further compare to a high-quality implementation of PPO5, a standard reinforcement learning algorithm that is known for its robustness. We run PPO with fixed hyperparameters chosen to maximize performance across domains and that reproduce strong published results of PPO on ProcGen37. To push the boundaries of reinforcement learning, we apply Dreamer to the challenging video game Minecraft, comparing it to strong previous algorithms. Finally, we analyze the importance of individual components of Dreamer and its robustness to different model sizes and computational budgets. All Dreamer agents are trained on a single Nvidia A100 GPU each, making it reproducible for many research labs.


agents are trained on a single Nvidia A100 GPU each, making it reproducible for many research labs. A public implementation of Dreamer that reproduces all results is available on the project website. Benchmarks We perform an extensive empirical study across 8 domains that include continuous and discrete actions, visual and low-dimensional inputs, dense and sparse rewards, different reward scales, 2D and 3D worlds, and procedural generation. Figure 1 summarizes the benchmark results, showing that Dreamer outperforms a wide range of previous expert algorithms across diverse domains. Crucially, Dreamer substantially outperforms PPO across all domains. • Atari This established benchmark contains 57 Atari 2600 games with a budget of 200M frames, posing a diverse range of challenges38.


contains 57 Atari 2600 games with a budget of 200M frames, posing a diverse range of challenges38. We use the sticky action simulator setting39. Dreamer outperforms the powerful MuZero algorithm8 while using only a fraction of the computational resources. Dreamer also outperforms the widely-used expert algorithms Rainbow40 and IQN41. • ProcGen This benchmark of 16 games features randomized levels and visual distractions to test the robustness and generalization of agents42. Within the budget of 50M frames, Dreamer matches the tuned expert algorithm PPG37 and outperforms Rainbow42,40. Our PPO agent with fixed hyperparameters matches the published score of the highly tuned official PPO implementation37.


ixed hyperparameters matches the published score of the highly tuned official PPO implementation37. • DMLab This suite of 30 tasks features 3D environments that test spatial and temporal reasoning43. In 100M frames, Dreamer exceeds the performance of the scalable IMPALA and R2D2+ agents35 at 1B environment steps, amounting to a data-efficiency gain of over 1000%. We note that these baselines were not designed for data-efficiency but serve as a valuable comparison point for the performance previously achievable at scale. 8


0 50 100 Env steps (%) 0 50 100 Return (%) 14 task mean Dreamer No obs symlog No retnorm (advnorm) No symexp twohot (Huber) No KL balance & free bits Without all 0 20 40 Env steps (106) 0 10 20 Return Crafter 400M 200M 100M 50M 25M 12M 0 100 200 Env steps (106) 0 250 500 DMLab Goals 0 50 100 Env steps (%) 0 50 100 Return (%) 14 task mean Dreamer No reward & value grads No reconstruction grads 0 10 20 Env steps (106) 0 9 18 Return Crafter 64 32 16 8 4 2 1 0 100 200 Env steps (106) 0 150 300 450 DMLab Goals Robustness techniques Learning signals Model size scaling Replay scaling a b c d Figure 6: Ablations and robust scaling of Dreamer. a, All individual robustness techniques contribute to the performance of Dreamer on average, although each individual technique may only affect some tasks.


e performance of Dreamer on average, although each individual technique may only affect some tasks. Training curves of individual tasks are included in the supplementary material. b, The performance of Dreamer predominantly rests on the unsupervised reconstruction loss of its world model, unlike most prior algorithms that rely predominantly on reward and value prediction gradients7,5,8. c, The performance of Dreamer increases monotonically with larger model sizes, ranging from 12M to 400M parameters. Notably, larger models not only increase task performance but also require less environment interaction. d, Higher replay ratios predictably increase the performance of Dreamer. Together with model size, this allows practitioners to improve task performance and data-efficiency by employing mor


del size, this allows practitioners to improve task performance and data-efficiency by employing more computational resources. • Atari100k This data-efficiency benchmark comntains 26 Atari games and a budget of only 400K frames, amounting to 2 hours of game time17. EfficientZero44 holds the state-of-the-art by combining online tree search, prioritized replay, and hyperparameter scheduling, but also resets levels early to increase data diversity, making a comparison difficult. Without this complexity, Dreamer outperforms the best remaining methods, including the transformer-based IRIS and TWM agents, the model-free SPR, and SimPLe45. • Proprio Control This benchmark contains 18 control tasks with continuous actions, proprioceptive vector inputs, and a budget of 500K environment steps46.


ks with continuous actions, proprioceptive vector inputs, and a budget of 500K environment steps46. The tasks range from classical control over locomotion to robot manipulation tasks, featuring dense and sparse rewards. Dreamer sets a new state-of-the-art on this benchmark, outperforming D4PG, DMPO, and MPO33. • Visual Control This benchmark consists of 20 continuous control tasks where the agent receives only high-dimensional images as input and has a budget of 1M environment steps46. Dreamer establishes a new state-of-the-art on this benchmark, outperforming DrQ-v2 and CURL47, which are specialized to visual environments and leverage data augmentation. 9


• BSuite This benchmark includes 23 environments with a total of 468 configurations that are specifically designed to test credit assignment, robustness to reward scale and stochasticity, memory, generalization, and exploration48. Dreamer establishes a new state-of-the-art on this benchmark, outperforming Boot DQN and other methods49. Dreamer improves over previous algorithms especially in the scale robustness category. Minecraft Collecting diamonds in the popular game Minecraft has been a long-standing challenge in artificial intelligence18,19,20. Every episode in this game is set in a unique randomly generated and infinite 3D world. Episodes last until the player dies or up to 36000 steps equaling 30 minutes, during which the player needs to discover a sequence of 12 items from sparse re


equaling 30 minutes, during which the player needs to discover a sequence of 12 items from sparse rewards by foraging for resources and crafting tools. It takes about 20 minutes for experienced human players to obtain diamonds20. We follow the block breaking setting of prior work19 because the provided action space would make it challenging for stochastic policies to keep a key pressed for a prolonged time. Because of the training time in this complex domain, extensive tuning would be difficult for Minecraft. Instead, we apply Dreamer out of the box with its default hyperparameters. As shown in Figures 1 and 5, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without using human data as was required by VPT20 or adaptive curricula19.


n Minecraft from scratch without using human data as was required by VPT20 or adaptive curricula19. All the Dreamer agents we trained on Minecraft discover diamonds in 100M environment steps. While several strong baselines progress to advanced items such as the iron pickaxe, none of them discovers a diamond. Ablations In Figure 6, we ablate the robustness techniques and learning signals on a diverse set of 14 tasks to understand their importance. The training curves of individual tasks are included in the supplementary material. We observe that all robustness techniques contribute to performance, most notably the KL objective of the world model, followed by return normalization and symexp twohot regression for reward and value prediction.


del, followed by return normalization and symexp twohot regression for reward and value prediction. In general, we find that each individual technique is critical on a subset of tasks but may not affect performance on other tasks. To investigate the effect of the world model, we ablate the learning signals of Dreamer by stopping either the task-specific reward and value prediction gradients or the task-agnostic reconstruction gradients from shaping its representations. Unlike previous reinforcement learning algorithms that often rely only on task-specific learning signals7,8, Dreamer rests predominantly on the unsupervised objective of its world model. This finding could allow for future algorithm variants that leverage pretraining on unsupervised data.


s finding could allow for future algorithm variants that leverage pretraining on unsupervised data. Scaling properties To investigate whether Dreamer can scale robustly, we train 6 model sizes ranging from 12M to 400M parameters, as well as different replay ratios on Crafter50 and a DMLab task43. The replay ratio affects the number of gradient updates performed by the agent. Figure 6 shows robust learning with fixed hyperparameters across the compared model sizes and replay ratios. Moreover, increasing the model size directly translates to both higher task performance and a lower data requirement. Increasing the number of gradient steps further reduces the interactions needed to learn successful behaviors.


the number of gradient steps further reduces the interactions needed to learn successful behaviors. The results show that Dreamer learns robustly across model sizes and replay ratios and that its performance and provides a predictable way for increasing performance given computational resources. 10


Previous work Developing general-purpose algorithms has long been a goal of reinforcement learning research. PPO5 is one of the most widely used algorithms and is relatively robust but requires large amounts of experience and often yields lower performance than specialized alternatives. SAC32 is a popular choice for continuous control and leverages experience replay for data-efficiency, but in practice requires tuning, especially for its entropy scale, and struggles under high-dimensional inputs51. MuZero8 plans using a value prediction model and has been applied to board games and Atari, but the authors did not release an implementation and the algorithm contains several complex components, making it challenging to reproduce.


entation and the algorithm contains several complex components, making it challenging to reproduce. Gato52 fits one large model to expert demonstrations of multiple tasks, but is only applicable when expert data is available. In comparison, Dreamer masters a diverse range of environments with fixed hyperparameters, does not require expert data, and its implementation is open source. Minecraft has been a focus of recent research. With MALMO53, Microsoft released a free version of the successful game for research purposes. MineRL18 offers several competition environments, which we rely on as the basis for our experiments. The MineRL competition supports agents in exploring and learning meaningful skills through a diverse human dataset18.


tion supports agents in exploring and learning meaningful skills through a diverse human dataset18. Voyager obtains items at a similar depth in the technology tree as Dreamer using API calls to a language model but operates on top of the MineFlayer bot scripting layer that was specifically engineered to the game and exposes high-level actions54. VPT20 trained an agent to play Minecraft through behavioral cloning based on expert data of keyboard and mouse actions collected by contractors and finetuning using reinforcement learning to obtain diamonds using 720 GPUs for 9 days. In comparison, Dreamer uses the MineRL competition action space to autonomously learn to collect diamonds from sparse rewards using 1 GPU for 9 days, without human data.


onomously learn to collect diamonds from sparse rewards using 1 GPU for 9 days, without human data. Conclusion We present the third generation of the Dreamer algorithm, a general reinforcement learning algorithm that masters a wide range of domains with fixed hyperparameters. Dreamer excels not only across over 150 tasks but also learns robustly across varying data and compute budgets, moving reinforcement learning toward a wide range of practical applications. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch, achieving a significant milestone in the field of artificial intelligence. As a high-performing algorithm that is based on a learned world model, Dreamer paves the way for future research directions, including teaching agents world


world model, Dreamer paves the way for future research directions, including teaching agents world knowledge from internet videos and learning a single world model across domains to allow artificial agents to build up increasingly general knowledge and competency. Acknowledgements We thank Mohammad Norouzi, Jessy Lin, Abbas Abdolmaleki, John Schulman, Adam Kosiorek, and Oleh Rybkin for insightful discussions. We thank Bobak Shahriari, Denis Yarats, Karl Cobbe, and Hubert Soyer for sharing training curves of baseline algorithms. We thank Daniel Furrer, Andrew Chen, and Dakshesh Garambha for help with Google Cloud infrastructure. 11


References 1. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587): 484, 2016. 2. OpenAI. OpenAI Five. https://blog.openai.com/openai-five/, 2018. 3. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. 4. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi.


744, 2022. 4. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328, 2022. 5. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 6. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. 7. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland


r, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015. 8. Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265, 2019. 9. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016. 10.


einforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016. 10. Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. Advances in neural information processing systems, 32, 2019. 11. Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, and Marc Toussaint. Reinforcement learning with neural radiance fields. arXiv preprint arXiv:2206.01634, 2022. 12. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017. 13.


n Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017. 13. Marcin Andrychowicz, Anton Raichuk, Piotr Sta´nczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990, 2020. 14. Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160–163, 1991. 12


15. Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2786–2793. IEEE, 2017. 16. David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 17. Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Modelbased reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019. 18. William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al.


tephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. The minerl competition on sample efficient reinforcement learning using human priors. arXiv e-prints, pages arXiv–1904, 2019. 19. Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2106.14876, 2021. 20. Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11795, 2022. 21.


t): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11795, 2022. 21. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019. 22. Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. 23. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 24. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018. 25. Yoshua Bengio, Nicholas Léonard, and Aaron Courville.


s. arXiv preprint arXiv:1811.04551, 2018. 25. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. 26. Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016. 27. Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011.10650, 2020. 28. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449–458. PMLR, 2017. 29.


forcement learning. In International Conference on Machine Learning, pages 449–458. PMLR, 2017. 29. Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. 30. Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991. 13


31. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. 32. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018. 33. Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018. 34. J Beau W Webber. A bi-symmetric log transformation for wide-range data. Measurement Science and Technology, 24(2):027001, 2012. 35. Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney.


4(2):027001, 2012. 35. Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018. 36. Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796–3803, 2019. 37. Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020–2027. PMLR, 2021. 38. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents.


s, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013. 39. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523–562, 2018. 40. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. 41. Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos.


e on Artificial Intelligence, 2018. 41. Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096–1105. PMLR, 2018. 42. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048–2056. PMLR, 2020. 43. Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016. 44. Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data.


aohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476–25488, 2021. 45. Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint arXiv:2209.00588, 2022. 14


46. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. 47. Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021. 48. Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568, 2019. 49. Olivia Dizon-Paradis, Stephen Wormald, Daniel Capecci, Avanti Bhandarkar, and Damon Woodard.


9. 49. Olivia Dizon-Paradis, Stephen Wormald, Daniel Capecci, Avanti Bhandarkar, and Damon Woodard. Investigating the practicality of existing reinforcement learning algorithms: A performance comparison. Authorea Preprints, 2023. 50. Danijar Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780, 2021. 51. Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019. 52. Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022. 53.


ay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022. 53. Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI, pages 4246–4247. Citeseer, 2016. 54. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. 55. Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022. 56. Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abd


an, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020. 57. Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. In International Conference on Machine Learning, pages 8545–8554. PMLR, 2020. 58. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015. 59. Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059–1071. PMLR, 2021. 60.


ut normalization. In International Conference on Machine Learning, pages 1059–1071. PMLR, 2021. 60. Liu Ziyin, Zhikang T Wang, and Masahito Ueda. Laprop: Separating momentum and adaptivity in adam. arXiv preprint arXiv:2002.04839, 2020. 15


61. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 62. Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Marc Bellemare, and Remi Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learning. arXiv preprint arXiv:1704.04651, 2017. 63. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. 64. Matthijs Van Keirsbilck, Alexander Keller, and Xiaodong Yang. Rethinking full connectivity in recurrent neural networks. arXiv preprint arXiv:1905.12340, 2019. 65.


thinking full connectivity in recurrent neural networks. arXiv preprint arXiv:1905.12340, 2019. 65. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523–562, 2018. 66. Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018. 16


Methods Baselines We employ the Proximal Policy Optimization (PPO) algorithm5, which has become a standard choice in the field, to compare Dreamer under fixed hyperparameters across all benchmarks. There are a large number of PPO implementations available publicly and they are known to substantially vary in task performance55. To ensure a comparison that is representative of the highest performance PPO can achieve under fixed hyperparameters across domains, we choose the high-quality PPO implementation available in the Acme framework56 and select its hyperparameters in Table 1 following recommendations55,13 and additionally tune its epoch batch size to be large enough for complex environments42, its learning rate, and its entropy scale.


batch size to be large enough for complex environments42, its learning rate, and its entropy scale. We match the discount factor to Dreamer because it works well across domains and is a common choice in the literature35,8. We choose the IMPALA network architecture that we have found performed better than alternatives42 and set the minibatch size to the largest possible for one A100 GPU. We verify the performance of our PPO implementation and hyperparameters on the ProcGen benchmark, where a highly tuned PPO implementation has been reported by the PPO authors37. We find that our implementation matches or slightly outperforms this performance reference. Parameter Value Observation normalization Yes Reward normalization Yes Reward clipping (stddev.) 10 Epoch batch 64 × 256 Number of epochs 3


n Yes Reward normalization Yes Reward clipping (stddev.) 10 Epoch batch 64 × 256 Number of epochs 3 Minibatch size 8 Minibatch length 256 Policy trust region 0.2 Value trust region No Advantage normalization Yes Entropy penalty scale 0.01 Discount factor 0.997 GAE lambda 0.95 Learning rate 3 × 10−4 Gradient clipping 0.5 Adam epsilon 10−5 Table 1: PPO hyperparameters used across all benchmarks. For Minecraft, we additionally tune and run the IMPALA and Rainbow algorithms because not successful end-to-end learning from scratch has been reported in the literature18. We use the Acme implementations56 of these algorithms, use the same IMPALA network we used for PPO, and tuned the learning rate and entropy regularizers.


use the same IMPALA network we used for PPO, and tuned the learning rate and entropy regularizers. For all other benchmarks, we compare to tuned expert algorithms reported in the literature as referenced in the results section. 17


Implementation Experience replay We implement Dreamer using a uniform replay buffer with an online queue57. Specifically, each minibatch is formed first from non-overlapping online trajectories and then filled up with uniformly sampled trajectories from the replay buffer. We store latent states into the replay buffer during data collection to initialize the world model on replayed trajectories, and write the fresh latent states of the training rollout back into the buffer. While prioritized replay58 is used by some of the expert algorithms we compare to and we found it to also improve the performance of Dreamer, we opt for uniform replay in our experiments for ease of implementation. We parameterize the amount of training via the replay ratio.


xperiments for ease of implementation. We parameterize the amount of training via the replay ratio. This is the fraction of time steps trained on per time step collected from the environment, without action repeat. Dividing the replay ratio by the time steps in a minibatch and by action repeat yields the ratio of gradient steps to env steps. For example, a replay ratio of 32 on Atari with action repeat of 4 and batch shape 16 × 64 corresponds to 1 gradient step every 128 env steps, or 1.5M gradient steps over 200M env steps. Optimizer We employ Adaptive Gradient Clipping (AGC)59, which clips per-tensor gradients if they exceed 30% of the L2 norm of the weight matrix they correspond to, with its default ϵ = 10−3.


they exceed 30% of the L2 norm of the weight matrix they correspond to, with its default ϵ = 10−3. AGC decouples the clipping threshold from the loss scales, allowing to change loss functions or loss scales without adjusting the clipping threshold. We apply the clipped gradients using the LaProp optimizer60 with ϵ = 10−20 and its default parameters β1 = 0.9 and β2 = 0.99. LaProp normalizes gradients by RMSProp and then smoothes them by momentum, instead of computing both momentum and normalizer on raw gradients as Adam does61. This simple change allows for a smaller epsilon and avoids occasional instabilities that we observed under Adam. Distributions The encoder, dynamics predictor, and actor distributions are mixtures of 99% the predicted softmax output and 1% of a uniform distribution6


tor distributions are mixtures of 99% the predicted softmax output and 1% of a uniform distribution62 to prevent zero probabilities and infinite log probabilities. The rewards and critic neural networks output a softmax distribution over exponentially spaced bins b ∈B and are trained towards twohot encoded targets: twohot(x)i .=      |bk+1 −x| / |bk+1 −bk| if i = k |bk −x| / |bk+1 −bk| if i = k + 1 0 else k .= |B| X j=1 δ(bj < x) (12) The output weights of twohot distributions are initialized to zero to ensure that the agent does not hallucinate rewards and values at initialization. For computing the expected prediction of the softmax distribution under bins that span many orders of magnitude, the summation order matters and positive and negative bins should be summed up separately, f


nitude, the summation order matters and positive and negative bins should be summed up separately, from small to large bins, and then added. Refer to the source code for an implementation. Networks Images are encoded using stride 2 convolutions to resolution 6 × 6 or 4 × 4 and then flattened and are decoded using transposed stride 2 convolutions, with sigmoid activation on the output. Vector inputs are symlog transformed and then encoded and decoded using 3-layer MLPs. The actor and critic neural networks are also 3-layer MLPs and the reward and continue predictors are 1-layer MLPs. The sequence model is a GRU63 with block-diagonal recurrent weights64 of 8 blocks to allow for a large number of memory units without quadratic increase in parameters and FLOPs.


cks to allow for a large number of memory units without quadratic increase in parameters and FLOPs. The input to the GRU at each time step is a linear embedding of the sampled latent zt, of the action at, and of the recurrent state to allow mixing between blocks. 18


Benchmarks Protocols Summarized in Table 2, we follow the standard evaluation protocols for the benchmarks where established. Atari38 uses 57 tasks with sticky actions65. The random and human reference scores used to normalize scores vary across the literature and we chose the most common reference values, replicated in Table 6. DMLab43 uses 30 tasks66 and we use the fixed action space36,35. We evaluate at 100M steps because running for 10B as in some prior work was infeasible. Because existing published baselines perform poorly at 100M steps, we compare to their performance at 1B steps instead, giving them a 10× data advantage. ProcGen uses the hard difficulty setting and the unlimited level set42.


them a 10× data advantage. ProcGen uses the hard difficulty setting and the unlimited level set42. Prior work compares at different step budgets42,37 and we compare at 50M steps due to computational cost, as there is no action repeat. For Minecraft Diamond purely from sparse rewards, we establish the evaluation protocol to report the episode return measured at 100M env steps, corresponding to about 100 days of in-game time. Atari100k17 includes 26 tasks with a budget of 400K env steps, 100K after action repeat. Prior work has used various environment settings, summarized in Table 10, and we chose the environments as originally introduced. Visual Control46,21 spans 20 tasks with an action repeat of 2.


vironments as originally introduced. Visual Control46,21 spans 20 tasks with an action repeat of 2. Proprioceptive Control follows the same protocol but we exclude the two quadruped tasks because of baseline availability in prior work47. Benchmark Tasks Env Steps Action Repeat Env Instances Replay Ratio GPU Days Model Size Minecraft 1 100M 1 64 32 8.9 200M DMLab 30 100M 4 16 32 2.9 200M ProcGen 16 50M 1 16 64 16.1 200M Atari 57 200M 4 16 32 7.7 200M Atari100K 26 400K 4 1 128 0.1 200M BSuite 23 — 1 1 1024 0.5 200M Proprio Control 18 500K 2 16 512 0.3 12M Visual Control 20 1M 2 16 512 0.1 12M Table 2: Benchmark overview. All agents were trained on a single Nvidia A100 GPU each. Environment instances In earlier experiments, we observed that the performance of both Dreamer and PPO is robust to


tances In earlier experiments, we observed that the performance of both Dreamer and PPO is robust to the number of environment instances. Based on the CPU resources available on our training machines, we use 16 environment instance by default. For BSuite, the benchmark requires using a single environment instance. We also use a single environment instance for Atari100K because the benchmark has a budget of 400K env steps whereas the maximum episode length in Atari is in principle 432K env steps. For Minecraft, we use 64 environments using remote CPU workers to speed up experiments because the environment is slower to step. Seeds and error bars We run 5 seeds for each Dreamer and PPO per benchmark, with the exception of 1 seed for ProcGen due to computational constraints, 10 seeds for BSuit


hmark, with the exception of 1 seed for ProcGen due to computational constraints, 10 seeds for BSuite as required by the benchmark, and 10 seeds for Minecraft to reliably report the fraction of runs that achieve diamonds. All curves show the mean over seeds with one standard deviation shaded. Computational choices All Dreamer and PPO agents in this paper were trained on a single Nvidia A100 GPU each. Dreamer uses the 200M model size by default. On the two control suitse, Dreamer the same performance using the substantially faster 12M model, making it more accessible to researchers. The replay ratio control the trade-off between computational cost and data efficiency as analyzed in Figure 6 and is chosen to fit the step budget of each benchmark. 19


Model sizes To accommodate different computational budgets and analyze robustness to different model sizes, we define a range of models ranging from 12M to 400M parameters shown in Table 3. The sizes are parameterized by the model dimension, which approximately increases in multiples of 1.5, alternating between powers of two and power of two scaled by 1.5. This yields tensor shapes that are multiples of 8 as required for hardware efficiency. Sizes of different network components derive from the model dimension. The MLPs have the model dimension as the number of hidden units. The sequence model has 8 times the number of recurrent units, split into 8 blocks of the same size as the MLPs. The convolutional encoder and decoder layers closest to the data use 16× fewer channels than the model dim


nvolutional encoder and decoder layers closest to the data use 16× fewer channels than the model dimension. Each latent also uses 16× fewer codes than the model dimension. The number of hidden layers and number of latents is fixed across model sizes. All hyperparamters, including the learning rate and batch size, are fixed across model sizes. Parameters 12M 25M 50M 100M 200M 400M Hidden size (d) 256 384 512 768 1024 1536 Recurrent units (8d) 1024 3072 4096 6144 8192 12288 Base CNN channels (d/16) 16 24 32 48 64 96 Codes per latent (d/16) 16 24 32 48 64 96 Table 3: Dreamer model sizes. The number of MLP hidden units defines the model dimension, from which recurrent units, convolutional channels, and number of codes per latent are derived.


on, from which recurrent units, convolutional channels, and number of codes per latent are derived. The number of layers and latents is constant across model sizes. Previous Dreamer generations We present the third generation of the Dreamer line of work. Where the distinction is useful, we refer to this algorithm as DreamerV3. The DreamerV1 algorithm21 was limited to continuous control, the DreamerV2 algorithm22 surpassed human performance on Atari, and the DreamerV3 algorithm enables out-of-the-box learning across diverse benchmarks. We summarize the changes that DreamerV3 introduces as follows: • Robustness techniques: Observation symlog, KL balance and free bits, 1% unimix for all categoricals, percentile return normalization, symexp twohot loss for the reward head and critic.


l categoricals, percentile return normalization, symexp twohot loss for the reward head and critic. • Network architecture: Block GRU, RMSNorm normalization, SiLu activation. • Optimizer: Adaptive gradient clipping (AGC), LaProp (RMSProp followed by momentum). • Replay buffer: Larger capacity, online queue, storing and updating latent states. 20


Hyperparameters Name Symbol Value General Replay capacity — 5 × 106 Batch size B 16 Batch length T 64 Activation — RMSNorm + SiLU Learning rate — 4 × 10−5 Gradient clipping — AGC(0.3) Optimizer — LaProp(ϵ = 10−20) World Model Reconstruction loss scale βpred 1 Dynamics loss scale βdyn 1 Representation loss scale βrep 0.1 Latent unimix — 1% Free nats — 1 Actor Critic Imagination horizon H 15 Discount horizon 1/(1 −γ) 333 Return lambda λ 0.95 Critic loss scale βval 1 Critic replay loss scale βrepval 0.3 Critic EMA regularizer — 1 Critic EMA decay — 0.98 Actor loss scale βpol 1 Actor entropy regularizer η 3 × 10−4 Actor unimix — 1% Actor RetNorm scale S Per(R, 95) −Per(R, 5) Actor RetNorm limit L 1 Actor RetNorm decay — 0.99 Table 4: Dreamer hyperparameters.


95) −Per(R, 5) Actor RetNorm limit L 1 Actor RetNorm decay — 0.99 Table 4: Dreamer hyperparameters. The same values are used across all benchmarks, including proprioceptive and visual inputs, continuous and discrete actions, and 2D and 3D domains. We do not use any hyperparameter annealing, prioritized replay, weight decay, or dropout. 21


Minecraft Game description With 100M monthly active users, Minecraft is one of the most popular video games worldwide. Minecraft features a procedurally generated 3D world of different biomes, including plains, forests, jungles, mountains, deserts, taiga, snowy tundra, ice spikes, swamps, savannahs, badlands, beaches, stone shores, rivers, and oceans. The world consists of 1 meter sized blocks that the player and break and place. There are about 30 different creatures that the player can interact with or fight. From gathered resources, the player can use over 350 recipes to craft new items and progress through the technology tree, all while ensuring safety and food supply to survive. There are many conceivable tasks in Minecraft and as a first step, the research community has focused on th


re many conceivable tasks in Minecraft and as a first step, the research community has focused on the salient task of obtaining a diamonds, a rare item found deep underground and requires progressing through the technology tree. Learning environment We built the Minecraft Diamond environment on top of MineRL to define a flat categorical action space and fix issues we discovered with the original environments via human play testing. For example, when breaking diamond ore, the item sometimes jumps into the inventory and sometimes needs to be collected from the ground. The original environment terminates episodes when breaking diamond ore so that many successful episodes end before collecting the item and thus without the reward.


nd ore so that many successful episodes end before collecting the item and thus without the reward. We remove this early termination condition and end episodes when the player dies or after 36000 steps, corresponding to 30 minutes at the control frequency of 20Hz. Another issue is that the jump action has to be held for longer than one control step to trigger a jump, which we solve by keeping the key pressed in the background for 200ms. We built the environment on top of MineRL v0.4.418, which offers abstract crafting actions. The Minecraft version is 1.11.2. Observations and rewards The agent observes a 64 × 64 × 3 first-person image, an inventory count vector for the over 400 items, a vector of maximum inventory counts since episode begin to tell the agent which milestones it has achieve


or of maximum inventory counts since episode begin to tell the agent which milestones it has achieved, a one-hot vector indicating the equipped item, and scalar inputs for the health, hunger, and breath levels. We follow the sparse reward structure of the MineRL competition environment18 that rewards 12 milestones leading up to the diamond, for obtaining the items log, plank, stick, crafting table, wooden pickaxe, cobblestone, stone pickaxe, iron ore, furnace, iron ingot, iron pickaxe, and diamond. The reward for each item is only given once per episode, and the agent has to learn to collect certain items multiple times to achieve the next milestone. To make the return easy to interpret, we give a reward of +1 for each milestone instead of scaling rewards based on how valuable each item is


give a reward of +1 for each milestone instead of scaling rewards based on how valuable each item is. Additionally, we give −0.01 for each lost heart and +0.01 for each restored heart, but did not investigate whether this is helpful. 22


Supplementary material Minecraft video predictions True Context Input Open Loop Prediction Model True Model True Model True Model True T = 0 Model 5 10 15 20 25 30 35 40 45 50 Figure 7: Multi-step predictions on Minecraft. The world model receives the first 5 frames as context input and the predicts 45 steps into the future given the action sequence and without access to intermediate images. 23


Minecraft additional results Method Return Dreamer 9.1 IMPALA 7.1 Rainbow 6.3 PPO 5.1 Table 5: Minecraft Diamond scores at 100M environment steps. 0 25M 50M 75M 100M Env steps 0 4 8 12 Return Minecraft Diamond Dreamer IMPALA Rainbow PPO Figure 8: Minecraft learning curves. 0 25 50 75 100 % of episodes Log 0 25 50 75 100 Planks 0 25 50 75 100 Stick 0 25 50 75 100 Crafting Table 0 25 50 75 100 % of episodes Wooden Pickaxe 0 25 50 75 100 Cobblestone 0 25 50 75 100 Furnace 0 25 50 75 100 Stone Pickaxe 100K 1M 10M 100M Env steps 0 20 40 60 80 % of episodes Iron Ore 100K 1M 10M 100M Env steps 0 20 40 60 80 Iron Ingot 100K 1M 10M 100M Env steps 0 6 12 18 24 Iron Pickaxe 100K 1M 10M 100M Env steps 0 0.2 0.5 0.8 1 Diamond Dreamer IMPALA Rainbow PPO Figure 9: Item success rates as a percentage of ep


0.2 0.5 0.8 1 Diamond Dreamer IMPALA Rainbow PPO Figure 9: Item success rates as a percentage of episodes. Dreamer obtains items at substantially higher rates than the baselines and continues to improve until the 100M step budget. At the budget, Dreamer obtains diamonds in 0.4% of episodes, leaving a challenge for future research. This metric differs from Figure 5, which shows that over the course of training, 100% of Dreamer agents obtain one or more diamonds regardless of episode boundaries, compared to 0% of the baseline agents. 24


Atari learning curves 0 25K 50K 75K Return Alien 0 2K 4K Amidar 0 15K 30K 45K Assault 0 300K 600K 900K Asterix 0 150K 300K Asteroids 0 500K 1M 1.5M Atlantis 0 400 800 1.2K Return Bank Heist 0 150K 300K 450K Battle Zone 0 80K 160K Beam Rider 0 8K 16K 24K Berzerk 80 160 240 Bowling 0 30 60 90 Boxing 0 250 500 750 Return Breakout 0 250K 500K 750K Centipede 0 300K 600K 900K ChopperComm 0 60K 120K 180K Crazy Climber 0 200K 400K 600K Defender 0 50K 100K 150K Demon Attack -15 0 15 30 Return Double Dunk 0 800 1.6K 2.4K Enduro -60 0 60 Fishing Derby 0 15 30 Freeway 0 25K 50K 75K Frostbite 0 40K 80K 120K Gopher 0 4K 8K 12K Return Gravitar 0 15K 30K 45K Hero 0 25 50 75 Ice Hockey 0 8K 16K 24K Jamesbond 0 4K 8K 12K Kangaroo 0 30K 60K 90K Krull 0 50K 100K 150K Return Kung Fu Master 0 1K 2K 3K Montezuma


0 4K 8K 12K Kangaroo 0 30K 60K 90K Krull 0 50K 100K 150K Return Kung Fu Master 0 1K 2K 3K MontezumaRev 0 20K 40K 60K Ms Pacman 0 40K 80K 120K Name This Game 0 150K 300K Phoenix -300 -150 0 Pitfall -15 0 15 Return Pong 0 15K 30K 45K Private Eye 0 150K 300K Qbert 0 40K 80K 120K Riverraid 0 200K 400K 600K Road Runner 0 40 80 120 Robotank 0 300K 600K Return Seaquest -32K -24K -16K Skiing 2K 4K 6K 8K Solaris 0 10K 20K 30K Space Invaders 0 250K 500K 750K Star Gunner -8 0 8 Surround -24 -16 -8 0 Return Tennis 0 150K 300K Time Pilot 0 150 300 Tutankham 0 250K 500K 750K Up N Down 0 600 1.2K 1.8K Venture 0 300K 600K 900K Video Pinball 0 200M Env steps 0 40K 80K Return Wizard Of Wor 0 200M Env steps 0 300K 600K 900K Yars Revenge 0 200M Env steps 0 25K 50K 75K Zaxxon 0 200M Env steps 0 10 20 30 Gamer


300K 600K 900K Yars Revenge 0 200M Env steps 0 25K 50K 75K Zaxxon 0 200M Env steps 0 10 20 30 Gamer Mean 0 200M Env steps 0 4 8 Gamer Median 0 200M Env steps 0 0.2 0.4 Record Mean Cap Dreamer MuZero PPO Figure 10: Atari learning curves. 25


Atari scores Task Random Gamer Record PPO MuZero Dreamer Environment steps — — — 200M 200M 200M Alien 228 7128 251916 5476 56835 10977 Amidar 6 1720 104159 817 1517 3612 Assault 222 742 8647 6673 42742 26010 Asterix 210 8503 1000000 47190 879375 441763 Asteroids 719 47389 10506650 2479 374146 348684 Atlantis 12850 29028 10604840 539721 1353617 1553222 Bank Heist 14 753 82058 946 1077 1083 Battle Zone 2360 37188 801000 27816 167412 419653 Beam Rider 364 16926 999999 7973 201154 37073 Berzerk 124 2630 1057940 1186 1698 10557 Bowling 23 161 300 118 133 250 Boxing 0 12 100 98 100 100 Breakout 2 30 864 299 799 384 Centipede 2091 12017 1301709 51833 774421 554553 Chopper Command 811 7388 999999 12667 8945 802698 Crazy Climber 10780 35829 219900 93176 184394 193204 Defender 2874 18689 6010500 382


667 8945 802698 Crazy Climber 10780 35829 219900 93176 184394 193204 Defender 2874 18689 6010500 38270 554492 579875 Demon Attack 152 1971 1556345 8229 142509 142109 Double Dunk –19 –16 22 16 23 24 Enduro 0 860 9500 1887 2369 2166 Fishing Derby –92 –39 71 43 58 82 Freeway 0 30 38 33 0 34 Frostbite 65 4335 454830 1123 17087 41888 Gopher 258 2412 355040 24792 122025 87600 Gravitar 173 3351 162850 3436 10301 12570 Hero 1027 30826 1000000 31967 36063 40677 Ice Hockey –11 1 36 12 26 57 Jamesbond 29 303 45550 1019 14872 24010 Kangaroo 52 3035 1424600 7769 14380 12229 Krull 1598 2666 104100 9193 11476 69858 Kung Fu Master 258 22736 1000000 32335 148936 154893 Montezuma Revenge 0 4753 1219200 2368 0 1852 Ms Pacman 307 6952 290090 7041 51310 24079 Name This Game 2292 8049 25220 19441 85331 77809 Ph


1852 Ms Pacman 307 6952 290090 7041 51310 24079 Name This Game 2292 8049 25220 19441 85331 77809 Phoenix 761 7243 4014440 31412 105593 316606 Pitfall –229 6464 114000 –2 0 0 Pong –21 15 21 19 21 20 Private Eye 25 69571 101800 73 100 26432 Qbert 164 13455 2400000 14554 102129 201084 Riverraid 1338 17118 1000000 14860 137983 48080 Road Runner 12 7845 2038100 423995 604083 150402 Robotank 2 12 76 63 70 132 Seaquest 68 42055 999999 1927 399764 356584 Skiing –17098 –4337 –3272 –29926 –30000 –29965 Solaris 1236 12327 111420 2368 5860 5851 Space Invaders 148 1669 621535 3489 3639 15005 Star Gunner 664 10250 77400 53439 127417 408961 Surround –10 6 6 6 9 9 Tennis –24 –8 21 –1 0 –3 Time Pilot 3568 5229 65300 17250 427209 314947 Tutankham 11 168 5384 225 235 395 Up N Down 533 11693 82840 83743 5229


229 65300 17250 427209 314947 Tutankham 11 168 5384 225 235 395 Up N Down 533 11693 82840 83743 522962 614065 Venture 0 1188 38900 953 0 0 Video Pinball 16257 17668 89218328 382306 775304 940631 Wizard Of Wor 564 4756 395300 10910 0 99136 Yars Revenge 3093 54577 15000105 137164 846061 675774 Zaxxon 32 9173 83700 13599 58115 78443 Gamer median (%) 0 100 3716 180 693 830 Gamer mean (%) 0 100 123001 892 3054 3381 Record mean (%) 0 13 100 21 66 74 Record mean capped (%) 0 13 100 21 34 38 Table 6: Atari scores. 26


ProcGen learning curves 0 8 16 24 32 Return Bigfish 0 3 6 9 12 Bossfight 3 6 9 Caveflyer 2 4 6 8 Chaser 3 6 9 Return Climber 6 8 10 Coinrun 3 6 9 Dodgeball 0 6 12 18 24 Fruitbot 2 4 6 8 Return Heist 2 4 6 8 Jumper 2 4 6 8 Leaper 2 4 6 8 Maze 4 8 12 Return Miner 0 25M 50M Env steps 2 4 6 8 10 Ninja 0 25M 50M Env steps 5 10 15 20 25 Plunder 0 25M 50M Env steps 0 8 16 24 Starpilot 0 25M 50M Env steps 0 0.2 0.3 0.5 0.6 Return Normalized Mean Dreamer PPG PPO Figure 11: ProcGen learning curves. 27


ProcGen scores Task Original PPO PPO PPG Dreamer Environment steps 50M 50M 50M 50M Bigfish 10.92 12.72 31.26 8.62 Bossfight 10.47 9.36 11.46 11.61 Caveflyer 6.03 6.71 10.02 9.42 Chaser 4.48 3.54 8.57 5.49 Climber 7.59 9.04 10.24 11.43 Coinrun 7.93 6.71 8.98 9.86 Dodgeball 4.80 3.44 10.31 10.93 Fruitbot 20.28 21.69 24.32 11.04 Heist 2.25 6.87 3.77 8.51 Jumper 5.09 6.13 5.84 9.17 Leaper 5.90 4.07 8.76 7.05 Maze 4.97 7.86 7.06 6.85 Miner 7.56 12.97 9.08 5.71 Ninja 6.16 3.62 9.38 9.82 Plunder 11.16 3.99 13.44 23.81 Starpilot 17.00 10.13 21.57 28.00 Normalized mean 41.16 42.80 64.89 66.01 Table 7: ProcGen scores. The PPO implementation we use throughout our paper under fixed hyperparameters performs on par or better than the original PPO, which its authors describe as highly tuned with near opt


rms on par or better than the original PPO, which its authors describe as highly tuned with near optimal hyperparameters37. 28


DMLab learning curves 0 50 100 150 Return Explore Goals Large 0 150 300 450 Explore Goals Small 20 40 60 Explore Objects Large 0 30 60 90 Explore Objects Small 0 15 30 45 Explore Rewards Few 20 40 60 Return Explore Rewards Many 0 25 50 75 Explore Obstructed Large 0 80 160 240 Explore Obstructed Small 0 80 160 240 Language Quantitative 0 80 160 240 Language Execute 0 250 500 Return Language Described 0 200 400 600 Language Located -0.3 -0.2 0 Lasertag One Large -0.2 -0.1 0 0.1 Lasertag One Small 0 4 8 12 Lasertag Three Large 0 8 16 24 Return Lasertag Three Small 15 30 45 Natlab Fixed Large 16 24 32 Natlab Randomized 5 10 15 20 Natlab Regrowth 8 16 24 32 Psychlab Visumotor 24 28 32 36 Return Psychlab Continuous 0 15 30 45 Psychlab Sequential 0 25 50 75 Psychlab Visual Search 0 3 6 9 Rooms Co


sychlab Continuous 0 15 30 45 Psychlab Sequential 0 25 50 75 Psychlab Visual Search 0 3 6 9 Rooms Collect Good 10 20 30 40 Rooms Exploit Deferred 15 30 45 Return Rooms Keys Doors 0 50M 100M Env steps 0 25 50 75 Rooms Select Nonmatching 0 50M 100M Env steps 8 16 24 32 Rooms Watermaze 0 50M 100M Env steps 0 25 50 75 Skymaze Hard 0 50M 100M Env steps 25 50 75 Skymaze Varied 0 50M 100M Env steps 0.2 0.4 0.6 Return Human Mean Cap Dreamer IMPALA PPO Figure 12: DMLab learning curves. 29


DMLab scores Task R2D2+ IMPALA IMPALA IMPALA PPO Dreamer Environment steps 10B 10B 1B 100M 100M 100M Explore Goal Locations Large 174.7 316.0 137.8 64.2 21.2 116.7 Explore Goal Locations Small 460.7 482.0 302.8 196.1 115.1 372.8 Explore Object Locations Large 60.6 91.0 55.1 34.3 22.5 63.9 Explore Object Locations Small 83.7 100.4 75.9 50.6 38.9 93.5 Explore Object Rewards Few 80.7 92.6 46.9 34.6 19.0 40.0 Explore Object Rewards Many 75.8 89.4 68.5 53.3 25.5 58.1 Explore Obstructed Goals Large 95.5 102.0 57.9 23.7 12.9 52.8 Explore Obstructed Goals Small 311.9 372.0 214.9 118.0 70.4 224.6 Language Answer Quantitative Question 344.4 362.0 304.7 1.0 0.3 266.0 Language Execute Random Task 497.4 465.4 140.8 –44.4 –2.5 223.7 Language Select Described Object 617.6 664.0 618.4 0.2 526.9 665.5 Lang


465.4 140.8 –44.4 –2.5 223.7 Language Select Described Object 617.6 664.0 618.4 0.2 526.9 665.5 Language Select Located Object 772.8 731.4 413.0 38.2 397.5 679.5 Lasertag One Opponent Large 0.0 0.0 0.0 –0.1 0.0 –0.1 Lasertag One Opponent Small 31.8 0.0 0.0 –0.1 0.0 0.0 Lasertag Three Opponents Large 28.6 32.2 10.4 –0.1 0.0 9.0 Lasertag Three Opponents Small 49.0 57.2 37.1 12.1 0.4 18.8 Natlab Fixed Large Map 60.6 63.4 53.8 13.0 15.8 50.5 Natlab Varying Map Randomized 42.4 47.0 40.5 35.3 29.6 31.2 Natlab Varying Map Regrowth 24.6 34.0 25.5 15.3 16.3 16.7 Psychlab Arbitrary Visuomotor Mapping 33.1 38.4 16.5 11.9 16.0 30.7 Psychlab Continuous Recognition 30.0 28.6 30.0 30.1 30.5 33.8 Psychlab Sequential Comparison 30.0 29.6 0.0 0.0 30.0 44.3 Psychlab Visual Search 79.9 80.0 0.0 0.0 76.6 40.1


equential Comparison 30.0 29.6 0.0 0.0 30.0 44.3 Psychlab Visual Search 79.9 80.0 0.0 0.0 76.6 40.1 Rooms Collect Good Objects Test 9.9 10.0 9.9 9.3 9.7 9.9 Rooms Exploit Deferred Effects Test 38.1 62.2 37.6 34.5 39.0 40.4 Rooms Keys Doors Puzzle 46.2 54.6 36.9 24.2 26.0 42.4 Rooms Select Nonmatching Object 63.6 39.0 63.2 4.0 2.7 73.1 Rooms Watermaze 49.0 47.0 50.1 23.6 21.2 26.1 Skymaze Irreversible Path Hard 76.0 80.0 46.4 7.7 0.0 80.9 Skymaze Irreversible Path Varied 76.0 100.0 69.8 32.7 31.3 87.7 Human mean capped (%) 85.4 85.1 66.3 31.0 35.9 71.4 Table 8: DMLab scores at 100M environment steps and larger budgets. The IMPALA agent corresponds to “IMPALA (deep)” presented by Kapturowski et al. 35 who made the learning curves available. 30


Atari100k learning curves 400 800 1.2K Return Alien 0 40 80 120 Amidar 200 400 600 800 Assault 400 800 1.2K Asterix 0 200 400 600 Return Bank Heist 0 8K 16K 24K Battle Zone 0 30 60 90 Boxing 3 6 9 12 Breakout 800 1.6K 2.4K Return Chopper Command 0 40K 80K 120K Crazy Climber 0 400 800 1.2K Demon Attack 0 3 6 Freeway 0 2K 4K 6K Return Frostbite 0 1.5K 3K 4.5K Gopher 0 5K 10K 15K Hero 0 250 500 750 Jamesbond 0 2K 4K 6K Return Kangaroo 2.5K 5K 7.5K Krull 0 15K 30K Kung Fu Master 600 1.2K 1.8K Ms Pacman -16 -8 0 Return Pong 0 2K 4K Private Eye 0 1.5K 3K Qbert 0 8K 16K 24K Road Runner 0 200K 400K Env steps 0 400 800 1.2K Return Seaquest 0 200K 400K Env steps 0 60K 120K 180K Up N Down 0 200K 400K Env steps 0 0.4 0.8 1.2 Gamer Mean 0 200K 400K Env steps 0 0.2 0.3 0.5 Gamer Median Dreamer PPO Figur


nv steps 0 0.4 0.8 1.2 Gamer Mean 0 200K 400K Env steps 0 0.2 0.3 0.5 Gamer Median Dreamer PPO Figure 13: Atari100k learning curves. 31


Atari100k scores Task Random Human PPO SimPLe SPR TWM IRIS Dreamer Environment steps — — 400K 400K 400K 400K 400K 400K Alien 228 7128 276 617 842 675 420 1118 Amidar 6 1720 26 74 180 122 143 97 Assault 222 742 327 527 566 683 1524 683 Asterix 210 8503 292 1128 962 1117 854 1062 Bank Heist 14 753 14 34 345 467 53 398 Battle Zone 2360 37188 2233 4031 14834 5068 13074 20300 Boxing 0 12 3 8 36 78 70 82 Breakout 2 30 3 16 20 20 84 10 Chopper Command 811 7388 1005 979 946 1697 1565 2222 Crazy Climber 10780 35829 14675 62584 36700 71820 59324 86225 Demon Attack 152 1971 160 208 518 350 2034 577 Freeway 0 30 2 17 19 24 31 0 Frostbite 65 4335 127 237 1171 1476 259 3377 Gopher 258 2412 368 597 661 1675 2236 2160 Hero 1027 30826 2596 2657 5859 7254 7037 13354 Jamesbond 29 303 41 100 366 362 463 540 K


5 2236 2160 Hero 1027 30826 2596 2657 5859 7254 7037 13354 Jamesbond 29 303 41 100 366 362 463 540 Kangaroo 52 3035 55 51 3617 1240 838 2643 Krull 1598 2666 3222 2205 3682 6349 6616 8171 Kung Fu Master 258 22736 2090 14862 14783 24555 21760 25900 Ms Pacman 307 6952 366 1480 1318 1588 999 1521 Pong –21 15 –20 13 –5 19 15 –4 Private Eye 25 69571 100 35 86 87 100 3238 Qbert 164 13455 317 1289 866 3331 746 2921 Road Runner 12 7845 602 5641 12213 9109 9615 19230 Seaquest 68 42055 305 683 558 774 661 962 Up N Down 533 11693 1502 3350 10859 15982 3546 46910 Gamer mean (%) 0 100 11 33 62 96 105 125 Gamer median (%) 0 100 2 13 40 51 29 49 Table 9: Atari100k scores at 400K environment steps, corresponding to 100k agent steps.


40 51 29 49 Table 9: Atari100k scores at 400K environment steps, corresponding to 100k agent steps. Setting SimPLe EffMuZero SPR IRIS TWM Dreamer Gamer score (%) 33 190 62 105 96 125 Gamer median (%) 13 109 40 29 51 49 GPU days 5.0 0.6 0.1 3.5 0.4 0.1 Online planning — X — — — — Data augmentation — — X — — — Non-uniform replay — X X — X — Separate hparams — — — X — — Increased resolution — X X — — — Uses life information — X X X X — Uses early resets — X — X — — Separate eval episodes X X X X X — Table 10: Evaluation protocols for the Atari 100k benchmark. Computational resources are converted to A100 GPU days. EfficientMuZero44 achieves the highest scores but changed the environment configuration from the standard17.


MuZero44 achieves the highest scores but changed the environment configuration from the standard17. IRIS uses a separate hyperparameter for its exploration strength on Freeway. 32


Proprioceptive control learning curves 0 60 120 180 240 Return Acrobot Swingup 0 300 600 900 Ball In Cup Catch 400 600 800 1K Cartpole Balance 0 250 500 750 1K Cartpole Bal. Sparse 0 200 400 600 800 Return Cartpole Swingup 0 250 500 750 Cartpole Sw. Sparse 0 200 400 600 Cheetah Run 0 250 500 750 1K Finger Spin 0 250 500 750 1K Return Finger Turn Easy 0 300 600 900 Finger Turn Hard 0 40 80 120 160 Hopper Hop 0 250 500 750 1K Hopper Stand 0 250 500 750 1K Return Pendulum Swingup 0 250 500 750 1K Reacher Easy 0 250 500 750 1K Reacher Hard 0 200 400 600 Walker Run 0 250K 500K Env steps 250 500 750 1K Return Walker Stand 0 250K 500K Env steps 0 250 500 750 1K Walker Walk 0 250K 500K Env steps 200 400 600 Task Mean 0 250K 500K Env steps 0 200 400 600 800 Task Median Dreamer D4PG DMPO PPO Figure


400 600 Task Mean 0 250K 500K Env steps 0 200 400 600 800 Task Median Dreamer D4PG DMPO PPO Figure 14: DeepMind Control Suite learning curves under proprioceptive inputs. 33


Proprioceptive control scores Task PPO DDPG DMPO D4PG Dreamer Environment steps 500K 500K 500K 500K 500K Acrobot Swingup 6 100 103 124 134 Ball In Cup Catch 632 917 968 968 962 Cartpole Balance 523 997 999 999 990 Cartpole Balance Sparse 930 992 999 974 990 Cartpole Swingup 240 864 860 875 852 Cartpole Swingup Sparse 7 703 438 752 491 Cheetah Run 82 596 650 624 614 Finger Spin 18 775 769 823 931 Finger Turn Easy 281 499 620 612 793 Finger Turn Hard 106 313 495 421 889 Hopper Hop 0 36 68 80 113 Hopper Stand 3 484 549 762 576 Pendulum Swingup 1 767 834 759 788 Reacher Easy 494 934 961 960 954 Reacher Hard 288 949 968 937 938 Walker Run 31 561 493 616 649 Walker Stand 159 965 975 947 964 Walker Walk 64 952 942 969 936 Task mean 94 771 801 792 871 Task median 215 689 705 733 754 Table 11: Deep


Walk 64 952 942 969 936 Task mean 94 771 801 792 871 Task median 215 689 705 733 754 Table 11: DeepMind Control Suite scores under proprioceptive inputs. 34


Visual control learning curves 0 80 160 240 Return Acrobot Swingup 0 250 500 750 1K Ball In Cup Catch 400 600 800 1K Cartpole Balance 0 250 500 750 1K Cartpole Bal. Sparse 0 200 400 600 800 Return Cartpole Swingup 0 250 500 750 Cartpole Sw. Sparse 0 200 400 600 800 Cheetah Run 0 250 500 750 1K Finger Spin 200 400 600 800 Return Finger Turn Easy 0 250 500 750 Finger Turn Hard 0 100 200 300 Hopper Hop 0 250 500 750 1K Hopper Stand 0 250 500 750 Return Pendulum Swingup 150 300 450 600 Quadruped Run 0 200 400 600 800 Quadruped Walk 0 250 500 750 1K Reacher Easy 0 250 500 750 Return Reacher Hard 0 200 400 600 800 Walker Run 0 500K 1M Env steps 250 500 750 1K Walker Stand 0 500K 1M Env steps 0 250 500 750 1K Walker Walk 0 500K 1M Env steps 200 400 600 800 Return Task Mean 0 500K 1M Env steps 0 2


500 750 1K Walker Walk 0 500K 1M Env steps 200 400 600 800 Return Task Mean 0 500K 1M Env steps 0 200 400 600 800 Task Median Dreamer DrQ-v2 CURL PPO Figure 15: DeepMind Control Suite learning curves under visual inputs. 35


Visual control scores Task PPO SAC CURL DrQ-v2 Dreamer Environment steps 1M 1M 1M 1M 1M Acrobot Swingup 3 4 4 166 229 Ball In Cup Catch 829 176 970 928 972 Cartpole Balance 516 937 980 992 993 Cartpole Balance Sparse 881 956 999 987 964 Cartpole Swingup 290 706 771 863 861 Cartpole Swingup Sparse 1 149 373 773 759 Cheetah Run 95 20 502 716 836 Finger Spin 118 291 880 862 589 Finger Turn Easy 253 200 340 525 878 Finger Turn Hard 79 94 231 247 904 Hopper Hop 0 0 164 221 227 Hopper Stand 4 5 777 903 903 Pendulum Swingup 1 592 413 843 744 Quadruped Run 88 54 149 450 617 Quadruped Walk 112 49 121 726 811 Reacher Easy 487 67 689 944 951 Reacher Hard 94 7 472 670 862 Walker Run 30 27 360 539 684 Walker Stand 161 143 486 978 976 Walker Walk 87 40 822 768 961 Task mean 94 81 479 770 861 Task median


lker Stand 161 143 486 978 976 Walker Walk 87 40 822 768 961 Task mean 94 81 479 770 861 Task median 206 226 525 705 786 Table 12: DeepMind Control Suite scores under visual inputs. 36


BSuite performance spectrum Basic Credit Assignment Exploration Generalization Memory Noise Scale Dreamer Boot DQN DQN PPO Figure 16: BSuite scores visualized by category48. Dreamer exceeds previous methods in the categories scale and memory. The scale category measure robustness to reward scales. 37


BSuite scores Task Random PPO AC-RNN DQN Boot DQN Dreamer Bandit 0.00 0.38 1.00 0.93 0.98 0.96 Bandit Noise 0.00 0.61 0.63 0.71 0.80 0.75 Bandit Scale 0.00 0.39 0.60 0.74 0.83 0.78 Cartpole 0.04 0.84 0.40 0.85 0.69 0.93 Cartpole Noise 0.04 0.77 0.20 0.82 0.69 0.93 Cartpole Scale 0.04 0.83 0.12 0.72 0.65 0.92 Cartpole Swingup 0.00 0.00 0.00 0.00 0.15 0.03 Catch 0.00 0.91 0.87 0.92 0.99 0.96 Catch Noise 0.00 0.54 0.27 0.58 0.68 0.53 Catch Scale 0.00 0.90 0.17 0.85 0.65 0.94 Deep Sea 0.00 0.00 0.00 0.00 1.00 0.00 Deep Sea Stochastic 0.00 0.00 0.00 0.00 0.90 0.00 Discounting Chain 0.20 0.24 0.39 0.25 0.22 0.40 Memory Len 0.00 0.17 0.70 0.04 0.04 0.65 Memory Size 0.00 0.47 0.29 0.00 0.00 0.59 Mnist 0.05 0.77 0.56 0.85 0.85 0.61 Mnist Noise 0.05 0.41 0.22 0.38 0.34 0.34 Mnist Scale 0.05 0.76 0.0


st 0.05 0.77 0.56 0.85 0.85 0.61 Mnist Noise 0.05 0.41 0.22 0.38 0.34 0.34 Mnist Scale 0.05 0.76 0.09 0.49 0.31 0.55 Mountain Car 0.10 0.10 0.10 0.93 0.93 0.92 Mountain Car Noise 0.10 0.10 0.10 0.89 0.82 0.87 Mountain Car Scale 0.10 0.10 0.10 0.85 0.56 0.90 Umbrella Distract 0.00 1.00 0.09 0.30 0.26 0.74 Umbrella Length 0.00 0.87 0.43 0.39 0.39 0.78 Basic 0.04 0.60 0.58 0.90 0.89 0.88 Credit assignment 0.03 0.76 0.37 0.59 0.56 0.75 Exploration 0.00 0.00 0.00 0.00 0.68 0.01 Generalization 0.06 0.47 0.19 0.68 0.60 0.70 Memory 0.00 0.32 0.49 0.02 0.02 0.62 Noise 0.02 0.54 0.24 0.51 0.61 0.62 Scale 0.04 0.60 0.22 0.73 0.60 0.82 Task mean (%) 3 49 32 54 60 66 Category mean (%) 3 47 30 49 57 63 Table 13: BSuite scores for each task averaged over environment configurations, as well as aggregated


ble 13: BSuite scores for each task averaged over environment configurations, as well as aggregated performance by category and over all tasks. 38


Robustness ablations 0 10M 20M 0 400K 800K 1.2M Return Atari Atlantis 0 10M 20M 0 100 200 300 Atari Breakout 0 10M 20M 0 1K 2K Atari Montezuma 0 2.5M 5M 4 8 12 16 Crafter Reward 0 10M 20M 0 100 200 300 Return DMLab Goals Small 0 10M 20M 0 25 50 DMLab Nonmatching 0 5M 10M 0 600 1.2K 1.8K PinPad Five 0 5M 10M 0 500 1K 1.5K PinPad Six 0 5M 10M 0 3 6 9 Return ProcGen Bossfight 0 5M 10M 2 4 6 8 ProcGen Caveflyer 0 4M 0 150 300 Proprio Control Dog Run 0 5M 10M 0 300 600 900 Proprio Control Reacher Hard 0 5M 10M Env Steps 0 80 160 240 Return Visual Control Acrobot Sparse 0 4M Env Steps 0 40 80 120 Visual Control Humanoid Run 0 50 100 Env Steps (%) 0 25 50 75 100 Return (%) Normalized Task Mean Dreamer No obs symlog No retnorm (advnorm) No symexp twohot (Huber) No KL balance & free bits Without al


mer No obs symlog No retnorm (advnorm) No symexp twohot (Huber) No KL balance & free bits Without all Figure 17: Individual learning curves for the robustness ablation experiment. All robustness techniques contribute to the overall performance of Dreamer, although each individual technique may only improve the performance on a subset of the tasks. 39


Learning signal ablations 0 20M 0 400K 800K 1.2M Return Atari Atlantis 0 20M 0 100 200 300 Atari Breakout 0 20M 0 800 1.6K 2.4K Atari Montezuma 0 5M 4 8 12 16 Crafter Reward 0 20M 0 100 200 300 Return DMLab Goals Small 0 20M 0 20 40 60 DMLab Nonmatching 0 10M 0 600 1.2K 1.8K PinPad Five 0 10M 0 500 1K PinPad Six 0 10M 0 3 6 9 Return ProcGen Bossfight 0 10M 2 4 6 8 ProcGen Caveflyer 0 6M 0 100 200 300 Proprio Control Dog Run 0 10M 0 300 600 900 Proprio Control Reacher Hard 0 10M Env Steps 0 100 200 300 Return Visual Control Acrobot Sparse 0 6M Env Steps 0 25 50 75 Visual Control Humanoid Run 0 100 Env Steps (%) 0 25 50 75 100 Return (%) Normalized Task Mean Dreamer No reward & value grads No reconstruction grads Figure 18: Individual learning curves for the learning signal ablation experime


reconstruction grads Figure 18: Individual learning curves for the learning signal ablation experiment. Dreamer relies predominantly on the undersupervised reconstruction objective of its world model and additional reward and value gradients further improve performance on a subset of tasks. 40


Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov∗† Archit Sharma∗† Eric Mitchell∗† Stefano Ermon†‡ Christopher D. Manning† Chelsea Finn† †Stanford University ‡CZ Biohub {rafailov,architsh,eric.mitchell}@cs.stanford.edu Abstract While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF).


d LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the


is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train. 1 Introduction Large unsupervised language models (LMs) trained on very large datasets acquire surprising capabilities [11, 7, 42, 8]. However, these models are trained on data generated by humans with a wide variety of goals, priorities, and skillsets.


ls are trained on data generated by humans with a wide variety of goals, priorities, and skillsets. Some of these goals and skillsets may not be desirable to imitate; for example, while we may want our AI coding assistant to understand common programming mistakes in order to correct them, nevertheless, when generating code, we would like to bias our model toward the (potentially rare) high-quality coding ability present in its training data. Similarly, we might want our language model to be aware of a common misconception believed by 50% of people, but we certainly do not want the model to claim this misconception to be true in 50% of queries about it! In other words, selecting the model’s desired responses and behavior from its very wide knowledge and abilities is crucial to building AI s


esired responses and behavior from its very wide knowledge and abilities is crucial to building AI systems that are safe, performant, and controllable [28]. While existing methods typically steer LMs to match human preferences using reinforcement learning (RL), ∗Equal contribution; more junior authors listed earlier. 37th Conference on Neural Information Processing Systems (NeurIPS 2023). arXiv:2305.18290v3 [cs.LG] 29 Jul 2024


Figure 1: DPO optimizes for human preferences while avoiding reinforcement learning. Existing methods for fine-tuning language models with human feedback first fit a reward model to a dataset of prompts and human preferences over pairs of responses, and then use RL to find a policy that maximizes the learned reward. In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification objective, fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form. we will show that the RL-based objective used by existing methods can be optimized exactly with a simple binary cross-entropy objective, greatly simplifying the preference learning pipeline.


with a simple binary cross-entropy objective, greatly simplifying the preference learning pipeline. At a high level, existing methods instill the desired behaviors into a language model using curated sets of human preferences representing the types of behaviors that humans find safe and helpful. This preference learning stage occurs after an initial stage of large-scale unsupervised pre-training on a large text dataset. While the most straightforward approach to preference learning is supervised fine-tuning on human demonstrations of high quality responses, the most successful class of methods is reinforcement learning from human (or AI) feedback (RLHF/RLAIF; [12, 2]). RLHF methods fit a reward model to a dataset of human preferences and then use RL to optimize a language model policy to p


ard model to a dataset of human preferences and then use RL to optimize a language model policy to produce responses assigned high reward without drifting excessively far from the original model. While RLHF produces models with impressive conversational and coding abilities, the RLHF pipeline is considerably more complex than supervised learning, involving training multiple LMs and sampling from the LM policy in the loop of training, incurring significant computational costs. In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning. We propose Direct Preference Optimization (DPO), an algorithm that implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a


that implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a KL-divergence constraint) but is simple to implement and straightforward to train. Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective. Like existing algorithms, DPO relies on a theoretical preference model (such as the Bradley-Terry model; [5]) that measures how well a given reward function aligns with empirical preference data. However, while existing methods use the preference model to define a preference loss to train a reward model and then train a policy that optimizes the l


del to define a preference loss to train a reward model and then train a policy that optimizes the learned reward model, DPO uses a change of variables to define the preference loss as a function of the policy directly. Given a dataset of human preferences over model responses, DPO can therefore optimize a policy using a simple binary cross entropy objective, producing the optimal policy to an implicit reward function fit to the preference data. Our main contribution is Direct Preference Optimization (DPO), a simple RL-free algorithm for training language models from preferences. Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using languag


g from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters. 2 Related Work Self-supervised language models of increasing scale learn to complete some tasks zero-shot [33] or with few-shot prompts [6, 27, 11]. However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and humanwritten completions [25, 38, 13, 41]. This ‘instruction-tuning’ procedure enables LLMs to generalize to instructions outside of the instruction-tuning set and generally increase their usability [13]. Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus sub


n judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation [20], summarization [40, 51], story-telling [51], and instruction-following [28, 34]. These methods first optimize a neural network reward function for compatibility with the dataset of preferences under a preference model such as the 2


Bradley-Terry model [5], then fine-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE [47], proximal policy optimization (PPO; [39]), or variants [34]. A closely-related line of work leverages LLMs fine-tuned for instruction following with human feedback to generate additional synthetic preference data for targeted attributes such as safety or harmlessness [2], using only weak supervision from humans in the form of a text rubric for the LLM’s annotations. These methods represent a convergence of two bodies of work: one body of work on training language models with reinforcement learning for a variety of objectives [35, 29, 48] and another body of work on general methods for learning from human preferences [12, 21].


, 29, 48] and another body of work on general methods for learning from human preferences [12, 21]. Despite the appeal of using relative human preferences, fine-tuning large language models with reinforcement learning remains a major practical challenge; this work provides a theoretically-justified approach to optimizing relative preferences without RL. Outside of the context of language, learning policies from preferences has been studied in both bandit and reinforcement learning settings, and several approaches have been proposed. Contextual bandit learning using preferences or rankings of actions, rather than rewards, is known as a contextual dueling bandit (CDB; [50, 14]). In the absence of absolute rewards, theoretical analysis of CDBs substitutes the notion of an optimal policy with


of absolute rewards, theoretical analysis of CDBs substitutes the notion of an optimal policy with a von Neumann winner, a policy whose expected win rate against any other policy is at least 50% [14]. However, in the CDB setting, preference labels are given online, while in learning from human preferences, we typically learn from a fixed batch of offline preference-annotated action pairs [49]. Similarly, preference-based RL (PbRL) learns from binary preferences generated by an unknown ‘scoring’ function rather than rewards [9, 37]. Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve first explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it [16, 9, 12, 36, 21].


latent scoring function (i.e. the reward model) and subsequently optimizing it [16, 9, 12, 36, 21]. We instead present a single stage policy learning approach that directly optimizes a policy to satisfy preferences. 3 Preliminaries We review the RLHF pipeline in Ziegler et al. (and later [40, 1, 28]). It usually includes three phases: 1) supervised fine-tuning (SFT); 2) preference sampling and reward learning and 3) RL optimization. SFT: RLHF typically begins by fine-tuning a pre-trained LM with supervised learning on high-quality data for the downstream task(s) of interest (dialogue, summarization, etc.), to obtain a model πSFT. Reward Modelling Phase: In the second phase the SFT model is prompted with prompts x to produce pairs of answers (y1, y2) ∼πSFT(y | x).


d phase the SFT model is prompted with prompts x to produce pairs of answers (y1, y2) ∼πSFT(y | x). These are then presented to human labelers who express preferences for one answer, denoted as yw ≻yl | x where yw and yl denotes the preferred and dispreferred completion amongst (y1, y2) respectively. The preferences are assumed to be generated by some latent reward model r∗(y, x), which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) [5] model being a popular choice (although more general Plackett-Luce ranking models [32, 23] are also compatible with the framework if we have access to several ranked answers). The BT model stipulates that the human preference distribution p∗can be written as: p∗(y1 ≻y2 | x) = exp (r∗(x, y1)) exp (


s that the human preference distribution p∗can be written as: p∗(y1 ≻y2 | x) = exp (r∗(x, y1)) exp (r∗(x, y1)) + exp (r∗(x, y2)). (1) Assuming access to a static dataset of comparisons D =  x(i), y(i) w , y(i) l N i=1 sampled from p∗, we can parametrize a reward model rϕ(x, y) and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss: LR(rϕ, D) = −E(x,yw,yl)∼D  log σ(rϕ(x, yw) −rϕ(x, yl))  (2) where σ is the logistic function. In the context of LMs, the network rϕ(x, y) is often initialized from the SFT model πSFT(y | x) with the addition of a linear layer on top of the final transformer layer that produces a single scalar prediction for the reward value [51].


of the final transformer layer that produces a single scalar prediction for the reward value [51]. To ensure a reward function with lower variance, prior works normalize the rewards, such that Ex,y∼D [rϕ(x, y)] = 0 for all x. RL Fine-Tuning Phase: During the RL phase, the learned reward function is used to provide feedback to the language model. Following prior works [17, 18], the optimization is formulated as max πθ Ex∼D,y∼πθ(y|x)  rϕ(x, y)  −βDKL  πθ(y | x) || πref(y | x)  , (3) 3


where β is a parameter controlling the deviation from the base reference policy πref, namely the initial SFT model πSFT. In practice, the language model policy πθ is also initialized to πSFT. The added constraint is important, as it prevents the model from deviating too far from the distribution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers. Due to the discrete nature of language generation, this objective is not differentiable and is typically optimized with reinforcement learning. The standard approach [51, 40, 1, 28] has been to construct the reward function r(x, y) = rϕ(x, y) −β(log πθ(y | x) −log πref(y | x)), and maximize using PPO [39].


reward function r(x, y) = rϕ(x, y) −β(log πθ(y | x) −log πref(y | x)), and maximize using PPO [39]. 4 Direct Preference Optimization Motivated by the challenges of applying reinforcement learning algorithms on large-scale problems such as fine-tuning language models, our goal is to derive a simple approach for policy optimization using preferences directly. Unlike prior RLHF methods, which learn a reward and then optimize it via RL, our approach leverages a particular choice of reward model parameterization that enables extraction of its optimal policy in closed form, without an RL training loop. As we will describe next in detail, our key insight is to leverage an analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functi


ward functions to optimal policies, which enables us to transform a loss function over reward functions into a loss function over policies. This change-of-variables approach avoids fitting an explicit, standalone reward model, while still optimizing under existing models of human preferences, such as the Bradley-Terry model. In essence, the policy network represents both the language model and the (implicit) reward. Deriving the DPO objective. We start with the same RL objective as prior work, Eq. 3, under a general reward function r. Following prior work [31, 30, 19, 15], it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective in Eq. 3 takes the form: πr(y | x) = 1 Z(x)πref(y | x) exp  1 β r(x, y)  , (4) where Z(x) = P y πref(y | x) ex


es the form: πr(y | x) = 1 Z(x)πref(y | x) exp  1 β r(x, y)  , (4) where Z(x) = P y πref(y | x) exp  1 β r(x, y)  is the partition function. See Appendix A.1 for a complete derivation. Even if we use the MLE estimate rϕ of the ground-truth reward function r∗, it is still expensive to estimate the partition function Z(x) [19, 15], which makes this representation hard to utilize in practice. However, we can rearrange Eq. 4 to express the reward function in terms of its corresponding optimal policy πr, the reference policy πref, and the unknown partition function Z(·). Specifically, we first take the logarithm of both sides of Eq. 4 and then with some algebra we obtain: r(x, y) = β log πr(y | x) πref(y | x) + β log Z(x).


of Eq. 4 and then with some algebra we obtain: r(x, y) = β log πr(y | x) πref(y | x) + β log Z(x). (5) We can apply this reparameterization to the ground-truth reward r∗and corresponding optimal model π∗. Fortunately, the Bradley-Terry model depends only on the difference of rewards between two completions, i.e., p∗(y1 ≻y2 | x) = σ(r∗(x, y1) −r∗(x, y2)). Substituting the reparameterization in Eq. 5 for r∗(x, y) into the preference model Eq. 1, the partition function cancels, and we can express the human preference probability in terms of only the optimal policy π∗and reference policy πref. Thus, the optimal RLHF policy π∗under the Bradley-Terry model satisfies the preference model: p∗(y1 ≻y2 | x) = 1 1 + exp  β log π∗(y2|x) πref(y2|x) −β log π∗(y1|x) πref(y1|x)  (6) The derivation is in


2 | x) = 1 1 + exp  β log π∗(y2|x) πref(y2|x) −β log π∗(y1|x) πref(y1|x)  (6) The derivation is in Appendix A.2. While Eq. 6 uses the Bradley-Terry model, we can similarly derive expressions under the more general Plackett-Luce models [32, 23], shown in Appendix A.3. Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parametrized policy πθ. Analogous to the reward modeling approach (i.e. Eq. 2), our policy objective becomes: LDPO(πθ; πref) = −E(x,yw,yl)∼D  log σ  β log πθ(yw | x) πref(yw | x) −β log πθ(yl | x) πref(yl | x)  . (7) This way, we fit an implicit reward using an alternative parameterization, whose optimal policy is simply πθ.


we fit an implicit reward using an alternative parameterization, whose optimal policy is simply πθ. Moreover, since our procedure is equivalent to fitting a reparametrized Bradley-Terry 4


model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution [4]. In Section 5, we further discuss theoretical properties of DPO in relation to other works. What does the DPO update do? For a mechanistic understanding of DPO, it is useful to analyze the gradient of the loss function LDPO. The gradient with respect to the parameters θ can be written as: ∇θLDPO(πθ; πref) = −βE(x,yw,yl)∼D  σ(ˆrθ(x, yl) −ˆrθ(x, yw)) | {z } higher weight when reward estimate is wrong  ∇θ log π(yw | x) | {z } increase likelihood of yw −∇θ log π(yl | x) | {z } decrease likelihood of yl  , where ˆrθ(x, y) = β log πθ(y|x) πref(y|x) is the reward implicitly defined by the language model πθ and reference model πref (more in Section 5).


he reward implicitly defined by the language model πθ and reference model πref (more in Section 5). Intuitively, the gradient of the loss function LDPO increases the likelihood of the preferred completions yw and decreases the likelihood of dispreferred completions yl. Importantly, the examples are weighed by how much higher the implicit reward model ˆrθ rates the dispreferred completions, scaled by β, i.e, how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naïve version of this method without the weighting coefficient can cause the language model to degenerate (Appendix Table 3). DPO outline.


e weighting coefficient can cause the language model to degenerate (Appendix Table 3). DPO outline. The general DPO pipeline is as follows: 1) Sample completions y1, y2 ∼πref(· | x) for every prompt x, label with human preferences to construct the offline dataset of preferences D = {x(i), y(i) w , yl)(i)}N i=1 and 2) optimize the language model πθ to minimize LDPO for the given πref and D and desired β. In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using πSFT, we initialize πref = πSFT whenever available. However, when πSFT is not available, we initialize πref by maximizing likelihood of preferred completions (x, yw), that is, πref = arg maxπ Ex,yw∼D [log


ref by maximizing likelihood of preferred completions (x, yw), that is, πref = arg maxπ Ex,yw∼D [log π(yw | x)]. This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and πref used by DPO. Further details related to the implementation and hyperparameters can be found in Appendix B. 5 Theoretical Analysis of DPO In this section, we give further interpretation of the DPO method, provide theoretical backing, and relate advantages of DPO to issues with actor critic algorithms used for RLHF (such as PPO [39]). 5.1 Your Language Model Is Secretly a Reward Model DPO is able to bypass both fitting an explicit reward and performing RL to learn the policy using a single maximum likelihood objective. Note the optimization objective Eq.


o learn the policy using a single maximum likelihood objective. Note the optimization objective Eq. 5 is equivalent to a Bradley-Terry model with a reward parameterization r∗(x, y) = β log π∗ θ(y|x) πref(y|x) and we optimize our parametric model πθ, equivalently to the reward model optimization in Eq. 2 under the change of variables. In this section we will build the theory behind this reparameterization, show that it does not constrain the class of learned reward models, and allows for the exact recovery of the optimal policy. We begin with by defining an equivalence relation between reward functions. Definition 1. We say that two reward functions r(x, y) and r′(x, y) are equivalent iff r(x, y) −r′(x, y) = f(x) for some function f.


ard functions r(x, y) and r′(x, y) are equivalent iff r(x, y) −r′(x, y) = f(x) for some function f. It is easy to see that this is indeed an equivalence relation, which partitions the set of reward functions into classes. We can state the following two lemmas: Lemma 1. Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution. Lemma 2. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem. The proofs are straightforward and we defer them to Appendix A.5. The first lemma is a well-known under-specification issue with the Plackett-Luce family of models [32]. Due to this under-specification, 5


we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 [4]. The second lemma states that all reward functions from the same class yield the same optimal policy, hence for our final objective, we are only interested in recovering an arbitrary reward function from the optimal class. We prove the following Theorem in Appendix A.6: Theorem 1. Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization r(x, y) = β log π(y|x) πref(y|x) for some model π(y | x) and a given reference model πref(y | x). Proof Sketch. Consider any reward function r(x, y), which induces a corresponding optimal model πr(y | x), specified by Eq. 4.


reward function r(x, y), which induces a corresponding optimal model πr(y | x), specified by Eq. 4. We will show that a reward function from the equivalence class of r can be represented using the reparameterization given above. We define the projection f as f(r; πref, β)(x, y) = r(x, y) −β log X y πref(y | x) exp  1 β r(x, y)  (8) The operator f simply normalizes the reward function with the logarithm of the partition function of πr. Since the added normalization term is only a function of the prefix x, f(r; πref, β)(x, y) is a reward function in the equivalence class of r(x, y). Finally, replacing r with the RHS of Eq. 5 (which holds for any reward function), we have f(r; πref, β)(x, y) = β log πr(y|x) πref(y|x).


Eq. 5 (which holds for any reward function), we have f(r; πref, β)(x, y) = β log πr(y|x) πref(y|x). That is, the projection f produces a member of the equivalence class of r with the desired form, and we do not lose any generality in our reward model from the proposed reparameterization. We can alternatively view Theorem 1 as specifying exactly which reward function within each equivalence class the DPO reparameterization selects, that is, the reward function satisfying: X y πref(y | x) exp  1 β r(x, y)  | {z } =π(y|x), using Thm. 1 reparam. = 1, (9) i.e., π(y | x) is a valid distribution (probabilities are positive and sum to 1). However, following Eq. 4, we can see that Eq. 9 is the partition function of the optimal policy induced by the reward function r(x, y).


that Eq. 9 is the partition function of the optimal policy induced by the reward function r(x, y). The key insight of the DPO algorithm is that we can impose certain constraints on the under-constrained Plackett-Luce (and Bradley-Terry in particular) family of preference models, such that we preserve the class of representable reward models, but explicitly make the optimal policy in Eq. 4 analytically tractable for all prompts x. 5.2 Instability of Actor-Critic Algorithms We can also use our framework to diagnose instabilities with standard actor-critic algorithms used for the RLHF, such as PPO. We follow the RLHF pipeline and focus on the RL fine-tuning step outlined in Section 3. We can draw connections to the control as inference framework [22] for the constrained RL problem outlined i


raw connections to the control as inference framework [22] for the constrained RL problem outlined in 3. We assume a parameterized model πθ(y | x) and minimize DKL[πθ(y|x) || π∗(y | x)] where π∗is the optimal policy from Eq. 7 induced by the reward function rϕ(y, x). With some algebra this leads to the optimization objective: max πθ Eπθ(y|x)  rϕ(x, y) −β log X y πref(y | x) exp  1 β rϕ(x, y)  | {z } f(rϕ,πref,β) −β log πθ(y | x) πref(y | x) | {z } KL  (10) This is the same objective optimized in prior works [51, 40, 1, 28] using the DPO-equivalent reward for the reward class of rϕ. In this setting, we can interpret the normalization term in f(rϕ, πref, β) as the soft value function of the reference policy πref.


t the normalization term in f(rϕ, πref, β) as the soft value function of the reference policy πref. While this term does not affect the optimal solution, without it, the policy gradient of the objective could have high variance, making learning unstable. We can accommodate for the normalization term using a learned value function, but that can also be difficult to optimize. Alternatively, prior works have normalized rewards using a human completion baseline, essentially a single sample Monte-Carlo estimate of the normalizing term. In contrast the DPO reparameterization yields a reward function that does not require any baselines. 6


0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 KL( ref) 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Reward IMDb Sentiment Generation DPO (Ours) Unlikelihood PPO (Our impl.) PPO-GT (Our impl.) PPO-GT (TRL) Preferred-FT 0.00 0.25 0.50 0.75 1.00 Sampling temperature 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Win rate TL;DR Summarization Win Rate vs Reference DPO PPO Preferred-FT SFT GPT-J Best of 128 Figure 2: Left. The frontier of expected reward vs KL to the reference policy. DPO provides the highest expected reward for all KL values, demonstrating the quality of the optimization. Right. TL;DR summarization win rates vs. human-written summaries, using GPT-4 as evaluator. DPO exceeds PPO’s best-case performance on summarization, while being more robust to changes in the sampling temperature.