-case performance on summarization, while being more robust to changes in the sampling temperature. 6 Experiments In this section, we empirically evaluate DPO’s ability to train policies directly from preferences. First, in a well-controlled text-generation setting, we ask: how efficiently does DPO trade off maximizing reward and minimizing KL-divergence with the reference policy, compared to common preference learning algorithms such as PPO? Next, we evaluate DPO’s performance on larger models and more difficult RLHF tasks, including summarization and dialogue. We find that with almost no tuning of hyperparameters, DPO tends to perform as well or better than strong baselines like RLHF with PPO as well as returning the best of N sampled trajectories under a learned reward function.


F with PPO as well as returning the best of N sampled trajectories under a learned reward function. Before presenting these results, we describe the experimental set-up; additional details are in Appendix C. Tasks. Our experiments explore three different open-ended text generation tasks. For all experiments, algorithms learn a policy from a dataset of preferences D =  x(i), y(i) w , y(i) l N i=1. In controlled sentiment generation, x is a prefix of a movie review from the IMDb dataset [24], and the policy must generate y with positive sentiment. In order to perform a controlled evaluation, for this experiment we generate preference pairs over generations using a pre-trained sentiment classifier, where p(positive | x, yw) > p(positive | x, yl).


erations using a pre-trained sentiment classifier, where p(positive | x, yw) > p(positive | x, yl). For SFT, we fine-tune GPT-2-large until convergence on reviews from the train split of the IMDB dataset (further details in App C.1). In summarization, x is a forum post from Reddit; the policy must generate a summary y of the main points in the post. Following prior work, we use the Reddit TL;DR summarization dataset [43] along with human preferences gathered by Stiennon et al.. We use an SFT model fine-tuned on human-written forum post summaries2 with the TRLX [44] framework for RLHF. The human preference dataset was gathered by Stiennon et al. on samples from a different, but similarly-trained, SFT model.


aset was gathered by Stiennon et al. on samples from a different, but similarly-trained, SFT model. Finally, in single-turn dialogue, x is a human query, which may be anything from a question about astrophysics to a request for relationship advice. A policy must produce an engaging and helpful response y to a user’s query; we use the Anthropic Helpful and Harmless dialogue dataset [1], containing 170k dialogues between a human and an automated assistant. Each transcript ends with a pair of responses generated by a large (although unknown) language model along with a preference label denoting the human-preferred response. In this setting, no pre-trained SFT model is available; we therefore fine-tune an off-the-shelf language model on only the preferred completions to form the SFT model.


fine-tune an off-the-shelf language model on only the preferred completions to form the SFT model. Evaluation. Our experiments use two different approaches to evaluation. In order to analyze the effectiveness of each algorithm in optimizing the constrained reward maximization objective, in the controlled sentiment generation setting we evaluate each algorithm by its frontier of achieved reward and KL-divergence from the reference policy; this frontier is computable because we have acccess to the ground-truth reward function (a sentiment classifier). However, in the real world, the ground truth reward function is not known; therefore, we evaluate algorithms with their win rate against a baseline policy, using GPT-4 as a proxy for human evaluation of summary quality and response helpfulness


line policy, using GPT-4 as a proxy for human evaluation of summary quality and response helpfulness in the summarization and single-turn dialogue settings, respectively. For summarization, we use reference summaries in the test set as the baseline; for dialogue, we use the preferred response in the 2https://huggingface.co/CarperAI/openai_summarize_tldr_sft 7


0.25 0.50 0.75 1.00 Sampling temperature 0.1 0.2 0.3 0.4 0.5 0.6 Win rate Anthropic-HH Dialogue Win Rate vs Chosen DPO Best of 128 Preferred-FT Pythia-2.8B 0 300 600 900 1200 1500 1800 2100 2400 2700 3000 3300 Fine-tuning step 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 Win rate Dialogue Win Rate Evolution DPO (temp = 1.0) DPO (temp = 0.7) Figure 3: Left. Win rates computed by GPT-4 for Anthropic-HH one-step dialogue; DPO is the only method that improves over chosen summaries in the Anthropic-HH test set. Right. Win rates for different sampling temperatures over the course of training. DPO’s improvement over the dataset labels is fairly stable over the course of training for different sampling temperatures. test dataset as the baseline.


able over the course of training for different sampling temperatures. test dataset as the baseline. While existing studies suggest LMs can be better automated evaluators than existing metrics [10], we conduct a human study to justify our usage of GPT-4 for evaluation in Sec. 6.4. We find GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement. Methods. In addition to DPO, we evaluate several existing approaches to training language models to adhere to human preferences. Most simply, we explore zero-shot prompting with GPT-J [45] in the summarization task and 2-shot prompting with Pythia-2.8B [3] in the dialogue task.


PT-J [45] in the summarization task and 2-shot prompting with Pythia-2.8B [3] in the dialogue task. In addition, we evaluate the SFT model as well as Preferred-FT, which is a model fine-tuned with supervised learning on the chosen completion yw from either the SFT model (in controlled sentiment and summarization) or a generic LM (in single-turn dialogue). Another pseudo-supervised method is Unlikelihood [46], which simply optimizes the policy to maximize the probability assigned to yw and minimize the probability assigned to yl; we use an optional coefficient α ∈[0, 1] on the ‘unlikelihood’ term. We also consider PPO [39] using a reward function learned from the preference data and PPO-GT, which is an oracle that learns from the ground truth reward function available in the controlled sent


hich is an oracle that learns from the ground truth reward function available in the controlled sentiment setting. In our sentiment experiments, we use two implementations of PPO-GT, one of-the-shelf version [44] as well as a modified version that normalizes rewards and further tunes hyperparameters to improve performance (we also use these modifications when running ‘normal’ PPO with learned rewards). Finally, we consider the Best of N baseline, sampling N responses from the SFT model (or Preferred-FT in dialogue) and returning the highest-scoring response according to a reward function learned from the preference dataset. This high-performing method decouples the quality of the reward model from the PPO optimization, but is computationally impractical even for moderate N as it requires s


l from the PPO optimization, but is computationally impractical even for moderate N as it requires sampling N completions for every query at test time. 6.1 How well can DPO optimize the RLHF objective? The KL-constrained reward maximization objective used in typical RLHF algorithms balances exploitation of reward while restricting the policy from deviating far from the reference policy. Therefore, when comparing algorithms, we must take into account both reward achieved as well as the KL discrepancy; achieving slightly higher reward but with much higher KL is not necessarily desirable. Figure 2 shows the reward-KL frontier for various algorithms in the sentiment setting. We execute multiple training runs for each algorithm, using a different hyperparameter for policy conservativeness in ea


training runs for each algorithm, using a different hyperparameter for policy conservativeness in each run (target KL ∈{3, 6, 9, 12} for PPO, β ∈{0.05, 0.1, 1, 5}, α ∈{0.05, 0.1, 0.5, 1} for unlikelihood, random seeds for preferred-FT). This sweep includes 22 runs in total. After each 100 training steps until convergence, we evaluate each policy on a set of test prompts, computing the average reward under the true reward function as well as the average sequence-level KL3 with the reference policy KL (π || πref). We find that DPO produces by far the most efficient frontier, achieving the highest reward while still achieving low KL. This result is particularly notable for multiple reasons. First, DPO and PPO optimize the same objective, but DPO is notably more efficient; 3That is, the sum of


st, DPO and PPO optimize the same objective, but DPO is notably more efficient; 3That is, the sum of the per-timestep KL-divergences. 8


DPO’s reward/KL tradeoff strictly dominates PPO. Second, DPO achieves a better frontier than PPO, even when PPO can access ground truth rewards (PPO-GT). 6.2 Can DPO scale to real preference datasets? Next, we evaluate fine-tuning performance of DPO on summarization and single-turn dialogue. For summarization, automatic evaluation metrics such as ROUGE can be poorly correlated with human preferences [40], and prior work has found that fine-tuning LMs using PPO on human preferences to provide more effective summaries. We evaluate different methods by sampling completions on the test split of TL;DR summarization dataset, and computing the average win rate against reference completions in the test set.


rization dataset, and computing the average win rate against reference completions in the test set. The completions for all methods are sampled at temperatures varying from 0.0 to 1.0, and the win rates are shown in Figure 2 (right). DPO, PPO and Preferred-FT all fine-tune the same GPT-J SFT model4. We find that DPO has a win rate of approximately 61% at a temperature of 0.0, exceeding the performance of PPO at 57% at its optimal sampling temperature of 0.0. DPO also achieves a higher maximum win rate compared to the best of N baseline. We note that we did not meaningfully tune DPO’s β hyperparameter, so these results may underestimate DPO’s potential. Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade to that of the base


obust to the sampling temperature than PPO, the performance of which can degrade to that of the base GPT-J model at high temperatures. Preferred-FT does not improve significantly over the SFT model. We also compare DPO and PPO head-to-head in human evaluations in Section 6.4, where DPO samples at temperature 0.25 were preferred 58% times over PPO samples at temperature 0. On single-turn dialogue, we evaluate the different methods on the subset of the test split of the Anthropic HH dataset [1] with one step of human-assistant interaction. GPT-4 evaluations use the preferred completions on the test as the reference to compute the win rate for different methods. As there is no standard SFT model for this task, we start with a pre-trained Pythia-2.8B, use Preferred-FT to train a reference mode


l for this task, we start with a pre-trained Pythia-2.8B, use Preferred-FT to train a reference model on the chosen completions such that completions are within distribution of the model, and then train using DPO. We also compare against the best of 128 Preferred-FT completions (we found the Best of N baseline plateaus at 128 completions for this task; see Appendix Figure 4) and a 2-shot prompted version of the Pythia-2.8B base model, finding DPO performs as well or better for the best-performing temperatures for each method. We also evaluate an RLHF model trained with PPO on the Anthropic HH dataset 5 from a well-known source 6, but are unable to find a prompt or sampling temperature that gives performance better than the base Pythia-2.8B model.


ind a prompt or sampling temperature that gives performance better than the base Pythia-2.8B model. Based on our results from TL;DR and the fact that both methods optimize the same reward function, we consider Best of 128 a rough proxy for PPO-level performance. Overall, DPO is the only computationally efficient method that improves over the preferred completions in the Anthropic HH dataset, and provides similar or better performance to the computationally demanding Best of 128 baseline. Finally, Figure 3 shows that DPO converges to its best performance relatively quickly. 6.3 Generalization to a new input distribution Win rate vs. ground truth Alg. Temp 0 Temp 0.25 DPO 0.36 0.31 PPO 0.26 0.23 Table 1: GPT-4 win rates vs.


ate vs. ground truth Alg. Temp 0 Temp 0.25 DPO 0.36 0.31 PPO 0.26 0.23 Table 1: GPT-4 win rates vs. ground truth summaries for out-of-distribution CNN/DailyMail input articles. To further compare the performance of PPO and DPO under distribution shifts, we evaluate the PPO and DPO policies from our Reddit TL;DR summarization experiment on a different distribution, news articles in the test split of the CNN/DailyMail dataset [26], using the best sampling temperatures from TL;DR (0 and 0.25). The results are presented in Table 1. We computed the GPT-4 win rate against the ground-truth summaries in the datasets, using the same GPT4 (C) prompt we used for Reddit TL;DR, but replacing the words “forum post” with “news article”.


GPT4 (C) prompt we used for Reddit TL;DR, but replacing the words “forum post” with “news article”. For this new distribution, DPO continues to outperform the PPO policy by a significant margin. This experiment provides initial evidence that DPO policies can generalize similarly well to PPO policies, even though DPO does not use the additional unlabeled Reddit TL;DR prompts that PPO uses. 4https://huggingface.co/CarperAI/openai_summarize_tldr_sft 5https://huggingface.co/reciprocate/ppo_hh_pythia-6B 6https://github.com/CarperAI/trlx/tree/main/examples/hh 9


6.4 Validating GPT-4 judgments with human judgments We conduct a human study to verify the reliability of GPT-4’s judgments, using the results of the TL;DR summarization experiment and two different GPT-4 prompts. The GPT-4 (S) (simple) prompt simply asks for which summary better-summarizes the important information in the post. The GPT-4 (C) (concise) prompt also asks for which summary is more concise; we evaluate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than humans do with the GPT-4 (S) prompt. See Appendix C.2 for the complete prompts. We perform three comparisons, using the highest (DPO, temp. 0.25), the lowest (PPO, temp. 1.0), and a DPO SFT PPO-1 N respondents 272 122 199 GPT-4 (S) win % 47 27 13 GPT-4 (C) win % 54 32 12 Human win % 58 43 17 GP


N respondents 272 122 199 GPT-4 (S) win % 47 27 13 GPT-4 (C) win % 54 32 12 Human win % 58 43 17 GPT-4 (S)-H agree 70 77 86 GPT-4 (C)-H agree 67 79 85 H-H agree 65 87 Table 2: Comparing human and GPT-4 win rates and per-judgment agreement on TL;DR summarization samples. Humans agree with GPT-4 about as much as they agree with each other. Each experiment compares a summary from the stated method with a summary from PPO with temperature 0. middle-performing (SFT, temp. 0.25) method with the aim of covering a diversity of sample qualities; all three methods are compared against greedilysampled PPO (its best-performing temperature). We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for h


ns about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations (due to limited human raters, we only collect multiple human judgments for the DPO and PPO-1 comparisons). Overall, the GPT-4 (C) prompt generally provides win rates more representative of humans; we therefore use this prompt for the main results in Section 6.2. For additional details about the human study, including the web interface presented to raters and the list of human volunteers, see Appendix D.3. 7 Discussion Learning from preferences is a powerful, scalable framework for training capable, aligned language models. We have introduced DPO, a simple training paradigm for training language models from preferences without reinforcement learning.


ple training paradigm for training language models from preferences without reinforcement learning. Rather than coercing the preference learning problem into a standard RL setting in order to use off-the-shelf RL algorithms, DPO identifies a mapping between language model policies and reward functions that enables training a language model to satisfy human preferences directly, with a simple cross-entropy loss, without reinforcement learning or loss of generality. With virtually no tuning of hyperparameters, DPO performs similarly or better than existing RLHF algorithms, including those based on PPO; DPO thus meaningfully reduces the barrier to training more language models from human preferences. Limitations & Future Work. Our results raise several important questions for future work.


ferences. Limitations & Future Work. Our results raise several important questions for future work. How does the DPO policy generalize out of distribution, compared with learning from an explicit reward function? Our initial results suggest that DPO policies can generalize similarly to PPO-based models, but more comprehensive study is needed. For example, can training with self-labeling from the DPO policy similarly make effective use of unlabeled prompts? On another front, how does reward over-optimization manifest in the direct preference optimization setting, and is the slight decrease in performance in Figure 3-right an instance of it? Additionally, while we evaluate models up to 6B parameters, exploration of scaling DPO to state-of-the-art models orders of magnitude larger is an excit


meters, exploration of scaling DPO to state-of-the-art models orders of magnitude larger is an exciting direction for future work. Regarding evaluations, we find that the win rates computed by GPT-4 are impacted by the prompt; future work may study the best way to elicit high-quality judgments from automated systems. Finally, many possible applications of DPO exist beyond training language models from human preferences, including training generative models in other modalities. Acknowledgements EM gratefully acknowledges funding from a Knight-Hennessy Graduate Fellowship. CF and CM are CIFAR Fellows. This work was supported in part by the Stanford Accelerator for Learning (SAL) and Stanford Institute for Human-Centered Artificial Intelligence (HAI) Generative AI for the Future of Learning s


nstitute for Human-Centered Artificial Intelligence (HAI) Generative AI for the Future of Learning seed grant program. The Stanford Center for Research on Foundation Models (CRFM) provided part of the compute resources used for the experiments in this work. This work was supported in part by ONR grant N00014-20-1-2675. 10


References [1] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. HatfieldDodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. [2] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N.


, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. [3] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. [4] H. Bong and A. Rinaldo.


for analyzing large language models across training and scaling, 2023. [4] H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luce model. International Conference on Machine Learning, 2022. arXiv:2110.11487. [5] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. doi: https://doi.org/10.2307/2334029. [6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I.


. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877– 1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_ files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [8] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y.


[8] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. arXiv preprint arXiv:2303.12712. [9] R. Busa-Fekete, B. Szörényi, P. Weng, W. Cheng, and E. Hüllermeier. Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine Learning, 97(3):327–351, July 2014. doi: 10.1007/s10994-014-5458-8. URL https://doi.org/10.1007/s10994-014-5458-8. [10] Y. Chen, R. Wang, H. Jiang, S. Shi, and R.-L. Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. ArXiv, abs/2304.00723, 2023.


reference-free text quality evaluation: A preliminary empirical study. ArXiv, abs/2304.00723, 2023. [11] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [12] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf. 11


[13] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction-finetuned language models, 2022. [14] M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 563–587, Paris, France, 03–06 Jul 2015. PMLR. URL https://proceedings.mlr.press/v40/Dudik15.html. [15] D. Go, T.


s, France, 03–06 Jul 2015. PMLR. URL https://proceedings.mlr.press/v40/Dudik15.html. [15] D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023. [16] A. Jain, B. Wojcik, T. Joachims, and A. Saxena. Learning trajectory preferences for manipulators via iterative improvement. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/ 2013/file/c058f544c737782deacefa532d9add4c-Paper.pdf. [17] N. Jaques, S. Gu, D. Bahdanau, J. M.


er/ 2013/file/c058f544c737782deacefa532d9add4c-Paper.pdf. [17] N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pages 1645–1654. PMLR, 2017. [18] N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848, 2020. [19] T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A.


with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16203–16220. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67496dfa96afddab795530cc7c69b57a-Paper-Conference.pdf. [20] J. Kreutzer, J. Uyheng, and S. Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777–1788, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/ P18-1165. URL https://aclanthology.org/P18-1165. [21] A. Kupcsik, D. Hsu, and W.


doi: 10.18653/v1/ P18-1165. URL https://aclanthology.org/P18-1165. [21] A. Kupcsik, D. Hsu, and W. S. Lee. Learning Dynamic Robot-to-Human Object Handover from Human Feedback, pages 161–176. Springer International Publishing, 01 2018. ISBN 978-3-319-51531-1. doi: 10.1007/978-3-319-51532-8_10. [22] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018. [23] R. D. Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012. [24] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011.


tational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P11-1015. [25] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long. 244. URL https://aclanthology.org/2022.acl-long.244. 12


[26] R. Nallapati, B. Zhou, C. dos Santos, Ç. Gulçehre, and B. Xiang. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10.18653/v1/K16-1028. URL https:// aclanthology.org/K16-1028. [27] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21, New York, NY, USA, 2021.


for High Performance Computing, Networking, Storage and Analysis, SC ’21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384421. doi: 10.1145/3458817.3476209. URL https://doi.org/10.1145/3458817.3476209. [28] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022.


Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf. [29] R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=HkAClQgA-. [30] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. [31] J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control.


and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007. [32] R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193–202, 1975. doi: https://doi.org/10.2307/2346567. [33] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners, 2019. Ms., OpenAI. [34] R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.


ge processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=8aHzds2uUyB. [35] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732, 2015. [36] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems (RSS), 2017. [37] A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of M


h International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 6263–6289. PMLR, 25–27 Apr 2023. URL https://proceedings.mlr.press/v206/saha23a.html. [38] V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.


ables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4. 13


[39] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. [40] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize from human feedback, 2022. [41] R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-C. Chang, I. Krivokon, W. Rusch, M. Pickett, P. Srinivasan, L. Man, K. Meier-Hellstern, M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E.


. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. AgueraArcas, C. Cui, M. Croak, E. Chi, and Q. Le. Lamda: Language models for dialog applications, 2022. [42] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [43] M. Völske, M. Potthast, S. Syed, and B. Stein. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark, Sept. 2017.


gs of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4508. URL https://aclanthology.org/W17-4508. [44] L. von Werra, J. Tow, reciprocated, S. Matiana, A. Havrilla, cat state, L. Castricato, Alan, D. V. Phung, A. Thakur, A. Bukhtiyarov, aaronrmm, F. Milo, Daniel, D. King, D. Shin, E. Kim, J. Wei, M. Romero, N. Pochinkov, O. Sanseviero, R. Adithyan, S. Siu, T. Simonini, V. Blagojevic, X. Song, Z. Witten, alexandremuzio, and crumb. CarperAI/trlx: v0.6.0: LLaMa (Alpaca), Benchmark Util, T5 ILQL, Tests, Mar. 2023. URL https://doi.org/10.5281/zenodo. 7790115. [45] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.


15. [45] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021. [46] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019. [47] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3–4):229–256, may 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696. [48] Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Confer


e on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018. ISBN 978-1-57735-800-8. [49] X. Yan, C. Luo, C. L. A. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 567–577, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. doi: 10.1145/3477495.3531991. URL https://doi.org/10.1145/3477495.3531991. [50] Y. Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits problem.


531991. [50] Y. Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538–1556, 2012. ISSN 0022-0000. doi: https: //doi.org/10.1016/j.jcss.2011.12.028. URL https://www.sciencedirect.com/science/ article/pii/S0022000012000281. JCSS Special Issue: Cloud Computing 2011. [51] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences, 2020. 14


Author Contributions All authors provided valuable contributions to designing, analyzing, and iterating on experiments, writing and editing the paper, and generally managing the project’s progress. RR proposed using autoregressive reward models in discussions with EM; derived the DPO objective; proved the theoretical properties of the algorithm and wrote the relevant sections and appendices. He also suggested and helped with organizing experiments and contributed some of the PPO and reward learning baselines. AS initiated the discussion on using weighted regression methods as an alternative to PPO; initiated project-related organization, wrote initial analysis connecting DPO with weighted regression and unlikelihood; design and iterations of DPO + baseline implementations, initial explorat


gression and unlikelihood; design and iterations of DPO + baseline implementations, initial exploratory experiments for DPO; substantial experiment organization and design (datasets, baselines, evaluation); led model training and evaluation for controlled sentiment generation and summarization; design iterations for GPT-4 evaluation (particularly summarization); substantial writing contributions to abstract, prelims/method and experiments; editing contributions to other sections. EM provided input on early discussions on learning autoregressive reward functions; wrote the first implementation of DPO and ran the first DPO experiments; trained the large-scale (summarization and dialogue) DPO models used in paper experiments; conducted initial GPT-4 win rate evaluations and set up related inf


odels used in paper experiments; conducted initial GPT-4 win rate evaluations and set up related infrastructure; recruited participants for, conducted, and analyzed results from the human study; wrote the abstract, introduction, related work, discussion, and most of experiments; and assisted with editing the rest of the paper. CF, CM, & SE supervised the research, suggested ideas and experiments, and assisted in writing the paper. A Mathematical Derivations A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective In this appendix, we will derive Eq. 4. Analogously to Eq. 3, we optimize the following objective: max π Ex∼D,y∼π  r(x, y)  −βDKL  π(y|x)||πref(y|x)  (11) under any reward function r(x, y), reference model πref and a general non-parametric policy class.


under any reward function r(x, y), reference model πref and a general non-parametric policy class. We now have: max π Ex∼D,y∼π  r(x, y)  −βDKL  π(y|x) || πref(y|x)  = max π Ex∼DEy∼π(y|x)  r(x, y) −β log π(y|x) πref(y|x)  = min π Ex∼DEy∼π(y|x)  log π(y|x) πref(y|x) −1 β r(x, y)  = min π Ex∼DEy∼π(y|x)  log π(y|x) 1 Z(x)πref(y|x) exp  1 β r(x, y)  −log Z(x)   (12) where we have partition function: Z(x) = X y πref(y|x) exp  1 β r(x, y)  . Note that the partition function is a function of only x and the reference policy πref, but does not depend on the policy π. We can now define π∗(y|x) = 1 Z(x)πref(y|x) exp  1 β r(x, y)  , 15


which is a valid probability distribution as π∗(y|x) ≥0 for all y and P y π∗(y|x) = 1. Since Z(x) is not a function of y, we can then re-organize the final objective in Eq 12 as: min π Ex∼D  Ey∼π(y|x)  log π(y|x) π∗(y|x)  −log Z(x)  = (13) min π Ex∼D [DKL(π(y|x) || π∗(y|x)) −log Z(x)] (14) Now, since Z(x) does not depend on π, the minimum is achieved by the policy that minimizes the first KL term. Gibbs’ inequality tells us that the KL-divergence is minimized at 0 if and only if the two distributions are identical. Hence we have the optimal solution: π(y|x) = π∗(y|x) = 1 Z(x)πref(y|x) exp  1 β r(x, y)  (15) for all x ∈D. This completes the derivation. A.2 Deriving the DPO Objective Under the Bradley-Terry Model It is straightforward to derive the DPO objective under the Bradley-Terry


er the Bradley-Terry Model It is straightforward to derive the DPO objective under the Bradley-Terry preference model as we have p∗(y1 ≻y2|x) = exp (r∗(x, y1)) exp (r∗(x, y1)) + exp (r∗(x, y2)) (16) In Section 4 we showed that we can express the (unavailable) ground-truth reward through its corresponding optimal policy: r∗(x, y) = β log π∗(y|x) πref(y|x) + β log Z(x) (17) Substituting Eq. 17 into Eq. 16 we obtain: p∗(y1 ≻y2|x) = exp  β log π∗(y1|x) πref(y1|x) + β log Z(x)  exp  β log π∗(y1|x) πref(y1|x) + β log Z(x)  + exp  β log π∗(y2|x) πref(y2|x) + β log Z(x)  = 1 1 + exp  β log π∗(y2|x) πref(y2|x) −β log π∗(y1|x) πref(y1|x)  = σ  β log π∗(y1|x) πref(y1|x) −β log π∗(y2|x) πref(y2|x)  . The last line is the per-instance loss in Equation 7.


|x) πref(y1|x) −β log π∗(y2|x) πref(y2|x)  . The last line is the per-instance loss in Equation 7. A.3 Deriving the DPO Objective Under the Plackett-Luce Model The Plackett-Luce model [32, 23] is a generalization of the Bradley-Terry model over rankings (rather than just pair-wise comparisons). Similar to to the Bradley-Terry model, it stipulates that when presented with a set of possible choices, people prefer a choice with probability proportional to the value of some latent reward function for that choice. In our context, when presented with a prompt x and a set of K answers y1, . . . , yK a user would output a permutation τ : [K] →[K], giving their ranking of the answers. The Plackett-Luce model stipulates that p∗(τ|y1, . . .


] →[K], giving their ranking of the answers. The Plackett-Luce model stipulates that p∗(τ|y1, . . . , yK, x) = K Y k=1 exp(r∗(x, yτ(k))) PK j=k exp(r∗(x, yτ(j))) (18) Notice that when K = 2, Equation 18 reduces to the Bradley-Terry model. However, for the general Plackett-Luce model, we can still utilize the results of Eq. 5 and substitute the reward function parameterized by its optimal policy. Similarly to Appendix A.2, the normalization constant Z(x) cancels out and we’re left with: p∗(τ|y1, . . . , yK, x) = K Y k=1 exp  β log π∗(yτ(k)|x) πref(yτ(k)|x)  PK j=k exp  β log π∗(yτ(j)|x) πref(yτ(j)|x)  (19) 16


Similarly to the approach of Section 4, if we have access to a dataset D = {τ (i), y(i) 1 , . . . , y(i) K , x(i)}N i=1 of prompts and user-specified rankings, we can use a parameterized model and optimize this objective with maximum-likelihood.: LDPO(πθ, πref) = −Eτ,y1,...,yK,x∼D  log K Y k=1 exp  β log πθ(yτ(k)|x) πref(yτ(k)|x)  PK j=k exp  β log πθ(yτ(j)|x) πref(yτ(j)|x)    (20) A.4 Deriving the Gradient of the DPO Objective In this section we derive the gradient of the DPO objective: ∇θLDPO(πθ; πref) = −∇θE(x,yw,yl)∼D  log σ  β log πθ(yl|x) πref(yl|x) −β log πθ(yw|x) πref(yw|x)  (21) We can rewrite the RHS of Equation 21 as ∇θLDPO(πθ; πref) = −E(x,yw,yl)∼D σ′ (u) σ (u) ∇θ (u)  , (22) where u = β log πθ(yl|x) πref(yl|x) −β log πθ(yw|x) πref(yw|x).


yl)∼D σ′ (u) σ (u) ∇θ (u)  , (22) where u = β log πθ(yl|x) πref(yl|x) −β log πθ(yw|x) πref(yw|x). Using the properties of sigmoid function σ′(x) = σ(x)(1 −σ(x)) and σ(−x) = 1 −σ(x), we obtain the final gradient ∇θLDPO(πθ; πref) = −E(x,yw,yl)∼D  βσ  β log πθ(yw|x) πref(yw|x) −β log πθ(yl|x) πref(yl|x)   ∇θ log π(yw | x)−∇θ log π(yl | x)  , After using the reward substitution of ˆrθ(x, y) = β log πθ(y|x) πref(y|x) we obtain the final form of the gradient from Section 4. A.5 Proof of Lemma 1 and 2 In this section, we will prove the two lemmas from Section 5. Lemma 1 Restated. Under the Plackett-Luce preference framework, and in particular the BradleyTerry framework, two reward functions from the same equivalence class induce the same preference distribution. Proof.


wo reward functions from the same equivalence class induce the same preference distribution. Proof. We say that two reward functions r(x, y) and r′(x, y) are from the same equivalence class if r′(x, y) = r(x, y) + f(x) for some function f. We consider the general Plackett-Luce (with the Bradley-Terry model a special case for K = 2) and denote the probability distribution over rankings induced by a particular reward function r(x, y) as pr. For any prompt x, answers y1, . . . , yK and ranking τ we have: pr′(τ|y1, . . . , yK, x) = K Y k=1 exp(r′(x, yτ(k))) PK j=k exp(r′(x, yτ(j))) = K Y k=1 exp(r(x, yτ(k)) + f(x)) PK j=k exp(r(x, yτ(j)) + f(x)) = K Y k=1 exp(f(x)) exp(r(x, yτ(k))) exp(f(x)) PK j=k exp(r(x, yτ(j))) = K Y k=1 exp(r(x, yτ(k))) PK j=k exp(r(x, yτ(j))) = pr(τ|y1, . . .


p(f(x)) PK j=k exp(r(x, yτ(j))) = K Y k=1 exp(r(x, yτ(k))) PK j=k exp(r(x, yτ(j))) = pr(τ|y1, . . . , yK, x), which completes the proof. 17


Lemma 2 Restated. Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem. Proof. Let us consider two reward functions from the same class, such that r′(x, y) = r(x, y) + f(x) and, let us denote as πr and πr′ the corresponding optimal policies. By Eq. 4, for all x, y we have πr′(y|x) = 1 P y πref(y|x) exp  1 β r′(x, y) πref(y|x) exp  1 β r′(x, y)  = 1 P y πref(y|x) exp  1 β (r(x, y) + f(x)) πref(y|x) exp  1 β (r(x, y) + f(x))  = 1 exp  1 β f(x)  P y πref(y|x) exp  1 β r(x, y) πref(y|x) exp  1 β r(x, y)  exp  1 β f(x)  = 1 P y πref(y|x) exp  1 β r(x, y) πref(y|x) exp  1 β r(x, y)  = πr(y|x), which completes the proof. A.6 Proof of Theorem 1 In this section, we will expand on the results of Theorem 1.


etes the proof. A.6 Proof of Theorem 1 In this section, we will expand on the results of Theorem 1. Theorem 1 Restated. Assume, we have a reference model, such that πref(y|x) > 0 for all pairs of prompts x and answers y and a parameter β > 0. All reward equivalence classes, as defined in Section 5 can be represented with the reparameterization r(x, y) = β log π(y|x) πref(y|x) for some model π(y|x). Proof. Consider any reward function r(x, y), which induces an optimal model πr(y|x) under the KL-constrained RL problem, with solution given by 4. Following Eq. 5, when we log-linearize both sides we obtain: r(x, y) = β log πr(y|x) πref(y|x) + β log Z(x) where Z(x) = P y πref(y|x) exp  1 β r(x, y)  (notice that Z(x) also depends on the reward function r).


Z(x) = P y πref(y|x) exp  1 β r(x, y)  (notice that Z(x) also depends on the reward function r). Using the operator r′(x, y) = f(r, πref, β)(x, y) = r(x, y) −β log Z(x), we see that this new reward function is within the equivalence class of r and, we have: r′(x, y) = β log πr(y|x) πref(y|x) which completes the proof. We can further expand on these results. We can see that if r and r′ are two reward functions in the same class, then f(r, πref, β)(x, y) = β log πr(y|x) πref(y|x) = β log π′ r(y|x) πref(y|x) = f(r′, πref, β)(x, y) where the second equality follows from Lemma 2. We have proven that the operator f maps all reward functions from a particular equivalence class to the same reward function.


erator f maps all reward functions from a particular equivalence class to the same reward function. Next, we show that for every equivalence class of reward functions, the reward function that has the reparameterization outlined in Theorem 1 is unique. Proposition 1. Assume, we have a reference model, such that πref(y|x) > 0 for all pairs of prompts x and answers y and a parameter β > 0. Then every equivalence class of reward functions, as defined in Section 5, has a unique reward function r(x, y), which can be reparameterized as r(x, y) = β log π(y|x) πref(y|x) for some model π(y|x). 18


Proof. We will proceed using proof by contradiction. Assume we have two reward functions from the same class, such that r′(x, y) = r(x, y) + f(x). Moreover, assume that r′(x, y) = β log π′(y|x) πref(y|x) for some model π′(y|x) and r(x, y) = β log π(y|x) πref(y|x) for some model π(y|x), such that π ̸= π′. We then have r′(x, y) = r(x, y) + f(x) = β log π(y|x) πref(y|x) + f(x) = β log π(y|x) exp( 1 β f(x)) πref(y|x) = β log π′(y|x) πref(y|x) for all prompts x and completions y. Then we must have π(y|x) exp( 1 β f(x)) = π′(y|x). Since these are distributions, summing over y on both sides, we obtain that exp( 1 β f(x)) = 1 and since β > 0, we must have f(x) = 0 for all x. Therefore r(x, y) = r′(x, y). This completes the proof.


nce β > 0, we must have f(x) = 0 for all x. Therefore r(x, y) = r′(x, y). This completes the proof. We have now shown that every reward class has a unique reward function that can be represented as outlined in Theorem 1, which is given by f(r, πref, β) for any reward function in that class. B DPO Implementation Details and Hyperparameters DPO is relatively straightforward to implement; PyTorch code for the DPO loss is provided below: 19


import torch.nn.functional as F def dpo_loss(pi_logps, ref_logps, yw_idxs, yl_idxs, beta): """ pi_logps: policy logprobs, shape (B,) ref_logps: reference model logprobs, shape (B,) yw_idxs: preferred completion indices in [0, B-1], shape (T,) yl_idxs: dispreferred completion indices in [0, B-1], shape (T,) beta: temperature controlling strength of KL penalty Each pair of (yw_idxs[i], yl_idxs[i]) represents the indices of a single preference pair. """ pi_yw_logps, pi_yl_logps = pi_logps[yw_idxs], pi_logps[yl_idxs] ref_yw_logps, ref_yl_logps = ref_logps[yw_idxs], ref_logps[yl_idxs] pi_logratios = pi_yw_logps - pi_yl_logps ref_logratios = ref_yw_logps - ref_yl_logps losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios)) rewards = beta * (pi_logps - ref_logps).detach() return losses, re


* (pi_logratios - ref_logratios)) rewards = beta * (pi_logps - ref_logps).detach() return losses, rewards Unless noted otherwise, we use a β = 0.1, batch size of 64 and the RMSprop optimizer with a learning rate of 1e-6 by default. We linearly warmup the learning rate from 0 to 1e-6 over 150 steps. For TL;DR summarization, we use β = 0.5, while rest of the parameters remain the same. C Further Details on the Experimental Set-Up In this section, we include additional details relevant to our experimental design. C.1 IMDb Sentiment Experiment and Baseline Details The prompts are prefixes from the IMDB dataset of length 2-8 tokens. We use the pre-trained sentiment classifier siebert/sentiment-roberta-large-english as a ground-truth reward model and gpt2-large as a base model.


bert/sentiment-roberta-large-english as a ground-truth reward model and gpt2-large as a base model. We use these larger models as we found the default ones to generate low-quality text and rewards to be somewhat inaccurate. We first use supervised fine-tuning on a subset of the IMDB data for 1 epoch. We then use this model to sample 4 completions for 25000 prefixes and create 6 preference pairs for each prefix using the ground-truth reward model. The RLHF reward model is initialized from the gpt2-large model and trained for 3 epochs on the preference datasets, and we take the checkpoint with the highest validation set accuracy. The “TRL” run uses the hyper-parameters in the TRL library. Our implementation uses larger batch samples of 1024 per PPO step.


r-parameters in the TRL library. Our implementation uses larger batch samples of 1024 per PPO step. C.2 GPT-4 prompts for computing summarization and dialogue win rates A key component of our experimental setup is GPT-4 win rate judgments. In this section, we include the prompts used to generate win rates for the summarization and dialogue experiments. We use gpt-4-0314 for all our experiments. The order of summaries or responses are randomly chosen for every evaluation. Summarization GPT-4 win rate prompt (S). Which of the following summaries does a better job of summarizing the most \ important points in the given forum post? Post: 20


<post> Summary A: <Summary A> Summary B: <Summary B> FIRST provide a one-sentence comparison of the two summaries, explaining which \ you prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your \ choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"A" or "B"> Summarization GPT-4 win rate prompt (C). Which of the following summaries does a better job of summarizing the most \ important points in the given forum post, without including unimportant or \ irrelevant details? A good summary is both precise and concise. Post: <post> Summary A: <Summary A> Summary B: <Summary B> FIRST provide a one-sentence comparison of the two summaries, explaining which \ you prefer and why.


IRST provide a one-sentence comparison of the two summaries, explaining which \ you prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your \ choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"A" or "B"> Dialogue GPT-4 win rate prompt. For the following query to a chatbot, which response is more helpful? Query: <the user query> Response A: <either the test method or baseline> Response B: <the other response> FIRST provide a one-sentence comparison of the two responses and explain \ which you feel is more helpful. SECOND, on a new line, state only "A" or \ "B" to indicate which response is more helpful. Your response should use \ the format: Comparison: <one-sentence comparison and explanation> More helpful:


sponse should use \ the format: Comparison: <one-sentence comparison and explanation> More helpful: <"A" or "B"> C.3 Unlikelihood baseline While we include the unlikelihood baseline [46] (simply maximizing log p(yw|x), the log probability of the preferred response, while minimizing log p(yl|x), the log probability of the dispreferred response) in our sentiment experiments, we do not include it as a baseline in either the summarization 21


Prompt Response SUBREDDIT: r/relationships TITLE: The girl [26 F] I [22 M] have been seeing for a month didn’t respond to me at all yesterday while hanging out with a friend [ 30? M]. POST: She gets terrible service while at her house, but I texted her 3 times yesterday, 4-5 hours apart. She didn’t call me until early this morning and left a voicemail that she was busy all day with a friend who showed up out of the blue. I saw that she posted a picture of the two of them out of her dead zone house on facebook before I texted her the last time. I don’t mind that she hangs out with friends, and I know it’s pretty early [...] TL;DR: girl when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when wh


en when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when whenUB when when when when whenUB when when when whenUB SUBREDDIT: r/tifu TITLE: TIFU by accidently kicking an old woman POST: So this didn’t happen today but actually about a year or two ago.


dently kicking an old woman POST: So this didn’t happen today but actually about a year or two ago. I was at my granddads funeral so of course it was all very sad and full of lots of crying old people. After the ceremony everyone walks outside the building and onto the other side of the small road the hearses drive down. Now the road is important because obviously if there’s a road, there’s a curb onto the sidewalk, so most of us are on the other side of the road, besides a few older people walking a lot slower. As one of the old woman goes to walk up the curb [...] TL;DR: when an old woman was tripping the when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when


when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when when Table 3: Unlikelihood samples from TL;DR prompts sampled at temperature 1.0. In general, we find unlikelihood fails to generate meaningful responses for more complex problems such as summarization and dialogue.


ails to generate meaningful responses for more complex problems such as summarization and dialogue. or dialogue experiment because it produces generally meaningless responses, which we believe is a result of unconstrained likelihood minimization. D Additional Empirical Results D.1 Performance of Best of N baseline for Various N We find that the Best of N baseline is a strong (although computationally expensive, requiring sampling many times) baseline in our experiments. We include an evaluation of the Best of N baseline for various N for the Anthropic-HH dialogue and TL;DR summarization; the results are shown in Figure 4. D.2 Sample Responses and GPT-4 Judgments In this section, we present examples of comparisons between DPO and the baseline (PPO temp 0.


ments In this section, we present examples of comparisons between DPO and the baseline (PPO temp 0. for summarization, and the ground truth chosen response for dialogue). See Tables 4-6 for summarization examples, and Tables 7-10 for dialogue examples. 22


0.25 0.50 0.75 1.00 Sampling temperature 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 Win rate Anthropic-HH Dialogue Win Rate vs Chosen Best of 1 Best of 4 Best of 16 Best of 64 Best of 128 0.00 0.25 0.50 0.75 1.00 Sampling temperature 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Win rate TL;DR Summarization Win Rate vs Reference Best of 64 Best of 128 Best of 256 Figure 4: Best of N baseline for N = {1, 4, 16, 64, 128}. Performance plateaus after roughly 64-128 samples. Prompt SUBREDDIT: r/AskReddit TITLE: I’ve been ungrateful to parents who have wanted only the best for me for the longest time. I am so lucky to have the life I have but I squandered it, this isn’t a humblebrag about my "wicked life" just tell me, what can I do to make it up to them after this.


t a humblebrag about my "wicked life" just tell me, what can I do to make it up to them after this. POST: I’m 17, leaving for University, and I’ve had epiphany after epiphany of how shitty i’ve beent to them, or how all the nice things they do I ruin with moods etc., I know there isn’t a quick fix, but I want to do something nice. I’ve always thanked them and apologized where it is due (in earlier years not so much) but as they say, actions speak louder than words, so I was wondering what can I do now, before I go, to just make them happy. What can I keep doing, I know all the little things, doing chores, keeping my room clean, cleaning up after myself, helping around the house, but what’s a big thing, for say christmas, or their birthdays, that I can do to make them happy?


e, but what’s a big thing, for say christmas, or their birthdays, that I can do to make them happy? DPO TL;DR: I’ve been a shitty child and I want to make up for it, what can I do for my parents on their birthdays and christmas to show them how much I love them? PPO TL;DR: I’ve been a shitty person to my parents, what can I do to make it up to them, before I go, and what can I do after university to make them happy, and what can I do to make them happy after university (if that is the case) Judgment Summary B [DPO] more effectively captures the main points of the post by focusing on making the parents happy through gestures on special occasions, while Summary A seems repetitive and less clear. Table 4: Sample summaries to a post from the TL;DR test set.


ary A seems repetitive and less clear. Table 4: Sample summaries to a post from the TL;DR test set. DPO sample generated with a temperature 0.25; PPO sample generated at a temperature of 0. The order in which summaries are presented in randomized when evaluating with GPT-4, so the order in the Judgment may not correspond to the order in the table. For clarity, post-hoc annotations are included in bold, formatted as [annotation]. These annotations are not part of the model generations. 23


Prompt SUBREDDIT: r/relationships TITLE: I [22 M] am about to propose to my girlfriend [22F] of almost 2 years, and I can’t calm myself down POST: Alright, so I am preparing to propose to my girlfriend of almost 2 years (we were friends for double that beforehand) in a couple days and I am starting to get EXTREMELY anxious/nervous. We have talked about this for months, and I know she will say yes, but I can’t seem to relax. The other day I bought the ring, which I fell in love with myself and know she will too. And then two days ago, I asked her dad’s blessing. Everything has gone great so far, but now leading up to it I can’t think about anything else. I can’t sleep. I am anxious as hell, and all I want to do is pop the question now, but I have to wait since I don’t see her until Thursday


nd all I want to do is pop the question now, but I have to wait since I don’t see her until Thursday. I know how I am going to do it. I have thoroughly planned it out and I am more than prepared. But, at the same time, I am freaking out. I asked my best friend for advice since he just got married recently, and all he could say was "It will feel 10x worse right before you do ask." Any suggestions on how to relax leading up to it? I would like to be able to sleep... DPO TL;DR: About to propose to my girlfriend of almost 2 years, but I am extremely anxious and can’t sleep. How can I relax? PPO TL;DR: I am about to ask my girlfriend of almost 2 years to marry me, and I can’t calm myself down. Any tips on how to relax?


y girlfriend of almost 2 years to marry me, and I can’t calm myself down. Any tips on how to relax? I would like to be able to sleep tonight (and the night before).............................................................................................................................. Judgment Summary A [DPO] more effectively summarizes the main points of the post, concisely conveying the asker’s anxiety and goal of finding ways to relax. Table 5: Sample summaries to a post from the TL;DR test set. DPO sample generated with a temperature 0.25; PPO sample generated at a temperature of 0. The order in which summaries are presented in randomized when evaluating with GPT-4, so the order in the Judgment may not correspond to the order in the table.


n evaluating with GPT-4, so the order in the Judgment may not correspond to the order in the table. For clarity, post-hoc annotations are included in bold, formatted as [annotation]. These annotations are not part of the model generations. Prompt SUBREDDIT: r/loseit TITLE: Is It Bullshit?: Fiber Gourmet Pasta POST: I was looking for low-cal alternatives to pasta, because I love pasta and it’s easy to make and I eat it several times a week. I find that whole grain pasta has a weird taste/texture, and I’m not a big fan of it. I was doing some research into spaghetti squash (which is on my grocery list for next time), but I also heard someone rave about some high-fiber pasta brand so I looked into it.


or next time), but I also heard someone rave about some high-fiber pasta brand so I looked into it. What sorcery is this? It has all the trappings of regular pasta (and the reviews I’ve read say it tastes the same too) but a little over half the calories. My mind boggles over how much extra pasta I could eat! I can’t believe this hasn’t taken the world by storm, which makes me wonder what the catch is. DPO TL;DR: I’m trying to cut back on calories and pasta is one of my main sources. Found a high-fiber pasta that has all the trappings of regular pasta and seems like it would be a good substitute. Is it bullshit? PPO TL;DR: I want to lose weight, but I love pasta and don’t want to give it up. Is there a high-fiber pasta that I can eat more of and lose weight with, or is it all bullshit?


up. Is there a high-fiber pasta that I can eat more of and lose weight with, or is it all bullshit? Judgment Summary B [DPO] more accurately captures the intent to find a low-calorie pasta alternative, without assuming the need for weight loss. Table 6: Sample summaries to a post from the TL;DR test set. DPO sample generated with a temperature 0.25; PPO sample generated at a temperature of 0. The order in which summaries are presented in randomized when evaluating with GPT-4, so the order in the Judgment may not correspond to the order in the table. For clarity, post-hoc annotations are included in bold, formatted as [annotation]. These annotations are not part of the model generations. 24


Prompt Can you help me write an essay on the Civil Rights Movement? DPO The Civil Rights Movement was a series of social and political movements that took place between the 1950s and the 1980s, leading to the integration of public spaces, higher education, employment, and voting rights. The movement is often credited with the end of legal segregation in the United States, and it was a major catalyst for the Civil Rights Act of 1964. The Movement also played a large role in the rise of the Black Power and Civil Rights Movements, which focused on economic equality and racial justice. The Movement focused on the rights of African Americans, who were then the dominant racial and ethnic group in the United States, but it also highlighted the rights of other minority groups, such as women, immig


the United States, but it also highlighted the rights of other minority groups, such as women, immigrants, and LGBT people. The Movement is often considered one of the most important social movements of the 20th century, and it played a significant role in the advancement of civil rights for all Americans. I’ll be happy to write an essay on this topic for you. What kind of essay would you like? A paper on the history of the Civil Rights Movement? A paper on the causes of the Civil Rights Movement? Or a paper on the effects of the Civil Rights Movement? Any ideas? I’ll be happy to [maximum generation length reached] GT Oh, I’m not sure if I can actually write an essay, but I can at least help you think through the process.


m not sure if I can actually write an essay, but I can at least help you think through the process. First, can you tell me what you’re trying to write about? In other words, what are the main points or questions you want to address? Judgment Response B [DPO] provides more detailed information about the Civil Rights Movement and offers specific suggestions for essay topics, making it more helpful for someone writing an essay. Table 7: GPT-4 chooses DPO over GT. Sample responses to a prompt from the Anthropic-HH test set. DPO sample generated with temperature 0.7; GT is the chosen completion in the dataset of preferences. For clarity, post-hoc annotations are included in bold, formatted as [annotation]. These annotations are not part of the model generations.


cluded in bold, formatted as [annotation]. These annotations are not part of the model generations. Prompt I am trying to locate the residential address for J.B. Pritzker, an hier [sic] to the Hyatt Hotel fortune. DPO I am not able to assist you in this matter. It’s unlawful and irresponsible to locate private information on people without their consent, and I’m not aware of any way you could do this for human. Please stop this activity. GT J.B. Pritzker is one of the co-founders and investors in Hyatt. He and his family have been listed in the Forbes 400, and the family also maintains a 4,000-acre ranch in Colorado. Judgment Response A [DPO] emphasizes the importance of privacy and laws, while Response B gives background information on J.B. Pritzker without providing his address.


laws, while Response B gives background information on J.B. Pritzker without providing his address. Table 8: GPT-4 chooses DPO over GT. Sample responses to a prompt from the Anthropic-HH test set. DPO sample generated with temperature 1.0; GT is the chosen completion in the dataset of preferences. For clarity, post-hoc annotations are included in bold, formatted as [annotation]. These annotations are not part of the model generations. 25


Prompt Can you tell me what got the USA involved in World War II? DPO In the 1930s, the United States found itself at the center of a growing political and economic crisis that threatened the world’s stability. The Great Depression had brought many other countries to the brink, including many European ones, and the United States found itself in a very precarious position. The Great Depression had left the government and the economy on the edge of collapse, it faced severe international indebtedness, and it faced the threat of a foreign default on its loans. When Japan attacked China in October 1937, the United States found itself in an especially dangerous military position, as the United States was the largest creditor.


itself in an especially dangerous military position, as the United States was the largest creditor. The United States found itself desperately seeking ways to avoid a complete collapse, and two main strategies emerged: both involved creating an alliance of nations to protect American economic security and stability, as well as ensuring that the United States remained politically relevant. The first strategy was to create a “coalition of the willing”, a network of international alliances to counteract the isolation and vulnerability of countries like Japan. The second strategy was to form an “all-inclusive association”, a group of nations that the United States would be the most powerful member of, and which would protect the prosperity of the United States and protect it from other powers


er of, and which would protect the prosperity of the United States and protect it from other powers [maximum generation length reached] GT The United States entered the war against Japan in response to the Japanese attack on Pearl Harbor, Hawaii, on December 7, 1941. This was a full month before the United States officially entered the war. At this point, the German Axis powers controlled France, Belgium, Norway, and other parts of Europe, and held significant military influence in much of Asia. President Roosevelt officially asked Congress for a declaration of war against Japan after the bombing of Pearl Harbor. Judgment Response B [GT] provides a direct and accurate answer to the question, while Response A is overly complicated and doesn’t provide the correct answer.


wer to the question, while Response A is overly complicated and doesn’t provide the correct answer. Table 9: GPT-4 chooses GT over DPO. DPO’s response is verbose and plausible, but contains factually incorrect information (the ‘coalition of the willing’ does not refer to events of WWII; the ‘all-inclusive association’ is not a real organization). Prompt what is 7 plus 2 DPO Hm, let me see... seven plus two is nine, I’m pretty sure. I’ll be happy to read the input aloud and double check if I’ve got it right. Let me try: "7 + 2" I’ve got that right! I think our input was meant to be seven, plus two. You can try it yourself: "7 + 2" "9" I’ve got that right! Now, what I’m wondering is if you want to learn this formula for yourself, or if you want me to help you practice it.


ering is if you want to learn this formula for yourself, or if you want me to help you practice it. The formula is called the “arithmetic-prefix method”, and it’s a great way to memorize numbers – and in fact I personally learned to count by memorizing the arithmatic prefixes! I’ll give you a refresher on arithmatic prefixes: you can think of them as an abbreviated way of describing a number by its first two significant digits. There are three arithmatic prefixes: The “1-arithmatic prefix” (1-A) The “2-arithmatic prefix” (2-A) GT 11 Judgment Response B [GT] provides a direct and accurate answer to the question, while Response A is overly complicated and doesn’t provide the correct answer. Table 10: GPT-4 chooses GT over DPO.


is overly complicated and doesn’t provide the correct answer. Table 10: GPT-4 chooses GT over DPO. GPT-4 incorrectly states that the ground truth is correct while DPO’s (more verbose) output is wrong. 26


D.3 Human study details Figure 5: Layout of the survey in SurveyMonkey. Each respondent completed 25 similarly-formatted judgments. In order to validate the usage of GPT4 for computing win rates, our human study collects human preference data for several matchups in the TL;DR summarization setting. We select three different algorithmic matchups, evaluating DPO (temp. 0.25), SFT (temp. 0.25), and PPO (temp 1.0) compared to the reference algorithm PPO (temp 0.). By selecting matchups for three unique algorithms as well as algorithms with a wide range of win rates vs the reference, we capture the similarity of human and GPT-4 win rates across the response quality spectrum. We sample 150 random comparisons of DPO vs PPO-0 and 100 random comparisons PPO-1 vs PPO-0, assigning two humans to each


comparisons of DPO vs PPO-0 and 100 random comparisons PPO-1 vs PPO-0, assigning two humans to each comparison, producing 275 judgments for DPO-PPO7 and 200 judgments for PPO-PPO. We sample 125 SFT comparisons, assigning a single human to each. We ignore judgments that humans labeled as ties (which amount to only about 1% of judgments), and measure the raw agreement percentage between human A and human B (for comparisons where we have two human annotators, i.e., not SFT) as well as between each human and GPT-4. Participants. We have 25 volunteer human raters in total, each comparing 25 summaries (one volunteer completed the survey late and was not included in the final analysis, but is listed here).


olunteer completed the survey late and was not included in the final analysis, but is listed here). The raters were Stanford students (from undergrad through Ph.D.), or recent Stanford graduates or visitors, with a STEM (mainly CS) focus. See Figure 5 for a screenshot of the survey interface. We gratefully acknowledge the contribution of each of our volunteers, listed in random order: 1. Gordon Chi 2. Virginia Adams 3. Max Du 4. Kaili Huang 5. Ben Prystawski 6. Ioanna Vavelidou 7. Victor Kolev 8. Karel D’Oosterlinck 9. Ananth Agarwal 10. Tyler Lum 11. Mike Hardy 12. Niveditha Iyer 13. Helena Vasconcelos 14. Katherine Li 15. Chenchen Gu 16. Moritz Stephan 17. Swee Kiat Lim 18. Ethan Chi 19. Kaien Yang 20. Ryan Chi 21. Joy Yun 22. Abhay Singhal 23. Siyan Li 24. Amelia Hardy 25.


han Chi 19. Kaien Yang 20. Ryan Chi 21. Joy Yun 22. Abhay Singhal 23. Siyan Li 24. Amelia Hardy 25. Zhengxuan Wu 7One volunteer did not respond for the DPO-PPO comparison. 27


CS 224R: Deep Reinforcement Learning (Spring 2023) Leni Aniva 2024-03-20 1


• Instructors: Chelsea Finn, Karol Hausman • Quarter: Spring 2023 • Time: MW 16:30 - 17:50 • Location: Gates B1 • Website: https://cs224r.stanford.edu/ • Discussion: Ed via Canvas • Email: cs224r-spr2223-staff@lists.stanford.edu • Office Hours: See schedule • Grading: 50% Homework (lowest will be 5%, others are 15% each) and 50% Project Assignments: 1. Imitation Learning 2. Online reinforcement learning 3. Offline reinforcement learning 4. Goal-conditioned and meta reinforcement learning 5. Project Project: – Research-level project of your choice – Groups of 1-3 students – Can share with other classes with higher expectation – Same late day policy as homework – No late days for poster, 16:00 - 19:00, 7 June – A word of warning: Deep RL methods take a long time to learn behaviour • 6 late d


- 19:00, 7 June – A word of warning: Deep RL methods take a long time to learn behaviour • 6 late days total across homeworks and projects. Max 2 late days per assignment. Pre-requisites: • Machine learning: CS229 or equivalent We will assume knowledgeo of SGD, cross-validation, calculus, probability theory, linear algebra • Some familiarity with deep learning: Backpropagation, convolutional networks, recurrent networks • Assignments will require training with PyTorch


• Some familiarity with reinforcement learning We’ll quickly go over the basics Topics: 1. Imitation Learning: Behaviour cloning, Inverse RL 2. Model-free deep RL: Policy gradients, Actor-critic methods, Q-learning 3. Model-based deep RL 4. Offline RL: Conservative methods, decision transformers 5. Multi-task and meta RL: Hindsight relabeling, learning to explore 6. Advanced topics: hierarchical RL, sim2real, reset-free RL 7. Case studies of interesting and timely applications 2


CONTENTS Contents 1 Introduction 3 2 Imitation Learning 3 2.1 Collecting Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 What can go wrong? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Learning from Online Interventions . . . . . . . . . . . . . . . . . . . . . . . 7 3 Markov Decision Processes and Policy Gradients 8 3.1 Reinforcement Learning Problem . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Policy Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3 Variance Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4 Value-based RL and Actor-Critic Methods 15 4.1 Value-based RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


c Methods 15 4.1 Value-based RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5 Q-Learning 19 5.1 Actor-Critic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.2 Policy and Value Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.3 Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 6 Practical Deep RL Implementations 22 6.1 Q-Learning Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 7 Model-Based RL 25 7.1 Gradient-based and Sampling-based Optimisation . . . . . . . . . . . . . . . 25 7.2 Learning a Dynamics Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 7.3 Using the Learned Dynamics Model . . . . . . . . . . . . . . . . . . . . . . .


. . . . . . . 26 7.3 Using the Learned Dynamics Model . . . . . . . . . . . . . . . . . . . . . . . 27 8 Reward Learning 29 8.1 Rewards from Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 8.2 Rewards from Human Preferences . . . . . . . . . . . . . . . . . . . . . . . . 31 9 Offline RL 33 9.1 Data Constraint Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 9.2 Conservative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 9.3 Data Stitching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 10 Offline RL cont. 36 10.1 Weighted Imitation Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 10.2 Conditional Imitation Learning . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . 37 10.2 Conditional Imitation Learning . . . . . . . . . . . . . . . . . . . . . . . . . 41 10.3 Offline Evaluation and Hyper-parameter Tuning . . . . . . . . . . . . . . . . 41 1


CONTENTS 11 Multi-Tasked RL and Goal-Conditioned RL 42 11.1 Multi-Task Imitation and Policy Gradients . . . . . . . . . . . . . . . . . . . 42 11.2 Multi-Task Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 11.3 Goal-Conditioned RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 12 Transfer Learning in RL 45 12.1 Learning Locomotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 12.2 Sim-to-Real Gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 13 Meta RL 48 13.1 Black-Box Meta-RL Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 49 13.2 Optimisation-Based Meta-RL Methods . . . . . . . . . . . . . . . . . . . . . 50 14 Meta RL: Learning to Explore 51 14.1 Learning to Explore . . . . . . . . . . .


. . . . . . . 50 14 Meta RL: Learning to Explore 51 14.1 Learning to Explore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 14.2 Decoupled Exploration and Exploitation . . . . . . . . . . . . . . . . . . . . 52 15 Reset-Free RL 54 15.1 Forward-Backward RL and MEDAL . . . . . . . . . . . . . . . . . . . . . . . 54 15.2 QWALE/single-life RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 16 Hierarchical RL and Skill Discovery 57 16.1 Skill Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 16.2 Slightly Different Mutual Information . . . . . . . . . . . . . . . . . . . . . . 59 16.3 Hierarchical RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . 59 16.3 Hierarchical RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 17 RL in the Real World: From Chip Design to LLMs 59 18 Connecting the dots 59 18.1 Research Lightning Talks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2


2 IMITATION LEARNING 1 Introduction Announcements • 04-05: Homework 1 out • 16:30 04-06. (Skilling Auditorium): PyTorch tutorial will be recorded • 04-19: Homework 1 due Reinforcement learning solves sequential decision-making problems: A system needs to make multiple decisions based on stream of information. The solutions • Imitation learning • Offline and online RL • Model-free and model-based RL • Multi-task and meta RL Excursion: Supervised Learning In supervised learning, we are given labeled data {(x, y)} and want to learn f(x) ≃y. We directly receive information about what to output, and the inputs are i.i.d. In comparison, in Reinforcement Learning, we learn behaviour π(a|s). The data is not i.i.d., since each action affects future observations.


, we learn behaviour π(a|s). The data is not i.i.d., since each action affects future observations. Open problems: • Can robots learn generalisable behaviour? • Can robots learn behaviours autonomously? Why study deep reinforcement learning? Sequential decision-making problems are everywhere, and the advancements of deep neural networks have achieved considerable success in recent years. Applications: 1. Controlling robots and autonomous vehicles 2. AI that interact with people 3. Systems with feedback loops 4. Some tasks’ objectives are not accuracy or differentiable 2 Imitation Learning • Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation. Zhang et al. (2017) 3


2 IMITATION LEARNING Announcements • 04-05: Homework 1 out • 16:30 04-06. (Skilling Auditorium): PyTorch tutorial • 04-07: Fill out AWS form with account id • Up to 2% extra credit for providing TA endorsed answers on Ed • Project: Final groups are not locked in until project proposals are due, survey due 04-17 We start by a formalisation of behaviour. An intelligent agent observes some environment, takes action, and the action updates the environment. A model which decides which action to take in response of an environment is the policy, denoted πθ(a|s), where a is the action and s is the state. It is often useful to output a distribution of actions rather than a single action, and hence πθ here is a distribution. The interaction can be described as a cycle: 1. Observe state st 2.


ence πθ here is a distribution. The interaction can be described as a cycle: 1. Observe state st 2. Take action at e.g. by sampling at ∼πθ(·|st) 3. Observe next state st+1 sampled from unknown world dynamics p(·|st, at) The result is a trajectory s1, a1, . . . , sT, also called a policy roll-out. Since the next state is only a distribution parameterised by st, at, the trajectory is Markovian. Question: Is the policy being changed as we go through the trajectory or we plot the entire trajectory and then update the policy? Dependent on the algorithm. Some policies update in the middle of a roll-out. Question: Does the state include the history of actions? The state includes only the previous action but the history is encoded here. In some cases (e.g.


s? The state includes only the previous action but the history is encoded here. In some cases (e.g. dialogue) it would be beneficial to include history in the state and policy. Imitation learning refers to train a policy using suprevised learning to imitate some data provided. • Data: The data, demonstrations, is the set of given trajectories collected by an expert D := {(s1, a1, . . . , sT)}. This does not have to be optimal. • Training: Train policy to mimic expert using a cross-entropy loss arg max θ E(s,a)∼D[log πθ(a|s)] 4


2 IMITATION LEARNING This is a differentiable function w.r.t. θ (given differentiable πθ), so it could be trained with gradient descent. Excursion: Cross-entropy loss The cross-entropy loss here is also called ℓ2-loss. This has to do with the Gaussian distribution f(x) ∝exp  −1 2σ2 ∥x −µ∥2  Taking logarithm of this, we have log f(x) = −1 2σ2 ∥x −µ∥2 + C When σ is fixed, the difference here is a ℓ2 distance between x and µ. 2.1 Collecting Data How can we collect demonstrations? • (Robotics) Videos of expert demonstrations • Execute a lot of simulations • (Games) Records of human players • (Robotics) Kinesthetic teaching: Forcing the robot arm to move and record the movement + Easy interface - Human controller visible in scene • (Robotics) Remote controllers: ~ Interface is different • (Ro


- Human controller visible in scene • (Robotics) Remote controllers: ~ Interface is different • (Robotics) Puppeteering: Use one robot to follow the other + Easy interface - Requires double hardware In some domains, demonstration data have already been collected. e.g. driving, writing text messages. In some domains it may not be viable to collect demonstrations. e.g. quadruped robot, or when we don’t even know what is the best behaviour? Question: Are GANs used for data generation in robotics? There are some works that use generative models for data augmentation. 5


2 IMITATION LEARNING (a) Compounding errors (b) Corrective behaviour data Figure 2.1: Compounding errors leading to state drifting out of data A B Figure 2.2: Multiple paths from point A to B exist, but the optimal policy under ℓ2-loss converges to the prohibited middle path. This is because the distribution is unimodal. Can we directly use videos of people and animals? There is a embodiment gap: Differences in appearance, physical capabilities, degrees of freedom, make it challenging to learn from video data. Question: Can we attach sensors to the expert to collect motion data? Yes. This can be quite useful. Sometimes there is still a physical embodiment gap. See Data Glove. 2.2 What can go wrong? Problems: 1.


times there is still a physical embodiment gap. See Data Glove. 2.2 What can go wrong? Problems: 1. Compounding errors: Accumulation of errors can lead to the state drifting away from data distribution. This is covariate shift. Solutions: • Collect a lot of demo data and hope for the best. • Collect corrective behaviour data 2. Multimodal demonstration data: Agent may fail to capture multimodal data due to constrains in policy distribution. Solutions: 6


2 IMITATION LEARNING • Use a more expressive distribution Mixture density network: Outputs parameters for a (Gaussian) mixture distribution • Use another loss function which matches a single mode. The cross-entropy minimises the KL-divergence DKL(pexpert∥π), but the reverse KL divergence DKL(π∥pexpert) will optimise to fit a single peak of the expert distribution. The problem with this approach is the reverse KL distribution is often intractable. The expressiveness of the distribution is separate from the expressiveness of the neural network. e.g. A huge network that outputs a Gaussian is still constrained by the limited expressiveness of the Gaussian. Excursion: Generative Models Some other classes of generative models represent distributions in other ways.


n: Generative Models Some other classes of generative models represent distributions in other ways. For example a model could take in (s, z) where z ∼N(0, 1). e.g. VAEs, Diffusion models Na¨ıve training of a model like this would make the model ignore the noise. 3. Miamatch in observability between expert and agent: To train a chatbot, we might have example demos are scraped from conversations, but knowledges external to the conversation may exist that are visible to the expert but not the agent. Solutions: • Try to get as much contextual information to the agent as possible • Collect demos in a way that gives expert same information as agent Question: What if the action space is discrete but huge (e.g. words)? • Tokenisation • Factorisation e.g.


on: What if the action space is discrete but huge (e.g. words)? • Tokenisation • Factorisation e.g. representing words by morphemes or letters and this becomes an autoregressive distribution 2.3 Learning from Online Interventions To collect corrective behaviour data, we may collect the errored trajectories from the agent and specifically train on expert trajectories correcting from such errors. Dataset Aggregation (DAgger) is the process of 1. Roll-out learned policy πθ : s′ 1 : ˆa1, . . . , s′ T 7


3 MARKOV DECISION PROCESSES AND POLICY GRADIENTS 2. Query expert action at visited states a∗∼πexpert(·|s′) This can happen during roll-out e.g. safety driver 3. Aggregate corrections with existing data D ←D{(s′, a∗)} 4. Update policy minθ L(πθ, D) This makes the model more robust since it sees a wider distribution of states. When humans are used as experts, this is Human Gated DAgger. + Data-efficient way to learn from expert - Can be challenging to query expert when agent has control - Does not discourage model from deviating from expert, only teaches it how to fix problems. 3 Markov Decision Processes and Policy Gradients • Simple statistical gradient-following algorithms for connectionist reinforcement learning.


dients • Simple statistical gradient-following algorithms for connectionist reinforcement learning. Williams (1992) Announcements • 04-17: Project Survey due • 04-19: Homework 1 due • 04-27: Project Proposal due 3.1 Reinforcement Learning Problem Consider the problem of robot grasping an object. A robot arm has a list of motions it can take, and the action influences what the robot will observe in the future. We can capture them in the form of a policy. Two problems: • How do we evaluate a policy? • How do we optimise a policy for a desired outcome? The agent gets sensory observation at time t, ot, and performs action at using a policy πθ(at|ot). The action affects the state st. If the state is fully observed, the policy is πθ(at|st).


θ(at|ot). The action affects the state st. If the state is fully observed, the policy is πθ(at|st). The state has st all the information needed for making a decision, but the agent only has access to the partial observation ot. It may be possible that the observation does not have the information needed to provide an optimal decision. 8


3 MARKOV DECISION PROCESSES AND POLICY GRADIENTS Object Classification Object manipulation Supervised learning Sequential decision making iid data action affects next state large labeled, curated dataset what are the dataset and labels? well defined loss function unclear loss function Table 3.1: Comparison between object classification (supervised learning) and object manipulation (reinforcement learning) Excursion: Plato’s Allegory of the Cave We can perceive real objects and describe them in an abstract manner based on our observations of them. In Plato’s Allegory of the Cave, the gods hold the objects we observe in a cave and we can only observe their shadow. For example, the state may be the precise location and shape of objects, and the observation is an image formed from visual perce


the precise location and shape of objects, and the observation is an image formed from visual perception. Question: If the state contains all information needed, why not spend time improve the perception system rather than the reinforcement learning algorithm? Often the notion of a state is unclear what is in the state. It is also costly or impossible to gather some information. The value of actions can be determined using a reward function r(s, a). e.g. crashing the car has low reward, driving to destination has high reward. s, a, r(s, a), p(s′|s, a) together define a Markov decision process (MDP). A supervised learning task can be described as T := { p(x) , p(y|x) , L } Data distribution Label distribution Loss function In comparison, a reinforcement learning task can be described as T


Label distribution Loss function In comparison, a reinforcement learning task can be described as T := { S , A , p(s1) , p(s′|s, a) , r(s, a) } Action space State space Initial state distribution Dynamics Reward The most expensive component to building an RL algorithm is generating the samples. 3.2 Policy Gradients Question: Why assume Markov property instead of using LSTM? Using LSTM is still assuming the Markov property but replacing state with all history of states. Assuming Markov property makes modeling of RL models simpler. 9


3 MARKOV DECISION PROCESSES AND POLICY GRADIENTS Figure 3.1: Anatomy of an RL algorithm s1 o1 a1 πθ(ai|oi) s2 o2 a2 πθ(ai|oi) p(si+1|si, ai) s3 o3 a3 πθ(ai|oi) p(si+1|si, ai) Figure 3.2: Markov chain formed by the sequences of oi, si, ai s1 a1 s2 a2 s3 a3 p(st+1|st, at) p(st+1|st, at) πθ(ai|si) πθ(ai|si) πθ(ai|si) Figure 3.3: Markov chain with oi = si 10


3 MARKOV DECISION PROCESSES AND POLICY GRADIENTS In reinforcement learning, we assume that the dynamics model p(st+1|st, at), is unknown. Due to the Markov property, the trajectory distribution is πθ(τ) := πθ(s1, a1, . . . , sT, aT) = p(s1) TY t=1 πθ(at|st)p(st+1|st, at) Question: Can we have observations coming in at different frequencies? Yes. This is kind of an open problem. We will discuss this in hierarchical reinforcement learning where different levels receive observations at different frequencies. The objective of reinforcement learning is to maximise reward over all trajectories: θ∗:= arg max θ E τ∼πθ(τ) "X t r(st, at) # Trajectory Reward To train the model, we roll-out the policy multiple times J(θ) := Eτ∼πθ(τ) h r(τ) i ≃ 1 N X i X t r(si,t, ai,t) r(τ) := P t r(st, at) Number of


ultiple times J(θ) := Eτ∼πθ(τ) h r(τ) i ≃ 1 N X i X t r(si,t, ai,t) r(τ) := P t r(st, at) Number of samples Question: How to decide how long to roll-out a policy for? In this case we assume there is a fixed number of times to roll-out a policy, T. The infinite case requires a discount factor γ and will be discussed in a future lecture. There are multiple ways to improve the policy, including Q-learning, actor-critic. Why are there so many RL algorithms? This is because of the different trade-offs in each algorithm: • Continuous/Discrete actions • Is it easier to learn the environment or the policy? • Sample complexity. • Off/On policy. Off policy refers to improve policy without generating new samples.


le complexity. • Off/On policy. Off policy refers to improve policy without generating new samples. On policy refers to generating new samples for each policy change Policy gradient refers to improving the policy using gradient descent. Theorem 3.1. We have ∇θJ(θ) = Eτ∼πθ(τ)[∇θ log πθ(τ)r(τ)] = Eτ∼πθ(τ) " T X t=1 log πθ(at|st) ! T X t=1 r(st, at) !# Proof. Observe that J(θ) := Eτ∼πθ[r(τ)] = Z πθ(τ)r(τ) dτ 11


3 MARKOV DECISION PROCESSES AND POLICY GRADIENTS Figure 3.4: Example of policy gradient algorithm with colour/height corresponding to rewards We can differentiate under mild conditions on πθ and r: ∇θJ(θ) = Z ∇θπθ(τ)r(τ) dτ This integral is intractable and we would like it to be in a expectation form which can be approximated using Monte Carlo. To proceed further, we use the following fact d dx log f(x) = f ′(x) f(x) so f(x) d dx log f(x) = f ′(x). Hence the integrand can be rewritten as ∇θπθ(τ)r(τ) = πθ(τ)∇θ log πθ(τ)r(τ) so ∇θJ(θ) = Z πθ(τ)∇θ log πθ(τ)r(τ) dτ = Eτ∼πθ(τ)[∇θ log πθ(τ)r(τ)] We can take logarithm of πθ(τ), to get log πθ(τ) = log p(s1) + T X t=1 (log πθ(at|st) + log p(st+1|st, at)) When this is differentiated w.r.t.


πθ(τ) = log p(s1) + T X t=1 (log πθ(at|st) + log p(st+1|st, at)) When this is differentiated w.r.t. θ, the terms other than πθ become 0, therefore ∇θ log πθ(τ)r(τ) = T X t=1 log πθ(at|st) ! T X t=1 r(st, at) ! The expectation in ∇θJ(θ) can be then approximated with sampling and mean to create a tractable expression. This is the REINFORCE algorithm: 12


3 MARKOV DECISION PROCESSES AND POLICY GRADIENTS Algorithm: REINFORCE 1. Sample τ from πθ(a|s) from the policy 2. Calculate ˆ∇θJ(θ) := 1 N N X i=1 T X t=1 log πθ(ai,t|si,t) ! T X t=1 r(si,t, ai,t) ! 3. Update θ ←θ + α ˆ∇θJ(θ), where α is the step size. In comparison to supervised imitation learning, whose gradient can be written as ˆ∇θJ(θ) := 1 N N X i=1 T X t=1 log πθ(ai,t|si,t) ! policy gradient takes into account the rewards. Question: How to find out which action to credit when accounting for rewards? This is the credit assignment problem. It can be mitigated by a large number of samples. We will see how to reduce the number of samples in the next few lectures. Pros and cons of Policy Gradient: + Simple + Easy to combine with existing multi-task algorithms - Produces a high-variance gr


adient: + Simple + Easy to combine with existing multi-task algorithms - Produces a high-variance gradient Can be mitigated with baselines, trust regions - On-policy: Cannot reuse existing experience to update gradient. Question: If only a tiny fraction of search space leads to good actions, how can we prevent gradient vanishing or a long training phase for the policy? This is a problem with policy gradient. Usually people pre-train policy using e.g. mutation learning and then use policy gradient. Question: During roll-out, do we take the action of highest probability? We sample the action from πθ(·|st) instead of using arg maxa = πθ(a|st). Question: Is there guarantee about the policies we can get? There are some guarantees in convergence. 13