, pp. 2601–2606. Cognitive Science Society. Singh, S., Lewis, R. L., Barto, A. G., Sorg, J. (2010). Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70–82. Special issue on Active Learning and Intrinsically Motivated Exploration in Robots: Advances and Challenges. Singh, S. P., Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, 22(1-3):123–158. Skinner, B. F. (1938). The Behavior of Organisms: An Experimental Analysis. Appleton-Century, New York. Skinner, B. F. (1958). Reinforcement today. American Psychologist, 13(3):94–99. Skinner, B. F. (1963). Operant behavior. American Psychologist, 18(8):503–515. Sofge, D. A., White, D. A. (1992).


(1963). Operant behavior. American Psychologist, 18(8):503–515. Sofge, D. A., White, D. A. (1992). Applied learning: Optimal control for manufacturing. In D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, pp. 259–281. Van Nostrand Reinhold, New York. Sorg, J. D. (2011). The Optimal Reward Problem:Designing E↵ective Reward for Bounded Agents. PhD thesis, University of Michigan, Ann Arbor. Sorg, J., Lewis, R. L., Singh, S. P. (2010). Reward design via online gradient ascent. In Advances in Neural Information Processing Systems 23, pp. 2190–2198. Curran Associates, Inc. Sorg, J., Singh, S. (2010). Linear options. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, pp. 31–38.


eedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, pp. 31–38. Sorg, J., Singh, S., Lewis, R. (2010). Internal rewards mitigate agent boundedness. In Proceedings of the 27th International Conference on Machine Learning, pp. 1007–1014. Spence, K. W. (1947). The role of secondary reinforcement in delayed reward learning. Psychological Review, 54(1):1–8. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overﬁtting. Journal of Machine Learning Research, 15(1):1929–1958. Staddon, J. E. R. (1983). Adaptive Behavior and Learning. Cambridge University Press.


References 511 Stanﬁll, C., Waltz, D. (1986). Toward memory-based reasoning. Communications of the ACM, 29(12):1213–1228. Steinberg, E. E., Keiﬂin, R., Boivin, J. R., Witten, I. B., Deisseroth, K., Janak, P. H. (2013). A causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience, 16(7):966–973. Sterling, P., Laughlin, S. (2015). Principles of Neural Design. MIT Press, Cambridge, MA. Sternberg, S. (1963). Stochastic learning theory. In: Handbook of Mathematical Psychology, Volume II, R. D. Luce, R. R. Bush, and E. Galanter (Eds.). John Wiley & Sons. Sugiyama, M., Hachiya, H., Morimura, T. (2013). Statistical Reinforcement Learning: Modern Machine Learning Approaches. Chapman & Hall/CRC. Suri, R. E., Bargas, J., Arbib, M. A. (2001).


dern Machine Learning Approaches. Chapman & Hall/CRC. Suri, R. E., Bargas, J., Arbib, M. A. (2001). Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience, 103(1):65–85. Suri, R. E., Schultz, W. (1998). Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Experimental Brain Research, 121(3):350–354. Suri, R. E., Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience, 91(3):871–890. Sutton, R. S. (1978a). Learning theory support for a single channel theory of the brain. Unpublished report. Sutton, R. S. (1978b). Single channel theory: A neuronal theory of learning. Brain Theory Newsletter, 4:72–75.


S. (1978b). Single channel theory: A neuronal theory of learning. Brain Theory Newsletter, 4:72–75. Center for Systems Neuroscience, University of Massachusetts, Amherst, MA. Sutton, R. S. (1978c). A uniﬁed theory of expectation in classical and instrumental conditioning. Bachelors thesis, Stanford University. Sutton, R. S. (1984). Temporal Credit Assignment in Reinforcement Learning. PhD thesis, University of Massachusetts, Amherst. Sutton, R. S. (1988). Learning to predict by the method of temporal di↵erences. Machine Learning, 3(1):9–44 (important erratum p. 377). Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the 7th International Workshop on Machine Learning, pp. 216–224.


mic programming. In Proceedings of the 7th International Workshop on Machine Learning, pp. 216–224. Morgan Kaufmann. Sutton, R. S. (1991a). Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin, 2(4):160–163. ACM, New York. Sutton, R. S. (1991b). Planning by incremental dynamic programming. In Proceedings of the 8th International Workshop on Machine Learning, pp. 353–357. Morgan Kaufmann. Sutton, R. S. (Ed.) (1992a). Reinforcement Learning. Kluwer Academic Press. Reprinting of a special double issue on reinforcement learning, Machine Learning, 8(3-4). Sutton, R. S. (1992b). Adapting bias by gradient descent: An incremental version of delta-bardelta. Proceedings of the Tenth National Conference on Artiﬁcial Intelligence, pp. 171–176, MIT Press. Sutton, R.


ings of the Tenth National Conference on Artiﬁcial Intelligence, pp. 171–176, MIT Press. Sutton, R. S. (1992c). Gain adaptation beats least squares? Proceedings of the Seventh Yale Workshop on Adaptive and Learning Systems, pp. 161–166, Yale University, New Haven, CT. Sutton, R. S. (1995a). TD models: Modeling the world at a mixture of time scales. In Proceedings of the 12th International Conference on Machine Learning, pp. 531–539. Morgan Kaufmann. Sutton, R. S. (1995b). On the virtues of linear learning and trajectory distributions. In Proceedings of the Workshop on Value Function Approximation at The 12th International Conference on Machine Learning.


512 References Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems 8, pp. 1038–1044. MIT Press, Cambridge, MA. Sutton, R. S. (2009). The grand challenge of predictive empirical abstract knowledge. Working Notes of the IJCAI-09 Workshop on Grand Challenges for Reasoning from Experiences. Sutton, R. S. (2015a) Introduction to reinforcement learning with function approximation. Tutorial at the Conference on Neural Information Processing Systems, Montreal, December 7, 2015. Sutton, R. S. (2015b) True online Emphatic TD(λ): Quick reference and implementation guide. ArXiv:1507.07147. Code is available in Python and C++ by downloading the source ﬁles of this arXiv paper as a zip archiv


de is available in Python and C++ by downloading the source ﬁles of this arXiv paper as a zip archive. Sutton, R. S., Barto, A. G. (1981a). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88(2):135–170. Sutton, R. S., Barto, A. G. (1981b). An adaptive network that constructs and uses an internal model of its world. Cognition and Brain Theory, 3:217–246. Sutton, R. S., Barto, A. G. (1987). A temporal-di↵erence model of classical conditioning. In Proceedings of the Ninth Annual Conference of the Cognitive Science Society, pp. 355-378. Erlbaum, Hillsdale, NJ. Sutton, R. S., Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel and J.


R. S., Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel and J. Moore (Eds.), Learning and Computational Neuroscience: Foundations of Adaptive Networks, pp. 497–537. MIT Press, Cambridge, MA. Sutton, R. S., Maei, H. R., Precup, D., Bhatnagar, S., Silver, D., Szepesv´ari, Cs., Wiewiora, E. (2009a). Fast gradient-descent methods for temporal-di↵erence learning with linear function approximation. In Proceedings of the 26th International Conference on Machine Learning, pp. 993–1000. ACM, New York. Sutton, R. S., Szepesv´ari, Cs., Maei, H. R. (2009b). A convergent O(d2) temporal-di↵erence algorithm for o↵-policy learning with linear function approximation. In Advances in Neural Information Processing Systems 21, pp. 1609–1616. Curran Associates, Inc.


on. In Advances in Neural Information Processing Systems 21, pp. 1609–1616. Curran Associates, Inc. Sutton, R. S., Mahmood, A. R., Precup, D., van Hasselt, H. (2014). A new Q(λ) with interim forward view and Monte Carlo equivalence. In Proceedings of the International Conference on Machine Learning, 31. JMLR W&CP 32(2). Sutton, R. S., Mahmood, A. R., White, M. (2016). An emphatic approach to the problem of o↵-policy temporal-di↵erence learning. Journal of Machine Learning Research, 17(73):1–29. Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, pp. 1057–1063. MIT Press, Cambridge, MA. Sutton, R.


es in Neural Information Processing Systems 12, pp. 1057–1063. MIT Press, Cambridge, MA. Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., Precup, D. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Proceedings of the Tenth International Conference on Autonomous Agents and Multiagent Systems, pp. 761–768, Taipei, Taiwan. Sutton, R. S., Pinette, B. (1985). The learning of world models by connectionist networks. In Proceedings of the Seventh Annual Conference of the Cognitive Science Society, pp. 54–64. Sutton, R. S., Precup, D., Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artiﬁcial Intelligence, 112(1-2):181–211. Sutton, R.


emporal abstraction in reinforcement learning. Artiﬁcial Intelligence, 112(1-2):181–211. Sutton, R. S., Rafols, E., Koop, A. (2006). Temporal abstraction in temporal-di↵erence networks. In Advances in neural information processing systems, pp. 1313–1320. Sutton, R. S., Singh, S. P., McAllester, D. A. (2000). Comparing policy-gradient algorithms. Unpublished manuscript.


References 513 Sutton, R. S., Szepesv´ari, Cs., Geramifard, A., Bowling, M., (2008). Dyna-style planning with linear function approximation and prioritized sweeping. In Proceedings of the 24th Conference on Uncertainty in Artiﬁcial Intelligence, pp. 528–536. Sutton, R. S., Tanner, B. (2005). Temporal-di↵erence networks. In Advances in Neural Information Processing Systems 17, p. 1377–1384. Szepesv´ari, Cs. (2010). Algorithms for reinforcement learning. In Synthesis Lectures on Artiﬁcial Intelligence and Machine Learning, 4(1):1–103. Morgan and Claypool. Szita, I. (2012). Reinforcement learning in games. In M. Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art, pp. 539–577. Springer-Verlag Berlin Heidelberg. Tadepalli, P., Ok, D. (1994).


ng: State-of-the-Art, pp. 539–577. Springer-Verlag Berlin Heidelberg. Tadepalli, P., Ok, D. (1994). H-learning: A reinforcement learning method to optimize undiscounted average reward. Technical Report 94-30-01. Oregon State University, Computer Science Department, Corvallis. Tadepalli, P., Ok, D. (1996). Scaling up average reward reinforcement learning by approximating the domain models and the value function. In Proceedings of the 13th International Conference on Machine Learning, pp. 471–479. Takahashi, Y., Schoenbaum, G., and Niv, Y. (2008). Silencing the critics: Understanding the e↵ects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Frontiers in Neuroscience, 2(1):86–99. Tambe, M., Newell, A., Rosenbloom, P. S. (1990).


itic model. Frontiers in Neuroscience, 2(1):86–99. Tambe, M., Newell, A., Rosenbloom, P. S. (1990). The problem of expensive chunks and its solution by restricting expressiveness. Machine Learning, 5(3):299–348. Tan, M. (1991). Learning a cost-sensitive internal representation for reinforcement learning. In L. A. Birnbaum and G. C. Collins (Eds.), Proceedings of the 8th International Workshop on Machine Learning, pp. 358–362. Morgan Kaufmann. Tanner, B. (2006). Temporal-Di↵erence Networks. MSc thesis, University of Alberta. Taylor, G., Parr, R. (2009). Kernelized value function approximation for reinforcement learning. In Proceedings of the 26th International Conference on Machine Learning, pp. 1017–1024. ACM, New York. Taylor, M. E., Stone, P. (2009).


onal Conference on Machine Learning, pp. 1017–1024. ACM, New York. Taylor, M. E., Stone, P. (2009). Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10:1633–1685. Tesauro, G. (1986). Simple neural models of classical conditioning. Biological Cybernetics, 55(2-3):187–200. Tesauro, G. (1992). Practical issues in temporal di↵erence learning. Machine Learning, 8(3-4):257–277. Tesauro, G. (1994). TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation, 6(2):215–219. Tesauro, G. (1995). Temporal di↵erence learning and TD-Gammon. Communications of the ACM, 38(3):58–68. Tesauro, G. (2002). Programming backgammon using self-teaching neural nets. Artiﬁcial Intelligence, 134(1-2):181–199.


. Programming backgammon using self-teaching neural nets. Artiﬁcial Intelligence, 134(1-2):181–199. Tesauro, G., Galperin, G. R. (1997). On-line policy improvement using Monte-Carlo search. In Advances in Neural Information Processing Systems 9, pp. 1068–1074. MIT Press, Cambridge, MA. Tesauro, G., Gondek, D. C., Lechner, J., Fan, J., Prager, J. M. (2012). Simulation, learning, and optimization techniques in Watson’s game strategies. IBM Journal of Research and Development, 56(3-4):16–1–16–11. Tesauro, G., Gondek, D. C., Lenchner, J., Fan, J., Prager, J. M. (2013). Analysis of Watson’s strategies for playing Jeopardy! Journal of Artiﬁcial Intelligence Research, 47:205–251. Tham, C. K. (1994). Modular On-Line Function Approximation for Scaling up Reinforcement Learning.


. Tham, C. K. (1994). Modular On-Line Function Approximation for Scaling up Reinforcement Learning. PhD thesis, University of Cambridge.


514 References Thathachar, M. A. L., Sastry, P. S. (1985). A new approach to the design of reinforcement schemes for learning automata. IEEE Transactions on Systems, Man, and Cybernetics, 15(1):168–175. Thathachar, M., Sastry, P. S. (2002). Varieties of learning automata: an overview. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 36(6):711–722. Thathachar, M., Sastry, P. S. (2011). Networks of Learning Automata: Techniques for Online Stochastic Optimization. Springer Science & Business Media. Theocharous, G., Thomas, P. S., Ghavamzadeh, M. (2015). Personalized ad recommendation for life-time value optimization guarantees. In Proceedings of the Twenty-Fourth International Joint Conference on Artiﬁcial Intelligence. AAAI Press, Palo Alto, CA. Thistlethwaite, D.


rnational Joint Conference on Artiﬁcial Intelligence. AAAI Press, Palo Alto, CA. Thistlethwaite, D. (1951). A critical review of latent learning and related experiments. Psychological Bulletin, 48(2):97–129. Thomas, P. S. (2014). Bias in natural actor–critic algorithms. In Proceedings of the 31st International Conference on Machine Learning, JMLR W&CP 32(1), pp. 441–448. Thomas, P. S. (2015). Safe Reinforcement Learning. PhD thesis, University of Massachusetts, Amherst. Thomas, P. S., Brunskill, E. (2017). Policy gradient methods for reinforcement learning with function approximation and action-dependent baselines. ArXiv:1706.06643. Thomas, P. S., Theocharous, G., Ghavamzadeh, M. (2015). High-conﬁdence o↵-policy evaluation.


06643. Thomas, P. S., Theocharous, G., Ghavamzadeh, M. (2015). High-conﬁdence o↵-policy evaluation. In Proceedings of the Twenty-Ninth AAAI Conference on Artiﬁcial Intelligence, pp. 3000–3006. AAAI Press, Menlo Park, CA. Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285–294. Thompson, W. R. (1934). On the theory of apportionment. American Journal of Mathematics, 57: 450–457. Thon, M. (2017). Spectral Learning of Sequential Systems. PhD thesis, Jacobs University Bremen. Thon, M., Jaeger, H. (2015). Links between multiplicity automata, observable operator models and predictive state representations: a uniﬁed learning framework. The Journal of Machine Learning Research, 16(1):103–147.


resentations: a uniﬁed learning framework. The Journal of Machine Learning Research, 16(1):103–147. Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. The Psychological Review, Series of Monograph Supplements, II(4). Thorndike, E. L. (1911). Animal Intelligence. Hafner, Darien, CT. Thorp, E. O. (1966). Beat the Dealer: A Winning Strategy for the Game of Twenty-One. Random House, New York. Tian, T. (in preparation) An Empirical Study of Sliding-Step Methods in Temporal Di↵erence Learning. M.Sc thesis, University of Alberta, Edmonton. Tieleman, T., Hinton, G. (2012). Lecture 6.5–RMSProp. COURSERA: Neural networks for machine learning 4.2:26–31. Tolman, E. C. (1932). Purposive Behavior in Animals and Men. Century, New York. Tolman, E.


2:26–31. Tolman, E. C. (1932). Purposive Behavior in Animals and Men. Century, New York. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55(4):189–208. Tsai, H.-S., Zhang, F., Adamantidis, A., Stuber, G. D., Bonci, A., de Lecea, L., Deisseroth, K. (2009). Phasic ﬁring in dopaminergic neurons is suﬃcient for behavioral conditioning. Science, 324(5930):1080–1084. Tsetlin, M. L. (1973). Automaton Theory and Modeling of Biological Systems. Academic Press, New York.


References 515 Tsitsiklis, J. N. (1994). Asynchronous stochastic approximation and Q-learning. Machine Learning, 16(3):185–202. Tsitsiklis, J. N. (2002). On the convergence of optimistic policy iteration. Journal of Machine Learning Research, 3:59–72. Tsitsiklis, J. N., Van Roy, B. (1996). Feature-based methods for large scale dynamic programming. Machine Learning, 22(1-3):59–94. Tsitsiklis, J. N., Van Roy, B. (1997). An analysis of temporal-di↵erence learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690. Tsitsiklis, J. N., Van Roy, B. (1999). Average cost temporal-di↵erence learning. Automatica, 35(11):1799–1808. Turing, A. M. (1948). Intelligent machinery. In B. Jack Copeland (Ed.) (2004), The Essential Turing, pp. 410–432.


(1948). Intelligent machinery. In B. Jack Copeland (Ed.) (2004), The Essential Turing, pp. 410–432. Oxford University Press, Oxford. Ungar, L. H. (1990). A bioreactor benchmark for adaptive network-based process control. In W. T. Miller, R. S. Sutton, and P. J. Werbos (Eds.), Neural Networks for Control, pp. 387–402. MIT Press, Cambridge, MA. Unnikrishnan, K. P., Venugopal, K. P. (1994). Alopex: A correlation-based learning algorithm for feedforward and recurrent neural networks. N eural Computation, 6(3): 469–490. Urbanczik, R., Senn, W. (2009). Reinforcement learning in populations of spiking neurons. Nature neuroscience, 12(3):250–252. Urbanowicz, R. J., Moore, J. H. (2009). Learning classiﬁer systems: A complete introduction, review, and roadmap.


J., Moore, J. H. (2009). Learning classiﬁer systems: A complete introduction, review, and roadmap. Journal of Artiﬁcial Evolution and Applications. 10.1155/2009/736398. Valentin, V. V., Dickinson, A., O’Doherty, J. P. (2007). Determining the neural substrates of goal-directed learning in the human brain. The Journal of Neuroscience, 27(15):4019–4026. van Hasselt, H. (2010). Double Q-learning. In Advances in Neural Information Processing Systems 23, pp. 2613–2621. Curran Associates, Inc. van Hasselt, H. (2011). Insights in Reinforcement Learning: Formal Analysis and Empirical Evaluation of Temporal-di↵erence Learning. SIKS dissertation series number 2011-04. van Hasselt, H. (2012). Reinforcement learning in continuous state and action spaces. In M. Wiering and M.


selt, H. (2012). Reinforcement learning in continuous state and action spaces. In M. Wiering and M. van Otterlo (Eds.), Reinforcement Learning: State-of-the-Art, pp. 207–251. Springer-Verlag Berlin Heidelberg. van Hasselt, H., Sutton, R. S. (2015). Learning to predict independent of span. ArXiv:1508.04582. Van Roy, B., Bertsekas, D. P., Lee, Y., Tsitsiklis, J. N. (1997). A neuro-dynamic programming approach to retailer inventory management. In Proceedings of the 36th IEEE Conference on Decision and Control, Vol. 4, pp. 4052–4057. van Seijen, H. (2011). Reinforcement Learning under Space and Time Constraints. University of Amsterdam PhD thesis. Hague: TNO. van Seijen, H. (2016). E↵ective multi-step temporal-di↵erence learning for non-linear function approximation. ArXiv:1608.05151.


ive multi-step temporal-di↵erence learning for non-linear function approximation. ArXiv:1608.05151. van Seijen, H., Sutton, R. S. (2013). Eﬃcient planning in MDPs by small backups. In: Proceedings of the 30th International Conference on Machine Learning, pp. 361–369. van Seijen, H., Sutton, R. S. (2014). True online TD(λ). In Proceedings of the 31st International Conference on Machine Learning, pp. 692–700. JMLR W&CP 32(1), van Seijen, H., Mahmood, A. R., Pilarski, P. M., Machado, M. C., Sutton, R. S. (2016). True online temporal-di↵erence learning. Journal of Machine Learning Research, 17(145):1–40. van Seijen, H., Van Hasselt, H., Whiteson, S., Wiering, M. (2009). A theoretical and empirical analysis of Expected Sarsa.


selt, H., Whiteson, S., Wiering, M. (2009). A theoretical and empirical analysis of Expected Sarsa. In IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pp. 177–184.


516 References van Seijen, H., Whiteson, S., van Hasselt, H., Wiering, M. (2011). Exploiting best-match equations for eﬃcient reinforcement learning. Journal of Machine Learning Research 12:2045– 2094. Varga, R. S. (1962). Matrix Iterative Analysis. Englewood Cli↵s, NJ: Prentice-Hall. Vasilaki, E., Fr´emaux, N., Urbanczik, R., Senn, W., Gerstner, W. (2009). Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLoS Computational Biology, 5(12). Viswanathan, R., Narendra, K. S. (1974). Games of stochastic automata. IEEE Transactions on Systems, Man, and Cybernetics, 4(1):131–135. Wagner, A. R. (2008). Evolution of an elemental theory of Pavlovian conditioning. Learning & Behavior, 36(3):253–265. Walter, W. G. (1950).


emental theory of Pavlovian conditioning. Learning & Behavior, 36(3):253–265. Walter, W. G. (1950). An imitation of life. Scientiﬁc American, 182(5):42–45. Walter, W. G. (1951). A machine that learns. Scientiﬁc American, 185(2):60–63. Waltz, M. D., Fu, K. S. (1965). A heuristic approach to reinforcement learning control systems. IEEE Transactions on Automatic Control, 10(4):390–398. Watkins, C. J. C. H. (1989). Learning from Delayed Rewards. PhD thesis, University of Cambridge. Watkins, C. J. C. H., Dayan, P. (1992). Q-learning. Machine Learning, 8(3-4):279–292. Werbos, P. J. (1977). Advanced forecasting methods for global crisis warning and models of intelligence. General Systems Yearbook, 22(12):25–38. Werbos, P. J. (1982). Applications of advances in nonlinear sensitivity analysis.


ok, 22(12):25–38. Werbos, P. J. (1982). Applications of advances in nonlinear sensitivity analysis. In R. F. Drenick and F. Kozin (Eds.), System Modeling and Optimization, pp. 762–770. Springer-Verlag. Werbos, P. J. (1987). Building and understanding adaptive systems: A statistical/numerical approach to factory automation and brain research. IEEE Transactions on Systems, Man, and Cybernetics, 17(1):7–20. Werbos, P. J. (1988). Generalization of back propagation with applications to a recurrent gas market model. Neural Networks, 1(4):339–356. Werbos, P. J. (1989). Neural networks for control and system identiﬁcation. In Proceedings of the 28th Conference on Decision and Control, pp. 260–265. IEEE Control Systems Society. Werbos, P. J. (1992).


onference on Decision and Control, pp. 260–265. IEEE Control Systems Society. Werbos, P. J. (1992). Approximate dynamic programming for real-time control and neural modeling. In D. A. White and D. A. Sofge (Eds.), Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, pp. 493–525. Van Nostrand Reinhold, New York. Werbos, P. J. (1994). The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting (Vol. 1). John Wiley and Sons. Wiering, M., Van Otterlo, M. (2012). Reinforcement Learning: State-of-the-Art. Springer-Verlag Berlin Heidelberg. White, A. (2015). Developing a Predictive Approach to Knowledge. PhD thesis, University of Alberta, Edmonton. White, D. J. (1969). Dynamic Programming. Holden-Day, San Francisco. White, D. J. (1985).


Edmonton. White, D. J. (1969). Dynamic Programming. Holden-Day, San Francisco. White, D. J. (1985). Real applications of Markov decision processes. Interfaces, 15(6):73–83. White, D. J. (1988). Further real applications of Markov decision processes. Interfaces, 18(5):55–61. White, D. J. (1993). A survey of applications of Markov decision processes. Journal of the Operational Research Society, 44(11):1073–1096. White, A., White, M. (2016). Investigating practical linear temporal di↵erence learning. In Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems, pp. 494–502.


References 517 Whitehead, S. D., Ballard, D. H. (1991). Learning to perceive and act by trial and error. Machine Learning, 7(1):45–83. Whitt, W. (1978). Approximations of dynamic programs I. Mathematics of Operations Research, 3(3):231–243. Whittle, P. (1982). Optimization over Time, vol. 1. Wiley, New York. Whittle, P. (1983). Optimization over Time, vol. 2. Wiley, New York. Wickens, J., K¨otter, R. (1995). Cellular models of reinforcement. In J. C. Houk, J. L. Davis and D. G. Beiser (Eds.), Models of Information Processing in the Basal Ganglia, pp. 187–214. MIT Press, Cambridge, MA. Widrow, B., Gupta, N. K., Maitra, S. (1973). Punish/reward: Learning with a critic in adaptive threshold systems. IEEE Transactions on Systems, Man, and Cybernetics, 3(5):455–465. Widrow, B., Ho↵, M. E.


d systems. IEEE Transactions on Systems, Man, and Cybernetics, 3(5):455–465. Widrow, B., Ho↵, M. E. (1960). Adaptive switching circuits. In 1960 WESCON Convention Record Part IV, pp. 96–104. Institute of Radio Engineers, New York. Reprinted in J. A. Anderson and E. Rosenfeld, Neurocomputing: Foundations of Research, pp. 126–134. MIT Press, Cambridge, MA, 1988. Widrow, B., Smith, F. W. (1964). Pattern-recognizing control systems. In J. T. Tou and R. H. Wilcox (Eds.), Computer and Information Sciences, pp. 288–317. Spartan, Washington, DC. Widrow, B., Stearns, S. D. (1985). Adaptive Signal Processing. Prentice-Hall, Englewood Cli↵s, NJ. Wiener, N. (1964). God and Golem, Inc: A Comment on Certain Points where Cybernetics Impinges on Religion. MIT Press, Cambridge, MA. Wiewiora, E. (2003).


rtain Points where Cybernetics Impinges on Religion. MIT Press, Cambridge, MA. Wiewiora, E. (2003). Potential-based shaping and Q-value initialization are equivalent. Journal of Artiﬁcial Intelligence Research, 19:205–208. Williams, R. J. (1986). Reinforcement learning in connectionist networks: A mathematical analysis. Technical Report ICS 8605. Institute for Cognitive Science, University of California at San Diego, La Jolla. Williams, R. J. (1987). Reinforcement-learning connectionist systems. Technical Report NU-CCS-87-3. College of Computer Science, Northeastern University, Boston. Williams, R. J. (1988). On the use of backpropagation in associative reinforcement learning. In Proceedings of the IEEE International Conference on Neural Networks, pp. I-263–I-270.


learning. In Proceedings of the IEEE International Conference on Neural Networks, pp. I-263–I-270. IEEE San Diego section and IEEE TAB Neural Network Committee. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256. Williams, R. J., Baird, L. C. (1990). A mathematical analysis of actor–critic architectures for learning optimal controls through incremental dynamic programming. In Proceedings of the Sixth Yale Workshop on Adaptive and Learning Systems, pp. 96–101. Center for Systems Science, Dunham Laboratory, Yale University, New Haven. Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., Niv, Y. (2014). Orbitofrontal cortex as a cognitive map of task space. Neuron, 81(2):267–279. Wilson, S. W.


(2014). Orbitofrontal cortex as a cognitive map of task space. Neuron, 81(2):267–279. Wilson, S. W. (1994). ZCS, A zeroth order classiﬁer system. Evolutionary Computation, 2(1):1–18. Wise, R. A. (2004). Dopamine, learning, and motivation. Nature Reviews Neuroscience, 5(6):1–12. Witten, I. H. (1976a). Learning to Control. University of Essex PhD thesis. Witten, I. H. (1976b). The apparent conﬂict between estimation and control—A survey of the two-armed problem. Journal of the Franklin Institute, 301(1-2):161–189.


518 References Witten, I. H. (1977). An adaptive optimal controller for discrete-time Markov environments. Information and Control, 34(4):286–295. Witten, I. H., Corbin, M. J. (1973). Human operators and automatic adaptive controllers: A comparative study on a particular control task. International Journal of Man–Machine Studies, 5(1):75–104. Woodbury, T., Dunn, C., and Valasek, J. (2014). Autonomous soaring using reinforcement learning for trajectory generation. In 52nd Aerospace Sciences Meeting, p. 0990. Woodworth, R. S. (1938). Experimental Psychology. New York: Henry Holt and Company. Xie, X., Seung, H. S. (2004). Learning in neural networks by reinforcement of irregular spiking. Physical Review E, 69(4):041909. Xu, X., Xie, T., Hu, D., Lu, X. (2005).


ment of irregular spiking. Physical Review E, 69(4):041909. Xu, X., Xie, T., Hu, D., Lu, X. (2005). Kernel least-squares temporal di↵erence learning. International Journal of Information Technology, 11(9):54–63. Yagishita, S., Hayashi-Takagi, A., Ellis-Davies, G. C. R., Urakubo, H., Ishii, S., Kasai, H. (2014). A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science, 345(6204):1616–1619. Yee, R. C., Saxena, S., Utgo↵, P. E., Barto, A. G. (1990). Explaining temporal di↵erences to create useful concepts for evaluating states. In Proceedings of the Eighth National Conference on Artiﬁcial Intelligence, pp. 882–888. AAAI Press, Menlo Park, CA. Yin, H. H., Knowlton, B. J. (2006). The role of the basal ganglia in habit formation.


nlo Park, CA. Yin, H. H., Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nature Reviews Neuroscience, 7(6):464–476. Young, P. (1984). Recursive Estimation and Time-Series Analysis. Springer-Verlag, Berlin. Yu, H. (2010). Convergence of least squares temporal di↵erence methods under general conditions. International Conference on Machine Learning 27, pp. 1207–1214. Yu, H. (2012). Least squares temporal di↵erence methods: An analysis under general conditions. SIAM Journal on Control and Optimization, 50(6):3310–3343. Yu, H. (2015). On convergence of emphatic temporal-di↵erence learning. In Proceedings of the 28th Annual Conference on Learning Theory, JMLR W&CP 40. Also ArXiv:1506.02582. Yu, H. (2016).


the 28th Annual Conference on Learning Theory, JMLR W&CP 40. Also ArXiv:1506.02582. Yu, H. (2016). Weak convergence properties of constrained emphatic temporal-di↵erence learning with constant and slowly diminishing stepsize. Journal of Machine Learning Research, 17(220):1–58. Yu, H. (2017). On convergence of some gradient-based temporal-di↵erences algorithms for o↵-policy learning. ArXiv:1712.09652. Yu, H., Mahmood, A. R., Sutton, R. S. (2017). On generalized bellman equations and temporaldi↵erence learning. ArXiv:17041.04463. A summary appeared in Proceedings of the Canadian Conference on Artiﬁcial Intelligence, pp. 3–14. Springer.


Index Page numbers in italics are recommended to be consulted ﬁrst. Page numbers in bold contain boxed algorithms. k-armed bandits, 25–45 absorbing state, 57 access-control queuing example, 256 action preferences, 322, 329, 336, 455 in bandit problems, 37, 42 action-value function, see value function, action action-value methods, 321 for bandit problems, 27 actor–critic, 21, 239, 321, 331–332, 338, 406 advantage, A2C, 338 one-step (episodic), 332 with eligibility traces (episodic), 332 with eligibility traces (continuing), 333 neural, 395–415 addiction, 409–410 afterstates, 137, 140, 181, 182, 191, 424, 430 agent–environment interface, 47–58, 466 all-actions algorithm, 326 AlphaGo, AlphaGo Zero, AlphaZero, 441–450 Andreae, John, 17, 21, 69, 89 ANN, see artiﬁcial neural networks application


ero, AlphaZero, 441–450 Andreae, John, 17, 21, 69, 89 ANN, see artiﬁcial neural networks applications and case studies, 421–457 approximate dynamic programming, 15 artiﬁcial intelligence, xvii, 1, 472, 478 artiﬁcial neural networks, 223–228, 238–240, 395–398, 423, 430, 436–450, 472 associative reinforcement learning, 45, 418 associative search, 41 asynchronous dynamic programming, 85, 88 Atari video game play, 436–441 auxiliary tasks, 460–461, 468, 474 average reward setting, 249–255, 258, 464 averagers, 264 backgammon, 11, 21, 182, 184, 421–426 backpropagation, 21, 225–227, 239, 407, 424, 436, 439 backup diagram, 60, 139 for dynamic programming, 59, 61, 64, 172 for Monte Carlo methods, 94 for Q-learning, 134 for TD(0), 121 for Sarsa, 129 for Expected Sarsa, 134 for Sarsa(λ), 304 for TD(λ)


or Q-learning, 134 for TD(0), 121 for Sarsa, 129 for Expected Sarsa, 134 for Sarsa(λ), 304 for TD(λ), 289 for Q(λ), 313 for Tree Backup(λ), 314 for Truncated TD(λ), 296 for n-step Q(σ), 155 for n-step Expected Sarsa, 146 for n-step Sarsa, 146 for n-step TD, 142 for n-step Tree Backup, 152 for Samuel’s Checker Player, 428 compound, 288 half backups, 62 backward view of eligibility traces, 288, 293 Baird’s counterexample, 261–264, 280, 283, 285 bandit algorithm, simple, 32 bandit problems, 25–45 basal ganglia, 386 baseline, 37–40, 329, 330, 338 behavior policy, 103, 110, see o↵-policy learning Bellman equation, 14 for v⇡, 59 for q⇡, 78 for optimal value functions: v⇤and q⇤, 63 di↵erential, 250 for options, 463 Bellman error, 268, 270, 272, 273 learnability of, 274–278 vector, 267–269 Bellman


for options, 463 Bellman error, 268, 270, 272, 273 learnability of, 274–278 vector, 267–269 Bellman operator, 267–269, 286 Bellman residual, 286, see Bellman error Bellman, Richard, 14, 71, 89, 241 binary features, 215, 222, 245, 304, 305 bioreactor example, 51 blackjack example, 93–94, 99, 106 blocking maze example, 166 bootstrapping, 89, 189, 308 n-step, 141–158, 255 and dynamic programming, 89 and function approximation, 208, 264–274


520 Index and Monte Carlo methods, 95 and stability, 263–265 and TD learning, 120 assessment of, 124–128, 248, 264, 291, 318 in psychology, 345, 349, 354, 355 parameter (λ or n), 291, 307, 399 BOXES, 18, 71, 237 branching factor, 173–177, 422 breakfast example, 5, 22 bucket-brigade algorithm, 19, 21, 139 catastrophic interference, 472 certainty-equivalence estimate, 128 chess, 4, 20, 54, 182, 450 classical conditioning, 20, 343–357 blocking, 371 and higher-order conditioning, 345–355 delay and trace conditioning, 344 Rescorla-Wagner model, 346–349 TD model, 349–357 classiﬁer systems, 19, 21 cli↵walking example, 132, 133 CMAC, see tile coding coarse coding, 215–220, 238 cognitive maps, 363–364 collective reinforcement learning, 404–407 complex backups, see compound update compound stimulus,


4 collective reinforcement learning, 404–407 complex backups, see compound update compound stimulus, 345, 346–356, 371, 382 compound update/backup, 288, 319 conditioned/unconditioned stimulus, conditioned response (CS/US, CR), 344 constant-↵MC, 120 contextual bandits, 41 continuing tasks, 54, 57, 70, 124, 249, 294 continuous action, 73, 244, 335–336 continuous state, 73, 223, 238 continuous time, 11, 71 control and prediction, 342 control theory, 4, 71 control variates, 150–152, 155, 281 and eligibility traces, 309–312 credit assignment, 11, 17, 19, 47, 294, 401 in psychology, 346, 361 structural, 385, 405, 407 critic, 18, 239, 346, 417, see actor–critic cumulant, 459 curiosity, 474 curse of dimensionality, 4, 14, 221, 231 cybernetics, xvii, 477 deadly triad, 264 deep learning, 12, 223, 44


dimensionality, 4, 14, 221, 231 cybernetics, xvii, 477 deadly triad, 264 deep learning, 12, 223, 441, 472–474, 479 deep reinforcement learning, 236 deep residual learning, 227 delayed reinforcement, 361–363 delayed reward, 1, 47, 249 dimensions of reinforcement learning methods, 189–191 direct and indirect RL, 162, 164, 192 discounting, 55, 199, 243, 249, 282, 324, 328, 427, 459 in pole balancing, 56 state dependent, 307 deprecated, 253, 256 distribution models, 159, 185 dopamine, 377, 381–387, 413–419 and addiction, 409–410 double learning, 134–136, 140 DP, see dynamic programming driving-home example, 122–123 Dyna architecture, 164, 161–170 dynamic programming, 13–15, 73–90, 174, 262 and artiﬁcial intelligence, 89 and function approximation, 241 and options, 463 and the deadly triad, 26


artiﬁcial intelligence, 89 and function approximation, 241 and options, 463 and the deadly triad, 264 computational eﬃciency of, 87 eligibility traces, 287–320, 350, 362, 398–403 accumulating, 300, 306, 310 replacing, 301, 306 dutch, 300–303 contingent/non-contingent, 399–403, 411 o↵-policy, 309–316 with state-dependent λ and γ, 309–316 Emphatic-TD methods, 234–235, 315 o↵-policy, 281–282 environment, 47–58 episodes, episodic tasks, 11, 54–58, 91 error reduction property, 144, 288 evaluative feedback, 17, 25, 47 evolution, 7, 359, 374, 471 evolutionary methods, 7, 8, 9, 11, 19 expected approximate value, 148, 155 Expected Sarsa, 133, see also Sarsa, Expected expected update, 75, 172–181, 189 experience replay, 440–441 explore/exploit dilemma, 3, 103, 472 exploring starts, 96, 98–100, 178


Index 521 feature construction, 210–223 ﬁnal time step (T), 54 Fourier basis, 211–215 function approximation, 195–200 gambler’s example, 84 game theory, 19 gazelle calf example, 5 general value functions (GVFs), 459–463, 474 generalized policy iteration (GPI), 86–87, 92, 97, 138, 189 genetic algorithms, 19 Gittins index, 43 gliding/soaring case study, 453–457 goal, see reward signal golf example, 61, 63, 66 gradient, 201 gradient descent, see stochastic gradient descent Gradient-TD methods, 278–281, 314–315 greedy or "-greedy as exploiting, 26–28 as shortsighted, 64 "-greedy policies, 100 gridworld examples, 60, 65, 76, 147 cli↵walking, 132 Dyna blocking maze, 166 Dyna maze, 164 Dyna shortcut maze, 167 windy, 130, 131 habitual and goal-directed control, 364–368 hedonistic neurons, 402–404


t maze, 167 windy, 130, 131 habitual and goal-directed control, 364–368 hedonistic neurons, 402–404 heuristic search, 181–183, 190 as sequences of backups, 183 in Samuel’s checkers player, 426 in TD-Gammon, 425 history of reinforcement learning, 13–22 Holland, John, 19, 21, 44, 139, 241 Hull, Clark, 16, 359, 360, 362–363 importance sampling, 103–117, 151, 257 ratio, 104, 148, 258 weighted and ordinary, 105, 106 and eligibility traces, 309–312 and inﬁnite variance, 106 discounting aware, 112–113 incremental implementation, 109 per-decision, 114–115 n-step, 148–156 incremental implementation of averages, 30–33 of weighted averages, 109 instrumental conditioning, 357–361, see also Law of E↵ect and motivation, 360–361 Thorndike’s puzzle boxes, 358 interest and emphasis, 234–235, 282, 316 inver


and motivation, 360–361 Thorndike’s puzzle boxes, 358 interest and emphasis, 234–235, 282, 316 inverse reinforcement learning, 470 Jack’s car rental example, 81–82, 137, 210 kernel-based function approximation, 232–233 Klopf, A. Harry, xv, xvii, 19–21, 402–404, 411 latent learning, 192, 363, 366 Law of E↵ect, 15–16, 45, 343, 358–361, 417 learning automata, 18 Least Mean Square (LMS) algorithm, 279, 301 Least-Squares TD (LSTD), 228–229 linear function approx., 204–209, 266–269 linear programming, 87, 90 local and global optima, 200 Markov decision process (MDP), 2, 14, 47–71 Markov property, 49, 115, 465–468 Markov reward process (MRP), 125 maximization bias, 134–136 maximum-likelihood estimate, 128 MC, see Monte Carlo methods Mean Square Bellman Error, BE, 268 Projected Bellman Error, PBE,


te, 128 MC, see Monte Carlo methods Mean Square Bellman Error, BE, 268 Projected Bellman Error, PBE, 269 Return Error, RE, 275 TD Error, TDE, 270 Value Error, VE, 199–200 memory-based function approx., 230–232 Michie, Donald, 17, 71, 117 Minsky, Marvin, 16, 17, 20, 89 model of the environment, 7, 159 model-based and model-free methods, 7, 159 in animal learning, 363–368 model-based reinforcement learning, 159–193 in neuroscience, 407–409 Monte Carlo methods, 91–117 ﬁrst- and every-visit MC, 92 ﬁrst-visit MC control, 101 ﬁrst-visit MC prediction, 92


522 Index gradient method for v⇡, 202 Monte Carlo ES (Exploring Starts), 99 o↵-policy control, 111, 110–112 o↵-policy prediction, 103–109, 110 Monte Carlo Tree Search (MCTS), 185–188 motivation, 360–361 mountain car example, 244–248, 305, 306 multi-armed bandits, 25–45 n-step methods, 141–158 Q(σ), 156 Sarsa, 147, 247 di↵erential, 255 o↵-policy, 149 TD, 144 Tree Backup, 154 truncated λ-return, 295 naughts and crosses, see tic-tac-toe neural networks, see artiﬁcial neural networks neurodynamic programming, 15 neuroeconomics, 413, 419 neuroscience, 4, 21, 377–419 nonstationarity, 30, 32–36, 44, 255 inherent, 91, 198 notation, xiii, xix observations, 464 o↵-policy methods, 257–286 vs on-policy methods, 100, 103 Monte Carlo, 103–115 Q-learning, 131 Expected Sarsa, 133–134 n-step, 148–156 n-ste


methods, 100, 103 Monte Carlo, 103–115 Q-learning, 131 Expected Sarsa, 133–134 n-step, 148–156 n-step Q(σ), 156 n-step Sarsa, 149 n-step Tree Backup, 154 and eligibility traces, 309–316 Emphatic-TD(λ), 315 GQ(λ), 315 GTD(λ), 314 HTD(λ), 315 Q(λ), 312–314 Tree Backup(λ), 312–314 reducing variance, 283–284 on-policy distribution, 175, 199, 208, 258, 262, 281, 282 vs uniform distribution, 176 on-policy methods, 100 actor–critic, 332, 333 approximate control, 244, 247, 251, 255 prediction, 202, 203, 209 Monte Carlo, 101, 100–103, 328, 330 n-step, 144, 147 Sarsa, 130, 129–131 TD(0), 120, 119–128 with eligibility traces, 293, 300, 305, 307 operant conditioning, see instrumental learning optimal control, 2, 14–15, 21 optimistic initial values, 34–35, 192 optimizing memory control, 432–436 options


ntrol, 2, 14–15, 21 optimistic initial values, 34–35, 192 optimizing memory control, 432–436 options, 461–464 models of, 462 pain and pleasure, 6, 16, 413 Partially Observable MDPs (POMDPs), 467 Pavlov, Ivan, 16, 343–345, 362 Pavlovian conditioning, see classical conditioning control, 343, 371, 373, 478 personalizing web services, 450–453 planning, 3, 5, 7, 11, 138, 159–193 in psychology, 363, 364, 366 with learned models, 161–168, 473 with options, 461, 463 policy, 6, 41, 58 hierarchical, 462 soft and "-soft, 100–103, 110 policy approximation, 321–324 policy evaluation, 74–76, see also prediction iterative, 75 policy gradient methods, 321–338 REINFORCE, 328, 330 actor–critic, 332, 333 policy gradient theorem, 324–326 proof, episodic case, 325 proof, continuing case, 334 policy improvement


y gradient theorem, 324–326 proof, episodic case, 325 proof, continuing case, 334 policy improvement, 76–80 theorem, 78, 101 policy iteration, 14, 80, 80–82 polynomial basis, 210–211 prediction, 74–76, see also policy evaluation and control, 342 Monte Carlo, 92–97 o↵-policy, 103–108 TD, 119–126 with approximation, 197–242 prior knowledge, 12, 34, 54, 137, 236, 324, 471


Index 523 prioritized sweeping, 170, 168–171 projected Bellman error, 285 vector, 267, 269 proximal TD methods, 286 pseudo termination, 282, 308 psychology, 4, 13, 19, 20, 341–376 Q(λ), Watkins’s, 312–314 Q-function, see action-value function Q-learning, 21, 131, 131–135 double, 136 Q-planning, 161 Q(σ), 156, 154–156 queuing example, 252 R-learning, 256 racetrack exercise, 111 radial basis functions (RBFs), 221–222 random walk, 95 5-state, 125, 126, 127 19-state, 144, 291 TD(λ) results on, 294, 295, 299 1000-state, 203–209, 217, 218 Fourier and polynomial bases, 214 real-time dynamic programming, 177–180 recycling robot example, 52 REINFORCE, 328, 326–331 with baseline, 330 reinforcement learning, 1–22 reinforcement signal, 380 representation learning, 473 residual-gradient algorithm, 272–


rning, 1–22 reinforcement signal, 380 representation learning, 473 residual-gradient algorithm, 272–274, 277 naive, 270, 271 return, 54–58 n-step, 143 for Q(σ), 155 for action values, 146 for Expected Sarsa, 148 for Tree Backup, 153 with control variates, 150, 151 with function approximation, 209 di↵erential, 250, 255, 334 ﬂat partial, 113 with state-dependent termination, 308 λ-return, 288–291 truncated, 296 reward prediction error hypothesis, 381–383, 387–395 reward signal, 1, 6, 48, 53, 361, 380, 383, 397 and reinforcement, 373–375, 380–381 design of, 469–472, 477 intrinsic, 474 sparse, 469–470 rod maneuvering example, 171 rollout algorithms, 183–185 root mean square (RMS) error, 125 safety, 434, 478 sample and expected updates, 121, 170–174 sample or simulation model, 115 sample-averag


ty, 434, 478 sample and expected updates, 121, 170–174 sample or simulation model, 115 sample-average method, 27 Samuel’s checkers player, 20, 241, 426–429 Sarsa, 130, 129–131, 244 vs Q-learning, 132 di↵erential, one-step, 251 Expected, 133–134, 140 n-step, 148 n-step o↵-policy, 150 double, 136 n-step, 147, 145–148, 247 di↵erential, 255 o↵-policy, 149 Sarsa(λ), 305, 303–307 true online, 307 Schultz, Wolfram, 387–395, 410 search control, 163 secondary reinforcement, 20, 346, 354, 369 selective bootstrap adaptation, 239 semi-gradient methods, 202, 258–259 SGD, see stochastic gradient descent Shannon, Claude, 16, 20, 71, 426 shaping, 360, 470 Skinner, B. F., 359–360, 375, 470, 479 soap bubble example, 95 soft and "-soft policies, 100–103, 110 soft-max, 322–323, 329, 336, 400, 445, 455 for ban


xample, 95 soft and "-soft policies, 100–103, 110 soft-max, 322–323, 329, 336, 400, 445, 455 for bandits, 37, 45 spike-timing-dependent plasticity (STDP), 401 state, 7, 48, 49 kth-order history approach, 468 and observations, 464–468 Markov property, 465–468 belief, 467 latent, 467 observable operator models (OOMs), 467 partially observable MDPs, 14, 467 predictive state representations, 467 state-update function, 465


524 Index state aggregation, 203–204 state-update function, 465 step-size parameter, 10, 31–33, 120, 125, 126 automatic adaptation, 238 in DQN, 439, 440 in psychological models, 347, 348 selecting manually, 222–223 with coarse coding, 216 with Fourier features, 213 with tile coding, 217, 223 stochastic approx. convergence conditions, 33 stochastic gradient descent (SGD), 200–204 in the Bellman error, 269–278 strong and weak methods, 4 supervised learning, xvii, 2, 17–19, 198 sweeps, 75, 160, see also prioritized sweeping synaptic plasticity, 379 Hebbian, 400 two-factor and three factor, 400 system identiﬁcation, 364 tabular solution methods, 23 target policy, 103, 110 of update, 31, 143, 198 TD, see temporal-di↵erence learning TD error, 121 n-step, 255 di↵erential, 250 with function approx


TD, see temporal-di↵erence learning TD error, 121 n-step, 255 di↵erential, 250 with function approximation, 270 TD(λ), 293, 292–295 truncated, 295–297 true online, 300, 299–301 TD-Gammon, 21, 421–426 temporal abstraction, 461–464 temporal-di↵erence learning, 10, 119–140 history of, 20–21 advantages of, 124–126 optimality of, 126–128 TD(0), 120, 203 TD(1), 294 TD(λ), 293, 292–295 true online, 300, 299–301 λ-return methods o↵-line, 290 online, 297–299 n-step, 144, 141–158, 209 termination function, 307, 459 Thompson sampling, 43, 45 Thorndike, Edward, see Law of E↵ect tic-tac-toe, 8–13, 17, 137 tile coding, 217–221, 223, 238, 246, 434, 435 Tolman, Edward, 364, 408 trace-decay parameter (λ), 287, 289, 290, 292 state dependent, 307 trajectory sampling, 174–177 transition probabilities, 49 Tre


87, 289, 290, 292 state dependent, 307 trajectory sampling, 174–177 transition probabilities, 49 Tree Backup n-step, 152–153, 154 Tree-Backup(λ), 312–314 trial-and-error, 1, 7, 15–21, 403, 404, see also instrumental conditioning true online TD(λ), 300, 299–301 Tsitsiklis and Van Roy’s Counterexample, 263 undiscounted continuing tasks, see average reward setting unsupervised learning, 2, 226 value, 6, 26, 47 value function, 6, 58–67 for a given policy: v⇡and q⇡, 58 for an optimal policy: v⇤and q⇤, 62 action, 58, 63, 65, 71, 129, 131 approximate action values: ˆq(s, a, w), 243 approximate state values: ˆv(s,w), 197 di↵erential, 243 vs evolutionary methods, 11 value iteration, 83, 82–84 value-function approximation, 198 Watkins, Chris, 15, 21, 89, 320 Watson (Jeopardy!


tion, 83, 82–84 value-function approximation, 198 Watkins, Chris, 15, 21, 89, 320 Watson (Jeopardy! player), 429–432 Werbos, Paul, 14, 21, 70, 89, 139, 239 Witten, Ian, 21, 70


525 Adaptive Computation and Machine Learning Francis Bach, Editor Bioinformatics: The Machine Learning Approach, Pierre Baldi and Søren Brunak Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto Graphical Models for Machine Learning and Digital Communication, Brendan J. Frey Learning in Graphical Models, Michael I. Jordan Causation, Prediction, and Search, second edition, Peter Spirtes, Clark Glymour, and Richard Scheines Principles of Data Mining, David Hand, Heikki Mannila, and Padhraic Smyth Bioinformatics: The Machine Learning Approach, second edition, Pierre Baldi and Søren Brunak Learning Kernel Classiﬁers: Theory and Algorithms, Ralf Herbrich Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, Bernhard Sch¨olkopf and


Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, Bernhard Sch¨olkopf and Alexander J. Smola Introduction to Machine Learning, Ethem Alpaydin Gaussian Processes for Machine Learning, Carl Edward Rasmussen and Christopher K.I. Williams Semi-Supervised Learning, Olivier Chapelle, Bernhard Sch¨olkopf, and Alexander Zien, Eds. The Minimum Description Length Principle, Peter D. Gr¨unwald Introduction to Statistical Relational Learning, Lise Getoor and Ben Taskar, Eds.


526 Probabilistic Graphical Models: Principles and Techniques, Daphne Koller and Nir Friedman Introduction to Machine Learning, second edition, Ethem Alpaydin Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation, Masashi Sugiyama and Motoaki Kawanabe Boosting: Foundations and Algorithms, Robert E. Schapire and Yoav Freund Machine Learning: A Probabilistic Perspective, Kevin P. Murphy Foundations of Machine Learning, Mehryar Mohri, Afshin Rostami, and Ameet Talwalker Introduction to Machine Learning, third edition, Ethem Alpaydin Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville Elements of Causal Inference, Jonas Peters, Dominik Janzing, and Bernhard Sch¨olkopf Machine Learning for Data Streams, with Practical Examples in MOA, Albert B


and Bernhard Sch¨olkopf Machine Learning for Data Streams, with Practical Examples in MOA, Albert Bifet, Ricard Gavald`a, Geo↵rey Holmes, Bernhard Pfahringer


Illustrating Reinforcement Learning from Human Feedback (RLHF)

+405

This article has been translated to Chinese 简体中文 and Vietnamese đọc tiếng việt .

Language models have shown impressive capabilities in the past few years by generating diverse and compelling text from human input prompts. However, what makes a "good" text is inherently hard to define as it is subjective and context dependent. There are many applications such as writing stories where you want creativity, pieces of informative text which should be truthful, or code snippets that we want to be executable.

Writing a loss function to capture these attributes seems intractable and most language models are still trained with a simple next token prediction loss (e.g. cross entropy).


st language models are still trained with a simple next token prediction loss (e.g. cross entropy). To compensate for the shortcomings of the loss itself people define metrics that are designed to better capture human preferences such as BLEU or ROUGE . While being better suited than the loss function itself at measuring performance these metrics simply compare generated text to references with simple rules and are thus also limited. Wouldn't it be great if we use human feedback for generated text as a measure of performance or go even one step further and use that feedback as a loss to optimize the model? That's the idea of Reinforcement Learning from Human Feedback (RLHF); use methods from reinforcement learning to directly optimize a language model with human feedback.


use methods from reinforcement learning to directly optimize a language model with human feedback. RLHF has enabled language models to begin to align a model trained on a general corpus of text data to that of complex human values.

RLHF's most recent success was its use in ChatGPT . Given ChatGPT's impressive abilities, we asked it to explain RLHF for us:

It does surprisingly well, but doesn't quite cover everything. We'll fill in those gaps!

RLHF: Let’s take it step by step

Reinforcement learning from Human Feedback (also referenced as RL from human preferences) is a challenging concept because it involves a multiple-model training process and different stages of deployment. In this blog post, we’ll break down the training process into three core steps:


ges of deployment. In this blog post, we’ll break down the training process into three core steps:

Pretraining a language model (LM),

gathering data and training a reward model, and

fine-tuning the LM with reinforcement learning.

To start, we'll look at how language models are pretrained.

Pretraining language models

As a starting point RLHF use a language model that has already been pretrained with the classical pretraining objectives (see this blog post for more details). OpenAI used a smaller version of GPT-3 for its first popular RLHF model, InstructGPT . In their shared papers, Anthropic used transformer models from 10 million to 52 billion parameters trained for this task. DeepMind has documented using up to their 280 billion parameter model Gopher .


ained for this task. DeepMind has documented using up to their 280 billion parameter model Gopher . It is likely that all these companies use much larger models in their RLHF-powered products.

This initial model can also be fine-tuned on additional text or conditions, but does not necessarily need to be. For example, OpenAI fine-tuned on human-generated text that was “preferable” and Anthropic generated their initial LM for RLHF by distilling an original LM on context clues for their “helpful, honest, and harmless” criteria. These are both sources of what we refer to as expensive, augmented data, but it is not a required technique to understand RLHF. Core to starting the RLHF process is having a model that responds well to diverse instructions .


. Core to starting the RLHF process is having a model that responds well to diverse instructions .

In general, there is not a clear answer on “which model” is the best for the starting point of RLHF. This will be a common theme in this blog – the design space of options in RLHF training are not thoroughly explored.

Next, with a language model, one needs to generate data to train a reward model , which is how human preferences are integrated into the system.

Reward model training

Generating a reward model (RM, also referred to as a preference model) calibrated with human preferences is where the relatively new research in RLHF begins. The underlying goal is to get a model or system that takes in a sequence of text, and returns a scalar reward which should numerically represent the human


akes in a sequence of text, and returns a scalar reward which should numerically represent the human preference. The system can be an end-to-end LM, or a modular system outputting a reward (e.g. a model ranks outputs, and the ranking is converted to reward). The output being a scalar reward is crucial for existing RL algorithms being integrated seamlessly later in the RLHF process.

These LMs for reward modeling can be both another fine-tuned LM or a LM trained from scratch on the preference data. For example, Anthropic has used a specialized method of fine-tuning to initialize these models after pretraining (preference model pretraining, PMP) because they found it to be more sample efficient than fine-tuning, but no one base model is considered the clear best choice for reward models.


ent than fine-tuning, but no one base model is considered the clear best choice for reward models.

The training dataset of prompt-generation pairs for the RM is generated by sampling a set of prompts from a predefined dataset (Anthropic’s data generated primarily with a chat tool on Amazon Mechanical Turk is available on the Hub, and OpenAI used prompts submitted by users to the GPT API). The prompts are passed through the initial language model to generate new text.

Human annotators are used to rank the generated text outputs from the LM. One may initially think that humans should apply a scalar score directly to each piece of text in order to generate a reward model, but this is difficult to do in practice.


to each piece of text in order to generate a reward model, but this is difficult to do in practice. The differing values of humans cause these scores to be uncalibrated and noisy. Instead, rankings are used to compare the outputs of multiple models and create a much better regularized dataset.

There are multiple methods for ranking the text. One method that has been successful is to have users compare generated text from two language models conditioned on the same prompt. By comparing model outputs in head-to-head matchups, an Elo system can be used to generate a ranking of the models and outputs relative to each-other. These different methods of ranking are normalized into a scalar reward signal for training.


other. These different methods of ranking are normalized into a scalar reward signal for training.

An interesting artifact of this process is that the successful RLHF systems to date have used reward language models with varying sizes relative to the text generation (e.g. OpenAI 175B LM, 6B reward model, Anthropic used LM and reward models from 10B to 52B, DeepMind uses 70B Chinchilla models for both LM and reward). An intuition would be that these preference models need to have similar capacity to understand the text given to them as a model would need in order to generate said text.

At this point in the RLHF system, we have an initial language model that can be used to generate text and a preference model that takes in any text and assigns it a score of how well humans perceive it.


nd a preference model that takes in any text and assigns it a score of how well humans perceive it. Next, we use reinforcement learning (RL) to optimize the original language model with respect to the reward model.

Fine-tuning with RL

Training a language model with reinforcement learning was, for a long time, something that people would have thought as impossible both for engineering and algorithmic reasons.
What multiple organizations seem to have gotten to work is fine-tuning some or all of the parameters of a copy of the initial LM with a policy-gradient RL algorithm, Proximal Policy Optimization (PPO).
Some parameters of the LM are frozen because fine-tuning an entire 10B or 100B+ parameter model is prohibitively expensive (for more, see Low-Rank Adaptation ( LoRA ) for LMs or the


rameter model is prohibitively expensive (for more, see Low-Rank Adaptation ( LoRA ) for LMs or the Sparrow LM from DeepMind) -- depending on the scale of the model and infrastructure being used. The exact dynamics of how many parameters to freeze, or not, is considered an open research problem.
PPO has been around for a relatively long time – there are tons of guides on how it works. The relative maturity of this method made it a favorable choice for scaling up to the new application of distributed training for RLHF. It turns out that many of the core RL advancements to do RLHF have been figuring out how to update such a large model with a familiar algorithm (more on that later).

Let's first formulate this fine-tuning task as a RL problem.


iliar algorithm (more on that later).

Let's first formulate this fine-tuning task as a RL problem. First, the policy is a language model that takes in a prompt and returns a sequence of text (or just probability distributions over text). The action space of this policy is all the tokens corresponding to the vocabulary of the language model (often on the order of 50k tokens) and the observation space is the distribution of possible input token sequences, which is also quite large given previous uses of RL (the dimension is approximately the size of vocabulary ^ length of the input token sequence). The reward function is a combination of the preference model and a constraint on policy shift.


e). The reward function is a combination of the preference model and a constraint on policy shift.

The reward function is where the system combines all of the models we have discussed into one RLHF process. Given a prompt, x ,  from the dataset, the text y is generated by the current iteration of the fine-tuned policy. Concatenated with the original prompt, that text is passed to the preference model, which returns a scalar notion of “preferability”, r θ r_\theta r θ  . In addition, per-token probability distributions from the RL policy are compared to the ones from the initial model to compute a penalty on the difference between them. In multiple papers from OpenAI, Anthropic, and DeepMind, this penalty has been designed as a scaled version of the Kullback–Leibler (KL) divergence betwee


d, this penalty has been designed as a scaled version of the Kullback–Leibler (KL) divergence between these sequences of distributions over tokens, r KL r_\text{KL} r KL  . The KL divergence term penalizes the RL policy from moving substantially away from the initial pretrained model with each training batch, which can be useful to make sure the model outputs reasonably coherent text snippets. Without this penalty the optimization can start to generate text that is gibberish but fools the reward model to give a high reward. In practice, the KL divergence is approximated via sampling from both distributions (explained by John Schulman here ). The final reward sent to the RL update rule is r = r θ − λ r KL r = r_\theta - \lambda r_\text{KL} r = r θ  − λ r KL  .


o the RL update rule is r = r θ − λ r KL r = r_\theta - \lambda r_\text{KL} r = r θ  − λ r KL  .

Some RLHF systems have added additional terms to the reward function. For example, OpenAI experimented successfully on InstructGPT by mixing in additional pre-training gradients (from the human annotation set) into the update rule for PPO. It is likely as RLHF is further investigated, the formulation of this reward function will continue to evolve.

Finally, the update rule is the parameter update from PPO that maximizes the reward metrics in the current batch of data (PPO is on-policy, which means the parameters are only updated with the current batch of prompt-generation pairs). PPO is a trust region optimization algorithm that uses constraints on the gradient to ensure the update step doe


st region optimization algorithm that uses constraints on the gradient to ensure the update step does not destabilize the learning process. DeepMind used a similar reward setup for Gopher but used synchronous advantage actor-critic (A2C) to optimize the gradients, which is notably different but has not been reproduced externally.

Technical detail note: The above diagram makes it look like both models generate different responses for the same prompt, but what really happens is that the RL policy generates text, and that text is fed into the initial model to produce its relative probabilities for the KL penalty. This initial model is untouched by gradient updates during training .

Optionally, RLHF can continue from this point by iteratively updating the reward model and the policy together


, RLHF can continue from this point by iteratively updating the reward model and the policy together. As the RL policy updates, users can continue ranking these outputs versus the model's earlier versions. Most papers have yet to discuss implementing this operation, as the deployment mode needed to collect this type of data only works for dialogue agents with access to an engaged user base. Anthropic discusses this option as Iterated Online RLHF (see the original paper ), where iterations of the policy are included in the ELO ranking system across models. This introduces complex dynamics of the policy and reward model evolving, which represents a complex and open research question.

Open-source tools for RLHF


model evolving, which represents a complex and open research question.

Open-source tools for RLHF

The first code released to perform RLHF on LMs was from OpenAI in TensorFlow in 2019.

Today, there are already a few active repositories for RLHF in PyTorch that grew out of this. The primary repositories are Transformers Reinforcement Learning ( TRL ), TRLX which originated as a fork of TRL, and Reinforcement Learning for Language models ( RL4LMs ).

TRL is designed to fine-tune pretrained LMs in the Hugging Face ecosystem with PPO. TRLX is an expanded fork of TRL built by CarperAI to handle larger models for online and offline training. At the moment, TRLX has an API capable of production-ready RLHF with PPO and Implicit Language Q-Learning ILQL at the scales required for LLM deployment (


eady RLHF with PPO and Implicit Language Q-Learning ILQL at the scales required for LLM deployment (e.g. 33 billion parameters). Future versions of TRLX will allow for language models up to 200B parameters. As such, interfacing with TRLX is optimized for machine learning engineers with experience at this scale.

RL4LMs offers building blocks for fine-tuning and evaluating LLMs with a wide variety of RL algorithms (PPO, NLPO, A2C and TRPO), reward functions and metrics. Moreover, the library is easily customizable, which allows training of any encoder-decoder or encoder transformer-based LM on any arbitrary user-specified reward function. Notably, it is well-tested and benchmarked on a broad range of tasks in recent work amounting up to 2000 experiments highlighting several practical insigh


range of tasks in recent work amounting up to 2000 experiments highlighting several practical insights on data budget comparison (expert demonstrations vs. reward modeling), handling reward hacking and training instabilities, etc.
RL4LMs current plans include distributed training of larger models and new RL algorithms.

Both TRLX and RL4LMs are under heavy further development, so expect more features beyond these soon.

There is a large dataset created by Anthropic available on the Hub.

What’s next for RLHF?

While these techniques are extremely promising and impactful and have caught the attention of the biggest research labs in AI, there are still clear limitations. The models, while better, can still output harmful or factually inaccurate text without any uncertainty.


odels, while better, can still output harmful or factually inaccurate text without any uncertainty. This imperfection represents a long-term challenge and motivation for RLHF – operating in an inherently human problem domain means there will never be a clear final line to cross for the model to be labeled as complete .

When deploying a system using RLHF, gathering the human preference data is quite expensive due to the direct integration of other human workers outside the training loop. RLHF performance is only as good as the quality of its human annotations, which takes on two varieties: human-generated text, such as fine-tuning the initial LM in InstructGPT, and labels of human preferences between model outputs.


fine-tuning the initial LM in InstructGPT, and labels of human preferences between model outputs.

Generating well-written human text answering specific prompts is very costly, as it often requires hiring part-time staff (rather than being able to rely on product users or crowdsourcing). Thankfully, the scale of data used in training the reward model for most applications of RLHF (~50k labeled preference samples) is not as expensive. However, it is still a higher cost than academic labs would likely be able to afford. Currently, there only exists one large-scale dataset for RLHF on a general language model (from Anthropic ) and a couple of smaller-scale task-specific datasets (such as summarization data from OpenAI ).


c ) and a couple of smaller-scale task-specific datasets (such as summarization data from OpenAI ). The second challenge of data for RLHF is that human annotators can often disagree, adding a substantial potential variance to the training data without ground truth.

With these limitations, huge swaths of unexplored design options could still enable RLHF to take substantial strides. Many of these fall within the domain of improving the RL optimizer. PPO is a relatively old algorithm, but there are no structural reasons that other algorithms could not offer benefits and permutations on the existing RLHF workflow. One large cost of the feedback portion of fine-tuning the LM policy is that every generated piece of text from the policy needs to be evaluated on the reward model (as it acts like


generated piece of text from the policy needs to be evaluated on the reward model (as it acts like part of the environment in the standard RL framework). To avoid these costly forward passes of a large model, offline RL could be used as a policy optimizer. Recently, new algorithms have emerged, such as implicit language Q-learning (ILQL) [ Talk on ILQL at CarperAI], that fit particularly well with this type of optimization. Other core trade-offs in the RL process, like exploration-exploitation balance, have also not been documented. Exploring these directions would at least develop a substantial understanding of how RLHF functions and, if not, provide improved performance.

We hosted a lecture on Tuesday 13 December 2022 that expanded on this post; you can watch it here !


e hosted a lecture on Tuesday 13 December 2022 that expanded on this post; you can watch it here !

Further reading

Here is a list of the most prevalent papers on RLHF to date. The field was recently popularized with the emergence of DeepRL (around 2017) and has grown into a broader study of the applications of LLMs from many large technology companies.
Here are some papers on RLHF that pre-date the LM focus:

TAMER: Training an Agent Manually via Evaluative Reinforcement (Knox and Stone 2008): Proposed a learned agent where humans provided scores on the actions taken iteratively to learn a reward model.

Interactive Learning from Policy-Dependent Human Feedback (MacGlashan et al. 2017): Proposed an actor-critic algorithm, COACH, where human feedback (both positive and negative) is used t


oposed an actor-critic algorithm, COACH, where human feedback (both positive and negative) is used to tune the advantage function.

Deep Reinforcement Learning from Human Preferences (Christiano et al. 2017): RLHF applied on preferences between Atari trajectories.

Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces (Warnell et al. 2018): Extends the TAMER framework where a deep neural network is used to model the reward prediction.

A Survey of Preference-based Reinforcement Learning Methods (Wirth et al. 2017): Summarizes efforts above with many, many more references.

And here is a snapshot of the growing set of "key" papers that show RLHF's performance for LMs:

Fine-Tuning Language Models from Human Preferences (Zieglar et al.


how RLHF's performance for LMs:

Fine-Tuning Language Models from Human Preferences (Zieglar et al. 2019): An early paper that studies the impact of reward learning on four specific tasks.

Learning to summarize with human feedback (Stiennon et al., 2020): RLHF applied to the task of summarizing text. Also, Recursively Summarizing Books with Human Feedback (OpenAI Alignment Team 2021), follow on work summarizing books.

WebGPT: Browser-assisted question-answering with human feedback (OpenAI, 2021): Using RLHF to train an agent to navigate the web.

InstructGPT: Training language models to follow instructions with human feedback (OpenAI Alignment Team 2022): RLHF applied to a general language model [ Blog post on InstructGPT].


OpenAI Alignment Team 2022): RLHF applied to a general language model [ Blog post on InstructGPT].

GopherCite: Teaching language models to support answers with verified quotes (Menick et al. 2022): Train a LM with RLHF to return answers with specific citations.

Sparrow: Improving alignment of dialogue agents via targeted human judgements (Glaese et al. 2022): Fine-tuning a dialogue agent with RLHF

ChatGPT: Optimizing Language Models for Dialogue (OpenAI 2022): Training a LM with RLHF for suitable use as an all-purpose chat bot.

Scaling Laws for Reward Model Overoptimization (Gao et al. 2022): studies the scaling properties of the learned preference model in RLHF.

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (Anthropic, 2022): A detailed doc


Harmless Assistant with Reinforcement Learning from Human Feedback (Anthropic, 2022): A detailed documentation of training a LM assistant with RLHF.

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (Ganguli et al. 2022): A detailed documentation of efforts to “discover, measure, and attempt to reduce [language models] potentially harmful outputs.”

Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning (Cohen at al. 2022): Using RL to enhance the conversational skill of an open-ended dialogue agent.

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (Ramamurthy and Ammanabrolu et al.


es, and Building Blocks for Natural Language Policy Optimization (Ramamurthy and Ammanabrolu et al. 2022): Discusses the design space of open-source tools in RLHF and proposes a new algorithm NLPO (Natural Language Policy Optimization) as an alternative to PPO.

Llama 2 (Touvron et al. 2023): Impactful open-access model with substantial RLHF details.

The field is the convergence of multiple fields, so you can also find resources in other areas:

Continual learning of instructions ( Kojima et al. 2021 , Suhr and Artzi 2022 ) or bandit learning from user feedback ( Sokolov et al. 2016 , Gao et al. 2022 )

Earlier history on using other RL algorithms for text generation (not all with human preferences), such as with recurrent neural networks ( Ranzato et al.


eneration (not all with human preferences), such as with recurrent neural networks ( Ranzato et al. 2015 ), an actor-critic algorithm for text prediction ( Bahdanau et al. 2016 ), or an early work adding human preferences to this framework ( Nguyen et al. 2017 ).

Citation: If you found this useful for your academic work, please consider citing our work, in text:

Lambert, et al., "Illustrating Reinforcement Learning from Human Feedback (RLHF)", Hugging Face Blog, 2022.

Lambert, et al., "Illustrating Reinforcement Learning from Human Feedback (RLHF)", Hugging Face Blog, 2022.

BibTeX citation:

@article{lambert2022illustrating,
  author = {Lambert, Nathan and Castricato, Louis and von Werra, Leandro and Havrilla, Alex},


ting,
  author = {Lambert, Nathan and Castricato, Louis and von Werra, Leandro and Havrilla, Alex},
  title = {Illustrating Reinforcement Learning from Human Feedback (RLHF)},
  journal = {Hugging Face Blog},
  year = {2022},
  note = {https://huggingface.co/blog/rlhf},
}

@article{lambert2022illustrating,
  author = {Lambert, Nathan and Castricato, Louis and von Werra, Leandro and Havrilla, Alex},
  title = {Illustrating Reinforcement Learning from Human Feedback (RLHF)},
  journal = {Hugging Face Blog},
  year = {2022},
  note = {https://huggingface.co/blog/rlhf},
}

Thanks to Robert Kirk for fixing some factual errors regarding specific implementations of RLHF. Thanks to Stas Bekman for fixing some typos or confusing phrases Thanks to Peter Stone , Khanh X.


. Thanks to Stas Bekman for fixing some typos or confusing phrases Thanks to Peter Stone , Khanh X. Nguyen and Yoav Artzi for helping expand the related works further into history. Thanks to Igor Kotenkov for pointing out a technical error in the KL-penalty term of the RLHF procedure, its diagram, and textual description.

Datasets mentioned in this article 1

More Articles from our Blog

Finetune Stable Diffusion Models with DDPO via TRL

Putting RL back in RLHF

Community

Is the last figure caption correct? "The above diagram makes it look like both models generate different responses for the same prompt, but what really happens is that the RL policy generates text, and that text is fed into the initial model to produce its relative probabilities for the KL penalty."


that text is fed into the initial model to produce its relative probabilities for the KL penalty."

To me it makes more sense to feed the prompt to the reference and the policy models then to compare the reference model probability distribution with the policy's. Any insight would be appreciated.

Best, Chiron

1 reply

Yeah, I was thinking the same.

Maybe Parameters frozen should be on Initial model? (last figure caption)

1 reply

Open-source or budget RLHF often uses PEFT (LoRA/adapters) freezing most parameters, while the biggest labs don't freeze any parameters during base model finetuning or RL phases.

· Sign up or log in to comment

+399

Datasets mentioned in this article 1


Part 1: Key Concepts in RL ¶

Table of Contents

Part 1: Key Concepts in RL What Can RL Do? Key Concepts and Terminology (Optional) Formalism

What Can RL Do?

Key Concepts and Terminology

(Optional) Formalism

Welcome to our introduction to reinforcement learning! Here, we aim to acquaint you with

the language and notation used to discuss the subject,

a high-level explanation of what RL algorithms do (although we mostly avoid the question of how they do it),

and a little bit of the core math that underlies the algorithms.

In a nutshell, RL is the study of agents and how they learn by trial and error. It formalizes the idea that rewarding or punishing an agent for its behavior makes it more likely to repeat or forego that behavior in the future.

What Can RL Do?


its behavior makes it more likely to repeat or forego that behavior in the future.

What Can RL Do? ¶

RL methods have recently enjoyed a wide variety of successes. For example, it’s been used to teach computers to control robots in simulation...

...and in the real world...

It’s also famously been used to create breakthrough AIs for sophisticated strategy games, most notably Go and Dota , taught computers to play Atari games from raw pixels, and trained simulated robots to follow human instructions .

Key Concepts and Terminology ¶

Agent-environment interaction loop.

The main characters of RL are the agent and the environment . The environment is the world that the agent lives in and interacts with.


gent and the environment . The environment is the world that the agent lives in and interacts with. At every step of interaction, the agent sees a (possibly partial) observation of the state of the world, and then decides on an action to take. The environment changes when the agent acts on it, but may also change on its own.

The agent also perceives a reward signal from the environment, a number that tells it how good or bad the current world state is. The goal of the agent is to maximize its cumulative reward, called return . Reinforcement learning methods are ways that the agent can learn behaviors to achieve its goal.

To talk more specifically what RL does, we need to introduce additional terminology.


eve its goal.

To talk more specifically what RL does, we need to introduce additional terminology. We need to talk about

states and observations,

action spaces,

policies,

trajectories,

different formulations of return,

the RL optimization problem,

and value functions.

States and Observations ¶

A state is a complete description of the state of the world. There is no information about the world which is hidden from the state. An observation is a partial description of a state, which may omit information.

In deep RL, we almost always represent states and observations by a real-valued vector, matrix, or higher-order tensor . For instance, a visual observation could be represented by the RGB matrix of its pixel values; the state of a robot might be represented by its joint angles and


he RGB matrix of its pixel values; the state of a robot might be represented by its joint angles and velocities.

When the agent is able to observe the complete state of the environment, we say that the environment is fully observed . When the agent can only see a partial observation, we say that the environment is partially observed .

You Should Know

Reinforcement learning notation sometimes puts the symbol for state, , in places where it would be technically more appropriate to write the symbol for observation, . Specifically, this happens when talking about how the agent decides an action: we often signal in notation that the action is conditioned on the state, when in practice, the action is conditioned on the observation because the agent does not have access to the state.


the action is conditioned on the observation because the agent does not have access to the state.

In our guide, we’ll follow standard conventions for notation, but it should be clear from context which is meant. If something is unclear, though, please raise an issue! Our goal is to teach, not to confuse.

Action Spaces ¶

Different environments allow different kinds of actions. The set of all valid actions in a given environment is often called the action space . Some environments, like Atari and Go, have discrete action spaces , where only a finite number of moves are available to the agent. Other environments, like where the agent controls a robot in a physical world, have continuous action spaces .


onments, like where the agent controls a robot in a physical world, have continuous action spaces . In continuous spaces, actions are real-valued vectors.

This distinction has some quite-profound consequences for methods in deep RL. Some families of algorithms can only be directly applied in one case, and would have to be substantially reworked for the other.

Policies ¶

A policy is a rule used by an agent to decide what actions to take. It can be deterministic, in which case it is usually denoted by :

or it may be stochastic, in which case it is usually denoted by :

Because the policy is essentially the agent’s brain, it’s not uncommon to substitute the word “policy” for “agent”, eg saying “The policy is trying to maximize reward.”


to substitute the word “policy” for “agent”, eg saying “The policy is trying to maximize reward.”

In deep RL, we deal with parameterized policies : policies whose outputs are computable functions that depend on a set of parameters (eg the weights and biases of a neural network) which we can adjust to change the behavior via some optimization algorithm.

We often denote the parameters of such a policy by or , and then write this as a subscript on the policy symbol to highlight the connection:

Deterministic Policies ¶

Example: Deterministic Policies. Here is a code snippet for building a simple deterministic policy for a continuous action space in PyTorch, using the torch.nn package:

torch.nn

pi_net = nn . Sequential ( nn . Linear ( obs_dim , 64 ), nn . Tanh (), nn .


.nn package:

torch.nn

pi_net = nn . Sequential ( nn . Linear ( obs_dim , 64 ), nn . Tanh (), nn . Linear ( 64 , 64 ), nn . Tanh (), nn . Linear ( 64 , act_dim ) )

This builds a multi-layer perceptron (MLP) network with two hidden layers of size 64 and activation functions. If obs is a Numpy array containing a batch of observations, pi_net can be used to obtain a batch of actions as follows:

obs

pi_net

obs_tensor = torch . as_tensor ( obs , dtype = torch . float32 ) actions = pi_net ( obs_tensor )

You Should Know

Don’t worry about it if this neural network stuff is unfamiliar to you—this tutorial will focus on RL, and not on the neural network side of things. So you can skip this example and come back to it later.


d not on the neural network side of things. So you can skip this example and come back to it later. But we figured that if you already knew, it could be helpful.

Stochastic Policies ¶

The two most common kinds of stochastic policies in deep RL are categorical policies and diagonal Gaussian policies .

Categorical policies can be used in discrete action spaces, while diagonal Gaussian policies are used in continuous action spaces.

Two key computations are centrally important for using and training stochastic policies:

sampling actions from the policy,

and computing log likelihoods of particular actions, .

In what follows, we’ll describe how to do these for both categorical and diagonal Gaussian policies.

Categorical Policies


escribe how to do these for both categorical and diagonal Gaussian policies.

Categorical Policies

A categorical policy is like a classifier over discrete actions. You build the neural network for a categorical policy the same way you would for a classifier: the input is the observation, followed by some number of layers (possibly convolutional or densely-connected, depending on the kind of input), and then you have one final linear layer that gives you logits for each action, followed by a softmax to convert the logits into probabilities.

Sampling. Given the probabilities for each action, frameworks like PyTorch and Tensorflow have built-in tools for sampling. For example, see the documentation for Categorical distributions in PyTorch , torch.multinomial , tf.distributions.Categorical ,


tation for Categorical distributions in PyTorch , torch.multinomial , tf.distributions.Categorical , or tf.multinomial .

Log-Likelihood. Denote the last layer of probabilities as . It is a vector with however many entries as there are actions, so we can treat the actions as indices for the vector. The log likelihood for an action can then be obtained by indexing into the vector:

Diagonal Gaussian Policies

A multivariate Gaussian distribution (or multivariate normal distribution, if you prefer) is described by a mean vector, , and a covariance matrix, . A diagonal Gaussian distribution is a special case where the covariance matrix only has entries on the diagonal. As a result, we can represent it by a vector.


covariance matrix only has entries on the diagonal. As a result, we can represent it by a vector.

A diagonal Gaussian policy always has a neural network that maps from observations to mean actions, . There are two different ways that the covariance matrix is typically represented.

The first way: There is a single vector of log standard deviations, , which is not a function of state: the are standalone parameters. (You Should Know: our implementations of VPG, TRPO, and PPO do it this way.)

The second way: There is a neural network that maps from states to log standard deviations, . It may optionally share some layers with the mean network.

Note that in both cases we output log standard deviations instead of standard deviations directly.


Note that in both cases we output log standard deviations instead of standard deviations directly. This is because log stds are free to take on any values in , while stds must be nonnegative. It’s easier to train parameters if you don’t have to enforce those kinds of constraints. The standard deviations can be obtained immediately from the log standard deviations by exponentiating them, so we do not lose anything by representing them this way.

Sampling. Given the mean action and standard deviation , and a vector of noise from a spherical Gaussian ( ), an action sample can be computed with

where denotes the elementwise product of two vectors. Standard frameworks have built-in ways to generate the noise vectors, such as torch.normal or tf.random_normal .


eworks have built-in ways to generate the noise vectors, such as torch.normal or tf.random_normal . Alternatively, you can build distribution objects, eg through torch.distributions.Normal or tf.distributions.Normal , and use them to generate samples. (The advantage of the latter approach is that those objects can also calculate log-likelihoods for you.)

Log-Likelihood. The log-likelihood of a -dimensional action , for a diagonal Gaussian with mean and standard deviation , is given by

Trajectories ¶

A trajectory is a sequence of states and actions in the world,

The very first state of the world, , is randomly sampled from the start-state distribution , sometimes denoted by :

State transitions (what happens to the world between the state at time , , and the state at , ), are governed b


itions (what happens to the world between the state at time , , and the state at , ), are governed by the natural laws of the environment, and depend on only the most recent action, . They can be either deterministic,

or stochastic,

Actions come from an agent according to its policy.

You Should Know

Trajectories are also frequently called episodes or rollouts .

Reward and Return ¶

The reward function is critically important in reinforcement learning. It depends on the current state of the world, the action just taken, and the next state of the world:

although frequently this is simplified to just a dependence on the current state, , or state-action pair .

The goal of the agent is to maximize some notion of cumulative reward over a trajectory, but this actually can mean a few things


maximize some notion of cumulative reward over a trajectory, but this actually can mean a few things. We’ll notate all of these cases with , and it will either be clear from context which case we mean, or it won’t matter (because the same equations will apply to all cases).

One kind of return is the finite-horizon undiscounted return , which is just the sum of rewards obtained in a fixed window of steps:

Another kind of return is the infinite-horizon discounted return , which is the sum of all rewards ever obtained by the agent, but discounted by how far off in the future they’re obtained. This formulation of reward includes a discount factor :

Why would we ever want a discount factor, though? Don’t we just want to get all rewards?


factor :

Why would we ever want a discount factor, though? Don’t we just want to get all rewards? We do, but the discount factor is both intuitively appealing and mathematically convenient. On an intuitive level: cash now is better than cash later. Mathematically: an infinite-horizon sum of rewards may not converge to a finite value, and is hard to deal with in equations. But with a discount factor and under reasonable conditions, the infinite sum converges.

You Should Know

While the line between these two formulations of return are quite stark in RL formalism, deep RL practice tends to blur the line a fair bit—for instance, we frequently set up algorithms to optimize the undiscounted return, but use discount factors in estimating value functions .

The RL Problem ¶


he undiscounted return, but use discount factors in estimating value functions .

The RL Problem ¶

Whatever the choice of return measure (whether infinite-horizon discounted, or finite-horizon undiscounted), and whatever the choice of policy, the goal in RL is to select a policy which maximizes expected return when the agent acts according to it.

To talk about expected return, we first have to talk about probability distributions over trajectories.

Let’s suppose that both the environment transitions and the policy are stochastic. In this case, the probability of a -step trajectory is:

The expected return (for whichever measure), denoted by , is then:

The central optimization problem in RL can then be expressed by

with being the optimal policy .

Value Functions ¶


ization problem in RL can then be expressed by

with being the optimal policy .

Value Functions ¶

It’s often useful to know the value of a state, or state-action pair. By value, we mean the expected return if you start in that state or state-action pair, and then act according to a particular policy forever after. Value functions are used, one way or another, in almost every RL algorithm.

There are four main functions of note here.

The On-Policy Value Function , , which gives the expected return if you start in state and always act according to policy :

The On-Policy Value Function , , which gives the expected return if you start in state and always act according to policy :

The On-Policy Action-Value Function , , which gives the expected return if you start in state , take an arbitr


icy Action-Value Function , , which gives the expected return if you start in state , take an arbitrary action (which may not have come from the policy), and then forever after act according to policy :

The On-Policy Action-Value Function , , which gives the expected return if you start in state , take an arbitrary action (which may not have come from the policy), and then forever after act according to policy :

The Optimal Value Function , , which gives the expected return if you start in state and always act according to the optimal policy in the environment:

The Optimal Value Function , , which gives the expected return if you start in state and always act according to the optimal policy in the environment:


ed return if you start in state and always act according to the optimal policy in the environment:

The Optimal Action-Value Function , , which gives the expected return if you start in state , take an arbitrary action , and then forever after act according to the optimal policy in the environment:

The Optimal Action-Value Function , , which gives the expected return if you start in state , take an arbitrary action , and then forever after act according to the optimal policy in the environment:

You Should Know

When we talk about value functions, if we do not make reference to time-dependence, we only mean expected infinite-horizon discounted return . Value functions for finite-horizon undiscounted return would need to accept time as an argument.


. Value functions for finite-horizon undiscounted return would need to accept time as an argument. Can you think about why? Hint: what happens when time’s up?

You Should Know

There are two key connections between the value function and the action-value function that come up pretty often:

and

These relations follow pretty directly from the definitions just given: can you prove them?

The Optimal Q-Function and the Optimal Action ¶

There is an important connection between the optimal action-value function and the action selected by the optimal policy. By definition, gives the expected return for starting in state , taking (arbitrary) action , and then acting according to the optimal policy forever after.


state , taking (arbitrary) action , and then acting according to the optimal policy forever after.

The optimal policy in will select whichever action maximizes the expected return from starting in . As a result, if we have , we can directly obtain the optimal action, , via

Note: there may be multiple actions which maximize , in which case, all of them are optimal, and the optimal policy may randomly select any of them. But there is always an optimal policy which deterministically selects an action.

Bellman Equations ¶

All four of the value functions obey special self-consistency equations called Bellman equations . The basic idea behind the Bellman equations is this:

The Bellman equations for the on-policy value functions are


behind the Bellman equations is this:

The Bellman equations for the on-policy value functions are

where is shorthand for , indicating that the next state is sampled from the environment’s transition rules; is shorthand for ; and is shorthand for .

The Bellman equations for the optimal value functions are

The crucial difference between the Bellman equations for the on-policy value functions and the optimal value functions, is the absence or presence of the over actions. Its inclusion reflects the fact that whenever the agent gets to choose its action, in order to act optimally, it has to pick whichever action leads to the highest value.

You Should Know

The term “Bellman backup” comes up quite frequently in the RL literature.


value.

You Should Know

The term “Bellman backup” comes up quite frequently in the RL literature. The Bellman backup for a state, or state-action pair, is the right-hand side of the Bellman equation: the reward-plus-next-value.

Advantage Functions ¶

Sometimes in RL, we don’t need to describe how good an action is in an absolute sense, but only how much better it is than others on average. That is to say, we want to know the relative advantage of that action. We make this concept precise with the advantage function.

The advantage function corresponding to a policy describes how much better it is to take a specific action in state , over randomly selecting an action according to , assuming you act according to forever after.


ate , over randomly selecting an action according to , assuming you act according to forever after. Mathematically, the advantage function is defined by

You Should Know

We’ll discuss this more later, but the advantage function is crucially important to policy gradient methods.

(Optional) Formalism ¶

So far, we’ve discussed the agent’s environment in an informal way, but if you try to go digging through the literature, you’re likely to run into the standard mathematical formalism for this setting: Markov Decision Processes (MDPs). An MDP is a 5-tuple, , where

is the set of all valid states,

is the set of all valid actions,

is the reward function, with ,

is the transition probability function, with being the probability of transitioning into state if you start in state and take actio


unction, with being the probability of transitioning into state if you start in state and take action ,

and is the starting state distribution.

The name Markov Decision Process refers to the fact that the system obeys the Markov property : transitions only depend on the most recent state and action, and no prior history.