3 MARKOV DECISION PROCESSES AND POLICY GRADIENTS τ πθ(τ) r(τ) Bias Figure 3.5: Biasing every reward r(s, a) by a constant c does not bias the policy gradient estimator, but it results in a more flat distribution which needs more samples to approximate. 3.3 Variance Reduction We have the policy gradient estimator ˆ∇θJ(θ). We want a gradient that is as efficient as possible, i.e. has as small of a variance of a possible, since it reduces the number of samples needed to accurately estimate the gradient. Consider if the rewards are inflated by a constant c, i.e. r(τ) ←r(τ) + c. Then we notice that the estimator for ∇θJ(θ) is unbiased: Eτ∼πθ(τ)[∇θ log πθ(τ)(r(τ) + c)] = ∇θJ(θ) + c Eτ∼πθ(τ)[ ∇θ log πθ(τ) ] = ∇θJ(θ) Score function Since the expectation of the score function is 0.


Eτ∼πθ(τ)[ ∇θ log πθ(τ) ] = ∇θJ(θ) Score function Since the expectation of the score function is 0. In practice, we set c := −1 N Pn i=1 r(τi) to minimise the variance. The intuitive explanation is that, if the critic always gives a high rating to the actions taken by the agent, we would need more samples to distinguish what the critic is expecting. Question: Why would you add constants to the reward function? We do not want to inflate the reward since it would increase the variance, but if we can evaluate the mean of rewards, we could subtract it from the rewards to reduce variance. Another tool to reduce variance is the causality trick. We can slightly change the objective so the reward is only summed from the current step to the end to reduce variance: ˆ∇θJ(θ) := 1 N N X i=1 T X t=1 log


only summed from the current step to the end to reduce variance: ˆ∇θJ(θ) := 1 N N X i=1 T X t=1 log πθ(ai,t|si,t) T X t′=t r(si,t′, ai,t′) ! Rewards “to go” Question: By doing causality trick, is the estimator still unbiased? Yes. 14


4 VALUE-BASED RL AND ACTOR-CRITIC METHODS 4 Value-based RL and Actor-Critic Methods • Trust Region Policy Optimization. Schulman et al. (2015) • Proximal Policy Optimization Algorithms. Schulman et al. (2017) • Training language models to follow instructions with human feedback. Ouyang et al. (2022) “Instruct GPT” • Solving Rubik’s Cube with a Robot Hand. Akkaya et al. (2019) Announcements • 04-17: Project Survey due • 04-19: Homework 1 due • 04-27: Project Proposal due 4.1 Value-based RL In the causality trick, the reward to-go is the reward from a single trajectory of actions. We could also design the reward based on the set of possible future actions. We could substitute in the “true” rewards to go to get a better estimate: ˆ∇θJ(θ) := 1 N N X i=1 T X t=1 log πθ(ai,t|si,t) ˆQ(st, at) Qπ(


ards to go to get a better estimate: ˆ∇θJ(θ) := 1 N N X i=1 T X t=1 log πθ(ai,t|si,t) ˆQ(st, at) Qπ(st, at) := T X t′=t Eπθ[r(st′, at′)|st, at] Qπ is the Q-function, representing the total reward from taking at in st. The value function is the total reward from st: V π(st) := Eat∼πθ(at|st)[Qπ(st, at)] The advantage function measures the amount by which action at is better than the average reward from state st: Aπ(st, at) := Qπ(st, at) −V π(st) We fit a model to estimate Qπ, V π, Aπ, and use the estimate ˆ∇θJ(θ) := 1 N N X i=1 T X t=1 log πθ(ai,t|si,t) ˆAπ(si,t, ai,t) Only V π needs to be fitted, since both Qπ and Aπ can be expressed in terms of V π: 15


4 VALUE-BASED RL AND ACTOR-CRITIC METHODS • Q: Qπ(st, at) = r(st, at) + T X t′=t+1 Eπθ[r(st′, at′)|st, at] = r(st, at) + T X t′=t+1 Eπθ[r(st′, at′)|st+1, at+1] ≃r(st, at) + V π(st+1) The last step is an approximation, since the expectation is evaluated on the stochastic distribution of p(st+1|st, at), but we are replacing it by the observed next state st+1 • A: Aπ(st, at) ≃r(st, at) + V π(st+1) −V π(st) The crux of this method is: If the value function changes at the next action, we can already determine the reward instead of having to roll-out the policy to the end. We want to fit ˆV π(si) (with a L2 loss, for example) to the ideal target yi,t := T X t′=t Eπθ[r(st′, at′)|si,t] We could use the Monte Carlo target: ˆyMC i,t := T X t′=t r(si,t′, ai,t′) Or we can use the bootstrap estimate, w


the Monte Carlo target: ˆyMC i,t := T X t′=t r(si,t′, ai,t′) Or we can use the bootstrap estimate, which is a type of autoregressive model using the previous step’s estimate: ˆyi,t := r(si,t, ai,t) + V π(si,t+1) ≃r(si,t, ai,t) + ˆV π(si,t+1) Previous fitted value function Question: If we don’t roll-out to the very end, do we get a greedy policy that only considers near future rewards? No since the value function considers all future rewards. Question: Does the number of trajectories with positive rewards matter? If only few trajectories lead to positive rewards, would it skew the estimate? If the trajectories with rewards are few, it becomes an exploration problem. Question: Is the previous step policy π the same as the current step policy?


n exploration problem. Question: Is the previous step policy π the same as the current step policy? We use a slightly outdated version of the network for the previous step policy. 16


4 VALUE-BASED RL AND ACTOR-CRITIC METHODS s1 s2 ⊥ p(s′|s, a) p(⊥|s, a) = 1 −γ Figure 4.1: Discount factor adds a terminal state to the Markov chain with probability of 1 −γ being reached in each step If the T episode length is ∞, the agent can potentially accumulate infinite rewards. The solution is to have a discount factor γ ∈[0, 1] (0.99 works well) and weight rewards higher sooner than later. yi,t ≃r(si,t, ai,t) + γ ˆV π(si,t+1) Introducing the discount factor γ changes the MDP: It introduces a terminal state with no rewards. The advantage function can be expressed in two ways. We can either use the value function based definition or the Monte Carlo definition: • ˆAπ C(st, at) := r(st, at) + γ ˆV π(st+1) −ˆV π(st) + Lower variance - Higher bias if value is wrong.


π C(st, at) := r(st, at) + γ ˆV π(st+1) −ˆV π(st) + Lower variance - Higher bias if value is wrong. • ˆAπ MC(st, at) := P∞ t′=t γt′−tr(st′, at′) −ˆV π(st) + No bias - Higher variance due to single-sample estimate We can combine the above two approaches by using the value function after n steps. This is N-step rewards: ˆAπ n(st, at) := t′+n X t′=t γt′−tr(st′, at′) −ˆV π(st) + γn ˆV π(st+n) Policy gradient can reuse old data via importance sampling. Excursion: Importance Sampling Suppose we want to evaluate Ex∼p(x)[f(x)] = Z p(x)f(x) dx but we cannot sample from p and we can merely evaluate its density. We can sample from another distribution q and use the importance p(x)/q(x): Ex∼p(x)[f(x)] = Z q(x)p(x) q(x)f(x) dx = Ex∼q(x) p(x) q(x)f(x)  The closer p(x) ≃q(x), the better the estimate.


= Z q(x)p(x) q(x)f(x) dx = Ex∼q(x) p(x) q(x)f(x)  The closer p(x) ≃q(x), the better the estimate. 17


4 VALUE-BASED RL AND ACTOR-CRITIC METHODS Suppose the older iteration’s parameter is θ and the new iteration parameter is θ′. We can perform importance sampling by substituting the expectation on πθ′ with one on πθ on policy gradient J(θ′) = Eτ∼πθ(τ) πθ′(τ) πθ(τ) r(τ)  ∇θ′J(θ′) = Eτ∼πθ(τ) ∇θ′πθ′(τ) πθ(τ) r(τ)  = Eτ∼πθ(τ) πθ′(τ) πθ(τ) ∇θ′ log πθ′(τ)r(t)  = Eτ∼πθ(τ) " TY t=1 πθ′(at|st) πθ(at|st) ∇θ′ log πθ′(τ)r(t) # where we have used the identity πθ′(τ) πθ(τ) =  p(s1) QT t=1 πθ′(at|st) p(st+1|st, a)  p(s1) QT t=1 πθ(at|st) p(st+1|st, a) = TY t=1 πθ′(at|st) πθ(at|st) The problem with importance distribution is that if the two distributions πθ, πθ′ are very different, the estimator becomes hard to evaluate.


s that if the two distributions πθ, πθ′ are very different, the estimator becomes hard to evaluate. The solutions of this: • Penalise the gradient step so θ = θ′. e.g. θ′ ←arg max(θ′ −θ)∇θJ(θ), ∥θ′ −θ∥2 ≤ϵ Since we care about the distribution not shifting, we can instead use a divergence penaliser θ′ ←arg max(θ′ −θ)∇θJ(θ), DKL(πθ′∥πθ) ≤ϵ • If the distribution changes too much, we can also perform roll-out again. Trust Region Policy Optimisation (TRPO) applies all the tricks: • Use advantage function to reduce the variance • Use importance sampling to take multiple gradient steps • Constrain the optimisation objective in the policy space Proximal Policy Optimisation (PPO) applies all the tricks in TRPO, and penalises large policy updates using a loss function on πθ′(a|s)/πθ(a|s).


l the tricks in TRPO, and penalises large policy updates using a loss function on πθ′(a|s)/πθ(a|s). It is the optimisation used in now state-of-the-art RL models. Example RL with human feedback (RLHF) is an algorithm where the reward function is augmented by human feedback oracle. In PPO for Large Language Models, we have a reward function R(s; p) for output s to 18


5 Q-LEARNING prompt p. The reward is higher when a human prefers the output. When we train a network to approximate R(s; p), it may fail to assign low scores for unnatural inputs s, so we penalise it with inputs that deviate from human language. ˆR(s; p) := R(s; p) −β log pRL(s) pPT(s) Policy density Pre-trained model density Hyperparameter The LLM is trained in 3 phases: 1. Collect human prompt and answer pairs and fine-tune GPT-3 with this data 2. A prompt and several models are sampled and a human labeler ranks the answers. The ranking is used to train a reward model. 3. Train the policy from (1) using reward function in (2). Question: Do we encode logic and other aspects in the reward function ˆR? No. 5 Q-Learning • Playing Atari with Deep Reinforcement Learning. Mnih et al.


reward function ˆR? No. 5 Q-Learning • Playing Atari with Deep Reinforcement Learning. Mnih et al. (2013) Announcements • 04-17: Project Survey Due • 04-19: Homework 1 Due, Homework 2 Out 5.1 Actor-Critic Methods The value formulation gives rise to the actor-critic algorithm. The policy πθ is the actor and the advantage function ˆAπ is the critic. Algorithm: Online Actor-Critic 1. Sample action a ∼πθ(a|s), get (s, a, s′, r) 2. Update ˆV π using target r + γ ˆV π(s′) 3. Evaluate ˆA(s, a) = r(s, a) + γ ˆV π(s′) −ˆV π(s) 4. Evaluate ˆ∇θJ(θ) := ∇θ log πθ(a|s) ˆAπ(s, a) 19


5 Q-LEARNING 5. Update θ ←θ + α ˆ∇θJ(θ) 5.2 Policy and Value Iterations The only reason we want the advantage, value, and Q functions is because we want a better estimate of the rewards to go. However, we could extract more information from them. The Q-function represents the sum of estimated future rewards, and we could engineer this function into a policy. Suppose we have a policy π(a|s). We can look at the advantage function Aπ(st, at) to find a better action a′ than the average action given by π: arg maxat Aπ(st, at). Taking this argmax is at least as good as a random action drawn from the policy π(at|st) no matter what the policy is. Hence the new policy can be π′(at|st) := 1{at = arg max a Aπ(st, a)} (5.1) Algorithm: Policy Iteration 1. Evaluate Aπ(s, a) 2.


π′(at|st) := 1{at = arg max a Aπ(st, a)} (5.1) Algorithm: Policy Iteration 1. Evaluate Aπ(s, a) 2. Set π ←π′ The Aπ(s, a) can be evaluated as Aπ(s, a) := r(s, a) + γ Ep(s′|s,a)[V π(s′)] −V π(s) Since V π(s) is independent on a, it does not factor into the arg max, so we can replace Aπ by Qπ arg max a Aπ(st, a) = arg max a Qπ(st, a) with Qπ(s, a) := r(s, a) + γ E[V π(s′)] With this we can skip policy update function entirely. Algorithm: Policy Iteration 1. Evaluate Qπ(s, a) 2. Set π ←π′ Since we can just let policy be arg maxa Q(s, a), we can skip policy entirely. This gives the Value Iteration algorithm 20


5 Q-LEARNING Algorithm: Value Iteration 1. Q(s, a) ←r(s, a) + γ Ep(s′|s,a)[V (s′)] 2. V (s) ←maxa Q(s, a) We can fit the policy function using some MSE loss: L(ϕ) := ∥Vϕ(s) −y∥, y := arg max a r(s, a) + γ Ep(s′|s,a)[Vϕ(s′)] The evaluation of maxa Q(s, a) maybe intractable, since the dynamics model generating the state s′ is unknown to us and evaluating this maximum requires evaluation of all counterfactual a’s. We cannot predict the reward provided by taking the action Vϕ(s). 5.3 Q-Learning The solution to the intractable dynamics model is by fitting the Qϕ : S × A →R function instead of the value function. Algorithm: Q-Learning 1. Sample {(si, ai, s′ i, ri)} using some policy, where s′ i is the next state 2. yi ←r(si, ai) + γ E[Vϕ(s′ i)] ≃r(si, ai) + γ maxa′ i Qϕ(s′ i, a′ i) 3.


re s′ i is the next state 2. yi ←r(si, ai) + γ E[Vϕ(s′ i)] ≃r(si, ai) + γ maxa′ i Qϕ(s′ i, a′ i) 3. Fit Qϕ(si, ai) to yi Now the Q-function is dependent on the action a, so we do not need to simulate the dynamics model. Qϕ(s, a) = r(s, a) + γ max a′ Qϕ(s′, a′) Question: What would happen if we have a continuous action space? We can perform some optimisation. We will cover this in the next lecture. Question: How do we get s′? s′ is the state the system will be in after taking action a (not a′). This fixes the state to the particular action a. For an optimal policy π⋆derived from Equation 5.1, it satisfies the Bellman Equation Q⋆(st, at) = Est+1∼p(st+1|st,at) h r(st, at) + γ max a′ Q∗(st+1, a′) i Q-Learning essentially learns a table of optimal actions to take for every single possible state


) i Q-Learning essentially learns a table of optimal actions to take for every single possible state. 21


6 PRACTICAL DEEP RL IMPLEMENTATIONS Algorithm: Fitted Q-iteration 1. Collect dataset {(si, ai, s′ i, ri)} using some policy, where s′ i is the next state. 2. yi ←r(si, ai) + γ maxa′ i Qϕ(s′ i, a′ i) (Bellman Equation) 3. ϕ ←arg minϕ 1 2 P i ∥Qϕ(si, ai) −yi∥2 4. Repeat (2) – (3) 5. π(a|s) := arg maxa Qϕ(s, a) + Q-Learning is an off-policy algorithm. + More efficient than on-policy methods + Can update the policy without seeing the reward. The algorithm does not have to experience the reward in any trajectory and decide by looking up the next step Qϕ + Easy to parallelise - Lots of tricks to make it work - Potentially harder to learn than just a policy Question: Is there a way to structure the neural net such that its easy to sample the argmax?


olicy Question: Is there a way to structure the neural net such that its easy to sample the argmax? If A is discrete, we could either use the action as a part of the input, or have multiple outputs as the logits of a categorical distribution. The latter usually works better. If A is continuous, we could have the model output a scalar f(s, a) ... 6 Practical Deep RL Implementations • Deep Reinforcement Learning with Double Q-learning. Hasselt et al. (2015) • QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Kalashnikov et al. (2018) Announcements • 04-19: Homework 1 Due, Homework 2 Out • 04-26: Project Proposal due 22


6 PRACTICAL DEEP RL IMPLEMENTATIONS 6.1 Q-Learning Tricks Q-Learning is not gradient descent algorithm. Fitting the Q-function requires a gradient descent but the Q-iteration step, which evaluates yi using Bellman Equation, does not use gradient descent. We can run Q-learning online: Algorithm: Online Q-iteration 1. Rollout policy to get {(si, ai, s′ i, ri)} 2. yi ←r(si, ai) + γ maxa′ i Qϕ(s′ i, a′ i) 3. ϕ ←arg minϕ 1 2 P i ∥Qϕ(si, ai) −yi∥2 4. Repeat (2) – (3) Potential problems in Q-learning: • Correlated samples in online Q-learning: If we take action ai and observe (si, ai, s′ i, ri), this is not iid data and sequential data are strongly correlated. Having strongly correlated data from the same trajectory, the fitted Q-function would only work for the tiny snippet of trajectory and not


he same trajectory, the fitted Q-function would only work for the tiny snippet of trajectory and not be able to generalise. One solution is Replay Buffers, a set B which stores historical trajectories. Usually B is large to decorrelate the trajectories Algorithm: Online Q-iteration with Replay Buffer 1. Sample {(si, ai, s′ i, ri)} from dataset B, a replay buffer initially populated by rolling out some policy. 2. yi ←r(si, ai) + γ maxa′ i Qϕ(s′ i, a′ i) 3. ϕ ←arg minϕ 1 2 P i ∥Qϕ(si, ai) −yi∥2 4. Repeat (2) – (3) for k iterations (commonly k = 1) 5. Rollout the Q-learning based policy and update B • The target of fitting is constantly moving The step ϕ ←arg minϕ 1 2 P i ∥Qϕ(si, ai) −yi∥2 is a well-defined stable regression step. Step (2) moves the target of regression.


Qϕ(si, ai) −yi∥2 is a well-defined stable regression step. Step (2) moves the target of regression. One solution is Target Networks. We save an earlier version ϕ′ of ϕ and fit the current step Qϕ function to the target yi created from Qϕ′ With this change, the step of fitting ϕ becomes a supervised regression problem 23


6 PRACTICAL DEEP RL IMPLEMENTATIONS Q a True Maximum max Q∗ Noisy Maximum max Qϕ Q∗ Q Figure 6.1: The maximum obtained from Q-learning overshoots the true maximum because it is evaluated from the imperfect noisy neural network output Algorithm: Classic Deep Q-Learning (DQN) 1. Save target network parameters ϕ′ (every N steps) 2. Sample {(si, ai, s′ i, ri)} from dataset B, a replay buffer initially populated by rolling out some policy. 3. yi ←r(si, ai) + γ maxa′ i Qϕ′(s′ i, a′ i) 4. ϕ ←arg minϕ 1 2 P i ∥Qϕ(si, ai) −yi∥2 5. Repeat (2) – (4) for k iterations (commonly k = 1) 6. Rollout the Q-learning based policy and update B Is Q-Learning accurate? The learned Q function Qϕ may overestimate the rewards since each step of Qϕ is a maximum of the noisy Qϕ of the next step.


ϕ may overestimate the rewards since each step of Qϕ is a maximum of the noisy Qϕ of the next step. Excursion: Imagine if we have two random variables X1, X2, then we have E[max(X1, X2)] ≥max(E[X1], E[X2]) This is because the two separate expectations eliminate the dependent component between the two variables. We can curb this effect using Double Q-Learning, which is a modification of target network with: y ←r + γQϕ′(s′, arg max a′ Qϕ(s′, a′)) where the value is samples from one Q-function (target network) and action from another Q-function (current network). In online Q-learning algorithm, the agent may get stuck in some state. In order to encourage exploration we may use the following policy instead. This is ϵ-greedy 24


7 MODEL-BASED RL exploration π(at|st) := ( 1 −ϵ at = arg maxat Qϕ(st, at) ϵ/(|A| −1) otherwise A common trick starts with higher ϵ and gradually drop to encourage exploration. 7 Model-Based RL • Deep Dynamics Models for Learning Dexterous Manipulation. Nagabandi et al. (2020) • When to Trust Your Model: Model-Based Policy Optimization. Janner et al. (2019) Announcements • 04-26: Project Proposal due; (Graded Lightly) • 05-03: Homework 2 due 7.1 Gradient-based and Sampling-based Optimisation Two common classes of optimisation methods exist. • Gradient-based optimisation or first-order optimisation relies on local gradient to guide the next sample + Scalable to high dimensions + Works well especially in overparameterised regimes - Requires nice optimisation landscape • Sampling-based optimis


ecially in overparameterised regimes - Requires nice optimisation landscape • Sampling-based optimisation or zeroth-order optimisation relies on sampling heuristic regions + Parallelisable + Requires no gradient information - Scales poorly to high dimensions Question: Is sampling-based optimisation more prone to getting stuck in local minima? In general it is not more prone. Question: Do people use hybrid methods? Yes A sample-based method is the Cross-Entropy Method (CEM). 25


7 MODEL-BASED RL θ L(θ) (a) Gradient-based optimisation θ L(θ) (b) Sample-based optimisation Figure 7.1: Gradient and Sampling based optimisations Algorithm: Cross-Entropy Method 1. Sample from distribution pi(θ) 2. Rank samples according to loss θ1, . . . , θK 3. Fit Gaussian distribution pi to “elite” samples θ1, . . . , θK Question: Why is this called the Cross-Entropy Method Step 3 fits the theoretical distribution to the empirical distribution using cross entropy. 7.2 Learning a Dynamics Model It would be useful if we could simulate the real world in reinforcement learning. Given samples si, s′ i, ai, we can fit a dynamics model pϕ(s, a) 7→s′: min ϕ X i ∥pϕ(sI, ai) −s′ i∥2 and use such a learned dynamics model to train the reinforcement learning algorithm.


I, ai) −s′ i∥2 and use such a learned dynamics model to train the reinforcement learning algorithm. When such a model exists, we could optimise the policy using ˆθ := arg max θ max t0+H X t=t0 r(st, at) + V (st0+H+1) Terminal function where the {(st, at)}H t=t0 sequence is obtained by rolling out the policy πθ on the dynamics model pϕ. H is the planning horizon. The terminal function is there to partially mitigate the short-sightedness problem, but it introduces the problem of having to train V . 26


7 MODEL-BASED RL Algorithm: Model-Based Reinforcement Learning 1. Collect data D by rolling out π 2. Train model pϕ(s′|s, a) using D 3. Optimise the objective function P t r(st, at) using gradient (backpropagation through time) or sample based optimisation 4. Execute planned actions and append the visiting tuples (s, a, s′) to D. This is to discourage the dynamics model from diverging from reality. Question: Is the horizon H much shorter than the actual horizon? Yes. This is a drawback and could cause the model to be greedy. This method could fail due to inaccuracy in pϕ. The policy may try to exploit the inaccuracy in the model. 7.3 Using the Learned Dynamics Model Another approach is to plan and re-plan using the model.


odel. 7.3 Using the Learned Dynamics Model Another approach is to plan and re-plan using the model. We optimise the action sequence at:t+H using pϕ and execute the first planned action at. To reduce cost, the re-planning can be done every 5 or 10 steps instead of just 1. At this point it is important to note the distinction between open and closed loop algorithms • Open-Loop: Agent sends a complete action sequence into the world without regards of the updated state. • Closed-Loop: Agent sends action into world and world feeds action back into agent Algorithm: Model-Predict Control 1. Rollout base policy π0(at|st) (e.g. random) to collect data D = {(s, a, s′)} 2. Learn dynamics model pϕ 3. Use model pϕ to optimise action sequence at:t+H 4.


a D = {(s, a, s′)} 2. Learn dynamics model pϕ 3. Use model pϕ to optimise action sequence at:t+H 4. Execute the first planned action and observe state s′ 5. Append (s, a, s′) to dataset D + Re-plan to correct for model errors - Computationally intensive + Simple 27


7 MODEL-BASED RL Figure 7.2: Augmenting existing data + Easy to plug in different reward functions - Only practical for short-horizon problems or very shaped reward functions Short horizon is because its both too computationally intensive and inaccurate to use the model for long horizons. Question: Does MPC always give the most optimal solution No. It could happen when the horizon is too short, or the optimiser is not powerful enough, or the model is inaccurate. Can we train a policy using a learned model? We can distill planner’s action into a policy + No longer computationally intensive at test time - Still limited to short-horizon problems unless a terminal function is used. To solve the short-horizon problem, we can also augment model-free RL methods using data from the model.


lve the short-horizon problem, we can also augment model-free RL methods using data from the model. With a given trajectory (si), we can augment data using the learned model pϕ by branching from s1. To mitigate this issue, we can create augmented trajectories by starting from later states si, which improves coverage without having to do long rollouts. Question: How do we know the reward for this? We either know the functional form of the reward and evaluate it on augmented data, or we can have the model predict both the reward and the state pϕ(r, s′|s, a). 28


8 REWARD LEARNING Algorithm: Model-Based Policy Optimisation 1. Rollout policy πθ and add data (s, a, s′) to dataset D 2. Update dynamics model pϕ(s′|s, a) (or pϕ(r, s′|s, a) if the reward function is unknown) from D 3. Collect synthetic rollouts using πθ in model pϕ from states in D and add to Dmodel 4. Update policy πθ and critic Q using Dmodel ∪D Usually in implementations, Dmodel and D have fixed sizes and are first-in-first-out. Sometimes older data are kept to prevent regression. In practice the policy would be stochastic to encourage exploration + Models are immensely useful if easy to learn + Models can be trained without reward labels (self-supervised) + Models are somewhat task-agnostic (can sometimes be transferred across rewards) - Models don’t optimise for task performance - S


ostic (can sometimes be transferred across rewards) - Models don’t optimise for task performance - Sometimes harder to learn than a policy 8 Reward Learning • Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition. Fu et al. (2018) • Deep reinforcement learning from human preferences. Christiano et al. (2017) Announcements • 04-26: Project Proposal due • 05-03: Homework 2 due Rewards in RL sometimes come from straightforward sources such as scores in computer games. In real world scenarios it could be unclear what the reward function may be. Often we use a proxy. In Direct Imitation Learning, the policy directly mimics the actions of an expert. - There is no reasoning about outcomes or dynamics in this method - The expert might have different degrees


is no reasoning about outcomes or dynamics in this method - The expert might have different degrees of freedom - Might not be possible to provide demonstrations 29


8 REWARD LEARNING + Figure 8.1: The goal classifier assigns a score of 1 to states close to the item and a reward of 0 for states farther away from the item 8.1 Rewards from Behaviour A Goal Classifier is a model r that discerns goal states from other states. e.g. If the agent is trying to find an item, it could be rewarded when it gets to a position (state) that is close to the item. The goal classifier could be trained to classify successful and unsuccessful states and use this goal classifier as the reward function. - The agent may learn artefacts related to the goal classifier and not focus on the task itself. Can we prevent the RL algorithm from exploiting the classifier’s weaknesses? During the RL training process, policy roll-outs generate fresh new samples.


assifier’s weaknesses? During the RL training process, policy roll-outs generate fresh new samples. We can add these sample states to the datasets D± used to train the goal classifier. Algorithm: Goal Classifier 1. Collect initial successful states D+ and unsuccessful states D− 2. Update classifier r using D+, D− 3. Roll-out policy π and obtain trajectories st, at, . . . 4. Update policy π using r based reward. 5. Add visited states st to D− 6. Repeat from (2) where to ensure the classifier receives balanced samples, • Keep D+ and D−the same size using first-in-first-out on D− • Or, when sampling batches from D± we sample half from D+ and half from D− (oversampling D+ undersampling D−) • Large enough D+ is needed to not overfit into the small D+ set • Or, we could regularise the classifier


ge enough D+ is needed to not overfit into the small D+ set • Or, we could regularise the classifier 30


8 REWARD LEARNING Excursion: Generative Adversarial Networks Generative Adversarial Networks are a class of models where two networks, a generator and a discriminator, play a two-player zero-sum game. The generator attempts to generate new samples which mimics a data distribution p(x) (from which we have samples from) and the discriminator attempts to tell apart the synthesised sample from real samples. The discriminator may pickup subtle details from the generator artefacts. Some techniques to regularise the discriminator in GAN: Gradient clipping, Spectral normalisation, instance noise Alternatively, in the case of not trying to reach a goal, we could use Generative Adversarial Imitation Learning: Algorithm: Generative Adversarial Imitation Learning 1.


Generative Adversarial Imitation Learning: Algorithm: Generative Adversarial Imitation Learning 1. Collect expert demonstration trajectories st, at, . . . from expert policy πexpert and add (s, a) pairs to D+. 2. Update classifier r using D+, D− 3. Roll-out policy π and obtain trajectories st, at, . . . 4. Update policy π using r based reward. 5. Add visited states (s, a) to D−and return to (2) + Practical framework for task specification ~ Adversarial training can be unstable - Requires examples of desired behavior or outcomes 8.2 Rewards from Human Preferences Suppose we have trajectories (full or partial roll-outs) τ+, τ−, where τ+ ≻τ−, i.e. humans feedback indicates τ+ is better than τ−. We want a reward that gives r(τ+) > r(τ−).


τ−, i.e. humans feedback indicates τ+ is better than τ−. We want a reward that gives r(τ+) > r(τ−). We can define the estimated probability of τ+ ≻τ−to be σ(rθ(τ+) −rθ(τ−)). Then we can maximise the log probability max θ Eτ+≻τ−[log σ(rθ(τ+) −rθ(τ−))] The states that are compared must have the same initial state. In LLM, the states are instead constrained to have the same prompt. This is to ensure comparisons are across the same category. 31


8 REWARD LEARNING Figure 8.2: Sukhbaatar et al. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. ICLR 2018 Algorithm: Learn a Reward Function from Human Preferences 1. Given dataset, sample batches of trajectories {τi} and ask humans to rank. 2. Compute their rewards rθ(τi) under the current reward model rθ 3. For all k 2  pairs per batch, compute ∇θ Eτ+≻τ−[log σ(rθ(τ+) −rθ(τ−))] 4. Update θ using the gradient from (3) and return to (2) This can be done in the loop of online RL. This is done to train ChatGPT: 1. Train f to do next-token prediction on massive dataset 2. Supervised fine-tuning to follow instructions 3. Train reward model rθ 4. Use PPO to maximise P rθ −DKL( ˆf∥f) where ˆf is a model trained on natural language to encourage f to generate natural output


DKL( ˆf∥f) where ˆf is a model trained on natural language to encourage f to generate natural outputs. The feedback can also come from another AI model since critique is easier than generation. Learning rewards from human preferences: + Pairwise preferences are easy to provide + Has been deployed at scale - May require supervision in the loop of RL Can RL agents provide their own goals using “unsupervised” RL? One example is that we can formulate goals using a two-player game with a goal setter and goal reacher. 32


9 OFFLINE RL 9 Offline RL • Conservative Q-Learning for Offline Reinforcement Learning. Kumar et al. (2020) • MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. Yu et al. (2021) Announcements • 05-03: Homework 2 due • 05-03: Homework 3 out Why would we do offline RL? Offline RL refers to RL on a given static dataset. Offline RL is useful when collection of new data or rolling out of policies is expensive. It can also utilise existing datasets. In offline RL, we assume we have a dataset {(s, a, s′, r)} sampled from some unknown behaviour policy πβ, which can be a mixture of policies. In this policy, s dim dπβ(·), a ∼πβ(·|s), s′ ∼p(·|s, a), r = r(s, a) The objective is to learn max θ X t Est∼dπθ(·),at∼πθ(·|st)[r(st, at)] This objective is online and cannot be evaluated pu


earn max θ X t Est∼dπθ(·),at∼πθ(·|st)[r(st, at)] This objective is online and cannot be evaluated purely from an offline dataset. Can we just use off-policy algorithms? e.g. the standard Q-learning algorithm min Q min X (s,a,s′)∈D Q(s, a) −(r(s, a) + γ max a′ Q(s′, a′))2 In most cases this does not work. The reason why this doesn’t work is that when Q is evaluated on an action a′ not in the dataset: • Q-function will be unreliable on OOD actions • max will seek out actions where Q-function is over-optimistic • After values propagate, Q-values will become substantially overestimated. Question: Could you do dataset attribution in offline RL to determine which part of the dataset lead to an action? Haven’t seen a previous work like this but it would be interesting to look at.


t lead to an action? Haven’t seen a previous work like this but it would be interesting to look at. Instead of using a maximum, we could replace maxa′ with Ea∼πnew(·|s)[Q(s, a)], πnew := arg max πnew Ea∼πnew[Q(s, a)] and πnew should be close to πβ. There are two ways to encourage closedness: 33


9 OFFLINE RL Action Reward Policy πβ πnew πoptimal Figure 9.1: KL Divergence policy may be closer to the behaviour policy than the optimal policy. 9.1 Data Constraint Methods • Absolute continuity: Allow πnew(a|s) > 0 only if πβ(a|s) ≥ϵ + Close to what we want - Challenging to implement in practice • Divergence: minimise DKL(πnew∥πβ) + May not be what we want - Easy to implement in practice. Both of these cannot be evaluated exactly and models use some imitation learning to fit ˆπβ to πβ. This leads to the data constraint methods: Algorithm: Data Constraint Methods Different ways of implementing data constraint methods: 1. Change actor update via Lagrange Multiplier λ θ ←arg max θ Es dim D,a∼πθ(·|s)[Q(s, a)] −λ DKL(πθ∥πβ) = arg max θ Es dim D,a∼πθ(·|s)[Q(s, a)] + λ log πβ(a|s) + λ H(πθ(·|s


∼πθ(·|s)[Q(s, a)] −λ DKL(πθ∥πβ) = arg max θ Es dim D,a∼πθ(·|s)[Q(s, a)] + λ log πβ(a|s) + λ H(πθ(·|s)) 2. Modify reward function ¯r(s, a) := r(s, a) −DKL(πθ∥πβ) Constraints by modeling πβ: + Intuitive 34


9 OFFLINE RL - Often this is too conservative in practice. Relaxing λ can mitigate this somewhat but risks running off the distribution support. 9.2 Conservative Methods We could also overestimation in offline RL. Without explicitly modeling behaviour policy, we can push down on large Q-values. ˆQ ←arg min Q max µ′ E(s,a,s′)∼D [(Q(s, a) −(r(s, a) + γ Eπ[Q(s′, a′)]))] + α Es∼D,a∼µ(·|s)[Q(s, a)] −α E(s,a)∼D[Q(s, a)] Standard Critic Update (Bellman) Push down on large Q values Push up on data Q values Without the push down term, we have ˆQπ ≤Qπ for all (s, a). With the push up term, this we no longer guarantee ˆQπ ≤Qπ for all (s, a) but we still have Eπ(a|s)[ ˆQπ(s, a)] ≤Eπ(a|s)[Qπ(s, a)]. This is known as Conservative Q-Learning (CQL).


still have Eπ(a|s)[ ˆQπ(s, a)] ≤Eπ(a|s)[Qπ(s, a)]. This is known as Conservative Q-Learning (CQL). Often we want to add a regulariser term that maximises the entropy H(µ) e.g. R(µ) := Es∼D[H(µ(·|s))]. The optimal µ(a|s) ∝exp(Q(s, a)). Thus Es∼D,a∼µ(·|s)[Q(s, a)] = log X a exp Q(s, a) Algorithm: Conservative Q-Learning 1. Update ˆQ using LCQL using dataset D 2. Update policy π: • If actions are discrete, π(a|s) = 1{a = arg maxa ˆQ(s, a)} • If actions are continuous, θ ←θ + η∇θ Es∼D,a∼πθ(·|s)[ ˆQπ(s, a)] In Model-Based Offline RL, we use the following model-generated state-action pairs instead of µ: E(s,a)∼ρ[Q(s, a)] Intuition: If model produces data that look clearly different from the real data, it’s easy for Q to make it look bad. Here a policy is needed to learn ρ.


ferent from the real data, it’s easy for Q to make it look bad. Here a policy is needed to learn ρ. The difference between this and µ is that it also attempts to push down on model generated states, whereas µ only pushes on the actions. Implicitly constrain data by penalising Q: + Simple + Works fairly well in practice - Need to tune α 35


10 OFFLINE RL CONT. Figure 9.2: Offline RL can stitch together trajectories s1 →s3, s7 →s9 and learn a policy that goes from s1 to s9. 9.3 Data Stitching Why Offline RL instead of imitation learning? Offline learning can stitch together different parts of the data with optimal behaviour by leveraging reward even when the data is sub-optimal. Imitation learning cannot outperform the expert. 10 Offline RL cont. • Offline Reinforcement Learning with Implicit Q-Learning. Kostrikov et al. (2021) Announcements • 05-03: Homework 2 due • 05-03: Homework 3 out (covers weighted and conditional imitation learning) • Two office hours moved from in-person to hybrid The key challenge in offline RL is the over-estimation of Q-values because of the shift between πβ and πθ.


challenge in offline RL is the over-estimation of Q-values because of the shift between πβ and πθ. We can • Explicitly constrain to the data by modeling πβ • Implicitly constrain to the data by under-estimating Q There are other ways to leverage reward in imitation learning. Pure imitation learning does not use reward information. If we have reward labels, can we imitate only good trajectories? One way is to filter the data: 36


10 OFFLINE RL CONT. r = 0 r = 1 Rs,a = 0 Rs,a = 1 Figure 10.1: The advantage at the intersection prefers the action towards the direction of higher reward. This allows the two trajectories to be stitched together. Algorithm: Filtered Behaviour Cloning (%BC) 1. Collect data D := {(s, a, s′, r)}: 2. Filter data ˜D := {τ : r(τ) > η} for some η such that ˜D = k% |D| 3. Imitation learning maxθ Es,a∼˜D log π(a|s) A policy trained like this is a very good baseline to test against. 10.1 Weighted Imitation Learning Could we weight each transition depending on how good the action is? We can measure the goodness of an action using the advantage function Aπ(s, a) = Qπ(s, a) −V π(s) to measure the advantage of taking action a relative to the average action on π.


= Qπ(s, a) −V π(s) to measure the advantage of taking action a relative to the average action on π. We can perform weighted imitation learning using A as the weight max θ E(s,a)∼D[log π(a|s) exp(A(s, a)) ] (10.1) Standard Imitation Learning Advantage weights Theorem 10.1. The advantage-weighted objective (Equation 10.1) approximates KL-constrained objective πnew := arg max π:DKL(π∥πβ)≤ϵ Ea∼π(·|s) Q(s, a) Proof. See Relative Entropy Policy Search. Peters et al. and On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference. Rawlik et al. (“Ψ-Learning”). For now we will use the advantage of the behaviour policy Aπβ. How can we estimate the advantage function? One way is to estimate the value function V πβ with Monte Carlo methods ˆV πβ := min V E(s,a)∼D[(V (s) −Rs,a )2


to estimate the value function V πβ with Monte Carlo methods ˆV πβ := min V E(s,a)∼D[(V (s) −Rs,a )2] 37


10 OFFLINE RL CONT. r = 0 r = 0 r = 1 Rs,a = 0 Rs,a = 0 Rs,a = 0 Rs,a = 1 Figure 10.2: The advantage function would not be able to distinguish the two actions at s1 when trained with Monte Carlo. Therefore it would not be able to stitch together the trajectory between the two red dots. where Rs,a is the empirical reward or reward to-go in the trajectory Rst,a = PT t′=t r(st′). then we can approximate ˆAπβ(s, a) = R(s, a) −ˆV πβ(s). This gives the Advantage-Weighted Regression algorithm. Algorithm: Advantage-Weighted Regression (AWR) with Monte Carlo Updates 1. Fit value function ˆV πβ ←arg min V E(s,a)∼D[(V (s) −Rs,a)2] 2. Train policy ˆπ ←arg max π E(s,a)∼D  log π(a|s) exp  1 α(Rs,a −ˆV πβ(s)  + Simple + Avoid querying or training on any OOD actions - Monte Carlo estimation is noisy -


β(s)  + Simple + Avoid querying or training on any OOD actions - Monte Carlo estimation is noisy - ˆAπβ assumes weaker policy than ˆAπθ AWR can fail when too many trajectories are needed to be stitched together, as shown in example Figure 10.2. In order to mitigate this issue we can train ˆQ using an iterative process: ˆQπ ←min Q E(s,a)∼D h (Q(s, a) −(r + γ Ea′∼πθ[Q(s′, a′)]))2i then we would have ˆAπ(s, a) = ˆQπ(s, a) −Ea′∼πθ(·|s)[ ˆQπ(s, a′)] 38


10 OFFLINE RL CONT. Algorithm: Advantage-Weighted Regression with Temporal Difference (TD) Updates 1. Fit Q-function ˆQπ ←min Q E(s,a)∼D hQ(s, a) −(r + γ Ea′∼πθ(·|s)[Q(s′, a′)]) 2i 2. Estimate advantage ˆAπ(s, a) = ˆQπ(s, a) −Ea′∼πθ(·|s)[ ˆQπ(s, a′)] 3. Train policy ˆπ ←arg max π E(s,a)∼D  log π(a|s) E  1 α ˆAπ(s, a)  + Policy still only trained on action data + Temporal difference updates instead of Monte Carlo - The evaluation of a ∼π(·|s) is particularly problematic because it possibly queries OOD actions! In AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. Nair et al. (2020), the evaluation of a ∼π(·|s) is replaced by a′ ∼D to avoid sampling out of distribution.


al. (2020), the evaluation of a ∼π(·|s) is replaced by a′ ∼D to avoid sampling out of distribution. This leads to the SARSA (State-Action-Reward-State-Action) algorithm: Algorithm: SARSA Update the ˆQ function with ˆQπ ←min Q E(s,a,s′,a′)∼D  Q(s, a) −(r(s, a) + γ Q(s′, a′) ) 2 A sample of V πβ(s′) where the tuple (s, a, s′, a′) represents two consecutive states in the trajectory. This ties Q with πβ. We can train a policy better than πβ using an asymmetric loss function. The L2 loss function targets the mean of a random variable. We can target a higher or lower percentile using expectile regression, with a loss function ℓτ 2(x) := ( (1 −τ)x2 x > 0 τx x ≤0 39


10 OFFLINE RL CONT. Figure 10.3: The expectile loss function ℓτ 2 and its use cases in regression from Kostrikov et al. (2021) Algorithm: Implicit Q-learning (IQL) 1. Fit V-function with expectile-loss (with large τ > 0.5): ˆV (s) ←arg min V E(s,a)∼D h ℓτ 2(V (s) −ˆQ(s, a)) i 2. Fit Q-function with MSE loss: ˆQπ ←min Q E(s,a,s′)∼D  Q(s, a) −(r + γ ˆV (s′)) 2 At the end, extract policy: ˆπ ←arg max π E(s,a)∼D  log π(a|s) exp  1 α( ˆQ(s, a) −ˆV (s))  + Never need to query OOD actions + Policy is only trained on actions in data + Decoupling actor and critic training, so computationally fast. Question: Do we still have to do it if we have access of the underlying policy πβ? Yes. • It could be used to augment the data in case a state is never associated with some action in D.


. • It could be used to augment the data in case a state is never associated with some action in D. • In Temporal Difference updates, the Q-function’s estimation of the remainder can be directly taken on the distribution a′ ∼πβ(·|s) if πβ is available. 40


10 OFFLINE RL CONT. Figure 10.4: Causal transformer used for conditional imitation learning 10.2 Conditional Imitation Learning For some datasets, filtered BC can work really well. The drawback with filtered BC is it discards data. Instead of discarding such data, we can condition the policy on the return achieved on trajectory τ on the state s (return to go). i.e. ˆθ ←arg max θ E(a,s)∼D[log π(a|s, Rs,a)] A policy trained in this manner can mimic both good and bad behaviours and the policy can be tuned to mimic both good and bad data. This is referred to as Upside-down RL, Reward-conditioned policies, or Decision transformers. The model can be sequential: - A policy trained like this cannot be expected to stitch trajectories.


he model can be sequential: - A policy trained like this cannot be expected to stitch trajectories. 10.3 Offline Evaluation and Hyper-parameter Tuning Can we evaluate a policy or compare the performance of two trained policies πθ1, πθ2 without rolling out the policy? There is no reliable way to evaluate off-line. • Roll-out policy and evaluate: + Accurate - Can be risky or expensive ~ Can be risky or expensive This makes the algorithm not fully off-line, at which point it may also be possible to train the policy on-line. • Evaluate in high-fidelity simulator or model + May be good enough in some situations - Developing simulator is hard • Some algorithms have heuristics 41


11 MULTI-TASKED RL AND GOAL-CONDITIONED RL Figure 10.5: Early stopping with CQL In CQL (Conservative Q-Learning), the peak and average Q-values during training can be used to evaluate the policy. By looking at the history of training, this heuristic can be used to early stop CQL before it overfits. See A Workflow for Offline Model-Free Robotic Reinforcement Learning. CoRL ’21 11 Multi-Tasked RL and Goal-Conditioned RL Announcements • 05-17: Homework 3 due The biggest challenge of the reinforcement learning methods so far is sample complexity. The algorithms take a long time and a large amount of data to train. 11.1 Multi-Task Imitation and Policy Gradients The classic supervised learning loss for a single task imitation learning is min θ L(θ, D) = min θ −E(s,a)∼D[log πθ(a|s)] In a multi-ta


s for a single task imitation learning is min θ L(θ, D) = min θ −E(s,a)∼D[log πθ(a|s)] In a multi-task imitation learning, the loss can be aggregated over several task datasets minθ P i L(θ, D). A reinforcement learning task is given by T := { S , A , p(s1) , p(s′|s, a) , r(s, a) } Action space State space Initial state distribution Dynamics Reward 42


11 MULTI-TASKED RL AND GOAL-CONDITIONED RL Question: Does the dynamics model have to be stochastic Not necessarily. A set of tasks {Ti} can be cast as a standard Markov decision process, with states being augmented with a task identifier zi: s := (¯s, zi). The pros and cons of multi-task learning are + Cross-task Generalisation + Easier exploration + Sequencing for long horizon tasks + Reset-free learning: If two tasks are complementary, they can reset each other’s terminal state and be trained in conjunction. + Per-task sample-efficiency gains - Poor generalisation cf. individual RL models This is exhibited in Metaworld (CoRL 2019), a benchmark about multi-task reinforcement learning algorithms. Each task is solvable by a single-task RL model.


bout multi-task reinforcement learning algorithms. Each task is solvable by a single-task RL model. Excursion: Gradient projection When training a multi-task model for task 1, 2, it may happen that the gradient g 1 := ∇θL1 and g 2 := ∇θL2 point in different directions. If the gradients g 1 · g 2 < 0, it may be difficult to train the model. A solution is to project conflicting gradients onto the normal planes of each other ˆg 2 := ( g 2 −projg1 g 2 g 1 · g 2 < 0 g 2 g 1 · g 2 ≥0 With the projection order randomised. This mitigates generalisation issues. The difference between reinforced and supervised learning algorithms is that the agent also decides the data distribution, and so far we have not leveraged this. 11.2 Multi-Task Q-Learning Consider an agent which is trained to play hockey.


e not leveraged this. 11.2 Multi-Task Q-Learning Consider an agent which is trained to play hockey. What if an agent performs a good pass while attempting to shoot a goal? The data does not have to go to waste and it can be labeled in hindsight as an example of passing. This is Hindsight Experience Replay (HER) 43


11 MULTI-TASKED RL AND GOAL-CONDITIONED RL Algorithm: Multi-Task RL with Re-labeling 1. Collect data Dk := {(s1:T, a1:T, zi, r1:T)} 2. Store data in replay buffer D ←D ∪Dk 3. Perform hindsight relabeling: Relabel experience in Dk for task Ti: D′ k := {(s1:T, a1:T, zj, r′ 1:T)} where r′ t = rj(st) 4. Store relabeled data in replay buffer D ←D ∪D′ Methods of choosing Ti in step (3): • Randomly • Tasks in which the trajectory get high reward • Other The mechanism of re-labeling is highly non-trivial and there has been a number of works in this (Eysenbach et al. Rewriting History with Inverse RL, Li et al. Generalized Hindsight for RL, Kalashnikov et al. MT-Opt, Kalashnikov et al. MT-Opt). We can apply re-labeling when • Reward function form is known, relatable • Dynamics consistent across tas


n apply re-labeling when • Reward function form is known, relatable • Dynamics consistent across tasks/goals • Using an off-policy algorithm For example, suppose two tasks are closing and opening a drawer. Can we use episodes from drawer opening task for drawer closing task? The answer is yes since the drawer closing task can be used as negative examples in Q-Learning, but not in policy gradient. 11.3 Goal-Conditioned RL Algorithm: Goal-Conditioned RL with Re-labeling 1. Collect data Dk := {(s1:T, a1:T, zi, r1:T)} 2. Store data in replay buffer D ←D ∪Dk 3. Perform hindsight relabeling: Relabel experience in Dk for task Ti: D′ k := {(s1:T, a1:T, zj, r′ 1:T)} where r′ t = −d(st, sT) 4. Store relabeled data in replay buffer D ←D ∪D′ Alternatively, any state in the trajectory can be used as re


elabeled data in replay buffer D ←D ∪D′ Alternatively, any state in the trajectory can be used as relabeling. 44


12 TRANSFER LEARNING IN RL Goal-Conditioned RL alleviates the exploration problem since any state can be used as a positive example. The reward is measured only by the distance to the target state d(st, sT). Example In Lynch, Khansari, Xiao, Kumar, Tompson, Levine, Sermanet. Learning Latent Plans from Play. ‘19, random human play data (with no task or goal) is fed into the policy as data. 12 Transfer Learning in RL Guest Lecture: Jie Tan Previous works using reinforcement learning on legged locomotion: 1. Learning to Walk via Deep Reinforcement Learning (Haarmoja et al. RSS 2019): The robot learns to walk on its own after 2 ours of training 2. Sim-to-Real (RSS 2018): Using domain randomisation to train a robot 3.


after 2 ours of training 2. Sim-to-Real (RSS 2018): Using domain randomisation to train a robot 3. Learning to Walk in Minutes (Rudin, 2021): Training thousands of robot clones in simulation at the same time reduces the overall training time. 4. Rapid Motor Adaptation (RSS 2021): Adapt to new domains in seconds. 5. Learned Gait Transitions (CoRL 2021): Using energy efficiency to automatically emerge natural gaits. 6. Learning Agile 7. Learning to Navigate Sidewalks (ICRA 2022): Combining perception and locomotion together to navigate sidewalks in urban area. 8. Learning Robust Perceptive Locomotion (Science Robotics 2022) Model Predictive Control (MPC) has made a lot of progress in the past years, but DRL has caught up and produced state-of-the-art results.


e a lot of progress in the past years, but DRL has caught up and produced state-of-the-art results. 12.1 Learning Locomotion There are two approaches to learn locomotion: In the real world, or in the simulation. The problem about real world learning - data efficiency, 60 million steps are needed to learn human-like locomotion behaviour which will take years. - Human supervision is required to reset the robot’s state. - Safety: Robotics can damage things in the real world or itself 45


12 TRANSFER LEARNING IN RL The problem setup is Learning to Walk in Real World with Minimum Human Efforts (CoRL 2020): • Observations: 8 motor angles, roll, pitch, previous action in the last 6 time-steps • Actions: 8 target motor angles • Reward function: A combination of velocity along a direction, turning rate, and a penalisation term for acceleration. Once we have this formulation, we have the Soft Actor-Critic (SAC) max π Eτ∼ρπ " T X t=0 r(st, at) # s.t.      Eπ[−log(πt(·|st))] ≥0 E(st,at)∼ρπ [P fs(st, at)] ≥0 Eπ[−log(πt(·|st))] ≥H Policy rollout of π where fs(st, at) := min( ˆϕ −|ϕt| , ˆρ −|ρt| ) Max safety pitch ˆϕ Max safety roll ˆρ Using Lagrangian multipliers, the goal is equivalent to max π min λ≥0 Eτ∼ρπ " T X t=0 r(st, at) + λfs(st, at) # This zero-sum two-player game forc


valent to max π min λ≥0 Eτ∼ρπ " T X t=0 r(st, at) + λfs(st, at) # This zero-sum two-player game forces fs ≥0. Algorithm: Dual Gradient Descent 1. Randomly initialise π, set λ = 0 2. Rollout policy π 3. Calculate policy gradient ∂L ∂θ 4. π ←π + α ∂L ∂π 5. Calculate gradient ∂L ∂λ 6. λ := max(0, λ −β ∂L ∂λ) 7. Return to step (2) The Sim-to-Real Gap is the difference between simulation and real world • Dynamics: The embodiment gap makes the motor control different • Perception: The visual sensors perceive differently in simulation and real world. What are the causes of the sim-to-real gap? 46


12 TRANSFER LEARNING IN RL • Unmodeled dynamics: Inaccurate contact models • Wrong simulation parameters • Latency • Actuator dynamics • Stochastic real environment • Noise, Numerical accuracy In the past few years, there has been an exponential growth on the papers for sim-to-real methods. 12.2 Sim-to-Real Gap There are three methods of bridging the sim-to-real gap. System Identification refers to determining the precise physical parameters to plug in to the simulation. This involves measuring: • Mass • Centre of mass • Motor resistance (viscous resistance) Once a model is built, we can test the model by comparing simulations and real world. System Identification can also be done automatically. Domain Randomisation is another method that improves the robustness of the policy.


e automatically. Domain Randomisation is another method that improves the robustness of the policy. - Trains efficiency for robustness - Need careful tuning for the range of domain randomisation Can we use a small amount of data to mitigate these limitations of DR? Domain Adaptation: Create a set of physical parameters µ, which would be randomised in DR, and encode it in z, a latent encoding of µ. Then condition the policy π(a|s, z) on z. When the policy is deployed to the real world, we freeze the policy weights π and search z to find the optimal policy. z is a low-dimensional vector so it can be optimised by simple optimisation algorithms e.g. Bayesian optimisation. Limitations: - The latent space may not contain the optimal version of the real world - Policy is not updated; Performance


nt space may not contain the optimal version of the real world - Policy is not updated; Performance does not necessarily improve - Adaption is slow (requires a few episodes) 47


13 META RL Rapid Motor Adaptation (RMA) uses an adaptation network which is called 10 times per second to assess the environment. This allows the robot to adapt to a variety of environments previously unseen in simulation. Does sim-to-real solve everything? • Some physical phenomena are difficult to model • Impossible to capture the diversity of real-world scenarios The complete picture of sim-to-real is that simulation and real-world are complementary methods. 13 Meta RL Announcements • 05-17: HW3 due, HW4 out • 05-24: Project Milestone due Multi-Task Learning is to solve multiple tasks and minimise a loss function of the form min θ T X i=1 Li(θ, Di) Transfer Learning refers to solving target task T2 after solving source task T1 by transferring knowledge from T1.


ng refers to solving target task T2 after solving source task T1 by transferring knowledge from T1. Meta-Learning refers to based on data from T1, . . . , Tn to solve a new task Tnew. In all settings, the tasks must share the same structure. Examples of meta-learning tasks: • A manipulator which can move objects according to only a few examples shown • A chat bot which adapts to user preferences In Meta supervised learning, the model is fed in training data (Di, x i) and tested with a new set of training data (D, x). In meta reinforcement learning, the model is fed (D, s) pairs and outputs action a. The caveat of meta reinforcement learning is that an additional component of data collection is involved. For example, a maze navigation model could be trained to quickly adapt to a new maze.


is involved. For example, a maze navigation model could be trained to quickly adapt to a new maze. There are two variants of Meta RL: • Episodic Variant: The data D are complete trajectories • Online Variant: The data (D, s, a) represent the next step action the model needs to take. 48


13 META RL Figure 13.1: Black-Box Meta-RL Model 13.1 Black-Box Meta-RL Methods In Black-Box methods, the model fθ(D, s) is a LSTM, NTM, or CNN. The training/support set is fed in as the beginning of the sequence and s, the query, is attached at the end. The actions at−1 does not need to be passed in since it is encoded in the state. This method is different from RL with a recurrent policy, since 1. Reward is passed as input 2. Hidden state maintained Algorithm: Training a Black-Box Meta-RL Model Training 1. Sample task Ti 2. Roll-out πθ(a|s, Dtr i ) for N episodes, under dynamics pi(s′|s, a) and reward ri(s, a). 3. Add to training buffer D 4. Optimise max θ ETi X t E(s,a)∼Ti,πθ[r(st, at)] Test 1. Sample new task T 2.


to training buffer D 4. Optimise max θ ETi X t E(s,a)∼Ti,πθ[r(st, at)] Test 1. Sample new task T 2. Roll-out policy π(a|s, Dtr j ) for at most N episodes + General and Expressive + A variety of design choices in architecture - Hard to optimise ~ Inherits sample efficiency from outer RL optimiser 49


13 META RL Figure 13.2: Multi-task Policy with experience as task classifier 13.2 Optimisation-Based Meta-RL Methods Fine-Tuning refers to learning from pre-trained parameters by repeating ϕ ←θ −α∇θL(θ, Dtr) Pre-trained parameters In meta-training, optimisations can be nested in each other to adapt to different tasks min θ X t L(θ −α∇θL(θ, Dtr t ), Dtr i ) There are two disadvantages to this method. One is that the gradient descent process is costly, so the inner gradient descent may be executed with only a single step. The other is that this requires evaluation of second order derivatives. This method is Model-Agnostic Meta-Learning (MAML). In optimisation-based meta RL, there are two loops of optimisation which MAML can go in. Algorithm: MAML Training 1. Sample task Ti 2.


e are two loops of optimisation which MAML can go in. Algorithm: MAML Training 1. Sample task Ti 2. Collect Dtr i by rolling out πθ 3. Inner loop adaptation: ϕi ←θ + α∇θJi(θ) 4. Collect Dts i by rolling out πϕi 5. Outer loop update: θ ←θ + β P task i ∇θJi(θ) Testing 50


14 META RL: LEARNING TO EXPLORE 1. Sample task T 2. Collect Dtr by rolling out πθ 3. Inner loop adaptation: ϕi ←θ + α∇θJ(θ) 14 Meta RL: Learning to Explore Announcements • 05-17: HW3 due, HW4 out • 05-24: Project Milestone due 14.1 Learning to Explore Suppose we wish to train an agent which learns to explore. We can directly optimise for end-to-end exploration and exploitation w.r.t. reward: + Simple + Leads to optimal strategy in principle - Challenging optimisation when exploration is hard. Example episodes during meta-training: • The agent could go to the goal by pure random chance. Training on this episode would reinforce this opportunistic behaviour and discourage generalisation. - Gets positive reward for current task but Dtrain i would not be different than for any other task • The


positive reward for current task but Dtrain i would not be different than for any other task • The agent goes to the wrong path and then the goal. ~ Provides signal on suboptimal exploration/exploitation strategy • The agent looks at the instruction and then follows the instruction + Optimal behaviour - Agent will not get extra rewards from this behaviour End-to-end approach for training is hard due to a chicken and egg problem: Execution (learning to cook) and Exploration (learning to find ingredients) must be simultaneously learned and learning either of them have low reward on their own. To mitigate these issues, we can use alternative exploration strategies 51


14 META RL: LEARNING TO EXPLORE Figure 14.1: Multi-task Policy with experience as task classifier • Posterior sampling/Thompson sampling (PEARL. Rakelly, Zhou, Quillen, Finn, Levine. ICML ’19) 1. Learn distribution over latent task variable p(z), q(z|Dtrain) and corresponding task policies π(a|s, z) 2. Sample z from current posterior and sample from policy π(a|s, z) - Can learn suboptimal exploration strategies - If each task takes a long time to complete the procedure will be extremely slow • Use intrinsic rewards (MAME. Gurumurthy, Kumar, Sycara. CoRL ’19) • Task dynamics and reward prediction (MetaCURE. Zhang, Wang, Hu, Chen, Fan, Zhang. ’20) Explicitly seek out states which inform about the dynamics. 1. Train dynamics-reward model f(s, r|s, a, Dtrain) 2.


out states which inform about the dynamics. 1. Train dynamics-reward model f(s, r|s, a, Dtrain) 2. Collect Dtrain so model is accurate. - Lots of distractors of complex high-dimensional state dynamics Overall, alternative exploration strategies are + Easy to optimise + Many based on principle strategies - Suboptimal by arbitrarily large amount in some environments 14.2 Decoupled Exploration and Exploitation Another solution is to extend the MetaCURE method. We label each training task with a unique id µ and trains an exploration policy to identify this id. Meta-train: • Execution policy πexec(a|s, µ) 52


14 META RL: LEARNING TO EXPLORE • Exploration policy πexp and task-identification model q(µi|Dtrain) The exploration policy generates Dtrain such that q maps this trajectory to the task identifier µ. Meta-test: 1. Rollout exploration policy πexp D 2. Predict task id µ ←q(·|D) 3. Rollout πexec(a|s, µ) Question: How is the exploration policy supervised? We train it using the goodness of deriving µ from D ∼πexp as the reward for πexp. The reward is a type of information gain. An informal way to state the reward is rt := accuracy of q(µ|τ1:t) −accuracy of q(µ|τ1:t−1) There is one more ingredient to the algorithm. We can add a information bottleneck which maps from µ (task id) to a latent embedding z.


algorithm. We can add a information bottleneck which maps from µ (task id) to a latent embedding z. Then we learn to explore by recovering this information Excursion: Information Bottleneck One way to add a bottleneck is to add noise in the representation ¯z := z+ϵ, ϵ ∼N(0, 1). We can either add this at train time or test time: + Will discard information - If done at test time, may discard useful information - If done at train time, the model can scale up the magnitude of z to bypass the bottleneck. We can: 1. Add gaussian noise during training 2. Prevent the model from increasing magnitude In Variational Information Bottleneck, we add noise before passing representation to the next layer: ¯z := z + ϵ. Then we add a loss term Ltrain + ∥z∥2.


efore passing representation to the next layer: ¯z := z + ϵ. Then we add a loss term Ltrain + ∥z∥2. This is equivalent to DKL(F(z|µi)∥N(0, 1)) We train the execution policy and encoder f(z|µ) using Eπexec[r] −DKL(f(z|µ)∥N(0, 1)) Here the execution policy is modified as πexec(a|s, z). 53


15 RESET-FREE RL [Fish] This is the Decoupled Reward-free ExplorAtion and Execution in Meta-Reinforcement Learning method (DREAM) (Informal) Theoretical Analysis shows 1. DREAM objective is consistent with end-to-end optimisation (under mild assumptions), and hence it can theoretically achieve optimum 2. Consider a bandit like setting with |A| arms In Task-i, arm i yields reward. In all Tasks, arm 0 reveals the rewarding arm. • RL2 requires O(|A|2 log |A|) samples for meta-optimisation • DREAM requires O(|A| log |A|) samples for meta-optimisation Decoupled Exploration and Exploitation: + Leads to optimal strategy in principle + Easy to optimise in practice - Requires task identifier 15 Reset-Free RL Guest Lecture by Archit Sharma Consider an RL problem with data s0, a0, . . . , sH.


5 Reset-Free RL Guest Lecture by Archit Sharma Consider an RL problem with data s0, a0, . . . , sH. At some point the agent needs to start from the beginning s′ 0. In simulated environment such as gym, we can use the handy reset function to reset the environment to the initial state, whereas a robot trained in the real world will have to have its state reset by a human operator. Autonomous RL refers to agents operating or training with minimal human supervision. We can evaluate autonomous RL using Deployed Policy Evaluation and Continuous Policy Evaluation. 15.1 Forward-Backward RL and MEDAL What happens when the episode length is increased? The deployed policy evaluation degrades a lot. In episodic learning, we always start from the initial state distribution and always reset to the initi


episodic learning, we always start from the initial state distribution and always reset to the initial state. In non-episodic learning, a few challenges arise 1. Exploration can cause the agent to drift far away. 2. The state distribution collapses since the agent learns to exploit one particular trajectory. 54


15 RESET-FREE RL We can learn a policy which resets the distribution back to the initial state distribution. This is not universally applicable due to irreversible processes in the real world. This is Forward-Backward RL (not a particular name in the literature). Assume we have a forward and backward reward functions rf(s, a), rb(s, a) where one reaches the goal state distribution ρg and one reaches the initial state distribution ρ0. We learn a forward policy πf and a backward policy πb. During rollout, we rollout H steps for πf and H steps for πb. - Requires an additional rb(s, a) + Simple to train. Can we do better than this? From the perspective of πf, it trains from the initial state ρ0 every time.


we do better than this? From the perspective of πf, it trains from the initial state ρ0 every time. What if the forward policy can practice from the easier states? This idea is closely related to curriculum learning. + Success from easier states can make it faster to learn from harder states. In comparison to episodic learning where the initial state distribution is fixed, the backward policy πb controls the initial state distribution. The key insight is to initialise forward policy πf at states an optimal policy would visit ρ∗. The key insight is to train πb to match expert states ρ∗by minimising DJS(ρb(s)∥ρ∗(s)). The problem is that we don’t have either distributions. Rolling out πb is approximately sampling ρb.


he problem is that we don’t have either distributions. Rolling out πb is approximately sampling ρb. We train πb by first training a classifier as a reward function c(s) → ( +1 s ∈demos −1 s ∼ρb(·) Now we can train policies πf, πb πf = arg max E " ∞ X t=0 γtrf(st, at) # πb = arg max −E " ∞ X t=0 γt log(1 −c(st+1)) # The classifier c(s) is c(s) = ρ∗(s) ρb(s) + ρ∗(s) ρb(s) ρ∗(s) = 1 −c(s) c(s) which can be used as the training objective: DKL(ρb(s)∥ρ∗(s)) = Z ρb(s) log ρb(s) ρ∗(s) ds = Z ρb(s) log 1 −c(s) c(s) ds 55


15 RESET-FREE RL Question: What if the initial classifier is suboptimal? The classifier in real cases pick up very quickly in comparison of the policy. The problem is this it he policy cannot catch up and reach the expert distribution. This is the Matching Expert Distributions for Autonomous Learning (MEDAL) algorithm - Requires expert demos - Adversarial training can be tricky + Prevents the agent from drifting away Challenge: We don’t have a reward function rf(s, a) for the forward policy. The solution is to use classifier-based rewards again. 15.2 QWALE/single-life RL In Single-Life RL, the agent has one life to complete some task given previous training data. The task has hurdles that have never been presented to the agent before.


en previous training data. The task has hurdles that have never been presented to the agent before. Deploying a policy and even fine-tuning online does not help the agent recover from a bad state. This problem is because of bias towards prior data. In Q-Weighted Adversarial Learning (QWALE), we train the agent to stay close to states visited in previous data. A technique to reach a state distribution is to use a reward classifier. c(s) → ( +1 s ∈prior data −1 s ∈online data Then we weigh all prior states when training the classifier by exp Q(s, a). QWALE trains policy π with the reward r(s′) −log(1 −c(s′)). Note: The slides aren’t available at the time of writing this and I can’t keep up with the speed of slide changing! Question: What if the state is continuous?


is and I can’t keep up with the speed of slide changing! Question: What if the state is continuous? The trained classifier c(s) will smooth out the states. Question: Why not build the environment in simulation and train in such an environment? It would be difficult to collect enough data to create a simulator which has high fidelity dynamics. We have not covered • How to handle irreversible states 56


16 HIERARCHICAL RL AND SKILL DISCOVERY [RL So far] • Autonomous agents are taking over the web Question: Can we have parallel reward functions? Can we have multiple goals/rewards? Combine reward functions? 16 Hierarchical RL and Skill Discovery Announcements • 05-24: Project Milestone due • 05-31: HW4 due so far, we have seen online, off-policy, and offline reinforcement learning. What if we want to have the robot autonomously discover interesting behaviours? Some research from biology shows that autonomous skill discovery is possible. A reason for skill discovery is that coming up with tasks is tricky. Example Suppose there is a robot in front of a table and it has a gripper. What are some of the tasks we can come up? e.g.


s a robot in front of a table and it has a gripper. What are some of the tasks we can come up? e.g. lifting an object, placing an object, pushing an object Hierarchical RL refers to performing tasks at various levels of abstractions. Excursion: Entropy Consider a distribution p(x). The entropy is defined as H p(x) := −Ex∼p(x)[log p(x)]. For one x, −log p(x) is higher if p(x) is lower. In one sense the entropy tells us how broad p(x) is. The KL Divergence is a type of “distance” (not really a metric since it’s asymmetric) between two distributions: DKL(q∥p) = Eq log q(x) log p(x)  = Eq log q(x) −Eq log p(x) = −Eq log p(x) −H q(x) The Mutual Information I(x; y) of a joint distribution is I(x; y) = DKL(p(x, y)∥p(x)p(y)) = E(x,y)∼p(x,y)  log p(x, y) p(x)p(y)  Mutual information represents


; y) = DKL(p(x, y)∥p(x)p(y)) = E(x,y)∼p(x,y)  log p(x, y) p(x)p(y)  Mutual information represents the amount of information one distribution has for the other. We have I(x, y) = H(p(y)) −H(p(y|x)) 57


16 HIERARCHICAL RL AND SKILL DISCOVERY x p(x) q(x) Figure 16.1: The minimiser of −Eq log p(x) compared to the minimiser of DKL(q∥p) 16.1 Skill Discovery The empowerment (defined by Polani et al.) is I(st+1; at) := H(st+1) −H(st+1|at) which represents the amount of control at has over the environment of the next state. In Soft Q-Learning, the objective is replaced by X t E(st,at)∼Q[r(st, at) + H(at|st)] and the policy by softmax Algorithm: Fitted Q-iteration 1. Collect dataset {(si, ai, s′ i, ri)} using some policy, where s′ i is the next state. 2. yi ←r(si, ai) + γ softmaxa′ i Qϕ(s′ i, a′ i) (Bellman Equation) 3. ϕ ←arg minϕ 1 2 P i ∥Qϕ(si, ai) −yi∥2 4. Repeat (2) – (3) 5. π(a|s) ∝exp A(st, a) In a multi-task setting with policy π(a|s, z) where z is the task index, we cannot just use MaxEn


a) In a multi-task setting with policy π(a|s, z) where z is the task index, we cannot just use MaxEnt RL for two reasons: 1. Action entropy is not the same as state entropy Agent can take different actions but land in similar states 2. MaxEnt policies are stochastic but not always controllable. Intuitively, we want low density for a fixed z and high diversity across z’s. 58


18 CONNECTING THE DOTS We would like a diversity-promoting reward function. π(a|s, z) := arg max π X z Es∼π(s|z)[r(s, z)] we can define r(s, z) := log p(z|s) This has a connection to mutual information I(z, s) = H(z) −H(z|s) Maximised by using uniform prior p(z) Minimised by maximising log p(z|s) 16.2 Slightly Different Mutual Information The problem with the above algorithm is that the learned skill may not be very useful or easy to use. max I(s′, z|s) = max(H(s′|s) −H(s′|s, z)) We are learning a skill-dynamics model q(s′|s, z) compared to conventional global dynamics p(s′|s, a). Skills are optimized specifically to make skill-dynamics easier to model. This is Dynamics-Aware Discovery of Skills. We can use skill-dynamics for model-based planning.


el. This is Dynamics-Aware Discovery of Skills. We can use skill-dynamics for model-based planning. Based on the z fed into the state-dynamics model q, we can predict which state the model will be in. Tasks can be learned zero-shot. 16.3 Hierarchical RL 17 RL in the Real World: From Chip Design to LLMs RL for Chip Design and LLMs (presented by Anna Goldie) Announcements • 05-31: HW4 due • 06-07: Poster due 18 Connecting the dots Announcements • 06-07: Poster due • 06-12: Project report due Recurring themes: 59


18 CONNECTING THE DOTS • Efficient learning requires controlling distribution shift • Learned functions can be exploited when optimised against • Trade-off between computational and data efficiency Challenges: • Core algorithms – Data/Computational Efficiency – Stability – Offline workflow • Assumptions – Formulating the problems in the context of MDPs – Is MDP even the right problem formulation? 18.1 Research Lightning Talks 1. Joey Hejna: Distance weighted supervised learning 2. Annie Xie: In-Context decision making (DPT) Question: Does the context get the entire history as input? If so how does it handle very long horizon tasks? 3. Q-Transformer: Scalable Offline Reinforcement Learning: 4. Paul Wohlhart: DeepRL@Scale 5. AlpacaFarm 60


Reinforcement Learning from Human Feedback A short introduction to RLHF and post-training focused on language models. Nathan Lambert 13 May 2026 Abstract Reinforcement learning from human feedback (RLHF) has become a crucial tool to build the latest machine learning systems at scale. The field grew around the core methods of RLHF into today’s broader suite of post-training techniques. In this book, we give a comprehensive introduction to the core methods for post-training models for people with some level of quantitative background, organized around the canonical RLHF recipe. The book starts with what RLHF does and why it was created, with seminal technical milestones in its young history and a primer on reinforcement learning context needed to understand the book.


in its young history and a primer on reinforcement learning context needed to understand the book. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, on-policy distillation, and direct alignment algorithms. The book also discusses broader topics, such as the origins of RLHF – both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. The book concludes with advanced topics – understudied or emerging research questions in synthetic data, tool-use, character training, and evaluation – and open questions for the field.


in synthetic data, tool-use, character training, and evaluation – and open questions for the field. The book is released with a variety of companion resources, including a codebase, a library to compare model completions from within post-training stages, and an educational course, to be a one-stop shop for learning all foundational concepts for post-training language models.


Contents 1 Introduction 7 1.1 RLHF in Three Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 What Does RLHF Do? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 Walkthrough of an RLHF Recipe . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.4 An Intuition for Post-Training . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.5 How We Got Here . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.6 Scope of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.6.1 Chapter Summaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.6.2 Target Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.6.3 How to Use This Book . . . . . . . . . . . . . . . . . . . . . .


. . . . . . . . . . . . . 16 1.6.3 How to Use This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.6.4 About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.7 Future of RLHF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2 A Tiny History of RLHF 18 2.1 Origins to 2018: RL on Preferences . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2 2019 to 2022: RL from Human Preferences on Language Models . . . . . . . 19 2.3 2023 to the Present: The ChatGPT Era . . . . . . . . . . . . . . . . . . . . . 20 3 Training Overview 21 3.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1.1 A Simple Example: The Thermostat . . . . . . . . . . . . . . . . . . .


. . . . . . . . . . 21 3.1.1 A Simple Example: The Thermostat . . . . . . . . . . . . . . . . . . . 22 3.1.2 Classic RL Example: CartPole . . . . . . . . . . . . . . . . . . . . . . 23 3.1.3 Manipulating the Standard RL Setup . . . . . . . . . . . . . . . . . . 24 3.1.4 Fine-Tuning and Regularization . . . . . . . . . . . . . . . . . . . . . . 25 3.1.5 Optimization Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.1.6 Subtle Advantages of RL in Post-Training Language Models . . . . . . 27 3.2 Canonical Training Recipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2.1 InstructGPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2.2 Tülu 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2.3 DeepSeek R1 . . .


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2.3 DeepSeek R1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4 Instruction Fine-Tuning 31 4.1 Chat Templates and the Structure of Instructions . . . . . . . . . . . . . . . . 31 4.2 Best Practices for Instruction Tuning . . . . . . . . . . . . . . . . . . . . . . . 34 4.3 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 5 Reward Modeling 36 5.1 Training a Bradley-Terry Reward Model . . . . . . . . . . . . . . . . . . . . . 36 5.2 The Default Reward Model Architecture . . . . . . . . . . . . . . . . . . . . . 39 5.3 Implementation Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.4 Reward Model Variants . . . . . . . . . . . . . . . .


. . . . . . . . . . . . . . . . . . . 39 5.4 Reward Model Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.4.1 Preference Margin Loss . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.4.2 Balancing Multiple Comparisons Per Prompt . . . . . . . . . . . . . . 41 5.4.3 K-Wise Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.5 Outcome Reward Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.6 Process Reward Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.7 Comparing Reward Model Types (and Value Functions) . . . . . . . . . . . . 47 rlhfbook.com 2


5.7.1 Inference Across Reward Model Types . . . . . . . . . . . . . . . . . . 48 5.8 Generative Reward Modeling (a.k.a. LLM-as-a-judge) . . . . . . . . . . . . . 49 5.9 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.10 Suggested Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 6 Reinforcement Learning 52 6.1 The Role of Reinforcement Learning in RLHF . . . . . . . . . . . . . . . . . . 52 6.2 Policy Gradient Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 6.2.1 Deriving the Policy Gradient . . . . . . . . . . . . . . . . . . . . . . . 55 6.2.2 Vanilla Policy Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.2.3 REINFORCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . . . . . 57 6.2.3 REINFORCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 6.2.4 REINFORCE Leave One Out (RLOO) . . . . . . . . . . . . . . . . . . 59 6.2.5 Proximal Policy Optimization (PPO) . . . . . . . . . . . . . . . . . . 60 6.2.6 Understanding the PPO Objective . . . . . . . . . . . . . . . . . . . . 62 6.2.7 Value Functions and PPO . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.2.8 Group Relative Policy Optimization (GRPO) . . . . . . . . . . . . . . 67 6.2.9 Group Sequence Policy Optimization (GSPO) . . . . . . . . . . . . . . 69 6.2.10 Clipped Importance Sampling Policy Optimization (CISPO) . . . . . 71 6.2.11 Comparing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 6.3 Implementation . . . . . . . . . . . . . . . . . .


. . . . . . . . . . . . . . . . . . . . . 72 6.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.3.1 Policy-Gradient Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.3.2 Loss Aggregation Tradeoffs . . . . . . . . . . . . . . . . . . . . . . . . 75 6.3.3 Asynchronous RL Systems . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.3.4 Truncated Importance Sampling . . . . . . . . . . . . . . . . . . . . . 81 6.3.5 Example: PPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.3.6 Example: GRPO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 6.4 Auxiliary Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.4.1 Generalized Advantage Estimation (GAE) . . . . . . . . . . . .


. . . . . . . . . . . . . . 86 6.4.1 Generalized Advantage Estimation (GAE) . . . . . . . . . . . . . . . . 86 6.4.2 Double Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.4.3 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.5 Suggested Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7 Reasoning and Inference-Time Scaling 91 7.1 The Role of RLVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.2 The Origins of New Reasoning Models . . . . . . . . . . . . . . . . . . . . . . 95 7.2.1 Why Does RL Work Now? . . . . . . . . . . . . . . . . . . . . . . . . . 95 7.2.2 RL Training vs. Inference-Time Scaling . . . . . . . . . . . . . . . . . 95 7.2.3 The Future (Beyond Reasoning) of RLVR .


nce-Time Scaling . . . . . . . . . . . . . . . . . 95 7.2.3 The Future (Beyond Reasoning) of RLVR . . . . . . . . . . . . . . . . 96 7.3 Understanding Reasoning Training Methods . . . . . . . . . . . . . . . . . . . 96 7.3.1 Reasoning Research Before OpenAI o1 or DeepSeek R1 . . . . . . . . 97 7.3.2 Early Reasoning Models . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.3.3 Common Practices in Training Reasoning Models . . . . . . . . . . . . 99 7.4 Looking Ahead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 8 Direct-Alignment Algorithms 103 8.1 Direct Preference Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8.1.1 How DPO Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8.1.2 DPO Derivation . . . . . . . . . .


. . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8.1.2 DPO Derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 rlhfbook.com 3


8.2 Numerical Concerns, Weaknesses, and Alternatives . . . . . . . . . . . . . . . 110 8.3 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 8.4 DAAs with Synthetic Preference Data . . . . . . . . . . . . . . . . . . . . . . 112 8.5 DAAs vs. RL: Online vs. Offline Data . . . . . . . . . . . . . . . . . . . . . . 113 8.6 Suggested Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 9 Rejection Sampling 115 9.1 Training Process, Step by Step . . . . . . . . . . . . . . . . . . . . . . . . . . 115 9.1.1 Generating Completions . . . . . . . . . . . . . . . . . . . . . . . . . . 116 9.1.2 Scoring Completions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 9.1.3 Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . . . . . . . . . . . 116 9.1.3 Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 9.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 9.3 Related: Best-of-N Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 9.4 Suggested Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 10 The Nature of Preferences 122 10.1 When Preference Replaces Correctness . . . . . . . . . . . . . . . . . . . . . . 122 10.2 The Origins of RLHF and Preferences . . . . . . . . . . . . . . . . . . . . . . 123 10.3 Specifying Objectives: From Logic of Utility to Reward Functions . . . . . . . 123 10.4 Tools for Optimizing Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . 123 10.4 Tools for Optimizing Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 10.5 Complexity of Optimizing Preferences . . . . . . . . . . . . . . . . . . . . . . 126 11 Preference Data 128 11.1 Why We Need Preference Data . . . . . . . . . . . . . . . . . . . . . . . . . . 128 11.2 Collecting Preference Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 11.2.1 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 11.2.2 Rankings vs. Ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 11.2.3 Multiturn Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 11.2.4 Structured Preference Data . . . . . . . . . . . . . . . . . . . . . . . . 136 11.2.5 Sourcing and Contracts . . . . . . . . . .


. . . . . . . . . . . . . . . . . . . . . . . 136 11.2.5 Sourcing and Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . 137 11.3 Bias: Things to Watch Out For in Data Collection . . . . . . . . . . . . . . . 139 11.4 Open Questions in RLHF Preference Data . . . . . . . . . . . . . . . . . . . . 139 12 Synthetic Data 141 12.1 The Roles of Synthetic Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 12.2 Distillation with Synthetic Data . . . . . . . . . . . . . . . . . . . . . . . . . 142 12.3 The Path to On-Policy, Teacher-Student Distillation . . . . . . . . . . . . . . 143 12.3.1 Adapting Knowledge-Distillation for LMs . . . . . . . . . . . . . . . . 143 12.3.2 From Offline to On-Policy Distillation . . . . . . . . . . . . . . . . . .


. . . . . . . 143 12.3.2 From Offline to On-Policy Distillation . . . . . . . . . . . . . . . . . . 145 12.3.3 Modern OPD Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 12.4 AI Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 12.4.1 Balancing AI and Human Feedback Data . . . . . . . . . . . . . . . . 148 12.4.2 Building Specific LLMs for Judgment . . . . . . . . . . . . . . . . . . 149 12.5 Constitutional AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 12.5.1 Further Reading on CAI . . . . . . . . . . . . . . . . . . . . . . . . . . 151 12.6 Rubrics: Prompt-Specific AI Feedback for Training . . . . . . . . . . . . . . . 151 13 Tool Use and Function Calling 155 13.1 Tool-Use Overview . . . . . . . . . . .


. . . . . . . 151 13 Tool Use and Function Calling 155 13.1 Tool-Use Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 rlhfbook.com 4


13.2 Interweaving Tool Calls in Generation . . . . . . . . . . . . . . . . . . . . . . 157 13.3 Multistep Tool Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 13.4 Model Context Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 13.5 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 14 Over-Optimization 163 14.1 Qualitative Over-Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 163 14.1.1 Managing Proxy Objectives . . . . . . . . . . . . . . . . . . . . . . . . 163 14.1.2 Over-Refusal and “Too Much RLHF” . . . . . . . . . . . . . . . . . . 166 14.2 Quantitative Over-Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 167 14.3 Misalignment and the Role of RLHF . . . . . . . . . .


. . . . . . . . . . . . . . . . . . 167 14.3 Misalignment and the Role of RLHF . . . . . . . . . . . . . . . . . . . . . . . 169 15 Regularization 170 15.1 KL Divergence in RL Optimization . . . . . . . . . . . . . . . . . . . . . . . . 170 15.1.1 Reference Model to Generations . . . . . . . . . . . . . . . . . . . . . 171 15.1.2 Implementation Example . . . . . . . . . . . . . . . . . . . . . . . . . 171 15.2 Implicit Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 15.2.1 SFT Memorizes, RL Generalizes . . . . . . . . . . . . . . . . . . . . . 172 15.2.2 Retaining by Doing: On-Policy Data Mitigates Forgetting . . . . . . . 173 15.2.3 RL’s Razor: Why Online RL Forgets Less . . . . . . . . . . . . . . . . 176 15.3 Other Types of Regularization . . . . . . .


L Forgets Less . . . . . . . . . . . . . . . . 176 15.3 Other Types of Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . 177 15.3.1 Pretraining Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 15.3.2 Margin-Based Regularization . . . . . . . . . . . . . . . . . . . . . . . 178 16 Evaluation 179 16.1 Prompting Formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 16.1.1 Few-Shot Prompting and Log-Likelihood Scoring . . . . . . . . . . . . 180 16.1.2 Chain-of-Thought Prompting . . . . . . . . . . . . . . . . . . . . . . . 182 16.1.3 Zero-Shot Instruction Following . . . . . . . . . . . . . . . . . . . . . . 182 16.1.4 Reasoning-Era Evaluation Prompts . . . . . . . . . . . . . . . . . . . .


. . . . . . . . 182 16.1.4 Reasoning-Era Evaluation Prompts . . . . . . . . . . . . . . . . . . . . 183 16.2 Why Many External Evaluation Comparisons Are Unreliable . . . . . . . . . 183 16.3 How Labs Actually Use Evaluations Internally to Improve Models . . . . . . 186 16.4 Contamination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 16.5 Tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 17 Crafting Model Character and Products 188 17.1 Character Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 17.1.1 Persona Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 17.1.2 The Assistant Axis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


. . . . . . 190 17.1.2 The Assistant Axis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 17.1.3 Persona Subnetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 17.2 Model Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 17.3 Product Cycles and What’s Next for RLHF . . . . . . . . . . . . . . . . . . . 198 Bibliography 200 A Definitions 222 A.1 Language Modeling Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 A.2 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 A.3 Natural Language Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 rlhfbook.com 5


A.4 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 A.5 RLHF-Only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 A.6 Extended Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 B Beyond “Just Style” 226 B.1 The Chattiness Balance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 C Practical Issues 231 C.1 Compute Costs of Post-Training . . . . . . . . . . . . . . . . . . . . . . . . . 231 C.2 Evaluation Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 C.3 Managing Training Performance Variance . . . . . . . . . . . . . . . . . . . . 232 C.4 Identifying Bad Training Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 rlhfbook.com 6


1 Introduction Reinforcement learning from Human Feedback (RLHF) is a technique used to incorporate human information into AI systems. RLHF emerged primarily as a method to solve hardto-specify problems. With systems that are designed to be used by humans directly, such problems emerge all the time due to the often inexpressible nature of an individual’s preferences. This encompasses every domain of content and interaction with a digital system. RLHF’s early applications were often in control problems and other traditional domains for reinforcement learning (RL), where the goal is to optimize a specific behavior to solve a task. The core idea to start the field of RLHF was, “Can we solve hard problems only with basic preference signals guiding the optimization process?” RLHF became most kn


d problems only with basic preference signals guiding the optimization process?” RLHF became most known through the release of ChatGPT and the subsequent rapid development of large language models (LLMs) and other foundation models. 1.1 RLHF in Three Steps The basic pipeline for RLHF involves three steps. First, a language model that can follow user questions must be trained (see Chapter 4). Second, human preference data must be collected for the training of a reward model of human preferences (see Chapter 5). Finally, the language model can be optimized with an RL optimizer of choice, by sampling generations and rating them with respect to the reward model (see Chapters 3 and 6). This book details key decisions and basic implementation examples for each step in this process.


). This book details key decisions and basic implementation examples for each step in this process. RLHF has been applied to many domains successfully, with complexity increasing as the techniques have matured. Early breakthrough experiments with RLHF were applied to deep reinforcement learning [1], summarization [2], following instructions [3], parsing web information for question-answering [4], and “alignment” [5]. A summary of the early RLHF recipes is shown below in fig. 1. Figure 1: A rendition of the early, three stage RLHF process with SFT, a reward model, and then optimization. In modern language model training, RLHF is one component of post-training. Post-training is a more complete set of techniques and best practices to make language models more useful for downstream tasks [6].


set of techniques and best practices to make language models more useful for downstream tasks [6]. Post-training can be summarized as a many-stage training process using three optimization methods: 1. Instruction / Supervised Fine-tuning (IFT/SFT), where we teach formatting and form the base of instruction-following abilities. This is largely about learning features in rlhfbook.com 7


language. 2. Preference Fine-tuning (PreFT), where we align to human preferences via RLHF and related methods (and get a smaller bump in capabilities at the same time). This is largely about style of language and subtle human preferences that are hard to quantify. 3. Reinforcement Learning with Verifiable Rewards (RLVR), the newest type of posttraining that boosts performance on verifiable domains with more RL training. RLHF lives within and dominates the second area, preference fine-tuning, which has more complexity than instruction tuning because it often involves proxy reward models of the true object and noisier data. At the same time, RLHF is far more established than the other popular RL method for language models, reinforcement learning with verifiable rewards.


an the other popular RL method for language models, reinforcement learning with verifiable rewards. For that reason, this book focuses on preference learning, but in order to completely grasp the role of RLHF, one needs to use these other training stages, so they are also explained in detail. As we consider the space of options and attention on these methods for crafting models we collectively use extensively, RLHF colloquially is what led to modern post-training. RLHF was the technique that enabled the massive success of the release of ChatGPT, so early in 2023 RLHF encompassed much of the interest in the general field of post-training. RLHF is now just one piece of post-training, so in this book we map why there was so much attention on RLHF early on, and how other methods emerged to com


s book we map why there was so much attention on RLHF early on, and how other methods emerged to complement it. Training language models is a very complex process, often involving large technical teams of 10s to 100s of people and millions of dollars in data and compute costs. This book serves three purposes to enable readers to grasp how RLHF and related models are used to craft leading models. First, the book distills cutting-edge research often hidden within large technology companies into clear topics and trade-offs, so readers can understand how models are made. Second, the book will allow users to set up basic code examples to get their hands dirty on fine-tuning these models themselves.


sers to set up basic code examples to get their hands dirty on fine-tuning these models themselves. Finally, beyond teaching the techniques for doing RLHF, this book is designed to distill intuition as to why RLHF is crucial to modern AI models. Due to the complexity of RLHF and how the state-of-the-art is often too complex to be done alone, this book focuses on enabling readers so they have the tools needed to get jobs and start research projects in the area. Others will just enjoy precisely understanding the inner workings of the technology that is the focus of countless discussions across the globe. A book that lays out an exact recipe of how to do RLHF for a specific need is impossible, which is why there is a large industry of companies providing RLHF training and related methods as a


which is why there is a large industry of companies providing RLHF training and related methods as a service worth millions of dollars. Still, rapid progress in AI systems means the readers who are equipped with this knowledge can approach more and more of their own training over time. 1.2 What Does RLHF Do? The biggest question around RLHF, yet one that is still hard to answer, is “What does RLHF training offer models?” In recent years, language models have shifted from academic experiments studied in the purview of benchmarks to general-purpose technology. RLHF is at the core of this transition. To start, let us consider how a language model at the end of autoregressive pretraining answers a query. A strong model, Llama 3.1 405B Base, will continue from “The president rlhfbook.com 8


of the united states in 2006 was”: Response: George W. Bush, the governor of Florida in 2006 was Jeb Bush, and John McCain was an Arizona senator in 2006 - who later lost to obama. September 1 – U.S. President Bush signs an executive order to provide more options for faith-based organizations when delivering social services that are paid for with federal funds. January 1 – The current deadline set by United States Congress in the Unlawful Internet Gambling Enforcement Act, aka UIGEA. search: Amendments to the 1961 International Convention for the Protection of New Varieties of Plants require plant breeders’ rights include farmer’s privilege. 2009 was a common year starting on Thursday of the Gregorian calendar, the 2009th year of the Common Era (ce) and Anno Domini (ad) designations, the 9


Gregorian calendar, the 2009th year of the Common Era (ce) and Anno Domini (ad) designations, the 9th year of the 3rd millennium, the 9th year of the 21st century, and the 10th and last year of the 2000s decade. What is clear is that this model is completing the sentence and adding other common internet metadata. In contrast, a model trained with RLHF (and other post-training tools) would answer the question very differently. Asking Tülu 3 405B the same question “The president of the united states in 2006 was,” it says concisely: Response: George W. Bush was the president of the United States in 2006. He served two terms in office, from January 20, 2001, to January 20, 2009. This change in style is downstream of the post-training methods described in this book.


y 20, 2009. This change in style is downstream of the post-training methods described in this book. The most compelling view of how RLHF works is to consider how style applies to interactions you have with language models. The style, closely related to tone or format, of information presented is crucial to how it is learned. Style is often used to describe the manner of the model. To see the difference, consider two example responses to the following question: User: Help, I have to give a presentation tomorrow and I’m panicking. A blunt, low-empathy style: Response: You’ll be fine. Practice your slides a few times, get some sleep, and stop overthinking it. Just present what you know and move on.