Alignment: SFT, RLHF, DPO & GRPO

Pretraining gives a model knowledge and the knack for continuing text. It does not give it manners — the instinct to follow instructions, be helpful, refuse the harmful, and prefer the better answer over the merely plausible one. Post-training installs all of that, and there are a few ways to do it: imitate good answers, or learn from which answers people preferred. Same model, same preferences, surprisingly different results.

1 Knowledge isn't manners

A freshly pretrained model is a strange thing to talk to. It knows an enormous amount, but its only skill is continuing text, so it'll happily complete your question with more questions, ramble, or echo the worst of its training data. It has no notion that it should be helpful, that it should follow an instruction, or that one answer might be better than another.

pretrained knows language SFT follows instructions ✓ preferred answer ✗ rejected answer aligned policy DPO · RLHF · GRPO
Alignment turns a raw next-token predictor into a helpful assistant in stages. Supervised fine-tuning teaches it to follow instructions; preference methods (DPO, RLHF, GRPO) then push it toward answers people prefer and away from ones they reject — learning from comparisons, not just single correct labels.

Post-training — alignment — is the stage that turns that raw model into something you'd want to use. The knowledge is already there; what's missing is a sense of what a good response looks like. And the cleanest signal for "good" turns out not to be a perfect example, but a comparison: this answer is better than that one. Most of modern alignment is about learning from those comparisons.

Alignment shapes behaviour, not knowledge. Pretraining decides what the model knows; post-training decides how it acts on it. The same weights become a helpful assistant or stay an unruly text-completer depending entirely on this stage.

2 SFT: imitate the good answers

The first and simplest step is supervised fine-tuning. You collect a set of high-quality (instruction, ideal response) pairs — written by humans or distilled from a stronger model — and train the model to reproduce them, exactly the next-token objective from Chapter 8, now on curated data. This alone is transformative: it teaches the model the assistant format and basic helpfulness, and turns the text-completer into something that answers.

But SFT can only imitate. It learns from positive examples — "produce this" — and never from "this is worse than that." If your demonstration is merely good and a better answer exists, SFT has no way to find it; it concentrates the model's probability on the example it was shown, whatever its quality. To go further, you need a signal about relative quality.

SFT imitates; it can't compare. Trained only on demonstrations, the model's ceiling is the quality of those demonstrations. It will happily learn a mediocre answer if that's what it was shown, with no mechanism to prefer something better.

3 RLHF: a reward model, then RL

Reinforcement learning from human feedback brings in the comparison signal. You show people two responses and ask which they prefer; from thousands of these pairs you train a reward model that scores any response with a number predicting human preference. Then you use reinforcement learning — classically PPO — to update the policy so it produces responses the reward model rates highly, with a leash (a KL penalty) keeping it from drifting too far from the SFT model.

This is powerful: the model can now exceed its demonstrations, discovering high-reward responses no human wrote, because it's optimizing a score rather than copying examples. It's also a lot of moving parts — a separate reward model, an RL loop that's famously finicky to stabilize, and the risk of the policy learning to game the reward model rather than genuinely improve.

RLHF optimizes a score, so it can beat the demos. By chasing a reward instead of imitating, the policy explores beyond what it was shown. The cost is complexity — a reward model plus a temperamental RL loop — and the ever-present danger of reward hacking.

4 DPO: skip the reward model

Direct preference optimization is the insight that you don't actually need the reward model or the RL loop. The math of RLHF can be rearranged so that the optimal policy is expressed directly in terms of the preference data — which means you can train the policy on the preference pairs with a single, stable supervised-style loss. No reward model, no PPO, no sampling loop.

The DPO loss simply pushes up the probability of each chosen response relative to its rejected partner, anchored to the reference model by a strength knob. It's far easier to implement and to stabilize than RLHF, which is why it became the default for preference tuning in the open-source world. The trade is that it learns only from the comparisons you give it — it can't explore for new high-reward responses the way an RL loop can.

DPO is RLHF's objective without the machinery. Same goal — make preferred answers more likely — reached with one stable loss instead of a reward model and an RL loop. Simpler and steadier, at the cost of the RL family's ability to explore.

5 GRPO: group-relative, for reasoning

Group relative policy optimization, popularized by the DeepSeek reasoning models, is an RL method tuned for tasks where you can check the answer — math, code, anything with a verifier. For each prompt it samples a whole group of responses, scores them, and computes each one's advantage as its reward minus the group's average. Responses better than their peers are reinforced; worse-than-average ones are pushed down.

The clever part is that the group's own average serves as the baseline, so GRPO needs no separate value model — a big simplification over PPO. Paired with a verifier that gives a clean reward (the code passes its tests, or it doesn't), it lets a model bootstrap strong reasoning by practising against itself, which is much of how the recent reasoning models were trained.

Score a group, reinforce the above-average. Using the group mean as the baseline removes the value model and makes RL cheap and stable. With a verifiable reward, GRPO turns "generate and check" into a self-improvement loop — the engine of modern reasoning training.

6 Reading the playground

The same starting policy — a distribution over six candidate responses, each with a hidden true reward — is post-trained three ways with the real update rules: SFT's imitation gradient, the DPO preference loss, and GRPO's group-relative policy gradient. Watch where they end up.

▦

Four distributions: the reference policy and the result of SFT, DPO, and GRPO — over the same six responses, with their true rewards shown beneath.

▶

Step through training and watch each method's probability mass move — SFT toward its demonstration, DPO toward the preferred pairs, GRPO toward the highest reward.

⚖

The comparison table — what each method needs, and where they diverge.

The reading is the setup. The playground is the point.

Post-train the same policy 6 candidate responses · hidden true rewards
Reference anchor β: 0.3
Steps trained: 0
Step 1 · where each method takes the policy probability over the six responses

Reference (pretrained)

where we start — a default leaning on the safe response 0

SFT

DPO

GRPO

true reward of each response (hidden from SFT & DPO)
Step 2 · what each method needs, and learns
methodsignal it usesextra machinerycan exceed demos?stability
SFTgood examples (positives only)noneno — imitatesvery stable
RLHFhuman preference pairsreward model + PPO loopyesfinicky
DPOpreference pairsnone (one loss)limited to the pairsstable
GRPOa verifier / reward, per groupRL loop, no value modelyes — exploresstable-ish

SFT imitates a demonstration; DPO leans on which answers were preferred; the RL family (RLHF, GRPO) optimizes a reward and can discover the best response on its own.