How frontier models train on outcomes in 2026

Sergio Paniego

sergiopaniego

Follow

This article is complementary material for Class 3: Reinforcement Learning of the Training an Agent series Ben and I are doing, where we train a coding agent step by step. This article does not explain how GRPO works. The class does that, with the three experiments. Here we show where the same ideas appear in the reports of frontier labs. Previous classes: SFT on traces and distillation.

It also pairs with Distillation in 2026 (so far), where we did the same exercise for distillation.

Something I really like when I study a subject is understanding its history, how it reached the point where it is today. Having at least a glimpse of where things come from adds a lot to my own understanding. This piece tries to do that for RL in post-training, and to keep it accessible. The goal is not to be exhaustive. It is to situate ourselves and set a common ground.

In the live session, Ben explained in depth how GRPO works, and then we went through three experiments: one with a toy reward, one with a verifiable reward, and one with a reward we broke on purpose. In that last run, the Trackio dashboard said the training was going great while the model got worse at coding. We tell the full story of that run in the recording.

The class is built on the same ideas frontier labs use. This article collects the receipts. We went through the public reports of the main labs (all of them linked in the table at the end) and pulled out what they say, in their own words. Three patterns stand out. RL appears in every one of them. Among the labs that describe their algorithm, very few run the original GRPO without changes. And the newest reports are already moving RL from single answers into environments (that is what Class 4 will be about).

How we got here (briefly)

RL in language models did not start with reasoning. The first wave was RLHF, the recipe behind InstructGPT. A reward model learned human preferences and PPO trained the model against it, sampling fresh answers at every step. DPO later removed that sampling loop and learned directly from fixed preference pairs. Ben compared both with GRPO in the class, so we will not repeat that here. This article is about the wave that came next, where the reward stopped being a preference and became a result.

OpenAI's o1 announcement (September 2024) put RL at the center of reasoning: "Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process." Performance "consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)". Beyond that, OpenAI gives no details about the algorithm. If you were around in late 2024, you may remember the test-time compute wave. I explored that half back then, in a Cookbook recipe. This article is about the training half.

Two months later, AI2's Tulu 3 gave the open community the term everyone now uses: "a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)". The reward here is not a model trained on human preferences. It is a check you can execute.

Then DeepSeek-R1 (January 2025) showed it working in the open: "the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL)". If you watched the live, this is the paper behind the "aha moment" Ben brought up, the point in training where the model "naturally learns to solve reasoning tasks with more thinking time". R1 trains with GRPO, the algorithm we walked through in the class. GRPO comes from an earlier paper, DeepSeekMath, and that is the real reference if you want to go to the source.

The pattern was not limited to reasoning models. Meta's Llama 4 (2025) already treated online RL as the core of its pipeline, alternating between "training the model and then using it to continually filter and retain only medium-to-hard difficulty prompts".

Rewards you can execute

In the class we rewarded code by executing its tests. Across these reports, that same pattern appears again and again, from a 7B model to frontier systems.

Qwen3 describes its Reasoning RL stage as GRPO over curated "query-verifier pairs". Every prompt comes with its own check. That is exactly what we did with MBPP and its asserts in our second experiment. Xiaomi's MiMo-7B curated "a dataset of 130K verifiable mathematics and programming problems for reinforcement learning" and added "a test-difficulty-driven code-reward scheme to alleviate sparse-reward issues". We hit this exact problem in our own runs. A sparse reward gives the model very little to learn from. MiMo's answer is to make the code reward sensitive to test difficulty, so each attempt carries more information than a single pass or fail.

Mistral's Magistral puts it directly: "We limit ourselves to problems with verifiable solutions; we use mathematical problems whose solution is a numerical answer or expression, and code problems with associated tests". Google's Gemma 3 lists, inside its RL finetuning phase, "code execution feedback [...], and ground-truth rewards for solving math problems [...]".

A case worth a closer look is Moonshot's Kimi K2. For everything a check cannot reach, it uses "a framework for general reinforcement learning from self-critic feedbacks", but the critic itself is anchored to the verifiable side: "On-policy rollouts generated from verifiable-reward prompts are used to continuously update the critic, a crucial step that distills objective performance signals from RLVR directly into its evaluation model". So even when the reward comes from a model, that model is trained on checks you can execute.

GLM-5 combines all of these signals in one system: "a hybrid reward system that integrates three complementary types of reward signals: rule-based reward functions, outcome reward models (ORMs), and generative reward models (GRMs)".

Our one-line summary in the class was that if you can execute a check, you have a reward function. In 2026, that idea sits at the center of how these models are trained.

GRPO became a family

The second pattern is that, among the labs that describe their algorithm in detail, GRPO is the usual starting point and each one changes something about it. We taught the original GRPO in the class for the same reason. It is where the family starts, it is the most widely used base, and it is the simplest version to learn. Once you know it, the variants are small changes on top.

ByteDance's DAPO lists its four changes: "1. Clip-Higher, which promotes the diversity of the system and avoids entropy collapse; 2. Dynamic Sampling, which improves training efficiency and stability; 3. Token-Level Policy Gradient Loss, which is critical in long-CoT RL scenarios; 4. Overlong Reward Shaping, which reduces reward noise and stabilizes training.". Dr. GRPO targets "an optimization bias in Group Relative Policy Optimization (GRPO), which artificially increases response length (especially for incorrect outputs)". Qwen's GSPO applies the correction at the sequence level instead of the token level: "GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization", and the paper credits it with having "contributed to the remarkable improvements in the latest Qwen3 models". MiniMax's CISPO "clips importance sampling weights rather than token updates".

Magistral runs its own list of changes on top of GRPO, and one of them maps straight to the class. They "filter out all groups with zero advantage when forming training batches". A group with zero advantage teaches the model nothing. It is the frac_reward_zero_std metric you can check in our Trackio dashboards, and here it is as a training rule at a frontier lab.

You can also see this evolution in TRL itself. As of August 2026, the loss_type of GRPOTrainer defaults to "dapo", not to the original GRPO loss. The docstring for the "grpo" option says it plainly: "Not recommended due to length bias—this approach tends to prefer shorter completions with positive advantages and longer ones with negative advantages." And GSPO is available through another option, importance_sampling_level="sequence". The defaults of the library already reflect this evolution. If you want to try any of these variants, the TRL paper index lists the exact configuration to reproduce each one (DAPO, Dr. GRPO, GSPO, CISPO and more).

The reward is becoming an environment

The newest reports go one step further. The model no longer just writes an answer. It acts (opens files, runs commands, uses tools) and the reward arrives at the end, based on what happened. Liquid AI describes it plainly for LFM2.5-2.6B, its small model built for on-device agents: "Each rollout runs in a dedicated sandbox with its own runtime. We optimize with GRPO, using an outcome-based reward that combines an LLM-as-a-judge rubric, programmatic checks, and a hard safety gate." Cursor trained its Composer model this way too, "specialized for software engineering through reinforcement learning (RL) in a diverse range of development environments". Kimi K3, GLM-5, Nemotron 3 Ultra and MiniMax describe versions of the same idea. What an environment is exactly, and how you train a model inside one, is the topic of Class 4, so we will stop here on purpose.

Where RL and distillation meet

The line between distillation, SFT and RL is getting blurry, and the reports show a clear pattern behind that blur. Labs use RL to build a capability in specialized models, and then use distillation to bring it into one final model.

DeepSeek-V4 trains domain experts with GRPO, "guided by reward models tailored to specific success criteria", and then merges them: "a single unified model is trained through on-policy distillation, wherein the unified model acts as the student learning to optimize the reverse KL loss with teacher models". In MiMo-V2-Flash, the RL does not happen where you would expect. Its "domain-specialized teachers (e.g., trained via large-scale reinforcement learning) provide dense and token-level reward" to the student through its Multi-Teacher On-Policy Distillation. The report attributes the RL to the teachers, not to the student itself. And LFM2.5-2.6B runs the same two-step at a much smaller size: each domain teacher gets "reinforcement learning with verifiable rewards (RLVR)" and the student absorbs them on-policy before its own agentic RL stage.

First, RL finds the behavior using outcome rewards. Then, distillation transfers it to the final model with a dense signal. So the techniques from Class 2 and Class 3 are not alternatives. In these pipelines, they are two consecutive stages.

All the reports at a glance

That was a lot of models and quotes in a row. So we do not get lost, here is everything again in one table, with all the links.

What this means for the concepts from the class

Almost everything we ran at 0.6B scale appears in these reports, just much bigger. Even entropy collapse, the problem DAPO's Clip-Higher exists to avoid, is visible in our broken run (the curves are public in the dashboards below). One difference matters here, though. Dr. GRPO fixes a length bias that comes from the training objective. Our broken run grew long for a simpler reason. Its reward directly paid it to produce more. The curves look similar, but the cause and the fix are different.

This is a review of public training descriptions, not of everything these labs do internally. It describes the landscape as of August 2026, and several of the reports were published in 2025. Gemma 4 is excluded because its report does not describe its post-training. Anthropic and Cohere were not investigated in this pass. OpenAI appears only through its o1 post.

If you want a broader discussion of how these recipes evolved, Nathan Lambert and Finbarr Timbers reviewed the frontier post-training recipes on Interconnects, covering many of the same reports from a different angle.

If you want to see these mechanics with your own eyes, the class materials are open: the recording, the slides, the scripts (three GRPO runs, one of them hacked on purpose), and the training dashboards. Class 4, on RL environments, is coming. Follow Ben and me to catch it live.