GRPO is mostly a systems problem

The GRPO config is small. The hard part is keeping rollout workers, trainers, rewards, and policy versions in the same story.

I came to GRPO through the systems side. At the API level, the method can look almost disappointingly small: choose a model, write a reward function, set num_generations, and let the trainer run. That is a useful interface, but it also hides the part where the job stops behaving like ordinary supervised fine-tuning. A GRPO run has to generate its own data, score that data, compute policy and reference log-probs, keep an inference runtime close to the trainer, and decide how fresh a rollout has to be before it is still worth training on.

The algorithm is not the hard part to explain. GRPO removes PPO's learned value model and estimates the baseline from a group of completions sampled for the same prompt. That does simplify the model census, and it is one reason GRPO is pleasant to work with, but the removed work does not disappear so much as move into rollout generation. Instead of training a second model to estimate values, you ask the current policy to produce many more completions, all the time, and you have to make sure those completions were produced by the policy version your loss thinks produced them.

This post is about that runtime. The math is included because it gives names to the moving pieces, but the useful questions are mostly operational ones: where do rewards run, what has to live in memory, where does generation happen, how do weights get synchronized, what changes when rollout is async, and which metrics tell you that the run is quietly becoming less on-policy than you intended.

At the level this post cares about, the loop looks more like this than like a single trainer call:

prompts -> rollout engine -> completions -> reward code or environments -> trainer update -> weight sync back to rollout engine

The arrows are where most of the practical questions live. Generation has to be fast enough, rewards have to be reliable enough, and weight synchronization has to keep the policy used for sampling close to the policy being optimized.

The part that fits in your head

For each prompt, GRPO samples a group of GG completions from the current policy and assigns a reward to each completion. The baseline for a completion is not predicted by a value model; it is computed from the rewards of the other completions in the same group:

$$A_i=\frac{r_i-\mathrm{mean}(r_1,\dots ,r_G)}{\mathrm{std}(r_1,\dots ,r_G)}A\_i\; =\; \backslash frac\{r\_i\; -\; \backslash operatorname\{mean\}(r\_1,\; \backslash dots,\; r\_G)\}\{\backslash operatorname\{std\}(r\_1,\; \backslash dots,\; r\_G)\}$$

The advantage $A_i$ is a sequence-level number. During the policy update it is broadcast over the tokens of that completion, while the probability ratio is still computed at the token level. With a clipped policy-gradient objective and a KL term against a frozen reference model, the loss has the usual PPO-like shape:

$$\mathcal{L}(\theta )=-\mathbb{E}[\min \text{\hspace{0.17em}⁣}({\rho }_{i,t}{A}_i,\mathrm{clip}({\rho }_{i,t},1-ϵ,1+ϵ)A_i\left)\right]+\beta D_{\mathrm{K}\mathrm{L}}\text{\hspace{0.17em}⁣}\left({\pi }_{\theta }\mathrm{\parallel }{\pi }_{\mathrm{r}\mathrm{e}\mathrm{f}}\right)\backslash mathcal\{L\}(\backslash theta)\; =\; -\backslash ,\backslash mathbb\{E\}\backslash left[\; \backslash min\backslash !\backslash big(\backslash rho\_\{i,t\}\backslash ,\; A\_i,\backslash \; \backslash operatorname\{clip\}(\backslash rho\_\{i,t\},\backslash ,\; 1-\backslash epsilon,\backslash ,\; 1+\backslash epsilon)\backslash ,\; A\_i\backslash big)\; \backslash right]\; +\; \backslash beta\backslash ,\; D\_\{\backslash mathrm\{KL\}\}\backslash !\backslash left(\backslash pi\_\backslash theta\; \backslash ,\backslash |\backslash ,\; \backslash pi\_\{\backslash mathrm\{ref\}\}\backslash right)$$

The intuition is simple enough: completions that score better than their siblings become more likely, and completions that score worse become less likely. The group gives you a local comparison without training a value model to estimate what reward should have been expected.

That trade is the source of most of the engineering that follows. A learned baseline has been replaced by repeated sampling, and repeated sampling is an inference workload. In SFT, examples are already sitting in the dataset. In GRPO, the examples for the next optimizer step have to be generated before the optimizer can do anything useful.

Rewards are part of the system

Classic RLHF usually adds a reward model to the loop. You train another transformer from preference data, freeze it, and use its scalar output as the reward. GRPO does not require rewards to come from a model. For tasks with a checkable answer, the reward can be a program: compare a math answer to ground truth, run generated code against tests, or parse a tool call and compare the arguments field by field.

That is the RLVR setting, reinforcement learning with verifiable rewards, and it fits naturally with GRPO because the trainer only needs scalar scores for the sampled completions. A tool-calling reward can be a normal Python function:

def tool_call_reward(completions, answers=None, **kwargs):
    rewards = []
    for completion, truth in zip(completions, answers):
        pred = normalize(parse_tool_calls(completion))
        rewards.append(1.0 if pred and pred == normalize(truth) else 0.0)
    return rewards

The small signature matters more than it first appears. The model sees the prompt, while the reward function can receive hidden dataset columns such as answers through the trainer's plumbing. That gives you a clean separation between what the policy is allowed to condition on and what the grader is allowed to inspect. It also means the reward parser is not just defensive code around the task; it is part of the task definition. If the parser accepts malformed JSON, ignores trailing text, or repairs arguments, then the model is being trained under those rules.

The reward function has to be robust in the boring sense too. During training it will see truncated generations, half-open tags, repeated fragments, and plain prose where a tool call should have been. Those completions should receive low scores, not exceptions. A parser crash is not a bad sample; it is a training-job crash, and on a multi-GPU run the difference is expensive.

There is a useful way to generalize this boundary. A single-turn reward function is the degenerate case of an environment: the policy emits one completion, receives one score, and the episode ends. Once the task becomes agentic, the environment is no longer a cheap function inside the trainer. The policy may call a tool, observe the result, call another tool, hit a timeout, and only receive reward at the end. At that point GRPO the objective may look similar, but the runtime now includes environment workers, sandboxes, browsers, or terminals. The reward path becomes a systems component with its own latency, failure modes, and scaling limits.

What GRPO removes, and what remains

It is helpful to count models. A PPO-style RLHF setup can involve a trainable policy, a trainable value model, a frozen reward model, and a frozen reference model. GRPO with verifiable rewards can reduce that to a trainable policy and a frozen reference. The value model became group statistics, and the reward model became code.

That is a real memory improvement, especially for full fine-tuning. A 7B model in bf16 is around 14 GB for weights alone, but the trainable policy costs much more once you include the fp32 master weights, Adam moments, and gradients. A rough mixed-precision Adam budget is about 16 bytes per parameter, or around 112 GB for a 7B policy before activations. The frozen reference is smaller because it is weights only, but it still has to be represented somewhere unless you set the KL coefficient to zero or use an adapter setup where the base model can serve as the reference.

The reference model is not just there for tradition. Verifiable rewards are narrow by design. An exact-match reward can say whether a tool call matched the target, but it does not say whether the model preserved general language quality, instruction following, or the rest of the distribution you cared about before RL. The KL term is the anchor to the pre-RL model. The coefficient $\beta$ controls how hard you pull back toward that anchor.

The reference also adds compute to each step. To compute the KL term, you need reference log-probs for the sampled completions. You also need policy log-probs for the update, and depending on the exact recipe you may keep old-policy log-probs when reusing generated data for multiple gradient updates. The result is that an RL step can include several forward passes over generated tokens before the backward pass is counted. This is one reason GRPO steps can feel so much heavier than SFT steps even when the training code looks compact.

LoRA gives a useful exception. If only the adapter is trained, disabling the adapter recovers the base model, so the base weights can act as the reference without keeping a second full copy. Full fine-tuning does not have that shortcut; the trained policy and the reference are genuinely different model states.

Generation becomes the workload

The main operational difference from SFT is that GRPO has to create its own training data. In SFT, the dataloader can prefetch static examples while the GPU alternates between forward and backward passes. In GRPO, each optimizer step starts with autoregressive generation from the current policy. If num_generations = 8, every prompt in the batch turns into eight completions before reward computation and training can begin.

The naive implementation is to call model.generate() inside the training process. That can be acceptable for small experiments, but it is usually not the shape you want to scale. A model sharded with ZeRO-3 or FSDP is arranged for training, not for high-throughput decoding. Serving runtimes such as vLLM are built around continuous batching, paged KV cache, request scheduling, and the other details that make autoregressive generation efficient.

For that reason, many GRPO stacks put an inference engine next to the trainer and ask it to produce rollouts. This introduces a second live copy of the policy. The trainer updates one copy; the inference engine samples from another. After an optimizer step, the trainer's policy has moved, while the rollout engine still has the previous weights until synchronization happens.

That synchronization is part of correctness, not just performance. The loss uses importance ratios, and those ratios are meaningful only if you know which policy generated the sampled tokens. If the rollout engine keeps sampling from weights that are too old, the run becomes increasingly off-policy while still looking superficially healthy. Nothing has to crash for the model to learn less than it should.

For a 7B bf16 model, pushing fresh weights to the rollout engine means moving about 14 GB. Within a node, fast interconnect can make that cheap enough that you do not think about it much at first. Across nodes, the same operation can become a visible part of step time. A large part of the engineering in modern open-source RL stacks is concerned with this boundary: how to keep the rollout policy fresh without spending all of the wall clock on weight transfer.

There is also a small but common accounting mistake. GRPO batch size is usually easier to reason about in completions, not prompts. If eight GPUs each process two sequences with eight gradient accumulation steps, that is 128 completions per optimizer step. With num_generations = 8, those 128 completions correspond to only 16 unique prompts. The group has to fit inside the optimization step because the baseline is computed from the group. Mixing prompt counts and completion counts is an easy way to misread the effective batch size.

Where rollout workers live

Once generation is on the critical path, placement becomes a first-order choice. In a colocated setup, the inference engine runs beside the trainer, often on the same node and sometimes on the same GPUs. The main advantage is that the system stays relatively compact. Weight sync is local, there is less RPC machinery, and policy freshness is easier to understand. The tradeoff is memory pressure: optimizer state, activations, the reference model, rollout KV cache, and generated batches all compete for the same HBM.

The TRL GRPO docs expose this directly. Colocated rollout is the default vLLM mode once use_vllm=True is enabled:

from trl import GRPOConfig

training_args = GRPOConfig(
    use_vllm=True,  # vllm_mode="colocate" by default
    vllm_gpu_memory_utilization=0.45,
)

Colocation is the mode I would usually try first when the model fits. It keeps the number of moving parts low while you are still validating the reward, batching, collapse metrics, and basic learning behavior. If generation is not yet starving the trainer, a separate rollout cluster is often just a more complicated way to debug the same training recipe.

Non-colocated rollout moves generation to separate GPUs or separate nodes. This lets you scale generation independently from training, which matters when completions are long or when environment interaction dominates latency. It also lets CPU-heavy environment workers live closer to the rollout service while the trainer GPUs focus on backprop.

In that shape, the rollout service is a separate process, usually on different GPUs from the trainer:

trl vllm-serve --model Qwen/Qwen2.5-72B --tensor_parallel_size 8

and the trainer is configured to talk to it:

from trl import GRPOConfig

training_args = GRPOConfig(
    use_vllm=True,
    vllm_mode="server",
    vllm_server_host="10.0.0.42",
)

The extra throughput comes with a distributed-systems bill. You now need queues, RPCs, versioned weights, retry behavior, and a policy for what happens when rollout is faster than training or training is faster than rollout. None of that is conceptually exotic, but it is now part of the RL job. A useful rule is to colocate until generation either starves the trainer or no longer fits comfortably beside it, and only then split rollout into its own service.

Async rollout and trajectory age

The simplest GRPO loop is synchronous: sync the rollout weights, generate a batch of completions, score them, train on them, then repeat. This is the cleanest loop to reason about because the rollout policy is fresh at sampling time. It also leaves performance on the table when generation or environment steps are slow, because trainer and rollout workers spend part of their time waiting for each other.

Async rollout tries to recover that idle time. Rollout workers keep generating and place completed trajectories into a queue, while the trainer consumes from that queue whenever it is ready. This can be the right shape for long completions and agentic tasks, but it changes the freshness assumption. A trajectory may have been generated by a policy version that is no longer the current trained policy by the time the trainer uses it.

The useful metric here is trajectory age. If a rollout worker sampled from policy version 100 and the trainer has advanced to version 103 by the time that trajectory is used, the trajectory has age three. Age zero means fresh on-policy data; larger ages are not automatically invalid, but they mean the update is correcting from an older behavior policy toward a newer trained policy.

rollout worker syncs weights:     policy v100
trainer optimizer steps:          v100 -> v101 -> v102 -> v103
trajectory enters training at:                         v103

trajectory age = 103 - 100 = 3 optimizer steps

Old log-probs still matter because they describe the policy that actually sampled the tokens and give the importance ratio its denominator. The problem is that as the trained policy moves farther away from the rollout policy, more tokens hit clipping, KL can rise, and the batch begins to describe a policy the trainer has already partly left behind. In that regime, async throughput can look good while optimization quality quietly degrades.

This is why I would not trust an async GRPO run without freshness metrics. At a minimum I would log policy version, queue depth, trajectory age percentiles, clip ratio, and KL. Those metrics tell you whether the queue is giving the trainer useful work or merely feeding it stale trajectories.

The usual control is simple: set a maximum age, and drop or downweight trajectories that exceed it. That gives up some of the throughput win, but it keeps the optimization closer to the objective you meant to run.

Failure modes worth logging early

The first failure mode is advantage collapse. GRPO learns from differences inside each group, so a group where every completion receives the same reward does not provide a useful gradient; all wrong and all correct are both uninformative for the group-relative update.

This makes task difficulty more important than it may appear from the config. If the base model already solves the task, most groups may come back with reward 1.0 across the board. If the base model cannot get partial success, groups may come back all zero. The useful region is the middle, where sampled completions disagree and the group baseline has something to compare. Measuring the base model pass rate before starting RL is one of the cheapest ways to avoid wasting a run. Dynamic sampling methods such as DAPO are motivated by the same issue: filter out zero-variance groups and spend the batch on examples that can produce learning signal.

The second failure mode is generation collapse. If the policy is pushed too hard, completions can become long, repetitive, or unparsable. When most samples hit the length cap and the reward parser gives zero, the run may not recover because the group no longer contains useful ordering information.

The metric pattern is simple enough to turn into a guardrail:

class StopOnCollapse(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        collapsed = (
            (logs.get("completions/clipped_ratio") or 0) >= 0.9
            and (logs.get("rewards/tool_call_reward/mean") or 0) <= 0
        )
        if collapsed:
            control.should_training_stop = True

The preventive measures are ordinary but important: use a conservative learning rate, clip gradients, keep a nonzero KL unless you have a good reason not to, set sane length limits, and make sure the reward does not accidentally make degenerate verbosity profitable. Modern GRPO recipes often add token-level normalization and batch-level reward scaling as well. These are small config choices, but they reduce two common sources of distorted updates: overweighting short completions and letting near-unanimous groups turn tiny reward differences into large advantages.

The small config sits on a larger machine

After all of this, the code you write can still be quite small:

from trl import GRPOConfig, GRPOTrainer

config = GRPOConfig(
    num_generations=8,
    max_completion_length=128,
    beta=0.1,
    loss_type="dapo",
    scale_rewards="batch",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    bf16=True,
)

trainer = GRPOTrainer(
    model="HuggingFaceTB/SmolLM3-3B",
    args=config,
    reward_funcs=[tool_call_reward, format_reward],
    train_dataset=dataset,
    callbacks=[StopOnCollapse()],
)
trainer.train()

That compact surface is a good thing because it lets you express the experiment without manually wiring every worker and tensor movement yourself. The danger is only forgetting what the surface stands on: rollout generation, reward parsing, reference forward passes, policy log-probs, optimizer state, group bookkeeping, optional inference servers, and weight synchronization.

The run that made these details concrete for me is the Hugging Face SageMaker GRPO example. It trains SmolLM3-3B with GRPO on Salesforce/xlam-function-calling-60k, using exact-match and format rewards, DeepSpeed ZeRO-3, and collapse metrics from the training logs. The SageMaker part is not essential to the argument. What is useful is that the abstract pieces show up as concrete choices: dataset columns passed to rewards, group size, KL, sharding, generation length, and the metrics that tell you whether the run is still healthy.

Conclusions

The main lesson is that GRPO simplifies one part of the RLHF stack while making another part more important. Removing the value model reduces model residency and removes a learned baseline, but the replacement baseline is built from sampled completions. That makes rollout generation, reward execution, policy synchronization, and freshness control central to the run.

The second lesson is that rollout architecture is not just a scaling detail. Colocated rollout keeps the system easier to reason about and tends to make freshness simpler, while non-colocated rollout can buy throughput, especially with long generations or slow environments, at the cost of queues and policy versions that have to be measured.

The third lesson is to be careful with async speedups. More generated samples are useful only if they are still close enough to the policy being trained. Trajectory age is the metric that makes that trade visible. Once you track it, GRPO starts to look less like a mysterious RL recipe and more like a familiar systems problem: keep the producer, consumer, and versioned state aligned well enough that the computation still means what you intended.