examples/gsm8k_grpo/README.md

# GSM8K + Plain GRPO Example

The minimum-viable end-to-end recipe a new user is most likely to want
from a GRPO framework: wire `ComposerReplicationTrainer` into a real
dataset (GSM8K) with a real verifiable reward (regex-extract `#### NUMBER`
and string-compare against gold) and run a couple of outer steps to
verify the training loop works.

## What this demonstrates

- `ComposerReplicationTrainer` with `alpha_sdpo=0` and `beta_replay=0`
  (plain GRPO — channels 2 and 3 disabled). This is the v0.1 recommended
  ablation baseline per `docs/USER_GUIDE.md` §8 Recipe A.
- A regex-based reward that returns `1.0` when the model's `#### NUMBER`
  line matches the gold answer, `0.0` otherwise. RLVR-style. No reward
  model.
- CPU-only execution. Slow but works without a GPU.

## Install

```bash
pip install -e ".[train]"
```

(Just `[train]` — no need for `[replay]`, `[replaysim]`, `[diloco]`,
`[serverless]`, `[prime-rl]`, or `[monarch]` for plain GRPO.)

## Run

```bash
python examples/gsm8k_grpo/run.py
```

Expected output: see `run.log`. ~60 seconds wall-clock on a modern CPU
for 2 outer steps with Qwen2.5-0.5B-Instruct + 100 GSM8K rows + 4
generations per prompt.

## What's missing (and why that's OK)

This example **does not** use the framework's novel channels (SDPO +
trace-replay DPO). For a 0.5B model on 100 prompts in 2 steps, plain
GRPO with a verifiable reward is the right baseline: simple, fast, and
the ablation point against which SDPO/replay-DPO improvements are
measured.

To extend this with SDPO, you'd need to:
1. Build a `data_collator` that produces `sdpo_loss_mask` +
   `ctx_teacher_input_ids` columns (the SDPO hint-conditioned context).
2. Set `alpha_sdpo > 0` in `ComposerReplicationTrainer.__init__`.

To extend with trace-replay DPO, you'd:
1. Run `composer_replication.teacher_replay.replay_trace` against your
   trace data + N teachers.
2. Convert teacher disagreement to DPO pairs via `extract_dpo_pairs`.
3. Optionally normalize via `composer_replication.replaysim.DJNormalizer`.
4. Build a `data_collator` that loads the DPO pairs into the batch.
5. Set `beta_replay > 0`.

A future `examples/gsm8k_grpo_with_sdpo/` will demonstrate (1) and (2)
end-to-end. As of Wave 15, the data-collator wiring for SDPO is documented
in `docs/USER_GUIDE.md` §6 but not yet shipped as a runnable example.

## Production scaling

For real runs:
- Replace `Qwen/Qwen2.5-0.5B-Instruct` with `Qwen/Qwen2.5-7B-Instruct`
  (or larger). Use `device_map="cuda"` and bf16.
- Increase `num_generations` to 8 or 16.
- Increase `max_completion_length` to 256-512.
- Train for 100+ steps (each step takes ~1 min on a single A100 for 7B).
- Add `vllm` or sglang for fast generation backend.

See `docs/INTEGRATION_RECIPES.md` Recipe A for the full TRL recipe.

## Cross-references

- `docs/USER_GUIDE.md` §8 — picking an RL backend
- `docs/INTEGRATION_RECIPES.md` Recipe A — TRL `GRPOTrainer` subclass
- `composer_replication/trainer/composer_trainer.py` — the
  `ComposerReplicationTrainer` source (read the `__init__` docstring for
  all channel-weight kwargs)