# verl Examples

This directory hosts curated, minimal-dependency examples that drive
`verl.trainer.main_ppo` with the current Hydra API. Algorithm-specific
extensions, research baselines, and non-trivial entry points live under
`recipe/`; prefer that directory if you need a custom loss or reward beyond
what these examples show.

## Conventions

All run scripts follow the same shape:

1. Canonical filename:

   ```
   run_<model>_<train-backend>.sh
   ```

   - `<model>`: a single canonical size per model family. E.g.
     `qwen3_8b`, `qwen3_30b_a3b`, `qwen3_235b_a22b`, `qwen3_vl_8b`,
     `deepseek_v3`, `mimo_7b`, `nemotron_nano_v3`.
   - `<train-backend>`: one of `fsdp`, `fsdp2`, `megatron`, `mindspeed`,
     `automodel`, or `veomni`. **Must be the last underscore-separated
     token before `.sh`**.

   Nothing follows `<train-backend>`. Per-example *features* — including
   the inference backend (`vllm`/`sglang`/`trtllm`), the platform
   (`DEVICE=gpu|npu`), the GPU machine type (`MACHINE=gb200`/`b200`/
   `blackwell`), Liger kernel, LoRA, FP8 quantization, sequence parallel
   size, server vs sync rollout, etc. — do **not** show up in filenames.
   They are exposed as env-var toggles inside the one canonical script.
   Do not add `_npu`, `_amd`, `_vllm`, `_sglang`, `_trtllm`, or `_fp8`
   script variants. For example, `sft/gsm8k/run_qwen2_5_0_5b_fsdp.sh`
   covers plain SFT and its `USE_LIGER=1`, `SP_SIZE=2`, `USE_PEFT=1`
   variants via env vars; `grpo_trainer/run_qwen3_8b_fsdp.sh` covers vLLM,
   SGLang, and TRT-LLM rollouts, CUDA/NPU platforms, and `MACHINE=gb200`
   (Blackwell) via toggles.

   This naming rule is enforced by the `check-example-naming` pre-commit
   hook (see `tests/special_sanity/check_example_naming.py`).

2. Every script exposes its important knobs in a user-adjustable region near
   the top. Derived defaults and device/backend-specific details belong below
   the "no user adjustment needed below" / "derived defaults" boundary.
   Use uppercase env vars for user-facing knobs, e.g.

   ```bash
   # ---- user-adjustable ----
   DEVICE=${DEVICE:-gpu}
   INFER_BACKEND=${INFER_BACKEND:-vllm}
   MODEL_PATH=${MODEL_PATH:-Qwen/Qwen3-8B}
   NNODES=${NNODES:-1}
   NDEVICES_PER_NODE=${NDEVICES_PER_NODE:-}
   TRAIN_BATCH_SIZE=${TRAIN_BATCH_SIZE:-1024}
   ROLLOUT_TP=${ROLLOUT_TP:-2}
   ROLLOUT_N=${ROLLOUT_N:-5}
   PROJECT_NAME=${PROJECT_NAME:-verl_grpo_gsm8k_math}
   EXPERIMENT_NAME=${EXPERIMENT_NAME:-qwen3_8b_grpo_vllm_fsdp}
   # ---- end user-adjustable ----
   ...
   ```

   Override anything you care about on the command line:

   ```bash
   DEVICE=npu MODEL_PATH=/my/local/qwen3-8b NDEVICES_PER_NODE=4 bash examples/grpo_trainer/run_qwen3_8b_fsdp.sh
   ```

   GPU and NPU paths should share the same `PROJECT_NAME` /
   `EXPERIMENT_NAME` form. Do not append `_npu` to project or experiment
   names just because `DEVICE=npu` is selected.

3. Defaults (unless a directory explicitly documents otherwise):

   - `data.train_files` + `data.val_files` = GSM8K + MATH for text LLMs
     (`geo3k` for vision, `dapo-math-17k` / `aime-2024` for scale-demo 235B /
     671B scripts).
   - `actor_rollout_ref.actor.use_dynamic_bsz=True`
   - `trainer.balance_batch=True`
   - `trainer.logger=["console","wandb"]`.

4. No deprecated Hydra knobs:

   - `ppo_megatron_trainer.yaml` → use `actor_rollout_ref.actor.model_engine=megatron`.
   - `actor_rollout_ref.rollout.mode=async` → removed; async rollout is no
     longer selected this way in example scripts.
   - `actor_rollout_ref.hybrid_engine=True` → removed; the trainer now
     enforces the supported hybrid-engine path internally.
   - `ppo_micro_batch_size` / `log_prob_micro_batch_size` → use the
     `_per_gpu` suffix.
   - `data.val_batch_size` → removed.
   - Top-level `reward_model.*` → use `reward_model.reward_model.*` /
     `reward.reward_model.*` as applicable.
   - `actor.ulysses_sequence_parallel_size` → use
     `actor_rollout_ref.actor.fsdp_config.ulysses_sequence_parallel_size`.

## Directory layout

### Algorithm trainers

Each directory holds a canonical recipe for one training algorithm. Adding a
new algorithm? If it needs its own trainer entry point or reward code, put it
under `recipe/` instead.

| Dir                                | Algorithm                      | `algorithm.adv_estimator` / `policy_loss.loss_mode` |
|------------------------------------|--------------------------------|-----------------------------------------------------|
| `ppo_trainer/`                     | PPO (actor + critic)           | `adv_estimator=gae`                                 |
| `grpo_trainer/`                    | GRPO                           | `adv_estimator=grpo`                                |
| `rloo_trainer/`                    | RLOO                           | `adv_estimator=rloo`                                |
| `remax_trainer/`                   | ReMax                          | `adv_estimator=remax`                               |
| `reinforce_plus_plus_trainer/`     | REINFORCE++ / baseline         | `adv_estimator=reinforce_plus_plus[_baseline]`      |
| `cispo_trainer/`                   | CISPO                          | `loss_mode=cispo`                                   |
| `dppo_trainer/`                    | DPPO (TV / KL variants)        | `loss_mode=dppo_tv \| dppo_kl`                      |
| `gdpo_trainer/`                    | GDPO                           | `adv_estimator=gdpo`                                |
| `gmpo_trainer/`                    | GMPO                           | `loss_mode=geo_mean`                                |
| `gpg_trainer/`                     | GPG                            | `adv_estimator=gpg`, `loss_mode=gpg`                |
| `gspo_trainer/`                    | GSPO                           | `loss_mode=gspo`                                    |
| `sapo_trainer/`                    | SAPO                           | `loss_mode=sapo`                                    |
| `otb_trainer/`                     | OTB                            | `adv_estimator=optimal_token_baseline`              |
| `mtp_trainer/`                     | DAPO + MTP (MiMo-7B)           | `adv_estimator=grpo`, MTP flags                     |
| `on_policy_distillation_trainer/`  | on-policy distillation         | GRPO + distillation loss                            |
| `flowgrpo_trainer/`                | Flow-GRPO (diffusion)          | image-gen specific                                  |

### Feature / infra

| Dir                  | Purpose                                                                                  |
|----------------------|------------------------------------------------------------------------------------------|
| `tuning/`            | LoRA (`tuning/lora/`) and scaling demos (`tuning/scaling/`).                             |
| `profile/`           | NPU profiler / torch-memory profiler runs.                                               |
| `sft/`               | Supervised fine-tuning examples.                                                         |
| `generation/`        | Rollout-only inference launches.                                                         |
| `vllm_omni/`         | vLLM omni backend examples.                                                              |
| `data_preprocess/`   | Scripts that produce the `$HOME/data/<dataset>/*.parquet` layout the run scripts expect. |
| `prefix_grouper/`    | Prefix-grouped rollout examples.                                                         |
| `rollout_correction/`| Rollout correction examples.                                                             |
| `router_replay/`     | Router replay examples.                                                                  |
| `tutorial/`          | Tutorials and cluster launchers (`ray/`, `slurm/`, `skypilot/`, `agent_loop_get_started/`). |

### Where are the algorithm research variants?

`recipe/` — e.g. `recipe/dapo`, `recipe/prime`, `recipe/retool`,
`recipe/r1`, `recipe/spin`, `recipe/gvpo`, `recipe/flowrl`, ...
They ship their own trainer entry points and reward code.