research/04-verl-trl.md

# VeRL vs. HF TRL — Deep-Dive Comparison Report

> **Generated:** 2026-05-25  
> **Scope:** Post-training framework selection for a "take any HF model, RL post-train it" goal, with particular focus on agentic-coding use-cases.

---

## Table of Contents

1. [VeRL Deep-Dive](#1-verl-deep-dive)
2. [TRL Deep-Dive](#2-trl-deep-dive)
3. [Algorithm Zoo — Current State of RL for LLMs (Late 2025)](#3-algorithm-zoo)
4. [Comparison Matrix](#4-comparison-matrix)
5. [Recommendation](#5-recommendation)
6. [Sources](#6-sources)

---

## 1. VeRL Deep-Dive

### 1.1 Overview

VeRL (**Volcano Engine Reinforcement Learning**) is ByteDance's production-grade, open-source RL training library for LLMs. Released publicly in 2024, it is the framework that powered DeepSeek-R1-style large-scale RL post-training runs and Qwen RL post-training. The headline paper is *HybridFlow* (Sheng et al., 2025), which formalises the underlying architecture.

> **GitHub:** https://github.com/volcengine/verl  
> **Stars:** >10 k (as of mid-2025)

### 1.2 Architecture — HybridFlow

VeRL's core design principle is the **HybridFlow** programming model, which decouples the RL *control plane* from the *compute plane*:

- **Single-Controller Orchestration:** A central `RayPPOTrainer` (Ray-based) coordinates all distributed workers. The controller treats the cluster as a set of remote high-level operators, making it easy to compose new algorithms.
- **Computation-Data Decoupling:** Workers execute independently and exchange state via `DataProto` objects, making computation flow reusable across different RL algorithms without re-implementation.
- **3D-HybridEngine:** A single worker can switch between *training mode* and *inference/rollout mode*, eliminating redundant model copies. During PPO/GRPO, the Actor is used for both generation and gradient updates via efficient resharding (e.g., FSDP sharded ↔ vLLM TP). This is the key memory efficiency win.
- **Flexible Resource Allocation:** Models can be colocated on the same GPU set, placed on separate GPU sets, or run in a hybrid configuration, enabling optimal hardware utilisation at scale.

### 1.3 Training Backends

| Layer | Options |
|---|---|
| **Distributed training** | FSDP / FSDP2 (research-friendly), Megatron-LM v0.13.1+ (production scale), MindSpeed-LLM (Ascend NPU) |
| **Rollout / inference** | vLLM (≥0.8.3), SGLang (fully supported, multi-node), TensorRT-LLM, HF Transformers (debug only) |
| **Hardware** | NVIDIA H100/A100, AMD, Ascend 910 |
| **Orchestration** | Ray (required) |

**Key insight:** VeRL treats the training engine and rollout engine as separable components. The `3D-HybridEngine` handles weight resharding between FSDP sharding patterns (needed for training) and Tensor-Parallel patterns (needed for vLLM/SGLang generation), without maintaining duplicate model copies.

### 1.4 Algorithm Zoo in VeRL

VeRL ships first-class implementations of:

| Algorithm | Status | Notes |
|---|---|---|
| **PPO** | Stable | Actor + Critic + Reference + Reward model; full pipeline |
| **GRPO** | Stable | Critic-free; group-relative advantages |
| **DAPO** | Stable | Decoupled clip + dynamic sampling + token-level PG loss |
| **RLOO** | Stable | REINFORCE Leave-One-Out; no critic |
| **ReMax** | Stable | Greedy baseline; no critic |
| **REINFORCE++** | Stable | Batch-global baseline with clipping |
| **SPIN** | Stable | Self-play via online DPO loss |
| **SPPO** | Stable | Self-play preference optimisation |
| **GPG** | Stable | Policy gradient variant for math/reasoning |
| **OTB** | Stable | Optimal Token Baseline for fine-grained credit |
| **SAPO** | Community | Smoothing-based actor-policy optimisation |
| **GSPO** | Community | Grouped Soft Policy Optimisation (sequence-level) |
| **DPO / Online DPO** | Supported | Via SPIN / DAPO extensions |

### 1.5 Agentic / Tool-Calling RL

VeRL has **first-class agentic RL support**:

- **AsyncServer / AgentLoop architecture:** An `asyncio`-based co-routine mechanism separates the `AgentLoop` (client that drives multi-turn trajectories) from the `AsyncServer` (vLLM/SGLang inference backend). During tool-call waits (e.g., code execution), GPU compute is not blocked — other inflight requests continue.
- **SandboxFusionTool:** Built-in code-execution sandbox for agentic coding tasks; allows model → `<tool_call>` → sandbox response → next step trajectories with rewards assigned at trajectory end.
- **Multi-turn tokenisation:** Supported but noted as complex; naive concatenation of per-turn token IDs can introduce distribution drift between the rollout policy and training policy.

### 1.6 Scale

| Tested configuration | Notes |
|---|---|
| Up to **671B parameters** | Confirmed in production (DeepSeek-scale) |
| **Trillion-parameter** GRPO | 64 H800 GPUs; GRPO with Megatron-LM backend |
| **8× H100 benchmark** | DeepSeek-R1-Distill-Qwen-1.5B, 28k context, batch 128 per DP: step time ~363s; gen throughput measured per-GPU |

A third-party benchmark (RLinf docs, Aug 2025) running VeRL v0.5.0 on 8× H100s with a 1.5B model (context 28,672 tokens):

- **Generation time:** 260.9 s/step  
- **Training time:** 66.5 s/step  
- **Total step time:** 363.6 s/step

VeRL's Megatron-LM backend + SGLang rollout is the performance-optimal path for >70B models.

### 1.7 Real-World Usage

- **DeepSeek-R1 lineage** — The architecture is directly inspired by DeepSeek's internal RLVR pipeline.
- **Qwen RL post-training** — Qwen3 and DAPO paper both used VeRL.
- **DAPO paper** (ByteDance, 2025) — Trained Qwen2.5-72B with VeRL; achieved new AIME 2024 SOTA.
- **Multiple open reproductions** of DeepSeek-R1-Zero use VeRL as the training backend.

### 1.8 Strengths

1. **Best-in-class throughput at scale** — 3D-HybridEngine + vLLM/SGLang eliminates memory redundancy.
2. **Widest algorithm coverage** — PPO through the latest DAPO/GSPO/OTB variants all natively supported.
3. **Production proven** — Used at 671B scale with Megatron-LM.
4. **First-class agentic loops** — AsyncServer decouples GPU from tool-call latency.
5. **Hardware agnostic** — NVIDIA, AMD, Ascend.
6. **Flexible resource allocation** — Colocated, separated, or hybrid GPU pooling.

### 1.9 Weaknesses / Challenges

1. **Steep learning curve** — Ray orchestration, multiple backend configs, FSDP vs. Megatron choice; not a 3-line quickstart.
2. **Multi-turn tokenisation complexity** — Risk of subtle off-policy drift if multi-turn chat templates are not handled carefully; noted as an active known issue.
3. **Off-policy instability** — Rollout correction is provided but requires careful tuning; naive replay buffers can cause policy collapse.
4. **Heavyweight infrastructure** — Requires Ray cluster; not ideal for single-GPU or commodity 4-GPU experiments.
5. **Documentation gaps** — Community recipes exist but the core docs lag behind code velocity.

---

## 2. TRL Deep-Dive

### 2.1 Overview

TRL (**Transformer Reinforcement Learning**) is Hugging Face's mainstream post-training library, designed around the HF ecosystem (Accelerate, PEFT, Transformers, Datasets). The philosophy is *accessible post-training for any HF model*, favouring simplicity and developer ergonomics over raw throughput at frontier scale.

> **GitHub:** https://github.com/huggingface/trl  
> **Version milestone:** TRL v1 released March 2026  
> **Stars:** >14 k

### 2.2 Trainer Taxonomy

TRL organises trainers into four categories:

#### Supervised
| Trainer | Description |
|---|---|
| `SFTTrainer` | Instruction-tuning / supervised fine-tuning; supports packing, PEFT, VLMs |
| `RewardTrainer` | Train scalar reward models from preference data |
| `PRMTrainer` | Process Reward Model training (step-level rewards) |

#### Preference / Offline Alignment
| Trainer | Description |
|---|---|
| `DPOTrainer` | Direct Preference Optimisation; supports VLMs and tool-calling |
| `BCOTrainer` | Binary Classifier Optimisation |
| `CPOTrainer` | Contrastive Preference Optimisation |
| `KTOTrainer` | KTO (binary signal, no pairs) |
| `ORPOTrainer` | Odds-Ratio Preference Optimisation |
| `GKDTrainer` | Generalised Knowledge Distillation |
| `NashMDTrainer` | Nash Mirror Descent online preference |

#### Online RL
| Trainer | Description |
|---|---|
| `GRPOTrainer` | **Primary online RL trainer.** Group Relative Policy Optimisation; stable; VLM + agentic support |
| `RLOOTrainer` | REINFORCE Leave-One-Out; supports VLMs |
| `PPOTrainer` | Proximal Policy Optimisation; **experimental** (noted as incomplete) |
| `OnlineDPOTrainer` | Online DPO with LLM-as-judge; **experimental** |
| `XPOTrainer` | Exploratory DPO (experimental) |

#### Other
| Trainer | Description |
|---|---|
| `MiniLLMTrainer` | Reverse-KL distillation |

### 2.3 GRPOTrainer — Key Design

`GRPOTrainer` is TRL's workhorse for RLVR-style training:

- **No critic model** — group-relative advantages, matching GRPO semantics from DeepSeek-R1.
- **vLLM integration** — co-located vLLM for fast rollout generation (June 2025 update: "NO GPU left behind" co-located vLLM).
- **Liger kernel integration** — May 2025 update; significant memory/speed improvements for GRPO training step.
- **VLM support** — Vision-language models trainable with GRPO as of August 2025.
- **Agentic workflows** — `GRPOTrainer` supports multi-step agentic rollouts; `OpenEnv` integration (October 2025) provides tool/environment loop scaffolding.

### 2.4 Distributed Backends

TRL relies on **HF Accelerate** as the distribution abstraction:

| Backend | Support level |
|---|---|
| DeepSpeed ZeRO-1/2/3 | Stable |
| FSDP v1 + v2 | Stable |
| PEFT / LoRA / QLoRA | Native; enables large model training on fewer GPUs |
| vLLM (co-located) | Integrated for online RL trainers (GRPO, RLOO, PPO) |

### 2.5 Scale Ceiling

TRL was designed for the **commodity to mid-scale cluster** range:

- Single GPU (with QLoRA) up through multi-node clusters.
- No native Megatron-LM tensor/pipeline parallelism — limits scaling for >70B full-parameter runs.
- No 3D-HybridEngine; actor model is held fully in training-mode sharding at all times, meaning rollout generation is bottlenecked by the training sharding strategy.
- Practical ceiling: **8–32 GPU clusters** for full-parameter runs of 7–70B models; beyond that, FSDP ZeRO-3 sharding overhead becomes limiting.

### 2.6 VLM and Tool-Calling

- **VLM alignment:** `SFTTrainer`, `DPOTrainer`, `GRPOTrainer`, `RLOOTrainer` all support VLMs (multimodal inputs via processor-aware collation).
- **Tool-calling:** `DPOTrainer` and `SFTTrainer` have explicit tool-calling support (formatting/masking of tool call tokens).
- **Agentic RL:** `GRPOTrainer` supports agentic workflows; `OpenEnv` (Oct 2025) adds an open tool-environment ecosystem. However, TRL does **not** have an async GPU-decoupled agent loop — tool-call latency stalls the training process.

### 2.7 Recent 2025 Highlights

| Date | Update |
|---|---|
| Jan 2025 | Open-R1: full DeepSeek-R1 reproduction using TRL |
| May 2025 | Liger kernels for GRPO — major memory/speed win |
| Jun 2025 | Co-located vLLM in TRL for online RL trainers |
| Aug 2025 | VLM alignment support in GRPOTrainer |
| Oct 2025 | OpenEnv: open agent environment ecosystem integration |
| Mar 2026 | TRL v1.0 release: stable API, architectural cleanup |

### 2.8 Strengths

1. **Developer ergonomics** — `GRPOTrainer(model, args, train_dataset, reward_funcs=...)` — fits in <50 lines of boilerplate.
2. **HF ecosystem native** — Any `AutoModel`, any HF dataset, any PEFT config, Weights & Biases, etc.
3. **PEFT/QLoRA** — Train large models (30–70B) on 4-GPU commodity rigs via quantised LoRA.
4. **Widest model coverage** — If it's on HF Hub, TRL can train it.
5. **VLM support** — Multimodal RL post-training out of the box.
6. **Active community** — Fast iteration; Open-R1 and dozens of community recipes.
7. **Process Reward Model training** — `PRMTrainer` is a notable capability VeRL lacks natively.

### 2.9 Weaknesses

1. **Scale ceiling** — No Megatron-LM; impractical for >70B full-parameter RL at production throughput.
2. **PPO is experimental** — The full 4-model PPO pipeline is not production-grade.
3. **No async agent loops** — GPU blocks during tool-call execution.
4. **Throughput gap vs. VeRL** — Without 3D-HybridEngine, memory layout switches between rollout and training are expensive.
5. **GRPO implementation quirks** — Naive GRPO without DAPO fixes (dynamic sampling, decoupled clip) can exhibit length bias and entropy collapse; not all fixes are default-on.

---

## 3. Algorithm Zoo — Current State of RL for LLMs (Late 2025)

The post-DeepSeek-R1 era produced an explosion of GRPO variants. Here is the taxonomy as of late 2025 / early 2026:

### 3.1 The GRPO Family (critic-free, group-relative)

| Algorithm | Key Innovation | Main Concern | Best For |
|---|---|---|---|
| **GRPO** (DeepSeek, 2024) | Group-relative advantages; no critic | Length bias; zero-signal groups; entropy collapse | Baseline for reasoning RL |
| **DAPO** (ByteDance, 2025) | Decoupled clip (ε_low ≠ ε_high) + dynamic sampling (filter zero-signal groups) + token-level PG loss + overlong shaping | More hyperparameters; GRPO family limitations | Long-CoT reasoning; production-scale RLVR |
| **Dr.GRPO** (Liu et al., 2025) | Removes 1/\|o_i\| length norm and σ_q std-dev division; equivalent to RLOO up to scaling | Less battle-tested | Correcting GRPO's statistical biases |
| **REINFORCE++** (Hu, 2025) | Batch-global baseline; no per-prompt grouping | Loses prompt-local difficulty signal | Avoiding group degeneracy; simple baseline |
| **GSPO** (Group Soft PO) | Sequence-level ratio via geometric mean; matches reward granularity | Newer; limited reproduction | Long-response MoE RL |
| **RLOO** (Ahmadian et al., 2024) | Leave-One-Out baseline; unbiased, no critic | Requires multi-sample generation | Variance reduction without critic overhead |
| **ReMax** | Greedy decoding as baseline | Greedy baseline may be poor for non-deterministic tasks | Low-cost critic-free training |

### 3.2 Actor-Critic Methods

| Algorithm | Key Feature | Status |
|---|---|---|
| **PPO** | Learned value function (GAE); token-level credit | Classic RLHF; high quality but expensive |
| **StepPO** (2025) | Step-level MDP + step-level credit assignment | Frontier for agentic RL; reduces sparse reward problem |

### 3.3 Off-Policy / Preference Methods

| Algorithm | Key Feature |
|---|---|
| **DPO** | Direct preference; offline; no RM |
| **Online DPO / SPIN / SPPO** | Self-play preference; iterative improvement |
| **CISPO** | IS-weight clipping (not objective clipping); asymmetric bounds; off-policy |
| **TOPR** | Sequence-level; asymmetric clipping by reward sign |

### 3.4 Reward Signal Paradigms

| Paradigm | Description | Use-case |
|---|---|---|
| **RLVR** (Rule-Verifiable Rewards) | Reward from deterministic verifier (math checker, test suite) | Coding, math, structured output |
| **Outcome Reward Model (ORM)** | Trained RM scoring final answer | General alignment |
| **Process Reward Model (PRM)** | Step-level rewards on reasoning trace | Long-CoT, complex reasoning |
| **LLM-as-Judge** | Strong LLM scores outputs | Quality tasks without verifier |

### 3.5 Converging Best Practices for Agentic-Coding RL

Based on the 2025 literature, the community is converging toward:

1. **Algorithm:** GRPO + DAPO fixes (dynamic sampling to filter zero-signal groups; decoupled clip; token-level loss) — or equivalently Dr.GRPO / REINFORCE++ for simpler implementations.
2. **Reward signal:** RLVR with test-suite execution (verifiable) — pass@k on code tests, format rewards.
3. **Multi-turn trajectories:** GRPO applied at trajectory level (sparse reward on final code output); StepPO-style step rewards are emerging for better credit assignment.
4. **Cold-start:** Brief SFT on curated CoT traces before RL (DeepSeek-R1 recipe) to avoid early entropy collapse.
5. **Context length:** Long context (16k–32k) is essential for coding; models with long context rollout support (SGLang/vLLM paged attention) are required.

---

## 4. Comparison Matrix

### 4.1 Feature Comparison

| Dimension | VeRL | TRL |
|---|---|---|
| **Primary abstraction** | HybridFlow dataflow graph + Ray workers | HF Trainer subclass + Accelerate |
| **Ease of entry** | ★★☆ (complex) | ★★★★★ (simple) |
| **Algorithm breadth** | ★★★★★ (PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, GSPO, OTB, SAPO, SPIN, SPPO, GPG) | ★★★★☆ (GRPO, RLOO, DPO variants; PPO experimental) |
| **Max tested scale** | 671B params, 100s of GPUs | ~70B with FSDP ZeRO-3; practical ceiling ~32 GPUs full-param |
| **Training backends** | FSDP, Megatron-LM, MindSpeed | FSDP, DeepSpeed ZeRO |
| **Rollout backends** | vLLM, SGLang, TensorRT-LLM, HF | vLLM (co-located), HF |
| **3D-HybridEngine** | ✅ (key differentiator) | ❌ |
| **Async agent loop** | ✅ AsyncServer + AgentLoop | ❌ (blocking) |
| **Agentic tool-calling RL** | ✅ (SandboxFusionTool, asyncio loop) | ⚠️ (GRPOTrainer + OpenEnv; blocking) |
| **VLM support** | ✅ (VeOmni stack) | ✅ (GRPOTrainer, DPOTrainer) |
| **PEFT / LoRA / QLoRA** | ⚠️ (partial; not primary use-case) | ✅ (native, core feature) |
| **Process Reward Model** | ❌ (native) | ✅ (PRMTrainer) |
| **HF Hub model load** | ✅ (via HF Transformers) | ✅ (native) |
| **Hardware (non-NVIDIA)** | ✅ AMD, Ascend | ⚠️ (primarily NVIDIA; DeepSpeed has AMD support) |
| **Production pedigree** | DeepSeek-R1, DAPO, Qwen RL | Open-R1, academic research, community |
| **Ray requirement** | ✅ Required | ❌ Not needed |
| **Documentation quality** | ★★★☆ | ★★★★★ |
| **Community size** | Medium (but growing fast) | Very large |

### 4.2 Throughput (Indicative)

| Scenario | VeRL | TRL |
|---|---|---|
| 1.5B model, 8× H100, context 28k | Step time ~363s (gen: 261s + train: 66s) | No published comparable; likely 1.5–3× slower without HybridEngine |
| 7B model, 8× A100, GRPO | Community reports: 2–4× faster than naive HF due to vLLM + resharding | With co-located vLLM: competitive at small scale; degrades at larger context |
| 70B+ full-param GRPO | ✅ Efficient with Megatron-LM + SGLang | ⚠️ Possible with FSDP ZeRO-3 but slow; practical limit |
| 70B+ QLoRA GRPO | Not optimised | ✅ TRL + QLoRA is the go-to recipe |

### 4.3 Agentic RL Specifically

| Capability | VeRL | TRL |
|---|---|---|
| Multi-turn rollout | ✅ | ✅ (limited) |
| Tool-call execution during rollout | ✅ Async (GPU not blocked) | ⚠️ Synchronous (GPU blocked) |
| Code sandbox | ✅ SandboxFusionTool | ❌ (user must integrate) |
| Reward on trajectory outcome | ✅ | ✅ (via reward_funcs) |
| Step-level credit assignment | ✅ (OTB, StepPO-compatible) | ❌ (trajectory-level only natively) |
| Multi-node rollout | ✅ (SGLang multi-node) | ⚠️ (experimental vLLM multi-node) |

---

## 5. Recommendation

### 5.1 Decision Framework

```
If target model size > 70B (full-param RL)         → VeRL + Megatron-LM
If agentic coding trajectories are core use-case    → VeRL (async tool loops)
If commodity GPUs (≤8× A100) + any HF model        → TRL (GRPOTrainer + vLLM)
If LoRA/QLoRA post-training is acceptable           → TRL
If rapid prototyping / research iteration           → TRL
If production-scale, low-latency RL pipeline        → VeRL
If VLM post-training (small-mid scale)              → TRL (simpler)
If VLM post-training (large scale)                  → VeRL (VeOmni)
```

### 5.2 For a "Take Any HF Model and RL Post-Train It" Framework

**Primary recommendation: TRL as the default, VeRL as the scale-out path.**

**Rationale:**

1. **TRL covers the 80% case:** Any HF model can be loaded, any reward function can be plugged in, and the `GRPOTrainer` with co-located vLLM gives competitive throughput up to ~70B models on reasonable hardware.

2. **TRL's ergonomics are essential for user adoption:** A framework goal of "any HF model" implies the interface must be familiar and accessible. TRL achieves this; VeRL does not.

3. **VeRL is the right backend for scale-out:** When users graduate to full-param 70B+ runs, or when async agentic trajectories are needed, VeRL is the right sub-backend. A framework could abstract both: use TRL for the training API surface, offer VeRL as a `backend="verl"` option for production scale.

4. **Algorithm-wise, GRPO + DAPO fixes is the current best practice** for agentic-coding RL. Both TRL (GRPOTrainer) and VeRL support this. Implementing DAPO's dynamic sampling filter and decoupled clip on top of TRL's GRPOTrainer is straightforward.

5. **Agentic coding gap:** TRL's missing async tool-execution loop is a real gap. For a framework targeting agentic coding post-training, this should be bridged — either by adopting VeRL's AgentLoop pattern or by implementing an async wrapper over TRL's rollout phase.

### 5.3 Suggested Architecture for the Framework

```
Framework Public API (HF-compatible)
    ↓
Trainer Abstraction Layer
    ├── Backend: TRL GRPOTrainer (default; <70B; commodity)
    │       ├── vLLM co-located rollout
    │       ├── GRPO + DAPO fixes (dynamic sampling, decoupled clip)
    │       └── Reward: RLVR (test execution) | LLM-judge | ORM
    └── Backend: VeRL (scale-out; ≥70B; H100 clusters; agentic)
            ├── 3D-HybridEngine + SGLang
            ├── Async AgentLoop + SandboxFusionTool
            └── Megatron-LM for 70B+ full-param

Reward Layer (shared)
    ├── Test-suite executor (RLVR for coding)
    ├── Format verifier
    ├── PRM (process reward; TRL PRMTrainer)
    └── LLM-as-judge

Algorithm Layer (shared config, maps to trainer)
    └── GRPO / DAPO / RLOO / PPO / DPO
```

---

## 6. Sources

### Framework Documentation
- VeRL GitHub: https://github.com/volcengine/verl
- TRL GitHub: https://github.com/huggingface/trl
- VeRL DeepWiki (architecture reference): https://deepwiki.com/search/what-is-verls-architecture-wha_d0f02939-74bd-4877-8821-2249dac5e72e
- TRL DeepWiki (trainer reference): https://deepwiki.com/search/what-trainers-does-trl-support_cb760bf9-4c30-47cc-8f80-1b10e71a53bf

### Algorithm Papers
- **GRPO / DeepSeek-R1-Zero:** DeepSeek-AI et al. (2025). *DeepSeek-R1.* https://arxiv.org/abs/2501.12948
- **DAPO:** Yu et al. (2025). *DAPO: Decoupled Clip and Dynamic Sampling Policy Optimization.* (ByteDance / VeRL team)
- **Dr.GRPO:** Liu et al. (2025). *Understanding GRPO: Dr.GRPO.* Referenced in RLHF book: https://rlhfbook.com/c/06-policy-gradients
- **REINFORCE++:** Hu (2025). *REINFORCE++: A Simple and Efficient Approach for Aligning LLMs.* Referenced in multiple 2025 papers.
- **RLOO:** Ahmadian et al. (2024). *Back to Basics: Revisiting REINFORCE-Style Optimization for Language Models.* 
- **GSPO:** Referenced in UC Berkeley Scalable AI lecture (Spring 2026): http://scalable-ai.eecs.berkeley.edu/assets/lecture_slides/lecture_15.pdf
- **StepPO:** arxiv.org/html/2604.18401v1 — *StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning*
- **ARPO:** arxiv.org/html/2507.19849v1 — *Agentic Reinforced Policy Optimization*

### Benchmarks & Comparisons
- VeRL v0.5.0 benchmark (8× H100, 1.5B model): https://rlinf.readthedocs.io/en/latest/rst_source/blog/compare_with_verl.html
- GRPO VRAM/cost analysis on H200/B200: https://www.spheron.network/blog/grpo-fine-tuning-gpu-cloud
- Oumi: Running GRPO in TRL and VeRL: https://oumi.ai/blog/run-grpo-training-in-oumi-using-the

### Blog Posts / Surveys
- UC Berkeley Scalable AI Lecture 15 (Spring 2026) — Algorithm comparison table: http://scalable-ai.eecs.berkeley.edu/assets/lecture_slides/lecture_15.pdf
- "From REINFORCE to Dr. GRPO" blog (Qingfeng, 2025): https://lancelqf.github.io/note/llm_post_training
- Sebastian Raschka — State of LLMs 2025: https://magazine.sebastianraschka.com/p/state-of-llms-2025
- RLHF and Post-Training Book (Nathan Lambert): https://rlhfbook.com/c/06-policy-gradients
- TRL blog — Liger GRPO (May 2025): Hugging Face blog
- TRL blog — Co-located vLLM (Jun 2025): Hugging Face blog
- TRL blog — VLM alignment (Aug 2025): Hugging Face blog
- TRL blog — OpenEnv (Oct 2025): Hugging Face blog
- TRL v1 release blog (Mar 2026): Hugging Face blog