Hugging Face's logo

Codeseys
/
composer-replication-framework

Reinforcement Learning
Transformers
English
post-training
distillation
agentic-coding
composer-2.5
cursor
kimi-k2
grpo
dapo
diloco
openenv
trl
verl
research
methodology
Model card Files
xet
 Community
composer-replication-framework / research /04-verl-trl.md
Codeseys's picture
Codeseys
Initial commit: Composer 2.5 Replication Framework — research synthesis
7165832
preview code
|
Raw
History Blame Contribute Delete
24.4 kB

VeRL vs. HF TRL — Deep-Dive Comparison Report

Generated: 2026-05-25
Scope: Post-training framework selection for a "take any HF model, RL post-train it" goal, with particular focus on agentic-coding use-cases.

Table of Contents

  1. VeRL Deep-Dive
  2. TRL Deep-Dive
  3. Algorithm Zoo — Current State of RL for LLMs (Late 2025)
  4. Comparison Matrix
  5. Recommendation
  6. Sources

1. VeRL Deep-Dive

1.1 Overview

VeRL (Volcano Engine Reinforcement Learning) is ByteDance's production-grade, open-source RL training library for LLMs. Released publicly in 2024, it is the framework that powered DeepSeek-R1-style large-scale RL post-training runs and Qwen RL post-training. The headline paper is HybridFlow (Sheng et al., 2025), which formalises the underlying architecture.

GitHub: https://github.com/volcengine/verl
Stars: >10 k (as of mid-2025)

1.2 Architecture — HybridFlow

VeRL's core design principle is the HybridFlow programming model, which decouples the RL control plane from the compute plane:

  • Single-Controller Orchestration: A central RayPPOTrainer (Ray-based) coordinates all distributed workers. The controller treats the cluster as a set of remote high-level operators, making it easy to compose new algorithms.
  • Computation-Data Decoupling: Workers execute independently and exchange state via DataProto objects, making computation flow reusable across different RL algorithms without re-implementation.
  • 3D-HybridEngine: A single worker can switch between training mode and inference/rollout mode, eliminating redundant model copies. During PPO/GRPO, the Actor is used for both generation and gradient updates via efficient resharding (e.g., FSDP sharded ↔ vLLM TP). This is the key memory efficiency win.
  • Flexible Resource Allocation: Models can be colocated on the same GPU set, placed on separate GPU sets, or run in a hybrid configuration, enabling optimal hardware utilisation at scale.

1.3 Training Backends

Layer Options
Distributed training FSDP / FSDP2 (research-friendly), Megatron-LM v0.13.1+ (production scale), MindSpeed-LLM (Ascend NPU)
Rollout / inference vLLM (≥0.8.3), SGLang (fully supported, multi-node), TensorRT-LLM, HF Transformers (debug only)
Hardware NVIDIA H100/A100, AMD, Ascend 910
Orchestration Ray (required)

Key insight: VeRL treats the training engine and rollout engine as separable components. The 3D-HybridEngine handles weight resharding between FSDP sharding patterns (needed for training) and Tensor-Parallel patterns (needed for vLLM/SGLang generation), without maintaining duplicate model copies.

1.4 Algorithm Zoo in VeRL

VeRL ships first-class implementations of:

Algorithm Status Notes
PPO Stable Actor + Critic + Reference + Reward model; full pipeline
GRPO Stable Critic-free; group-relative advantages
DAPO Stable Decoupled clip + dynamic sampling + token-level PG loss
RLOO Stable REINFORCE Leave-One-Out; no critic
ReMax Stable Greedy baseline; no critic
REINFORCE++ Stable Batch-global baseline with clipping
SPIN Stable Self-play via online DPO loss
SPPO Stable Self-play preference optimisation
GPG Stable Policy gradient variant for math/reasoning
OTB Stable Optimal Token Baseline for fine-grained credit
SAPO Community Smoothing-based actor-policy optimisation
GSPO Community Grouped Soft Policy Optimisation (sequence-level)
DPO / Online DPO Supported Via SPIN / DAPO extensions

1.5 Agentic / Tool-Calling RL

VeRL has first-class agentic RL support:

  • AsyncServer / AgentLoop architecture: An asyncio-based co-routine mechanism separates the AgentLoop (client that drives multi-turn trajectories) from the AsyncServer (vLLM/SGLang inference backend). During tool-call waits (e.g., code execution), GPU compute is not blocked — other inflight requests continue.
  • SandboxFusionTool: Built-in code-execution sandbox for agentic coding tasks; allows model → <tool_call> → sandbox response → next step trajectories with rewards assigned at trajectory end.
  • Multi-turn tokenisation: Supported but noted as complex; naive concatenation of per-turn token IDs can introduce distribution drift between the rollout policy and training policy.

1.6 Scale

Tested configuration Notes
Up to 671B parameters Confirmed in production (DeepSeek-scale)
Trillion-parameter GRPO 64 H800 GPUs; GRPO with Megatron-LM backend
8× H100 benchmark DeepSeek-R1-Distill-Qwen-1.5B, 28k context, batch 128 per DP: step time ~363s; gen throughput measured per-GPU

A third-party benchmark (RLinf docs, Aug 2025) running VeRL v0.5.0 on 8× H100s with a 1.5B model (context 28,672 tokens):

  • Generation time: 260.9 s/step
  • Training time: 66.5 s/step
  • Total step time: 363.6 s/step

VeRL's Megatron-LM backend + SGLang rollout is the performance-optimal path for >70B models.

1.7 Real-World Usage

  • DeepSeek-R1 lineage — The architecture is directly inspired by DeepSeek's internal RLVR pipeline.
  • Qwen RL post-training — Qwen3 and DAPO paper both used VeRL.
  • DAPO paper (ByteDance, 2025) — Trained Qwen2.5-72B with VeRL; achieved new AIME 2024 SOTA.
  • Multiple open reproductions of DeepSeek-R1-Zero use VeRL as the training backend.

1.8 Strengths

  1. Best-in-class throughput at scale — 3D-HybridEngine + vLLM/SGLang eliminates memory redundancy.
  2. Widest algorithm coverage — PPO through the latest DAPO/GSPO/OTB variants all natively supported.
  3. Production proven — Used at 671B scale with Megatron-LM.
  4. First-class agentic loops — AsyncServer decouples GPU from tool-call latency.
  5. Hardware agnostic — NVIDIA, AMD, Ascend.
  6. Flexible resource allocation — Colocated, separated, or hybrid GPU pooling.

1.9 Weaknesses / Challenges

  1. Steep learning curve — Ray orchestration, multiple backend configs, FSDP vs. Megatron choice; not a 3-line quickstart.
  2. Multi-turn tokenisation complexity — Risk of subtle off-policy drift if multi-turn chat templates are not handled carefully; noted as an active known issue.
  3. Off-policy instability — Rollout correction is provided but requires careful tuning; naive replay buffers can cause policy collapse.
  4. Heavyweight infrastructure — Requires Ray cluster; not ideal for single-GPU or commodity 4-GPU experiments.
  5. Documentation gaps — Community recipes exist but the core docs lag behind code velocity.

2. TRL Deep-Dive

2.1 Overview

TRL (Transformer Reinforcement Learning) is Hugging Face's mainstream post-training library, designed around the HF ecosystem (Accelerate, PEFT, Transformers, Datasets). The philosophy is accessible post-training for any HF model, favouring simplicity and developer ergonomics over raw throughput at frontier scale.

GitHub: https://github.com/huggingface/trl
Version milestone: TRL v1 released March 2026
Stars: >14 k

2.2 Trainer Taxonomy

TRL organises trainers into four categories:

Supervised

Trainer Description
SFTTrainer Instruction-tuning / supervised fine-tuning; supports packing, PEFT, VLMs
RewardTrainer Train scalar reward models from preference data
PRMTrainer Process Reward Model training (step-level rewards)

Preference / Offline Alignment

Trainer Description
DPOTrainer Direct Preference Optimisation; supports VLMs and tool-calling
BCOTrainer Binary Classifier Optimisation
CPOTrainer Contrastive Preference Optimisation
KTOTrainer KTO (binary signal, no pairs)
ORPOTrainer Odds-Ratio Preference Optimisation
GKDTrainer Generalised Knowledge Distillation
NashMDTrainer Nash Mirror Descent online preference

Online RL

Trainer Description
GRPOTrainer Primary online RL trainer. Group Relative Policy Optimisation; stable; VLM + agentic support
RLOOTrainer REINFORCE Leave-One-Out; supports VLMs
PPOTrainer Proximal Policy Optimisation; experimental (noted as incomplete)
OnlineDPOTrainer Online DPO with LLM-as-judge; experimental
XPOTrainer Exploratory DPO (experimental)

Other

Trainer Description
MiniLLMTrainer Reverse-KL distillation

2.3 GRPOTrainer — Key Design

GRPOTrainer is TRL's workhorse for RLVR-style training:

  • No critic model — group-relative advantages, matching GRPO semantics from DeepSeek-R1.
  • vLLM integration — co-located vLLM for fast rollout generation (June 2025 update: "NO GPU left behind" co-located vLLM).
  • Liger kernel integration — May 2025 update; significant memory/speed improvements for GRPO training step.
  • VLM support — Vision-language models trainable with GRPO as of August 2025.
  • Agentic workflows — GRPOTrainer supports multi-step agentic rollouts; OpenEnv integration (October 2025) provides tool/environment loop scaffolding.

2.4 Distributed Backends

TRL relies on HF Accelerate as the distribution abstraction:

Backend Support level
DeepSpeed ZeRO-1/2/3 Stable
FSDP v1 + v2 Stable
PEFT / LoRA / QLoRA Native; enables large model training on fewer GPUs
vLLM (co-located) Integrated for online RL trainers (GRPO, RLOO, PPO)

2.5 Scale Ceiling

TRL was designed for the commodity to mid-scale cluster range:

  • Single GPU (with QLoRA) up through multi-node clusters.
  • No native Megatron-LM tensor/pipeline parallelism — limits scaling for >70B full-parameter runs.
  • No 3D-HybridEngine; actor model is held fully in training-mode sharding at all times, meaning rollout generation is bottlenecked by the training sharding strategy.
  • Practical ceiling: 8–32 GPU clusters for full-parameter runs of 7–70B models; beyond that, FSDP ZeRO-3 sharding overhead becomes limiting.

2.6 VLM and Tool-Calling

  • VLM alignment: SFTTrainer, DPOTrainer, GRPOTrainer, RLOOTrainer all support VLMs (multimodal inputs via processor-aware collation).
  • Tool-calling: DPOTrainer and SFTTrainer have explicit tool-calling support (formatting/masking of tool call tokens).
  • Agentic RL: GRPOTrainer supports agentic workflows; OpenEnv (Oct 2025) adds an open tool-environment ecosystem. However, TRL does not have an async GPU-decoupled agent loop — tool-call latency stalls the training process.

2.7 Recent 2025 Highlights

Date Update
Jan 2025 Open-R1: full DeepSeek-R1 reproduction using TRL
May 2025 Liger kernels for GRPO — major memory/speed win
Jun 2025 Co-located vLLM in TRL for online RL trainers
Aug 2025 VLM alignment support in GRPOTrainer
Oct 2025 OpenEnv: open agent environment ecosystem integration
Mar 2026 TRL v1.0 release: stable API, architectural cleanup

2.8 Strengths

  1. Developer ergonomics — GRPOTrainer(model, args, train_dataset, reward_funcs=...) — fits in <50 lines of boilerplate.
  2. HF ecosystem native — Any AutoModel, any HF dataset, any PEFT config, Weights & Biases, etc.
  3. PEFT/QLoRA — Train large models (30–70B) on 4-GPU commodity rigs via quantised LoRA.
  4. Widest model coverage — If it's on HF Hub, TRL can train it.
  5. VLM support — Multimodal RL post-training out of the box.
  6. Active community — Fast iteration; Open-R1 and dozens of community recipes.
  7. Process Reward Model training — PRMTrainer is a notable capability VeRL lacks natively.

2.9 Weaknesses

  1. Scale ceiling — No Megatron-LM; impractical for >70B full-parameter RL at production throughput.
  2. PPO is experimental — The full 4-model PPO pipeline is not production-grade.
  3. No async agent loops — GPU blocks during tool-call execution.
  4. Throughput gap vs. VeRL — Without 3D-HybridEngine, memory layout switches between rollout and training are expensive.
  5. GRPO implementation quirks — Naive GRPO without DAPO fixes (dynamic sampling, decoupled clip) can exhibit length bias and entropy collapse; not all fixes are default-on.

3. Algorithm Zoo — Current State of RL for LLMs (Late 2025)

The post-DeepSeek-R1 era produced an explosion of GRPO variants. Here is the taxonomy as of late 2025 / early 2026:

3.1 The GRPO Family (critic-free, group-relative)

Algorithm Key Innovation Main Concern Best For
GRPO (DeepSeek, 2024) Group-relative advantages; no critic Length bias; zero-signal groups; entropy collapse Baseline for reasoning RL
DAPO (ByteDance, 2025) Decoupled clip (ε_low ≠ ε_high) + dynamic sampling (filter zero-signal groups) + token-level PG loss + overlong shaping More hyperparameters; GRPO family limitations Long-CoT reasoning; production-scale RLVR
Dr.GRPO (Liu et al., 2025) Removes 1/|o_i| length norm and σ_q std-dev division; equivalent to RLOO up to scaling Less battle-tested Correcting GRPO's statistical biases
REINFORCE++ (Hu, 2025) Batch-global baseline; no per-prompt grouping Loses prompt-local difficulty signal Avoiding group degeneracy; simple baseline
GSPO (Group Soft PO) Sequence-level ratio via geometric mean; matches reward granularity Newer; limited reproduction Long-response MoE RL
RLOO (Ahmadian et al., 2024) Leave-One-Out baseline; unbiased, no critic Requires multi-sample generation Variance reduction without critic overhead
ReMax Greedy decoding as baseline Greedy baseline may be poor for non-deterministic tasks Low-cost critic-free training

3.2 Actor-Critic Methods

Algorithm Key Feature Status
PPO Learned value function (GAE); token-level credit Classic RLHF; high quality but expensive
StepPO (2025) Step-level MDP + step-level credit assignment Frontier for agentic RL; reduces sparse reward problem

3.3 Off-Policy / Preference Methods

Algorithm Key Feature
DPO Direct preference; offline; no RM
Online DPO / SPIN / SPPO Self-play preference; iterative improvement
CISPO IS-weight clipping (not objective clipping); asymmetric bounds; off-policy
TOPR Sequence-level; asymmetric clipping by reward sign

3.4 Reward Signal Paradigms

Paradigm Description Use-case
RLVR (Rule-Verifiable Rewards) Reward from deterministic verifier (math checker, test suite) Coding, math, structured output
Outcome Reward Model (ORM) Trained RM scoring final answer General alignment
Process Reward Model (PRM) Step-level rewards on reasoning trace Long-CoT, complex reasoning
LLM-as-Judge Strong LLM scores outputs Quality tasks without verifier

3.5 Converging Best Practices for Agentic-Coding RL

Based on the 2025 literature, the community is converging toward:

  1. Algorithm: GRPO + DAPO fixes (dynamic sampling to filter zero-signal groups; decoupled clip; token-level loss) — or equivalently Dr.GRPO / REINFORCE++ for simpler implementations.
  2. Reward signal: RLVR with test-suite execution (verifiable) — pass@k on code tests, format rewards.
  3. Multi-turn trajectories: GRPO applied at trajectory level (sparse reward on final code output); StepPO-style step rewards are emerging for better credit assignment.
  4. Cold-start: Brief SFT on curated CoT traces before RL (DeepSeek-R1 recipe) to avoid early entropy collapse.
  5. Context length: Long context (16k–32k) is essential for coding; models with long context rollout support (SGLang/vLLM paged attention) are required.

4. Comparison Matrix

4.1 Feature Comparison

Dimension VeRL TRL
Primary abstraction HybridFlow dataflow graph + Ray workers HF Trainer subclass + Accelerate
Ease of entry ★★☆ (complex) ★★★★★ (simple)
Algorithm breadth ★★★★★ (PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, GSPO, OTB, SAPO, SPIN, SPPO, GPG) ★★★★☆ (GRPO, RLOO, DPO variants; PPO experimental)
Max tested scale 671B params, 100s of GPUs ~70B with FSDP ZeRO-3; practical ceiling ~32 GPUs full-param
Training backends FSDP, Megatron-LM, MindSpeed FSDP, DeepSpeed ZeRO
Rollout backends vLLM, SGLang, TensorRT-LLM, HF vLLM (co-located), HF
3D-HybridEngine ✅ (key differentiator) ❌
Async agent loop ✅ AsyncServer + AgentLoop ❌ (blocking)
Agentic tool-calling RL ✅ (SandboxFusionTool, asyncio loop) ⚠️ (GRPOTrainer + OpenEnv; blocking)
VLM support ✅ (VeOmni stack) ✅ (GRPOTrainer, DPOTrainer)
PEFT / LoRA / QLoRA ⚠️ (partial; not primary use-case) ✅ (native, core feature)
Process Reward Model ❌ (native) ✅ (PRMTrainer)
HF Hub model load ✅ (via HF Transformers) ✅ (native)
Hardware (non-NVIDIA) ✅ AMD, Ascend ⚠️ (primarily NVIDIA; DeepSpeed has AMD support)
Production pedigree DeepSeek-R1, DAPO, Qwen RL Open-R1, academic research, community
Ray requirement ✅ Required ❌ Not needed
Documentation quality ★★★☆ ★★★★★
Community size Medium (but growing fast) Very large

4.2 Throughput (Indicative)

Scenario VeRL TRL
1.5B model, 8× H100, context 28k Step time ~363s (gen: 261s + train: 66s) No published comparable; likely 1.5–3× slower without HybridEngine
7B model, 8× A100, GRPO Community reports: 2–4× faster than naive HF due to vLLM + resharding With co-located vLLM: competitive at small scale; degrades at larger context
70B+ full-param GRPO ✅ Efficient with Megatron-LM + SGLang ⚠️ Possible with FSDP ZeRO-3 but slow; practical limit
70B+ QLoRA GRPO Not optimised ✅ TRL + QLoRA is the go-to recipe

4.3 Agentic RL Specifically

Capability VeRL TRL
Multi-turn rollout ✅ ✅ (limited)
Tool-call execution during rollout ✅ Async (GPU not blocked) ⚠️ Synchronous (GPU blocked)
Code sandbox ✅ SandboxFusionTool ❌ (user must integrate)
Reward on trajectory outcome ✅ ✅ (via reward_funcs)
Step-level credit assignment ✅ (OTB, StepPO-compatible) ❌ (trajectory-level only natively)
Multi-node rollout ✅ (SGLang multi-node) ⚠️ (experimental vLLM multi-node)

5. Recommendation

5.1 Decision Framework 

If target model size > 70B (full-param RL)         → VeRL + Megatron-LM
If agentic coding trajectories are core use-case    → VeRL (async tool loops)
If commodity GPUs (≤8× A100) + any HF model        → TRL (GRPOTrainer + vLLM)
If LoRA/QLoRA post-training is acceptable           → TRL
If rapid prototyping / research iteration           → TRL
If production-scale, low-latency RL pipeline        → VeRL
If VLM post-training (small-mid scale)              → TRL (simpler)
If VLM post-training (large scale)                  → VeRL (VeOmni)

5.2 For a "Take Any HF Model and RL Post-Train It" Framework

Primary recommendation: TRL as the default, VeRL as the scale-out path.

Rationale:

  1. TRL covers the 80% case: Any HF model can be loaded, any reward function can be plugged in, and the GRPOTrainer with co-located vLLM gives competitive throughput up to ~70B models on reasonable hardware.

  2. TRL's ergonomics are essential for user adoption: A framework goal of "any HF model" implies the interface must be familiar and accessible. TRL achieves this; VeRL does not.

  3. VeRL is the right backend for scale-out: When users graduate to full-param 70B+ runs, or when async agentic trajectories are needed, VeRL is the right sub-backend. A framework could abstract both: use TRL for the training API surface, offer VeRL as a backend="verl" option for production scale.

  4. Algorithm-wise, GRPO + DAPO fixes is the current best practice for agentic-coding RL. Both TRL (GRPOTrainer) and VeRL support this. Implementing DAPO's dynamic sampling filter and decoupled clip on top of TRL's GRPOTrainer is straightforward.

  5. Agentic coding gap: TRL's missing async tool-execution loop is a real gap. For a framework targeting agentic coding post-training, this should be bridged — either by adopting VeRL's AgentLoop pattern or by implementing an async wrapper over TRL's rollout phase.

5.3 Suggested Architecture for the Framework 

Framework Public API (HF-compatible)
    ↓
Trainer Abstraction Layer
    ├── Backend: TRL GRPOTrainer (default; <70B; commodity)
    │       ├── vLLM co-located rollout
    │       ├── GRPO + DAPO fixes (dynamic sampling, decoupled clip)
    │       └── Reward: RLVR (test execution) | LLM-judge | ORM
    └── Backend: VeRL (scale-out; ≥70B; H100 clusters; agentic)
            ├── 3D-HybridEngine + SGLang
            ├── Async AgentLoop + SandboxFusionTool
            └── Megatron-LM for 70B+ full-param

Reward Layer (shared)
    ├── Test-suite executor (RLVR for coding)
    ├── Format verifier
    ├── PRM (process reward; TRL PRMTrainer)
    └── LLM-as-judge

Algorithm Layer (shared config, maps to trainer)
    └── GRPO / DAPO / RLOO / PPO / DPO

6. Sources

Framework Documentation

Algorithm Papers

  • GRPO / DeepSeek-R1-Zero: DeepSeek-AI et al. (2025). DeepSeek-R1. https://arxiv.org/abs/2501.12948
  • DAPO: Yu et al. (2025). DAPO: Decoupled Clip and Dynamic Sampling Policy Optimization. (ByteDance / VeRL team)
  • Dr.GRPO: Liu et al. (2025). Understanding GRPO: Dr.GRPO. Referenced in RLHF book: https://rlhfbook.com/c/06-policy-gradients
  • REINFORCE++: Hu (2025). REINFORCE++: A Simple and Efficient Approach for Aligning LLMs. Referenced in multiple 2025 papers.
  • RLOO: Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE-Style Optimization for Language Models.
  • GSPO: Referenced in UC Berkeley Scalable AI lecture (Spring 2026): http://scalable-ai.eecs.berkeley.edu/assets/lecture_slides/lecture_15.pdf
  • StepPO: arxiv.org/html/2604.18401v1 — StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning
  • ARPO: arxiv.org/html/2507.19849v1 — Agentic Reinforced Policy Optimization

Benchmarks & Comparisons

Blog Posts / Surveys