research/deepread/08-rl-infra.md

Deep-Read: RL Infra & Frameworks — Critical Findings

Cluster 8 of the dataset-pipeline review series Reviewer: automated critical pipeline, 2026-06-09 Primary sources fetched: TRL GRPOTrainer live docs (v1.5.1, huggingface.co/docs/trl/en/grpo_trainer), vLLM co-locate blog (June 2025, huggingface.co/blog/vllm-colocate), verl GitHub README (main branch, verl-project/verl — 21.9k stars), SWE-MiniSandbox paper (arXiv:2602.11210v5), secure EKS sandboxes article (AWS Builder Center). Repo files inspected: research/04-verl-trl.md, research/03-monarch-torchforge-openenv.md, docs/adrs/ADR-006-rl-frameworks.md, docs/adrs/ADR-008-drgrpo-sdpo-live-channel.md, composer_replication/datagen/env.py, composer_replication/trainer/composer_trainer.py, composer_replication/recipes/prime_rl/composer_loss.py.

---

1. TRL GRPOTrainer: Does It Actually Support Multi-Turn Agentic Rollouts?

1.1 What the live TRL docs say (as of TRL v1.5.1, fetched 2026-06-10)

The TRL GRPOTrainer docs (primary source, not the repo's research/04 note written in May 2025) describe two distinct agentic mechanisms:

Mechanism A — tools parameter: Pass a list of Python callables. GRPOTrainer runs a tool-call loop. Quote from docs: "GRPO supports agent training through the tools argument in GRPOTrainer." The loop has a hard cap max_tool_calling_iterations (default: unlimited, stops on no-tool-call response or max_model_length). Each tool call is synchronous — the training GPU waits.

Mechanism B — environment_factory parameter: Pass a callable that creates environment instances. "GRPOTrainer creates one environment instance per rollout and exposes the environment's public methods as tools." Requires transformers>=5.2.0. Marked experimental: "This feature is experimental and may change or be removed at any time without prior notice." The reset() method can return a string that gets appended to the last user message. rollout_func is similarly experimental.

Mechanism C — rollout_func (custom rollout): A callable that receives prompts and the trainer, returns {"prompt_ids", "completion_ids", "logprobs"}. Also experimental. This is the escape hatch for fully custom multi-turn generation.

Key constraint confirmed from primary source: TRL has no async GPU-decoupled agent loop. The docs explicitly state the training-inference mismatch and handle it via Truncated Importance Sampling (vllm_importance_sampling_correction=True by default), not by async GPU handoff. When a tool call is executing, the GPU waits. This is not a flaw in the repo's research note — research/04-verl-trl.md correctly identified this gap — but the docs now show TRL has partially closed the multi-turn gap via tools / environment_factory.

1.2 What research/04-verl-trl.md claims vs. primary source

Claim in research/04	Primary source (TRL docs, 2026)	Verdict
"TRL does NOT have an async GPU-decoupled agent loop"	Confirmed	CORRECT
"OpenEnv integration (October 2025)"	Confirmed; environment_factory + TRL's OpenEnv guide	CORRECT
"VLM support"	Confirmed — tools can return list of content blocks incl. images	CORRECT
"GRPOTrainer supports multi-step agentic rollouts" (04:173)	Confirmed via tools + environment_factory	CORRECT
TRL v1.0 released March 2026	Confirmed; docs show versions v1.0.0 through v1.5.1	CORRECT
Default loss_type is "dapo"	CONFIRMED from source: loss_type: str = 'dapo' in GRPOConfig	CORRECT
Default scale_rewards is...	CONFIRMED: default is "group" (not False/"none")	CORRECT

1.3 Critical discovery: TRL's default is DAPO, not GRPO

The TRL GRPOConfig shows loss_type = 'dapo' as the default. ADR-008 claims to configure loss_type="dr_grpo" to match Composer 2.5. The source confirms "dr_grpo" is a valid value (uses max_completion_length as the constant denominator). This is consistent with ADR-008's decision.

However: ADR-008 states "KL estimator (k1 vs k3) is not configured or asserted" as an OPEN item. The TRL docs show beta=0.0 as default (KL term disabled). If beta=0, the k1/k3 distinction is moot — there is no KL term in the loss at all. The ADR-008 open item is therefore low-priority when beta=0 (the current default). If the repo ever enables beta>0 to use k1 KL-in-loss (distinct from the k1-in-reward path the trainer already implements), the open item becomes relevant.

1.4 scale_rewards drift assertion

ADR-008 checks str(cfg.scale_rewards).lower() in ("none","false"). Primary source confirms scale_rewards accepts: True/"group" (default), "batch", False/"none". The check is correct.

---

2. The Colocate-vLLM Blog: What It Actually Says

Primary source: "No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" (June 3, 2025, huggingface.co/blog/vllm-colocate).

What the blog confirms:

• Co-locate mode (vllm_mode="colocate") runs training and vLLM in the same process, sharing GPUs. No REST API overhead.
• Speedups measured: 1.43× for 1.5B model, 1.35× for 7B model, 1.73× for 7B with TP>1. For 72B model (Qwen2.5-Math-72B): co-locate is ~1.26× faster than plain TRL with 4 fewer GPUs.
• vLLM sleep mode (level 2) is not yet merged into TRL upstream (as of the blog date) due to a segfault on exit (vLLM issue #16993). The docs now show vllm_enable_sleep_mode as a parameter, implying it was eventually merged, but the blog notes a real production bug.
• FSDP + co-locate + LoRA has known issues: "GRPO + FSDP + LoRA + VLLM colocate" doesn't work; DeepSpeed ZeRO-3 is the recommended path. FSDP1 has a NaN bug with co-located vLLM (issue #14443).

What research/04 says: Does not cite the co-locate blog specifically but correctly describes the "NO GPU left behind" feature (June 2025 update, row in §2.7 table). No material misread.

What the repo's SageMaker smoke recipe uses: The SageMaker GRPO smoke (from git history context) uses use_vllm=False for initial tests, which is fine — the co-locate mode requires enough GPU memory for both model and vLLM, and a single g5.2xlarge (1× A10G, 24 GB) may not accommodate it.

---

3. VeRL: Agentic Mode and the AsyncServer

Primary source: verl GitHub README (main branch, fetched 2026-06-10). Key finding from README:

"[2026/05] uni-agent is released: a unified agent framework to build, run, and train LLM agents at scale, built on top of verl." "[2026/01] transfer_queue, fully_async_policy, one_step_off_policy ... are kept under verl/experimental since they are planned to be merged into the main library." Feature list includes: "Multi-turn with tool calling", "Sandbox Fusion Integration", "SGLang, verl, OpenBMB and Tsinghua University: Pioneering End-to-End Multi-Turn RLHF"

The verl README confirms multi-turn tool-calling exists and uni-agent was released May 2026 as a unified agent framework. The AsyncServer/AgentLoop architecture described in research/04 is consistent with what the README describes, though the README doesn't use those exact terms. The experimental async features (fully_async_policy, transfer_queue) are available but not yet in main.

What research/04 claims about VeRL agentic support:

• "First-class agentic RL support" with AsyncServer/AgentLoop — the README confirms the direction but notes these are under verl/experimental. The research/04 characterization of "first-class" slightly overclaims what is in the stable API; the full async path is experimental.
• SandboxFusionTool — mentioned in the README as a documented integration ("Sandbox Fusion Integration" link). Consistent.
• "Multi-turn tokenisation: noted as complex; naive concatenation of per-turn token IDs can introduce distribution drift" — confirmed by the README community blog link "When Reasoning Models Break Tokenization: The Hidden Complexity of Multiturn Training."

---

4. PRIME-RL in the Repo

4.1 What ADR-006 claims

ADR-006 claims PRIME-RL ships a CustomLossConfig with import_path for dropping in a Python loss function, exposing LossInputs with trainer_logprobs, inference_logprobs, teacher_logprobs, advantages, loss_mask. It was used to train INTELLECT-1 (10B, 30 nodes) and INTELLECT-2 (32B QwQ).

4.2 What composer_replication/recipes/prime_rl/composer_loss.py confirms

The code reads (lines 21-28):

@dataclass
class LossInputs:
    trainer_logprobs:   Float[Tensor, ' seq']
    inference_logprobs: Float[Tensor, ' seq']
    teacher_logprobs:   Float[Tensor, ' seq'] | None
    advantages:         Float[Tensor, ' seq']
    loss_mask:          Bool[Tensor, ' seq']

This is marked as "verified against PrimeIntellect-ai/prime-rl src/prime_rl/trainer/rl/loss.py lines 13-22." The code correctly raises NotImplementedError when alpha_sdpo > 0 (logits not available, only log-probs). This is a real constraint, not a placeholder.

4.3 The DPPO upstream loss — a subtle accuracy point

The composer_loss.py reproduces the upstream DPPO loss verbatim (lines 40-60 of the file). It uses:

probs_diff = exp(trainer_logprobs) - exp(inference_logprobs)  # probability-space diff

This is notably not a log-ratio but a probability-space difference gating the drop/keep mask. This is PRIME-RL's design, not a repo mistake. But it means the DPPO channel is more like PRIME-RL's INTELLECT-style training than standard GRPO — the repo's framing in ADR-006 as "channels 1+3" needs to be understood in that context: channel 1 is DPPO-shaped (probability-gated policy update), not raw GRPO.

---

5. The Key Question: Is TRL's Single-Submit reward_fn a Dead End for Multi-Turn?

5.1 What env.py::reward_fn actually does

def reward_fn(self, prompts, completions, *, task_id, **kwargs) -> list[float]:
    ...
    for comp, tid in zip(completions, task_id):
        task = self.registry[tid]
        self.reset(task)
        if self._replay is not None:
            res = self._replay(self, comp)
        else:
            res = self.step({"type": "submit"})   # <-- single submit
        rewards.append(res.reward)

The fallback path (no _replay function) treats the entire completion as a single submit — this is an outcome reward on a single-turn completion. This is what the unit tests exercise and what a standard TRL reward_funcs call would do.

The intended multi-turn path is _replay: a callable that takes (env, completion) and drives multi-turn turns by parsing the agent's encoded tool-call history from the completion string. This is a custom deserializer that replays the agent turns and grades at the end.

5.2 Is this a dead end for multi-turn RL?

For current TRL integration: partly yes, mostly no.

The single-submit fallback IS a dead end for genuine multi-turn RL credit assignment — it cannot grade intermediate tool-call steps. But there are two viable paths:

Path A — _replay + rollout_func (TRL experimental): The rollout_func parameter in GRPOTrainer can drive multi-turn generation externally (running the env's step() loop), serialize the full trajectory into completion tokens, then call reward_fn which uses _replay to deserialize and grade. This makes the reward_fn the grader, not the rollout driver. This works in TRL today but requires the experimental rollout_func interface.

Path B — environment_factory (TRL experimental): Pass FeatureDeletionEnv (or an adapter) as environment_factory. GRPOTrainer calls reset() and then uses the env's public methods as tools. The reward_fn is replaced by a reward function that reads environments[i].reward after generation. This is the more principled path for true multi-turn RL and is what TRL's environment_factory was designed for. It requires transformers>=5.2.0 and is still experimental.

Path C — VeRL's AgentLoop: For the tree-of-work (multiple parallel rollout branches per prompt, credit assigned at trajectory end), VeRL's AsyncServer+AgentLoop is architecturally the right fit. Each branch is a coroutine; GPU is not blocked during sandbox.exec() calls. The repo acknowledges this in research/04 §5.3 recommendation.

5.3 The honest migration path

The current TRL single-submit reward_fn is:

• Correct for the Phase 1 use case: offline dataset generation where the model produces a diff and we grade it. This is the "GRPO on completions" paradigm.
• Insufficient for genuine multi-turn RL over FeatureDeletionEnv episodes, especially the tree-of-work vision where the model takes tool-call steps, explores branches, and gets rewards at trajectory end.

Migration path (in order of complexity):

1. Immediate (low cost): Use TRL's environment_factory with FeatureDeletionEnv as the adapter. The env's step() becomes a tool. Grade via reward_funcs reading env.reward. Marked experimental but low integration cost. This supports genuine multi-turn GRPO with the current TRL host.

2. Medium term (single-GPU scale): Implement rollout_func that drives the env loop directly, returns serialized trajectories with log-probs. Full control over multi-turn; TRL handles the GRPO update.

3. Scale-out (multi-GPU, async, tree-of-work): Migrate to VeRL's AgentLoop. The FeatureDeletionEnv maps onto verl's SandboxFusionTool protocol. The tree-of-work branching requires N parallel rollout workers per prompt, which VeRL's asyncio architecture supports and TRL's synchronous loop does not.

The tree-of-work IS multi-turn. The vision in framework/composer-replication-framework.md of a "multi-model Monte-Carlo tree-of-work" requires:

• Many concurrent rollout branches per prompt
• Reward propagated back through the tree (not just at leaf)
• Asynchronous sandbox execution without blocking GPU

None of these are provided by TRL's current GRPOTrainer (even with tools/environment_factory). VeRL's experimental fully_async_policy + AgentLoop is the right substrate. The repo's research/04 correctly identifies this but the ADR layer has not formally acknowledged this migration requirement.

---

6. Sandboxing for Code Execution at Scale

6.1 What the secure-EKS article says (primary source)

"gVisor added negligible launch latency... handles isolation for most agent workloads." "Cold start was around 5 seconds per sandbox" for Kata+Firecracker. "EKS Managed Node Groups do not work yet: they override the CPU Options stanza needed for nested virtualization, forcing the use of self-managed node groups." "Managed sandbox platforms skip Kubernetes entirely. E2B and Vercel Sandbox provision Firecracker microVMs directly... sandbox creation in under a second, versus ~5 seconds Kata with Firecracker on EKS."

6.2 What SWE-MiniSandbox (arXiv:2602.11210v5) adds

Abstract: "SWE-MiniSandbox lowers disk usage to approximately 5% of that required by container-based pipelines and reduces environment preparation time to about 25% of the container baseline." Uses "kernel-level mechanisms" (not containers) with "lightweight environment pre-caching." Empirical performance comparable to container-based pipelines on SWE-bench-style tasks.

This is directly relevant to the repo's FeatureDeletionEnv/Sandbox design. The current sandbox.py uses LocalSubprocessSandbox (plain subprocess, Docker-gated for real tests) — essentially no isolation for the subprocess case. For production RL training at scale with multiple rollout workers, the SWE-MiniSandbox approach (kernel-level isolation without per-task container builds) could reduce env setup from minutes to seconds.

6.3 What the repo's sandbox.py actually provides

sandbox.py defines:

• LocalSubprocessSandbox — runs commands via subprocess.run in the repo tree. No container, no kernel isolation. The security model relies on the denylist + cache scrub (commented as "INSUFFICIENT as a primary control").
• DockerSandbox (in docker_sandbox.py) — real isolation, referenced in tests.

The gap: For RL training at scale (many parallel rollout workers), neither LocalSubprocessSandbox nor per-task Docker containers are adequate:

• Subprocess: no isolation, reward hacking possible via Python import tricks (acknowledged in code comments).
• Docker: isolation is good, but per-task container boot is slow (typical: 2–5s without pre-warming), and at 8 rollouts × N prompts × G generations = hundreds of container launches per training step.

The repo's research/review-sandbox.json presumably tracks this; the production path requires pre-warmed sandbox pools (gVisor RuntimeClass for speed, Kata/Firecracker for stronger isolation).

---

7. Misreads, Overclaims, and Gaps in Repo Research

7.1 OVERCLAIM: VeRL "first-class" agentic RL

research/04 §1.5: "VeRL has first-class agentic RL support" and describes AsyncServer/AgentLoop as stable. The verl README (main branch, 2026-06-10) shows:

• transfer_queue, fully_async_policy, one_step_off_policy are kept under verl/experimental — "planned to be merged into the main library."
• uni-agent (May 2026) provides the higher-level agent framework, but it's a separate release on top of verl, not part of the stable library.

The agentic async path exists but is experimental in the stable API. The characterization as "first-class" is slightly ahead of the actual maturity. The repo should note this when recommending VeRL for the tree-of-work.

7.2 MISS: TRL now has environment_factory + tools for multi-turn

research/04 (written May 2025) describes TRL as having only synchronous reward functions. The live TRL docs (v1.5.1, 2026) show environment_factory and tools for genuine multi-turn generation loops. These are experimental but available. The research/04 comparison matrix says "TRL: agentic tool-calling RL ⚠️ (blocking)" — this remains accurate (it IS blocking), but misses that TRL now provides the environment_factory interface which would allow FeatureDeletionEnv to drive multi-turn episodes inside GRPOTrainer without a custom rollout_func. This is not mentioned in any ADR.

7.3 MISS: TRL default loss_type="dapo", NOT "grpo" or "dr_grpo"

ADR-008 correctly targets loss_type="dr_grpo", but research/04 (the background research) does not explicitly state that TRL 1.x defaults to DAPO loss. The live docs confirm this. The drift assertion in make_dr_grpo_config is the right mitigation.

7.4 GAP: scale_rewards default is "group", not True

The GRPOConfig shows scale_rewards: str = 'group' (a string, not a bool). ADR-008's assertion str(cfg.scale_rewards).lower() in ("none","false") correctly handles both the old bool (False) and new string ("none") forms. But the docs show True and "group" are equivalent (both mean group-level std scaling). The assertion is correct; this is a documentation note, not a bug.

7.5 GAP: KL estimator — TRL default is k3, not k1

The TRL docs show the KL approximator formula:

D_KL[π_θ || π_ref] = π_ref/π_θ - log(π_ref/π_θ) - 1

This is the k3 estimator (Schulman et al., 2020). ADR-008 claims Composer 2.5 uses k1 (-log r = log π_θ/π_ref). The ADR notes this as an OPEN item. Since beta=0.0 by default in TRL, the KL term is disabled and this doesn't affect training unless beta>0. However, composer_trainer.py implements the k1-in-reward path via kl_in_reward.py — this is a separate mechanism from TRL's in-loss KL. Verify: the k1-in-reward path computes log(π_θ/π_ref) at reward time and folds it into advantages, while TRL's in-loss k3 term (when beta>0) would add a different term. If both are enabled simultaneously, they would double-count KL. The safe configuration is: k1-in-reward active, beta=0 (TRL in-loss KL disabled). The code appears to do this but there's no explicit assertion that beta=0 when kl_in_reward=True.

7.6 MISS: num_iterations=1 narrowed claim

ADR-008 acknowledges that num_iterations=1 controls GRPO inner-loop reuse, not dataset-level epochs. Primary source confirms: "Number of iterations per batch (denoted as μ in the algorithm)." The ADR's narrowed claim is correct.

7.7 MISS: TRL default optimizer is adamw_torch_fused, not adam

ADR-008 has an OPEN item: "Adam is claimed but optim is not set." The GRPOConfig docs show:

optim: transformers.training_args.OptimizerNames | str = 'adamw_torch_fused'

Default is adamw_torch_fused (AdamW with fused CUDA kernel), not plain adam. If Composer 2.5 uses Adam (without weight decay), the ADR's open item remains relevant: set weight_decay=0.0 and optim="adam" explicitly to match. The default AdamW has weight decay (though weight_decay=0.0 is already the GRPOConfig default, making it numerically equivalent to Adam in this specific case). However, adamw_torch_fused ≠ adam in terms of the optimizer implementation; to be precise, set optim="adamw_8bit" or optim="paged_adamw_8bit" (memory efficient) or just optim="adam_torch" if plain Adam is intended.

---

8. verl uni-agent — A New Development Not in Research/04

The verl README (May 2026): "uni-agent is released: a unified agent framework to build, run, and train LLM agents at scale, built on top of verl." This is a post-cutoff development (research/04 was written May 2025) that the repo has not incorporated. uni-agent could be the production-ready path for multi-turn agentic RL with verl, potentially superseding the lower-level AgentLoop/AsyncServer integration that ADR-006 contemplates.

Implication for the repo: Before committing to a custom VeRL AgentLoop integration, evaluate whether uni-agent already provides the FeatureDeletionEnv integration pattern out of the box. This could significantly reduce the engineering surface.

---

9. Sandboxing Recommendation Gaps

9.1 The SWE-MiniSandbox approach is not referenced anywhere in the repo

arXiv:2602.11210 (Feb 2026) directly addresses the production gap: container-free RL training with 5% disk usage and 25% env setup time vs containers. The paper's "kernel-level mechanisms" (likely Linux namespaces + cgroups without a full container runtime) with pre-caching is directly applicable to FeatureDeletionEnv at scale. The repo's sandbox design doesn't reference this work.

9.2 The repo's docker_sandbox.py is production-blocking for RL at scale

Per-task Docker container boots at GRPO scale (G=8 completions/prompt, B=4 per-device batch, many workers) means O(B*G) = O(32) container launches per training step. Without pre-warming or snapshot-based fast boot, this is the dominant latency. The gVisor RuntimeClass approach (negligible overhead, per the AWS article) or SWE-MiniSandbox's kernel-namespace approach are both faster alternatives.

---

10. Summary of Critical Findings

Finding	Severity	Affected Files
VeRL async agent loop is EXPERIMENTAL, not "first-class" stable	MEDIUM — overclaim	research/04 §1.5, ADR-006
TRL environment_factory (multi-turn) not in any ADR	MEDIUM — miss	ADR-008, env.py
k1-in-reward + beta=0 assertion missing (double-KL risk if beta>0 ever set)	MEDIUM — correctness gap	composer_trainer.py, ADR-008
optim default is adamw_torch_fused, not adam	LOW — fidelity gap	ADR-008 OPEN item
TRL loss_type defaults to "dapo" (not GRPO), correctly handled	INFO — confirmed correct	ADR-008, make_dr_grpo_config
env.py::reward_fn single-submit path is dead end for tree-of-work	HIGH — architecture gap	env.py, no ADR exists
uni-agent (verl, May 2026) not evaluated — may supersede custom AgentLoop	MEDIUM — miss	ADR-006
SWE-MiniSandbox approach not referenced (5% disk, 25% setup time)	MEDIUM — miss	sandbox.py, docker_sandbox.py
EKS Managed Node Groups incompatible with Kata+Firecracker (nested virt)	INFO — production gotcha	no ADR

---

11. Migration Path for Multi-Turn Agentic RL (Honest Assessment)

The current repo architecture (TRL reward_fn with single-submit fallback) is:

Phase 1 — GRPO on completions (current): The model generates a single diff/completion, the reward function grades it. This is viable, shippable, and correct. The TRL host is appropriate. No migration needed for this phase.

Phase 2 — Multi-turn FeatureDeletionEnv (agentic GRPO): The model takes tool-call steps (bash, file edits, test runs). Reward at trajectory end. Migration options in order:

1. TRL environment_factory adapter (experimental, weeks of work): Wrap FeatureDeletionEnv as a TRL environment. Methods become tools. Blocking GPU during sandbox execution — OK for small scale (≤8 GPUs), not for high parallelism.

2. TRL rollout_func (experimental, 1–2 weeks): Custom rollout that drives the env loop, serializes trajectories. Full control; TRL handles GRPO update.

3. VeRL AsyncServer + FeatureDeletionEnv as SandboxFusionTool adapter (2–4 weeks): GPU not blocked during sandbox calls. Required for tree-of-work fan-out at scale. The repo's ADR-006/ADR-008 have this on the roadmap but it's not implemented.

Phase 3 — Tree-of-Work (MCTS, multi-branch): This REQUIRES verl's async architecture. TRL cannot support N parallel branches per prompt with GPU-efficient execution. The uni-agent framework on top of verl should be evaluated first before building a custom AgentLoop integration.

---

Sources: TRL docs fetched 2026-06-10 (huggingface.co/docs/trl/en/grpo_trainer); vLLM co-locate blog (huggingface.co/blog/vllm-colocate); verl README (github.com/volcengine/verl/blob/main/README.md); SWE-MiniSandbox arXiv:2602.11210v5; AWS Builder Center EKS sandboxes article. Repo files: research/04, research/03, ADR-006, ADR-008, env.py, composer_trainer.py, recipes/prime_rl/composer_loss.py.