Hugging Face's logo

Codeseys
/
composer-replication-framework

Reinforcement Learning
Transformers
English
post-training
distillation
agentic-coding
composer-2.5
cursor
kimi-k2
grpo
dapo
diloco
openenv
trl
verl
research
methodology
Model card Files
xet
 Community
composer-replication-framework / research /03-monarch-torchforge-openenv.md
Codeseys's picture
Codeseys
Initial commit: Composer 2.5 Replication Framework — research synthesis
7165832
preview code
|
Raw
History Blame Contribute Delete
19.6 kB

Monarch + TorchForge + OpenEnv for RL Post-Training

This note surveys Meta’s PyTorch-native post‑training stack—Monarch (distributed actor framework), TorchForge (RL post‑training library), and OpenEnv (open environment standard)—with a focus on applicability to a production RL post-training pipeline. It covers what each component provides, how they compose, production maturity, and a comparison with VeRL, TRL, and OpenRLHF.

Stack overview

  • Monarch (pytorch/monarch): A single‑controller, mesh‑centric distributed programming framework for PyTorch. Exposes the cluster (hosts→processes→actors) as programmable arrays with fast actor messaging, RDMA data plane, and distributed tensors. It targets heterogeneous, asynchronous ML workflows (e.g., RL post‑training) where orchestration logic is awkward in SPMD-only models. [Docs] [Blog]
  • TorchForge (meta-pytorch/forge): A PyTorch‑native RL post‑training library built on Monarch. Provides service/actor abstractions for common RL components (generator/inference, trainer/learner, rewarders, stores) and ships reference recipes (SFT, GRPO). Integrates with vLLM for rollout generation, TorchTitan for training, and TorchStore for fast weight/tensor exchange. Note: as of 2026, the repo states development is paused and LLM training is consolidating into TorchTitan. [Repo] [Blog]
  • OpenEnv (meta-pytorch/OpenEnv + Hub): A standard and hub for agentic/RL environments—typed reset/step/close API, WebSocket transport, Dockerized isolation, and MCP (Model Context Protocol) tool-calling integration. Environments publish to a Hugging Face Hub collection; trainers (TRL, TorchForge, VeRL, Unsloth) consume via a stable client without per‑env adapters. [HF blog] [Spec/RFCs] [Repo] [TRL guide]

Where this shines:

  • A coherent PyTorch‑native control plane (Monarch) plus post‑training orchestration (Forge) and a portable environment substrate (OpenEnv) reduce glue code and make async, tool‑using RL feasible at scale.
  • Monarch’s separation of control plane (message passing, supervision) and data plane (RDMA buffers, distributed tensors) is well aligned with disaggregated RL stacks—high‑throughput inference and weight sync paths can be optimized independently of controller logic.

Monarch deep‑dive

What it is

  • Single‑controller model: you write one Python program (the controller) that orchestrates distributed resources directly; the cluster is exposed as structured meshes you can slice/index like arrays.
  • Key abstractions:
    • ProcessMesh: an array of processes (often 1 proc/GPU) across hosts.
    • ActorMesh: a collection of stateful actors spawned onto the process mesh; vectorized messaging across all/slices.
    • RDMA buffers and data plane: register any CPU/GPU memory and perform one‑sided transfers (libibverbs) for zero‑copy paths; integrates with distributed tensor operations.
    • Distributed tensors: PyTorch‑native DTensor integration so actors operate on sharded tensors that "feel" local.
    • Supervision trees: fault handling modeled after actor systems; fail‑fast by default with opt‑in, scoped recovery.
    • Lower‑level runtime: hyperactor (Rust) underpins actor messaging and supervision; hyperactor_mesh provides vectorized actor operations.
  • Environments: local dev server, multi‑GPU nodes, Kubernetes jobs (via monarch‑kubernetes), and HPC clusters.

Why not Ray Actors?

  • Ray is a general distributed runtime (actors, tasks, object store, autoscaler) across many domains. Monarch is PyTorch‑native and oriented around meshes of processes/actors with tight integration to DTensor and an explicit RDMA data plane.
  • Programming model: Monarch’s "program clusters like arrays" meshes and single‑controller orchestration feel like NumPy/PyTorch over clusters; Ray’s API is broader but less tensor/mesh‑centric.
  • Data movement: Monarch explicitly separates a lightweight control plane from a high‑performance data plane (RDMA, direct GPU‑GPU); Ray relies on its object store and networking stack.
  • Fit for post‑training: RL pipelines often need orchestrated SPMD components (trainer FSDP, inference TP/PP, rewarders) plus asynchronous control; Monarch’s controller + meshes model this cleanly.

Evidence and references

  • Intro + model: "Introducing PyTorch Monarch" (2025‑10‑22), "Monarch: an API to your supercomputer" (2026‑04‑08) and v0.5 docs detail ProcessMesh/ActorMesh, supervision, and RDMA/data‑plane separation.
  • Activity: ~1k stars, active releases through v0.4/v0.5, K8s support added, many contributors; used for agentic development, telemetry (distributed SQL), and even as a VeRL backend in validation experiments.

Caveats

  • Monarch is powerful but new; while the programming model is minimal, you still craft orchestration code. Higher‑level libraries (Forge) reduce that, but Forge’s development pause (see next section) is relevant.

TorchForge recipes & API

What Forge ships

  • Purpose: "focus on algorithms, not infra"—service abstractions built on Monarch actors.
  • Reference recipes:
    • SFT quickstart (Llama/Qwen variants).
    • GRPO end‑to‑end (Qwen3 1.7B/8B/32B reference configs), including multi‑node scale demos.
  • Architecture and integrations:
    • Generator (policy inference): vLLM‑backed service/actors for high‑throughput autoregressive generation; can run as colocated services or as external vLLM servers.
    • Trainer/Learner: Trainer actors running on TorchTitan (FSDP, TP/PP/CP) to update weights; supports async and synchronous coordination patterns.
    • Rewarders: reward model services/actors; Forge blogs highlight RLVR setups with Weaver‑style verifiers as drop‑in reward sources.
    • TorchStore: RDMA‑accelerated tensor/weight exchange to keep generators near‑on‑policy (direct GPU‑GPU state_dict transfers; resharding support).
    • OpenEnv: environments are consumed via a standard client; tool‑calling environments (MCP) supported through OpenEnv, not bespoke adapters.

Developer experience

  • Single config and Python entrypoint spin up a job where the controller orchestrates Generator/Trainer/Rewarders as ActorMeshes.
  • Service abstractions manage:
    • Spawning/placement across nodes
    • Load balancing and routing
    • Fault tolerance and retries
  • Explicit toggles for synchronicity (sync PPO‑like loops ↔ fully async off‑policy) without rewriting rollout logic.

What’s included vs missing

  • Included out of the box: SFT, GRPO; end‑to‑end RLVR demo with Weaver (verifier ensemble) at 512‑GPU scale in the blog; vLLM integration; TorchTitan trainer; TorchStore weight sync.
  • Not first‑class in the public materials: built‑in DPO/PPO/ORPO recipes (though PPO‑like sync is described conceptually), SGLang integration (VeRL supports SGLang, Forge highlights vLLM), or an extensive cookbook (tutorials "coming soon" in docs).

Status and activity

  • Repo banner: "Development paused—LLM training consolidating in TorchTitan." Last pushes in 2026, ~685 stars, 100+ open issues; examples and CI present.
  • Takeaway: Useful as a reference of patterns built on Monarch and TorchStore; for greenfield, plan to lean more on TorchTitan (training core) + OpenEnv + TRL/VeRL for algorithm coverage.

OpenEnv protocol

Core idea

  • OpenEnv standardizes how agents/trainers interact with real or simulated environments using a typed, Gymnasium‑like API: reset(), step(), close(), state(). Observations/actions are schemas (dataclasses), enabling type safety and IDE support.
  • Transport and isolation: environments run as servers—WebSocket is the default (supports many concurrent sessions per container); HTTP control plane exists for orchestration; Dockerized packaging for reproducibility and sandboxing.
  • MCP integration: RFC‑003 maps MCP tool list/call to OpenEnv actions so environments can expose tools via the Model Context Protocol. This supports tool‑calling agents and ML trainers with the same environment surface.

Hub and publishing flow

  • Authors publish an environment (Docker image + Python client) to the Hugging Face OpenEnv Hub. Users:
    • Inspect tools, schemas, and try environments as a Human Agent in‑browser.
    • Connect trainers (TRL, TorchForge, VeRL, Unsloth) by referencing the Hub ID—no per‑env adapters.
  • Scaling: documented patterns and benchmarks show 100s to 10Ks of concurrent sessions by switching from HTTP (1:1 session/container) to WebSocket multiplexing and scaling containers behind Envoy.

Tool‑calling, async, and harnesses

  • MCP tools are exposed safely alongside the environment’s RL API, with reserved name checks (not allowing reset/step/state as tools) to preserve orchestration boundaries.
  • RFC‑005 adds “agentic harness” integration: some envs wrap a full agent harness (e.g., OpenClaw). Production endpoints stream harness events; training keeps episode control by mapping turns to step() transitions.

Adoption signals

  • HF launch blog (Meta × HF) with examples; TRL has a first‑party OpenEnv integration guide; OpenEnv repo ~1.5–2k stars, active RFCs and releases; third‑party writeups (e.g., Turing’s calendar environment) and community envs (games, coding, REPL, web nav) on the Hub.

The combined pipeline (Monarch + TorchForge + OpenEnv)

A canonical post‑training topology looks like:

  • Controller: Monarch single Python program orchestrating meshes.
  • Generator service (ActorMesh): vLLM‑backed policy inference over prompts from datasets or environments; can be colocated or external microservices.
  • Environments: OpenEnv servers (Dockerized, WebSocket) providing tool‑using or simulator environments. Generators interact via OpenEnv client; for tool‑calling flows, the same environment exposes MCP list/call mapped to actions.
  • Rewarders: reward model(s) or verifiers (e.g., Weaver) as services. Reward functions can be synchronous or delayed (RFC‑004 delayed rewards).
  • Trainer (ActorMesh): TorchTitan‑powered learner updating the policy (FSDP/TP/PP/CP as needed).
  • Weight/tensor sync: TorchStore for state_dict exchange and DTensor‑aware resharding; Monarch RDMA paths provide direct GPU‑to‑GPU sync to reduce iteration latency.

Operational considerations

  • Synchronicity: pattern toggles between sync PPO‑style loops (tighter on‑policy, lower throughput) and async off‑policy (higher throughput, some staleness). Forge surfaces this without reworking rollout code.
  • Inference plane: vLLM usually runs as separate pods, discoverable by the controller; can also run in‑process for small scales.
  • Reward serving: either colocated fast RMs (transformers classification heads) or verifier ensembles (e.g., Weaver) via RPC. Monarch meshes and services route traffic intelligently.
  • Telemetry: Monarch integrates a distributed SQL telemetry plane for introspection across actors (useful in debugging coordination pathologies—queue depth, policy staleness, etc.).

Reference: PyTorch blog post shows Forge + Weaver at 512‑GPU scale for RLVR, with Monarch handling coordination and TorchStore accelerating weight sync.

Comparison vs VeRL / TRL / OpenRLHF

Criteria and synthesis

  • Programming model

    • Monarch + Forge: single‑controller, actor/mesh orchestration in Python; services abstract placement, retries, routing. Tight PyTorch/DTensor/RDMA integration.
    • VeRL (HybridFlow): hybrid model—single‑controller logic with multi‑controller efficiency; built on Ray but exposes a clean single‑controller interface; can run with vLLM/SGLang. Mature production framing; strong community and docs.
    • TRL: library‑first, Trainer APIs (GRPO, PPO [experimental], Online DPO, DPO, Reward modeling, SFT). Integrates with vLLM; now has OpenEnv integration to drive stateful envs via environment_factory. Minimal infra; you supply the orchestration.
    • OpenRLHF: PPO‑style RLHF focus; strong PPO pipelines and examples; less emphasis on stateful, tool‑using environments; infra glue typically on users.

  • Algorithm coverage

    • Forge: SFT + GRPO references; PPO described as synchronization pattern but not a first‑class shipped recipe; no built‑in DPO/Online DPO.
    • VeRL: PPO/GRPO and more; productionized alignment/TRL variants; broader set of recipes; integrates RMs and multiple inference engines.
    • TRL: very broad—SFT, GRPO, PPO (exp), Online DPO, DPO, reward modeling, etc.
    • OpenRLHF: strong PPO RLHF, some preference‑optimization variants via community forks.

  • Environment integration

    • Forge: consumes OpenEnv environments; tool‑calling via MCP thanks to OpenEnv; demoed with coding sandbox and others.
    • VeRL: OpenEnv‑compatible (via Hub clients) and has its own env adapters historically; strong ecosystem around vLLM/SGLang rollouts.
    • TRL: first‑party OpenEnv integration guide with GRPOTrainer; clean developer UX.
    • OpenRLHF: generally Gym/Gymnasium‑style or custom envs; can use OpenEnv with adapters but not first‑party yet.

  • Scale ceiling and performance

    • Monarch + Forge: RDMA data plane + TorchStore for zero‑copy weight/tensor sync; meshes support thousands of GPUs; validated RLVR at 512 GPUs.
    • VeRL: proven scale; Ray scheduling maturity; broad industry adopters and talks; benchmark claims of high throughput; supports vLLM/SGLang/in‑proc HF.
    • TRL: depends on your training backend (Deepspeed, Titan, PEFT) and rollout engine (vLLM). Good scaling stories but orchestration is user‑owned.
    • OpenRLHF: similar—performance comes from chosen backends; less built‑in orchestration.

  • Production readiness

    • Monarch: active development, releases, docs—credible but new; requires engineering buy‑in to its model.
    • Forge: marked "development paused; consolidating into TorchTitan"—use patterns as reference; expect Titan + TRL/VeRL for go‑forward.
    • OpenEnv: fast‑moving but already widely referenced (HF blog, TRL integration, RFCs, Hub adoption). Clear isolation + transport story; scaling guides published.
    • VeRL: strong community traction and ecosystem of integrations; production‑minded design (HybridFlow); multi‑engine support.
    • TRL: de‑facto OSS standard for post‑training algorithms; v1 emphasizes robustness; extensive examples and docs.
    • OpenRLHF: widely used for PPO RLHF; simpler but narrower API.

Fit for our framework

  • Using Monarch as the control substrate: Feasible and attractive if we want a single‑controller Python program to coordinate learners, generators, rewarders, and environment clients with strong fault handling and a high‑performance data plane. Monarch does not conflict with our gradient synchronization method (e.g., DiLoCo/local‑SGD)—those live inside the trainer (TorchTitan/Deepspeed/etc.). Monarch sits above as orchestration.
  • TorchForge as the RL layer: Good as a pattern reference for service abstractions, but given the "development paused" status, we should not bet on Forge as a moving foundation. Instead:
    • Prefer TorchTitan for the training core (supports FSDP/TP/PP/CP and context parallelism),
    • Pair with TRL or VeRL for algorithm coverage (GRPO, PPO, DPO, reward modeling),
    • Keep OpenEnv as the environment substrate,
    • Re‑implement needed Forge‑like services (generator/rewarder/store) using Monarch where it adds value, or start with VeRL’s backend and migrate selectively.
  • DiLoCo compatibility: No inherent conflict. DiLoCo controls intra‑trainer gradient sync; Monarch/VeRL/TRL govern inter‑component orchestration. If we keep Titan + DiLoCo inside the learner and use Monarch to coordinate rollout and envs, they are complementary.
  • Inference engine: vLLM is first‑class across Forge/VeRL/TRL; SGLang is supported in VeRL; nothing prevents adding SGLang actor services in a Monarch stack if desired.

Recommended adoption path

  1. Standardize environments via OpenEnv (use Hub IDs in all experiments).
  2. Choose training core: TorchTitan (preferred) or Deepspeed, and decide on algorithm library: TRL for breadth or VeRL for a production‑oriented RL stack with hybrid single‑controller flavor.
  3. Use vLLM rollouts as an external service initially; add Monarch‑managed generator/rewarder services only if we need advanced placement/fault semantics or RDMA‑accelerated weight sync with TorchStore.
  4. If we want Monarch, adopt it incrementally—start by running Titan trainers under Monarch Job API + ActorMeshes for rollout and rewarders; keep algorithm logic in TRL/VeRL.

Open questions we should validate

  • Monarch vs Ray swap‑costs in downstream libraries: PyTorch notes that even when a framework exposes a clean single‑controller interface, Ray API usage may surface elsewhere—how invasive is a Monarch backend in VeRL/TRL codepaths we care about?
  • Weight freshness vs throughput: With TorchStore + RDMA, what iteration times do we achieve for 7B/32B policies at 8–32 generators? What update cadence avoids excessive off‑policyness while keeping generators saturated?
  • Reward serving patterns: For verifier‑heavy tasks (math/code), what is the optimal topology—RM colocated per generator vs shared verifiers; how do we saturate them without becoming the bottleneck?
  • Environment scaling: For target benchmarks (e.g., web nav + coding), can we reach 5–10k concurrent env sessions using the documented WebSocket multiplexing + Envoy patterns; does the Hub infra suffice or do we need cluster‑native deployments from day one?
  • Telemetry and observability: Monarch’s distributed SQL telemetry sounds promising; do we integrate this or rely on W&B + Prometheus? How painful is cross‑actor correlation in practice?

Sources

Appendix: quick status snapshot (as of May 2026; see linked pages for live numbers)

  • Monarch: ~1k stars, v0.4/v0.5 docs, K8s support, active commits through Apr 2026.
  • TorchForge: ~685 stars, last pushes in 2026, readme notes development paused (consolidate in TorchTitan).
  • OpenEnv: ~1.5–2k stars, active RFCs (MCP, delayed rewards, harnesses), v0.3.0 released May 2026, HF Hub org and catalog live.