composer_replication/recipes/monarch/monarch_actor_layout.md

Monarch actor mesh — design for hosting the framework's training topology

Status: Design + skeleton. Real Monarch integration is post-replication work (ADR-006 explicitly defers it to v0.2+).

ADR: 006

What Monarch is

Monarch (https://github.com/meta-pytorch/monarch, BSD-3) is Meta's actor- mesh runtime — a thin coordination layer over Python processes that lets you describe a training topology as a graph of typed actors, then run that topology on top of any cluster manager (k8s, Slurm, raw ssh).

Per ADR-006, Monarch is the only Meta PyTorch agentic-stack component that's actively shipping (v0.4.1 stable, v0.5 dev daily) and not paused. TorchForge, the original "agent" piece, is paused per its own repo banner.

Why Monarch fits the framework's design

The framework already has an N-actor topology even without Monarch:

• Trainer (channel 1: GRPO; channel 2: SDPO; channel 3: trace-replay DPO)
• Generator (rollout / vllm)
• Rewarder (RLVR test runner / verifiers protocol)
• N teachers (channel 3: external OpenRouter calls)
• DiLoCo replicas (N copies of trainer, syncing via object store)

PRIME-RL gives us the trainer/generator/rewarder split for free. Monarch takes that further: each of those becomes a Monarch actor, and the framework gains:

1. Heterogeneous executor support — actors run wherever Monarch's backend places them (Modal, k8s, on-prem cluster). Composes naturally with our ServerlessExecutor Protocol.
2. Failure recovery — Monarch handles actor crashes + restarts; the framework's DiLoCo state is durable in object storage, so a restarted trainer replica can resume from the last outer round.
3. Hot-swap of actor implementations — switch teacher backends from "OpenRouter" to "local vllm" by changing one Monarch actor binding.

Actor topology (proposed)

┌───────────────────────────────────────────────────────────────┐
│                  ComposerReplicationMesh                       │
│                                                                │
│   ┌──────────────┐   ┌──────────────┐   ┌──────────────────┐  │
│   │  Trainer × N │←─│  Generator    │←─│  Rewarder         │  │
│   │  (DiLoCo     │   │  (vllm)       │   │  (verifiers)      │  │
│   │   replicas)  │   └──────────────┘   └──────────────────┘  │
│   └──────┬───────┘                                              │
│          │                                                      │
│          │   Channel 2: same-model hint-conditioned forward    │
│          │   Channel 3: cross-model OpenRouter teachers        │
│          ▼                                                      │
│   ┌──────────────┐                                              │
│   │ TeacherPool  │ ── OpenRouter (Claude, GPT, DeepSeek, ...) │
│   │ (channel 3)  │                                              │
│   └──────────────┘                                              │
│                                                                │
│   ┌──────────────────────────────────────────────────────────┐ │
│   │  ObjectStore (s3://, hf://, file://)                     │ │
│   │  · DiLoCo pseudo-gradients (round_N/rank_R.pt)           │ │
│   │  · Replay datasets (NormalizedDPOPair JSONL)             │ │
│   └──────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────┘

Mapping to Monarch primitives

from monarch import Actor, mesh, endpoint

class TrainerActor(Actor):
    """Hosts the GRPO trainer + composer 3-channel loss."""
    @endpoint
    async def train_outer_step(self, batch_id: int): ...

class GeneratorActor(Actor):
    """vllm rollout server — generates trajectories on demand."""
    @endpoint
    async def rollout(self, prompts: list[str]) -> list[str]: ...

class RewarderActor(Actor):
    """Runs verifiers protocol — RLVR-style test execution."""
    @endpoint
    async def score(self, completions: list[str]) -> list[float]: ...

class TeacherPoolActor(Actor):
    """Channel 3 — OpenRouter calls to N external teachers."""
    @endpoint
    async def replay(self, states: list[dict]) -> list[dict]: ...

# Topology
trainers   = mesh.spawn(TrainerActor, n=4, gpu="A100")
generator  = mesh.spawn(GeneratorActor, n=1, gpu="A100")
rewarder   = mesh.spawn(RewarderActor, n=1, gpu=None)
teachers   = mesh.spawn(TeacherPoolActor, n=1, gpu=None)

Status of this directory

• monarch_actor_layout.md — this file (design)
• actors.py — skeleton actor definitions; do not import without monarch installed
• composer_mesh.py — composition glue; not yet implemented

Open questions (deferred to v0.2)

• Does Monarch v0.5's Slurm backend hand-shake cleanly with HF Jobs? (HF Jobs runs each "job" as an independent container; Monarch wants to manage the lifecycle. Possible mismatch.)
• Can the TrainerActor host the framework's ComposerReplicationTrainer unmodified, or does it need to be split into step_init / step_compute endpoints to fit Monarch's async actor model?

References

• Monarch repo: https://github.com/meta-pytorch/monarch
• ADR-006: docs/adrs/ADR-006-rl-frameworks.md
• Reconnaissance: docs/research/RL_FRAMEWORKS_LANDSCAPE.md § Monarch