Online Disaggregation Roadmap

Live disaggregated training: a rollout pool runs the frozen target model autoregressively over a fixed prompt set and streams its hidden states across the network into a separate trainer pool. This folds PR #618's online roadmap into the consolidated tree and applies the shared decisions: train-with-decode only (the online target is frozen, so the streamed data is independent of draft weights — there is no staleness and nothing to re-sync, and the whole weight-sync / hot-update / on-policy axis is dropped here, kept only as the explicit out-of-scope note below); and Ray is OPEN/undecided for the O2 scale-out layer — presented as a candidate with a decision gate against a lighter torchrun/rcli alternative, not as a commitment.

Substrate is canonical and shared with the sibling docs: SampleRef (control plane, tensor-free)

FeatureStore (tensors: Local / SharedDir / Mooncake) + FeatureDataLoader → TrainBatch. There is no separate hidden-state stream source of truth — FeatureDataLoader over SampleRef + FeatureStore is the stream; online/offline/disagg vary only in the ref source + the FeatureStore backend, shielded from training. See domain-refactor for the domain Trainer/managers that wrap the runtime training seam and for Phase B's TargetEngine that O1.3 depends on, eval-and-breadth for the orthogonal eval/algorithm track, and the top-level plan.

What we keep (do not rebuild)

The data plane is competitive — arguably ahead of the comparables on rigor (three backends behind one contract, contract-tested, with a durability seam). The gap is the orchestration + live-engine + async control loop. Reuse, do not rewrite:

Data plane — MooncakeFeatureStore zero-copy (data_plane/mooncake_store.py) with hard-pin replication and per-generation key sets; LocalFeatureStore / SharedDirFeatureStore behind the same FeatureStore contract (data_plane/feature_store.py).
Control-plane vocabulary — SampleRef (tensor-free; assert_no_tensors in contracts.py); SampleRefQueue with lease/ack and the reserved partition seam (data_plane/sample_ref_queue.py); DataFlowController commit/lease/ack + reconcile_on_restart (control_plane/controller.py); the MetadataStore ABC with InMemoryMetadataStore / SQLiteMetadataStore (control_plane/metadata_store.py).
Streaming seam — StreamingRefChannel / StreamingRefQueue (data_plane/streaming_ref_channel.py) already carry committed refs cross-process with in-flight accounting and a close-and-drain protocol.
Trainer seam — TrainerCore / DraftTrainStrategy / TrainingBackend + StepContext (training/), wrapped (not replaced) by the domain Trainer.

Comparables (from #618)

Axis	SpecForge (today)	TorchSpec	DeepSpec
Data plane (Mooncake zero-copy, tensor-free metadata)	At parity / cleaner — 3 backends, one contract, contract-tested, durability seam	`EagleMooncakeStore` (per-tensor keys)	disk cache, no streaming
Orchestration / scale-out	GAP — library + composable builders, controller in-process	Ray actors + placement groups + async controllers + `EnginePool`	single 8-GPU node, no Ray
Live generation (train-with-decode)	GAP — in-process generator stub today; no server	live autoregressive rollout streaming	offline regen-to-disk
Cross-pool backpressure	policy exists, off by default	pool-byte feedback + capacity await	n/a
Online weight sync / staleness	n/a — out of scope (frozen target)	also absent (not needed)	absent
Fault tolerance	`reconcile_on_restart` + checkpoint resume	checkpoint resume (≈ parity)	n/a

Read: the data plane is at or ahead of parity; the competitive gap is orchestration + the live engine + the async control loop. DeepSpec's separate edge (breadth + eval) is orthogonal and lives in eval-and-breadth.

O1.1 — Shared cross-process control plane · size M · CPU · status: in-review

Goal Producer and consumer attach to one durable control plane instead of each building its own InMemoryMetadataStore, so commit/lease/ack/dedup and restart-reconcile work across processes (~PR #624).
Target state A separate producer and consumer share commit/lease/ack via a durable store; the "single-host in-process index" caveat is gone for the shared-store path; reconcile_on_restart is wired into the launcher and resume=True re-streams the committed-but-unacked tail.
Implementation
- control_plane/metadata_store.py — SQLiteMetadataStore is the end-to-end dev/single-host durable tier behind the existing MetadataStore ABC; a thin RedisMetadataStore is a later subclass behind the same interface (chosen at O2 if multi-node demands it).
- data_plane/sample_ref_queue.py — SampleRefQueue operates over the shared store rather than purely in-process.
- control_plane/controller.py — DataFlowController(metadata_store=...); the producer's commit_samples and the consumer's lease_train_refs / ack_train_refs land in the same durable store; reconcile_on_restart resolves committed-but-unacked refs on attach.
- launch.py — build_disagg_online_producer / build_disagg_online_consumer already accept metadata_store / metadata_db_path (O1.1 wiring); _resolve_metadata_store points both halves at the same durable path, and the consumer's resume=True reconciles before training.
Tests / gates Two controllers over one store: producer commits, consumer leases/acks across processes; restart reconcile hands already-trained ids back as skip_ids and re-streams the tail; at-least-once dedup. tests/test_runtime/test_disagg_online_launch.py, tests/test_runtime/test_disagg_online.py.
Done when A separate producer and consumer share commit/lease/ack via the durable store, and the single-host-in-process-index caveat is removed for the shared-store path.

O1.2 — Async streaming loop + one-process builder · size M · CPU / small-GPU · status: in-review

Goal Replace synchronous drain-then-fit with an async streaming loop, using the existing in-process generator as a stub so this milestone carries no engine risk (~PR #625).
Target state End-to-end live 1+1 run with in-process generation: producer pulls prompts → generates → put + commit ref; consumer leases → get → trains → acks at the optimizer step; a bounded sample pool throttles via the channel's in-flight watermark.
Implementation
- launch.py — the composable builders already exist today: build_disagg_online_producer returns (workers, drive_producer); build_disagg_online_consumer returns (trainer, loader); run_disagg_online_interleaved runs the producer on a background thread while trainer.fit consumes on the main thread, with symmetric hang-free shutdown (trainer-done sets should_stop; producer-done closes the channel; producer-raise closes the channel then re-raises). build_disagg_online_runtime (aliased build_disagg_online_eagle3_runtime) composes the two halves and dispatches to run_disagg_online_interleaved.
- The producer's drive_producer(should_stop=...) applies channel backpressure (pauses while channel.in_flight_remote() exceeds in_flight_high_watermark) and closes the channel on exit.
- In-process generator stub: SGLangAdapter.generate_features over the SpecForge Eagle3TargetModel.generate_eagle3_data path (inference/sglang_adapter.py), driven by RolloutWorker (inference/rollout_worker.py).
Tests / gates Tiny-model 1+1 online run: trainer consumes a streamed (not precomputed) ref set; metrics align with the offline baseline on the same prompts/seed. Interleaved shutdown is hang-free in all three orderings. tests/test_runtime/test_disagg_online_launch.py, tests/test_runtime/test_streaming_ref_channel.py.
Done when An end-to-end live 1+1 run drives training from an in-process generator over the streaming channel, with no precomputed features.

O1.3 — Live SGLang-server hidden-state capture · size L · GPU + engine · status: in review (PR #641, reforward transport; spike findings + upstream proposal in ./o13-capture-spike.md)

🔴 GATING RISK — run the capture spike first. The unknown is how far Eagle3TargetModel.generate_eagle3_data is from server-backed capture. The spike is the difference between O1.3 being days vs. a quarter; SpecForge is SGLang's own project so reuse is plausible (TorchSpec needed an SGLang codebase patch). Run it before/with O1.1.

Goal Replace the in-process generator with a real SGLang server emitting aux + final hidden states straight into MooncakeFeatureStore.
Target state A live SGLang server feeds training with zero precomputed features: server generates → hidden states land in the store → consumer trains; acceptance metrics match the offline baseline.
Implementation
- Depends on domain-refactor Phase B's TargetEngine abstraction (SGLangServerEngine) — O1.3 is the online consumer of that engine.
- inference/sglang_adapter.py — a server-backed FeatureSource (launch/attach an SGLang server) alongside the existing in-process SGLangAdapter; the capture hook is engine-side, writing aux + final hidden states to the FeatureStore. Reuse the CaptureConfig / verify_capture contract (inference/capture.py) so an aux-layer-id / width / target-dim mismatch fails loudly. See inference/sglang_patch_inventory.md for the patch surface.
- inference/rollout_worker.py — the producer uses the server-backed source; committed SampleRefs flow through the same StreamingRefChannel.
Tests / gates (gated GPU) Server generates → hidden states resident in the store → consumer trains; capture verification passes; acceptance vs. offline. tests/test_runtime/test_mooncake_store.py for the store path.
Done when A live SGLang server feeds training with zero precomputed features and acceptance metrics match the in-process / offline baseline.

O2 — Scale-out orchestration (Ray = OPEN) · size L · infra · status: later

Goal Move from 1+1 to N producers + M trainers with per-DP-rank leasing, cross-pool backpressure, and a managed streaming-pool lifetime — and decide the orchestration layer.
Target state Independent N producers and M trainers scale without re-leasing or dropping samples; a slow trainer throttles the producer (bounded memory, not OOM); the online MooncakeFeatureStore bugs that become reachable at scale are closed; the orchestration layer is chosen against an explicit gate.

Orchestration: Ray candidate vs. lighter alternative (decision gate)

Ray is a candidate for the scale-out orchestration layer, not a commitment and not a non-goal. It is likely necessary for multi-node N-producer/M-trainer scale-out and high-bandwidth isolated-pool runs (its actor/placement model maps cleanly onto independent rollout and trainer pools with separate GPU counts/sharding, matching the comparables' shape). The lighter alternative is torchrun + the existing rcli multi-node launch driving the already-composable build_disagg_online_producer / build_disagg_online_consumer builders across nodes, with the shared durable MetadataStore (O1.1) as the only cross-process coordination point — no actor framework.

Trade-off Ray buys dynamic placement, actor restart/health, and a uniform multi-pool control surface, at the cost of a heavy runtime dependency, a new failure surface, and packaging/operability overhead on the GPU boxes. torchrun + rcli keeps the dependency surface minimal and reuses the existing launch path, but leaves placement, fan-out, and restart as bespoke launcher logic.
Decision gate — adopt Ray only if the N+M / multi-node runs demonstrably need dynamic placement or per-actor restart that the static torchrun + rcli + shared-store path cannot deliver (e.g. heterogeneous pool sizes that must reshard at runtime, or actor-level fault recovery that bash-level relaunch cannot meet the availability target for). If a static launch over the shared store carries N+M with acceptable restart behaviour, stay on the lighter path. Re-evaluate at the first true multi-node scale-out run.

dp_partition leasing

Implementation data_plane/sample_ref_queue.py — the partition seam exists today: dp_partition(sample_id, num_partitions) computes a stable DP-rank assignment, and lease(..., partition=(index, num_partitions)) restricts a lease to a DP shard via _lease_partition_locked. The reserved partition_key producer-side routing hint is accepted-and-ignored today; turn on per-DP-rank leasing so every sample is leased exactly once across a reshard. control_plane/controller.py gains multi-trainer lease dispatch.
Tests / gates N+M run; every sample leased exactly once across a reshard; no double-train; changing num_partitions re-distributes the same committed pool without re-leasing or dropping.

Cross-pool backpressure

Implementation control_plane/backpressure.py exists and is off by default: BackpressureController with should_pause_prompts / cap_prompt_grant / cap_train_lease and a high/low watermark BackpressureConfig. Wire the controller into the dispatch loop so it returns pool-size feedback and blocks granting when the pool is full; the producer throttles instead of OOM-ing the store. control_plane/controller.py reports capacity.
Tests / gates A slow trainer makes the producer throttle (bounded store memory) rather than OOM.

Streaming-pool lifetime + known online MooncakeFeatureStore bugs

Implementation data_plane/mooncake_store.py — add consume-then-GC / TTL lifetime for the streaming pool, and close the two online-only bugs that become reachable at scale, both of which the shared metadata index (O1.1) unblocks:
- release-pending re-put gc-clobber — a re-put bumps the per-sample generation and removes the prior key set; a parked failed-free in _release_pending plus gc() can clobber the fresh blob. Needs a tombstone-then-free protocol tied to the shared index.
- cross-process abort tombstone — under a lease-deferred remove, producer.abort marks only its own _freed, so the bytes linger and a separate consumer (empty _freed) still resolves the aborted ref → stale. This is the expectedFailure test_cross_process_abort_under_lease_defer_is_known_gap in tests/test_runtime/test_mooncake_store.py:445; flip it to a hard assertion once the shared index lands.
Tests / gates Re-put-while-pending no longer deletes the fresh blob; cross-process abort blocks a separate consumer (the expectedFailure becomes a real assertion); consume-once streaming is correct cross-process.
Done when N producers and M trainers scale without re-leasing/dropping; trainer-behind-rollout is a bounded-memory backpressure signal; the streaming-pool bugs are closed; the orchestration decision gate has been evaluated against a real N+M run.

O3 — Hardening · size L · infra · status: later

Goal Make the live disaggregated path production-operable.
Target state RDMA path runs without per-op buffer churn; actor/process restart and health are handled; the run is observable.
Implementation
- RDMA buffer-pool registration in data_plane/mooncake_store.py — replace the per-op register_buffer / unregister_buffer around _store_put_tensor / _store_get_tensor with a registered pool before flipping DEFAULT protocol from tcp to rdma.
- Actor / process restart + health: restart the rollout/trainer/controller processes (under whichever orchestration O2 selects) with reconcile_on_restart driving recovery from the shared store; health endpoints on RolloutWorker.health / SGLangAdapter.health.
- Observability: pool depth, in-flight, lease/ack rates, backpressure snapshots (BackpressureController.snapshot), capture-verification counts.
Tests / gates Kill-and-restore a producer/trainer mid-run and confirm no lost or double-trained samples; RDMA path registers buffers once; metrics emitted.
Done when The live disaggregated run survives a process restart with no data loss and runs the RDMA path with one-time buffer registration.

Provenance, not weight sync

draft_weight_version is retained only as provenance metadata on each committed sample — it is recorded by RolloutWorker._put_metadata / _offline_metadata (inference/rollout_worker.py), threaded through contracts.py, the metadata store, the controller, and every FeatureStore backend's metadata round-trip. It exists for debugging/audit. There is no staleness gate, no version registry, and no consumer-side check keyed on it. Because the online target is frozen, the streamed hidden states are independent of draft weights.

Dependency graph

                          capture spike (gates O1.3)
                                   │
O1.1 ──▶ O1.2 ──▶ O1.3 ──▶ O2 ──▶ O3
 │                  │
 │                  └─ depends on domain Phase B TargetEngine (SGLangServerEngine)
 │
 └─ shared MetadataStore unblocks the O2 online-store bug fixes (re-put gc-clobber,
    cross-process abort tombstone)

eval-and-breadth track ── independent (orthogonal to disaggregation)

Explicitly out of scope

Draft-weight sync, weight-version registry, hot draft-update, staleness gate, WeightPublisher / WeightVersion / serving accept-length gate / ServingTrafficStream / serving-traffic capture — not needed for train-with-decode. The online target is frozen; the draft is not in the generation loop; the data is independent of draft weights, so there is no staleness and nothing to re-sync. draft_weight_version survives as provenance only (above).
True on-policy / draft-coupled training (draft proposes → target verifies → train on realized acceptance, requiring live draft-weight sync) — a separate research bet, not addressed here. No public framework ships it.
Elastic / dynamic engine scaling at runtime — engines are fixed at init; the comparables do the same.