# SpecForge Architecture Plan (Reconciled) > **Status**: Draft — reconciles the original "SpecForge Redesign Plan" (a from-scratch, > torchrun-native, HTTP-only target architecture) with the **landed DataFlow `runtime/` > spine** (control/data/inference/training planes, SampleRef + FeatureStore + Mooncake). > > **Supersedes** the previous redesign draft. That draft's detailed component sketches > (CheckpointManager, Evaluator, Pydantic config, MLA draft, SGLang export, train-with-decode) > remain valid as the **domain-layer target** and are preserved verbatim in > [`docs/redesign-draft-legacy.md`](docs/redesign-draft-legacy.md); this document > re-frames *where they sit* now that the DataFlow runtime exists, and corrects the one bet > the original draft got wrong for our actual workloads (see §6). --- ## 0. TL;DR — the decision There were two architecture efforts in flight for the same goal (organize SpecForge so new draft models / training modes / disaggregation compose cleanly without a `train_XXX.py` per method): - **The redesign draft** (this doc's predecessor): clean domain decomposition (`models/drafts` registry, `TargetEngine`, `HiddenStateStream`, `DraftTrainStrategy`, `Trainer`, config/eval/export/cli) — but **explicitly torchrun-native with no Ray/Mooncake**; disaggregation was "the target is an HTTP service" (`RemoteStream`). - **The DataFlow `runtime/` spine** (landed: the `dataflow-up-1..20` stack, M1–M7/O1): a metadata-only **control plane** + a tensor-carrying **data plane** (`SampleRef` + `FeatureStore` incl. **Mooncake** + `StreamingRefChannel` + durable metadata + backpressure + resume), with disaggregated producer/consumer and an interleaved online loop. **The fact that settles it:** there is a real **multi-node / >100 GB/s / isolated-pool** requirement (separate rollout and trainer pools, no shared FS / no viable HTTP path for the hidden-state volume). That **breaks the original draft's central bet** ("HTTP/gRPC to SGLang is sufficient; Mooncake only matters >100 GB/s which we never hit"). So the runtime's Mooncake control/data plane is **necessary, not premature**. **Decision: do not pick one — layer them.** - **Canonical substrate = the DataFlow `runtime/` control + data plane.** `SampleRef` (metadata on the control plane) + `FeatureStore` (tensors, Local/SharedDir/Mooncake) + `FeatureDataLoader` → `TrainBatch` is the canonical data path. online / offline / disaggregated converge at `SampleRef`. - **Front-end = the redesign draft's domain layer.** `TargetEngine` / `DraftTrainStrategy` / `Trainer` / `CheckpointManager` / `Evaluator` / config / eval / export / cli become the user-facing surface. The **canonical "stream" is `FeatureDataLoader` over `SampleRef` + `FeatureStore`** — we do **not** re-introduce a `HiddenStateStream` as a separate source of truth (the loader already plays that role). The online/offline/disaggregated variation lives in the *ref source* (`OfflineManifestReader` / `RolloutWorker` / `StreamingRefChannel`) + the *transport* (`FeatureStore`: Local for colocated, Mooncake/SharedDir for isolated pools), and is shielded from training by `FeatureDataLoader → TrainBatch`. (`RemoteStream`-over-HTTP is just one more ref source / engine — the light cross-node option, not the base abstraction.) This is, in the predecessor's own terms, pulling **feature #32 ("Mooncake streaming backend") forward from Phase 7 to now** because the requirement demands it — and implementing it with the already-built `runtime/` work, behind the clean domain interfaces. **Two scope decisions that bound this plan** (consolidating with the online-disaggregation roadmap, #618): 1. **Frozen target — no weight sync.** "Train-with-decode" means a *frozen* target streams hidden states; the draft is never in the generation loop. Weight sync / hot draft-update / weight-version registry / on-policy are **out of scope**; `draft_weight_version` is provenance only. (This removes the predecessor's W4 weight-lifecycle workload.) 2. **Ray is an open decision, not a non-goal.** It is a *candidate* for the O2 scale-out orchestrator (multi-node N-producer/M-trainer) — likely necessary, but not committed. Until the decision gate fires we keep the home-grown metadata-only control plane and add nothing. **Per-phase target/implementation detail lives in [`docs/roadmap/`](docs/roadmap/)** (domain, online-disaggregation [folds in #618], eval & breadth); this document is the architecture; the roadmap is the build order. --- ## 1. Why each side is right (and where each is incomplete) Two orthogonal slicings. The runtime is sliced by **plane** (how bytes/metadata move); the draft is sliced by **domain object** (draft / target / stream / strategy). They compose; they do not conflict. | Layer | Verdict | Rationale | |---|---|---| | **control plane** (`runtime/control_plane`) | **Keep — canonical. No predecessor equivalent.** | Metadata-only (every entrypoint runs `assert_no_tensors`), lease/ack/dedup/reconcile/backpressure behind a `MetadataStore` seam. This is exactly the cross-pool machinery the isolated-pool requirement needs, and the HTTP design has nothing like it. | | **data plane** (`runtime/data_plane`) | **Keep — canonical; it *is* the stream.** | `SampleRef` (metadata) ÷ `FeatureStore` (tensors) ÷ `FeatureDataLoader` (materialize) is a *finer* decomposition than the predecessor's coarse `HiddenStateStream`, and it's the better substrate for backpressure / lease-ack / resume / Mooncake. `FeatureDataLoader` over `SampleRef`+`FeatureStore` already plays the role plan.md gave `HiddenStateStream`, so **no separate `HiddenStateStream` source-of-truth is introduced** — `seek()`/prefetch land on the loader/ref-source; `FeatureStore` is the transport swap point. | | **inference plane** (`runtime/inference`) | **Converge to the predecessor's `TargetEngine`.** | Today it conflates "wrap a target engine" + "capture" inside `SGLangAdapter`, and is bound to `generate_eagle3_data` / EAGLE3 names. The `FeatureSource` Protocol already exists (good); the gap is a real `TargetEngine` abstraction (hf / sglang / **sglang_server** / custom) + de-EAGLE3-ifying. This is the layer that most borrows the draft. | | **training plane** (`runtime/training`) | **Keep the seam; fill the managers.** | `DraftTrainStrategy` / `TrainerCore` / `TrainingBackend` (+ the `StepContext` added for Domino) is *already* the predecessor's `training/strategies` shape — keep it. What's missing are the **managers**, not the seam: `no_sync()` accumulation, full optimizer/scheduler/RNG resume, `CheckpointManager` (rotation/best), `Evaluator`. | **Landed (merged — the spine):** - `runtime/` planes (M1–M7/O1): control/data/inference/training, `SampleRef`, `FeatureStore` (Local/SharedDir/Mooncake), `StreamingRefChannel`, durable `MetadataStore`, disagg producer/consumer + interleaved online loop. **In the stacked PRs #627/#628/#629 (validated, in review — not yet merged):** - **Composable launch** (`StrategySpec` registry + parameterized `launch.py`): adding a model is a spec entry, not a `build_*_runtime` family. eagle3 / **dflash** / **domino** all train end-to-end through one strategy-parameterized path (validated: 197 `tests/test_runtime` OK on H200). Domino added `StepContext{global_step, total_steps}` threaded through `forward_loss` — the one deliberate contract extension for schedule-dependent loss. **Explicitly not yet implemented in `runtime/`** (flagged in-source — `contracts.py`, `trainer.py`, `controller.py`, both `DESIGN.md`s, `runtime/README.md`): live frozen-target online capture from a real SGLang server (O1.3), full optimizer/scheduler resume, and `no_sync()` accumulation. These are the §3 gaps. (The in-source weight-publication NOTEs — `WeightVersion` / `WeightPublisher` / `update_draft_weights` — are **descoped**: the target is frozen, see §8.) --- ## 2. Target architecture ### 2.1 Canonical data path (unchanged from the runtime, this is the spine) ``` ref source ─┐ OfflineManifestReader │ (offline: re-iterable refs) RolloutWorker │ (online: produced into a SampleRefQueue) StreamingRefChannel │ (disaggregated: cross-process/pool stream) ▼ DataFlowController ── metadata only (SampleRef), assert_no_tensors │ FeatureStore ── tensors only: LocalFeatureStore (mem://, colocated) SharedDirFeatureStore (shared mount) MooncakeFeatureStore (RDMA, isolated pools) ▼ FeatureDataLoader (per_sample_transform + collate) ─► TrainBatch ▼ DraftTrainStrategy.forward_loss(batch, ctx) ─► TrainerCore / TrainerController ─► FSDP ``` ### 2.2 Domain layer on top (user-facing abstractions over the substrate) The substrate's source of truth is `SampleRef` + `FeatureStore` + `FeatureDataLoader` (§2.1). The domain layer does **not** re-introduce a `HiddenStateStream` as a parallel source of truth — `FeatureDataLoader` already *is* the stream. The domain layer is what user/training code sees: ``` PRODUCE (inference): TargetEngine ── the engine rollout wraps to produce features; the abstraction that replaces the EAGLE3-bound SGLangAdapter. HFTargetEngine / SGLangTargetEngine / CustomTargetEngine (in-process) SGLangServerEngine (frozen target as a live SGLang server, cross-node). ONE engine, two feature transports (the engine is identical; only WHERE features land differs): · capture transport — engine-side hook writes hidden states INTO a FeatureStore (Mooncake/SharedDir). This is W3 / online O1.3. · inline-HTTP transport — features serialized in the HTTP response, no shared store (RemoteStream-style). This is the light W3′ path, the one case features do NOT live in a FeatureStore. RolloutWorker ── drives a TargetEngine → writes tensors to FeatureStore → commits SampleRef to the control plane. Stays at the domain↔substrate seam. CONSUME (training): Trainer ── owns the lifecycle: loop / eval / checkpoint. WRAPS the TrainerCore/TrainerController seam; does NOT replace it. (No weight-sync — the target is frozen, §8.) DraftTrainStrategy ── per-algorithm forward+loss (eagle3 TTT / dflash block / domino). The per-step seam; lives in training/strategies (relocated from runtime/training in E0) — already plan-shaped. CheckpointManager / Evaluator / lr_scheduler / fsdp seam ── the managers (G1). COMPOSE: modeling/draft ── DRAFT_REGISTRY (@register_draft) for draft *architecture* classes (llama / deepseek-MLA / dflash-qwen3). Separate axis from strategy. StrategyRegistry ── per-algorithm spec (today's StrategySpec, converged here). config / cli / export ── first-class run surface (predecessor §4.3–4.8, carried forward). ``` The online / offline / disaggregated distinction is **not visible to `Trainer`/strategy** — it is fully absorbed by (ref source + `FeatureStore`) behind `FeatureDataLoader → TrainBatch`. ### 2.3 Module layout (runtime spine + domain layer) ``` specforge/ ├── runtime/ # SUBSTRATE ONLY — the canonical DataFlow spine (keep as-is) │ ├── contracts.py # SampleRef, TrainBatch, PromptTask, *Strategy literal │ ├── control_plane/ # metadata-only: controller, metadata_store, backpressure │ └── data_plane/ # FeatureStore (Local/SharedDir/Mooncake), FeatureDataLoader, │ # SampleRefQueue, StreamingRefChannel, offline/disagg readers │ ├── inference/ # TOP-LEVEL — the single home for all rollout/capture execution │ ├── rollout_worker.py # (from runtime/inference/rollout_worker.py) │ ├── capture.py # CaptureConfig (from runtime/inference/capture.py) │ ├── feature_source.py # the FeatureSource Protocol (the worker's only contract) │ ├── target_engine/ # TargetEngine, EXTRACTED from modeling/target/*TargetModel │ │ ├── base.py factory.py # (adapters then wrap an engine, §G2) │ │ ├── hf.py sglang.py sglang_server.py custom.py # per-backend generic engines │ │ └── sglang_capture_backend.py # sglang version-pinned glue │ └── adapters/ │ └── eagle3.py dflash.py # (from runtime/inference/{sglang,dflash}_adapter.py) │ ├── training/ # TOP-LEVEL — the single home for all training execution (NO facade) │ ├── trainer.py # domain Trainer: owns loop/eval/checkpoint (B3) │ ├── controller.py # TrainerCore + TrainerController seam (from runtime/training/trainer.py) │ ├── backend.py # FSDPTrainingBackend + no_sync (from runtime/training/backend.py) │ ├── checkpoint.py resume.py evaluator.py # NEW managers (§3) — born here │ └── strategies/ # StrategySpec registry + per-algorithm strategies (converges HERE) │ ├── base.py # DraftTrainStrategy ABC + StepContext (from runtime/training/strategy.py) │ ├── registry.py # StrategySpec / resolve_strategy (from runtime/training/registry.py) │ └── eagle3.py dflash.py domino.py │ ├── modeling/ # MODEL DEFINITIONS ONLY — no orchestration, no capture factory │ ├── draft/ # DRAFT_REGISTRY + @register_draft (NEW registry.py — predecessor §4.2) │ │ └── base.py registry.py llama3_eagle.py deepseek_eagle3.py(MLA) dflash.py flex_attention.py │ └── target/ # target model nn.Modules + model-specific glue ONLY │ └── target_head.py custom_backend/ sglang_backend/ # (capture orchestration moved to inference/) │ │ # (NO separate data/streams package — FeatureDataLoader over │ # SampleRef+FeatureStore IS the stream. Ref sources │ # (offline/rollout/streaming) live in runtime/data_plane; │ # live frozen-target capture is just another ref source) │ ├── launch.py # TOP-LEVEL — topology assembly only (spec-driven builders; from runtime/launch.py) ├── eval/ export/ config/ cli.py # NEW — carried forward from predecessor §4.4–4.8 └── core/ optimizer.py tracker.py distributed.py lr_scheduler.py # kept verbatim (predecessor §1) ``` > **One implementation home per concern; `runtime/` is substrate-only.** The two structural moves > (both gated by the byte-identical suite, executed as the move-only step `E0` — see > [docs/roadmap/domain-refactor.md](docs/roadmap/domain-refactor.md)): (1) extract `TargetEngine` > out of the EAGLE3-bound `modeling/target/*TargetModel` into top-level `inference/target_engine/` > (the `adapters/` then wrap an engine); (2) lift the training seam > (`TrainerCore`/`TrainerController`/`TrainingBackend`/strategy+registry) up into top-level > `training/`. `runtime/` shrinks to the substrate (control + data plane + contracts) — it is > **not** a home for domain/algorithm code, and there is **no facade package**. The control + data > planes stay exactly where they are; the online/offline/disaggregated distinction stays absorbed > by (ref source + `FeatureStore`) behind `FeatureDataLoader`, invisible to `training/`. --- ## 3. Gaps to close (prioritized) — what the domain layer adds on top of the spine These are the items the landed `runtime/` does **not** have, drawn from the predecessor and from the in-source `NOTE`s. Each lands behind the canonical spine without re-plumbing it. ### G1 — Training managers (highest leverage; the spine works but is bare) - **`no_sync()` accumulation.** `runtime/training` has **zero** `no_sync()` — FSDP all-reduces every micro-step, defeating `accumulation_steps`. (Predecessor §4.2 #10; verify with one profiler check: one all-reduce per `optimizer.step()`.) - **Full resume.** `save_checkpoint` persists training state only; no optimizer/scheduler/RNG restore, no rotation, no best-tracking. Add `CheckpointManager` (predecessor §4.3) + a `seek()`-equivalent on the colocated streams. - **`Evaluator` + `EvalCache`.** No `simulated_acc_len` / per-position-acc / best-checkpoint. (Predecessor §4.4 — per-position acc aggregated across batches *before* the geometric sum.) - **lr WSD + per-strategy `fsdp_wrap_policy()`** (predecessor §4.6, §4.2). ### G2 — `TargetEngine` abstraction (inference convergence) - Introduce a `TargetEngine` ABC **extracted from the `modeling/target/*TargetModel` classes** (`Eagle3TargetModel`, `DFlashTargetModel`); the `runtime/inference` adapters (`SGLangAdapter`/`DFlashAdapter`) then **wrap** a `TargetEngine` rather than being the engine, and stop binding to `generate_eagle3_data` / EAGLE3 names. Keep the existing `FeatureSource` Protocol. Add `SGLangServerEngine` (live SGLang server) as the cross-node engine — see §2.2 for its two feature transports (capture-into-FeatureStore for W3/O1.3, inline-HTTP for the light W3′). ### G3 — Live online capture (frozen target; **no** weight sync) - Replace the in-process generator with **live SGLang-server hidden-state capture**: a *frozen* target streams aux+final hidden states into `MooncakeFeatureStore`; the producer commits `SampleRef`s. The cross-process control plane + async loop are in-review; live capture (the gating spike) is next; scale-out (Ray = **open**) and hardening follow. - **Weight sync / hot draft-update / on-policy are explicitly out of scope.** The target is frozen, so the streamed data is independent of draft weights — there is nothing to re-sync and no staleness. `draft_weight_version` is kept **only as provenance**. - Detailed phases (O1–O3): [`docs/roadmap/online-disaggregation.md`](docs/roadmap/online-disaggregation.md). ### G4 — Composition + models (predecessor Phase 1) - `modeling/draft` `DRAFT_REGISTRY` (`@register_draft`) for draft **architectures** (separate axis from strategy); **MLA Eagle3 draft** (deepseek/Kimi); converge `StrategySpec` → `training/strategies` registry. Note: draft-arch registry and strategy registry are two registries, not one. ### G5 — Run surface (predecessor Phase 3) - Pydantic `config/` + `specforge` CLI; `export/to_sglang` (+ the documented MLA weight-name map) and `export/to_hf` with vocab pruning. --- ## 4. Topologies & the colocated lightweight path The workloads differ only in which *ref source* + `FeatureStore` + `TargetEngine` compose. All consume **one** `FeatureDataLoader → TrainBatch` iterator; the trainer/strategy/backend are identical regardless (no per-topology stream class). **In every case the target is frozen** — the draft is never in the generation loop, so the streamed features do not depend on draft training progress (no staleness, nothing to re-sync). | # | Workload | Ref source (→ FeatureDataLoader) | FeatureStore | Control plane | |---|---|---|---|---| | W1 | Offline (precomputed) | `OfflineManifestReader` (refs) | Local (`file://`/`mem://`) | **no-op** | | W2 | In-process online (frozen target) | `RolloutWorker` → `SampleRefQueue` | Local (`mem://`) | **no-op** | | W3 | Disaggregated online (frozen target; isolated pools, high BW) | `RolloutWorker` (producer pool) → `StreamingRefChannel`, `SGLangServerEngine` *capture transport* | **Mooncake** | **active** (lease/ack/reconcile/backpressure) | | W3′ | Disaggregated online (light/cross-node) | `SGLangServerEngine` *inline-HTTP transport* (RemoteStream-style source) | n/a — features inline over HTTP, no shared store | minimal | > **"Train-with-decode" = live *frozen-target* generation** — i.e. W2 (colocated) or W3 > (disaggregated), **not** a separate workload. The predecessor's dual-purpose > serve-and-push-weights "W4" is **out of scope**: a frozen target means there is nothing to push. > See [`docs/roadmap/online-disaggregation.md`](docs/roadmap/online-disaggregation.md). **Colocated lightweight path (W1/W2): keep it, but as a *no-op control plane*, not a fork.** The control-plane machinery (lease/ack/reconcile/backpressure/durable metadata/cross-process transport) only earns its keep when producer and consumer are decoupled. For W1/W2 (same process) it is pure overhead. Resolution: - **Single canonical path** through `SampleRef` + `FeatureStore`, so colocated and disaggregated produce **byte-identical batches for free** (today's property — "disaggregation changes *where* features live, not their values"). We do **not** fork a second trainer/launch path. - "Lightweight" = for colocated, the controller's lease/ack/metadata-store are **opt-in / no-op**, `FeatureStore` is `LocalFeatureStore(mem://)`, and there is **no SQLite, no cross-process, no backpressure**. The colocated ref sources (`OfflineManifestReader` / in-process `RolloutWorker`) flow through the spine but skip the heavy bits. - Cost to own honestly: this preserves the free byte-identical guarantee **only if** we keep one path. A numerical-equivalence gate (§7) locks "colocated stream output == disagg stream output" so the no-op path can never silently diverge. --- ## 5. What changed from the predecessor draft (explicit deltas) 1. **"No Mooncake in Phase 1-6" → revised.** The Mooncake control/data plane is now the **canonical** disaggregation backend, because the isolated-pool / >100 GB/s requirement is real and HTTP cannot serve it. The predecessor's `RemoteStream`-over-HTTP remains as the **light** second backend (W3′). Mooncake is used as a *transport*; the scale-out **orchestration** layer (Ray vs. home-grown) is an **open decision** (see §6, §8 and [`docs/roadmap/online-disaggregation.md`](docs/roadmap/online-disaggregation.md) §O2), no longer a flat non-goal. 2. **Primary abstraction / `HiddenStateStream` dropped.** Predecessor: a coarse `HiddenStateStream` produces `TrainBatch` and is the source of truth. Reconciled: `SampleRef` + `FeatureStore` + `FeatureDataLoader` is canonical, and `FeatureDataLoader` **already is** the stream — so `HiddenStateStream` is **not introduced** as a parallel abstraction. Topology variation lives in (ref source + `FeatureStore`); `seek()`/prefetch land on the loader/ref-source. This is strictly finer and is what makes lease/ack/backpressure/Mooncake clean. 3. **`runtime/` is not "thrown away."** The predecessor's "what we throw away" never mentioned `runtime/` (it predated it). The DataFlow spine is the foundation, not debt. 4. **Module layout** gains a top-level `runtime/` (the spine) under the domain layer; `models/` (drafts + targets), `training/` (Trainer + managers), `eval/`, `export/`, `config/`, `cli.py` are the layer on top. No separate `data/streams/` package — the loader is the stream. 5. **W4 weight lifecycle → dropped.** The predecessor's serve-and-push-weights workload + weight registry are **out of scope**. "Train-with-decode" is *frozen-target* live generation (W2/W3); weight sync / hot draft-update / staleness gate / on-policy are explicitly cut, aligning with #618. `draft_weight_version` survives only as provenance metadata. 6. **Ray reframed from non-goal → open.** It is a *candidate* for the O2 scale-out orchestrator (multi-node N-producer/M-trainer), not committed and not forbidden; the decision gate lives in the roadmap. Everything else in the predecessor (workloads W1–W3, the domain abstractions, the testing discipline, the MLA/export/eval/config detail) **carries forward unchanged**. --- ## 6. Tradeoffs (updated) ### Mooncake transport vs HTTP/gRPC — *transport bet reversed* The predecessor bet "HTTP is sufficient, Mooncake is gated behind profiling." For the **data transport** that bet is now wrong (isolated-pool / >100 GB/s requirement) — Mooncake is in, canonically. The `RemoteStream`-over-HTTP path stays as the light cross-node backend (W3′). ### Scale-out orchestration: Ray vs. home-grown — *OPEN* Today's metadata-only control plane handles a single producer-pool ↔ trainer scope. **O2** (multi-node N-producer/M-trainer scale-out) needs an orchestration layer, and **whether that is Ray or a home-grown scheduler is undecided** — likely necessary, but not committed. It is **not** a non-goal anymore; the decision gate (when ≥1 of: >1 producer pool, cross-node failover, multi-job pool sharing) lives in [`docs/roadmap/online-disaggregation.md`](docs/roadmap/online-disaggregation.md) §O2. Until then we keep the home-grown control plane and add nothing. ### One canonical path vs a light colocated fork Chosen: **one path** (spine everywhere) + control-plane-as-no-op for colocated. Keeps byte-identical free and avoids two implementations; costs an equivalence gate. (See §4.) ### Two registries, not one `modeling/draft` `DRAFT_REGISTRY` (architecture) and `training/strategies` (algorithm) are **separate** axes — an algorithm (eagle3) runs on multiple draft architectures. Don't merge them; the current `StrategySpec` is the *strategy* registry and should converge there, not into `modeling/draft`. *(The predecessor's other tradeoffs — Pydantic vs OmegaConf, MLA cache compressed vs expanded, registry vs HF AutoModel — are unchanged and carry forward.)* --- ## 7. Migration path (incremental, gated) The spine is already landed, so migration is "grow the domain layer on top + close the gaps," not a rewrite. Every phase that touches the training path passes the numerical-equivalence gate (below) against the prior commit. **Per-phase target/implementation/tests/done-when detail lives in [`docs/roadmap/`](docs/roadmap/)** — the phases below are the index; the online track there also folds in the former online-disaggregation roadmap (#618). - **Phase A — composable launch (in review).** `StrategySpec` registry + parameterized `launch.py`; eagle3/dflash/domino end-to-end in the stacked PRs #627/#628/#629 (validated, in review). 197 `tests/test_runtime` OK. - **Phase B — domain abstractions (no behavior change).** Introduce `TargetEngine` (wrap the existing adapters; de-EAGLE3 the names) and the domain `Trainer` wrapping the kept `runtime/training` core. No `HiddenStateStream` — `FeatureDataLoader` stays the stream. Gate: byte-identical batches/loss vs the pre-refactor path. - **Phase C — colocated no-op control plane + equivalence gate.** Make lease/ack/metadata opt-in; add the colocated≡disagg equivalence test. - **Phase D — training managers (G1).** `no_sync()`, full optimizer/scheduler/RNG resume, `CheckpointManager`, `Evaluator`. Gate: loss/eval parity at fixed steps. - **Phase E0 — layout consolidation (move-only).** Relocate the training seam + target engine into the top-level `training/` / `inference/` homes; `runtime/` shrinks to substrate. Pure `git mv` + import shims; gate: suite + B byte-identical gate unchanged (no functional diff). - **Phase E — `modeling/draft` registry + MLA draft + config/CLI/export (G4/G5).** Adding a draft arch = one decorated file; one validated YAML per run; export-loop test. - **Online track (parallel; G3) — live *frozen-target* capture.** O1.1 shared control plane + O1.2 async loop (in review) → **O1.3** live SGLang-server hidden-state capture (next; gated by a throughput spike) → **O2** scale-out (orchestrator: Ray = open) → **O3** hardening. **No weight sync.** Detail: [`docs/roadmap/online-disaggregation.md`](docs/roadmap/online-disaggregation.md). - **Eval track (parallel) — E1** acceptance-length eval harness → **E2** algorithm breadth (new algo = a `StrategySpec` + loss). Detail: [`docs/roadmap/eval-and-breadth.md`](docs/roadmap/eval-and-breadth.md). Doc debt to fix alongside Phase B: revise the predecessor's "No Mooncake / HTTP is sufficient" statements (now §5/§6) so the code and the plan stop contradicting each other. --- ## 8. Non-goals (updated) - **No weight sync / hot draft-update / on-policy training.** The target is **frozen**; the draft is never in the generation loop. No `WeightPublisher`, no weight-version registry, no staleness gate. `draft_weight_version` is provenance metadata only. - **No two-stack fork for colocated.** One canonical data path; colocated is the spine with the control plane as a no-op, not a parallel implementation. - **No vLLM target backend** (possible via `TargetEngine`, not prioritized). - **No multi-engine load balancing / multi-job inference-pool sharing** for now — gated behind ≥5 concurrent jobs sharing one target. **Open decisions (NOT non-goals):** - **Ray (or a home-grown scheduler) for O2 scale-out** — likely necessary for multi-node N-producer/M-trainer; undecided. Decision gate in §6 / the online roadmap. *(Reversed from the predecessor: "No Mooncake" → Mooncake transport is now canonical for W3; "No Ray" → Ray is now an open scale-out decision, not a flat non-goal.)* --- ## 9. Success criteria | Area | Criterion | |---|---| | Composable launch (in review) | eagle3/dflash/domino train via one strategy-parameterized path; full `test_runtime` green. | | Abstractions (B) | `TargetEngine` (wrapping existing adapters) + domain `Trainer` produce byte-identical batches/loss vs the direct spine path. | | Colocated (C) | W1/W2 run with control plane as no-op; colocated≡disagg equivalence gate passes. | | Managers (D) | resume reproduces the no-resume loss curve; one all-reduce per optimizer step; best-checkpoint tracked. | | Drafts/MLA (E) | MLA Eagle3 trains + loads in SGLang; new draft arch = one `@register_draft` file. | | Online (O1.3) | a live **frozen-target** SGLang server feeds training with zero precomputed features; loss/eval matches the offline baseline on the same prompts+seed. (Scale-out O2 / Ray = open.) | --- ## 10. Testing strategy (gating) Carries the predecessor's gates, plus the reconciliation-specific one. - **10.1 Numerical-equivalence gate** (per PR touching `core/` / `training/` / `runtime/data_plane/` / `modeling/draft`): fixed seed, 3×4 batches; per-step loss `atol/rtol=1e-4` at steps 0/1/100/500/1000; per-position eval acc + `simulated_acc_len` `atol=1e-3`. - **10.2 Colocated ≡ disaggregated gate (NEW):** same data through the colocated (no-op control plane, Local store) and the disaggregated (control plane, SharedDir/Mooncake) paths must yield identical `TrainBatch`es and identical loss. This is what licenses the lightweight path. - **10.3 Smoke tests:** each draft arch trains 20 steps under TP=1 / TP=2 / TP=2+SP=2; checkpoint save+resume matches no-resume (validates `seek`); eval cache miss==hit. - **10.4 Distributed correctness:** MLA + Yunchang USP with asymmetric head dims; gradient accumulation = one all-reduce per `optimizer.step()`. - **10.5 Export-loop test:** train MLA draft 100 steps → export → load in SGLang → 32 gens, acceptance > 0 (catches weight-name-map regressions). - **10.6 Online-capture parity gate (O1.3):** a live frozen-target capture run produces features + loss matching the offline-precomputed baseline on the same prompts/seed (the target is frozen, so this must hold exactly up to nondeterminism tolerance). No weight-sync gate — weight sync is out of scope.