reference/code/SpecForge/docs/roadmap/eval-and-breadth.md

# Eval + Algorithm-Breadth Track

This track runs **parallel and orthogonal** to the domain refactor (./domain-refactor.md)
and the online/disagg work (./online-disaggregation.md): it is where DeepSpec's edge actually
shows up — a trustworthy acceptance-length number and a low-cost path to new draft algorithms.
It has **no Ray and no weight-sync content** by construction: the eval target is the same
**frozen** online/offline target the rest of the system uses (see the weight-sync-out-of-scope
decision in ../../plan.md), so there is nothing to re-sync and no staleness to model. Both
phases build directly on the canonical Substrate (SampleRef + FeatureStore + FeatureDataLoader)
and the runtime training seam (`TrainerCore` / `DraftTrainStrategy` / `StrategySpec`) rather than
replacing any of it.

Sibling tracks: ./domain-refactor.md · ./online-disaggregation.md · root plan ../../plan.md.

---

### E1 — Acceptance-length eval harness · size M · GPU · status: in review
- **Goal** Produce a correct, cache-backed `simulated_acc_len` / `avg_loss` / `avg_acc` eval
  pass that is batch-size-independent and tracks the best checkpoint. This is the same
  `Evaluator` the domain refactor's Phase D wires into the domain `Trainer` — see Phase D in
  ./domain-refactor.md and cross-link back here.
- **Target state** Calling eval over any `FeatureDataLoader` stream yields per-position accuracy
  aggregated across the **whole** eval pass *before* the geometric sum; the number does not move
  when you change micro-batch size; repeat runs over an unchanged (eval-data, target, revision,
  tokenizer, template, aux-layers, seqlen) tuple hit an on-disk cache instead of recomputing
  hidden states; the controller records and can restore the best checkpoint by
  `eval/simulated_acc_len`.
- **Implementation**
  - **Fix the existing bug first.** `specforge/runtime/training/trainer.py::TrainerController.evaluate`
    today means per-batch scalars: `TrainerCore._result` collapses the per-position vectors
    (`acces`, `acceptance_rates`, `plosses`) to a single `_scalar` per batch, then `evaluate`
    averages those scalars across batches. That destroys per-position structure and makes any
    acceptance-length derived from it batch-size-dependent. The harness must aggregate the raw
    per-position **sum/count** tensors that `Eagle3TrainStrategy.forward_loss` already emits
    (`acc_corrects`, `acc_denoms` in `StepOutput.metrics`) — do **not** reduce them to scalars in
    the eval path.
  - Add `specforge/eval/evaluator.py` (top-level domain layer, per plan.md §2.3 — **not** under
    `runtime/`) with `Evaluator` + `EvalConfig`, following the
    sketch in ../../docs/redesign-draft-legacy.md (§4.4 Evaluation system). It consumes a
    `forward_fn` (a thin wrapper over `TrainerCore.eval_step`) and an eval stream that is a plain
    `FeatureDataLoader` over SampleRef+FeatureStore — there is no separate eval source of truth.
  - Aggregation contract (the load-bearing bit): keep running `per_pos_acc_sum` and
    `per_pos_acc_count`, each shape `[ttt_length]`, accumulated over **all** micro-batches; only
    after the full pass compute `per_position_acc = sum / count.clamp_min(1)` and feed *that*
    single aggregated vector into the geometric sum
    `acc_0 + acc_0·acc_1 + acc_0·acc_1·acc_2 + …`. The common bug — treating each batch's
    per-position vector as if it were positions — is explicitly what this ordering prevents.
  - Emit `{eval/avg_loss, eval/avg_acc, eval/simulated_acc_len}`; `avg_loss` is
    token-weighted (`Σ loss·num_tokens / Σ num_tokens`), `avg_acc` is the position-0 aggregated
    accuracy.
  - Add `specforge/eval/cache.py` with `EvalCache`. Key on
    **eval-data path + target path + target revision + tokenizer path + chat template +
    aux-layer ids + max seqlen** (§4.4 `cache_key`). Missing any field silently serves stale
    tensors after a target swap or template change. Keep this **separate** from the tokenization
    cache (`data/cache.py`) — different keys, different lifecycle; do not merge.
  - Best-checkpoint tracking: have `TrainerController` (trainer.py) keep `best_metric` /
    `best_checkpoint_uri` keyed on `eval/simulated_acc_len`, updated in the eval hook of `fit`,
    and persist a `best` pointer next to the step checkpoints written by `save_checkpoint`.
  - Strategy-agnostic by construction: EAGLE3 supplies per-position vectors; DFlash/Domino supply
    a single-position `accuracy`, which is the `ttt_length=1` degenerate case of the same
    aggregation (geometric sum of one term). No branching in the evaluator.
- **Tests / gates**
  - **Batch-size invariance gate:** the same fixed eval set produces identical
    `simulated_acc_len` at micro-batch sizes 1, 4, 16 (the regression that proves the
    aggregate-before-geometric-sum ordering).
  - Geometric-sum unit test on a hand-computed per-position vector.
  - `EvalCache` key test: flipping any one keyed field (target revision, template, aux layers,
    seqlen, tokenizer) changes the key; a clean re-run with all fields equal is a cache hit.
  - colocated==disagg equivalence: eval metrics match within tolerance whether the stream is a
    `LocalFeatureStore` (mem://) colocated loader or a disagg FeatureStore (reuses the track-wide
    numerical-equivalence gate).
- **Done when** `evaluate` returns the three metrics from a single full-pass aggregation, the
  batch-size-invariance gate is green, the cache hits on an unchanged tuple and misses on any
  changed field, and the controller restores the best checkpoint by `eval/simulated_acc_len`.
- **Landed split (July 2026):** the Evaluator (aggregate-before-geometric-sum, token-weighted
  loss AND token-weighted scalar accuracy, DP-reduced, batch-size-invariance gates) and durable
  best-checkpoint tracking landed with domain-refactor Phase D (#637); `EvalConfig` + `EvalCache`
  (injective JSON-encoded key, atomic-rename produce-once) landed as the E1 PR. One scope note:
  the launch builders still take a pre-produced `hidden_states_path`, so the cache is consumed
  programmatically (`EvalCache.for_config` + `get_or_produce`) until Phase E's typed config + CLI
  own eval-data production and wire it in front.

---

### E2 — Algorithm breadth · size L · GPU · status: later
- **Goal** Make "add a new speculative-decoding algorithm" a small, well-bounded contribution:
  a `DraftTrainStrategy` + a `StrategySpec` entry (+ optionally a draft arch), reusing the entire
  runtime spine. The first new targets are **MTP** and **Medusa**.
- **Target state** dflash and domino are **already landed** through the strategy registry
  (Phase A) — see `Eagle3TrainStrategy` / `DFlashTrainStrategy` / `DominoTrainStrategy` in
  `specforge/runtime/training/strategy.py` and their three `register_strategy(...)` entries in
  `specforge/runtime/training/registry.py`. Because of that, breadth is no longer "write a new
  `build_*_runtime` family" — it is "add a `StrategySpec` + a loss." Adding MTP or Medusa touches
  only the strategy class, one registry entry, and (if its draft network is new) one file under
  the draft model package.
- **Implementation** — what a new algorithm **must add**:
  1. **A `DraftTrainStrategy` subclass** in `specforge/runtime/training/strategy.py` (or a sibling
     module imported there). It must set `name`, `required_features`, implement
     `trainable_module()` and `forward_loss(batch, ctx) -> StepOutput`, and — if its persisted
     weights are a subset of the wrapped module — `checkpoint_state_filter`. `forward_loss` returns
     `StepOutput(loss, metrics)`; put per-position `acc_corrects`/`acc_denoms` (or a single
     `accuracy`) in `metrics` so the E1 `Evaluator` works unchanged. Only read `StepContext` if the
     loss depends on *where in training* you are (Domino's decayed `lambda_base` is the existing
     precedent — every other strategy ignores `ctx`).
       - *MTP*: multi-token-prediction heads → emit per-position accuracy like EAGLE3 (TTT-style),
         reuse the `ploss_decay`-style weighting.
       - *Medusa*: independent per-head losses over a shared trunk → a single combined loss; per-head
         accuracy can be reported as the per-position vector E1 already aggregates.
  2. **One `StrategySpec` entry** via `register_strategy(...)` in `registry.py`, declaring:
     `make_strategy`, `required_features` (frozenset, drives `CaptureConfig.from_strategy` +
     loader validation), `uses_target_head`, the offline data path
     (`make_offline_reader` / `make_offline_transform` / `make_offline_collate` /
     `offline_target_repr`), `make_online_collate`, `make_adapter` (None ⇒ default `SGLangAdapter`),
     and `supports_online`. If a data path isn't wired yet, leave that factory `None` — the builder
     raises an actionable `NotImplementedError` instead of silently feeding EAGLE3-shaped features.
  3. **Optionally, a draft architecture** under the draft-model package
     (`specforge/modeling/draft/`; domain Phase E adds a `registry.py` there —
     alongside `dflash.py` / `llama3_eagle.py`) — only if the algorithm needs a network the existing draft
     models don't provide. Medusa heads or MTP heads would live here; an algorithm that reuses an
     existing draft body (as Domino reuses DFlash's draft with a different head/loss) needs **no**
     new arch file.
- **What it reuses (unchanged):** `TrainerCore` / `TrainerController` (trainer.py),
  `FSDPTrainingBackend` (backend.py), `FeatureDataLoader` + FeatureStore (Substrate), checkpointing,
  the online capture adapters, and the E1 `Evaluator`. The topology builders stay
  `topologies + one spec per model` rather than `topologies × models` — the model is the
  `strategy=` parameter resolved through `resolve_strategy`.
- **Tests / gates**
  - A new strategy registers and resolves: `resolve_strategy("mtp")` / `available_strategies()`
    includes it; `required_features` round-trips through loader validation.
  - `forward_loss` emits metrics in the shape E1 consumes (per-position sum/count or a scalar
    `accuracy`); E1 produces a finite `simulated_acc_len` for the new strategy with no evaluator
    changes.
  - Offline↔online schema parity for the new strategy where both paths exist (same
    `required_features`, equivalent collate output).
  - `NotImplementedError` is raised (not a wrong-feature run) when an unwired data path is invoked.
- **Done when** at least one new algorithm (MTP **or** Medusa) is registered as a `StrategySpec`,
  trains through the unchanged `TrainerCore`, is evaluated by the unchanged E1 `Evaluator`, and the
  only files touched are its strategy class, its registry entry, and (if needed) one draft arch file.