Wave 21: deep-read critical review — 8 source clusters re-read, findings verified

8 Sonnet readers re-fetched every primary source (Composer 2.5 blog verbatim,
Composer 2 techreport 2603.24477 full HTML, SWE-smith/SWE-Gym/R2E-Gym/SWE-bench,
SDPO/OPSD, GRPO family + Comedy-of-Estimators, DiLoCo line, world-model cluster
+ anti-evidence, LATS/ToT/rStar/Tree-GRPO/SWE-Search/Symphony/Socratic-SWE,
TRL/verl/SkyRL live docs, SWE-MiniSandbox) and cross-checked the repo's research
notes + vision docs against them. 2 adversarial critics (fidelity, design);
every P0/P1 finding independently verified against code + source quotes — 0
refuted. (Feasibility critic stalled on capacity; flagged unreviewed.)

Top verified findings:
- SDPO "mathematically the same" claim is WRONG (Cursor cites it as background;
ours is a third, blog-inspired design) — to be corrected.
- Envisioned tree pipeline had 4 structural breaks: seed-trace/oracle
disjointness, NO rollout harness (SFT corpus had no producer), divergence
gate uncomputable (whitespace-stub normalizer), no Sandbox.fork().
- Zero benchmark decontamination anywhere; no secrets/PII gate; golden_diff
serialization leak; two unreconciled S3 contracts.
- Buy-vs-build inversion: pip install swesmith ships what we planned to build;
its PR Mirror ≡ our gold-patch reversion and is VALIDATED best-of-five by
its ablation (Table 5) — the engine for "point at a repo".
- Fabricated numbers circulating as Cursor-stated (69.3%, "24 generators",
85% compute); CWM misread (mid-training, not RL-time aux head); Streaming
DiLoCo citation doubly wrong; cost figures mislabeled.

13-synthesis-architecture.md: the Stage-0 "point at a repo -> corpus" pipeline
(swesmith buy + rollout harness + ingest gates + canonical trajectory IR +
reconciled contract + acceptance probe, ~900 LOC) and the staged vision ladder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Files changed (13) hide show

research/deepread/00-grounding.md +255 -0
research/deepread/01-cursor-artifacts.md +326 -0
research/deepread/02-swe-task-synthesis.md +340 -0
research/deepread/03-sdpo-opsd.md +556 -0
research/deepread/04-grpo-family.md +421 -0
research/deepread/05-diloco.md +309 -0
research/deepread/06-worldmodel.md +298 -0
research/deepread/07-trace-replay-tree.md +291 -0
research/deepread/08-rl-infra.md +292 -0
research/deepread/10-critic-fidelity.md +169 -0
research/deepread/11-critic-design.md +180 -0
research/deepread/12-verified-findings.md +131 -0
research/deepread/13-synthesis-architecture.md +197 -0

research/deepread/00-grounding.md ADDED Viewed

	@@ -0,0 +1,255 @@

+# Grounding Map: Dataset-Generation Pipeline — What Exists vs What Is Claimed vs What Is Envisioned
+**Agent:** REPO-GROUNDING
+**Date:** 2026-06-09
+**Scope:** composer_replication/datagen/*.py, teacher_replay.py, trainer/composer_trainer.py, loss.py,
+hint_generator.py, docs/adrs/ADR-010 + ADR-002 + ADR-013, research/design-F1..F5,
+research/notes/final_report_socratic-mcts-swe-worldmodel-8f6dea.md, docs/COMPOSER_RECIPE_MAPPING.md,
+docs/BACKLOG_RESOLUTION_2026-06-09.md
+---
+## (1) Exact Current Dataset-Generation Capability
+### FeatureDeletionTask schema (`datagen/schema.py`)
+Six load-bearing fields and what produces each today:
+| Field | Type | Producer today | Notes |
+|---|---|---|---|
+| `task_id` | `str` | `SweBenchAdapter.to_task()` — copied from `instance["instance_id"]` or `instance["task_id"]` | `"unknown"` if missing |
+| `repo` | `str` | `instance["repo"]` via `SweBenchAdapter.to_task()` | e.g. `"getmoto/moto"` |
+| `base_commit` | `str` | `instance["base_commit"]` | no code to `git checkout` this commit exists today |
+| `broken_image` | `str` | `SweBenchAdapter.image_for(instance)` — either `instance["docker_image"]` (SWE-rebench) or the conventional `swebench/sweb.eval.x86_64.{iid}:latest` | This tag is a **pre-built SWE-bench eval image**; no code in the repo pulls or builds these images |
+| `fail_to_pass` | `tuple[str,...]` | `_as_tuple(instance["FAIL_TO_PASS"])` — handles JSON-encoded string OR list | validated non-empty in `__post_init__` |
+| `pass_to_pass` | `tuple[str,...]` | `_as_tuple(instance["PASS_TO_PASS"])` | may be empty |
+| `test_command` | `str` | `SweBenchAdapter.default_test_command` = `"python -m pytest -q"` | hardcoded; not read from instance |
+| `deleted_symbols` | `tuple[str,...]` | **never populated by SweBenchAdapter** — hardcoded `()` in every substrate inversion | the monitor can't do symbol-provenance checks without this |
+| `golden_diff` | `str` | `instance["patch"]` | held out of repr; used only by validator |
+| `granularity` | `str` | hardcoded `"feature"` in `SweBenchAdapter.to_task()` | CREATE-half escalation (function→file→feature) not wired to anything |
+| `difficulty_prior` | `float` | `instance["difficulty"]` if present (SWE-rebench) else `0.5` | |
+| `upstream_license` | `str` | `instance["license_name"]` | copyleft filter in `is_redistributable()` strips GPL/AGPL/LGPL |
+### What SweBenchAdapter actually does and does NOT do
+`SweBenchAdapter.to_task(instance: dict)` is a **pure schema inversion** — it takes one SWE-bench-shaped dict and maps it to a `FeatureDeletionTask`. It does NOT:
+- Pull or build a Docker image
+- Apply the gold patch in reverse (`git apply -R`)
+- Run any tests
+- Discover test node IDs
+- Populate `deleted_symbols` (always empty)
+- Escalate `granularity` beyond the static `"feature"`
+The broken-repo Docker image is **assumed to exist pre-built** (the SWE-bench project publishes these images; SWE-rebench carries its own `docker_image` field). The full pipeline step "revert gold patch → scrub caches → freeze image" is the documented `[~]` gate in ADR-010 — implemented in concept (the 4-gate validator interface exists, `scrub_tree` is built, `LocalSubprocessSandbox` and `DockerSandbox` are built) but there is no code in the repo that actually clones a repo, runs `git apply -R <gold_patch>`, builds a Docker image, and pushes it to a registry.
+### What FeatureDeletionEnv does during training (`datagen/env.py`)
+- `reset(task)` — boots the sandbox (by image tag), returns a text prompt listing failing tests. The prompt exposes `task.repo`, `task.fail_to_pass`, `task.test_command` but NEVER `golden_diff` or `deleted_symbols`.
+- `step(action)` — delegates to `sandbox.exec(action)`, returning observation text; grades on `submit` or turn limit.
+- `_grade()` — runs `sandbox.run_tests(test_command, fail_to_pass + pass_to_pass)`, computes pass-fraction over `fail_to_pass`, gates to 0 if `pass_to_pass` guard is broken OR `HackMonitor.flag()` fires.
+- `reward_fn(prompts, completions, *, task_id, **kwargs)` — TRL `RewardFunc` face; dispatches through `reset`/`step`; feeds fractional credit (not binary) to `DifficultyCurriculum.update`.
+### Safeguards implemented
+- `scrub_tree(workdir)` — physically removes `__pycache__`, `.mypy_cache`, `.pytest_cache`, `.git`, `.hg`, `*.pyc/.pyo/.class` before episode start. This is the PRIMARY control (added in Wave 2; was absent before).
+- `SANDBOX_DENYLIST` — blocks `find`, `strings`, `unzip`, `jar`, `javap`, decompilers, `git`. First-token-only check; bypassable via `sh -c "..."`. Documented as defense-in-depth, NOT the wall.
+- `HackMonitor.flag()` — layer 1: substring scan for cache/decompiler signatures in trajectory actions (not in `submit_patch`). Layer 2: patch-provenance — if a deleted symbol reappears verbatim in the patch AND the trajectory shows a cache/bytecode artifact being read (normalized to defeat `"__py"+"cache__"` obfuscation), flags the trajectory.
+- `DockerSandbox` — `network_mode='none'`, `read_only=True`, `cap_drop=['ALL']`, `no-new-privileges`, `pids_limit=256`, `mem_limit=1g`, optional gVisor `runtime='runsc'`.
+### What ingestion/claude_code.py can ingest today
+`ClaudeCodeIngester.ingest(path: Path) -> Iterator[TraceState]`:
+- Input: Claude Code session JSONL at `~/.claude/projects/<encoded>/<sessionId>.jsonl`
+- Output: one `TraceState` per assistant TURN (`state_id`, `messages`, `student_action`)
+- Skips: subagent files (`agent-` prefix), sidechain records (`isSidechain: True`), `summary` / `attachment` / `queue-operation` / `file-history-snapshot` records
+- `student_action`: JSON-serialized list of text + tool_use + thinking blocks (thinking KEPT in student_action, STRIPPED from teacher-facing messages if `strip_thinking=True`)
+- `tool_error` flag: structurally set on `user` messages where any `tool_result` block has `is_error: true` — this is the SDPO error-site detection signal
+- `state_id`: `f"{path.stem}::{state_idx:04d}"`
+- Does NOT handle: OpenHands traces, SWE-smith trajectories, any format other than Claude Code JSONL
+---
+## (2) Envisioned Pipeline End-to-End (S3 Contract Prefixes, Tree Controller, Outer Loop)
+From `research/design-F1-systems-framing.md`, `research/design-F2-aws-datagen.md`, and `research/notes/final_report_socratic-mcts-swe-worldmodel-8f6dea.md` §5/§8/§10.
+1. **Seed trace ingestion (Stage a):** `ClaudeCodeIngester.ingest()` over `s3://composer-datagen-386931836011-us-west-2/raw/claude_code/**/*.jsonl` → Parquet at `traces/v1/run_id=<id>/part-*.parquet` via AWS Glue 5.0 Spark ETL job (`glue_ingest_job.py`, ~80 LOC, NOT YET BUILT).
+2. **Schema inversion (Stage c1):** `SweBenchAdapter.to_task()` per SWE-bench row → `FeatureDeletionTask` JSONL at `tasks/v1/run_id=<id>/manifest.jsonl` (one task per line, array index = line number). Pure CPU; runs inside the Glue job or a Lambda. License gate (`is_redistributable()`) applied here.
+3. **N-teacher replay (Stage b):** `teacher_replay.replay_trace()` generalized from flat OpenRouter to `BedrockBatchTeacherPool` — write one shared `replay/v1/run_id=<id>/input/states.jsonl`, submit one `CreateModelInvocationJob` per teacher, write `.jsonl.out` per teacher to `replay/v1/.../teacher=<slug>/`. An EMR Serverless aggregation step joins all N outputs by `state_id` → `list[TeacherCallResult]`. (`teacher_replay_bedrock.py`, ~180 LOC, NOT YET BUILT).
+4. **Multi-model tree expansion (the core delta — NOT BUILT):** A `tree_controller.py` (~250–350 LOC, design-only) that, for each `TraceState` node, fires N models, applies each candidate action through `FeatureDeletionEnv.step()` to get a real next observation, branches again from the new state, grades leaves with `_grade()`. Expansion is gated on pre-expansion divergence between sibling next-action distributions (to avoid O(N^D) explosion). Emits six typed S3 prefixes (see step 8).
+5. **Sandbox materialization + 4-gate validation (Stage c2):** AWS Batch array jobs on EC2 Spot, one child per task. Each child reads `AWS_BATCH_JOB_ARRAY_INDEX`, looks up its task in the S3 manifest, boots `DockerSandbox`/`LocalSubprocessSandbox`, runs `validator.validate_task()` (4 gates), writes `task_grades/v1/run_id=<id>/<task_id>.json`. (`datagen/aws/batch_validate.py`, ~120 LOC, NOT YET BUILT).
+6. **DPO pair extraction + normalization (Stage d):** `extract_dpo_pairs()` (already built in `teacher_replay.py`) on the fan-in of teacher outputs → `DPOPair` rows → `DJNormalizer` data-juicer op-graph → EMR Serverless Spark for cross-partition dedup → `corpus/v1/run_id=<id>/dpo/part-*.parquet` and `corpus/sft/part-*.parquet`. (`replaysim/emr_normalize_job.py`, ~100 LOC, NOT YET BUILT).
+7. **Orchestration:** AWS Step Functions Standard Workflow: `Ingest(Glue) → InvertSchema(Lambda) → [Bedrock batch ×N (Map)] → FanIn(EMR-Serverless) → ExtractDPO+SynthTasks → SandboxValidate(Batch array, .sync) → Normalize(EMR-Serverless) → WriteManifest(Lambda)`. (`infra/datagen_stepfunctions.json` + `infra/datagen_stack.py`, ~250 LOC IaC, NOT YET BUILT).
+8. **S3 typed dataset contract (full set):**
+   - `raw/claude_code/**/*.jsonl` — input seed traces
+   - `traces/v1/run_id=<id>/part-*.parquet` — TraceState rows (Stage a output)
+   - `tasks/v1/run_id=<id>/manifest.jsonl` — FeatureDeletionTask rows (Stage c1 output)
+   - `tasks/golden/run_id=<id>/` — golden_diff ACL-isolated prefix (deny-by-default; NEVER co-located with policy-visible tasks/)
+   - `replay/v1/run_id=<id>/input/states.jsonl` — shared Bedrock batch input
+   - `replay/v1/run_id=<id>/teacher=<slug>/*.jsonl.out` — per-teacher Bedrock batch output
+   - `task_grades/v1/run_id=<id>/<task_id>.json` — validator + _grade() results
+   - `corpus/v1/run_id=<id>/sft/part-*.parquet` — clean winning trajectories (SFT-first floor)
+   - `corpus/v1/run_id=<id>/dpo/part-*.parquet` — DPO pairs (normalized DPOPair)
+   - `dpo_pairs/` — divergence-derived DPO pairs from the tree (sibling winners vs losers)
+   - `rl_task_pool/` — FeatureDeletionTask registry + DifficultyCurriculum priors
+   - `divergence_pairs/` — divergence-annotated nodes (where sibling next-action distributions forked)
+   - `wm_tuples/` — (state, action, next_state, outcome) for ALL branches incl. failures (world-model training target)
+   - `holdout/` — disjoint held-out eval anchor (HeldoutSplit; NEVER fed back)
+   - `diloco/rendezvous/round_<NNNNNN>/rank_<RRRR>.pt` — DiLoCo outer-sync (already used by existing allreduce.py)
+   - `manifests/run_id=<id>.json` — run-level manifest (counts, cost, lineage, schema_version, parent_run_id for flywheel)
+9. **SFT-first stage:** Read `sft_corpus/` (clean `_grade()` gate-1 passing trajectories), run `compose_loss` with `alpha_sdpo=0, beta_replay=0` (reduces to `_lm_response_ce` — next-token CE masked to response tokens), write `ckpt_sft/`. (`pipeline/sft_floor.py`, ~60 LOC, NOT YET BUILT).
+10. **Inner RL loop:** `ComposerReplicationTrainer` (trl.GRPOTrainer subclass) on `rl_task_pool/` with `FeatureDeletionEnv.reward_fn`; `total = grpo + α·sdpo + β·trace_replay_dpo`; DiLoCo outer-sync via S3; `HeldOutGuard` kill-switch now wired (Wave 3).
+11. **Flywheel:** Improved student generates next outer loop's seed traces; learned deliberation-confidence becomes the next round's divergence gate.
+---
+## (3) Unbuilt Components the Vision Depends On
+Every item below is design-only or a skeleton; none has real production code.
+| Component | File Estimate | Source | Status |
+|---|---|---|---|
+| `datagen/tree_controller.py` — the core delta: env-step between branches, `_grade()` at leaves, divergence-gated expansion, six typed S3 prefix writes | ~250–350 LOC | design-F1, final_report §1/§5/§6 | **0% built** — no file exists |
+| `SiblingBootstrapGenerator` in `hint_generator.py` — select max-reward sibling → emit "a working approach looks like: …" → feed `ctx_teacher` splice | ~60 LOC | design-F5 Tier 1 / final_report §1/§6 | **0% built** — not a class in hint_generator.py at all |
+| `pipeline/s3_layout.py` — typed writers for all six S3 dataset prefixes; the OUTER→INNER contract | ~80 LOC | design-F1 §4 | **0% built** — no `pipeline/` directory exists |
+| `pipeline/sft_floor.py` — SFT-first driver: read `sft_corpus/`, run TRL SFTTrainer or `compose_loss` `alpha=beta=0`, write `ckpt_sft/` | ~60 LOC | design-F1 §2 / design-F5 d | **0% built** |
+| `teacher_replay_bedrock.py` — `BedrockBatchTeacherPool`: submit one Bedrock `CreateModelInvocationJob` per teacher, poll, parse `.jsonl.out` back into `list[TeacherCallResult]` | ~180 LOC | design-F2 §b | **0% built** |
+| `datagen/aws/batch_validate.py` — AWS Batch array-child entrypoint: read `BATCH_JOB_ARRAY_INDEX` → manifest line → `DockerSandbox` + `validator` + `_grade()` → write `task_grades/` | ~120 LOC | design-F2 §c2 | **0% built** — `datagen/aws/` subdirectory does not exist |
+| `datagen/aws/glue_ingest_job.py` — Glue Spark entrypoint wrapping `ClaudeCodeIngester.ingest` in `mapPartitions`; write `traces/` Parquet | ~80 LOC | design-F2 §a | **0% built** |
+| `replaysim/emr_normalize_job.py` — EMR Serverless Spark entrypoint wrapping `DJNormalizer` per partition + Spark cross-partition dedup; write `corpus/dpo/` + `corpus/sft/` Parquet | ~100 LOC | design-F2 §d | **0% built** |
+| `datagen/aws/s3_contract.py` — S3 layout constants, `RunManifest` dataclass, Parquet/JSONL serializers, `recordId==state_id` join helpers, `schema_version`/`split` column injection | ~120 LOC | design-F2 §contract | **0% built** |
+| `infra/datagen_stepfunctions.json` + `infra/datagen_stack.py` — Step Functions state machine + IAM roles (Bedrock batch service role, Batch Spot compute env, EMR Serverless, Glue) | ~250 LOC IaC | design-F2 §orchestration | **0% built** — `infra/` directory does not exist |
+| `trainer/composer_trainer.py` world-model head — parameter-isolated next-state adapter + `<deliberate>` token as second SDPO mode | ~40 LOC delta | design-F1 §4 / final_report §2 | **0% built** — grep confirms no `world_model`/`WorldModel`/`next_state_head`/`<deliberate>` anywhere in `composer_replication/` |
+| Broken-repo image builder — code to clone a repo at `base_commit`, apply `git apply -R <golden_diff>`, run `scrub_tree`, build and push a Docker image to ECR | unspecified | ADR-010 §decision / design-F2 §c2 | **0% built** — there is NO code anywhere in the repo that manufactures a broken-repo Docker image from scratch |
+| `EKSExecutor` (now skeleton-built in Wave 2) + Argo Workflows controller for outer loop | Wave-2 executor skeleton built; Argo controller design-only | design-F1 §AWS / final_report §8 | **skeleton built** — `eks.py` is a functional executor (IndexedJob dispatch) but the Argo outer-loop controller is 0% |
+| `verl AsyncServer` backend for tool-heavy tree | — | final_report §8 | **0% built** — design note only |
+| Offline LLM-judge hack monitor (EvilGenie-style, Bedrock) | — | design-F5 §Tier 4 | **0% built** |
+---
+## (4) Seams Where "Point at an Arbitrary OSS Repo" Breaks the Current Code
+The `SweBenchAdapter` is designed to consume pre-packaged SWE-bench-shaped datasets, not arbitrary GitHub repos. The breaks are structural:
+### Break 1: `broken_image` assumes a pre-built SWE-bench image exists
+`SweBenchAdapter.image_for()` returns either `instance["docker_image"]` (SWE-rebench) or the convention `swebench/sweb.eval.x86_64.{iid}:latest`. For an arbitrary OSS repo there is no such image. A fresh repo would need:
+- Clone at `base_commit`
+- Install the project's Python/Java/etc. toolchain
+- Apply `git apply -R <golden_diff>` to manufacture the broken state
+- Run `scrub_tree()` to strip caches
+- Build a Docker image that encapsulates this broken state
+- Push the image to a registry accessible by `DockerSandbox.boot()`
+None of this code exists. `DockerSandbox.boot(image)` raises `RuntimeError("DockerSandbox.boot: image {image!r} not found locally and could not be pulled (the container is --network none). Pull it on the host first.")` if the image is absent.
+### Break 2: `test_command` is hardcoded
+`SweBenchAdapter.default_test_command = "python -m pytest -q"`. A fresh repo may use `make test`, `npm test`, `cargo test`, `mvn verify`, or any other test runner. There is no test-discovery logic anywhere in the repo.
+### Break 3: `fail_to_pass` and `pass_to_pass` require pre-existing test labels
+SWE-bench instances ship with `FAIL_TO_PASS` and `PASS_TO_PASS` as pre-identified pytest node IDs. For an arbitrary repo the mapping from "the code change" to "which tests exercise the deleted symbols" must be derived — e.g., via coverage analysis or AST-reachability. `FeatureDeletionTask.__post_init__` raises `ValueError` if `fail_to_pass` is empty. The 4-gate validator's Gate 2 (deletion breaks the feature) cannot be verified without pre-identified test node IDs.
+### Break 4: `deleted_symbols` is never populated
+`SweBenchAdapter` hardcodes `deleted_symbols=()`. The `HackMonitor._patch_provenance_hack()` check (`monitor.py:157-182`) skips the symbol-reappearance test if `deleted_symbols` is empty — so the provenance layer of the hack monitor is effectively a no-op on all SweBenchAdapter-derived tasks. For a fresh repo, AST analysis to identify the deleted symbols would be required.
+### Break 5: No copyleft scrub for arbitrary repos
+`is_redistributable()` reads `upstream_license` from `instance["license_name"]`. For a fresh GitHub repo there is no pre-populated license field; the repo license must be detected (e.g., via SPDX scanning) before the copyleft filter can be applied.
+### Break 6: No env setup for non-Python repos
+`LocalSubprocessSandbox.run_tests` runs `subprocess.run(cmd, shell=True, ...)` against the working tree with a hard-coded 600s timeout. It has no virtualenv creation, no dependency installation, no multi-language support. `DockerSandbox` depends on a pre-baked image that already has the environment. A fresh Python repo would need `pip install -e .` run inside the image, and a non-Python repo would need a completely different image and test runner.
+---
+## (5) What ingestion/claude_code.py Can Ingest Today
+`ClaudeCodeIngester.ingest(path)` handles exactly one format: **Claude Code session JSONL** at `~/.claude/projects/<encoded>/<sessionId>.jsonl`.
+Supported record types handled:
+- `type="user"` — string content or list of text/tool_result blocks → OpenAI-style user message; `tool_error` structural flag set if any `tool_result` block has `is_error: true`
+- `type="assistant"` — list of text/thinking/tool_use blocks → one `TraceState` with `student_action` (full blocks including thinking) and `messages` (history, optionally with thinking stripped)
+Record types silently skipped:
+- `type="summary"` — Claude Code conversation summary records
+- `type="attachment"`, `"queue-operation"`, `"file-history-snapshot"`, `"last-prompt"`, `"system"` — auxiliary records
+- `isSidechain: True` records — subagent traces (skipped in v0.1 per ADR-002)
+- Files starting with `agent-` — subagent session files by naming convention
+Structural features:
+- `state_id = f"{path.stem}::{state_idx:04d}"` — stable within-session identifier
+- `strip_thinking` flag (default True) — strips `[THINKING] ...` lines from the teacher-facing `messages` history but keeps them in `student_action`
+- Injects synthetic system prompt at `messages[0]` (`"You are a senior software engineer..."`)
+- Version check: warns on schema version outside `2.x.x`
+NOT handled by this ingester:
+- OpenHands trajectory format (planned for v0.2 per ADR-002)
+- SWE-smith trajectories (planned for v0.2)
+- Cline VS Code export
+- Aider chat history
+- SWE-bench leaderboard trajectory submissions
+- Any binary or non-JSONL format
+---
+## Critical Cross-Checks: What the Repo Claims vs What Exists
+### Claim 1: "Feature Deletion generator" (Composer 2.5 blog says "point at a repo")
+**What the blog says (COMPOSER_RECIPE_MAPPING.md):** "take a repo with passing tests, delete some code, ask the agent to reimplement to pass tests."
+**What the repo does:** Inverts *existing* SWE-bench-shaped instances — reverts their gold patch. There is NO code that: (a) points at an arbitrary OSS repo, (b) identifies deletable symbols, (c) synthesizes a broken state beyond SWE-bench's pre-packaged ones. The ADR correctly scopes this as "Option A — invert OSS substrates" vs "Option B — greenfield repo scraping." The blog's "point at a repo" vision is Option B, which was *explicitly rejected*.
+### Claim 2: "25× synthetic data"
+**What the blog says:** Composer 2.5 uses 25× more synthetic tasks than Composer 2 (COMPOSER_RECIPE_MAPPING.md §2).
+**What the repo has:** A schema adapter for 5 existing OSS datasets (SWE-bench-Lite ~300, SWE-Gym ~2.4k, R2E-Gym ~8.1k, SWE-rebench ~21.3k, OpenHands/Nemotron ~59k). ADR-010 notes ~15 node-days to invert all SWE-rebench tasks. No actual inverted task corpus has been generated. The 25× claim refers to the *training run*; the repo has the generation machinery for the inversion shape but not the greenfield synthesis needed for genuine novel task minting.
+### Claim 3: "Dynamic difficulty curriculum — select for AND create harder tasks"
+**What Composer 2.5 says:** "We both select for and create harder tasks dynamically throughout the run."
+**What the repo has:** The SELECT-FOR half: `DifficultyCurriculum` with p̂(1−p̂) frontier weighting, retire/quarantine thresholds, and effort tilt on turns/think-tokens (Wave 20). The CREATE half (escalating deletion span, coupling complexity, multi-feature targets during the run) is explicitly listed as MISSING in design-F5 row b2. `granularity` is set statically to `"feature"` for all SweBenchAdapter tasks; no escalation logic exists.
+### Claim 4: `deleted_symbols` enables AST-provenance monitoring
+**What ADR-010 says:** "signature + patch-provenance monitor" that detects if deleted symbols reappear via cache reads.
+**Reality:** `deleted_symbols=()` on every `SweBenchAdapter`-derived task (line 81 in substrates.py: hardcoded empty tuple). `HackMonitor._patch_provenance_hack()` returns False immediately when `deleted_symbols` is empty (`reappeared = [s for s in deleted_symbols if s and s in patch]` → empty list). The provenance layer of the monitor is a dead code path for all currently-generable tasks.
+### Claim 5: The tree controller and world-model head are part of the system
+**What design docs say:** "roughly nine-tenths of it" is reuse (final_report §6 reuse-vs-build table).
+**Reality:** The tree controller is 0/0 — no file, no function, no class. Confirmed by exhaustive grep: no `SiblingBootstrap`, `world_model`, `WorldModel`, `next_state_head`, `tree_controller`, `MCTS`, `deliberate_token` anywhere in `composer_replication/`. The "nine-tenths reuse" claim is accurate for the Composer recipe replication; the tree itself (the framework's own addition) is entirely design.
+### Claim 6: The broken-repo image is manufactured by the pipeline
+**What design-F2 says:** Step c2 involves "pull the substrate's frozen image, `git apply -R` the gold patch, `scrub_tree()`, run the test command, confirm FAIL_TO_PASS actually fails."
+**Reality:** This describes what SHOULD happen in the Batch array child. No such code is written. `SweBenchAdapter.image_for()` returns a string tag; that tag must be pre-pulled on the host before `DockerSandbox.boot()` can use it (`RuntimeError` on image-not-found). The full broken-image manufacture pipeline (clone → revert → scrub → build → push) is a gap.
+---
+## Summary of Unbuilt vs Built
+### BUILT and tested (production-ready CPU, Docker-gated where noted):
+- `FeatureDeletionTask` schema + `FeatureDeletionEnv` (reset/step/_grade/reward_fn)
+- `SweBenchAdapter` schema inversion (pure dict transform)
+- `FakeSandbox`, `LocalSubprocessSandbox`, `DockerSandbox` (hardware-gated e2e green in Wave 1/2)
+- `scrub_tree()` primary reward-hack control
+- `HackMonitor` (signature + patch-provenance, obfuscation-resistant)
+- `DifficultyCurriculum` (SELECT-FOR half + effort tilt)
+- `validate_task()` 4-gate solvability validator
+- `ClaudeCodeIngester` (Claude Code JSONL only)
+- `behavior_rewards.py` — `c_length`, `EffortWeights`, `LengthEffortPenalty`, `UnfinishedTodoPenalty`, `LeftoverCoTPenalty`, `CommunicationReward` (Wave 20)
+- `kl_in_reward.py` — k1-in-reward path opt-in (Wave 20)
+- `HeldOutGuard` + `HeldoutSplit` + wired into trainer (Wave 2/3)
+- `EKSExecutor` skeleton + `SageMakerExecutor` skeleton (Wave 2)
+### DESIGN-ONLY (no code):
+- Tree controller (`datagen/tree_controller.py`)
+- `SiblingBootstrapGenerator` in `hint_generator.py`
+- `pipeline/s3_layout.py`, `pipeline/sft_floor.py`
+- `teacher_replay_bedrock.py` (BedrockBatchTeacherPool)
+- `datagen/aws/batch_validate.py`, `datagen/aws/glue_ingest_job.py`, `datagen/aws/s3_contract.py`
+- `replaysim/emr_normalize_job.py`
+- `infra/datagen_stepfunctions.json`, `infra/datagen_stack.py`
+- World-model next-state head in trainer
+- Argo Workflows outer-loop controller
+- Broken-repo image builder (clone → git apply -R → build → push)
+- CREATE half of difficulty curriculum (mint harder tasks during run)
+- SFT-first training stage
+- Offline LLM-judge hack monitor

research/deepread/01-cursor-artifacts.md ADDED Viewed

	@@ -0,0 +1,326 @@

+# Deep-Read: Cursor Artifacts — Composer 2.5 Blog + Composer 2 Technical Report
+> **Written:** 2026-06-09.
+> **Primary sources fetched directly:**
+> - Composer 2.5 blog — full body retrieved from vault note `introducing-composer-25-cursor` (sourced from `https://cursor.com/blog/composer-2-5`).
+> - Composer 2 Technical Report — full HTML body retrieved from `https://arxiv.org/html/2603.24477` (arXiv:2603.24477 v2, 26 Mar 2026), stored as vault note `composer-2-technical-report` (~11 900 words, all sections including appendices).
+> **Repo notes cross-checked:** `research/01-composer-2.5.md`, `research/09-composer-blog-delta-2026.md`, `research/10-composer2-techreport-mining.md`, `research/06-feature-deletion-datagen.md`, `research/07-sdpo-hint-generator.md`.
+> **Tag conventions:** `[SOURCE-VERBATIM]` = quote taken verbatim from the fetched primary source. `[REPO-CLAIM]` = what the repo's notes assert. `[FINDING]` = discrepancy or new fact. `[CONFIRM]` = repo claim verified against primary.
+---
+## PART 1 — What the primary sources actually say (verbatim extractions)
+### 1A. Synthetic data — every word the 2.5 blog says
+The entirety of the "Synthetic data" section of the Composer 2.5 blog (fetched verbatim):
+> [SOURCE-VERBATIM] *"During RL training, Composer's coding ability improves substantially to the point where it begins to get most training problems correct. To continue increasing intelligence, we both select for and create harder tasks dynamically throughout the run. Composer 2.5 is trained with 25x more synthetic tasks than Composer 2.*
+>
+> *We use a range of approaches for creating synthetic tasks that are grounded in real codebases. For example, one synthetic approach is feature deletion. For these tasks the agent is given a codebase with a large set of tests, and asked to delete code and files in such a way that the codebase remains functional while specific testable features are removed. The synthetic task is to reimplement the feature, and the tests are used as a verifiable reward.*
+>
+> *One downstream consequence of large scale synthetic task creation is that it can cause unexpected reward hacking. As the model became more adept, Composer 2.5 was able to find increasingly sophisticated workarounds to solve the task at hand. In one example, the model found a leftover Python type-checking cache and reverse-engineered the format to find a deleted function signature. In another, it was able to find and decompile Java bytecode to reconstruct a third-party API. We were able to find and diagnose these problems using agentic monitoring tools, but they demonstrate the increasing care necessary for large scale RL."*
+**Critical observations from this text:**
+1. **The "25x more synthetic tasks" is relative to Composer 2**, not an absolute count. No absolute count is given in any source. The Composer 2 report does not mention synthetic tasks at all (it uses real-problem distributions).
+2. **Feature deletion is explicitly described as a TWO-PHASE task:** Phase 1 ("task construction") involves a deleter that *"deletes code and files in such a way that the codebase remains functional while specific testable features are removed."* Phase 2 is the reimplementation task for the agent under training. The blog does NOT say who/what performs the deletion (human? model? program?).
+3. **"A range of approaches"** — the blog explicitly says feature deletion is "one synthetic approach" among a "range." No other generators are named anywhere in either source. The total number of generators is completely unspecified.
+4. **"Select for and create harder tasks dynamically throughout the run"** — this is a SINGLE sentence that bundles two distinct operations: (a) online difficulty filtering/selection ("select for"), and (b) active task generation ("create"). Neither operation's implementation is described.
+5. **"Agentic monitoring tools"** — the only stated mitigation for reward hacking. Absolutely no technical detail about these tools is given. No reward penalties, no static analysis, no sandbox specifications.
+6. **The dynamic curriculum signal is implicit.** The mechanism that drives "select for harder" is unstated. The Composer 2 report (§4) provides the only concrete handle: *"In later stages of training, we use simple heuristics—such as number of turns and thinking tokens of rollouts—to upsample increasingly harder data points."* This is from Composer **2**, not 2.5 — but it is the only stated heuristic.
+### 1B. Targeted RL with textual feedback — every word the 2.5 blog says
+Full "Targeted RL with textual feedback" section (verbatim):
+> [SOURCE-VERBATIM] *"Credit assignment during RL is becoming an increasingly difficult challenge as rollouts can span hundreds of thousands of tokens. When a reward is computed over an entire rollout, it may be hard for the model to tell which specific decision helped or hurt the outcome. This is especially limiting when we want to discourage a localized behavior, such as a bad tool call, a confusing explanation, or a style violation. The final reward can tell us that something went wrong, but it is a noisy signal for where it went wrong.*
+>
+> *To address this, we trained Composer 2.5 with targeted textual feedback.[footnote 1] The idea is to provide feedback directly at the point in the trajectory where the model could have behaved better. For a target model message, we construct a short hint describing the desired improvement, insert that hint into the local context, and use the resulting model distribution as a teacher. We use the policy with the original context as the student and add an on-policy distillation KL loss that moves the student's token probabilities toward the teacher's. This gives us a localized training signal for the behavior we want to change, while still retaining the broader RL objective over the full trajectory.*
+>
+> *As an illustration of the text feedback process, consider a long rollout that includes a tool call error where the model attempts to call a tool that is not available. During the rollout, the model will receive a "Tool not found" error and continue making additional valid tool calls. The fact that it hit one error in the process of hundreds of tool calls will have a minimal impact on its final reward.*
+>
+> *With text feedback, we can target this specific mistake by inserting a hint in the context of the problematic turn, such as "Reminder: Available tools…" with a list of available tools. This hint changes the probabilities for the teacher, lowering those for the wrong tool and increasing those for a valid replacement. For that turn only, we then update the student weights towards to the new probabilities.*
+>
+> *During the Composer 2.5 run, we applied this method to a variety of model behaviors, from coding style to model communication."*
+**Footnote 1** (verbatim list from the blog's closing line): "For more background on this approach see Self-Distillation Enables Continual Learning, Reinforcement Learning via Self-Distillation, and Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models."
+**Critical observations:**
+1. **The teacher = hint-conditioned forward pass of the SAME weights.** The blog says "use the resulting model distribution as a teacher" after inserting the hint into the local context. This is not a separate model. The weights do not change for the teacher pass.
+2. **Student = policy with the ORIGINAL context.** The student is trainable; the teacher is stop-gradient.
+3. **The KL is applied "for that turn only"** — not over the full trajectory. This is a LOCALIZED loss.
+4. **HOW HINTS ARE CONSTRUCTED is never stated.** The blog says "we construct a short hint describing the desired improvement." The construction mechanism (heuristic templates? LLM judge? human?) is completely absent.
+5. **Behavior targets explicitly named:** "coding style," "model communication," and implicitly "tool use" (from the example). The blog also separately mentions "effort calibration" as a behavioral improvement. That is four distinct behavior classes.
+6. **The interaction with the main RL objective is stated:** "while still retaining the broader RL objective over the full trajectory." The SDPO loss is ADDITIVE to the main reward — it does not replace it.
+### 1C. RL algorithm — from the Composer 2 Technical Report (§4.1)
+This is NOT in the 2.5 blog. The RL algorithm is fully specified only in the Composer 2 report:
+> [SOURCE-VERBATIM] *"We use a policy gradient algorithm with multiple samples per prompt and a fixed group size. We operate in the single-epoch regime, i.e., the same prompt is never trained on twice. We utilize Adam as our underlying optimizer and update the full parameter set."*
+> [SOURCE-VERBATIM] *"As in Dr. GRPO, we found that it is crucial to minimize the bias in the gradients that can arise from transforming the underlying advantage. Following this work, we remove the length standardization term from GRPO as it introduces a length bias. We do not normalize group advantages by their standard deviation, as it results in the degenerate case where small behavioral differences get massively upweighted within a group where every rollout achieves equal correctness."*
+> [SOURCE-VERBATIM] *"We did not see benefits with overlong masking at small scale and opted not to mask rollouts that exceed the maximum sequence length."*
+KL regularization (§4.1):
+> [SOURCE-VERBATIM] *"Similar to prior work, we use a Kullback–Leibler divergence for regularization, KL(q‖p) = E_{x~q}[−log r(x)], r(x)=p(x)/q(x). Many open-source implementations of RL estimate KL with the estimator k3=(r−1)−log r... However, Amini et al. shows that the variance increases drastically as p and q diverge. See Figure 4: for large KL values, the variance of the estimate is extremely large. Therefore, we use the standard estimator k1=−log r instead."*
+Dynamic curriculum (§4):
+> [SOURCE-VERBATIM] *"In later stages of training, we use simple heuristics—such as number of turns and thinking tokens of rollouts—to upsample increasingly harder data points."*
+Agent behavior shaping (§4.2):
+> [SOURCE-VERBATIM] *"For behavior and communication, we apply an array of auxiliary rewards to ensure the model provides a good experience. These include rewards for coding style, communication, and product-specific penalties for poor tool calls, such as creating to-do list items and then leaving them unfinished. During RL training, we monitor the model for emergent behaviors and occasionally introduce additional behavior rewards as needed. For example, we observed that the model would start to leave long chains-of-thought in comments or collapse to using the terminal tool only."*
+Nonlinear length penalty (§4.2):
+> [SOURCE-VERBATIM] *"C_length{k,q}(x) = ((1 + kx)^{1−q} − 1) / (k(1−q)), where k and q are hyperparameters which define the curvature of the penalty, and the input x is a weighted combination of thinking tokens, tool calling tokens, tool output tokens, final message tokens, number of tool calls, and number of turns of a rollout."*
+Self-summarization (§4.1):
+> [SOURCE-VERBATIM] *"Each training rollout can involve multiple generations chained together by summaries, rather than a single prompt–response pair. We use the final reward for all tokens produced by the model in the chain. This upweights both the agent responses in good trajectories and also the self-summarizations that made them work."*
+### 1D. Infrastructure (Anyrun) — Composer 2 Technical Report (§6.2)
+> [SOURCE-VERBATIM] *"Environments are run on top of Anyrun, an internal compute platform built for running untrusted code at scale. This is the same compute platform that powers Cloud Agents and Automations in the Cursor product."*
+> [SOURCE-VERBATIM] *"Within a cluster, a distributed set of Anyrun managers schedule pods, scale cloud compute provisioned across multiple regions, and perform state reconciliation to manage hundreds of thousands of pods per cluster. Each pod is a dedicated Firecracker VM capable of running a full development environment, including a browser and GUI for computer use."*
+> [SOURCE-VERBATIM] *"Each Anyrun cluster is capable of scheduling more than 500 pods per second."*
+Weight sync (§6.2):
+> [SOURCE-VERBATIM] *"Every training step, we synchronize updated weights to the inference engine by uploading to a shared S3 bucket. To minimize transfer size, we use delta compression: each rank caches its previous upload and transmits only the diff against the new weights. Because RL updates are small, even with full-parameter training these diffs compress to a handful of gigabytes for the 1T-parameter model."*
+Inference partner:
+> [SOURCE-VERBATIM] *"We partner with Fireworks AI to run RL inference."*
+### 1E. Sharded Muon and dual mesh HSDP — from the 2.5 blog only
+The full text of the "Sharded Muon and dual mesh HSDP" section (verbatim):
+> [SOURCE-VERBATIM] *"For continued pretraining, we use Muon with distributed orthogonalization. After forming the momentum update, we run Newton-Schulz at the model's natural granularity: per attention head for attention projections, and per expert for stacked MoE weights.*
+>
+> *The main cost is orthogonalizing expert weights. For sharded parameters, we batch same-shaped tensors, all-to-all shards into complete matrices, run Newton-Schulz, then all-to-all the result back to the original sharded layout. These transfers are asynchronous: while one task is waiting on communication, the optimizer runtime advances other Muon tasks, overlapping network and compute. This is equivalent to full-matrix Muon, but keeps the shard group busy; on the 1T model, optimizer step time is 0.2s.*
+>
+> *This interacts closely with how we use HSDP for MoE models. HSDP forms multiple FSDP replicas and all-reduces gradients across corresponding shards. We use separate HSDP layouts for non-expert and expert weights: non-expert weights are comparatively small, so their FSDP groups can stay narrow, often within a node or rack, while expert weights hold most of the parameters and most of the Muon compute, so they use a wider expert sharding mesh.*
+>
+> *Keeping these layouts separate also lets independent parallelism dimensions overlap: CP=2 and EP=8 can run on 8 GPUs instead of requiring 16 in a single shared mesh. This avoids wide communication for small non-expert state while spreading expert optimizer work over many GPUs."*
+**Critical observation:** Muon + HSDP is described in the 2.5 blog as being used for **continued pretraining**. The Composer 2 report (§6.1) describes a different sharding scheme for Composer 2 (FSDP + decoupled EP + CP, with AdamW/Adam, no Muon). These are two different systems for two different model versions.
+---
+## PART 2 — Critical discrepancies between the repo's research notes and the primary sources
+### FINDING-1: research/01 claims benchmarks not in either blog
+**REPO-CLAIM (research/01):**
+> *"CursorBench 69.3%, Terminal-Bench 2.0 parity"* — attributed to the 2.5 blog.
+> *"On Cursor's internal CursorBench, Composer 2.5 scored 69.3% (or ~61-63% depending on the specific benchmark version cited)"*
+**SOURCE FACT:** The **2.5 blog contains NO benchmark numbers at all.** The 2.5 blog is a brief, methods-focused post with zero numerical results. Benchmark numbers only appear in the **Composer 2 technical report** (Table 1): Composer 2 = 61.3 / 73.7 / 61.7. The 69.3% figure appears nowhere in either primary source.
+**SEVERITY:** High. The 69.3% figure and "Terminal-Bench 2.0 parity" are fabrications or secondary-source-only claims presented as if Cursor-stated. The audit note at the top of research/01 correctly flags this as "[NOT in the Cursor blog]," which means this file's own audit notice is accurate — but the body of the file still asserts these numbers as though Cursor-stated. Any pipeline design that uses these numbers as performance targets is using unverified figures.
+### FINDING-2: The "25x more synthetic tasks" claim is correctly reported but its scope is misread
+**REPO-CLAIM (research/01, research/06):** Accurately quotes the 25x figure. research/06 correctly identifies it as Composer 2.5 vs Composer 2.
+**SOURCE FACT:** [CONFIRM] The 2.5 blog says verbatim: *"Composer 2.5 is trained with 25x more synthetic tasks than Composer 2."* The absolute number is NEVER stated. The Composer 2 report gives **no figure** for how many synthetic tasks Composer 2 used — it describes only "a problem distribution that reflects the most common use cases" with a category histogram (Fig. 3). There is therefore no baseline from which "25x more" can be translated to an absolute count.
+**FINDING:** research/06 says "Reaching a '25×-spirit' pool of ~50k–60k tasks" but this number has no grounding in the primary sources. With no Composer 2 synthetic-task count, 25x is not convertible to an absolute. **The 50k–60k estimate is entirely [EXTRAPOLATED]** and is not sourced from the primary documents.
+### FINDING-3: Feature deletion task structure — the "two agents" framing is inferred, not stated
+**REPO-CLAIM (research/06, §1):** Interprets feature deletion as a "two-agent / two-phase structure the blog implies" with a distinct "deleter" and a "reimplementation agent."
+**SOURCE FACT:** The blog says *"the agent is given a codebase with a large set of tests, and **asked to delete code and files** in such a way that the codebase remains functional while specific testable features are removed."* The subject "the agent" is grammatically the SAME agent doing deletion and then the synthetic task is reimplementation. The blog does NOT cleanly describe two separate agents — it could be interpreted as: (1) an agent tasked to do the deletion [two-phase construction], or (2) the blog is describing the construction process generally. The "deleter model vs. program" question (flagged as an open question in research/06) is therefore genuinely open — and the current analysis of "two agents" may be over-reading the blog's grammar.
+**RECOMMENDATION:** Treat the deleter as unknown. The programmatic AST-deletion approach in research/06 is well-reasoned but is [EXTRAPOLATED]. The blog could equally describe a model-driven deletion step.
+### FINDING-4: The "other 24 generators" claim is hallucinated
+**REPO-CLAIM (research/01, §2):** *"Feature Deletion + 24 unnamed generators"* — a specific count of 24 additional generators.
+**SOURCE FACT:** The blog says: *"We use a **range of approaches** for creating synthetic tasks."* "Range" has no numeric content. Neither the blog nor the Composer 2 report names ANY other generator beyond feature deletion. "24 unnamed generators" is an unsourced extrapolation that has been transcribed as if it were a blog claim.
+**SEVERITY:** Medium. This affects how the dataset-pipeline scope is scoped. The correct statement is: "feature deletion is the one named generator; the total count and names of others are unknown."
+### FINDING-5: The RL algorithm in research/01 is labeled [EXTRAPOLATED] — now resolved
+**REPO-CLAIM (research/01):** *"RL Algorithm: Use a PPO or GRPO variant, modified for long-horizon sparse rewards"* — labeled as [EXTRAPOLATED] by the audit note. This was correct at the time of writing.
+**SOURCE FACT (Composer 2 Technical Report, §4.1):** [CONFIRM] The algorithm is now known from the primary source: **Dr. GRPO-style** (Liu et al., arXiv:2503.20783). Specific modifications:
+- Remove the length-standardization term.
+- Do NOT normalize group advantages by their standard deviation.
+- k1 KL estimator (−log r), not k3.
+- Adam optimizer, single-epoch regime, full-parameter update.
+- Overlong masking EXPLICITLY REJECTED.
+research/10 correctly captures all of this as [REPORT-VERIFIED]. The issue is that research/01 has never been updated to reflect the Composer 2 findings.
+### FINDING-6: "85% of total compute is post-training" — confirmed as community consensus, not Cursor-stated
+**REPO-CLAIM (research/01, §Overview):** *"roughly 85% of the total compute budget for Composer 2.5 was spent on Cursor's proprietary post-training and Reinforcement Learning (RL) pipeline."*
+**SOURCE FACT:** Neither the 2.5 blog nor the Composer 2 technical report contains any compute budget percentages. The 2.5 blog contains NO compute cost figures. The Composer 2 report mentions only that the production training job "spanned 3 GPU regions + 4 CPU regions." This 85% figure is community speculation and should not be stated as a fact in any pipeline design document.
+### FINDING-7: Sharded Muon is a 2.5 CPT optimization, NOT a Composer 2 optimization
+**REPO-CLAIM (research/01):** Presents Muon as a Composer 2.5 feature; also discusses alongside "Dual Mesh HSDP."
+**SOURCE FACT from the 2.5 blog:** Muon + HSDP is explicitly for "continued pretraining" only. The blog section is titled "Sharded Muon and dual mesh HSDP" and begins: "For continued pretraining, we use Muon."
+**SOURCE FACT from Composer 2 Technical Report (§6.1):** Composer 2 uses **AdamW** for CPT and **Adam** for RL. No Muon. Sharding is FSDP + decoupled EP + Context Parallelism (CP). "HSDP" does not appear in the Composer 2 report — the relevant construct is FSDP + CP.
+**FINDING:** The 2.5 blog's "HSDP" formulation differs architecturally from Composer 2's FSDP+CP system. They are described as separate evolutionary steps. research/01's treatment of these as the same system is correct for the evolution, but care must be taken not to conflate Composer 2 sharding details with the 2.5 blog's Muon+HSDP description.
+### FINDING-8: "Anyrun" — correct source attribution now confirmed
+**REPO-CLAIM (research/01, audit note):** "Anyrun environment harness with LSP/file-I/O/terminal — likely from the Composer 2 report, not 2.5."
+**SOURCE FACT:** [CONFIRM] Anyrun is mentioned in BOTH sources. The 2.5 blog does not name Anyrun explicitly; the Composer 2 Technical Report (§6.2) provides full internals. research/09 already correctly flagged and confirmed this. The audit note in research/01 is accurate.
+### FINDING-9: The hint-generation mechanism remains the #1 reproducibility gap — research/07 correctly frames this
+**REPO-CLAIM (research/07):** Correctly identifies hint generation as fully unstated by Cursor. Proposes a layered (a)→(b)→(c)→(f) generator taxonomy.
+**SOURCE FACT:** [CONFIRM] Neither the 2.5 blog nor the Composer 2 Technical Report contains any description of how hints are constructed. The Composer 2 report does not even contain the hint mechanism (it is a 2.5-only feature; Composer 2 uses auxiliary scalar rewards instead). The entire hint-generation design in research/07 is well-reasoned extrapolation from SDPO/OPSD papers, appropriately labeled [EXTRAPOLATED].
+### FINDING-10: Targeted textual feedback is confirmed ABSENT from Composer 2
+**REPO-CLAIM (research/10):** Correctly states "The Composer 2 technical report contains NO hint-generation / teacher-student textual-feedback mechanism."
+**SOURCE FACT (Composer 2 Technical Report, §4.2):** [CONFIRM] Composer 2 uses ONLY auxiliary scalar rewards for behavior shaping, plus the nonlinear length penalty. The targeted-textual-feedback method is exclusively a Composer 2.5 contribution. This is a critical boundary for dataset-pipeline design: anything built to replicate Composer 2.5's SDPO layer cannot be validated against the Composer 2 report.
+### FINDING-11: The CPT→RL causal claim is in the Composer 2 report, not any blog
+**REPO-CLAIM (research/09, §1):** Correctly flags "cross-entropy loss is predictive of downstream RL performance" as coming from the Composer 2 report.
+**SOURCE FACT (Composer 2 Technical Report, §3.1):** [CONFIRM] Verbatim: *"cross-entropy loss is indeed predictive of downstream RL performance"* — demonstrated via Qwen3-Coder-30B-A3B at three log-spaced CPT compute levels. This is the empirical justification for doing CPT at all and is a Composer 2 finding, not stated in the 2.5 blog.
+**IMPLICATION for pipeline design:** Starting from Qwen3-Coder-7B or similar already-code-tuned models as proposed in research/06 is supported by this finding. However, note that the Composer 2 team's CPT recipe (3-phase: 32k bulk → 256k long-context extension → SFT) is not replicable without knowing the data mix percentages and token counts, which remain unstated.
+### FINDING-12: Dynamic curriculum implementation — only Composer 2 provides a concrete handle
+**REPO-CLAIM (research/09, research/06):** The 2.5 blog says "select for and create harder tasks dynamically throughout the run" — interpreted as an online curriculum.
+**SOURCE FACT:** The 2.5 blog gives ZERO implementation detail for the curriculum. The ONLY concrete curriculum handle in either primary source is from the **Composer 2 Technical Report** (§4): *"In later stages of training, we use simple heuristics—such as number of turns and thinking tokens of rollouts—to upsample increasingly harder data points."*
+**FINDING:** This is a Composer 2 mechanism being mapped onto the 2.5 description. Whether Composer 2.5 uses the same heuristic (turns + thinking tokens) or a different one is unknown. The PassRateCurriculum in research/06 uses pass-rate as the difficulty signal, which is a reasonable [EXTRAPOLATED] alternative to Cursor's stated turns/thinking-tokens heuristic, but is NOT what Cursor described.
+### FINDING-13: Reward structure — test pass-fraction vs. correctness/succinctness/principles
+**REPO-CLAIM (research/06):** Designs reward as "test pass fraction (|FAIL_TO_PASS passing| / |FAIL_TO_PASS|), masked by the hack monitor."
+**SOURCE FACT (Composer 2 Technical Report, §2):** *"a reward is given based on the code's correctness, succinctness, and conformance to software engineering principles."* The Composer 2 reward is multi-dimensional — not purely test pass-fraction. The "succinctness" and "conformance to software engineering principles" components are only partially captured by the length penalty; they imply additional rubrics or reward signals beyond pass/fail.
+**FINDING:** For Feature Deletion specifically, test pass-fraction is the natural verifiable reward and is consistent with the 2.5 blog's description. But the broader reward structure for non-feature-deletion RL tasks includes quality and style components not reducible to test pass rates.
+### FINDING-14: Self-summarization is present in Composer 2, not mentioned in 2.5 blog
+**REPO-CLAIM (research/01):** Does not mention self-summarization.
+**SOURCE FACT (Composer 2 Technical Report, §4.1):** Self-summarization is described as introduced in Composer 1.5 and carried into Composer 2. The 2.5 blog does not mention it at all. It is therefore likely carried into 2.5 but is not described as a new 2.5 feature. Research notes should treat this as a "likely continued from Composer 2" feature, not a 2.5 innovation.
+### FINDING-15: MoE router replay — a critical infrastructure detail not in any repo note
+**SOURCE FACT (Composer 2 Technical Report, §6.2):** *"during inference, the engine returns the selected expert indices for every token at every MoE layer, and during the training forward pass the router's expert assignment is overridden to match."* Extended by filtering replayed experts below a plausibility threshold. This addresses the MoE numerical mismatch between inference and training forward passes.
+**FINDING:** This detail is critical for any MoE-base RL implementation (relevant if the framework uses Kimi K2.5 or another MoE). It is described in research/10 correctly, but it is absent from research/01 and the design docs. Any replication plan that assumes "just run GRPO on a MoE" will encounter numerical divergence issues that this router-replay mechanism is specifically designed to solve.
+### FINDING-16: Base model selection — Composer 2 used specific criteria that precluded agentic benchmarks
+**SOURCE FACT (Composer 2 Technical Report, Appendix B):** *"We intentionally do not consider coding agent benchmarks when testing base models. We find that such benchmarks are less predictive of final performance, as agentic and long-horizon capabilities can drastically change during the RL stage."* Base model was selected on: FreshBench (factual coding knowledge), State Tracking (LoCoDiff-style), and internal codebase perplexity.
+**FINDING:** This is an important methodological point absent from research/01. When the replication framework selects a base model (currently proposed as Qwen3-Coder variants), the selection should be made on intrinsic knowledge benchmarks (perplexity on domain code, state tracking) rather than SWE-bench or similar agentic benchmarks — consistent with Cursor's own methodology.
+---
+## PART 3 — What the sources DO NOT say (reproducibility gaps)
+These are the load-bearing unknown quantities for building the dataset-generation pipeline:
+| Gap | What's stated | What's missing |
+|---|---|---|
+| **Hint construction** | "we construct a short hint" | HOW — templates? LLM judge? human? learned? |
+| **Synthetic generator inventory** | "a range of approaches… for example, one… is feature deletion" | Any other generator name, count, or weighting |
+| **Synthetic task absolute count** | "25x more than Composer 2" | Absolute number; Composer 2 baseline count |
+| **Dynamic curriculum implementation** | Composer 2: "turns + thinking tokens to upsample"; 2.5: "select for and create" | 2.5's specific curriculum signal; whether pass-rate is used |
+| **Feature deletion: deletion agent** | "the agent… asked to delete code and files" | Whether "the agent" is a model, program, or human pipeline |
+| **Feature deletion: target selection heuristic** | "delete code and files in such a way that the codebase remains functional while specific testable features are removed" | HOW deletion targets are selected from a repo |
+| **Feature deletion: languages** | Python (type-check cache) and Java (bytecode) implied by reward-hack examples | No explicit language list |
+| **Reward-hacking mitigations** | "agentic monitoring tools" | No technical specification |
+| **CPT data mix** | "large code-dominated data mix"; "3 phases" (32k bulk → 256k → SFT) | Token counts, percentages, data sources |
+| **Behavioral reward signals** | Scalar "rewards for coding style, communication" (Composer 2); 2.5 extends to hint-distillation | No RM architecture, no rubric spec |
+| **Muon optimizer for CPT** | "For continued pretraining, we use Muon with distributed orthogonalization" | Learning rate, hyperparameters, data ordering during CPT |
+| **SpaceXAI / Colossus 2 model** | "training a significantly larger model from scratch, using 10x more total compute" | Everything; this is a future model not Composer 2.5 |
+---
+## PART 4 — Implications for building the dataset-generation pipeline
+Based on the primary-source analysis, the following are the **only verifiably-sourced facts** the pipeline design can be grounded on:
+### What IS in the sources and can be directly implemented:
+1. **Feature deletion as described:** give agent a test-covered repo → delete a testable feature (keeping repo otherwise functional) → reward = passing the tests. The blog's formulation implies the deletion maintains a `PASS_TO_PASS` guard (the "codebase remains functional" requirement). This is the exact formulation in research/06 and is faithful.
+2. **Verifiable test-based reward:** "the tests are used as a verifiable reward." Test pass-fraction is the correct scalar. No golden patch is needed at reward time.
+3. **Online curriculum — turns + thinking-token upsampling (Composer 2 handle):** The only stated heuristic. Difficulty = rollout length / thinking-token count. This is simpler than the pass-rate curriculum in research/06 but is the only Cursor-sourced signal.
+4. **Targeted textual feedback — exact mechanism:** (a) construct hint, (b) insert into local context, (c) teacher = hint-conditioned forward pass of same weights (stop-grad), (d) student = policy without hint, (e) on-policy KL loss on student for that turn only, (f) stack on top of the main RL objective.
+5. **Dr. GRPO modifications (Composer 2):** Remove length-standardization, remove std-norm advantage normalization, use k1 KL (−log r), Adam optimizer, single-epoch regime.
+6. **Auxiliary scalar rewards for behavior (Composer 2):** coding style, communication, tool-call quality penalties. These are the Composer 2 behavior-shaping mechanism; in 2.5 they are SUPPLEMENTED (not replaced) by the hint-distillation channel.
+7. **Reward = correctness + succinctness + SE principles (Composer 2):** Multi-dimensional, not just test pass. The nonlinear length penalty C_length is the explicit succinctness component.
+### What CANNOT be directly sourced and requires [EXTRAPOLATION]:
+1. The hint construction mechanism — all of research/07 is [EXTRAPOLATED] (well-reasoned but not Cursor-stated).
+2. Any synthetic generator beyond feature deletion.
+3. The absolute scale of the task bank (50k, 60k, or any other number).
+4. The pass-rate curriculum (a reasonable design choice but not what Cursor describes for Composer 2 or 2.5).
+5. The CPT data mix specifics.
+6. The reward-hacking detection tooling specifics.
+---
+## PART 5 — Verdict on the repo's existing research notes
+| Note | Accuracy vs. primary sources |
+|---|---|
+| `research/01-composer-2.5.md` | Body is substantially from secondary sources. Audit note at top is accurate in flagging its own errors. The benchmark numbers (69.3%, Terminal-Bench parity) are not in either primary source. The "24 unnamed generators" claim is hallucinated. Should NOT be used directly as design input; use `research/09` and `research/10` instead. |
+| `research/09-composer-blog-delta-2026.md` | High accuracy. All verbatim quotes verified against fetched blog. Deltas correctly identified. One minor gap: does not flag that the "25x" baseline count (Composer 2 synthetic task count) is also unstated in the Composer 2 report. |
+| `research/10-composer2-techreport-mining.md` | High accuracy. All [REPORT-VERIFIED] claims confirmed against the full HTML. Corrections (optimizer Adam not Muon; FSDP+CP not HSDP; hint mechanism absent) are correct. The "k1 in reward KL" claim (also referenced in commit bd37412) is verified against §4.1. |
+| `research/06-feature-deletion-datagen.md` | Well-constructed design brief. Correctly labeled [BLOG-VERIFIED] vs [EXTRAPOLATED]. One concern: the "50k–60k tasks" scale estimate has no primary-source grounding. The PassRateCurriculum (pass-rate as difficulty) is [EXTRAPOLATED] and departs from Cursor's stated heuristic (turns + thinking tokens). Otherwise faithful. |
+| `research/07-sdpo-hint-generator.md` | Well-constructed. Correctly identifies the open question. The taxonomy (a)→(b)→(c)→(d)→(f) is well-grounded in SDPO/OPSD papers. The "successful-sibling bootstrap" as a fallback is directly grounded in SDPO's ablation of "sample solution" feedback type. The single issue: the layered design is predicated on the collator's existing `hint_generator` hook, which should be verified in the actual codebase before committing to this design. |
+---
+## Sources
+- **Composer 2.5 blog** — full body, vault note `introducing-composer-25-cursor`, URL `https://cursor.com/blog/composer-2-5` (fetched 2026-06-09).
+- **Composer 2 Technical Report** — full HTML body, vault note `composer-2-technical-report`, URL `https://arxiv.org/html/2603.24477` (arXiv:2603.24477 v2, 26 Mar 2026). 11 904 words, all sections including appendices.
+- **arXiv abstract note** — vault note `260324477-composer-2-technical-report`, cross-checks author list and abstract wording.
+- **Prior internal notes** — `research/01`, `research/09`, `research/10`, `research/06`, `research/07` as cross-reference for repo claims.

research/deepread/02-swe-task-synthesis.md ADDED Viewed

	@@ -0,0 +1,340 @@

+# Deep-Read: SWE Task Synthesis Prior Art — Critical Review
+**Cluster:** Open-source task-synthesis literature ("point at a repo, build a dataset")
+**Date:** 2026-06-09
+**Reviewer:** Claude (critical-review pipeline, subagent)
+**Sources fetched:** SWE-smith HTML (arXiv:2504.21798, 26k words), SWE-Gym HTML (arXiv:2412.21139, 10k words), R2E-Gym HTML (arXiv:2504.07164, 14k words), SWE-bench abstract (arXiv:2310.06770), SWE-MiniSandbox (arXiv:2602.11210), SWE-RL (arXiv:2502.18449), DeepSWE blog (Together AI), SkyRL GitHub, SWE-smith HF dataset page, SWE-smith GitHub README.
+**Repo artifacts compared:** `composer_replication/datagen/substrates.py` (SweBenchAdapter), `research/06-feature-deletion-datagen.md`, `docs/adrs/ADR-010-feature-deletion-datagen.md`.
+---
+## 1. SWE-smith (arXiv:2504.21798) — Deep Read
+### 1.1 What it actually does (task synthesis mechanics)
+SWE-smith's core insight is stated verbatim in §2: "Conceptually, this is a simple inversion of SWE-bench's approach, which instead prioritizes identifying task instances, and then attempts to build an environment for each." SWE-smith **builds an execution environment first**, then synthesizes many tasks within it. This is the opposite of SWE-bench's PR-mining approach.
+**Environment construction (§2.1):**
+1. Target the top-5000 most-downloaded PyPI packages by stars (≥1000 stars), excluding the 12 SWE-bench test repos.
+2. Run SWE-agent on the latest commit for ≤100 steps to auto-install + run test suite.
+3. Manually verify installation instructions and check >80% tests pass.
+4. Build one Docker image per repo (not per task — this is the key scalability win).
+5. Total: 128 repos selected, ~7 min human labor per repo for step 2 correction, ~1 min for test parser.
+6. Total human labor: ~20 hours to create the entire 50k dataset.
+**Four bug-synthesis strategies (Table 1 from the paper):**
+| Strategy | Yield % | # Instances | Cost/instance | Median F2P | Median Lines Edited |
+|---|---|---|---|---|---|
+| LM Modify | 56.0% | 17,887 | 0.38¢ | 4 | 3 |
+| LM Rewrite | 35.0% | 4,173 | 3.93¢ | 4 | 24 |
+| Procedural | 40.2% | 15,641 | 0.00¢ | 7 | 5 |
+| Combine (Cross-bug) | 96.9% | 10,092 | 0.00¢ | 15 | 11 |
+| PR Mirror (Invert PRs) | 33.8% | 2,344 | 5.53¢ | 3 | 14 |
+| **Total** | **50.1%** | **50,137** | **2.32¢** | | |
+**Key detail on bug strategies:**
+- **LM Modify:** Prompt LM to "introduce errant modifications" to a function. Yield 56% (LM doesn't always break a test). Input: full function.
+- **LM Rewrite:** Given only function header + docstring, ask LM to re-implement. Yield 35% (lower because it's not explicitly asked to break anything). Generates longer changes (24 lines median). Input: signature only, so LM can't see the original.
+- **Procedural Modification (§B.2):** Zero-cost AST transformations. 13 types (see §B.2 verbatim):
+  - Class: Remove Functions (removes method(s) + references); Remove Parent (removes base class); Shuffle Methods.
+  - Control: Invert If/Else (inverts if-else bodies).
+  - Flow: Shuffle Lines (shuffles lines of function).
+  - Expressions: Change Constants (±1 to numeric); Break Chains (removes operators); Swap Operands; Change Operator (+→-).
+  - Removal: Remove Loops (for/while); Remove Conditionals (if); Remove Assignments; Remove Wrappers (try/with).
+  - Applied with a `likelihood` parameter so modifications don't make tasks too hard. Filtered by function/class criteria (complexity min/max gates).
+- **Combine Bugs (§B.3):** Combine already-validated bugs from same file or module into a multi-bug task. 96.9% yield because each component already passes validation. Creates complex multi-site tasks.
+- **PR Mirror (§B.4):** For each PR in the repo's history, prompt LM to "undo" the PR's changes in the current codebase (NOT checkout the base commit — this is the key difference from SWE-bench). LM rewrites each affected file. Expensive (rewrites whole files) and lower yield (33.8%). Most reflective of SWE-bench distribution (PR Mirror trajectories are the best training data per Table 5).
+**Issue text generation (§2.1, last subsection):** Provide LM with (the diff, source code of a random F2P test, test execution output), ask for GitHub-style issue text with reproduction code. Empirically as good as real issues (28.2% vs 28.0% on SWE-bench Verified in Table 5).
+**Validation:** Apply bug patch, run test suite, keep only bugs that break ≥1 test. Time limit: 2 minutes per test run (bugs causing infinite loops discarded).
+**Scale and storage (Table 2 in §2.2 comparison):**
+- SWE-smith: 50k tasks, 128 repos, 295 GB environments.
+- SWE-Gym: 2.4k tasks, 11 repos, 6 TB environments.
+- R2E-Gym subset: 4.6k tasks, 10 repos, 4 TB environments.
+- SWE-fixer (Xie et al. 2025a): 115k tasks, 856 repos — but NO execution environments.
+- SWE-bench-train: 19k tasks, 37 repos — NO executable environments.
+The one Docker image per repo (vs. SWE-bench's one image per task) is the mechanism: estimated 500x storage reduction at 50k tasks. SWE-bench at 50k would need 50-150 TB; SWE-smith uses 295 GB.
+**Training results (§3-4):**
+- SWE-agent-LM-32B (Qwen 2.5 Coder 32B fine-tuned on 5,016 SWE-smith trajectories) achieves 40.2% on SWE-bench Verified.
+- Expert model for trajectory collection: claude-3-7-sonnet-20250219 + SWE-agent, ≤75 steps, $2 cost limit.
+- PR Mirror and LM Rewrite trajectories produce the best-performing models; LM Modify has a steep drop-off.
+- GRPO-style RL on SWE-smith is explicitly mentioned in the SWE-smith GitHub README: "Perform GRPO style reinforcement learning using SkyRL."
+**Licensing (Table 6, §A.4):**
+- 128 repos cover Apache 2.0 (majority), BSD 2/3-Clause, MIT, GNU GPLv3, LGPL v2.1 and v3, ISC.
+- GPLv3 repos: Cog-Creators/Red-DiscordBot and adrienverge/yamllint. LGPL: chardet/chardet, paramiko/paramiko, pylint-dev/astroid, Knio/dominate.
+- Paper states: "We inspected the repositories with custom licenses and confirmed they allowed for the use cases exercised in our work." (Note: does NOT explicitly address whether derivative diffs are redistributable — they exercise the bugs, not redistribute.)
+- **Dataset HF license:** The SWE-smith HF dataset page (`SWE-bench/SWE-smith`) shows **59,136 rows** (note: larger than the 50,137 paper reports — likely includes additional variants). The page says "We will no longer actively update this dataset. Recommend the language-specific `SWE-bench/SWE-smith-[lang]` datasets."
+- **Toolkit license: MIT** (confirmed from GitHub).
+**`pip install` / API availability:**
+- The GitHub README shows a Python API: `from swesmith.profiles import registry; rp = registry.get_from_inst(task); container = rp.get_container(task)` — returns a Docker container with the task initialized.
+- Install: `pip install swesmith` from source (requires Docker, Ubuntu 22.04.4 LTS, does NOT support Windows or MacOS officially).
+- The toolkit is a full pipeline: create environments → synthesize task instances → keep tasks that break ≥1 unit test → generate issue text.
+---
+## 2. SWE-Gym (arXiv:2412.21139) — Deep Read
+### 2.1 What it actually provides
+SWE-Gym is **not a synthesis pipeline** — it is an existing dataset of 2,438 real-world task instances from 11 Python repos with pre-built executable environments. It is the "first training environment" for SWE agents (per abstract).
+**Collection method (§3 of paper):** Same approach as SWE-bench (mine PRs), but:
+- Applied to 11 **different** repos from SWE-bench's 12 repos (deliberately non-overlapping for train/test separation).
+- Per-task Docker images (6 TB total — this is the bottleneck SWE-smith solved).
+- 66,894 raw task instances in SWE-Gym-Raw (no executable envs); 2,438 have full envs.
+**Table 2 statistics (SWE-Gym vs SWE-bench):**
+- Average gold patch: 69.8 lines edited, 2.5 files, 4.1 functions (much larger than SWE-bench's 32.8/1.7/3.0).
+- Average F2P tests: 10.0 (vs SWE-bench 9.0).
+- Average total tests: 760.8 (vs SWE-bench 132.5) — much more test coverage per instance.
+- Average codebase: 971 non-test files, 340k lines.
+**Training results:**
+- 491 trajectories from GPT-4o and Claude 3.5 Sonnet → Qwen 2.5 Coder 32B fine-tuned → +12.3%/+13.6% on SWE-bench Lite/Verified.
+- With verifier (OSRM): +11.4% more → 32.0% on Verified, 26.0% on Lite.
+- Scaling: performance still improving at 491 trajectories, no saturation yet.
+**License:** SWE-Gym tooling is MIT. Per-instance instances inherit upstream repo licenses (paper does not provide a per-instance breakdown, unlike SWE-rebench).
+**Availability:** `SWE-Gym/SWE-Gym` on HuggingFace. Docker images hosted on Docker Hub as `xingyaoww/sweb.eval.*`. Pre-built images reduce env-build cost to near-zero for adopters.
+---
+## 3. R2E-Gym (arXiv:2504.07164) — Deep Read
+### 3.1 The SweGen pipeline — the most novel test synthesis approach
+R2E-Gym's key contribution for this cluster is **SweGen**: a method to generate executable training environments **without human-written issues or unit tests**, directly from commits.
+**SweGen procedure (§2):**
+1. **Repo selection:** Use SEART GitHub search to find Python repos with many commits.
+2. **Commit curation:** Extract commit history; filter with rule-based + LLM heuristics for "interesting" code changes.
+3. **Build scripts:** Semi-manually collect dependency pins and installation procedures per commit.
+4. **Test collection:** Use existing tests from the commit to identify F2P cases (failing before the commit, passing after — the natural oracle).
+5. **Test generation for commits without tests:** Synthesize F2P test cases using an LLM when no test exists. Appended test generation details in Appendix A (not fully extracted, but confirmed it exists).
+6. **Backtranslation for problem statements:** Instead of human GitHub issues, use LLM to backtranslate the commit diff into an issue-style problem statement. Key insight: include the F2P test execution trace in the backtranslation prompt to generate specific, non-generic statements. 27.8% vs 28.0% (synthetic vs real issues) — statistically indistinguishable.
+7. **Decontamination:** Remove repos overlapping with SWE-bench test set → R2E-Gym-Subset (4,578 tasks, 10 repos).
+**Scale:** 8,135 tasks total, from more repos than SWE-Gym. SweGen enables 2.5x more problems than PR-based collection alone. But 4 TB environments for the subset alone — still a storage problem.
+**Oracle quality:**
+- When commits lack existing tests: LLM synthesizes them. The paper claims "Real vs Synthetic: Real 28.0%, Synthetic 27.8%" — essentially identical performance. This is a strong result for synthetic oracle quality.
+- However, the paper does NOT quantify what % of commits had no existing tests (and thus needed synthesized tests) — this is a gap that ADR-010 does not flag.
+**Training results (Table 3):**
+- R2E-Gym-trained 32B model: 34.4% on SWE-bench Verified (vs SWE-Gym 20.6%, +13.8%).
+- With hybrid verifier (execution-based + execution-free): 51% on SWE-bench Verified — SOTA for open-weight models at paper time.
+- DeepSWE (R2E-Gym + GRPO, Qwen3-32B): 42.2% Pass@1, 59% with test-time scaling.
+**License:** Apache 2.0 for tooling (R2E-Gym GitHub). Per-instance upstream licenses inherited.
+**API:** Available on HuggingFace at `R2E-Gym/R2E-Gym-V1` and `R2E-Gym/R2E-Gym-Subset`. Docker images hosted. The code at `r2e-gym.github.io` is the generation pipeline — not a clean `pip install swesmith`-style API, but usable.
+---
+## 4. SWE-bench (arXiv:2310.06770) — Schema Reference
+The canonical task schema (ICLR 2024):
+- Input: `(codebase, issue_description)` at `base_commit`.
+- Output: `git diff` patch resolving the issue.
+- Oracle: run unit tests from `test_patch` against the patched codebase.
+- Fields: `FAIL_TO_PASS` (tests that must go red→green), `PASS_TO_PASS` (tests that must stay green).
+- Construction: mine GitHub issues → find corresponding PRs → filter for PRs that modify test files and resolve the issue → build per-version Docker environments.
+- **The hard part:** building per-version Docker environments is the most costly step (~hundreds of hours human labor at scale). SWE-smith's core contribution is eliminating this by using one image per repo at HEAD.
+- **The oracle property:** FAIL_TO_PASS is a constructive oracle — the tests already existed and were known to exercise the feature (they were in the PR). This is what makes it "verifiable for free."
+- Scale: 2,294 eval instances, 12 Python repos. Training split: 19,008 (no exec environments).
+- License: CC-BY-4.0 for the dataset.
+---
+## 5. Newer Work (2025-2026)
+### 5.1 SWE-MiniSandbox (arXiv:2602.11210, Feb 2026)
+Container-free RL training for SWE agents. Key result: kernel-level isolation (not Docker) reduces disk usage to ~5% of container-based pipelines and setup time to ~25%. Evaluation performance comparable to container baseline. Not a dataset synthesis method, but directly relevant to the sandbox cost problem ADR-010 acknowledges (Docker required, CPU-pool generation cost). **MISSED by research/06 and ADR-010** — this was published after the main research was done (Feb 2026).
+### 5.2 SWE-RL (arXiv:2502.18449, NeurIPS 2025 Main Track)
+RL on real GitHub software evolution data (issues, PRs, code history). No synthetic bug injection — uses rule-based reward (similarity score between ground truth and LLM-generated solution). Llama3-SWE-RL-70B: 41.0% on SWE-bench Verified. Key finding: RL on SWE data transfers to 5 out-of-domain tasks (math, code reasoning, general language). Uses 11M training pairs from open-source software evolution. Reward = similarity (not binary test pass) — this is weaker than the test-execution oracle in SWE-smith/R2E-Gym.
+### 5.3 DeepSWE (Together AI + Agentica, Jul 2025 blog)
+Qwen3-32B + pure RL (GRPO via rLLM framework) on R2E-Gym environments. 4,500 SWE tasks, 64 H100s for 6 days. Achieves 42.2% Pass@1, 59% with test-time scaling (16 attempts). **This is the closest living demonstration of the Composer-style "RL on SWE tasks" recipe using an open-source substrate.**
+### 5.4 SkyRL (NovaSky-AI, 2025)
+Explicitly mentioned in SWE-smith README: "Perform GRPO style reinforcement learning using SkyRL." SkyRL is a full-stack RL library (Berkeley Sky Computing Lab) with SWE-bench environments in `skyrl-gym`. The GitHub shows direct integration: SkyRL + SWE-smith = GRPO on SWE-smith tasks. This is exactly the missing integration link for this repo's `reward_fn` + RL loop design.
+### 5.5 SWE-fixer (Xie et al. 2025a, referenced in SWE-smith Table 2)
+115k instances, 856 repos — largest by task count. But **NO execution environments** — no Docker, no test execution. SWE-fixer trained SWE-fixer-72B achieving 32.8% on SWE-bench Verified. The comparison in SWE-smith Table 2 shows SWE-fixer has zero executable environments, meaning it relies on string-similarity rewards, not test-execution verification. **Not a direct competitor for this repo's verifiable-reward approach.**
+---
+## 6. Comparison Against the Repo's Current State
+### 6.1 What `substrates.py` (SweBenchAdapter) actually does
+`substrates.py` implements **schema inversion only**: it takes an existing SWE-bench-shaped instance dict and converts it to a `FeatureDeletionTask` by reversing the gold patch. It does NOT:
+- Synthesize new tasks from an arbitrary repo.
+- Call SWE-smith's API or any synthesis engine.
+- Implement AST/procedural deletion (Path B from research/06).
+- Build execution environments for new repos.
+- Generate issue text.
+The adapter only handles the "adopt existing substrates" half of the design space. This is correct for the ADR-010 v0.0-v0.1 scope, but it's important to be explicit about what is NOT yet built.
+### 6.2 What research/06 and ADR-010 claim vs. what the sources actually say
+**Correct claims (verified):**
+1. `[research/06 §4, ADR-010]` "SWE-bench FAIL_TO_PASS / PASS_TO_PASS schema is the universal schema across SWE-bench, SWE-Gym, R2E-Gym, SWE-rebench." **VERIFIED.** All four use this schema. SWE-smith also uses it (per §A.1: "FAIL_TO_PASS: The unit tests that break when the test suite is run with the bug patch applied. PASS_TO_PASS: The unit tests that do not break.").
+2. `[research/06 §4]` "R2E-Gym: 8.1K executable envs." **VERIFIED.** Paper abstract states "more than 8.1K tasks."
+3. `[research/06 §4]` "SWE-rebench: 21,336 tasks, 3,468 repos, CC-BY-4.0." **UNVERIFIED here** (SWE-rebench paper arXiv:2505.20411 not directly fetched — only the Nebius infrastructure blog post was available). The infrastructure blog describes the pipeline but does not give the exact 21,336 count. Cannot confirm from fetched sources.
+4. `[research/06 §2, ADR-010]` "Online pass-rate curriculum: 'select for and create harder tasks dynamically.'" **VERIFIED verbatim** in the Cursor blog quote in research/06 §1.
+5. `[research/06 §3]` "Python bytecode/type-check cache recovery is a real reward-hacking vector." **VERIFIED verbatim** in the Cursor blog (quoted in §1 of research/06).
+6. `[ADR-010]` "SWE-smith env-construction: ~20h human labor for 128 repos." **VERIFIED.** Paper §2.1: "Creating SWE-smith took one author ~20h of human labor."
+7. `[ADR-010]` "SWE-smith costs $1,360 to create." **VERIFIED.** Paper §2.2: "$1360 to create ($1000 to generate bugs, $160 for auto repo installation, $200 to generate issues for 10K bugs)."
+**Errors and overclaims in the repo's research notes:**
+1. **`[research/06 §4, ADR-010]` "SWE-Gym: 2.4k tasks, 11 repos, 6 TB."** PARTIALLY CORRECT. SWE-Gym has 2,438 tasks, 11 repos. The 6 TB figure for environments is cited from the SWE-smith Table 2 comparison. However, research/06 states "SWE-Gym (arXiv:2412.21139, ICML 2025)" — the venue claim is correct (ICML 2025 per the paper's header).
+2. **`[research/06 §4]` "R2E-Gym-Subset = non-overlapping w/ SWE-bench."** VERIFIED but requires precision: R2E-Gym-Subset (4,578 tasks, 10 repos) is non-overlapping with SWE-bench TEST SET repos. The full R2E-Gym (8,135 tasks) may overlap with SWE-bench training repos. The paper decontaminates against the test set only.
+3. **`[research/06 §4, ADR-010]` "OpenHands/Nemotron-SWE-v1: 59K agent trajectories."** The HF dataset page for `SWE-bench/SWE-smith` shows 59,136 rows — this is the SWE-smith dataset (with updates), NOT the Nemotron dataset. Research/06 correctly cites `nvidia/Nemotron-SWE-v1` separately. But the 59K figure is ambiguous: SWE-smith also has ~59k rows on HF. The Nemotron dataset is a different thing. This is a minor labeling confusion but not wrong in substance.
+4. **`[research/06 §4]` "SWE-Gym purpose-built for training (train split, not a held-out benchmark → no contamination worry)"** — OVERCLAIM. SWE-Gym paper itself states its 11 repos are separate from SWE-bench's 12, but the decontamination is at the repo level, not task level. Using SWE-Gym for Feature Deletion and then evaluating on SWE-bench should be safe, but the claim "no contamination worry" is stronger than the paper asserts.
+5. **`[research/06 §5, Path B]` "Coverage-mapped AST deletion using libcst — select deletion targets by coverage selectivity."** This is original to the repo and NOT from any of the fetched papers. SWE-smith's procedural approach uses random AST transforms without coverage-guided targeting. R2E-Gym uses real commits (not synthetic deletions). The coverage-selectivity framing is a valid engineering idea but is `[EXTRAPOLATED]` — research/06 correctly tags it `[EXTRAPOLATED]` but the ADR text promotes it to a stated capability without that tag. **ADR-010 should make clearer that Path B (coverage-mapped AST deletion) is not from any prior work — it would need to be built from scratch.**
+6. **`[ADR-010 §2, Decision Drivers]` "Reuse existing verified OSS substrates... they already guarantee test-exercises-the-code via FAIL_TO_PASS."** PARTIALLY OVERCLAIM. SWE-smith (§2.1) explicitly notes that yield rates are limited because some bug candidates "did not actually introduce relevant issues" or "lack test coverage for the change." The FAIL_TO_PASS guarantee is not automatic — it is the *output* of the validation step (test execution), not something inherited from the schema. The ADR implies the guarantee is pre-built; in reality it requires running the test suite to confirm it.
+7. **`[ADR-010 post-review]` "OPEN: Gate 2 ('deletion breaks the feature') does not verify reachability."** This self-identified open item is correct and important. The sources confirm: SWE-smith's validation only checks that some test breaks (not which code causes it). R2E-Gym also checks test pass/fail without coverage verification. This is a systemic gap in the field, not just in the repo.
+**Missing from the repo's research notes:**
+1. **SWE-smith's PR Mirror strategy is the most important one for this repo.** Per Table 5 of the paper: PR Mirror trajectories produce the best-performing models (trajectories from PR mirrors > LM Rewrite ≈ Procedural > LM Modify). The repo's feature deletion via gold-patch reversion (SweBenchAdapter.to_task) is exactly analogous to SWE-smith's PR Mirror strategy. The paper's ablation directly validates the repo's core approach — but neither research/06 nor ADR-010 cite this result.
+2. **SWE-smith's "one Docker image per repo" architecture.** Research/06 and ADR-010 discuss reusing per-instance Docker images from SWE-Gym/SWE-rebench. SWE-smith shows that the correct architecture for new repos is one image per repo (not per task), which is 500x more storage-efficient. If the repo wants to extend beyond the existing substrates, this is the right architecture. Not mentioned.
+3. **SWE-smith `pip install swesmith` — it is a real, usable API.** The GitHub README shows that `rp.get_container(task)` returns a Docker container with the task initialized. The repo could `pip install swesmith` to get task synthesis capabilities today, rather than building a separate synthesis pipeline. ADR-010 discusses options A/B/C but does not mention that Option A ("invert OSS substrates") could be powered by the `swesmith` toolkit directly.
+4. **R2E-Gym's LLM-synthesized tests.** When commits lack existing tests, R2E-Gym synthesizes them with an LLM. Research/06 describes this capability as needing separate building ("the hard part: Composer deletes features where tests exist; SWE-smith generates bugs and validates against existing tests; R2E-Gym synthesizes tests"), but research/06 does not propose actually synthesizing tests — it only uses existing tests from substrates. The gap: if we want to point at an arbitrary repo (the user's stated goal), most arbitrary repos will have commits without comprehensive test coverage. SweGen's test synthesis capability (confirmed equivalent to real tests at 27.8% vs 28.0%) is directly relevant and not mentioned in ADR-010.
+5. **SkyRL + SWE-smith = working GRPO pipeline.** The SWE-smith README explicitly says "Perform GRPO style reinforcement learning using SkyRL." This is a working open-source stack (SkyRL is MIT licensed, 2k GitHub stars). ADR-010's `reward_fn` design reinvents what SkyRL + SWE-smith already provide. Not evaluated in the ADR.
+6. **SWE-MiniSandbox (arXiv:2602.11210, 2026).** Container-free RL at 5% disk / 25% setup time of Docker. Directly addresses ADR-010's acknowledged "sandbox/Docker cost" concern. Not considered — was published after ADR-010.
+7. **DeepSWE: living demonstration of "RL + SWE-smith/R2E-Gym."** Qwen3-32B + GRPO on 4,500 R2E-Gym tasks = 42.2% Pass@1 / 59% with TTS. This is published evidence that the core architecture in ADR-010 (RL + feature-deletion env + GRPO reward_fn) works at scale in the open. Not cited.
+---
+## 7. ADOPT vs BUILD — Concrete Recommendation
+### 7.1 What we can `pip install` TODAY
+**`pip install swesmith` (MIT, from source, requires Docker):**
+- Provides: environment construction from arbitrary GitHub repos, all 4 bug synthesis strategies (LM Modify, LM Rewrite, Procedural with 13 transform types, PR Mirror/Combine), issue text generation, task validation.
+- What the repo currently builds manually in `composer_replication/datagen/` is a subset of what `swesmith` provides.
+- **Recommendation:** Use `swesmith` as the synthesis engine for new repos rather than rebuilding. The repo's `SweBenchAdapter` can remain as the inversion layer for pre-existing substrates.
+**`datasets.load_dataset("SWE-bench/SWE-smith")` (dataset available, MIT toolkit, mixed upstream licenses):**
+- 59k tasks, 128 repos, 295 GB environments (vs. 50,137 in paper — dataset has grown since publication).
+- All tasks have executable Docker environments via `swesmith.profiles.registry`.
+- **This is the largest immediately usable Feature-Deletion dataset**: every SWE-smith task is a `(broken_repo, FAIL_TO_PASS, PASS_TO_PASS)` tuple. The "patch" field IS the gold diff (already reversed in the task construction). `SweBenchAdapter.to_task()` works on SWE-smith instances unchanged.
+- **License risk:** GPLv3 repos (2 repos: Red-DiscordBot, yamllint) are present. The existing `is_redistributable()` copyleft filter in `substrates.py` would catch these. Apache/BSD/MIT majority is clean.
+**`datasets.load_dataset("R2E-Gym/R2E-Gym-V1")` (Apache-2.0 toolkit, mixed upstream licenses):**
+- 8,135 tasks, SweGen-synthesized. Includes synthesized tests for commits without existing tests.
+- **Key advantage over SWE-smith for this repo's "point at any repo" vision:** SweGen works from commits, not PRs, and synthesizes tests when none exist. This is the mechanism that unlocks arbitrary repos.
+- Pre-built Docker environments. Direct integration with OpenHands scaffold.
+**What needs to be built (not available as pip install):**
+1. **Coverage-guided deletion target selection (research/06 Path B).** No existing library does coverage-selectivity-based AST deletion. `libcst` for AST manipulation is available but the targeting logic (which function to delete based on test coverage and selectivity) is novel. This is the "hard part" for arbitrary-repo synthesis without prior substrates.
+2. **The online difficulty curriculum.** `DifficultyCurriculum` in `composer_replication/datagen/curriculum.py` is correctly identified as needing to be built. SWE-smith/R2E-Gym/SWE-Gym do NOT provide this.
+3. **Anti-reward-hacking sandbox.** The `LocalSubprocessSandbox` + `SANDBOX_DENYLIST` + `HackMonitor` in the repo are custom implementations. No OSS library provides this. SWE-MiniSandbox (arXiv:2602.11210) provides container-free isolation but not the semantic hack detection.
+4. **TRL `reward_fn` adapter wired to test execution.** Not provided by any of the surveyed toolkits (SkyRL has its own non-TRL RL stack; SWE-smith's GRPO support is via SkyRL, not TRL GRPOTrainer).
+### 7.2 Integration architecture recommendation
+```
+[Adopt] swesmith toolkit → environment construction for new repos
+[Adopt] SWE-smith dataset → 59k pre-built Feature-Deletion tasks (via SweBenchAdapter)
+[Adopt] R2E-Gym-Subset → 4.6k tasks with SweGen synthetic tests
+[Adopt] SWE-Gym → 2.4k tasks, clean train split, per-task Docker images
+[Adopt] SWE-rebench → scale (21k) + difficulty priors (cold-start p̂)
+[Build] DifficultyCurriculum (unique to this recipe)
+[Build] LocalSubprocessSandbox + HackMonitor (unique to this recipe)
+[Build] TRL reward_fn adapter (SkyRL uses veRL, not TRL)
+[Build] Coverage-guided Path B synthesis (unique, unlocks arbitrary repos)
+[Consider] SWE-MiniSandbox for container-free RL at scale (2026, 5% disk overhead)
+```
+The current `substrates.py` correctly handles the [Adopt] paths. What is missing is the `swesmith` integration for new-repo synthesis (the "25x more synthetic tasks" vision requires going beyond existing substrates).
+---
+## 8. Critical Flags Summary
+| Flag | Severity | Location | Issue |
+|---|---|---|---|
+| **Missing: swesmith API exists** | HIGH | ADR-010 Option A | `pip install swesmith` provides what Option A describes building. The ADR does not evaluate it as a dependency. |
+| **Missing: SkyRL+SWE-smith = working GRPO stack** | HIGH | ADR-010 §Decision | SkyRL (MIT) + SWE-smith already implements GRPO on SWE tasks. The repo's TRL `reward_fn` reinvents this without acknowledging the prior art. |
+| **Missing: SweGen synthesizes tests for commitless repos** | HIGH | research/06 §4, ADR-010 | R2E-Gym's test synthesis approach (commits without tests → LLM synthesizes F2P tests, equivalent to real tests) is the key mechanism for "point at any repo" synthesis. Not discussed. |
+| **Overclaim: FAIL_TO_PASS 'guaranteed'** | MEDIUM | ADR-010 §Decision Drivers | The guarantee requires running tests; it is not inherited from the schema. SWE-smith validation explicitly filters out candidates with no test coverage. |
+| **Overclaim: Path B (coverage-AST deletion) is built** | MEDIUM | ADR-010 capability list | Path B is `[EXTRAPOLATED]` in research/06 but the ADR implies it's implemented. It is not; `substrates.py` only does Path A (gold-patch reversion). |
+| **Missing: SWE-MiniSandbox (2026)** | MEDIUM | ADR-010 §Negative consequences | Container-free RL at 5% disk / 25% setup overhead addresses the ADR's Docker cost concern. Published Feb 2026, after ADR-010. |
+| **Missing: DeepSWE as validation** | LOW | research/06 | Provides evidence that GRPO + R2E-Gym = 42% Pass@1 (the core recipe) works. Not cited. |
+| **Correct: PR Mirror = Feature Deletion** | CONFIRM | research/06 §5 Path A | SWE-smith ablations directly validate the repo's core approach. The best SWE-smith training data (PR Mirror) is exactly gold-patch-reversion Feature Deletion. |
+| **Correct: SWE-smith costs ~$1360 for 50k tasks** | CONFIRM | ADR-010 §7 cost model | Cost model in research/06 is consistent with SWE-smith's reported $1360 for 50k tasks ($0.027/task vs research/06's estimate of $0.02-$0.10/task). |
+| **Correct: ADR-010 OPEN items are honest** | CONFIRM | ADR-010 post-review | The "gate 2 does not prove reachability" open item correctly identifies a gap that exists across ALL surveyed work (SWE-smith, R2E-Gym, SWE-Gym all have this gap). |
+---
+## 9. Key Numbers for Architecture Reference
+| Paper | Tasks | Repos | Env Size | Oracle source | Test-exec? | License |
+|---|---|---|---|---|---|---|
+| SWE-bench | 2,294 eval / 19,008 train | 12 | ~per-instance large | Human PRs | eval only | CC-BY-4.0 |
+| SWE-Gym | 2,438 | 11 | 6 TB | Human PRs | YES | MIT (tooling) |
+| R2E-Gym | 8,135 (4,578 subset) | 13 | 4 TB | Commits + LLM tests | YES | Apache-2.0 |
+| SWE-smith | 50,137 (59,136 HF) | 128 | 295 GB | LM/AST bug injection | YES | MIT (toolkit) |
+| SWE-rebench | ~21k | 3,468 | per-instance | Human PRs (automated) | YES | CC-BY-4.0 |
+| SWE-fixer | 115k | 856 | none | Human PRs | NO | - |
+SWE-smith's 295 GB for 50k tasks vs 6 TB for 2.4k (SWE-Gym) = the per-repo Docker image architecture is a 500x storage efficiency win. This is the right architecture for any new synthesis beyond existing substrates.
+---
+## 10. Sources
+1. **SWE-smith HTML full text** — `research/notes/bugs-scaling-data-for-software-engineering-agents.md` (26,442 words, arxiv.org/html/2504.21798)
+2. **SWE-Gym HTML full text** — `research/notes/training-software-engineering-agents-and-verifiers-with-swe-gym.md` (10,865 words, arxiv.org/html/2412.21139)
+3. **R2E-Gym HTML full text** — `research/notes/r2e-gym-scaling-open-weights-software-engineering-agents-with-procedural-synthet.md` (14,810 words, arxiv.org/html/2504.07164)
+4. **SWE-bench abstract** — `research/notes/231006770-swe-bench-can-language-models-resolve-real-world-github-issues.md`
+5. **SWE-smith GitHub README** — `research/notes/github-swe-benchswe-smith-neurips-2025-db-spotlight-scaling-data-for-swe-agents.md`
+6. **SWE-smith HF dataset page** — `research/notes/swe-benchswe-smith-datasets-at-hugging-face.md`
+7. **SkyRL GitHub** — `research/notes/github-novasky-aiskyrl-skyrl-a-modular-full-stack-rl-library-for-llms-github.md`
+8. **SWE-RL abstract** — `research/notes/250218449-swe-rl-advancing-llm-reasoning-via-reinforcement-learning-on-open-soft.md`
+9. **SWE-MiniSandbox abstract** — `research/notes/260211210-swe-minisandbox-container-free-reinforcement-learning-for-building-sof.md`
+10. **DeepSWE blog** — `research/notes/deepswe-training-a-fully-open-sourced-state-of-the-art-coding-agent-by-scaling-r.md`
+11. **SWE-rebench infrastructure blog** — `research/notes/behind-swe-rebench-infrastructure-to-collect-massive-datasets-of-swe-tasks-and-e.md`
+12. **Repo artifacts:** `composer_replication/datagen/substrates.py`, `research/06-feature-deletion-datagen.md`, `docs/adrs/ADR-010-feature-deletion-datagen.md`

research/deepread/03-sdpo-opsd.md ADDED Viewed

	@@ -0,0 +1,556 @@

+# Deep-Read: SDPO / OPSD — Critical Audit of Cluster 2
+**Date**: 2026-06-09
+**Sources fetched from primary HTML**:
+- SDPO: arXiv:2601.20802v2 (Hübotter et al., ETH Zürich / MIT / Stanford)
+  — note `reinforcement-learning-via-self-distillation-2` (148 K body)
+- OPSD: arXiv:2601.18734v3 (Zhao et al., ICML 2026)
+  — note `self-distilled-reasoner-on-policy-self-distillation-for-large-language-models-2` (52 K body)
+- OPSD code repo: github.com/siyan-zhao/OPSD (README + key args)
+- SDPO code repo: github.com/lasgroup/SDPO (listed in abstract; fetch returned empty body)
+**Repo files audited**: `composer_replication/opsd.py`, `composer_replication/trainer/composer_trainer.py::_compute_sdpo_loss`, `composer_replication/hint_generator.py`, `research/07-sdpo-hint-generator.md`, `research/11-sdpo-alignment-indices.md`, `docs/adrs/ADR-007`, `ADR-008`, `ADR-009`, `ADR-011`.
+---
+## 1. What the primary sources actually say
+### 1.1 SDPO (arXiv:2601.20802v2) — core method
+**Exact loss (Eq. 1)**:
+```
+L_SDPO(θ) := Σ_t KL( π_θ(·|x, y_{<t}) ‖ stopgrad(π_θ(·|x, f, y_{<t})) )
+```
+KL direction is **forward KL — KL(student ‖ teacher)**, i.e. `KL(π_θ || q_θ)`. The
+student is in the first argument. This is the "reverse KL from the teacher's
+perspective" but forward from the student's perspective (student wants to match teacher).
+The paper writes it `KL(π_θ ‖ stopgrad(q_θ))`.
+**Stability improvements (§2.3)**:
+1. Regularized teacher: EMA of student params OR interpolation between current teacher
+   and initial teacher `q_{θ_ref}`. The paper calls these "trust-region" and "EMA"
+   teachers (Table 4). Non-regularized teacher (`q_θ`, the live student) diverges.
+2. Symmetric JSD: the paper adopts JSD as the distillation loss for stability —
+   citing Agarwal et al. 2024 on-policy distillation. The pseudocode (Fig 14) calls
+   this `divergence(logprobs_student, logprobs_teacher)` with no fixed default — the
+   paper reports using JSD.
+**Top-K approximation (Appendix A.3)**:
+The paper approximates the full-vocab KL with top-K tokens of the **student** distribution:
+```
+L_SDPO ≈ Σ_t [ Σ_{ŷ_t ∈ topK(π_θ)} π_θ(ŷ_t|x,y_{<t}) · log(π_θ(ŷ_t) / q_θ(ŷ_t))
+              + tail_term ]
+```
+The tail term aggregates the remaining probability mass. Default K=100.
+**Teacher context construction (Table 2, verbatim)**:
+```
+User:  {prompt}
+Correct solution: {successful_previous_rollout}    (skipped if unavailable)
+The following is feedback from your unsuccessful earlier attempt:
+{environment_output}                               (skipped if no env output or if solved)
+Correctly solve the original question.
+Assistant: {original_response}                     (the student's original attempt, for log-prob re-eval)
+```
+Critical nuance: the `original_response` is placed in the teacher context so the
+model can re-evaluate log-probs of `y` under the teacher. **The student's original
+attempt is always appended as the response the teacher evaluates** — this is how both
+student and teacher evaluate log-probs of the same token sequence `y`.
+Token alignment: **there is no shift / hint-insertion at an "error turn."** The teacher
+sees `[prompt, feedback, original_response]` and both student and teacher evaluate
+log-probs of the SAME `original_response` tokens. No additional tokens are inserted
+into the response; the prefix is longer for the teacher (it has feedback), but the
+response token sequence evaluated is identical.
+**Three feedback types ablated (§4.6, Table 6)**:
+1. **Sample solution** (`f = own solution`): a successful sibling rollout from the GRPO
+   group; always student-generated (no expert model). Teacher accuracy: 42.4%.
+2. **Environment output** (`f = output`): runtime errors, failing unit tests, etc.
+   Teacher accuracy: 32.5%.
+3. **Student's original attempt** (`f = y`): the repo calls out that including the
+   original attempt in the feedback (not just in the response slot) **reduces teacher
+   diversity** (biases teacher toward student; "Same output": 30% vs ~10–13%).
+4. **Combined** (`f = output + own solution`): best trained student accuracy (48.3%).
+   Excluding `f = y` (the original attempt as part of conditioning) is key.
+**Failure modes reported**:
+- Non-regularized teacher (`q_θ`) diverges / training collapses (Table 4: 36.1% vs
+  50.6% for trust-region teacher).
+- Performance depends on model in-context learning ability: SDPO underperforms GRPO on
+  weaker models (Qwen3-0.6B); hybrid SDPO+GRPO (λ=0.9 GRPO + 0.1 SDPO) is more
+  robust (§4.5).
+- Uninformative or misleading environment feedback: SDPO cannot learn from it.
+- SDPO adds small compute overhead (additional forward for log-prob re-computation of
+  teacher context); minor for large models, non-negligible for small models.
+- Including the student's own attempt in the teacher conditioning (not just as the
+  response to re-evaluate) reduces diversity; the correct template excludes it from
+  the conditioning prefix.
+**SDPO operates over the full rollout, not at isolated "error turns"**. The loss
+sums over all tokens `t` in the response `y`. There is no error-site detection
+step in the SDPO paper.
+### 1.2 OPSD (arXiv:2601.18734v3) — core method
+**Exact loss (Eq. 6–8)**:
+```
+L_OPSD(θ) = E_{(x,y*)~S} [ E_{ŷ~p_S(·|x)} [ D(p_T ‖ p_S)(ŷ|x) ] ]
+where
+D(p_T ‖ p_S)(ŷ|x) := (1/|ŷ|) Σ_{n=1}^{|ŷ|} D( p_T(·|x, y*, ŷ_{<n}) ‖ p_S(·|x, ŷ_{<n}) )
+```
+**Divergence D** can be: forward KL, reverse KL, or JSD_β. The paper defines:
+```
+JSD_β(p_T ‖ p_S) = β·KL(p_T ‖ m) + (1-β)·KL(p_S ‖ m)
+m = β·p_T + (1-β)·p_S
+```
+**Direction convention**: `D(p_T ‖ p_S)` — teacher in first arg, student in second. For the JSD:
+- β=0 → KL(p_S ‖ m) (approaches pure KL(p_S ‖ p_T) as m→p_T; forward KL w.r.t. teacher)
+- β=1 → KL(p_T ‖ m) (approaches pure KL(p_T ‖ p_S); reverse KL w.r.t. teacher / forward KL w.r.t. student)
+The GKD paper (arXiv:2306.13649) that OPSD cites defines JSD_β with the **same
+convention**: `JSD_β(p ‖ q) = β·KL(p||M) + (1-β)·KL(q||M)`.
+**Per-token pointwise clipping**: OPSD introduces this explicitly:
+```
+D_clip^(f)(p_T ‖ p_S) = (1/|ŷ|) Σ_n Σ_v min(l_{n,v}^(f), τ)
+where l_{n,v}^(f) = p_T(v|·) · f( p_S(v|·) / p_T(v|·) )
+```
+This clips per vocab-entry contribution. Default τ=0.05 (from README: `--jsd_token_clip 0.05`).
+Non-thinking mode results in README use 1e-7 (Qwen3-8B) and 1e-6 (Qwen3-4B, 1.7B).
+**Teacher context**: `p_T(·|x, y*, ŷ_{<n})` — teacher sees the ground-truth answer `y*`
+(a reference CoT / verified reasoning trace from the dataset) prepended to the problem,
+then evaluates the student's prefix `ŷ_{<n}`. Same token sequence for both distributions
+evaluated at each step `n`.
+**`--reason_first` flag (from GitHub README)**: Prepend an explicit rationalization to the
+teacher context before distillation. This is OPSD's self-introspection lever: the teacher
+is first asked to rationalize why `y*` is correct, then that rationalization is folded into
+the conditioning. Not the main results configuration; requires `--use_peft`.
+**Results**: On Qwen3-1.7B (AIME24/25/HMMT25), OPSD +OPSD vs. base: 37.1% → 43.4%
+(Avg@12). Outperforms GRPO (37.7%) and SFT (35.8%). Token-efficient: generation capped
+at 1024 tokens vs. GRPO's 16K.
+---
+## 2. Audit: `composer_replication/opsd.py` — "byte-for-byte OPSD parity" claim
+### 2.1 JSD formula — CORRECT, with a subtle direction note
+The code implements:
+```python
+JSD = β·KL(teacher||M) + (1-β)·KL(student||M)
+M = logsumexp([ log p_student + log(1-β), log p_teacher + log(β) ])
+```
+The OPSD paper (Eq. 7) defines:
+```
+JSD_β(p_T ‖ p_S) = β·KL(p_T‖m) + (1-β)·KL(p_S‖m)
+```
+where `m = β·p_T + (1-β)·p_S`.
+The code `kl_teacher = F.kl_div(mixture_log_probs, teacher_log_probs, ...)` uses
+PyTorch semantics where `F.kl_div(input=log_q, target=log_p, log_target=True)`
+computes `KL(p||q) = Σ p(x)·(log p(x) - log q(x))`. So `kl_teacher` computes
+`KL(teacher||mixture)` and `kl_student` computes `KL(student||mixture)`.
+The final JSD: `β·kl_teacher + (1-β)·kl_student` = `β·KL(teacher||M) + (1-β)·KL(student||M)`.
+This matches the OPSD paper's `JSD_β(p_T ‖ p_S)` exactly. **CORRECT.**
+### 2.2 β convention docstring — INVERTED vs. both papers
+The `opsd.py` docstring says:
+```
+β = 0  → KL(teacher || student)  (reverse KL — mode-covering for student)
+β = 1  → KL(student || teacher)  (forward KL — mode-seeking for student)
+```
+From the OPSD GitHub README:
+> `--beta`: Interpolation weight for the JSD mixture distribution.
+> **Beta=0 means forward KL and 1 means reverse KL.**
+The repo docstring has the β=0 and β=1 labels **swapped** relative to the OPSD upstream.
+When β=0: `JSD_0 = 0·KL(teacher||M) + 1·KL(student||M)`. In the limit (degenerate β=0),
+M approaches p_teacher, so this approaches `KL(student||teacher)` — which is **forward KL**
+(student → teacher), **mode-seeking for student**. The README says "Beta=0 means forward KL"
+which matches this analysis.
+The code *implementation* is correct (the formula computes the right mixture). The *docstring*
+labels β=0 as "reverse KL" and β=1 as "forward KL", which contradicts both the upstream README
+and the mathematical analysis. This is a documentation error, not a numerical error.
+**VERDICT**: Implementation is numerically correct. Docstring direction labels are inverted.
+### 2.3 `reduction="batchmean"` behavior — MINOR DIVERGENCE from upstream
+The repo `opsd.py` comment says:
+> "batchmean" matches upstream OPSD: divides by `mask.sum()` when labels are given,
+> else by the leading dim of jsd (= batch size).
+The OPSD paper (Algorithm 1) normalizes by `|ŷ|` (sequence length, token-mean):
+```
+ℓ(x,y*) = D(p_T‖p_S)(ŷ|x) = (1/|ŷ|) Σ_n D(...)
+L_OPSD(θ) = (1/|B|) Σ_{(x,y*)∈B} ℓ(x,y*)
+```
+The repo divides by `mask.sum()` (number of valid/masked tokens in the batch), which is
+equivalent to OPSD's normalization only when every example has the same number of
+error-turn tokens. When batch sizes vary (real training), this differs from the paper's
+per-sequence average followed by batch average. In practice this difference is negligible
+for stability, but it is technically not byte-for-byte OPSD parity on the reduction.
+**VERDICT**: The `reduction="batchmean"` logic is borrowed from the OPSD upstream code
+(which uses the same `mask.sum()` convention). The docstring's "matches upstream" claim
+is accurate for the code, but the code diverges from the paper's stated per-sequence
+normalization. Not a material issue.
+### 2.4 `token_clip` parameter — CORRECT semantics, but per-token vs. per-(token,vocab) distinction
+The repo implements `token_clip` as a **per-position** JSD clip:
+```python
+jsd = jsd.clamp(max=token_clip)  # jsd shape is (B, T, V) or (n_valid, V)
+```
+The OPSD paper's pointwise clipping (Section 3.3) clips **per-(position, vocab-entry)**:
+`min(l_{n,v}^(f), τ)` for each vocab entry v at each position n.
+The upstream OPSD code (`--jsd_token_clip`) appears to apply the same per-(position,vocab)
+clip. The repo's clamp on the jsd tensor before reduction would clip the full-vocab
+contribution per position (since jsd has shape (B,T,V) before masking) — this is
+equivalent to per-(position,vocab) clipping, which is correct.
+**VERDICT**: Implementation appears correct. The parameter name (`token_clip`) is slightly
+misleading (it clips per-token-vocab-entry, not just per-token), but the semantics match.
+---
+## 3. Critical structural mismatch: Composer/repo framing vs. SDPO mechanics
+### 3.1 ERROR-TURN MASKING — NOT IN SDPO
+The repo implements SDPO as an error-turn-masked loss:
+- `_compute_sdpo_loss` applies JSD only at `error-turn tokens` (via `sdpo_loss_mask`).
+- The data collator detects error sites in a trace and constructs a teacher context
+  with a hint inserted at the error turn (`ctx_teacher = ctx_student + hint`).
+- The hint shifts teacher response tokens right, requiring explicit alignment indices
+  (ADR-011).
+**The SDPO paper has no error-turn masking.** SDPO applies the KL loss to ALL tokens `t`
+in the rollout response:
+> "L_SDPO(θ) := Σ_t KL(π_θ(·|x, y_{<t}) ‖ stopgrad(π_θ(·|x, f, y_{<t})))"
+The SDPO teacher context includes the full feedback; both student and teacher evaluate
+log-probs of the **same response tokens** `y`. There is no "hint inserted into the
+response" — the feedback is in the conditioning prefix, not intercalated into the
+response sequence. Therefore the teacher response tokens are **not shifted** and token
+alignment is trivially preserved: both contexts evaluate the same sequence `y`.
+**The repo's architecture (hint at error turn → response token shift → alignment indices)**
+is an interpretation of Composer 2.5's "hint" mechanism, not a feature of SDPO. SDPO's
+feedback is in the prompt/conditioning context; it does not intercalate text into the
+middle of a response.
+**VERDICT**: The repo's error-turn-masking design is a reasonable extension of the
+Composer blog's described mechanism ("insert hint at error turn") but is **NOT**
+SDPO as described in the paper. The Composer blog's mechanism is itself not fully
+described and may or may not match SDPO mechanics.
+### 3.2 TEACHER CONTEXT — CRITICAL DIFFERENCE
+**SDPO teacher context** (Table 2):
+```
+[prompt, feedback_f, original_response_y]
+```
+The teacher evaluates log-probs of `original_response_y` given `[prompt, feedback_f]`.
+Teacher prefix = `[prompt, feedback_f]`. Response = `y` (same as student). No hint is
+inserted *into* `y`.
+**Repo teacher context**:
+```
+ctx_teacher = ctx_student + hint_at_error_turn
+```
+The hint is *intercalated* into the response sequence at an error turn. Teacher prefix
+= student prefix. Response = `y_before_error + hint + y_after_error`. Teacher response
+tokens are LONGER than student response tokens and SHIFTED.
+This is architecturally different from SDPO. The alignment problem (ADR-008, ADR-011)
+arises precisely because the repo's teacher context design inserts hint text into the
+response, which SDPO does not do.
+**VERDICT**: The repo's teacher context construction is a novel design inspired by the
+Composer blog. It is not what SDPO does. The ADR-008 "trust-gap" and the entire
+ADR-011 alignment index complexity are artifacts of this departure from SDPO, not
+corrections to SDPO.
+### 3.3 OPSD vs. SDPO as the loss source
+The repo header in `opsd.py` says the loss is:
+> "lifted from siyan-zhao/OPSD::OPSDTrainer.generalized_jsd_loss (MIT)"
+And:
+> "SDPO paper: Hübotter et al. … formalizes the same loss as Composer 2.5's
+> 'Targeted RL with Textual Feedback.'"
+These are TWO DIFFERENT methods with related but not identical losses:
+- **OPSD loss** (Eq. 7–8): `JSD_β(p_T ‖ p_S)` with teacher having `y*` (ground truth).
+  Normalization: per-sequence average then batch average. Pointwise vocab clipping.
+  Training runs ~100 steps. Fixed teacher (initial checkpoint, not live).
+- **SDPO loss** (Eq. 1): `KL(π_θ(·|x,y_{<t}) ‖ stopgrad(π_θ(·|x,f,y_{<t})))` where
+  KL is applied per-position over the full response. The paper adopts JSD as a stability
+  improvement (§2.3) but the base formulation is reverse KL (student ‖ teacher).
+  Teacher is regularized via EMA or trust-region. No per-vocab clipping in the paper
+  (top-K approximation instead).
+The repo correctly implements the OPSD JSD formula (which SDPO also uses for stability).
+The claim "verified port of siyan-zhao/OPSD::OPSDTrainer.generalized_jsd_loss" is
+accurate for the loss kernel. The claim "Composer 2.5's 'Targeted RL with Textual Feedback'"
+is an assertion that Composer uses the same loss — this is not confirmed anywhere in the
+Cursor blog or Composer 2 tech report.
+---
+## 4. Audit: `_compute_sdpo_loss` in `composer_trainer.py`
+### 4.1 Gradient flow — CORRECT
+```python
+student_logits = model(input_ids=inputs["input_ids"]).logits
+with torch.no_grad():
+    teacher_logits = model(input_ids=inputs["ctx_teacher_input_ids"]).logits
+```
+Teacher is `no_grad` — matches SDPO's `stopgrad(π_θ(·|x,f,y_{<t}))`. Student has
+gradient. Correct.
+### 4.2 Alignment index machinery — NECESSARY GIVEN THE DESIGN, BUT NOT FROM SDPO
+The `student_response_idx` / `teacher_response_idx` machinery (ADR-011) is needed
+because the hint is inserted into the teacher response sequence. This complexity does
+not exist in SDPO or OPSD because those methods never insert text into the response.
+The repo's `strict_sdpo_alignment` guard is a correct defense against the problem it
+has created for itself.
+### 4.3 Batch-level masking — CORRECT for the repo's error-turn interpretation
+The loss is masked to error-turn tokens only (`aligned_labels` with -100 elsewhere).
+This means the SDPO channel only trains on error recovery tokens, not the full rollout.
+SDPO trains on the full rollout. For Composer's intent (correcting error turns), the
+masking is reasonable, but it produces a loss that is more like a targeted distillation
+at error sites than SDPO's full-rollout advantage assignment.
+---
+## 5. Audit: `research/07-sdpo-hint-generator.md` — Accuracy check
+### 5.1 Three feedback types from SDPO paper — CORRECTLY REPORTED
+research/07 correctly identifies the three types:
+1. Sample solution (successful sibling rollout)
+2. Environment output (runtime errors)
+3. Student's original attempt
+The paper (Table 6 results, which research/07 did NOT have access to) shows:
+- Best configuration: `f = output + own solution` (48.3% accuracy)
+- Including `f = y` (original attempt as conditioning, not as response) **hurts diversity**
+  and slightly reduces final accuracy (44.5% vs 48.3%)
+Research/07 correctly notes the sibling rollout is "always generated by the student, not
+an expert model" — confirmed in the paper: "We emphasize that these sample solutions are
+always generated by the student, as in GRPO, and do not require an expert model."
+### 5.2 "Successful sibling rollout as implicit feedback" claim — CORRECTLY REPORTED
+The abstract: "SDPO also outperforms baselines in standard RLVR environments that only
+return scalar feedback by using successful rollouts as implicit feedback for failed attempts."
+Research/07 cites this correctly and uses it as the basis for the `SiblingBootstrapGenerator`.
+### 5.3 OPSD `--reason_first` flag — CORRECTLY DESCRIBED
+The OPSD README confirms: `--reason_first False: Prepend an explicit rationalization to
+the teacher context before distillation.` Research/07 correctly calls this "OPSD's own
+knob for same-model introspection."
+### 5.4 `--jsd_token_clip default 0.05` — CORRECTLY CITED
+Confirmed from OPSD README: `--jsd_token_clip 0.05` is the default.
+---
+## 6. Audit: `SiblingBootstrapGenerator` — Is it supported by the papers?
+The repo's `hint_generator.py` sketch (lines 319–331) and `research/07` §6.3:
+```python
+class SiblingBootstrapGenerator:
+    def generate(self, ctx):
+        sibs = ctx.get("sibling_rollouts") or []
+        winners = [s for s in sibs if s.get("reward", 0.0) > 0.0]
+        if not winners:
+            return None
+        best = max(winners, key=lambda s: s["reward"])
+        snippet = (best.get("solution_excerpt") or "")[:200]
+        return ("Reminder: a working approach for this task looks like:\n"
+                f"{snippet}\nAdapt this to the current step.")
+```
+**What the SDPO paper actually does** (Table 2 template):
+```
+Correct solution: {successful_previous_rollout}
+```
+The successful rollout is passed as the full solution (or relevant excerpt) in the
+teacher context. The teacher then evaluates log-probs of the student's original
+response given this context.
+**Key difference**: In SDPO, the sibling solution goes into the teacher's conditioning
+prefix. The teacher does not generate a new hint; it just re-evaluates the student's
+response log-probs with the solution visible. In the repo, the sibling solution is
+used to *generate a hint string* that gets inserted into the response sequence.
+This is an extrapolation beyond what the SDPO paper supports. SDPO's "successful rollout
+as implicit feedback" mechanism does NOT:
+1. Generate a "Reminder: a working approach..." hint string.
+2. Insert text into the student's response sequence.
+3. Require error-turn detection.
+The SDPO sibling mechanism IS:
+1. Condition the teacher on the full successful solution.
+2. Re-evaluate ALL student response token log-probs under that teacher.
+3. Apply the KL loss across the entire response.
+**VERDICT**: The `SiblingBootstrapGenerator` as sketched is an extrapolation from SDPO's
+mechanism, not a faithful implementation of it. The paper supports using a sibling rollout
+as teacher conditioning context; it does not support generating a textual hint from it
+to splice into the response. The Composer blog's "hint" framing is the source of this
+architectural decision; SDPO is cited as inspiration but is not the mechanism.
+Research/07 acknowledges this at several points ("A working approach looks like: …" in
+the class comment vs the actual SDPO template) but does not flag it as a divergence — it
+presents the sibling-bootstrap hint approach as if it naturally follows from SDPO.
+---
+## 7. Audit: `research/11-sdpo-alignment-indices.md`
+### 7.1 Problem correctly identified
+ADR-011 correctly identifies that inserting a hint into the teacher context shifts teacher
+response tokens right. The alignment indices machinery (`_mask_to_padded_indices`,
+`student_response_idx`, `teacher_response_idx`, sentinel handling) is a sound engineering
+solution to the problem the repo's design creates.
+### 7.2 Root cause attribution — MISLEADING
+ADR-011 and the trainer comments frame the alignment problem as an SDPO issue that the
+papers fail to address ("the exact trust-gap flagged in ADR-008"). This is not accurate.
+The alignment problem does not exist in SDPO or OPSD because those methods never insert
+text into the response sequence. The alignment problem is entirely self-created by the
+repo's decision to implement the Composer blog's "hint at error turn" as a text insertion
+into the teacher's response sequence.
+---
+## 8. Audit: ADR-007, ADR-008, ADR-009 — Key claims
+### ADR-007 — JSD as "the kernel of SDPO arXiv:2601.20802"
+The ADR says `generalized_jsd_loss` is "verified port of siyan-zhao/OPSD, the kernel of
+SDPO arXiv:2601.20802." This telescopes two papers. The JSD is the kernel of OPSD
+(the Zhao et al. paper, 2601.18734). SDPO (Hübotter et al., 2601.20802) uses JSD as a
+**stability improvement** over the base KL loss; the primary SDPO loss is the KL. Both
+papers use the same JSD formula (citing GKD paper 2306.13649). The conflation is not
+consequential for the loss code but creates confusion in documentation.
+### ADR-008 — "SDPO needs full vocabulary logits"
+Confirmed. SDPO Appendix A.3 discusses top-K approximation of the KL because "naively
+computing the KL divergence between student and teacher requires holding full logits of
+both models in memory." The repo's claim about needing full logits is correct; PRIME-RL's
+log-probs-only interface is correctly identified as incompatible with the SDPO channel.
+### ADR-008 — Dr. GRPO as the Composer algorithm
+This is sourced from research/10 (Composer 2 tech report mining). Not audited here (out
+of scope for this cluster).
+### ADR-009 — "How Cursor generates that hint is unstated"
+Confirmed true. The Composer 2 tech report (arXiv:2603.24477) is cited as unread in
+research/07 §8 and ADR-009. ADR-009 correctly acknowledges the open question.
+---
+## 9. Summary of findings
+| Claim | Source | Verdict |
+|---|---|---|
+| JSD formula in opsd.py is numerically correct | OPSD Eq. 7 | CORRECT |
+| β=0 = "reverse KL" in docstring | OPSD README: "β=0 = forward KL" | INVERTED label |
+| "byte-for-byte OPSD parity" | OPSD code | Mostly correct; β direction label wrong; reduction differs from paper's per-sequence normalization; otherwise matches upstream code |
+| Error-turn masking is from SDPO | SDPO paper | FALSE — SDPO applies loss to full rollout, no error-turn detection |
+| Teacher context = ctx_student + hint_at_error_turn | SDPO paper | FALSE — SDPO teacher = [prompt, feedback, student_response]; feedback is in prefix, not intercalated |
+| SiblingBootstrapGenerator follows from SDPO "successful rollout as implicit feedback" | SDPO §4.6 | EXTRAPOLATION — SDPO conditions teacher on full solution; repo generates a hint string and inserts it into response sequence |
+| Alignment indices machinery (ADR-011) addresses SDPO misalignment | SDPO paper | MISLEADING — problem is self-created by hint-insertion design; does not exist in SDPO |
+| SDPO needs full vocabulary logits (ADR-008) | SDPO Appendix A.3 | CORRECT |
+| Three feedback types in research/07 | SDPO §4.6 | CORRECTLY REPORTED |
+| --jsd_token_clip default 0.05 | OPSD README | CORRECT |
+| --reason_first flag | OPSD README | CORRECTLY DESCRIBED |
+| "Successful rollouts as implicit feedback" claim | SDPO abstract | CORRECTLY CITED |
+| Teacher is stop-grad, student has gradient | SDPO Eq. 1 | CORRECT in opsd.py and composer_trainer.py |
+---
+## 10. Recommendations
+1. **Fix the β docstring** in `opsd.py` to match the OPSD upstream convention:
+   β=0 → forward KL (KL(student‖teacher)), β=1 → reverse KL (KL(teacher‖student)).
+2. **Clarify the architectural departure from SDPO** in `composer_trainer.py` docstring
+   and `research/07`: the repo implements a Composer-blog-inspired error-turn hint
+   injection, which is an extension beyond SDPO. SDPO uses the feedback in the prompt
+   prefix and evaluates the full response; the repo intercalates text into the response.
+3. **Reconsider framing of `SiblingBootstrapGenerator`**: it is an original design choice,
+   not an SDPO mechanism. The SDPO "sibling as implicit feedback" mechanism would look like:
+   build a teacher context `[prompt, successful_sibling_rollout, original_response]` and
+   apply KL over the whole original response — without generating a hint string or
+   error-turn detection. This would be simpler and more faithful to SDPO.
+4. **Teacher regularization is not implemented**: the SDPO paper shows a non-regularized
+   teacher diverges (Table 4: 36.1% vs. 50.6%). The repo's teacher is the live model
+   weights at each step with no EMA or trust-region regularization. For production SDPO
+   runs this is a gap. (The `sdpo_jsd_beta` default of 0.5 uses symmetric JSD which is
+   one of SDPO's stability improvements, but the teacher regularization is absent.)
+5. **SDPO's original attempt placement**: the paper includes the student's original
+   response as the sequence being log-prob-evaluated (i.e., the "response" slot in the
+   teacher context). The repo's collator instead masks specific error-turn tokens within
+   a modified response. These are architecturally different. The paper-accurate approach
+   would re-evaluate log-probs of the entire original response under the hint-conditioned
+   teacher, not just the tokens after the error.
+6. **Failure mode from SDPO paper**: the strongest limitation is model capability
+   dependence — SDPO underperforms GRPO on weak models (Qwen3-0.6B); SDPO+GRPO with
+   λ=0.9 is recommended for weaker base models. This is not documented in the repo's
+   SDPO usage guidance.
+---
+## 11. What the papers do NOT say (repo-claimed but unconfirmed in sources)
+- That Composer 2.5's "Targeted RL with Textual Feedback" is SDPO specifically (the
+  Cursor blog does not cite SDPO; it describes a mechanism consistent with SDPO but
+  the connection is an inference, not a citation).
+- That error-turn masking is part of SDPO.
+- That the repo's hint-at-error-turn teacher context is the SDPO mechanism.
+- That the alignment index problem (ADR-011) is an issue in SDPO.
+- How Cursor generates the hint (confirmed absent in all Cursor artifacts).

research/deepread/04-grpo-family.md ADDED Viewed

	@@ -0,0 +1,421 @@

+# Deep-Read: GRPO Objective Family — Critical Review
+**Cluster:** Policy-optimization objective family
+**Date:** 2026-06-09
+**Reviewer:** Claude Code subagent (critical-review pipeline)
+**Sources fetched and read (full HTML bodies):**
+- arXiv:2402.03300 — DeepSeekMath / original GRPO (Shao et al.)
+- arXiv:2503.20783 — Dr.GRPO "Understanding R1-Zero-Like Training" (Liu et al.)
+- arXiv:2503.14476 — DAPO (Yu et al., ByteDance/Tsinghua)
+- arXiv:2507.18071 — GSPO (Zheng et al., Qwen/Alibaba)
+- arXiv:2506.13585 — CISPO / MiniMax-M1 (MiniMax)
+- arXiv:2512.21852 — "A Comedy of Estimators" (Shah, Obando-Ceron et al.) — **abstract only** (HTML 404; PDF binary; full text unavailable via hyperresearch; findings below rely on the abstract + cross-refs in the repo)
+- arXiv:2603.24477 — Composer 2 Technical Report (full body via `research/notes/composer-2-technical-report.md`)
+- **Repo files read:** `composer_replication/trainer/composer_trainer.py`, `composer_replication/trainer/kl_in_reward.py`, `docs/adrs/ADR-014-policy-optimization-objective-menu.md`, `research/10-composer2-techreport-mining.md`, `research/notes/channel-1-drgrpo-base-po-objective-menu-*.md`, `research/design-F5-fidelity-audit.md`
+---
+## 1. Exact loss-math delta: each objective vs original GRPO
+### 1.1 Original GRPO (arXiv:2402.03300, Shao et al. 2024)
+**Loss (Eq. 11 in the DeepSeekMath paper):**
+```
+J_GRPO(θ) = E_{q, {o_i}~π_old}
+    (1/G) Σ_i (1/|o_i|) Σ_t { min[r_{i,t} · Â_{i,t}, clip(r_{i,t}, 1-ε, 1+ε)·Â_{i,t}]
+                               - β · D_KL[π_θ || π_ref] }
+Â_{i,t} = (R(q,o_i) - mean({R_j})) / std({R_j})
+r_{i,t} = π_θ(o_{i,t}|q,o_{i,<t}) / π_old(o_{i,t}|q,o_{i,<t})   # token-level IS ratio
+```
+Key design choices:
+- **Per-sequence length normalization** (`1/|o_i|` inside the sum) — averages loss over tokens *within* each response
+- **Advantage = group-mean-centered, then divided by group std** (std-normalization)
+- **KL in the LOSS** (not in the reward): `β · D_KL[π_θ||π_ref]` added as a per-token penalty inside the surrogate
+- KL estimated via the k3 (Schulman) estimator: `D_KL ≈ π_ref/π_θ - log(π_ref/π_θ) - 1`, i.e., `exp(ref_logp - logp) - (ref_logp - logp) - 1` (verified at DeepSeekMath full text L1723-1826, and `composer_trainer.py` docstring referencing trl grpo_trainer.py ~L2513)
+### 1.2 Dr. GRPO (arXiv:2503.20783, Liu et al., Sea AI Lab)
+**Exact delta from GRPO:**
+```
+CHANGE 1: Remove 1/|o_i| length normalization
+  → loss is now sum over tokens (not mean), normalized only by batch/group size 1/G
+  → in implementation: masked_mean divides by MAX_TOKENS (a constant), not mask.sum(axis=dim)
+    (Listing 1, Dr.GRPO paper)
+CHANGE 2: Remove std(R) normalization from advantage
+  Â_{i,t}^{Dr.GRPO} = R(q,o_i) - mean({R_j})   ← NO /std
+```
+**Why (paper's argument, Sec. 3.1–3.2):**
+- Length normalization introduces *response-level length bias*: for positive advantages, shorter correct responses get larger gradient; for negative advantages, longer wrong responses are penalized less → policy artificially lengthens incorrect responses
+- Std-normalization introduces *question-level difficulty bias*: easy/hard questions (std≈0) get up-weighted in the batch regardless of their value
+**Empirical result:** Dr.GRPO achieves better token efficiency and prevents "wild length growth" for incorrect responses; accuracy is maintained or improved (Appendix C ablation, Fig. 9 3-seed comparison).
+**Connection to RLOO:** The unbiased advantage `Â_{i,t} = R(q,o_i) - mean({R_j})` is proportional to the REINFORCE Leave-One-Out estimator: `G/(G-1) · Â_{i,t} = Â^{RLOO}_{i,t}` (App. A). This is a principled unbiased baseline without a value model.
+### 1.3 DAPO (arXiv:2503.14476, Yu et al., ByteDance)
+**Four changes from GRPO. Exact deltas:**
+**DAPO-1: Clip-Higher (Decoupled Clip)**
+```
+PPO/GRPO: clip(r, 1-ε, 1+ε)   # symmetric, same bound both directions
+DAPO:     clip(r, 1-ε_low, 1+ε_high)  where ε_high > ε_low
+  → paper: ε_low=0.2, ε_high=0.28
+```
+Purpose: allow larger upward updates on positive-advantage responses (more entropy diversity) while keeping the lower bound conservative (no degradation on negative responses).
+**DAPO-2: Dynamic Sampling (filters zero-gradient groups)**
+```
+If all G responses in a group have reward=0 or reward=1, the group gradient is zero
+→ DAPO filters these "degenerate" groups and resamples until a mixed-reward group appears
+```
+**DAPO-3: Token-Level Policy Gradient Loss**
+DAPO normalizes loss by total tokens across the batch (not per-sequence). Effectively same as Dr.GRPO's length-unbiased formulation but motivated differently (stability in long-CoT).
+**DAPO-4: Overlong Reward Shaping (masking truncated responses)**
+```
+Default: assign punitive reward to truncated (overlong) responses
+DAPO: instead mask the loss on truncated responses entirely
+→ paper calls this "Overlong Filtering" (Section 3.4)
+```
+**DAPO KL treatment:** `beta=0.0` �� KL is REMOVED entirely. No reference policy.
+**repo's DAPO preset (`PO_OBJECTIVES["dapo"]`):**
+```python
+{
+    "loss_type": "dapo",
+    "scale_rewards": "none",
+    "epsilon": 0.2,
+    "epsilon_high": 0.28,        # correct per paper
+    "mask_truncated_completions": True,   # correct = DAPO's overlong filter
+    "beta": 0.0,                 # correct: KL removed
+    "importance_sampling_level": "token",
+    "num_iterations": 1,
+}
+```
+**Assessment: ACCURATE to the paper.** The `epsilon_high=0.28` matches the paper's recommended value. `mask_truncated_completions=True` maps to DAPO's "Overlong Filtering." `beta=0.0` is correct — DAPO removes KL.
+**Composer 2's rejection of DAPO overlong masking:** The Composer 2 technical report (research/notes/composer-2-technical-report.md, L362) states verbatim:
+> "We did not see benefits with overlong masking at small scale and opted not to mask rollouts that exceed the maximum sequence length. Our self-summary system (discussed below) also limits the occurrence of these cases in practice."
+The repo's research/10-composer2-techreport-mining.md records this correctly at line 51: "DAPO overlong-rollout masking explicitly TRIED and REJECTED ('did not see benefits at small scale')."
+**VERDICT on repo's research/10 claim about DAPO overlong masking:** VERIFIED. The Composer 2 report does explicitly try and reject DAPO overlong masking. The claim "not adopted at small scale" is accurate.
+### 1.4 GSPO (arXiv:2507.18071, Zheng et al., Qwen Team, Alibaba)
+**Key insight:** GRPO's instability for large/MoE models stems from token-level importance sampling weights accumulating high variance over long sequences. The IS weight at each token position is a single sample from a distribution — when the policy updates, MoE routing changes, making token-level ratios for the same token in two different forward passes unreliable.
+**GSPO objective:**
+```
+J_GSPO(θ) = E_{x, {y_i}~π_old} [ (1/G) Σ_i min(s_i(θ)·Â_i, clip(s_i(θ), 1-ε, 1+ε)·Â_i) ]
+where s_i(θ) = (π_θ(y_i|x) / π_old(y_i|x))^{1/|y_i|}
+             = exp((1/|y_i|) Σ_t log(π_θ(y_{i,t}|x,y_{i,<t}) / π_old(y_{i,t}|x,y_{i,<t})))
+Â_i = (r(x,y_i) - mean({r_j})) / std({r_j})   # advantage still uses std-norm!
+```
+**Key delta from GRPO:**
+- IS ratio is the **geometric mean** of per-token ratios = the |y_i|-th root of the sequence-level ratio
+- Clipping is done at the **sequence level** (one clip decision per response), not per token
+- Tokens within an unclipped response contribute equally to the gradient (no per-token IS weighting)
+- Advantage still uses std-normalization (unlike Dr.GRPO) — GSPO does NOT remove std-norm
+**Gradient analysis:** GSPO token-level gradient = `s_i(θ) · (1/|y_i|) Σ_t Â_i · ∇_θ log π_θ(y_{i,t})`. The scalar `s_i(θ)` is the same for all tokens in a response — stable, does not accumulate per-token noise.
+**MoE benefit (Section 5.3):** With MoE models, expert activations change between π_old and π_θ forward passes (~10% of experts per layer differ after one update for Qwen3-30B-A3B). This makes token-level IS ratios volatile. GSPO's sequence-level ratio depends only on the sequence likelihood — stable even when some expert routing changes. Eliminates need for "Routing Replay" strategy.
+**repo's GSPO preset:**
+```python
+"gspo": {
+    "loss_type": "grpo",            # trl has no literal "gspo"
+    "scale_rewards": "group",       # ← std-norm kept (correct for GSPO!)
+    "importance_sampling_level": "sequence",
+    "num_iterations": 1,
+}
+```
+**Assessment: CORRECT** — TRL expresses GSPO as grpo-loss + sequence IS, and `scale_rewards="group"` (std-norm) matches the GSPO paper. The comment in `composer_trainer.py:818` accurately notes "trl has no literal 'gspo'."
+**ONE SUBTLE ISSUE:** The GSPO paper's clipping ranges are of a **different magnitude** than GRPO. GSPO uses `ε ≈ 3e-4` (left) and `4e-4` (right) vs GRPO's `0.2/0.27`. The repo's GSPO preset does not set custom clipping ranges — it uses whatever TRL's default is for `loss_type="grpo"`. The `epsilon` and `epsilon_high` are likely to be GRPO-scale defaults (0.2), which would be dramatically wrong for GSPO's geometric-mean ratio. **This is an unaddressed subtlety** — the GSPO paper explicitly states the clipping ranges "differ by orders of magnitude" between GSPO and GRPO (Section 4.1, Fig. 2). The repo's GSPO preset needs custom `epsilon`/`epsilon_high` at GSPO scale (e.g., 3e-4/4e-4).
+### 1.5 CISPO (arXiv:2506.13585, MiniMax-M1)
+**CISPO key idea:** Rather than clipping the PPO surrogate objective (which zeroes gradient for out-of-range IS weights), CISPO clips the IS weight itself as a **stop-gradient coefficient** — every token always contributes to gradient.
+**CISPO objective:**
+```
+J_CISPO(θ) = E_{q, {o_i}~π_old} [
+    (1/Σ_i|o_i|) Σ_i Σ_t sg(clip(r_{i,t}(θ), 1-ε_low^IS, 1+ε_high^IS)) · Â_{i,t} · log π_θ(o_{i,t}|q,o_{i,<t})
+]
+```
+where `sg(·)` = stop-gradient. So the IS weight `r̂_{i,t}` is a **detached constant coefficient** on log π, not a differentiable term.
+**Key deltas from GRPO/PPO:**
+1. **No min(r·A, clip(r,...)·A)** surrogate — just `r̂(detached) · log π` (pure weighted REINFORCE)
+2. IS weight is clipped but stop-grad'd — zeroing a token's IS weight sets its gradient contribution to zero; NOT clipping means every token contributes, even far-from-policy ones
+3. In MiniMax-M1's implementation: `ε_low^IS` is not applied (no lower bound); only `ε_high^IS` is tuned (~5 in their experiments, which is "ScaleRL" range)
+4. **KL removed**: no KL penalty, beta=0
+5. Advantage uses group-relative std-norm (like original GRPO, unlike Dr.GRPO)
+**Why it's more efficient than GRPO/DAPO (paper's argument):** GRPO/DAPO zero gradients for tokens whose IS ratio falls outside the clip window. In long-response reasoning, MANY tokens get zeroed. CISPO avoids this — the IS weight is merely down-weighted, never zeroed. Result: 2× speedup vs DAPO empirically on Qwen2.5-32B math (Fig. 2 of MiniMax-M1 report).
+**repo's CISPO preset:**
+```python
+"cispo": {
+    "loss_type": "cispo",
+    "scale_rewards": "none",      # ← Dr.GRPO regime, no std-norm
+    "epsilon_high": 5.0,          # ← matches ScaleRL range
+    "importance_sampling_level": "token",
+    "num_iterations": 1,
+}
+```
+**Assessment: MOSTLY CORRECT but two issues:**
+1. The MiniMax-M1 paper uses `scale_rewards` with group std-norm (the paper's advantage formula includes std-normalization). Setting `scale_rewards="none"` deviates from the paper — though Dr.GRPO argument applies here too, this is a deliberate design choice not explicitly sourced from the CISPO paper itself.
+2. The KL treatment is not explicitly set to `beta=0` in the CISPO preset. DAPO sets `beta=0.0` explicitly, but CISPO doesn't — this relies on TRL's default. Should be made explicit to match the paper ("There is no KL penalty term in CISPO similar to other recent works" — MiniMax-M1 L1170-1174).
+---
+## 2. Comedy of Estimators (arXiv:2512.21852) — what it claims, what the repo says
+**Critical limitation:** The HTML experimental version returns 404; PDF is binary. Only the abstract and cross-references within the repo's research notes are available for full-text analysis. The abstract (v3, 2026-03-18) states:
+> "we observe that, in on-policy settings: (1) estimator configurations with **biased gradients can result in training instabilities**; and (2) using estimator configurations resulting in **unbiased gradients leads to better performance on in-domain as well as out-of-domain tasks**."
+> "We also investigate the performance resulting from different KL configurations in off-policy settings and observe that KL regularization can help stabilize off-policy RL training resulting from asynchronous setups."
+**Models tested (from abstract):** Qwen2.5-7B, Llama-3.1-8B-Instruct, Qwen3-4B-Instruct-2507.
+**What the repo claims about it:**
+The repo (`research/design-F5-fidelity-audit.md:10`, `kl_in_reward.py` docstring, `composer_trainer.py:79`) claims:
+> "arXiv:2512.21852 ('A Comedy of Estimators') — k1-in-reward improves OOD generalization; k3-in-reward can collapse."
+**ACCURACY ASSESSMENT:** The abstract supports the general claim that biased-gradient configurations cause instabilities and unbiased ones improve OOD performance. However the specific claim "k3-in-reward can collapse" requires verification against the full paper text — the abstract doesn't explicitly say which estimator is biased in which placement. The repo's characterization is **plausible but partially unverified** (full text unavailable). The abstract's claim about "unbiased gradients → better OOD" supports the repo's k1-in-reward motivation, but the specific k1-vs-k3 ranking and which placement (in-reward vs in-loss) is correct cannot be confirmed from the abstract alone.
+**What the repo says about verl:** `kl_in_reward.py:12` claims "verl adopted k1-in-reward as its *only* reverse-KL option." This cannot be verified against source without reading verl's codebase — but the research/design-F5-fidelity-audit.md records it as a reference alongside TRL issue #4967 (also not read directly).
+**SCALE / TASK CAVEAT:** The Comedy of Estimators was tested on 7B/8B models on reasoning tasks. Composer 2 runs a 1.04T/32B-active MoE on agentic coding. Generalization of 7B math findings to 1T MoE coding is an extrapolation the repo does not explicitly flag.
+---
+## 3. k1-in-reward implementation: faithfulness audit
+### 3.1 What the Composer 2 report specifies
+Composer 2 Technical Report §4.1 (verified verbatim, `research/notes/composer-2-technical-report.md:442-497`):
+> "Many open-source implementations of RL estimate KL with the estimator k3 = (r−1) − log r... However, Amini et al. shows [that] the variance increases drastically as p and q diverge... Therefore, we use the standard estimator **k1 = −log r** instead."
+This is KL applied **in the reward** (PPO-style), not in the loss:
+> "Similar to prior work... we use a Kullback–Leibler divergence for regularization, KL(q||p) = E_{x~q}[-log r(x)]"
+So the Composer 2 choice is:
+- **k1 estimator**: `KL_i ≈ Σ_t (logp_{i,t} - ref_logp_{i,t})` (= `-log r` where r = π_ref/π_θ... wait, need to check sign)
+**Sign convention note:** In the Composer 2 report, `r(x) = p(x)/q(x)` where p=π_ref (reference) and q=π_θ (current). So `−log r = −log(π_ref/π_θ) = logp - ref_logp` per token. This is the KL(π_θ||π_ref) direction.
+**In the repo's kl_in_reward.py:**
+```python
+# k1 estimator:
+per_token = (policy_logps - ref_logps) * completion_mask   # logp - ref_logp
+return per_token.sum(dim=-1)
+```
+And `apply_kl_in_reward` subtracts `coef*(KL_i - group_mean(KL))` from advantages.
+**Sign check:** `adv'_i = adv_i - coef*(KL_i - mean(KL))`. If KL_i is large (policy has drifted far from reference), the advantage is reduced — penalizing divergence. This matches the standard RLHF KL penalty direction. ✓
+**The penalty direction in original GRPO/PPO-RLHF:** The standard KL-in-reward is:
+`reward'_i = reward_i - coef * KL(π_θ||π_ref)`, where KL ≥ 0.
+With k1: `KL ≈ Σ_t (logp - ref_logp)`. If policy is very close to reference, `logp ≈ ref_logp`, KL ≈ 0. If policy drifts positive (puts more mass on this response), `logp > ref_logp`, KL > 0, penalty increases. This is the reverse KL direction (π_θ||π_ref), appropriate for preventing drift. ✓
+### 3.2 The algebra: fold-then-baseline identity
+The `kl_in_reward.py` docstring and `apply_kl_in_reward` implement:
+```
+adv'_i = adv_i - coef*(KL_i - group_mean(KL))
+       = [R_i - group_mean(R)] - coef*[KL_i - group_mean(KL)]
+       = [R_i - coef*KL_i] - group_mean([R - coef*KL])
+       = reward'_i - group_mean(reward')
+```
+This identity is exact under a **linear** (group-mean) baseline with **no std-normalization** (`scale_rewards="none"`). The docstring correctly states this requires the Dr.GRPO regime.
+**With std-normalization (vanilla GRPO):** the identity breaks because `group_std(R - coef*KL) ≠ group_std(R) - coef*group_std(KL)` in general. `validate_kl_in_reward_config` correctly raises if `scale_rewards` is not in {none, false}.
+**FAITHFUL IMPLEMENTATION? YES** for the on-policy Dr.GRPO regime. The identity algebra is mathematically sound and unit-tested.
+### 3.3 Deferred penalty case (aligned non-vLLM)
+The `_generate_and_score_completions` override handles two cases:
+1. **vLLM path**: `old_per_token_logps` is available at scoring time → apply penalty immediately, fold into advantages
+2. **Aligned non-vLLM path**: `old_per_token_logps` may be None at scoring time → stash `_kl_in_reward_applied=False`, defer to `_grpo_loss_kl_in_reward` which gets `per_token_logps` from the model forward pass
+**Issue in case 2:** The deferred path re-runs the model forward pass inside `torch.no_grad()` (`_grpo_loss_kl_in_reward:308-322`) to get `policy_logps`. This is the **sampling-time policy** (old policy), but by the time `_compute_loss` is called, gradient updates may have already been applied (if using multiple inner iterations). With `num_iterations=1` (the Dr.GRPO default), there's exactly one update epoch per batch, so this is fine. With `num_iterations>1`, the deferred path would compute KL against the *updated* policy instead of the *sampling-time* old policy — a subtle correctness bug. The `validate_kl_in_reward_config` does NOT guard against `num_iterations>1`. **This is a missing precondition.**
+### 3.4 Per-token vs per-sequence: where the KL penalty lands
+The repo computes the KL penalty as a **per-sequence sum** (not per-token mean):
+```python
+per_token = (policy_logps - ref_logps) * completion_mask
+return per_token.sum(dim=-1)   # ← SUM, not mean
+```
+Then `apply_kl_in_reward` subtracts `coef * (KL_i - group_mean(KL))` from the **scalar advantage** (one number per sequence).
+This is the correct "in-reward" treatment: the penalty is on the total sequence reward, not added token-by-token to the loss as verl's per-token path would. The per-token sum equals the total sequence KL contribution.
+**Compared to verl:** The original GRPO/PPO approach adds `β * KL_per_token` inside the per-token surrogate loss. The repo instead adjusts the advantage (scalar) by the summed KL. The gradient signal is different:
+- In-loss (per-token): every token gets a gradient from the KL term
+- In-reward (advantage adjustment): the KL modifies the *advantage*, which then scales all token gradients uniformly
+The "in-reward" vs "in-loss" distinction affects whether the KL gradient flows through the IS ratio or not. In the in-reward approach (the repo's), the KL is folded into the advantage before the surrogate is computed — so the surrogate objective is `min(r_t * (A + KL_correction), clip(...) * (A + KL_correction))`. In the in-loss approach (TRL's native k3), the KL is added directly as `β * k3_per_token` outside the surrogate clip.
+---
+## 4. Findings on the repo's research notes accuracy
+### 4.1 research/10: Dr.GRPO claims — ACCURATE
+The mining note (`research/10-composer2-techreport-mining.md:43-55`) states:
+- "Built on Dr. GRPO [34=Liu et al., 2503.20783]" ✓ (verified against Composer 2 report §4.1)
+- "Remove the length-standardization term" ✓
+- "Do NOT normalize group advantages by std" ✓
+- "k1 estimator (-log r)" ✓ (verbatim from report §4.1 L488-497)
+- "DAPO overlong-rollout masking explicitly tried and rejected" ✓ (verbatim from report L362)
+### 4.2 research/10: DAPO overlong masking rejection — ACCURATELY ATTRIBUTED
+The note correctly attributes the DAPO overlong masking rejection to small-scale experiments. The Composer 2 report states "at small scale" — this qualifier is preserved in research/10. ACCURATE.
+### 4.3 The k1-in-reward claim for verl — PARTIALLY UNVERIFIED
+`kl_in_reward.py:12` claims: "verl adopted k1-in-reward as its *only* reverse-KL option." This appears in the docstring as motivation but has not been verified against verl source code in this review. The claim is consistent with verl's documented `kl_penalty="kl"` option (= `logp - ref_logp` = k1) but verl also supports `kl_penalty="low_var_kl"` (= k3). **Claiming it's the "only" option is likely an overclaim** — verl supports both k1 and k3 (see `research/notes/eks-primary-sagemaker-hybrid-architecture-and-the-minimal-repo-delta.md:58` which mentions `kl_coef=0.001` without specifying which estimator). **This claim should be softened to "verl defaults to / recommends k1-in-reward."**
+### 4.4 Comedy of Estimators claim — SUPPORTED BUT IMPRECISE
+The repo claims the Comedy of Estimators paper shows "k1-in-reward improves OOD generalization; k3-in-reward can collapse." The abstract says "biased gradient configurations → instabilities; unbiased → better OOD." The repo's characterization maps the abstract's claim to specific estimators (k1 vs k3) and specific placements (in-reward vs in-loss). This mapping is directionally correct based on the mathematical argument (k1-in-reward has unbiased gradient; k3-in-loss has biased gradient for the in-reward objective) but the paper's own empirical scope (7B/8B models, reasoning tasks) is not mentioned in the repo's framing.
+**Specific gap:** The Comedy of Estimators paper (v3 2026-03-18) was updated after the repo's k1-in-reward implementation (Wave 20, 2026-06-09). The v3 abstract now includes Qwen3-4B-Instruct-2507, suggesting the empirical results were expanded. Whether the v3 conclusions changed from v1 is unknown (no full text).
+### 4.5 TRL k3 in production — CORRECTLY DIAGNOSED AND GUARDED
+The repo's honest finding (`composer_trainer.py:708-716`, `make_dr_grpo_config` docstring):
+> "GRPOTrainer._compute_loss uses the k3 estimator `exp(ref_logp - logp) - (ref_logp - logp) - 1` (trl/trainer/grpo_trainer.py ~L2513), NOT the k1 estimator. k3 is Schulman's low-variance, always-non-negative KL approximation; k1 is its unbiased but higher-variance counterpart."
+This is correct. The original GRPO paper (DeepSeekMath) explicitly uses the k3 estimator in the loss:
+```
+D_KL[π_θ||π_ref] = π_ref(o_{i,t}|q,o_{i,<t}) / π_θ(o_{i,t}|q,o_{i,<t})
+                   - log(π_ref(o_{i,t}|q,o_{i,<t}) / π_θ(o_{i,t}|q,o_{i,<t})) - 1
+```
+(DeepSeekMath full text L1826, verified in this review)
+The comment "k3 = k1 + O((Δlogp)²)" is also correct mathematically (Taylor expansion), and the repo correctly notes the delta is small for r≈1.
+---
+## 5. Issues, Gaps, and Open Items
+### ISSUE 1: GSPO preset missing GSPO-scale clipping ranges — **UNADDRESSED, LOW-MEDIUM RISK**
+The GSPO paper (arXiv:2507.18071, Section 5.1) explicitly states:
+> "In GSPO, we set the left and right clipping ranges in Equation (5) to **3e-4 and 4e-4**, respectively. We compare against GRPO... and set the left and right clipping ranges in Equation (2) to **0.2 and 0.27**... Note that we observe a difference of **two orders of magnitude** in the fractions of clipped tokens between GSPO and GRPO."
+The repo's GSPO preset does not set `epsilon` or `epsilon_high`, so it inherits TRL defaults (likely 0.2). This means the GSPO preset would actually behave more like a GRPO run because the clipping range is 2 orders of magnitude too wide — almost nothing gets clipped, making the sequence-level ratio identical to no-clip REINFORCE. **The GSPO preset is architecturally correct but operationally wrong without custom epsilon values.**
+**Fix needed:**
+```python
+"gspo": {
+    "loss_type": "grpo",
+    "scale_rewards": "group",
+    "importance_sampling_level": "sequence",
+    "epsilon": 3e-4,        # ← ADD: GSPO-scale clipping
+    "epsilon_high": 4e-4,   # ← ADD: GSPO-scale clipping
+    "num_iterations": 1,
+}
+```
+### ISSUE 2: CISPO preset missing explicit beta=0 — **MINOR, EASY FIX**
+The MiniMax-M1 paper explicitly states no KL penalty in CISPO. The `cispo` preset doesn't set `beta=0.0` explicitly (unlike the `dapo` preset which does). If TRL defaults beta to something non-zero, the CISPO behavior would diverge from the paper. Should add `"beta": 0.0` to the `cispo` preset for safety.
+### ISSUE 3: kl_in_reward deferred path and num_iterations>1 — **CORRECTNESS BUG, LOW RISK IN PRACTICE**
+As noted in §3.3, the deferred penalty path (`_kl_in_reward_applied=False`) computes KL against the model's current weights at `_compute_loss` time. With `num_iterations>1`, this is no longer the sampling-time policy. `validate_kl_in_reward_config` does not guard against `num_iterations>1`. Since all current presets use `num_iterations=1`, this is currently moot but should be added as a precondition check.
+**Fix needed in `validate_kl_in_reward_config`:**
+```python
+# Add after existing checks:
+num_iterations = kwargs.get("num_iterations", 1)
+if int(num_iterations) > 1:
+    raise ValueError(
+        "kl_in_reward=True requires num_iterations=1: with >1 inner update epochs, "
+        "the deferred-penalty path computes KL against the updated policy, not the "
+        "sampling-time old policy. This is incorrect."
+    )
+```
+### ISSUE 4: CISPO scale_rewards="none" vs paper's std-norm — **DOCUMENTED DESIGN CHOICE, NOT A BUG**
+The CISPO paper uses group std-norm in the advantage. The repo sets `scale_rewards="none"` for CISPO. This is a deliberate choice to keep consistency with the Dr.GRPO regime. It is not a mistake but should be explicitly documented in the preset comment — currently the comment says "eps_max≈5 (ScaleRL)" but doesn't explain the std-norm deviation from the paper.
+### ISSUE 5: "verl only reverse-KL option" overclaim in docstring — **DOCUMENTATION INACCURACY**
+The `kl_in_reward.py:12` docstring claims "verl adopted k1-in-reward as its *only* reverse-KL option." This appears to be an overclaim — verl supports both `kl_penalty="kl"` (k1) and `kl_penalty="low_var_kl"` (k3-like). The phrase "only" should be removed or qualified.
+### ISSUE 6: Comedy of Estimators task/scale scope not flagged — **MINOR DOCUMENTATION GAP**
+The repo uses the Comedy of Estimators paper as motivation for k1-in-reward on a 1T MoE agentic system, but the paper's empirical scope is 7B/8B models on math reasoning. This extrapolation is not flagged anywhere in the codebase. Should add a caveat in the docstring noting the scale/task gap.
+---
+## 6. Summary Table
+| Paper | Loss math delta vs GRPO | KL treatment | repo accuracy |
+|---|---|---|---|
+| GRPO (2402.03300) | Baseline: 1/|o_i| length-norm + std-norm advantage | k3 in loss, beta=0.04 | Reference |
+| Dr.GRPO (2503.20783) | Remove 1/|o_i| AND /std(R) | k3 in loss (TRL) OR k1-in-reward (opt-in) | ACCURATE |
+| DAPO (2503.14476) | Decoupled clip-higher (ε_high>ε_low) + overlong filter + dynamic sampling | beta=0 (KL removed) | ACCURATE (ε_high=0.28 correct) |
+| GSPO (2507.18071) | Sequence-level IS ratio (geomean); sequence-level clip | KL not primary focus | ARCHITECTURALLY CORRECT but OPERATIONALLY WRONG (missing GSPO-scale epsilon 3e-4/4e-4) |
+| CISPO (2506.13585) | Detached clipped IS weight as scalar coefficient on log π; every token keeps gradient | beta=0 (no KL) | MOSTLY CORRECT; missing explicit beta=0; scale_rewards deviation from paper (deliberate) |
+| Comedy of Estimators (2512.21852) | KL estimator comparison: k1 (unbiased) vs k3 (biased-gradient in some placements) → unbiased → better OOD | k1-in-reward recommended | SUPPORTED but full-text unavailable; scale/task scope not flagged |
+---
+## 7. Load-Bearing Verbatim Quotes
+**Dr.GRPO paper (arXiv:2503.20783, Sec. 3.2):**
+> "To avoid the aforementioned optimization bias in GRPO, we propose to simply remove the `1/|o_i|` and `std({R(q,o_1),...,R(q,o_G)})` normalization terms."
+**Composer 2 Technical Report (§4.1, verified via research/notes/composer-2-technical-report.md:442-497):**
+> "Many open-source implementations of RL estimate KL with the estimator k3 = (r−1) − log r... the variance increases drastically as p and q diverge... Therefore, we use the standard estimator **k1 = −log r** instead."
+> "We did not see benefits with overlong masking at small scale and opted not to mask rollouts that exceed the maximum sequence length."
+**GSPO paper (arXiv:2507.18071, Sec. 5.1):**
+> "Note that we observe a difference of **two orders of magnitude** in the fractions of clipped tokens between GSPO and GRPO (while adjusting the clipping ranges does not alter the disparity in magnitude)."
+**CISPO paper (arXiv:2506.13585, Sec. 3.1):**
+> "Rather than clipping the token updates as in PPO/GRPO, we instead clip the importance sampling weight in Eq. 3 to stabilize training... **CISPO... always leverages all tokens for gradient computations.**"
+> "There is no KL penalty term in CISPO similar to other recent works."
+---
+## 8. Priority Action Items
+1. **(HIGH)** Fix GSPO preset: add `"epsilon": 3e-4, "epsilon_high": 4e-4`. Without GSPO-scale clipping, the sequence IS ratio will rarely be clipped and the behavior degrades to no-clip sequence-level REINFORCE — not what Qwen's GSPO experiments demonstrate.
+2. **(MEDIUM)** Add `"beta": 0.0` to CISPO preset for explicitness.
+3. **(LOW)** Add `num_iterations=1` guard to `validate_kl_in_reward_config` for the deferred-penalty correctness.
+4. **(LOW)** Soften the "verl *only* reverse-KL option" claim in `kl_in_reward.py:12` to "verl defaults to / recommends k1-in-reward."
+5. **(LOW)** Add a docstring note to the k1-in-reward motivation flagging the Comedy of Estimators paper was tested at 7B/8B math scale, not at 1T MoE agentic scale — the extrapolation is reasonable but should be disclosed.

research/deepread/05-diloco.md ADDED Viewed

	@@ -0,0 +1,309 @@

+# Deep-read critical review: DiLoCo / Streaming DiLoCo family
+**Date:** 2026-06-09
+**Sources read verbatim:**
+- arXiv:2311.08105 (DiLoCo, Douillard et al. 2023/2024, HTML full text, 9503 words)
+- arXiv:2501.18512 (Streaming DiLoCo, Douillard et al. 2025, HTML full text, 19297 words)
+- arXiv:2502.12996 (Eager Updates, Kale, Douillard, Donchev 2025, abstract)
+- torchft/local_sgd.py (live main branch, fetched 2026-06-10)
+**Repo artefacts cross-checked:**
+- `composer_replication/diloco/__init__.py`
+- `composer_replication/diloco/serverless/allreduce.py`
+- `docs/adrs/ADR-003-diloco-impl.md`
+- `docs/adrs/ADR-005-serverless-diloco.md`
+- `research/02-diloco-family.md`
+- `research/design-F4-decoupled-diloco-s3.md`
+---
+## 1. Primary-source extraction: DiLoCo (arXiv:2311.08105)
+### 1.1 Algorithm
+Algorithm 1 (paper §2) is:
+1. Outer step t = 1…T
+2. Each worker i: θ_i^(t) ← θ^(t-1) (re-initialized from global params)
+3. H inner steps: θ_i^(t) ← InnerOpt(θ_i^(t), ∇L) using AdamW
+4. Outer gradient: **Δ^(t) = (1/k) Σ_i ( θ^(t-1) − θ_i^(t) )**
+5. Outer update: θ^(t) ← OuterOpt(θ^(t-1), Δ^(t))
+**Sign convention (paper, verbatim):** "the delta in parameters space…is computed per worker…Δ^(t) ← (1/k) Σ_i (θ^(t-1) − θ_i^(t))". This is θ_initial minus θ_local — the **negative of the local update direction**.
+### 1.2 Outer optimizer hyperparameters (paper §3.2 ablation)
+Paper reports tuning outer optimizer across SGD, SGDM, Nesterov, Adam. Results in Table 5:
+> "the setting with outer learning rate equal to **0.7** and outer momentum equal to **0.9** is very robust, and it is adopted for all our experiments throughout."
+Chosen values (bold in Table 5): **outer LR = 0.7, outer momentum = 0.9, Nesterov.**
+### 1.3 Sync frequency H
+Figure 4 sweeps H ∈ {50, 100, 250, **500**, 1000, 2000}. Main experiment default: **H = 500**.
+> "communicating more frequently than H = 500 steps leads to diminishing returns. Moreover, the performance degradation is very mild up to H = 1000 steps."
+H = 2000 causes meaningful degradation. H = 500 chosen as best trade-off.
+### 1.4 Heterogeneous / unreliable workers
+Section 3.1 "Adaptive compute pool": workers can join/leave; final perplexity tracks total compute budget regardless of schedule. "Models quality is affected by the total amount of compute, but not as much by how such computed is allocated over time."
+Section 3.1 "Asynchronous Communication": outer gradients dropped with probability up to 50%. At 50% drop rate, perplexity degrades only 2.1% relative to perfect communication.
+**Limitation (§5):** "The version of DiLoCo presented here assumes that all workers are **homogeneous**. However, in practice workers might operate at wildly different speed." Heterogeneous/async DiLoCo is explicitly listed as future work.
+### 1.5 What the original paper does NOT cover
+- No scaling laws for optimal H vs model size.
+- No quantization (mentioned pruning in appendix: 50% sign-pruning costs only +0.39% PPL).
+- Models only up to 400M params.
+- All experiments start from a 24k-step pretrained checkpoint (cold-start is studied but as secondary).
+- No measurement of real wall-clock with actual network constraints.
+---
+## 2. Primary-source extraction: Streaming DiLoCo (arXiv:2501.18512)
+### 2.1 Three contributions
+1. **Fragment streaming (§2.2):** Partition model into P fragments. Each fragment syncs every H steps, but fragments are staggered (offsets t_p). Peak bandwidth reduced by factor |p|/L (fragment size/total layers).
+2. **Overlapping communication with computation (§2.3):** τ parameter (inner overlap delay). Fragment's allreduce initiated at step t, results applied τ steps later. Workers keep training during those τ steps. For heterogeneous workers: use per-worker τ values. Robust to τ up to ~5 inner steps.
+3. **Low-precision outer gradients (§2.4):** Outer gradients (what's communicated, not optimizer state) quantized to **FP4 = E3M0** (1 sign bit, 3 exponent bits, 0 mantissa bits). Accumulation in FP32. No regression found at 4-bit. Applied at send time, before allreduce.
+**Combined result:** "reducing required bandwidth by two orders of magnitude" (abstract). Table 1 shows Data-Parallel 441 TB vs Streaming DiLoCo 1.10 TB (≈400× total bits reduction for 1B model).
+### 2.2 Outer hyperparameters in Streaming DiLoCo
+> "The main hyperparameter of DiLoCo is its outer learning rate; we tuned it to be optimal at small scale at **0.4**, and kept it fixed across all scales."
+Streaming paper uses **outer LR = 0.4**, not 0.7. Momentum: not explicitly restated; inherits from original paper's Nesterov momentum = 0.9.
+H values in experiments: **H = 30 and H = 100** (not H = 500 from the original paper).
+### 2.3 Fragment scheduling
+Algorithm 2: condition `t - t_p mod H == 0` to decide which fragment syncs at step t. Fragmented offsets allow continuous streaming. "As we increase model scale, the fragment definition…is maintained, which means that larger models have more fragments."
+### 2.4 Heterogeneous workers
+§3.3.2 (Overlapping with slack between workers): per-worker τ_m handles execution speed differences. "the loss degradation is limited under a delay of up to **5 inner steps**." Above τ ≈ 5, degradation increases.
+### 2.5 Memory overhead
+66% more memory than Data-Parallel (outer parameters copy + Nesterov state). For Streaming: only active fragment's outer state needs to be in HBM; rest can be on CPU. For 100B model with 3-layer fragment out of 108 layers: ~2% additional memory.
+### 2.6 "Liu et al. 2024a" citation in Streaming paper
+The Streaming paper references "Liu et al. (2024a)" = Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A. Rusu, Jiajun Shen, Arthur Szlam, Marc'Aurelio Ranzato. "Asynchronous local-sgd training for language modeling." arXiv:2401.09135. This is cited in the context of per-worker slack for heterogeneous workers.
+---
+## 3. Primary-source extraction: torchft/local_sgd.py
+### 3.1 Sign convention (verbatim from `_save_grads`)
+```python
+def _save_grads(self) -> None:
+    with torch.no_grad():
+        for name, p in self._model_fragment.named_parameters():
+            ...
+            pseudogradient = (
+                self.original_parameters[name].to(p.device) - local_param
+            )
+            self._grads[name] = pseudogradient
+```
+`original_parameters[name]` = θ_initial, `local_param` = θ_local.
+**torchft computes: pseudogradient = θ_initial − θ_local** (same sign as paper's Δ).
+### 3.2 Outer optimizer application chain (verbatim from `perform_sync`)
+```
+1. _save_local_parameters()  # saves θ_local into _local_parameters
+2. restore_parameters()      # p.data ← θ_initial (from original_parameters)
+3. _set_grads()              # p.grad ← averaged_pseudogradient
+4. _outer_optimizer.step()   # SGD Nesterov: p.data ← θ_initial - lr*Nesterov(avg_pseudograd)
+5. save_parameters()         # original_parameters ← p.data (θ_outer_updated)
+6. _merge_parameters()       # p.data ← alpha*p.data + (1-alpha)*_local_parameters
+   # For alpha=0.0 (vanilla): p.data ← θ_local
+```
+Post-sync state: `p.data = θ_local`, `original_parameters = θ_outer_updated`.
+Next outer round: `restore_parameters()` sets `p.data = θ_outer_updated`; H inner steps produce `θ_local_next`.
+Pseudograd_next = θ_outer_updated − θ_local_next. **This is faithful to Algorithm 1.**
+### 3.3 Fragment rotation logic
+```python
+def _current_fragment(self) -> int:
+    step = self._manager.current_step()
+    return step % len(self._fragments)
+```
+For vanilla (1 fragment): always fragment 0. For Streaming with P fragments: round-robin by `step % P`.
+`start_quorum()` is called at `step = sync_every - fragment_sync_delay` (prepare_sync time).
+`current_step()` is read at `step = sync_every` (perform_sync time).
+### 3.4 `_use_async_quorum` constraint
+`DiLoCo.__init__` raises `ValueError` if `manager._use_async_quorum` is truthy. Must be False for synchronous quorum (as in the paper's design).
+---
+## 4. Repo correctness findings
+### F1. Sign convention — CORRECT
+`__init__.py` lines 17–38 documents the convention: "pseudograd = θ_initial - θ_local (per torchft's `_save_grads()`)". The code's outer SGD uses standard `p.data ← p.data - lr * grad` where `p.data = θ_initial` (restored before outer step) and `grad = avg_pseudograd = avg(θ_initial - θ_local)`. The arithmetic is correct and matches Algorithm 1. The sign-convention test in `spikes/008-streaming-diloco/tests/test_diloco_smoke.py` is appropriate insurance.
+**No bug. No mischaracterization.**
+### F2. Outer optimizer hyperparams — MINOR DIVERGENCE
+Repo default: `outer_lr=0.7, outer_momentum=0.9, nesterov=True` (lines 69–72 of `__init__.py`). These match the **original DiLoCo paper** (§3.2, Table 5 bold values).
+The **Streaming DiLoCo paper** uses `outer_lr=0.4` tuned at small scale. The repo uses original-paper values, which is appropriate for v0.1 (vanilla DiLoCo). But if someone enables Streaming by setting `fragment_sync_delay>0` and `model_fragments=[f0,...,fN]`, they will use lr=0.7 where the Streaming paper used lr=0.4. This is not a correctness bug but may underperform Streaming DiLoCo in practice.
+**Recommendation:** Add a note in `make_diloco_outer_loop` docstring: "Streaming DiLoCo (Douillard et al. 2025) tunes outer_lr=0.4; the default 0.7 is optimal for vanilla DiLoCo."
+### F3. Default sync_every=100 vs paper's H=500
+Repo default: `sync_every=100` (line 72 of `__init__.py`).
+Original DiLoCo paper main experiment: **H=500**. OpenDiLoCo: H=125. Streaming paper: H=30 or H=100.
+The default H=100 is consistent with the Streaming paper and OpenDiLoCo but does NOT match the original paper's "main experiment default." The ADR-003 docstring says "Default hyperparams (DiLoCo paper §3.2): outer_lr = 0.7, outer_momentum = 0.9, Nesterov" — this is correct for lr/momentum but the paper's main default H is 500, not 100.
+**Not a correctness bug** (H=100 is within the paper's tested range and performs well). But the docstring claiming it cites "DiLoCo paper §3.2" for the H default is misleading — §3.2 chooses H=500.
+**ADR-005** says "H = 500-1000 inner steps" — this is consistent with the original paper's claimed range, though the practical default in code is 100.
+### F4. Author misattribution of Streaming DiLoCo — INCORRECT
+`__init__.py` line 8: `"Streaming DiLoCo (Liu et al. 2025)"`.
+`design-F4-decoupled-diloco-s3.md` line 109: `"Streaming DiLoCo (Liu et al. 2025, 'Eager Updates for Overlapped Communication in DiLoCo', arXiv:2501.18512)"`.
+**All three facts here are wrong:**
+- arXiv:2501.18512 is authored by **Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, et al.** — not "Liu et al."
+- The title of arXiv:2501.18512 is "**Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch**" — not "Eager Updates for Overlapped Communication in DiLoCo."
+- "Eager Updates for Overlapped Communication and Computation in DiLoCo" is a **separate paper**, arXiv:2502.12996, by Satyen Kale, Arthur Douillard, Yanislav Donchev — also not "Liu et al."
+- "Liu et al. (2024a)" is the correct citation for the **Async Local-SGD** paper (arXiv:2401.09135), not Streaming DiLoCo.
+The confusion appears to be: design-F4 imported "Liu et al. 2025" from a different context (possibly confusing with Dr.GRPO or another Liu et al. paper), attached it to the wrong arXiv ID, and applied the wrong title. The `__init__.py` propagated the wrong author name from design-F4.
+**Correct attributions:**
+- Vanilla DiLoCo: Douillard et al. 2023/2024, arXiv:2311.08105.
+- Streaming DiLoCo: **Douillard et al. 2025**, arXiv:2501.18512.
+- Eager Updates: Kale, Douillard, Donchev 2025, arXiv:2502.12996 (companion/workshop paper with noted text overlap with 2501.18512).
+**Files to fix:** `composer_replication/diloco/__init__.py` line 8; `research/design-F4-decoupled-diloco-s3.md` line 109.
+### F5. "Streaming degrades to vanilla" — IMPRECISE BUT DEFENSIBLE
+`design-F4-decoupled-diloco-s3.md` line 116 states:
+> "prepare_sync blocks for the full S3 rendezvous and `fragment_sync_delay` buys zero overlap — **Streaming degrades to vanilla**, correctly but without the comm/compute overlap benefit."
+This is directionally true but imprecise. With synchronous `ObjectStoreAllReduce`:
+- **What is PRESERVED:** fragment streaming (per-fragment partial sync, multiple fragments synced in staggered rotation). A 4-fragment model still syncs only 1/4 of parameters per outer step boundary; peak bandwidth is still reduced.
+- **What is LOST:** the τ (fragment_sync_delay) overlap benefit — inner training steps can no longer run while the allreduce is in flight because the allreduce blocks.
+So the correct characterization is: "synchronous allreduce loses **overlap** (Contribution 2 of Streaming DiLoCo), but **fragment partial sync** (Contribution 1) still works. fragment_sync_delay=0 with multiple fragments still gives partial-sync bandwidth savings."
+The statement "degrades to vanilla" implies full-model-sync-at-H, which is not what happens for multi-fragment configurations. However, for the current Spike 008 configuration (single fragment, fragment_sync_delay=0), this degrades to exactly vanilla DiLoCo, which makes the statement accidentally correct for that specific case. The broader claim for multi-fragment Streaming is imprecise.
+**design-F4 correctly notes the fix (non-blocking PUT returning deferred Work) and correctly defers it to Phase 5.** The characterization is acceptable as long as readers understand it applies to overlap loss, not to fragment streaming.
+### F6. Quantization — NOT IMPLEMENTED, CORRECTLY OMITTED
+`MockManager.allreduce` signature: `def allreduce(self, tensor, **_kwargs)` — the `should_quantize` keyword is silently absorbed by `**_kwargs` and passed to `ObjectStoreAllReduce`, which has no quantization logic. FP32 tensors are serialized directly via `torch.save`.
+The Streaming paper's FP4 (E3M0) outer gradient quantization is not implemented. This is appropriate for v0.1 vanilla scope. The paper shows: E3M0 quantization cuts communicated bits by 8× (FP32→FP4) with no regression. The framework loses this efficiency benefit but is algorithmically correct.
+**research/02** line 244 incorrectly describes Streaming DiLoCo as using "FP16 outer state" for compression. The paper uses **FP4 outer gradients** (what is transmitted), NOT FP16 optimizer state. These are different things. FP16 is used by OpenDiLoCo (separately) to cut payload 2×; Streaming DiLoCo uses FP4 = 8× reduction.
+### F7. H range and sync cadence claims — MINOR INACCURACY in research/02
+`research/02` Table §2 says Streaming DiLoCo uses "Continuous partial" sync frequency. This is accurate. But the bandwidth reduction column "~100× peak BW + frequency" is underestimated. The paper's Table 1 shows ≈400× total bits reduction (not 100×) for a 1B model. The "two orders of magnitude" phrasing in the abstract means ≥100×; the actual measured result is ≥400×.
+### F8. Heterogeneous worker handling — PARTIALLY MISSING in research/02
+`research/02` says Streaming DiLoCo has "better tolerance since communication is continuous, but still synchronous." This misses the per-worker τ mechanism the Streaming paper introduces. The paper explicitly shows: with τ_1=1, varying τ_2 up to 5 inner steps shows robust degradation curve. This is a first-class mechanism for heterogeneous workers, not just an incidental property.
+The correct characterization: Streaming DiLoCo with per-worker τ tolerates timing heterogeneity of up to ~5 inner steps without significant degradation.
+---
+## 5. ADR-003 correctness summary
+| Claim | Verdict |
+|---|---|
+| torchft computes pseudogradient = θ_initial − θ_local | CORRECT (verified from source) |
+| Outer optimizer sign: no negation needed in our wrapper | CORRECT (SGD subtracts, pseudograd is already "subtract away from θ_local") |
+| fragment_sync_delay > 0 requires CUDA streams | CORRECT (torchft uses `torch.Stream` for overlap; without a stream, overlap is serial) |
+| Spike 008 uses vanilla (single fragment, delay=0) | CORRECT |
+| Streaming is "a configuration-flag away" | CORRECT (same API, just different params) |
+| torchft is Meta-maintained BSD-3 | CORRECT |
+| Gap | Verdict |
+|---|---|
+| Default H=100 attributed to "DiLoCo paper §3.2" | MISLEADING — paper's main default is H=500; H=100 comes from OpenDiLoCo / Streaming paper range |
+| "Liu et al. 2025" for Streaming DiLoCo | WRONG — should be Douillard et al. 2025 |
+---
+## 6. ADR-005 correctness summary
+| Claim | Verdict |
+|---|---|
+| DiLoCo outer sync is once per H=500-1000 inner steps | CORRECT for original paper; Streaming paper uses H=30-100 |
+| Pseudo-gradient size ~2 GB for 1B model in bf16 | CORRECT: 1B params × 2 bytes/param = 2 GB |
+| Object-store rendezvous is bandwidth-efficient | CORRECT — well-reasoned; consistent with paper's communication profile |
+---
+## 7. DiLoCo scaling laws
+No dedicated DiLoCo scaling-laws paper exists as of search date. The Streaming DiLoCo paper (§3.2.1) shows scaling experiments from 35M to 4B parameters but does not derive scaling laws for optimal H or outer optimizer settings. Table 4 of the original paper shows modest scaling (60M, 150M, 400M). DiLoCoX (arXiv:2506.21263) scales to 107B but is a different framework.
+**The repo makes no explicit scaling-law claims for DiLoCo**, so no finding here.
+---
+## 8. Action items (priority-ordered)
+1. **[HIGH] Fix author attribution in two files:**
+   - `composer_replication/diloco/__init__.py` line 8: change `"Streaming DiLoCo (Liu et al. 2025)"` to `"Streaming DiLoCo (Douillard et al. 2025)"`.
+   - `research/design-F4-decoupled-diloco-s3.md` line 109: correct from `"Liu et al. 2025, 'Eager Updates…', arXiv:2501.18512"` to `"Douillard et al. 2025, 'Streaming DiLoCo…', arXiv:2501.18512"`.
+2. **[MEDIUM] Add outer_lr note for Streaming:**
+   - `make_diloco_outer_loop` docstring: note that Streaming DiLoCo (2501.18512) uses outer_lr=0.4 tuned at small scale, while the default 0.7 is optimal for vanilla.
+3. **[MEDIUM] Fix research/02 compression claim:**
+   - Line 244: "FP16 outer state" → "FP4 (E3M0) outer gradients" (what is communicated; accumulation stays FP32).
+   - Bandwidth reduction: "~100× peak BW + frequency" → "≥400× total bits + peak BW reduction" to match Table 1 of the Streaming paper.
+4. **[LOW] Clarify "degrades to vanilla" in design-F4:**
+   - The current text is accurate for the v0.1 single-fragment case. For multi-fragment configurations, tighten to: "fragment partial-sync bandwidth savings are preserved; only the τ overlap benefit is lost with synchronous allreduce."
+5. **[LOW] Fix H=100 default source attribution:**
+   - `__init__.py` docstring says "Default hyperparams (DiLoCo paper §3.2)" — add that §3.2 uses H=500 for main experiments; H=100 matches Streaming/OpenDiLoCo range.
+6. **[FUTURE] Phase-5 upgrade per design-F4:**
+   - Non-blocking `allreduce_async` in `ObjectStoreAllReduce` to realize genuine τ overlap on S3. ~60 LOC, deferred post-Phase-4.
+---
+## 9. What the papers say that the repo does NOT cover (gaps, not bugs)
+- **FP4 quantization of outer gradients** (Streaming paper §2.4): not implemented. Worth noting as Phase-5 item alongside the async allreduce.
+- **Per-worker τ for heterogeneous devices** (Streaming paper §3.3.2): the API supports different `fragment_sync_delay` values but there is no orchestration layer to set per-replica τ based on observed step latency. This is more of an RL orchestration concern than a DiLoCo wrapper concern.
+- **Decoupled DiLoCo** (DeepMind blog 2025, Gemma 4): a Pathways-style async variant not in the literature yet; no implementation expected.
+- **Async outer gradient application** (arXiv:2401.09135, Liu et al. 2024a = Async Local-SGD): delayed Nesterov (DN) optimizer + Dynamic Local Updates (DyLU) for heterogeneous workers. Not needed for v0.1 but relevant if serverless executors have highly variable step times.

research/deepread/06-worldmodel.md ADDED Viewed

	@@ -0,0 +1,298 @@

+# Deep-Read: World-Model / Deliberation Literature — Critical Review
+**Cluster 6 / Novel-Extension Guard**
+**Reviewer:** critical pipeline subagent
+**Date:** 2026-06-09
+**Sources fetched:** MuZero (1911.08265), DreamerV3 (2301.04104), CWM (2510.02387), Chain-of-World (2603.03195), foresight-governance (2601.03905), From-Word-to-World (2512.18832), Predictive-Causal Gap (2605.05029), Reasoning-Tool-Compete/DART (2602.00994), Myopic Planning (2605.06840), Negative Gradient/LLD (2505.18830), RAFT (2504.11343), Near-miss negatives (2503.14391).
+**Final report reviewed:** `research/notes/final_report_socratic-mcts-swe-worldmodel-8f6dea.md` sections 2–4.
+---
+## Executive Summary
+The report's world-model / deliberation section (§2–4) is well-structured and intellectually honest about uncertainty, but contains five factual misreadings of primary sources, three overclaims dressed as conditional commitments, and two omissions that materially weaken the evidence base. The most serious finding is a CWM misread: the paper does NOT "train on all trajectories for the world-model head, reserving success-filtering only for the RL reward" in the sense the report implies — CWM uses a *mid-training stage* architecture, not an auxiliary head on a policy network, and the "train on all" decision applies to a separate, structurally distinct training phase, not an add-on loss riding the policy head. The report imports this result as license for an "aux head trains on all" design that the source does not demonstrate. The other misreadings are: (a) CWM's 65.8% score requires test-time scaling and is not the base score; (b) the Chain-of-World (2603.03195) paper is a robotics/embodied VLA paper, not an SWE paper; (c) the foresight-governance paper (2601.03905) is VLM/VQA, not SWE; (d) the Predictive-Causal Gap paper (2605.05029) is a single-author preprint with linear-Gaussian proofs and a small Duffing-GRU sweep — the report presents it as if the SWE mixed-timescale argument is a theorem about the proposed system, which it is not. The overclaims are: the report commits "parameter isolation eliminates the interference risk" when 2602.00994 shows interference on parameter-isolated LoRA modules too; the report treats Foresight@k as a standard metric when it is a proposed construct with no published baseline; and the "two hard prune gates resolve the central question" framing obscures that neither gate addresses the predictive-causal gap the report itself invokes. The omissions are: no paper in the cluster studies next-state-prediction as an auxiliary loss on a policy network for software engineering tasks — the exact configuration proposed — so the evidentiary basis for the aux-loss design rests entirely on analogical transfer from CWM (different architecture) and MuZero/Dreamer (different domain). The report acknowledges uncertainty but does not flag this as the null-evidence zone it is.
+---
+## Section 2: World-Model Goal — Source-by-Source Findings
+### Finding 2.1 — CWM (arXiv:2510.02387): Misread of "trains on all" + score overclaim [CRITICAL]
+**What the report says (§2, line 33):**
+> "Meta's Code World Model mid-trains a 32B model on observation-action trajectories to predict next program state, reaching **65.8% on SWE-bench Verified** — crucially training *on all* trajectories for the world-model head, reserving success-filtering only for the RL reward [13]."
+**What the source actually says:**
+The CWM paper uses a three-phase training pipeline (pre-training → mid-training → post-training/RL). The "train on all" decision is a mid-training data decision for an entire separate training stage, verbatim:
+> "Because our goal with the ForagerAgent data is to learn a comprehensive world model of agentic interactions with code environments, **we do not filter trajectories based on whether they succeed at bug or issue resolution**." (Section 2.2, ForagerAgent)
+This is *not* an auxiliary loss added to a policy head. CWM is mid-trained as a *general purpose* next-state predictor in a dedicated training phase, separate from RL, using 3M ForagerAgent trajectories from 10.2k images. The world modeling capability is baked into the base model before policy optimization begins. The RL stage (Section 5.3.1) then applies success-filtered rewards *on top of* a model that already has world-modeling capability from mid-training.
+**Why this matters for the proposed design:** The report uses this citation as support for "a world-model aux head can train on all branches during RL." CWM does NOT support this. CWM supports "dedicate a mid-training stage to train-on-all dynamics learning before RL." These are architecturally distinct: CWM's world-modeling capability is in the base weights, not in an auxiliary head receiving gradients simultaneously with RL. The interference risk (2602.00994) the report cites elsewhere applies precisely to the simultaneous-gradient version CWM does NOT use.
+**Score overclaim:** The 65.8% SWE-bench Verified score requires test-time scaling (multiple candidates + ranking). CWM's base score (single attempt, no retry) is lower. The report does not make this distinction. The verbatim from the CWM abstract: "it reaches pass@1 scores of **65.8% on SWE-bench Verified (with test-time scaling)**." The base score "is computed with a single attempt per instance (no retries, majority voting, or parallel candidates), averaged over multiple runs." CWM's figure-2 caption explicitly notes this gap. Citing "65.8%" without the "(with test-time scaling)" qualifier is a misread of the headline number.
+**Verdict:** Overclaim on two dimensions. The aux-head-on-policy design does not have CWM support. The score should be cited as "65.8% with test-time scaling" or the base score should be stated alongside.
+---
+### Finding 2.2 — Chain-of-World (arXiv:2603.03195): Wrong domain attribution [HIGH]
+**What the report says (§2, line 35):**
+> "The latent-motion line carries the same discipline into 2026: factorize dynamics into a compact latent and predict the consequential terminal state, not the full frame [53]."
+The source list cites [53] as: "Chain of World: World Model Thinking in Latent Motion — arXiv:2603.03195 (CVPR 2026; disentangled latent-motion world model predicts terminal state instead of reconstructing redundant background)."
+**What the source actually says:**
+CoWVLA (Chain-of-World VLA) is a Vision-Language-Action model for **robotics**. It is submitted to CVPR 2026 under cs.CV. The authors are from Li Auto, Harbin Institute of Technology, and BAAI. The experimental benchmarks are robotic simulation benchmarks (manipulator tasks: grasping cups, placing objects). The architecture uses a video VAE to extract latent motion from physical-world video frames and predicts the terminal visual frame of a robot arm action segment.
+This paper has **no relevance to software engineering or LLM-based code agents**. Its "compact latent, predict terminal state, not full frame" insight is about pixel-space robot dynamics. The analogy to "don't reconstruct the full next repo state, predict the decision-relevant delta" is the report author's inference, not a claim the source makes. Using it as a citation for SWE world modeling is a domain transfer that the source does not support.
+**Verdict:** Improper citation. The paper should either be removed or explicitly noted as "robotics analogy, not SWE evidence."
+---
+### Finding 2.3 — Foresight Governance Paper (arXiv:2601.03905): Domain-transfer not flagged adequately [MEDIUM]
+**What the report says (§2, line 29):**
+> "handed a world model as a tool, agents invoke it under 1% of the time, misuse it ~15%, *degrade* when forced, and consult it *less* as they grow more capable — the bottleneck is foresight *governance* [11]"
+**What the source actually says:**
+The abstract is verbatim: "Across diverse agentic and visual question answering tasks, we observe that some agents rarely invoke simulation (fewer than 1%), **frequently misuse predicted rollouts (approximately 15%)**, and often exhibit inconsistent or even degraded performance (up to 5%) when simulation is available or enforced."
+The report's numbers are accurate. However, the paper's experimental domain is "agentic and VQA tasks" — Vision-Language Models (VLMs) on visual question answering. The degradation figures are from VLM/VQA settings, not SWE task settings. The paper itself notes the bottleneck is foresight governance in those domains.
+The report acknowledges this in §7: "the world-model-as-tool foresight result [11] is VLM/VQA." However, in §2 where this result is deployed as a structural argument for "it does not emerge from scale," the domain caveat is absent. A reader of §2 alone would not know this is VLM evidence being applied to SWE.
+**Verdict:** The numbers are accurately quoted, but the domain caveat is deferred to §7 and absent from §2 where the argument is made. Minor but addressable.
+---
+### Finding 2.4 — Predictive-Causal Gap (arXiv:2605.05029): Scope overstated [MEDIUM]
+**What the report says (§2, line 37):**
+> "across 2,695 networks **mean causal fidelity is 0.49** (only 2.5% exceed 0.70), and at high dimension (N=100) the optimal encoder becomes **causally blind (~1e-8) *while achieving 92% lower prediction error*** [18]. A SWE repo is exactly mixed-timescale..."
+**What the source actually says:**
+The paper is a single-author preprint (Kejun Liu, single affiliation) studying linear-Gaussian dynamics with a theorem and a nonlinear Duffing-GRU sweep. The 2,695-network count and the fidelity numbers are accurate. The theorem proves the gap for linear-Gaussian systems.
+The SWE-specific generalization ("A SWE repo is exactly mixed-timescale") is the report's inference, not the paper's claim. The paper does state implications for "world models" in general, but its empirical evidence is limited to linear-Gaussian dynamics and the Duffing oscillator nonlinear extension. There is no SWE experiment, no code model, no NLP experiment.
+Further, the paper uses "operational grounding" as a partial mitigation ("operational grounding — restricting the loss to system observables — partially suppresses the gap"). The report correctly notes "The value-equivalent target reduces but, by the theorem, never eliminates the gap" — this is accurate per the abstract. But the "never eliminates" is the theorem's conclusion for linear-Gaussian systems; the practical magnitude of the gap for an LLM trained on code SWE trajectories is unknown.
+**Verdict:** The report uses this paper's impossibility theorem to argue against an aux-head configuration that CWM does not use anyway (see Finding 2.1). The theorem is real and relevant, but applying it as if it proves the SWE aux-head will fail is an extrapolation the paper does not support. It is best-used as a risk flag, not a structural argument.
+---
+### Finding 2.5 — MuZero (arXiv:1911.08265) and DreamerV3 (arXiv:2301.04104): The "value-equivalent" translation is plausible but not direct evidence [MEDIUM]
+**What the report says (§2, line 33):**
+> "MuZero and Dreamer add the design discipline: learn the *value-equivalent* latent — predict reward, value, the signed `FAIL_TO_PASS` delta, the predicted `tool_error` kind — never reconstruct the full state [14][15]."
+**What the sources say:**
+MuZero (arXiv:1911.08265) learns a latent model that predicts reward, policy, and value function for MCTS planning in board games and Atari. DreamerV3 (arXiv:2301.04104) learns a RSSM world model that predicts compact latent states for imagined rollouts. Both papers operate in fully observable, discrete/continuous control domains with clear reward signals.
+The "value-equivalent latent" framing is from Schrittwieser et al. and is accurately invoked. However, neither paper has experiments in NLP, code generation, or multi-step software engineering. The translation from "predict reward/value/policy in Atari" to "predict FAIL_TO_PASS delta + tool_error kind for SWE" is the report's design inference.
+This is not a misread — it is analogical reasoning from RL theory, which is legitimate. But the report presents these as "design discipline" rather than "analogical design inspiration," which is a subtle overclaim. MuZero and Dreamer provide no direct evidence that their latent-representation principle transfers to transformer-based LLM policy training on code.
+**Verdict:** Sound analogy but presented as established principle. The report should note: MuZero/Dreamer motivate the value-equivalent design direction; they do not demonstrate it works in the LLM-policy-training regime.
+---
+## Section 3: Aux-Head as "Second SDPO Mode" — Overclaim Analysis
+### Finding 3.1 — "Parameter isolation eliminates the interference risk" is overclaimed [HIGH]
+**What the report says (§2, line 39):**
+> "three 2026 results pull the other way, hard, and they are why the aux loss must be a *separate* head and an *ablation*. First, interference: 'Reasoning and Tool-use Compete in Agentic RL' shows training reasoning and tool-use into one parameter set induces misaligned gradients, and decoupling into separate adapters (DART) beats every joint baseline across thirteen benchmarks [16] — stacking a next-state head onto the *same* policy head is exactly the configuration it indicts."
+The report then concludes (§2, line 39): the solution is a "parameter-isolated head or adapter."
+**What arXiv:2602.00994 actually shows:**
+DART decouples reasoning and tool-use into **separate LoRA modules** — but these LoRA modules share the same base model weights (frozen). The gradient interference is between the two LoRA adapters, not just between a head and a base. The paper's solution is to use *disjoint parameter sets* for the two capabilities. This means parameter isolation to a separate LoRA module does reduce interference, but the report's implication that a "parameter-isolated head" fully eliminates the problem is not the paper's finding.
+Specifically: DART's separate LoRA modules still share the frozen base, and the paper's ablation shows even with LoRA decoupling, there is residual interference. "Approaches the 2-Agent upper bound" (abstract) means it does not fully close the gap. The "two-Agent upper bound" is the theoretical ceiling achieved by having two separate models — separate LoRA on the same base does not achieve this.
+**Verdict:** The report correctly identifies that joint parameter training is the risk and that isolation helps. The overclaim is that isolation *eliminates* the risk. The source shows isolation *reduces* interference but does not eliminate it. The correct framing: "parameter isolation substantially reduces but does not eliminate gradient interference; a fully separate model achieves the upper bound."
+---
+### Finding 3.2 — "Foresight@k" is proposed, not standard [MEDIUM]
+**What the report says (§2, line 43):**
+> "**Foresight@k** — the lift in terminal pass-fraction when the deliberation token is allowed versus suppressed, sampling fixed — is **the kill ablation**: if it is ≈0, the token is a no-op and is cut [11][2]."
+**What the sources say:**
+Neither arXiv:2601.03905 nor any other cited source defines or uses "Foresight@k" as a metric. The term appears to be coined by the report. This is fine — defining a novel metric is reasonable — but the report presents it as a standard metric with source citations, which is misleading.
+**Verdict:** The metric is the report's proposal. The citations [11][2] do not define or use Foresight@k. The report should explicitly mark this as a proposed metric ("we define Foresight@k as...") rather than implying it is established.
+---
+### Finding 3.3 — The aux-loss-as-"second SDPO mode" claim is architecturally creative but unsupported [MEDIUM]
+**What the report says (§2, line 41):**
+> "A 'predict-the-outcome' target is the same shape: splice the *realized post-action observation* [...] into the teacher context as the privileged info, and distill the student toward the distribution it would have had if it had foreseen that outcome. [...] Because the teacher is stop-grad, a wrong predicted-outcome hint is *bounded-bad*..."
+This is presented as follows: the aux objective is a "second SDPO mode" riding `generalized_jsd_loss`, not a new loss term.
+**Assessment of the claim:**
+The architectural argument is internally consistent and clever. SDPO is hint-conditioned distillation; if the hint is the realized observation, then distillation toward what the model would have predicted with that hint is "predict the next state." The stop-grad safety argument is valid.
+However: the synthesis note in the vault (`latent-what-if-deliberation`) accurately identifies this as the report's *design inference*, not something any source supports directly. The gap-fill note (`gap-fill-counter-evidence`) explicitly identifies the missing ablation: "No SWE-specific next-state-head null result exists yet — that exact ablation is the cheapest decisive experiment we could run ourselves." The final report presents this design as a committed design with supporting evidence, when the supporting evidence is analogical (CWM uses a different architecture; MuZero/Dreamer operate in different domains).
+**Verdict:** The design argument is sound but the evidentiary claim is overclaimed. The source base for "aux-next-state-loss as second SDPO mode improves SWE agent performance" is zero. All sources are analogical or domain-different. The report should state this explicitly: "no prior work has tested this exact configuration; the nearest existential proof is CWM's mid-training architecture, which is structurally different."
+---
+## Section 4: Prune-vs-Train-on-All — Source Fidelity Check
+### Finding 4.1 — RAFT [2504.11343]: The report's characterization is accurate [OK]
+**What the report says:** "RAFT/rejection-sampling is competitive with GRPO at far less complexity, and GRPO's advantage comes from *discarding all-wrong prompts* — a pruning move [25]."
+**What the source says (arXiv:2504.11343 abstract):** "a simple rejection sampling baseline, RAFT, which trains only on positively rewarded samples, yields competitive performance than GRPO and PPO. Our ablation studies reveal that GRPO's main advantage arises from discarding prompts with entirely incorrect responses, rather than from its reward normalization."
+**Verdict:** The report's characterization is verbatim-accurate. No issue.
+---
+### Finding 4.2 — Negative Gradient / LLD [2505.18830]: Characterization is accurate [OK]
+**What the report says:** "the 'squeezing'/lazy-likelihood-displacement pathology, where the likelihood of *correct* responses barely rises or even drops under blanket per-token penalties [26]."
+**What the source says (arXiv:2505.18830 abstract):** "we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training [...] identifying the source of LLD as the naive penalization of all tokens in incorrect responses with the same strength."
+**Verdict:** Accurately characterized. No issue.
+---
+### Finding 4.3 — Near-Miss Negatives [2503.14391]: Characterization is accurate but domain is MCQA [OK with caveat]
+**What the report says:** "positives-only training structurally cannot decrease the likelihood of plausible-but-wrong near-misses [27]."
+**What the source says (arXiv:2503.14391 abstract):** "while training with positive examples fails to significantly decrease the likelihood of plausible but incorrect answers, training with negative examples more accurately identifies them."
+The experimental setting is multiple-choice QA benchmarks (MCQA), not SWE. The report acknowledges this in §7: "the near-miss-calibration result [27] is MCQA." In §4 itself, this caveat is absent.
+**Verdict:** Accurately quoted, domain gap present but not flagged in §4.
+---
+### Finding 4.4 — CWM "trains on all" re-enters §4 without correcting §2 [HIGH]
+**What the report says (§4, lines 82-83):**
+> "World-model next-state target — the single best foresight lever [...]; no policy penalty at all (§2) [13][27]."
+> "The head is therefore not a bolt-on; it is the mechanism that makes train-on-all *safe* for the policy, because it relocates failed-branch signal off the policy gradient."
+This invokes CWM [13] again as the model for "aux next-state head trains on all." As established in Finding 2.1, CWM's "train on all" is in the mid-training stage on a model that is NOT simultaneously receiving policy-gradient updates. CWM's design does not demonstrate that a simultaneously-trained aux head receiving failed-branch signal is safe for the policy.
+**Verdict:** The §4 argument inherits the §2 misread. The "trains on all" CWM citation does not support the aux-head-on-policy configuration. This is the report's most consequential misread because it is the load-bearing justification for the "failed branch → world model head (safe)" two-harvest design.
+---
+## Missing Evidence / What the Sources Do NOT Say
+### Missing 1: No paper tests next-state-prediction as aux loss on a policy network for SWE
+The entire cluster (MuZero, Dreamer, CWM, Chain-of-World, 2512.18832) provides zero direct evidence for the proposed configuration: an auxiliary next-state-prediction objective appended to a policy network (as a separate head/adapter) during RL on software engineering tasks.
+- MuZero: game-playing RL with a separate planning model
+- DreamerV3: latent world model where policy is trained *inside* the world model's imagined rollouts — structurally opposite to "add aux head to existing policy"
+- CWM: dedicated mid-training stage, not aux head during RL
+- Chain-of-World: robotics, not SWE
+- 2512.18832 ("From Word to World"): tests prompting and SFT for next-state prediction, not RL training with aux head
+**The evidentiary gap is total for the specific proposed configuration.** The report should acknowledge: "There is no published ablation of an aux next-state loss on a policy LLM during code RL. CWM is the existence proof for mid-training dynamics; MuZero/Dreamer motivate the value-equivalent latent target. The specific aux-head-during-RL design is ours to test."
+---
+### Missing 2: The DART paper's scope is narrow
+DART (2602.00994) is on retrieval-augmented QA and NL2SQL — not SWE, not multi-step agent tasks with long-horizon tool use. The interference result is between two capabilities (reasoning vs tool-use) in a shared LoRA. Applying it to "next-state-prediction head vs policy head" is another analogical transfer that the source does not make.
+---
+### Missing 3: The Myopic Planning paper (2605.06840) is not verified in the vault with full content
+The vault note for 2605.06840 only has the abstract. The report cites specific causal pruning findings. The full paper is not in the vault as fetched content. The abstract confirms the causal CoT-pruning direction is described, but the specific intervention details ("causal CoT-pruning intervention confirms move selection is driven by shallow depth-1 nodes") are derived from the paper's full text, which was not independently verified from the source. This should be checked directly.
+---
+## What the Literature DOES Support (for balance)
+1. **Explicit dynamics training (mid-training or SFT) beats zero-shot prompting**: 2512.18832 demonstrates SFT on trajectories lifts ALFWorld/SciWorld accuracy to 99%/98%. CWM demonstrates mid-training on dynamics produces a strong SWE base. These are real, direct endorsements of *some form* of dynamics training.
+2. **Train-on-all for world model, filter for RL reward**: CWM explicitly does this, and the motivating argument ("comprehensive world model") is stated in the paper. The design principle is supported, just not via aux head during RL.
+3. **Value-equivalent / decision-relevant targets**: MuZero's design principle — predict only what matters (reward, value, policy) — is well-established RL theory. Its application to code (predict FAIL_TO_PASS delta, not full repo state) is a sound architectural translation.
+4. **Structured negatives fix near-miss calibration positives-only cannot**: 2503.14391 supports this in MCQA. The SWE transfer is an inference.
+5. **Foresight governance bottleneck**: 2601.03905 clearly identifies this bottleneck in VLM/VQA. The principle is general.
+6. **Simultaneous gradient interference**: 2602.00994 is a real empirical finding in agentic RL. The prescriptive consequence (use parameter isolation) is supported.
+---
+## Verdict on Report Commitments (§2–4)
+| Commitment in final report | Source support | Verdict |
+|---|---|---|
+| CWM "trains on all" for world-model head | CWM trains on all in mid-training stage, not during RL aux-head | **MISREAD — cite correctly as mid-training** |
+| CWM reaches "65.8% SWE-bench Verified" | Correct but with test-time scaling; base score is lower | **INCOMPLETE — add qualifier** |
+| Chain-of-World supports "predict terminal state, not full frame" for code | CoWVLA is robotics VLA, not SWE | **WRONG DOMAIN — remove or note as robotics analogy** |
+| Predictive-Causal Gap proves SWE repo is dangerous for aux loss | Theorem is linear-Gaussian; SWE application is author inference | **OVERSTATED — demote to risk flag** |
+| Parameter isolation eliminates interference risk | DART shows isolation reduces, not eliminates | **OVERCLAIM — soften to "substantially reduces"** |
+| Foresight@k is the kill ablation | Metric is proposed by report, not a standard metric | **MARK AS PROPOSED METRIC** |
+| Aux loss as "second SDPO mode" has evidentiary support | No source tests this configuration | **ZERO DIRECT EVIDENCE — flag as null-evidence design proposal** |
+| "Two hard prune gates resolve the central question" | Gates don't address predictive-causal gap | **INTERNAL INCONSISTENCY — the theorem is invoked then resolved by a design that doesn't address it** |
+| MuZero/Dreamer provide "design discipline" for SWE aux head | They motivate value-equivalent targets; no LLM-SWE experiments | **OVERCLAIM — demote to "analogical motivation"** |
+---
+## Recommended Corrections
+1. **§2, CWM citation**: Rewrite as: "Meta's CWM mid-trains a 32B model in a dedicated pre-RL stage on 3M observation-action trajectories without success filtering, reaching 65.8% on SWE-bench Verified with test-time scaling. The train-on-all decision is for the mid-training dynamics stage, not an auxiliary head during RL." Remove the implication that CWM licenses aux-head-on-policy train-on-all.
+2. **§2, Chain-of-World [53]**: Flag explicitly as robotics VLA. Either remove or rewrite as: "In the robotics domain, CoWVLA (CVPR 2026) demonstrates the same latent-terminal-state design for embodied agents, providing design-level motivation for the analogous SWE architecture."
+3. **§2, Predictive-Causal Gap**: Rewrite as a risk flag: "The Predictive-Causal Gap theorem (linear-Gaussian dynamics; 2695 networks) establishes that predictive objectives can be accurate and causally blind simultaneously. Applied to SWE by analogy, a next-state head could improve token-level prediction while failing to learn decision-relevant dynamics. This is a structural risk, not a demonstrated outcome for LLM SWE training."
+4. **§2, Parameter isolation**: Change "parameter-isolated head or adapter, never fused into the policy head [16]" to "parameter-isolated head or adapter substantially reduces gradient interference (DART reduces but does not fully close the interference gap to the 2-Agent upper bound [16])."
+5. **§2, Foresight@k**: Add "(we define this metric; it has no published baseline)" at first use.
+6. **§2, null-evidence flag**: Add a box or explicit paragraph: "Direct evidence gap: no published paper has tested an auxiliary next-state-prediction objective as an add-on loss during RL on a code policy network. All cited support is analogical transfer from: (a) mid-training architectures (CWM), (b) dedicated world-model planning systems (MuZero, Dreamer), or (c) non-SWE domains (Chain-of-World, 2512.18832). The proposed aux-head-during-RL design is a research hypothesis requiring the P4 ablation in §4, not a design with established support."
+7. **§4, CWM [13] re-citation**: Correct to: "CWM's mid-training precedent motivates train-on-all dynamics learning; direct evidence for the aux-head variant of this design during RL is absent."
+8. **§3, foresight domain caveat**: Bring the "VLM/VQA" caveat from §7 into §2 at first use of 2601.03905.
+---
+## What These Sources Collectively Say About (a), (b), (c)
+**Question (a): Does next-state-prediction auxiliary head help agent policies (vs hurt via gradient interference)?**
+Direct evidence: **None in SWE or code domain.** CWM does it in mid-training, not as aux head. DART shows simultaneous parameter optimization on reasoning+tool-use interferes; parameter isolation helps substantially. 2512.18832 shows SFT (not RL with aux head) helps next-state prediction transfer. The honest position: unknown for the specific proposed configuration; plausible but untested.
+**Question (b): Does training on failure trajectories help/hurt and under what routing?**
+Direct evidence for routing: 2503.14391 (MCQA near-miss: negatives help near-miss calibration, positives-only cannot decrease plausible-wrong likelihood). 2505.18830 (raw uniform negative gradient destabilizes). 2504.11343 (positives-only RAFT is competitive on pass@1). Together: **raw negatives hurt pass@1, structured negatives fix calibration, SWE-specific ablation unrun.** CWM's mid-training: all trajectories useful for dynamics, not for policy. The "two-harvest" design (negatives to world model, not policy) is consistent with all sources but not tested in the proposed configuration.
+**Question (c): Do "deliberation tokens" / think-before-act distillation have support?**
+Partial support: CWM discusses "reasoning about environment feedback to improve agentic code generation" as future work and shows early prototype (Figure 5) of trace-conditioned reasoning. 2512.18832 shows SFT on trajectories enables explicit next-state reasoning. But "deliberation token" as a trainable gate with RL on placement is not in any source. 2605.06840 suggests CoT deliberation content is generated but causally ignored by the model — a direct challenge to the governance-RL-on-token-placement idea. 2601.03905 shows even explicit simulation access fails. The think-before-act idea has motivational support but no direct SWE ablation and one result (myopic planning) that challenges whether the token's content would be consumed.
+---
+*End of findings. Full-length file: `/Users/baladita/Documents/DevBox/composer-replication-framework/research/deepread/06-worldmodel.md`*

research/deepread/07-trace-replay-tree.md ADDED Viewed

	@@ -0,0 +1,291 @@

+# Deep-Read Critical Review: Trace-Replay / Multi-Teacher Distillation / MCTS-for-LLM-Agents
+## Cluster 07 — Channel 3 + Tree-of-Work
+**Reviewer:** agent (Claude Sonnet 4.6)
+**Date:** 2026-06-10
+**Scope:** `composer_replication/teacher_replay.py`, `research/05-trace-replay-distillation.md`, `research/notes/final_report_socratic-mcts-swe-worldmodel-8f6dea.md` §§1,3,6,7,10, `docs/adrs/ADR-002-trace-source.md`, `docs/adrs/ADR-013-lma-integration-channel-ladder.md`
+---
+## 0. TL;DR
+Channel 3 (`teacher_replay.py`) and the tree-of-work design are **neither pure novelty nor pure reinvention**. The specific combination — frozen-trace replay at every decision step across N heterogeneous frontier models, consensus-disagreement-as-DPO-signal, divergence-gated recursive branching with an execution oracle, and world-model auxiliary loss — is genuinely novel synthesis. But the repo's research note (`research/05`) **overclaims** novelty in three specific ways, the final report's cost estimates are **internally inconsistent**, the "closest precedent" identification (rStar) is partially wrong, and there is a **missing piece** (agent-trace distillation at code-repo scale already exists in published form at direct overlap) that neither `research/05` nor the final report adequately cites. This review quotes sources verbatim, identifies every misread, and gives a verdict on each cost figure.
+---
+## 1. What Was Fetched / Read
+Primary sources fetched and read:
+- **LATS** (arXiv:2310.04406, Zhou et al. 2023): Language Agent Tree Search — MCTS over agent reasoning/acting/planning steps with external env feedback and LM-valued nodes.
+- **rStar** (arXiv:2408.06195, Qi et al. 2024): mutual self-play two-SLM MCTS for math reasoning; generator+discriminator role split.
+- **rStar-Math** (arXiv:2501.04519, Guan et al. 2025): MCTS-driven step-verified data synthesis for math SLMs; trains PPM (process preference model) from tree rollouts.
+- **Tree-of-Thoughts** (arXiv:2305.10601, Yao et al. 2023): foundational ToT framework, BFS/DFS over thought trees, LM self-evaluation.
+- **SWE-Search** (arXiv:2410.20285, Antoniades et al. 2024): MCTS over SWE-bench tasks; hybrid LM value function, Value Agent, Discriminator Agent; 23% relative gain over no-MCTS; scales with inference-time compute.
+- **Tree-GRPO** (arXiv:2509.21240, Ji et al. 2025/ICLR 2026): tree-structured group-relative policy optimization; shared-prefix branching; proves intra-tree group-relative advantage == step-level DPO.
+- **Socratic-SWE** (arXiv:2606.07412, Xiao et al. 2026): closed-loop self-evolving SWE agent; trace-derived Agent Skill Registry; 4-gate verifier; solver-gradient alignment reward; 50.40% SWE-bench Verified after 3 iters.
+- **Symphony** (arXiv:2601.22623, Zhu et al. 2026/NeurIPS 2025): heterogeneous LM pool in MCTS planning; argues single-agent MCTS has insufficient branch diversity; heterogeneous pool improves rollout diversity.
+- **SWE-Gym** (arXiv:2412.21139, Pan et al. 2024/ICML 2025): 2,438 real-world Python task environments; inference-time scaling via trained verifiers; 32.0%/26.0% on SWE-bench Verified/Lite.
+- **Code World Model MCTS** (arXiv:2405.15383, Dainese et al. 2024): MCTS-guided code world model generation; GIF-MCTS; NeurIPS 2024.
+- **Tree Search for LM Agents** (arXiv:2407.01476, Koh et al. 2024): best-first tree search for web agents; 39.7% relative gain on VisualWebArena with GPT-4o; effective at realistic tasks.
+Repo files read in full:
+- `composer_replication/teacher_replay.py` (281 LOC)
+- `research/05-trace-replay-distillation.md`
+- `research/notes/final_report_socratic-mcts-swe-worldmodel-8f6dea.md` (§§1–10, full)
+- `docs/adrs/ADR-002-trace-source.md`
+- `docs/adrs/ADR-013-lma-integration-channel-ladder.md`
+---
+## 2. What the Literature Actually Says
+### 2a. Replaying a trace prefix across N heterogeneous models then branching — has anyone done it?
+**Short answer: The exact frozen-trace multi-teacher replay mechanism is not directly published. But the components are deeply covered and combinatorially close work exists.**
+The relevant lineage:
+1. **Tree-of-Thoughts / LATS / SWE-Search** all expand nodes via a policy (one model, sometimes self-evaluated) and do NOT fix a human-collected trace then replay each step with multiple heterogeneous external models. The branching is forward-generative, not retrospective-replay.
+2. **rStar** (the closest analogue cited in `research/05`) uses two SLMs in fixed roles (generator + discriminator) at trajectory-level evaluation. The discriminator evaluates the full trajectory, not each step position independently. Quote from the abstract: "another SLM, with capabilities similar to the target SLM, acts as a **discriminator to verify each trajectory** generated by the target SLM." This is trajectory-level verdict, not per-step replay with N teachers.
+3. **rStar-Math** does per-step MCTS rollouts, but these are single-policy forward rollouts (not fixed-trace, not multi-teacher). The PPM then scores steps. No external teacher heterogeneity.
+4. **SWE-Gym** trains verifiers on agent trajectories and scales inference via them, but the training data collection does not use multi-teacher per-step replay.
+5. **AgentTrek** (cited in `research/05`) uses "guided replay demonstrations" from web tutorials — semantically similar name but operationally different: it is single-teacher demonstration collection, not multi-teacher disagreement harvesting.
+**Verdict on (a):** Nobody has published the exact frozen-trace-prefix → N-teacher parallel replay → disagreement-as-DPO mechanism at the agentic-step level. The combination is genuinely novel. However, `research/05`'s claim that "No published work systematically freezes a trace at step t, replays that exact state with N different teachers, harvests variance as per-step supervision" is **accurate** — but `research/05` then goes on to undercite the work that comes closest, which is more than rStar.
+### 2b. Execution-oracle-graded tree search over real repos at scale — what does the literature actually say and what does it cost?
+**Published execution-oracle tree search at repo scale:**
+- **SWE-Search** (arXiv:2410.20285): MCTS over SWE-bench using a hybrid value function (LLM-estimated numerical + qualitative). **Critical nuance:** the value function is LLM-estimated, NOT a true execution oracle. It uses a "Value Agent" that provides qualitative feedback. Full pytest runs are expensive; SWE-Search's value is LM-evaluated fitness, not test-suite-pass-fraction. The 23% relative gain is impressive but it is LM-valued MCTS, not execution-oracle MCTS.
+- **SWE-Gym verifiers** (arXiv:2412.21139): trains a verifier (a model that learns to score trajectory quality) on trajectories, then uses it for inference-time selection. The verifier *learns* execution outcome but is not the oracle itself during search.
+- **Socratic-SWE** (arXiv:2606.07412): uses a 4-gate verifier — Format/Grounding/Execution/Semantics — for task validation, NOT as the tree search value function. The training reward is `Valid(τ)·cos(g_τ, G_val)` — solver-gradient alignment, NOT test-suite pass-fraction. The execution gate is a solvability filter, not a step-level oracle.
+**Only the repo's own `FeatureDeletionEnv._grade()` (and CWM arXiv:2510.02387, which trains a model on execution trajectories) use raw test-suite pass-fraction as the MCTS/RL reward at training time.** The final report correctly identifies this as the critical differentiator.
+**Cost data for execution-oracle tree search:**
+The literature does NOT provide a clean "$X per trace" figure for execution-oracle MCTS at real repo scale. The closest:
+- DeepSWE (cited as [43] in the final report) ran 64 H100s for six days on 0/1 sparse reward outcome-RL with no branching tree — making the per-trace cost of a *flat* outcome-RL run tractable but providing no per-branch cost.
+- SWE-rebench infrastructure (nebius.com, [54]) evaluates thousands of SWE instances/hour via distributed container orchestration — providing a throughput floor but not a per-branch dollar figure.
+### 2c. Cross-model disagreement as a branch gate — is it a useful signal?
+**Tree-GRPO** (arXiv:2509.21240, ICLR 2026) provides the strongest theoretical backing: it proves that intra-tree group-relative advantage estimation is *equivalent* to step-level direct preference learning. This directly validates the repo's claim that "sibling divergence → DPO signal" is principled. Quote from abstract: "we demonstrate that the objective of intra-tree level group relative policy optimization is **equivalent to that of step-level direct preference learning**."
+However, this is about **same-model** intra-tree branching (shared prefixes, different continuations), not heterogeneous-model branching. The equivalence proof does not cover cross-family disagreement.
+**Symphony** (arXiv:2601.22623, NeurIPS 2025) argues explicitly that "single-agent MCTS yields insufficient branch diversity and a heterogeneous LM pool improves rollout diversity and exploration." This endorses cross-model disagreement as a diversity mechanism — but note: Symphony tests this in planning tasks, not repo-level SWE.
+**Counterpoint:** arXiv:2604.02460 (Single-Agent LLMs Outperform Multi-Agent, Tran & Kiela 2026) makes the data-processing inequality argument that under equal compute, single-agent is information-theoretically superior. The SWE-specific result is unrun (as the final report correctly acknowledges).
+### 2d. DPO-pair extraction from sibling branches
+Tree-GRPO (arXiv:2509.21240) provides the clearest theoretical backing for this being well-founded. The repo's `extract_dpo_pairs` implements a simpler heuristic (Counter over normalized actions, break-on-first-pair per state) — this is a sound but weaker version of what Tree-GRPO formalizes. The repo's implementation does not implement the intra-tree advantage weighting Tree-GRPO derives.
+---
+## 3. Critical Review of `research/05-trace-replay-distillation.md`
+### Finding 05-R1: Overclaim on rStar as "closest precedent" (MISREAD)
+`research/05` Section "The Closest Published Precedent" frames rStar as the closest precedent and describes it as having "multi-model step-level evaluation." This is partially wrong:
+- rStar's discriminator evaluates **full trajectories**, not individual steps. Quote from rStar abstract: "acts as a discriminator to **verify each trajectory**." The per-step MCTS action expansion in rStar uses the *generator* SLM alone; the discriminator only votes on completed trajectories.
+- The correct characterization: rStar is close because it uses two models with agreement as signal, but the granularity is trajectory-level, not step-level.
+**More accurate closest precedent:** Tree-GRPO (arXiv:2509.21240) is closer to the DPO-extraction mechanism. `research/05` does not cite Tree-GRPO at all — this paper was published September 2025 and is directly relevant.
+### Finding 05-R2: Missing citation — Tree-GRPO (MISS)
+`research/05` does not cite Tree-GRPO (arXiv:2509.21240), which:
+- Proves intra-tree group-relative advantage == step-level DPO (exactly what `extract_dpo_pairs` is trying to compute informally).
+- Shows that shared-prefix branching is a principled way to generate process supervision from outcome reward alone.
+- Is published at ICLR 2026, directly relevant.
+`research/05` was presumably written before Tree-GRPO's acceptance was known, but this is now a material gap. The final report (§7, footnote [44]) cites Tree-GRPO correctly — so this was caught downstream but never propagated back to `research/05`.
+### Finding 05-R3: Missing citation — SWE-Search (MISS)
+`research/05` does not cite SWE-Search (arXiv:2410.20285), which applies MCTS specifically to SWE-bench tasks (the exact domain), achieves 23% relative improvement, and demonstrates that the value signal scales with inference-time compute. This is the most directly relevant published tree-search SWE result. The final report cites it as [51] — again, a gap that exists only in `research/05`.
+### Finding 05-R4: AgentTrek description is misleading (MISREAD)
+`research/05` describes AgentTrek as "Large-scale multimodal trajectory dataset from web tutorials. Guided replay demonstrations." The characterization "Guided replay demonstrations" is misleading — it implies similarity to multi-teacher replay. AgentTrek generates demonstrations by following web tutorial steps with a single agent; there is no replay of a frozen trace across multiple models. The "connection" note ("Demonstrates feasibility of guided/counterfactual replay") overstates the analogy.
+### Finding 05-R5: "Trace-Freezing + Multi-Teacher Replay: Novel" claim is directionally right but mis-attributed (PARTIAL OVERCLAIM)
+Section "What IS Novel" states: "No published work systematically freezes a trace at step t, replays that exact state with N different teachers, harvests variance as per-step supervision." This specific combination IS genuinely novel. But the claim "Step-Level Multi-Teacher Preference Data: Traditional multi-teacher: response-level preferences; PRMs: single-teacher step evaluation; Gap: No multi-teacher per-step comparison" understates what Tree-GRPO achieves: Tree-GRPO generates multiple continuations from a shared prefix (a frozen state) and extracts step-level preference signal — which is mechanically very close to what Channel 3 does (though with a single model rather than heterogeneous teachers).
+### Finding 05-R6: Cost estimates are internally consistent in `research/05` but inconsistent with the final report (FLAG)
+`research/05` states the "Baseline" cost as:
+```
+$0.008/step × 1000 × 8 = $64 per trace
+```
+This is "8 teachers × 1000 steps" ungated. The final report's footnote [6] says "~$0.98 flat vs ~$64 ungated" — the $0.98 figure comes from Spike 001's actual measurement on 50 synthetic states with 3 teachers (reported in `teacher_replay.py` docstring: "Verified economic floor (✅ spike 001): $0.98 mean per-trace cost ungated, $0.30/trace projected with VOI gating.").
+**The inconsistency:** `research/05` constructs the $64 figure from 8 teachers × 1000 steps, while `teacher_replay.py` uses DEFAULT_TEACHERS with only 3 teachers. The final report uses $0.98 (3 teachers, real trace) and $64 (8 teachers, 1000 steps) in the same sentence, implying $64 is the cost of "the tree" not "flat Channel 3 with 8 teachers." This is confusing:
+- $0.98 per trace = real Spike 001 measurement, N=3 teachers, ~50 synthetic states
+- $64 per trace = constructed estimate from research/05, N=8 teachers, 1000 steps, flat (no branching)
+- A true branching tree at depth D with N branches is O(N^D) which is *worse* than $64
+The final report says "a true branching tree is O(N^D), strictly worse than either flat figure" (§10), which is correct — but the $64 figure is NOT a "branching tree" cost, it is a flat all-steps N=8 estimate. Calling it "ungated tree" in the final report (§10: "~$64 ungated tree") is a misnomer. It should be "$64 flat 8-teacher 1000-step, vs O(N^D) for a true tree."
+---
+## 4. Critical Review of the Final Report (§§1,3,6,7,10 of `final_report_socratic-mcts-swe-worldmodel-8f6dea.md`)
+### Finding FR-R1: The "flat (depth-1)" characterization of Channel 3 is accurate (CORRECT)
+§1: "It is flat (depth-1): the teachers never take their candidate action or continue, so the 'tree' is a set of depth-1 stars hung off a pre-existing linear human trace." This is exactly right — confirmed by reading `replay_trace()` (lines 178-188) and `extract_dpo_pairs()` (lines 206-262).
+### Finding FR-R2: "Fitness is teacher plurality, not execution" characterization is accurate (CORRECT)
+§1: "its fitness is teacher plurality, not execution: selection is a Counter over normalized actions, nothing runs against a test suite." Confirmed by reading `_normalize_action()` and `extract_dpo_pairs()` — the pairing logic is purely textual counter-based with whitespace normalization. There is no execution oracle in Channel 3's current form.
+**Critical gap:** The normalization in `_normalize_action()` (lines 196-203) is very weak — it only normalizes whitespace and lowercases. For real agentic traces with structured tool calls, this will produce near-zero agreement (two tool calls that are semantically identical but differently formatted will count as disagreements). The docstring acknowledges this: "For real agentic traces, this should parse the tool call (name + args) and return a canonical form." This is a **known unimplemented critical path** for the channel to produce any real signal.
+### Finding FR-R3: The Socratic-SWE description has a materially wrong claim (MISREAD, HIGH SEVERITY)
+§1 describes Socratic-SWE: "Socratic-SWE (2606.07412) is the closest published analogue — a closed-loop self-evolving SWE trainer that distills traces into skills, generates targeted repair tasks, gates them through a four-stage execution validator, scores by solver-gradient alignment, and reaches 50.40% on SWE-bench Verified after three iterations. But it generates repair tasks; it does NOT inject bugs."
+This is accurate as far as it goes. However, the final report then says: "Bug injection — manufacturing a broken state by reverting a gold patch and rewarding re-derivation — is the repo's own FeatureDeletionEnv, the analogue of Cursor's 'feature deletion'."
+The implicit contrast — "they do repair tasks, we do feature deletion / bug injection" — is valid. But the report slightly overstates the contrast by not noting that Socratic-SWE's task generation pipeline is *also* execution-validated (Format/Grounding/Execution/Semantics gate) and also targets agent-specific weaknesses (via skill distillation). The gap is real (no bug injection, no per-step branching) but smaller than the report implies.
+**Bigger issue:** The claim "it does NOT inject bugs" deserves a citation to the Socratic-SWE paper to confirm this is an architectural choice and not an oversight. Quote from the Socratic-SWE abstract: "Existing synthetic data methods **typically create tasks through fixed mutation or bug-injection** procedures, making the resulting distributions largely independent of the agent's own weaknesses and training progress. We introduce Socratic-SWE, a closed-loop self-evolution framework that **reuses the agent's historical solving traces** as a source of training signal." So Socratic-SWE explicitly positions against bug injection. The final report correctly notes this but doesn't quote it, which weakens the citation.
+### Finding FR-R4: "heterogeneous-per-turn branching + execution-oracle fitness + divergence-derived textual feedback + world-model aux loss — is the novel synthesis" claim is correctly hedged (CORRECT)
+§1: "The multi-model per-turn tree is a recombination whose ingredients all exist (SWE-Search expands nodes with one policy [51]; Symphony does heterogeneous-LM planning [52]; Channel 3 queries N teachers, flat) but whose specific combination [...] is the novel synthesis. Claim the synthesis, not the parts." This is well-calibrated. The component-level citations are verified correct.
+**One gap:** The paper arXiv:2407.01476 ("Tree Search for Language Model Agents," Koh et al. 2024) is in the vault and applies best-first tree search to realistic web agent tasks. This is another component-level precedent that should be in the synthesis inventory but is not cited in §1 or §7.
+### Finding FR-R5: The cost estimate claim "$0.98 mean per-trace cost ungated" needs qualification (POTENTIAL OVERCLAIM)
+The $0.98 figure appears both in `teacher_replay.py`'s docstring ("Verified economic floor (✅ spike 001): $0.98 mean per-trace cost ungated") and in the final report (§10, [6]).
+However, Spike 001 used 50 **hand-crafted synthetic states** (ADR-002: "Spike 001 used 50 hand-crafted synthetic states for the cost-floor measurement"). Real Claude Code sessions have 125–2,830 tool_use messages per session (ADR-002). If a "trace" is a full session with ~1,400 states (median), the actual cost of flat N=3 replay would be roughly:
+```
+1,400 steps × 3 teachers × (prompt_tokens × cost_in + completion_tokens × cost_out)
+```
+The DEFAULT_TEACHERS pricing in `teacher_replay.py` at 200 max_tokens:
+- claude-opus-4.7: $15/M in + $75/M out → at 2k prompt tokens + 200 completion: ~$0.045/call
+- gpt-5: $1.25/M in + $10/M out → ~$0.004/call
+- deepseek-v4-pro: $1.10/M in + $4.40/M out → ~$0.003/call
+Total per-step: ~$0.052
+For 1,400 steps: **~$73 per session** (not $0.98).
+The $0.98 figure was correct for 50 synthetic states with short messages. At real session scale, the cost is one to two orders of magnitude higher. The docstring's "Verified economic floor" language suggests the $0.98 is the per-trace figure for the spike's trace definition (50 steps), not per full-session. The final report's §10 is careful about this distinction but `teacher_replay.py`'s docstring is misleading: it should say "per 50-step trace (spike 001 benchmark)," not just "per-trace."
+### Finding FR-R6: The $64 "ungated tree" terminology is a misnomer (FLAG, see 05-R6 above)
+§10: "flat Channel-3 replay is ~$0.98/trace at N=3 and ~$64/trace at the eight-teacher × thousand-step scale, both flat O(N·T); a true branching tree is O(N^D), strictly worse than either flat figure."
+The statement is mathematically correct but the label "~$64 ungated tree" (used in the final report's in-text references) is misleading because $64 is a flat (non-branching) extrapolation from `research/05`. It is not a measured figure, and the "eight teachers" scenario (N=8) does not correspond to any actual configuration in the codebase (DEFAULT_TEACHERS has N=3). The label should be "$64 flat-replay extrapolation at N=8, T=1000, unmeasured."
+### Finding FR-R7: The heterogeneity pushback is correctly represented (CORRECT)
+§7 cites arXiv:2604.02460 (Tran & Kiela 2026) and arXiv:2601.12307 (Rethinking multi-agent) correctly. Both papers are in the vault with correct titles/years. The data-processing-inequality framing for single-agent efficiency is correctly attributed.
+**One gap:** The final report does not note that arXiv:2604.02460 studies multi-hop *reasoning* tasks (QA), not SWE tasks. The domain transfer is uncertain. This is the kind of caveat that belongs in §7's pushback discussion.
+### Finding FR-R8: The _normalize_action issue is a silent failure mode not surfaced in the final report (MISS)
+The final report discusses the $0.98→$64 cost gap and the O(N^D) blowup, but does not flag that `extract_dpo_pairs` will produce near-zero pairs on real agentic traces because `_normalize_action` only normalizes whitespace. On real Claude Code traces with structured `tool_use` blocks (JSON-formatted), two semantically identical "run bash command X" steps formatted slightly differently will count as disagreements, and two structurally different calls with identical normalized strings will count as agreements. The pair-extraction logic is likely to produce mostly noise on real traces without a proper tool-call parser.
+This is acknowledged in the code comment ("For real agentic traces, this should parse the tool call (name + args) and return a canonical form") but is not surfaced as a risk in the final report's §10 failure modes section.
+### Finding FR-R9: The world-model aux loss discussion correctly cites the key interference and causal-gap papers (CORRECT)
+§2 cites arXiv:2602.00994 (Reasoning and Tool-use Compete in Agentic RL) and arXiv:2605.05029 (The Predictive-Causal Gap) correctly. Both are in the vault with correct titles. The conclusion — parameter-isolated head, ablation-gated — is appropriate given the evidence.
+### Finding FR-R10: The "strip_thinking=False" critical path claim is not verified by any cited paper (UNVERIFIED)
+§6: "strip_thinking must be False: ~67% of real Claude Code error-recovery turns are pure thinking." The 67% figure appears to come from internal repo analysis (ADR-002 / claude_code.py) rather than any external source. This is likely accurate (it's an empirical measurement from real sessions) but should be marked as an internal finding, not stated as if it has external validation.
+---
+## 5. Is the Channel-3/Tree Design Novel or Reinventing Something That Exists?
+### Novel (good):
+1. **Frozen-trace per-step replay across N heterogeneous frontier models with consensus-DPO harvesting**: No published paper does exactly this. The closest are Tree-GRPO (shared-prefix same-model branching) and rStar (two-model trajectory verification). The heterogeneous-teacher-consensus mechanism applied to a frozen captured trace is original.
+2. **Combining execution-oracle fitness with multi-model disagreement for divergence-gated branching**: SWE-Search uses LM-estimated value (not execution oracle). Socratic-SWE uses execution validation as a filter, not as the MCTS fitness. Using test-suite pass-fraction as the tree's value function is the most meaningful differentiator.
+3. **Two-loop architecture (outer MCTS/datagen + inner RL trainer) with the world-model head as the bridge**: The specific coupling where failed branches become world-model training targets (not policy gradient targets) is a design synthesis not found in any single paper.
+### Reinventing (needs citation):
+1. **MCTS over real code repositories**: SWE-Search (arXiv:2410.20285) does exactly this and is 18 months old. The repo cites it correctly as [51] in the final report but `research/05` doesn't mention it. The "23% relative improvement" from SWE-Search should inform the expected gain from adding search (whether at training or inference time).
+2. **Tree-structured DPO signal from shared prefix branches**: Tree-GRPO (arXiv:2509.21240) formalizes exactly this. The repo's `extract_dpo_pairs` is a weaker/informal version. The repo should acknowledge that it is implementing a version of what Tree-GRPO proves, and should consider whether the formal Tree-GRPO objective would be stronger than the current Counter-based heuristic.
+3. **Per-step process supervision from MCTS**: rStar-Math (arXiv:2501.04519) and Math-Shepherd do this for math. The adaptation to code/SWE is novel, but the core mechanism is not. `research/05` does acknowledge this in the PRM section, but understates how directly rStar-Math applies (it performs MCTS rollouts to generate step-verified training data, which is structurally similar to what the tree-of-work aims to do).
+4. **World-model head trained on agent trajectories**: CWM (arXiv:2510.02387) trains a 32B model on observation-action trajectories for next-state prediction and achieves 65.8% on SWE-bench Verified. This is directly cited in the final report as [13] and correctly described. No reinvention issue.
+---
+## 6. Cost Estimate Plausibility
+| Estimate | Source | Plausibility | Issues |
+|---|---|---|---|
+| **$0.98/trace ungated** | Spike 001 measurement, 50 synthetic states, N=3 | Accurate *for 50-step traces*. Misleading at full-session scale (~1400 steps → ~$73 with same model prices). | `teacher_replay.py` docstring should specify the scope. |
+| **$64/trace flat** | `research/05` constructed estimate, N=8 teachers, T=1000 steps | Plausible *as an extrapolation* for a specific 8-teacher 1000-step scenario. Not measured, not a codebase config. | Calling it "ungated tree" is a misnomer. DEFAULT_TEACHERS has N=3, not N=8. |
+| **O(N·decision-points) with divergence gating** | Final report §3, citing research/05 | Plausible as a *theoretical claim* if gating is very aggressive (VOI/entropy > τ fires only at a small fraction of steps). The 60-80% reduction estimate from "PRM literature" is from `research/05`'s analysis, not a specific paper citation. | No empirical measurement of what fraction of real agentic trace steps are "high-uncertainty." The claim needs a specific PRM citation. |
+| **$0.30/trace with VOI gating** | `teacher_replay.py` docstring, "projected" | Speculative projection from $0.98 × ~0.3 reduction. Not measured. | The 70% reduction rate should cite a specific paper or be labeled "estimated." |
+**Overall cost plausibility verdict:** The $0.98 and $64 figures are internally consistent as upper/lower bounds for specific parameterizations, but neither reflects the actual cost of running Channel 3 on real full-session Claude Code traces (which would be far higher). The divergence-gating reduction estimate (60-80%) is reasonable but unverified for agentic traces specifically. The final report correctly identifies the O(N^D) combinatorial blowup as the dominant cost concern for a true branching tree — this framing is sound.
+---
+## 7. Summary of Findings
+| ID | Severity | Type | Location | Finding |
+|---|---|---|---|---|
+| 05-R1 | Medium | MISREAD | research/05 | rStar is trajectory-level verdict, not per-step replay — overclaims closeness |
+| 05-R2 | High | MISS | research/05 | Tree-GRPO (arXiv:2509.21240) not cited; directly formalizes the DPO-from-branching mechanism |
+| 05-R3 | High | MISS | research/05 | SWE-Search (arXiv:2410.20285) not cited; most direct execution-oracle-adjacent MCTS-on-SWE precedent |
+| 05-R4 | Low | MISREAD | research/05 | AgentTrek "guided/counterfactual replay" connection overstated |
+| 05-R5 | Low | PARTIAL OVERCLAIM | research/05 | Tree-GRPO's shared-prefix branching is closer to the step-level preference extraction than rStar |
+| 05-R6 | Medium | INCONSISTENCY | research/05 + teacher_replay.py | $64 "8-teacher flat" extrapolation mislabeled as "ungated tree"; N=8 not a codebase config |
+| FR-R1 | — | CORRECT | final_report §1 | Depth-1 / flat characterization of Channel 3 is accurate |
+| FR-R2 | — | CORRECT | final_report §1 | Teacher-plurality vs execution-oracle distinction is accurate |
+| FR-R3 | Medium | MISREAD (mild) | final_report §1 | Socratic-SWE contrast slightly overstated; execution-gating detail understated |
+| FR-R4 | — | CORRECT | final_report §1 | Novel synthesis vs ingredient-level precedents is well-calibrated |
+| FR-R5 | High | POTENTIAL OVERCLAIM | final_report §10 + teacher_replay.py | $0.98/trace docstring applies to 50-step synthetic traces; at real full-session scale (~1400 steps), cost would be ~$73/session |
+| FR-R6 | Medium | FLAG | final_report §10 | "$64 ungated tree" label is misleading; it is a flat extrapolation with N=8, T=1000 |
+| FR-R7 | — | CORRECT | final_report §7 | Heterogeneity pushback cited correctly |
+| FR-R8 | High | MISS | final_report §10 | `_normalize_action` whitespace-only normalization is a critical silent failure path on real agentic traces; not in §10 failure modes |
+| FR-R9 | — | CORRECT | final_report §2 | World-model interference and causal-gap papers cited correctly |
+| FR-R10 | Low | UNVERIFIED | final_report §6 | 67% of error-recovery turns are pure thinking: internal empirical claim, should not be stated as externally validated |
+---
+## 8. Actionable Recommendations
+1. **Fix `_normalize_action` before any real-trace replay**: The whitespace-only normalization is the most likely silent failure path. Implement tool-call parsing that canonicalizes (tool_name, args) tuples. This is acknowledged in the code comment but not in any ADR or the final report's failure mode list. **Add to ADR-002 consequences or open a new ADR.**
+2. **Correct `teacher_replay.py` docstring**: Change "Verified economic floor (✅ spike 001): $0.98 mean per-trace cost ungated" to "Verified economic floor for 50-step synthetic-state benchmark traces (Spike 001): $0.98 mean per-trace. Full Claude Code sessions (~1,400 tool-use steps) would cost ~$70–80 flat ungated at DEFAULT_TEACHERS pricing."
+3. **Cite Tree-GRPO in `research/05`** and consider adopting its formal objective in `extract_dpo_pairs` rather than the Counter heuristic. The Tree-GRPO advantage estimator (intra-tree group-relative) is a stronger principled basis for the DPO pair extraction.
+4. **Add SWE-Search to `research/05`** and note that its 23% relative gain represents the expected floor for adding any tree search to a SWE agent. The tree-of-work must beat this at equal compute on a per-training-dollar basis, not just at inference time.
+5. **Standardize the $64 figure's label**: In the final report's in-text citations and in `research/05`, consistently label it "flat N=8 T=1000 extrapolation (not measured, not a codebase config)" rather than "ungated tree."
+6. **Add the `_normalize_action` failure mode to §10's failure modes list** in any future revision of the final report.
+7. **The tree-of-work core claim stands**: The combination of execution-oracle fitness + frozen-trace multi-teacher replay + divergence-gated recursive branching + typed signal routing (world-model head for failed branches, contrastive DPO for near-misses) is a genuine novel synthesis. The final report's pre-registered ablation design (P0–P6 axis + heterogeneity control) is the right scientific vehicle — do not abandon it based on the misreads identified here, which are correction-level, not foundation-level problems.

research/deepread/08-rl-infra.md ADDED Viewed

	@@ -0,0 +1,292 @@

+# Deep-Read: RL Infra & Frameworks — Critical Findings
+**Cluster 8 of the dataset-pipeline review series**
+**Reviewer:** automated critical pipeline, 2026-06-09
+**Primary sources fetched:** TRL GRPOTrainer live docs (v1.5.1, huggingface.co/docs/trl/en/grpo_trainer), vLLM co-locate blog (June 2025, huggingface.co/blog/vllm-colocate), verl GitHub README (main branch, verl-project/verl — 21.9k stars), SWE-MiniSandbox paper (arXiv:2602.11210v5), secure EKS sandboxes article (AWS Builder Center). Repo files inspected: `research/04-verl-trl.md`, `research/03-monarch-torchforge-openenv.md`, `docs/adrs/ADR-006-rl-frameworks.md`, `docs/adrs/ADR-008-drgrpo-sdpo-live-channel.md`, `composer_replication/datagen/env.py`, `composer_replication/trainer/composer_trainer.py`, `composer_replication/recipes/prime_rl/composer_loss.py`.
+---
+## 1. TRL GRPOTrainer: Does It Actually Support Multi-Turn Agentic Rollouts?
+### 1.1 What the live TRL docs say (as of TRL v1.5.1, fetched 2026-06-10)
+The TRL GRPOTrainer docs (primary source, not the repo's research/04 note written in May 2025) describe **two distinct agentic mechanisms**:
+**Mechanism A — `tools` parameter:** Pass a list of Python callables. GRPOTrainer runs a tool-call loop. Quote from docs: "GRPO supports agent training through the `tools` argument in `GRPOTrainer`." The loop has a hard cap `max_tool_calling_iterations` (default: unlimited, stops on no-tool-call response or `max_model_length`). Each tool call is synchronous — the training GPU waits.
+**Mechanism B — `environment_factory` parameter:** Pass a callable that creates environment instances. "GRPOTrainer creates one environment instance per rollout and exposes the environment's public methods as tools." Requires `transformers>=5.2.0`. Marked **experimental**: "This feature is experimental and may change or be removed at any time without prior notice." The `reset()` method can return a string that gets appended to the last user message. `rollout_func` is similarly experimental.
+**Mechanism C — `rollout_func` (custom rollout):** A callable that receives prompts and the trainer, returns `{"prompt_ids", "completion_ids", "logprobs"}`. Also experimental. This is the escape hatch for fully custom multi-turn generation.
+**Key constraint confirmed from primary source:** TRL has **no async GPU-decoupled agent loop**. The docs explicitly state the training-inference mismatch and handle it via Truncated Importance Sampling (`vllm_importance_sampling_correction=True` by default), not by async GPU handoff. When a tool call is executing, the GPU waits. This is not a flaw in the repo's research note — `research/04-verl-trl.md` correctly identified this gap — but the docs now show TRL has partially closed the *multi-turn* gap via `tools` / `environment_factory`.
+### 1.2 What `research/04-verl-trl.md` claims vs. primary source
+| Claim in research/04 | Primary source (TRL docs, 2026) | Verdict |
+|---|---|---|
+| "TRL does NOT have an async GPU-decoupled agent loop" | Confirmed | CORRECT |
+| "OpenEnv integration (October 2025)" | Confirmed; `environment_factory` + TRL's OpenEnv guide | CORRECT |
+| "VLM support" | Confirmed — tools can return `list` of content blocks incl. images | CORRECT |
+| "GRPOTrainer supports multi-step agentic rollouts" (04:173) | Confirmed via `tools` + `environment_factory` | CORRECT |
+| TRL v1.0 released March 2026 | Confirmed; docs show versions v1.0.0 through v1.5.1 | CORRECT |
+| Default `loss_type` is `"dapo"` | **CONFIRMED from source**: `loss_type: str = 'dapo'` in GRPOConfig | CORRECT |
+| Default `scale_rewards` is... | **CONFIRMED: default is `"group"`** (not `False`/`"none"`) | CORRECT |
+### 1.3 Critical discovery: TRL's default is DAPO, not GRPO
+The TRL GRPOConfig shows `loss_type = 'dapo'` as the default. ADR-008 claims to configure `loss_type="dr_grpo"` to match Composer 2.5. The source confirms `"dr_grpo"` is a valid value (uses `max_completion_length` as the constant denominator). **This is consistent with ADR-008's decision.**
+However: ADR-008 states "KL estimator (k1 vs k3) is not configured or asserted" as an OPEN item. The TRL docs show `beta=0.0` as default (KL term disabled). If `beta=0`, the k1/k3 distinction is moot — there is no KL term in the loss at all. The ADR-008 open item is therefore **low-priority when beta=0** (the current default). If the repo ever enables beta>0 to use k1 KL-in-loss (distinct from the k1-in-reward path the trainer already implements), the open item becomes relevant.
+### 1.4 `scale_rewards` drift assertion
+ADR-008 checks `str(cfg.scale_rewards).lower() in ("none","false")`. Primary source confirms `scale_rewards` accepts: `True`/`"group"` (default), `"batch"`, `False`/`"none"`. The check is correct.
+---
+## 2. The Colocate-vLLM Blog: What It Actually Says
+Primary source: "No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" (June 3, 2025, huggingface.co/blog/vllm-colocate).
+**What the blog confirms:**
+- Co-locate mode (`vllm_mode="colocate"`) runs training and vLLM in the same process, sharing GPUs. No REST API overhead.
+- Speedups measured: 1.43× for 1.5B model, 1.35× for 7B model, 1.73× for 7B with TP>1. For 72B model (Qwen2.5-Math-72B): **co-locate is ~1.26× faster than plain TRL with 4 fewer GPUs**.
+- vLLM sleep mode (level 2) is **not yet merged into TRL upstream** (as of the blog date) due to a segfault on exit (vLLM issue #16993). The docs now show `vllm_enable_sleep_mode` as a parameter, implying it was eventually merged, but the blog notes a real production bug.
+- FSDP + co-locate + LoRA has known issues: "GRPO + FSDP + LoRA + VLLM colocate" doesn't work; DeepSpeed ZeRO-3 is the recommended path. FSDP1 has a NaN bug with co-located vLLM (issue #14443).
+**What `research/04` says:** Does not cite the co-locate blog specifically but correctly describes the "NO GPU left behind" feature (June 2025 update, row in §2.7 table). No material misread.
+**What the repo's SageMaker smoke recipe uses:** The SageMaker GRPO smoke (from git history context) uses `use_vllm=False` for initial tests, which is fine — the co-locate mode requires enough GPU memory for both model and vLLM, and a single g5.2xlarge (1× A10G, 24 GB) may not accommodate it.
+---
+## 3. VeRL: Agentic Mode and the AsyncServer
+Primary source: verl GitHub README (main branch, fetched 2026-06-10). Key finding from README:
+> "[2026/05] uni-agent is released: a unified agent framework to build, run, and train LLM agents at scale, built on top of verl."
+> "[2026/01] transfer_queue, fully_async_policy, one_step_off_policy ... are kept under verl/experimental since they are planned to be merged into the main library."
+> Feature list includes: "Multi-turn with tool calling", "Sandbox Fusion Integration", "SGLang, verl, OpenBMB and Tsinghua University: Pioneering End-to-End Multi-Turn RLHF"
+The verl README confirms multi-turn tool-calling exists and `uni-agent` was released May 2026 as a unified agent framework. The `AsyncServer`/`AgentLoop` architecture described in `research/04` is consistent with what the README describes, though the README doesn't use those exact terms. The experimental async features (`fully_async_policy`, `transfer_queue`) are available but not yet in main.
+**What `research/04` claims about VeRL agentic support:**
+- "First-class agentic RL support" with `AsyncServer`/`AgentLoop` — the README confirms the direction but notes these are under `verl/experimental`. The research/04 characterization of "first-class" **slightly overclaims** what is in the stable API; the full async path is experimental.
+- `SandboxFusionTool` — mentioned in the README as a documented integration ("Sandbox Fusion Integration" link). Consistent.
+- "Multi-turn tokenisation: noted as complex; naive concatenation of per-turn token IDs can introduce distribution drift" — confirmed by the README community blog link "When Reasoning Models Break Tokenization: The Hidden Complexity of Multiturn Training."
+---
+## 4. PRIME-RL in the Repo
+### 4.1 What ADR-006 claims
+ADR-006 claims PRIME-RL ships a `CustomLossConfig` with `import_path` for dropping in a Python loss function, exposing `LossInputs` with `trainer_logprobs`, `inference_logprobs`, `teacher_logprobs`, `advantages`, `loss_mask`. It was used to train INTELLECT-1 (10B, 30 nodes) and INTELLECT-2 (32B QwQ).
+### 4.2 What `composer_replication/recipes/prime_rl/composer_loss.py` confirms
+The code reads (lines 21-28):
+```python
+@dataclass
+class LossInputs:
+    trainer_logprobs:   Float[Tensor, ' seq']
+    inference_logprobs: Float[Tensor, ' seq']
+    teacher_logprobs:   Float[Tensor, ' seq'] | None
+    advantages:         Float[Tensor, ' seq']
+    loss_mask:          Bool[Tensor, ' seq']
+```
+This is marked as "verified against PrimeIntellect-ai/prime-rl `src/prime_rl/trainer/rl/loss.py` lines 13-22." The code correctly raises `NotImplementedError` when `alpha_sdpo > 0` (logits not available, only log-probs). **This is a real constraint, not a placeholder.**
+### 4.3 The DPPO upstream loss — a subtle accuracy point
+The `composer_loss.py` reproduces the upstream DPPO loss verbatim (lines 40-60 of the file). It uses:
+```python
+probs_diff = exp(trainer_logprobs) - exp(inference_logprobs)  # probability-space diff
+```
+This is notably **not** a log-ratio but a probability-space difference gating the drop/keep mask. This is PRIME-RL's design, not a repo mistake. But it means the DPPO channel is more like PRIME-RL's INTELLECT-style training than standard GRPO — the repo's framing in ADR-006 as "channels 1+3" needs to be understood in that context: channel 1 is DPPO-shaped (probability-gated policy update), not raw GRPO.
+---
+## 5. The Key Question: Is TRL's Single-Submit `reward_fn` a Dead End for Multi-Turn?
+### 5.1 What `env.py::reward_fn` actually does
+```python
+def reward_fn(self, prompts, completions, *, task_id, **kwargs) -> list[float]:
+    ...
+    for comp, tid in zip(completions, task_id):
+        task = self.registry[tid]
+        self.reset(task)
+        if self._replay is not None:
+            res = self._replay(self, comp)
+        else:
+            res = self.step({"type": "submit"})   # <-- single submit
+        rewards.append(res.reward)
+```
+**The fallback path** (no `_replay` function) treats the entire completion as a single submit — this is an outcome reward on a single-turn completion. This is what the unit tests exercise and what a standard TRL `reward_funcs` call would do.
+**The intended multi-turn path** is `_replay`: a callable that takes `(env, completion)` and drives multi-turn turns by parsing the agent's encoded tool-call history from the `completion` string. This is a **custom deserializer** that replays the agent turns and grades at the end.
+### 5.2 Is this a dead end for multi-turn RL?
+**For current TRL integration: partly yes, mostly no.**
+The single-submit fallback IS a dead end for genuine multi-turn RL credit assignment — it cannot grade intermediate tool-call steps. But there are two viable paths:
+**Path A — `_replay` + `rollout_func` (TRL experimental):** The `rollout_func` parameter in GRPOTrainer can drive multi-turn generation externally (running the env's `step()` loop), serialize the full trajectory into `completion` tokens, then call `reward_fn` which uses `_replay` to deserialize and grade. This makes the `reward_fn` the grader, not the rollout driver. This works in TRL **today** but requires the experimental `rollout_func` interface.
+**Path B — `environment_factory` (TRL experimental):** Pass `FeatureDeletionEnv` (or an adapter) as `environment_factory`. GRPOTrainer calls `reset()` and then uses the env's public methods as tools. The `reward_fn` is replaced by a reward function that reads `environments[i].reward` after generation. This is the more principled path for true multi-turn RL and is what TRL's `environment_factory` was designed for. It requires `transformers>=5.2.0` and is still experimental.
+**Path C — VeRL's AgentLoop:** For the tree-of-work (multiple parallel rollout branches per prompt, credit assigned at trajectory end), VeRL's `AsyncServer`+`AgentLoop` is architecturally the right fit. Each branch is a coroutine; GPU is not blocked during `sandbox.exec()` calls. The repo acknowledges this in `research/04` §5.3 recommendation.
+### 5.3 The honest migration path
+The current TRL single-submit `reward_fn` is:
+- **Correct** for the Phase 1 use case: offline dataset generation where the model produces a diff and we grade it. This is the "GRPO on completions" paradigm.
+- **Insufficient** for genuine multi-turn RL over FeatureDeletionEnv episodes, especially the tree-of-work vision where the model takes tool-call steps, explores branches, and gets rewards at trajectory end.
+**Migration path (in order of complexity):**
+1. **Immediate (low cost):** Use TRL's `environment_factory` with `FeatureDeletionEnv` as the adapter. The env's `step()` becomes a tool. Grade via `reward_funcs` reading `env.reward`. Marked experimental but low integration cost. This supports genuine multi-turn GRPO with the current TRL host.
+2. **Medium term (single-GPU scale):** Implement `rollout_func` that drives the env loop directly, returns serialized trajectories with log-probs. Full control over multi-turn; TRL handles the GRPO update.
+3. **Scale-out (multi-GPU, async, tree-of-work):** Migrate to VeRL's `AgentLoop`. The `FeatureDeletionEnv` maps onto verl's `SandboxFusionTool` protocol. The tree-of-work branching requires N parallel rollout workers per prompt, which VeRL's asyncio architecture supports and TRL's synchronous loop does not.
+**The tree-of-work IS multi-turn.** The vision in `framework/composer-replication-framework.md` of a "multi-model Monte-Carlo tree-of-work" requires:
+- Many concurrent rollout branches per prompt
+- Reward propagated back through the tree (not just at leaf)
+- Asynchronous sandbox execution without blocking GPU
+None of these are provided by TRL's current `GRPOTrainer` (even with `tools`/`environment_factory`). VeRL's experimental `fully_async_policy` + `AgentLoop` is the right substrate. The repo's `research/04` correctly identifies this but the ADR layer has not formally acknowledged this migration requirement.
+---
+## 6. Sandboxing for Code Execution at Scale
+### 6.1 What the secure-EKS article says (primary source)
+> "gVisor added negligible launch latency... handles isolation for most agent workloads."
+> "Cold start was around 5 seconds per sandbox" for Kata+Firecracker.
+> "EKS Managed Node Groups do not work yet: they override the CPU Options stanza needed for nested virtualization, forcing the use of self-managed node groups."
+> "Managed sandbox platforms skip Kubernetes entirely. E2B and Vercel Sandbox provision Firecracker microVMs directly... sandbox creation in under a second, versus ~5 seconds Kata with Firecracker on EKS."
+### 6.2 What SWE-MiniSandbox (arXiv:2602.11210v5) adds
+Abstract: "SWE-MiniSandbox lowers disk usage to approximately 5% of that required by container-based pipelines and reduces environment preparation time to about 25% of the container baseline." Uses "kernel-level mechanisms" (not containers) with "lightweight environment pre-caching." Empirical performance comparable to container-based pipelines on SWE-bench-style tasks.
+This is directly relevant to the repo's `FeatureDeletionEnv`/`Sandbox` design. The current `sandbox.py` uses `LocalSubprocessSandbox` (plain subprocess, Docker-gated for real tests) — essentially no isolation for the subprocess case. For production RL training at scale with multiple rollout workers, the SWE-MiniSandbox approach (kernel-level isolation without per-task container builds) could reduce env setup from minutes to seconds.
+### 6.3 What the repo's sandbox.py actually provides
+`sandbox.py` defines:
+- `LocalSubprocessSandbox` — runs commands via `subprocess.run` in the repo tree. No container, no kernel isolation. The security model relies on the denylist + cache scrub (commented as "INSUFFICIENT as a primary control").
+- `DockerSandbox` (in `docker_sandbox.py`) — real isolation, referenced in tests.
+**The gap:** For RL training at scale (many parallel rollout workers), neither `LocalSubprocessSandbox` nor per-task Docker containers are adequate:
+- Subprocess: no isolation, reward hacking possible via Python import tricks (acknowledged in code comments).
+- Docker: isolation is good, but per-task container boot is slow (typical: 2–5s without pre-warming), and at 8 rollouts × N prompts × G generations = hundreds of container launches per training step.
+The repo's `research/review-sandbox.json` presumably tracks this; the production path requires pre-warmed sandbox pools (gVisor RuntimeClass for speed, Kata/Firecracker for stronger isolation).
+---
+## 7. Misreads, Overclaims, and Gaps in Repo Research
+### 7.1 OVERCLAIM: VeRL "first-class" agentic RL
+`research/04` §1.5: "VeRL has first-class agentic RL support" and describes `AsyncServer`/`AgentLoop` as stable. The verl README (main branch, 2026-06-10) shows:
+- `transfer_queue`, `fully_async_policy`, `one_step_off_policy` are kept under `verl/experimental` — "planned to be merged into the main library."
+- `uni-agent` (May 2026) provides the higher-level agent framework, but it's a separate release on top of verl, not part of the stable library.
+The agentic async path **exists** but is experimental in the stable API. The characterization as "first-class" is slightly ahead of the actual maturity. The repo should note this when recommending VeRL for the tree-of-work.
+### 7.2 MISS: TRL now has `environment_factory` + `tools` for multi-turn
+`research/04` (written May 2025) describes TRL as having only synchronous reward functions. The live TRL docs (v1.5.1, 2026) show `environment_factory` and `tools` for genuine multi-turn generation loops. These are experimental but available. The research/04 comparison matrix says "TRL: agentic tool-calling RL ⚠️ (blocking)" — this remains accurate (it IS blocking), but misses that TRL now provides the `environment_factory` interface which would allow `FeatureDeletionEnv` to drive multi-turn episodes inside GRPOTrainer without a custom `rollout_func`. This is not mentioned in any ADR.
+### 7.3 MISS: TRL default `loss_type="dapo"`, NOT `"grpo"` or `"dr_grpo"`
+ADR-008 correctly targets `loss_type="dr_grpo"`, but research/04 (the background research) does not explicitly state that TRL 1.x defaults to DAPO loss. The live docs confirm this. The drift assertion in `make_dr_grpo_config` is the right mitigation.
+### 7.4 GAP: `scale_rewards` default is `"group"`, not `True`
+The GRPOConfig shows `scale_rewards: str = 'group'` (a string, not a bool). ADR-008's assertion `str(cfg.scale_rewards).lower() in ("none","false")` correctly handles both the old bool (`False`) and new string (`"none"`) forms. But the docs show `True` and `"group"` are equivalent (both mean group-level std scaling). The assertion is correct; this is a documentation note, not a bug.
+### 7.5 GAP: KL estimator — TRL default is k3, not k1
+The TRL docs show the KL approximator formula:
+```
+D_KL[π_θ || π_ref] = π_ref/π_θ - log(π_ref/π_θ) - 1
+```
+This is the **k3 estimator** (Schulman et al., 2020). ADR-008 claims Composer 2.5 uses k1 (`-log r = log π_θ/π_ref`). The ADR notes this as an OPEN item. Since `beta=0.0` by default in TRL, the KL term is disabled and this doesn't affect training unless `beta>0`. However, `composer_trainer.py` implements the k1-in-reward path via `kl_in_reward.py` — this is a separate mechanism from TRL's in-loss KL. Verify: the k1-in-reward path computes `log(π_θ/π_ref)` at reward time and folds it into advantages, while TRL's in-loss k3 term (when beta>0) would add a different term. If both are enabled simultaneously, they would double-count KL. The safe configuration is: **k1-in-reward active, beta=0 (TRL in-loss KL disabled)**. The code appears to do this but there's no explicit assertion that `beta=0` when `kl_in_reward=True`.
+### 7.6 MISS: `num_iterations=1` narrowed claim
+ADR-008 acknowledges that `num_iterations=1` controls GRPO inner-loop reuse, not dataset-level epochs. Primary source confirms: "Number of iterations per batch (denoted as μ in the algorithm)." The ADR's narrowed claim is correct.
+### 7.7 MISS: TRL default optimizer is `adamw_torch_fused`, not `adam`
+ADR-008 has an OPEN item: "Adam is claimed but `optim` is not set." The GRPOConfig docs show:
+```
+optim: transformers.training_args.OptimizerNames | str = 'adamw_torch_fused'
+```
+Default is `adamw_torch_fused` (AdamW with fused CUDA kernel), not plain `adam`. If Composer 2.5 uses Adam (without weight decay), the ADR's open item remains relevant: set `weight_decay=0.0` and `optim="adam"` explicitly to match. The default AdamW has weight decay (though `weight_decay=0.0` is already the GRPOConfig default, making it numerically equivalent to Adam in this specific case). However, `adamw_torch_fused` ≠ `adam` in terms of the optimizer implementation; to be precise, set `optim="adamw_8bit"` or `optim="paged_adamw_8bit"` (memory efficient) or just `optim="adam_torch"` if plain Adam is intended.
+---
+## 8. verl `uni-agent` — A New Development Not in Research/04
+The verl README (May 2026): "uni-agent is released: a unified agent framework to build, run, and train LLM agents at scale, built on top of verl." This is a post-cutoff development (research/04 was written May 2025) that the repo has not incorporated. `uni-agent` could be the production-ready path for multi-turn agentic RL with verl, potentially superseding the lower-level `AgentLoop`/`AsyncServer` integration that ADR-006 contemplates.
+**Implication for the repo:** Before committing to a custom VeRL `AgentLoop` integration, evaluate whether `uni-agent` already provides the FeatureDeletionEnv integration pattern out of the box. This could significantly reduce the engineering surface.
+---
+## 9. Sandboxing Recommendation Gaps
+### 9.1 The `SWE-MiniSandbox` approach is not referenced anywhere in the repo
+arXiv:2602.11210 (Feb 2026) directly addresses the production gap: container-free RL training with 5% disk usage and 25% env setup time vs containers. The paper's "kernel-level mechanisms" (likely Linux namespaces + cgroups without a full container runtime) with pre-caching is directly applicable to `FeatureDeletionEnv` at scale. The repo's sandbox design doesn't reference this work.
+### 9.2 The repo's `docker_sandbox.py` is production-blocking for RL at scale
+Per-task Docker container boots at GRPO scale (G=8 completions/prompt, B=4 per-device batch, many workers) means O(B*G) = O(32) container launches per training step. Without pre-warming or snapshot-based fast boot, this is the dominant latency. The gVisor RuntimeClass approach (negligible overhead, per the AWS article) or SWE-MiniSandbox's kernel-namespace approach are both faster alternatives.
+---
+## 10. Summary of Critical Findings
+| Finding | Severity | Affected Files |
+|---|---|---|
+| VeRL async agent loop is EXPERIMENTAL, not "first-class" stable | MEDIUM — overclaim | research/04 §1.5, ADR-006 |
+| TRL `environment_factory` (multi-turn) not in any ADR | MEDIUM — miss | ADR-008, env.py |
+| k1-in-reward + beta=0 assertion missing (double-KL risk if beta>0 ever set) | MEDIUM — correctness gap | composer_trainer.py, ADR-008 |
+| `optim` default is `adamw_torch_fused`, not `adam` | LOW — fidelity gap | ADR-008 OPEN item |
+| TRL `loss_type` defaults to `"dapo"` (not GRPO), correctly handled | INFO — confirmed correct | ADR-008, make_dr_grpo_config |
+| `env.py::reward_fn` single-submit path is dead end for tree-of-work | HIGH — architecture gap | env.py, no ADR exists |
+| `uni-agent` (verl, May 2026) not evaluated — may supersede custom AgentLoop | MEDIUM — miss | ADR-006 |
+| SWE-MiniSandbox approach not referenced (5% disk, 25% setup time) | MEDIUM — miss | sandbox.py, docker_sandbox.py |
+| EKS Managed Node Groups incompatible with Kata+Firecracker (nested virt) | INFO — production gotcha | no ADR |
+---
+## 11. Migration Path for Multi-Turn Agentic RL (Honest Assessment)
+The current repo architecture (TRL `reward_fn` with single-submit fallback) is:
+**Phase 1 — GRPO on completions (current):** The model generates a single diff/completion, the reward function grades it. This is viable, shippable, and correct. The TRL host is appropriate. No migration needed for this phase.
+**Phase 2 — Multi-turn FeatureDeletionEnv (agentic GRPO):** The model takes tool-call steps (bash, file edits, test runs). Reward at trajectory end. Migration options in order:
+1. **TRL `environment_factory` adapter (experimental, weeks of work):** Wrap `FeatureDeletionEnv` as a TRL environment. Methods become tools. Blocking GPU during sandbox execution — OK for small scale (≤8 GPUs), not for high parallelism.
+2. **TRL `rollout_func` (experimental, 1–2 weeks):** Custom rollout that drives the env loop, serializes trajectories. Full control; TRL handles GRPO update.
+3. **VeRL AsyncServer + `FeatureDeletionEnv` as SandboxFusionTool adapter (2–4 weeks):** GPU not blocked during sandbox calls. Required for tree-of-work fan-out at scale. The repo's ADR-006/ADR-008 have this on the roadmap but it's not implemented.
+**Phase 3 — Tree-of-Work (MCTS, multi-branch):** This REQUIRES verl's async architecture. TRL cannot support N parallel branches per prompt with GPU-efficient execution. The `uni-agent` framework on top of verl should be evaluated first before building a custom AgentLoop integration.
+---
+*Sources: TRL docs fetched 2026-06-10 (huggingface.co/docs/trl/en/grpo_trainer); vLLM co-locate blog (huggingface.co/blog/vllm-colocate); verl README (github.com/volcengine/verl/blob/main/README.md); SWE-MiniSandbox arXiv:2602.11210v5; AWS Builder Center EKS sandboxes article. Repo files: research/04, research/03, ADR-006, ADR-008, env.py, composer_trainer.py, recipes/prime_rl/composer_loss.py.*

research/deepread/10-critic-fidelity.md ADDED Viewed

	@@ -0,0 +1,169 @@

+# Critic: FIDELITY-TO-SOURCES
+**Reviewer:** fidelity critic (critical-review pipeline)
+**Date:** 2026-06-10
+**Primary sources re-verified this pass:** Composer 2.5 blog full body (vault note `introducing-composer-25-cursor`, re-read verbatim), deep-reads `00-grounding.md` and `01`–`08` (all read in full), plus direct re-reads of `docs/COMPOSER_RECIPE_MAPPING.md`, `research/01-composer-2.5.md`, `research/06-feature-deletion-datagen.md`, `docs/adrs/ADR-010`, `composer_replication/opsd.py`, `composer_replication/teacher_replay.py`, `composer_replication/trainer/kl_in_reward.py`, `composer_replication/diloco/__init__.py`, `framework/composer-replication-framework.md`, `docs/VISION_VALIDATION.md`.
+Severity scale: **P0** = claim must be corrected (factually wrong vs primary source); **P1** = needs a caveat/qualifier (true-ish but overclaims or hides scope); **P2** = wording/precision fix.
+---
+## P0 — claims that must be corrected
+**1. [P0] Benchmark numbers (69.3%, Terminal-Bench 2.0 parity) presented as Cursor-stated — they appear in NO primary source.**
+- *Source fact:* The Composer 2.5 blog (re-fetched, full body) contains **zero benchmark numbers**. The only numerics are pricing ($0.50/$2.50, $3.00/$15.00), "25x more synthetic tasks," "10x more total compute," "0.2s" optimizer step, CP=2/EP=8. The Composer 2 tech report's Table 1 has 61.3/73.7/61.7 for Composer **2** (deep-read 01, FINDING-1).
+- *Repo claim:* `research/01-composer-2.5.md:63-64`: "Composer 2.5 scored 69.3% … On public agentic benchmarks like *Terminal-Bench 2.0*, it hit 69.3%. On *SWE-bench Multilingual*, it achieved parity with or slightly surpassed OpenAI's GPT-5.5."
+- *Problem:* The file's own audit notice (line 8) flags this, but the body still asserts the numbers as fact, and the audit note says only "not in the 2.5 blog" — it does not say the numbers are unverified in ANY primary source. Anyone citing §Performance gets fabricated/secondary numbers.
+- *Correction:* Strike or rewrite §"Performance Characteristics" to: "Neither the 2.5 blog nor the Composer 2 technical report publishes Composer 2.5 benchmark numbers. Figures like 69.3% circulate in secondary commentary (e.g., the Lushbinary developer-guide blog in the vault) and must not be used as replication targets."
+**2. [P0] "Feature Deletion + 24 other (unnamed) generators" — the count 24 is invented.**
+- *Source fact (blog, verbatim):* "We use a **range of approaches** for creating synthetic tasks that are grounded in real codebases. For example, **one** synthetic approach is feature deletion." No count, no other names, anywhere.
+- *Repo claim:* `docs/COMPOSER_RECIPE_MAPPING.md:75` (mapping row b): "Feature Deletion + 24 other (unnamed) generators"; `research/06-feature-deletion-datagen.md:330`: "The other ~24 generators"; propagated into `research/09` line 23 ("captured 'Feature Deletion + 24 unnamed generators'").
+- *Problem:* "24" appears to be a back-formation from "25x" — but 25x is a **task-count multiplier vs Composer 2**, not a generator count. Two unrelated quantities have been conflated.
+- *Correction:* "Feature deletion is the only named generator. The blog says 'a range of approaches'; the number, names, and weighting of other generators are unknown. The '25x' figure counts synthetic *tasks* relative to Composer 2 (whose baseline count is also unstated), not generators."
+**3. [P0] SDPO declared "mathematically the same" as Composer 2.5's mechanism — the equivalence is asserted nowhere by Cursor, and SDPO's published mechanics materially differ.**
+- *Source facts:* (a) The blog cites the three self-distillation papers only as "For more **background** on this approach see…" (footnote 1, verbatim). (b) SDPO (arXiv:2601.20802, Eq. 1) applies the KL **over the full rollout response** with feedback in the **conditioning prefix**; it has **no error-turn detection, no hint intercalated into the response, and requires a regularized (EMA/trust-region) teacher** (deep-read 03 §1.1, §3.1–3.2). The blog's mechanism is turn-localized ("For that turn only, we then update the student weights…") — closer to the repo's implementation than to SDPO.
+- *Repo claims:* `docs/COMPOSER_RECIPE_MAPPING.md:25`: "This is **mathematically the same** as Composer's targeted-textual-feedback method"; same doc §Citations: SDPO is "The direct formalization of Composer's hint-distill"; `composer_replication/opsd.py:13-14`: "arXiv:2601.20802 (formalizes the same loss as Composer 2.5's 'Targeted RL with Textual Feedback')".
+- *Problem:* This is the load-bearing identification the whole Channel-2 architecture rests on ("Lift the OPSD/SDPO loss directly… exact same mechanism Cursor uses" — mapping doc §Implementation handles). Cursor never says SDPO is its mechanism; and the repo's actual implementation (error-turn masking + hint spliced into the teacher's response sequence + ADR-011 alignment indices) is **neither** SDPO **nor** confirmed Composer — it is a blog-inspired original design (deep-read 03 verdicts: "FALSE — SDPO applies loss to full rollout"; "FALSE — feedback is in prefix, not intercalated").
+- *Correction:* "Cursor cites SDPO/OPSD as *background*. SDPO is the closest published formalization of a hint-conditioned same-weights teacher with an on-policy distillation KL, but it differs from the blog's described mechanism in localization (full-rollout vs per-turn) and from our implementation in teacher-context construction (feedback-in-prefix vs hint-at-error-turn). Our SDPO channel is a blog-faithful, SDPO-*inspired* design — not a port of SDPO and not a confirmed match to Cursor's internal method."
+**4. [P0] Streaming DiLoCo misattributed: wrong authors AND wrong title attached to arXiv:2501.18512.**
+- *Source fact:* arXiv:2501.18512 is "Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch," **Douillard et al. 2025**. "Eager Updates for Overlapped Communication and Computation in DiLoCo" is a *different* paper (arXiv:2502.12996, Kale/Douillard/Donchev). "Liu et al." is the Async Local-SGD paper (arXiv:2401.09135) (deep-read 05, F4).
+- *Repo claims:* `composer_replication/diloco/__init__.py:8`: "Streaming DiLoCo (Liu et al. 2025)"; `research/design-F4-decoupled-diloco-s3.md:109`: "Liu et al. 2025, 'Eager Updates for Overlapped Communication in DiLoCo', arXiv:2501.18512".
+- *Correction:* Fix both files to "Streaming DiLoCo (Douillard et al. 2025, arXiv:2501.18512)"; cite Eager Updates separately as Kale et al. 2025, arXiv:2502.12996.
+**5. [P0] CWM cited as licensing "train-on-all for the world-model aux head during RL" — CWM does this in a separate mid-training stage, not as an aux head riding policy gradients.**
+- *Source fact (CWM, arXiv:2510.02387 §2.2, verbatim):* "Because our goal with the ForagerAgent data is to learn a comprehensive world model … **we do not filter trajectories based on whether they succeed**." That is a **mid-training data decision** for a dedicated pre-RL stage; the world-modeling capability is in the base weights before RL begins (deep-read 06, Finding 2.1, 4.4).
+- *Repo claim:* `research/notes/final_report_socratic-mcts-swe-worldmodel-8f6dea.md` §2: CWM "crucially training *on all* trajectories **for the world-model head**, reserving success-filtering only for the RL reward"; §4 re-uses it as the justification that the aux head "makes train-on-all *safe* for the policy."
+- *Problem:* This is the single load-bearing citation for the two-harvest design (failed branches → world-model head). The cited architecture is structurally different; the simultaneous-gradient configuration the repo proposes is exactly the one the also-cited interference paper (2602.00994) warns about.
+- *Correction:* "CWM mid-trains on unfiltered trajectories in a dedicated pre-RL stage; it provides an existence proof for train-on-all *dynamics learning*, not for an auxiliary next-state head trained simultaneously with RL. No published work tests our exact configuration."
+---
+## P1 — claims needing caveats / qualifiers
+**6. [P1] CWM's "65.8% on SWE-bench Verified" cited without the test-time-scaling qualifier.**
+- *Source:* CWM abstract: "pass@1 scores of 65.8% on SWE-bench Verified **(with test-time scaling)**"; the single-attempt base score is lower (deep-read 06, Finding 2.1).
+- *Correction:* Always cite as "65.8% with test-time scaling" or give the base score alongside.
+**7. [P1] Chain-of-World (arXiv:2603.03195) used as evidence for SWE world-modeling — it is a robotics VLA paper.**
+- *Source:* CoWVLA: latent-motion world model for robot manipulation, CVPR 2026, cs.CV (deep-read 06, Finding 2.2). No SWE/code content.
+- *Repo claim:* final report §2 deploys it for "predict the consequential terminal state" in the SWE design.
+- *Correction:* Keep only as explicitly-flagged robotics analogy or remove.
+**8. [P1] "85% of total compute is post-training" still stated as fact in the research/01 body.**
+- *Source:* Neither blog nor tech report contains any compute-budget ratio (deep-read 01, FINDING-6).
+- *Repo claim:* `research/01-composer-2.5.md:14`: "roughly 85% of the total compute budget for Composer 2.5 was spent on Cursor's proprietary post-training…". The header audit note flags it, but the body asserts it unhedged.
+- *Correction:* Inline-tag the sentence: "[community speculation — not in any Cursor source]".
+**9. [P1] The repo's implemented "feature deletion" is gold-patch reversion over pre-labeled SWE datasets — not the blog's mechanism — yet downstream docs say it "matches the recipe."**
+- *Source fact (blog, verbatim):* "the agent is given a codebase with a large set of tests, and **asked to delete code and files** in such a way that the codebase remains functional while specific testable features are removed." The published mechanism is (likely agentic) *deletion from a functional, test-covered codebase*. The repo's `SweBenchAdapter` instead reverts human PR gold patches of pre-packaged instances — the analogue of SWE-smith's **PR Mirror** strategy, a *third* construction the blog never describes (deep-reads 00 Claim 1, 02 §6.1).
+- *Repo claims:* ADR-010 is honest about Option A vs B, but its Consequences section says the subsystem delivers the recipe; `docs/COMPOSER_RECIPE_MAPPING.md:42` paraphrases the blog as "take a repo with passing tests, delete some code" (passive — drops the agentic deleter and the "remains functional" constraint); the project framing ("point-at-a-repo feature-deletion task synthesis") attributes to the blog a phrase ("point at a repo") that appears nowhere in it.
+- *Correction:* "We implement the *inversion analogue* of feature deletion (revert gold patch of an existing SWE instance — SWE-smith's PR-Mirror shape, which SWE-smith's ablations show yields the best training data). The blog's actual generator — deleting code/files from a live, functional codebase so that targeted tests fail — is unbuilt (Option B / Path B). 'Point at a repo' is our reconstruction, not blog language."
+**10. [P1] ADR-010: "Online difficulty gate matches the actual recipe" — only half the recipe, and with a different signal.**
+- *Source facts:* Blog: "we both **select for and create** harder tasks dynamically throughout the run" — two operations. The only Cursor-stated difficulty signal is Composer 2's "number of turns and thinking tokens of rollouts—to upsample increasingly harder data points" (tech report §4) — and that is a Composer-2 mechanism mapped onto 2.5 (deep-read 01, FINDING-12).
+- *Repo state:* `DifficultyCurriculum` implements only the SELECT half, primarily on a pass-rate frontier (extrapolated, not Cursor-stated); the CREATE half (minting harder tasks mid-run) is 0% built; `granularity` is hardcoded `"feature"` (grounding doc, Claim 3). Wave 20's effort tilt on turns/think-tokens partially aligns with the stated heuristic.
+- *Correction:* "The select-for half is built (pass-rate frontier [EXTRAPOLATED] + turns/think-token effort tilt [Composer-2-sourced]); the create-harder half of the published recipe is not implemented."
+**11. [P1] ADR-010 decision driver: substrates "already guarantee test-exercises-the-code via FAIL_TO_PASS."**
+- *Source fact:* SWE-smith (§2.1) shows the F2P property is the *output of running validation* (overall yield ~50.1%; candidates lacking test coverage are filtered out by execution), not a schema-inherited guarantee; no surveyed dataset verifies *reachability* of the deleted code from the failing tests (deep-read 02, §6.2 item 6; ADR-010's own OPEN item concedes Gate 2 doesn't verify reachability).
+- *Correction:* "F2P labels are trustworthy because the upstream pipelines *executed* the tests; any new inversion we mint must re-run gates 1–3 to re-establish the property, and reachability remains unverified across the field."
+**12. [P1] Mapping doc states the KL direction as `KL(teacher || student)` — unsupported by the blog and opposite to the paper the doc equates it to.**
+- *Source facts:* Blog: "an on-policy distillation KL loss that moves the student's token probabilities toward the teacher's" — direction unspecified. SDPO Eq. 1: `KL(π_θ ‖ stopgrad(q_θ))` — **student first** (deep-read 03 §1.1).
+- *Repo claim:* `docs/COMPOSER_RECIPE_MAPPING.md:20`: "Loss = on-policy KL divergence: `KL( teacher_logits_at_turn_t || student_logits_at_turn_t )`".
+- *Correction:* "Direction unspecified in the blog; SDPO uses KL(student‖teacher) (adopting JSD for stability); our implementation uses the OPSD generalized JSD."
+**13. [P1] `opsd.py` β-convention docstring is inverted vs the upstream it claims byte-parity with.**
+- *Source:* OPSD README: "Beta=0 means forward KL and 1 means reverse KL." Repo docstring labels β=0 "reverse KL" / β=1 "forward KL." Code is numerically correct; the labels would misconfigure anyone choosing β by docstring (deep-read 03 §2.2).
+- *Correction:* Swap the labels to match upstream.
+**14. [P1] SDPO-fidelity claim without SDPO's teacher regularization.**
+- *Source:* SDPO Table 4: non-regularized live-weights teacher **diverges** (36.1% vs 50.6% with trust-region teacher); EMA/trust-region regularization is a core stability component (deep-read 03 §1.1, rec. 4). Also §4.5: SDPO underperforms GRPO on weak models; λ=0.9 GRPO + 0.1 SDPO hybrid recommended.
+- *Repo state:* teacher = live weights each step, no EMA/trust-region; no usage guidance on the weak-model failure mode.
+- *Correction:* Document both gaps wherever the SDPO channel is described as paper-faithful; implement or explicitly defer teacher regularization.
+**15. [P1] `SiblingBootstrapGenerator` framed as following from SDPO's "successful rollouts as implicit feedback" — it is an extrapolation.**
+- *Source:* SDPO's sibling mechanism puts the successful sibling rollout **into the teacher's conditioning prefix** and applies the KL over the *entire* original response. It does not generate a "Reminder: a working approach looks like…" hint string, insert text into the response, or detect error turns (deep-read 03 §6).
+- *Repo claim:* `research/07` + `hint_generator.py` sketch present the sibling-hint design as the natural SDPO-supported fallback.
+- *Correction:* "SDPO supports sibling-as-conditioning-context; our hint-string splice at error turns is our own Composer-blog-inspired variant. The paper-faithful alternative (sibling in prefix, full-response KL) is simpler and should be the ablation baseline."
+**16. [P1] "$0.98 mean per-trace cost ungated — verified economic floor" hides the trace definition; real sessions cost ~2 orders of magnitude more.**
+- *Source:* Spike 001 measured 50 hand-crafted synthetic states at N=3 teachers. Real Claude Code sessions have 125–2,830 tool-use messages (ADR-002); at DEFAULT_TEACHERS pricing a ~1,400-step session is ~$70–80 flat (deep-read 07, FR-R5).
+- *Repo claims:* `teacher_replay.py` docstring ("Verified economic floor … $0.98 mean per-trace cost ungated"); `docs/VISION_VALIDATION.md` ("$0.98/trace verified … economic floor is established").
+- *Correction:* "Verified floor for 50-step synthetic-state benchmark traces; full-session flat replay is ~$70–80 ungated; $0.30/trace VOI-gated figure is a projection, not a measurement."
+**17. [P1] "$64 ungated tree" is mislabeled — it is a flat N=8, T=1000 extrapolation, not a tree and not a codebase config.**
+- *Source:* research/05 constructs $0.008 × 1000 × 8 = $64 flat; DEFAULT_TEACHERS is N=3; a true branching tree is O(N^D), strictly worse (deep-read 07, 05-R6/FR-R6).
+- *Correction:* Label it "flat 8-teacher × 1000-step extrapolation (unmeasured)" everywhere; never "tree."
+**18. [P1] `kl_in_reward.py`: "verl adopted k1-in-reward as its *only* reverse-KL option" — overclaim.**
+- *Source:* verl supports `kl_penalty="kl"` (k1) **and** `kl_penalty="low_var_kl"` (k3-family) (deep-read 04 §4.3).
+- *Correction:* "verl defaults to / recommends k1-in-reward" — drop "only."
+**19. [P1] Comedy of Estimators (arXiv:2512.21852) used as "k1-in-reward improves OOD; k3-in-reward can collapse" — full text never read, and the 7B/8B-math→1T-MoE-agentic extrapolation is unflagged.**
+- *Source:* Only the abstract was obtainable (HTML 404); it supports "biased gradients → instability; unbiased → better OOD" but does not state the specific estimator-placement ranking the repo asserts (deep-read 04 §2, §4.4).
+- *Correction:* Soften to "consistent with the abstract's biased-vs-unbiased finding; specific k1/k3-placement ranking unverified (full text unavailable); empirical scope is 7B/8B reasoning models."
+**20. [P1] GSPO preset claims to implement GSPO but inherits GRPO-scale clipping — two orders of magnitude off the paper's settings.**
+- *Source:* GSPO paper §5.1: clipping ranges 3e-4/4e-4, "a difference of two orders of magnitude in the fractions of clipped tokens between GSPO and GRPO." The repo preset sets no epsilon → TRL defaults (~0.2) → effectively unclipped sequence-level REINFORCE (deep-read 04, ISSUE 1).
+- *Correction:* Add `epsilon=3e-4, epsilon_high=4e-4` to the preset or annotate it "architecture-only; not operationally GSPO without GSPO-scale epsilons." (Companion: CISPO preset should set `beta=0.0` explicitly per MiniMax-M1, and document its deliberate `scale_rewards="none"` deviation from the paper's std-norm.)
+**21. [P1] research/05's novelty framing: rStar named "closest precedent" (misread) while the actually-closest works are uncited.**
+- *Source facts:* rStar's discriminator verifies **full trajectories**, not per-step states ("acts as a discriminator to verify each trajectory" — abstract). Tree-GRPO (arXiv:2509.21240) proves intra-tree group-relative advantage ≡ step-level DPO — the formal version of `extract_dpo_pairs`; SWE-Search (arXiv:2410.20285) is MCTS on SWE-bench itself. Neither is cited in research/05; `framework/composer-replication-framework.md:17` still says "Closest precedent: rStar-Math … Multi-teacher *frozen-trace replay* is open territory" (deep-read 07, 05-R1/R2/R3).
+- *Correction:* The novelty claim (frozen-trace × N heterogeneous teachers × disagreement-DPO) survives, but the provenance section must cite Tree-GRPO and SWE-Search as the nearest neighbors and correct the rStar granularity description.
+**22. [P1] "Roughly nine-tenths of it is reuse" (final report §6) conflates design-reuse with build status.**
+- *Source:* Exhaustive grep confirms tree controller, SiblingBootstrapGenerator, world-model head, `<deliberate>` token, pipeline/, infra/, broken-image builder are **0% built** (grounding doc §3, Claim 5).
+- *Correction:* "Nine-tenths of the *recipe-replication* layer reuses existing code; the framework's own additions (tree, world-model head, AWS datagen pipeline) are entirely design-stage."
+**23. [P1] World-model aux-head design presented with citation support that is wholly analogical; "parameter isolation eliminates the interference risk" overclaims DART.**
+- *Source facts:* No paper in the cluster tests a next-state aux loss on a policy LLM during code RL — the evidentiary gap for the exact proposed configuration is total (deep-read 06, Missing 1). DART (2602.00994) shows separate LoRA modules *reduce* interference but do not reach the 2-Agent upper bound (Finding 3.1); its domain is RAG-QA/NL2SQL, not SWE.
+- *Correction:* Add an explicit null-evidence flag to §2 of the final report and to any ADR that inherits it; change "eliminates" → "substantially reduces"; the P4 ablation is a research hypothesis test, not a validation of established results.
+**24. [P1] VeRL "first-class agentic RL support" — the async path is experimental.**
+- *Source:* verl README: `fully_async_policy`, `transfer_queue` live under `verl/experimental`; `uni-agent` (May 2026) is a separate layer (deep-read 08 §7.1).
+- *Correction:* "VeRL's agentic/async path exists but is experimental; evaluate `uni-agent` before committing to a custom AgentLoop."
+---
+## P2 — wording / precision
+**25. [P2] research/01 §5: "During **post-training**, Cursor employs Sharded Muon and Dual Mesh HSDP."** Blog (verbatim): "For **continued pretraining**, we use Muon…". Fix the stage attribution (the mapping doc already has it right). Also do not conflate with Composer 2's FSDP+CP/AdamW system (deep-read 01, FINDING-7).
+**26. [P2] research/06's "two-agent / two-phase structure the blog implies."** The blog's grammar has one "the agent … asked to delete"; whether the deleter is a model, program, or pipeline is unstated. research/06 already lists this as an open question (line 329) — align the §1 framing with it: "deleter unknown; blog grammar suggests an agent" (deep-read 01, FINDING-3).
+**27. [P2] research/06 "~50k–60k tasks" as the "25×-spirit" pool.** No Composer-2 baseline count exists, so 25x is not convertible to an absolute. The "spirit" hedge is present; add "[EXTRAPOLATED — no primary-source baseline]" at the number (deep-read 01, FINDING-2).
+**28. [P2] Grounding doc Claim 1 presents a paraphrase as a blog quote.** `research/deepread/00-grounding.md` Claim 1: "What the blog says (COMPOSER_RECIPE_MAPPING.md): 'take a repo with passing tests, delete some code, ask the agent to reimplement to pass tests.'" — that sentence is the mapping doc's paraphrase, not blog text. Use the real blog sentence when quoting "what the blog says."
+**29. [P2] research/02 DiLoCo compression details.** "FP16 outer state" → Streaming DiLoCo quantizes **outer gradients to FP4 (E3M0)**, FP32 accumulation; bandwidth claim "~100×" → measured ≈400× total-bits reduction (Table 1) (deep-read 05, F6/F7).
+**30. [P2] DiLoCo H default source attribution.** `diloco/__init__.py` docstring credits "DiLoCo paper §3.2" for defaults, but §3.2's main-experiment H is 500; the repo's `sync_every=100` matches the Streaming/OpenDiLoCo range. Also note Streaming's outer_lr=0.4 vs the vanilla 0.7 default (deep-read 05, F2/F3).
+**31. [P2] "Foresight@k" cited as if sourced.** The metric is coined by the final report; citations [11][2] do not define it. Mark "(we define this metric)" at first use (deep-read 06, Finding 3.2).
+**32. [P2] SWE-rebench "21,336 tasks" count never verified against the SWE-rebench paper** (only the Nebius infra blog was fetched); and the 59k-row figure is ambiguous between SWE-smith-on-HF and Nemotron-SWE-v1 (deep-read 02, §6.2 items 3 and unverified item 3). Tag both counts "[UNVERIFIED-COUNT]" until arXiv:2505.20411 is read.
+**33. [P2] `_normalize_action` whitespace-only normalization is acknowledged in code but absent from every risk list.** On real tool-call traces, Channel-3 pair extraction will be mostly noise; the final report's §10 failure modes and ADR-002 consequences should both name it (deep-read 07, FR-R8). (Fidelity angle: the "DPO-pair extractor, 7 unit tests" line in VISION_VALIDATION implies more readiness than the known-stub normalizer supports.)
+---
+## Confirmed faithful (for balance — no action)
+- The verbatim 25x sentence, feature-deletion paragraph, reward-hacking anecdotes (Python type-check cache, Java bytecode), and "agentic monitoring tools" are quoted accurately in `research/06`, `research/09`, ADR-010, and COMPOSER_RECIPE_MAPPING — re-verified against the fetched blog body this pass.
+- Dr.GRPO claims in `research/10` (length-norm removal, no std-norm, k1 estimator, DAPO overlong masking tried-and-rejected, Adam, single-epoch) — all verified verbatim against the Composer 2 report (deep-read 04 §4.1–4.2).
+- The k1-in-reward fold-then-baseline algebra and its Dr.GRPO-regime precondition are mathematically sound (deep-read 04 §3.2); one latent `num_iterations>1` guard is missing (engineering, not fidelity).
+- Channel 3's depth-1/flat and teacher-plurality-not-execution self-descriptions in the final report are accurate (deep-read 07, FR-R1/R2).
+- Channel-3 and tree-of-work provenance is honestly labeled "NOVEL — our addition, not part of Cursor's recipe" in `framework/composer-replication-framework.md`, VISION_VALIDATION, and the spike layer — the provenance boundary between "Composer's recipe" and "our additions" is consistently drawn. The residual issue is nearest-neighbor citation completeness (finding 21), not provenance dishonesty.
+- ADR-010's OPEN items (Gate-2 reachability, deleted_symbols emptiness) are honest self-flags that match what the sources show is an unsolved field-wide gap.
+- TRL claims (default `loss_type="dapo"`, `scale_rewards` handling, drift assertions in `make_dr_grpo_config`) verified against live docs (deep-read 08 §1).
+---
+## Severity totals
+- **P0: 5** (findings 1–5)
+- **P1: 19** (findings 6–24)
+- **P2: 9** (findings 25–33)

research/deepread/11-critic-design.md ADDED Viewed

	@@ -0,0 +1,180 @@

+# Design Critic: Architecture of the Envisioned Dataset-Generation Pipeline
+**Reviewer:** DESIGN CRITIC (critical-review pipeline, subagent)
+**Date:** 2026-06-10
+**Inputs:** `research/deepread/00-grounding.md` + `01`–`08`, `research/design-F1-systems-framing.md`, `research/design-F2-aws-datagen.md`, `docs/adrs/ADR-010`, and direct reads of `composer_replication/datagen/{env,substrates,validator,sandbox}.py`, `teacher_replay.py`, `safety/holdout.py`, `hint_generator.py`.
+**Scope:** Attack the ARCHITECTURE of the envisioned dataset pipeline — buy-vs-build, the S3 contract, tree-controller implementability, package placement, the minimal end-to-end pipeline, and components no design mentions.
+---
+## P0 — Foundational architecture defects (the design as drawn cannot produce the promised artifact)
+### 1. The tree-of-work's seed traces and its execution oracle operate on DISJOINT data — a chicken-and-egg no design resolves. **[P0]**
+F1's diagram and the final report (§5: *"ingest a seed trace, expand the divergence-gated tree across N models, execute every branch in a sandbox, grade leaves by `_grade()`"*) compose two components that cannot currently meet:
+- Seed traces come from `ClaudeCodeIngester` over `~/.claude/projects/**.jsonl` — sessions against the user's **local working directories** at arbitrary commits, with no Docker image, no pinned dependencies, no `FAIL_TO_PASS` labels, no `FeatureDeletionTask`.
+- The execution oracle (`FeatureDeletionEnv.step()`/`_grade()`) requires `task.broken_image` (a frozen container) and pre-identified test node IDs (`__post_init__` raises on empty `fail_to_pass`).
+You cannot `env.step()` a Claude Code trace action: there is no environment that materializes the repo state the trace was recorded against. The grounding map's Breaks 1–6 document this for "arbitrary OSS repo" but nobody applies it to the trace-replay-tree, where it is fatal: **every branch of the tree needs an executable sandbox, and Claude Code traces have none.** The only traces that are tree-expandable are traces collected *inside* FeatureDeletionEnv episodes — which do not exist because nothing in the repo runs an agent inside the env (see Finding 2).
+**Recommendation:** Invert the bootstrap order. Phase 1 of the tree must seed from **env-grounded traces** (agent rollouts on FeatureDeletionTask episodes, where reset state is reproducible by `task_id`), not Claude Code sessions. Demote Claude Code traces to (a) flat Channel-3 replay (DPO text pairs, no execution) and (b) SFT style data — uses that need no oracle. Document this split in a new ADR; the current F1 diagram is misleading and will misdirect the build.
+### 2. No agent rollout harness exists — the SFT corpus has no producer. **[P0]**
+The "SFT-first competence floor" (F1 §2, final report §5) reads `sft_corpus/` = "clean winning trajectories." Trace the producers: `teacher_replay` emits **single next-actions** per frozen state, not episodes. `env.reward_fn`'s fallback treats the whole completion as one `submit` (08 §5.1 calls this "a dead end for genuine multi-turn"). The tree controller is 0% built. **Nothing in the repo, built or designed, drives a multi-turn agent loop (LLM → tool call → `sandbox.exec` → observation → LLM) to completion and serializes the trajectory.** SWE-smith collected its 5k SFT trajectories with SWE-agent + Claude; SWE-Gym with OpenHands. Every design doc skips this component; F2's four stages (ingest/replay/validate/normalize) produce tasks and DPO pairs but *no SFT trajectories at all* — yet F2's stage (d) claims to write `corpus/sft/`.
+**Recommendation:** Add an explicit `rollout harness` component to the pipeline: adopt **mini-swe-agent or SWE-agent** (battle-tested, supports any API model) as the expert-trajectory collector against `FeatureDeletionEnv` tasks, with `_grade()==1.0` + `HackMonitor`-clean as the SFT admission filter. ~200–400 LOC of adapter, not a new agent. This is the single highest-priority build item — without it the minimal pipeline (Finding 16) cannot terminate in an SFT corpus.
+### 3. Divergence-gating is not computable from what the current components emit. **[P0]**
+The tree's economics depend entirely on the divergence gate (final report §3: it turns O(N^D) into "O(N · decision-points)"). The gate needs "pre-expansion divergence between sibling next-action distributions." But:
+- Teacher APIs (OpenRouter, Bedrock batch) return **text**, not distributions. Bedrock batch (`CreateModelInvocationJob`) does not return logprobs usable for a cross-model divergence measure (and cross-model token distributions live in different vocabularies anyway — KL between them is undefined without a common action space).
+- The only equality/divergence measure in the codebase is `_normalize_action()` — whitespace-collapse + lowercase. Deepread 07 (FR-R8, HIGH) already established this produces "mostly noise on real traces": semantically identical tool calls in different JSON formatting count as disagreement, so the gate would fire on nearly every node, **silently degrading the tree to the O(N^D) ungated cost the gate exists to prevent.** The cost-control mechanism and the known-broken normalizer are the same component.
+**Recommendation:** Define divergence over a **canonical action algebra**, not text: parse every candidate into `(tool_name, normalized_args)` (AST-normalize code args, path-normalize file args), and gate on (i) tool-name disagreement, (ii) arg-level edit distance over the canonical form, with an escalation tier that asks a cheap judge model only when (i)/(ii) is ambiguous. Build and *unit-test the gate's firing rate on 5 real traces* (expected: fires on <20% of steps) **before** writing `tree_controller.py`. The tool-call parser is the prerequisite, not a polish item.
+### 4. The tree requires a sandbox fork/snapshot primitive that neither the Sandbox API nor any design has. **[P0]**
+`tree_controller.py` (design: "apply each candidate action through `FeatureDeletionEnv.step()` to get a real next observation, branch again from the new state"). Branching N ways from a mid-episode state requires N **independent copies of the mutated working tree**. The `Sandbox` protocol has `boot/exec/run_tests/trajectory` — no `fork()`, no `snapshot()/restore()`. `DockerSandbox` boots one container per episode with `read_only=True` rootfs. The two realizable options are architecturally very different and nobody has chosen:
+1. **Replay-from-root:** re-boot the image and re-execute the action prefix for every branch — cost O(depth) sandbox-execs per node, multiplying the already-dominant sandbox cost (final report §10 names per-branch sandbox isolation "the throughput ceiling of the whole idea").
+2. **Filesystem snapshot fork:** overlayfs/`docker commit`/CRIU per node — fast but pins the design to one isolation backend and conflicts with gVisor/Kata choices in F2 stage (c).
+**Recommendation:** Extend the `Sandbox` protocol with `fork() -> Sandbox` *now*, implement it as overlayfs-upper-dir copy for `LocalSubprocessSandbox` and `docker commit`+boot for `DockerSandbox`, and measure fork latency in a spike before committing to tree depth >1. If fork costs >5s, the honest fallback is depth-1 trees (N candidate actions, each executed one step + graded by a cheap proxy) — which is most of the DPO value at a fraction of the machinery.
+### 5. Zero benchmark decontamination anywhere in the pipeline. **[P0]**
+The pipeline trains on SWE-bench-family substrates and the program will inevitably report SWE-bench Verified numbers (every comparison point in the research notes — DeepSWE 42.2%, SWE-smith 40.2%, Socratic-SWE 50.4% — is SWE-bench Verified). The substrates have **different and partial** decontamination policies: R2E-Gym decontaminates only its *Subset* against SWE-bench test repos (deepread 02 §6.2.2: "The full R2E-Gym (8,135 tasks) may overlap"); SWE-rebench spans 3,468 repos with no stated guarantee; the deepread 02 review explicitly flags research/06's "no contamination worry" as an OVERCLAIM. `HeldoutSplit` guards only the *internal* train/holdout partition — it has no concept of an external benchmark. No design doc (F1, F2, ADR-010) contains the word "decontamination."
+**Recommendation:** Add a mandatory `decontaminate(tasks, benchmark)` gate at stage (c1), enforced at three levels: (i) repo-level — drop any task whose `repo` appears in SWE-bench Lite/Verified/full or the chosen eval suite; (ii) instance-level — drop tasks whose `golden_diff` or `fail_to_pass` node IDs match an eval instance; (iii) record the decontamination manifest (benchmark name + version hash + drop count) in `manifests/run_id.json`. ~50 LOC + a pinned eval-instance index. This must exist before the *first* corpus is generated, because retro-filtering a published corpus is a credibility event.
+### 6. Buy-vs-build: the designs build the one component that is buyable, and skip the integration deepread 02 already identified. **[P0]**
+ADR-010 chose "Option A: invert OSS substrates" and rejected "Option B: greenfield repo scraping" — correct in 2026-05. But the user's new ask ("point at a repo") **is Option B**, and the design response (grounding map: "Broken-repo image builder — clone, `git apply -R`, scrub, build, push — unspecified, 0% built") is to hand-build exactly what `pip install swesmith` (MIT) ships: environment construction from arbitrary GitHub repos (~7 min human/repo, one image per repo = 500× storage win over per-task images), four bug-synthesis strategies, validation, issue-text generation, and `rp.get_container(task)` returning a booted container. Deepread 02 flagged this at HIGH ("the ADR does not evaluate it as a dependency") and noted SWE-smith's **PR Mirror** strategy — the exact gold-patch-reversion mechanic of `SweBenchAdapter` — produces the *best* training data in SWE-smith's own ablation (Table 5). The repo's core approach is validated by prior art it doesn't cite, and its missing half is shipped by a library it doesn't depend on.
+**Recommendation:** Commit a revised ADR: **[BUY]** swesmith for env-construction + bug synthesis on new repos; **[BUY]** SWE-smith 59k / R2E-Gym-Subset 4.6k / SWE-Gym 2.4k datasets through the existing `SweBenchAdapter` (02 confirms it works on SWE-smith instances unchanged); **[BUILD]** only the genuinely novel pieces — `DifficultyCurriculum`, `HackMonitor`/scrub, decontamination, the rollout harness (Finding 2), and (later, ablation-gated) coverage-guided deletion targeting. The "point-at-a-repo" feature becomes a ~100 LOC `SweSmithProfileAdapter` instead of an unspecified image-builder subsystem. Budget reality check from 02: SWE-smith built 50k tasks for **$1,360 + 20 human-hours**; any in-house builder must beat that.
+---
+## P1 — Significant design errors (buildable, but will mislead or bite)
+### 7. There are TWO unreconciled S3 contracts; the "6-prefix contract" is neither complete nor singular. **[P1]**
+F1 commits six prefixes (`sft_corpus/ dpo_pairs/ rl_task_pool/ divergence_pairs/ wm_tuples/ holdout/` under `runs/<run_id>/`). F2 commits a different layout (`traces/v1/run_id=<id>/`, `tasks/v1/...`, `replay/v1/...`, `task_grades/v1/...`, `corpus/v1/run_id=<id>/{sft,dpo}/`, `manifests/`). The grounding map (§2 step 8) pastes **both** into one list — so `corpus/v1/.../dpo/` and `dpo_pairs/` both exist for the same artifact, with different partitioning conventions (Hive `run_id=` vs path `runs/<id>/`), different versioning (F2 prefixes carry `v1`, F1 prefixes carry none), and two different buckets named across the docs (`amazon-sagemaker-...` in F1, `composer-datagen-...` in F2). Additionally: `divergence_pairs/` and `dpo_pairs/` describe one lineage (divergence-annotated nodes → extracted pairs) split across two prefixes, inviting drift; there is no prefix for quarantined/retired tasks (the curriculum produces them) nor for eval results.
+**Recommendation:** Write `s3_contract.py` FIRST and make both design docs subordinate to it. One layout: `s3://<bucket>/<contract_version>/run_id=<id>/<artifact>/` where artifact ∈ {traces, tasks, tasks_golden, replay, grades, corpus_sft, corpus_dpo, wm_tuples, holdout, quarantine, manifest.json} — every artifact versioned by the single top-level `contract_version`, every prefix owned by exactly one writer stage, `divergence_pairs` folded into `corpus_dpo` as a `provenance` column rather than a sibling prefix. Until this file exists, no AWS stage should be built (they would each encode one of the two divergent layouts).
+### 8. `golden_diff` leaks into the policy-visible manifest: `repr=False` is not serialization-exclusion. **[P1]**
+F2 (open questions) correctly demands `golden_diff` live in a deny-by-default `tasks/golden/` prefix. But stage (c1) writes "FeatureDeletionTask rows" to `tasks/v1/run_id=<id>/manifest.jsonl`, and any naive serializer (`dataclasses.asdict`, `json.dumps(vars(task))`) **includes `golden_diff`** — `field(repr=False)` only affects `__repr__`. The Batch validator children legitimately need the gold diff (Gate 4), but `rl_task_pool/` (read by the training env, whose prompt renderer carefully hides golden) would carry it in plaintext one `json.loads` away from any reward-hacking trajectory that reads its own task manifest. The safeguard exists in the prompt renderer and nowhere in the storage contract.
+**Recommendation:** In `s3_contract.py`, define two explicit serializers — `to_policy_row(task)` (drops `golden_diff` AND `deleted_symbols`) and `to_validator_row(task)` (full) — and make the policy-row writer the *only* code path that can populate `rl_task_pool/`. Add a unit test asserting `"golden_diff" not in json` for policy rows. Enforce the prefix split with bucket policy, not convention.
+### 9. The F2 architecture is a five-service orchestration for a pipeline that has never run once locally. **[P1]**
+F2 commits Glue 5.0 + Bedrock batch + EMR Serverless + AWS Batch + Step Functions + Lambda + CDK (~250 LOC IaC) before a single task has been validated end-to-end on a laptop (ADR-010's own post-review: the gates passed "against FakeSandbox materializers"; the Docker e2e is still `[~]`). Every stage's per-service rationale in F2 is individually sound, but the composition is premature: the corpus that matters first is O(10²–10³) tasks (SWE-smith's full 50k cost 20 human-hours and one machine), which is a **single-node workload**. The Step Functions DAG also has no idempotency/restart semantics, no run-level budget envelope (only `teacher_replay`'s in-process `max_total_usd`), and a 24h Bedrock-batch SLA in the middle of what should be a same-day iteration loop during development.
+**Recommendation:** Build the pipeline as **stage functions with a local driver first** (`python -m composer_replication.pipeline.run --stage all --tasks 200`), with S3 used only as a dumb artifact store via the `s3_contract.py` writers. Promote individual stages to managed services only when a measured bottleneck demands it, in this order: (1) AWS Batch for sandbox validation (the only genuinely parallel-heavy stage), (2) Bedrock batch when replay volume × cost crosses the 50%-discount break-even, (3) Step Functions only when >1 unattended run/week. Glue and EMR Serverless are likely never needed at this corpus scale — F2's own "live caveat" already concedes the ingester "is not intrinsically Spark-shaped."
+### 10. No secrets/PII gate at trace ingest — raw Claude Code sessions go to S3 verbatim. **[P1]**
+F2 stage (a) uploads `~/.claude/projects/**.jsonl` to `raw/claude_code/` and Parquet-izes them. Claude Code session files contain the user's local file contents, env-var echoes, API keys in tool outputs, internal hostnames, and proprietary code from *whatever repos the user worked on* — none of which passed any license, secrets, or PII filter. The copyleft filter (`is_redistributable`) applies only to SWE-substrate tasks, not to traces. The flywheel then trains on this and (per the publications/ directory) the corpus is intended to be shareable.
+**Recommendation:** Insert a mandatory scrub stage between ingest and storage: secrets scanning (gitleaks/trufflehog rule pack over message contents), path anonymization, and a per-session allowlist (only sessions from designated repos enter the corpus). Record scrub stats in the run manifest. This is ~1 day of work and belongs in `ClaudeCodeIngester` itself so no unscrubbbed `TraceState` can exist downstream.
+### 11. No canonical trajectory IR — three trace shapes are about to become five. **[P1]**
+Today: Claude Code JSONL → `TraceState` (messages + `student_action` as serialized block-list). Planned: Bedrock `.jsonl.out` rows, tree-controller branch trajectories, rollout-harness episodes (Finding 2), OpenHands/SWE-smith trajectories (ADR-002 v0.2). Each design names its own shape; `_normalize_action`'s whitespace hack is the symptom of the missing abstraction. Without one normalized trajectory schema, every pairwise consumer (DPO extractor, SFT formatter, wm_tuple writer, replay submitter) needs format-specific code, and the "trace format normalization" cost grows quadratically.
+**Recommendation:** Define a `CanonicalTrajectory` schema (list of `Turn{role, content, tool_calls: [(name, canonical_args)], tool_results, error_kind}`) in `datagen/schema.py` as the single internal currency; every ingester is a `X -> CanonicalTrajectory` adapter, and `extract_dpo_pairs`/SFT formatting/wm_tuples consume only it. This also gives the divergence gate (Finding 3) its action algebra for free.
+### 12. The flywheel has no cross-generation dedup — it will feed itself duplicates. **[P1]**
+The flywheel (F1: "improved student generates the next round's seed traces") loops the corpus into itself. Dedup today: data-juicer `document_deduplicator` is per-batch only; F2 adds Spark `dropDuplicates` *within one run's* normalize stage. Nothing dedups **across `run_id`s**, so generation N+1's SFT corpus will contain near-copies of generation N's winning trajectories (same tasks, similar solutions), compounding each cycle — a known self-training collapse accelerant. The held-out guard detects collapse after the fact; dedup prevents its cheapest cause.
+**Recommendation:** Add MinHash/LSH near-dup dedup keyed on `(task_id, canonical_action_sequence)` across all prior run manifests at corpus-write time, plus a per-task cap on retained trajectories (e.g., ≤K winners per task across all generations). Store the dedup index alongside `manifests/`.
+### 13. License handling is field-deep, not repo-deep — and absent for the "point-at-a-repo" path. **[P1]**
+`is_redistributable()` lowercases `instance["license_name"]` and substring-matches `("gpl","agpl","lgpl")`. For arbitrary repos there is no `license_name` (grounding Break 5); for SWE-smith instances the license lives in the toolkit's repo profiles, not the instance dict (02 §1: 2 GPLv3 + 4 LGPL repos in SWE-smith would need mapping); the substring check also misclassifies (e.g., "GPL-2.0-with-classpath-exception" semantics, dual-licensed repos) and ignores the difference between *using* a repo for training and *redistributing* derivative diffs (02 notes SWE-smith only claims the former). Repo-ingest is exactly where this must run, and no design places it there.
+**Recommendation:** At repo ingest, run SPDX detection (`licensee`/`askalono`) on the cloned tree, store the SPDX id + detection confidence on the task, and split policy into two explicit gates: `trainable(license)` (permissive + most copyleft OK for internal training) and `redistributable(license)` (permissive only) — applied at corpus-*publish* time, not generation time, so copyleft repos still contribute non-redistributed training signal. Keep the existing function as the redistribution gate, fix it to exact-SPDX matching.
+### 14. `wm_tuples/` ("ALL branches incl. failures") is an unbounded write path serving an ablation-gated consumer. **[P1]**
+The world-model head — the sole consumer of `wm_tuples/` — has, per deepread 06, **zero direct evidence** for its configuration ("no published paper has tested an auxiliary next-state-prediction objective during RL on a code policy") and is explicitly an ablation arm, not a premise. Yet the S3 contract gives it the highest-volume prefix in the system (every `env.step()` observation of every branch, including all failures — observations are full test logs/file contents), with no size estimate, no sampling policy, no retention/lifecycle rule in any design. The pipeline's storage architecture is load-bearing on its most speculative research bet.
+**Recommendation:** Make wm_tuple emission opt-in per run (`emit_wm_tuples: bool` in the run config, default off until the P4 ablation is scheduled), store observations as content-addressed blobs with dedup (test logs repeat massively), and attach an S3 lifecycle rule (expire after N days unless pinned by an ablation manifest). Do not let the typed-routing elegance ("failed branch is gold for the world model") force eager collection before the consumer exists.
+### 15. The "create harder tasks dynamically" half of the curriculum is absent from every pipeline design — yet it is the blog's actual mechanism. **[P1]**
+Deepread 01 confirms the verbatim Composer claim: "we both select for **and create** harder tasks dynamically throughout the run." The repo has SELECT-FOR (`DifficultyCurriculum`); the CREATE half (escalate deletion granularity function→file→feature, combine bugs, multi-feature targets minted *during* the run) appears in no design — `granularity` is hardcoded `"feature"`, and F1/F2's outer loop has no stage that takes curriculum state as *input* to task synthesis. Meanwhile SWE-smith's **Combine-Bugs** strategy (96.9% yield, zero cost, 15 median F2P — 02 §1) is precisely a CREATE-half mechanism sitting in the buyable toolkit.
+**Recommendation:** Add a `task_escalation` stage to the outer loop contract now (input: curriculum pass-rates per task; output: new combined/escalated task candidates into the validation queue), implemented first as SWE-smith Combine-Bugs over already-validated per-repo bugs. Even if deferred, the *stage boundary* must exist in the orchestration design, or the pipeline hard-codes a select-only curriculum and the eventual retrofit will break the Step Functions/driver topology.
+---
+## P2 — Real gaps, lower blast radius
+### 16. Where the pipeline should live: extend the monorepo with isolated extras; do NOT split a package. **[P2]**
+F1/F2 propose `composer_replication/pipeline/`, `composer_replication/datagen/aws/`, and root `infra/`. This is broadly right; a separate datagen package/repo would be wrong now: the shared dataclasses (`FeatureDeletionTask`, `TraceState`, `DPOPair`, future `CanonicalTrajectory`) ARE the contract, and splitting them across repos forces schema version-pinning with zero external consumers. The risks to manage are dependency bleed (boto3/pyspark/docker must not enter the training image) and entrypoint importability.
+**Recommendation:** (a) `composer_replication/datagen/` stays the pure library (schema, env, validator, monitor, curriculum — no cloud deps); (b) new `composer_replication/pipeline/` holds stage drivers + `s3_contract.py`, all cloud/Spark imports lazy and gated behind a `[pipeline]` extra; (c) `infra/` at repo root for IaC only; (d) CI check that `import composer_replication.datagen` succeeds with no extras installed. Revisit a package split only when a second project actually consumes the datagen library.
+### 17. No state/node ID contract for tree-generated states. **[P2]**
+`state_id = f"{path.stem}::{idx:04d}"` is unique only within one session file; tree-controller branches, rollout-harness episodes, and Bedrock `recordId` joins all need globally unique, lineage-encoding IDs (parent pointer, branch index, run_id), and `recordId==state_id` is named "the universal join key" without ever specifying the tree extension.
+**Recommendation:** Commit `node_id = {run_id}/{trace_id}/{path-from-root as branch indices}` in `s3_contract.py`; parent derivable by truncation, collision-free by construction.
+### 18. No dataset versioning, cards, or reproducibility pins. **[P2]**
+`manifests/run_id.json` carries counts/cost/lineage but no: substrate dataset revision hashes (HF datasets mutate — SWE-smith grew from 50,137 to 59,136 rows post-publication, per 02), swesmith/toolchain versions, Docker image digests (tags like `:latest` in `image_for()` are mutable!), prompt-template hashes, or a generated dataset card (composition, license mix, decontamination statement, known limitations). Irreproducible corpora are unpublishable and undebuggable.
+**Recommendation:** Pin image **digests** not tags in `FeatureDeletionTask.broken_image`; record substrate `(dataset_id, revision)` and generator git SHA in the manifest; auto-generate a dataset card per run from the manifest (the HF `datasets` card template is fine). ~100 LOC total.
+### 19. DiLoCo rendezvous traffic co-located in the dataset bucket. **[P2]**
+Both F1 and F2 put `diloco/rendezvous/` (hot, high-frequency, delete-heavy training sync) in the same bucket as the immutable corpus. This entangles IAM (training nodes get write access to a bucket containing `tasks/golden/`), lifecycle policies, and cost attribution.
+**Recommendation:** Separate bucket (or at minimum a separate top-level prefix with its own bucket policy statement and aggressive expiry), keeping the dataset bucket append-only for everything except `quarantine/`.
+### 20. No corpus-quality acceptance metric — the pipeline has no definition of "done/good." **[P2]**
+Every stage has pass/fail gates for *tasks*, but nothing measures whether the resulting *corpus* is any good before it consumes GPU budget. SWE-Gym/SWE-smith both validated with small SFT probes (491 and 5,016 trajectories respectively, measurable lift on held-out).
+**Recommendation:** Define the corpus acceptance test as part of the pipeline: SFT a small model (e.g., Qwen3-Coder-7B, LoRA) on each new corpus generation and require a measurable delta on the internal holdout (and decontaminated SWE-bench subset) before the corpus is promoted to `status=accepted` in its manifest. This is the dataset analogue of the trainer's `HeldOutGuard`.
+### 21. Orchestration restart semantics and budget envelopes are unspecified. **[P2]**
+F2's Step Functions design has `.sync` integrations but no statement of stage idempotency (re-running stage (b) after partial Bedrock job failure double-writes `replay/`?), no run-level cost ceiling (the only budget control in the system is `replay_trace`'s in-process `max_total_usd=5.0`), and no poison-task quarantine path at the orchestration level (a task that wedges Batch children retries forever).
+**Recommendation:** Make every stage write-once per `(run_id, stage, attempt)` with a completion marker object; add a `budget_usd` field to the run manifest enforced by the driver before each paid stage; route repeatedly-failing array indices to `quarantine/` after `retryStrategy` exhaustion.
+---
+## The MINIMAL pipeline (one pointed-at repo → real SFT corpus), and what the full vision adds in what order
+**Minimal (Stage 0 — local, no new AWS services, ~1–2 weeks):**
+1. `pip install swesmith`; build the repo profile (env image, test parser) — *buy*, ~7 min human + automation [Finding 6].
+2. Synthesize candidate tasks: PR Mirror first (best data per SWE-smith Table 5 = the repo's own gold-patch-reversion mechanic), procedural/Combine as volume fallback — *buy*.
+3. SPDX license gate at clone + benchmark decontamination check (repo ∉ eval suite) — *build*, ~80 LOC [Findings 5, 13].
+4. 4-gate `validate_task()` in `DockerSandbox` (exists) + `scrub_tree` — *have*; wire the swesmith container into `Sandbox.boot` — ~50 LOC.
+5. Expert trajectory collection: mini-swe-agent/SWE-agent + a frontier model over validated tasks, `$cap` per task — *adopt + adapt*, ~200–400 LOC [Finding 2]. **This is the critical missing component.**
+6. Admission filter: `_grade()==1.0` + `HackMonitor` clean + `pass_to_pass` guard — *have*.
+7. Format to messages-schema SFT rows via `CanonicalTrajectory`; MinHash dedup; `HeldoutSplit` with `check_content=True`; write Parquet + manifest + dataset card via `s3_contract.py` — *build*, ~250 LOC [Findings 7, 11, 12, 18].
+Total new code ≈ 600–900 LOC plus two adopted dependencies. Output: a real, decontaminated, license-clean, deduped, carded SFT corpus from one repo — and, as a free byproduct, the env-grounded traces the tree needs (Finding 1).
+**Then, strictly in this order:**
+- **Stage 1:** AWS Batch array validation + Bedrock batch replay when volume justifies (Finding 9); DPO channel on env-grounded traces after the tool-call parser fixes `_normalize_action` (Finding 3).
+- **Stage 2:** Depth-1 tree (N candidates, one env-step each, oracle-graded) — requires `Sandbox.fork()` spike (Finding 4); divergence-gated depth>1 only after the gate's firing rate is measured.
+- **Stage 3:** Curriculum CREATE-half via Combine-Bugs (Finding 15); flywheel with cross-generation dedup (Finding 12); `wm_tuples` emission only when the P4 ablation is scheduled (Finding 14).
+- **Stage 4:** Step Functions/Argo orchestration once runs are routine (Finding 21).
+---
+## Severity tally
+| Severity | Count | Findings |
+|---|---|---|
+| **P0** | 6 | 1 (trace/oracle disjointness), 2 (no rollout harness), 3 (divergence gate uncomputable), 4 (no sandbox fork), 5 (no decontamination), 6 (buy-vs-build inversion) |
+| **P1** | 9 | 7 (two S3 contracts), 8 (golden_diff serialization leak), 9 (premature 5-service orchestration), 10 (no secrets/PII gate), 11 (no trajectory IR), 12 (no cross-generation dedup), 13 (license gate too shallow), 14 (wm_tuples unbounded/speculative), 15 (CREATE-half absent) |
+| **P2** | 6 | 16 (package placement), 17 (node ID contract), 18 (versioning/cards), 19 (bucket co-location), 20 (corpus acceptance metric), 21 (restart/budget semantics) |

research/deepread/12-verified-findings.md ADDED Viewed

	@@ -0,0 +1,131 @@

+# Verified Findings — Consolidated Critical Review (Verification Agent)
+**Date:** 2026-06-09
+**Inputs verified:** `research/deepread/10-critic-fidelity.md`, `research/deepread/11-critic-design.md`, cross-checked against repo code, the deep-reads (`00`–`08`), and the fetched Composer 2.5 blog body (`research/notes/introducing-composer-25-cursor.md`).
+> **MISSING INPUT:** `research/deepread/09-critic-feasibility.md` did not exist at verification time
+> (polled for ~12 minutes; directory contains only 00–08, 10, 11). The feasibility critic's findings
+> are therefore **unverified and not consolidated here**. If that file appears later, a delta
+> verification pass is required before its P0s are acted on.
+**Method:** every P0 (and every P1 naming a specific file/claim) was re-checked against the actual
+repo file at the cited line, the verbatim blog text, or the deep-read's primary-source quotes.
+Verdicts: **CONFIRMED** / **REFUTED** / **OVERSTATED** (true core, inflated framing).
+---
+## Part 1 — Verification ledger
+### Fidelity critic (10) — P0s
+| # | Claim | Verdict | Evidence |
+|---|---|---|---|
+| F-1 | Benchmark numbers (69.3%, Terminal-Bench 2.0 parity) asserted as Cursor-stated; in no primary source | **CONFIRMED** | `research/01-composer-2.5.md:63-64` asserts "it hit 69.3%" under "Cursor claims…"; fetched blog body contains zero benchmark numbers (only "not well captured by existing benchmarks", note line 26). Header audit note flags but body asserts unhedged. |
+| F-2 | "Feature Deletion + 24 other (unnamed) generators" — count invented | **CONFIRMED** | Blog verbatim (note line 50): "a range of approaches… one synthetic approach is feature deletion" — no count. `docs/COMPOSER_RECIPE_MAPPING.md:75`, `research/06:330` ("The other ~24 generators"), `research/09:23` all carry the fabricated 24. |
+| F-3 | SDPO declared "mathematically the same" as Composer's mechanism — unsupported identification | **CONFIRMED** | `docs/COMPOSER_RECIPE_MAPPING.md:25` ("mathematically the same"), `:151` ("direct formalization"); `opsd.py:13-14` ("formalizes the same loss as Composer 2.5's…"). Blog cites the papers only as "For more background on this approach see" (note lines 67/148). Deep-read 03 (lines 93, 252, 280-281, 308): SDPO loss is full-rollout with feedback-in-prefix; repo's design is turn-localized hint-splice — neither SDPO nor confirmed Composer. |
+| F-4 | Streaming DiLoCo misattributed (wrong authors AND wrong title on arXiv:2501.18512) | **CONFIRMED** | `composer_replication/diloco/__init__.py:8` "Streaming DiLoCo (Liu et al. 2025)"; `research/design-F4-decoupled-diloco-s3.md:109` attaches "Eager Updates…" title to 2501.18512. Deep-read 05 F4: 2501.18512 = Douillard et al., "Streaming DiLoCo with overlapping communication"; Eager Updates = arXiv:2502.12996 (Kale et al.). |
+| F-5 | CWM cited as licensing train-on-all for an RL-time aux head; CWM does it in a separate mid-training stage | **CONFIRMED** | Final report line 33: "crucially training *on all* trajectories for the world-model head, reserving success-filtering only for the RL reward [13]" (+ ref line 258). Deep-read 06 quotes CWM §2.2 verbatim ("we do not filter trajectories…") and documents the three-phase pipeline: world-modeling is in base weights pre-RL, not an aux head riding policy gradients. |
+### Fidelity critic (10) — P1s naming specific files/claims
+| # | Claim | Verdict | Evidence |
+|---|---|---|---|
+| F-6 | CWM "65.8%" cited without test-time-scaling qualifier | **CONFIRMED** | Final report line 33 cites bare "65.8% on SWE-bench Verified"; CWM abstract per deep-read 06: "65.8% … (with test-time scaling)". |
+| F-7 | Chain-of-World (2603.03195) is robotics VLA, used for SWE | **CONFIRMED** (via deep-read 06 finding 2.2) | Deep-read 06 exec summary: "robotics/embodied VLA paper, not an SWE paper". |
+| F-8 | "85% post-training compute" stated as fact in research/01 body | **CONFIRMED** | `research/01-composer-2.5.md:14` asserts it unhedged; header (line 5) flags it as "community consensus, not Cursor-stated" — body never inline-tags. |
+| F-9 | Implemented "feature deletion" is gold-patch reversion, not the blog's delete-from-functional-codebase mechanism | **CONFIRMED** | Blog verbatim (note line 50): agent "asked to delete code and files… codebase remains functional". `substrates.py` SweBenchAdapter reverts gold patches of pre-labeled instances; mapping doc line ~42 paraphrase drops the agentic deleter. "Point at a repo" appears nowhere in the blog. |
+| F-10 | Difficulty curriculum: only SELECT half built; CREATE half 0% | **CONFIRMED** | `curriculum.py` header quotes the blog's "select for and create"; `substrates.py:80` hardcodes `granularity="feature"`; no task-escalation stage anywhere. (= Design #15; merged below.) |
+| F-12 | Mapping doc states `KL(teacher‖student)` — direction unsupported by blog, opposite of SDPO Eq. 1 | **CONFIRMED** | `COMPOSER_RECIPE_MAPPING.md` (~line 19): "KL( teacher_logits_at_turn_t \|\| student_logits_at_turn_t )". Blog: "moves the student's token probabilities toward the teacher's" — directionless. SDPO Eq. 1 per deep-read 03 line 308: `KL(π_θ ‖ stopgrad(·))`, student first. |
+| F-13 | `opsd.py` β-convention docstring inverted vs upstream | **CONFIRMED** | `opsd.py:55-59` labels β=0 "reverse KL"; OPSD README per deep-read 03 (§2.2, line 168 "INVERTED", table line 497): "Beta=0 means forward KL". Code numerically correct; labels swapped. |
+| F-14 | SDPO channel lacks the paper's EMA/trust-region teacher regularization | **CONFIRMED** | Deep-read 03 lines 32-34, 81-82: non-regularized live teacher diverges (Table 4: 36.1% vs 50.6%); repo teacher = live weights, no EMA. |
+| F-16 | "$0.98/trace verified economic floor" hides trace definition; real sessions ~2 OOM more | **CONFIRMED** | `teacher_replay.py:7-8` unqualified docstring; `VISION_VALIDATION.md:63,76`; `ADR-002:60` (125→2,830 tool-use messages/session); deep-read 07 FR-R5: ~$73/full session. (VISION_VALIDATION's Objection 2 partially self-flags; the docstring does not.) |
+| F-17 | "$64 ungated tree" is a flat N=8×T=1000 extrapolation, not a tree | **CONFIRMED** | `research/05:256` ($0.008×1000×8 flat math); `comparisons.md:33` and final report line 66 label it the branching/"ungated" cost. True tree is O(N^D), strictly worse. |
+| F-18 | `kl_in_reward.py`: verl k1 "*only* reverse-KL option" overclaim | **CONFIRMED** | `kl_in_reward.py:12` verbatim; deep-read 04 §4.3: verl also supports `kl_penalty="low_var_kl"` (k3-family). |
+| F-20 | GSPO preset inherits GRPO-scale clipping — 2 OOM off the paper | **CONFIRMED** | `composer_trainer.py` "gspo" preset (~line 814) sets no epsilon → TRL default ~0.2; GSPO paper §5.1: 3e-4/4e-4. Companion confirmed: "cispo" preset omits explicit `beta`. |
+| F-21 | rStar named "closest precedent" (misread granularity); Tree-GRPO + SWE-Search uncited | **CONFIRMED** | `framework/composer-replication-framework.md:17-18` ("Closest precedent: rStar-Math… open territory"); grep: zero hits for Tree-GRPO/2509.21240/SWE-Search/2410.20285 in `research/05`. |
+| F-22 | "Nine-tenths is reuse" conflates design-reuse with build status | **CONFIRMED** | Final report line 133 ("the substrate already exists — roughly nine-tenths of it"); grounding doc Claim 5: tree, wm-head, pipeline/, infra/ are 0% built. |
+| F-23 | "Parameter isolation eliminates interference" overclaims DART; aux-head evidence wholly analogical | **CONFIRMED** | Deep-read 06: 2602.00994 shows interference persists on isolated LoRA modules; "evidentiary gap is total for the specific proposed configuration". |
+| F-24 | VeRL "first-class agentic RL" — async path is experimental | **CONFIRMED** | Deep-read 08 lines 66-73: `fully_async_policy`/`transfer_queue` under `verl/experimental`; "slightly overclaims". |
+P2s (F-25…F-33) spot-checked where cheap; no refutations found. F-33 (`_normalize_action` absent from risk lists) merges into Design #3.
+### Design critic (11) — P0s
+| # | Claim | Verdict | Evidence |
+|---|---|---|---|
+| D-1 | Tree seed traces (Claude Code) and execution oracle (FeatureDeletionEnv) operate on disjoint data | **CONFIRMED** | `design-F1:18,70-74` seeds the tree from `ingestion/claude_code.py` TraceStates (local `~/.claude/projects` JSONL — no image, no F2P); `env.py:59` requires `task.broken_image`; `schema.py:35-38` `__post_init__` raises on empty `fail_to_pass`. No design reconciles them. |
+| D-2 | No agent rollout harness exists — SFT corpus has no producer | **CONFIRMED** (minor nuance) | Deep-read 08 §5.1: `reward_fn` fallback = single submit, "dead end for genuine multi-turn"; `teacher_replay` emits single next-actions; F1 line 136 names "tree controller" (0% built) as the sft_corpus producer; F2 stage (d) writes `corpus/sft` with no trajectory-generating stage upstream. Nuance: the tree-controller *design* does call `env.step()`, but expands counterfactual branches from (unexecutable, per D-1) frozen traces — it is not an episode-completing rollout loop. |
+| D-3 | Divergence gating not computable from what components emit | **CONFIRMED** | `teacher_replay.py:195-203`: `_normalize_action` = whitespace-collapse + lowercase, docstring self-admits skeleton status; teachers return text (no cross-model distributions); deep-read 07 FR-R8 (HIGH): "mostly noise on real traces" → gate fires everywhere → silent O(N^D). |
+| D-4 | Sandbox lacks fork/snapshot primitive the tree requires | **CONFIRMED** | `sandbox.py:99-106` Protocol = boot/exec/run_tests/trajectory/is_command_allowed only; grep "fork\|snapshot" → only "fork-bomb guard"; `docker_sandbox.py:15,157` `read_only=True` rootfs. |
+| D-5 | Zero benchmark decontamination anywhere | **CONFIRMED** | grep "decontamin/contamination" across `composer_replication/`, `docs/`, F1, F2, ADR-010 → zero hits; `holdout.py:171` `HeldoutSplit` is internal-split-only; deep-read 02 flags R2E-Gym full-set overlap risk. |
+| D-6 | Buy-vs-build inversion: repo hand-builds what swesmith ships; PR-Mirror validation uncited | **CONFIRMED** | No `swesmith` in `pyproject.toml` or ADR-010 (grep empty); deep-read 02 lines 230, 251, 306: PR Mirror ≡ repo's gold-patch reversion, best training data per SWE-smith Table 5, "$1,360 + 20 human-hours" verified — "neither research/06 nor ADR-010 cite this result". |
+### Design critic (11) — P1s naming specific files/claims
+| # | Claim | Verdict | Evidence |
+|---|---|---|---|
+| D-7 | Two unreconciled S3 contracts | **CONFIRMED** (one sub-point softened) | F1 lines 86-93 (`runs/<id>/sft_corpus/ dpo_pairs/ …`) vs F2 lines 133-143 (`traces/v1/run_id=<id>`, `corpus/v1/...{sft,dpo}`); grounding step 8 (lines 91-93) pastes both — `corpus/v1/.../dpo/` AND `dpo_pairs/` coexist. Nuance: F2:129 offers "reuse the existing amazon-sagemaker-… **or** a dedicated composer-datagen-…" — two bucket *options*, not a hard contradiction. |
+| D-8 | `golden_diff` leaks via naive serialization; `repr=False` ≠ exclusion | **CONFIRMED** | `schema.py:29` `field(default="", repr=False)`; F2:100,135 writes "FeatureDeletionTask rows" to `tasks/v1/.../manifest.jsonl`; `dataclasses.asdict`/`vars()` include repr=False fields. Only `env.py:73` (prompt renderer) hides it. |
+| D-9 | Five-service AWS orchestration before one local end-to-end run | **CONFIRMED** | F2 names Glue/EMR Serverless/AWS Batch/Bedrock batch/Step Functions/Lambda (46 hits); ADR-010:99-103 concedes gates passed "against FakeSandbox", Docker e2e `[~]`. |
+| D-10 | No secrets/PII gate at trace ingest | **CONFIRMED** | F2:29 uploads raw `~/.claude/projects/**.jsonl` to `raw/claude_code/`; grep secrets/PII/gitleaks/trufflehog in F2 → zero; `scrub_tree` strips caches, not secrets; `is_redistributable` applies to tasks only. |
+| D-12 | No cross-generation dedup in the flywheel | **CONFIRMED** | F2:114-118: data-juicer `document_deduplicator` per-batch + Spark `dropDuplicates` within one run; `parent_run_id` (F2:151) threads lineage but nothing dedups across runs. |
+| D-13 | License gate is field-deep substring matching, absent for repo-ingest | **CONFIRMED** | `substrates.py:86-90`: `lic = task.upstream_license.lower(); return not any(c in lic for c in _COPYLEFT)`; no SPDX detection at clone path; trainable-vs-redistributable distinction absent. |
+| D-14 | `wm_tuples/` highest-volume prefix serving an ablation-gated, zero-direct-evidence consumer | **CONFIRMED** | F1:90 "wm_tuples/ ← … ALL branches"; deep-read 06: "no published paper has tested an auxiliary next-state-prediction objective during RL on a code policy"; no sampling/lifecycle policy in any design. |
+| D-15 | CREATE-half of curriculum absent from every pipeline design | **CONFIRMED** | Same evidence as F-10; merged below as one finding. |
+| D-11 | No canonical trajectory IR (3 trace shapes → 5) | **CONFIRMED** | `_normalize_action` stub is the symptom; Claude Code JSONL/TraceState, Bedrock `.jsonl.out`, planned tree + rollout + OpenHands shapes each named in separate docs with no shared schema (no `datagen/schema.py` trajectory type exists). |
+D-16…D-21 (P2): spot-checked — `state_id` format confirmed (`claude_code.py:181`), `diloco_rendezvous/` co-located in F1's run layout (F1:93). No refutations.
+**Verification summary: 0 REFUTED, 2 minor nuances (D-2, D-7), everything else CONFIRMED.** Both critics were accurate against source; no fabrication detected.
+---
+## Part 2 — FINAL consolidated finding list (ranked by severity × confidence, deduplicated)
+### Tier 1 — P0, high confidence, load-bearing for the dataset-pipeline build
+1. **[V1] The Channel-2 identity claim is wrong: SDPO is *background*, not Composer's mechanism — and the repo's implementation is a third design.** Cursor's blog cites OPSD/SDPO only as "for more background"; SDPO's published loss is full-rollout with feedback-in-prefix and a *regularized* teacher; the repo's turn-localized hint-splice (live-weights teacher, no EMA) is its own blog-inspired variant. Fix `COMPOSER_RECIPE_MAPPING.md:25,151`, `opsd.py:13-14`; also fix the KL-direction line (F-12), the inverted β docstring (F-13), and document/implement teacher regularization (F-14). *(Fidelity F-3 + F-12 + F-13 + F-14.)*
+2. **[V2] The envisioned tree-of-work pipeline cannot execute as drawn: seed traces and the execution oracle are disjoint, no rollout harness exists to produce the SFT corpus, the divergence gate is uncomputable from text + a whitespace normalizer, and the Sandbox has no fork primitive.** Four independently confirmed structural breaks that compose into one verdict: Phase 1 must seed from env-grounded rollouts (adopt mini-swe-agent/SWE-agent as the trajectory collector), demote Claude Code traces to flat Channel-3/SFT-style uses, build a canonical tool-call action algebra before any `tree_controller.py`, and spike `Sandbox.fork()` before committing to depth>1. *(Design D-1 + D-2 + D-3 + D-4 + fidelity F-33.)*
+3. **[V3] Zero benchmark decontamination anywhere in code or designs.** The pipeline trains on SWE-bench-family substrates and will be scored on SWE-bench Verified; substrates have partial/divergent decontamination; `HeldoutSplit` is internal-only; the word does not appear in F1/F2/ADR-010. Must exist before the first corpus is generated. *(Design D-5.)*
+4. **[V4] Buy-vs-build inversion: the repo plans to hand-build what `pip install swesmith` ships, while its own core mechanic (gold-patch reversion) is SWE-smith's PR Mirror — validated as the *best* training data by an ablation the repo never cites.** The user's "point-at-a-repo" ask is ~100 LOC of swesmith adapter, not an unspecified image-builder subsystem ($1,360 + 20 human-hours is the budget bar to beat). Relatedly, the implemented "feature deletion" is the inversion *analogue*, not the blog's delete-from-functional-codebase mechanism — and downstream docs should say so. *(Design D-6 + fidelity F-9.)*
+### Tier 2 — P0 fidelity corrections (high confidence, citation/claim integrity)
+5. **[V5] Fabricated quantities circulating as Cursor-stated facts:** (a) benchmark numbers 69.3%/Terminal-Bench parity appear in no primary source (`research/01:63-64`); (b) "24 other generators" is a back-formation from the 25x *task* multiplier (`COMPOSER_RECIPE_MAPPING.md:75`, `research/06:330`, `research/09:23`); (c) "85% post-training compute" asserted unhedged in the research/01 body. Strike or inline-tag all three. *(Fidelity F-1 + F-2 + F-8.)*
+6. **[V6] World-model aux-head's load-bearing citation is a misread: CWM trains-on-all in a dedicated *mid-training* stage, not as an aux head riding RL policy gradients; 65.8% requires the test-time-scaling qualifier; Chain-of-World is robotics; "parameter isolation eliminates interference" overclaims DART.** The exact proposed configuration has zero published evidence — keep it an ablation arm and gate `wm_tuples/` emission (highest-volume prefix, most speculative consumer) behind the scheduled ablation. *(Fidelity F-5 + F-6 + F-7 + F-23 + design D-14.)*
+7. **[V7] Streaming DiLoCo citation is doubly wrong** — arXiv:2501.18512 is Douillard et al., "Streaming DiLoCo with overlapping communication"; "Eager Updates…" is arXiv:2502.12996 (Kale et al.); "Liu et al." is the Async Local-SGD paper. Fix `diloco/__init__.py:8` and `design-F4:109`. Low blast radius, unambiguous. *(Fidelity F-4.)*
+### Tier 3 — P1, high confidence (will bite during the build)
+8. **[V8] Two unreconciled S3 contracts (F1 vs F2) + `golden_diff` serialization leak.** Write `s3_contract.py` first with one layout and two explicit serializers (`to_policy_row` drops `golden_diff`/`deleted_symbols`; unit-test the absence); fold `divergence_pairs` into `corpus_dpo` provenance; separate the DiLoCo rendezvous prefix/bucket. *(Design D-7 + D-8 + D-19.)*
+9. **[V9] No secrets/PII scrub at Claude Code trace ingest** — raw sessions (file contents, keys in tool outputs, proprietary code) go to S3 verbatim; the only license filter applies to tasks, and it is a lowercase substring match with no SPDX detection and no trainable-vs-redistributable split for the repo-ingest path. *(Design D-10 + D-13.)*
+10. **[V10] The blog's CREATE-half of the dynamic curriculum ("select for **and create** harder tasks") is absent from code and from every pipeline design** — `granularity` hardcoded `"feature"`, no escalation stage in the outer loop. Cheapest first implementation: SWE-smith Combine-Bugs (96.9% yield) as a `task_escalation` stage; at minimum reserve the stage boundary in the orchestration contract. *(Fidelity F-10 ≡ design D-15.)*
+11. **[V11] Cost claims mislabeled in ways that distort planning:** "$0.98/trace verified floor" is a 50-step synthetic-state benchmark (real sessions ~$70–80 flat at 125–2,830 steps/session); "$64 ungated tree" is a flat 8-teacher×1000-step extrapolation, not a tree (true tree is O(N^D)). Fix `teacher_replay.py:7-8`, `VISION_VALIDATION.md`, `comparisons.md:33`, final report §10/§3. *(Fidelity F-16 + F-17.)*
+12. **[V12] Premature five-service AWS orchestration + flywheel hygiene gaps:** F2 commits Glue/EMR/Batch/Bedrock-batch/Step-Functions before one local e2e run (ADR-010 gates passed against FakeSandbox); no cross-generation dedup (self-training collapse accelerant); no canonical trajectory IR; no restart/budget semantics. Build a local stage-driver first; add MinHash cross-run dedup + `CanonicalTrajectory` schema. *(Design D-9 + D-12 + D-11 + D-21.)*
+### Tier 4 — P1/P2 precision fixes (confirmed, lower blast radius)
+13. **[V13] Preset/config drift vs papers:** GSPO preset missing 3e-4/4e-4 epsilons (operationally not GSPO); CISPO preset should set `beta=0.0` explicitly; `kl_in_reward.py:12` "only reverse-KL option" → "default/recommended"; Comedy-of-Estimators citation rests on abstract-only. *(Fidelity F-20 + F-18 + F-19.)*
+14. **[V14] Provenance/novelty bookkeeping:** cite Tree-GRPO (2509.21240) and SWE-Search (2410.20285) as nearest neighbors and fix the rStar granularity misread (`framework/…:17`, `research/05`); re-scope "nine-tenths reuse" to the recipe-replication layer (tree/wm-head/pipeline are 0% built); note verl async path is experimental. *(Fidelity F-21 + F-22 + F-24.)*
+15. **[V15] Remaining P2 wording fixes from the fidelity critic (F-25…F-33)** — stage attribution of Muon, deleter-unknown framing, 25x non-convertibility, grounding-doc paraphrase-as-quote, DiLoCo FP4/H-default details, Foresight@k "(we define this)", SWE-rebench unverified counts — all consistent with the deep-reads; apply as a batch documentation pass. Plus design P2s: node-ID contract, image-digest pinning + dataset cards, corpus acceptance probe (small-model SFT delta before GPU spend).
+---
+## Coverage note
+The design critic's minimal-pipeline proposal (swesmith + SPDX/decontamination gates + SWE-agent
+rollout harness + canonical trajectory schema + `s3_contract.py`, ≈600–900 LOC) is consistent with
+every confirmed finding above and is the recommended Stage-0 build order. The feasibility critic
+(09) was never produced; its absence means cost/timeline feasibility claims in the designs have had
+only the fidelity-angle checks (V11) — treat any standalone feasibility assertions as unreviewed.

research/deepread/13-synthesis-architecture.md ADDED Viewed

	@@ -0,0 +1,197 @@

+# Synthesis: Critical-Review Verdict + The Dataset-Generation Pipeline Architecture
+> **Date:** 2026-06-09. **Inputs:** the 8 deep-reads (`01`–`08`), grounding map (`00`),
+> verified findings (`12`), design-critic minimal pipeline (`11` §MINIMAL).
+> **Provenance:** 8 source-cluster readers re-fetched and re-read every primary source
+> (Composer 2.5 blog verbatim, Composer 2 techreport arXiv:2603.24477 full HTML,
+> SWE-smith/SWE-Gym/R2E-Gym/SWE-bench full texts, SDPO/OPSD, Dr.GRPO/DAPO/GSPO/CISPO +
+> Comedy-of-Estimators, DiLoCo/Streaming-DiLoCo, MuZero/Dreamer/CWM + the anti-evidence
+> cluster, LATS/ToT/rStar/Tree-GRPO/SWE-Search/Symphony/Socratic-SWE, TRL/verl/SkyRL live
+> docs, SWE-MiniSandbox); 2 adversarial critics' findings were then independently
+> verified line-by-line — **0 refuted**.
+---
+## Part A — The critical-review verdict in one page
+**The recipe-replication layer (Channels 1+2 mechanics, env, safeguards) is solid and
+mostly faithful. The *story we tell about it* has confirmed fidelity defects, and the
+*envisioned dataset pipeline* has four structural breaks that would have made it
+unbuildable as drawn.** The good news: every break has a cheap, evidence-backed fix, and
+the biggest fix is a *buy* (SWE-smith), not a build.
+### What survived adversarial review intact
+- Feature-deletion-by-gold-patch-reversion as the task mechanic — **independently
+  validated** by SWE-smith's ablation (its "PR Mirror" strategy *is* our mechanic and
+  produces the **best** training data of its five strategies, Table 5).
+- The execution-oracle reward (`_grade()` masked pass-fraction), the 4-gate validator,
+  `scrub_tree` as primary anti-hack control, fractional-credit curriculum (SELECT half).
+- The k1-in-reward KL fix (Composer-2 §4.1 verbatim confirms k1 = −log r in reward,
+  citing the same variance argument) and the behavior-rewards bank (§4.2 verbatim).
+- The DiLoCo-over-S3 substrate (math verified against torchft; live-S3 validated).
+- The honest-provenance discipline itself: Channel 3 + tree are OUR additions, not
+  Cursor's — re-confirmed.
+### The four structural breaks in the envisioned pipeline (all CONFIRMED)
+1. **Seed-trace/oracle disjointness.** The tree was drawn growing off Claude Code traces,
+   but those traces have no executable environment (no `broken_image`, no
+   `fail_to_pass`) — `FeatureDeletionEnv` literally cannot `reset()` on them. The tree
+   must seed from **env-grounded rollouts**, which don't exist yet because…
+2. **No rollout harness.** Nothing in the repo runs an agent loop against
+   `FeatureDeletionEnv` to completion. The SFT corpus has **no producer**. This is the
+   single highest-priority build item.
+3. **Divergence gate uncomputable.** `_normalize_action` is a whitespace-collapse stub;
+   teachers return free text; there is no tool-call action algebra to compare. Ungated,
+   the tree is O(N^D).
+4. **No `Sandbox.fork()`.** Branching from a mid-trajectory state requires state
+   cloning the Sandbox protocol doesn't have.
+### The five worst fidelity defects (all CONFIRMED, now to be corrected)
+1. "SDPO is mathematically the same as Composer's mechanism" — **wrong**; Cursor cites
+   SDPO/OPSD only as *background*; SDPO's published loss is full-rollout with
+   feedback-in-prefix + EMA-regularized teacher; ours is a turn-localized hint-splice
+   with a live teacher. It is a *third design* (blog-inspired), and must be labeled so.
+2. Fabricated numbers circulating as Cursor-stated: 69.3% CursorBench / Terminal-Bench
+   parity (in no primary source), "24 other generators" (back-formed from "25x"),
+   "85% post-training compute" (community speculation).
+3. CWM misread: it trains-on-all in a separate **mid-training stage**, not as an aux
+   head riding RL gradients; its 65.8% requires test-time scaling; Chain-of-World is a
+   **robotics** paper. The world-model head has **zero** direct published evidence for
+   the exact proposed configuration — it stays an ablation arm.
+4. Streaming DiLoCo citation doubly wrong (2501.18512 = Douillard et al. "Streaming
+   DiLoCo"; "Eager Updates" = 2502.12996 Kale et al.).
+5. Cost figures mislabeled: "$0.98/trace" is a 50-state synthetic benchmark (real
+   sessions ≈ $70–80 flat); "$64 ungated tree" is a flat 8×1000 extrapolation, not a
+   tree.
+### Missing pieces no design mentioned (all now in the architecture)
+**Benchmark decontamination** (zero mentions anywhere — training substrates overlap
+SWE-bench eval repos), **secrets/PII scrub at trace ingest**, **SPDX license detection
+at repo ingest**, **canonical trajectory IR**, **cross-generation dedup**,
+**`golden_diff` serialization leak** (`repr=False` does not survive `asdict()`),
+**corpus acceptance probe**, **two unreconciled S3 contracts**.
+---
+## Part B — The architecture: "point at a repo → enhanced dataset"
+**The committed answer to the user's question: yes — point at an open-source repo and
+build the dataset. The engine is SWE-smith (buy), the trajectories come from a rollout
+harness (build), and our vision enhancements (oracle-graded multi-candidate divergence,
+typed signal routing) layer on top as Stage 1+ — strictly after Stage 0 produces one
+real corpus end-to-end locally.**
+### Why SWE-smith is the engine (the buy-vs-build verdict)
+- `pip install swesmith` (MIT) ships: env construction from arbitrary GitHub repos
+  (one Docker image **per repo**, 500× more storage-efficient than per-task), five
+  bug-synthesis strategies (LM Modify 56% yield / LM Rewrite 35% / Procedural-AST 40%
+  at $0 / Combine 96.9% at $0 / PR Mirror 33.8%), issue-text generation, and
+  validation-by-test-execution. 50k tasks for **$1,360 + 20 human-hours** total.
+- Its **PR Mirror ≡ our gold-patch reversion** — and its ablation shows PR Mirror
+  trajectories train the best models. The repo's core mechanic is *independently
+  validated*; what we were about to hand-build is exactly what the toolkit ships.
+- Its **Combine-Bugs** (96.9% yield, $0) is the cheapest implementation of the blog's
+  "create harder tasks" CREATE-half — multi-bug escalation for free.
+- R2E-Gym's SweGen covers the "commits without tests" case (LLM-synthesized F2P tests,
+  27.8% vs 28.0% — indistinguishable from real) — the fallback when a pointed-at repo
+  has thin test coverage. Adopt as data (R2E-Gym-Subset), not code, for now.
+- Composer-2 reward nuance (from the techreport): reward = correctness + succinctness +
+  SE-principles. Our pass-fraction is the correctness core; `behavior_rewards.py`
+  (Wave 20) already carries the succinctness/style components.
+### The Stage-0 pipeline (local, no new AWS services)
+```
+            point at repo URL (or HF substrate, or trace dir)
+                              │
+   ┌──────────────────────────▼───────────────────────────┐
+   │ 1. INGEST GATE  (datagen/repo_gate.py)                │
+   │    SPDX license detect → trainable/redistributable    │
+   │    tier; BENCHMARK DECONTAMINATION (repo ∉ SWE-bench/ │
+   │    Verified/Lite/Multimodal eval repo lists)          │
+   └──────────────────────────┬───────────────────────────┘
+                              ▼
+   ┌──────────────────────────────────────────────────────┐
+   │ 2. TASK SYNTHESIS (buy: swesmith)                     │
+   │    swesmith profile → env image → PR-Mirror first,    │
+   │    Combine-Bugs for escalation (CREATE-half), 13      │
+   │    procedural AST transforms for volume               │
+   │    → SwesmithAdapter.to_task() → FeatureDeletionTask  │
+   └──────────────────────────┬───────────────────────────┘
+                              ▼
+   ┌──────────────────────────────────────────────────────┐
+   │ 3. VALIDATE (have): 4-gate validate_task() in         │
+   │    DockerSandbox + scrub_tree                         │
+   └──────────────────────────┬───────────────────────────┘
+                              ▼
+   ┌──────────────────────────────────────────────────────┐
+   │ 4. ROLLOUT HARNESS (build — the critical missing      │
+   │    component): agent loop over FeatureDeletionEnv     │
+   │    (prompt → act → env.step → … → submit → _grade()), │
+   │    pluggable policy (frontier API / local model),     │
+   │    $cap per task. Output: CanonicalTrajectory.        │
+   └──────────────────────────┬───────────────────────────┘
+                              ▼
+   ┌──────────────────────────────────────────────────────┐
+   │ 5. ADMIT + TYPE (have + build): _grade()==1.0 +       │
+   │    HackMonitor-clean + PASS_TO_PASS → sft/;           │
+   │    near-misses → dpo candidates; failures → withheld  │
+   │    (wm_tuples only when the P4 ablation is scheduled) │
+   └──────────────────────────┬───────────────────────────┘
+                              ▼
+   ┌───────────────────────────────���──────────────────────┐
+   │ 6. CORPUS (build): CanonicalTrajectory → messages-    │
+   │    schema SFT rows via to_policy_row() (golden_diff   │
+   │    PROVABLY absent — unit-tested); MinHash dedup      │
+   │    (cross-generation aware); HeldoutSplit; secrets    │
+   │    scrub; ONE reconciled S3/local layout + manifest   │
+   │    + dataset card  (pipeline/s3_contract.py)          │
+   └──────────────────────────┬───────────────────────────┘
+                              ▼
+   7. ACCEPTANCE PROBE: small-model SFT (LoRA) on each corpus
+      generation; require measurable holdout delta before
+      promote-to-accepted.  (the dataset analogue of HeldOutGuard)
+```
+**Free byproduct:** step 4's trajectories are *env-grounded* — exactly the seed nodes
+the tree-of-work needs, fixing structural break #1 without extra work. Claude Code
+traces are demoted to flat Channel-3 / SFT-style uses (their honest capability).
+### What the vision enhancements add, strictly in order
+- **Stage 1 — DPO channel + tool-call algebra:** parse tool calls into a canonical
+  action form (`ToolCall(name, normalized_args)`), fix `_normalize_action`, extract
+  DPO pairs from env-grounded multi-candidate rollouts (N samples per state, each
+  env-stepped + graded — depth-1, no fork needed). Bedrock batch / AWS Batch only when
+  volume justifies.
+- **Stage 2 — depth-1 tree → divergence gate measurement:** N candidates per decision
+  point, one env-step each, oracle-graded. Measure the divergence gate's firing rate on
+  real traces BEFORE building depth>1. `Sandbox.fork()` spike gates depth>1.
+- **Stage 3 — curriculum CREATE-half (Combine-Bugs escalation), flywheel with
+  cross-generation dedup, wm_tuples emission only when P4 ablation scheduled.**
+- **Stage 4 — Step Functions/Argo orchestration once runs are routine.**
+### Build manifest (Stage 0, ~900 LOC + 1 adopted dep)
+| # | Module | What | ~LOC |
+|---|---|---|---|
+| 1 | `datagen/repo_gate.py` | SPDX license detection (LICENSE/classifier heuristics) + trainable-vs-redistributable tiers + benchmark-decontamination list (SWE-bench{,-Lite,-Verified,-Multimodal}+SWE-Gym eval repos) + gate verdict dataclass | ~150 |
+| 2 | `datagen/swesmith_adapter.py` | swesmith task instance → `FeatureDeletionTask` (mirror of SweBenchAdapter; handles swesmith's image naming, F2P/P2P, strategy provenance field) + optional thin synthesis driver behind `[swesmith]` extra | ~120 |
+| 3 | `datagen/trajectory.py` | `CanonicalTrajectory` IR: steps of (obs, ToolCall-or-text action, result, error flag), provenance, grade; adapters: ClaudeCode TraceState→IR; IR→SFT messages; IR→`to_policy_row()` (golden_diff/deleted_symbols PROVABLY dropped) | ~180 |
+| 4 | `datagen/rollout_harness.py` | `RolloutPolicy` protocol (pluggable: API model / local / scripted-fake); `collect_trajectory(env, task, policy, max_turns, budget)` loop; admission filter (`_grade()==1.0` + monitor-clean + guard) | ~220 |
+| 5 | `pipeline/s3_contract.py` | THE single reconciled layout (file:// + s3:// via fsspec): `runs/<run_id>/{tasks,traj,corpus_sft,corpus_dpo,holdout,quarantine}/` + run manifest (counts, cost, lineage, schema_version, parent_run_id) + dataset card writer | ~200 |
+| 6 | `pipeline/dedup.py` | MinHash near-dup detection over SFT rows; cross-generation aware (accepts prior-run signature file) | ~120 |
+| 7 | `pipeline/build_corpus.py` | the local stage-driver wiring 1→6: `build_corpus(source, out, policy, budget)`; source = swesmith repo / SWE-* substrate rows / trace dir | ~180 |
+| 8 | Fidelity-fix batch | research/01 + COMPOSER_RECIPE_MAPPING strikes/tags; opsd.py β + "same loss" claims; diloco citation; kl_in_reward wording; teacher_replay cost docstring; ADR-016 records all of Part A | docs |
+Everything testable CPU-only with fakes (swesmith synthesis itself needs Docker+Linux —
+the adapter and driver are tested on fixture instances, the live path is
+`skipif`-gated like the existing Docker e2e).
+### Pre-registered falsifiers (kept from the original vision, now evidence-bounded)
+- Heterogeneous-N-models vs single-model-N-samples at equal compute (Symphony vs
+  data-processing-inequality — genuinely two-sided, SWE-specific answer unrun).
+- World-model aux head: build only if the P0–P6 ladder's P4/P6 beat P0–P3 on
+  foresight+calibration (CWM supports mid-training train-on-all, NOT an RL-time aux
+  head — the proposed configuration is in a null-evidence zone).
+- Tree depth>1: build only if the measured divergence-gate firing rate makes
+  O(N·decision-points) real, and only after `Sandbox.fork()` exists.