# Verified Findings — Consolidated Critical Review (Verification Agent)

**Date:** 2026-06-09
**Inputs verified:** `research/deepread/10-critic-fidelity.md`, `research/deepread/11-critic-design.md`, cross-checked against repo code, the deep-reads (`00`–`08`), and the fetched Composer 2.5 blog body (`research/notes/introducing-composer-25-cursor.md`).

> **MISSING INPUT:** `research/deepread/09-critic-feasibility.md` did not exist at verification time
> (polled for ~12 minutes; directory contains only 00–08, 10, 11). The feasibility critic's findings
> are therefore **unverified and not consolidated here**. If that file appears later, a delta
> verification pass is required before its P0s are acted on.

**Method:** every P0 (and every P1 naming a specific file/claim) was re-checked against the actual
repo file at the cited line, the verbatim blog text, or the deep-read's primary-source quotes.
Verdicts: **CONFIRMED** / **REFUTED** / **OVERSTATED** (true core, inflated framing).

---

## Part 1 — Verification ledger

### Fidelity critic (10) — P0s

| # | Claim | Verdict | Evidence |
|---|---|---|---|
| F-1 | Benchmark numbers (69.3%, Terminal-Bench 2.0 parity) asserted as Cursor-stated; in no primary source | **CONFIRMED** | `research/01-composer-2.5.md:63-64` asserts "it hit 69.3%" under "Cursor claims…"; fetched blog body contains zero benchmark numbers (only "not well captured by existing benchmarks", note line 26). Header audit note flags but body asserts unhedged. |
| F-2 | "Feature Deletion + 24 other (unnamed) generators" — count invented | **CONFIRMED** | Blog verbatim (note line 50): "a range of approaches… one synthetic approach is feature deletion" — no count. `docs/COMPOSER_RECIPE_MAPPING.md:75`, `research/06:330` ("The other ~24 generators"), `research/09:23` all carry the fabricated 24. |
| F-3 | SDPO declared "mathematically the same" as Composer's mechanism — unsupported identification | **CONFIRMED** | `docs/COMPOSER_RECIPE_MAPPING.md:25` ("mathematically the same"), `:151` ("direct formalization"); `opsd.py:13-14` ("formalizes the same loss as Composer 2.5's…"). Blog cites the papers only as "For more background on this approach see" (note lines 67/148). Deep-read 03 (lines 93, 252, 280-281, 308): SDPO loss is full-rollout with feedback-in-prefix; repo's design is turn-localized hint-splice — neither SDPO nor confirmed Composer. |
| F-4 | Streaming DiLoCo misattributed (wrong authors AND wrong title on arXiv:2501.18512) | **CONFIRMED** | `composer_replication/diloco/__init__.py:8` "Streaming DiLoCo (Liu et al. 2025)"; `research/design-F4-decoupled-diloco-s3.md:109` attaches "Eager Updates…" title to 2501.18512. Deep-read 05 F4: 2501.18512 = Douillard et al., "Streaming DiLoCo with overlapping communication"; Eager Updates = arXiv:2502.12996 (Kale et al.). |
| F-5 | CWM cited as licensing train-on-all for an RL-time aux head; CWM does it in a separate mid-training stage | **CONFIRMED** | Final report line 33: "crucially training *on all* trajectories for the world-model head, reserving success-filtering only for the RL reward [13]" (+ ref line 258). Deep-read 06 quotes CWM §2.2 verbatim ("we do not filter trajectories…") and documents the three-phase pipeline: world-modeling is in base weights pre-RL, not an aux head riding policy gradients. |

### Fidelity critic (10) — P1s naming specific files/claims

| # | Claim | Verdict | Evidence |
|---|---|---|---|
| F-6 | CWM "65.8%" cited without test-time-scaling qualifier | **CONFIRMED** | Final report line 33 cites bare "65.8% on SWE-bench Verified"; CWM abstract per deep-read 06: "65.8% … (with test-time scaling)". |
| F-7 | Chain-of-World (2603.03195) is robotics VLA, used for SWE | **CONFIRMED** (via deep-read 06 finding 2.2) | Deep-read 06 exec summary: "robotics/embodied VLA paper, not an SWE paper". |
| F-8 | "85% post-training compute" stated as fact in research/01 body | **CONFIRMED** | `research/01-composer-2.5.md:14` asserts it unhedged; header (line 5) flags it as "community consensus, not Cursor-stated" — body never inline-tags. |
| F-9 | Implemented "feature deletion" is gold-patch reversion, not the blog's delete-from-functional-codebase mechanism | **CONFIRMED** | Blog verbatim (note line 50): agent "asked to delete code and files… codebase remains functional". `substrates.py` SweBenchAdapter reverts gold patches of pre-labeled instances; mapping doc line ~42 paraphrase drops the agentic deleter. "Point at a repo" appears nowhere in the blog. |
| F-10 | Difficulty curriculum: only SELECT half built; CREATE half 0% | **CONFIRMED** | `curriculum.py` header quotes the blog's "select for and create"; `substrates.py:80` hardcodes `granularity="feature"`; no task-escalation stage anywhere. (= Design #15; merged below.) |
| F-12 | Mapping doc states `KL(teacher‖student)` — direction unsupported by blog, opposite of SDPO Eq. 1 | **CONFIRMED** | `COMPOSER_RECIPE_MAPPING.md` (~line 19): "KL( teacher_logits_at_turn_t \|\| student_logits_at_turn_t )". Blog: "moves the student's token probabilities toward the teacher's" — directionless. SDPO Eq. 1 per deep-read 03 line 308: `KL(π_θ ‖ stopgrad(·))`, student first. |
| F-13 | `opsd.py` β-convention docstring inverted vs upstream | **CONFIRMED** | `opsd.py:55-59` labels β=0 "reverse KL"; OPSD README per deep-read 03 (§2.2, line 168 "INVERTED", table line 497): "Beta=0 means forward KL". Code numerically correct; labels swapped. |
| F-14 | SDPO channel lacks the paper's EMA/trust-region teacher regularization | **CONFIRMED** | Deep-read 03 lines 32-34, 81-82: non-regularized live teacher diverges (Table 4: 36.1% vs 50.6%); repo teacher = live weights, no EMA. |
| F-16 | "$0.98/trace verified economic floor" hides trace definition; real sessions ~2 OOM more | **CONFIRMED** | `teacher_replay.py:7-8` unqualified docstring; `VISION_VALIDATION.md:63,76`; `ADR-002:60` (125→2,830 tool-use messages/session); deep-read 07 FR-R5: ~$73/full session. (VISION_VALIDATION's Objection 2 partially self-flags; the docstring does not.) |
| F-17 | "$64 ungated tree" is a flat N=8×T=1000 extrapolation, not a tree | **CONFIRMED** | `research/05:256` ($0.008×1000×8 flat math); `comparisons.md:33` and final report line 66 label it the branching/"ungated" cost. True tree is O(N^D), strictly worse. |
| F-18 | `kl_in_reward.py`: verl k1 "*only* reverse-KL option" overclaim | **CONFIRMED** | `kl_in_reward.py:12` verbatim; deep-read 04 §4.3: verl also supports `kl_penalty="low_var_kl"` (k3-family). |
| F-20 | GSPO preset inherits GRPO-scale clipping — 2 OOM off the paper | **CONFIRMED** | `composer_trainer.py` "gspo" preset (~line 814) sets no epsilon → TRL default ~0.2; GSPO paper §5.1: 3e-4/4e-4. Companion confirmed: "cispo" preset omits explicit `beta`. |
| F-21 | rStar named "closest precedent" (misread granularity); Tree-GRPO + SWE-Search uncited | **CONFIRMED** | `framework/composer-replication-framework.md:17-18` ("Closest precedent: rStar-Math… open territory"); grep: zero hits for Tree-GRPO/2509.21240/SWE-Search/2410.20285 in `research/05`. |
| F-22 | "Nine-tenths is reuse" conflates design-reuse with build status | **CONFIRMED** | Final report line 133 ("the substrate already exists — roughly nine-tenths of it"); grounding doc Claim 5: tree, wm-head, pipeline/, infra/ are 0% built. |
| F-23 | "Parameter isolation eliminates interference" overclaims DART; aux-head evidence wholly analogical | **CONFIRMED** | Deep-read 06: 2602.00994 shows interference persists on isolated LoRA modules; "evidentiary gap is total for the specific proposed configuration". |
| F-24 | VeRL "first-class agentic RL" — async path is experimental | **CONFIRMED** | Deep-read 08 lines 66-73: `fully_async_policy`/`transfer_queue` under `verl/experimental`; "slightly overclaims". |

P2s (F-25…F-33) spot-checked where cheap; no refutations found. F-33 (`_normalize_action` absent from risk lists) merges into Design #3.

### Design critic (11) — P0s

| # | Claim | Verdict | Evidence |
|---|---|---|---|
| D-1 | Tree seed traces (Claude Code) and execution oracle (FeatureDeletionEnv) operate on disjoint data | **CONFIRMED** | `design-F1:18,70-74` seeds the tree from `ingestion/claude_code.py` TraceStates (local `~/.claude/projects` JSONL — no image, no F2P); `env.py:59` requires `task.broken_image`; `schema.py:35-38` `__post_init__` raises on empty `fail_to_pass`. No design reconciles them. |
| D-2 | No agent rollout harness exists — SFT corpus has no producer | **CONFIRMED** (minor nuance) | Deep-read 08 §5.1: `reward_fn` fallback = single submit, "dead end for genuine multi-turn"; `teacher_replay` emits single next-actions; F1 line 136 names "tree controller" (0% built) as the sft_corpus producer; F2 stage (d) writes `corpus/sft` with no trajectory-generating stage upstream. Nuance: the tree-controller *design* does call `env.step()`, but expands counterfactual branches from (unexecutable, per D-1) frozen traces — it is not an episode-completing rollout loop. |
| D-3 | Divergence gating not computable from what components emit | **CONFIRMED** | `teacher_replay.py:195-203`: `_normalize_action` = whitespace-collapse + lowercase, docstring self-admits skeleton status; teachers return text (no cross-model distributions); deep-read 07 FR-R8 (HIGH): "mostly noise on real traces" → gate fires everywhere → silent O(N^D). |
| D-4 | Sandbox lacks fork/snapshot primitive the tree requires | **CONFIRMED** | `sandbox.py:99-106` Protocol = boot/exec/run_tests/trajectory/is_command_allowed only; grep "fork\|snapshot" → only "fork-bomb guard"; `docker_sandbox.py:15,157` `read_only=True` rootfs. |
| D-5 | Zero benchmark decontamination anywhere | **CONFIRMED** | grep "decontamin/contamination" across `composer_replication/`, `docs/`, F1, F2, ADR-010 → zero hits; `holdout.py:171` `HeldoutSplit` is internal-split-only; deep-read 02 flags R2E-Gym full-set overlap risk. |
| D-6 | Buy-vs-build inversion: repo hand-builds what swesmith ships; PR-Mirror validation uncited | **CONFIRMED** | No `swesmith` in `pyproject.toml` or ADR-010 (grep empty); deep-read 02 lines 230, 251, 306: PR Mirror ≡ repo's gold-patch reversion, best training data per SWE-smith Table 5, "$1,360 + 20 human-hours" verified — "neither research/06 nor ADR-010 cite this result". |

### Design critic (11) — P1s naming specific files/claims

| # | Claim | Verdict | Evidence |
|---|---|---|---|
| D-7 | Two unreconciled S3 contracts | **CONFIRMED** (one sub-point softened) | F1 lines 86-93 (`runs/<id>/sft_corpus/ dpo_pairs/ …`) vs F2 lines 133-143 (`traces/v1/run_id=<id>`, `corpus/v1/...{sft,dpo}`); grounding step 8 (lines 91-93) pastes both — `corpus/v1/.../dpo/` AND `dpo_pairs/` coexist. Nuance: F2:129 offers "reuse the existing amazon-sagemaker-… **or** a dedicated composer-datagen-…" — two bucket *options*, not a hard contradiction. |
| D-8 | `golden_diff` leaks via naive serialization; `repr=False` ≠ exclusion | **CONFIRMED** | `schema.py:29` `field(default="", repr=False)`; F2:100,135 writes "FeatureDeletionTask rows" to `tasks/v1/.../manifest.jsonl`; `dataclasses.asdict`/`vars()` include repr=False fields. Only `env.py:73` (prompt renderer) hides it. |
| D-9 | Five-service AWS orchestration before one local end-to-end run | **CONFIRMED** | F2 names Glue/EMR Serverless/AWS Batch/Bedrock batch/Step Functions/Lambda (46 hits); ADR-010:99-103 concedes gates passed "against FakeSandbox", Docker e2e `[~]`. |
| D-10 | No secrets/PII gate at trace ingest | **CONFIRMED** | F2:29 uploads raw `~/.claude/projects/**.jsonl` to `raw/claude_code/`; grep secrets/PII/gitleaks/trufflehog in F2 → zero; `scrub_tree` strips caches, not secrets; `is_redistributable` applies to tasks only. |
| D-12 | No cross-generation dedup in the flywheel | **CONFIRMED** | F2:114-118: data-juicer `document_deduplicator` per-batch + Spark `dropDuplicates` within one run; `parent_run_id` (F2:151) threads lineage but nothing dedups across runs. |
| D-13 | License gate is field-deep substring matching, absent for repo-ingest | **CONFIRMED** | `substrates.py:86-90`: `lic = task.upstream_license.lower(); return not any(c in lic for c in _COPYLEFT)`; no SPDX detection at clone path; trainable-vs-redistributable distinction absent. |
| D-14 | `wm_tuples/` highest-volume prefix serving an ablation-gated, zero-direct-evidence consumer | **CONFIRMED** | F1:90 "wm_tuples/ ← … ALL branches"; deep-read 06: "no published paper has tested an auxiliary next-state-prediction objective during RL on a code policy"; no sampling/lifecycle policy in any design. |
| D-15 | CREATE-half of curriculum absent from every pipeline design | **CONFIRMED** | Same evidence as F-10; merged below as one finding. |
| D-11 | No canonical trajectory IR (3 trace shapes → 5) | **CONFIRMED** | `_normalize_action` stub is the symptom; Claude Code JSONL/TraceState, Bedrock `.jsonl.out`, planned tree + rollout + OpenHands shapes each named in separate docs with no shared schema (no `datagen/schema.py` trajectory type exists). |

D-16…D-21 (P2): spot-checked — `state_id` format confirmed (`claude_code.py:181`), `diloco_rendezvous/` co-located in F1's run layout (F1:93). No refutations.

**Verification summary: 0 REFUTED, 2 minor nuances (D-2, D-7), everything else CONFIRMED.** Both critics were accurate against source; no fabrication detected.

---

## Part 2 — FINAL consolidated finding list (ranked by severity × confidence, deduplicated)

### Tier 1 — P0, high confidence, load-bearing for the dataset-pipeline build

1. **[V1] The Channel-2 identity claim is wrong: SDPO is *background*, not Composer's mechanism — and the repo's implementation is a third design.** Cursor's blog cites OPSD/SDPO only as "for more background"; SDPO's published loss is full-rollout with feedback-in-prefix and a *regularized* teacher; the repo's turn-localized hint-splice (live-weights teacher, no EMA) is its own blog-inspired variant. Fix `COMPOSER_RECIPE_MAPPING.md:25,151`, `opsd.py:13-14`; also fix the KL-direction line (F-12), the inverted β docstring (F-13), and document/implement teacher regularization (F-14). *(Fidelity F-3 + F-12 + F-13 + F-14.)*

2. **[V2] The envisioned tree-of-work pipeline cannot execute as drawn: seed traces and the execution oracle are disjoint, no rollout harness exists to produce the SFT corpus, the divergence gate is uncomputable from text + a whitespace normalizer, and the Sandbox has no fork primitive.** Four independently confirmed structural breaks that compose into one verdict: Phase 1 must seed from env-grounded rollouts (adopt mini-swe-agent/SWE-agent as the trajectory collector), demote Claude Code traces to flat Channel-3/SFT-style uses, build a canonical tool-call action algebra before any `tree_controller.py`, and spike `Sandbox.fork()` before committing to depth>1. *(Design D-1 + D-2 + D-3 + D-4 + fidelity F-33.)*

3. **[V3] Zero benchmark decontamination anywhere in code or designs.** The pipeline trains on SWE-bench-family substrates and will be scored on SWE-bench Verified; substrates have partial/divergent decontamination; `HeldoutSplit` is internal-only; the word does not appear in F1/F2/ADR-010. Must exist before the first corpus is generated. *(Design D-5.)*

4. **[V4] Buy-vs-build inversion: the repo plans to hand-build what `pip install swesmith` ships, while its own core mechanic (gold-patch reversion) is SWE-smith's PR Mirror — validated as the *best* training data by an ablation the repo never cites.** The user's "point-at-a-repo" ask is ~100 LOC of swesmith adapter, not an unspecified image-builder subsystem ($1,360 + 20 human-hours is the budget bar to beat). Relatedly, the implemented "feature deletion" is the inversion *analogue*, not the blog's delete-from-functional-codebase mechanism — and downstream docs should say so. *(Design D-6 + fidelity F-9.)*

### Tier 2 — P0 fidelity corrections (high confidence, citation/claim integrity)

5. **[V5] Fabricated quantities circulating as Cursor-stated facts:** (a) benchmark numbers 69.3%/Terminal-Bench parity appear in no primary source (`research/01:63-64`); (b) "24 other generators" is a back-formation from the 25x *task* multiplier (`COMPOSER_RECIPE_MAPPING.md:75`, `research/06:330`, `research/09:23`); (c) "85% post-training compute" asserted unhedged in the research/01 body. Strike or inline-tag all three. *(Fidelity F-1 + F-2 + F-8.)*

6. **[V6] World-model aux-head's load-bearing citation is a misread: CWM trains-on-all in a dedicated *mid-training* stage, not as an aux head riding RL policy gradients; 65.8% requires the test-time-scaling qualifier; Chain-of-World is robotics; "parameter isolation eliminates interference" overclaims DART.** The exact proposed configuration has zero published evidence — keep it an ablation arm and gate `wm_tuples/` emission (highest-volume prefix, most speculative consumer) behind the scheduled ablation. *(Fidelity F-5 + F-6 + F-7 + F-23 + design D-14.)*

7. **[V7] Streaming DiLoCo citation is doubly wrong** — arXiv:2501.18512 is Douillard et al., "Streaming DiLoCo with overlapping communication"; "Eager Updates…" is arXiv:2502.12996 (Kale et al.); "Liu et al." is the Async Local-SGD paper. Fix `diloco/__init__.py:8` and `design-F4:109`. Low blast radius, unambiguous. *(Fidelity F-4.)*

### Tier 3 — P1, high confidence (will bite during the build)

8. **[V8] Two unreconciled S3 contracts (F1 vs F2) + `golden_diff` serialization leak.** Write `s3_contract.py` first with one layout and two explicit serializers (`to_policy_row` drops `golden_diff`/`deleted_symbols`; unit-test the absence); fold `divergence_pairs` into `corpus_dpo` provenance; separate the DiLoCo rendezvous prefix/bucket. *(Design D-7 + D-8 + D-19.)*

9. **[V9] No secrets/PII scrub at Claude Code trace ingest** — raw sessions (file contents, keys in tool outputs, proprietary code) go to S3 verbatim; the only license filter applies to tasks, and it is a lowercase substring match with no SPDX detection and no trainable-vs-redistributable split for the repo-ingest path. *(Design D-10 + D-13.)*

10. **[V10] The blog's CREATE-half of the dynamic curriculum ("select for **and create** harder tasks") is absent from code and from every pipeline design** — `granularity` hardcoded `"feature"`, no escalation stage in the outer loop. Cheapest first implementation: SWE-smith Combine-Bugs (96.9% yield) as a `task_escalation` stage; at minimum reserve the stage boundary in the orchestration contract. *(Fidelity F-10 ≡ design D-15.)*

11. **[V11] Cost claims mislabeled in ways that distort planning:** "$0.98/trace verified floor" is a 50-step synthetic-state benchmark (real sessions ~$70–80 flat at 125–2,830 steps/session); "$64 ungated tree" is a flat 8-teacher×1000-step extrapolation, not a tree (true tree is O(N^D)). Fix `teacher_replay.py:7-8`, `VISION_VALIDATION.md`, `comparisons.md:33`, final report §10/§3. *(Fidelity F-16 + F-17.)*

12. **[V12] Premature five-service AWS orchestration + flywheel hygiene gaps:** F2 commits Glue/EMR/Batch/Bedrock-batch/Step-Functions before one local e2e run (ADR-010 gates passed against FakeSandbox); no cross-generation dedup (self-training collapse accelerant); no canonical trajectory IR; no restart/budget semantics. Build a local stage-driver first; add MinHash cross-run dedup + `CanonicalTrajectory` schema. *(Design D-9 + D-12 + D-11 + D-21.)*

### Tier 4 — P1/P2 precision fixes (confirmed, lower blast radius)

13. **[V13] Preset/config drift vs papers:** GSPO preset missing 3e-4/4e-4 epsilons (operationally not GSPO); CISPO preset should set `beta=0.0` explicitly; `kl_in_reward.py:12` "only reverse-KL option" → "default/recommended"; Comedy-of-Estimators citation rests on abstract-only. *(Fidelity F-20 + F-18 + F-19.)*

14. **[V14] Provenance/novelty bookkeeping:** cite Tree-GRPO (2509.21240) and SWE-Search (2410.20285) as nearest neighbors and fix the rStar granularity misread (`framework/…:17`, `research/05`); re-scope "nine-tenths reuse" to the recipe-replication layer (tree/wm-head/pipeline are 0% built); note verl async path is experimental. *(Fidelity F-21 + F-22 + F-24.)*

15. **[V15] Remaining P2 wording fixes from the fidelity critic (F-25…F-33)** — stage attribution of Muon, deleter-unknown framing, 25x non-convertibility, grounding-doc paraphrase-as-quote, DiLoCo FP4/H-default details, Foresight@k "(we define this)", SWE-rebench unverified counts — all consistent with the deep-reads; apply as a batch documentation pass. Plus design P2s: node-ID contract, image-digest pinning + dataset cards, corpus acceptance probe (small-model SFT delta before GPU spend).

---

## Coverage note

The design critic's minimal-pipeline proposal (swesmith + SPDX/decontamination gates + SWE-agent
rollout harness + canonical trajectory schema + `s3_contract.py`, ≈600–900 LOC) is consistent with
every confirmed finding above and is the recommended Stage-0 build order. The feasibility critic
(09) was never produced; its absence means cost/timeline feasibility claims in the designs have had
only the fidelity-angle checks (V11) — treat any standalone feasibility assertions as unreviewed.