research/deepread/12-verified-findings.md

Verified Findings — Consolidated Critical Review (Verification Agent)

Date: 2026-06-09 Inputs verified: research/deepread/10-critic-fidelity.md, research/deepread/11-critic-design.md, cross-checked against repo code, the deep-reads (00–08), and the fetched Composer 2.5 blog body (research/notes/introducing-composer-25-cursor.md).

MISSING INPUT: research/deepread/09-critic-feasibility.md did not exist at verification time (polled for ~12 minutes; directory contains only 00–08, 10, 11). The feasibility critic's findings are therefore unverified and not consolidated here. If that file appears later, a delta verification pass is required before its P0s are acted on.

Method: every P0 (and every P1 naming a specific file/claim) was re-checked against the actual repo file at the cited line, the verbatim blog text, or the deep-read's primary-source quotes. Verdicts: CONFIRMED / REFUTED / OVERSTATED (true core, inflated framing).

---

Part 1 — Verification ledger

Fidelity critic (10) — P0s

#	Claim	Verdict	Evidence
F-1	Benchmark numbers (69.3%, Terminal-Bench 2.0 parity) asserted as Cursor-stated; in no primary source	CONFIRMED	research/01-composer-2.5.md:63-64 asserts "it hit 69.3%" under "Cursor claims…"; fetched blog body contains zero benchmark numbers (only "not well captured by existing benchmarks", note line 26). Header audit note flags but body asserts unhedged.
F-2	"Feature Deletion + 24 other (unnamed) generators" — count invented	CONFIRMED	Blog verbatim (note line 50): "a range of approaches… one synthetic approach is feature deletion" — no count. docs/COMPOSER_RECIPE_MAPPING.md:75, research/06:330 ("The other ~24 generators"), research/09:23 all carry the fabricated 24.
F-3	SDPO declared "mathematically the same" as Composer's mechanism — unsupported identification	CONFIRMED	docs/COMPOSER_RECIPE_MAPPING.md:25 ("mathematically the same"), :151 ("direct formalization"); opsd.py:13-14 ("formalizes the same loss as Composer 2.5's…"). Blog cites the papers only as "For more background on this approach see" (note lines 67/148). Deep-read 03 (lines 93, 252, 280-281, 308): SDPO loss is full-rollout with feedback-in-prefix; repo's design is turn-localized hint-splice — neither SDPO nor confirmed Composer.
F-4	Streaming DiLoCo misattributed (wrong authors AND wrong title on arXiv:2501.18512)	CONFIRMED	composer_replication/diloco/__init__.py:8 "Streaming DiLoCo (Liu et al. 2025)"; research/design-F4-decoupled-diloco-s3.md:109 attaches "Eager Updates…" title to 2501.18512. Deep-read 05 F4: 2501.18512 = Douillard et al., "Streaming DiLoCo with overlapping communication"; Eager Updates = arXiv:2502.12996 (Kale et al.).
F-5	CWM cited as licensing train-on-all for an RL-time aux head; CWM does it in a separate mid-training stage	CONFIRMED	Final report line 33: "crucially training on all trajectories for the world-model head, reserving success-filtering only for the RL reward [13]" (+ ref line 258). Deep-read 06 quotes CWM §2.2 verbatim ("we do not filter trajectories…") and documents the three-phase pipeline: world-modeling is in base weights pre-RL, not an aux head riding policy gradients.

Fidelity critic (10) — P1s naming specific files/claims

#	Claim	Verdict	Evidence
F-6	CWM "65.8%" cited without test-time-scaling qualifier	CONFIRMED	Final report line 33 cites bare "65.8% on SWE-bench Verified"; CWM abstract per deep-read 06: "65.8% … (with test-time scaling)".
F-7	Chain-of-World (2603.03195) is robotics VLA, used for SWE	CONFIRMED (via deep-read 06 finding 2.2)	Deep-read 06 exec summary: "robotics/embodied VLA paper, not an SWE paper".
F-8	"85% post-training compute" stated as fact in research/01 body	CONFIRMED	research/01-composer-2.5.md:14 asserts it unhedged; header (line 5) flags it as "community consensus, not Cursor-stated" — body never inline-tags.
F-9	Implemented "feature deletion" is gold-patch reversion, not the blog's delete-from-functional-codebase mechanism	CONFIRMED	Blog verbatim (note line 50): agent "asked to delete code and files… codebase remains functional". substrates.py SweBenchAdapter reverts gold patches of pre-labeled instances; mapping doc line ~42 paraphrase drops the agentic deleter. "Point at a repo" appears nowhere in the blog.
F-10	Difficulty curriculum: only SELECT half built; CREATE half 0%	CONFIRMED	curriculum.py header quotes the blog's "select for and create"; substrates.py:80 hardcodes granularity="feature"; no task-escalation stage anywhere. (= Design #15; merged below.)
F-12	Mapping doc states KL(teacher‖student) — direction unsupported by blog, opposite of SDPO Eq. 1	CONFIRMED	COMPOSER_RECIPE_MAPPING.md (~line 19): "KL( teacher_logits_at_turn_t || student_logits_at_turn_t )". Blog: "moves the student's token probabilities toward the teacher's" — directionless. SDPO Eq. 1 per deep-read 03 line 308: KL(π_θ ‖ stopgrad(·)), student first.
F-13	opsd.py β-convention docstring inverted vs upstream	CONFIRMED	opsd.py:55-59 labels β=0 "reverse KL"; OPSD README per deep-read 03 (§2.2, line 168 "INVERTED", table line 497): "Beta=0 means forward KL". Code numerically correct; labels swapped.
F-14	SDPO channel lacks the paper's EMA/trust-region teacher regularization	CONFIRMED	Deep-read 03 lines 32-34, 81-82: non-regularized live teacher diverges (Table 4: 36.1% vs 50.6%); repo teacher = live weights, no EMA.
F-16	"$0.98/trace verified economic floor" hides trace definition; real sessions ~2 OOM more	CONFIRMED	teacher_replay.py:7-8 unqualified docstring; VISION_VALIDATION.md:63,76; ADR-002:60 (125→2,830 tool-use messages/session); deep-read 07 FR-R5: ~$73/full session. (VISION_VALIDATION's Objection 2 partially self-flags; the docstring does not.)
F-17	"$64 ungated tree" is a flat N=8×T=1000 extrapolation, not a tree	CONFIRMED	research/05:256 ($0.008×1000×8 flat math); comparisons.md:33 and final report line 66 label it the branching/"ungated" cost. True tree is O(N^D), strictly worse.
F-18	kl_in_reward.py: verl k1 "only reverse-KL option" overclaim	CONFIRMED	kl_in_reward.py:12 verbatim; deep-read 04 §4.3: verl also supports kl_penalty="low_var_kl" (k3-family).
F-20	GSPO preset inherits GRPO-scale clipping — 2 OOM off the paper	CONFIRMED	composer_trainer.py "gspo" preset (~line 814) sets no epsilon → TRL default ~0.2; GSPO paper §5.1: 3e-4/4e-4. Companion confirmed: "cispo" preset omits explicit beta.
F-21	rStar named "closest precedent" (misread granularity); Tree-GRPO + SWE-Search uncited	CONFIRMED	framework/composer-replication-framework.md:17-18 ("Closest precedent: rStar-Math… open territory"); grep: zero hits for Tree-GRPO/2509.21240/SWE-Search/2410.20285 in research/05.
F-22	"Nine-tenths is reuse" conflates design-reuse with build status	CONFIRMED	Final report line 133 ("the substrate already exists — roughly nine-tenths of it"); grounding doc Claim 5: tree, wm-head, pipeline/, infra/ are 0% built.
F-23	"Parameter isolation eliminates interference" overclaims DART; aux-head evidence wholly analogical	CONFIRMED	Deep-read 06: 2602.00994 shows interference persists on isolated LoRA modules; "evidentiary gap is total for the specific proposed configuration".
F-24	VeRL "first-class agentic RL" — async path is experimental	CONFIRMED	Deep-read 08 lines 66-73: fully_async_policy/transfer_queue under verl/experimental; "slightly overclaims".

P2s (F-25…F-33) spot-checked where cheap; no refutations found. F-33 (_normalize_action absent from risk lists) merges into Design #3.

Design critic (11) — P0s

#	Claim	Verdict	Evidence
D-1	Tree seed traces (Claude Code) and execution oracle (FeatureDeletionEnv) operate on disjoint data	CONFIRMED	design-F1:18,70-74 seeds the tree from ingestion/claude_code.py TraceStates (local ~/.claude/projects JSONL — no image, no F2P); env.py:59 requires task.broken_image; schema.py:35-38 __post_init__ raises on empty fail_to_pass. No design reconciles them.
D-2	No agent rollout harness exists — SFT corpus has no producer	CONFIRMED (minor nuance)	Deep-read 08 §5.1: reward_fn fallback = single submit, "dead end for genuine multi-turn"; teacher_replay emits single next-actions; F1 line 136 names "tree controller" (0% built) as the sft_corpus producer; F2 stage (d) writes corpus/sft with no trajectory-generating stage upstream. Nuance: the tree-controller design does call env.step(), but expands counterfactual branches from (unexecutable, per D-1) frozen traces — it is not an episode-completing rollout loop.
D-3	Divergence gating not computable from what components emit	CONFIRMED	teacher_replay.py:195-203: _normalize_action = whitespace-collapse + lowercase, docstring self-admits skeleton status; teachers return text (no cross-model distributions); deep-read 07 FR-R8 (HIGH): "mostly noise on real traces" → gate fires everywhere → silent O(N^D).
D-4	Sandbox lacks fork/snapshot primitive the tree requires	CONFIRMED	sandbox.py:99-106 Protocol = boot/exec/run_tests/trajectory/is_command_allowed only; grep "fork|snapshot" → only "fork-bomb guard"; docker_sandbox.py:15,157 read_only=True rootfs.
D-5	Zero benchmark decontamination anywhere	CONFIRMED	grep "decontamin/contamination" across composer_replication/, docs/, F1, F2, ADR-010 → zero hits; holdout.py:171 HeldoutSplit is internal-split-only; deep-read 02 flags R2E-Gym full-set overlap risk.
D-6	Buy-vs-build inversion: repo hand-builds what swesmith ships; PR-Mirror validation uncited	CONFIRMED	No swesmith in pyproject.toml or ADR-010 (grep empty); deep-read 02 lines 230, 251, 306: PR Mirror ≡ repo's gold-patch reversion, best training data per SWE-smith Table 5, "$1,360 + 20 human-hours" verified — "neither research/06 nor ADR-010 cite this result".

Design critic (11) — P1s naming specific files/claims

#	Claim	Verdict	Evidence
D-7	Two unreconciled S3 contracts	CONFIRMED (one sub-point softened)	F1 lines 86-93 (runs/<id>/sft_corpus/ dpo_pairs/ …) vs F2 lines 133-143 (traces/v1/run_id=<id>, corpus/v1/...{sft,dpo}); grounding step 8 (lines 91-93) pastes both — corpus/v1/.../dpo/ AND dpo_pairs/ coexist. Nuance: F2:129 offers "reuse the existing amazon-sagemaker-… or a dedicated composer-datagen-…" — two bucket options, not a hard contradiction.
D-8	golden_diff leaks via naive serialization; repr=False ≠ exclusion	CONFIRMED	schema.py:29 field(default="", repr=False); F2:100,135 writes "FeatureDeletionTask rows" to tasks/v1/.../manifest.jsonl; dataclasses.asdict/vars() include repr=False fields. Only env.py:73 (prompt renderer) hides it.
D-9	Five-service AWS orchestration before one local end-to-end run	CONFIRMED	F2 names Glue/EMR Serverless/AWS Batch/Bedrock batch/Step Functions/Lambda (46 hits); ADR-010:99-103 concedes gates passed "against FakeSandbox", Docker e2e [~].
D-10	No secrets/PII gate at trace ingest	CONFIRMED	F2:29 uploads raw ~/.claude/projects/**.jsonl to raw/claude_code/; grep secrets/PII/gitleaks/trufflehog in F2 → zero; scrub_tree strips caches, not secrets; is_redistributable applies to tasks only.
D-12	No cross-generation dedup in the flywheel	CONFIRMED	F2:114-118: data-juicer document_deduplicator per-batch + Spark dropDuplicates within one run; parent_run_id (F2:151) threads lineage but nothing dedups across runs.
D-13	License gate is field-deep substring matching, absent for repo-ingest	CONFIRMED	substrates.py:86-90: lic = task.upstream_license.lower(); return not any(c in lic for c in _COPYLEFT); no SPDX detection at clone path; trainable-vs-redistributable distinction absent.
D-14	wm_tuples/ highest-volume prefix serving an ablation-gated, zero-direct-evidence consumer	CONFIRMED	F1:90 "wm_tuples/ ← … ALL branches"; deep-read 06: "no published paper has tested an auxiliary next-state-prediction objective during RL on a code policy"; no sampling/lifecycle policy in any design.
D-15	CREATE-half of curriculum absent from every pipeline design	CONFIRMED	Same evidence as F-10; merged below as one finding.
D-11	No canonical trajectory IR (3 trace shapes → 5)	CONFIRMED	_normalize_action stub is the symptom; Claude Code JSONL/TraceState, Bedrock .jsonl.out, planned tree + rollout + OpenHands shapes each named in separate docs with no shared schema (no datagen/schema.py trajectory type exists).

D-16…D-21 (P2): spot-checked — state_id format confirmed (claude_code.py:181), diloco_rendezvous/ co-located in F1's run layout (F1:93). No refutations.

Verification summary: 0 REFUTED, 2 minor nuances (D-2, D-7), everything else CONFIRMED. Both critics were accurate against source; no fabrication detected.

---

Part 2 — FINAL consolidated finding list (ranked by severity × confidence, deduplicated)

Tier 1 — P0, high confidence, load-bearing for the dataset-pipeline build

1. [V1] The Channel-2 identity claim is wrong: SDPO is background, not Composer's mechanism — and the repo's implementation is a third design. Cursor's blog cites OPSD/SDPO only as "for more background"; SDPO's published loss is full-rollout with feedback-in-prefix and a regularized teacher; the repo's turn-localized hint-splice (live-weights teacher, no EMA) is its own blog-inspired variant. Fix COMPOSER_RECIPE_MAPPING.md:25,151, opsd.py:13-14; also fix the KL-direction line (F-12), the inverted β docstring (F-13), and document/implement teacher regularization (F-14). (Fidelity F-3 + F-12 + F-13 + F-14.)

2. [V2] The envisioned tree-of-work pipeline cannot execute as drawn: seed traces and the execution oracle are disjoint, no rollout harness exists to produce the SFT corpus, the divergence gate is uncomputable from text + a whitespace normalizer, and the Sandbox has no fork primitive. Four independently confirmed structural breaks that compose into one verdict: Phase 1 must seed from env-grounded rollouts (adopt mini-swe-agent/SWE-agent as the trajectory collector), demote Claude Code traces to flat Channel-3/SFT-style uses, build a canonical tool-call action algebra before any tree_controller.py, and spike Sandbox.fork() before committing to depth>1. (Design D-1 + D-2 + D-3 + D-4 + fidelity F-33.)

3. [V3] Zero benchmark decontamination anywhere in code or designs. The pipeline trains on SWE-bench-family substrates and will be scored on SWE-bench Verified; substrates have partial/divergent decontamination; HeldoutSplit is internal-only; the word does not appear in F1/F2/ADR-010. Must exist before the first corpus is generated. (Design D-5.)

4. [V4] Buy-vs-build inversion: the repo plans to hand-build what pip install swesmith ships, while its own core mechanic (gold-patch reversion) is SWE-smith's PR Mirror — validated as the best training data by an ablation the repo never cites. The user's "point-at-a-repo" ask is ~100 LOC of swesmith adapter, not an unspecified image-builder subsystem ($1,360 + 20 human-hours is the budget bar to beat). Relatedly, the implemented "feature deletion" is the inversion analogue, not the blog's delete-from-functional-codebase mechanism — and downstream docs should say so. (Design D-6 + fidelity F-9.)

Tier 2 — P0 fidelity corrections (high confidence, citation/claim integrity)

5. [V5] Fabricated quantities circulating as Cursor-stated facts: (a) benchmark numbers 69.3%/Terminal-Bench parity appear in no primary source (research/01:63-64); (b) "24 other generators" is a back-formation from the 25x task multiplier (COMPOSER_RECIPE_MAPPING.md:75, research/06:330, research/09:23); (c) "85% post-training compute" asserted unhedged in the research/01 body. Strike or inline-tag all three. (Fidelity F-1 + F-2 + F-8.)

6. [V6] World-model aux-head's load-bearing citation is a misread: CWM trains-on-all in a dedicated mid-training stage, not as an aux head riding RL policy gradients; 65.8% requires the test-time-scaling qualifier; Chain-of-World is robotics; "parameter isolation eliminates interference" overclaims DART. The exact proposed configuration has zero published evidence — keep it an ablation arm and gate wm_tuples/ emission (highest-volume prefix, most speculative consumer) behind the scheduled ablation. (Fidelity F-5 + F-6 + F-7 + F-23 + design D-14.)

7. [V7] Streaming DiLoCo citation is doubly wrong — arXiv:2501.18512 is Douillard et al., "Streaming DiLoCo with overlapping communication"; "Eager Updates…" is arXiv:2502.12996 (Kale et al.); "Liu et al." is the Async Local-SGD paper. Fix diloco/__init__.py:8 and design-F4:109. Low blast radius, unambiguous. (Fidelity F-4.)

Tier 3 — P1, high confidence (will bite during the build)

8. [V8] Two unreconciled S3 contracts (F1 vs F2) + golden_diff serialization leak. Write s3_contract.py first with one layout and two explicit serializers (to_policy_row drops golden_diff/deleted_symbols; unit-test the absence); fold divergence_pairs into corpus_dpo provenance; separate the DiLoCo rendezvous prefix/bucket. (Design D-7 + D-8 + D-19.)

9. [V9] No secrets/PII scrub at Claude Code trace ingest — raw sessions (file contents, keys in tool outputs, proprietary code) go to S3 verbatim; the only license filter applies to tasks, and it is a lowercase substring match with no SPDX detection and no trainable-vs-redistributable split for the repo-ingest path. (Design D-10 + D-13.)

10. [V10] The blog's CREATE-half of the dynamic curriculum ("select for and create harder tasks") is absent from code and from every pipeline design — granularity hardcoded "feature", no escalation stage in the outer loop. Cheapest first implementation: SWE-smith Combine-Bugs (96.9% yield) as a task_escalation stage; at minimum reserve the stage boundary in the orchestration contract. (Fidelity F-10 ≡ design D-15.)

11. [V11] Cost claims mislabeled in ways that distort planning: "$0.98/trace verified floor" is a 50-step synthetic-state benchmark (real sessions ~$70–80 flat at 125–2,830 steps/session); "$64 ungated tree" is a flat 8-teacher×1000-step extrapolation, not a tree (true tree is O(N^D)). Fix teacher_replay.py:7-8, VISION_VALIDATION.md, comparisons.md:33, final report §10/§3. (Fidelity F-16 + F-17.)

12. [V12] Premature five-service AWS orchestration + flywheel hygiene gaps: F2 commits Glue/EMR/Batch/Bedrock-batch/Step-Functions before one local e2e run (ADR-010 gates passed against FakeSandbox); no cross-generation dedup (self-training collapse accelerant); no canonical trajectory IR; no restart/budget semantics. Build a local stage-driver first; add MinHash cross-run dedup + CanonicalTrajectory schema. (Design D-9 + D-12 + D-11 + D-21.)

Tier 4 — P1/P2 precision fixes (confirmed, lower blast radius)

13. [V13] Preset/config drift vs papers: GSPO preset missing 3e-4/4e-4 epsilons (operationally not GSPO); CISPO preset should set beta=0.0 explicitly; kl_in_reward.py:12 "only reverse-KL option" → "default/recommended"; Comedy-of-Estimators citation rests on abstract-only. (Fidelity F-20 + F-18 + F-19.)

14. [V14] Provenance/novelty bookkeeping: cite Tree-GRPO (2509.21240) and SWE-Search (2410.20285) as nearest neighbors and fix the rStar granularity misread (framework/…:17, research/05); re-scope "nine-tenths reuse" to the recipe-replication layer (tree/wm-head/pipeline are 0% built); note verl async path is experimental. (Fidelity F-21 + F-22 + F-24.)

15. [V15] Remaining P2 wording fixes from the fidelity critic (F-25…F-33) — stage attribution of Muon, deleter-unknown framing, 25x non-convertibility, grounding-doc paraphrase-as-quote, DiLoCo FP4/H-default details, Foresight@k "(we define this)", SWE-rebench unverified counts — all consistent with the deep-reads; apply as a batch documentation pass. Plus design P2s: node-ID contract, image-digest pinning + dataset cards, corpus acceptance probe (small-model SFT delta before GPU spend).

---

Coverage note

The design critic's minimal-pipeline proposal (swesmith + SPDX/decontamination gates + SWE-agent rollout harness + canonical trajectory schema + s3_contract.py, ≈600–900 LOC) is consistent with every confirmed finding above and is the recommended Stage-0 build order. The feasibility critic (09) was never produced; its absence means cost/timeline feasibility claims in the designs have had only the fidelity-angle checks (V11) — treat any standalone feasibility assertions as unreviewed.