composer-replication-framework / docs /adrs /ADR-010-feature-deletion-datagen.md

feat(b4-gpu+b6): GPU train-proof on A10G + docker-gated substrate E2E test

c009712 about 2 months ago

14.7 kB

status: accepted
date: 2026-05-29T00:00:00.000Z
deciders:
  - Codeseys
  - ARIA

ADR-010: Build a FeatureDeletionEnv synthetic-data subsystem over OSS SWE substrates

Context and Problem Statement

Composer 2.5 trained on "25× more synthetic tasks than Composer 2," with Feature Deletion as a named generator: take a repo with passing tests, delete code/features, the agent must reimplement to make the tests pass — tests are the verifiable reward. The Cursor blog also notes the data is a dynamic difficulty curriculum ("we both select for and create harder tasks dynamically throughout the run"); the Composer 2 tech report adds the curriculum is keyed on rollout #turns + thinking-token count (research/10-composer2-techreport-mining.md). The blog reports real reward-hacking (decompiling Java bytecode, reverse-engineering Python type-check caches to recover deleted signatures) mitigated by "agentic monitoring tools."

The framework has no synthetic-data-generation subsystem at all — this is genuinely greenfield. Codeseys' ask is to bring Composer's dataset-generation approach into the framework as a real, reusable subsystem (useful beyond this project). Research doc research/06-feature-deletion-datagen.md designs it and identifies ready OSS substrates: SWE-bench-Lite, SWE-Gym (2.4k), R2E-Gym (8.1k), SWE-rebench (21.3k), Nemotron/OpenHands (59k) — each ships (repo, gold_patch, FAIL_TO_PASS, PASS_TO_PASS, test_cmd) tuples that invert directly into Feature-Deletion tasks (revert the gold patch → broken repo; FAIL_TO_PASS tests = the reward signal; PASS_TO_PASS = the don't-break- existing-behavior guard).

Relationship to prior ADRs — `preserves`

No prior ADR governs synthetic data generation. ADR-002 (trace source) chose Claude Code JSONL for ingestion of real traces — orthogonal to generating tasks. This ADR preserves everything; it adds a new composer_replication.datagen package.

Decision Drivers

Reuse existing verified OSS substrates rather than scraping repos from scratch — they already guarantee test-exercises-the-code via FAIL_TO_PASS.
The env must fit the framework's RL side (TRL GRPOTrainer + verifiers / OpenEnv-compatible) so generated tasks feed the same training loop.
Reward-hacking is a real reported failure, not theoretical — safeguards must be in the design, not deferred.
The online difficulty curriculum is part of the recipe, not optional.
Subsystem should be reusable for the owner's other project (general "verifiable-reward task generation over a code repo").

Considered Options

A. FeatureDeletionEnv that inverts OSS SWE substrates (revert gold patch) + online pass-rate difficulty gate + sandbox + signature/patch-provenance reward-hacking safeguards (chosen)
B. Greenfield repo-scraping generator (clone arbitrary GitHub repos, delete AST nodes, hope tests cover them)
C. Skip generation; reuse SWE-bench-lite tasks as-is without a deletion/inversion layer

Decision Outcome

Chosen: Option A — a composer_replication.datagen package with a FeatureDeletionTask dataclass, a FeatureDeletionEnv (Gym/OpenEnv-style reset/step/reward where reward = masked test-pass fraction), substrate adapters that invert the 5 OSS datasets by reverting their gold patch, a 4-gate solvability validator (broken repo fails FAIL_TO_PASS, passes PASS_TO_PASS, gold patch restores green, deletion is reachable from tests), an online pass-rate difficulty gate, and reward-hacking safeguards (pre-task scrub of __pycache__/.mypy_cache/.class/.git; allowlisted sandbox without find/strings/unzip/decompilers; signature + patch-provenance monitor that masks reward when deleted symbols reappear via non-implementation paths — including string-concat-obfuscated cache reads). A TRL reward_fn(prompts, completions, **kwargs) -> list[float] adapter wires it to the RL loop.

Consequences

Positive: Verifiable reward for free (tests already exist + are known to exercise the code via FAIL_TO_PASS); no need to generate or trust new tests.
Positive: Reusable general subsystem — "invert a solved-repo dataset into a reimplement-to-pass task" works for the owner's other project.
Positive: Online difficulty gate matches the actual recipe.
Negative: Bounded to what the OSS substrates cover (Python-dominant; SWE-bench is Python/JS-heavy). Other languages need new substrates. Documented as a known coverage limit.
Negative: Running tests in a sandbox requires Docker images per substrate; CPU-pool generation has real wall-clock cost (~15 node-days to invert all 21k SWE-rebench tasks per research/06). Mitigated by reusing the substrates' published Docker images and generating lazily.
Negative: Reward-hacking safeguards are a moving target; the signature + patch-provenance monitor is heuristic and will have false negatives. Mitigated by treating it as defense-in-depth (sandbox lockdown is the primary control) and logging suspected hacks for review.
Neutral: Adds a [datagen] optional extra (datasets, docker SDK).

Pros and Cons of the Options

A. Invert OSS substrates

Good: tests guaranteed to exercise the deleted code; verified datasets; reusable; matches recipe.
Bad: language/domain coverage bounded by the substrates; sandbox/Docker cost.

B. Greenfield repo scraping

Good: unbounded repo universe.
Bad: no guarantee remaining tests exercise the deleted code (the hard part SWE-bench already solved); huge validation burden; reinvents SWE-Gym. High effort, low marginal value.

C. Reuse SWE-bench-lite as-is

Good: zero build.
Bad: not Feature Deletion — it's issue-fixing; no controlled difficulty knob; no deletion mechanic; doesn't bring Composer's data-gen method in, just consumes an existing benchmark. Fails the actual ask.

Post-acceptance cross-family review (2026-05-29)

The 5-family review (see ADR-008's review section) was 3 REJECT / 1 ACCEPT-WITH-FIXES on ADR-010 — the harshest of the three. The reviewers' core, correct objection: the [x] gates were satisfied against FakeSandbox materializers that directly assign pass/fail booleans, so they prove the control-flow plumbing, NOT that feature-deletion-and-reimplement works on a real repo. ADR-010's central claim ("invert OSS SWE substrates") is exactly the part that is [~] (Docker-deferred) — the accepted gates are the easy half. This is fair; the gate language below is re-scoped accordingly. Findings verified and remediated where possible without Docker:

[FIXED — P0, 4/4 reviewers] Reward-hack sandbox lockdown was the ADR's claimed PRIMARY control but was UNIMPLEMENTED. LocalSubprocessSandbox.boot only recorded the image string; no cache/.git scrub existed, so the (bypassable) command denylist was the only defense. Worse, the denylist checks only the first whitespace token, so python -c "import marshal,dis; …read __pycache__/*.pyc", /usr/bin/strings, sh -c "strings x" all bypass it, and the HackMonitor regex is defeated by string-concat ("__py"+"cache__"). Fix: boot() now physically scrubs __pycache__, .mypy_cache, .pytest_cache, .git, .hg, and *.pyc/*.pyo/*.class from the working tree before the episode — the actual primary control. The denylist is now documented as cheap defense-in-depth, not the wall. Tested: test_local_sandbox_scrubs_caches_and_git_on_boot.
[FIXED — P0, 3/4 reviewers] Curriculum recorded partial multi-feature reward as a full pass. reward_fn updated the curriculum with int(res.reward > 0), so a 0.5 (1-of-2 features) reward logged as a 100% pass; p_hat crossed tau_easy and the task was retired before the policy learned the second feature. Fix: the curriculum now takes FRACTIONAL credit (n_pass: float), and reward_fn feeds the pass-fraction; a guard-broken or hack-flagged rollout contributes 0 credit but still counts as an exposure (so hack-only tasks trend to quarantine, not phantom passes). Tested: test_partial_multifeature_reward_is_fractional_credit_not_a_full_pass, test_guard_broken_trajectory_does_not_pollute_curriculum_with_a_pass.
[FIXED] reward_fn could silently return the wrong reward count. The zip(completions, task_id) truncated on length mismatch, returning fewer rewards than TRL expects and corrupting the advantage computation. Fix: explicit length-guard raises. Tested: test_reward_fn_rejects_length_mismatch.
[FIXED] Shell injection / parameterized-test breakage in run_tests (DeepSeek + Gemini). f"{test_command} {node_ids}" with shell=True breaks on SWE-bench parametrized node ids (test_x.py::test_y[a b]) and is injectable. Fix: shlex.quote each node id.
[FIXED] Duplicate "unzip" in SANDBOX_DENYLIST (multiple reviewers).
[OPEN — honest, the central re-scope] Gate 2 ("deletion breaks the feature") and the "deletion reachable from tests" claim do NOT verify reachability (GPT-5.5 P0, Grok P0). Gate 2 only checks that FAIL_TO_PASS tests fail in the broken state — it does not prove the failure was caused by the reverted feature rather than an unrelated breakage. A real reachability check (coverage of the changed region by the failing tests, or revert-provenance) needs the live Docker materializers. This is the same [~] gate as the substrate-inversion e2e — see below.
[RESOLVED — ADR-012] HackMonitor was a substring matcher, not the AST-provenance monitor the ADR advertised (DeepSeek P0). It flagged cache/decompiler signatures in the trajectory but did no symbol-reappearance analysis, and was bypassable by string-concat. With the scrub now in place as the primary control, the monitor is correctly-scoped defense-in-depth. ADR-012 re-scoped the language to "signature + patch-provenance monitor" (not "AST") and added a patch-provenance layer: a deleted symbol reappearing verbatim in the agent's patch alongside a cache/bytecode read — normalized to defeat string-concat obfuscation ("__py"+"cache__") — is now flagged.
[OPEN — recipe fidelity] Curriculum ignores rollout-turns and thinking-token count (DeepSeek, GPT-5.5). The Composer 2 tech report keys the curriculum on these; the implementation tracks only pass-rate. Follow-up: extend DifficultyCurriculum.update to accept and weight on turn/think-token signals.

The three OPEN items all require the Docker substrate e2e to close (real reachability, real provenance on a materialized repo). That gate remains the honest [~] — see the unblocked-by note. The fixable correctness defects (scrub, fractional curriculum, length-guard, shlex) are fixed and tested.

Acceptance gate (must be green before status flips to accepted)

Core gates green as of 2026-05-29 (19 tests in composer_replication/datagen/tests/test_feature_deletion.py, all CPU via FakeSandbox). The single Docker-dependent gate (real substrate inversion) is implemented but its live run is the documented unblocked-by step — see note.

FeatureDeletionTask dataclass + FeatureDeletionEnv (reset/step/reward) implemented; reward = masked test-pass fraction — test_reward_is_pass_fraction_when_guard_ok, test_reward_graded_for_multi_feature (0.5 for 1-of-2), test_reward_zeroed_when_functional_guard_broken. golden_diff held out of repr (test_golden_diff_not_in_repr).
[] SWE-bench-Lite substrate adapter: schema inversion implemented + tested (SweBenchAdapter.to_task — test_swebench_adapter_inverts_instance, JSON-or-list FAIL_TO_PASS handling, copyleft filter). The live revert-gold-patch → broken-repo → test-run path requires a substrate Docker image; LocalSubprocessSandbox + validate_task are wired for it, and the gate is exercised in unit form via FakeSandbox materializers (test_validator_accepts_well_formed_task). UPDATE 2026-05-29 (B6): the skipif(docker) end-to-end test now EXISTS — composer_replication/datagen/tests/test_docker_substrate_e2e.py runs the 4 gates against a REAL python:3.11-slim container with a real subprocess pytest runner on a minimal synthetic feature-deletion task (proving the inversion mechanics on real hardware), plus a cache-scrub-in-container test. It SKIPS cleanly in this no-Docker CPU env and ACTIVELY RUNS the moment a Docker host executes the suite. Still `[]` because (a) no Docker in this env to demonstrate a green run, and (b) the full SWE-bench-Lite image variant (pull one published instance image, revert its real gold patch) is a follow-up reusing the same driver. The mechanics gate is now ready-to-close, not just wired.
4-gate solvability validator implemented; test_validator_rejects_unreachable_deletion (deletion that doesn't break the target → gate 2 fails) and test_validator_rejects_when_guard_breaks (gate 3 fails).
Reward-hacking safeguard: SANDBOX_DENYLIST blocks find/strings/unzip/decompilers/git (test_sandbox_denies_decompiler_and_cache_tools); HackMonitor flags cache/bytecode-provenance hacks (test_monitor_flags_cache_provenance_hack) and passes clean reimplementation (test_monitor_passes_clean_reimplementation); reward is masked to 0 when a hack is detected even if tests "pass" (test_reward_masked_when_hack_detected).
Online difficulty gate: DifficultyCurriculum up-weights the frontier (~0.5 pass-rate) over aced tasks and retires aced ones (test_curriculum_upweights_frontier_over_solved); quarantines all-fail tasks after min_exposures (test_curriculum_quarantines_impossible_task). NOTE: quarantine uses the raw observed rate, not the Laplace-smoothed p_hat (smoothing is for weighting, not the have-we-ever-passed decision).
TRL reward_fn(prompts, completions, *, task_id, **kwargs) -> list[float] adapter returns one float in [0,1] per completion = masked pass-fraction (test_reward_fn_returns_one_float_per_completion); requires the task_id column (test_reward_fn_requires_task_id).
[datagen] optional extra added to pyproject.toml (datasets + docker); pure-Python core needs only datasets.

More Information

research/06-feature-deletion-datagen.md — full design: substrates w/ HF ids + licenses, deletion mechanics, safeguards, env + reward_fn sketches, cost verdict.
Substrates: SWE-bench (arXiv:2310.06770), SWE-Gym (arXiv:2412.21139), R2E-Gym, SWE-rebench.
Composer data-gen: docs/COMPOSER_RECIPE_MAPPING.md §2, research/10-composer2-techreport-mining.md §1.2 (curriculum).