auto-sync 2026-07-03T16:06:56Z workspace (part 5)

workspace/reports/2026-07-03.md

@@ -239,6 +239,20 @@
   - `runs/ctt_base_context_obs_learned_dominance_context_val_to_test`: context learned dominance selected success `0.3264`, coverage `0.5069`
   - ablations `success`, `success_weighted`, `context_success_weighted`, `tangent`, and `context_tangent` range from `0.2847` to `0.3125`
   - interpretation: visual-stat support plus validation-calibrated context dominance is the best held-out selected-success diagnostic so far, improving over base `0.2917` and the previous learned ridge row `0.3056`. It still misses the selected-success gate by a large margin and leaves a `0.2431` selector gap.
+- Leakage-hardened train-only calibration rollout prep:
+  - updated `scripts/eval_ctt_generated_rollout.py` with `--exclude-self-source`, which excludes train source charts with the same `chart_id` or `state_hash` as the target chart.
+  - source chart metadata is now loaded during measured rollout so `base_context_obs` source features use the same deployment-visible observation embeddings as train/proxy CTT, instead of silently falling back to zero observation features.
+  - updated `scripts/slurm/eval_ctt_generated_rollout.sbatch` with `EXCLUDE_SELF_SOURCE=1`.
+  - added a regression test for source-pool self-exclusion in `tests/test_ctt.py`.
+  - verification: `python -m py_compile scripts/eval_ctt_generated_rollout.py scripts/eval_dominance_selector.py scripts/eval_learned_dominance_selector.py scripts/build_ctt_rollout_comparison.py scripts/eval_metrics.py cil/metrics.py` passed; `.venv/bin/python -m pytest tests/test_ctt.py tests/test_metrics.py tests/test_cil_metrics.py tests/test_dominance_selector.py tests/test_chart_features.py -q` passed with `24 passed`.
+- Submitted train-only measured calibration rollout jobs:
+  - jobs: `15115695`, `15115696`, `15115697`
+  - source index: `data/cil_charts_rgb_refs/train/index.json`
+  - target index: `data/cil_charts_rgb_refs/train/index.json`
+  - checkpoints: `runs/ctt_residual_base_context_obs_seed{0,1,2}/model.pt`
+  - out dirs: `runs/ctt_residual_base_context_obs_rollout_train_cal_seed{0,1,2}`
+  - settings: `K=8`, `MAX_TARGET_CHARTS=144`, `NEIGHBORS=8`, `EXCLUDE_SELF_SOURCE=1`, CPU ManiSkill rollout.
+  - status at submission: pending in Slurm queue.
 
 ## Failed Jobs And Log Excerpts
 
@@ -325,15 +339,18 @@ Interpretation: the test proposal oracle clears the internal 50% support target,
 - Added RGB-ref/embedding audit evidence, the three-seed `base_context_obs` proxy row, measured validation/test rollout aggregates, and visual-stat dominance selector diagnostics. The draft describes this as a deterministic visual-stat support plus selector diagnostic that improves held-out selected success, but still not object-centric, learned visual-language, or enough for selected-action deployment evidence.
 - Added `cil/chart_features.py`, `scripts/audit_chart_feature_sources.py`, `scripts/export_chart_observation_embeddings.py`, and `scripts/slurm/train_ctt_feature_proxy.sbatch` to the reproducibility artifact list.
 - Reframed the title/abstract around `Counterfactual Action Atlas`; CTT is now the current transport generator inside the Atlas, not the whole novelty.
+- Tightened the abstract to avoid overclaiming a successful learned generator: the paper now says the Atlas measures local do-action geometry and audits whether deployment-clean generators reach positive support.
+- Added the train-calibration leakage rule to the split contract: when target charts are from train, retrieval excludes same chart id and same restored-state hash.
+- Added measured-rollout reproducibility notes for source metadata loading and self-source exclusion.
 - Updated limitations: current evidence includes validation/test measured rollout, but the selected action fails the acceptance gate; real-robot recovery, unsafe-contact measurement, and dominance-calibrated deployment remain missing.
 - Added local-atlas/proxy-comparison scripts to the reproducibility artifact list.
 - Added generated-rollout script and Slurm wrapper to the reproducibility artifact list.
 
 ## Next Actions
 
-1. Replace the current weak train-only utility-energy/dominance model with richer visual-language/object-centric chart tokens; checkpoint scorers reached only `23.61%`--`28.47%` selected success across three seeds, and context metadata reached only `29.17%`.
-2. Move beyond deterministic RGB statistics toward learned visual-language/object-centric chart tokens; the best current visual-stat learned context dominance row reaches `0.3264` selected success but remains far below the gate.
-3. Train a stronger deployment-clean dominance model on train-only data, because validation-calibrated ridge/context selectors improve held-out success but still leave a large selector gap.
-4. Add unsafe execution rate and calibrated fallback rate to measured rollout artifacts.
-5. Improve chart token from base-action summary to visual-language/object-centric geometry to reduce SupportGap and SelectorRegret.
+1. Wait for train-only calibration rollout jobs `15115695`--`15115697`; aggregate them with `scripts/build_ctt_rollout_comparison.py`.
+2. Train/evaluate train-calibrated dominance selectors on held-out test measured rows. This avoids choosing thresholds/models directly from validation measured outcomes as the best result.
+3. Replace the current weak train-only utility-energy/dominance model with richer visual-language/object-centric chart tokens; checkpoint scorers reached only `23.61%`--`28.47%` selected success across three seeds, and context metadata reached only `29.17%`.
+4. Move beyond deterministic RGB statistics toward learned visual-language/object-centric chart tokens; the best current visual-stat learned context dominance row reaches `0.3264` selected success but remains far below the gate.
+5. Add unsafe execution rate and calibrated fallback rate to measured rollout artifacts.
 6. Rerun validation/test measured rollouts after the selector is changed, not only proxy support sweeps.

workspace/reports/draft_audit_ctt.md

@@ -103,6 +103,10 @@ Current artifact evidence:
 - `scripts/eval_ctt_generated_rollout.py` and
   `scripts/slurm/eval_ctt_generated_rollout.sbatch` implement measured
   generated-candidate rollout rows for OutcomePTR/SupportGap/SelectorRegret.
+  The rollout path now loads source metadata for deployment-visible chart
+  features such as `base_context_obs`, and exposes `--exclude-self-source` /
+  `EXCLUDE_SELF_SOURCE=1` so train-split calibration cannot retrieve the target
+  chart's own positives by `chart_id` or `state_hash`.
 - `runs/ctt_residual_rollout_val69_seed{0,1,2}` and
   `runs/ctt_val_rollout_comparison` contain the first full validation measured
   rollout results for residual CTT.
@@ -181,9 +185,12 @@ so deployment success remains unclaimed.
 Required fix:
 
 - The abstract now says CIL-Atlas measures local causal action geometry and CTT
-  is the current transport instance.
+  is the current transport instance; it no longer presents the generator as a
+  solved deployment method.
 - Keep the test result phrased as support evidence: proposal oracle passes
   50%, selected action fails, calibrated dominance is required next.
+- For train-only calibration rollouts, use `--exclude-self-source` so source
+  retrieval stays train-only but does not self-copy the target chart.
 
 ### A2. Generic Generator Instead Of CTT