auto-sync 2026-07-03T18:45:41Z workspace (part 2)
Browse files
workspace/reports/2026-07-03.md
CHANGED
|
@@ -20,6 +20,7 @@
|
|
| 20 |
- Added `scripts/audit_action_bounds.py` to audit whether exported chart actions lie inside the ManiSkill deployment action space and to write a full run artifact under `runs/action_bound_audit_rgb_refs`.
|
| 21 |
- Extended `scripts/audit_action_bounds.py` with per-dimension action/base-action/base-branch max, mean-absolute, violation-rate, and scale diagnostics so the action-representation mismatch is attributable by action dimension rather than only by row-level max excess.
|
| 22 |
- Added `--execution-action-scale` to `scripts/eval_ctt_generated_rollout.py` and `EXECUTION_ACTION_SCALE` to the Slurm wrapper. The scale is applied before action-bound diagnostics, optional clipping, and simulator execution, and is logged in the decoder plus each measured row.
|
|
|
|
| 23 |
- Extended `scripts/eval_learned_dominance_selector.py` with logged `--feature-set` and `--target` variants, including success-weighted and tangent-code diagnostics, while keeping the original utility-margin default unchanged.
|
| 24 |
- Extended `scripts/eval_learned_dominance_selector.py` again with deployment-visible `context` and `context_tangent` feature sets using target task id, instruction hash, source task id, and source-target task match. These features do not read evaluator-only val/test outcomes.
|
| 25 |
- Extended `scripts/eval_dominance_selector.py` with `--score-source checkpoint`, cached chart embeddings/base scores, and single-thread torch scoring so a standalone train-only utility-energy checkpoint can be used for dominance evaluation.
|
|
@@ -113,6 +114,11 @@
|
|
| 113 |
- `15118838`, `15118839`, `15118840`: validation `base_context_obs`, seeds 0/1/2, 69 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
|
| 114 |
- `15118841`, `15118842`, `15118843`: test `base_context_obs`, seeds 0/1/2, 48 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
|
| 115 |
- final scheduler state: all six full validation/test jobs completed with exit code `0:0`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
## Jobs Completed
|
| 118 |
|
|
@@ -284,6 +290,11 @@
|
|
| 284 |
- test, 144 rows: base success `0.2292`, selected success `0.0903`, proposal oracle `0.1667`, hidden oracle `0.7292`, OutcomePTR@8 `0.3750`, success support gap `0.5625`, success selector gap `0.0764`, generated action-bound violation rate `0.1997`, base action-bound violation rate `0.0000`.
|
| 285 |
- NCAR is now exported in measured reports only for rows with a nontrivial base-to-oracle utility gap; scaled validation NCAR-to-hidden is `1.7011` over 174 rows and scaled test NCAR-to-hidden is `1.1567` over 111 rows.
|
| 286 |
- interpretation: global max-fit scaling fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations. This rejects a naive scalar post-processing fix; action representation/decoder correction is now required.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 287 |
- No-clipping visual-stat CTT measured rollout refresh jobs `15117739`--`15117744` completed with exit code `0:0`.
|
| 288 |
- validation seed artifacts: `runs/ctt_residual_base_context_obs_rollout_val69_noclip_seed{0,1,2}`
|
| 289 |
- test seed artifacts: `runs/ctt_residual_base_context_obs_rollout_test_noclip_seed{0,1,2}`
|
|
@@ -440,7 +451,7 @@ Interpretation: the test proposal oracle clears the internal 50% support target,
|
|
| 440 |
|
| 441 |
Safety metric status: regenerated measured rollout reports now expose `base_safety_label_known`, `generated_safety_label_coverage_at_8`, and `selected_safety_label_known_at_8`. For the visual-stat, object-layout, held-out test, and train-calibration rollout rows generated before the action-bound hook, each coverage number is `0.0000`; the legacy validation aggregate without `candidate_outcomes` reports `n/a`. For action-bound smoke/no-clipping/scaled artifacts, coverage is `1.0000` because `safety_violation_source=action_bounds`. These are action-space validity labels only, not collision/contact labels.
|
| 442 |
|
| 443 |
-
Action-bound audit status: new rollout artifacts now log action-bound safety labels. The first smoke has 100% known labels and 100% action-bound violation for base/generated candidates. The full chart DB audit shows base-action violation rate 1.0000 in train/val/test and branch-level violation rate about 0.308. The no-clipping refresh has 100% known labels and about 0.98 generated action-bound violation rate. The scaled0.215 refresh has 100% known labels and base action-bound violation rate `0.0000`, but generated violation rate remains `0.1920` on validation and `0.1997` on test while proposal support collapses. Collision/contact safety remains unmeasured.
|
| 444 |
|
| 445 |
## Leakage Audit Status
|
| 446 |
|
|
@@ -498,6 +509,7 @@ Action-bound audit status: new rollout artifacts now log action-bound safety lab
|
|
| 498 |
- Added CILBench benchmark-track and branch-family language matching the advisor spine: Ranking, Generation, Deployment, and Recovery tracks all share the same same-state chart primitive.
|
| 499 |
- Added NCAR to the canonical metric section and evaluator artifacts. The evaluator reports NCAR only when the base-to-oracle utility gap is nontrivial, avoiding misleading normalized-regret blowups.
|
| 500 |
- Added scaled0.215 validation/test aggregate tables via `\input{../runs/ctt_base_context_obs_val_scaled0215_rollout_comparison/table}` and `\input{../runs/ctt_base_context_obs_test_scaled0215_rollout_comparison/table}`. The paper treats global scaling as a negative diagnostic: it fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations.
|
|
|
|
| 501 |
- Verification after the NCAR/scaled-refresh paper patch:
|
| 502 |
- `.venv/bin/python -m py_compile cil/metrics.py cil/__init__.py scripts/eval_metrics.py scripts/build_ctt_rollout_comparison.py scripts/eval_ctt_generated_rollout.py scripts/audit_action_bounds.py tests/test_metrics.py tests/test_ctt.py`
|
| 503 |
- `.venv/bin/python -m pytest tests/test_ctt.py tests/test_metrics.py tests/test_cil_metrics.py tests/test_dominance_selector.py tests/test_chart_features.py -q`: 34 passed.
|
|
|
|
| 20 |
- Added `scripts/audit_action_bounds.py` to audit whether exported chart actions lie inside the ManiSkill deployment action space and to write a full run artifact under `runs/action_bound_audit_rgb_refs`.
|
| 21 |
- Extended `scripts/audit_action_bounds.py` with per-dimension action/base-action/base-branch max, mean-absolute, violation-rate, and scale diagnostics so the action-representation mismatch is attributable by action dimension rather than only by row-level max excess.
|
| 22 |
- Added `--execution-action-scale` to `scripts/eval_ctt_generated_rollout.py` and `EXECUTION_ACTION_SCALE` to the Slurm wrapper. The scale is applied before action-bound diagnostics, optional clipping, and simulator execution, and is logged in the decoder plus each measured row.
|
| 23 |
+
- Added `--execution-action-transform {identity,tanh}` and optional `--execution-action-scale-vector` to `scripts/eval_ctt_generated_rollout.py`, plus `EXECUTION_ACTION_TRANSFORM` and `EXECUTION_ACTION_SCALE_VECTOR` in the Slurm wrapper. The new `tanh` convention is a bounded decoder diagnostic: it maps decoded controls smoothly into finite env action bounds before action-bound diagnostics and rollout, and logs the transform/vector in the decoder and each measured row.
|
| 24 |
- Extended `scripts/eval_learned_dominance_selector.py` with logged `--feature-set` and `--target` variants, including success-weighted and tangent-code diagnostics, while keeping the original utility-margin default unchanged.
|
| 25 |
- Extended `scripts/eval_learned_dominance_selector.py` again with deployment-visible `context` and `context_tangent` feature sets using target task id, instruction hash, source task id, and source-target task match. These features do not read evaluator-only val/test outcomes.
|
| 26 |
- Extended `scripts/eval_dominance_selector.py` with `--score-source checkpoint`, cached chart embeddings/base scores, and single-thread torch scoring so a standalone train-only utility-energy checkpoint can be used for dominance evaluation.
|
|
|
|
| 114 |
- `15118838`, `15118839`, `15118840`: validation `base_context_obs`, seeds 0/1/2, 69 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
|
| 115 |
- `15118841`, `15118842`, `15118843`: test `base_context_obs`, seeds 0/1/2, 48 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
|
| 116 |
- final scheduler state: all six full validation/test jobs completed with exit code `0:0`.
|
| 117 |
+
- Submitted bounded-decoder tanh visual-stat CTT measured rollout diagnostics after global scaling fixed base bounds but collapsed proposal support:
|
| 118 |
+
- smoke `15119536`: validation `base_context_obs`, seed 0, one chart, K=2, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=1.0`, `EXECUTION_ACTION_TRANSFORM=tanh`
|
| 119 |
+
- `15119566`, `15119567`, `15119568`: validation `base_context_obs`, seeds 0/1/2, 69 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_TRANSFORM=tanh`
|
| 120 |
+
- `15119569`, `15119570`, `15119571`: test `base_context_obs`, seeds 0/1/2, 48 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_TRANSFORM=tanh`
|
| 121 |
+
- latest scheduler state: smoke completed; full validation/test jobs pending by priority.
|
| 122 |
|
| 123 |
## Jobs Completed
|
| 124 |
|
|
|
|
| 290 |
- test, 144 rows: base success `0.2292`, selected success `0.0903`, proposal oracle `0.1667`, hidden oracle `0.7292`, OutcomePTR@8 `0.3750`, success support gap `0.5625`, success selector gap `0.0764`, generated action-bound violation rate `0.1997`, base action-bound violation rate `0.0000`.
|
| 291 |
- NCAR is now exported in measured reports only for rows with a nontrivial base-to-oracle utility gap; scaled validation NCAR-to-hidden is `1.7011` over 174 rows and scaled test NCAR-to-hidden is `1.1567` over 111 rows.
|
| 292 |
- interpretation: global max-fit scaling fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations. This rejects a naive scalar post-processing fix; action representation/decoder correction is now required.
|
| 293 |
+
- Bounded tanh decoder smoke completed:
|
| 294 |
+
- artifact: `runs/ctt_residual_base_context_obs_rollout_safety_tanh_smoke_seed0`, job `15119536`, completed `0:0`
|
| 295 |
+
- setting: one validation chart, K=2, `--disable-action-clipping --execution-action-transform tanh --execution-action-scale 1.0`
|
| 296 |
+
- result: base and both generated candidates have known action-bound labels and zero action-bound violations (`base_action_clip_max_abs=0.0`, generated max excess `[0.0, 0.0]`). Base progress is `0.1471`; generated candidate progress is `[0.2366, 0.0794]`; OutcomePTR@2 is `1.0000`.
|
| 297 |
+
- interpretation: tanh is the first bounded decoder convention that both removes action-bound violations and improves over base on the one-chart smoke without global magnitude collapse. It is still only a smoke until full validation/test jobs `15119566`--`15119571` complete.
|
| 298 |
- No-clipping visual-stat CTT measured rollout refresh jobs `15117739`--`15117744` completed with exit code `0:0`.
|
| 299 |
- validation seed artifacts: `runs/ctt_residual_base_context_obs_rollout_val69_noclip_seed{0,1,2}`
|
| 300 |
- test seed artifacts: `runs/ctt_residual_base_context_obs_rollout_test_noclip_seed{0,1,2}`
|
|
|
|
| 451 |
|
| 452 |
Safety metric status: regenerated measured rollout reports now expose `base_safety_label_known`, `generated_safety_label_coverage_at_8`, and `selected_safety_label_known_at_8`. For the visual-stat, object-layout, held-out test, and train-calibration rollout rows generated before the action-bound hook, each coverage number is `0.0000`; the legacy validation aggregate without `candidate_outcomes` reports `n/a`. For action-bound smoke/no-clipping/scaled artifacts, coverage is `1.0000` because `safety_violation_source=action_bounds`. These are action-space validity labels only, not collision/contact labels.
|
| 453 |
|
| 454 |
+
Action-bound audit status: new rollout artifacts now log action-bound safety labels. The first smoke has 100% known labels and 100% action-bound violation for base/generated candidates. The full chart DB audit shows base-action violation rate 1.0000 in train/val/test and branch-level violation rate about 0.308. The no-clipping refresh has 100% known labels and about 0.98 generated action-bound violation rate. The scaled0.215 refresh has 100% known labels and base action-bound violation rate `0.0000`, but generated violation rate remains `0.1920` on validation and `0.1997` on test while proposal support collapses. The tanh bounded-decoder smoke has 100% known labels and zero base/generated action-bound violations on one chart; full validation/test tanh jobs are pending. Collision/contact safety remains unmeasured.
|
| 455 |
|
| 456 |
## Leakage Audit Status
|
| 457 |
|
|
|
|
| 509 |
- Added CILBench benchmark-track and branch-family language matching the advisor spine: Ranking, Generation, Deployment, and Recovery tracks all share the same same-state chart primitive.
|
| 510 |
- Added NCAR to the canonical metric section and evaluator artifacts. The evaluator reports NCAR only when the base-to-oracle utility gap is nontrivial, avoiding misleading normalized-regret blowups.
|
| 511 |
- Added scaled0.215 validation/test aggregate tables via `\input{../runs/ctt_base_context_obs_val_scaled0215_rollout_comparison/table}` and `\input{../runs/ctt_base_context_obs_test_scaled0215_rollout_comparison/table}`. The paper treats global scaling as a negative diagnostic: it fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations.
|
| 512 |
+
- Added bounded-decoder tanh support to the rollout artifact path and paper limitations language. The draft treats the one-chart tanh smoke as a diagnostic only and waits for full validation/test tanh aggregates before any method claim.
|
| 513 |
- Verification after the NCAR/scaled-refresh paper patch:
|
| 514 |
- `.venv/bin/python -m py_compile cil/metrics.py cil/__init__.py scripts/eval_metrics.py scripts/build_ctt_rollout_comparison.py scripts/eval_ctt_generated_rollout.py scripts/audit_action_bounds.py tests/test_metrics.py tests/test_ctt.py`
|
| 515 |
- `.venv/bin/python -m pytest tests/test_ctt.py tests/test_metrics.py tests/test_cil_metrics.py tests/test_dominance_selector.py tests/test_chart_features.py -q`: 34 passed.
|