anhtld commited on
Commit
dd4ed55
·
verified ·
1 Parent(s): 7e43e84

auto-sync 2026-07-03T18:45:41Z workspace (part 2)

Browse files
Files changed (1) hide show
  1. workspace/reports/2026-07-03.md +13 -1
workspace/reports/2026-07-03.md CHANGED
@@ -20,6 +20,7 @@
20
  - Added `scripts/audit_action_bounds.py` to audit whether exported chart actions lie inside the ManiSkill deployment action space and to write a full run artifact under `runs/action_bound_audit_rgb_refs`.
21
  - Extended `scripts/audit_action_bounds.py` with per-dimension action/base-action/base-branch max, mean-absolute, violation-rate, and scale diagnostics so the action-representation mismatch is attributable by action dimension rather than only by row-level max excess.
22
  - Added `--execution-action-scale` to `scripts/eval_ctt_generated_rollout.py` and `EXECUTION_ACTION_SCALE` to the Slurm wrapper. The scale is applied before action-bound diagnostics, optional clipping, and simulator execution, and is logged in the decoder plus each measured row.
 
23
  - Extended `scripts/eval_learned_dominance_selector.py` with logged `--feature-set` and `--target` variants, including success-weighted and tangent-code diagnostics, while keeping the original utility-margin default unchanged.
24
  - Extended `scripts/eval_learned_dominance_selector.py` again with deployment-visible `context` and `context_tangent` feature sets using target task id, instruction hash, source task id, and source-target task match. These features do not read evaluator-only val/test outcomes.
25
  - Extended `scripts/eval_dominance_selector.py` with `--score-source checkpoint`, cached chart embeddings/base scores, and single-thread torch scoring so a standalone train-only utility-energy checkpoint can be used for dominance evaluation.
@@ -113,6 +114,11 @@
113
  - `15118838`, `15118839`, `15118840`: validation `base_context_obs`, seeds 0/1/2, 69 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
114
  - `15118841`, `15118842`, `15118843`: test `base_context_obs`, seeds 0/1/2, 48 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
115
  - final scheduler state: all six full validation/test jobs completed with exit code `0:0`.
 
 
 
 
 
116
 
117
  ## Jobs Completed
118
 
@@ -284,6 +290,11 @@
284
  - test, 144 rows: base success `0.2292`, selected success `0.0903`, proposal oracle `0.1667`, hidden oracle `0.7292`, OutcomePTR@8 `0.3750`, success support gap `0.5625`, success selector gap `0.0764`, generated action-bound violation rate `0.1997`, base action-bound violation rate `0.0000`.
285
  - NCAR is now exported in measured reports only for rows with a nontrivial base-to-oracle utility gap; scaled validation NCAR-to-hidden is `1.7011` over 174 rows and scaled test NCAR-to-hidden is `1.1567` over 111 rows.
286
  - interpretation: global max-fit scaling fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations. This rejects a naive scalar post-processing fix; action representation/decoder correction is now required.
 
 
 
 
 
287
  - No-clipping visual-stat CTT measured rollout refresh jobs `15117739`--`15117744` completed with exit code `0:0`.
288
  - validation seed artifacts: `runs/ctt_residual_base_context_obs_rollout_val69_noclip_seed{0,1,2}`
289
  - test seed artifacts: `runs/ctt_residual_base_context_obs_rollout_test_noclip_seed{0,1,2}`
@@ -440,7 +451,7 @@ Interpretation: the test proposal oracle clears the internal 50% support target,
440
 
441
  Safety metric status: regenerated measured rollout reports now expose `base_safety_label_known`, `generated_safety_label_coverage_at_8`, and `selected_safety_label_known_at_8`. For the visual-stat, object-layout, held-out test, and train-calibration rollout rows generated before the action-bound hook, each coverage number is `0.0000`; the legacy validation aggregate without `candidate_outcomes` reports `n/a`. For action-bound smoke/no-clipping/scaled artifacts, coverage is `1.0000` because `safety_violation_source=action_bounds`. These are action-space validity labels only, not collision/contact labels.
442
 
443
- Action-bound audit status: new rollout artifacts now log action-bound safety labels. The first smoke has 100% known labels and 100% action-bound violation for base/generated candidates. The full chart DB audit shows base-action violation rate 1.0000 in train/val/test and branch-level violation rate about 0.308. The no-clipping refresh has 100% known labels and about 0.98 generated action-bound violation rate. The scaled0.215 refresh has 100% known labels and base action-bound violation rate `0.0000`, but generated violation rate remains `0.1920` on validation and `0.1997` on test while proposal support collapses. Collision/contact safety remains unmeasured.
444
 
445
  ## Leakage Audit Status
446
 
@@ -498,6 +509,7 @@ Action-bound audit status: new rollout artifacts now log action-bound safety lab
498
  - Added CILBench benchmark-track and branch-family language matching the advisor spine: Ranking, Generation, Deployment, and Recovery tracks all share the same same-state chart primitive.
499
  - Added NCAR to the canonical metric section and evaluator artifacts. The evaluator reports NCAR only when the base-to-oracle utility gap is nontrivial, avoiding misleading normalized-regret blowups.
500
  - Added scaled0.215 validation/test aggregate tables via `\input{../runs/ctt_base_context_obs_val_scaled0215_rollout_comparison/table}` and `\input{../runs/ctt_base_context_obs_test_scaled0215_rollout_comparison/table}`. The paper treats global scaling as a negative diagnostic: it fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations.
 
501
  - Verification after the NCAR/scaled-refresh paper patch:
502
  - `.venv/bin/python -m py_compile cil/metrics.py cil/__init__.py scripts/eval_metrics.py scripts/build_ctt_rollout_comparison.py scripts/eval_ctt_generated_rollout.py scripts/audit_action_bounds.py tests/test_metrics.py tests/test_ctt.py`
503
  - `.venv/bin/python -m pytest tests/test_ctt.py tests/test_metrics.py tests/test_cil_metrics.py tests/test_dominance_selector.py tests/test_chart_features.py -q`: 34 passed.
 
20
  - Added `scripts/audit_action_bounds.py` to audit whether exported chart actions lie inside the ManiSkill deployment action space and to write a full run artifact under `runs/action_bound_audit_rgb_refs`.
21
  - Extended `scripts/audit_action_bounds.py` with per-dimension action/base-action/base-branch max, mean-absolute, violation-rate, and scale diagnostics so the action-representation mismatch is attributable by action dimension rather than only by row-level max excess.
22
  - Added `--execution-action-scale` to `scripts/eval_ctt_generated_rollout.py` and `EXECUTION_ACTION_SCALE` to the Slurm wrapper. The scale is applied before action-bound diagnostics, optional clipping, and simulator execution, and is logged in the decoder plus each measured row.
23
+ - Added `--execution-action-transform {identity,tanh}` and optional `--execution-action-scale-vector` to `scripts/eval_ctt_generated_rollout.py`, plus `EXECUTION_ACTION_TRANSFORM` and `EXECUTION_ACTION_SCALE_VECTOR` in the Slurm wrapper. The new `tanh` convention is a bounded decoder diagnostic: it maps decoded controls smoothly into finite env action bounds before action-bound diagnostics and rollout, and logs the transform/vector in the decoder and each measured row.
24
  - Extended `scripts/eval_learned_dominance_selector.py` with logged `--feature-set` and `--target` variants, including success-weighted and tangent-code diagnostics, while keeping the original utility-margin default unchanged.
25
  - Extended `scripts/eval_learned_dominance_selector.py` again with deployment-visible `context` and `context_tangent` feature sets using target task id, instruction hash, source task id, and source-target task match. These features do not read evaluator-only val/test outcomes.
26
  - Extended `scripts/eval_dominance_selector.py` with `--score-source checkpoint`, cached chart embeddings/base scores, and single-thread torch scoring so a standalone train-only utility-energy checkpoint can be used for dominance evaluation.
 
114
  - `15118838`, `15118839`, `15118840`: validation `base_context_obs`, seeds 0/1/2, 69 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
115
  - `15118841`, `15118842`, `15118843`: test `base_context_obs`, seeds 0/1/2, 48 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=0.215`
116
  - final scheduler state: all six full validation/test jobs completed with exit code `0:0`.
117
+ - Submitted bounded-decoder tanh visual-stat CTT measured rollout diagnostics after global scaling fixed base bounds but collapsed proposal support:
118
+ - smoke `15119536`: validation `base_context_obs`, seed 0, one chart, K=2, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_SCALE=1.0`, `EXECUTION_ACTION_TRANSFORM=tanh`
119
+ - `15119566`, `15119567`, `15119568`: validation `base_context_obs`, seeds 0/1/2, 69 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_TRANSFORM=tanh`
120
+ - `15119569`, `15119570`, `15119571`: test `base_context_obs`, seeds 0/1/2, 48 charts, K=8, `CLIP_ACTIONS=0`, `EXECUTION_ACTION_TRANSFORM=tanh`
121
+ - latest scheduler state: smoke completed; full validation/test jobs pending by priority.
122
 
123
  ## Jobs Completed
124
 
 
290
  - test, 144 rows: base success `0.2292`, selected success `0.0903`, proposal oracle `0.1667`, hidden oracle `0.7292`, OutcomePTR@8 `0.3750`, success support gap `0.5625`, success selector gap `0.0764`, generated action-bound violation rate `0.1997`, base action-bound violation rate `0.0000`.
291
  - NCAR is now exported in measured reports only for rows with a nontrivial base-to-oracle utility gap; scaled validation NCAR-to-hidden is `1.7011` over 174 rows and scaled test NCAR-to-hidden is `1.1567` over 111 rows.
292
  - interpretation: global max-fit scaling fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations. This rejects a naive scalar post-processing fix; action representation/decoder correction is now required.
293
+ - Bounded tanh decoder smoke completed:
294
+ - artifact: `runs/ctt_residual_base_context_obs_rollout_safety_tanh_smoke_seed0`, job `15119536`, completed `0:0`
295
+ - setting: one validation chart, K=2, `--disable-action-clipping --execution-action-transform tanh --execution-action-scale 1.0`
296
+ - result: base and both generated candidates have known action-bound labels and zero action-bound violations (`base_action_clip_max_abs=0.0`, generated max excess `[0.0, 0.0]`). Base progress is `0.1471`; generated candidate progress is `[0.2366, 0.0794]`; OutcomePTR@2 is `1.0000`.
297
+ - interpretation: tanh is the first bounded decoder convention that both removes action-bound violations and improves over base on the one-chart smoke without global magnitude collapse. It is still only a smoke until full validation/test jobs `15119566`--`15119571` complete.
298
  - No-clipping visual-stat CTT measured rollout refresh jobs `15117739`--`15117744` completed with exit code `0:0`.
299
  - validation seed artifacts: `runs/ctt_residual_base_context_obs_rollout_val69_noclip_seed{0,1,2}`
300
  - test seed artifacts: `runs/ctt_residual_base_context_obs_rollout_test_noclip_seed{0,1,2}`
 
451
 
452
  Safety metric status: regenerated measured rollout reports now expose `base_safety_label_known`, `generated_safety_label_coverage_at_8`, and `selected_safety_label_known_at_8`. For the visual-stat, object-layout, held-out test, and train-calibration rollout rows generated before the action-bound hook, each coverage number is `0.0000`; the legacy validation aggregate without `candidate_outcomes` reports `n/a`. For action-bound smoke/no-clipping/scaled artifacts, coverage is `1.0000` because `safety_violation_source=action_bounds`. These are action-space validity labels only, not collision/contact labels.
453
 
454
+ Action-bound audit status: new rollout artifacts now log action-bound safety labels. The first smoke has 100% known labels and 100% action-bound violation for base/generated candidates. The full chart DB audit shows base-action violation rate 1.0000 in train/val/test and branch-level violation rate about 0.308. The no-clipping refresh has 100% known labels and about 0.98 generated action-bound violation rate. The scaled0.215 refresh has 100% known labels and base action-bound violation rate `0.0000`, but generated violation rate remains `0.1920` on validation and `0.1997` on test while proposal support collapses. The tanh bounded-decoder smoke has 100% known labels and zero base/generated action-bound violations on one chart; full validation/test tanh jobs are pending. Collision/contact safety remains unmeasured.
455
 
456
  ## Leakage Audit Status
457
 
 
509
  - Added CILBench benchmark-track and branch-family language matching the advisor spine: Ranking, Generation, Deployment, and Recovery tracks all share the same same-state chart primitive.
510
  - Added NCAR to the canonical metric section and evaluator artifacts. The evaluator reports NCAR only when the base-to-oracle utility gap is nontrivial, avoiding misleading normalized-regret blowups.
511
  - Added scaled0.215 validation/test aggregate tables via `\input{../runs/ctt_base_context_obs_val_scaled0215_rollout_comparison/table}` and `\input{../runs/ctt_base_context_obs_test_scaled0215_rollout_comparison/table}`. The paper treats global scaling as a negative diagnostic: it fixes base action-bound validity but collapses proposal support and leaves nonzero generated action-bound violations.
512
+ - Added bounded-decoder tanh support to the rollout artifact path and paper limitations language. The draft treats the one-chart tanh smoke as a diagnostic only and waits for full validation/test tanh aggregates before any method claim.
513
  - Verification after the NCAR/scaled-refresh paper patch:
514
  - `.venv/bin/python -m py_compile cil/metrics.py cil/__init__.py scripts/eval_metrics.py scripts/build_ctt_rollout_comparison.py scripts/eval_ctt_generated_rollout.py scripts/audit_action_bounds.py tests/test_metrics.py tests/test_ctt.py`
515
  - `.venv/bin/python -m pytest tests/test_ctt.py tests/test_metrics.py tests/test_cil_metrics.py tests/test_dominance_selector.py tests/test_chart_features.py -q`: 34 passed.