anhtld commited on
Commit
733a6d7
·
verified ·
1 Parent(s): 273ece9

document CTT proxy gate diagnostics

Browse files
Files changed (1) hide show
  1. README.md +29 -2
README.md CHANGED
@@ -259,12 +259,15 @@ CTT training/proxy/rollout:
259
 
260
  - `scripts/train_ctt.py`: trains residual or gated residual CTT.
261
  - `scripts/eval_ctt_proxy.py`: proxy support evaluation with PPTC,
262
- NegativeNear, distance, diversity, and collapse metrics.
 
263
  - `scripts/eval_ctt_generated_rollout.py`: measured rollout harness that
264
  restores states, decodes generated tangents, executes candidates, and writes
265
  measured candidate rows.
266
  - `scripts/eval_ctt_rollout.py`: measured-output wrapper.
267
- - `scripts/build_ctt_proxy_comparison.py`: proxy comparison/gate table.
 
 
268
  - `scripts/build_ctt_rollout_comparison.py`: measured rollout aggregation.
269
  - `scripts/summarize_ctt_runs.py`: global CSV/Markdown summary. Markdown output
270
  may be generated transiently, but the persistent overview is this README.
@@ -412,6 +415,7 @@ High-value run directories:
412
  - `runs/ctt_gated_residual_full_seed{0,1,2}`: full gated residual CTT training.
413
  - `runs/ctt_base_context_obs_test_envclip_k16_rollout_comparison`: current
414
  strongest measured support artifact.
 
415
  - `runs/ctt_base_context_obs_train_cal_envclip_k16_rollout_comparison`: train
416
  calibration rows for K=16 `env_clip`.
417
  - `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test`: K=16 LCB
@@ -496,6 +500,29 @@ Key held-out test values:
496
 
497
  Interpretation: support is strong; selector is not.
498
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
499
  ### Best current K=16 train-clean learned selector
500
 
501
  Run:
 
259
 
260
  - `scripts/train_ctt.py`: trains residual or gated residual CTT.
261
  - `scripts/eval_ctt_proxy.py`: proxy support evaluation with PPTC,
262
+ NegativeNear, PosCloserThanNeg, distance, diversity, and collapse metrics.
263
+ Use `--no-markdown-report` for README-only runs.
264
  - `scripts/eval_ctt_generated_rollout.py`: measured rollout harness that
265
  restores states, decodes generated tangents, executes candidates, and writes
266
  measured candidate rows.
267
  - `scripts/eval_ctt_rollout.py`: measured-output wrapper.
268
+ - `scripts/build_ctt_proxy_comparison.py`: proxy comparison/gate table with
269
+ by-task/by-seed JSON outputs. Use `--no-markdown-report` for README-only
270
+ runs.
271
  - `scripts/build_ctt_rollout_comparison.py`: measured rollout aggregation.
272
  - `scripts/summarize_ctt_runs.py`: global CSV/Markdown summary. Markdown output
273
  may be generated transiently, but the persistent overview is this README.
 
415
  - `runs/ctt_gated_residual_full_seed{0,1,2}`: full gated residual CTT training.
416
  - `runs/ctt_base_context_obs_test_envclip_k16_rollout_comparison`: current
417
  strongest measured support artifact.
418
+ - `runs/ctt_val_proxy_comparison`: CTT-vs-local-atlas proxy support gate.
419
  - `runs/ctt_base_context_obs_train_cal_envclip_k16_rollout_comparison`: train
420
  calibration rows for K=16 `env_clip`.
421
  - `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test`: K=16 LCB
 
500
 
501
  Interpretation: support is strong; selector is not.
502
 
503
+ ### CTT validation proxy support gate
504
+
505
+ Run:
506
+
507
+ ```text
508
+ runs/ctt_val_proxy_comparison
509
+ ```
510
+
511
+ This is a proxy geometry gate, not rollout success. CTT variants pass by
512
+ improving mean distance to target positives while staying within the
513
+ NegativeNear@0.20 safety slack; they do not beat local-atlas on PPTC thresholds.
514
+ The gated residual variant fails the safety gate.
515
+
516
+ | Method | PPTC@0.20 | PPTC@0.40 | Neg@0.20 | Pos<Neg | MeanPos | Collapse | Gate |
517
+ | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
518
+ | local-atlas | 0.4058 | 0.6812 | 0.0368 | 0.5998 | 0.7203 | 0.0661 | baseline |
519
+ | CTT residual full | 0.1981 | 0.6087 | 0.0296 | 0.7352 | 0.4509 | 0.0681 | pass |
520
+ | CTT residual base+context+obs | 0.2464 | 0.6425 | 0.0343 | 0.7717 | 0.4347 | 0.0681 | pass |
521
+ | CTT gated residual full | 0.2319 | 0.6135 | 0.0527 | 0.7248 | 0.4337 | 0.0681 | fail |
522
+
523
+ Interpretation: CTT improves support distance geometry, but the story is still
524
+ diagnostic until measured rollout and selection close the outcome gap.
525
+
526
  ### Best current K=16 train-clean learned selector
527
 
528
  Run: