document CTT proxy gate diagnostics
Browse files
README.md
CHANGED
|
@@ -259,12 +259,15 @@ CTT training/proxy/rollout:
|
|
| 259 |
|
| 260 |
- `scripts/train_ctt.py`: trains residual or gated residual CTT.
|
| 261 |
- `scripts/eval_ctt_proxy.py`: proxy support evaluation with PPTC,
|
| 262 |
-
NegativeNear, distance, diversity, and collapse metrics.
|
|
|
|
| 263 |
- `scripts/eval_ctt_generated_rollout.py`: measured rollout harness that
|
| 264 |
restores states, decodes generated tangents, executes candidates, and writes
|
| 265 |
measured candidate rows.
|
| 266 |
- `scripts/eval_ctt_rollout.py`: measured-output wrapper.
|
| 267 |
-
- `scripts/build_ctt_proxy_comparison.py`: proxy comparison/gate table
|
|
|
|
|
|
|
| 268 |
- `scripts/build_ctt_rollout_comparison.py`: measured rollout aggregation.
|
| 269 |
- `scripts/summarize_ctt_runs.py`: global CSV/Markdown summary. Markdown output
|
| 270 |
may be generated transiently, but the persistent overview is this README.
|
|
@@ -412,6 +415,7 @@ High-value run directories:
|
|
| 412 |
- `runs/ctt_gated_residual_full_seed{0,1,2}`: full gated residual CTT training.
|
| 413 |
- `runs/ctt_base_context_obs_test_envclip_k16_rollout_comparison`: current
|
| 414 |
strongest measured support artifact.
|
|
|
|
| 415 |
- `runs/ctt_base_context_obs_train_cal_envclip_k16_rollout_comparison`: train
|
| 416 |
calibration rows for K=16 `env_clip`.
|
| 417 |
- `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test`: K=16 LCB
|
|
@@ -496,6 +500,29 @@ Key held-out test values:
|
|
| 496 |
|
| 497 |
Interpretation: support is strong; selector is not.
|
| 498 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 499 |
### Best current K=16 train-clean learned selector
|
| 500 |
|
| 501 |
Run:
|
|
|
|
| 259 |
|
| 260 |
- `scripts/train_ctt.py`: trains residual or gated residual CTT.
|
| 261 |
- `scripts/eval_ctt_proxy.py`: proxy support evaluation with PPTC,
|
| 262 |
+
NegativeNear, PosCloserThanNeg, distance, diversity, and collapse metrics.
|
| 263 |
+
Use `--no-markdown-report` for README-only runs.
|
| 264 |
- `scripts/eval_ctt_generated_rollout.py`: measured rollout harness that
|
| 265 |
restores states, decodes generated tangents, executes candidates, and writes
|
| 266 |
measured candidate rows.
|
| 267 |
- `scripts/eval_ctt_rollout.py`: measured-output wrapper.
|
| 268 |
+
- `scripts/build_ctt_proxy_comparison.py`: proxy comparison/gate table with
|
| 269 |
+
by-task/by-seed JSON outputs. Use `--no-markdown-report` for README-only
|
| 270 |
+
runs.
|
| 271 |
- `scripts/build_ctt_rollout_comparison.py`: measured rollout aggregation.
|
| 272 |
- `scripts/summarize_ctt_runs.py`: global CSV/Markdown summary. Markdown output
|
| 273 |
may be generated transiently, but the persistent overview is this README.
|
|
|
|
| 415 |
- `runs/ctt_gated_residual_full_seed{0,1,2}`: full gated residual CTT training.
|
| 416 |
- `runs/ctt_base_context_obs_test_envclip_k16_rollout_comparison`: current
|
| 417 |
strongest measured support artifact.
|
| 418 |
+
- `runs/ctt_val_proxy_comparison`: CTT-vs-local-atlas proxy support gate.
|
| 419 |
- `runs/ctt_base_context_obs_train_cal_envclip_k16_rollout_comparison`: train
|
| 420 |
calibration rows for K=16 `env_clip`.
|
| 421 |
- `runs/ctt_base_context_obs_dominance_envclip_k16_train_to_test`: K=16 LCB
|
|
|
|
| 500 |
|
| 501 |
Interpretation: support is strong; selector is not.
|
| 502 |
|
| 503 |
+
### CTT validation proxy support gate
|
| 504 |
+
|
| 505 |
+
Run:
|
| 506 |
+
|
| 507 |
+
```text
|
| 508 |
+
runs/ctt_val_proxy_comparison
|
| 509 |
+
```
|
| 510 |
+
|
| 511 |
+
This is a proxy geometry gate, not rollout success. CTT variants pass by
|
| 512 |
+
improving mean distance to target positives while staying within the
|
| 513 |
+
NegativeNear@0.20 safety slack; they do not beat local-atlas on PPTC thresholds.
|
| 514 |
+
The gated residual variant fails the safety gate.
|
| 515 |
+
|
| 516 |
+
| Method | PPTC@0.20 | PPTC@0.40 | Neg@0.20 | Pos<Neg | MeanPos | Collapse | Gate |
|
| 517 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
|
| 518 |
+
| local-atlas | 0.4058 | 0.6812 | 0.0368 | 0.5998 | 0.7203 | 0.0661 | baseline |
|
| 519 |
+
| CTT residual full | 0.1981 | 0.6087 | 0.0296 | 0.7352 | 0.4509 | 0.0681 | pass |
|
| 520 |
+
| CTT residual base+context+obs | 0.2464 | 0.6425 | 0.0343 | 0.7717 | 0.4347 | 0.0681 | pass |
|
| 521 |
+
| CTT gated residual full | 0.2319 | 0.6135 | 0.0527 | 0.7248 | 0.4337 | 0.0681 | fail |
|
| 522 |
+
|
| 523 |
+
Interpretation: CTT improves support distance geometry, but the story is still
|
| 524 |
+
diagnostic until measured rollout and selection close the outcome gap.
|
| 525 |
+
|
| 526 |
### Best current K=16 train-clean learned selector
|
| 527 |
|
| 528 |
Run:
|