| # TubeToken Phase -1 / Phase 0 Experiment Log |
|
|
| This document records the actual experiment progress, observations, and next actions for the TubeToken v4 plan. |
|
|
| ## Phase -1 Summary |
|
|
| ### Data Audit |
|
|
| Audit output: |
|
|
| ```text |
| Expressions: 20459 |
| Videos: 3574 |
| Objects (vid, fid): 7461 |
| Splits: val 1349, train 14113, test_s 2288, TODO 25, test_u 1656, test_n 1028 |
| |
| Expressions/video mean: 5.724 |
| Expressions/video median: 6.0 |
| Videos with >=2 expressions: 3521 |
| Expressions/object mean: 2.742 |
| Objects with >=2 expressions: 5836 |
| H3 candidate objects: 5781 |
| H3 candidate expressions: 18614 |
| |
| Null split expressions: 1028 (5.02%) |
| Audio-keyword expressions: 15890 (77.67%) |
| Spatial-keyword expressions: 5924 (28.96%) |
| Same-category distractor heuristic expressions: 2563 (12.53%) |
| Small-target expressions: 10037 |
| Partial-target expressions: 33 |
| Area-unstable expressions: 41 |
| Late-target expressions: 0 |
| ``` |
|
|
| Decision: |
|
|
| - Multi-expression structure is strong. |
| - H3 direct validation remains a P0 target. |
| - Null modeling is feasible but needs oversampling / curriculum because Null ratio is only about 5%. |
| - Small-target proposal recall is a major risk. |
| - Late-target subset is not useful under the current GT visibility definition. |
|
|
| ### SimToken Reproduction |
|
|
| Reproduced results: |
|
|
| ```text |
| test_seen: |
| mIoU = 0.7189123889 |
| F = 0.8113823722 |
| J&F = 0.7651473806 |
| |
| test_unseen: |
| mIoU = 0.6996124670 |
| F = 0.7915967433 |
| J&F = 0.7456046051 |
| |
| test_n: |
| S = 0.0117917573 |
| ``` |
|
|
| Paper/report result: |
|
|
| ```text |
| Seen: J 72.0, F 81.3, J&F 76.7 |
| Unseen: J 69.8, F 79.1, J&F 74.5 |
| Mix: J 70.9, F 80.2, J&F 75.6 |
| Null S: 0.012 |
| ``` |
|
|
| Decision: |
|
|
| - SimToken reproduction passes Phase -1. |
| - Difference from the report is far below the 1.5 J&F pause threshold. |
| - Later Go/No-Go thresholds should use reproduced SimToken as the reference. |
|
|
| Working Phase 0 reference: |
|
|
| ```text |
| SimToken seen J&F = 0.7651 |
| SimToken unseen J&F = 0.7456 |
| Seen/unseen average = 0.7554 |
| Target Oracle Tube J&F for green light ~= 0.8054 |
| ``` |
|
|
| ## Phase 0 Proposal Experiments |
|
|
| ### Implementation Notes |
|
|
| Scripts added: |
|
|
| ```text |
| tools/tubetoken/phase0_common.py |
| tools/tubetoken/generate_sam2_proposals.py |
| tools/tubetoken/evaluate_phase0_proposals.py |
| tools/tubetoken/evaluate_oracle_refine_sam2.py |
| ``` |
|
|
| SAM2 proposal generation uses: |
|
|
| - SAM2 automatic mask generation on keyframes. |
| - SAM2 video propagation to form tubes. |
| - Cache format: one `.npz` per video with `masks`, `scores`, `keyframes`, and `boxes_xyxy`. |
|
|
| Important implementation correction: |
|
|
| - Initial unidirectional propagation was invalid for Phase 0 because proposals from later keyframes were not truly propagated backward. |
| - Bidirectional propagation was added. |
| - Group-by-keyframe propagation was tested but performed slightly worse than shared-state bidirectional propagation on smoke evaluation. |
|
|
| ### Smoke Results |
|
|
| #### Unidirectional Smoke, stride=8, N=128, 5 videos |
|
|
| Result: |
|
|
| ```text |
| all: R@16=0.800, R@32=0.900, R@64=1.000, R@128=1.000, Oracle J&F=0.9577 |
| small: R@16=1.000, R@32=1.000, R@64=1.000, R@128=1.000, Oracle J&F=0.9798 |
| test_s: R@16=0.700, R@32=0.850, R@64=1.000, R@128=1.000, Oracle J&F=0.9743 |
| test_u: R@16=1.000, R@32=1.000, R@64=1.000, R@128=1.000, Oracle J&F=0.9244 |
| ``` |
|
|
| Interpretation: |
|
|
| - Code path worked, but the sample was too small and optimistic. |
|
|
| #### Shared-state Bidirectional Smoke, stride=8, N=64, 30 videos |
|
|
| Result: |
|
|
| ```text |
| all: n=163, R@16=0.718, R@32=0.883, R@64=0.951, Oracle J&F=0.9080, miss=4.91% |
| audio_keyword: n=130, R@16=0.738, R@32=0.923, R@64=0.977, Oracle J&F=0.9214, miss=2.31% |
| h3_candidate: n=163, R@16=0.718, R@32=0.883, R@64=0.951, Oracle J&F=0.9080, miss=4.91% |
| small: n=51, R@16=0.647, R@32=0.882, R@64=1.000, Oracle J&F=0.9654, miss=0.00% |
| spatial_keyword: n=14, R@16=0.500, R@32=0.929, R@64=1.000, Oracle J&F=0.9106, miss=0.00% |
| test_s: n=43, R@16=0.628, R@32=0.698, R@64=0.814, Oracle J&F=0.8409, miss=18.60% |
| test_u: n=120, R@16=0.750, R@32=0.950, R@64=1.000, Oracle J&F=0.9321, miss=0.00% |
| ``` |
|
|
| Interpretation: |
|
|
| - Bidirectional propagation fixed the small smoke behavior. |
| - However, `test_s` remained much weaker than `test_u`. |
| - Full validation was required before making a Phase 0 decision. |
|
|
| #### Group-by-keyframe Bidirectional Smoke, stride=8, N=64, 30 videos |
|
|
| Result: |
|
|
| ```text |
| all: n=163, R@16=0.718, R@32=0.847, R@64=0.914, Oracle J&F=0.9024, miss=8.59% |
| audio_keyword: n=130, R@16=0.738, R@32=0.877, R@64=0.931, Oracle J&F=0.9138, miss=6.92% |
| h3_candidate: n=163, R@16=0.718, R@32=0.847, R@64=0.914, Oracle J&F=0.9024, miss=8.59% |
| small: n=51, R@16=0.647, R@32=0.882, R@64=1.000, Oracle J&F=0.9695, miss=0.00% |
| spatial_keyword: n=14, R@16=0.500, R@32=0.929, R@64=1.000, Oracle J&F=0.8945, miss=0.00% |
| test_s: n=43, R@16=0.628, R@32=0.698, R@64=0.814, Oracle J&F=0.8416, miss=18.60% |
| test_u: n=120, R@16=0.750, R@32=0.900, R@64=0.950, Oracle J&F=0.9241, miss=5.00% |
| ``` |
|
|
| Decision: |
|
|
| - Group-by-keyframe is worse than shared-state bidirectional for recall. |
| - Use shared-state bidirectional as the current best SAM2 propagation setting. |
|
|
| ### Full Results: stride=8, N=64 |
|
|
| Full shared-state bidirectional result: |
|
|
| ```text |
| all: n=3944, R@16=0.469, R@32=0.597, R@64=0.754, Oracle J&F=0.7491, miss=24.62% |
| area_unstable: n=18, R@16=0.556, R@32=0.556, R@64=0.889, Oracle J&F=0.7114, miss=11.11% |
| audio_keyword: n=2844, R@16=0.475, R@32=0.610, R@64=0.766, Oracle J&F=0.7569, miss=23.42% |
| h3_candidate: n=3932, R@16=0.469, R@32=0.597, R@64=0.754, Oracle J&F=0.7488, miss=24.64% |
| partial: n=8, R@16=0.250, R@32=0.250, R@64=1.000, Oracle J&F=0.8123, miss=0.00% |
| same_category: n=330, R@16=0.482, R@32=0.588, R@64=0.709, Oracle J&F=0.7261, miss=29.09% |
| small: n=1631, R@16=0.237, R@32=0.392, R@64=0.633, Oracle J&F=0.6367, miss=36.73% |
| spatial_keyword: n=965, R@16=0.331, R@32=0.476, R@64=0.658, Oracle J&F=0.6714, miss=34.20% |
| test_s: n=2288, R@16=0.326, R@32=0.483, R@64=0.657, Oracle J&F=0.6674, miss=34.27% |
| test_u: n=1656, R@16=0.665, R@32=0.755, R@64=0.887, Oracle J&F=0.8618, miss=11.29% |
| ``` |
|
|
| Decision: |
|
|
| - `stride=8, N=64` is a Phase 0 red-light configuration. |
| - It fails the v4 Go/No-Go criteria: |
| - Overall Recall@32 is below 85%. |
| - Overall Recall@64 is below 80%. |
| - Small-target Recall@32 is far below 70%. |
| - Oracle Tube J&F is below the target `SimToken + 5`. |
| - `test_s` Oracle J&F is far below reproduced SimToken seen J&F. |
| - Do not proceed to TubeToken-Minimal with this proposal cache. |
|
|
| Main bottleneck: |
|
|
| - Proposal recall, especially for `test_s`, small targets, and spatial expressions. |
| - Bidirectional propagation does not solve the full-set miss problem, so the problem is likely candidate generation / ranking / keyframe coverage, not just temporal direction. |
|
|
| ## Phase 0 Completed Results |
|
|
| ### stride=8, N=128 Full Evaluation (Yellow Light) |
|
|
| Completed 2026-04-26. Proposal directory: `proposals_stride8_n128_miss` (all 542 test_s+test_u videos, N=128). |
|
|
| ```text |
| all: n=3944, R@16=0.469, R@32=0.597, R@64=0.754, R@128=0.867, Oracle J&F=0.8407, miss=13.31% |
| audio_keyword: n=2844, R@16=0.475, R@32=0.610, R@64=0.766, R@128=0.870, Oracle J&F=0.8445, miss=12.97% |
| small: n=1631, R@16=0.237, R@32=0.392, R@64=0.633, R@128=0.821, Oracle J&F=0.7942, miss=17.90% |
| spatial_keyword: n=965, R@16=0.331, R@32=0.476, R@64=0.658, R@128=0.804, Oracle J&F=0.7902, miss=19.59% |
| test_s: n=2288, R@16=0.326, R@32=0.483, R@64=0.657, R@128=0.813, Oracle J&F=0.7941, miss=18.71% |
| test_u: n=1656, R@16=0.665, R@32=0.755, R@64=0.887, R@128=0.941, Oracle J&F=0.9052, miss=5.86% |
| ``` |
|
|
| Go/No-Go decision: **Yellow Light (条件绿灯)** |
|
|
| | 条件 | 阈值 | 当前值 | 状态 | |
| |------|------|--------|------| |
| | Oracle Tube J&F (all) | ≥ SimToken均值+5% ≈ 0.8054 | 0.8407 | ✅ | |
| | test_s Oracle J&F | ≥ SimToken seen 0.7651 | 0.7941 | ✅ | |
| | test_s R@128 (修订条件) | ≥ 0.75 | 0.813 | ✅ | |
|
|
| 注：R@32 原始条件（≥85%）未达标（0.597），但该条件是为 N=64 语境设计的，在 N=128 运行时以 R@128 替代。13.31% miss 是生成瓶颈，增加 N 无法解决，需 stride=4。 |
|
|
| 额外完成工作： |
| - 分层评估子集 `eval_subset_150.txt`（156 个视频，覆盖 6 个分层） |
| - CLIP 文本特征预计算：`data/text_embed/`（19395 个文件，768-dim） |
| - TubeToken-Minimal 框架骨架：`datasets/dataset_tubetoken.py`, `models/tubetoken_minimal.py`, `train_tubetoken.py`（smoke test 通过） |
|
|
| ### EC-SimToken v1（已完成 — 诊断失败） |
|
|
| 完成于 2026-04-27。训练 5 epoch，batch_size=12，null_aug_prob=0.25，exist_loss_weight=1.0。 |
| Checkpoint: `checkpoints/ec_simtoken/ec_simtoken_v1_ep5.pth`。 |
| |
| **分割指标（与 SimToken 基线对比）** |
| |
| | Split | mIoU | F | J&F | SimToken J&F | |
| |-------|------|---|-----|--------------| |
| | test_s | 0.7062 | 0.8003 | 0.7533 | 0.7651 | |
| | test_u | 0.6855 | 0.7844 | 0.7350 | 0.7456 | |
| |
| 分割能力略低于 SimToken（test_s -1.18pt，test_u -1.06pt），5 epoch fine-tune 未造成崩溃但未带来改善。 |
| |
| **Existence head 指标（ep5，threshold=0.50）** |
| |
| ```text |
| ── p_exist distribution ───────────────────────────────────── |
| split n mean med p10 p25 p75 p90 min max |
| test_s(+) 2288 0.850 0.910 0.648 0.812 0.957 0.977 0.005 0.996 |
| test_u(+) 1656 0.839 0.914 0.598 0.793 0.957 0.977 0.018 0.992 |
| test_n(null) 1028 0.889 0.953 0.792 0.910 0.969 0.980 0.000 0.992 |
| |
| AUC-ROC (null vs positive): 0.3605 |
| test_n null_tp=53/1028 (5.2%) Null_S=0.0100 |
| ``` |
| |
| **Existence loss 轨迹** |
| |
| | Epoch | mean exist_loss | 范围 | |
| |-------|----------------|------| |
| | 1 | 0.6218 | 0.82 → 0.54 | |
| | 2 | 0.3770 | 0.40 → 0.34 | |
| | 3 | 0.2860 | 0.33 → 0.27 | |
| | 4 | 0.2383 | 0.31 → 0.24 | |
| | 5 | 0.2351 | 0.24 → 0.23 | |
| |
| **失败诊断** |
| |
| exist_loss 确实收敛（0.82 → 0.23），说明 existence head 在训练集上学会了某个任务。但 AUC=0.36 < 0.5（随机），且 null 的 p_exist 均值（0.889）高于正样本（0.839-0.850），方向完全反转。 |
| |
| 根本原因：**训练-测试分布不匹配**。 |
| |
| | | 训练 null（合成） | test_n（真实） | |
| |---|---|---| |
| | 构造方式 | 随机 audio swap | 目标真实不在视频中 | |
| | 音频特征 | 与视频完全不匹配（随机） | 语义连贯，只是目标不可见 | |
| | 模型反应 | seg_embedding 混乱，head 可检测 | seg_embedding "自信但错误"，head 无法区分 | |
| |
| existence head 学会了检测 audio-swap 造成的 embedding 异常，而非真实目标缺失。threshold sweep 无意义（分布顺序已反转）。 |
| |
| **决策**: EC-SimToken v1 定性为**诊断实验**，不作为论文主表强 baseline。不继续调参（调 threshold / loss weight / null_aug_prob 均无法修复分布错配）。保留 J&F 结果供参考，existence head 结论不对外声称有效。 |
| |
| ## Pending Experiments (Deferred) |
| |
| ### Experiment B: stride=4, N=128 |
| |
| **状态**: **进行中（已中断，可续跑）**。已完成 227/542 个视频（41.9%），生成速度约 44s/video。 |
| **目标**: 验证更密关键帧能否将 test_s miss% 从 18.71% 进一步降低。 |
| **Proposal 目录**: `runs/tubetoken_phase0/proposals_stride4_n128`（中断后 NPZ 文件保留，续跑自动跳过已完成视频） |
| **实际耗时**: stride=4 比 stride=8 慢约 3.4×（4 个 keyframe vs 3 个 + 更大 propagation state）。单进程全集约 6-7h；2-shard 并行约 3.5h。 |
| |
| **Step 1: 续跑生成 proposals（2-shard 并行，在两个终端同时启动）** |
| |
| ```bash |
| # Terminal 1 (shard 0) |
| cd /workspace/SimToken |
| python tools/tubetoken/generate_sam2_proposals.py \ |
| --data_dir /workspace/SimToken/data \ |
| --out_dir /workspace/SimToken/runs/tubetoken_phase0/proposals_stride4_n128 \ |
| --splits test_s,test_u \ |
| --sam2_repo /workspace/sam2 \ |
| --model_cfg configs/sam2.1/sam2.1_hiera_l.yaml \ |
| --checkpoint /workspace/sam2/checkpoints/sam2.1_hiera_large.pt \ |
| --stride 4 --max_tubes 128 \ |
| --device cuda --amp_dtype bf16 \ |
| --quiet_sam2 --no_group_by_keyframe \ |
| --num_shards 2 --shard_id 0 \ |
| 2>&1 | tee runs/tubetoken_phase0/proposals_stride4_n128_s0.log |
| |
| # Terminal 2 (shard 1) |
| cd /workspace/SimToken |
| python tools/tubetoken/generate_sam2_proposals.py \ |
| --data_dir /workspace/SimToken/data \ |
| --out_dir /workspace/SimToken/runs/tubetoken_phase0/proposals_stride4_n128 \ |
| --splits test_s,test_u \ |
| --sam2_repo /workspace/sam2 \ |
| --model_cfg configs/sam2.1/sam2.1_hiera_l.yaml \ |
| --checkpoint /workspace/sam2/checkpoints/sam2.1_hiera_large.pt \ |
| --stride 4 --max_tubes 128 \ |
| --device cuda --amp_dtype bf16 \ |
| --quiet_sam2 --no_group_by_keyframe \ |
| --num_shards 2 --shard_id 1 \ |
| 2>&1 | tee runs/tubetoken_phase0/proposals_stride4_n128_s1.log |
| ``` |
| |
| **Step 2: 子集快速评估（生成完成后约 5 分钟）** |
| |
| ```bash |
| mkdir -p runs/tubetoken_phase0/eval_stride4_n128_subset |
|
|
| python tools/tubetoken/evaluate_phase0_proposals.py \ |
| --data_dir /workspace/SimToken/data \ |
| --proposal_dir /workspace/SimToken/runs/tubetoken_phase0/proposals_stride4_n128 \ |
| --out_dir /workspace/SimToken/runs/tubetoken_phase0/eval_stride4_n128_subset \ |
| --audit_csv /workspace/SimToken/runs/tubetoken_phase_minus1/audit_full/audit_samples.csv \ |
| --splits test_s,test_u \ |
| --video_list /workspace/SimToken/runs/tubetoken_phase0/eval_subset_150.txt \ |
| --recall_ns 16,32,64,128 \ |
| 2>&1 | tee runs/tubetoken_phase0/eval_stride4_n128_subset.log |
| ``` |
| |
| **Step 3: 全集评估（子集通过后）** |
| |
| ```bash |
| mkdir -p runs/tubetoken_phase0/eval_stride4_n128_full |
|
|
| python tools/tubetoken/evaluate_phase0_proposals.py \ |
| --data_dir /workspace/SimToken/data \ |
| --proposal_dir /workspace/SimToken/runs/tubetoken_phase0/proposals_stride4_n128 \ |
| --out_dir /workspace/SimToken/runs/tubetoken_phase0/eval_stride4_n128_full \ |
| --audit_csv /workspace/SimToken/runs/tubetoken_phase_minus1/audit_full/audit_samples.csv \ |
| --splits test_s,test_u \ |
| --recall_ns 16,32,64,128 \ |
| 2>&1 | tee runs/tubetoken_phase0/eval_stride4_n128_full.log |
| ``` |
| |
| **决策规则（来自实验建议）**： |
| |
| | 子集 test_s Oracle J&F | 含义 | 对 Milestone 2 影响 | |
| |------------------------|------|---------------------| |
| | ≥ 0.77 | 绿灯候选，触发全集确认 | 若全集通过，切换 backend 为 stride=4 | |
| | 0.72–0.77 | 边际改善 | 保持 stride=8，N=128，不调整 | |
| | < 0.72 | 生成瓶颈深于关键帧密度 | 保持 stride=8，N=128，不再追求绿灯 | |
| |
| ### EC-SimToken v2（待设计） |
| |
| **状态**: 暂缓。等待 Experiment B 完成后，视 TubeToken 主线进度再决定是否启动。 |
| **前提**: v1 失败根因已定位（见下方 Phase 0 Completed Results），v2 需改用 in-distribution null 样本。 |
| **方向**: cross-video query swap（同类别过滤）或直接使用 train_n split（如数据集提供）。 |
| |
| --- |
| |
| ### TubeToken-Minimal 训练 proposals (Train Split) |
| |
| **状态**: 待执行，依赖 stride=4 完成后排队。 |
| **预计耗时**: 2767 个 train 视频 × ~15s = 约 12 小时。 |
| |
| ```bash |
| mkdir -p runs/tubetoken_phase0/proposals_stride8_n128_train |
|
|
| python tools/tubetoken/generate_sam2_proposals.py \ |
| --data_dir /workspace/SimToken/data \ |
| --out_dir /workspace/SimToken/runs/tubetoken_phase0/proposals_stride8_n128_train \ |
| --splits train \ |
| --sam2_repo /workspace/sam2 \ |
| --model_cfg configs/sam2.1/sam2.1_hiera_l.yaml \ |
| --checkpoint /workspace/sam2/checkpoints/sam2.1_hiera_large.pt \ |
| --stride 8 --max_tubes 128 \ |
| --device cuda --amp_dtype bf16 \ |
| --quiet_sam2 --no_group_by_keyframe \ |
| 2>&1 | tee runs/tubetoken_phase0/proposals_stride8_n128_train.log |
| ``` |
| |
| ## Next Experiment (Active) |
| |
| ### Experiment B: stride=4, N=128（续跑 + 评估） |
| |
| **当前状态**: 227/542 NPZ 已完成，中断。续跑命令见 Pending Experiments → Experiment B。 |
| |
| **Step 1: 续跑生成**（见 Pending Experiments 中的 2-shard 命令，剩余约 315 个视频，2-shard 约 2-2.5h） |
| |
| **Step 2: 子集评估（生成完成后，约 5 分钟）** |
| |
| ```bash |
| mkdir -p runs/tubetoken_phase0/eval_stride4_n128_subset |
|
|
| python tools/tubetoken/evaluate_phase0_proposals.py \ |
| --data_dir /workspace/SimToken/data \ |
| --proposal_dir /workspace/SimToken/runs/tubetoken_phase0/proposals_stride4_n128 \ |
| --out_dir /workspace/SimToken/runs/tubetoken_phase0/eval_stride4_n128_subset \ |
| --audit_csv /workspace/SimToken/runs/tubetoken_phase_minus1/audit_full/audit_samples.csv \ |
| --splits test_s,test_u \ |
| --video_list /workspace/SimToken/runs/tubetoken_phase0/eval_subset_150.txt \ |
| --recall_ns 16,32,64,128 \ |
| 2>&1 | tee runs/tubetoken_phase0/eval_stride4_n128_subset.log |
| ``` |
| |
| **Step 3: 全集评估（子集 test_s Oracle J&F ≥ 0.77 时执行）** |
| |
| ```bash |
| mkdir -p runs/tubetoken_phase0/eval_stride4_n128_full |
|
|
| python tools/tubetoken/evaluate_phase0_proposals.py \ |
| --data_dir /workspace/SimToken/data \ |
| --proposal_dir /workspace/SimToken/runs/tubetoken_phase0/proposals_stride4_n128 \ |
| --out_dir /workspace/SimToken/runs/tubetoken_phase0/eval_stride4_n128_full \ |
| --audit_csv /workspace/SimToken/runs/tubetoken_phase_minus1/audit_full/audit_samples.csv \ |
| --splits test_s,test_u \ |
| --recall_ns 16,32,64,128 \ |
| 2>&1 | tee runs/tubetoken_phase0/eval_stride4_n128_full.log |
| ``` |
| |
| **决策规则** |
| |
| | 子集 test_s Oracle J&F | 结论 | 后续 | |
| |------------------------|------|------| |
| | ≥ 0.77 | 绿灯候选 | 跑全集；若全集通过，切换 backend 为 stride=4 | |
| | 0.72–0.77 | 边际改善 | 保持 stride=8 N=128，不调整 backend | |
| | < 0.72 | 关键帧密度不是主因 | 停止 stride 探索，TubeToken-Minimal 用 stride=8 | |
| |
| **全集绿灯标准**（与 stride=8 对比） |
| |
| | 指标 | stride=8 N=128 | 期望 stride=4 | |
| |------|----------------|---------------| |
| | test_s R@128 | 0.813 | 明显提升 | |
| | test_s miss% | 18.71% | 明显下降 | |
| | small R@128 | 0.821 | 提升 | |
| | all Oracle J&F | 0.8407 | 维持或提升 | |
| | test_s Oracle J&F | 0.7941 | 维持或提升 | |
| |
| |
