assa8945 commited on
Commit
7fd7ebb
·
verified ·
1 Parent(s): 0e52ae6

Add RESEARCH_LOG.md (MoE: WaveRep-DINOv3 replaces E3; Setting G results)

Browse files
Files changed (1) hide show
  1. RESEARCH_LOG.md +106 -0
RESEARCH_LOG.md ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # MoE detector — research log (Setting G)
2
+
3
+ *Last updated: 2026-10-12. Private working log; numbers are from local runs, single test set (Setting G), not yet peer reviewed.*
4
+
5
+ Working title: **Route by Reliability — degradation-aware mixture of forensic experts for AI-generated video detection.**
6
+ Test set: **Setting G** (repost-chain re-encoding; 9,468 runnable videos, 3,010 real / 6,458 fake, 7 generators,
7
+ re-encode depth 0/1/5/10/20 hops through chains of 6 simulated platforms). See the benchmark log for Setting G details.
8
+
9
+ ## Protocol (all MoE results)
10
+ - **No G labels in any fitted part.** Platt scaling per expert on `calib_fit`; router heads on `router_train`;
11
+ temperature τ on `router_val`; decision threshold = Youden J on `calib_select`; test on G.
12
+ - Router / calibration videos: 9,577 videos (5,246 real from ActivityNet/InternVid; fakes from CogVideoX 3,064,
13
+ HunyuanVideo 767, Kling 250, OpenSora 199, Sora 51; GenVidBench 500 of them) re-encoded 0/1/5/10/20 hops through
14
+ **platforms disjoint from G** (`whatsapp_hd`, `wechat_chat`, `generic`, and their bad-connection variants).
15
+ - Metrics: ACC, Macro F1, AUC, TPR (fake), TNR (real), TPR@1%FPR. AUC and Macro F1 are always reported together.
16
+ - Bootstrap: 1,000 paired resamples grouped by source video name, 95% percentile CI.
17
+
18
+ ## Experts
19
+ | ID | Expert | Training | G AUC / Macro F1 (alone) |
20
+ |---|---|---|---|
21
+ | E1 | **WaveRep** (released, DINOv2 ViT-B/14 reg4, `weights_dinov2_G4.ckpt`), mean frame logit | authors' data, frozen | 0.749 / 0.563 (native 0.5) |
22
+ | E2 | **D3** (XCLIP-16, std of 2nd-order frame-feature distance), real if dis_std > 5 | training-free | 0.695 / 0.646 |
23
+ | E3 (old) | DINOv3 ViT-L/16 **temporal-difference** frozen features + linear probe | expert_train (CogVideoX fakes only) | 0.606 / 0.560 |
24
+ | **E3′ (new)** | **WaveRep-DINOv3**: WaveRep recipe re-implemented on DINOv3 ViT-L/16, full fine-tune, 5 s clips, re-encode aug. | gen4 (4 modern generators, see below) | **0.871 / 0.756** |
25
+
26
+ ## Main result — Setting G (n = 9,468)
27
+
28
+ | Method | ACC | Macro F1 | AUC | TPR fake | TNR real | TPR@1%FPR |
29
+ |---|---|---|---|---|---|---|
30
+ | Old router E1–E3 (τ=2) | 0.746 | 0.718 | 0.826 | 0.774 | 0.684 | 0.349 |
31
+ | Old mean of z (E1, E2, E3) | 0.740 | 0.713 | 0.819 | 0.768 | 0.679 | 0.340 |
32
+ | E3′ WaveRep-DINOv3 alone | 0.794 | 0.751 | 0.871 | 0.887 | 0.594 | 0.432 |
33
+ | **Router E1, E2, E3′ (τ=2)** | 0.805 | 0.785 | 0.887 | 0.816 | 0.783 | 0.432 |
34
+ | **Mean of z (E1, E2, E3′)** | 0.799 | 0.784 | 0.894 | 0.778 | 0.846 | 0.513 |
35
+ | **Logistic stacker on z (router_train)** | 0.813 | 0.795 | 0.896 | 0.818 | 0.804 | 0.494 |
36
+ | [reference, uses G labels] in-G logistic on z | 0.820 | 0.792 | 0.896 | 0.866 | 0.720 | 0.506 |
37
+
38
+ Per hop AUC / Macro F1 (new experts): mean of z 0.952/0.872, 0.904/0.795, 0.815/0.646, 0.747/0.644, 0.753/0.640 for
39
+ 0/1/5/10/20 hops; old router 0.927/0.815, 0.820/0.726, 0.682/0.582, 0.665/0.580, 0.632/0.549.
40
+
41
+ Replacing **E1** (instead of E3) with E3′ is worse: router 0.884 / 0.761, mean of z 0.886 / 0.755. Keep WaveRep.
42
+
43
+ ### Findings
44
+ 1. Replacing the old E3 by WaveRep-DINOv3 gives +0.06–0.07 AUC and +0.07–0.08 Macro F1 over the old MoE at every hop depth.
45
+ 2. **Routing still adds nothing over simple fusion**: router 0.887 vs mean of z 0.894; gates stay near uniform (0.28–0.40).
46
+ Oracle-hop router on the old experts: 0.811 vs mean 0.819. The logistic stacker matches the in-G label-fitted reference.
47
+ 3. Expert contributions (paired bootstrap on G, threshold 0, E3′ seed 0 / seed 1):
48
+ - add WaveRep to (D3, E3′): ΔAUC +0.002 [+0.001,+0.004] / +0.006 [+0.004,+0.008]; ΔTPR@1%FPR **+0.057 [+0.026,+0.079] / +0.089 [+0.047,+0.113]**
49
+ - add E3′ to (WaveRep, D3): **ΔAUC +0.073 [+0.065,+0.080] / +0.071 [+0.063,+0.078]**; ΔMacro F1 +0.050 / +0.059
50
+ - add D3 to (WaveRep, E3′): ΔAUC +0.019 [+0.017,+0.021] / +0.018 [+0.016,+0.021]
51
+ 4. **WaveRep and E3′ are complementary.** GenVideo (clean, older generators, n=977): WaveRep 0.998 / 0.955 vs E3′ 0.948 / 0.818
52
+ (WaveRep wins on all 10 generators). G (modern generators, repost chains): E3′ wins on every hop, platform and 6/7 generators;
53
+ WaveRep wins only on Relight-Video (0.664 vs 0.578), the one G generator absent from E3′ training.
54
+
55
+ ## WaveRep re-implementation (wr3) — E3′ development
56
+ Official WaveRep training code is not released; we re-implemented the paper recipe (Corvi et al., NeurIPS 2025,
57
+ arXiv 2506.16802): end-to-end ViT + linear head, BCE on random frame crops, balanced batches, WaveRep augmentation
58
+ (p=0.1 on fake crops: 3-level Haar fully-separable wavelet transform; baseband + horizontal + vertical bands from a real
59
+ crop, the 9 diagonal high×high bands kept), JPEG/blur p=0.1, resize of both classes to the same short-side range,
60
+ Adam lr 1e-5 cosine, wd 1e-6, batch 32, 20k iterations, crop 448, model selection on val AUC (held-out source groups).
61
+ Re-encode augmentation (`--reenc`): p=0.3, 1–3 H.264 passes (CRF 23–35, scale 0.6–1.0) of a 16-frame window around the sampled frame.
62
+
63
+ **gen4 training fakes** (image + caption → video; each fake from a different real expert_train source video:
64
+ its first frame + a Qwen2.5-VL-7B caption): Wan2.2-TI2V-5B (704×1056, 40 steps), HunyuanVideo-1.5 480p I2V step-distilled
65
+ (12 steps), LTX-2 19B I2V + distilled LoRA (8 steps), SkyReels-V3 R2V-14B (540P, 8 steps). H.264 CRF 18–28, 24 fps.
66
+ 2 s set: 49 frames, 300/gen. 5 s set: 121 frames, 300/gen. Reals: expert_train ActivityNet/InternVid (49 frames at 8 fps).
67
+
68
+ | Run (G, Youden thr on calib_select) | ACC | Macro F1 | AUC | TPR | TNR | TPR@1% | GenVideo AUC |
69
+ |---|---|---|---|---|---|---|---|
70
+ | WaveRep released (DINOv2-B) | 0.565 | 0.563 | 0.749 | 0.364 | 0.995 | 0.398 | 0.998 |
71
+ | DINOv2-B, 2 s | 0.747 | 0.687 | 0.811 | 0.868 | 0.486 | 0.284 | 0.715 |
72
+ | DINOv3-L, 2 s | 0.753 | 0.711 | 0.822 | 0.829 | 0.588 | 0.369 | 0.902 |
73
+ | DINOv3-L, 2 s + reenc | 0.770 | 0.723 | 0.846 | 0.867 | 0.562 | 0.358 | 0.912 |
74
+ | DINOv2-B, 5 s + reenc (seed 0 / 1) | 0.749 / 0.762 | 0.663 / 0.704 | 0.830 / 0.837 | 0.919 / 0.884 | 0.384 / 0.501 | 0.303 / 0.322 | 0.832 / 0.869 |
75
+ | DINOv2-L, 5 s + reenc | 0.753 | 0.674 | 0.832 | 0.912 | 0.412 | 0.316 | 0.795 |
76
+ | **DINOv3-L, 5 s + reenc (seed 0 / 1) = E3′** | 0.796 / 0.793 | 0.756 / 0.748 | **0.871 / 0.865** | 0.881 / 0.892 | 0.613 / 0.582 | 0.432 / 0.402 | 0.948 / 0.944 |
77
+
78
+ - DINOv3 > DINOv2 also at equal size (DINOv2-L 0.832 ≈ DINOv2-B 0.834): the gain comes from pre-training, not size.
79
+ - 2 s vs 5 s, 100 fakes/gen, 2 seeds each: G AUC 0.802 vs 0.825, Macro F1 0.706 vs 0.712 (sources differ between the two sets).
80
+ - More fakes help more than longer clips (2 s: 100/gen 0.802 vs 300/gen 0.846).
81
+
82
+ ## Negative / null results (kept for the record)
83
+ - Old E3 adds no ranking to (E1, E2): ΔAUC 0.000; it shifts the threshold (F1 +0.021 [0.013, 0.030], TNR −0.16, TPR@1% −0.053).
84
+ It calls G reals fake at all hops (TNR 0.17–0.31): a real-class shift, not a compression detector.
85
+ - Router E1–E3 seed 0 = 0.816 AUC, but 5-seed mean 0.798 ± 0.015; oracle-hop router 0.811 (routing claim not supported).
86
+ - Label-free accuracy estimation (SML, GNB-EM): no gain (GNB-EM best TPR@1% 0.410). Self-probe by re-encoding: fails (compression saturates after one re-encode).
87
+ - E3 v2 (fixed 125 ms step) 0.594 vs 0.606; fps is not a shortcut in G (AUC 0.50).
88
+ - Training-free trajectory statistics (dense DINOv3, Wan2.1 / CogVideoX VAE latents): best G 0.703; combined new temporal expert 0.655, no fusion gain.
89
+ - Other frozen E3 candidates (linear probe, CogVideoX-only training data): SigLIP 2 0.733 / 0.612, DINOv3 RGB 0.677 / 0.591,
90
+ V-JEPA 2 0.598 / 0.516; none improves the (WaveRep, D3) fusion. In-domain AUC 0.85–0.93 vs G 0.60–0.73 → training data was the main limit.
91
+ - Common failures of WaveRep and D3 on G: 1,158 / 1,165 joint failures are missed fakes (SkyReels 44%, Relight 33%, 5–20 hops 26–27%).
92
+
93
+ ## Known issues and caveats
94
+ - **D3 prediction bug in the benchmark CSVs**: `tools/build_setting_{F,G}_per_video.py` swap predictions when they flip the
95
+ score sign, so `pred_D3` is inverted on rows with 5/10/20 hops in F and G. Correct rule: fake iff `score_D3 ≥ −5`.
96
+ Corrected D3 on G (n=9,468): 0.686 / 0.646 / 0.695 / 0.750 / 0.550 / 0.013. All MoE scripts use the correct rule;
97
+ the project CSV fix is still pending approval.
98
+ - **Generator overlap**: gen4 uses Wan2.2, HunyuanVideo, LTX and SkyReels families, which are also in G. G is therefore not an
99
+ unseen-generator test for E3′. Unseen-generator evidence: Relight-Video (E3′ weak, 0.54–0.58) and GenVideo.
100
+ - Not yet checked: overlap between WaveRep's own training generators and GenVideo ("COG", "OpenSora").
101
+ - Single test set; E3′ results from 2 seeds; no CIs yet for the router-vs-mean comparison with E3′.
102
+
103
+ ## Next steps
104
+ 1. Leave-one-generator-out retrain of E3′ (train on 3 gen4 generators, test on the 4th) — answers the generator-overlap objection.
105
+ 2. Bootstrap CIs for router / mean of z / stacker with the new experts; tune the fusion threshold with the same Youden rule.
106
+ 3. Check WaveRep training generators vs GenVideo.