Add RESEARCH_LOG.md (MoE: WaveRep-DINOv3 replaces E3; Setting G results)
Browse files- RESEARCH_LOG.md +106 -0
RESEARCH_LOG.md
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MoE detector — research log (Setting G)
|
| 2 |
+
|
| 3 |
+
*Last updated: 2026-10-12. Private working log; numbers are from local runs, single test set (Setting G), not yet peer reviewed.*
|
| 4 |
+
|
| 5 |
+
Working title: **Route by Reliability — degradation-aware mixture of forensic experts for AI-generated video detection.**
|
| 6 |
+
Test set: **Setting G** (repost-chain re-encoding; 9,468 runnable videos, 3,010 real / 6,458 fake, 7 generators,
|
| 7 |
+
re-encode depth 0/1/5/10/20 hops through chains of 6 simulated platforms). See the benchmark log for Setting G details.
|
| 8 |
+
|
| 9 |
+
## Protocol (all MoE results)
|
| 10 |
+
- **No G labels in any fitted part.** Platt scaling per expert on `calib_fit`; router heads on `router_train`;
|
| 11 |
+
temperature τ on `router_val`; decision threshold = Youden J on `calib_select`; test on G.
|
| 12 |
+
- Router / calibration videos: 9,577 videos (5,246 real from ActivityNet/InternVid; fakes from CogVideoX 3,064,
|
| 13 |
+
HunyuanVideo 767, Kling 250, OpenSora 199, Sora 51; GenVidBench 500 of them) re-encoded 0/1/5/10/20 hops through
|
| 14 |
+
**platforms disjoint from G** (`whatsapp_hd`, `wechat_chat`, `generic`, and their bad-connection variants).
|
| 15 |
+
- Metrics: ACC, Macro F1, AUC, TPR (fake), TNR (real), TPR@1%FPR. AUC and Macro F1 are always reported together.
|
| 16 |
+
- Bootstrap: 1,000 paired resamples grouped by source video name, 95% percentile CI.
|
| 17 |
+
|
| 18 |
+
## Experts
|
| 19 |
+
| ID | Expert | Training | G AUC / Macro F1 (alone) |
|
| 20 |
+
|---|---|---|---|
|
| 21 |
+
| E1 | **WaveRep** (released, DINOv2 ViT-B/14 reg4, `weights_dinov2_G4.ckpt`), mean frame logit | authors' data, frozen | 0.749 / 0.563 (native 0.5) |
|
| 22 |
+
| E2 | **D3** (XCLIP-16, std of 2nd-order frame-feature distance), real if dis_std > 5 | training-free | 0.695 / 0.646 |
|
| 23 |
+
| E3 (old) | DINOv3 ViT-L/16 **temporal-difference** frozen features + linear probe | expert_train (CogVideoX fakes only) | 0.606 / 0.560 |
|
| 24 |
+
| **E3′ (new)** | **WaveRep-DINOv3**: WaveRep recipe re-implemented on DINOv3 ViT-L/16, full fine-tune, 5 s clips, re-encode aug. | gen4 (4 modern generators, see below) | **0.871 / 0.756** |
|
| 25 |
+
|
| 26 |
+
## Main result — Setting G (n = 9,468)
|
| 27 |
+
|
| 28 |
+
| Method | ACC | Macro F1 | AUC | TPR fake | TNR real | TPR@1%FPR |
|
| 29 |
+
|---|---|---|---|---|---|---|
|
| 30 |
+
| Old router E1–E3 (τ=2) | 0.746 | 0.718 | 0.826 | 0.774 | 0.684 | 0.349 |
|
| 31 |
+
| Old mean of z (E1, E2, E3) | 0.740 | 0.713 | 0.819 | 0.768 | 0.679 | 0.340 |
|
| 32 |
+
| E3′ WaveRep-DINOv3 alone | 0.794 | 0.751 | 0.871 | 0.887 | 0.594 | 0.432 |
|
| 33 |
+
| **Router E1, E2, E3′ (τ=2)** | 0.805 | 0.785 | 0.887 | 0.816 | 0.783 | 0.432 |
|
| 34 |
+
| **Mean of z (E1, E2, E3′)** | 0.799 | 0.784 | 0.894 | 0.778 | 0.846 | 0.513 |
|
| 35 |
+
| **Logistic stacker on z (router_train)** | 0.813 | 0.795 | 0.896 | 0.818 | 0.804 | 0.494 |
|
| 36 |
+
| [reference, uses G labels] in-G logistic on z | 0.820 | 0.792 | 0.896 | 0.866 | 0.720 | 0.506 |
|
| 37 |
+
|
| 38 |
+
Per hop AUC / Macro F1 (new experts): mean of z 0.952/0.872, 0.904/0.795, 0.815/0.646, 0.747/0.644, 0.753/0.640 for
|
| 39 |
+
0/1/5/10/20 hops; old router 0.927/0.815, 0.820/0.726, 0.682/0.582, 0.665/0.580, 0.632/0.549.
|
| 40 |
+
|
| 41 |
+
Replacing **E1** (instead of E3) with E3′ is worse: router 0.884 / 0.761, mean of z 0.886 / 0.755. Keep WaveRep.
|
| 42 |
+
|
| 43 |
+
### Findings
|
| 44 |
+
1. Replacing the old E3 by WaveRep-DINOv3 gives +0.06–0.07 AUC and +0.07–0.08 Macro F1 over the old MoE at every hop depth.
|
| 45 |
+
2. **Routing still adds nothing over simple fusion**: router 0.887 vs mean of z 0.894; gates stay near uniform (0.28–0.40).
|
| 46 |
+
Oracle-hop router on the old experts: 0.811 vs mean 0.819. The logistic stacker matches the in-G label-fitted reference.
|
| 47 |
+
3. Expert contributions (paired bootstrap on G, threshold 0, E3′ seed 0 / seed 1):
|
| 48 |
+
- add WaveRep to (D3, E3′): ΔAUC +0.002 [+0.001,+0.004] / +0.006 [+0.004,+0.008]; ΔTPR@1%FPR **+0.057 [+0.026,+0.079] / +0.089 [+0.047,+0.113]**
|
| 49 |
+
- add E3′ to (WaveRep, D3): **ΔAUC +0.073 [+0.065,+0.080] / +0.071 [+0.063,+0.078]**; ΔMacro F1 +0.050 / +0.059
|
| 50 |
+
- add D3 to (WaveRep, E3′): ΔAUC +0.019 [+0.017,+0.021] / +0.018 [+0.016,+0.021]
|
| 51 |
+
4. **WaveRep and E3′ are complementary.** GenVideo (clean, older generators, n=977): WaveRep 0.998 / 0.955 vs E3′ 0.948 / 0.818
|
| 52 |
+
(WaveRep wins on all 10 generators). G (modern generators, repost chains): E3′ wins on every hop, platform and 6/7 generators;
|
| 53 |
+
WaveRep wins only on Relight-Video (0.664 vs 0.578), the one G generator absent from E3′ training.
|
| 54 |
+
|
| 55 |
+
## WaveRep re-implementation (wr3) — E3′ development
|
| 56 |
+
Official WaveRep training code is not released; we re-implemented the paper recipe (Corvi et al., NeurIPS 2025,
|
| 57 |
+
arXiv 2506.16802): end-to-end ViT + linear head, BCE on random frame crops, balanced batches, WaveRep augmentation
|
| 58 |
+
(p=0.1 on fake crops: 3-level Haar fully-separable wavelet transform; baseband + horizontal + vertical bands from a real
|
| 59 |
+
crop, the 9 diagonal high×high bands kept), JPEG/blur p=0.1, resize of both classes to the same short-side range,
|
| 60 |
+
Adam lr 1e-5 cosine, wd 1e-6, batch 32, 20k iterations, crop 448, model selection on val AUC (held-out source groups).
|
| 61 |
+
Re-encode augmentation (`--reenc`): p=0.3, 1–3 H.264 passes (CRF 23–35, scale 0.6–1.0) of a 16-frame window around the sampled frame.
|
| 62 |
+
|
| 63 |
+
**gen4 training fakes** (image + caption → video; each fake from a different real expert_train source video:
|
| 64 |
+
its first frame + a Qwen2.5-VL-7B caption): Wan2.2-TI2V-5B (704×1056, 40 steps), HunyuanVideo-1.5 480p I2V step-distilled
|
| 65 |
+
(12 steps), LTX-2 19B I2V + distilled LoRA (8 steps), SkyReels-V3 R2V-14B (540P, 8 steps). H.264 CRF 18–28, 24 fps.
|
| 66 |
+
2 s set: 49 frames, 300/gen. 5 s set: 121 frames, 300/gen. Reals: expert_train ActivityNet/InternVid (49 frames at 8 fps).
|
| 67 |
+
|
| 68 |
+
| Run (G, Youden thr on calib_select) | ACC | Macro F1 | AUC | TPR | TNR | TPR@1% | GenVideo AUC |
|
| 69 |
+
|---|---|---|---|---|---|---|---|
|
| 70 |
+
| WaveRep released (DINOv2-B) | 0.565 | 0.563 | 0.749 | 0.364 | 0.995 | 0.398 | 0.998 |
|
| 71 |
+
| DINOv2-B, 2 s | 0.747 | 0.687 | 0.811 | 0.868 | 0.486 | 0.284 | 0.715 |
|
| 72 |
+
| DINOv3-L, 2 s | 0.753 | 0.711 | 0.822 | 0.829 | 0.588 | 0.369 | 0.902 |
|
| 73 |
+
| DINOv3-L, 2 s + reenc | 0.770 | 0.723 | 0.846 | 0.867 | 0.562 | 0.358 | 0.912 |
|
| 74 |
+
| DINOv2-B, 5 s + reenc (seed 0 / 1) | 0.749 / 0.762 | 0.663 / 0.704 | 0.830 / 0.837 | 0.919 / 0.884 | 0.384 / 0.501 | 0.303 / 0.322 | 0.832 / 0.869 |
|
| 75 |
+
| DINOv2-L, 5 s + reenc | 0.753 | 0.674 | 0.832 | 0.912 | 0.412 | 0.316 | 0.795 |
|
| 76 |
+
| **DINOv3-L, 5 s + reenc (seed 0 / 1) = E3′** | 0.796 / 0.793 | 0.756 / 0.748 | **0.871 / 0.865** | 0.881 / 0.892 | 0.613 / 0.582 | 0.432 / 0.402 | 0.948 / 0.944 |
|
| 77 |
+
|
| 78 |
+
- DINOv3 > DINOv2 also at equal size (DINOv2-L 0.832 ≈ DINOv2-B 0.834): the gain comes from pre-training, not size.
|
| 79 |
+
- 2 s vs 5 s, 100 fakes/gen, 2 seeds each: G AUC 0.802 vs 0.825, Macro F1 0.706 vs 0.712 (sources differ between the two sets).
|
| 80 |
+
- More fakes help more than longer clips (2 s: 100/gen 0.802 vs 300/gen 0.846).
|
| 81 |
+
|
| 82 |
+
## Negative / null results (kept for the record)
|
| 83 |
+
- Old E3 adds no ranking to (E1, E2): ΔAUC 0.000; it shifts the threshold (F1 +0.021 [0.013, 0.030], TNR −0.16, TPR@1% −0.053).
|
| 84 |
+
It calls G reals fake at all hops (TNR 0.17–0.31): a real-class shift, not a compression detector.
|
| 85 |
+
- Router E1–E3 seed 0 = 0.816 AUC, but 5-seed mean 0.798 ± 0.015; oracle-hop router 0.811 (routing claim not supported).
|
| 86 |
+
- Label-free accuracy estimation (SML, GNB-EM): no gain (GNB-EM best TPR@1% 0.410). Self-probe by re-encoding: fails (compression saturates after one re-encode).
|
| 87 |
+
- E3 v2 (fixed 125 ms step) 0.594 vs 0.606; fps is not a shortcut in G (AUC 0.50).
|
| 88 |
+
- Training-free trajectory statistics (dense DINOv3, Wan2.1 / CogVideoX VAE latents): best G 0.703; combined new temporal expert 0.655, no fusion gain.
|
| 89 |
+
- Other frozen E3 candidates (linear probe, CogVideoX-only training data): SigLIP 2 0.733 / 0.612, DINOv3 RGB 0.677 / 0.591,
|
| 90 |
+
V-JEPA 2 0.598 / 0.516; none improves the (WaveRep, D3) fusion. In-domain AUC 0.85–0.93 vs G 0.60–0.73 → training data was the main limit.
|
| 91 |
+
- Common failures of WaveRep and D3 on G: 1,158 / 1,165 joint failures are missed fakes (SkyReels 44%, Relight 33%, 5–20 hops 26–27%).
|
| 92 |
+
|
| 93 |
+
## Known issues and caveats
|
| 94 |
+
- **D3 prediction bug in the benchmark CSVs**: `tools/build_setting_{F,G}_per_video.py` swap predictions when they flip the
|
| 95 |
+
score sign, so `pred_D3` is inverted on rows with 5/10/20 hops in F and G. Correct rule: fake iff `score_D3 ≥ −5`.
|
| 96 |
+
Corrected D3 on G (n=9,468): 0.686 / 0.646 / 0.695 / 0.750 / 0.550 / 0.013. All MoE scripts use the correct rule;
|
| 97 |
+
the project CSV fix is still pending approval.
|
| 98 |
+
- **Generator overlap**: gen4 uses Wan2.2, HunyuanVideo, LTX and SkyReels families, which are also in G. G is therefore not an
|
| 99 |
+
unseen-generator test for E3′. Unseen-generator evidence: Relight-Video (E3′ weak, 0.54–0.58) and GenVideo.
|
| 100 |
+
- Not yet checked: overlap between WaveRep's own training generators and GenVideo ("COG", "OpenSora").
|
| 101 |
+
- Single test set; E3′ results from 2 seeds; no CIs yet for the router-vs-mean comparison with E3′.
|
| 102 |
+
|
| 103 |
+
## Next steps
|
| 104 |
+
1. Leave-one-generator-out retrain of E3′ (train on 3 gen4 generators, test on the 4th) — answers the generator-overlap objection.
|
| 105 |
+
2. Bootstrap CIs for router / mean of z / stacker with the new experts; tune the fusion threshold with the same Youden rule.
|
| 106 |
+
3. Check WaveRep training generators vs GenVideo.
|