MoE_detector / aug_study /phase4 /mix_reference.md
assa8945's picture
Augmentation study: RUNBOOK v2.1 complete (Phase 2b, final models, pre-registered selection, mix_setting_v1 Phase 4, band reliance, DINOv2 references)
f06a441 verified
|
Raw History Blame Contribute Delete
3.51 kB
# Phase 4 reference rows: DINOv2 WaveRep baselines (n = 9939; report only, no selection)
| system | n | AUC | F1 | TPR | TNR | TPR1 | AUC_lo | AUC_hi | F1_lo | F1_hi |
|:------------|-----:|------:|------:|------:|------:|-------:|---------:|---------:|--------:|--------:|
| (E2, E3p) | 9939 | 0.842 | 0.735 | 0.844 | 0.627 | 0.301 | 0.834 | 0.849 | 0.727 | 0.744 |
| (E2, F3_s0) | 9939 | 0.849 | 0.739 | 0.836 | 0.648 | 0.335 | 0.842 | 0.857 | 0.731 | 0.748 |
| E1 alone | 9939 | 0.637 | 0.495 | 0.336 | 0.87 | 0.141 | 0.627 | 0.648 | 0.484 | 0.504 |
| (E2, E1) | 9939 | 0.759 | 0.583 | 0.474 | 0.85 | 0.145 | 0.75 | 0.769 | 0.573 | 0.591 |
| D2wr alone | 9939 | 0.737 | 0.638 | 0.816 | 0.452 | 0.157 | 0.728 | 0.747 | 0.629 | 0.648 |
| (E2, D2wr) | 9939 | 0.795 | 0.696 | 0.878 | 0.492 | 0.159 | 0.787 | 0.803 | 0.686 | 0.705 |
| system | vs | dAUC | dAUC_lo | dAUC_hi | dF1 | dF1_lo | dF1_hi |
|:-----------|:------------|-------:|----------:|----------:|-------:|---------:|---------:|
| (E2, E1) | (E2, E3p) | -0.083 | -0.092 | -0.074 | -0.153 | -0.164 | -0.142 |
| (E2, E1) | (E2, F3_s0) | -0.09 | -0.1 | -0.081 | -0.157 | -0.169 | -0.146 |
| (E2, D2wr) | (E2, E3p) | -0.046 | -0.053 | -0.039 | -0.04 | -0.05 | -0.03 |
| (E2, D2wr) | (E2, F3_s0) | -0.054 | -0.061 | -0.047 | -0.044 | -0.054 | -0.033 |
| | (E2, D2wr) | (E2, E1) | (E2, E3p) | (E2, F3_s0) | D2wr alone | E1 alone |
|:----------------------------------|:--------------|:--------------|:--------------|:--------------|:--------------|:--------------|
| ('family', 'HunyuanVideo-1.5') | 0.895 / 0.654 | 0.839 / 0.734 | 0.963 / 0.746 | 0.971 / 0.76 | 0.82 / 0.595 | 0.683 / 0.652 |
| ('family', 'LTX-2') | 0.802 / 0.664 | 0.764 / 0.66 | 0.832 / 0.72 | 0.831 / 0.724 | 0.767 / 0.627 | 0.659 / 0.612 |
| ('family', 'Relight-Video') | 0.646 / 0.4 | 0.673 / 0.556 | 0.641 / 0.455 | 0.615 / 0.459 | 0.543 / 0.368 | 0.552 / 0.518 |
| ('family', 'SkyReels-V3') | 0.631 / 0.54 | 0.588 / 0.533 | 0.7 / 0.613 | 0.7 / 0.616 | 0.619 / 0.519 | 0.53 / 0.483 |
| ('family', 'Wan-animate/replace') | 0.802 / 0.406 | 0.78 / 0.563 | 0.826 / 0.476 | 0.906 / 0.502 | 0.799 / 0.386 | 0.701 / 0.574 |
| ('family', 'Wan2.2') | 0.833 / 0.644 | 0.814 / 0.706 | 0.874 / 0.716 | 0.9 / 0.732 | 0.721 / 0.569 | 0.646 / 0.615 |
| ('platforms', 'measured only') | 0.87 / 0.731 | 0.863 / 0.762 | 0.914 / 0.813 | 0.932 / 0.838 | 0.873 / 0.704 | 0.83 / 0.716 |
| ('platforms', 'mixed') | 0.76 / 0.683 | 0.715 / 0.479 | 0.79 / 0.69 | 0.788 / 0.684 | 0.664 / 0.598 | 0.537 / 0.356 |
| ('platforms', 'old only') | 0.828 / 0.698 | 0.816 / 0.686 | 0.896 / 0.792 | 0.906 / 0.803 | 0.813 / 0.699 | 0.767 / 0.606 |
| ('re-encodes', np.int64(1)) | 0.849 / 0.714 | 0.838 / 0.725 | 0.904 / 0.803 | 0.919 / 0.821 | 0.841 / 0.703 | 0.792 / 0.663 |
| ('re-encodes', np.int64(5)) | 0.792 / 0.706 | 0.748 / 0.519 | 0.819 / 0.711 | 0.825 / 0.723 | 0.698 / 0.618 | 0.572 / 0.389 |
| ('re-encodes', np.int64(10)) | 0.741 / 0.671 | 0.684 / 0.443 | 0.792 / 0.694 | 0.779 / 0.667 | 0.651 / 0.596 | 0.497 / 0.311 |
| ('re-encodes', np.int64(20)) | 0.696 / 0.644 | 0.676 / 0.429 | 0.736 / 0.588 | 0.711 / 0.582 | 0.59 / 0.544 | 0.516 / 0.344 |