ahmedyradwan02 commited on
Commit
2bb0d13
·
verified ·
1 Parent(s): 640229d

ONNX: batch axis, attribution outputs, float16 file, usual ranges, serving facts

Browse files
Files changed (6) hide show
  1. README.md +32 -10
  2. model.fp16.onnx +3 -0
  3. model.onnx +2 -2
  4. serving.json +48 -0
  5. usual_ranges.fp16.json +220 -0
  6. usual_ranges.json +220 -0
README.md CHANGED
@@ -71,7 +71,11 @@ the frame scores.
71
  |---|---|---|
72
  | `model.safetensors` | trained adapters, routers and head; the frozen CLIP ViT-L/14 is not included and is downloaded from OpenAI on first use | 415 MB |
73
  | `config.yaml` | model and data settings, read by the `mars` package | 1 KB |
74
- | `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone | 1.63 GB |
 
 
 
 
75
 
76
  This is the checkpoint behind the DFD-HR results in the MARS paper.
77
 
@@ -139,12 +143,30 @@ pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime f
139
 
140
  | name | shape, type | meaning |
141
  |---|---|---|
142
- | input `frames` | (1, 64, 3, 448, 448), float32 | one clip in [-1, 1] |
143
- | input `n_valid` | (1,), int64 | number of real frames in the clip, 1 to 64 |
144
- | output `fake_prob` | (1,), float32 | P(manipulated) for the clip |
145
- | output `logits` | (1, 2), float32 | authentic and manipulated logits |
146
-
147
- Clip length, frame stride and image size are also stored in the file's metadata.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
148
 
149
  ```python
150
  import av
@@ -154,7 +176,7 @@ from huggingface_hub import hf_hub_download
154
  from PIL import Image
155
 
156
  session = ort.InferenceSession(
157
- hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"),
158
  providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
159
  )
160
  meta = session.get_modelmeta().custom_metadata_map
@@ -187,8 +209,8 @@ def score(path: str) -> float:
187
  print(score("video.mp4"))
188
  ```
189
 
190
- The ONNX graph runs in float32, while the paper's evaluation ran PyTorch in bfloat16, so
191
- scores differ slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with
192
  CUDA is installed, `import torch` before creating the session also works.
193
 
194
  ## Training
 
71
  |---|---|---|
72
  | `model.safetensors` | trained adapters, routers and head; the frozen CLIP ViT-L/14 is not included and is downloaded from OpenAI on first use | 415 MB |
73
  | `config.yaml` | model and data settings, read by the `mars` package | 1 KB |
74
+ | `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone, float32 | 1.64 GB |
75
+ | `model.fp16.onnx` | the same graph in float16, inputs and outputs still float32; less GPU memory | 0.83 GB |
76
+ | `usual_ranges.json` | percentiles of the ONNX video score on the authentic and manipulated test videos, with `model.onnx` | 3 KB |
77
+ | `usual_ranges.fp16.json` | the same with `model.fp16.onnx` | 3 KB |
78
+ | `serving.json` | measured GPU memory and time per clip with ONNX Runtime (both files, batch 1 and 2), opset, outputs, encoder chunk, ONNX Runtime version | 1 KB |
79
 
80
  This is the checkpoint behind the DFD-HR results in the MARS paper.
81
 
 
143
 
144
  | name | shape, type | meaning |
145
  |---|---|---|
146
+ | input `frames` | (B, 64, 3, 448, 448), float32 | B clips in [-1, 1] |
147
+ | input `n_valid` | (B,), int64 | number of real frames in each clip, 1 to 64 |
148
+ | output `fake_prob` | (B,), float32 | P(manipulated) for each clip |
149
+ | output `logits` | (B, 2), float32 | authentic and manipulated logits |
150
+ | output `frame_probs` | (B, 64), float32 | P(manipulated) of each frame on its own, 0 on padded frames |
151
+
152
+ The batch size B is free. Clip length, frame stride and image size are also stored in the
153
+ file's metadata.
154
+
155
+ Both files take the full 64-frame clip. Inside the graph the encoder processes 16 frames of
156
+ each clip at a time; frames are encoded independently, so this lowers memory without
157
+ changing the outputs. On an A100, one clip peaks at about 3.5 GiB of GPU memory with
158
+ `model.onnx` and 2.6 GiB with `model.fp16.onnx` (details in [`serving.json`](serving.json)).
159
+ The float16 video scores are within 0.007 of float32 on UADFV and Celeb-DF. Use `model.onnx`
160
+ on CPU, or as the source for a TensorRT engine.
161
+
162
+ The clip logits are the mean of the per-frame logits, so `fake_prob` is the sigmoid of the
163
+ mean logit of `frame_probs` over the real frames, not the mean of `frame_probs`. For the clip
164
+ score, use `fake_prob`; do not average `frame_probs`.
165
+
166
+ [`usual_ranges.json`](usual_ranges.json) gives the percentiles (0 to 100) of the video score
167
+ on the authentic and on the manipulated test videos listed under Results, computed with
168
+ `model.onnx`; [`usual_ranges.fp16.json`](usual_ranges.fp16.json) is the same for `model.fp16.onnx`.
169
+ Use them to place a score in context, not as a threshold.
170
 
171
  ```python
172
  import av
 
176
  from PIL import Image
177
 
178
  session = ort.InferenceSession(
179
+ hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"), # or "model.fp16.onnx" for less GPU memory
180
  providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
181
  )
182
  meta = session.get_modelmeta().custom_metadata_map
 
209
  print(score("video.mp4"))
210
  ```
211
 
212
+ Neither ONNX file runs in bfloat16 like the evaluation behind Results, so scores differ
213
+ slightly from that table. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with
214
  CUDA is installed, `import torch` before creating the session also works.
215
 
216
  ## Training
model.fp16.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3661c660c8d3de0ed1b7deb0c4eadb6ce5b97ec7c4753b79e160e6e8d4cb2b8f
3
+ size 827548529
model.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a5bd0b1e1c69d6e9bb4be3bff2deb7284b8deeb146a43134c7939bd407f622a9
3
- size 1634194132
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:24a21d8bcd4297f37c1f8be966daf6e92f06b9b29dfa8d94a5a8164b688fcbd6
3
+ size 1638346050
serving.json ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "onnxruntime": "onnxruntime-gpu 1.32.0",
3
+ "measured_on": "NVIDIA A100-SXM4-80GB, CUDA execution provider, arena_extend_strategy kSameAsRequested, cudnn_conv_algo_search HEURISTIC; peak is process total in GiB including the CUDA context; time is the median of 5 runs after 2 warm-up runs",
4
+ "onnx_opset": 17,
5
+ "batch_axis": "dynamic",
6
+ "n_frames": 64,
7
+ "encoder_chunk": 16,
8
+ "outputs": [
9
+ "fake_prob",
10
+ "logits",
11
+ "frame_probs"
12
+ ],
13
+ "files": {
14
+ "model.onnx": {
15
+ "precision": "float32",
16
+ "peak_vram_gb": {
17
+ "batch1": 3.45,
18
+ "batch2": 4.92
19
+ },
20
+ "seconds_per_clip": {
21
+ "batch1": 1.49,
22
+ "batch2": 1.43
23
+ }
24
+ },
25
+ "model.fp16.onnx": {
26
+ "precision": "float16",
27
+ "peak_vram_gb": {
28
+ "batch1": 2.55,
29
+ "batch2": 3.88
30
+ },
31
+ "seconds_per_clip": {
32
+ "batch1": 0.98,
33
+ "batch2": 0.94
34
+ }
35
+ }
36
+ },
37
+ "pytorch_bf16_reference": {
38
+ "precision": "bfloat16 autocast",
39
+ "peak_vram_gb": {
40
+ "batch1": 4.69,
41
+ "batch2": 4.97
42
+ },
43
+ "seconds_per_clip": {
44
+ "batch1": 0.75,
45
+ "batch2": 0.75
46
+ }
47
+ }
48
+ }
usual_ranges.fp16.json ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "real": [
3
+ 9.1e-05,
4
+ 0.003165,
5
+ 0.007157,
6
+ 0.013732,
7
+ 0.020334,
8
+ 0.027734,
9
+ 0.033919,
10
+ 0.043242,
11
+ 0.050797,
12
+ 0.057607,
13
+ 0.064636,
14
+ 0.070414,
15
+ 0.076433,
16
+ 0.083303,
17
+ 0.089844,
18
+ 0.095798,
19
+ 0.102051,
20
+ 0.107849,
21
+ 0.113737,
22
+ 0.119594,
23
+ 0.126099,
24
+ 0.131316,
25
+ 0.137817,
26
+ 0.143066,
27
+ 0.14917,
28
+ 0.155518,
29
+ 0.160034,
30
+ 0.165184,
31
+ 0.170654,
32
+ 0.177018,
33
+ 0.182996,
34
+ 0.188599,
35
+ 0.193726,
36
+ 0.198854,
37
+ 0.204221,
38
+ 0.209589,
39
+ 0.214712,
40
+ 0.220337,
41
+ 0.226301,
42
+ 0.231425,
43
+ 0.236328,
44
+ 0.241671,
45
+ 0.247681,
46
+ 0.253662,
47
+ 0.258955,
48
+ 0.264075,
49
+ 0.270264,
50
+ 0.276123,
51
+ 0.281143,
52
+ 0.286133,
53
+ 0.292236,
54
+ 0.297607,
55
+ 0.304063,
56
+ 0.310547,
57
+ 0.31665,
58
+ 0.32251,
59
+ 0.329102,
60
+ 0.334961,
61
+ 0.341553,
62
+ 0.348633,
63
+ 0.354736,
64
+ 0.362061,
65
+ 0.369385,
66
+ 0.376492,
67
+ 0.383545,
68
+ 0.390393,
69
+ 0.397954,
70
+ 0.406248,
71
+ 0.414062,
72
+ 0.422852,
73
+ 0.431885,
74
+ 0.440154,
75
+ 0.450195,
76
+ 0.459717,
77
+ 0.470649,
78
+ 0.480164,
79
+ 0.490479,
80
+ 0.500977,
81
+ 0.511719,
82
+ 0.524414,
83
+ 0.537109,
84
+ 0.547852,
85
+ 0.560059,
86
+ 0.57105,
87
+ 0.583496,
88
+ 0.597876,
89
+ 0.612998,
90
+ 0.631199,
91
+ 0.645996,
92
+ 0.662271,
93
+ 0.679688,
94
+ 0.697886,
95
+ 0.719727,
96
+ 0.744141,
97
+ 0.765625,
98
+ 0.792065,
99
+ 0.828184,
100
+ 0.866255,
101
+ 0.906279,
102
+ 0.950077,
103
+ 0.999023
104
+ ],
105
+ "fake": [
106
+ 0.000322,
107
+ 0.005215,
108
+ 0.009109,
109
+ 0.01474,
110
+ 0.022139,
111
+ 0.03284,
112
+ 0.045513,
113
+ 0.06334,
114
+ 0.082512,
115
+ 0.103088,
116
+ 0.120667,
117
+ 0.140928,
118
+ 0.159077,
119
+ 0.1766,
120
+ 0.195283,
121
+ 0.213513,
122
+ 0.229233,
123
+ 0.244751,
124
+ 0.260742,
125
+ 0.276968,
126
+ 0.293164,
127
+ 0.308872,
128
+ 0.325195,
129
+ 0.338623,
130
+ 0.352822,
131
+ 0.367798,
132
+ 0.379857,
133
+ 0.394819,
134
+ 0.411133,
135
+ 0.425781,
136
+ 0.439697,
137
+ 0.453989,
138
+ 0.469727,
139
+ 0.483154,
140
+ 0.498184,
141
+ 0.513672,
142
+ 0.52832,
143
+ 0.541992,
144
+ 0.556636,
145
+ 0.568975,
146
+ 0.583496,
147
+ 0.597627,
148
+ 0.611328,
149
+ 0.623838,
150
+ 0.635254,
151
+ 0.646973,
152
+ 0.658691,
153
+ 0.674316,
154
+ 0.68457,
155
+ 0.696289,
156
+ 0.709473,
157
+ 0.723311,
158
+ 0.73584,
159
+ 0.749023,
160
+ 0.760254,
161
+ 0.77138,
162
+ 0.78418,
163
+ 0.797061,
164
+ 0.808594,
165
+ 0.819365,
166
+ 0.82959,
167
+ 0.840744,
168
+ 0.850137,
169
+ 0.860352,
170
+ 0.870117,
171
+ 0.879932,
172
+ 0.890137,
173
+ 0.898926,
174
+ 0.907227,
175
+ 0.915039,
176
+ 0.924316,
177
+ 0.931641,
178
+ 0.939941,
179
+ 0.947266,
180
+ 0.953125,
181
+ 0.959717,
182
+ 0.96582,
183
+ 0.97168,
184
+ 0.976562,
185
+ 0.980469,
186
+ 0.985352,
187
+ 0.98877,
188
+ 0.991211,
189
+ 0.993652,
190
+ 0.995605,
191
+ 0.99707,
192
+ 0.998047,
193
+ 0.999023,
194
+ 0.999512,
195
+ 0.999512,
196
+ 0.999837,
197
+ 1.0,
198
+ 1.0,
199
+ 1.0,
200
+ 1.0,
201
+ 1.0,
202
+ 1.0,
203
+ 1.0,
204
+ 1.0,
205
+ 1.0,
206
+ 1.0
207
+ ],
208
+ "n_real": 11098,
209
+ "n_fake": 12435,
210
+ "test_sets": [
211
+ "ffpp",
212
+ "celebdf",
213
+ "dftimit",
214
+ "uadfv",
215
+ "hififace_paired",
216
+ "genvideo",
217
+ "df26"
218
+ ],
219
+ "runtime": "onnxruntime-gpu 1.32.0 fp16, model.fp16.onnx"
220
+ }
usual_ranges.json ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "real": [
3
+ 9.1e-05,
4
+ 0.003143,
5
+ 0.007131,
6
+ 0.013785,
7
+ 0.02035,
8
+ 0.027829,
9
+ 0.033729,
10
+ 0.043095,
11
+ 0.050553,
12
+ 0.058023,
13
+ 0.064771,
14
+ 0.070544,
15
+ 0.076469,
16
+ 0.083347,
17
+ 0.089711,
18
+ 0.095807,
19
+ 0.102122,
20
+ 0.108215,
21
+ 0.114043,
22
+ 0.119682,
23
+ 0.126083,
24
+ 0.131439,
25
+ 0.137844,
26
+ 0.143327,
27
+ 0.149318,
28
+ 0.155558,
29
+ 0.160023,
30
+ 0.165324,
31
+ 0.170334,
32
+ 0.177086,
33
+ 0.183172,
34
+ 0.188871,
35
+ 0.193849,
36
+ 0.198904,
37
+ 0.204573,
38
+ 0.209873,
39
+ 0.215128,
40
+ 0.220286,
41
+ 0.226417,
42
+ 0.231478,
43
+ 0.236716,
44
+ 0.242341,
45
+ 0.247905,
46
+ 0.254273,
47
+ 0.259144,
48
+ 0.264214,
49
+ 0.270441,
50
+ 0.276064,
51
+ 0.281567,
52
+ 0.286621,
53
+ 0.29245,
54
+ 0.297579,
55
+ 0.304448,
56
+ 0.310857,
57
+ 0.316792,
58
+ 0.322257,
59
+ 0.329321,
60
+ 0.335256,
61
+ 0.342264,
62
+ 0.348735,
63
+ 0.355019,
64
+ 0.362545,
65
+ 0.369679,
66
+ 0.376894,
67
+ 0.383503,
68
+ 0.391081,
69
+ 0.398633,
70
+ 0.40614,
71
+ 0.4142,
72
+ 0.423441,
73
+ 0.431969,
74
+ 0.441437,
75
+ 0.450826,
76
+ 0.460384,
77
+ 0.471334,
78
+ 0.47993,
79
+ 0.491037,
80
+ 0.501301,
81
+ 0.511784,
82
+ 0.524508,
83
+ 0.53693,
84
+ 0.548948,
85
+ 0.560428,
86
+ 0.571869,
87
+ 0.584066,
88
+ 0.599095,
89
+ 0.61321,
90
+ 0.630941,
91
+ 0.646079,
92
+ 0.662357,
93
+ 0.680184,
94
+ 0.69828,
95
+ 0.719263,
96
+ 0.743963,
97
+ 0.76628,
98
+ 0.791845,
99
+ 0.829163,
100
+ 0.866628,
101
+ 0.905704,
102
+ 0.950159,
103
+ 0.999104
104
+ ],
105
+ "fake": [
106
+ 0.00032,
107
+ 0.005218,
108
+ 0.009081,
109
+ 0.014721,
110
+ 0.022088,
111
+ 0.032722,
112
+ 0.045357,
113
+ 0.063397,
114
+ 0.082877,
115
+ 0.10301,
116
+ 0.120719,
117
+ 0.140095,
118
+ 0.159242,
119
+ 0.17675,
120
+ 0.195467,
121
+ 0.21329,
122
+ 0.229752,
123
+ 0.244674,
124
+ 0.259957,
125
+ 0.276616,
126
+ 0.293312,
127
+ 0.308799,
128
+ 0.324263,
129
+ 0.337707,
130
+ 0.3531,
131
+ 0.368397,
132
+ 0.380784,
133
+ 0.395837,
134
+ 0.411657,
135
+ 0.425691,
136
+ 0.440024,
137
+ 0.454213,
138
+ 0.469752,
139
+ 0.483112,
140
+ 0.498041,
141
+ 0.51376,
142
+ 0.527799,
143
+ 0.54097,
144
+ 0.556939,
145
+ 0.569282,
146
+ 0.583418,
147
+ 0.597487,
148
+ 0.611513,
149
+ 0.624201,
150
+ 0.635425,
151
+ 0.647072,
152
+ 0.659005,
153
+ 0.674139,
154
+ 0.684488,
155
+ 0.696546,
156
+ 0.709589,
157
+ 0.723177,
158
+ 0.735987,
159
+ 0.748802,
160
+ 0.760542,
161
+ 0.771247,
162
+ 0.783933,
163
+ 0.797384,
164
+ 0.80936,
165
+ 0.819677,
166
+ 0.829755,
167
+ 0.840717,
168
+ 0.850627,
169
+ 0.860297,
170
+ 0.870136,
171
+ 0.879982,
172
+ 0.890237,
173
+ 0.899022,
174
+ 0.90713,
175
+ 0.914941,
176
+ 0.92409,
177
+ 0.931538,
178
+ 0.939605,
179
+ 0.947222,
180
+ 0.953233,
181
+ 0.959611,
182
+ 0.965933,
183
+ 0.971492,
184
+ 0.976719,
185
+ 0.980607,
186
+ 0.98531,
187
+ 0.988646,
188
+ 0.991231,
189
+ 0.99358,
190
+ 0.995663,
191
+ 0.99702,
192
+ 0.998113,
193
+ 0.998895,
194
+ 0.999332,
195
+ 0.999629,
196
+ 0.999776,
197
+ 0.999878,
198
+ 0.999935,
199
+ 0.999958,
200
+ 0.999974,
201
+ 0.999983,
202
+ 0.99999,
203
+ 0.999994,
204
+ 0.999997,
205
+ 0.999999,
206
+ 1.0
207
+ ],
208
+ "n_real": 11098,
209
+ "n_fake": 12435,
210
+ "test_sets": [
211
+ "ffpp",
212
+ "celebdf",
213
+ "dftimit",
214
+ "uadfv",
215
+ "hififace_paired",
216
+ "genvideo",
217
+ "df26"
218
+ ],
219
+ "runtime": "onnxruntime-gpu 1.32.0 fp32, model.onnx"
220
+ }