abinazebinoy commited on
Commit
e91f9b4
Β·
1 Parent(s): 0adbe06

docs(f17): profiling write-up for the 30-signal parallelization

Browse files

Real measured timings for 27 of the 30 signals (synthetic 1600x1200
JPEG, 5 runs each): ~5.4s sequential sum, ~2.1s parallel floor
(statistical bundle is the slowest single signal). Structural
estimate for DIRE/CLIP/own-embedding (couldn't execute torch in the
profiling sandbox) based on their actual model architectures -- DIRE
runs 20 sequential UNet forward passes through SD 2.1, almost
certainly the dominant cost.

Documents the concrete reason DIRE/CLIP/own-embedding are deferred:
DIRE's DDIMScheduler is cached and shared process-wide, and both
add_noise() and the denoising loop mutate it -- a real thread-safety
question that needs resolving before those three join the parallel
pool, not a hypothetical one.

Files changed (1) hide show
  1. PROFILING_F17.md +125 -0
PROFILING_F17.md ADDED
@@ -0,0 +1,125 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # F-17 Profiling β€” Parallelizing the 30 Signals
2
+
3
+ ## Method
4
+
5
+ 27 of the 30 signals (19-signal statistical bundle + 8 classical forensic detectors +
6
+ `image_type_classifier`) are pure numpy/opencv/scipy/scikit-image β€” no torch dependency β€” so they
7
+ were timed for real: a synthetic 1600Γ—1200 JPEG (smooth gradient + sensor-like noise, quality 88),
8
+ 5 runs each, mean/min/max recorded. The 3 torch-dependent signals (DIRE, CLIP, own-embedding)
9
+ couldn't be executed in the profiling session's sandbox (no working torch install there) β€” their
10
+ relative cost was estimated by reading the model architecture directly instead.
11
+
12
+ ## Real measured numbers (mean of 5 runs, synthetic 1600Γ—1200 JPEG)
13
+
14
+ | Signal | Mean | Notes |
15
+ |---|---|---|
16
+ | Statistical bundle (19 signals, 1 call) | 2116.5ms | Internally: BasicSignals 1396.7ms, UltraSignals 264.5ms, AdvancedSignals 232.2ms, CovarianceSignals 149.8ms |
17
+ | JPEG ghost | 1026.1ms | |
18
+ | DCT frequency | 758.6ms | |
19
+ | Noiseprint | 557.7ms | |
20
+ | Noise map | 481.6ms | |
21
+ | CFA | 154.8ms | |
22
+ | Image type classifier | 126.4ms | Gates prnu/ela/metadata β€” must resolve before those three, independent of everything else |
23
+ | PRNU | 92.0ms | |
24
+ | ELA | 75.2ms | |
25
+ | Metadata | 0.1ms | |
26
+
27
+ **Sum if sequential: ~5.4s. Slowest single signal (the parallel floor for this group): ~2.1s
28
+ (statistical bundle).**
29
+
30
+ Within the statistical bundle itself, F-16 already decomposed it into 4 independent components
31
+ sharing only a read-only `ImageContext` β€” `BasicSignals` alone is ~66% of the bundle's own total
32
+ (1.4s of 2.1s), so there's a second, smaller layer of parallelization available inside the bundle
33
+ if the outer-level win alone isn't enough later. Not implemented in this pass β€” flagged for
34
+ whoever picks up further F-17 work.
35
+
36
+ ## DIRE / CLIP / own-embedding β€” structural estimate, not measured
37
+
38
+ Read directly from `dire_detector.py`, `clip_detector.py`, `own_embedding_detector.py`:
39
+
40
+ - **DIRE**: loads a full Stable Diffusion 2.1 pipeline, runs **20 sequential UNet forward passes**
41
+ (~860M params) at 512Γ—512, plus a VAE encode/decode. Almost certainly the dominant cost of the
42
+ whole pipeline on CPU β€” likely an order of magnitude past the 2.1s statistical bundle.
43
+ - **CLIP**: one forward pass, ViT-B/32 (~88M params). Fast relative to DIRE.
44
+ - **own-embedding**: one forward pass, EfficientNet-B0 (~5M params). Fastest of the three.
45
+
46
+ To get a real DIRE number, run this in an environment with working torch (this repo's own dev
47
+ environment, not necessarily the profiling sandbox):
48
+
49
+ ```python
50
+ import time, statistics
51
+ from backend.services.dire_detector import DIREDetector
52
+
53
+ with open("some_real_test_image.jpg", "rb") as f:
54
+ image_bytes = f.read()
55
+
56
+ times = []
57
+ d = DIREDetector()
58
+ d.detect(image_bytes, "warmup.jpg") # first call also pays model-load cost β€” exclude it
59
+ for _ in range(3):
60
+ t0 = time.perf_counter()
61
+ d.detect(image_bytes, "test.jpg")
62
+ times.append(time.perf_counter() - t0)
63
+ print(f"DIRE mean: {statistics.mean(times):.2f}s (min {min(times):.2f}s, max {max(times):.2f}s)")
64
+ ```
65
+
66
+ ## The scheduler-sharing risk (why DIRE/CLIP/own-embedding are deliberately NOT parallelized yet)
67
+
68
+ DIRE's `DDIMScheduler` is loaded once and cached process-wide (`ModelCache`, F-7) β€”
69
+ `cached_model['scheduler']` β€” and every `DIREDetector` instance shares that *same object*. Both
70
+ `_add_noise()` and the denoising loop call mutating methods on it (`scheduler.add_noise()`,
71
+ `scheduler.set_timesteps()`, `scheduler.step()`). `denoise_steps` is currently hardcoded to `20`,
72
+ so a `set_timesteps(20)` race between two concurrent DIRE calls likely produces the same values
73
+ either way β€” probably not silently wrong today. But that's incidental, not a guarantee, and this
74
+ is exactly the risk the original F-17 planning flagged as "a plausible interaction risk nobody has
75
+ tested yet." Resolve this (explicit thread-safety test, or a per-instance scheduler clone instead
76
+ of sharing the cached one) before enabling concurrent DIRE calls.
77
+
78
+ ## What was actually shipped this round (partial F-17)
79
+
80
+ The 9 mutually-independent, non-torch signal calls β€” the statistical bundle + 8 classical
81
+ forensic detectors β€” now run in a `ThreadPoolExecutor` inside `AdvancedEnsembleDetector.detect()`,
82
+ instead of one after another. Every module involved was grepped for module-level caches/globals
83
+ before this change; none exist (all are pure functions over their own local data), so there's no
84
+ DIRE-scheduler-style shared-state risk in this half of the change.
85
+
86
+ **A real trap found and fixed during verification**: the profiling/development sandbox turned out
87
+ to have exactly 1 CPU core. On 1 core, 9 threads contending for that core measured **~1.5x SLOWER**
88
+ than plain sequential β€” pure context-switch overhead, zero real parallelism, not a hypothetical
89
+ concern. Fixed by capping `max_workers = min(len(_signal_tasks), max(1, os.cpu_count() or 1))`, so
90
+ the pool degrades to one-thread-at-a-time (matching sequential performance) on a constrained host
91
+ instead of regressing, and scales up to real concurrency on whatever's actually available.
92
+
93
+ The project's actual production target (Hugging Face Spaces CPU Basic) is confirmed 2 vCPU β€” real,
94
+ though bounded (not the full 9-way ideal), benefit is expected there. DIRE/CLIP/own-embedding
95
+ remain sequential, tracked as a separate follow-up gated on the scheduler-sharing question above
96
+ and a real DIRE number from the script in this doc.
97
+
98
+ ## Verification performed
99
+
100
+ - Ran the real, patched `AdvancedEnsembleDetector.detect()` (torch import stubbed just enough to
101
+ satisfy `dire_detector.py`/`clip_detector.py`'s type annotations at import time; `dire_detector`/
102
+ `clip_detector`/`own_detector` swapped for instant fakes so timing measures only the 9-signal
103
+ pool this change touches) β€” 3 repeated runs produced identical `ai_probability` and identical
104
+ 30-signal sets, confirming no race condition/data corruption from the thread pool.
105
+ - Directly verified the `max_workers` capping logic against 3 scenarios: `cpu_count()=1` β†’ 1
106
+ worker, `cpu_count()=64` β†’ 9 workers (capped at task count, not core count), `cpu_count()=None`
107
+ β†’ 1 worker (not passed through as `None`, which `ThreadPoolExecutor` would otherwise interpret
108
+ as "pick a default based on core count," silently reintroducing the oversubscription bug this
109
+ fix exists to prevent).
110
+ - Full `backend/` still `py_compile`-clean after the change.
111
+ - Could **not** measure real multi-core speedup in the profiling sandbox (1 core, by definition
112
+ can't demonstrate parallelism). Recommend running the timing harness below on this repo's own
113
+ environment (the 2-vCPU HF Space, or local dev hardware) to confirm the real-world win:
114
+
115
+ ```python
116
+ import time
117
+ from backend.services.advanced_ensemble_detector import AdvancedEnsembleDetector
118
+
119
+ with open("some_real_test_image.jpg", "rb") as f:
120
+ image_bytes = f.read()
121
+
122
+ t0 = time.perf_counter()
123
+ AdvancedEnsembleDetector(image_bytes, "test.jpg").detect()
124
+ print(f"Full detect(): {time.perf_counter() - t0:.2f}s")
125
+ ```