Spaces:
Running
Running
Commit Β·
e91f9b4
1
Parent(s): 0adbe06
docs(f17): profiling write-up for the 30-signal parallelization
Browse filesReal measured timings for 27 of the 30 signals (synthetic 1600x1200
JPEG, 5 runs each): ~5.4s sequential sum, ~2.1s parallel floor
(statistical bundle is the slowest single signal). Structural
estimate for DIRE/CLIP/own-embedding (couldn't execute torch in the
profiling sandbox) based on their actual model architectures -- DIRE
runs 20 sequential UNet forward passes through SD 2.1, almost
certainly the dominant cost.
Documents the concrete reason DIRE/CLIP/own-embedding are deferred:
DIRE's DDIMScheduler is cached and shared process-wide, and both
add_noise() and the denoising loop mutate it -- a real thread-safety
question that needs resolving before those three join the parallel
pool, not a hypothetical one.
- PROFILING_F17.md +125 -0
PROFILING_F17.md
ADDED
|
@@ -0,0 +1,125 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# F-17 Profiling β Parallelizing the 30 Signals
|
| 2 |
+
|
| 3 |
+
## Method
|
| 4 |
+
|
| 5 |
+
27 of the 30 signals (19-signal statistical bundle + 8 classical forensic detectors +
|
| 6 |
+
`image_type_classifier`) are pure numpy/opencv/scipy/scikit-image β no torch dependency β so they
|
| 7 |
+
were timed for real: a synthetic 1600Γ1200 JPEG (smooth gradient + sensor-like noise, quality 88),
|
| 8 |
+
5 runs each, mean/min/max recorded. The 3 torch-dependent signals (DIRE, CLIP, own-embedding)
|
| 9 |
+
couldn't be executed in the profiling session's sandbox (no working torch install there) β their
|
| 10 |
+
relative cost was estimated by reading the model architecture directly instead.
|
| 11 |
+
|
| 12 |
+
## Real measured numbers (mean of 5 runs, synthetic 1600Γ1200 JPEG)
|
| 13 |
+
|
| 14 |
+
| Signal | Mean | Notes |
|
| 15 |
+
|---|---|---|
|
| 16 |
+
| Statistical bundle (19 signals, 1 call) | 2116.5ms | Internally: BasicSignals 1396.7ms, UltraSignals 264.5ms, AdvancedSignals 232.2ms, CovarianceSignals 149.8ms |
|
| 17 |
+
| JPEG ghost | 1026.1ms | |
|
| 18 |
+
| DCT frequency | 758.6ms | |
|
| 19 |
+
| Noiseprint | 557.7ms | |
|
| 20 |
+
| Noise map | 481.6ms | |
|
| 21 |
+
| CFA | 154.8ms | |
|
| 22 |
+
| Image type classifier | 126.4ms | Gates prnu/ela/metadata β must resolve before those three, independent of everything else |
|
| 23 |
+
| PRNU | 92.0ms | |
|
| 24 |
+
| ELA | 75.2ms | |
|
| 25 |
+
| Metadata | 0.1ms | |
|
| 26 |
+
|
| 27 |
+
**Sum if sequential: ~5.4s. Slowest single signal (the parallel floor for this group): ~2.1s
|
| 28 |
+
(statistical bundle).**
|
| 29 |
+
|
| 30 |
+
Within the statistical bundle itself, F-16 already decomposed it into 4 independent components
|
| 31 |
+
sharing only a read-only `ImageContext` β `BasicSignals` alone is ~66% of the bundle's own total
|
| 32 |
+
(1.4s of 2.1s), so there's a second, smaller layer of parallelization available inside the bundle
|
| 33 |
+
if the outer-level win alone isn't enough later. Not implemented in this pass β flagged for
|
| 34 |
+
whoever picks up further F-17 work.
|
| 35 |
+
|
| 36 |
+
## DIRE / CLIP / own-embedding β structural estimate, not measured
|
| 37 |
+
|
| 38 |
+
Read directly from `dire_detector.py`, `clip_detector.py`, `own_embedding_detector.py`:
|
| 39 |
+
|
| 40 |
+
- **DIRE**: loads a full Stable Diffusion 2.1 pipeline, runs **20 sequential UNet forward passes**
|
| 41 |
+
(~860M params) at 512Γ512, plus a VAE encode/decode. Almost certainly the dominant cost of the
|
| 42 |
+
whole pipeline on CPU β likely an order of magnitude past the 2.1s statistical bundle.
|
| 43 |
+
- **CLIP**: one forward pass, ViT-B/32 (~88M params). Fast relative to DIRE.
|
| 44 |
+
- **own-embedding**: one forward pass, EfficientNet-B0 (~5M params). Fastest of the three.
|
| 45 |
+
|
| 46 |
+
To get a real DIRE number, run this in an environment with working torch (this repo's own dev
|
| 47 |
+
environment, not necessarily the profiling sandbox):
|
| 48 |
+
|
| 49 |
+
```python
|
| 50 |
+
import time, statistics
|
| 51 |
+
from backend.services.dire_detector import DIREDetector
|
| 52 |
+
|
| 53 |
+
with open("some_real_test_image.jpg", "rb") as f:
|
| 54 |
+
image_bytes = f.read()
|
| 55 |
+
|
| 56 |
+
times = []
|
| 57 |
+
d = DIREDetector()
|
| 58 |
+
d.detect(image_bytes, "warmup.jpg") # first call also pays model-load cost β exclude it
|
| 59 |
+
for _ in range(3):
|
| 60 |
+
t0 = time.perf_counter()
|
| 61 |
+
d.detect(image_bytes, "test.jpg")
|
| 62 |
+
times.append(time.perf_counter() - t0)
|
| 63 |
+
print(f"DIRE mean: {statistics.mean(times):.2f}s (min {min(times):.2f}s, max {max(times):.2f}s)")
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
## The scheduler-sharing risk (why DIRE/CLIP/own-embedding are deliberately NOT parallelized yet)
|
| 67 |
+
|
| 68 |
+
DIRE's `DDIMScheduler` is loaded once and cached process-wide (`ModelCache`, F-7) β
|
| 69 |
+
`cached_model['scheduler']` β and every `DIREDetector` instance shares that *same object*. Both
|
| 70 |
+
`_add_noise()` and the denoising loop call mutating methods on it (`scheduler.add_noise()`,
|
| 71 |
+
`scheduler.set_timesteps()`, `scheduler.step()`). `denoise_steps` is currently hardcoded to `20`,
|
| 72 |
+
so a `set_timesteps(20)` race between two concurrent DIRE calls likely produces the same values
|
| 73 |
+
either way β probably not silently wrong today. But that's incidental, not a guarantee, and this
|
| 74 |
+
is exactly the risk the original F-17 planning flagged as "a plausible interaction risk nobody has
|
| 75 |
+
tested yet." Resolve this (explicit thread-safety test, or a per-instance scheduler clone instead
|
| 76 |
+
of sharing the cached one) before enabling concurrent DIRE calls.
|
| 77 |
+
|
| 78 |
+
## What was actually shipped this round (partial F-17)
|
| 79 |
+
|
| 80 |
+
The 9 mutually-independent, non-torch signal calls β the statistical bundle + 8 classical
|
| 81 |
+
forensic detectors β now run in a `ThreadPoolExecutor` inside `AdvancedEnsembleDetector.detect()`,
|
| 82 |
+
instead of one after another. Every module involved was grepped for module-level caches/globals
|
| 83 |
+
before this change; none exist (all are pure functions over their own local data), so there's no
|
| 84 |
+
DIRE-scheduler-style shared-state risk in this half of the change.
|
| 85 |
+
|
| 86 |
+
**A real trap found and fixed during verification**: the profiling/development sandbox turned out
|
| 87 |
+
to have exactly 1 CPU core. On 1 core, 9 threads contending for that core measured **~1.5x SLOWER**
|
| 88 |
+
than plain sequential β pure context-switch overhead, zero real parallelism, not a hypothetical
|
| 89 |
+
concern. Fixed by capping `max_workers = min(len(_signal_tasks), max(1, os.cpu_count() or 1))`, so
|
| 90 |
+
the pool degrades to one-thread-at-a-time (matching sequential performance) on a constrained host
|
| 91 |
+
instead of regressing, and scales up to real concurrency on whatever's actually available.
|
| 92 |
+
|
| 93 |
+
The project's actual production target (Hugging Face Spaces CPU Basic) is confirmed 2 vCPU β real,
|
| 94 |
+
though bounded (not the full 9-way ideal), benefit is expected there. DIRE/CLIP/own-embedding
|
| 95 |
+
remain sequential, tracked as a separate follow-up gated on the scheduler-sharing question above
|
| 96 |
+
and a real DIRE number from the script in this doc.
|
| 97 |
+
|
| 98 |
+
## Verification performed
|
| 99 |
+
|
| 100 |
+
- Ran the real, patched `AdvancedEnsembleDetector.detect()` (torch import stubbed just enough to
|
| 101 |
+
satisfy `dire_detector.py`/`clip_detector.py`'s type annotations at import time; `dire_detector`/
|
| 102 |
+
`clip_detector`/`own_detector` swapped for instant fakes so timing measures only the 9-signal
|
| 103 |
+
pool this change touches) β 3 repeated runs produced identical `ai_probability` and identical
|
| 104 |
+
30-signal sets, confirming no race condition/data corruption from the thread pool.
|
| 105 |
+
- Directly verified the `max_workers` capping logic against 3 scenarios: `cpu_count()=1` β 1
|
| 106 |
+
worker, `cpu_count()=64` β 9 workers (capped at task count, not core count), `cpu_count()=None`
|
| 107 |
+
β 1 worker (not passed through as `None`, which `ThreadPoolExecutor` would otherwise interpret
|
| 108 |
+
as "pick a default based on core count," silently reintroducing the oversubscription bug this
|
| 109 |
+
fix exists to prevent).
|
| 110 |
+
- Full `backend/` still `py_compile`-clean after the change.
|
| 111 |
+
- Could **not** measure real multi-core speedup in the profiling sandbox (1 core, by definition
|
| 112 |
+
can't demonstrate parallelism). Recommend running the timing harness below on this repo's own
|
| 113 |
+
environment (the 2-vCPU HF Space, or local dev hardware) to confirm the real-world win:
|
| 114 |
+
|
| 115 |
+
```python
|
| 116 |
+
import time
|
| 117 |
+
from backend.services.advanced_ensemble_detector import AdvancedEnsembleDetector
|
| 118 |
+
|
| 119 |
+
with open("some_real_test_image.jpg", "rb") as f:
|
| 120 |
+
image_bytes = f.read()
|
| 121 |
+
|
| 122 |
+
t0 = time.perf_counter()
|
| 123 |
+
AdvancedEnsembleDetector(image_bytes, "test.jpg").detect()
|
| 124 |
+
print(f"Full detect(): {time.perf_counter() - t0:.2f}s")
|
| 125 |
+
```
|