aditya-rAj19 commited on
Commit
e323c6f
Β·
1 Parent(s): 4a2b4cf

Enable GPU for InsightFace: auto-add CUDA12 + cuDNN9 DLL dirs on Windows

Browse files

onnxruntime-gpu silently fell back to CPU (LoadLibrary error 126) because
cuDNN 9 keeps its CUDA-12 DLLs in a versioned subfolder (CUDNN\vX\bin\12.x)
that isn't on PATH. detector._ensure_cuda_dlls() auto-discovers the CUDA toolkit
bin + cuDNN 12.x bin and prepends them before importing onnxruntime. Verified:
InsightFace, GFPGAN and RealESRGAN all run on the RTX 4060. No-op on Linux.

+ GPU_ENHANCEMENT_PLAN.md runbook (incl. batch test harness).

Files changed (2) hide show
  1. GPU_ENHANCEMENT_PLAN.md +100 -0
  2. core/detector.py +35 -0
GPU_ENHANCEMENT_PLAN.md ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # GPU Enhancement Runbook β€” DeepFace Studio
2
+
3
+ Executed once `torch+cu121` **and** `onnxruntime-gpu` are installed. Goal: best
4
+ realistic output, working **both locally (RTX 4060)** and on **HF Pro GPU**.
5
+
6
+ ---
7
+
8
+ ## Phase 1 β€” Verify the GPU stack
9
+ 1. `torch.cuda.is_available()` β†’ True, `torch.cuda.get_device_name(0)` β†’ RTX 4060.
10
+ 2. `onnxruntime.get_available_providers()` β†’ contains `CUDAExecutionProvider`.
11
+ 3. Load InsightFace and confirm it initialises on `CUDAExecutionProvider`
12
+ (check `_ORT_PROVIDERS` in `core/detector.py` is `[CUDA, CPU]`).
13
+ 4. Confirm `core/super_res.py::_device()` and `core/head_swap.py::_device()`
14
+ both return `cuda`.
15
+ 5. **Fail-safe**: if onnxruntime-gpu can't find cuDNN, install matching cuDNN 9
16
+ or pin `onnxruntime-gpu==1.19.2` (CUDA 12 + cuDNN 9).
17
+
18
+ ## Phase 2 β€” Render + benchmark a real swap on GPU
19
+ 1. Run the full pipeline (head β†’ face β†’ GFPGAN β†’ skin-match β†’ hair β†’ blend β†’
20
+ RealESRGAN Γ—4) on a real source + a location target.
21
+ 2. Save the output, **view it**, and time it (expect a few seconds vs ~60s CPU).
22
+ 3. Confirm only the main subject is swapped (no background/poster faces).
23
+
24
+ ## Phase 3 β€” Quality tuning (now that there's GPU headroom)
25
+ 1. **Working resolution**: raise `resize_keep_aspect(..., 1024)` in `web_app.py`
26
+ to **1280–1536** (GPU can handle it β†’ sharper swap, better landmarks).
27
+ 2. **RealESRGAN Γ—4**: real GPU upscale for the download (currently Lanczos on
28
+ CPU). Tune `realesrgan_weight` (~0.5) so skin isn't plastic.
29
+ 3. **GFPGAN**: confirm it runs on GPU; keep `upscale=1` (RealESRGAN does scaling).
30
+ 4. **Skin tone**: keep `match_skin_to_source` at 0.75; re-evaluate visually.
31
+ 5. **Blend**: keep Laplacian 4-level over the face mask; check no hard seam.
32
+ 6. Re-render and visually compare before/after each change (no guessing).
33
+
34
+ ## Phase 4 β€” Real hair swap (HairFastGAN, GPU)
35
+ 1. `python scripts/setup_hairfast.py` β†’ clones `external/HairFastGAN` + weights,
36
+ patches ops, sets up the MSVC/CUDA launcher.
37
+ 2. `pip install git+https://github.com/openai/CLIP.git face_alignment lpips kornia`.
38
+ 3. Set `HAIRFAST_LOCAL=1`; `core/hair_transfer.py` auto-uses the local GPU path.
39
+ 4. Wire it into the pipeline: after the face swap, transfer the **source's
40
+ hairstyle** onto the result (real hair, not the BiSeNet warp).
41
+ 5. Render + view; first call is slow (op compile), then ~15–20s on the 4060.
42
+ 6. **VRAM guard**: 8GB is tight β€” run HairFastGAN, free InsightFace/GFPGAN
43
+ between stages if needed; fall back gracefully if OOM.
44
+
45
+ ## Phase 5 β€” Automated test harness (the "100 tests per location")
46
+ Feasible on GPU (~3-5s/swap). 100 Γ— 19 locations Γ— 2 genders β‰ˆ 3,800 swaps β‰ˆ
47
+ 3-5 hrs on the RTX 4060 β€” run as a batch job.
48
+
49
+ **5a. Source face dataset**
50
+ - Need a folder of diverse SOURCE faces: light / medium / deep skin tones,
51
+ round / oval / long face shapes, short / long / curly hair, with / without
52
+ glasses and beard, balanced male + female.
53
+ - Source options: a face dataset (e.g. FFHQ samples / a provided folder).
54
+ Drop them in `tests/faces/male/` and `tests/faces/female/`. The harness
55
+ scales to however many are provided (10, 50, 100…).
56
+
57
+ **5b. Build `scripts/batch_test.py`**
58
+ - For each gender β†’ each location β†’ each source face: run the full GPU pipeline.
59
+ - Save outputs to `tests/output/<gender>/<location>/<source>.jpg`.
60
+ - Build a **contact sheet (grid)** per location for quick visual QA.
61
+ - Compute metrics per swap via `core/quality_checker.py`
62
+ (alignment, blend, Ξ”E, naturalness) β†’ write `tests/report.csv`.
63
+
64
+ **5c. Analyse + tune**
65
+ - Aggregate metrics: flag locations / source-types with low scores
66
+ (e.g. deep skin tone + bright location, or long-hair sources).
67
+ - **View** the worst cases; tune the specific stage (skin-match strength,
68
+ blend, head-swap region, hair) and re-run that subset.
69
+ - Iterate until the score distribution + visual quality are consistently good
70
+ across all skin tones, face shapes and hairstyles.
71
+
72
+ **5d. Reality on "fine-tune the model itself"**
73
+ - The `inswapper_128` weights can't be fine-tuned here (no training pipeline /
74
+ dataset / time). What we *can* tune is the **pipeline around it** (resolution,
75
+ head/face/hair order, skin-match, blend, RealESRGAN, HairFastGAN). The harness
76
+ drives that tuning with real numbers + visuals. For a genuinely different
77
+ identity model, the upgrade path is a stronger swapper (e.g. SimSwap-HQ /
78
+ inswapper-512 / a GPU diffusion swapper) β€” also GPU-only.
79
+
80
+ ## Phase 6 β€” GPU deploy for HF Pro
81
+ 1. `requirements-gpu.txt`: `torch==2.3.0+cu121`, `torchvision==0.18.0+cu121`
82
+ (cu121 index), `onnxruntime-gpu==1.19.2`, + the rest of the deps.
83
+ 2. `Dockerfile.gpu`: base `nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04`
84
+ (CUDA 12 + cuDNN 9), install Python 3.10, torch cu121, onnxruntime-gpu, deps,
85
+ build the React frontend (stage 1), copy app, `CMD ./startup.sh`.
86
+ 3. README: note "switch HF Space to GPU hardware + use Dockerfile.gpu".
87
+ 4. Keep the CPU `Dockerfile` as fallback so nothing breaks before the upgrade.
88
+
89
+ ## Phase 7 β€” Ship
90
+ 1. `npm run build` (frontend), syntax-check backend, run `pytest`.
91
+ 2. Commit + push; confirm CI green and HF deploy green.
92
+ 3. On HF: switch Space to **GPU hardware**, point to `Dockerfile.gpu`, set the
93
+ existing secrets; verify a live GPU swap.
94
+
95
+ ---
96
+
97
+ ### Guardrails
98
+ - **Verify every change by rendering + viewing the image** β€” no blind tuning.
99
+ - Keep CPU fallback intact (so the app still runs without a GPU).
100
+ - Don't break the green HF deploy until the GPU Space is confirmed working.
core/detector.py CHANGED
@@ -1,7 +1,42 @@
1
  import os
 
 
2
  import cv2
3
  import cv2.data
4
  import numpy as np
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  try:
6
  import onnxruntime as ort
7
  _ORT_PROVIDERS = (
 
1
  import os
2
+ import sys
3
+ import glob
4
  import cv2
5
  import cv2.data
6
  import numpy as np
7
+
8
+
9
+ def _ensure_cuda_dlls():
10
+ """
11
+ Windows: make CUDA 12 + cuDNN 9 DLLs discoverable so onnxruntime-gpu can load
12
+ CUDAExecutionProvider β€” otherwise it silently falls back to CPU (LoadLibrary
13
+ error 126). cuDNN 9 ships its CUDA-12 DLLs in a versioned subfolder
14
+ (…\\CUDNN\\vX\\bin\\12.x) that isn't on PATH by default. We auto-discover the
15
+ CUDA-toolkit bin + the cuDNN 12.x bin and prepend them to PATH.
16
+ No-op on Linux (the HF Docker/CUDA base image already puts CUDA on the path).
17
+ """
18
+ if sys.platform != "win32":
19
+ return
20
+ pf = os.environ.get("ProgramFiles", r"C:\Program Files")
21
+ patterns = [
22
+ os.path.join(pf, "NVIDIA GPU Computing Toolkit", "CUDA", "v12*", "bin"),
23
+ os.path.join(pf, "NVIDIA", "CUDNN", "v*", "bin", "12*"), # cuDNN9 for CUDA12
24
+ os.path.join(pf, "NVIDIA", "CUDNN", "v*", "bin"),
25
+ ]
26
+ added = set()
27
+ for pat in patterns:
28
+ for d in sorted(glob.glob(pat), reverse=True): # newest version first
29
+ if os.path.isdir(d) and d not in added:
30
+ added.add(d)
31
+ os.environ["PATH"] = d + os.pathsep + os.environ.get("PATH", "")
32
+ try:
33
+ os.add_dll_directory(d)
34
+ except Exception:
35
+ pass
36
+
37
+
38
+ _ensure_cuda_dlls()
39
+
40
  try:
41
  import onnxruntime as ort
42
  _ORT_PROVIDERS = (