File size: 2,523 Bytes
24dd68d df7fff2 c47eba8 d783468 a99f23b d783468 24dd68d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | # Backend scripts
Utility / probe scripts for tagging accuracy, throughput, and model audits.
## Tag recall benchmark (ML vs WD)
UI: **Debug → Tag recall benchmark**.
Pinned suite: `backend/app/data/tag_recall_suite.json`
Cache (gitignored): `sample_data/tag_recall_suite/`
```text
.venv\Scripts\python.exe backend\scripts\fetch_tag_recall_suite.py --rebuild
.venv\Scripts\python.exe backend\scripts\run_tag_recall_eval.py
```
Latest report: `backend/scripts/out/tag_recall_eval_latest.json`.
False-positive rates on the same suite (selected Settings folders → taxonomy
evidence tags):
```text
.venv\Scripts\python.exe backend\scripts\run_tag_fp_eval.py --also-wd-threshold
```
UI: **Debug → Tag false-positive benchmark**.
Latest report: `backend/scripts/out/tag_fp_eval_latest.json`.
## THR3SHR debug: real-life vs anime eval
UI: **Debug → Real-life vs anime**.
API: `POST /api/debug/realism-eval` with
`{"count_per_class": 20, "compare_models": true}`.
CLI (thorough, multi-model + local video probe):
```text
.venv\Scripts\python.exe backend\scripts\run_realism_debug_eval.py 20
```
Latest report: `backend/scripts/out/realism_debug_eval_latest.json`.
Photos: RandomUser people portraits (remote). Anime: Safebooru typical +
`realistic` / `photorealistic` / `3d` edges. Routing uses production
`real_life` taxonomy.
## Real-life vs anime (`real_life` bucket)
Production `backend/app/data/taxonomy.json` includes a priority-`0` `real_life`
folder (aliases: `photo`, `Real Life`, …). Select that folder in the UI to
route accidental camera photos away from content buckets.
### Probe
```text
.venv\Scripts\python.exe backend\scripts\debug_realism_probe.py
```
Report: `backend/scripts/out/realism_probe_report.json`.
Findings (12 photos + 12 anime):
- **GO** with WD, best model **`wd_eva02_large`**
- WD v3 has **`realistic` / `photorealistic`** only (no `photo_(medium)` / `3d`)
- Anime max `realistic` ≈ 0.004 on EVA02; photos often 0.11–0.99
- Inference always includes `realistic`/`photorealistic` when score ≥ **0.10**
even if below `wd_general_threshold` (so moon/sky shots are not dropped)
### Evidence policy
| Tag | Weight | Notes |
|-----|--------|--------|
| `photorealistic` | 1.0 | Hard (WD + ML) |
| `realistic` | 1.0 | Hard on WD (probe-safe vs anime) |
| `photo_(medium)` | 1.0 | Hard, **ML only** |
| `3d` | 0.45 | Soft, **ML only** |
Prop/framing tags (`photo_(object)`, `selfie`, `photo_background`, …) are
ignored so they never win the folder alone.
|