REFACTOR_PLAN.md

# Search-UI — Aggressive Refactor Plan

This document tracks the autonomous refactor begun on `claude/refactor-aggressive`.
It is the canonical source for what the refactor is doing, the order of work,
and the gates between phases.

## Goal

Cut LOC by ~50%, separate ingestion from serving, replace hand-rolled infra
with proven libraries, modernize the frontend. App must stay functional at
every phase boundary.

## North-star metrics

| Metric | Before | Target |
|---|---|---|
| Backend Python LOC | ~35,850 | ~10,000 |
| Frontend JS/JSX LOC | ~11,820 | ~5,000 |
| `*_routes.py` files | 13 | ≤ 5 |
| SQLite DB files | 5 | 1 |
| Web-process cold-start RSS | 450 MB → **76 MB** (Session 4) | < 500 MB ✓ (cold) |
| Tests collecting cleanly | 85/443 → 443/443 after Phase 0 | 443/443 |
| Tests passing (sandbox, no model network) | 440/443 | 443/443 in user env |

## Phases & status

| Phase | Title | Status |
|---|---|---|
| 0 | Safety net (conftest, golden tests, smoke script) | DONE |
| 1 | Subtraction: dead code, redundant routers, rerankers, dedup | DONE |
| 2 | Single database + merge tool + Alembic | MERGE TOOL VALIDATED on real DBs (Session 3): found+fixed a vec0 INTEGER-PRIMARY-KEY-alias copy bug; merged 5→1 (3.2 GB), all 15 key row counts exact, golden diff identical except one benign tie-swap of two equal-score (0.71191) semantic hits. Flipped to the single DB (Session 3). Alembic SCOPED FOUNDATION added (opt-in); full cutover deliberately deferred — see "Phase 2 Alembic foundation" below. |
| 3a | ML out of web request path: CLI extraction | **Cutover step 1 (lazy torch imports) DONE & runtime-verified (Session 4): cold-start web RSS 450→76 MB, torch absent at boot, models load lazily on first query, results byte-identical (golden + eval). CLI scaffold from Session 3 stands. Steps 3–4 (enqueue ingestion / relocate modules) still open.** |
| 3b | Query-time embedding sidecar (optional but recommended) | **DROPPED for current deployments (Session 4 decision). On the single-container HF Docker Space, web+sidecar share one container's RAM, so a sidecar does NOT lower container memory — its only real benefit is a true multi-machine split (web on HF, model on a GPU box), which isn't the deployment and would add per-query network latency + an always-on paid service. Revisit only if/when web and ML run on separate machines.** |
| 3c | Video lifecycle: prune source MP4s after ingestion | DONE |
| 4 | Route consolidation: 13 → 6 coherent routers | DONE at 6 (Session 3 decision). 13→6 via cloud PRs #34–39 (search←content+analysis, face←speaker, catalog←status, workflow←AD+settings+sync). Stopped at 6 deliberately — the last two merges would chase a counter at the cost of multi-concern files; see "Phase 4 progress" below. |
| 5 | Frontend: TypeScript + React Router + TanStack Query + Tailwind | DEFERRED — needs runtime verification |
| 6 | Face: DeepFace → insightface (ONNX) | DEFERRED — needs runtime verification |
| 7 | Desktop story (drop Electron or → Tauri) | DEFERRED — needs decision |
| 8 | Smart-search reckoning (instrument & decide) | DEFERRED — needs production data |

## Phase 0 — Safety net (1–2 days)

Build the regression detector before changing anything.

**Deliverables**:
- `conftest.py` at repo root that fixes the two-import-conventions test mess
- `backend/__init__.py` so tests can import either flat or namespaced
- `tests/golden/` query fixture set (run against user's real DB to populate)
- `scripts/snapshot_golden.py` — capture top-N results per query
- `scripts/diff_golden.py` — re-run and diff
- `scripts/smoke_test.sh` — boot backend, hit 20 critical endpoints
- All collection errors fixed (target: 131/131 tests collect, even if some skip)
- Tag pre-refactor state for rollback

**Gate**: all tests collect cleanly. Golden snapshot scripts run end-to-end
against a dummy DB. Smoke script returns exit 0.

## Phase 1 — Subtraction (2–3 days)

Pure deletion. No new deps, no architecture change.

**Delete outright**:
- `frontend/src/designMockups/` (13 files, ~485 LOC) — DONE
- `MockupReview.jsx` + `?view=mockups` routing — DONE
- `main.py:/api/hello`, `/api/data` placeholders — DONE
- `llm_router.py` + `llm_client.py` (orphaned, ~896 LOC) — DONE
- `MiniLM` fallback in `ai_features.py` — DONE
- Mixedbread legacy embedding code — DEFERRED to Phase 2
  (still used by search_semantic.py for whole-document index;
  removing it now breaks search against legacy DB rows. Will be
  retired via Alembic backfill migration during DB consolidation.)
- `pywebview` — DEFERRED to Phase 7 (desktop decision)

**Consolidate**:
- 3 rerankers → `search_visual_rerank_rules.py` — DONE (common+event+rules
  merged; bug found and fixed in _best_label_score case-sensitivity)
- 8 face files → defer to Phase 4. On inspection, the small ones
  (face_search_common 40 LOC, face_person_index 87 LOC, face_route_common
  48 LOC) sit at the bottom of the dep tree and can't be merged upward
  without circularity. The 5 mixin files map to real concerns (db, storage,
  people, recognition, review) — collapsing them touches a 2000+ line class
  and isn't worth the regression risk in this phase.
- Duplicate `_get_vod_categories()` → one helper in `media_metadata.py` — TODO

**Policy change**: drop the 600-line hard limit from `CLAUDE.md`. Replace with
guidance: "files should do one thing; never split a coherent concept just to
satisfy a counter."

**Gate**: golden tests pass. Tests still collect. LOC down ≥ 5k.

## Phase 2 — Single DB + Alembic (4–5 days)

**Deliverables**:
- `backend/schema_version.py` — one connection factory, one DB file
- `backend/migrations/` — Alembic with baseline migration
- `scripts/migrate_to_single_db.py` — merge 5 source DBs into 1, verify row counts
- Remove every `CREATE TABLE IF NOT EXISTS` from app boot code
- Replace `_schema_metadata` per-db with single `alembic_version`

**Gate**: migration script runs cleanly on a copy of real DBs.
Golden tests pass against single DB. Row counts match.

**Risk mitigation**: original 5 DBs untouched until 2 weeks of normal use.

## Phase 3 — ML out of web process

### 3a: CLI extraction

**Deliverables**:
- `backend/cli.py` with subcommands: `jws ingest vod`, `jws ingest subtitles`,
  `jws ingest video`, `jws ingest faces`, `jws ingest images`, `jws reindex embeddings`
- All ML imports (`torch`, `deepface`, `transformers`, `whisper`, `transnetv2`)
  moved into `backend/jwsearch/ingest/`
- Web process imports only `sentence-transformers` for query embeddings
  (or none if 3b ships)
- Endpoints that previously kicked off processing now enqueue jobs

### 3b: Query-time embedding sidecar (optional)

**Deliverables**:
- `backend/jwsearch/embed_service.py` — 100-line FastAPI process holding Qwen3
- Main web process makes HTTP calls to it
- Web process imports zero ML libraries

### 3c: Video lifecycle

**Deliverables**:
- `jws prune videos --keep-thumbnails --keep-embeddings` CLI command
- New column `source_deleted_at` on the videos table
- New env flag `SEARCH_UI_KEEP_SOURCE_VIDEOS` (default `false`)
- Ingestion workflow deletes MP4 after extraction if flag is unset
- `content_status` treats videos with `source_deleted_at` set as "complete"

**Rationale**: thumbnails (~50 MB/video) are 10× smaller than source MP4s
(~350 MB/video). JW.org streams playback via `progressiveDownloadURL` already.
Re-extraction only needs re-download (bandwidth, not storage).

**Gate**: web process cold-start under 5s, RSS under 500 MB. Background
ingest produces identical indexed data (golden tests pass).
A pruned video still plays via `ClipPlayer` (poster + streaming URL).

## Phase 4 — Route consolidation (3–5 days)

13 routers → 4:

| New router | Replaces |
|---|---|
| `search.py` | search_routes, content_routes, analysis_routes (scripture) |
| `catalog.py` | catalog_routes, status_routes, publication_routes (read) |
| `people.py` | face_routes, face_route_persons, speaker_routes |
| `jobs.py` | workflow_routes, processing_routes, sync_routes, audio_description_routes, settings_routes |

Plus: `services.py` (centralized service factory), `errors.py` (global handlers),
`schemas.py` (Pydantic DTOs).

**Gate**: golden tests pass. Frontend works without changes (URLs preserved).

## Phases 5–8 — Deferred (need runtime verification or production data)

5: Frontend modernization, 6: Face re-platform, 7: Desktop story, 8: Router decision.
Documented in chat; not started in this autonomous session.

## Session 1 actual outcome (2026-05-26)

Phases 0, 1, 3c shipped. Phase 2 scaffolded (merge tool only).
Phases 3a, 3b, 4, 5, 6, 7, 8 deferred — they need runtime verification,
production data, or are best sequenced after the user flips to the
single DB.

**Net change**: ~5,000 LOC removed. 18 new tests added. Test suite:
455/458 passing (3 pre-existing HuggingFace-network failures unchanged).
Branch: `claude/refactor-aggressive`.

**What the user needs to do next** to continue the refactor:

1. **Run the golden snapshot** against current backend with real data:
   ```
   python scripts/snapshot_golden.py --base-url http://localhost:8001 \
       --output tests/golden/snapshot.json
   git add tests/golden/snapshot.json && git commit
   ```
   Without a baseline snapshot, Phase 4+ can't detect ranking regressions.

2. **Merge the databases.** The merge tool now reconstructs regular
   tables, vec0 (sqlite-vec) embeddings, AND FTS5 full-text indices,
   preserving rowids so embedding→metadata joins survive. Steps:
   ```
   # Dry run first — reports per-source row counts, writes nothing
   python scripts/merge_databases.py --output ~/searchui-merged.db --dry-run
   # Then for real (needs: pip install sqlite-vec)
   python scripts/merge_databases.py --output ~/searchui-merged.db
   ```
   Verify the merged DB serves searches correctly, THEN flip the app by
   pointing all DB env vars at it:
   ```
   export SEARCH_UI_SEARCH_DB_PATH=~/searchui-merged.db
   export SEARCH_UI_IMAGE_DB_PATH=~/searchui-merged.db
   export SEARCH_UI_FACE_DB_PATH=~/searchui-merged.db
   export SEARCH_UI_SPEAKER_DB_PATH=~/searchui-merged.db
   export SEARCH_UI_PUBLICATIONS_DB_PATH=~/searchui-merged.db
   ```
   Run the golden diff (`scripts/diff_golden.py`) against the flipped
   app to confirm no ranking drift. Keep the original 5 DBs untouched
   for two weeks as the rollback path before archiving.

   Still TODO in Phase 2: replace the per-DB `CREATE TABLE IF NOT EXISTS`
   bootstrap + `_schema_metadata` with Alembic migrations so the single
   DB has real schema versioning. The merge tool keeps only the first
   source's `_schema_metadata` row — Alembic's `alembic_version` will
   supersede it.

3. **Try the video prune** in dry-run mode first:
   ```
   python scripts/prune_source_videos.py --dry-run
   ```
   Expect ~1 TB of disk reclaimed across 3,713 videos.

## Session 2 actual outcome (2026-05-28)

Completed the Phase 2 merge tool (vec0 + FTS5 reconstruction with rowid
preservation), then ran full pre-merge QC on the whole `claude/refactor-aggressive`
branch (18 commits, net −2,952 LOC):

- **Safety review** (subagent): zero dangling references — every deleted module
  (`llm_client`, `llm_router`, the two visual rerankers) and removed endpoint
  (`/api/hello`, `/api/data`, `designMockups`) has zero remaining referents.
- **Standards review** (subagent): zero MUST-FIX. Fixed SHOULD-FIX items —
  `SEARCH_UI_KEEP_SOURCE_VIDEOS` was silently ignored on the batch path
  (`process_all_local_videos` hard-coded `delete_video_after=True`); defaulted
  to the `None` sentinel + added a regression test. Removed a phantom
  `--keep-largest` doc line and two dead imports.
- **Security review** (skill): no vulnerabilities. The SQL-building merge tool
  and file-deleting prune script are operator CLIs whose only external inputs
  are trusted env/CLI values and the app's own schema.

Test suite: **460 passed**, 3 pre-existing HuggingFace-network failures. Branch
merged to `main` via PR. Continuation handoff for the remaining phases lives in
`CONTINUATION_PROMPT.md`.

**Next (see CONTINUATION_PROMPT.md):** Phase 4 route consolidation → Phase 2
Alembic → Phase 3a CLI-ingestion scaffold, each as an atomic-commit branch with
subagent QC and a PR for Glenn to merge after an app smoke-test.

## Phase 4 progress (Session 2, branch claude/phase4-route-consolidation)

**Done & verified (13 → 10 route modules):**
- Added `backend/tests/test_app_boot.py` — assembles the real `create_app()`
  (startup checks off via `SEARCH_UI_STARTUP_CHECKS=false`) and asserts the full
  /api surface + no duplicate (path, method) registrations. This is the
  regression guard that makes consolidation verifiable without the live app.
- `search_routes.py` ← absorbed `content_routes.py` + `analysis_routes.py`
  (deduped the byte-identical `_get_default_*_service` helpers; dropped a dead
  `import json`).
- `face_routes.py` ← absorbed `speaker_routes.py` (kept the module-level speaker
  `router` singleton).
- Each step verified by a **byte-identical 128-route manifest** + full suite
  green (462 passed). Old files deleted; importing tests use module aliases.

**Naming note:** `search.py` is the search ENGINE module, so the consolidated
router keeps the `*_routes.py` convention rather than the plan's `search.py`.
Targets are now: `search_routes`, `catalog_routes`, `face_routes`,
`workflow_routes` (4 ≤ 5).

**Final state (Session 3): 13 → 6 routers. Phase 4 closed here.**
Cloud PRs #34–39 carried it past the Session-2 "13→10" note: `catalog_routes`
absorbed `status_routes`, and `workflow_routes` absorbed `audio_description` +
`settings` + `sync`. Current routers (6): `search_routes`, `catalog_routes`
(+status), `face_routes` (+speaker), `workflow_routes` (+AD/settings/sync),
`processing_routes`, `publication_routes`.

**Decision: stop at 6. Do NOT force the last two merges.** A deep-dive review
(Session 3) found both remaining merges trade maintainability for a smaller
count — exactly what CLAUDE.md's file-size guidance warns against:
- **`processing → workflow`** → a ~1,790-LOC file with **five** unrelated
  concerns (orchestration, AD, settings, sync, batch-processing) wired by two
  independent runtimes (`WorkflowRuntime`, `ProcessingRuntime`) that share no
  code. Strictly worse. Rejected.
- **`publication → catalog`** → a ~1,158-LOC, three-concern `catalog_routes`,
  and `test_publication_routes.py`'s `importlib.reload` would pollute sibling
  catalog/status tests. Publications is its own concern (separate JW
  publications API, DB, and image search). Marginal count gain, real risk.
  Rejected.

The `≤5` North-star was a ceiling, not a quota. 6 coherent routers is the
maintainable resting place; the churn budget is better spent on search quality.
If a future session wants 5, the publication→catalog merge is the only
defensible one and must first replace that test's reload with attribute-patching.

## Session 3 — DB consolidation flipped to single DB (2026-05-29)

Validated and **flipped the live app onto the merged single DB**. Steps taken
and what's needed to keep/rollback it:

- **Merged DB:** `/Users/avsadmin/searchui-merged.db` (3.2 GB; built with
  `scripts/merge_databases.py` after fixing the vec0 PK-alias bug). All 15 key
  row counts match the per-source dry-run exactly.
- **Durable launcher:** `scripts/run-backend-merged.sh` sets the five env vars
  (override location via `SEARCH_UI_MERGED_DB`); see CLAUDE.md Development
  Workflow. Merged to `main` 2026-05-29.
- **Flip is LIVE in the running backends** (`:8001` and `:8002`) via these env
  vars on the uvicorn launch (settings.db stays separate — it is NOT merged):
  ```
  SEARCH_UI_SEARCH_DB_PATH=/Users/avsadmin/searchui-merged.db
  SEARCH_UI_IMAGE_DB_PATH=/Users/avsadmin/searchui-merged.db
  SEARCH_UI_FACE_DB_PATH=/Users/avsadmin/searchui-merged.db
  SEARCH_UI_SPEAKER_DB_PATH=/Users/avsadmin/searchui-merged.db
  SEARCH_UI_PUBLICATIONS_DB_PATH=/Users/avsadmin/searchui-merged.db
  ```
- **To make it durable** (survive a manual restart): start the backend with
  `scripts/run-backend-merged.sh` (it sets those env vars for you), per CLAUDE.md's
  Development Workflow. The app reads `os.environ` directly — there is no `.env`
  loader — so starting it the *plain* way (without the launcher) reverts to the
  5 source DBs.
- **Rollback:** unset the five env vars and restart → back to the original 5
  DBs, which are left untouched. **RETENTION: do NOT archive/delete the original
  5 source DBs (`database.db`, `images_database.db`, `faces_database.db`,
  `speakers_database.db`, `publications_database.db`) before ~2026-06-12** (≈2
  weeks of normal single-DB use). They are the rollback path until then.
- **Golden baseline re-captured on the merged DB** (`tests/golden/snapshot.json`).
  Smoke-tested keyword/semantic/hybrid/image-content/title/scripture/
  publication-image on the flipped `:8001` — all return sensible results.
- **Known cosmetic diff vs the 5-DB world:** one semantic query's two
  equal-score (0.71191) hits swap tie-order (vec0 index rebuild). Same set,
  same scores. A deterministic tie-break (ORDER BY distance, natural_key) is a
  good follow-up under search-quality.

## Phase 2 Alembic foundation (Session 3) — SCOPED, opt-in

Added a safe foundation for schema versioning without the risky full cutover
(the Plan agent showed a full cutover is currently unsafe — see "deferred").

**Delivered:**
- `backend/schema_version.py` — `ensure_alembic_version()` idempotently brings a DB under
  Alembic: existing DB → `stamp` baseline; empty file → `upgrade`; already
  versioned → no-op. Plus `get_primary_db_path()`.
- Wired into `app_runtime.create_app_runtime()` **behind `SEARCH_UI_ALEMBIC_MANAGE`
  (OFF by default)** — so merging changes nothing until opted in. When enabled,
  the live merged DB gets stamped at baseline (`alembic_version` row; safe,
  reversible, no data change). `init_db()` still owns the schema; failure is
  logged, not fatal.
- Tests: stamp/upgrade/idempotent detection + flag-gated boot wiring.

**Deferred (do NOT do without a separate, carefully-verified PR):**
- Replacing the no-op baseline with a verbatim live-schema migration, and
  removing the per-subsystem `init_db()` `CREATE TABLE IF NOT EXISTS`. Blockers:
  Alembic autogenerate can't model the 12 vec0/FTS5 virtual tables (baseline
  must be hand-authored raw DDL with sqlite-vec loaded at migrate time), and
  `_schema_metadata` carries embedding model/recipe info `alembic_version` does
  not replace.

**Finding — orphan tables not in source.** `video_concepts`, `video_concepts_fts`,
`video_concept_embeddings` are READ by `search_semantic.py` but have **no
`CREATE` statement in `backend/`** — they exist only because some ingestion path
(outside the web backend) created them in the live DB. A fresh web-only install
would lack them. Their live DDL (captured for the future baseline):
```sql
CREATE TABLE video_concepts (natural_key TEXT NOT NULL, language TEXT NOT NULL,
  summary TEXT NOT NULL, topics_json TEXT NOT NULL, keywords_json TEXT NOT NULL,
  concept_text TEXT NOT NULL, content_hash TEXT, recipe_version INTEGER NOT NULL,
  recipe_payload TEXT NOT NULL, indexed_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
  visual_cues_json TEXT NOT NULL DEFAULT '[]', PRIMARY KEY (natural_key, language));
CREATE VIRTUAL TABLE video_concepts_fts USING fts5(natural_key, language, concept_text);
CREATE VIRTUAL TABLE video_concept_embeddings USING vec0(natural_key TEXT, language TEXT, embedding float[1024]);
```
## Phase 3a CLI scaffold (Session 3) — additive, ML untouched

Added `backend/cli.py`: a stdlib-argparse ingestion entrypoint so ingestion can
run OUTSIDE the web process. **Purely additive** — it calls the existing
processing functions unchanged and does NOT move ML off the web request path
(that flip needs Glenn's cold-start RSS verification; see cutover below).

**Subcommands wired (each calls an existing function; heavy imports are
function-local so `cli.py --help` and the parser load no torch):**
`ingest vod` · `ingest subtitles` · `ingest video [--all]` · `ingest images
--source {publications|web}` · `process subtitles` · `reindex embeddings` ·
`reindex subtitles`. Tested via parser-routing + dispatch (faked modules) + a
subprocess guard asserting `torch`/`process_video` are absent after import.

**Deferred:** `ingest faces` — faces have no standalone ingestion function; they
run inside `process_video`'s thumbnail step. A standalone face re-index is
net-new orchestration (pair it with Phase 6).

**ML import boundary (what the cutover must move):** the web process imports
torch eagerly today via `search.py` (`from sentence_transformers import ...`,
top-level) and `search_images.py` (`import torch` / `from transformers import
...`, top-level), both pulled in at boot through `app_runtime`. DeepFace/TF
(face_search), Whisper (transcription), TransNetV2 (scene detect), CLAP/VLM
(scene_processing) are lazy *with respect to web boot* — the boot path never
imports them, so they stay out of web RSS. (Note: `video_scene_detect` itself
imports `search_images` at module level, so importing it directly still pulls
torch; only the TransNet model load is lazy. Relevant for step 4's relocation.)

**Cutover plan (separate PR, gated on Glenn's runtime check — do NOT do blind):**
1. Make torch import-lazy in `search.py` + `search_images.py` (move the imports
   into `get_embedding_model()` / `get_siglip_model()`; `TYPE_CHECKING` for
   annotations). Web boot then imports no torch.
2. Decide query-time embeddings (the real fork): keyword/title/scripture/
   image-category need NO model; semantic/hybrid + image-content text→embedding
   DO. Either ship the **3b embedding sidecar** (web imports zero ML — only way
   to hold <500 MB steady-state) or keep the model lazy in-process (cold-start
   is lean but the first semantic query loads torch into web RSS).
3. Make heavy-ingestion endpoints **enqueue a job / shell out to `cli.py`**
   instead of running ML in the request worker.
4. (optional) relocate ML modules under `backend/jwsearch/ingest/`.

**Gate (Glenn verifies in his runtime, not a sandbox):** boot `uvicorn main:app`
with the single-DB env vars, no warm queries; `ps -o rss= -p <pid>` after
`/api/health` 200 → target < 512000 KB; assert `torch` absent from the web
process; re-run `scripts/diff_golden.py` (no ranking drift) and confirm
background-ingest output is byte-identical to in-process.

## Session 4 — Tie-break + Phase 3a cold-start cutover (2026-06-01)

Baseline confirmed green first: `main`==`origin/main` (50ef139), 628 tests
passing, merged-DB flip live, all six search families sane, golden diff clean.

**1. Deterministic semantic tie-break (search-quality follow-up from Session 3) —
committed.** sqlite-vec rejects a secondary `ORDER BY` on KNN queries, so
equal-distance hits arrived in index-dependent order (the 0.71191 tie-swap noted
in Session 3). Fixed in the three Python ranking sites (`search_semantic`,
`search_video_concepts`, `search_hybrid`) by breaking score ties on
`natural_key`. Verified on the real merged DB: the two previously-drifting golden
queries now have identical key sets + identical (key,score) multisets — only
equal-score hits reorder, now deterministically. search-eval (n=150):
title/keyword unchanged; hybrid recall@1 24.67→24.00% (1 sample, tie-ambiguity
noise, now *stable* vs rebuild-dependent). The hybrid wobble traces to the
semantic tie-break propagating into hybrid RRF ranks (hybrid reuses
search_semantic's order), not the concept sort. Golden re-baselined deterministic.
Added `test_search_tie_break.py`.

**2. Phase 3a cutover step 1 (lazy torch imports) — committed & runtime-verified.**
Moved torch/transformers/sentence-transformers out of module scope into
function-local imports in `search.py`, `search_images.py`,
`image_siglip_inference.py` (the three boot-path torch importers). **Cold-start
web RSS 450 MB → 76 MB; torch absent at boot** (new subprocess guard
`test_web_boot_ml_free.py`). Models still load lazily on first query (first
semantic → 828 MB, +visual → 1071 MB); results byte-identical (golden: all 12
match; eval unchanged). 632 tests pass.

**Decision (Glenn): do Option A (lazy in-process), NOT the 3b sidecar.** Rationale
is the HF deployment shape: the Space is a *single* Docker container (free tier,
one uvicorn). A sidecar in the same container doesn't lower container RAM (the
model sits in one process either way); it only helps a true multi-machine split,
which isn't the deployment and would add per-query network latency + an always-on
paid service. Steady-state ~1.07 GB fits the 16 GB Space fine. The cold-start win
(faster Space wake; word/title/scripture searches ready instantly without loading
ML) is the real benefit and is now banked. Adjacent known issue, NOT addressed
here: CPU semantic/visual latency on the free Space (inherent to no-GPU; would
need caching / lighter model / precompute — a separate conversation).

**3. Single-video ingestion moved off the web worker — committed & runtime-verified.**
`POST /api/process-video` now shells out to `python -m cli ingest video` (new
`--result-json` gives a clean JSON artifact; the helper inherits env so the
subprocess targets the same DBs). Verified: endpoint returns HTTP 200 with the
identical result dict while the **web worker stays 76 MB / 0 torch** and a
separate subprocess (~1.3 GB) does all the ML. Response contract + delete
semantics unchanged. 640 tests pass.

**Still open in 3a (not started):**
- The **streaming bulk endpoints** (`process-all-videos`, `-v2`, `retry-failed`)
  still run `process_video` inline — they stream live SSE progress, so moving
  them out-of-process needs a progress-bridge design (subprocess → file/queue →
  SSE). Deliberately deferred as a separate, designed piece.
- Composite workflows (`/api/update-content`, `reprocess-existing`,
  `nuclear-rebuild`) — no single CLI subcommand yet.
- `index-publication-images` / `crawl-web-images` → `cli ingest images`.
- Optionally relocate ML modules under `backend/jwsearch/ingest/`.

## Guardrails

1. Golden tests pass at every commit.
2. No phase ships without a rollback path.
3. CLAUDE.md updated as part of the phase that invalidates a rule.
4. Atomic commits. Subagent code review at each phase boundary.
5. No model swaps and dependency cleanup in the same commit.