DECISIONS.md

DECISIONS.md

Running log of implementation decisions where the spec (PROJECT_SPEC_dog_reunification_V1.md) left room, plus anything discovered that contradicts the spec. Newest entries at the bottom of each section.

Implementer choices (from spec §19)

• Frontend styling: Tailwind CSS. Faster to build responsive/mobile-first UI; utility classes keep styling co-located with components. (spec §5, §19)
• Auth: JWT access tokens (Bearer), password hashing via passlib[bcrypt]. Stateless, simple to consume from the SPA, and matches the OpenAPI/codegen story. Refresh tokens deferred. (spec §11, §19)
• Dog-detector crop: planned, never built. The intent was an optional crop-to-dog pass before embedding. It was not needed once the embedder was fine-tuned on full-body photos as well as face crops, so no detector module exists and Picture.is_probably_not_dog is never set. (spec §8, §19)

Decisions where the spec left room

• Python 3.13 is used (local interpreter); spec asked for 3.11+. No incompatibilities expected.
• Finders without accounts do not create a users row. Their contact lives on the cases row (finder_name/finder_email/finder_phone), per spec §7.1 note. De-dupe into lightweight finder users deferred. (spec §7.1)
• Embeddings stored in their own embeddings table (per spec §7.5/§7.6 refinement), vector as float32 BLOB, L2-normalized at write time so cosine == dot product.
• Single config object via pydantic-settings, loaded from .env. All thresholds/radius levels are named config values with documented defaults (spec §0, §15).

Resolved spec questions

• REVIEW_THRESHOLD / STRONG_THRESHOLD govern the case matcher only. The public photo search returns the top N ranked results with no cutoff, deliberately: a page of low scores tells a searcher their dog is probably not in the database, where an empty page is ambiguous. Retune with scripts/eval_matching.py if the case workflow is put into real use (spec §9.3, §9.6).
• RADIUS_LEVELS default [0, 10, 25, 50, 100, -1] where -1 == nationwide (no filter).
• Nearby-shelter lookup is a bundled static list for MVP (spec §12).

Estimated-breed candidate gate (extends spec §9.3 metadata gate)

• New swap point: BreedClassifier (backend/app/ml/breed.py), structured exactly like Embedder: Protocol + MockBreedClassifier default + lazily-imported HFBreedClassifier + get_breed_classifier()/reset_breed_classifier_cache() singletons. It produces breed labels only, used purely as a candidate pre-filter — never for similarity (that matcher is selected separately).
• Predictions stored in their own versioned breed_predictions table (mirrors embeddings): per-picture, keyed by breed-model name/version, one row per top-K rank. Match time uses only the active breed model's rows; other versions count as "no predictions" → fail-open.
• Fail-open, symmetric top-K gate (open choice #1: chose symmetric for recall). A candidate is excluded only on a definite conflict: both query and candidate have predictions under the active model and their top-K label sets do not intersect. Missing predictions on either side never exclude — same recall-first discipline as _metadata_compatible and geo.within_radius. The gate runs after the geo + color/size gates and before the cosine work, so it prunes the expensive comparison. BREED_FILTER_ENABLED=false disables it entirely (predictions still stored).
• BREED_TOP_K default 7 (open choice #3; anywhere in 5–10), exposed as config to tune vs recall.
• Human-entered breed is deliberately not folded in (open choice #2: no, settled). KnownDog.breed / UnknownDog.est_breed are free text, and owners are often wrong about mixed breeds. A confident-but-wrong human label would exclude the correct dog outright, so the gate stays purely model-driven to keep it recall-first.
• Mock-first / HF opt-in. Default BREED_CLASSIFIER=mock is deterministic and weight-free (reuses the mock embedder's pixel hashing so identical images yield identical breed sets — this keeps the seeded Rex match working). The real HF model (jhoppanne/Dogs-Breed-Image-Classification-V1, ~43 MB) is enabled only via config and lazily imports torch/transformers, exactly like the real embedders. Labels are read from model.config.id2label at load time (no hardcoded breed list).
• Best-effort in the upload pipeline. Breed prediction is wrapped in try/except so an upload never fails if the breed model errors or isn't downloaded; the embedding stays required.
• Migrations: versions/ previously relied on create_all; added a baseline autogenerated migration covering the full schema including breed_predictions. Existing dev DBs predate the table — delete backend/data/app.db and re-seed, or run alembic upgrade head then python -m scripts.process_dataset --all to populate predictions for already-uploaded photos.

Batch loading & test-data management (datasets)

• New datasets table + nullable dataset_id FK on known_dogs, unknown_dogs, users (Alembic 31362988755b, chained on the a18bdb0cac22 baseline; batch-mode ALTERs with named FK constraints since SQLite batch mode requires names). Cases/matches/pictures link transitively through the dogs/users, not via their own dataset_id.
• Purge is application-level and transactional, not DB-cascade. Pictures link to dogs by convention (subject_type/subject_id, no FK) and embeddings/breed_predictions FK pictures with no ON DELETE, so a clean purge deletes children before parents in one transaction (notifications → breed_predictions/embeddings → matches → pictures → cases → unknown_dogs → known_dogs → users → dataset) and returns per-type counts. Everything-or-nothing via commit/rollback. (services/datasets.py)
• Purge also deletes media files. The picture storage keys are collected before the rows are deleted, then the files are removed after the DB commit succeeds (best-effort: a failed unlink only leaves an orphan, never inconsistent data; a rolled-back purge deletes nothing). Count returned as media_files. Earlier builds left media on disk (the storage step was a no-op), which accumulated orphans across purges/re-seeds; scripts/prune_media.py removes any orphans not referenced by a pictures row (dry-run by default) and stays as a safety net for manual DB resets.
• One dog per folder, multiple pictures (default). The CSV folder column is the identity key (each DogFaceNet subfolder == one real dog). The loader groups a folder's rows into a single KnownDog/UnknownDog and attaches every image as a Picture (first = primary); a known dog gets one optional lost case, a found dog one found Case. This supersedes the original "one row == one dog" behavior, which created a separate same-named dog per photo — wrong for multi-photo identities and bad for matching (1-picture dogs). Pass group_by_folder=False / --one-dog-per-image for the legacy per-row behavior. Owners/finders are still de-duped by email and linked to the dataset; found_zip equals the dog's known ZIP so default-radius (level 0 = same ZIP) matching works. Note: datasets loaded before this fix have one dog per image — purge and re-load them.
• Holdout default: 1 image, or 2 when a dog has 5+ images and --holdout is not given; always leaves ≥1 registration image (a single-image dog yields 0 holdouts, logged).
• --mark-lost also opens a lost Case per marked known dog (status lost, last_known_zip set) so the known dataset is actually matchable; --mark-lost-pct controls the share (default 100).
• Embedding reuse: extracted embed_picture() in services/images.py, used by both the upload pipeline and embed-all — embedding generation is never reimplemented.
• Many-to-many match: run_matching_for_case gained an optional candidate_dataset_id to scope the candidate pool; the /admin/datasets/{id}/match?candidate_dataset_id= endpoint passes it through.
• Progress feedback = polling. No realtime channel existed, so dataset loads run in a worker thread tracked by a process-local in-memory job registry (services/jobs.py); the dashboard polls GET /admin/jobs/{id}. Ephemeral by design (single-process local target, A8); a durable queue is a documented future swap.
• Schema change ⇒ re-seed. Existing dev DBs lack dataset_id; create_all does not ALTER, so delete backend/data/app.db and re-seed, or alembic upgrade head.

Storage-only load (defer embeddings)

• process_and_store_picture gained generate_embedding / generate_breed flags (default True, so normal uploads/seed are unchanged). The batch loader's new skip_embeddings option sets both to False — images are validated, normalized (EXIF stripped), stored, and thumbnailed, but no embeddings or breed predictions are generated. Exposed as --skip-embeddings (CLI), skip_embeddings (load-job form), and a "Load images into storage only" checkbox in the dashboard. Matching needs embeddings, so "Run matching after load" is force-disabled when skipping.
• Batch embedding generator: the existing POST /admin/datasets/{id}/embed-all (idempotent — only embeds pictures lacking an embedding under the active model) is now surfaced as an "Embed" button per dataset row on the Data Loading page, next to "Purge" (in addition to the dataset detail page's "Run Embeddings"). This is the intended two-step flow: bulk-load into storage now, generate embeddings later. embed-all runs synchronously; fine for the mock embedder. (If a real CNN embedder on very large datasets makes the request too long, move embed_all onto the background job system used by load — a documented future step.)

Real re-ID embedder: HuggingFace model (EMBEDDER=hf)

• New HFEmbedder (backend/app/ml/embedder.py) behind the existing Embedder swap point — no API/schema changes. It loads an HF image-classification model (EMBEDDER_HF_MODEL, default jhoppanne/Dogs-Breed-Image-Classification-V1) and uses its penultimate, pre-classifier pooled features as the re-ID vector: last hidden state → global-average-pool (CNN feature maps) or mean-over-tokens (transformer sequences) → L2-normalize. Breed labels are NOT used for similarity — that stays the separate BreedClassifier. Embedding dim is inferred at load with one dummy forward (architecture-agnostic).
• torch/transformers lazily imported, exactly like HFBreedClassifier; default EMBEDDER=mock still needs none of it. torch is installed CPU-only.
• Versioning makes the swap safe: embeddings are tagged model_name="hf-embed", model_version=<repo name>. Matching only compares embeddings from the active model, so after switching EMBEDDER=mock→hf you must regenerate vectors (admin embed-all); old mock vectors are simply ignored, not mixed.
• Same model, two roles: this repo can use the model both as the breed gate (BREED_CLASSIFIER=hf, labels) and the re-ID embedder (EMBEDDER=hf, features). They load independent copies for interface cleanliness; sharing one load is a possible future optimization.
• Perf note: CPU embedding of the full DogFaceNet set (~8k images) via embed-all is slow and runs synchronously today — fine to run once, but a background-job version of embed-all is the documented next step for large real-model runs.

Breed filtering (HF softmax labels)

• Same HF model, the classifier head this time. BreedClassifier=hf runs jhoppanne/Dogs-Breed-Image-Classification-V1's softmax and stores the top-K labels in breed_predictions (separate from the re-ID embeddings). BREED_TOP_K=10 so the UI can filter by top-1 through top-10.
• Filter semantics (per-photo, recall-friendly): a dog matches breed B at depth k if any of its photos predicts B within rank < k. breed_k=1 = "top match"; 5/10 = top-K. Displayed predicted_breeds is the dog-level aggregate (distinct labels ordered by best rank then score across the dog's photos), so a dog matched on one photo's top-1 may show a different label first — expected for multi-photo dogs.
• Active breed model is auto-detected from the DB (active_breed_model_in_db, prefers non-mock, most-used) — reading predictions never loads the HF model. estimated_breeds_for was refactored to use this too, so match-candidate hydration no longer risks loading the breed model.
• Endpoints: GET /admin/breeds (distinct labels + counts) and GET /admin/dogs?breed=&breed_k= (filter). Profiles now include predicted_breeds.
• Tests pin BREED_CLASSIFIER=mock (conftest) so the suite never loads the HF model; the .env default is hf for real use. Generate predictions with python -m scripts.process_dataset --all (--limit N for a partial run).

Public "search by photo" (home page)

• What: anonymous visitors can upload a dog photo (+ optional ZIP) on the home page and get the found/unknown dogs already in the system ranked by photo similarity. POST /search/by-photo (multipart file, optional zip, top_k). No auth, no DB writes.
• Read-only by design: it is a lookup, not a report. It never creates a KnownDog/Case, so it doesn't pollute the matching pipeline or notify anyone. The copy steers users who find a likely match to the real "report my dog as lost" flow (where confirmed, mediated matching happens).
• Transient embedding: images.embed_bytes() runs the same validate→normalize→encode pipeline as stored uploads, writes to a tempfile, embeds via the active model, deletes the temp file. The query photo is never persisted — same vector a stored picture would get, zero footprint.
• Scoring reuses the matcher: matching.search_unknowns_by_vectors() scores the active found/unknown pool (pending|lost|at_shelter) by the same dog-level max cosine aggregate the case matcher uses (_dog_level_score). No REVIEW_THRESHOLD cut here — this is a "show me the closest" ranked list, so we return the top-K regardless of absolute score (recall-first; the model eval showed scores run uniformly high, so a threshold would mostly mislead).
• ZIP is optional + fail-open: with a ZIP we scope to DEFAULT_SEARCH_RADIUS=100 mi (fail-open on unknown ZIPs, like the case matcher) and surface per-result distance_miles; empty ZIP = nationwide. Results are always ranked by score, not distance.
• Results carry their own photos so the UI viewer (PhotoTools modal) shows every angle without a separate auth'd detail endpoint (the admin /admin/dog/... route stays admin-only).
• Tests: backend/tests/test_search.py (ranking, no-auth, persists-nothing, ZIP distance, bad image) and frontend/src/components/__tests__/PhotoTools.test.tsx (upload→render, photo viewer).

Public breed-estimation toy (home page)

• What: a for-fun home-page widget — upload a photo, get the top N (3/5/7/10) breed labels. POST /search/breed (multipart file, top_n). No auth, no DB writes.
• Reuses the breed swap point, not a new model: images.predict_breeds_bytes() mirrors embed_bytes (same validate→normalize→encode→tempfile→cleanup) and calls the active BreedClassifier (get_breed_classifier). With BREED_CLASSIFIER=hf it's the real softmax; with mock (tests/offline) it's deterministic pseudo-labels. The endpoint slices to top_n, so the practical max is bounded by BREED_TOP_K (10 in .env).
• Distinct from the match-time breed gate: this is display/entertainment only and never touches matching; it just exposes the classifier the gate already uses.
• Note: the home page presents this together with photo-search in a single PhotoTools widget — one upload, a large centered preview, and buttons to run matching, breed estimation, or both.
• Tests: test_search.py (top_n length, deterministic labels, clamp to 10, bad image) + frontend/src/components/__tests__/PhotoTools.test.tsx (upload→ranked list, run-both).

ZIP filtering fix (geo centroids were missing)

• Bug: photo-search ZIP filtering (and case-match radius) silently did nothing. Two causes: (1) the app default zip_centroid_file was CWD-relative (./data/...), so launching uvicorn from backend/ looked in backend/data/ and loaded 0 centroids; (2) the shipped centroid CSV was a 31-row national demo set, while the DogFaceNet demo data uses ~480 real Texas ZIPs — 0 of which had centroids. within_radius fails open on unknown ZIPs, so every candidate passed → ZIP no-op, distance always null.
• Fix 1 (path): config.zip_centroid_file now defaults to an absolute path resolved from the repo root (parents[2]/data/zip_centroids.csv), so it's found regardless of CWD. .env / ZIP_CENTROID_FILE still overrides.
• Fix 2 (coverage): a one-off generator produced data/zip_centroids.csv (539 rows) covering the national demo ZIPs (kept verbatim so matching tests' distances are unchanged) plus every Texas ZIP in prepare_dogfacenet.ZIP_CODES, mapped to a metro-level centroid by ZIP3 prefix. Inter-metro distances are realistic (Houston↔Dallas ≈ 225 mi) so radius filtering works; intra-metro distance ≈ 0. A real per-ZIP gazetteer is a drop-in CSV replacement later.
• Verified: 1395/1395 demo dogs now have centroids; a Houston search at 100 mi keeps 382/1395 (drops other metros). Test: test_search_by_photo_zip_excludes_out_of_radius.

One model pass per stored image (embedding + breed together)

• Problem: every stored picture forwarded through the model TWICE — once in HFEmbedder.embed (penultimate features) and once in HFBreedClassifier.predict (logits) — even though both use the same model (jhoppanne/Dogs-Breed-Image-Classification-V1). Batch workflows were worse: running embed_dataset then backfill_breeds = two full passes over the dataset.
• Fix: HFEmbedder.embed_and_breed(paths, top_k) does ONE forward with output_hidden_states=True and returns both the pooled re-ID vector (from hidden_states[-1]) and the softmax breed labels (from logits). Verified the vector is identical to the old standalone embed() — existing embeddings stay valid, no regeneration needed.
• Orchestration: images.embed_and_breed_picture(db, pic) uses the single pass when _same_hf_model() (embedder=hf, breed=hf, same repo id); otherwise falls back to the two separate models (mock defaults / mismatched ids), embedding required + breed best-effort. Breed rows are tagged via hf_breed_name_version() so they match HFBreedClassifier's tags exactly.
• Wired in: process_and_store_picture (live uploads) calls it when both are requested; datasets.embed_all (admin "Run Embeddings + Breeds" button) now reports embedded + breeds; and scripts/process_dataset.py is the batch path that replaces embed_dataset + backfill_breeds (one pass instead of two). embed_dataset remains for mock / split configs; backfill_breeds has since been removed.
• Scope: applies to pictures stored in the system. The public front-page toys (embed_bytes, predict_breeds_bytes) are intentionally separate one-off queries and out of scope.
• Tests: test_embed_breed_single_pass.py — a fake same-HF embedder proves the model runs exactly once and both rows are written with the right tags; plus the mock-fallback idempotency.

Purge fix: datasets that share users (reused test emails)

• Bug: purging "DogFaceNet known" failed with FOREIGN KEY constraint failed on DELETE FROM users and rolled the whole transaction back (dataset left fully intact). Root cause: the prepare scripts mint owner emails owner1@test.com…, and batch_loader._get_or_create_user dedupes users by email — so a second dataset (MPDD) whose CSV reused those emails attached its 96 dogs to the FIRST dataset's users. Purging the first dataset then tried to delete users still owning the second dataset's dogs.
• Fix (purge_dataset): a user may be legitimately shared across datasets, so instead of force- deleting all of the dataset's users, only delete users with no surviving references (KnownDog.owner_id / Case.person_id after the dataset's own rows are gone) and detach the rest (dataset_id = NULL) so the dataset row can be removed without a dangling FK. New users_detached count is returned. Also extended match cleanup to remove matches whose candidate is one of the purged dataset's dogs (not just by case), so a cross-dataset match can't be left pointing at a deleted dog.
• Verified live: purged DogFaceNet known (1393 dogs, 5969 pictures + media, 11938 embeddings, 59690 breed preds; 1297 users deleted, 96 detached). MPDD (#3/#4) fully intact, all owners valid, 0 media orphans. "DogFaceNet unknown" purge was checked and had cleaned up correctly (no DB or media orphans).
• Latent data-hygiene note: reused test emails still ENTANGLE datasets (one user owning dogs in several). The purge now tolerates it, but future prep runs (YT-BB-Dog, Sibetan) will keep sharing owner1@test.com… unless the prepare scripts namespace emails per dataset. Recommended follow-up.
• Test: test_purge_retains_user_shared_with_another_dataset (shared owner across two datasets).

Removed color/size as entered information (subjective/unreliable)

• Why: color and size are subjective, inconsistently reported, and were pruning candidates in the matcher — a wrong color/size guess could hide a correct match.
• Matching: deleted the _metadata_compatible color/size gate and the apply_metadata_filter path in run_matching_for_case. Matching now gates only on ZIP radius + estimated breed (model softmax), then ranks by embedding similarity. (Breed filtering was explicitly wanted; kept.)
• Inputs removed: KnownDogCreate/KnownDogUpdate, LostCaseCreate.new_dog_color/size, FoundSightedCreate, and the /cases/found + /cases/sighted form params no longer accept color/size. Frontend forms (MyDogs add-dog, ReportLost new-dog, ReportFound/Sighted) dropped the fields, and the now-always-empty color/size display was removed from the owner views (MyDogs card, DogDetail).
• Deliberately kept: the color/size DB columns (nullable) and the *Out response schemas — no destructive migration, and any already-loaded/legacy data still round-trips. Batch-loader CSVs and admin dataset views still carry their (often "unknown") values; these are inert now that matching ignores them. FastAPI/Pydantic ignore the extra fields, so old clients/CSVs don't break.
• Verified: 71 backend + 14 frontend tests pass, clean typecheck/build.

Per-dataset + combined re-ID metrics (scripts/eval_datasets.py)

• New read-only eval over the loaded (known+unknown) dataset pairs, auto-detected by name (" known" + " unknown"). Reuses DB embeddings (active model).
• Intra-subject similarity: per dog, cosine over all pairs of its OWN photos (min/max/avg) → eval_out/intra_subject_similarity.csv, plus a per-dataset summary and a sampled different-dog baseline for context.
• Retrieval quality: found dog → ranked known gallery with the production dog-level max-cosine score; recall@1/5/10 (+counts), mean/median rank, MRR, true-vs-best-wrong separation. Reported per dataset AND combined (shared gallery; the other dataset's dogs become distractors). Identity is namespaced per source (base:folder) so colliding folder numbers across datasets never cross-match → this is how "filter matching by dataset" is realized for the eval. Summary → eval_out/retrieval_summary.csv.
• Measures the MODEL (pure embedding retrieval, full gallery). Deliberately does NOT apply the production ZIP-radius / breed gates: on this synthetic data each dog's found+known share a ZIP, so a ZIP gate would inflate recall as an artifact rather than reflect the model.

Photo-level retrieval / CMC (scripts/eval_photo_retrieval.py)

• Different protocol from eval_datasets.py (which is dog-level: whole found dog vs whole known dogs). Here every INDIVIDUAL photo (on the recombined pre-split whole dogs) is a query matched against ALL other photos, its own excluded; a hit @k means a same-dog photo is in the top-k. Reported per dataset then combined (shared photo gallery; other dataset = distractors). Recall@1/5/10/20 + mean/ median rank + MRR. Identity namespaced per source so folder-number collisions don't cross-match.
• _cmc computes each query's nearest-same-dog rank as 1 + #{sim > s_true} with s_true and the count taken from the SAME similarity row (one matmul) — a first attempt used two matmuls and a float mismatch nudged true matches off rank 1 (recall@1 low by ~5pts). Verified elementwise identical to a brute-force argsort on MPDD.
• Results (hf-embed): MPDD r@1/5/10/20 = 65.1/81.0/87.3/91.8%; DogFaceNet 62.9/79.8/85.6/90.1%; Combined 62.7/79.5/85.3/89.8%. Lower than the dog-level eval (single-photo query is weaker than aggregating a whole dog + larger per-photo gallery). Summary -> eval_out/photo_retrieval_cmc.csv.

Outlier photo removal + re-eval (scripts/remove_outliers.py)

• Quarantined the 22 flagged removable_outlier photos (10 MPDD + 12 DogFaceNet) — each the single isolated photo whose removal lifts its dog's MIN pairwise similarity back above 0.5. Files moved to data/outlier_photos/ (reversible); Picture+Embedding+BreedPrediction rows deleted. Every affected dog kept >=2 photos. Verified 0 DB-keys-missing-on-disk afterward.
• Re-ran both eval scripts on the cleaned data (counts −22 exactly):
  • Consistency: dogs with min<0.5 fell MPDD 15->5, DFN 13->1 (remaining low-min dogs have >1 bad photo, so no single removable outlier left).
  • Photo-level CMC improved: MPDD r@1/5/10/20 65.1/81.0/87.3/91.8 -> 66.7/82.6/88.9/93.2; DFN 62.9/79.8/85.6/90.1 -> 63.0/79.9/85.8/90.2; Combined 62.7/79.5/85.3/89.8 -> 62.9/79.6/85.5/90.0. MPDD moved most (small set); DFN barely (12/8363).
• NOTE (pre-existing, unrelated): ~5240 orphan media files on disk with no DB row (from the earlier DogFaceNet reload). Invisible to the DB-based evals; cleanup via scripts/prune_media.py when wanted.

Fix: load progress bar freezes on large batch loads

Two independent causes, both hit only on LONG loads (many polls over many minutes):

• Frontend (root cause): the poll's catch { clearInterval(t) } permanently stopped polling on a SINGLE failed request, so the bar froze at its last value while the load kept running server-side. Now tolerates a run of transient failures (~10s / 20 polls) before giving up, and surfaces a "Lost connection" error only if the job is truly unreachable (failsRef counter).
• Backend (why polls failed): SQLite had no busy_timeout and used the default rollback journal, so the status poll's admin-auth SELECT could transiently hit "database is locked" during the load's long write transaction. Added PRAGMA journal_mode=WAL (readers don't block on the writer) + PRAGMA busy_timeout=10000. Bonus: the test suite got ~35% faster under WAL.
• Tests: new "keeps polling through a transient poll failure" AdminData test; all backend/frontend green.
• Not changed (documented follow-up): the whole load is still ONE transaction (commit at the end), so the SQLAlchemy session grows for very large loads. Periodic commit + expunge would bound memory and smooth progress further, at the cost of all-or-nothing semantics.

Batch encoding progress bar (background embed job)

• The admin "Run Embeddings + Breeds" button was a synchronous blocking POST — no progress, and on a large dataset it just spun / could time out. Converted encoding to a background JOB reusing the same registry + polling the loader uses.
• Backend: datasets.embed_all(db, dataset, *, progress=None, commit_every=25) now commits + reports {processed,total,dataset_id} every 25 pictures (and expunge_all to bound the session) instead of one commit at the end — safe because encoding is additive/idempotent. New endpoint POST /admin/datasets/{id}/embed-all-job runs it in a worker thread and returns a job id; the old synchronous embed-all endpoint stays for small sets / tests.
• Frontend: shared useJobPolling hook (resilient polling, same transient-failure tolerance as the load bar) + JobProgressBar component. DatasetDetail's encode button now starts the job, shows a live bar, and prints the summary + refreshes stats on completion.
• Tests: test_embed_all_reports_progress (monotonic progress events), test_embed_all_job_runs_in_ background (start->poll->done), and a DatasetDetail UI test (bar -> summary). All suites green.

Palette: warm tan/brown theme (replaced generic blue/white)

• The app consistently uses brand-* and gray-* Tailwind tokens, so retheming is a two-scale change in tailwind.config.js — no component rewrites:
  • brand → warm tan→coffee-brown scale (was blue). Recolors buttons, links, active nav, focus rings, progress bars, and breed chips.
  • gray → warmed taupe/sand neutrals (overrides Tailwind's cool default gray). Warms every surface, border, and text token the app already uses (page bg gray-50 = warm cream, cards stay white for a paper-on-desk feel).
  • New accent = a muted dusty-denim blue (deliberately not the default bright blue); the 4 generic blue-100/blue-700 "found/unknown/open" badges now use it as a cool counterpoint.
• Contrast checked on the key pairs (brand-600/white ~6.6:1, brand-700 on brand-50, gray-900 on gray-50, accent-700 on accent-100) — all comfortably readable. tsc/tests/build green.

Date plausibility gate in matching

• A dog can't be FOUND before it was LOST, so a lost dog is only compared to found dogs whose found date is on/after lost_date - grace. Added _date_compatible(lost_date, found_date) in matching.py, applied (pre-cosine, alongside the ZIP + breed gates) in BOTH directions of run_matching_for_case: lost-query→found-pool and found-query→lost-pool. Uses each case's event_date (the user-entered lost/found date, not the report timestamp).
• Recall-first / fail-open: missing dates never exclude; only a definite conflict (found earlier than lost minus grace) prunes. Config: DATE_FILTER_ENABLED=true, MATCH_DATE_GRACE_DAYS=2 (grace absorbs imprecise dates). The admin test-match (pure re-ID check) intentionally skips this gate.
• Batch-loaded datasets set both cases' event_date to the load date, so their found/lost dates are equal → always compatible (dataset matching/eval unaffected).
• Tests: found-before-lost excluded despite identical photo; found within grace still matches.

Narrow-first case matching + widen the constraints

Directive: searches start narrow (location + top-10 breeds + top-10 matches) then widen. Scope chosen: lost/found case matching, MANUAL widen button (keep it). What was already true vs added:

• Location: already narrow-first (starts at radius_levels[0]=0) with the manual /widen ladder. ✓
• Top-10 breed gate: already applied (BREED_TOP_K=10, breed_filter_enabled). ✓
• Top-10 matches: TOP_N was 5 → now 10 (the narrow-pass result cap).
• "Then widen the constraints": the widen ladder previously loosened only location. Now the WIDEST rung (nationwide, radius -1) also drops the breed gate (nationwide = radius == -1; the breed gate is skipped there). So the manual widen progresses location → … → breed, with no schema change (state derived from the existing search_radius_miles). Date/ZIP/threshold gates unchanged.
• Tests: narrow pass caps at 10 matches; breed gate excludes a disjoint-breed dog at radius 0 but matches it once widened to nationwide.

Privacy: shelter/vet location is public (only homes stay private)

• Reversed the earlier rule that hid current_location_detail (the shelter/vet holding a found dog). A shelter/vet is a public place, and telling a matched owner where to reclaim their dog is the point — so it's now included in the hydrated match candidate (hydrate.py) and UnknownDogOut. Home locations remain protected: never collected below ZIP granularity, contact stays mediated.
• Frontend: MatchCard shows "📍 Being held at <shelter/vet>"; the found-report field is relabeled (public, "use the shelter/vet name — not a home address").
• Test flipped: test_unknown_dog_shelter_shown_but_location_stays_zip_level asserts the shelter name IS returned while location stays ZIP-level. Updated model/schema/spec-§14 comments.

Removed the "sighted" workflow (found = you have the dog)

• A found report now always means the reporter HAS the dog: in their own custody (pending) or dropped at a shelter/vet (at_shelter). Removed the seen-but-not-caught "sighted" flow entirely.
• Backend: deleted POST /cases/sighted and the shared _create_found_sighted helper (folded into a single create_found_case); removed CaseType.sighted and the now-unused UnknownDogStatus.lost ("sighted, not in custody"); dropped lost from the found/unknown matching pool filter (both the case matcher and the public photo search); renamed FoundSightedCreate→FoundCreate; updated seed.py's far-away demo dog from sighted→found.
• Data migration (raw SQL, enums already removed): 1 sighted case → found, 1 lost unknown → pending.
• Frontend: removed the /report/sighted route + home button; ReportFound is found-only (no kind prop); api.createReport(kind)→api.createFound; CaseType and MatchCard copy updated.
• All backend/frontend tests green; typecheck + build clean.

Removed seeded demo data (generated-dot "dogs")

• The synthetic make_sample_images dogs (Rex, Maple, 2 demo found dogs) and their info were noise now that real datasets are loaded. New scripts/purge_seed_data.py deletes every non-dataset dog (dataset_id IS NULL) + its pictures/embeddings/breeds/cases/matches/notifications/media, plus the demo owner users — but ONLY non-admin users owning no dataset dog, so the admin and the 96 detached (dataset_id-NULL) MPDD owners are preserved.
• Ran it: removed 2 known + 2 unknown dogs, 2 owners (alice/bob), 3 cases, 1 match, 4 pictures + 4 media files. Verified 0 non-dataset dogs remain, admin intact, datasets intact (1489 known + 1489 unknown), media 8862 = 8862 on disk (0 orphans).
• scripts/seed.py slimmed to create ONLY the admin account (no synthetic dogs), so re-seeding can't reintroduce them; README updated. make_sample_images.py stays (the test suite uses it).
• All backend tests green.

Generated realistic cases for every loaded dog (one-off script, since removed)

• One LOST case per known dog + one FOUND case per unknown dog, rebuilt cleanly (old cases/matches purged first). Pairing is by dataset + folder identity ("DogN" ↔ "Test found dog N").
• Constraints honored: lost dates within the last 2 days; each found date is on/after its paired lost date and never in the future; each found ZIP equals the paired lost ZIP or a neighboring ZIP (another ZIP sharing the metro centroid — 40% of found reports); known-dog ZIPs (which were null) taken from the paired found dog's ZIP.
• Each known dog got a fresh owner owner{i} (owner0…owner1488) as its owner + the lost case's reporter. The ~2825 old shared/entangled test owners (now orphaned) were removed; admin kept.
• Verified all 1489 pairs: dates in range, found≥lost, all found ZIPs same-metro as their lost ZIP, every known dog lost with an owner. Cases are NOT auto-matched (run matching separately if wanted).

Admin dashboard: all cases, owners, and run-matching-on-demand

• All Cases (/admin/cases, page + GET /admin/cases): every case, newest first, hydrated with its dog (thumb/name/kind), reporter (owner for lost, finder for found), ZIP, date, status, and match count. Filter by type (all/lost/found), paginated. Replaced the old thin list[CaseOut].
• Owners (/admin/owners, page + GET /admin/owners): each owner with the dogs they own (thumb/name/status), paginated.
• Run matching on demand (POST /admin/cases/{id}/run-match): runs the PRODUCTION matcher (run_matching_for_case — narrow-first ZIP/breed/date gates, persists ranked Match rows) for a chosen case and returns the hydrated matches. Surfaced as a "Run matching" button per row on the All Cases page, opening a modal of the ranked candidates (MatchCard). Distinct from Test Matching (/admin/test-match), which is the non-persisting pure-re-ID diagnostic.
• Admin dashboard links updated (All Cases / All Dogs / Owners / Test Matching / Data Loading).
• Tests: test_admin_all_cases_and_owners, test_admin_run_case_match. All suites green (79 backend, 16 frontend); verified live on the 2978-case dataset.

Fix: run-matching 500 (matching must not load the model) + case detail view

• Bug: POST /admin/cases/{id}/run-match 500'd. Root cause: run_matching_for_case called get_embedder() / get_breed_classifier() just to read the active model's name/version — which loads the HF model over the network. On a server without a HF token that gets rate-limited/offline, the load fails → 500. Matching only needs the PRECOMPUTED embeddings, never the model.
• Fix: added active_embed_model_in_db(db) (mirrors active_breed_model_in_db) and switched run_matching_for_case + rank_dog_against_opposite to pick the model from stored rows — no model load, works fully offline. Removed the now-dead _active_breed_model(). The public photo search keeps get_embedder() (it must embed the uploaded photo anyway). Verified run-match returns 200 with the hf-embed vectors under EMBEDDER=mock HF_HUB_OFFLINE=1 (zero network calls).
• Case detail: new GET /admin/cases/{id} returns the case + its subject dog (all photos) + its persisted matches (each candidate hydrated with photos). All-Cases rows are now clickable → CaseDetailModal showing the dog's photos and the ranked matches, with a Run/Re-run matching button inside. Tests: test_admin_case_detail; all suites green (80 backend, 16 frontend).

All cases moved to Houston ZIPs

• The case generator drew ZIPs from HOUSTON_ZIPS (the 77002–77099 subset of the prepare list) instead of the spread-across-Texas dataset ZIPs. Conventions kept: one deterministic ZIP per dog (per-folder seeded), found ZIP = same or a neighboring (other Houston) ZIP (~40%), owner ZIP = the dog's ZIP, lost dates within the last 2 days, found ≥ lost.
• Owner creation is now idempotent (upsert owner{i}@example.com) so re-running doesn't hit the email UNIQUE constraint.
• Re-ran: 1489 lost + 1489 found cases, all ZIPs Houston (95 distinct), 0 constraint violations.
• Consequence: all Houston ZIPs share one centroid, so at radius 0 every dog is in range of every other — ZIP no longer narrows the pool; matching leans on photos (+ breed/date gates). Matches were cleared by the rebuild; re-run matching (per case in the admin All Cases view, or in bulk) if wanted.

Side-by-side photo comparison (case dog vs each match)

• MatchCard now takes optional queryPhotos + queryLabel. When present, clicking a match's thumbnail opens a TWO-COLUMN modal: the case's own dog photos on the left, the candidate match's photos on the right (with the % match). Without those props it falls back to the single-dog gallery.
• Owner view (CaseDetail): new owner/admin-authorized GET /cases/{id}/dog returns the case's dog (kind + all photos); CaseDetail fetches it and passes it to each MatchCard.
• Admin view (AdminCases → CaseDetailModal): passes the case's subject-dog photos (already in the detail payload) to each MatchCard. So comparison works in both regular and admin case viewing.
• Tests: test_case_dog_endpoint_returns_photos_and_guards_owner (photos + 403 for non-owner). All suites green (81 backend, 16 frontend).

Removed human-entered breed; display the model's estimated breed (top 3)

• Owners/finders are unreliable on breed, so it's no longer collected. Removed the breed input from KnownDogCreate/KnownDogUpdate, LostCaseCreate.new_dog_breed, FoundCreate.est_breed, the /cases/found est_breed form param, and the frontend forms (MyDogs add-dog, ReportLost new-dog).
• Everywhere breed was shown, it now shows the top-3 estimated breeds (model's HF-softmax predictions, already surfaced as estimated_breeds / predicted_breeds): MatchCard (card + modal
  • comparison), DogPhotosModal, PhotoSearch results, AdminDogs cards, AdminTestMatch candidates, DogDetail (removed the human "Breed" field), and the ReportLost dog picker. Shared estimatedBreeds() helper (top-3, "Breed not estimated" fallback).
• DB columns (KnownDog.breed, UnknownDog.est_breed) kept nullable for backward compat (same as color/size); they're just no longer collected or displayed. Matching never used human breed anyway (the gate uses the model's breed_predictions).
• All suites green (81 backend, 16 frontend), clean build.

Found-report 500 fix + owner re-run + admin delete buttons

• Found-report 500 (root cause): each found report ran rematch_open_lost_cases_against, which re-ran FULL matching for EVERY open lost case (1490 now) — O(cases×candidates), holding the SQLite write lock for minutes → timeout/500 and locking out other requests (the Houston co-location made every dog a radius-0 candidate, worsening it). The found case's OWN matching already surfaces + notifies matching lost dogs, so the auto-rematch is redundant. Made it bounded + geo-filtered and OFF by default (REMATCH_MAX_CASES=0); raise it to opt in for small sets. Also fixed a latent FK bug: run_matching_for_case deleted prior pending matches without detaching notifications that referenced them (nulls Notification.match_id first now) — this surfaced once re-matching a case that had produced a strong-match notification.
• Owner re-run matching: POST /cases/{id}/rematch (owner/admin, re-runs at current radius) + a "Re-run matching" button on the owner CaseDetail (next to Widen). Two tests updated to re-run explicitly now that auto-rematch is off by default.
• Admin delete (temporary, for re-demoing): DELETE /admin/dogs/{kind}/{id} (dog + pictures + media + embeddings + breeds + its cases + matches referencing it) and DELETE /admin/cases/{id} (case + matches, dog kept). Delete buttons on AdminDogs cards and AdminCases rows.
• Tests: owner rematch (+403), admin delete dog/case (+404/400). All green (84 backend, 16 frontend).

Admin dashboard as a button hub + admin nav + delete people

• Admin home (Admin.tsx) is now just a grid of bigger link cards (Pending Matches, All Cases, All Dogs, Owners, Test Matching, Data Loading). The pending-matches review view moved to its own page AdminPending.tsx at /admin/pending.
• Nav: when an admin is logged in, "My Dogs"/"My Cases" are hidden (only the Admin link + Log out).
• Delete people: DELETE /admin/owners/{id} (+ delete_person service) removes a person and their dogs/pictures/media/embeddings/breeds/cases/matches; refuses to delete an admin. Delete button per owner on the Owners page (dogs + cases already had delete buttons).
• Test: test_admin_delete_person (deletes owner+dogs, 404 on re-delete, 400 on admin). All green.

1-decimal match % + tie-break by most-recent report

• Match percentages now show one decimal (e.g. 94.4%) so close scores stop collapsing onto the same integer: ConfidenceBar, MatchCard header, and the PhotoSearch result badges.
• Matching ranking (run_matching_for_case): primary sort is the similarity ROUNDED to one decimal (matches the display); candidates that tie at that precision are ordered most-recently-reported first, using the candidate's case created_at (fetched alongside the date-gate query; ties with no report time sort last). CandidateScore gained a reported_at field.
• Tests: test_equal_score_matches_break_ties_by_most_recent_report; two frontend tests updated for the decimal. All green (87 backend, 16 frontend).