Feat: living-library cost-skip on re-extract (Phase 2)

POST /upload now accepts merge_into (the cached library CSV). _run_pipeline
seeds it as the job's places_full.csv and runs extract.run(resume=True), so
_warm_start_from_csv skips the LLM on posts already in the library (keyed on
instagram_url). new_urls excludes known posts, so the cost estimate and
progress message reflect only genuinely new ones ("X already in your library,
skipped"). After extraction the result is folded back with _merge_rows
(keep-existing-wins) so known posts keep status/coords and any place the new
export omitted is re-added.

Client sends the cached library on the Extract upload. tests/test_merge.py
gains a warm-start section that patches extract.analyze_batch to prove known
posts never reach the LLM and keep their visited status.

A 600-post re-export where 550 are already mapped now only pays to extract the
50 new ones. Phase 3 (polish) is next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -102,6 +102,16 @@ flags "nothing matched" (likely wrong file → client offers Replace). Client (`
 `{forceReplace}`) sends `merge_into` from `localStorage.savedLibrary` unless restoring. Pinned
 by `tests/test_merge.py` (CI guardrail).
 
+**`POST /upload` living-library cost-skip (Phase 2).** `/upload` also accepts `merge_into`
+(the cached library CSV). `_run_pipeline` writes it into the job dir as the seed `places_full.csv`
+and runs `extract.run(resume=True)`, so `_warm_start_from_csv` skips the **LLM** on posts already
+in the library (keyed on `instagram_url`) — `new_urls` excludes known posts so the cost estimate
+and progress message reflect only new ones. After extraction the result is folded back with
+`_merge_rows(existing, extracted)` (keep-existing-wins) so known posts keep status/coords and any
+library place the new export omitted is re-added. Geocoding re-runs on the merged CSV (cheap via
+the geocode cache; hosted/client-geocode only pins coord-less rows). `tests/test_merge.py` patches
+`extract.analyze_batch` to prove known posts never reach the LLM.
+
 **Downloads** live in the Browse sub-panel (`#download-bar`), revealed when `showSubTabBar` is called.
 Buttons: `#download-btn` (KML), `#download-geojson-btn`, `#download-csv-btn` — onclick set in
 both `showDownload` (post-extraction) and `handleImportFile` (post-import).

README.md

@@ -332,6 +332,7 @@ The UI lets you:
 - **KML round-trip** — city and state are embedded in each Placemark's description so re-importing the KML preserves full location data (not just country)
 - **Remembers your library (this device)** — after a run, the app caches your places in the browser (`localStorage`) and keeps the copy in sync as you mark places visited, edit fields, or pin them. Come back later and a **📦 Restore your last library** card brings everything back — visited status, edits, and pins included — with no re-upload and no re-extraction. It's stored only on your device and never uploaded; clearing your browser data removes it, so the Download buttons remain your permanent backup. Edits are flushed to the cache when you switch away from or close the tab, so a change made right before leaving isn't lost
 - **Living library — imports merge, not replace** — when you already have a library loaded and drop another KML/CSV, it's **merged into** your library keyed on the Instagram link (keep-existing-wins): your visited status and manual edits are preserved and only genuinely new places are added ("N new places added"). If nothing in the file matches your library, you're offered the choice to replace instead. This is the foundation for incrementally growing one map over time
+- **Re-export costs nothing for places you already have** — when you re-run Extract on a fuller export months later, the app sends your existing library along, so the pipeline **skips the AI call on posts already in your map** and only extracts genuinely new ones (your visited status and pins are kept). A 600-post re-export where 550 are already mapped only pays to extract the 50 new ones
 
 ### Web UI providers
 

TODO.md

@@ -40,9 +40,11 @@ The paid competitors (Someday Map, Rezz, ReelsMap, Drawer) are all closed, cloud
     curation, returns `added`/`merged`/`no_overlap`; client sends the cached library,
     shows an "N new places added" banner, and offers Replace when nothing matched.
     Pinned by `tests/test_merge.py` (CI). Unlocks the Shortcut/capture path.
-  - [ ] **Phase 2 — server-side cost-skip for bulk re-export.** Pass the cached library
-    to the Extract pipeline so `_warm_start_from_csv` skips the LLM on posts already in
-    the library (saves $), and report "X already known, extracting Z new".
+  - [x] **Phase 2 — server-side cost-skip for bulk re-export (done).** `/upload` accepts
+    `merge_into`; `_run_pipeline` seeds it as the job CSV and runs `extract.run(resume=True)`
+    so `_warm_start_from_csv` skips the LLM on known posts (`new_urls` excludes them; progress
+    says "X already in your library, skipped"), then `_merge_rows` folds results back so
+    nothing is lost. `tests/test_merge.py` proves known posts never reach `analyze_batch`.
   - [ ] **Phase 3 — polish.** New-to-top sort + "new" card badge; an explicit
     merge-vs-replace toggle on the import card; a dedup/merge report.
 - [ ] **HF Spaces live deploy** — code-complete (see `deploy/hf-spaces/`); remaining step is the manual "create the Space + push" in `deploy/hf-spaces/SETUP.md`, then add the live URL to README.

tests/test_merge.py

@@ -15,8 +15,10 @@ Run: python3 tests/test_merge.py
 
 import csv
 import io
+import json
 import shutil
 import sys
+import tempfile
 from pathlib import Path
 
 ROOT = Path(__file__).parent.parent
@@ -28,6 +30,7 @@ except Exception as exc:
     print(f"SKIP: TestClient unavailable ({exc}). `pip install httpx` to run.")
     sys.exit(0)
 
+from pipeline import extract as extract_mod
 from pipeline import geocode as geocode_mod
 from pipeline.extract import FIELDNAMES
 from web import app as webapp
@@ -147,6 +150,65 @@ finally:
     for j in created:
         shutil.rmtree(webapp.JOBS_DIR / j, ignore_errors=True)
 
+# ── Phase 2: warm-start cost-skip (extract.run skips the LLM on known posts) ───
+# When the job CSV is seeded with the existing library, a re-export must not pay
+# the LLM again for posts already extracted — only genuinely new posts hit it,
+# and the known post's status survives.
+seen_urls: list[str] = []
+
+
+def _recording_analyze_batch(client, posts, model=None, provider="anthropic", ollama_url=""):
+    out = []
+    for p in posts:
+        seen_urls.append(p["url"])
+        row = {k: "UNKNOWN" for k in FIELDNAMES}
+        row.update(name="Koffee Mameya", city="Tokyo", country="Japan", category="Cafe")
+        out.append(row)
+    return out
+
+
+_orig_ab = extract_mod.analyze_batch
+extract_mod.analyze_batch = _recording_analyze_batch
+tmp = Path(tempfile.mkdtemp())
+try:
+    json_p = tmp / "saved_posts.json"
+    csv_p = tmp / "places_full.csv"
+
+    # Export contains a KNOWN post (aaa, already curated) + a NEW post (ccc).
+    json_p.write_text(json.dumps([
+        {"label_values": [{"label": "URL", "value": "https://instagram.com/p/aaa"},
+                          {"label": "Caption", "value": "Ichiran ramen Tokyo"}],
+         "timestamp": 1700000000},
+        {"label_values": [{"label": "URL", "value": "https://instagram.com/p/ccc"},
+                          {"label": "Caption", "value": "Amazing new cafe @koffee_mameya in Tokyo"}],
+         "timestamp": 1700000100},
+    ]), encoding="utf-8")
+
+    # Seed the existing library: aaa already extracted and marked visited.
+    with open(csv_p, "w", newline="", encoding="utf-8") as f:
+        w = csv.DictWriter(f, fieldnames=FIELDNAMES)
+        w.writeheader()
+        w.writerow({**{k: "" for k in FIELDNAMES}, "name": "Ichiran", "city": "Tokyo",
+                    "country": "Japan", "category": "Restaurant", "status": "visited",
+                    "instagram_url": "https://instagram.com/p/aaa"})
+
+    extract_mod.run(str(json_p), str(csv_p), resume=True, provider="anthropic",
+                    api_key="sk-test-unused", batch_size=10)
+
+    check("warm-start: LLM called only for the NEW post (known aaa skipped)",
+          seen_urls == ["https://instagram.com/p/ccc"])
+
+    with open(csv_p) as f:
+        out_rows = {r["instagram_url"]: r for r in csv.DictReader(f)}
+    check("warm-start: known + new both in output",
+          set(out_rows) == {"https://instagram.com/p/aaa", "https://instagram.com/p/ccc"})
+    check("warm-start: known post keeps visited status",
+          out_rows["https://instagram.com/p/aaa"]["status"] == "visited")
+finally:
+    extract_mod.analyze_batch = _orig_ab
+    shutil.rmtree(tmp, ignore_errors=True)
+
+
 print()
 if failures:
     print(f"FAIL — {len(failures)} merge check(s) failed:")

web/app.py

@@ -166,7 +166,7 @@ def _acquire_slot(job_id: str) -> None:
 
 def _run_pipeline(job_id: str, json_bytes: bytes, model: str,
                    provider: str = "anthropic", ollama_url: str = "http://localhost:11434",
-                   api_key: str | None = None) -> None:
+                   api_key: str | None = None, merge_into: str = "") -> None:
     job_dir = JOBS_DIR / job_id
     job_dir.mkdir(exist_ok=True)
     json_path = job_dir / "saved_posts.json"
@@ -178,6 +178,22 @@ def _run_pipeline(job_id: str, json_bytes: bytes, model: str,
     try:
         json_path.write_bytes(json_bytes)
 
+        # ── Living-library merge: seed the job CSV with the caller's existing
+        # library so extract's warm-start skips the LLM on posts already known
+        # (keyed on instagram_url). Results are merged back below so nothing in
+        # the library is lost if a slimmer export omits it.
+        existing_rows: list[dict] = []
+        existing_urls: set[str] = set()
+        if merge_into:
+            try:
+                existing_rows = _parse_import_csv(merge_into.encode())
+            except ValueError:
+                existing_rows = []
+            if existing_rows:
+                _write_csv(job_id, existing_rows)   # seed for warm-start
+                existing_urls = {r.get("instagram_url") for r in existing_rows
+                                 if r.get("instagram_url")}
+
         # ── Extract ────────────────────────────────────────────────────────
         _update(job_id, step="extract", progress=10,
                 message="Reading your saved posts…")
@@ -189,25 +205,35 @@ def _run_pipeline(job_id: str, json_bytes: bytes, model: str,
                              f"({len(posts)} posts, max {MAX_POSTS}). Run the Docker "
                              f"or CLI version locally for libraries this big."))
             return
-        new_urls = {p["url"] for p in posts}
+        # Only posts not already in the library need the LLM.
+        new_urls = {p["url"] for p in posts if p["url"] not in existing_urls}
+        known = len(posts) - len(new_urls)
         est = estimate_cost(posts, new_urls, model, provider=provider)
 
         provider_label = f"Ollama/{model}" if provider == "ollama" else f"Claude/{model}"
         pre_filtered = est.get("prefiltered", 0)
-        llm_posts = est.get("llm_posts", len(posts))
+        llm_posts = est.get("llm_posts", len(new_urls))
         pre_note = f" ({pre_filtered} pre-filtered, no LLM call)" if pre_filtered else ""
+        known_note = f" — {known} already in your library, skipped" if known else ""
         _update(job_id, step="extract", progress=15,
-                message=f"Extracting {llm_posts} posts with {provider_label}{pre_note}…",
+                message=f"Extracting {llm_posts} new posts with {provider_label}{pre_note}{known_note}…",
                 total_posts=len(posts), prefiltered=pre_filtered)
 
         with contextlib.redirect_stdout(io.StringIO()):
             extract_mod.run(str(json_path), str(csv_path), model=model,
-                            provider=provider, ollama_url=ollama_url, resume=False,
-                            batch_size=10, api_key=api_key)
+                            provider=provider, ollama_url=ollama_url,
+                            resume=bool(existing_rows), batch_size=10, api_key=api_key)
 
         with open(csv_path) as f:
             extracted = list(csv.DictReader(f))
 
+        # Merge the fresh extraction back into the existing library
+        # (keep-existing-wins): preserves status/coords on known posts and
+        # re-adds any library places the new export happened to omit.
+        if existing_rows:
+            extracted, _added = _merge_rows(existing_rows, extracted)
+            _write_csv(job_id, extracted)
+
         # ── Geocode ────────────────────────────────────────────────────────
         if CLIENT_GEOCODE:
             # Hosted/shared deploy: skip server-side geocoding (public OSM bulk
@@ -529,6 +555,7 @@ async def upload(
     ollama_model: str = Form(DEFAULT_OLLAMA_MODEL),
     ollama_url: str = Form(None),
     api_key: str = Form(""),
+    merge_into: str = Form(""),
 ):
     provider, active_model = resolve_provider(
         provider, model, ollama_model, ollama_enabled=OLLAMA_ENABLED
@@ -549,7 +576,7 @@ async def upload(
         target=_run_pipeline,
         args=(job_id, contents, active_model, provider,
               ollama_url or OLLAMA_URL,
-              api_key.strip() or None),
+              api_key.strip() or None, merge_into),
         daemon=True,
     )
     thread.start()

web/static/app.js

@@ -803,6 +803,10 @@ runBtn.addEventListener('click', async () => {
   fd.append('model', selectedModel());
   fd.append('ollama_model', selectedOllamaModel());
   fd.append('api_key', ($('anthropic-api-key').value || '').trim());
+  // Living library: send the cached library so the pipeline warm-starts and
+  // skips the LLM on posts already extracted (saves cost on a re-export).
+  const _lib = _readSavedLibrary();
+  if (_lib && _lib.csv) fd.append('merge_into', _lib.csv);
 
   let res;
   try {