Hardening: multi-user safety, rename to places, geocoder cache, robust chatbot

Tier 1 (honesty + safety):
- Rename restaurants → places end-to-end: output files (places_full.csv,
places_map.kml, places_map.geojson, places_skipped.csv, places_override.csv),
HTTP download filenames, page title, headings, SSE messages, gitignore.
Internal LLM prompt key left as-is (benchmark oracle depends on it).
- Fix roulette cross-user data leak: /roulette no longer falls back to the
globally-newest CSV across all jobs (which leaked one user's places to
another on a shared deploy). job_id is now required; client already sends it.

Tier 2 (hosting readiness):
- Job TTL + persisted state: each job's state mirrors to <job_dir>/state.json
so progress survives a restart; job dirs older than JOB_TTL_HOURS (default 24)
are deleted on startup and on each new upload.
- Geocoder: persistent place→coords cache (.geocode_cache.json) so duplicates
and re-runs skip the network; NOMINATIM_URL env to point at a self-hosted
instance; HOSTED=1 refuses bulk geocoding against public OSM (usage policy).
- Robust chatbot handoff: auto-chunk large exports into per-file downloads,
accept multiple pasted replies and merge/dedup them, and parse tolerantly
(markdown fences, prose, trailing commas, truncated arrays salvaged).

TODO: added Hardening (today), Backlog (in-app map preview — revisit end of
day), and Tier 3 Growth (mobile consumption docs, offline targets, persistent
library, HTML map export, remix loop) above the Beli V2 section.

Note: pre-existing prefilter false-negative (unrelated, prefilter.py untouched).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.dockerignore

@@ -8,4 +8,5 @@ data/
 tests/fixtures/benchmark_results.json
 *.checkpoint.json
 *.batch_pending.json
+.geocode_cache.json
 .DS_Store

.gitignore

@@ -7,16 +7,24 @@ data/
 # Web UI job artifacts
 jobs/
 
-# Legacy paths (in case any exist at root)
+# Generated outputs (in case any exist at root)
 saved_posts.json
 saved_collections.json
 saved_locations.json
+places_full.csv
+places_skipped.csv
+places_override.csv
+places_map.kml
+places_map.geojson
+# Legacy (pre-rename) output names
 restaurants_full.csv
 restaurants_skipped.csv
 restaurants_override.csv
 restaurants_map.kml
 restaurants.csv
 *.checkpoint.json
+*.batch_pending.json
+.geocode_cache.json
 tests/benchmark_results.json
 
 # Python

README.md

@@ -1,6 +1,6 @@
-# Instagram Saved Restaurants → Map
+# Instagram Saved Places → Map
 
-Turn your Instagram saved posts into a searchable, filterable restaurant map — **for free, forever, without giving anyone your data.**
+Turn your Instagram saved posts into a searchable, filterable map of every place you've saved — restaurants, bars, cafes, hotels, shops, hikes, viewpoints, museums and more — **for free, forever, without giving anyone your data.**
 
 Every paid alternative (Someday Map, Rezz, ReelsMap, Drawer) is a subscription app that sends your saves to their servers and charges you monthly. This project runs entirely on your machine: Ollama handles AI extraction locally, Nominatim handles geocoding for free, and the only output is files on your disk.
 
@@ -10,7 +10,7 @@ Every paid alternative (Someday Map, Rezz, ReelsMap, Drawer) is a subscription a
 
 1. **Extracts** — reads every saved post caption using AI (Claude or a free local Ollama model), identifies posts about places worth visiting (restaurants, bars, cafes, hotels, shops, hikes, viewpoints, museums, and more), and pulls out name, city, category, cuisine, price range, highlight, and occasion
 2. **Geocodes** — looks up coordinates via Nominatim (OpenStreetMap) with country + city + proximity validation to avoid false pins
-3. **Exports** — writes `restaurants_full.csv`, `restaurants_map.kml` (Google My Maps), and `restaurants_map.geojson` (Felt, Mapbox, etc.)
+3. **Exports** — writes `places_full.csv`, `places_map.kml` (Google My Maps), and `places_map.geojson` (Felt, Mapbox, etc.)
 
 ## Why not the paid apps?
 
@@ -208,16 +208,16 @@ python main.py saved_posts.json --transcribe --whisper-model medium  # more accu
 
 | File | Description |
 |---|---|
-| `restaurants_full.csv` | One row per place: name, city, category, cuisine, highlight, price_range, lat, lng, creator, saved_at, instagram_url |
-| `restaurants_map.kml` | Google My Maps–ready file, organised by country → city. Pins show cuisine, must-order dish, and link back to Instagram. |
-| `restaurants_skipped.csv` | Reels that couldn't be downloaded or weren't food. Review manually to fill gaps. |
-| `restaurants_override.csv` | Manual override template — fill in `maps_url` or `lat`/`lng` for ungeocoded restaurants. |
+| `places_full.csv` | One row per place: name, city, category, cuisine, highlight, price_range, lat, lng, creator, saved_at, instagram_url |
+| `places_map.kml` | Google My Maps–ready file, organised by country → city. Pins show cuisine, must-order dish, and link back to Instagram. |
+| `places_skipped.csv` | Reels that couldn't be downloaded or weren't food. Review manually to fill gaps. |
+| `places_override.csv` | Manual override template — fill in `maps_url` or `lat`/`lng` for ungeocoded places. |
 
 ## Fixing missing map pins
 
-After each run, `restaurants_override.csv` lists every restaurant with no coordinates. To add a pin manually:
+After each run, `places_override.csv` lists every restaurant with no coordinates. To add a pin manually:
 
-1. Open `restaurants_override.csv`
+1. Open `places_override.csv`
 2. Search for the restaurant on Google Maps, copy the URL
 3. Paste it into the `maps_url` column (or enter `lat`/`lng` directly)
 4. Re-run: `python main.py saved_posts.json --no-extract --no-geocode --yes`
@@ -229,7 +229,7 @@ Supported URL formats:
 ## Import into Google My Maps
 
 1. Go to [mymaps.google.com](https://mymaps.google.com) → **Create a new map**
-2. Click **Import** on the base layer → upload `restaurants_map.kml`
+2. Click **Import** on the base layer → upload `places_map.kml`
 3. Done — pins are organised into folders by country and city
 
 ## Docker (self-hosted, batteries included)
@@ -253,6 +253,9 @@ Open [http://localhost:8000](http://localhost:8000). On first run, `llama3.2` (~
 | `OLLAMA_URL` | `http://ollama:11434` | Ollama API endpoint (set automatically by compose) |
 | `OLLAMA_ENABLED` | `true` | Set to `false` to hide the Ollama provider (for hosted deployments) |
 | `DEFAULT_OLLAMA_MODEL` | `llama3.2` | Model pulled on first run and pre-selected in the UI |
+| `JOB_TTL_HOURS` | `24` | Per-upload job directories are deleted after this many hours |
+| `NOMINATIM_URL` | public OSM | Geocoder endpoint — point at a self-hosted Nominatim for hosted use |
+| `HOSTED` | unset | Set to `1` on a multi-user deploy: refuses to bulk-geocode against public OSM |
 
 ### Tier comparison
 
@@ -299,10 +302,10 @@ The UI lets you:
 | **ChatGPT / Claude.ai (free chatbot)** | $0 | Captions to chatbot | None |
 
 **ChatGPT / Claude.ai path (no API key needed):**
-1. Drop your export file and click **Prepare extraction file** — the app pre-filters your posts and generates a numbered JSON file
-2. Download the JSON, copy the extraction prompt, and paste both into ChatGPT or Claude.ai (free tier)
-3. Paste the chatbot's JSON response back into the web UI
-4. The app geocodes the results and builds your map — same output as any other path
+1. Drop your export file and click **Prepare extraction file** — the app pre-filters your posts and generates a numbered JSON file (large libraries are split into several files that each fit a free chatbot's reply limit)
+2. Download each file, copy the extraction prompt, and paste both into ChatGPT or Claude.ai (free tier)
+3. Paste each reply back into the web UI (click **Add response** per file), then process
+4. The app geocodes the results and builds your map — same output as any other path. Response parsing is tolerant of markdown fences, prose, trailing commas, and truncated replies.
 
 **BYOK (Bring Your Own Key):** Select the Claude provider and paste your Anthropic API key into the password field. It is used for that request only and never stored on the server.
 
@@ -316,6 +319,10 @@ The geocoder runs three validation checks before accepting any result:
 
 If any check fails, the row is left blank rather than accepting a wrong pin.
 
+**Caching** — every lookup is cached to `.geocode_cache.json` (place → coordinates), so duplicate venues and re-runs never re-hit the network. Delete the file to force a fresh geocode.
+
+**Endpoint & hosted use** — geocoding defaults to the public OSM Nominatim endpoint, which is fine for personal/local use. OSM's usage policy **forbids bulk automated use**, so a shared/hosted deployment must run its own geocoder: set `NOMINATIM_URL` to your instance. With `HOSTED=1`, the pipeline refuses to bulk-geocode against the public endpoint rather than risk getting the server IP banned.
+
 ## Benchmark & quality testing
 
 The `tests/` directory contains a regression suite that measures extraction quality across all models.

TODO.md

@@ -128,6 +128,36 @@ KML requires Google My Maps. The primary output should be a self-contained file
 
 ---
 
+## Hardening — multi-user safety & correctness (today)
+
+Surfaced in the architecture review. These make the product safe to host and honest about what it is.
+
+- [ ] **Rename restaurants → places end-to-end** — output files (`places_full.csv`, `places_map.kml`, `places_map.geojson`, `places_skipped.csv`, `places_override.csv`), download filenames, README title, and user-facing UI/CLI text. The product handles hikes/museums/hotels now; users shouldn't download a file called "restaurants".
+- [ ] **Fix roulette cross-user data leak** — `/roulette` with no `job_id` scanned `jobs/*/` and returned the globally newest CSV — on a hosted deploy that returns *another user's* saved places. Require an explicit `job_id` (the client already sends it); drop the global-newest fallback.
+- [ ] **Job TTL + persisted state** — job state lived only in a process-memory dict (lost on restart) and `jobs/` grew forever. Persist each job's state to `state.json`, recover it in the progress endpoint, and delete job dirs older than `JOB_TTL_HOURS` (default 24) on startup and on each new upload.
+- [ ] **Geocoder: cache + configurable endpoint + hosted guard** — persistent geocode cache (place → coords) so duplicates and re-runs don't re-hit Nominatim; `NOMINATIM_URL` env to point at a self-hosted instance; refuse bulk geocoding against the public OSM endpoint when `HOSTED=1` (OSM's usage policy forbids it — a hosted deploy would get IP-banned).
+- [ ] **Robust chatbot handoff** — auto-chunk exports above the free-tier limit into multiple files, accept multiple pasted responses and merge them, and make response parsing tolerant of preamble text, trailing commas, and truncated arrays.
+
+---
+
+## Backlog — revisit at end of today's fixes
+
+- [ ] **In-app map preview (QC, not consumption)** — a Leaflet/OSM preview in the results view whose job is to *catch geocode errors before export* (a venue pinned in the wrong country is invisible in the card grid). Consumption stays in Google My Maps / offline apps via the exported file. Lower priority than the hardening work — revisit once that lands.
+
+---
+
+## Tier 3 — Growth features
+
+Build after the hardening pass. These extend reach without compromising the local-first, no-account story.
+
+- [ ] **Close the mobile consumption loop (docs)** — the deliverable is meant to live on your phone, so document the last mile: opening a custom Google My Map in the Google Maps app and navigating to a pin. Cheap, directly serves the "usable on your phone while travelling" goal.
+- [ ] **Offline / privacy-first map targets** — KML and GPX import cleanly into Organic Maps and OsmAnd: fully offline, no account, works abroad with no data plan. Arguably a better recommendation than Google My Maps for the privacy-first traveller, and we already produce the file — mostly a docs + small export addition.
+- [ ] **Persistent library without a server DB** — let the user download a "library file" and re-upload it to merge, preserving visited status, edits, and "new since last run" across re-imports. Keyed on Instagram post ID. Stays true to local-first (no backend database). Supersedes the deferred re-import item above.
+- [ ] **Self-contained interactive HTML map export** — single bundled file (Leaflet/MapLibre), pins link back to the original reel; opens on a phone browser with no import step. See the on-hold item under "Output".
+- [ ] **"Remix this map" share loop** — publish a static map file anyone can view and one-click copy into their own instance. $0 hosting for OSS; mirrors Someday Map's virality mechanic.
+
+---
+
 ## V2: Beli / Google Maps Saved integration
 
 - [ ] **find_place_ids.py** — Google Places Text Search API, validate against expected country + city. ~$2 for 173 requests.

docker-compose.yml

@@ -25,6 +25,12 @@ services:
     environment:
       OLLAMA_URL: http://ollama:11434
       OLLAMA_ENABLED: "true"
+      JOB_TTL_HOURS: "24"          # delete each upload's job dir after N hours
+      # Personal/local use geocodes against public OSM Nominatim (fine for one user).
+      # For a MULTI-USER hosted deploy, OSM forbids bulk use — run your own geocoder
+      # and uncomment these (plus a nominatim/photon service):
+      # HOSTED: "1"
+      # NOMINATIM_URL: http://nominatim:8080
     volumes:
       - ./jobs:/app/jobs
       - ./data:/app/data

main.py

@@ -15,8 +15,8 @@ Usage
   python main.py <path/to/saved_posts.json> --no-extract --transcribe  # transcribe-only run
 
 Output (written to --output-dir, default: same folder as the JSON file)
-  restaurants_full.csv   — one row per restaurant with coordinates + reel URL
-  restaurants_map.kml    — Google My Maps–ready KML organised by country › city
+  places_full.csv   — one row per restaurant with coordinates + reel URL
+  places_map.kml    — Google My Maps–ready KML organised by country › city
 
 Environment
 ───────────
@@ -117,8 +117,8 @@ def main() -> None:
     output_dir = Path(args.output_dir).resolve() if args.output_dir else input_path.parent
     output_dir.mkdir(parents=True, exist_ok=True)
 
-    csv_path = output_dir / "restaurants_full.csv"
-    kml_path = output_dir / "restaurants_map.kml"
+    csv_path = output_dir / "places_full.csv"
+    kml_path = output_dir / "places_map.kml"
 
     # Load .env starting from the input file's directory
     _load_env(input_path.parent)
@@ -252,7 +252,7 @@ def main() -> None:
     # ── Step 2b: Manual overrides ─────────────────────────────────────────────
     import csv as _csv
     from pipeline import override as override_mod
-    override_path = output_dir / "restaurants_override.csv"
+    override_path = output_dir / "places_override.csv"
     with open(csv_path, encoding="utf-8") as f:
         csv_rows = list(_csv.DictReader(f))
     csv_rows, applied = override_mod.apply(csv_rows, override_path)

pipeline/chatbot.py

@@ -8,9 +8,18 @@ Flow:
 """
 
 import json
+import re
 from . import prefilter as prefilter_mod
 
 CHUNK_WARN_THRESHOLD = 400  # show a warning above this many posts (fits in most free-tier windows)
+CHUNK_SIZE = 350            # posts per downloadable file; comfortably under free-tier output limits
+
+
+def chunk_export(export_posts: list[dict], size: int = CHUNK_SIZE) -> list[list[dict]]:
+    """Split the export into chunks small enough for a free chatbot to answer in one reply."""
+    if size <= 0 or not export_posts:
+        return [export_posts]
+    return [export_posts[i:i + size] for i in range(0, len(export_posts), size)]
 
 _PROMPT = """\
 You are a place extraction assistant. The attached JSON file contains Instagram post captions and hashtags.
@@ -119,24 +128,68 @@ def _clean(val) -> str:
     return "UNKNOWN" if not s or s.lower() in ("null", "unknown", "n/a", "") else s
 
 
-def parse_response(json_text: str, post_mapping: dict) -> list[dict]:
-    """Parse the chatbot's JSON response into standard row dicts.
+def _extract_json_array(json_text: str) -> list:
+    """Best-effort extraction of a JSON array from a chatbot reply.
 
-    Raises ValueError if the text cannot be parsed as a JSON array.
-    Non-place entries and entries with no useful data are silently dropped.
+    Tolerates: markdown fences, prose before/after the array, trailing commas,
+    and a response truncated mid-array (salvages the complete objects).
     """
     text = json_text.strip()
-    # Strip markdown fences some chatbots add despite instructions
     if text.startswith("```"):
         text = text.split("\n", 1)[-1].rsplit("```", 1)[0].strip()
 
-    try:
-        data = json.loads(text)
-    except json.JSONDecodeError as exc:
-        raise ValueError(f"Could not parse as JSON: {exc}") from exc
+    start = text.find("[")
+    end = text.rfind("]")
+    candidate = text[start:end + 1] if (start != -1 and end > start) else text
+
+    attempts = [
+        candidate,
+        re.sub(r",\s*([\]}])", r"\1", candidate),  # drop trailing commas
+    ]
+    # Salvage a truncated array: keep up to the last complete object, then close it
+    if start != -1:
+        last_obj = text.rfind("}")
+        if last_obj > start:
+            salvage = re.sub(r",\s*([\]}])", r"\1", text[start:last_obj + 1] + "]")
+            attempts.append(salvage)
+
+    for attempt in attempts:
+        try:
+            data = json.loads(attempt)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(data, list):
+            return data
+
+    raise ValueError(
+        "Could not read a JSON array from the response. Paste the chatbot's full "
+        "reply — it should start with [ and end with ]."
+    )
+
 
-    if not isinstance(data, list):
-        raise ValueError("Expected a JSON array at the top level — got a single object or non-array value.")
+def parse_responses(texts: list[str], post_mapping: dict) -> list[dict]:
+    """Parse and merge one or more chatbot replies (one per export chunk)."""
+    seen: set = set()
+    merged: list[dict] = []
+    for t in texts:
+        if not t or not t.strip():
+            continue
+        for row in parse_response(t, post_mapping):
+            dedup = row.get("instagram_url") or id(row)
+            if dedup in seen:
+                continue
+            seen.add(dedup)
+            merged.append(row)
+    return merged
+
+
+def parse_response(json_text: str, post_mapping: dict) -> list[dict]:
+    """Parse one chatbot JSON reply into standard row dicts.
+
+    Raises ValueError if no JSON array can be recovered.
+    Non-place entries and entries with no useful data are silently dropped.
+    """
+    data = _extract_json_array(json_text)
 
     rows: list[dict] = []
     for item in data:

pipeline/export.py

@@ -228,5 +228,5 @@ def run(input_csv: str, output_kml: str) -> None:
     print(f"Wrote {output_kml}  ({pinned}/{len(rows)} rows have coordinates)")
     print("\nTo import into Google My Maps:")
     print("  1. mymaps.google.com → Create a new map")
-    print("  2. Click 'Import' on the base layer → upload restaurants_map.kml")
+    print("  2. Click 'Import' on the base layer → upload places_map.kml")
     print("  3. Rename the map 'Instagram Saved Places'")

pipeline/geocode.py

@@ -18,16 +18,47 @@ Blank is always better than a wrong pin.
 import csv
 import json
 import math
+import os
 import ssl
 import time
 import urllib.request
 import urllib.parse
+from pathlib import Path
 
 import certifi
 
 SSL_CTX = ssl.create_default_context(cafile=certifi.where())
 MAX_ADDRESS_KM = 50  # max distance from the expected city centre for address hits
 
+# Geocoding endpoint. Defaults to public OSM Nominatim for local use; point at a
+# self-hosted instance for any shared/hosted deployment (OSM forbids bulk use).
+NOMINATIM_URL = os.environ.get("NOMINATIM_URL", "https://nominatim.openstreetmap.org").rstrip("/")
+_PUBLIC_NOMINATIM = "nominatim.openstreetmap.org"
+USER_AGENT = "InstagramPlacesMapper/1.0 (+https://github.com/)"
+
+# Persistent place→coords cache so duplicate venues and re-runs skip the network.
+GEOCODE_CACHE_PATH = Path(
+    os.environ.get("GEOCODE_CACHE", str(Path(__file__).parent.parent / ".geocode_cache.json"))
+)
+
+
+def _cache_key(name: str, city: str, state: str, country: str, address: str) -> str:
+    return "|".join(s.strip().lower() for s in (name, city, state, country, address))
+
+
+def _load_cache() -> dict:
+    try:
+        return json.loads(GEOCODE_CACHE_PATH.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return {}
+
+
+def _save_cache(cache: dict) -> None:
+    try:
+        GEOCODE_CACHE_PATH.write_text(json.dumps(cache), encoding="utf-8")
+    except OSError:
+        pass
+
 
 # ── Country normalisation ─────────────────────────────────────────────────────
 
@@ -131,8 +162,8 @@ def _city_centre(city: str, country_code: str | None) -> tuple[float, float] | N
     query = city if not country_code else f"{city}, {country_code}"
     params = urllib.parse.urlencode({"q": query, "format": "json", "limit": 1, "addressdetails": 0})
     req = urllib.request.Request(
-        f"https://nominatim.openstreetmap.org/search?{params}",
-        headers={"User-Agent": "InstagramRestaurantMapper/1.0"},
+        f"{NOMINATIM_URL}/search?{params}",
+        headers={"User-Agent": USER_AGENT},
     )
     try:
         with urllib.request.urlopen(req, timeout=10, context=SSL_CTX) as resp:
@@ -164,8 +195,8 @@ def _nominatim(query: str, expected_country: str, expected_city: str) -> tuple[f
         "addressdetails": 1,
     })
     req = urllib.request.Request(
-        f"https://nominatim.openstreetmap.org/search?{params}",
-        headers={"User-Agent": "InstagramRestaurantMapper/1.0"},
+        f"{NOMINATIM_URL}/search?{params}",
+        headers={"User-Agent": USER_AGENT},
     )
     try:
         with urllib.request.urlopen(req, timeout=10, context=SSL_CTX) as resp:
@@ -272,6 +303,13 @@ def geocode_one(row: dict) -> tuple[str, str]:
 
 def run(input_csv: str, output_csv: str | None = None) -> list[dict]:
     """Geocode every row in input_csv and write results to output_csv (or in-place)."""
+    if os.environ.get("HOSTED", "").lower() in ("1", "true", "yes") and _PUBLIC_NOMINATIM in NOMINATIM_URL:
+        raise RuntimeError(
+            "Refusing to bulk-geocode against public Nominatim in hosted mode. "
+            "Set NOMINATIM_URL to a self-hosted instance — OSM's usage policy "
+            "forbids bulk automated use of the public endpoint."
+        )
+
     target = output_csv or input_csv
 
     with open(input_csv, encoding="utf-8") as f:
@@ -279,6 +317,8 @@ def run(input_csv: str, output_csv: str | None = None) -> list[dict]:
 
     print(f"Loaded {len(rows)} rows. Geocoding with country + city + proximity validation...\n")
 
+    cache = _load_cache()
+
     for row in rows:
         row["lat"] = ""
         row["lng"] = ""
@@ -287,21 +327,30 @@ def run(input_csv: str, output_csv: str | None = None) -> list[dict]:
         name = row.get("name", "")
         city = row.get("city", "")
         country = row.get("country", "")
-        print(f"  [{i}/{len(rows)}] {name} — {city}, {country}")
-
-        lat, lng = geocode_row(
-            name,
-            city,
-            row.get("state", ""),
-            country,
-            row.get("address", ""),
-        )
+
+        key = _cache_key(name, city, row.get("state", ""), country, row.get("address", ""))
+        if key in cache:
+            lat, lng = cache[key]
+            print(f"  [{i}/{len(rows)}] {name} — {city}, {country}  (cached)")
+        else:
+            print(f"  [{i}/{len(rows)}] {name} — {city}, {country}")
+            lat, lng = geocode_row(
+                name,
+                city,
+                row.get("state", ""),
+                country,
+                row.get("address", ""),
+            )
+            cache[key] = [lat, lng]
+
         row["lat"] = f"{lat:.7f}" if lat is not None else ""
         row["lng"] = f"{lng:.7f}" if lng is not None else ""
 
         status = f"→ {lat:.4f}, {lng:.4f}" if lat is not None else "→ not found (left blank)"
         print(f"          {status}")
 
+    _save_cache(cache)
+
     with open(target, "w", newline="", encoding="utf-8") as f:
         writer = csv.DictWriter(f, fieldnames=rows[0].keys())
         writer.writeheader()

pipeline/override.py

@@ -2,7 +2,7 @@
 Manual override for restaurants that geocoding couldn't place.
 
 Workflow:
-  1. After geocoding, restaurants_override.csv is generated (or updated)
+  1. After geocoding, places_override.csv is generated (or updated)
      with all rows that still have no lat/lng.
   2. User opens it, finds the restaurant on Google Maps, and pastes the URL
      into the maps_url column (or enters lat/lng directly).

pipeline/transcribe.py

@@ -6,7 +6,7 @@ For each post URL not already in the extracted CSV, this module:
   2. Transcribes it with Whisper (local, free — requires openai-whisper + torch)
   3. Re-runs Claude extraction on the transcript
   4. Appends new finds to the extracted CSV
-  5. Writes restaurants_skipped.csv with URLs that failed or were not food,
+  5. Writes places_skipped.csv with URLs that failed or were not food,
      so the user can manually review and add them
 
 Install extra deps first:
@@ -144,7 +144,7 @@ def run(
 
     New rows are appended to extracted_csv with lat/lng blank.
     All posts that couldn't be downloaded or weren't food are written to
-    restaurants_skipped.csv for manual review.
+    places_skipped.csv for manual review.
 
     Download failures are never checkpointed — they always retry so that
     adding browser auth takes effect without clearing the checkpoint.
@@ -157,7 +157,7 @@ def run(
         return
 
     checkpoint_path = Path(str(Path(extracted_csv).with_suffix("")) + _CHECKPOINT_SUFFIX)
-    skipped_csv_path = Path(extracted_csv).parent / "restaurants_skipped.csv"
+    skipped_csv_path = Path(extracted_csv).parent / "places_skipped.csv"
 
     checkpoint = _load_checkpoint(checkpoint_path) if resume else {}
     already_done = sum(1 for u in skipped if u["url"] in checkpoint)
@@ -172,7 +172,7 @@ def run(
     client = anthropic.Anthropic()
 
     new_rows: list[dict] = []
-    skipped_records: list[dict] = []  # for restaurants_skipped.csv
+    skipped_records: list[dict] = []  # for places_skipped.csv
 
     print()
     bar = tqdm(skipped, desc="Transcribing reels", unit="reel", dynamic_ncols=True)

web/app.py

@@ -37,6 +37,7 @@ JOBS_DIR.mkdir(exist_ok=True)
 
 OLLAMA_ENABLED = os.environ.get("OLLAMA_ENABLED", "true").lower() not in ("0", "false", "no")
 OLLAMA_URL     = os.environ.get("OLLAMA_URL", "http://localhost:11434")
+JOB_TTL_HOURS  = float(os.environ.get("JOB_TTL_HOURS", "24"))
 
 _BENCHMARK_RESULTS_PATH = Path(__file__).parent.parent / "tests" / "benchmark_results.json"
 
@@ -48,17 +49,66 @@ def _load_benchmark_scores() -> dict:
     except (FileNotFoundError, json.JSONDecodeError):
         return {}
 
-app = FastAPI(title="Instagram Restaurant Mapper")
+app = FastAPI(title="Instagram Places Mapper")
 templates = Jinja2Templates(directory=str(Path(__file__).parent / "templates"))
 
-# In-memory job state (survives for the lifetime of the server process)
+
+@app.on_event("startup")
+def _on_startup() -> None:
+    _cleanup_old_jobs()
+
+# In-memory job state, mirrored to <job_dir>/state.json so it survives a restart.
 _jobs: dict[str, dict] = {}
 _lock = threading.Lock()
 
 
+def _state_path(job_id: str) -> Path:
+    return JOBS_DIR / job_id / "state.json"
+
+
+def _persist_state(job_id: str, state: dict) -> None:
+    """Mirror a job's state to disk so the progress endpoint survives a restart."""
+    try:
+        job_dir = JOBS_DIR / job_id
+        job_dir.mkdir(exist_ok=True)
+        _state_path(job_id).write_text(json.dumps(state), encoding="utf-8")
+    except OSError:
+        pass  # disk mirror is best-effort; in-memory state is authoritative
+
+
+def _get_state(job_id: str) -> dict:
+    """Return a job's state from memory, falling back to the on-disk mirror."""
+    with _lock:
+        if job_id in _jobs:
+            return dict(_jobs[job_id])
+    try:
+        return json.loads(_state_path(job_id).read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return {"step": "unknown", "message": "Job not found"}
+
+
 def _update(job_id: str, **kwargs) -> None:
     with _lock:
         _jobs[job_id].update(kwargs)
+        snapshot = dict(_jobs[job_id])
+    _persist_state(job_id, snapshot)
+
+
+def _cleanup_old_jobs() -> None:
+    """Delete job directories older than JOB_TTL_HOURS (best-effort)."""
+    import shutil
+    import time as _time
+    cutoff = _time.time() - JOB_TTL_HOURS * 3600
+    for d in JOBS_DIR.glob("*"):
+        if not d.is_dir():
+            continue
+        try:
+            if d.stat().st_mtime < cutoff:
+                shutil.rmtree(d, ignore_errors=True)
+                with _lock:
+                    _jobs.pop(d.name, None)
+        except OSError:
+            pass
 
 
 def _haversine_km(lat1, lon1, lat2, lon2) -> float:
@@ -75,9 +125,9 @@ def _run_pipeline(job_id: str, json_bytes: bytes, model: str,
     job_dir = JOBS_DIR / job_id
     job_dir.mkdir(exist_ok=True)
     json_path = job_dir / "saved_posts.json"
-    csv_path  = job_dir / "restaurants_full.csv"
-    kml_path  = job_dir / "restaurants_map.kml"
-    override_path = job_dir / "restaurants_override.csv"
+    csv_path  = job_dir / "places_full.csv"
+    kml_path  = job_dir / "places_map.kml"
+    override_path = job_dir / "places_override.csv"
 
     try:
         json_path.write_bytes(json_bytes)
@@ -107,7 +157,7 @@ def _run_pipeline(job_id: str, json_bytes: bytes, model: str,
             extracted = list(csv.DictReader(f))
 
         _update(job_id, step="geocode", progress=55,
-                message=f"Found {len(extracted)} restaurants — geocoding…",
+                message=f"Found {len(extracted)} places — geocoding…",
                 extracted=len(extracted))
 
         # ── Geocode ────────────────────────────────────────────────────────
@@ -143,9 +193,9 @@ def _run_pipeline(job_id: str, json_bytes: bytes, model: str,
 def _run_geocode_export(job_id: str) -> None:
     """Geocode + export an already-written CSV (no extraction step)."""
     job_dir  = JOBS_DIR / job_id
-    csv_path = job_dir / "restaurants_full.csv"
-    kml_path = job_dir / "restaurants_map.kml"
-    override_path = job_dir / "restaurants_override.csv"
+    csv_path = job_dir / "places_full.csv"
+    kml_path = job_dir / "places_map.kml"
+    override_path = job_dir / "places_override.csv"
     try:
         _update(job_id, step="geocode", progress=30,
                 message="Geocoding places…")
@@ -192,7 +242,7 @@ class RegeocodeRequest(BaseModel):
 
 
 def _csv_path(job_id: str) -> Path:
-    return JOBS_DIR / job_id / "restaurants_full.csv"
+    return JOBS_DIR / job_id / "places_full.csv"
 
 
 def _read_csv(job_id: str) -> list[dict] | None:
@@ -265,10 +315,13 @@ async def upload(
         active_model = model if model in MODELS else DEFAULT_MODEL
 
     contents = await file.read()
+    _cleanup_old_jobs()
     job_id = uuid.uuid4().hex
 
+    initial = {"step": "queued", "progress": 0, "message": "Starting…"}
     with _lock:
-        _jobs[job_id] = {"step": "queued", "progress": 0, "message": "Starting…"}
+        _jobs[job_id] = initial
+    _persist_state(job_id, initial)
 
     thread = threading.Thread(
         target=_run_pipeline,
@@ -287,8 +340,7 @@ async def progress(job_id: str):
     async def _generate():
         import asyncio
         while True:
-            with _lock:
-                state = dict(_jobs.get(job_id, {"step": "unknown", "message": "Job not found"}))
+            state = _get_state(job_id)
             yield f"data: {json.dumps(state)}\n\n"
             if state.get("step") in ("done", "error"):
                 break
@@ -300,11 +352,11 @@ async def progress(job_id: str):
 
 @app.get("/download/{job_id}")
 async def download(job_id: str):
-    kml = JOBS_DIR / job_id / "restaurants_map.kml"
+    kml = JOBS_DIR / job_id / "places_map.kml"
     if not kml.exists():
         return HTMLResponse("Not found", status_code=404)
     return FileResponse(str(kml), media_type="application/vnd.google-earth.kml+xml",
-                        filename="restaurants_map.kml")
+                        filename="places_map.kml")
 
 
 @app.get("/download/{job_id}/geojson")
@@ -316,7 +368,7 @@ async def download_geojson(job_id: str):
     return Response(
         content=geojson_str,
         media_type="application/geo+json",
-        headers={"Content-Disposition": 'attachment; filename="restaurants_map.geojson"'},
+        headers={"Content-Disposition": 'attachment; filename="places_map.geojson"'},
     )
 
 
@@ -326,7 +378,7 @@ async def download_csv(job_id: str):
     if not csv_p.exists():
         return HTMLResponse("Not found", status_code=404)
     return FileResponse(str(csv_p), media_type="text/csv",
-                        filename="restaurants_full.csv")
+                        filename="places_full.csv")
 
 
 @app.get("/results/{job_id}")
@@ -383,7 +435,7 @@ async def reexport(job_id: str):
     csv_p = _csv_path(job_id)
     if not csv_p.exists():
         return JSONResponse({"error": "Job not found"}, status_code=404)
-    kml_p = JOBS_DIR / job_id / "restaurants_map.kml"
+    kml_p = JOBS_DIR / job_id / "places_map.kml"
 
     def _run():
         with contextlib.redirect_stdout(io.StringIO()):
@@ -394,7 +446,8 @@ async def reexport(job_id: str):
 
 
 class ChatbotProcessRequest(BaseModel):
-    response_json: str
+    response_json: str = ""
+    responses: list[str] | None = None  # one per export chunk (preferred)
 
 
 @app.post("/chatbot-prepare")
@@ -402,6 +455,7 @@ async def chatbot_prepare(file: UploadFile = File(...)):
     """Receive an Instagram JSON/zip, run prefilter, return the chatbot export package."""
     import zipfile as _zipfile
     contents = await file.read()
+    _cleanup_old_jobs()
 
     if file.filename and file.filename.lower().endswith(".zip"):
         try:
@@ -432,8 +486,10 @@ async def chatbot_prepare(file: UploadFile = File(...)):
         json.dumps(result["post_mapping"]), encoding="utf-8"
     )
 
+    pending = {"step": "chatbot_pending", "progress": 0, "message": "Waiting for chatbot response…"}
     with _lock:
-        _jobs[job_id] = {"step": "chatbot_pending", "progress": 0, "message": "Waiting for chatbot response…"}
+        _jobs[job_id] = pending
+    _persist_state(job_id, pending)
 
     return {
         "job_id":       job_id,
@@ -443,6 +499,8 @@ async def chatbot_prepare(file: UploadFile = File(...)):
         "skipped":      result["skipped"],
         "prompt":       chatbot_mod.get_prompt(len(result["export_posts"])),
         "warn_large":   len(result["export_posts"]) > chatbot_mod.CHUNK_WARN_THRESHOLD,
+        "chunk_size":   chatbot_mod.CHUNK_SIZE,
+        "chunk_count":  len(chatbot_mod.chunk_export(result["export_posts"])),
     }
 
 
@@ -459,8 +517,9 @@ async def chatbot_process(job_id: str, body: ChatbotProcessRequest):
         post_mapping = json.load(f)
 
     from pipeline import chatbot as chatbot_mod
+    texts = body.responses if body.responses else [body.response_json]
     try:
-        rows = chatbot_mod.parse_response(body.response_json, post_mapping)
+        rows = chatbot_mod.parse_responses(texts, post_mapping)
     except ValueError as exc:
         return JSONResponse({"error": str(exc)}, status_code=422)
 
@@ -470,7 +529,7 @@ async def chatbot_process(job_id: str, body: ChatbotProcessRequest):
             status_code=422,
         )
 
-    csv_path = job_dir / "restaurants_full.csv"
+    csv_path = job_dir / "places_full.csv"
     with open(csv_path, "w", newline="", encoding="utf-8") as f:
         writer = csv.DictWriter(f, fieldnames=FIELDNAMES)
         writer.writeheader()
@@ -490,24 +549,19 @@ async def chatbot_process(job_id: str, body: ChatbotProcessRequest):
 async def roulette(job_id: str = "", lat: float = None, lng: float = None,
                    radius_km: float = 40, city: str = "", all_statuses: bool = False,
                    category: str = ""):
-    """Pick a random restaurant from the most recent job's CSV."""
-    import random
+    """Pick a random place from a specific job's CSV.
 
-    # Find CSV: prefer the given job, otherwise the newest job with a CSV
-    csv_path = None
-    if job_id:
-        p = JOBS_DIR / job_id / "restaurants_full.csv"
-        if p.exists():
-            csv_path = p
+    A job_id is required — there is deliberately no "newest job wins" fallback,
+    which would leak one user's places to another on a shared deployment.
+    """
+    import random
 
-    if csv_path is None:
-        candidates = sorted(JOBS_DIR.glob("*/restaurants_full.csv"),
-                            key=lambda p: p.stat().st_mtime, reverse=True)
-        if candidates:
-            csv_path = candidates[0]
+    if not job_id:
+        return {"error": "No job selected. Run a pipeline first."}
 
-    if csv_path is None:
-        return {"error": "No restaurant data yet. Run a pipeline first."}
+    csv_path = JOBS_DIR / job_id / "places_full.csv"
+    if not csv_path.exists():
+        return {"error": "No place data for this job yet. Run a pipeline first."}
 
     with open(csv_path, encoding="utf-8") as f:
         rows = [r for r in csv.DictReader(f)
@@ -517,7 +571,7 @@ async def roulette(job_id: str = "", lat: float = None, lng: float = None,
     if city:
         rows = [r for r in rows if r.get("city", "").strip().lower() == city.lower()]
         if not rows:
-            return {"error": f"No restaurants found in {city}."}
+            return {"error": f"No places found in {city}."}
     elif lat is not None and lng is not None:
         def _ok(r):
             try:

web/templates/index.html

@@ -3,7 +3,7 @@
 <head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
-<title>The Belagurian · Restaurant Map</title>
+<title>The Belagurian · Places Map</title>
 <style>
   :root {
     --ig1: #e1306c;
@@ -711,7 +711,7 @@
   <div class="logo-ring">
     <div class="logo-inner">🍜</div>
   </div>
-  <h1>Restaurant Mapper</h1>
+  <h1>Places Mapper</h1>
   <p class="tagline">Turn your Instagram saved posts into a Google My Maps import</p>
 </div>
 
@@ -820,14 +820,12 @@
     </ol>
     <!-- Step outputs (hidden until prepare runs) -->
     <div id="chatbot-step2" style="display:none;margin-top:16px;">
-      <div style="display:flex;gap:10px;flex-wrap:wrap;margin-bottom:12px;">
-        <a id="chatbot-download-btn" class="btn btn-outline" download="instagram_places.json" style="text-decoration:none;">
-          ⬇ Download posts JSON
-        </a>
+      <div style="margin-bottom:12px;">
+        <div id="chatbot-downloads" style="display:flex;gap:10px;flex-wrap:wrap;margin-bottom:10px;"></div>
         <button class="btn btn-outline" id="chatbot-copy-prompt-btn" type="button">📋 Copy prompt</button>
       </div>
-      <div id="chatbot-warn-large" style="display:none;background:#fff8e1;border:1px solid #f0c040;border-radius:6px;padding:10px 12px;font-size:0.8rem;margin-bottom:12px;">
-        ⚠️ Your library has <strong id="chatbot-post-count"></strong> posts to analyse. Most free AI tiers handle this fine, but if you hit a limit, split the JSON into two halves and run each separately.
+      <div id="chatbot-chunk-note" style="display:none;background:#eef4ff;border:1px solid #c7d9ff;border-radius:6px;padding:10px 12px;font-size:0.8rem;margin-bottom:12px;">
+        Your library is split into <strong id="chatbot-chunk-count"></strong> files so each fits a free chatbot's reply limit. Run <em>each</em> file through the chatbot, and paste every reply below with <strong>Add response</strong> before processing.
       </div>
       <label style="font-size:0.82rem;font-weight:600;display:block;margin-bottom:6px;">
         Paste the chatbot's response here:
@@ -835,6 +833,10 @@
       <textarea id="chatbot-response-area" rows="6"
                 placeholder='[{"post_number": 1, "is_place": true, "category": "Restaurant", …}, …]'
                 style="width:100%;padding:8px 10px;border:1px solid var(--border);border-radius:6px;font-size:0.78rem;font-family:monospace;resize:vertical;"></textarea>
+      <div style="display:flex;gap:10px;align-items:center;margin-top:8px;">
+        <button class="btn btn-outline" id="chatbot-add-response-btn" type="button" style="display:none;">➕ Add response</button>
+        <span id="chatbot-added-count" style="font-size:0.8rem;color:var(--muted);"></span>
+      </div>
       <button class="btn" id="chatbot-process-btn" type="button" style="margin-top:10px;width:100%;" disabled>
         Process response &amp; build map →
       </button>
@@ -895,7 +897,7 @@
   <div class="steps">
     <div class="step" id="s-extract">
       <div class="step-icon">🤖</div>
-      <span id="extract-step-label">Extract restaurants with AI</span>
+      <span id="extract-step-label">Extract places with AI</span>
     </div>
     <div class="step" id="s-geocode">
       <div class="step-icon">📍</div>
@@ -1254,6 +1256,14 @@ $('btn-chatbot').addEventListener('click', () => setProvider('chatbot'));
 // ── Chatbot handoff ────────────────────────────────────────────────────────
 let _chatbotJobId = null;
 let _chatbotPromptText = '';
+let _chatbotChunkCount = 1;
+let _chatbotResponses = [];   // accumulated pasted replies (multi-chunk mode)
+
+function _updateAddedCount() {
+  const el = $('chatbot-added-count');
+  if (_chatbotChunkCount <= 1) { el.textContent = ''; return; }
+  el.textContent = `${_chatbotResponses.length} of ${_chatbotChunkCount} added`;
+}
 
 $('chatbot-prepare-btn').addEventListener('click', async () => {
   const prepBtn = $('chatbot-prepare-btn');
@@ -1282,14 +1292,37 @@ $('chatbot-prepare-btn').addEventListener('click', async () => {
   const data = await res.json();
   _chatbotJobId = data.job_id;
   _chatbotPromptText = data.prompt;
+  _chatbotChunkCount = data.chunk_count || 1;
+  _chatbotResponses = [];
 
-  const blob = new Blob([JSON.stringify(data.export_posts, null, 2)], { type: 'application/json' });
-  $('chatbot-download-btn').href = URL.createObjectURL(blob);
-
-  if (data.warn_large) {
-    $('chatbot-post-count').textContent = data.post_count;
-    $('chatbot-warn-large').style.display = 'block';
-  }
+  // Split the export into chunk files and render a download link for each
+  const posts = data.export_posts;
+  const size = data.chunk_size || posts.length || 1;
+  const dl = $('chatbot-downloads');
+  dl.innerHTML = '';
+  for (let c = 0; c < _chatbotChunkCount; c++) {
+    const slice = posts.slice(c * size, (c + 1) * size);
+    const blob = new Blob([JSON.stringify(slice, null, 2)], { type: 'application/json' });
+    const a = document.createElement('a');
+    a.className = 'btn btn-outline';
+    a.style.textDecoration = 'none';
+    a.download = _chatbotChunkCount > 1
+      ? `instagram_places_${c + 1}of${_chatbotChunkCount}.json`
+      : 'instagram_places.json';
+    a.href = URL.createObjectURL(blob);
+    a.textContent = _chatbotChunkCount > 1
+      ? `⬇ File ${c + 1} of ${_chatbotChunkCount}`
+      : '⬇ Download posts JSON';
+    dl.appendChild(a);
+  }
+
+  const multi = _chatbotChunkCount > 1;
+  $('chatbot-chunk-note').style.display      = multi ? 'block' : 'none';
+  $('chatbot-add-response-btn').style.display = multi ? 'inline-block' : 'none';
+  if (multi) $('chatbot-chunk-count').textContent = _chatbotChunkCount;
+  $('chatbot-process-btn').disabled = true;
+  $('chatbot-response-area').value = '';
+  _updateAddedCount();
 
   $('chatbot-step2').style.display = 'block';
   prepBtn.innerHTML = '✓ File ready — see steps below';
@@ -1303,14 +1336,31 @@ $('chatbot-copy-prompt-btn').addEventListener('click', () => {
   });
 });
 
+// Single-chunk: enable Process as soon as there's text. Multi-chunk: Process is
+// gated on accumulated responses, so the textarea only feeds "Add response".
 $('chatbot-response-area').addEventListener('input', () => {
-  $('chatbot-process-btn').disabled = !$('chatbot-response-area').value.trim();
+  if (_chatbotChunkCount <= 1) {
+    $('chatbot-process-btn').disabled = !$('chatbot-response-area').value.trim();
+  }
+});
+
+$('chatbot-add-response-btn').addEventListener('click', () => {
+  const area = $('chatbot-response-area');
+  const txt = area.value.trim();
+  if (!txt) return;
+  _chatbotResponses.push(txt);
+  area.value = '';
+  _updateAddedCount();
+  $('chatbot-process-btn').disabled = _chatbotResponses.length === 0;
 });
 
 $('chatbot-process-btn').addEventListener('click', async () => {
   const processBtn = $('chatbot-process-btn');
-  const responseText = $('chatbot-response-area').value.trim();
-  if (!responseText || !_chatbotJobId) return;
+  // Include anything still sitting in the textarea (single-chunk, or a forgotten "Add")
+  const pending = $('chatbot-response-area').value.trim();
+  const responses = _chatbotResponses.slice();
+  if (pending) responses.push(pending);
+  if (responses.length === 0 || !_chatbotJobId) return;
 
   processBtn.innerHTML = '<span class="spinner"></span>Processing…';
   processBtn.disabled = true;
@@ -1320,7 +1370,7 @@ $('chatbot-process-btn').addEventListener('click', async () => {
     res = await fetch('/chatbot-process/' + _chatbotJobId, {
       method: 'POST',
       headers: { 'Content-Type': 'application/json' },
-      body: JSON.stringify({ response_json: responseText }),
+      body: JSON.stringify({ responses }),
     });
     if (!res.ok) throw new Error(await res.text());
   } catch (err) {
@@ -1395,7 +1445,7 @@ runBtn.addEventListener('click', async () => {
   const modelLabel = activeProvider === 'ollama'
     ? `Ollama / ${selectedOllamaModel()}`
     : `Claude / ${selectedModel()}`;
-  $('extract-step-label').textContent = `Extract restaurants with ${modelLabel}`;
+  $('extract-step-label').textContent = `Extract places with ${modelLabel}`;
 
   $('upload-card').style.display = 'none';
   $('progress-section').style.display = 'block';
@@ -1545,7 +1595,7 @@ $('spin-btn').addEventListener('click', async () => {
   $('spin-btn').disabled = false;
 
   // Auto-flip to "All places" when all unvisited spots are exhausted
-  if (r.error && r.error.includes('No unvisited restaurants')) {
+  if (r.error && r.error.includes('No unvisited places')) {
     _setRouletteAllStatuses(true);
     $('spin-btn').click();
     return;