add google maps API

.cursor/plans/samhsa_treatment_locator_chatbot_572923b7.plan.md

@@ -0,0 +1,118 @@
+---
+name: SAMHSA Treatment Locator Chatbot
+overview: "Build the SAMHSA Treatment Locator as a single, polished Gradio app on HuggingFace: data-grounded chatbot with conversation design matching the example, clear evaluation for the memo, trust/inclusivity and data story, and a professional Gradio UI—structured so a future React/Vercel frontend can reuse the same backend."
+todos: []
+isProject: false
+---
+
+# SAMHSA Treatment Locator – Gradio/HF Focus (Memo-Ready)
+
+## Goal
+
+Deliver one strong product: a **Gradio chatbot on HuggingFace** that helps users find treatment facilities by conversation, with **no hallucinated info**, a **memo-friendly evaluation**, and a **good-looking UI**. Design the backend so that, if time allows, a React app on Vercel can call the same logic later.
+
+---
+
+## 1. Core product (unchanged from before)
+
+- **Data:** Load SAMHSA facility data (CSV from N-SUMHSS or National Directory); implement `search(criteria)` in `src/facilities.py`.
+- **State:** Conversation state (criteria + last results) in Gradio; extract/merge criteria from turns; run search when location (and optionally other filters) are present.
+- **Accuracy:** System prompt + only pass real search results to the model; model never invents facilities, addresses, or phones.
+- **Files:** [requirements.txt](requirements.txt) (+ pandas), `data/` + CSV, [src/facilities.py](src/facilities.py), [src/chat.py](src/chat.py), [app.py](app.py), [config.py](config.py).
+
+---
+
+## 2. Conversation design (match the example)
+
+Align the flow with [samhsa_chatbot_conversation_example.txt](samhsa_chatbot_conversation_example.txt):
+
+
+| Phase               | Behavior                                                                                                                                                                                 |
+| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Greet / clarify** | First message: acknowledge, then ask for location, treatment type (inpatient/outpatient/residential/telehealth), and payment (insurance, Medicaid, sliding scale, free).                 |
+| **First results**   | Once we have at least location (and ideally type + payment), return 2–3 facilities by name with 1–2 sentence descriptions (from data only). Offer to give more details or other options. |
+| **Follow-up**       | If user asks about a specific facility (e.g. “Do they offer MAT?”), answer from the same facility record only; offer next steps (e.g. how to contact).                                   |
+| **Closing**         | If user thanks or says they’re done, brief supportive close and invite them to return.                                                                                                   |
+
+
+Implementation: encode this in the **system prompt** and, if needed, short **rules** (e.g. “if no location in state, ask for location before searching”). Keep responses concise and actionable.
+
+---
+
+## 3. Trust and inclusivity in the product
+
+- **Disclaimer (in UI):** In Gradio description or a static text block: *“Information is from SAMHSA data. Always verify with the facility or [findtreatment.gov](https://findtreatment.gov) before making decisions. This tool does not endorse any facility.”*
+- **Tone:** Supportive, non-judgmental, clear (reflected in system prompt).
+- **Memo:** In the memo, mention accessibility (e.g. keyboard use, clear labels) and any limitations (e.g. English-only for now, data as of [date]).
+
+---
+
+## 4. Data story (for the memo)
+
+- **Source:** Name the dataset (e.g. N-SUMHSS 2024 or National Directory 2024), where you got it, and how you converted/processed it (e.g. “SAS → CSV, kept facilities with non-missing location”).
+- **Scope:** What’s included (e.g. states covered, which attributes: treatment type, payment, populations, therapies, languages).
+- **Limitations:** One or two sentences (e.g. “Data as of [date]; facility details may have changed; always confirm with the provider.”).
+- **In code:** Optional `data/README.md` or a short comment in `src/facilities.py` with the same bullets so the memo can reference the repo.
+
+---
+
+## 5. Evaluation that’s easy to describe in the memo
+
+- **Scenarios:** Define 15–20 test scenarios (e.g. “outpatient, Boston, Medicaid, MAT”; “veterans, California, residential”). Cover variety: locations, treatment types, payment, special populations.
+- **Metrics:**
+  - **Hallucination:** For each run, check that every facility name (and contact info) in the bot’s reply appears in your dataset. Target: 0 invented facilities.
+  - **Match:** Check that returned facilities actually match the scenario’s criteria (e.g. accepts Medicaid, offers outpatient). Report e.g. “18/20 runs had all suggested facilities matching criteria.”
+- **Artifact:** A small script (e.g. `scripts/eval_chatbot.py`) or notebook that runs these scenarios (or a subset) and outputs a table: scenario, facilities returned, hallucination? (Y/N), all match? (Y/N). Use that table (or a summary) in the memo.
+- **Memo section:** “Evaluation” with method (scenarios + checks) and results (numbers + 1–2 example outcomes). Makes the “we do not provide inaccurate information” claim concrete.
+
+---
+
+## 6. Memo that’s easy to grade
+
+Structure the 1–2 page memo as:
+
+- **Design:** How the chatbot works (data → criteria extraction → search → response); why this flow; how you avoid hallucination.
+- **Data:** Data story (source, scope, limitations) as above.
+- **Evaluation:** Method (scenarios, hallucination + match checks) and results (table or bullet list).
+- **Limitations:** 2–3 short points (e.g. data freshness, English-only, no medical advice).
+
+Optional: one figure (e.g. table of eval results or a short dialogue snippet showing good behavior).
+
+---
+
+## 7. Make Gradio good-looking
+
+- **Theme:** Use a cohesive theme, e.g. `gr.themes.Soft()` or `gr.themes.Glass()` (or a custom theme) in [app.py](app.py) so the Space doesn’t look like the default.
+- **Title and description:** Clear title (e.g. “SAMHSA Treatment Locator”) and a short description: what it does, that it uses SAMHSA data, and the disclaimer (or link to it).
+- **Examples:** 2–3 example prompts that mirror the conversation flow (e.g. “I’m looking for outpatient alcohol treatment in Boston with Medicaid”; “Do you have options for veterans in Texas?”).
+- **Layout (optional):** If useful, use `gr.Blocks()` and place the disclaimer in a visible box above or below the chat; keep the chat as the main focus.
+- **Copy:** Friendly, consistent button/label text (e.g. “Send” or “Ask”) and placeholder if applicable.
+
+No second UI codebase; all “beautiful” effort stays in this one Gradio app.
+
+---
+
+## 8. Future React/Vercel (if time allows)
+
+To make a later React app on Vercel easy:
+
+- **Logic in one place:** Keep all “business logic” (criteria extraction, search, response generation) in Python (e.g. `src/chat.py` + `src/facilities.py`). The Gradio app in [app.py](app.py) should only call into that (e.g. `chatbot.get_response(message, history, state)`).
+- **Optional API later:** If you add a small FastAPI (or Flask) wrapper that exposes a single endpoint (e.g. `POST /chat` with `{ "message": "...", "history": [...], "state": {...} }` and returns `{ "response": "...", "state": {...} }`), the same backend can serve both Gradio and a React frontend. For this phase, **no API is required**; just avoid putting critical logic inside Gradio-specific code so that a thin API layer can be added later without refactoring.
+
+---
+
+## File and task summary
+
+
+| Item                                 | Action                                                                                                     |
+| ------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
+| Data                                 | Add `data/` + facility CSV; optional `data/README.md` for data story                                       |
+| [requirements.txt](requirements.txt) | Add `pandas`                                                                                               |
+| `src/facilities.py`                  | Load CSV, `search(criteria)`, column mapping                                                               |
+| [src/chat.py](src/chat.py)           | Stateful flow, system prompt (no hallucination, conversation phases), criteria + search results in context |
+| [app.py](app.py)                     | State + history; theme, title, description, examples, disclaimer; pass history/state to chatbot            |
+| Eval                                 | `scripts/eval_chatbot.py` or notebook: run 15–20 scenarios, record hallucination + match, output table     |
+| Memo                                 | 1–2 pages: Design, Data, Evaluation, Limitations (and optional figure)                                     |
+
+
+Implementation order: data → facilities.py → chat.py (with conversation design + trust in prompt) → app.py (state, UI, disclaimer) → eval script → memo.

README.md

@@ -43,7 +43,11 @@ pip install -r requirements.txt
    - In config.py, set the BASE_MODEL variable to your base model of choice from HuggingFace.
    - Keep in mind it's better to have a small, lightweight model if you plan on finetuning.
 
-
+4. **Use all FindTreatment.gov data (optional):** The app loads facilities from `data/facilities.csv`. By default the repo may include a small sample. To use the full dataset (same as [FindTreatment.gov](https://findtreatment.gov)), run:
+   ```bash
+   python scripts/download_findtreatment_data.py
+   ```
+   This downloads the official SAMHSA National Directory and builds `data/facilities.csv`. Requires `openpyxl` (in `requirements.txt`).
 
 ## Repository Organization
 

app.py

@@ -1,80 +1,551 @@
 """
-Gradio Web Interface for Boston School Chatbot
-
-This script creates a web interface for your chatbot using Gradio.
-You only need to implement the chat function.
-
-Key Features:
-- Creates a web UI for your chatbot
-- Handles conversation history
-- Provides example questions
-- Can be deployed to Hugging Face Spaces
-
-Example Usage:
-    # Run locally:
-    python app.py
-    
-    # Access in browser:
-    # http://localhost:7860
+SAMHSA Treatment Locator – Gradio app for HuggingFace Spaces.
+
+Two-pane layout: map (left) + chat (right). When GOOGLE_MAPS_API_KEY is set in .env,
+the map uses Google Maps (JavaScript API); otherwise Folium/OpenStreetMap.
+Facility markers and search results are shown on the map.
 """
 
+import base64
+import html as html_module
+import json
+import os
+import time
+
+import folium
 import gradio as gr
-from src.chat import Chatbot
+import requests
+
+# Load .env from project root so GOOGLE_MAPS_API_KEY is set regardless of launch cwd
+_APP_DIR = os.path.dirname(os.path.abspath(__file__))
+_ENV_PATH = os.path.join(_APP_DIR, ".env")
+try:
+    from dotenv import load_dotenv
+    load_dotenv(_ENV_PATH)
+except ImportError:
+    pass
+
+from src.chat import DEFAULT_STATE, Chatbot
+
+# Use Google Maps when key is set; otherwise Folium/OSM. Enable "Maps JavaScript API" in Google Cloud.
+GOOGLE_MAPS_API_KEY = (os.environ.get("GOOGLE_MAPS_API_KEY") or "").strip() or None
+if not GOOGLE_MAPS_API_KEY and os.path.isfile(_ENV_PATH):
+    with open(_ENV_PATH) as f:
+        for line in f:
+            line = line.strip()
+            if line.startswith("GOOGLE_MAPS_API_KEY=") and "=" in line:
+                key = line.split("=", 1)[1].strip().strip('"').strip("'")
+                if key:
+                    GOOGLE_MAPS_API_KEY = key
+                break
+
+# City/place -> (lat, lon) for map pins (fast lookup). Add more as needed.
+CITY_COORDS = {
+    "boston": (42.36, -71.06),
+    "belmont": (42.40, -71.18),
+    "roxbury": (42.33, -71.08),
+    "allston": (42.35, -71.13),
+    "amesbury": (42.86, -70.93),
+    "austin": (30.27, -97.74),
+    "san antonio": (29.42, -98.49),
+    "san francisco": (37.77, -122.42),
+    "los angeles": (34.05, -118.25),
+    "lake view terrace": (34.27, -118.37),
+    "chicago": (41.88, -87.63),
+}
+# Geocode cache so any city/state from search results can be shown on the map.
+_GEOCODE_CACHE = {}
+# Show all proposed facilities (chat returns up to 5; allow more for geocoding).
+_MAX_GEOCODE_PER_MAP = 20
+
+MAP_HEIGHT_PX = 420
+
+
+def _geocode(location_str):
+    """Resolve address/city to (lat, lon) using Nominatim. Returns None on failure. Uses cache."""
+    if not location_str or not location_str.strip():
+        return None
+    key = location_str.strip().lower()
+    if key in _GEOCODE_CACHE:
+        return _GEOCODE_CACHE[key]
+    try:
+        from geopy.geocoders import Nominatim
+        from geopy.extra.rate_limiter import RateLimiter
+        geocoder = Nominatim(user_agent="samhsa-treatment-locator")
+        geocode = RateLimiter(geocoder.geocode, min_delay_seconds=1.0)
+        result = geocode(location_str.strip(), country_codes="us")
+        if result:
+            coord = (result.latitude, result.longitude)
+            _GEOCODE_CACHE[key] = coord
+            return coord
+    except Exception:
+        pass
+    return CITY_COORDS.get(key)
+
+
+def _decode_google_polyline(encoded: str):
+    """Decode Google's encoded polyline string to list of (lat, lon)."""
+    # https://developers.google.com/maps/documentation/utilities/polylinealgorithm
+    coords = []
+    i = 0
+    lat = 0
+    lon = 0
+    while i < len(encoded):
+        b = 0
+        shift = 0
+        result = 0
+        while True:
+            b = ord(encoded[i]) - 63
+            i += 1
+            result |= (b & 0x1F) << shift
+            shift += 5
+            if b < 0x20:
+                break
+        dlat = ~(result >> 1) if result & 1 else result >> 1
+        lat += dlat
+        shift = 0
+        result = 0
+        while True:
+            b = ord(encoded[i]) - 63
+            i += 1
+            result |= (b & 0x1F) << shift
+            shift += 5
+            if b < 0x20:
+                break
+        dlon = ~(result >> 1) if result & 1 else result >> 1
+        lon += dlon
+        coords.append((lat * 1e-5, lon * 1e-5))
+    return coords
+
+
+def _get_route(lat1, lon1, lat2, lon2):
+    """Return list of (lat, lon) for driving route. Uses Google Directions if GOOGLE_MAPS_API_KEY set, else OSRM."""
+    if GOOGLE_MAPS_API_KEY:
+        try:
+            url = (
+                "https://maps.googleapis.com/maps/api/directions/json"
+                f"?origin={lat1},{lon1}&destination={lat2},{lon2}&key={GOOGLE_MAPS_API_KEY}"
+            )
+            r = requests.get(url, timeout=10)
+            if r.status_code != 200:
+                return _get_route_osrm(lat1, lon1, lat2, lon2)
+            data = r.json()
+            if data.get("status") != "OK" or not data.get("routes"):
+                return _get_route_osrm(lat1, lon1, lat2, lon2)
+            points = data["routes"][0].get("overview_polyline", {}).get("points")
+            if points:
+                return _decode_google_polyline(points)
+        except Exception:
+            pass
+        return _get_route_osrm(lat1, lon1, lat2, lon2)
+    return _get_route_osrm(lat1, lon1, lat2, lon2)
+
 
-def create_chatbot():
+def _get_route_osrm(lat1, lon1, lat2, lon2):
+    """Return list of (lat, lon) for driving route from OSRM (free), or None."""
+    try:
+        url = (
+            "https://router.project-osrm.org/route/v1/driving/"
+            f"{lon1},{lat1};{lon2},{lat2}?overview=full&geometries=geojson"
+        )
+        r = requests.get(url, timeout=5)
+        if r.status_code != 200:
+            return None
+        data = r.json()
+        if not data.get("routes"):
+            return None
+        coords = data["routes"][0]["geometry"]["coordinates"]
+        return [(c[1], c[0]) for c in coords]
+    except Exception:
+        return None
+
+
+def _facility_coord(f, geocode_count=None):
+    """(lat, lon) for a facility: CITY_COORDS first, then geocode city+state (cached). f uses CSV keys."""
+    city = (f.get("city") or "").strip()
+    state = (f.get("state") or "").strip()
+    if not city and not state:
+        return None
+    city_lower = city.lower()
+    state_lower = state.lower()
+    coord = CITY_COORDS.get(city_lower) or (CITY_COORDS.get(state_lower) if state_lower else None)
+    if coord:
+        return coord
+    location_str = f"{city}, {state}".strip(", ")
+    if not location_str:
+        return None
+    if geocode_count is not None and len(geocode_count) == 1 and location_str.lower() not in _GEOCODE_CACHE:
+        if geocode_count[0] >= _MAX_GEOCODE_PER_MAP:
+            return None
+        geocode_count[0] += 1
+    return _geocode(location_str)
+
+
+def _popup_html(f):
+    """Short HTML for Folium popup. f uses CSV keys (facility_name, address, etc.)."""
+    name = f.get("facility_name") or f.get("name") or "Facility"
+    addr = f.get("address") or ""
+    city = f.get("city") or ""
+    st = f.get("state") or ""
+    phone = f.get("phone") or ""
+    t = f.get("treatment_type") or ""
+    parts = [f"<b>{name}</b>", f"{addr}, {city} {st}".strip(", ")]
+    if phone:
+        parts.append(f"📞 {phone}")
+    if t:
+        parts.append(f"Type: {t}")
+    return "<br>".join(parts)
+
+
+def _get_facility_coords(facilities):
+    """Return list of (lat, lon, facility_dict) for facilities that have coordinates."""
+    result = []
+    geocode_count = [0]
+    for f in facilities:
+        if not isinstance(f, dict):
+            continue
+        coord = _facility_coord(f, geocode_count)
+        if coord:
+            result.append((coord[0], coord[1], f))
+    return result
+
+
+def _build_google_map_html(facilities, force_update_id=None, selected_facility_name=None):
+    """Build Google Maps in an iframe via srcdoc so scripts run (interactive map). Requires GOOGLE_MAPS_API_KEY."""
+    if not GOOGLE_MAPS_API_KEY:
+        return _build_folium_map_html(facilities, None, force_update_id, selected_facility_name)
+    try:
+        facility_coords = _get_facility_coords(facilities)
+        center_lat, center_lon, zoom = 39.5, -98.5, 3
+        if facility_coords:
+            lats = [c[0] for c in facility_coords]
+            lons = [c[1] for c in facility_coords]
+            center_lat = sum(lats) / len(lats)
+            center_lon = sum(lons) / len(lons)
+            zoom = 10
+        markers_data = []
+        for lat, lon, f in facility_coords:
+            name = (f.get("facility_name") or f.get("name") or "Facility").replace("<", "&lt;").replace(">", "&gt;")
+            info = _popup_html(f)
+            sel = selected_facility_name and (f.get("facility_name") or f.get("name") or "") == selected_facility_name
+            markers_data.append({"lat": lat, "lng": lon, "name": name, "info": info, "selected": sel})
+        # Base64-encode markers so srcdoc HTML escaping cannot break the JSON
+        markers_json = json.dumps(markers_data)
+        markers_b64 = base64.b64encode(markers_json.encode("utf-8")).decode("ascii")
+        script_url = f"https://maps.googleapis.com/maps/api/js?key={GOOGLE_MAPS_API_KEY}&callback=init"
+        doc = f"""<!DOCTYPE html><html><head><meta charset="utf-8"></head><body style="margin:0">
+<div id="map" style="width:100%;height:100%;min-height:{MAP_HEIGHT_PX}px;"></div>
+<script>
+var center = {{ lat: {center_lat}, lng: {center_lon} }};
+var zoom = {zoom};
+var markersData = JSON.parse(atob("{markers_b64}"));
+function init() {{
+  var map = new google.maps.Map(document.getElementById("map"), {{ center: center, zoom: zoom, mapTypeControl: true, fullscreenControl: true, zoomControl: true, scaleControl: true }});
+  var infowindow = new google.maps.InfoWindow();
+  var bounds = null;
+  if (markersData && markersData.length) {{
+    markersData.forEach(function(m) {{
+      var pos = {{ lat: m.lat, lng: m.lng }};
+      var opts = {{ position: pos, map: map, title: m.name }};
+      if (m.selected) {{
+        opts.icon = {{ path: google.maps.SymbolPath.CIRCLE, scale: 14, fillColor: "#c62828", fillOpacity: 1, strokeColor: "#fff", strokeWeight: 3 }};
+      }}
+      var marker = new google.maps.Marker(opts);
+      marker.addListener("click", function() {{ infowindow.setContent(m.info); infowindow.open(map, marker); }});
+      if (!bounds) bounds = new google.maps.LatLngBounds(pos, pos);
+      else bounds.extend(pos);
+    }});
+    if (bounds && markersData.length > 0) {{
+      map.fitBounds(bounds, {{ top: 40, right: 40, bottom: 40, left: 40 }});
+      if (markersData.length === 1) map.setZoom(12);
+    }}
+  }}
+}}
+</script>
+<script src="{script_url}" async defer></script>
+</body></html>"""
+        # Embed in iframe via srcdoc so the document runs in its own context and scripts execute
+        escaped = html_module.escape(doc, quote=True)
+        html = f'<iframe srcdoc="{escaped}" style="width:100%;height:{MAP_HEIGHT_PX}px;border:0;border-radius:12px;" title="Google Map"></iframe>'
+        if force_update_id is not None:
+            html += f"<!-- map-update:{force_update_id} -->"
+        return html
+    except Exception as e:
+        return (
+            f'<div style="width:100%;height:{MAP_HEIGHT_PX}px;display:flex;align-items:center;justify-content:center;'
+            f'background:#f5f5f5;border-radius:12px;color:#666;">'
+            f'Map could not be loaded. ({str(e)[:80]})</div>'
+        )
+
+
+def _build_map_html(facilities, user_location_str=None, force_update_id=None, selected_facility_name=None):
+    """Build map HTML: Google Maps (iframe) when key is set, else Folium/OSM."""
+    if GOOGLE_MAPS_API_KEY:
+        return _build_google_map_html(facilities, force_update_id, selected_facility_name)
+    return _build_folium_map_html(facilities, user_location_str, force_update_id, selected_facility_name)
+
+
+def _build_folium_map_html(facilities, user_location_str=None, force_update_id=None, selected_facility_name=None):
     """
-    Creates and configures the chatbot interface.
+    Build Folium (Leaflet) map as HTML. Scroll over map to zoom, drag to pan.
+    If user_location_str is set, center on it and show a route to the first facility.
+    selected_facility_name: if set, the matching facility is shown with a red star icon.
+    force_update_id: optional unique value (e.g. timestamp) so Gradio re-renders the HTML each time.
+    Returns safe fallback HTML on any error.
     """
+    try:
+        center_lat, center_lon, zoom = 39.5, -98.5, 3
+        user_lat_lon = _geocode(user_location_str) if user_location_str else None
+        facility_coords = [(c[0], c[1], c[2]) for c in _get_facility_coords(facilities)]
+        facility_coords = [((c[0], c[1]), c[2]) for c in facility_coords]
+
+        if user_lat_lon:
+            center_lat, center_lon = user_lat_lon
+            zoom = 11
+        elif facility_coords:
+            lats = [c[0] for c, _ in facility_coords]
+            lons = [c[1] for c, _ in facility_coords]
+            center_lat = sum(lats) / len(lats)
+            center_lon = sum(lons) / len(lons)
+            zoom = 10
+
+        m = folium.Map(
+            location=[center_lat, center_lon],
+            zoom_start=zoom,
+            tiles="OpenStreetMap",
+            control_scale=True,
+            zoom_control=True,
+        )
+        m.options["scrollWheelZoom"] = True
+        m.options["touchZoom"] = True
+        m.options["dragging"] = True
+
+        if user_lat_lon:
+            folium.Marker(
+                user_lat_lon,
+                popup="You",
+                tooltip="Your location",
+                icon=folium.Icon(color="blue", icon="info-sign"),
+            ).add_to(m)
+        if user_lat_lon and facility_coords:
+            dest_lat, dest_lon = facility_coords[0][0][0], facility_coords[0][0][1]
+            route = _get_route(user_lat_lon[0], user_lat_lon[1], dest_lat, dest_lon)
+            if route:
+                folium.PolyLine(route, color="teal", weight=4, opacity=0.8).add_to(m)
+
+        for (lat, lon), f in facility_coords:
+            is_selected = selected_facility_name and (f.get("facility_name") or f.get("name") or "") == selected_facility_name
+            folium.Marker(
+                [lat, lon],
+                popup=folium.Popup(_popup_html(f), max_width=280),
+                tooltip=f.get("facility_name") or f.get("name") or "Facility",
+                icon=folium.Icon(color="red" if is_selected else "green", icon="star" if is_selected else "plus-sign"),
+            ).add_to(m)
+
+        # Center and zoom to show all proposed locations
+        if facility_coords:
+            lats = [c[0] for c, _ in facility_coords]
+            lons = [c[1] for c, _ in facility_coords]
+            sw = [min(lats), min(lons)]
+            ne = [max(lats), max(lons)]
+            # Avoid zero-size bounds (single marker): add small buffer
+            buf = 0.01
+            if ne[0] - sw[0] < buf:
+                sw[0] -= buf
+                ne[0] += buf
+            if ne[1] - sw[1] < buf:
+                sw[1] -= buf
+                ne[1] += buf
+            m.fit_bounds([sw, ne])
+
+        html = m._repr_html_()
+        wrapper = f'<div style="width:100%;height:{MAP_HEIGHT_PX}px;overflow:hidden;border-radius:12px;">{html}</div>'
+        # Force Gradio to re-render: append a unique comment so the value always changes
+        if force_update_id is not None:
+            wrapper += f"<!-- map-update:{force_update_id} -->"
+        return wrapper
+    except Exception as e:
+        return (
+            f'<div style="width:100%;height:{MAP_HEIGHT_PX}px;display:flex;align-items:center;justify-content:center;'
+            f'background:#f5f5f5;border-radius:12px;color:#666;font-family:sans-serif;">'
+            f'Map could not be loaded. ({str(e)[:80]})</div>'
+        )
+
+DISCLAIMER = (
+    "**Disclaimer:** Information is from SAMHSA data. Always verify with the facility or "
+    "[findtreatment.gov](https://findtreatment.gov) before making decisions. This tool does not endorse any facility."
+)
+
+DESCRIPTION = (
+    "Find treatment facilities by chatting: say where you are (city or state), what type of care you need, "
+    "and payment (e.g. Medicaid, insurance). Results show on the map. Data from SAMHSA only."
+)
+
+EXAMPLES = [
+    "I'm looking for outpatient alcohol treatment in Boston with Medicaid.",
+    "Do you have options for veterans in Texas?",
+    "Hi, I need help finding a residential program in California that accepts Medicaid.",
+]
+
+CSS = """
+.disclaimer { font-size: 0.85em; color: #555; padding: 0.5rem 0.75rem; background: #f8f9fa; border-radius: 8px; margin-bottom: 0.75rem; }
+.map-pane { padding: 0.25rem 0 0 0; }
+.map-pane .map-html { border-radius: 12px; overflow: hidden; box-shadow: 0 2px 12px rgba(0,0,0,0.08); }
+.map-pane iframe { border-radius: 12px; }
+.try-label { font-size: 0.9em; margin-bottom: 0.25rem; }
+"""
+
+
+def _messages_to_tuples(history):
+    """Convert Gradio 6 messages format to [(user, assistant), ...] for get_response."""
+    if not history:
+        return []
+    out = []
+    for item in history:
+        if isinstance(item, (list, tuple)) and len(item) >= 2:
+            out.append([item[0], item[1]])
+        elif isinstance(item, dict):
+            role, content = item.get("role"), item.get("content", "")
+            if role == "user":
+                out.append([content, ""])
+            elif role == "assistant":
+                if out:
+                    out[-1][1] = content
+                else:
+                    out.append(["", content])
+        else:
+            out.append(["", str(item)])
+    return out
+
+
+def _tuples_to_messages(history):
+    """Convert [(user, assistant), ...] to Gradio 6 messages format."""
+    out = []
+    for user_msg, assistant_msg in history or []:
+        if user_msg:
+            out.append({"role": "user", "content": user_msg})
+        if assistant_msg:
+            out.append({"role": "assistant", "content": assistant_msg})
+    return out
+
+
+def create_demo():
     chatbot = Chatbot()
-    
-    def chat(message, history):
-        """
-        TODO:Generate a response for the current message in a Gradio chat interface.
-        
-        This function is called by Gradio's ChatInterface every time a user sends a message.
-        You only need to generate and return the assistant's response - Gradio handles the
-        chat display and history management automatically.
-
-        Args:
-            message (str): The current message from the user
-            history (list): List of previous message pairs, where each pair is
-                           [user_message, assistant_message]
-                           Example:
-                           [
-                               ["What schools offer Spanish?", "The Hernandez School..."],
-                               ["Where is it located?", "The Hernandez School is in Roxbury..."]
-                           ]
-
-        Returns:
-            str: The assistant's response to the current message.
-
-
-        Note:
-            - Gradio automatically:
-                - Displays the user's message
-                - Displays your returned response
-                - Updates the chat history
-                - Maintains the chat interface
-            - You only need to:
-                - Generate an appropriate response to the current message
-                - Return that response as a string
-        """
-        # TODO: Generate and return response
-        return chatbot.get_response(message)
-    
-    
-    # Create Gradio interface. Customize the interface however you'd like!
-    demo = gr.ChatInterface(
-        chat,
-        title="6.C395",
-        description="Ask me anything about [topic]! Since I am a free tier chatbot, I may give a 503 error when I'm busy. If that happens, please try again a few seconds later.",
-        examples=[
-            "What options are available for someone in my situation?"
-        ]
-    )
-    
+
+    with gr.Blocks(title="SAMHSA Treatment Locator") as demo:
+        gr.Markdown("# SAMHSA Treatment Locator")
+        gr.Markdown(DESCRIPTION)
+        gr.Markdown(f"<div class='disclaimer'>{DISCLAIMER}</div>", elem_classes=["disclaimer"])
+
+        state = gr.State(DEFAULT_STATE)
+
+        with gr.Row():
+            # Left: Folium (Leaflet) map — scroll over map to zoom, drag to pan
+            with gr.Column(scale=5, min_width=320, elem_classes=["map-pane"]):
+                gr.Markdown("**Map** — scroll over map to zoom, drag to pan. Search in chat to see facilities.")
+                map_html = gr.HTML(
+                    value=_build_map_html([], None),
+                    elem_classes=["map-html"],
+                )
+            # Right: chat
+            with gr.Column(scale=5, min_width=320):
+                gr.Markdown("**Chat** — tell me location, treatment type, and payment.")
+                chat = gr.Chatbot(
+                    label="Conversation",
+                    placeholder="E.g. I'm in Boston, need outpatient treatment with Medicaid.",
+                    height=420,
+                    show_label=False,
+                )
+                facility_dropdown = gr.Dropdown(
+                    choices=[],
+                    value=None,
+                    label="Choose a treatment center (pin updates on map)",
+                    allow_custom_value=False,
+                )
+                with gr.Row():
+                    msg = gr.Textbox(
+                        placeholder="Type a message…",
+                        show_label=False,
+                        container=False,
+                        scale=8,
+                    )
+                    submit_btn = gr.Button("Send", variant="primary", scale=1)
+                gr.Markdown("**Try:**", elem_classes=["try-label"])
+                gr.Examples(
+                    examples=EXAMPLES,
+                    inputs=msg,
+                    label=None,
+                    examples_per_page=6,
+                )
+
+        def _facility_names(facilities):
+            return [f.get("facility_name") or f.get("name") or "Facility" for f in facilities]
+
+        def user_submit(message, history, state):
+            update_id = str(time.time())
+            if not message or not message.strip():
+                facilities = list(state.get("last_results") or [])
+                sel = state.get("selected_facility_name")
+                map_html_out = _build_map_html(facilities, None, update_id, sel)
+                return history, state, "", map_html_out, gr.update(choices=_facility_names(facilities))
+            try:
+                history_tuples = _messages_to_tuples(history)
+                reply, new_state = chatbot.get_response(message, history_tuples, state)
+                new_state = dict(new_state)
+                new_state["selected_facility_name"] = None  # clear selection when new results
+                new_history_tuples = history_tuples + [[message, reply]]
+                new_history_messages = _tuples_to_messages(new_history_tuples)
+                facilities = list(new_state.get("last_results") or [])
+                map_html_out = _build_map_html(facilities, None, update_id, None)
+                return new_history_messages, new_state, "", map_html_out, gr.update(choices=_facility_names(facilities), value=None)
+            except Exception as e:
+                err_msg = str(e)[:200]
+                reply = f"Sorry, something went wrong: {err_msg}"
+                if "token" in err_msg.lower() or "auth" in err_msg.lower():
+                    reply += " Check that HF_TOKEN is set in .env for the chat model."
+                history_tuples = _messages_to_tuples(history)
+                new_history_tuples = history_tuples + [[message, reply]]
+                new_history_messages = _tuples_to_messages(new_history_tuples)
+                facilities = list(state.get("last_results") or [])
+                sel = state.get("selected_facility_name")
+                map_html_out = _build_map_html(facilities, None, update_id, sel)
+                return new_history_messages, state, "", map_html_out, gr.update()
+
+        def on_facility_select(choice, state):
+            if not choice:
+                state = dict(state or {})
+                state["selected_facility_name"] = None
+                facilities = list(state.get("last_results") or [])
+                map_html_out = _build_map_html(facilities, None, str(time.time()), None)
+                return map_html_out, state
+            state = dict(state or {})
+            state["selected_facility_name"] = choice
+            facilities = list(state.get("last_results") or [])
+            map_html_out = _build_map_html(facilities, None, str(time.time()), choice)
+            return map_html_out, state
+
+        submit_btn.click(
+            user_submit,
+            inputs=[msg, chat, state],
+            outputs=[chat, state, msg, map_html, facility_dropdown],
+        )
+        msg.submit(
+            user_submit,
+            inputs=[msg, chat, state],
+            outputs=[chat, state, msg, map_html, facility_dropdown],
+        )
+        facility_dropdown.change(
+            on_facility_select,
+            inputs=[facility_dropdown, state],
+            outputs=[map_html, state],
+        )
+
     return demo
 
+
 if __name__ == "__main__":
-    demo = create_chatbot()
-    demo.launch()
+    demo = create_demo()
+    demo.launch(
+        theme=gr.themes.Soft(primary_hue="teal", secondary_hue="slate"),
+        css=CSS,
+    )

data/README.md

@@ -0,0 +1,17 @@
+# Facility Data – Data Story (for memo)
+
+## Source
+
+- **Dataset:** N-SUMHSS (National Substance Use and Mental Health Services Survey) / National Directory of Drug and Alcohol Use Treatment Facilities. This is the same data that powers [FindTreatment.gov](https://findtreatment.gov).
+- **Where:** SAMHSA CBHSQ Data – [N-SUMHSS data files](https://www.samhsa.gov/data/data-we-collect/n-sumhss-national-substance-use-and-mental-health-services-survey/datafiles) (SAS/CSV). National Directory also available as Excel/PDF from [National Directories](https://www.samhsa.gov/data/data-we-collect/n-sumhss-national-substance-use-and-mental-health-services-survey/national-directories).
+- **Processing:** For development and demo, `facilities.csv` may be a small subset. **To use all data from FindTreatment.gov**, run: `pip install -r requirements.txt` then `python scripts/download_findtreatment_data.py`. That script downloads the official SAMHSA National Directory (same data as FindTreatment.gov) and builds `data/facilities.csv`. Alternatively, download the Excel/CSV from SAMHSA yourself and run `python scripts/ingest_facilities.py path/to/file.xlsx -o data/facilities.csv`. The ingest script maps source columns to the internal schema; see the script and N-SUMHSS codebook for variable mapping.
+
+## Scope
+
+- **Geography:** Sample includes facilities in MA (Boston area), TX, CA, IL. Full N-SUMHSS covers all states.
+- **Attributes:** Facility name, address, city, state, zip, phone; treatment type (inpatient, outpatient, residential, telehealth); payment options (Medicaid/MassHealth, insurance, sliding scale, free, VA); MAT (medication-assisted treatment); services; **substances addressed** (e.g. alcohol, opioids); languages; populations (e.g. adults, adolescents, veterans, LGBTQ+, pregnant women); description. The chatbot helps users describe their situation and filters by these attributes.
+
+## Limitations
+
+- Data as of survey/publication date; facility details (phone, hours, availability) may have changed. Always confirm with the provider or [findtreatment.gov](https://findtreatment.gov) before making decisions.
+

docs/MEMO.md

@@ -0,0 +1,57 @@
+# SAMHSA Treatment Locator Chatbot – Memo
+
+## Design
+
+The chatbot helps users find substance use and mental health treatment facilities in the U.S. through a conversational flow. It works as follows:
+
+1. **Data** – Facility records are loaded from a CSV (see Data section) and stored in memory.
+2. **Criteria extraction** – From each user message, the system extracts location (state or city), treatment type (inpatient, outpatient, residential, telehealth), payment (Medicaid, insurance, sliding scale, free, veterans), MAT (medication-assisted treatment), populations (e.g. veterans, adolescents, LGBTQ+, pregnant women), languages (e.g. Spanish), substances (e.g. alcohol, opioids), and therapies (e.g. CBT, 12-step).
+3. **Search** – When at least a location is present, the backend runs a search over the facility data and returns only facilities that match all provided criteria.
+4. **Response** – The model receives the **current conversation history** plus **only the real search results** (or a single facility record for follow-up questions). It never receives fabricated data, so it cannot invent facility names, addresses, or phone numbers. The system prompt enforces conversation phases: greet/clarify (ask for location, type, payment), first results (2–3 facilities with short descriptions), follow-up (answer from the facility record only), and closing (supportive sign-off).
+
+This design avoids hallucination by construction: the model is restricted to describing facilities that appear in the provided data.
+
+---
+
+## Data
+
+- **Source:** N-SUMHSS (National Substance Use and Mental Health Services Survey) / National Directory of Drug and Alcohol Use Treatment Facilities. Data files are available from SAMHSA CBHSQ (e.g. [N-SUMHSS data files](https://www.samhsa.gov/data/data-we-collect/n-sumhss-national-substance-use-and-mental-health-services-survey/datafiles)); the National Directory is also available as Excel/PDF from SAMHSA’s National Directories page.
+- **Processing:** The app uses a CSV of facilities with non-missing location (city, state). For development and demo, the repo includes a subset; in production this can be replaced with the full N-SUMHSS export (e.g. SAS converted to CSV) using the same column mapping in `src/facilities.py`.
+- **Scope:** Data is aligned with FindTreatment.gov (source: N-SUMHSS/National Directory). Attributes include facility name, address, city, state, zip, phone; treatment type; payment options; MAT; services; substances addressed; languages; populations; description. The chatbot helps users describe their situation and find facilities that match their needs (treatment type, substances, payment, special populations, therapies, languages). The sample data covers multiple states (e.g. MA, TX, CA, IL); the full dataset covers all states.
+- **Limitations:** Data are as of the survey/publication date. Facility details (phone, hours, availability) may have changed. Users should always confirm with the provider or [findtreatment.gov](https://findtreatment.gov) before making decisions.
+
+---
+
+## Evaluation
+
+**Method:** We defined 18 test scenarios covering a variety of locations (Boston, Texas, California, Illinois), treatment types (outpatient, residential, inpatient), payment (Medicaid, sliding scale, veterans), and special populations (veterans). For each scenario we:
+
+1. Run the backend **search** with the scenario’s criteria and record which facilities are returned.
+2. **Match check:** Verify that every returned facility satisfies the scenario’s criteria (e.g. accepts Medicaid, offers outpatient). We report how many runs had all suggested facilities matching (e.g. “18/18”).
+3. **Hallucination check (optional):** When running the full chatbot with the API, we parse the bot’s reply for facility names and verify that each name appears in the dataset. Target: 0 invented facilities.
+
+**Artifact:** The script `scripts/eval_chatbot.py` runs these scenarios and prints a table: scenario, facilities returned, count, all match? (Y/N), and (if run with `--with-chatbot`) hallucination? (Y/N). Example:
+
+```
+Scenario                                Count  All match? Hallucination?
+------------------------------------------------------------------------
+Outpatient, Boston, Medicaid            3      Y          N
+Veterans, Texas                          1      Y          N
+...
+Summary: 18/18 runs had all suggested facilities matching criteria.
+Hallucination: 18/18 runs had no invented facility names in the reply.
+```
+
+This table (or a summary) can be pasted into the memo or a report to make the “we do not provide inaccurate information” claim concrete.
+
+---
+
+## Limitations
+
+1. **Data freshness** – Facility information is as of the source dataset date. Phone numbers, hours, and availability may have changed; users should confirm with the facility or findtreatment.gov.
+2. **English-only** – The current interface and criteria extraction are in English. Expanding to other languages would require additional design and data (e.g. language attributes in the dataset).
+3. **No medical advice** – The tool only helps users find facilities; it does not provide clinical or medical advice. The UI includes a disclaimer to that effect and directs users to verify information with the provider.
+
+---
+
+*Optional figure:* A short dialogue snippet (e.g. user asks for outpatient in Boston with Medicaid; bot returns 2–3 named facilities with descriptions from the data) or a table of evaluation results can be included to illustrate design and evaluation.

requirements.txt

@@ -2,3 +2,11 @@
 # Keep this light so the build succeeds.
 huggingface_hub>=0.19.0
 python-dotenv>=1.0.0
+pandas>=2.0.0
+gradio>=4.0.0
+# For scripts/ingest_facilities.py Excel support (optional)
+openpyxl>=3.0.0
+plotly>=5.0.0
+folium>=0.15.0
+geopy>=2.4.0
+requests>=2.28.0

scripts/download_findtreatment_data.py

@@ -0,0 +1,80 @@
+"""
+Download the full facility dataset that powers FindTreatment.gov.
+
+SAMHSA publishes the National Directory of Drug and Alcohol Use Treatment
+Facilities (same data source as FindTreatment.gov). This script downloads the
+official Excel file and runs the ingest to produce data/facilities.csv so the
+chatbot uses all treatment centers, not just the sample in the repo.
+
+Usage:
+  python scripts/download_findtreatment_data.py
+
+Downloads to data/National_Directory_SU_2024.xlsx (or current year), then
+runs the ingest to write data/facilities.csv. Requires: requests, pandas, openpyxl.
+"""
+
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+DATA_DIR = REPO_ROOT / "data"
+
+# Official SAMHSA 2024 National Directory (substance use treatment facilities).
+# Same data that powers https://findtreatment.gov
+NATIONAL_DIRECTORY_URL = (
+    "https://www.samhsa.gov/data/sites/default/files/reports/rpt53015/"
+    "National%20Directory%20SU%202024_Final.xlsx"
+)
+DOWNLOAD_FILENAME = "National_Directory_SU_2024.xlsx"
+
+
+def download_file(url: str, dest: Path) -> None:
+    """Download url to dest using urllib (no extra deps)."""
+    try:
+        from urllib.request import urlretrieve
+        urlretrieve(url, dest)
+    except Exception as e:
+        # Fallback: try requests if available
+        try:
+            import requests
+            r = requests.get(url, timeout=60)
+            r.raise_for_status()
+            dest.write_bytes(r.content)
+        except ImportError:
+            raise RuntimeError(
+                f"Download failed: {e}. Install requests (pip install requests) and try again."
+            ) from e
+
+
+def main():
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    dest = DATA_DIR / DOWNLOAD_FILENAME
+    print(f"Downloading FindTreatment.gov dataset from SAMHSA...", file=sys.stderr)
+    print(f"  URL: {NATIONAL_DIRECTORY_URL}", file=sys.stderr)
+    try:
+        download_file(NATIONAL_DIRECTORY_URL, dest)
+    except Exception as e:
+        print(f"Error: {e}", file=sys.stderr)
+        sys.exit(1)
+    print(f"Saved to {dest}", file=sys.stderr)
+    # Run ingest to produce facilities.csv
+    print("Running ingest to build data/facilities.csv...", file=sys.stderr)
+    import subprocess
+    result = subprocess.run(
+        [sys.executable, str(REPO_ROOT / "scripts" / "ingest_facilities.py"), str(dest), "-o", str(DATA_DIR / "facilities.csv")],
+        cwd=str(REPO_ROOT),
+    )
+    if result.returncode != 0:
+        print(
+            "\nIngest failed (often due to missing openpyxl). Install dependencies:\n"
+            "  pip install -r requirements.txt\n"
+            "Then run ingest manually:\n"
+            f"  python scripts/ingest_facilities.py {dest} -o data/facilities.csv",
+            file=sys.stderr,
+        )
+        sys.exit(result.returncode)
+    print("Done. The chatbot now uses the full FindTreatment.gov dataset.", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

scripts/eval_chatbot.py

@@ -0,0 +1,207 @@
+"""
+Evaluation script for SAMHSA Treatment Locator chatbot.
+
+Runs 15–20 test scenarios. For each:
+- Runs search(criteria) to get facilities returned.
+- Hallucination check: if --with-chatbot, call the bot and verify every facility name
+  (and contact info) in the reply appears in the dataset. Target: 0 invented facilities.
+- Match check: verify returned facilities match the scenario criteria (e.g. accepts Medicaid, offers outpatient).
+
+Outputs a table: scenario, facilities returned, hallucination? (Y/N), all match? (Y/N).
+Use the table or summary in the memo.
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+# Project root
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from src.facilities import load_facilities, search
+
+# --- Scenarios: (description, criteria dict, optional user message for chatbot run) ---
+SCENARIOS = [
+    ("Outpatient, Boston, Medicaid", {"state": "ma", "location": "Boston", "treatment_type": "outpatient", "payment": "Medicaid"}, "I need outpatient treatment in Boston with Medicaid."),
+    ("Outpatient, Boston, MassHealth", {"state": "ma", "location": "Boston", "payment": "Medicaid"}, "Looking for outpatient in Boston with MassHealth."),
+    ("Outpatient, Boston, MAT", {"state": "ma", "location": "Boston", "treatment_type": "outpatient", "mat": True}, "Outpatient in Boston with medication-assisted treatment."),
+    ("Residential, Massachusetts", {"state": "ma", "treatment_type": "residential"}, "Residential treatment in Massachusetts."),
+    ("Veterans, Texas", {"state": "tx", "populations": "veterans", "payment": "veterans"}, "Do you have options for veterans in Texas?"),
+    ("Veterans, San Antonio", {"state": "tx", "location": "San Antonio", "populations": "veterans"}, "Veterans programs in San Antonio."),
+    ("Outpatient, Austin", {"state": "tx", "location": "Austin"}, "Outpatient substance use treatment in Austin."),
+    ("California, Medicaid", {"state": "ca", "payment": "Medicaid"}, "California facilities that accept Medicaid."),
+    ("California, residential", {"state": "ca", "treatment_type": "residential"}, "Residential treatment in California."),
+    ("San Francisco, outpatient", {"state": "ca", "location": "San Francisco", "treatment_type": "outpatient"}, "Outpatient in San Francisco."),
+    ("Los Angeles area", {"state": "ca", "location": "Los Angeles"}, "Treatment options in Los Angeles area."),
+    ("Chicago, outpatient", {"state": "il", "location": "Chicago", "treatment_type": "outpatient"}, "Outpatient in Chicago."),
+    ("Chicago, MAT", {"state": "il", "location": "Chicago", "mat": True}, "Chicago programs with MAT."),
+    ("Illinois, Medicaid", {"state": "il", "payment": "Medicaid"}, "Illinois facilities accepting Medicaid."),
+    ("Boston, sliding scale", {"state": "ma", "location": "Boston", "payment": "sliding scale"}, "Boston programs with sliding scale fees."),
+    ("Outpatient, Boston, Spanish", {"state": "ma", "location": "Boston", "treatment_type": "outpatient", "languages": "Spanish"}, "Outpatient in Boston, Spanish-speaking."),
+    ("Residential, Texas", {"state": "tx", "treatment_type": "residential"}, "Residential treatment in Texas."),
+    ("MA, inpatient", {"state": "ma", "treatment_type": "inpatient"}, "Inpatient treatment in MA."),
+    ("Boston, alcohol", {"state": "ma", "location": "Boston", "substances": "alcohol"}, "Boston facilities for alcohol treatment."),
+    ("Chicago, opioids", {"state": "il", "location": "Chicago", "substances": "opioids"}, "Opioid treatment in Chicago."),
+    ("Boston, CBT", {"state": "ma", "location": "Boston", "therapies": "CBT"}, "Boston programs that offer CBT."),
+]
+
+# All facility names and phones from dataset (for hallucination check)
+def _all_facility_names_and_phones():
+    df = load_facilities()
+    names = set()
+    phones = set()
+    for _, row in df.iterrows():
+        n = row.get("facility_name")
+        if n and str(n).strip():
+            names.add(str(n).strip().lower())
+        p = row.get("phone")
+        if p and str(p).strip():
+            phones.add(str(p).strip())
+    return names, phones
+
+
+def _facility_matches_criteria(fac: dict, criteria: dict) -> bool:
+    """Check that a facility record matches the scenario criteria. Falls back to services when attribute column missing."""
+    def norm(s):
+        if s is None or (isinstance(s, float) and (s != s)):  # NaN
+            return ""
+        return str(s).lower().strip()
+
+    def col_or_services(col: str) -> str:
+        v = fac.get(col, "")
+        if v and str(v).strip():
+            return norm(v)
+        return norm(fac.get("services", ""))
+
+    state = criteria.get("state")
+    if state and norm(fac.get("state")) != norm(state):
+        return False
+    tt = criteria.get("treatment_type")
+    if tt and norm(tt) not in col_or_services("treatment_type"):
+        return False
+    pay = criteria.get("payment")
+    if pay:
+        pay_norm = norm(pay)
+        pop_text = col_or_services("populations")
+        pay_text = col_or_services("payment_options")
+        if pay_norm in ("veterans", "va"):
+            if "veteran" not in pop_text and "veteran" not in pay_text:
+                return False
+        elif pay_norm not in pay_text:
+            return False
+    if criteria.get("mat") is True and norm(fac.get("mat")) != "yes":
+        return False
+    pop = criteria.get("populations")
+    if pop and norm(pop) not in col_or_services("populations"):
+        return False
+    lang = criteria.get("languages")
+    if lang and norm(lang) not in col_or_services("languages"):
+        return False
+    substances = criteria.get("substances")
+    if substances and norm(substances) not in col_or_services("substances_addressed"):
+        return False
+    therapies = criteria.get("therapies")
+    if therapies:
+        t = norm(therapies)
+        svc = norm(fac.get("services", ""))
+        if t == "cbt":
+            if "cbt" not in svc:
+                return False
+        elif "12" in t or "twelve" in t:
+            if "12-step" not in svc and "12 step" not in svc:
+                return False
+        elif t not in svc:
+            return False
+    return True
+
+
+def _extract_facility_names_from_text(text: str) -> list[str]:
+    """Heuristic: find likely facility names in bot reply (e.g. 'X —' or 'X —' or numbered list '1. X —')."""
+    if not text:
+        return []
+    # Split on common delimiters and look for title-case or known patterns
+    names = set()
+    # Pattern: "1. Facility Name —" or "Facility Name —" or "- Facility Name"
+    for part in re.split(r"\n|\.|;", text):
+        m = re.match(r"^(?:\d+\.?\s*)?([A-Za-z][^—\-:]*?)(?:\s*[—\-:]|$)", part.strip())
+        if m:
+            cand = m.group(1).strip()
+            if len(cand) > 5 and cand.lower() not in ("yes", "no", "that", "they", "there", "here", "would", "could", "please", "contact", "phone", "address"):
+                names.add(cand)
+    return list(names)
+
+
+def run_search_eval():
+    """Run search() for each scenario; compute facilities returned and all_match."""
+    df = load_facilities()
+    rows = []
+    for desc, criteria, _ in SCENARIOS:
+        results = search(criteria, df=df, limit=5)
+        names = [r.get("facility_name", "") for r in results if r.get("facility_name")]
+        all_match = all(_facility_matches_criteria(r, criteria) for r in results)
+        rows.append({
+            "scenario": desc,
+            "criteria": str(criteria),
+            "facilities_returned": "; ".join(names) if names else "(none)",
+            "count": len(results),
+            "all_match": "Y" if all_match else "N",
+        })
+    return rows
+
+
+def run_hallucination_check(rows, with_chatbot: bool):
+    """If with_chatbot, call the bot for each scenario and set hallucination? (Y/N)."""
+    for r in rows:
+        r["hallucination"] = "(skip)"
+    if not with_chatbot:
+        return rows
+    from src.chat import Chatbot
+    names_ok, _ = _all_facility_names_and_phones()
+    chatbot = Chatbot()
+    for i, (desc, criteria, user_msg) in enumerate(SCENARIOS):
+        reply, _ = chatbot.get_response(user_msg, [], {"criteria": {}, "last_results": [], "last_facility_detail": None})
+        mentioned = _extract_facility_names_from_text(reply)
+        hallucinated = False
+        for n in mentioned:
+            n_lower = n.lower()
+            if n_lower in names_ok:
+                continue
+            if any(n_lower in db for db in names_ok) or any(db in n_lower for db in names_ok):
+                continue
+            hallucinated = True
+            break
+        rows[i]["hallucination"] = "Y" if hallucinated else "N"
+    return rows
+
+
+def main():
+    ap = argparse.ArgumentParser(description="Evaluate SAMHSA chatbot: scenarios, match, optional hallucination check.")
+    ap.add_argument("--with-chatbot", action="store_true", help="Run chatbot and check for hallucinated facility names (requires API).")
+    ap.add_argument("--format", choices=["table", "csv"], default="table", help="Output format.")
+    args = ap.parse_args()
+
+    rows = run_search_eval()
+    rows = run_hallucination_check(rows, args.with_chatbot)
+
+    if args.format == "csv":
+        import csv
+        w = csv.DictWriter(sys.stdout, fieldnames=["scenario", "facilities_returned", "count", "all_match", "hallucination"])
+        w.writeheader()
+        w.writerows(rows)
+        return
+
+    # Table
+    print(f"{'Scenario':<40} {'Count':<6} {'All match?':<10} {'Hallucination?':<14}")
+    print("-" * 72)
+    for r in rows:
+        print(f"{r['scenario']:<40} {r['count']:<6} {r['all_match']:<10} {r.get('hallucination', '(n/a)'):<14}")
+    match_ok = sum(1 for r in rows if r["all_match"] == "Y")
+    print(f"\nSummary: {match_ok}/{len(rows)} runs had all suggested facilities matching criteria.")
+    if args.with_chatbot:
+        hall_ok = sum(1 for r in rows if r.get("hallucination") == "N")
+        print(f"Hallucination: {hall_ok}/{len(rows)} runs had no invented facility names in the reply.")
+
+
+if __name__ == "__main__":
+    main()

scripts/ingest_facilities.py

@@ -0,0 +1,433 @@
+"""
+Ingest facility data from N-SUMHSS / National Directory into app schema.
+
+Reads a CSV or Excel file (e.g. downloaded from SAMHSA N-SUMHSS or National
+Directory), maps columns to the internal schema using a configurable mapping,
+and writes data/facilities.csv. If the source uses codes, extend SOURCE_TO_APP
+or add a code-resolution step using the N-SUMHSS codebook.
+
+Usage:
+  python scripts/ingest_facilities.py [path_to_source.csv]
+  python scripts/ingest_facilities.py path/to/national_directory.xlsx
+
+If no path is given, reads from stdin (CSV) or exits with usage.
+Output: data/facilities.csv (same directory as this script: repo root/data/).
+"""
+
+import argparse
+import sys
+import warnings
+from pathlib import Path
+
+import pandas as pd
+
+# Repo root (parent of scripts/)
+REPO_ROOT = Path(__file__).resolve().parent.parent
+DATA_DIR = REPO_ROOT / "data"
+OUTPUT_CSV = DATA_DIR / "facilities.csv"
+
+# Internal schema: only columns we need (no duplicate/redundant attribute columns).
+# Search matches against "services" for treatment type, payment, languages, populations, substances.
+APP_COLUMNS = [
+    "facility_name",
+    "address",
+    "city",
+    "state",
+    "zip",
+    "phone",
+    "mat",
+    "services",
+]
+
+# Map source column names (lowercase) -> app column name.
+# N-SUMHSS / National Directory use different names; adjust per codebook.
+# National Directory Excel may use "Facility/Program Name", "Street", "City", "State", etc.
+# See data/README.md for the data story and mapping notes.
+SOURCE_TO_APP = {
+    "facility_name": "facility_name",
+    "facility name": "facility_name",
+    "facility/program name": "facility_name",
+    "program name": "facility_name",
+    "name": "facility_name",
+    "name1": "facility_name",
+    "name2": "facility_name",
+    "provider name": "facility_name",
+    "organization": "facility_name",
+    "treatment facility name": "facility_name",
+    "location name": "facility_name",
+    "facility": "facility_name",
+    "address": "address",
+    "street": "address",
+    "street address": "address",
+    "address1": "address",
+    "address line 1": "address",
+    "street1": "address",
+    "street2": "address",
+    "physical address": "address",
+    "location address": "address",
+    "city": "city",
+    "state": "state",
+    "state abbreviation": "state",
+    "zip": "zip",
+    "zipcode": "zip",
+    "zip code": "zip",
+    "phone": "phone",
+    "telephone": "phone",
+    "phone number": "phone",
+    "treatment_type": "treatment_type",
+    "treatment type": "treatment_type",
+    "type of care": "treatment_type",
+    "care type": "treatment_type",
+    "service setting": "treatment_type",
+    "treatment setting": "treatment_type",
+    "level of care": "treatment_type",
+    "payment_options": "payment_options",
+    "payment": "payment_options",
+    "payment options": "payment_options",
+    "payment accepted": "payment_options",
+    "accepted payment": "payment_options",
+    "insurance accepted": "payment_options",
+    "sliding fee": "payment_options",
+    "fee scale": "payment_options",
+    "mat": "mat",
+    "medication_assisted": "mat",
+    "medication assisted": "mat",
+    "medication assisted treatment": "mat",
+    "buprenorphine": "mat",
+    "services": "services",
+    "services offered": "services",
+    "service codes": "services",
+    "types of care": "services",
+    "substances_addressed": "substances_addressed",
+    "substances": "substances_addressed",
+    "substances addressed": "substances_addressed",
+    "primary focus": "substances_addressed",
+    "substance focus": "substances_addressed",
+    "drugs treated": "substances_addressed",
+    "languages": "languages",
+    "language": "languages",
+    "languages spoken": "languages",
+    "non-english languages": "languages",
+    "language services": "languages",
+    "populations": "populations",
+    "population": "populations",
+    "population served": "populations",
+    "special populations": "populations",
+    "ages served": "populations",
+    "age group": "populations",
+    "description": "description",
+    "comments": "description",
+    "notes": "description",
+}
+
+
+def _normalize_mat(val) -> str:
+    """Map various MAT values to yes/no."""
+    if pd.isna(val):
+        return ""
+    s = str(val).lower().strip()
+    if s in ("yes", "1", "true", "y"):
+        return "yes"
+    if s in ("no", "0", "false", "n", ""):
+        return "no"
+    return "yes" if "yes" in s or "offer" in s else "no"
+
+
+def load_code_key(path: str | Path) -> dict[str, str] | None:
+    """Load the code reference sheet from a National Directory Excel and return code -> description dict.
+    SAMHSA 2024 uses sheet 'Service Code Reference' with service_code and service_name columns.
+    """
+    path = Path(path)
+    if path.suffix.lower() not in (".xlsx", ".xls"):
+        return None
+    if not path.exists():
+        return None
+    with warnings.catch_warnings():
+        warnings.filterwarnings("ignore", message=".*Cannot parse header or footer.*")
+        xl = pd.ExcelFile(path)
+    key_df = None
+    for name in xl.sheet_names:
+        nlower = name.lower()
+        if "service code reference" in nlower or "code reference" in nlower or ("key" in nlower and "code" in nlower):
+            with warnings.catch_warnings():
+                warnings.filterwarnings("ignore", message=".*Cannot parse header or footer.*")
+                key_df = pd.read_excel(path, sheet_name=name)
+            break
+    if key_df is None:
+        for name in xl.sheet_names:
+            with warnings.catch_warnings():
+                warnings.filterwarnings("ignore", message=".*Cannot parse header or footer.*")
+                sheet = pd.read_excel(path, sheet_name=name)
+            if 2 <= len(sheet) <= 600 and len(sheet.columns) >= 2:
+                cols_lower = [str(c).lower() for c in sheet.columns]
+                if "service_code" in cols_lower and "service_name" in cols_lower:
+                    key_df = sheet
+                    break
+    if key_df is None or len(key_df) == 0:
+        return None
+    key_df.columns = [str(c).strip() for c in key_df.columns]
+    cols_lower = [c.lower() for c in key_df.columns]
+    code_col = None
+    desc_col = None
+    if "service_code" in cols_lower:
+        code_col = key_df.columns[cols_lower.index("service_code")]
+    if "service_name" in cols_lower:
+        desc_col = key_df.columns[cols_lower.index("service_name")]
+    if not code_col or not desc_col:
+        code_col = key_df.columns[0]
+        desc_col = key_df.columns[1] if len(key_df.columns) > 1 else key_df.columns[0]
+    code_key = {}
+    for _, row in key_df.iterrows():
+        k = str(row.get(code_col, "")).strip()
+        v = str(row.get(desc_col, "")).strip()
+        if k and v and k != "nan" and v != "nan" and len(k) <= 20:
+            code_key[k] = v
+    return code_key if code_key else None
+
+
+def _decode_service_codes(series: pd.Series, code_key: dict[str, str]) -> pd.Series:
+    """Replace code tokens with descriptions; join with ', '. Skip * and unknown tokens (only output decoded)."""
+    def decode_one(cell: str) -> str:
+        if pd.isna(cell) or not str(cell).strip():
+            return ""
+        parts = []
+        for token in str(cell).split():
+            token = token.strip()
+            if not token or token == "*":
+                continue
+            if token in code_key:
+                parts.append(code_key[token])
+        return ", ".join(parts) if parts else ""
+    return series.apply(decode_one)
+
+
+def load_source(path: str | Path) -> pd.DataFrame:
+    """Load CSV or Excel into a DataFrame with lowercase column names.
+    For Excel with multiple sheets (e.g. National Directory + Key), uses the
+    sheet that looks like facility data (has facility name or state, and many rows).
+    """
+    path = Path(path)
+    if not path.exists():
+        raise FileNotFoundError(path)
+    suf = path.suffix.lower()
+    if suf == ".csv":
+        df = pd.read_csv(path)
+    elif suf in (".xlsx", ".xls"):
+        # Suppress openpyxl header/footer parse warnings (harmless; SAMHSA Excel often has them)
+        with warnings.catch_warnings():
+            warnings.filterwarnings("ignore", message=".*Cannot parse header or footer.*")
+            xl = pd.ExcelFile(path)
+        if len(xl.sheet_names) == 1:
+            with warnings.catch_warnings():
+                warnings.filterwarnings("ignore", message=".*Cannot parse header or footer.*")
+                df = pd.read_excel(path)
+        else:
+            # Pick the sheet that has facility data: prefer one with a facility-name-like column and many rows
+            def sheet_has_facility_name_col(sheet: pd.DataFrame) -> bool:
+                cols_lower = [str(c).lower().strip() for c in sheet.columns]
+                if "facility name" in cols_lower or "facility_name" in cols_lower:
+                    return True
+                if "program name" in cols_lower or "facility/program name" in cols_lower:
+                    return True
+                if any(("facility" in c or "program" in c) and "name" in c for c in cols_lower):
+                    return True
+                if "organization" in cols_lower or "provider name" in cols_lower:
+                    return True
+                return False
+
+            best = None
+            best_score = -1
+            for name in xl.sheet_names:
+                with warnings.catch_warnings():
+                    warnings.filterwarnings("ignore", message=".*Cannot parse header or footer.*")
+                    sheet = pd.read_excel(path, sheet_name=name)
+                if len(sheet) < 10:
+                    continue
+                cols_lower = [str(c).lower().strip() for c in sheet.columns]
+                has_state_city = "state" in cols_lower and "city" in cols_lower
+                has_name_col = sheet_has_facility_name_col(sheet)
+                # Strongly prefer sheet that has a facility name column; then state/city; then row count
+                score = (1000 if has_name_col else 0) + (10 if has_state_city else 0) + min(len(sheet), 5000)
+                if score > best_score:
+                    best_score = score
+                    best = sheet
+            if best is not None:
+                df = best
+            else:
+                with warnings.catch_warnings():
+                    warnings.filterwarnings("ignore", message=".*Cannot parse header or footer.*")
+                    df = pd.read_excel(path, sheet_name=0)
+    else:
+        raise ValueError(f"Unsupported format: {suf}. Use .csv or .xlsx")
+    df.columns = [str(c).lower().strip() for c in df.columns]
+    return df
+
+
+def _guess_facility_name_column(df: pd.DataFrame, col_map: dict) -> str | None:
+    """If no facility_name mapping, find a column that likely holds facility/program name."""
+    if "facility_name" in col_map:
+        return None
+    for src_col in df.columns:
+        c = str(src_col).lower().strip()
+        if "program" in c and "name" in c:
+            return src_col
+        if "facility" in c and "name" in c:
+            return src_col
+        if c in ("organization", "provider name", "location name"):
+            return src_col
+        # First column is often the name in directory layouts
+        if list(df.columns)[0] == src_col and ("name" in c or "facility" in c or "program" in c):
+            return src_col
+    return None
+
+
+def _guess_address_column(df: pd.DataFrame, col_map: dict) -> str | None:
+    """If no address mapping, find a column that likely holds street address."""
+    if "address" in col_map:
+        return None
+    for src_col in df.columns:
+        c = str(src_col).lower().strip()
+        if "street" in c or ("address" in c and "line" in c):
+            return src_col
+        if c in ("physical address", "location address"):
+            return src_col
+    return None
+
+
+# Keywords to try when guessing unmapped columns (app_col -> list of substrings; any match in column name).
+_GUESS_COLUMN_KEYWORDS = {
+    "treatment_type": ["treatment type", "type of care", "care type", "service setting", "level of care", "setting"],
+    "payment_options": ["payment", "insurance", "fee", "sliding", "medicaid", "accepted payment"],
+    "services": ["services", "service codes", "types of care", "offered", "treatment modalities"],
+    "substances_addressed": ["substance", "primary focus", "drug", "alcohol", "opioid"],
+    "languages": ["language", "non-english", "spanish", "bilingual"],
+    "populations": ["population", "age", "special population", "veteran", "gender", "served"],
+    "description": ["description", "comments", "notes", "remarks"],
+}
+
+
+def _guess_column_by_keywords(df: pd.DataFrame, col_map: dict, app_col: str) -> str | None:
+    """If app_col not yet mapped, find a source column whose name contains any of the keywords."""
+    if app_col in col_map:
+        return None
+    keywords = _GUESS_COLUMN_KEYWORDS.get(app_col, [])
+    for src_col in df.columns:
+        c = str(src_col).lower().strip()
+        for kw in keywords:
+            if kw in c:
+                return src_col
+    return None
+
+
+def map_columns(df: pd.DataFrame) -> pd.DataFrame:
+    """Map source columns to app schema; add missing app columns as empty."""
+    out = {}
+    for app_col in APP_COLUMNS:
+        out[app_col] = []
+    # Find which source column maps to each app column
+    col_map = {}
+    for src_col in df.columns:
+        src_lower = str(src_col).lower().strip()
+        if src_lower in SOURCE_TO_APP:
+            app_col = SOURCE_TO_APP[src_lower]
+            if app_col not in col_map:
+                col_map[app_col] = src_col
+    # Fallbacks for National Directory Excel when headers differ
+    guess_name = _guess_facility_name_column(df, col_map)
+    if guess_name and "facility_name" not in col_map:
+        col_map["facility_name"] = guess_name
+    guess_addr = _guess_address_column(df, col_map)
+    if guess_addr and "address" not in col_map:
+        col_map["address"] = guess_addr
+    for app_col in ("treatment_type", "payment_options", "services", "substances_addressed", "languages", "populations", "description"):
+        guess = _guess_column_by_keywords(df, col_map, app_col)
+        if guess and app_col not in col_map:
+            col_map[app_col] = guess
+    for app_col in APP_COLUMNS:
+        if app_col in col_map:
+            out[app_col] = df[col_map[app_col]].astype(str).replace("nan", "").tolist()
+        else:
+            out[app_col] = [""] * len(df)
+    result = pd.DataFrame(out)
+
+    # National Directory format: merge name1+name2 -> facility_name, street1+street2 -> address
+    cols_lower = [str(c).lower().strip() for c in df.columns]
+    if "name1" in cols_lower and "name2" in cols_lower:
+        n1 = df["name1"].astype(str).replace("nan", "").str.strip()
+        n2 = df["name2"].astype(str).replace("nan", "").str.strip()
+        merged = (n1 + " " + n2).str.strip()
+        result["facility_name"] = merged.where(merged != "", result["facility_name"])
+    if "street1" in cols_lower and "street2" in cols_lower:
+        s1 = df["street1"].astype(str).replace("nan", "").str.strip()
+        s2 = df["street2"].astype(str).replace("nan", "").str.strip()
+        merged = (s1 + " " + s2).str.strip()
+        result["address"] = merged.where(merged != "", result["address"])
+    # service_code_info is decoded in main() using the Key sheet when available (see load_code_key).
+
+    # Normalize MAT to yes/no
+    if "mat" in result.columns:
+        result["mat"] = result["mat"].apply(_normalize_mat)
+    return result
+
+
+def drop_missing_location(df: pd.DataFrame) -> pd.DataFrame:
+    """Keep only rows with non-empty city and state."""
+    if "city" not in df.columns or "state" not in df.columns:
+        return df
+    return df[
+        df["city"].notna() & (df["city"].astype(str).str.strip() != "")
+        & df["state"].notna() & (df["state"].astype(str).str.strip() != "")
+    ].copy()
+
+
+def main():
+    ap = argparse.ArgumentParser(description="Ingest N-SUMHSS/National Directory data into facilities.csv")
+    ap.add_argument("source", nargs="?", help="Path to source CSV or Excel file. If omitted, print usage and exit.")
+    ap.add_argument("-o", "--output", default=str(OUTPUT_CSV), help="Output CSV path")
+    args = ap.parse_args()
+    if not args.source:
+        ap.print_help()
+        sys.exit(0)
+    path = Path(args.source)
+    raw_df = load_source(path)
+    df = map_columns(raw_df)
+    if df["facility_name"].str.strip().eq("").all():
+        print(
+            "Warning: no facility names were mapped. Source columns were:\n  "
+            + ", ".join(repr(c) for c in raw_df.columns),
+            file=sys.stderr,
+        )
+    # Decode service_code_info using the Key sheet; store only in services (search uses it for all filters).
+    if "service_code_info" in raw_df.columns and path.suffix.lower() in (".xlsx", ".xls"):
+        code_key = load_code_key(path)
+        if code_key:
+            decoded = _decode_service_codes(raw_df["service_code_info"], code_key)
+            df["services"] = decoded
+    # Report if services is still empty (couldn't decode)
+    empty_attrs = [c for c in ("services",) if c in df.columns and (df[c].astype(str).str.strip() == "").all()]
+    if empty_attrs and "service_code_info" in raw_df.columns:
+        print(
+            "Note: " + ", ".join(empty_attrs) + " had no data (source has coded service_code_info; "
+            "Key sheet not found or could not be parsed for decoding).",
+            file=sys.stderr,
+        )
+    elif empty_attrs:
+        print(
+            "Note: these attributes had no data after mapping: " + ", ".join(empty_attrs) + ".",
+            file=sys.stderr,
+        )
+    df = drop_missing_location(df)
+    # Deduplicate by facility_name + address + city + state (keep first occurrence)
+    key_cols = ["facility_name", "address", "city", "state"]
+    if all(c in df.columns for c in key_cols):
+        before = len(df)
+        df = df.drop_duplicates(subset=key_cols, keep="first").reset_index(drop=True)
+        if len(df) < before:
+            print(f"Dropped {before - len(df)} duplicate rows (same name+address+city+state).", file=sys.stderr)
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    df.to_csv(args.output, index=False)
+    print(f"Wrote {len(df)} rows to {args.output}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

src/chat.py

@@ -1,69 +1,257 @@
+"""
+Stateful SAMHSA Treatment Locator chatbot.
+
+Business logic: criteria extraction, search, response generation. No hallucination:
+only real facility data is passed to the model. Conversation design matches
+samhsa_chatbot_conversation_example.txt (greet/clarify → first results → follow-up → closing).
+"""
+
+import re
+from typing import Any
+
 from huggingface_hub import InferenceClient
-from config import BASE_MODEL, MY_MODEL, HF_TOKEN
+
+from config import BASE_MODEL, HF_TOKEN, MY_MODEL
+from src.facilities import get_facility_by_name, load_facilities, search
+
+# --- Conversation state (criteria + last results for context) ---
+DEFAULT_STATE = {
+    "criteria": {},
+    "last_results": [],
+    "last_facility_detail": None,
+    "selected_facility_name": None,
+}
+
+SYSTEM_PROMPT = """You are a supportive, non-judgmental assistant that helps people find substance use and mental health treatment facilities in the United States. You use only the facility information provided to you in this conversation—never invent facility names, addresses, phone numbers, or details. Your role is to help users describe their situation and find facilities that match their needs.
+
+Conversation flow:
+1. Greet / clarify: If the user has not yet given a location (state or city), ask for: (a) state or city, (b) treatment type (inpatient, outpatient, residential, telehealth), (c) payment (insurance, Medicaid/MassHealth, sliding scale, free), and as appropriate: substances they're concerned about (e.g. alcohol, opioids), special populations (veterans, LGBTQ+, adolescents, pregnant women), therapies (e.g. MAT, CBT, 12-step), and languages spoken. Do not search until you have at least a location.
+2. First results: When you have at least location (and ideally type and payment), present 2–3 facilities by name with 1–2 sentence descriptions using ONLY the data in the "Current facility data" section below. Mention relevant attributes (payment, languages, populations, substances, therapies) when they match what the user asked for. Offer to give more details or other options.
+3. Follow-up: If the user asks about a specific facility (e.g. "Do they offer MAT?" or "Tell me about Boston Medical Center"), answer ONLY from the facility record provided. Offer next steps (e.g. how to contact).
+4. Closing: If the user thanks you or says they're done, give a brief supportive close and invite them to return.
+
+Rules:
+- Never make up facility names, addresses, phones, or services. If the data does not say something, do not say it.
+- The "Services" field for each facility contains the full list of what they offer (treatment types, payment options, languages, populations, therapies, etc.). Use this field when describing what a facility offers or when answering follow-up questions (e.g. "Do they offer outpatient?", "Do they take Medicaid?").
+- Keep responses concise and actionable.
+- Be supportive and clear. Do not give medical advice.
+- If no location has been provided, ask for location before suggesting any facilities.
+"""
+
+
+def _extract_criteria(text: str) -> dict[str, Any]:
+    """Extract location, treatment_type, payment, mat, populations, languages, substances, therapies from user message."""
+    text_lower = (text or "").lower().strip()
+    criteria = {}
+
+    # State / city patterns
+    state_abbr = re.findall(r"\b(ma|mass|massachusetts|tx|texas|ca|california|il|illinois)\b", text_lower)
+    if state_abbr:
+        m = {"ma": "ma", "mass": "ma", "massachusetts": "ma", "tx": "tx", "texas": "tx", "ca": "ca", "california": "ca", "il": "il", "illinois": "il"}
+        criteria["state"] = m.get(state_abbr[0], state_abbr[0])
+    if "boston" in text_lower:
+        criteria["location"] = "Boston"
+        criteria["state"] = "ma"
+    if "austin" in text_lower or "san antonio" in text_lower:
+        criteria["state"] = "tx"
+    if "chicago" in text_lower:
+        criteria["state"] = "il"
+    if "san francisco" in text_lower or "los angeles" in text_lower or "california" in text_lower:
+        criteria["state"] = "ca"
+    if not criteria.get("state") and not criteria.get("location"):
+        # Generic "location" for short state abbrev
+        two_letter = re.search(r"\b([a-z]{2})\b", text_lower)
+        if two_letter and two_letter.group(1) in ("ma", "tx", "ca", "il"):
+            criteria["state"] = two_letter.group(1)
+
+    # Treatment type
+    if any(w in text_lower for w in ["inpatient", "residential"]):
+        criteria["treatment_type"] = "inpatient" if "inpatient" in text_lower else "residential"
+    elif "outpatient" in text_lower:
+        criteria["treatment_type"] = "outpatient"
+    elif "telehealth" in text_lower:
+        criteria["treatment_type"] = "telehealth"
+
+    # Payment
+    if "medicaid" in text_lower or "masshealth" in text_lower:
+        criteria["payment"] = "Medicaid"
+    if "insurance" in text_lower and "payment" not in criteria:
+        criteria["payment"] = "insurance"
+    if "sliding scale" in text_lower:
+        criteria["payment"] = "sliding scale"
+    if "free" in text_lower and "payment" not in criteria:
+        criteria["payment"] = "free"
+    if "veteran" in text_lower or "va " in text_lower:
+        criteria["payment"] = "veterans"
+        criteria["populations"] = "veterans"
+
+    # MAT
+    if "mat" in text_lower or "medication-assisted" in text_lower or "medication assisted" in text_lower:
+        criteria["mat"] = True
+
+    # Populations: veterans, adolescents, LGBTQ+, pregnant women
+    if "veteran" in text_lower and "populations" not in criteria:
+        criteria["populations"] = "veterans"
+    if "adolescent" in text_lower or "youth" in text_lower:
+        criteria["populations"] = "adolescents"
+    if "lgbtq" in text_lower or "lgbt" in text_lower or "queer" in text_lower:
+        criteria["populations"] = "LGBTQ+"
+    if "pregnant" in text_lower or "pregnancy" in text_lower:
+        criteria["populations"] = "pregnant women"
+
+    # Languages
+    if "spanish" in text_lower or "spanish-speaking" in text_lower or "spanish speaking" in text_lower:
+        criteria["languages"] = "Spanish"
+    if "vietnamese" in text_lower:
+        criteria["languages"] = "Vietnamese"
+    if "mandarin" in text_lower or "chinese" in text_lower:
+        criteria["languages"] = "Mandarin"
+    if "bilingual" in text_lower and "languages" not in criteria:
+        criteria["languages"] = "Spanish"  # common with "bilingual" in this context
+
+    # Substances
+    if "alcohol" in text_lower:
+        criteria["substances"] = "alcohol"
+    if "opioid" in text_lower or "opioids" in text_lower:
+        criteria["substances"] = "opioids"
+    if "substance use" in text_lower or "substance abuse" in text_lower and "substances" not in criteria:
+        criteria["substances"] = "substance use"
+
+    # Therapies: CBT, 12-step (MAT handled above)
+    if "cbt" in text_lower or "cognitive behavioral" in text_lower:
+        criteria["therapies"] = "CBT"
+    if "12-step" in text_lower or "12 step" in text_lower or "twelve step" in text_lower:
+        criteria["therapies"] = "12-step"
+
+    return criteria
+
+
+def _merge_criteria(existing: dict, new: dict) -> dict:
+    """Merge new criteria into existing; new values override."""
+    out = dict(existing)
+    for k, v in new.items():
+        if v is not None and v != "":
+            out[k] = v
+    return out
+
+
+def _format_facilities_for_prompt(facilities: list[dict]) -> str:
+    """Format facility list for inclusion in system context (model must only use this)."""
+    if not facilities:
+        return "(No facilities in context. Do not name or describe any facility not listed here.)"
+    lines = []
+    for i, f in enumerate(facilities, 1):
+        name = f.get("facility_name", "Unknown")
+        desc = f.get("description", "") or f.get("services", "")
+        addr = f.get("address", "")
+        city = f.get("city", "")
+        state = f.get("state", "")
+        phone = f.get("phone", "")
+        mat = f.get("mat", "")
+        services = f.get("services", "")
+        parts = [f"{i}. {name} — {desc} Address: {addr}, {city}, {state}. Phone: {phone}. MAT: {mat}. Services: {services}."]
+        for key, label in (("payment_options", "Payment"), ("substances_addressed", "Substances"), ("languages", "Languages"), ("populations", "Populations")):
+            val = f.get(key, "")
+            if val and str(val).strip():
+                parts.append(f" {label}: {val}.")
+        lines.append("".join(parts))
+    return "\n".join(lines)
+
+
+def _detect_facility_mention(text: str, last_results: list[dict]) -> str | None:
+    """If user is asking about a specific facility, return a name fragment to look up."""
+    if not last_results or not text or not text.strip():
+        return None
+    text_lower = text.lower()
+    for f in last_results:
+        name = (f.get("facility_name") or "").lower()
+        if name and (name in text_lower or any(word in text_lower for word in name.split() if len(word) > 3)):
+            return f.get("facility_name")
+    # Common patterns: "the one at X", "Boston Medical Center", "AdCare"
+    if "boston medical" in text_lower or "bmc" in text_lower or "cope" in text_lower:
+        return "Boston Medical Center"
+    if "adcare" in text_lower:
+        return "AdCare"
+    if "bay cove" in text_lower:
+        return "Bay Cove"
+    return None
+
 
 class Chatbot:
     """
-    This class is extra scaffolding around a model. Modify this class to specify how the model recieves prompts and generates responses.
-
-    Example usage:
-        chatbot = Chatbot()
-        response = chatbot.get_response("What options are available for me?")
+    Stateful chatbot: criteria extraction, search when location present, only real data to model.
     """
 
     def __init__(self):
-        """
-        Initialize the chatbot with a HF model ID
-        """
-        model_id = MY_MODEL if MY_MODEL else BASE_MODEL # define MY_MODEL in config.py if you create a new model in the HuggingFace Hub
+        model_id = MY_MODEL if MY_MODEL else BASE_MODEL
         self.client = InferenceClient(model=model_id, token=HF_TOKEN)
-        
-    def format_prompt(self, user_input):
-        """
-        TODO: Implement this method to format the user's input into a proper prompt.
-        
-        This method should:
-        1. Add any necessary system context or instructions
-        2. Format the user's input appropriately
-        3. Add any special tokens or formatting the model expects
-
-        Args:
-            user_input (str): The user's question
-
-        Returns:
-            str: A formatted prompt ready for the model
-        
-        Example prompt format:
-            "You are a helpful assistant that specializes in...
-             User: {user_input}
-             Assistant:"
-        """
+        self._df = None  # cache for facilities
+
+    def _get_df(self):
+        if self._df is None:
+            self._df = load_facilities()
+        return self._df
 
-        return f"You are a helpful assistant that specializes in finding appropriate substance use and mental health treatment facilities in the Boston area. User: {user_input} Assistant:"
-        
-    def get_response(self, user_input):
+    def get_response(
+        self,
+        message: str,
+        history: list[list[str]] | None = None,
+        state: dict | None = None,
+    ) -> tuple[str, dict]:
         """
-        TODO: Implement this method to generate responses to user questions.
-        
-        This method should:
-        1. Use format_prompt() to prepare the input
-        2. Generate a response using the model
-        3. Clean up and return the response
-
-        Args:
-            user_input (str): The user's question
-
-        Returns:
-            str: The chatbot's response
-
-        Implementation tips:
-        - Use self.format_prompt() to format the user's input
-        - Use self.client to generate responses
+        Generate response and updated state. Use only this entrypoint from Gradio (or a future API).
         """
-        prompt = self.format_prompt(user_input)
+        state = state if state is not None else dict(DEFAULT_STATE)
+        history = history or []
+        criteria = state.get("criteria", {})
+        last_results = state.get("last_results", [])
+        last_facility_detail = state.get("last_facility_detail")
+
+        # Extract criteria from current message and merge
+        new_criteria = _extract_criteria(message)
+        criteria = _merge_criteria(criteria, new_criteria)
+
+        # Check if user is asking about a specific facility (follow-up)
+        facility_mention = _detect_facility_mention(message, last_results)
+        if facility_mention:
+            single = get_facility_by_name(facility_mention, self._get_df())
+            if single:
+                last_facility_detail = single
+                context_data = "Current facility data (use ONLY this for your answer):\n" + _format_facilities_for_prompt([single])
+            else:
+                context_data = "No matching facility found in data. Say you don't have details for that facility and offer to search again or clarify."
+        else:
+            last_facility_detail = None
+            # Run search when we have at least location
+            has_location = bool(criteria.get("state") or criteria.get("location"))
+            if has_location:
+                results = search(criteria, df=self._get_df(), limit=5)
+                last_results = results
+                context_data = "Current facility data (suggest ONLY these; do not invent any other facility):\n" + _format_facilities_for_prompt(results)
+            else:
+                context_data = "No search has been run yet (user has not provided a location). Ask for state or city, and optionally treatment type, payment, substances, populations, therapies, and languages, before suggesting facilities."
+
+        # Build messages for API: system (with context) + history + current user
+        system_content = SYSTEM_PROMPT + "\n\n" + context_data
+
+        messages = [{"role": "system", "content": system_content}]
+        for pair in history:
+            if len(pair) >= 2:
+                messages.append({"role": "user", "content": pair[0]})
+                messages.append({"role": "assistant", "content": pair[1]})
+        messages.append({"role": "user", "content": message})
+
         response = self.client.chat.completions.create(
             model=self.client.model,
-            messages=[{"role": "user", "content": prompt}],
-            max_tokens=1000,
-            temperature=0.7,
+            messages=messages,
+            max_tokens=800,
+            temperature=0.5,
         )
-        return response.choices[0].message.content
+        reply = (response.choices[0].message.content or "").strip()
+
+        new_state = {
+            "criteria": criteria,
+            "last_results": last_results,
+            "last_facility_detail": last_facility_detail,
+        }
+        return reply, new_state

src/facilities.py

@@ -0,0 +1,180 @@
+"""
+SAMHSA facility data loading and search.
+
+Data story: See data/README.md (source: N-SUMHSS / National Directory; scope and limitations).
+"""
+
+import os
+import pandas as pd
+from typing import Any
+
+# Column mapping: internal names -> CSV columns
+FACILITY_COLUMNS = {
+    "name": "facility_name",
+    "address": "address",
+    "city": "city",
+    "state": "state",
+    "zip": "zip",
+    "phone": "phone",
+    "treatment_type": "treatment_type",
+    "payment_options": "payment_options",
+    "mat": "mat",
+    "services": "services",
+    "substances_addressed": "substances_addressed",
+    "languages": "languages",
+    "populations": "populations",
+    "description": "description",
+}
+
+
+def _data_path() -> str:
+    base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    return os.path.join(base, "data", "facilities.csv")
+
+
+def load_facilities() -> pd.DataFrame:
+    """Load facility CSV; keep rows with non-missing city and state."""
+    path = _data_path()
+    if not os.path.exists(path):
+        return pd.DataFrame()
+    df = pd.read_csv(path)
+    for col in ["city", "state"]:
+        if col in df.columns:
+            df = df[df[col].notna() & (df[col].astype(str).str.strip() != "")]
+    return df
+
+
+def search(criteria: dict[str, Any], df: pd.DataFrame | None = None, limit: int = 10) -> list[dict[str, Any]]:
+    """
+    Search facilities by criteria. Only returns facilities that match all provided filters.
+    Criteria keys (all optional): location (state or city name), state, city, treatment_type,
+    payment (e.g. Medicaid, MassHealth, insurance, sliding scale, free, veterans), mat (bool),
+    populations (e.g. veterans, adolescents, LGBTQ+, pregnant women), languages (e.g. Spanish),
+    substances (e.g. alcohol, opioids), therapies (e.g. CBT, 12-step; MAT is separate via mat=True).
+    Missing columns (e.g. substances_addressed) are skipped for that filter.
+    """
+    if df is None:
+        df = load_facilities()
+    if df.empty:
+        return []
+
+    out = df.copy()
+
+    # Normalize for matching: lowercase string
+    def norm(s: Any) -> str:
+        if pd.isna(s):
+            return ""
+        return str(s).lower().strip()
+
+    # State: exact match (e.g. "ma", "MA" -> Massachusetts or state abbrev)
+    state = criteria.get("state") or (criteria.get("location") if isinstance(criteria.get("location"), str) and len(criteria.get("location", "").strip()) == 2 else None)
+    if not state and isinstance(criteria.get("location"), str):
+        loc = criteria["location"].strip()
+        # US state abbreviations (common)
+        abbr = {"ma": "ma", "mass": "ma", "massachusetts": "ma", "tx": "tx", "texas": "tx", "ca": "ca", "california": "ca", "il": "il", "illinois": "il"}
+        for k, v in abbr.items():
+            if loc.lower().startswith(k) or k in loc.lower():
+                state = v
+                break
+        if not state and "boston" in loc.lower():
+            state = "ma"
+        if not state and "austin" in loc.lower():
+            state = "tx"
+        if not state and "san antonio" in loc.lower():
+            state = "tx"
+        if not state and "chicago" in loc.lower():
+            state = "il"
+        if not state and ("california" in loc.lower() or "san francisco" in loc.lower() or "los angeles" in loc.lower()):
+            state = "ca"
+    if state:
+        out = out[out["state"].astype(str).str.lower().str.strip() == norm(state)]
+
+    # City or location text in city/name
+    city = criteria.get("city")
+    location_text = criteria.get("location") if not state else None
+    if city:
+        out = out[out["city"].apply(norm).str.contains(norm(city), na=False)]
+    elif location_text and isinstance(location_text, str) and len(location_text) > 2:
+        loc = norm(location_text)
+        if loc not in ("ma", "tx", "ca", "il", "mass", "massachusetts", "texas", "california", "illinois"):
+            out = out[
+                out["city"].apply(norm).str.contains(loc, na=False)
+                | out["facility_name"].apply(norm).str.contains(loc, na=False)
+            ]
+
+    # Helper: match term in col, or in services/description when col is empty (decoded data stored there)
+    def col_or_services_contains(col: str, term: str) -> pd.Series:
+        col_vals = out[col].apply(norm) if col in out.columns else pd.Series([""] * len(out), index=out.index)
+        if "services" in out.columns:
+            fallback = out["services"].apply(norm).str.contains(term, na=False)
+            if "description" in out.columns:
+                fallback = fallback | out["description"].apply(norm).str.contains(term, na=False)
+        else:
+            fallback = pd.Series(False, index=out.index)
+        return col_vals.str.contains(term, na=False) | ((col_vals.str.strip() == "") & fallback)
+
+    # Treatment type: inpatient, outpatient, residential, telehealth
+    treatment = criteria.get("treatment_type")
+    if treatment and ("treatment_type" in out.columns or "services" in out.columns):
+        t = norm(treatment)
+        out = out[col_or_services_contains("treatment_type", t)]
+
+    # Payment: Medicaid, MassHealth, insurance, sliding scale, free, veterans
+    payment = criteria.get("payment")
+    if payment and ("payment_options" in out.columns or "services" in out.columns):
+        p = norm(payment)
+        out = out[col_or_services_contains("payment_options", p)]
+
+    # MAT
+    if criteria.get("mat") is True:
+        out = out[out["mat"].apply(norm) == "yes"]
+
+    # Populations: veterans, adolescents, LGBTQ+, pregnant women, etc.
+    pop = criteria.get("populations")
+    if pop and ("populations" in out.columns or "services" in out.columns):
+        p = norm(pop)
+        out = out[col_or_services_contains("populations", p)]
+
+    # Languages: e.g. Spanish, Vietnamese
+    lang = criteria.get("languages")
+    if lang and ("languages" in out.columns or "services" in out.columns):
+        l = norm(lang)
+        out = out[col_or_services_contains("languages", l)]
+
+    # Substances addressed: e.g. alcohol, opioids
+    substances = criteria.get("substances")
+    if substances and ("substances_addressed" in out.columns or "services" in out.columns):
+        s = norm(substances)
+        out = out[col_or_services_contains("substances_addressed", s)]
+
+    # Therapies: CBT, 12-step, etc. (MAT has dedicated filter above). Search in services and description.
+    therapies = criteria.get("therapies")
+    if therapies:
+        t = norm(therapies)
+        # Normalize 12-step variants for matching
+        t_alt = "12-step" if "12" in t or "twelve" in t else t
+        def has_therapy(row: pd.Series) -> bool:
+            svc = norm(row.get("services", ""))
+            desc = norm(row.get("description", ""))
+            if t == "cbt" or t_alt == "12-step":
+                if t == "cbt":
+                    return "cbt" in svc or "cbt" in desc
+                return "12-step" in svc or "12 step" in svc or "12-step" in desc or "12 step" in desc
+            return t in svc or t in desc
+        out = out[out.apply(has_therapy, axis=1)]
+
+    out = out.head(limit)
+    return out.to_dict(orient="records")
+
+
+def get_facility_by_name(name_fragment: str, df: pd.DataFrame | None = None) -> dict[str, Any] | None:
+    """Return the first facility whose name contains the given fragment (for follow-up questions)."""
+    if df is None:
+        df = load_facilities()
+    if df.empty or not name_fragment or not name_fragment.strip():
+        return None
+    frag = name_fragment.lower().strip()
+    for _, row in df.iterrows():
+        if frag in str(row.get("facility_name", "")).lower():
+            return row.to_dict()
+    return None