feature: streetview agent — multimodal Gemma 4 vision

The single deepest Gemma 4 capability we hadn't been using. Adds a
6th data agent that fetches a Google Street View image at the
property and asks Gemma 4 (vision) to identify visible flood-risk
indicators: basement-level windows, ground-floor HVAC, drainage
infrastructure, evidence of past water damage, elevation vs street.

Backend:
- New backend/app/tools/streetview.py — Google Street View Static
API client. Metadata-first pattern (free, doesn't count against
quota): walks radii None→50m→200m→500m looking for a real
panorama before fetching the image. Computes a bearing from the
panorama's actual location back to the geocoded address so the
camera aims AT the building (without this, default heading shows
the road, not the property — verified with a Drexel Blvd test
case where unaimed heading produced property_visible=false but
the aimed shot found a real impermeable-paving indicator).
Image returned as a data: URL ready to inline into the vision
request.
- New backend/app/agents/streetview_agent.py — sends the data URL
to Gemma 4 with the OpenAI-format image_url content-part. System
prompt explicitly tells the model NOT to fabricate features it
can't see (returning indicators=[] with low confidence is the
right answer when the property isn't clearly visible). Output
schema: indicators[], property_visible, image_quality,
elevation_vs_street, overall_visual_risk, confidence, summary.
Image data URL passed through to the dossier so the UI can show
the thumbnail Gemma actually saw.
- Orchestrator now builds the data-agent pack per-request (so the
streetview agent can receive the request's language for its
summary copy). Streetview runs in parallel with the other 5
data agents.

Frontend:
- AGENTS list gets a 'Streetview surveyor' entry (so the agents
screen shows it lighting up live).
- mapDossier surfaces raw.streetview.
- New §03 'What we saw at the property' section appears when
Street View has coverage. Shows the actual image side-by-side
with the model's interpretation: visual-risk badge, elevation
badge, confidence badge, bullet list of indicators with severity
+ 1-line implication. Falls back gracefully to the model's
honest summary when no specific indicators are visible.
- Subsequent section numbers shift §04→§05, §05→§06 when streetview
is shown.

Vision capability verified on the :free tier with BYOK in
scripts/smoke_test_vision.py. Image fetch + base64 + heading
calculation cost is ~150ms; the Gemma 4 vision call itself is
~3-5s. Total pipeline still ~30-40s.

Requires GOOGLE_MAPS_API_KEY env var (added to backend/app/config.py).
Free tier: 28K Street View Static requests/month against the
$200/month Maps Platform credit. For hackathon usage = $0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

app/agents/orchestrator.py

@@ -22,6 +22,7 @@ from app.agents.fema_agent import run_fema_agent
 from app.agents.local_agent import run_local_agent
 from app.agents.news_agent import run_news_agent
 from app.agents.risk_agent import run_risk_agent
+from app.agents.streetview_agent import run_streetview_agent
 from app.agents.weather_agent import run_weather_agent
 from app.tools.geocoder import geocode_address
 
@@ -56,14 +57,33 @@ async def _archive(ctx: GeoCtx) -> dict:
     )
 
 
+# The streetview agent needs language for its summary copy. Curried in
+# at agent-pack-construction time below.
+def _make_streetview(language: str) -> AgentFn:
+    async def _run(ctx: GeoCtx) -> dict:
+        return await run_streetview_agent(
+            ctx["lat"], ctx["lon"], ctx.get("display_name", ""),
+            language=language,
+        )
+    return _run
+
+
 # Order here is the order the frontend renders agent rows in.
-DATA_AGENTS: dict[str, AgentFn] = {
-    "fema": _fema,
-    "local": _local,
-    "weather": _weather,
-    "news": _news,
-    "archive": _archive,
-}
+# streetview is the only multimodal one — placed last in the data row.
+def _data_agents_for(language: str) -> dict[str, AgentFn]:
+    return {
+        "fema": _fema,
+        "local": _local,
+        "weather": _weather,
+        "news": _news,
+        "archive": _archive,
+        "streetview": _make_streetview(language),
+    }
+
+
+# Static handle used by the frontend to enumerate agent ids before a
+# request fires (see AGENTS array in index.html).
+DATA_AGENTS: dict[str, AgentFn] = _data_agents_for("en")
 
 
 def sse(event: str, data: dict) -> str:
@@ -94,8 +114,12 @@ async def run_assessment(
         "county": geo.get("county", ""),
     })
 
+    # Build the data-agent pack with the request's language so the
+    # streetview agent's summary text matches the dossier language.
+    data_agents = _data_agents_for(language)
+
     # Announce all data agents up front so the UI can render rows.
-    for name in DATA_AGENTS:
+    for name in data_agents:
         yield sse("agent_update", {
             "agent": name,
             "status": "working",
@@ -113,7 +137,7 @@ async def run_assessment(
     })
 
     # Run data agents in parallel; stream results in completion order.
-    tasks = {name: asyncio.create_task(fn(ctx)) for name, fn in DATA_AGENTS.items()}
+    tasks = {name: asyncio.create_task(fn(ctx)) for name, fn in data_agents.items()}
     task_to_name = {t: n for n, t in tasks.items()}
     pending = set(tasks.values())
     results: dict[str, dict] = {}
@@ -203,6 +227,7 @@ def _compile_dossier(geo: GeoCtx, results: dict) -> dict:
         "weather": results.get("weather", {}),
         "news": results.get("news", {}),
         "archive": results.get("archive", {}),
+        "streetview": results.get("streetview", {}),
         "risk": results.get("risk", {}),
         "advisor": results.get("advisor", {}),
     }

app/agents/streetview_agent.py

@@ -0,0 +1,122 @@
+"""
+Street View flood-indicator agent — the multimodal Gemma 4 showcase.
+
+Fetches a Google Street View image at the property, then asks Gemma 4
+(vision) to identify visible flood-risk features: basement-level
+windows, ground-floor HVAC, below-grade entries, drainage
+infrastructure, evidence of prior water damage, elevation relative
+to street grade.
+
+This is the only agent that exercises Gemma 4's multimodal capability.
+Verified to work on the :free tier with BYOK in scripts/smoke_test_vision.py.
+
+Returns the verdict alongside the image (data URL) so the dossier UI
+can show a thumbnail next to the findings.
+"""
+import json
+
+from app.data.languages import prompt_directive
+from app.llm.client import call_gemma4, extract_text, parse_json_response
+from app.tools.streetview import fetch_streetview_for
+
+
+SYSTEM_PROMPT = """You are a flood-risk surveyor analyzing a street-level photograph of a residential or small-commercial property. Your job is to identify visible features in the image that affect flood vulnerability.
+
+Critical rules:
+- Only describe features you can actually see. NEVER fabricate a feature to fill the JSON.
+- If the image is partially obstructed, low quality, or doesn't show the property clearly, say so via "confidence": "low" and a short summary explaining what's missing.
+- Distinguish "I see X and it implies Y" from "I'd want to verify Y by looking inside."
+
+What to look for, in plain language:
+- Basement-level windows (small, below sidewalk level, sometimes with metal grating) — vulnerable to surface water entering directly
+- Ground-floor HVAC units, water heaters, electrical meters mounted at low elevation — expensive to replace if submerged
+- Below-grade entries, basement stairwells, driveways sloping toward the building
+- Visible downspouts that discharge into hard surfaces or storm drains rather than landscaping
+- Storm drains, swales, retaining walls, French drains, sandbags
+- Watermarks, staining, repair patches, or rust — evidence of past water exposure
+- The property's elevation relative to the street and neighboring buildings (sunken vs raised)
+- Proximity to obvious water features (canals, low-lying parks, creeks)
+
+Always respond with valid JSON only."""
+
+
+async def run_streetview_agent(
+    lat: float,
+    lon: float,
+    address: str,
+    language: str = "en",
+) -> dict:
+    sv = await fetch_streetview_for(lat, lon)
+
+    if not sv.get("available"):
+        return {
+            "available": False,
+            "summary": (
+                "No street-level imagery available for this address — "
+                "Google Street View has no panorama within 500m."
+            ),
+            "raw": sv,
+        }
+
+    user_prompt = f"""Examine this street-level photo of the property at {address}.
+
+The photo was captured by Google Street View in {sv.get('capture_date') or 'an unknown date'} from
+approximately ({sv.get('pano_lat'):.5f}, {sv.get('pano_lon'):.5f})
+{f"— {sv.get('radius_m')}m from the geocoded address" if sv.get('radius_m') else "— at the geocoded address"}.
+
+Identify visible flood-risk indicators. Return a JSON object with:
+{{
+  "indicators": [
+    {{
+      "feature": "<short name, e.g. 'basement-level windows'>",
+      "location_in_image": "<e.g. 'lower-left of the building facade'>",
+      "risk_implication": "<1 sentence on what this means for flood risk>",
+      "severity": "low" | "moderate" | "high"
+    }}
+  ],
+  "property_visible": true | false,
+  "image_quality": "good" | "partial" | "poor",
+  "elevation_vs_street": "below_grade" | "at_grade" | "elevated" | "unclear",
+  "overall_visual_risk": "low" | "moderate" | "high",
+  "confidence": "low" | "medium" | "high",
+  "summary": "<1 sentence for the status feed>"
+}}
+
+If you can't see the property clearly (image obstructed, wrong angle, distant
+street), set property_visible=false and confidence=low; don't invent indicators.
+
+Return ONLY the JSON object."""
+
+    response = await call_gemma4(
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT + prompt_directive(language)},
+            {"role": "user", "content": [
+                {"type": "text", "text": user_prompt},
+                {"type": "image_url", "image_url": {"url": sv["image_data_url"]}},
+            ]},
+        ],
+        temperature=0.2,
+        max_tokens=1500,
+    )
+
+    text = extract_text(response)
+    parsed = parse_json_response(text) or {}
+
+    # Always attach the image data URL + provenance so the UI can render
+    # the thumbnail and credit Google for the imagery.
+    parsed["available"] = True
+    parsed["image_data_url"] = sv["image_data_url"]
+    parsed["pano_id"] = sv.get("pano_id")
+    parsed["capture_date"] = sv.get("capture_date")
+    parsed["copyright"] = sv.get("copyright")
+    parsed["pano_distance_m"] = sv.get("radius_m")
+
+    if "summary" not in parsed:
+        parsed["summary"] = "Street-level survey complete"
+    if not parsed.get("indicators"):
+        parsed.setdefault("indicators", [])
+    parsed.setdefault("raw_text", text if not parse_json_response(text) else None)
+    if parsed["raw_text"] is None:
+        parsed.pop("raw_text", None)
+
+    return parsed

app/config.py

@@ -5,6 +5,7 @@ import os
 # paste flows (HF Spaces secrets UI adds a trailing \n which makes httpx
 # reject the Authorization header as an illegal value).
 OPENROUTER_API_KEY = os.environ.get("OPENROUTER_API_KEY", "").strip()
+GOOGLE_MAPS_API_KEY = os.environ.get("GOOGLE_MAPS_API_KEY", "").strip()
 OPENROUTER_BASE = "https://openrouter.ai/api/v1"
 
 MODEL_PRIMARY = "google/gemma-4-31b-it:free"

app/tools/streetview.py

@@ -0,0 +1,147 @@
+"""
+Google Street View Static API tool.
+
+Strategy (per Google's guidance):
+  1. Hit /streetview/metadata first — FREE, doesn't count against
+     billing quota. Tells us if there's actual panorama coverage.
+  2. If status == OK at the requested point or within a small radius,
+     fetch the actual image at the panorama's true lat/lon (which may
+     differ from the input by tens of meters if Google snapped to the
+     nearest street).
+  3. If no coverage anywhere within 500m, return None — the agent
+     surfaces an honest "no street-level imagery" finding instead
+     of feeding the gray "no imagery" placeholder to the vision
+     model (which would hallucinate features in gray pixels).
+
+The image is returned as a data: URL (base64-encoded) so we can
+inline it directly into the Gemma 4 vision request without making
+the upstream provider re-fetch (which we learned the hard way fails
+when User-Agent rules block the upstream).
+"""
+import base64
+import math
+from typing import Optional
+
+import httpx
+
+from app.config import GOOGLE_MAPS_API_KEY
+
+METADATA_URL = "https://maps.googleapis.com/maps/api/streetview/metadata"
+IMAGE_URL = "https://maps.googleapis.com/maps/api/streetview"
+
+# Try increasingly wider radii until we find a panorama.
+# Most addresses with coverage hit on the first attempt; the 500m
+# fallback catches addresses on private roads or driveways where the
+# nearest pano is a block or two away.
+SEARCH_RADII_M = (None, 50, 200, 500)
+
+# Bias the camera angle slightly downward — flood-risk indicators
+# (basement windows, ground-floor HVAC, drainage) are at street level
+# and below.
+DEFAULT_PITCH = -5
+DEFAULT_FOV = 90
+
+
+def _bearing_deg(from_lat: float, from_lon: float, to_lat: float, to_lon: float) -> float:
+    """Compute the initial bearing (in degrees, 0=N, 90=E) from one
+    coordinate to another. Used to aim the Street View camera from the
+    captured panorama back toward the user's geocoded address."""
+    phi1 = math.radians(from_lat)
+    phi2 = math.radians(to_lat)
+    dlon = math.radians(to_lon - from_lon)
+    x = math.sin(dlon) * math.cos(phi2)
+    y = math.cos(phi1) * math.sin(phi2) - math.sin(phi1) * math.cos(phi2) * math.cos(dlon)
+    return (math.degrees(math.atan2(x, y)) + 360.0) % 360.0
+
+
+async def _metadata(client: httpx.AsyncClient, lat: float, lon: float, radius: Optional[int]) -> dict:
+    params = {
+        "location": f"{lat},{lon}",
+        "key": GOOGLE_MAPS_API_KEY,
+    }
+    if radius is not None:
+        params["radius"] = str(radius)
+    resp = await client.get(METADATA_URL, params=params)
+    if resp.status_code != 200:
+        return {"status": f"HTTP_{resp.status_code}"}
+    return resp.json() or {}
+
+
+async def fetch_streetview_for(lat: float, lon: float, size: str = "640x480") -> dict:
+    """
+    Fetch the best available Street View image near (lat, lon).
+
+    Returns:
+        On success:
+          {
+            "available": True,
+            "image_data_url": "data:image/jpeg;base64,...",
+            "pano_id": str,
+            "pano_lat": float, "pano_lon": float,
+            "capture_date": "YYYY-MM",
+            "radius_m": int (which radius found it; None = exact),
+          }
+        On no coverage:
+          {"available": False, "reason": "no panorama within 500m"}
+        On config error:
+          {"available": False, "reason": "GOOGLE_MAPS_API_KEY not set"}
+    """
+    if not GOOGLE_MAPS_API_KEY:
+        return {"available": False, "reason": "GOOGLE_MAPS_API_KEY not set"}
+
+    async with httpx.AsyncClient(timeout=20) as client:
+        meta = None
+        used_radius = None
+        for radius in SEARCH_RADII_M:
+            meta = await _metadata(client, lat, lon, radius)
+            if meta.get("status") == "OK":
+                used_radius = radius
+                break
+
+        if not meta or meta.get("status") != "OK":
+            return {
+                "available": False,
+                "reason": f"no Street View panorama within {SEARCH_RADII_M[-1]}m",
+                "metadata_status": meta.get("status") if meta else "unknown",
+            }
+
+        pano_loc = meta.get("location") or {}
+        pano_lat = pano_loc.get("lat", lat)
+        pano_lon = pano_loc.get("lng", lon)
+
+        # Aim the camera back at the user's actual address. Without
+        # this, the API picks a default heading that often shows the
+        # road instead of the building.
+        heading = _bearing_deg(pano_lat, pano_lon, lat, lon)
+
+        img_params = {
+            "size": size,
+            "location": f"{pano_lat},{pano_lon}",
+            "heading": f"{heading:.1f}",
+            "fov": str(DEFAULT_FOV),
+            "pitch": str(DEFAULT_PITCH),
+            "key": GOOGLE_MAPS_API_KEY,
+            "return_error_code": "true",  # 4xx instead of placeholder image
+        }
+        img_resp = await client.get(IMAGE_URL, params=img_params)
+        if img_resp.status_code != 200:
+            return {
+                "available": False,
+                "reason": f"image fetch returned HTTP {img_resp.status_code}",
+            }
+
+        b64 = base64.b64encode(img_resp.content).decode("ascii")
+        ct = img_resp.headers.get("content-type", "image/jpeg").split(";")[0].strip()
+
+        return {
+            "available": True,
+            "image_data_url": f"data:{ct};base64,{b64}",
+            "image_bytes": len(img_resp.content),
+            "pano_id": meta.get("pano_id"),
+            "pano_lat": pano_lat,
+            "pano_lon": pano_lon,
+            "heading_deg": heading,
+            "capture_date": meta.get("date"),
+            "copyright": meta.get("copyright"),
+            "radius_m": used_radius,
+        }

static/index.html

@@ -540,6 +540,9 @@ const AGENTS = [
   { id: "archive", name: "Archivist",
     finding: "Cook County: 11 NOAA flood events in 10yr, 4 FEMA disaster declarations. 2008, 2013, 2020, 2023 caused widespread basement flooding.",
     pin: { x: 62, y: 70, label: "11 events / 10y", tone: "amber" }, delay: 1100 },
+  { id: "streetview", name: "Streetview surveyor",
+    finding: "Multimodal: feeding Google Street View imagery to Gemma 4 vision to spot ground-floor flood-risk indicators.",
+    pin: null, delay: 1800 },
   { id: "risk",    name: "Risk analyst",
     finding: "Synthesizing… Score 74/100. AEP ≈ 5.1%, 30-yr cumulative probability of basement flood ≈ 79%. FEMA gap material.",
     pin: null, delay: 1600 },
@@ -1108,6 +1111,7 @@ const mapDossier = (raw) => {
     mitigating_factors: risk.mitigating_factors || [],
     reasoning_trace: risk.reasoning_trace || "",
     advisor_tldr,
+    streetview: raw.streetview || {},
   };
 };
 
@@ -1201,8 +1205,63 @@ const DossierScreen = ({ onBack, dossier }) => {
           )}
         </Section>
 
+        {D.streetview?.available && (
+          <Section
+            num="§03"
+            icon={<Icon name="pin" color="var(--accent)" size={16}/>}
+            iconBg="var(--accent-soft)"
+            title="What we saw at the property"
+            badge={<span className="risk-tag purple">Gemma 4 vision</span>}>
+            <div style={{display:"grid", gridTemplateColumns:"260px 1fr", gap:18, alignItems:"start"}}>
+              <div>
+                <img
+                  src={D.streetview.image_data_url}
+                  alt="Google Street View of the property"
+                  style={{width:"100%", borderRadius:10, border:"1px solid var(--line)", display:"block"}}
+                />
+                <div style={{fontSize:11, color:"var(--ink-4)", marginTop:6, fontFamily:"'JetBrains Mono', monospace"}}>
+                  {D.streetview.copyright || "© Google"}
+                  {D.streetview.capture_date && ` · captured ${D.streetview.capture_date}`}
+                  {D.streetview.pano_distance_m != null && ` · ${D.streetview.pano_distance_m}m from address`}
+                </div>
+              </div>
+              <div>
+                <p style={{margin:"0 0 10px"}}>
+                  We fed this Street View frame to Gemma 4 with vision turned on
+                  and asked it to flag visible flood-risk indicators. The model
+                  was instructed not to fabricate features it couldn't actually see.
+                </p>
+                <div style={{display:"flex", gap:8, flexWrap:"wrap", marginBottom:12, fontSize:12}}>
+                  <span className={`risk-tag ${D.streetview.overall_visual_risk === "high" ? "coral" : D.streetview.overall_visual_risk === "moderate" ? "amber" : "teal"}`}>
+                    visual risk: {D.streetview.overall_visual_risk || "unclear"}
+                  </span>
+                  {D.streetview.elevation_vs_street && D.streetview.elevation_vs_street !== "unclear" && (
+                    <span className="risk-tag neutral">{D.streetview.elevation_vs_street.replace(/_/g, " ")}</span>
+                  )}
+                  <span className="risk-tag neutral">confidence: {D.streetview.confidence || "unknown"}</span>
+                </div>
+                {D.streetview.indicators?.length > 0 ? (
+                  <ul style={{margin:0, paddingLeft:16, fontSize:13, color:"var(--ink-2)", lineHeight:1.55}}>
+                    {D.streetview.indicators.map((ind, i) => (
+                      <li key={i} style={{marginBottom:8}}>
+                        <strong>{ind.feature}</strong>
+                        {ind.severity && <span className="small" style={{marginLeft:6}}> · {ind.severity}</span>}
+                        <div style={{color:"var(--ink-3)", fontSize:12, marginTop:2}}>{ind.risk_implication}</div>
+                      </li>
+                    ))}
+                  </ul>
+                ) : (
+                  <p style={{color:"var(--ink-3)", fontSize:13, margin:0}}>
+                    {D.streetview.summary || "No specific indicators visible from the available angle."}
+                  </p>
+                )}
+              </div>
+            </div>
+          </Section>
+        )}
+
         <Section
-          num="§03"
+          num={D.streetview?.available ? "§04" : "§03"}
           icon={<Icon name="warn" color="var(--coral)" size={16}/>}
           iconBg="var(--coral-soft)"
           title="Why FEMA's flood map isn't the whole story"
@@ -1256,7 +1315,7 @@ const DossierScreen = ({ onBack, dossier }) => {
         </Section>
 
         <Section
-          num="§04"
+          num={D.streetview?.available ? "§05" : "§04"}
           icon={<Icon name="drop" color="var(--accent)" size={16}/>}
           iconBg="var(--accent-soft)"
           title="The raw signals we looked at"
@@ -1270,7 +1329,7 @@ const DossierScreen = ({ onBack, dossier }) => {
         </Section>
 
         <Section
-          num="§05"
+          num={D.streetview?.available ? "§06" : "§05"}
           icon={<Icon name="news" color="var(--amber)" size={16}/>}
           iconBg="var(--amber-soft)"
           title="Recent local flood news">