Safety fix: never return an edible verdict from a single photo

.github/workflows/observability.yml

@@ -0,0 +1,56 @@
+name: observability
+
+on:
+  push:
+    paths:
+      - "observability/**"
+      - "pipeline/**"
+      - ".github/workflows/observability.yml"
+  pull_request:
+    paths:
+      - "observability/**"
+      - "pipeline/**"
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_PASSWORD: forager
+          POSTGRES_DB: forager_fs_obs
+        ports:
+          - 5434:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 2s
+          --health-timeout 5s
+          --health-retries 15
+    env:
+      FORAGER_OBS_DSN: postgresql://postgres:forager@localhost:5434/forager_fs_obs
+    steps:
+      # Two checkouts side by side, because observability/requirements.txt
+      # installs the shared package as `-e ../forager-obs`. Actions cannot check
+      # out above the workspace, so this repo goes in a subdirectory too and the
+      # later steps set working-directory.
+      - uses: actions/checkout@v4
+        with:
+          path: forager-field-station
+      - uses: actions/checkout@v4
+        with:
+          repository: thefullnacho/forager-obs
+          path: forager-obs
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Install deps (incl. editable forager-obs)
+        working-directory: forager-field-station
+        run: pip install -r observability/requirements.txt
+      - name: Shared safety rules (no DB, no models)
+        working-directory: forager-obs
+        run: python -m pytest -q
+      - name: Two-photo safety tests + observability schema/views
+        working-directory: forager-field-station
+        run: |
+          python -m pytest observability/tests/ tests/test_two_photo.py -q

.pytest_cache/.gitignore

@@ -0,0 +1,2 @@
+# Created by pytest automatically.
+*

.pytest_cache/CACHEDIR.TAG

@@ -0,0 +1,4 @@
+Signature: 8a477f597d28d172789f06886806bc55
+# This file is a cache directory tag created by pytest.
+# For information about cache directory tags, see:
+#	https://bford.info/cachedir/spec.html

.pytest_cache/README.md

@@ -0,0 +1,8 @@
+# pytest cache directory #
+
+This directory contains data from the pytest's cache plugin,
+which provides the `--lf` and `--ff` options, as well as the `cache` fixture.
+
+**Do not** commit this to version control.
+
+See [the docs](https://docs.pytest.org/en/stable/how-to/cache.html) for more information.

.pytest_cache/v/cache/nodeids

@@ -0,0 +1,13 @@
+[
+  "observability/tests/test_observability.py::test_abstain_is_never_toxic_as_edible",
+  "observability/tests/test_observability.py::test_positive_recall_view",
+  "observability/tests/test_observability.py::test_safety_view_reports_zero_for_safe_handling",
+  "observability/tests/test_observability.py::test_writer_derives_toxic_as_edible",
+  "tests/test_two_photo.py::test_confident_deadly_mushroom_is_not_shown_safe",
+  "tests/test_two_photo.py::test_deadly_can_surface_on_single_photo",
+  "tests/test_two_photo.py::test_single_photo_edible_only_when_confident",
+  "tests/test_two_photo.py::test_single_photo_never_commits_edible",
+  "tests/test_two_photo.py::test_single_photo_never_surfaces_edible",
+  "tests/test_two_photo.py::test_two_agreeing_photos_can_commit_safe",
+  "tests/test_two_photo.py::test_two_photos_no_deadly_shown_edible"
+]

STATUS.md

@@ -0,0 +1,79 @@
+# STATUS
+
+Running log of where the Field Station actually is. Newest entry at the top. Append, do not
+rewrite.
+
+Cross-repo facts live in the Forager wiki at `~/Documents/Forager/forager-wiki/`.
+
+---
+
+## 2026-08-15 — shared observability package; the dev harness stops shipping to the Space
+
+**The dev harness was being uploaded to the public Space.** `deploy.py`'s IGNORE list excluded
+`scripts/*` but not `observability/*`, so the whole Postgres harness went up with every deploy.
+Now excluded. Verified the Space runtime never imported it: `app.py`, `pipeline/` and `game/`
+are clean.
+
+**`forager-obs` adopted.** The `toxic_as_edible` rule was implemented here inline in
+`SessionWriter` and separately in forager_ml as a property on `InferenceOutcome`, with the
+flagship 0.0 claim resting on both. It now comes from
+`github.com/thefullnacho/forager-obs` (public, `main`), an editable sibling install, along with
+connect/migrate, the batch-writer contract (`SessionWriter` is now a `BaseWriter`) and the
+val-set walk including `k_groups`.
+
+**The session-grained schema stays here.** forager_ml's is image-grained. Those answer
+different questions and are deliberately not merged — recorded in the wiki so a later pass
+does not "fix" it.
+
+**Compose collision fixed.** This repo and forager_ml both bound host port 5433 *and* derived
+the same Compose project name from their parent directory (`observability` in both), so
+bringing up the second silently recreated the first's container. Now 5434 /
+`forager-fs-obs`, and both run side by side for the first time.
+
+**Verified:** 4 observability tests + 4 two-photo tests green against a live Postgres on 5434,
+and the Space's pipeline imports unchanged.
+
+### In flight
+
+- **`two-photo-safety` branch is 1 commit ahead and this repo has no git remote.** It is a
+  local repo that deploys to Hugging Face via `deploy.py`.
+- **The Space has not been redeployed.** The `deploy.py` exclusion takes effect on the next
+  `python deploy.py <user>/forager-field-station`, so `observability/` is still present in the
+  currently published Space.
+
+### Next concrete action
+
+Decide whether to redeploy the Space now (which removes the dev harness from the public repo)
+or fold it into the next content deploy.
+
+## 2026-10-25 — merge + GitHub remote
+
+**Changed:** `two-photo-safety` fast-forwarded into `master` (two-photo safety protocol,
+observability harness, confidence-banded single-photo gate, shared `forager-obs`). Repo now has
+a private GitHub remote, `thefullnacho/forager-field-station`, default branch `master`; both
+branches pushed. `assets/` (local backyard photos, video) is gitignored.
+
+### In flight
+
+- **The Space has still not been redeployed.** `observability/` is still in the published
+  Space until the next `deploy.py` run. [non-production] Approve the redeploy (it publishes).
+- The `forager_ml` convergence drift check stays local-only. Its reason (no remote here) is
+  gone, so it could move into CI now.
+
+### Next concrete action
+
+Space redeploy deferred (2026-10-25): fold it into the next content deploy, which will also
+remove `observability/` from the public Space.
+
+## 2026-10-27 — safety fix deploy blocked
+
+The fix that removes the 0.85 single-photo edible shortcut (`209f6b6`) is on master but **not
+live**. The live Space is `build-small-hackathon/Forager-Field-Notes`, not the
+`HomesteaderLabs/forager-field-station` named in the docs (that one doesn't exist). The
+HomesteaderLabs token has read-only access to the hackathon org, so the upload was refused (403).
+Nothing changed on the Space; it still serves the 2026-06-14 build and is sleeping. A new Space
+under HomesteaderLabs returned 402: Gradio on free CPU needs PRO.
+
+- [non-production] Get write access back on `build-small-hackathon`, or choose a new home
+  (PRO for HomesteaderLabs, or a different host), then re-run `deploy.py`.
+- [non-production] Until then, decide whether the live Space should keep serving the shortcut.

app.py

@@ -15,7 +15,8 @@ import time
 
 import gradio as gr
 
-from pipeline.convergence import build_result
+from pipeline.convergence import build_result  # noqa: F401  (re-exported for callers)
+from pipeline.session import resolve_session
 from pipeline.infer import Pipeline
 from game.ui import build_game_tab, build_stump_tab
 
@@ -73,7 +74,17 @@ def _loading() -> str:
 def _card(r) -> str:
     label, color, glyph = TIER.get(r.safety, TIER["UNKNOWN"])
 
-    if r.abstained:
+    if r.abstained and r.reason == "need_second_photo":
+        # Leaning toward an edible ID, but the safety rule withholds it until a
+        # second photo confirms. Not a failure — a request.
+        color, label, glyph = INK_UNK, "SECOND PHOTO", "◎"
+        title, sub = "ADD A SECOND ANGLE", "an edible ID needs two photos that agree"
+        rows = [
+            f"<div class='rdt-row'>{r.key_diff}</div>",
+            f"<div class='rdt-meta'>photograph the same specimen from another angle, "
+            f"then scan again</div>",
+        ]
+    elif r.abstained:
         color, label, glyph = INK_UNK, "UNKNOWN", "?"
         title, sub = "UNKNOWN", "not confident enough — refusing by default"
         rows = [
@@ -112,14 +123,20 @@ def _card(r) -> str:
     )
 
 
-def identify(image):
-    """Generator: show the vine animation, then the readout."""
-    if image is None:
+def identify(image, image2=None):
+    """Generator: show the vine animation, then the readout.
+
+    Takes an optional second photo. The safety rule (pipeline.session) means a
+    single photo never surfaces an edible verdict — it asks for the second — so
+    both inputs feed one resolve_session call.
+    """
+    images = [im for im in (image, image2) if im is not None]
+    if not images:
         yield _idle()
         return
     t0 = time.time()
     yield _loading()
-    result = build_result(PIPE.identify(image))
+    result = resolve_session(PIPE, images)
     elapsed = time.time() - t0
     if elapsed < 1.6:                       # let the vine finish drawing
         time.sleep(1.6 - elapsed)
@@ -358,6 +375,9 @@ with gr.Blocks(title="Forager's Field Station") as demo:
                 with gr.Column(scale=1):
                     img = gr.Image(type="pil", label="SPECIMEN", sources=["upload", "webcam"],
                                    elem_classes="eink-input", height=300)
+                    img2 = gr.Image(type="pil", label="SECOND ANGLE — confirms an edible ID",
+                                    sources=["upload", "webcam"], elem_classes="eink-input",
+                                    height=180)
                     btn = gr.Button("▸ SCAN SPECIMEN", variant="primary", elem_classes="eink-scan")
                     if os.path.isdir(EXAMPLES_DIR):
                         samples = [[os.path.join(EXAMPLES_DIR, f)] for f in (
@@ -371,8 +391,9 @@ with gr.Blocks(title="Forager's Field Station") as demo:
             gr.HTML(QUOTE_BAR)
             gr.Markdown(SAFETY_NOTICE, elem_id="notice")
 
-            btn.click(identify, inputs=img, outputs=out)
-            img.change(identify, inputs=img, outputs=out)
+            btn.click(identify, inputs=[img, img2], outputs=out)
+            img.change(identify, inputs=[img, img2], outputs=out)
+            img2.change(identify, inputs=[img, img2], outputs=out)
 
         with gr.Tab("⚔ BEAT THE MACHINE"):
             build_game_tab(PIPE)

pipeline/convergence.py

@@ -37,6 +37,9 @@ class ForagerResult:
     expert_model:    str
     abstained:       bool
     reason:          str     # why we abstained (or "" when committed)
+    energy:          float | None = None  # winning call's OOD energy — telemetry,
+                                          # not a gate (energy doesn't separate
+                                          # deadly-as-safe); kept for drift monitoring.
 
     @property
     def is_deadly(self) -> bool:
@@ -71,7 +74,7 @@ def _commit(domain: str, c: dict) -> ForagerResult:
         domain=domain, species=species, scientific_name=meta["scientific"], confidence=conf,
         safety=meta["safety"], lookalike=meta["lookalike"], key_diff=meta["key_diff"],
         low_confidence=conf < LOW_CONFIDENCE_THRESHOLD, expert_model=c["expert"],
-        abstained=False, reason="",
+        abstained=False, reason="", energy=c.get("energy"),
     )
 
 

pipeline/infer.py

@@ -122,6 +122,11 @@ class Pipeline:
         idx = int(probs.argmax())
         return self._classes[name][idx], float(probs[idx]), _energy(logits)
 
+    def _run_probs(self, name: str, x: np.ndarray) -> tuple[np.ndarray, float]:
+        """Returns (full softmax prob vector, energy) — for multi-photo fusion."""
+        logits = self._sessions[name].run(None, {"input": x})[0][0]
+        return _softmax(logits), _energy(logits)
+
     def identify(self, img: Image.Image) -> dict:
         """
         Returns a dict describing the call:
@@ -148,3 +153,55 @@ class Pipeline:
 
         call = {"expert": ename, "species": species, "confidence": conf, "energy": round(energy, 4)}
         return {**out, "abstain": False, "calls": [call]}
+
+    def identify_many(self, images: list[Image.Image]) -> dict:
+        """Fuse K photos of one specimen into a single call.
+
+        Late-fusion by GEOMETRIC MEAN of the experts' softmax vectors (treats
+        each photo as independent evidence: agreement sharpens the peak,
+        disagreement flattens it), plus a CONSENSUS requirement — every photo's
+        own top class must match the fused top class. This is the K>=2 protocol
+        validated in scripts/multiview_smoketest.py (deadly-shown-edible -> 0.00%
+        at K=2). K=1 degenerates exactly to identify()'s single call.
+
+        Abstains (and says why) when the photos disagree on domain or class, so
+        a confident-but-wrong single frame can't carry an edible verdict.
+        """
+        if not images:
+            return {"domain": "unknown", "domain_confidence": 0.0,
+                    "abstain": True, "reason": "low_confidence"}
+        if len(images) == 1:
+            return self.identify(images[0])
+
+        # ── Stage 1: every photo must route to the SAME in-domain domain ──────
+        routed = [self._run(ROUTER, preprocess(img)) for img in images]
+        domains = {d for d, _, _ in routed}
+        dconf = min(c for _, c, _ in routed)
+        domain = next(iter(domains))
+        out = {"domain": domain if len(domains) == 1 else "mixed",
+               "domain_confidence": dconf}
+        if len(domains) != 1 or dconf < ROUTER_CONFIDENCE_THRESHOLD or domain not in DOMAIN_EXPERTS:
+            reason = ("off_domain" if len(domains) == 1 and domain not in DOMAIN_EXPERTS
+                      else "uncertain_domain")
+            return {**out, "abstain": True, "reason": reason}
+
+        # ── Stage 2: run the domain's expert on each photo, fuse, check consensus
+        ename = DOMAIN_EXPERTS[domain]
+        classes = self._classes[ename]
+        per_photo = [self._run_probs(ename, preprocess(img)) for img in images]
+        prob_rows = np.stack([p for p, _ in per_photo])           # [K, C]
+        energy = float(np.mean([e for _, e in per_photo]))
+
+        logp = np.log(np.clip(prob_rows, 1e-9, 1.0)).mean(axis=0)  # geometric mean
+        fused = np.exp(logp - logp.max())
+        fused /= fused.sum()
+        top = int(fused.argmax())
+
+        per_photo_top = {int(row.argmax()) for row in prob_rows}
+        if per_photo_top != {top}:
+            # Photos disagree on the species — refuse rather than commit.
+            return {**out, "abstain": True, "reason": "no_consensus"}
+
+        call = {"expert": ename, "species": classes[top],
+                "confidence": float(fused[top]), "energy": round(energy, 4)}
+        return {**out, "abstain": False, "calls": [call]}

pipeline/session.py

@@ -0,0 +1,64 @@
+"""session.py — the two-photo safety policy that sits over the resolver.
+
+The core safety rule, stated plainly:
+
+    You are never shown an edible verdict on the strength of a single photo.
+
+A single photo can show DEADLY (over-warning is always safe) or abstain. A
+SAFE/CAUTION identification is only surfaced when a SECOND photo agrees with the
+first (geometric-mean fusion + consensus, in infer.identify_many). So the second
+photo is requested exactly, and only, when the system is leaning toward telling
+you something is edible — the one moment extra rigor is worth the friction.
+
+This is what closes the ~1% single-shot deadly-shown-edible residual to 0.00%
+(scripts/multiview_smoketest.py).
+"""
+
+from PIL import Image
+
+from .convergence import ForagerResult, build_result
+from .infer import Pipeline
+
+# There is no confidence bar that lets one photo commit edible. A 0.85 bar used to:
+# the val sets had put every single-photo deadly-as-edible miss at <= 0.772. Over
+# the full val sets, two toxic mushroom photos commit as SAFE above it (lion's
+# mane at 0.91, chanterelle at 0.87), and val is the optimistic case. A confident
+# wrong answer looks exactly like a confident right one, so no threshold on one
+# photo's confidence can hold; only a second look can.
+
+
+def resolve_session(pipe: Pipeline, images: list[Image.Image]) -> ForagerResult:
+    """Run the K-photo pipeline and apply the single-photo edible gate.
+
+    - 0 photos -> abstain.
+    - DEADLY (any photo count) -> surface it. Over-warning is safe.
+    - 1 photo, SAFE/CAUTION, at any confidence -> withhold; ask for a second photo.
+    - 2+ photos -> the fused/consensus verdict from identify_many stands.
+    """
+    result = build_result(pipe.identify_many(images))
+
+    if (len(images) == 1
+            and not result.abstained
+            and result.safety in ("SAFE", "CAUTION")):
+        return _need_second_photo(result)
+    return result
+
+
+def _need_second_photo(provisional: ForagerResult) -> ForagerResult:
+    """Turn a single-photo edible lean into a 'add a second photo' refusal,
+    carrying the provisional species so the UI can say what it's leaning toward
+    without committing to it."""
+    return ForagerResult(
+        domain=provisional.domain,
+        species="unknown",
+        scientific_name=provisional.scientific_name,
+        confidence=provisional.confidence,
+        safety="UNKNOWN",
+        lookalike=provisional.lookalike,
+        key_diff=(f"Leaning {provisional.species.replace('_', ' ')}, but an edible call "
+                  f"needs a second photo from a different angle to confirm."),
+        low_confidence=True,
+        expert_model=provisional.expert_model,
+        abstained=True,
+        reason="need_second_photo",
+    )

tests/test_two_photo.py

@@ -0,0 +1,108 @@
+"""Two-photo safety protocol tests, against the real ONNX models.
+
+The load-bearing guarantee: no edible verdict is ever surfaced from a single
+photo, and two disagreeing photos abstain rather than commit.
+"""
+
+import os
+import sys
+
+import pytest
+from PIL import Image
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from pipeline.infer import Pipeline  # noqa: E402
+from pipeline.session import resolve_session  # noqa: E402
+
+ML = "/home/alex/Documents/Forager/forager_ml"
+
+
+@pytest.fixture(scope="module")
+def pipe():
+    return Pipeline()
+
+
+def _img(path):
+    return Image.open(path)
+
+
+def _first(cls_dir, n=1):
+    files = sorted(f for f in os.listdir(cls_dir)
+                   if f.lower().endswith((".jpg", ".jpeg", ".png")))
+    return [os.path.join(cls_dir, f) for f in files[:n]]
+
+
+# ── Policy: single photo never surfaces an edible verdict ────────────────────
+
+def test_single_photo_never_commits_edible(pipe):
+    """A single photo never surfaces an edible (SAFE/CAUTION) verdict, at any
+    confidence: it is withheld and routed to a second photo."""
+    safe_dir = f"{ML}/berry_dataset_split/val/blackberry_common"
+    if not os.path.isdir(safe_dir):
+        pytest.skip("val data not present")
+    for path in _first(safe_dir, 12):
+        r = resolve_session(pipe, [_img(path)])
+        assert not (not r.abstained and r.safety in ("SAFE", "CAUTION")), (
+            f"single photo committed {r.safety} at {r.confidence:.3f} for {path}")
+        if r.abstained:
+            assert r.reason in ("need_second_photo", "low_confidence",
+                                "uncertain_domain", "off_domain")
+
+
+def test_confident_deadly_mushroom_is_not_shown_safe(pipe):
+    """The case that removed the 0.85 shortcut: toxic-catch-all photos the
+    mushroom expert calls lion's mane (0.91) and chanterelle (0.87)."""
+    d = f"{ML}/high_value_dataset_split/val/high_value_toxics"
+    paths = [f"{d}/599814795.jpg", f"{d}/616602478.jpg"]
+    if not all(os.path.isfile(p) for p in paths):
+        pytest.skip("val data not present")
+    for path in paths:
+        r = resolve_session(pipe, [_img(path)])
+        assert r.abstained and r.reason == "need_second_photo", (
+            f"{path} surfaced as {r.species} {r.safety} at {r.confidence:.3f}")
+
+
+def test_two_agreeing_photos_can_commit_safe(pipe):
+    """Two photos of the same SAFE class should be allowed to commit (the whole
+    point — the second photo unlocks an edible verdict)."""
+    safe_dir = f"{ML}/berry_dataset_split/val/blackberry_common"
+    if not os.path.isdir(safe_dir):
+        pytest.skip("val data not present")
+    committed = 0
+    paths = _first(safe_dir, 12)
+    # try a few pairs; at least one agreeing pair should commit SAFE
+    for i in range(0, len(paths) - 1, 2):
+        r = resolve_session(pipe, [_img(paths[i]), _img(paths[i + 1])])
+        if not r.abstained and r.safety == "SAFE":
+            committed += 1
+    assert committed >= 1, "no agreeing SAFE pair ever committed — fusion too strict"
+
+
+def test_deadly_can_surface_on_single_photo(pipe):
+    """Over-warning is safe: a DEADLY read on one photo is allowed through."""
+    deadly_dir = f"{ML}/berry_dataset_split/val/pokeweed_toxic"
+    if not os.path.isdir(deadly_dir):
+        pytest.skip("val data not present")
+    # We don't require every pokeweed to read DEADLY on one shot, only that the
+    # policy PERMITS a single-photo DEADLY (never converts it to need_second).
+    for path in _first(deadly_dir, 8):
+        r = resolve_session(pipe, [_img(path)])
+        if r.safety == "DEADLY":
+            assert not r.abstained
+            return
+    pytest.skip("no single-shot DEADLY in sample (still safe: they abstained)")
+
+
+def test_two_photos_no_deadly_shown_edible(pipe):
+    """The core claim: fuse two photos of DEADLY berry specimens; none may be
+    surfaced as edible (SAFE/CAUTION committed)."""
+    for cls in ("pokeweed_toxic", "canada_moonseed_deadly"):
+        d = f"{ML}/berry_dataset_split/val/{cls}"
+        if not os.path.isdir(d):
+            pytest.skip("val data not present")
+        paths = _first(d, 12)
+        for i in range(0, len(paths) - 1, 2):
+            r = resolve_session(pipe, [_img(paths[i]), _img(paths[i + 1])])
+            assert not (not r.abstained and r.safety in ("SAFE", "CAUTION")), (
+                f"{cls} pair surfaced as {r.safety} — toxic-as-edible at K=2")