[NOTICKET] revert: unwire F-2 service-secret gate — unarmable with a browser-only caller

The gate (#37) shipped inert 2026-07-23, meant to be armed once the caller sent
`X-Dataeyond-Service-Secret`. Verified in E2E-Frontend `src/services/agenticApi.ts`
that the sole caller of this service is the browser SPA, which sends only
`Content-Type` and which we do not own and cannot change. Arming the gate (setting
`dataeyond__service__secret`) would therefore 401 the entire app. A wired-but-unarmable
gate is a footgun: any operator who sets the env var takes prod down, with nothing in
the code preventing it.

Lead decision (Rifqi, 2026-07-27): unwire it and record the live surface as
unauthenticated by design until Go forwards a verified per-user identity (#43).

Changes — un-wiring only, nothing else touched:
- main.py: dropped `dependencies=_guard` from all six router mounts; commented the
`_guard` line, the `service_auth` import, and the ENFORCED/DISABLED boot-log block;
the boot log now states plainly that the surface is unauthenticated.
- src/middlewares/service_auth.py: kept in-tree (comment-out-don't-delete) with a
PARKED banner; the function is correct and unchanged, just unmounted.
- src/config/settings.py: `dataeyond_service_secret` field commented; `extra="allow"`
means a stale `.env` value is ignored, not an error.
- Docs: API_CONTRACT "Authentication" now says "None — do not expose beyond the demo"
with the history; REPO_STATUS security section updated; DEV_PLAN #37 -> paused with
the reasoning, #43 promoted to the sole auth path, decisions line added.

NOT touched: F-1 tenant predicates (#38) stay as defensive-in-depth; F-22 UUID
validation stays; CORS left at ["*"] deliberately (tightening needs the FE origin as
config, which we chose not to set now). Restoring the gate is a one-edit reversal,
documented beside the router mounts.

Verification: suite 456 passed / 0 failed / 7 skipped (unchanged — nothing asserted
the wiring; test_service_auth.py still passes against the parked function). Ruff on
touched paths identical to HEAD (7 E501 / 1 S104 / 1 I001, all pre-existing).
`import main` OK; all 10 live /api routes still mount.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

API_CONTRACT_BE_PYTHON.md

@@ -36,19 +36,15 @@ The frontend uses this service during the analysis conversation flow:
 | `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |
 | `GET` | `/api/v1/charts` | Retrieve chart(s) produced by `render_chart` for one assistant answer (added 2026-07-13). |
 
-## Authentication (added 2026-07-23)
+## Authentication
 
-Every endpoint in the table above sits behind a **shared service secret**. Callers send it as a header on every request:
+**None. The live surface is unauthenticated — do not expose it beyond the demo.**
 
-```
-X-Dataeyond-Service-Secret: <secret>
-```
-
-A missing or incorrect value returns `401` with `{"detail": "Missing or invalid service credentials."}`. `GET /` and `GET /health` are exempt (deployment health probes).
+The service accepts `user_id` and `analysis_id` as ordinary request fields with no verification, and there is no caller authentication in front of any endpoint.
 
-**Rollout state — currently inert.** The gate is **disabled unless** the service is deployed with `dataeyond__service__secret` set. Until that env var is configured, requests without the header succeed exactly as before, so no frontend change is required to keep working today. Send the header as soon as the secret is issued; it is safe to send before the gate is armed (an unconfigured service ignores it).
+**History (2026-07-23 → 2026-07-27).** A shared service-secret gate (`X-Dataeyond-Service-Secret`, router-level, constant-time compare) shipped 2026-07-23, inert until `dataeyond__service__secret` was set. It was **unwired on 2026-07-27** (DEV_PLAN #37): the only caller of this service is the browser SPA, which cannot be changed to send the header, so the gate could never be armed without a 401 outage. The gate code is parked in `src/middlewares/service_auth.py` (not deleted) and restores in one edit if the caller situation changes.
 
-**What this is and is not.** It authenticates the *calling service*, not the end user — it does not identify which user is making the request, and it is not a substitute for the frontend's own auth. It exists because this service currently accepts `user_id` and `analysis_id` as ordinary request fields with no verification, and is reachable directly. Per-user authorization will move to a verified identity forwarded by the Go service; when it does, `user_id` will come from the verified token rather than the request body, and the request fields documented below may become redundant.
+**The real fix (DEV_PLAN #43).** Per-user authorization requires a verified identity forwarded by the Go service; once Go forwards a token, `user_id` comes from the verified claims rather than the request body. Until then the `user_id` predicates in the stores (F-1) are defensive-in-depth only, not an access control.
 
 ## Common Concepts
 

DEV_PLAN.md

@@ -238,13 +238,13 @@ plus the live report bug on analysis `966224d4…`. Same status legend as §0.
 | 34 | **Report floor extension** — a successful `retrieve_data` **that returned rows** clears the floor | Rifqi | ✅ | Same root cause; fixes a hard **409** for a session where every question is R2/R2b-shaped. Guardrail-adjacent, authorised 2026-07-23. NOT the "Floor Fixer" failure mode: the floor still asks "did we produce a real result" — empty retrievals, `check_*`-only and fully-failed runs all still fail it |
 | 35 | **`GET …/records` `substantive` flag** repointed to the body predicate + contract §records updated | Rifqi | ✅ | The curation list was contradicting the artifact it curates. Behavioral, non-breaking: no field added/removed/retyped |
 | 36 | **CK5b** — quality checkpoint covers analyze-free plans | Rifqi | ✅ | CK5 only inspected `analyze_*` tasks, so an all-null aggregate column on an R2/R2b plan reached the answer unflagged. `check_*` excluded (uncounted tables legitimately carry nulls) |
-| 37 | **F-2 service-secret gate** — `X-Dataeyond-Service-Secret`, router-level dependency | Rifqi | 🔎 | Code shipped 2026-07-23, **inert until `dataeyond__service__secret` is set**. Go makes no outbound call to Python (verified in the Go source), so the "Go fronts Python" premise in the code comments is not wired and the surface is currently open. **Action: Rifqi sets the secret on the HF Space + the FE/Go caller, then re-verify** |
+| 37 | **F-2 service-secret gate** — `X-Dataeyond-Service-Secret`, router-level dependency | Rifqi | ⏸️ | Shipped inert 2026-07-23. **UNWIRED 2026-07-27 (lead decision).** Verified in `E2E-Frontend-Data-Eyond/src/services/agenticApi.ts` that the **browser SPA is the only caller** of Python and we don't own it — it sends only `Content-Type`, and cannot be changed by us to send the header. So the gate could never be armed without a 401 outage: a wired-but-unarmable gate is a footgun (any operator setting `dataeyond__service__secret` breaks prod). Guard dependency removed from all six router mounts in `main.py`; `service_auth.py` kept in-tree but parked (comment-out-don't-delete); the `dataeyond_service_secret` setting commented. A prepared FE proxy (Node `server.js` holding the secret and injecting the header server-side) is the way to arm it later without exposing the secret to the browser — but that needs FE-repo access we don't have. **Net: the live surface is unauthenticated by design; real auth = #43. F-1 tenant predicates (#38) stay** as defensive-in-depth. |
 | 38 | **F-1 tenant scoping** — `user_id` predicate on the six analysis-keyed reads | Rifqi | ✅ | `CatalogStore.get_by_analysis` filtered on `analysis_id` alone where Go filters on both; the catalog payload carries the owner's `user_id`, so `DbExecutor`'s ownership check compared the victim's id against itself and passed → cross-tenant **query execution against a customer DB**. Defence-in-depth only until #37 is armed (`user_id` is caller-supplied, and `GET /traceability` leaks it) |
 | 39 | **Stale tests resolved** — the two long-standing suite failures | Rifqi | ✅ | Both encoded the pre-2026-07-13 user-scope fallback that `reader.py` deliberately removed. Not product bugs. Suite is now **394 passed / 0 failed / 7 skipped** — fully green for the first time |
 | 40 | ~~**F-3** — scope `GET /charts` + `GET /traceability` by `user_id`~~ | Rifqi | ⛔ | **RESOLVED 2026-07-23 — DECLINED by Rifqi. Do not re-open without his sign-off.** Both endpoints keep their existing lookup keys: `/traceability` by `(analysis_id, message_id)`, `/charts` by `message_id` alone (the 2026-07-13 lead decision). F-3 proposed adding a `user_id` parameter to both; an optional-param version was implemented on 2026-07-23 and then **reverted in full** (endpoints, charts-store predicates, and the contract notes) once the decision was restated — no FE change is required and none should be requested. **Accepted consequence:** both endpoints remain unauthenticated capability URLs over real customer data (`charts[].spec.plotly.data` is actual table values; the traceability payload carries 5-row previews, the executed SQL, and the owner's `user_id`). **The service-secret gate (#37) is therefore the only control protecting them** — which raises #37 from important to load-bearing. The stores' optional `user_id` parameters are kept as dead capability for a future Go-forwarded identity (#43); `PostgresChartStore` was returned to its message_id-only form. |
 | 41 | **F-12 / F-13** — bound the planner catalog render and the tabular blob read | Rifqi | ✅ | Shipped 2026-07-23. **F-13:** neither storage backend could report an object size, so `object_size()` was added to both (S3 `head_object`→`ContentLength`, Azure `get_blob_properties().size`, each returning None rather than raising) and `TabularExecutor` now refuses >500 MB **before** downloading, with a post-download byte check as the fallback when the probe is unavailable. **F-12:** `render()` gained TWO ceilings — `_MAX_TABLES=150` and `_MAX_CATALOG_CHARS=250_000` — because a table-count cap alone leaves wide tables unbounded (20 tables × 300 cols is as fatal as 400 × 30). Measured after: 400×30 went from ~241k tokens to ~63k; 100×30 (~45k tokens) still renders **in full**, so no realistic catalog is touched. Truncation emits an explicit "N more tables not shown" line so the planner knows it saw a subset, and logs — that log is the signal to retune. 12 new tests |
 | 42 | **F-20 observability** — `degraded_seam=<name>` on every never-throw / silent-drop path | Rifqi | 🔄 | The 2026-07-23 report bug was invisible by construction: the record was dropped with zero logging. **Partially shipped 2026-07-23** — the 10 seams where silent degradation is user-visible now emit a stable `degraded_seam` field (+ `repr(e)` instead of `str(e)`, so an empty-`str()` Fernet error is no longer a blank log): `input_guard_fail_open`, `analysis_catalog_read`, `report_floor_record_read`, `traceability_persist`, `traceability_flush`, `chart_persist` (×2), `report_input_persist` (×2), `analysis_state_ensure`. **Remaining:** the other ~76 `except Exception` sites, most of which are in unwired routers (`db_client`, `data_catalog`, `users`) or non-live paths — deliberately not swept, since a blanket edit across unwired code is exactly the drive-by §7A forbids. Control flow unchanged throughout (§5.4) |
-| 43 | **Go identity contract** — what does Go forward, and when? | Rifqi ↔ Harry | ⬜ new | Needed to replace #37's shared secret with real per-user authorization. Until then the store predicates in #38 are defensive only |
+| 43 | **Go identity contract** — what does Go forward, and when? | Rifqi ↔ Harry | ⬜ | **Now the sole path to caller auth** after #37 was unwired 2026-07-27. Python cannot authenticate the caller alone; either Go forwards a verified per-user token, or the FE (once we can change it) forwards the Go bearer token it already holds (`orchestrationApi.ts` shows the FE has one). Until then the live surface is unauthenticated and the #38 predicates are defensive only. Options for arming interim protection when FE access returns: the `server.js` BFF proxy (secret server-side) or route agentic calls through Go |
 | 44 | **F-17 / F-18 / F-25 compiler-parity batch** | Rifqi | ✅ new | Shipped 2026-07-23. Three execution-verified review findings that had **no task row** — the tracker jumped from #39 to #40 and lost them. **F-17 (High):** the bare-select check was gated on `if ir.group_by`, so a mixed select with `group_by=[]` passed; Postgres then failed loudly but the pandas path silently DROPPED the column while `output_columns` still advertised it → a real-looking table with a fabricated all-null column. Fixed in `validator.py` (fires whenever any agg is present — no false positives possible) + a presence backstop in `TabularExecutor`. **F-18:** `astype(str)` ran before `na=False`, so `NULL LIKE '%an%'` matched the literal `"nan"`/`"None"`. **F-25:** `SqlCompiler` raised on an empty `in`/`not_in` where pandas and `_column_values`' own docstring implement the empty-set semantics — a two-step plan whose first step legitimately returned zero rows hard-failed instead of answering. 16 new parity tests; one stale test updated (it pinned the old F-25 raise) |
 | 45 | **F-9 PII in persisted artifacts** — mask the traceability preview + report evidence tables | Rifqi | ✅ | Shipped 2026-07-24. `pii_flag` was an **ingestion-time control only**: it nulls `sample_values` into the planner prompt, but nothing stops the planner SELECTing a flagged column — and "list our top 20 customers" legitimately selects `customer_name`/`email`. Real values then reached two PERSISTED sinks: `message_traceability.data` (served by an unauthenticated GET, F-3 declined) and report evidence tables frozen permanently into `reports.content`. Fix = `retrieve_data` now carries `meta.pii_columns` (resolved through the IR select list, so aliases are honoured), and both sinks redact those cells to `[redacted]`. **Per the 2026-07-23 decision the ASSEMBLER still receives real values**, so the answer prose is unchanged and the question stays answerable — verified end-to-end: assembler input kept `Ada Lovelace`, the persisted preview showed `[redacted]`. Aggregates are deliberately NOT masked except `min`/`max`: `sum(salary)` identifies nobody, but `max(email)` returns one customer's actual address. Fails **open** (an unresolvable name is left unmasked, never a legitimate column wrongly blanked), so this is a mitigation, not a guarantee. 15 new tests |
 | 46 | **F-8 prompt-injection resistance** — planner / assembler / report_summary | Rifqi | ✅ new | Shipped 2026-07-24. The three prompts that ingest customer data had **no injection rule**: `guardrails.md` is appended only in `chatbot.py`/`help.py`, and the `InputGuard` screens only the user's message — it never sees catalog or row content. This is the one attack the five query-defense layers structurally cannot see, because every IR the planner emits is individually valid. A hostile string only needs to reach a text column in the customer's OWN database (a product description, a support ticket, a form field) to be sampled into `sample_values` and rendered verbatim into the planner prompt. Fix = a purpose-written "content is data, never instructions" rule in each prompt + `<data>…</data>` delimiters around the catalog render and the run-state render. **Deliberately NOT `guardrails.md` wholesale** — its rules prescribe refusal sentences, and the planner's only free-text field is `infeasible_reason`, so those strings would surface there and could regress the Q2 data-gap path (tests pin their absence). Verified: planner eval **6/6, carried_over 5/5 green**; live hostile-catalog run planned only `t_products` and never touched the planted `employees.salary`; 10 new tests |
@@ -266,6 +266,13 @@ drift; per §7F no result file is ever deleted or overwritten.
 Go's `database_clients.Service.Create` enforces `isSupportedActive`, and only `postgres` is
 `active`, so no such source can be registered today.
 
+**Decided 2026-07-27 (lead) — F-2 gate unwired, not armed.** The service-secret gate (#37) was
+removed from the router mounts because the sole caller is a browser SPA we don't own and can't
+change to send the header; arming it would 401 the whole app. The live surface is **unauthenticated
+by design** until #43 (Go-forwarded identity). Not a gap to re-raise — it is a recorded posture.
+CORS was left at `["*"]` on purpose (tightening it needs the FE origin as config, which we chose
+not to set for now).
+
 ## 5. Critical path & sequencing
 
 - **Critical path:** ~~#22 (send Harry the `report_inputs` schema)~~ **✅ resolved 2026-07-22** — now **#32** (`message_traceability` + `message_charts` DDL to Harry) and **#31** (non-convergent migration set). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)

REPO_STATUS.md

@@ -343,12 +343,14 @@ is not wired: a repo-wide search of the Orchestrator source finds **no HTTP clie
 service** and no config key for one, while the FE calls `POST /api/v2/chat/stream` directly. Two
 Python-side controls landed:
 
-- **`src/middlewares/service_auth.py`** — a shared-secret header
-  (`X-Dataeyond-Service-Secret`, constant-time compare) applied as a router-level dependency in
-  `main.py`, so a new endpoint cannot be added without it. **Blank env var = disabled**, so the
-  change is inert until `dataeyond__service__secret` is set on both sides; boot logs state which
-  mode is active. It authenticates the *caller service*, not the user — an interim measure until
-  Go forwards a verified per-request identity.
+- **Service-secret gate — UNWIRED 2026-07-27 (DEV_PLAN #37).** A shared-secret header
+  (`X-Dataeyond-Service-Secret`) shipped 2026-07-23 as a router-level dependency, inert until
+  `dataeyond__service__secret` was set. It was **removed from the router mounts** on 2026-07-27:
+  the only caller is the browser SPA, which we don't own and can't change to send the header, so
+  arming the gate would 401 the whole app — a wired-but-unarmable gate is a footgun. The code is
+  parked in `src/middlewares/service_auth.py` (not deleted) and restores in one edit. **Net: the
+  live surface is unauthenticated by design**; the durable fix is a Go-forwarded per-user identity
+  (DEV_PLAN #43).
 - **Tenant predicates.** `CatalogStore.get_by_analysis` filtered on `analysis_id` **alone** while
   Go's equivalent always filters `analysis_id AND user_id` (`catalog_repo.go`). Because the
   catalog payload also carries the *owner's* `user_id`, `DbExecutor`'s ownership check then

main.py

@@ -2,10 +2,15 @@
 
 from contextlib import asynccontextmanager
 
-from fastapi import Depends, FastAPI
+from fastapi import FastAPI
 from src.middlewares.logging import configure_logging, get_logger
 from src.middlewares.cors import add_cors_middleware
-from src.middlewares.service_auth import is_enforced, require_service_secret
+# F-2 service-secret gate UNWIRED 2026-07-27 (lead decision, DEV_PLAN #37). The only
+# caller of this service is the browser SPA, which we don't own and can't change to
+# send the header, so the gate could never be armed without a 401 outage. Restore by
+# re-adding these imports + the `_guard` dependency on each router mount below.
+# from fastapi import Depends
+# from src.middlewares.service_auth import is_enforced, require_service_secret
 from src.middlewares.rate_limit import limiter, _rate_limit_exceeded_handler
 from slowapi.errors import RateLimitExceeded
 # --- pr/5 Phase 1: unwire non-AI routers (Go owns these now). ---
@@ -42,16 +47,13 @@ async def lifespan(app: FastAPI):
         logger.info("Database initialized")
     else:
         logger.info("Skipping database initialization (SKIP_INIT_DB=true)")
-    # Make the security posture visible at boot: an unset secret means the whole
-    # surface is open, which is a deployment fact worth seeing in the logs rather
-    # than inferring from the absence of 401s.
-    if is_enforced():
-        logger.info("Service-secret gate ENFORCED on all live routers")
-    else:
-        logger.warning(
-            "Service-secret gate DISABLED (dataeyond__service__secret unset) — "
-            "every endpoint is reachable without credentials"
-        )
+    # F-2 service-secret gate UNWIRED 2026-07-27 (DEV_PLAN #37): the live surface is
+    # unauthenticated by design — Python cannot authenticate a browser-only caller it
+    # doesn't front. The real fix is a verified per-user identity forwarded by Go
+    # (DEV_PLAN #43). Do not expose this service beyond the demo.
+    logger.warning(
+        "No caller authentication on the live surface (F-2 unwired — see DEV_PLAN #37)"
+    )
     yield
 
 
@@ -76,19 +78,22 @@ app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
 # app.include_router(db_client_router)     # unwired: Go registers DB client
 # app.include_router(data_catalog_router)  # unwired: Go handles the catalog
 # app.include_router(chat_router)  # unwired: v2 chat replaces it (drops v1 cache ops routes)
-# F-2 (2026-07-23): every live router is mounted behind the service-secret gate.
-# Applied here rather than per route so a new endpoint cannot be added without it.
-# The dependency is a NO-OP until `dataeyond__service__secret` is set, so this is
-# inert for local dev / the current FE until the secret is deployed on both sides.
-# `/` and `/health` stay open for the HF Space health probe.
-_guard = [Depends(require_service_secret)]
-
-app.include_router(report_router, dependencies=_guard)
-app.include_router(tools_router, dependencies=_guard)
-app.include_router(help_router, dependencies=_guard)
-app.include_router(traceability_router, dependencies=_guard)  # KM-691: GET /api/v1/traceability
-app.include_router(charts_router, dependencies=_guard)  # W2: GET /api/v1/charts (§4.5)
-app.include_router(chat_v2_router, dependencies=_guard)  # pr/5 Phase 2: POST /api/v2/chat/stream
+# F-2 service-secret gate UNWIRED 2026-07-27 (lead decision, DEV_PLAN #37). It shipped
+# 2026-07-23 as a router-level dependency, inert until `dataeyond__service__secret` was
+# set. But the sole caller is the browser SPA (verified in E2E-Frontend `agenticApi.ts`),
+# which we don't own and can't change to send `X-Dataeyond-Service-Secret` — so the gate
+# could never be armed without a 401 outage. A wired-but-unarmable gate is a footgun, so
+# the dependency is removed here. `src/middlewares/service_auth.py` stays in-tree (parked,
+# not deleted). To restore: re-add the imports above and
+#   _guard = [Depends(require_service_secret)]
+# then pass `dependencies=_guard` to each mount below. Real auth = DEV_PLAN #43.
+
+app.include_router(report_router)
+app.include_router(tools_router)
+app.include_router(help_router)
+app.include_router(traceability_router)  # KM-691: GET /api/v1/traceability
+app.include_router(charts_router)  # W2: GET /api/v1/charts (§4.5)
+app.include_router(chat_v2_router)  # pr/5 Phase 2: POST /api/v2/chat/stream
 
 
 @app.get("/")

src/config/settings.py

@@ -99,15 +99,14 @@ class Settings(BaseSettings):
         alias="dataeyond__db__credential__key"
     )
 
-    # Shared service secret guarding the live surface (F-2, 2026-07-23). Callers send
-    # it as the `X-Dataeyond-Service-Secret` header. **Blank by default and blank =
-    # disabled**, so nothing breaks until it is deployed on both sides; setting it is
-    # what arms the gate. Interim measure only — it authenticates the *caller service*,
-    # not the user, and is meant to be replaced by verified per-request identity once
-    # Go forwards one. See src/middlewares/service_auth.py.
-    dataeyond_service_secret: str = Field(
-        alias="dataeyond__service__secret", default=""
-    )
+    # Shared service secret for the F-2 gate — UNWIRED 2026-07-27 (DEV_PLAN #37). The
+    # gate is no longer mounted (see main.py), so this value is read by nothing. Kept
+    # commented rather than deleted so the whole feature restores in one place. If a
+    # deployment still has `dataeyond__service__secret` in its .env it is simply
+    # ignored (extra="allow" on Settings). Restore alongside the main.py dependency.
+    # dataeyond_service_secret: str = Field(
+    #     alias="dataeyond__service__secret", default=""
+    # )
 
 
 # Singleton instance

src/middlewares/service_auth.py

@@ -1,5 +1,15 @@
 """Service-secret gate for the live Python surface (F-2, 2026-07-23).
 
+⚠️ PARKED / UNWIRED 2026-07-27 (lead decision, DEV_PLAN #37). This module is no longer
+mounted — `main.py` dropped the `Depends(require_service_secret)` dependency from every
+router. It stays in-tree (comment-out-don't-delete) so it can be restored in one edit,
+but it currently does NOTHING regardless of whether `dataeyond__service__secret` is set.
+Reason: the sole caller is the browser SPA (E2E-Frontend), which we don't own and can't
+change to send the header, so arming the gate would 401 the whole app. The durable fix
+is a verified per-user identity forwarded by Go (DEV_PLAN #43). Restore instructions are
+in `main.py` beside the router mounts.
+
+
 Python has no authentication of its own. The comments in `api/v1/traceability.py`
 and `api/v1/charts.py` say "No auth — Go fronts Python", but Go does not: a
 repo-wide search of the Orchestrator source finds no HTTP client pointed at this