document S4.5 D1 regression root cause

docs/evidence/THOUGHT_ENGINE_NODE_D1_REGRESSION_V1.md

@@ -0,0 +1,198 @@
+# THOUGHT ENGINE NODE D1 REGRESSION V1
+
+Date: 2026-08-18
+Stage: S4.5C Node -> D1 root-cause continuation
+
+## Symptom
+
+The deployed Thought Engine Node remained healthy at `/health`, but D1-backed reads took about 10 seconds and returned healthy-looking empty state (`/sessions` => `{"total":0,"sessions":[]}`). Session creation through the Bridge also failed downstream.
+
+## Cloudflare control
+
+Authoritative D1 database: `thought-vault`
+
+Database UUID: `ed9b375d-bf00-477b-942e-379828b70e81`
+
+Authoritative pre-smoke row counts from direct remote Wrangler control:
+
+- sessions: 7
+- thought_units: 60
+- thought_edges: 53
+- thought_reviews: 4
+- witness_events: 60
+- pattern_registry: 0
+- version_history: 0
+
+No data loss is supported by the evidence. Historical rows are present.
+
+Schema control completed from `C:\Users\kryst` context:
+
+- `pattern_registry` (7 columns)
+- `version_history` (6)
+- `witness_events` (7)
+- `thought_reviews` (7)
+- `thought_edges` (6)
+- `thought_units` (9)
+- `sessions` (9)
+- Cloudflare internal `_cf_KV`
+
+`sessions` columns:
+
+1. session_id TEXT primary key
+2. title TEXT
+3. created_by TEXT NOT NULL default 'system'
+4. created_at TEXT NOT NULL default datetime('now')
+5. updated_at TEXT NOT NULL default datetime('now')
+6. status TEXT NOT NULL default 'active'
+7. root_thought_id TEXT
+8. active_thought_id TEXT
+9. metadata TEXT default '{}'
+
+Both schema queries completed successfully against the intended UUID in about 0.14-0.21 ms SQL execution time.
+
+### Wrangler context anomaly
+
+Launching Wrangler from the production Orchestrator working directory (`C:\Users\kryst\Infrastructure\mcp-servers\mcp-orchestrator`) caused Wrangler to attempt account `ae406365ceacd48b5b3bfa299f2644be` and return Cloudflare authentication error code 10000, while Wrangler's authentication fallback displayed account `8543b906ec5121d56ff7bf7ede12525d`.
+
+The Desktop Commander process environment did not contain `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_D1_DATABASE_ID`, or `CLOUDFLARE_D1_API_TOKEN`. Re-running the schema commands from `C:\Users\kryst` immediately targeted the correct `thought-vault` UUID and succeeded. This is treated as a local Wrangler working-context anomaly, not evidence about the deployed Node.
+
+## Node source and deployment identity
+
+Original local / remote / deployed revision: `1fce18e542306fd929363902119b0a496d16ef79`
+
+Diagnostic deployed revision: `e48c9052b54c9e1eb221a80027a0a2e66569fbaa`
+
+Rollback revision: `1fce18e542306fd929363902119b0a496d16ef79`
+
+HF Space: `Kode-Animator/thought-engine-node`
+
+HF runtime reached `RUNNING` on diagnostic revision `e48c9052...`.
+
+Pre-existing untracked `test_smoke.py` remained untouched and was not committed.
+
+## D1 client implementation
+
+`d1_client.py` reads:
+
+- `CLOUDFLARE_ACCOUNT_ID`
+- `CLOUDFLARE_D1_DATABASE_ID`
+- `CLOUDFLARE_D1_API_TOKEN`
+
+Request construction:
+
+`https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/d1/database/{DATABASE_ID}/query`
+
+Headers:
+
+- `Authorization: Bearer <token>`
+- `Content-Type: application/json`
+
+Body:
+
+- `{"sql": <sql>}`
+- optional `params`
+
+Client behavior:
+
+- `httpx.AsyncClient(timeout=10.0)`
+- no retry loop
+- `raise_for_status()` on HTTP response
+- broad exception catch returned `[]`
+
+HF metadata confirmed all three Cloudflare values exist as Space Secrets. Hugging Face intentionally hides secret values after creation, so the configured account ID and database UUID could not be compared directly without changing runtime observability or manually replacing secrets. Token value was never exposed.
+
+## API compatibility control
+
+Current Cloudflare D1 documentation still specifies:
+
+`POST /accounts/{account_id}/d1/database/{database_id}/query`
+
+with Bearer API token authorization and `sql` / optional `params` JSON body. The Node request shape matches the current documented API. No API-path compatibility regression is supported.
+
+## HF -> Cloudflare boundary
+
+Existing HF logs before instrumentation repeatedly contained only:
+
+`D1 Connection Error:`
+
+because the broad exception handler logged only `str(e)`.
+
+A one-line diagnostic logging change was developed with TDD so the existing log also records `type(e).__name__`.
+
+Test: `test_d1_client.py`
+
+RED proof: `ReadTimeout` was absent from the original blank log.
+
+GREEN proof: after the logging-only change, the test passed and `git diff --check` was clean.
+
+After deploying `e48c9052...`, a fresh read-only call to `/sessions` returned HTTP 200 with `{"total":0,"sessions":[]}` after about 10.6 seconds.
+
+The corresponding HF runtime log recorded:
+
+`D1 Connection Error [ConnectTimeout]:`
+
+Therefore the Node does not reach an HTTP response from Cloudflare. Authentication acceptance/rejection and D1 query acceptance/rejection are downstream of the first observed failure.
+
+Boundary classification:
+
+- DNS: not independently isolated
+- TCP connect: not independently isolated from TLS setup by current `httpx.ConnectTimeout` evidence
+- TLS handshake: not independently isolated
+- HTTP response: NOT REACHED
+- Cloudflare auth: NOT REACHED / NOT TESTED
+- D1 request acceptance: NOT REACHED / NOT TESTED
+- observed timeout class: `httpx.ConnectTimeout`
+
+## Root cause
+
+Classification: D - HF runtime cannot successfully establish the Cloudflare API connection.
+
+First broken boundary: deployed HF Thought Engine Node -> `api.cloudflare.com:443` connection establishment, before HTTP response.
+
+Evidence:
+
+1. Direct remote Wrangler queries to the intended D1 database succeed and historical data is present.
+2. Node source uses the current documented Cloudflare D1 REST path.
+3. Deployed Node `/sessions` consistently consumes roughly the 10-second httpx timeout and returns false empty state.
+4. Typed runtime instrumentation proves the thrown exception is `httpx.ConnectTimeout`.
+5. No Cloudflare HTTP status is observed, so credential/schema/API response errors cannot be the first failure.
+
+Confidence: HIGH for classification D and the first broken boundary; insufficient evidence to attribute the connect timeout specifically to DNS, TCP routing, TLS handshake, provider egress policy, or an upstream network incident.
+
+## Fix status
+
+Primary repair: NOT PERFORMED.
+
+Reason: no evidence-backed minimal Node-side fix has yet been established for the HF -> Cloudflare connection-establishment timeout. Changing credentials, database identifiers, API path, timeout duration, retries, DNS behavior, IPv4/IPv6 preference, or provider settings without further evidence would violate the no-fix-without-root-cause rule.
+
+Database mutation: NONE.
+
+Smoke session: NOT CREATED because database connectivity is not restored.
+
+Historical Node session restoration: NOT ACHIEVED.
+
+Bridge historical list proof: NOT RUN because the downstream Node path remains broken.
+
+## Secondary defect
+
+Independently confirmed: any D1 exception is swallowed into `[]`, allowing unavailable D1 state to masquerade as zero records.
+
+The diagnostic commit improves log observability only. It does not yet change API error semantics. Per S4.5 ordering, the primary connectivity failure should be resolved before the secondary empty-state masquerade is repaired.
+
+## Untouched systems
+
+Bridge revision: `b17615cc0ab007b2dce33ebee9f5f258934c3e96` (unchanged)
+
+Candidate changes: NONE
+
+Federation changes: NONE
+
+Production Orchestrator changes: NONE
+
+D1 schema/data mutation: NONE
+
+## Next boundary
+
+S5 remains blocked.
+
+The next investigation must refine `ConnectTimeout` into DNS vs TCP vs TLS/provider-egress evidence, or establish a provider-side incident/configuration fact, before selecting a repair.