docs: build-time deep dive notes (root cause, verified results, next bottleneck)

Captures the 2026-08-10 research: scoped-connection root cause + evidence,
what shipped and where (hf-test + MCP Space df44611, rollback 3d735c7),
verified timings (e2e 655s, MCP 9.2min), API research findings, and the
model-semantics phase (~5 min of sequential LLM batches) as the documented
next bottleneck with optimization options — deferred, no code changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/build_time_deep_dive_2026_08_10.md

@@ -0,0 +1,124 @@
+# Build-time deep dive — 2026-08-10
+
+Research notes from the investigation into "why does the model take so long to
+create." Companion to `docs/sre_slow_tml_import.md` (now RESOLVED) and
+`docs/async_tml_import_handoff.md` (superseded as a fix, kept as the 504
+safety net). Probe script: `tests/ts_table_create_deepdive.py`.
+
+---
+
+## 1. Root cause of the ~250s table imports
+
+ThoughtSpot's logical-table TML import **scans all external-warehouse metadata
+visible to the connection**. This is documented behavior for the v2 connection
+APIs: when the database scope is omitted, *"all databases in the data warehouse
+are accessible for metadata operations."* Our connection TML had no `database`
+property, and the Snowflake role (`SE_ROLE`) can see **~507 databases** — so
+every table create/join import paid a full metadata scan.
+
+Measured 2026-08-10, same instance, same trivial 2-column table:
+
+| Connection | Table import time |
+|---|---:|
+| Unscoped (app behavior before the fix) | **246–251s** |
+| Scoped to `DEMOBUILD` (1,003 schemas / 6,084 tables of accumulated demo debris) | **33s** |
+| Scoped to a fresh, tiny database | **0.7s** |
+
+Key evidence that killed the earlier theories:
+
+- **Prod (secloud) measured identical ~250s unscoped** — never sebe-specific,
+  never "instance load."
+- Duplicate-name create errors return in 0.17s; model TML imports are
+  near-instant — parsing/validation/auth were never the cost.
+- Flat ~250s per call regardless of payload (1 trivial table vs 4-table batch)
+  — a scan proportional to *visible metadata*, not to the payload.
+- July's conclusion ("server-side, not reducible, raise with SRE") was wrong;
+  the async-import work from July remains useful only as the 504 safety net.
+
+## 2. The fix that shipped (commit `21ee86d` + follow-ups)
+
+1. **Scoped connections** — every demo connection now carries a
+   `database` property in its TML (`create_connection_tml` /
+   `create_connection_with_reconcile` require it; fail loudly without).
+2. **Monthly-rotating demo database** — demos write to
+   `<SNOWFLAKE_DATABASE>_<YYYY_MM>` (e.g. `DEMOBUILD_2026_08`). Single source:
+   `get_demo_database()` in `snowflake_auth.py`, consumed by the session
+   params, `cdw_connector`, env injection, and the deploy path.
+   `ensure_demo_database()` creates it on connect. The admin setting
+   `SNOWFLAKE_DATABASE` stays the base name.
+3. **Test harnesses** (`e2e_quality.py`, `settings_test.py`) resolve the
+   database from the model TML / `get_demo_database()` — no hardcoded
+   `DEMOBUILD` anywhere.
+
+Operational notes:
+
+- `get_demo_database()` is **memoized per process** — a long-running Space
+  crosses a month boundary only on restart/redeploy. Harmless (old DB keeps
+  working), just means rotation isn't mid-process.
+- **Old-demo redeploys**: data written before 2026-08-10 lives in old
+  `DEMOBUILD`; re-running ThoughtSpot deployment for such a schema will fail
+  "schema not found" (deploy looks in the rotated DB). Already-deployed old
+  demos are untouched. If old-schema redeploys become a need, thread the
+  original database through instead.
+- Old monthly DBs can be dropped as their demos expire — that, not cleanup
+  jobs, is the bloat story now.
+
+## 3. Verified results (same day)
+
+| Path | Before | After |
+|---|---|---|
+| App e2e, per test (Aug 7–10 pre-fix record) | 1,084–2,711s, 12/24 timeouts | **655s, Grade B, 0 timeouts** (TreeHouse) |
+| MCP solo build (6 rehearsals on old deploy) | 16–20 min | **9.2 min, PASS, zero errors** (Tixr, owner=jack) |
+
+Both paths verified writing to `DEMOBUILD_2026_08` with scoped connections.
+MCP Space `thoughtspot-demoprep/mcp` deployed at `df44611`; rollback point is
+`3d735c7` (`git push hf-mcp 3d735c7:main --force`).
+
+## 4. Related API research (documented, for future reference)
+
+- The v2 connection APIs also accept a **`databases` list** in
+  `data_warehouse_config` (persisted scope across multiple DBs) — an
+  alternative to the single `database` property if we ever need >1 DB visible.
+- The UI's "edit connection → select tables" path is
+  `POST /api/rest/2.0/connections/{id}/update` with an `externalDatabases`
+  hierarchy — a table-registration path that skips TML entirely. Not needed
+  now (scoped TML import is <1s) but a known fallback.
+- **No join/relationship API exists in REST v2.0** — TML is the only public
+  way to define joins; our two-phase create+joins stays.
+- `skip_diff_check: true` (10.6.0.cl+) is an undocumented-beyond-one-line
+  import knob; untested, might shave validation time. Not needed at current
+  speeds.
+- Search indexing on table add is a background process (docs) and was ruled
+  out as the latency source.
+
+## 5. The NEXT bottleneck: model semantics (~5 min, LLM-bound)
+
+Current solo MCP build profile (Tixr, 9.2 min total):
+
+| Phase | Time | Bound by |
+|---|---:|---|
+| Dataset + DDL authoring | ~2.5 min | LLM |
+| Snowflake load + TS tables + joins | **~30s** | (was ~8–10 min) |
+| **Model create + semantic enrichment** | **~5 min** | **LLM, sequential** |
+| Liveboard create + enhance | ~2 min | MCP + LLM titles |
+
+The model phase is NOT ThoughtSpot: model TML import is near-instant. The time
+is `model_semantic_updater.py` — per-column description + 3–5 synonyms +
+Spotter `ai_context` for every model column, generated in **sequential batches
+of 25 columns per LLM call** (a 60–110 column model = 3–5 back-to-back calls),
+plus one model-description call.
+
+Optimization options (deliberately NOT implemented — decision 2026-08-10 was
+to hold code changes):
+
+1. **Parallelize the batches** — the batch calls are independent; running them
+   concurrently collapses ~4–5 min to roughly the latency of one call (~1 min).
+   Contained change in `model_semantic_updater.generate_column_semantics`.
+   Biggest win, lowest risk.
+2. **Overlap phases** — column set is known from the blueprint before Snowflake
+   loads; semantics could generate during the data-load wait and be applied
+   after model creation. Bigger restructuring, saves the same wall-clock.
+3. **Cheaper model** — descriptions/synonyms are formulaic; a Haiku-class model
+   would cut latency and cost. Quality check needed on `ai_context` hints.
+
+With option 1 alone, a solo build lands ~5–6 min end to end.

sprint_2026_04.md

@@ -257,6 +257,8 @@ should tell. KPI targets, growth trends, and anomaly patterns live in the matrix
 ### Sprint 4 (this sprint)
 
 - [x] **Table-create latency root cause + fix (Aug 10)** — ~250s/import was the unscoped TS connection scanning all ~507 Snowflake DBs visible to SE_ROLE (measured identical on sebe AND secloud; NOT instance load, July SRE docs corrected). Fix: (1) connection TML now scoped via required `database` property (`create_connection_tml`), (2) demos write to monthly-rotating DB `<SNOWFLAKE_DATABASE>_<YYYY_MM>` (`get_demo_database()`/`ensure_demo_database()` in `snowflake_auth.py`) since bloated DEMOBUILD (1,003 schemas) still cost 33s vs 0.7s fresh. Probe: `tests/ts_table_create_deepdive.py` ✅
+- [x] **Perf fix verified + deployed (Aug 10)** — app e2e: 655s Grade B, 0 timeouts (pre-fix record: 1,084–2,711s, 12/24 timeouts). Deployed to `hf-test` AND MCP Space `thoughtspot-demoprep/mcp` (`df44611`, rollback `3d735c7`); MCP Tixr rehearsal on new code: PASS, 9.2 min vs 16–20 min. Full research notes: `docs/build_time_deep_dive_2026_08_10.md` ✅
+- [ ] **Next bottleneck: model semantics ~5 min** — `model_semantic_updater.generate_column_semantics()` runs sequential LLM batches (25 cols/call); parallelizing collapses it to ~1 min → ~5–6 min total builds. Deliberately deferred (no code changes during demo window); see deep-dive doc §5
 
 - [x] **Branching / Release Strategy** — `main`/`develop`/hotfix workflow documented in CLAUDE.md; `develop` branch confirmed; `v1.0.0` tagged on `main`; `hf-test` remote wired ✅
 - [x] **Settings Redesign — accordion** — Right panel: TS Environment + AI Model always visible; collapsible `⚙️ Settings` accordion with Liveboard Name, Data Size, Geo Scope, Tag Name, Column Naming Style, Object Naming Prefix, Share With ✅
@@ -289,4 +291,4 @@ should tell. KPI targets, growth trends, and anomaly patterns live in the matrix
 
 ---
 
-*Last updated: July 23, 2026 — share fix (`$message` required-variable 400 on model + liveboard auto-share)*
+*Last updated: August 10, 2026 — table-create perf fix verified + deployed (test + MCP); build-time deep dive documented*