docs: portal/new-vision session notes

- docs/: AgentSpot portal handoff (8/24), new-vision execution notes,
temperature tuning notes (from parallel work sessions)
- demo_content/legal_demo intentionally NOT tracked: it is its own git repo
(would only commit as an empty gitlink)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/agentspot_portal_handoff_2026_08_24.md

@@ -0,0 +1,76 @@
+# Handoff — AgentSpot-first architecture + portal (`build_portal`) work
+*Written 2026-08-25 (session of 2026-08-24). Branch: `develop_dg` in `demoprep_dg`. Nothing committed from this session — this doc and one memory file are the only outputs.*
+
+---
+
+## Where this session ended
+
+Discussion phase is **done**. Direction is agreed with boone; next session starts **building**, pending three small decisions (bottom of this doc).
+
+## Context shift (important — supersedes older framing)
+
+- **AgentSpot** (new ThoughtSpot product, CEO-pushed) is now the **primary consumer** of DemoPrep. Most users call the MCP from AgentSpot; few use the Gradio app directly.
+- The MCP is the future surface of this project. The app is **not** being retired or given away, but its likely future is admin console + run history viewer.
+- boone will supply AgentSpot docs/research. **Do not research AgentSpot independently** — he asked us not to. Known so far (unverified, from his account): tool-call timeout ~**480s**; AgentSpot's `status()` polling **resets that timer** — the polling pattern is load-bearing, not just UX.
+- Memory file: `agentspot-first-direction.md` in the project memory dir.
+
+## Architecture review (published, still current)
+
+Full review artifact: https://claude.ai/code/artifact/e9726077-27e8-4798-8ab5-940df582280b
+
+Key conclusions (all verified against code this session):
+
+1. **Do not rewrite the app on the MCP — the dependency already points the right way.** `mcp_server.py:410` constructs `ChatDemoInterface(user_email=owner)`, the same controller the app uses (`chat_interface.py:6110`). The MCP is a 656-line thin adapter.
+2. **Real problem:** the engine is trapped in `chat_interface.py` (7,137 lines, UI + orchestration). `demoprep_app/controllers/chat.py` is an 8-line shim importing back OUT of the monolith. Headless callers must import Gradio and set `DEMOPREP_NO_AUTH=true`. Fix = extract `run_build()` into `demoprep_app/pipeline/`, using `_run_build()` in `mcp_server.py` as the spec — it already names every input/output.
+3. **The 3-build cap is a guess, not a measurement.** `mcp_server.py:162`: `MCP_MAX_CONCURRENT_BUILDS` env var, default `"3"`. The app allows 10 for the same work (`app.py:97`). Builds are I/O-bound (LLM + Snowflake + TS REST), 1k–50k rows. Recommendation: raise to 8 on the MCP Space now (env var only, no code), measure, reconcile the two limits.
+4. **Concurrency safety is fine** — per-build controller/owner/TS-target, no shared mutable state, env written once at startup. Nothing to unpick before raising the cap.
+5. **Blocker for wide rollout: in-memory build state.** `_active_builds` / `_recent_builds` (cap 50) in `mcp_server.py` reset on Space restart — in-flight 15–20 min builds vanish silently. Completed runs DO persist (`demo_history` via `supabase_client.py:251`; accepted briefs via `log_mcp_payload()` at `:765`). Durable *live* run state is the prerequisite for everything else (multi-container, `build_portal(run_id)`, restart-safe `status()`).
+6. **Decompose MCP tools by user job, not pipeline stage.** boone floated separate create-data / create-liveboard calls; we argued (and he accepted) that agents orchestrating multi-stage 20-min workflows is fragile. Keep `build_demo_from_brief` as the flagship one-shot; add tools per real job: `build_portal` (below), later `regenerate_liveboard`.
+7. Pre-rollout checklist items still open: per-user MCP auth (currently one shared bearer; `MCP_SHARE_WITH` hardcodes boone's email), schema-name collision window at `chat_interface.py:2945` (second-granularity — add run_id), Snowflake cost/debris ownership, visible queue story instead of "busy, retry".
+
+## The new work: `build_portal` (the portal-rebrand skill → MCP tool)
+
+### What the skill is
+`rebrand-thoughtspot-portal` in **github.com/thoughtspot/tse_demos** (repo is PUBLIC — read it directly, no auth needed). Files: `SKILL.md`, `scripts/apply-spec.mjs`, `references/{spec-schema.md, spec.example.json, build-playbook.md, intake.md}`, plus `template-tse/` at repo root (Vite + React + @thoughtspot/visual-embed-sdk, brand-neutral "Northwind" build with all features toggleable).
+
+Shape: **interview → `spec.json` → deterministic codemod (~90%, <1s) → hand-finish → `npm run build`**. This is DemoPrep's own house pattern (LLM authors blueprint → deterministic engine executes).
+
+### The key insight (drives the whole design)
+Everything the skill's interview asks for, **DemoPrep already knows authoritatively because it created it**: host URL, liveboard GUID (guaranteed to be a liveboard — kills the skill's #1 blank-embed failure mode), model GUID, filter/date/metric columns (from the blueprint), company name/site/logo (from research/`website_data`), brand palette (liveboard creator already derives it). So: **don't absorb the skill — absorb the spec contract.** DemoPrep authors `spec.json` from a run; the interview collapses to preference questions.
+
+### Agreed v1 design (one deliverable — boone rejected two phases; they collapse anyway since spec generation IS step 1 of the tool)
+```
+build_portal(run_id) → author spec.json from run result → run apply-spec.mjs codemod
+  → npm run build → create/update STATIC HF Space thoughtspot-?/<company>-portal
+  → return URL (async: returns immediately, AgentSpot polls status())
+```
+- **Hosting = static HF Spaces** (boone's call: no Vercel unless forced; already paying HF). Confirmed feasible from this repo: `huggingface_hub==0.26.2` already in requirements.txt, and a static Space already exists in their org (`thoughtspot-dp-demoprep-doc.static.hf.space`, referenced at `chat_interface.py:5061`). Static Spaces are free. URL scheme: `<owner>-<name>.static.hf.space`.
+- **Scope = the skill's "Basic" mode only** (Analytics + Spotter + Ask-AI + theme) — 100% codemod, zero hand-finish. Advanced mode (custom-action modals = mandatory hand-rebuild) stays a Claude Code skill flow, fed by the same spec.
+- **Vendor `template-tse` pinned to a commit** in the MCP image; add node/npm to the Dockerfile (currently `python:3.11-slim` + git + curl). Deliberate upgrades only — the template is Mani's team's and evolves.
+- **Auth v1 = "rides existing TS session"** (static site can't hold secrets; SEs are logged into the cluster). Trusted-auth portals need a token endpoint — the MCP Space could serve one later; NOT v1.
+
+### The two unknowns to verify FIRST (before writing the tool)
+1. **CSP/CORS wildcard**: does ThoughtSpot's CSP visual-embed-hosts / CORS allowlist accept `*.hf.space` (or `*.static.hf.space`)? If yes → one-time setup per cluster, fully self-serve. If exact-host-only → `build_portal` must return the exact allowlist instruction (or automate via security-settings API on instances we control). **Test on sebe first.**
+2. **Local end-to-end proof**: hand-write one `spec.json` from a real recent run (pull GUIDs/columns from `demo_history` or a fresh build), run `apply-spec.mjs` + `npm run build` locally, push to a throwaway static Space, load it against sebe. Proves codemod + hosting + embed before any MCP code.
+
+## Decisions boone still owes (asked, not yet answered — ask again at session start)
+1. **Which HF org** do portals land in — `thoughtspot-dp` (app + docs Spaces) or `thoughtspot-demoprep` (MCP Space)?
+2. **Public or private** Spaces for v1? (Private requires HF login to view — awkward mid-demo; public is likely fine since data still requires TS auth.)
+3. Go-ahead on the verification spike (item above) — he was asked "want me to start?" and the session ended before an answer.
+
+## Sequenced plan (agreed order — each unblocks the next)
+1. Raise `MCP_MAX_CONCURRENT_BUILDS` to 8 on the MCP Space (env var, minutes) + instrument per-build memory/CPU.
+2. **Durable run state** in Supabase (phase/timestamps/result as the build progresses — extend the existing `demo_history` / `log_mcp_payload` pattern). Prerequisite for `build_portal(run_id)` and restart-safe `status()`.
+3. Extract `run_build()` into `demoprep_app/pipeline/` (use `mcp_server.py::_run_build` as spec; point MCP at it first, then the app's GO button; delete `DEMOPREP_NO_AUTH`). Behind the e2e suite.
+4. `build_portal` v1 per the design above.
+5. Per-user MCP identity before company-wide rollout (shared bearer → per-user; fix `MCP_SHARE_WITH` hardcode).
+(1, the CSP test, and 2 can run in parallel; 4 needs 2.)
+
+## Standing constraints (from CLAUDE.md — do not violate)
+- Never push to any remote without explicit instruction; never restart the app.
+- venv: `source ./demoprep/bin/activate` (this clone also has `.venv/` — check which is live).
+- Scratch/temp files → `scratch/`; docs → `dev_notes/` or `docs/`; ask before new root files.
+- e2e runs as mike.boone (TEST_USER in .env) on sebe; never edit `.env` without per-change approval.
+
+## Uncommitted state in this clone (pre-existing, not from this session)
+Modified: `tests/quality_results/latest_prod_summary.md`, `latest_test_summary.md`; untracked: five `tests/quality_results/*_quality_run.md` files (Aug 7–10). Quality-run artifacts — leave for boone.

docs/new_vision_execution_notes.md

@@ -0,0 +1,96 @@
+## DemoPrep New Vision 2 - Execution Notes
+
+This document explains the current pipeline, where call volume comes from, and what to improve next.
+
+### 1) End-to-end flow (current)
+
+1. User starts from chat or UI flow.
+2. Research stage builds company + industry context.
+3. DDL stage generates schema.
+4. Population stage runs LegitData (classification -> value generation -> repair -> quality gate -> write).
+5. ThoughtSpot deploy stage creates connection/tables/model.
+6. Liveboard stage (HYBRID):
+   - MCP builds initial answers/liveboard.
+   - TML post-processing polishes layout, groups, KPIs, and color.
+
+Primary files:
+- `chat_interface.py`
+- `legitdata_bridge.py`
+- `legitdata_project/legitdata/generator.py`
+- `thoughtspot_deployer.py`
+- `liveboard_creator.py`
+
+### 2) Why junk categorical data appeared
+
+Observed failures (for example: branch/employee/department-like values) were primarily from fallback paths:
+- Name-like columns were sometimes treated as generic person names or weak fallbacks.
+- Some business dimensions (for example `department`) were not semantically typed strongly enough.
+- Validation focused on classic business categorical enums, but not enough on org/person realism.
+
+New Vision 2 fixes now in place:
+- Stronger semantic routing for branch/department/person patterns.
+- Domain-backed generation for branch/org/department semantics.
+- Stricter sanitization for name-like business columns.
+- Validator/repair coverage extended for these semantics.
+
+### 3) LLM/API call budget model
+
+Call volume depends heavily on table/column count and how often AI/search fallback is needed.
+
+Deterministic-ish stages:
+- Research: usually ~2 LLM calls (company + industry).
+- DDL: usually ~1 LLM call.
+- Liveboard questions: ~1 LLM call for question generation (if AI path enabled).
+- Liveboard answers: roughly one ThoughtSpot answer call per question (`num_visualizations`, often 6-8).
+
+Variable stages (largest source of volume):
+- LegitData classification/generation:
+  - AI classification calls (column-dependent).
+  - AI value generation calls for `AI_GEN` columns (chunked by `MAX_AI_ROWS_PER_CALL`).
+  - Web search calls for `SEARCH_REAL` columns.
+  - Fallback may reduce quality and also shift call mix.
+
+Practical planning heuristic per run:
+- Small run: 10-30 LLM-equivalent calls.
+- Medium run: 30-80 LLM-equivalent calls.
+- Large run (many AI_GEN/search columns): 80+.
+
+### 4) How to measure calls today
+
+Existing instrumentation:
+- Prompt logs are written to `logs/prompts/*.md`.
+- `prompt_logger.py` captures stage, model, timing, and token usage (when available).
+
+Recommended operational check:
+1. Run one demo end-to-end.
+2. Open latest prompt log file.
+3. Summarize by stage:
+   - `research_company`
+   - `research_industry`
+   - `ddl`
+   - `liveboard_questions`
+   - any population-related logged calls
+4. Record:
+   - call count by stage
+   - total tokens in/out
+   - longest stage duration
+
+### 5) Improvement backlog (priority order)
+
+1. Add a per-run call budget report in UI output:
+   - counts by stage
+   - token totals
+   - top slow calls
+2. Add stricter quality gates for semantic text realism:
+   - reject single-token junk for person/org-like dimensions
+   - require domain membership for strict business categorical fields
+3. Add hybrid post-MCP enhancements:
+   - deterministic chart diversity targets
+   - safer dimension-selection heuristics for noisy fields
+   - stronger duplicate-question prevention
+4. Add automatic "bad-dimension suppression" list at liveboard time:
+   - hide suspect columns if data quality score is below threshold
+
+### 6) Branching/project note
+
+`DemoPrep_new_vision2` is intended as a parallel workspace for faster iteration on realism + hybrid polish without blocking the original project flow.

docs/newvision_temperature_tuning.md

@@ -0,0 +1,37 @@
+## New Vision Temperature / Predictability Notes
+
+This note captures deferred guidance for LLM creativity settings in demo generation.
+
+### Recommendation (Default)
+- Keep generation deterministic-leaning for structure-critical stages.
+- Suggested defaults:
+  - Research: `temperature=0.3-0.4`
+  - DDL creation: `temperature=0.15-0.25`
+  - Data value generation (AI-assisted columns): `temperature=0.2-0.35`
+  - Liveboard question generation: `temperature=0.4-0.6`
+
+### Why
+- Lower temperature improves repeatability and reduces schema/data drift.
+- Higher temperature can produce novelty but often increases variance, noise, and QA repair load.
+
+### Deferred Experiment Plan
+Run the same 4-case sample set across 3 profiles and compare quality reports + liveboard usefulness:
+
+1. **Stable profile**
+   - research 0.3, ddl 0.2, data 0.2, liveboard 0.45
+2. **Balanced profile**
+   - research 0.4, ddl 0.25, data 0.3, liveboard 0.6
+3. **Creative profile**
+   - research 0.6, ddl 0.35, data 0.45, liveboard 0.75
+
+### Compare Metrics
+- semantic pass ratio
+- volatility breaches
+- smoothness score
+- outlier explainability
+- kpi consistency
+- number of repair actions
+
+### Current Decision
+- Do not implement temperature-control rewiring in this coding pass.
+- Revisit after reviewing the first 4 sample outputs.