docs/new_vision_execution_notes.md

## DemoPrep New Vision 2 - Execution Notes

This document explains the current pipeline, where call volume comes from, and what to improve next.

### 1) End-to-end flow (current)

1. User starts from chat or UI flow.
2. Research stage builds company + industry context.
3. DDL stage generates schema.
4. Population stage runs LegitData (classification -> value generation -> repair -> quality gate -> write).
5. ThoughtSpot deploy stage creates connection/tables/model.
6. Liveboard stage (HYBRID):
   - MCP builds initial answers/liveboard.
   - TML post-processing polishes layout, groups, KPIs, and color.

Primary files:
- `chat_interface.py`
- `legitdata_bridge.py`
- `legitdata_project/legitdata/generator.py`
- `thoughtspot_deployer.py`
- `liveboard_creator.py`

### 2) Why junk categorical data appeared

Observed failures (for example: branch/employee/department-like values) were primarily from fallback paths:
- Name-like columns were sometimes treated as generic person names or weak fallbacks.
- Some business dimensions (for example `department`) were not semantically typed strongly enough.
- Validation focused on classic business categorical enums, but not enough on org/person realism.

New Vision 2 fixes now in place:
- Stronger semantic routing for branch/department/person patterns.
- Domain-backed generation for branch/org/department semantics.
- Stricter sanitization for name-like business columns.
- Validator/repair coverage extended for these semantics.

### 3) LLM/API call budget model

Call volume depends heavily on table/column count and how often AI/search fallback is needed.

Deterministic-ish stages:
- Research: usually ~2 LLM calls (company + industry).
- DDL: usually ~1 LLM call.
- Liveboard questions: ~1 LLM call for question generation (if AI path enabled).
- Liveboard answers: roughly one ThoughtSpot answer call per question (`num_visualizations`, often 6-8).

Variable stages (largest source of volume):
- LegitData classification/generation:
  - AI classification calls (column-dependent).
  - AI value generation calls for `AI_GEN` columns (chunked by `MAX_AI_ROWS_PER_CALL`).
  - Web search calls for `SEARCH_REAL` columns.
  - Fallback may reduce quality and also shift call mix.

Practical planning heuristic per run:
- Small run: 10-30 LLM-equivalent calls.
- Medium run: 30-80 LLM-equivalent calls.
- Large run (many AI_GEN/search columns): 80+.

### 4) How to measure calls today

Existing instrumentation:
- Prompt logs are written to `logs/prompts/*.md`.
- `prompt_logger.py` captures stage, model, timing, and token usage (when available).

Recommended operational check:
1. Run one demo end-to-end.
2. Open latest prompt log file.
3. Summarize by stage:
   - `research_company`
   - `research_industry`
   - `ddl`
   - `liveboard_questions`
   - any population-related logged calls
4. Record:
   - call count by stage
   - total tokens in/out
   - longest stage duration

### 5) Improvement backlog (priority order)

1. Add a per-run call budget report in UI output:
   - counts by stage
   - token totals
   - top slow calls
2. Add stricter quality gates for semantic text realism:
   - reject single-token junk for person/org-like dimensions
   - require domain membership for strict business categorical fields
3. Add hybrid post-MCP enhancements:
   - deterministic chart diversity targets
   - safer dimension-selection heuristics for noisy fields
   - stronger duplicate-question prevention
4. Add automatic "bad-dimension suppression" list at liveboard time:
   - hide suspect columns if data quality score is below threshold

### 6) Branching/project note

`DemoPrep_new_vision2` is intended as a parallel workspace for faster iteration on realism + hybrid polish without blocking the original project flow.