docs/new_vision_execution_notes.md

DemoPrep New Vision 2 - Execution Notes

This document explains the current pipeline, where call volume comes from, and what to improve next.

1) End-to-end flow (current)

1. User starts from chat or UI flow.
2. Research stage builds company + industry context.
3. DDL stage generates schema.
4. Population stage runs LegitData (classification -> value generation -> repair -> quality gate -> write).
5. ThoughtSpot deploy stage creates connection/tables/model.
6. Liveboard stage (HYBRID):
   • MCP builds initial answers/liveboard.
   • TML post-processing polishes layout, groups, KPIs, and color.

Primary files:

• chat_interface.py
• legitdata_bridge.py
• legitdata_project/legitdata/generator.py
• thoughtspot_deployer.py
• liveboard_creator.py

2) Why junk categorical data appeared

Observed failures (for example: branch/employee/department-like values) were primarily from fallback paths:

• Name-like columns were sometimes treated as generic person names or weak fallbacks.
• Some business dimensions (for example department) were not semantically typed strongly enough.
• Validation focused on classic business categorical enums, but not enough on org/person realism.

New Vision 2 fixes now in place:

• Stronger semantic routing for branch/department/person patterns.
• Domain-backed generation for branch/org/department semantics.
• Stricter sanitization for name-like business columns.
• Validator/repair coverage extended for these semantics.

3) LLM/API call budget model

Call volume depends heavily on table/column count and how often AI/search fallback is needed.

Deterministic-ish stages:

• Research: usually ~2 LLM calls (company + industry).
• DDL: usually ~1 LLM call.
• Liveboard questions: ~1 LLM call for question generation (if AI path enabled).
• Liveboard answers: roughly one ThoughtSpot answer call per question (num_visualizations, often 6-8).

Variable stages (largest source of volume):

• LegitData classification/generation:
  • AI classification calls (column-dependent).
  • AI value generation calls for AI_GEN columns (chunked by MAX_AI_ROWS_PER_CALL).
  • Web search calls for SEARCH_REAL columns.
  • Fallback may reduce quality and also shift call mix.

Practical planning heuristic per run:

• Small run: 10-30 LLM-equivalent calls.
• Medium run: 30-80 LLM-equivalent calls.
• Large run (many AI_GEN/search columns): 80+.

4) How to measure calls today

Existing instrumentation:

• Prompt logs are written to logs/prompts/*.md.
• prompt_logger.py captures stage, model, timing, and token usage (when available).

Recommended operational check:

1. Run one demo end-to-end.
2. Open latest prompt log file.
3. Summarize by stage:
   • research_company
   • research_industry
   • ddl
   • liveboard_questions
   • any population-related logged calls
4. Record:
   • call count by stage
   • total tokens in/out
   • longest stage duration

5) Improvement backlog (priority order)

1. Add a per-run call budget report in UI output:
   • counts by stage
   • token totals
   • top slow calls
2. Add stricter quality gates for semantic text realism:
   • reject single-token junk for person/org-like dimensions
   • require domain membership for strict business categorical fields
3. Add hybrid post-MCP enhancements:
   • deterministic chart diversity targets
   • safer dimension-selection heuristics for noisy fields
   • stronger duplicate-question prevention
4. Add automatic "bad-dimension suppression" list at liveboard time:
   • hide suspect columns if data quality score is below threshold

6) Branching/project note

DemoPrep_new_vision2 is intended as a parallel workspace for faster iteration on realism + hybrid polish without blocking the original project flow.