Spaces:
Running
Running
File size: 3,777 Bytes
eaeb114 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | ## DemoPrep New Vision 2 - Execution Notes
This document explains the current pipeline, where call volume comes from, and what to improve next.
### 1) End-to-end flow (current)
1. User starts from chat or UI flow.
2. Research stage builds company + industry context.
3. DDL stage generates schema.
4. Population stage runs LegitData (classification -> value generation -> repair -> quality gate -> write).
5. ThoughtSpot deploy stage creates connection/tables/model.
6. Liveboard stage (HYBRID):
- MCP builds initial answers/liveboard.
- TML post-processing polishes layout, groups, KPIs, and color.
Primary files:
- `chat_interface.py`
- `legitdata_bridge.py`
- `legitdata_project/legitdata/generator.py`
- `thoughtspot_deployer.py`
- `liveboard_creator.py`
### 2) Why junk categorical data appeared
Observed failures (for example: branch/employee/department-like values) were primarily from fallback paths:
- Name-like columns were sometimes treated as generic person names or weak fallbacks.
- Some business dimensions (for example `department`) were not semantically typed strongly enough.
- Validation focused on classic business categorical enums, but not enough on org/person realism.
New Vision 2 fixes now in place:
- Stronger semantic routing for branch/department/person patterns.
- Domain-backed generation for branch/org/department semantics.
- Stricter sanitization for name-like business columns.
- Validator/repair coverage extended for these semantics.
### 3) LLM/API call budget model
Call volume depends heavily on table/column count and how often AI/search fallback is needed.
Deterministic-ish stages:
- Research: usually ~2 LLM calls (company + industry).
- DDL: usually ~1 LLM call.
- Liveboard questions: ~1 LLM call for question generation (if AI path enabled).
- Liveboard answers: roughly one ThoughtSpot answer call per question (`num_visualizations`, often 6-8).
Variable stages (largest source of volume):
- LegitData classification/generation:
- AI classification calls (column-dependent).
- AI value generation calls for `AI_GEN` columns (chunked by `MAX_AI_ROWS_PER_CALL`).
- Web search calls for `SEARCH_REAL` columns.
- Fallback may reduce quality and also shift call mix.
Practical planning heuristic per run:
- Small run: 10-30 LLM-equivalent calls.
- Medium run: 30-80 LLM-equivalent calls.
- Large run (many AI_GEN/search columns): 80+.
### 4) How to measure calls today
Existing instrumentation:
- Prompt logs are written to `logs/prompts/*.md`.
- `prompt_logger.py` captures stage, model, timing, and token usage (when available).
Recommended operational check:
1. Run one demo end-to-end.
2. Open latest prompt log file.
3. Summarize by stage:
- `research_company`
- `research_industry`
- `ddl`
- `liveboard_questions`
- any population-related logged calls
4. Record:
- call count by stage
- total tokens in/out
- longest stage duration
### 5) Improvement backlog (priority order)
1. Add a per-run call budget report in UI output:
- counts by stage
- token totals
- top slow calls
2. Add stricter quality gates for semantic text realism:
- reject single-token junk for person/org-like dimensions
- require domain membership for strict business categorical fields
3. Add hybrid post-MCP enhancements:
- deterministic chart diversity targets
- safer dimension-selection heuristics for noisy fields
- stronger duplicate-question prevention
4. Add automatic "bad-dimension suppression" list at liveboard time:
- hide suspect columns if data quality score is below threshold
### 6) Branching/project note
`DemoPrep_new_vision2` is intended as a parallel workspace for faster iteration on realism + hybrid polish without blocking the original project flow.
|