feat: replace Spotter Viz method with Spotter Viz Story tab output

Remove SPOTTER_VIZ as a liveboard creation method (HYBRID is now the
only method). Add post-liveboard Spotter Viz Story generation — a
conversational sequence of NL prompts for ThoughtSpot's Spotter Viz
agent, displayed in a new Gradio tab. Update .gitignore to allow
legitdata_project source files to be tracked.

Co-authored-by: Cursor <cursoragent@cursor.com>

.gitignore

@@ -221,5 +221,7 @@ scratch/
 
 # Sprint documents (local working docs)
 
-# LegitData project (third-party/separate project - needs proper integration strategy)
-legitdata_project/
+# LegitData project - track source, ignore build artifacts
+legitdata_project/venv/
+legitdata_project/*.egg-info/
+legitdata_project/.DS_Store

CLAUDE.md

@@ -195,37 +195,22 @@ When user says "create a test for X":
 ```
 Liveboard Creation:
   HYBRID: chat_interface.py → create_liveboard_from_model_mcp() → enhance_mcp_liveboard()
-  SPOTTER_VIZ: chat_interface.py → create_liveboard_from_model() → enhance_mcp_liveboard()
 
 DO NOT use create_visualization_tml() directly - that's internal low-level code
 ```
 
 ---
 
-## Liveboard Creation - Two-Method System
+## Liveboard Creation — HYBRID Method
 
-**Settings UI:** Admin tab → "Liveboard Creation Method" dropdown
-**Two options:** HYBRID (default) and SPOTTER_VIZ
+**Single method:** HYBRID (MCP + TML post-processing)
 
-| Method | Speed | Dependency | Best For |
-|--------|-------|------------|----------|
-| **HYBRID** | ~60-90s | MCP server | AI-driven question selection |
-| **SPOTTER_VIZ** | ~20-30s | Direct REST API only | Production demos, reliability |
-
-### HYBRID Method (MCP + TML Post-Processing)
+### Pipeline
 1. MCP creates liveboard via `agent.thoughtspot.app` (bearer auth)
-2. TML post-processing enhances with groups, KPIs, colors, layout
+2. `enhance_mcp_liveboard()` post-processes with groups, KPIs, colors, layout
 - **Entry:** `create_liveboard_from_model_mcp()` → `enhance_mcp_liveboard()`
 
-### SPOTTER_VIZ Method (Direct API + TML)
-1. LiveboardCreator builds complete TML from outlier patterns + AI
-2. Deploys via REST API `/metadata/tml/import`
-3. Same TML post-processing as HYBRID (groups, KPIs, colors, layout)
-- **Entry:** `create_liveboard_from_model()` → `enhance_mcp_liveboard()`
-- No MCP dependency — uses same auth as table/model deployment
-
-### Shared Post-Processing: enhance_mcp_liveboard()
-Both methods share the same post-processing function:
+### Post-Processing: enhance_mcp_liveboard()
 1. Exports the liveboard TML
 2. Classifies visualizations by type (KPI, trend, categorical)
 3. Adds Groups with proper `group_layouts` (Golden Demo style)
@@ -234,9 +219,13 @@ Both methods share the same post-processing function:
 6. Applies brand colors (liveboard-level + group-level)
 7. Re-imports the enhanced TML
 
-### Backward Compatibility
-- Old settings values "TML" or "MCP" are mapped to "HYBRID" in the router
-- `USE_MCP_LIVEBOARD=true` env var still works (maps to HYBRID)
+### Spotter Viz Story (Post-Liveboard Output)
+After liveboard creation, a **Spotter Viz Story** is generated:
+- A conversational sequence of natural language prompts for ThoughtSpot's Spotter Viz agent
+- Uses company context, use case, outlier patterns, and liveboard visualizations
+- Displayed in the "Spotter Viz Story" tab in the app
+- Can be manually entered into Spotter Viz to recreate/refine the liveboard
+- Future: will be automated when the Spotter Viz API is published
 
 ### KPI Requirements
 - **For sparklines and percent change comparisons:**

chat_interface.py

@@ -149,6 +149,7 @@ class ChatDemoInterface:
         # New tab content
         self.live_progress_log = []  # Real-time deployment progress
         self.demo_pack_content = ""   # Generated demo pack markdown
+        self.spotter_viz_story = ""   # Spotter Viz story (NL prompts for Spotter Viz agent)
     
     def load_default_settings(self):
         """Load settings from Supabase or defaults"""
@@ -1937,6 +1938,135 @@ To change settings, use:
 - **Ask questions**: Let the AI demonstrate natural language
 - **End with action**: Show how insights lead to decisions""")
     
+    def _generate_spotter_viz_story(self, company_name: str, use_case: str, 
+                                     model_name: str = None, liveboard_name: str = None) -> str:
+        """Generate a Spotter Viz story — a conversational sequence of NL prompts
+        that can be entered into ThoughtSpot's Spotter Viz agent to build a liveboard.
+        
+        Uses the build_prompt() system with stage="spotter_viz_story" + LLM call.
+        Falls back to a template-based story if LLM fails.
+        """
+        try:
+            from prompts import build_prompt
+            from demo_personas import parse_use_case
+            
+            v, f = parse_use_case(use_case or '')
+            vertical = v or "Generic"
+            function = f or "Generic"
+            
+            # Build company context for the prompt
+            company_context = f"Company: {company_name}\nUse Case: {use_case}"
+            if model_name:
+                company_context += f"\nData Source/Model: {model_name}"
+            if liveboard_name:
+                company_context += f"\nLiveboard Name: {liveboard_name}"
+            
+            # Add research context if available
+            if hasattr(self, 'demo_builder') and self.demo_builder:
+                research = getattr(self.demo_builder, 'company_summary', '') or ''
+                if research:
+                    company_context += f"\n\nCompany Research:\n{research[:1500]}"
+            
+            prompt = build_prompt(
+                stage="spotter_viz_story",
+                vertical=vertical,
+                function=function,
+                company_context=company_context,
+            )
+            
+            # Make LLM call
+            from litellm import completion
+            llm_model = self.settings.get('model', 'claude-sonnet-4')
+            self.log_feedback(f"🎬 Generating Spotter Viz story ({llm_model})...")
+            
+            response = completion(
+                model=llm_model,
+                messages=[{"role": "user", "content": prompt}],
+                max_tokens=2000,
+                temperature=0.7,
+            )
+            
+            story = response.choices[0].message.content.strip()
+            
+            # Add header
+            header = f"""# Spotter Viz Story: {company_name}
+## {use_case}
+
+*Copy these prompts into ThoughtSpot Spotter Viz to build this liveboard interactively.*
+
+---
+
+"""
+            return header + story
+            
+        except Exception as e:
+            self.log_feedback(f"⚠️ Spotter Viz story generation failed: {e}")
+            # Fallback: build a basic template from what we know
+            return self._build_fallback_spotter_story(company_name, use_case, model_name)
+    
+    def _build_fallback_spotter_story(self, company_name: str, use_case: str, 
+                                       model_name: str = None) -> str:
+        """Build a basic Spotter Viz story without LLM, using available context."""
+        data_source = model_name or f"{company_name} model"
+        
+        # Get spotter questions from outlier system
+        spotter_qs = []
+        try:
+            from demo_personas import parse_use_case
+            from outlier_system import get_outliers_for_use_case
+            v, f = parse_use_case(use_case or '')
+            if v or f:
+                outlier_config = get_outliers_for_use_case(v or "Generic", f or "Generic")
+                for op in outlier_config.required:
+                    for sq in op.spotter_questions[:1]:
+                        spotter_qs.append(sq)
+        except:
+            pass
+        
+        story = f"""# Spotter Viz Story: {company_name}
+## {use_case}
+
+*Copy these prompts into ThoughtSpot Spotter Viz to build this liveboard interactively.*
+
+---
+
+### Step 1: Set Context
+> "Create a new liveboard for {company_name} {use_case} using the {data_source} data source."
+
+**Expected result:** Empty liveboard created with the correct data source connected.
+
+### Step 2: Add Key KPIs
+> "Add KPI cards showing the main metrics with weekly sparklines."
+
+**Expected result:** KPI tiles with sparkline trends at the top of the liveboard.
+
+### Step 3: Add Trend Analysis
+> "Add a line chart showing how the primary metric has trended over the last 12 months."
+
+**Expected result:** Time-series visualization showing monthly trends.
+
+### Step 4: Add Category Breakdown
+> "Show a bar chart breaking down performance by the main dimension."
+
+**Expected result:** Categorical breakdown chart.
+
+### Step 5: Add Comparison
+> "Add a comparison showing this period vs. last period."
+
+**Expected result:** Period-over-period comparison visualization.
+"""
+        if spotter_qs:
+            story += "\n### Step 6: Explore with Spotter Questions\n"
+            for i, q in enumerate(spotter_qs[:3]):
+                story += f'> "{q}"\n\n'
+        
+        story += """
+---
+
+*Refine the liveboard further by asking Spotter Viz to adjust colors, reorganize tiles, or add filters.*
+"""
+        return story
+
     def run_research(self, company, use_case):
         """Run the research phase"""
         import time
@@ -3383,6 +3513,22 @@ Ask these questions to showcase ThoughtSpot's AI capabilities:
                 except Exception as e:
                     safe_print(f"Could not generate demo pack: {e}", flush=True)
                     self.demo_pack_content = f"*Demo pack generation failed: {e}*"
+                
+                # Generate Spotter Viz Story
+                try:
+                    model_name_for_story = results.get('model', None)
+                    liveboard_name_for_story = results.get('liveboard', None)
+                    self.spotter_viz_story = self._generate_spotter_viz_story(
+                        company_name=company_name,
+                        use_case=use_case,
+                        model_name=model_name_for_story,
+                        liveboard_name=liveboard_name_for_story
+                    )
+                    safe_print("Spotter Viz Story generated - check the Spotter Viz Story tab.", flush=True)
+                    self.live_progress_log.append("Spotter Viz Story generated")
+                except Exception as e:
+                    safe_print(f"Could not generate Spotter Viz story: {e}", flush=True)
+                    self.spotter_viz_story = f"*Spotter Viz story generation failed: {e}*"
             
             # Build final response
             if results.get('success'):
@@ -3769,6 +3915,14 @@ def create_chat_interface():
                     elem_classes=["demo-pack-content"]
                 )
             
+            with gr.Tab("🎬 Spotter Viz Story"):
+                gr.Markdown("### Spotter Viz Story — Natural Language Liveboard Builder")
+                gr.Markdown("*Conversational prompts you can enter into ThoughtSpot Spotter Viz to recreate this liveboard.*")
+                spotter_viz_story_display = gr.Markdown(
+                    value="Spotter Viz story will be generated after liveboard creation.\n\n**What is Spotter Viz?**\nSpotter Viz is an AI agent in ThoughtSpot that creates, structures, and styles Liveboards through natural language prompts. The agent reviews the data, proposes layouts, generates KPIs and visualizations, and allows conversational refinement.",
+                    elem_classes=["spotter-viz-story-content"]
+                )
+            
             with gr.Tab("⚙️ Settings"):
                 settings_components = create_settings_tab()
             
@@ -3792,13 +3946,16 @@ def create_chat_interface():
                 )
         
         # Create update function for tabs
+        spotter_viz_default = "Spotter Viz story will be generated after liveboard creation.\n\n**What is Spotter Viz?**\nSpotter Viz is an AI agent in ThoughtSpot that creates, structures, and styles Liveboards through natural language prompts. The agent reviews the data, proposes layouts, generates KPIs and visualizations, and allows conversational refinement."
+        
         def update_all_tabs(controller):
             if controller is None:
                 return (
                     "",
                     "-- DDL will appear here after generation",
                     "Progress will appear here during deployment...",
-                    "Demo pack will be generated after deployment completes.\n\nThis will include:\n- Key insights/outliers\n- Spotter questions to ask\n- Talking points for the demo"
+                    "Demo pack will be generated after deployment completes.\n\nThis will include:\n- Key insights/outliers\n- Spotter questions to ask\n- Talking points for the demo",
+                    spotter_viz_default
                 )
             
             # Get live progress from controller (captures deployment output)
@@ -3809,11 +3966,16 @@ def create_chat_interface():
             demo_pack = getattr(controller, 'demo_pack_content', '')
             demo_pack_text = demo_pack if demo_pack else "Demo pack will be generated after deployment completes.\n\nThis will include:\n- Key insights/outliers\n- Spotter questions to ask\n- Talking points for the demo"
             
+            # Get Spotter Viz story from controller
+            spotter_story = getattr(controller, 'spotter_viz_story', '')
+            spotter_story_text = spotter_story if spotter_story else spotter_viz_default
+            
             return (
                 "\n".join(controller.ai_feedback_log),
                 controller.ddl_code if controller.ddl_code else "-- DDL will appear here after generation",
                 live_progress_text,
-                demo_pack_text
+                demo_pack_text,
+                spotter_story_text
             )
         
         # Wire up tab updates on chat interactions
@@ -3821,7 +3983,7 @@ def create_chat_interface():
         chat_components['chatbot'].change(
             fn=update_all_tabs,
             inputs=[chat_controller_state],
-            outputs=[ai_feedback_display, ddl_display, live_progress_display, demo_pack_display]
+            outputs=[ai_feedback_display, ddl_display, live_progress_display, demo_pack_display, spotter_viz_story_display]
         )
         
         # Load settings from Supabase on startup (uses SETTINGS_SCHEMA)
@@ -4133,9 +4295,9 @@ def create_settings_tab():
             
             liveboard_method = gr.Dropdown(
                 label="Liveboard Creation Method",
-                choices=["HYBRID", "SPOTTER_VIZ"],
+                choices=["HYBRID"],
                 value="HYBRID",
-                info="HYBRID: MCP + TML polish (AI-driven). SPOTTER_VIZ: Direct API + TML (faster, no MCP dependency)."
+                info="HYBRID: MCP creates liveboard + TML post-processing for styling, groups, and KPI fixes."
             )
     
     # Existing Model Section

liveboard_creator.py

@@ -2513,7 +2513,7 @@ def create_liveboard_from_model(
     Create and deploy a Liveboard via TML (Spotter Viz path).
     
     This is the direct API approach — builds complete TML and imports it.
-    Used by both the legacy TML path and the new SPOTTER_VIZ method.
+    Used by the direct TML liveboard creation path.
 
     Args:
         ts_client: Authenticated ThoughtSpotDeployer instance

prompts.py

@@ -555,6 +555,33 @@ Create a bullet outline demo script with:
 - The "aha moment" reveal
 - Spotter questions to ask live
 - Closing value proposition""",
+
+    "spotter_viz_story": """You are creating a Spotter Viz story for ThoughtSpot's AI-powered liveboard builder.
+
+Spotter Viz is an AI agent in ThoughtSpot that creates, structures, and styles Liveboards through natural language prompts. Users type conversational requests and the agent builds the dashboard step by step, allowing iterative refinement.
+
+Your job is to write a sequence of natural language prompts that a user would type into Spotter Viz to build a liveboard for this demo scenario. The prompts should be conversational, specific, and progressively build the liveboard from scratch.
+
+{context}
+
+---
+
+Write a Spotter Viz story as a numbered sequence of prompts. Format each step as:
+
+### Step N: [Brief label]
+> "[The exact prompt to type into Spotter Viz]"
+
+**Expected result:** [1 sentence describing what Spotter Viz should create]
+
+Rules:
+- Step 1 should set context: company name, data source, and overall goal
+- Steps 2-3 should add KPIs with sparklines (mention time granularity)
+- Steps 4-6 should add key visualizations (charts, breakdowns, comparisons)
+- Steps 7-8 should refine: add styling, rename the liveboard, organize into groups/tabs
+- Final step should call out the "aha moment" — the key data story or outlier to highlight
+- Use the company name and real column/metric names from the context above
+- Keep prompts natural — write how a business user would actually talk
+- Include 6-10 steps total""",
 }
 
 DEFAULT_TEMPLATE = """You are helping create a {vertical} {function} demo.

sprint_2026_02.md

@@ -88,15 +88,32 @@
 
 ### Done
 
-
-#### Feb 6, 2026 - Spotter Viz Method Added (Design Doc Implementation)
-- [x] **Settings dropdown updated**: `["HYBRID", "SPOTTER_VIZ"]` replaces old 3-option dropdown ✅
-- [x] **SPOTTER_VIZ routing in deployer**: Direct TML path with outliers + model_columns ✅
-- [x] **Backward compatibility**: old "TML"/"MCP" values mapped to "HYBRID" ✅
-- [x] **Shared post-processing**: Both methods use `enhance_mcp_liveboard()` ✅
-- [x] **create_liveboard_from_model() extended**: accepts `outliers` and `model_columns` params ✅
-- [x] **CLAUDE.md updated**: Two-method system documented ✅
-- [ ] **To test**: Run SPOTTER_VIZ end-to-end with Retail Sales
+#### Feb 6, 2026 - Spotter Viz Story Tab (Post-Liveboard Output)
+- [x] **SPOTTER_VIZ method removed** — HYBRID is the only liveboard creation method ✅
+- [x] **Spotter Viz Story generator** — `_generate_spotter_viz_story()` in chat_interface.py ✅
+  - Uses `build_prompt(stage="spotter_viz_story")` + LLM call
+  - Fallback: `_build_fallback_spotter_story()` builds template without LLM
+  - Takes company name, use case, model name, liveboard name
+- [x] **New "🎬 Spotter Viz Story" tab** in Gradio app ✅
+  - Displays after liveboard creation
+  - Conversational sequence of NL prompts for Spotter Viz agent
+  - Can be manually entered into ThoughtSpot Spotter Viz
+- [x] **Wired into pipeline** — called after demo pack generation in deploy flow ✅
+- [x] **CLAUDE.md updated** — documents Spotter Viz Story as post-liveboard output ✅
+- [ ] **To test**: Run end-to-end and verify story generation
+
+#### Feb 6, 2026 - Spotter Viz API Investigation
+- [x] **Tested ThoughtSpot AI endpoints on sebe staging cluster** ✅
+  - `ai/conversation/create` + `ai/conversation/{id}/converse` — WORKS (Spotter NL → search tokens)
+  - `ai/answer/create` — WORKS (single-shot NL → TS search tokens)
+  - `ai/agent/conversation/create` — schema unknown (ContextPayloadV2Input enum not discoverable)
+  - Documented in `dev_notes/SPOTTER_VIZ_API_TEST_RESULTS.md`
+- [x] **Decision**: Spotter Viz API not ready for liveboard creation — use story output instead ✅
+
+#### Feb 6, 2026 - Spotter Viz Method Added (REVERTED)
+- ~~SPOTTER_VIZ routing in deployer~~ — Removed (was just TML pipeline renamed)
+- ~~Settings dropdown `["HYBRID", "SPOTTER_VIZ"]`~~ — Simplified to `["HYBRID"]`
+- [x] **Backward compatibility preserved**: old "TML"/"MCP" values still map to "HYBRID" ✅
 
 
 #### Feb 6, 2026 - Gradio Compat + Hybrid Liveboard Layout Fix

thoughtspot_deployer.py

@@ -2123,14 +2123,10 @@ class ThoughtSpotDeployer:
                             use_mcp = os.getenv('USE_MCP_LIVEBOARD', 'false').lower() == 'true'
                             method = 'MCP' if use_mcp else 'HYBRID'
                         
-                        # Normalize method name and backward compatibility
-                        method = method.upper()
-                        # Map old values to current options
-                        if method in ('TML', 'MCP'):
-                            log_progress(f"[INFO] Mapping legacy method '{method}' → HYBRID")
-                            method = 'HYBRID'
-                        if method not in ['HYBRID', 'SPOTTER_VIZ']:
-                            log_progress(f"[WARN] Unknown liveboard method '{method}', defaulting to HYBRID")
+                        # Normalize method — HYBRID is the only supported method
+                        method = method.upper() if method else 'HYBRID'
+                        if method != 'HYBRID':
+                            log_progress(f"[INFO] Mapping method '{method}' → HYBRID (only supported method)")
                             method = 'HYBRID'
                         
                         log_progress(f"Creating liveboard ({method} method)...")
@@ -2144,95 +2140,9 @@ class ThoughtSpotDeployer:
                                 'use_case': use_case or 'General Analytics'
                             }
                             
-                            if method == 'SPOTTER_VIZ':
-                                # Spotter Viz: Direct REST API + TML — no MCP dependency
-                                from liveboard_creator import create_liveboard_from_model, enhance_mcp_liveboard
-                                
-                                # Get actual column names from ThoughtSpot model
-                                model_columns = self.get_model_columns(model_guid)
-                                if not model_columns:
-                                    log_progress(f"   ⚠️ Could not get model columns, falling back to DDL")
-                                    model_columns = []
-                                    for table_name, columns_list in tables.items():
-                                        for col in columns_list:
-                                            model_columns.append(col)
-                                
-                                # Get outlier patterns from the vertical×function system
-                                outlier_dicts = []
-                                try:
-                                    from outlier_system import get_outliers_for_use_case
-                                    from demo_personas import parse_use_case
-                                    uc_vertical, uc_function = parse_use_case(use_case or '')
-                                    if uc_vertical or uc_function:
-                                        outlier_config = get_outliers_for_use_case(
-                                            uc_vertical or "Generic", 
-                                            uc_function or "Generic"
-                                        )
-                                        for op in outlier_config.required:
-                                            outlier_dicts.append({
-                                                'title': op.name,
-                                                'insight': op.viz_talking_point,
-                                                'viz_type': op.viz_type,
-                                                'show_me_query': op.viz_question,
-                                                'kpi_companion': True,
-                                                'spotter_questions': op.spotter_questions,
-                                            })
-                                        for op in outlier_config.optional[:2]:
-                                            outlier_dicts.append({
-                                                'title': op.name,
-                                                'insight': op.viz_talking_point,
-                                                'viz_type': op.viz_type,
-                                                'show_me_query': op.viz_question,
-                                                'kpi_companion': False,
-                                                'spotter_questions': op.spotter_questions,
-                                            })
-                                        if outlier_dicts:
-                                            log_progress(f"   [SPOTTER] Using {len(outlier_dicts)} outlier patterns from {uc_vertical}×{uc_function}")
-                                except Exception as outlier_err:
-                                    log_progress(f"   [SPOTTER] Outlier loading skipped: {outlier_err}")
-                                
-                                log_progress(f"   [SPOTTER] Model: {model_name}, GUID: {model_guid}")
-                                log_progress(f"   [SPOTTER] Using {len(model_columns)} columns from ThoughtSpot model")
-                                log_progress(f"   Step 1/2: Building liveboard via TML...")
-                                
-                                try:
-                                    liveboard_result = create_liveboard_from_model(
-                                        ts_client=self,
-                                        model_id=model_guid,
-                                        model_name=model_name,
-                                        company_data=company_data,
-                                        use_case=use_case or 'General Analytics',
-                                        num_visualizations=8,
-                                        liveboard_name=liveboard_name,
-                                        llm_model=llm_model,
-                                        outliers=outlier_dicts if outlier_dicts else None,
-                                        model_columns=model_columns
-                                    )
-                                except Exception as spotter_error:
-                                    import traceback
-                                    error_trace = traceback.format_exc()
-                                    log_progress(f"   [SPOTTER ERROR] {type(spotter_error).__name__}: {str(spotter_error)}")
-                                    liveboard_result = {'success': False, 'error': str(spotter_error), 'traceback': error_trace}
-                                
-                                # Spotter Viz: Add TML enhancement (same as Hybrid post-processing)
-                                if liveboard_result.get('success') and liveboard_result.get('liveboard_guid'):
-                                    log_progress(f"   Step 2/2: Enhancing with TML post-processing...")
-                                    enhance_result = enhance_mcp_liveboard(
-                                        liveboard_guid=liveboard_result['liveboard_guid'],
-                                        company_data=company_data,
-                                        ts_client=self,
-                                        add_groups=True,
-                                        fix_kpis=True,
-                                        apply_brand_colors=True
-                                    )
-                                    if enhance_result.get('success'):
-                                        log_progress(f"   [OK] Enhancement applied: {', '.join(enhance_result.get('enhancements', []))}")
-                                    else:
-                                        log_progress(f"   [WARN] Enhancement partial: {enhance_result.get('message', '')[:80]}")
-                                
-                            elif method == 'HYBRID':
-                                # MCP and HYBRID both use MCP for creation
-                                # HYBRID adds TML post-processing enhancement
+                            if method == 'HYBRID':
+                                # HYBRID: MCP creates liveboard, TML post-processes
+
                                 from liveboard_creator import create_liveboard_from_model_mcp, enhance_mcp_liveboard
                                 
                                 # Get actual column names from ThoughtSpot model (not DDL)