feat: March 2026 sprint — new vision merge, pipeline improvements, settings refactor

Core changes:
- Merged DemoPrep_new_vision2: StorySpec outlier injection engine, domain packs,
quality validator, 1600-line generator with deterministic seeds + trend signals
- Consolidated outlier_system.py + 3 data_adjuster files → demo_personas.py liveboard_questions
- Rewrote liveboard AI fill prompt: no redundant KPIs, business insight focus
- Fixed "Show me" title strip, Spotter Viz Story now persona-driven (Part 1) + AI (Part 2)
- Fixed progress meter stuck on Data during ThoughtSpot deployment
- Fixed liveboard name: UI field value now always takes priority over DB default
- Settings UI: split into Default Settings (AI Model, Use Case, LB Name) + App Settings
- Added session_logger.py, admin log viewer, state isolation (per-session loggers)
- TS environment dropdown, front page redesign, progress meter Init→Complete
- Supabase session logging with file fallback
- Sharing after every build (model + liveboard)
- Sage indexing retry on 10004, fallback TML skips invalid column refs
- Fixed domain-specific NAME column generation (DRUG_NAME was using fake.name())
- Batch test runner + test cases copied from new_vision2 branch

New files: session_logger.py, sprint_2026_03.md, legitdata storyspec/domain/quality packages,
tests/newvision_*, tests/test_mcp_liveboard.py
Removed: outlier_system.py, chat_data_adjuster.py, conversational_data_adjuster.py,
data_adjuster.py, sprint_2026_01.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.gitignore

@@ -111,6 +111,7 @@ env.bak/
 venv.bak/
 demo_wire/  # Your virtual environment
 demo_wire/**
+demoprep/   # Local virtual environment
 
 # Spyder project settings
 .spyderproject

CLAUDE.md

@@ -33,7 +33,7 @@ See `dev_notes/USE_CASE_FLOW.md` for the full use case framework documentation.
 
 ## Current Sprint
 
-**ALWAYS READ FIRST**: `sprint_2026_02.md` (in root)
+**ALWAYS READ FIRST**: `sprint_2026_03.md` (in root)
 
 For overall project status, see `PROJECT_STATUS.md` in root.
 
@@ -58,7 +58,7 @@ Example:
 - [x] snake_case naming ✅ - `_to_snake_case()` in deployer
 ```
 
-**Previous sprints:** `sprint_2026_01.md` (January - closed)
+**Previous sprints:** `sprint_2026_02.md` (February/March - closed), `sprint_2026_01.md` (January - closed)
 
 ---
 
@@ -112,6 +112,7 @@ Example:
 2. **No silent fallbacks** — if something is missing, error immediately with a clear message
    - ❌ WRONG: `value = os.getenv('KEY', 'some_default')`
    - ✅ RIGHT: `value = get_admin_setting('KEY')` → raises error if empty
+   - **User identity is required**: Never use fallback users like `default@user.com` or `USER_EMAIL` env defaults for settings. If authenticated username/email is missing, fail immediately with a clear error.
 
 3. **Single source of truth** — every piece of data should have one canonical source
    - Admin settings live in Supabase, read via `get_admin_setting()`
@@ -135,6 +136,7 @@ Example:
 2. **No silent fallbacks** - if something is missing, error immediately with a clear message
    - WRONG: `value = os.getenv('KEY', 'some_default')`
    - RIGHT: `value = get_admin_setting('KEY')` raises error if empty
+   - User identity is required: Never use fallback users like `default@user.com` or `USER_EMAIL` env defaults for settings. If authenticated username/email is missing, fail immediately with a clear error.
 
 3. **Single source of truth** - every piece of data should have one canonical source
    - Admin settings live in Supabase, read via `get_admin_setting()`

cdw_connector.py

@@ -3,9 +3,9 @@ CDW Connector - Sprint 2A
 Snowflake deployment functionality for generated schemas and data
 """
 
-import os
 from datetime import datetime
 from typing import Dict, List, Tuple, Optional
+import re
 
 try:
     import snowflake.connector
@@ -19,18 +19,17 @@ try:
 except ImportError:
     SQLPARSE_AVAILABLE = False
 
-try:
-    from dotenv import load_dotenv
-    load_dotenv(override=True)
-except ImportError:
-    pass
-
 try:
     from snowflake_auth import get_snowflake_connection_params
     SNOWFLAKE_AUTH_AVAILABLE = True
 except ImportError:
     SNOWFLAKE_AUTH_AVAILABLE = False
 
+try:
+    from supabase_client import inject_admin_settings_to_env
+except ImportError:
+    inject_admin_settings_to_env = None
+
 class SnowflakeDeployer:
     """Handle Snowflake deployment for generated schemas and data"""
     
@@ -42,14 +41,11 @@ class SnowflakeDeployer:
     def connect(self) -> Tuple[bool, str]:
         """Connect to Snowflake using OKTA authentication"""
         try:
-            # Validate required environment variables
-            required_vars = ['SNOWFLAKE_ACCOUNT', 'SNOWFLAKE_KP_USER', 'SNOWFLAKE_KP_PK', 'SNOWFLAKE_WAREHOUSE', 'SNOWFLAKE_DATABASE']
-            missing_vars = [var for var in required_vars if not os.getenv(var)]
-            
-            if missing_vars:
-                return False, f"Missing required environment variables: {missing_vars}"
-            
-            # Key pair authentication - let's fix the implementation
+            # Keep legacy env-based consumers aligned with Supabase admin settings.
+            if inject_admin_settings_to_env:
+                inject_admin_settings_to_env()
+
+            # Key pair authentication via centralized auth helper
             if not SNOWFLAKE_AUTH_AVAILABLE:
                 return False, "snowflake_auth module not available"
             
@@ -117,8 +113,10 @@ class SnowflakeDeployer:
             
             # Parse and execute DDL statements
             statements = self._parse_ddl_statements(ddl_statements, schema_name)
+            expected_tables = self._extract_expected_table_names(statements)
             
             executed_count = 0
+            failed_statements = []
             for statement in statements:
                 if statement.strip():
                     try:
@@ -126,8 +124,32 @@ class SnowflakeDeployer:
                         cursor.execute(statement)
                         executed_count += 1
                     except Exception as e:
-                        print(f"Warning: Failed to execute statement: {str(e)}")
-                        continue
+                        failed_statements.append((statement, str(e)))
+
+            # Retry failed statements once to handle dependency-order issues.
+            if failed_statements:
+                print(f"⚠️ First DDL pass had {len(failed_statements)} failures. Retrying once...")
+                retry_failures = []
+                for statement, first_error in failed_statements:
+                    try:
+                        cursor.execute(statement)
+                        executed_count += 1
+                    except Exception as e:
+                        retry_failures.append((statement, first_error, str(e)))
+                failed_statements = retry_failures
+
+            if failed_statements:
+                preview_errors = []
+                for statement, first_error, second_error in failed_statements[:3]:
+                    stmt_preview = statement.replace("\n", " ")[:120]
+                    preview_errors.append(
+                        f"- {stmt_preview} | first_error={first_error} | retry_error={second_error}"
+                    )
+                error_summary = "\n".join(preview_errors)
+                raise Exception(
+                    f"DDL deployment failed after retry; {len(failed_statements)} statements still failing.\n"
+                    f"{error_summary}"
+                )
             
             # Check autocommit setting 
             print(f"🔍 DEBUG: Connection autocommit: {self.connection.autocommit}")
@@ -144,6 +166,17 @@ class SnowflakeDeployer:
             print(f"🔍 DEBUG: Current schema: {current_schema}")
             print(f"🔍 DEBUG: Schema verification completed - using current schema context")
 
+            # Verify expected tables actually exist before declaring success.
+            if expected_tables:
+                cursor.execute(f'SHOW TABLES IN SCHEMA "{schema_name}"')
+                existing_tables = {row[1].upper() for row in cursor.fetchall()}
+                missing_tables = sorted(tbl for tbl in expected_tables if tbl not in existing_tables)
+                if missing_tables:
+                    raise Exception(
+                        "DDL deployment completed but required tables are missing in schema "
+                        f"{schema_name}: {missing_tables}"
+                    )
+
             cursor.close()
 
             success_message = f"Schema '{schema_name}' created successfully with {executed_count} tables"
@@ -168,7 +201,12 @@ class SnowflakeDeployer:
             clean_text = ddl_text.replace("```sql", "").replace("```", "").strip()
             
             # Split by semicolon
-            statements = [stmt.strip() for stmt in clean_text.split(';') if stmt.strip()]
+            statements = []
+            for stmt in clean_text.split(';'):
+                stmt = stmt.strip()
+                if not stmt:
+                    continue
+                statements.append(self._sanitize_create_table_statement(stmt))
             processed_statements = []
             
             # Separate tables with and without foreign keys
@@ -190,6 +228,101 @@ class SnowflakeDeployer:
         except Exception as e:
             print(f"Error parsing DDL statements: {str(e)}")
             return []
+
+    def _sanitize_create_table_statement(self, statement: str) -> str:
+        """
+        Remove CHECK constraints from CREATE TABLE statements.
+        This deploy path rejects CHECK constraints, so we sanitize deterministically.
+        """
+        if "CREATE TABLE" not in statement.upper():
+            return statement
+
+        open_idx = statement.find("(")
+        if open_idx == -1:
+            return statement
+
+        close_idx = self._find_matching_paren(statement, open_idx)
+        if close_idx == -1:
+            return statement
+
+        inner = statement[open_idx + 1:close_idx]
+        chunks = self._split_top_level_csv(inner)
+
+        sanitized_chunks = []
+        removed_any = False
+        for chunk in chunks:
+            stripped = chunk.strip()
+            if not stripped:
+                continue
+
+            upper = stripped.upper()
+            is_table_check = upper.startswith("CHECK") or ("CHECK" in upper and upper.startswith("CONSTRAINT"))
+            if is_table_check:
+                removed_any = True
+                continue
+
+            cleaned = self._remove_inline_check_clause(stripped)
+            if cleaned != stripped:
+                removed_any = True
+            if cleaned.strip():
+                sanitized_chunks.append(cleaned.strip())
+
+        if not removed_any:
+            return statement
+
+        rebuilt_inner = ",\n  ".join(sanitized_chunks)
+        rebuilt = f"{statement[:open_idx + 1]}\n  {rebuilt_inner}\n{statement[close_idx:]}"
+        print("⚠️ Sanitized unsupported CHECK constraint(s) from CREATE TABLE statement.")
+        return rebuilt
+
+    def _find_matching_paren(self, text: str, open_idx: int) -> int:
+        """Find matching closing parenthesis index for text[open_idx] == '('."""
+        depth = 0
+        for i in range(open_idx, len(text)):
+            ch = text[i]
+            if ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth -= 1
+                if depth == 0:
+                    return i
+        return -1
+
+    def _split_top_level_csv(self, text: str) -> List[str]:
+        """Split by commas that are not nested in parentheses."""
+        parts = []
+        start = 0
+        depth = 0
+        for i, ch in enumerate(text):
+            if ch == "(":
+                depth += 1
+            elif ch == ")":
+                depth = max(0, depth - 1)
+            elif ch == "," and depth == 0:
+                parts.append(text[start:i])
+                start = i + 1
+        parts.append(text[start:])
+        return parts
+
+    def _remove_inline_check_clause(self, chunk: str) -> str:
+        """Remove inline `CHECK (...)` clause from a column definition chunk."""
+        m = re.search(r"\bCHECK\b", chunk, flags=re.IGNORECASE)
+        if not m:
+            return chunk
+
+        start = m.start()
+        i = m.end()
+        while i < len(chunk) and chunk[i].isspace():
+            i += 1
+        if i >= len(chunk) or chunk[i] != "(":
+            # Bare CHECK token; remove token only as fallback.
+            return (chunk[:start] + chunk[m.end():]).strip()
+
+        end = self._find_matching_paren(chunk, i)
+        if end == -1:
+            return chunk
+
+        return (chunk[:start] + " " + chunk[end + 1:]).strip()
     
     def _qualify_table_name(self, statement: str, schema_name: str) -> str:
         """Ensure table names are properly schema-qualified"""
@@ -223,6 +356,32 @@ class SnowflakeDeployer:
         except Exception as e:
             print(f"Error qualifying table name: {str(e)}")
             return statement  # Return original if processing fails
+
+    def _extract_expected_table_names(self, statements: List[str]) -> List[str]:
+        """Extract CREATE TABLE targets for post-deploy existence verification."""
+        expected = []
+        table_pattern = re.compile(
+            r'CREATE\s+(?:OR\s+REPLACE\s+)?TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?'
+            r'(?:(?:"[^"]+"|[A-Za-z0-9_]+)\.)?(?:"([^"]+)"|([A-Za-z0-9_]+))',
+            re.IGNORECASE
+        )
+
+        for statement in statements:
+            match = table_pattern.search(statement)
+            if not match:
+                continue
+            table_name = (match.group(1) or match.group(2) or "").strip()
+            if table_name:
+                expected.append(table_name.upper())
+
+        # Keep order but remove duplicates.
+        seen = set()
+        ordered_unique = []
+        for name in expected:
+            if name not in seen:
+                seen.add(name)
+                ordered_unique.append(name)
+        return ordered_unique
     
     def list_demo_schemas(self) -> List[Dict]:
         """List all demo schemas in the database"""

chat_data_adjuster.py

@@ -1,163 +0,0 @@
-"""
-Simple Chat Interface for Data Adjustment
-
-A basic command-line chat to test the conversational data adjuster.
-User can keep making adjustments until they type 'done' or 'exit'.
-"""
-
-from dotenv import load_dotenv
-import os
-from conversational_data_adjuster import ConversationalDataAdjuster
-
-load_dotenv()
-
-
-def chat_loop():
-    """Main chat loop for data adjustment"""
-    
-    print("""
-╔════════════════════════════════════════════════════════════╗
-║                                                            ║
-║          Data Adjustment Chat                              ║
-║                                                            ║
-╚════════════════════════════════════════════════════════════╝
-
-Commands:
-  - Type your adjustment request naturally
-  - "done" or "exit" to quit
-  - "help" for examples
-
-Examples:
-  - "increase 1080p Webcam sales to 50B"
-  - "set profit margin to 25% for electronics"  
-  - "make Tablet revenue 100 billion"
-
-""")
-    
-    # Initialize adjuster
-    database = os.getenv('SNOWFLAKE_DATABASE')
-    schema = "20251116_140933_AMAZO_SAL"
-    model_id = "3c97b0d6-448b-440a-b628-bac1f3d73049"
-    
-    print(f"📊 Connected to: {database}.{schema}")
-    print(f"🎯 Model: {model_id}\n")
-    
-    adjuster = ConversationalDataAdjuster(database, schema, model_id)
-    adjuster.connect()
-    
-    tables = adjuster.get_available_tables()
-    print(f"📋 Available tables: {', '.join(tables)}\n")
-    print("="*80)
-    print("Ready! Type your adjustment request...")
-    print("="*80 + "\n")
-    
-    while True:
-        # Get user input
-        user_input = input("\n💬 You: ").strip()
-        
-        if not user_input:
-            continue
-        
-        # Check for exit commands
-        if user_input.lower() in ['done', 'exit', 'quit', 'bye']:
-            print("\n👋 Goodbye!")
-            break
-        
-        # Check for help
-        if user_input.lower() == 'help':
-            print("""
-📚 Help - How to make adjustments:
-
-Format: "make/increase/set [entity] [metric] to [value]"
-
-Examples:
-  ✅ "increase 1080p Webcam revenue to 50 billion"
-  ✅ "set profit margin to 25% for electronics"
-  ✅ "make Laptop sales 100B"
-  ✅ "increase customer segment premium revenue by 30%"
-
-You'll see 3 strategy options - type A, B, or C to pick one.
-""")
-            continue
-        
-        try:
-            # Step 1: Parse the request
-            print(f"\n🤔 Parsing your request...")
-            adjustment = adjuster.parse_adjustment_request(user_input, tables)
-            
-            if 'error' in adjustment:
-                print(f"❌ {adjustment['error']}")
-                print("💡 Try rephrasing or type 'help' for examples")
-                continue
-            
-            # Step 2: Analyze current data
-            print(f"📊 Analyzing current data...")
-            analysis = adjuster.analyze_current_data(adjustment)
-            
-            if analysis['current_total'] == 0:
-                print(f"⚠️  No data found for '{adjustment['entity_value']}'")
-                print("💡 Try a different product/entity name")
-                continue
-            
-            # Step 3: Generate strategies
-            strategies = adjuster.generate_strategy_options(adjustment, analysis)
-            
-            # Step 4: Present options
-            adjuster.present_options(adjustment, analysis, strategies)
-            
-            # Step 5: Get user's strategy choice
-            print("\n" + "="*80)
-            choice = input("Which strategy? [A/B/C or 'skip']: ").strip().upper()
-            
-            if choice == 'SKIP' or not choice:
-                print("⏭️  Skipping this adjustment")
-                continue
-            
-            # Find the chosen strategy
-            chosen = None
-            for s in strategies:
-                if s['id'] == choice:
-                    chosen = s
-                    break
-            
-            if not chosen:
-                print(f"❌ Invalid choice: {choice}")
-                continue
-            
-            # Step 6: Confirm
-            print(f"\n⚠️  About to execute: {chosen['name']}")
-            print(f"   This will affect {chosen.get('details', {}).get('rows_affected', 'some')} rows")
-            confirm = input("   Confirm? [yes/no]: ").strip().lower()
-            
-            if confirm not in ['yes', 'y']:
-                print("❌ Cancelled")
-                continue
-            
-            # Step 7: Execute
-            result = adjuster.execute_strategy(chosen)
-            
-            if result['success']:
-                print(f"\n✅ {result['message']}")
-                print(f"🔄 Data updated! Refresh your ThoughtSpot liveboard to see changes.")
-            else:
-                print(f"\n❌ Failed: {result.get('error')}")
-        
-        except KeyboardInterrupt:
-            print("\n\n⚠️  Interrupted")
-            break
-        except Exception as e:
-            print(f"\n❌ Error: {e}")
-            import traceback
-            print(traceback.format_exc())
-    
-    # Cleanup
-    adjuster.close()
-    print("\n✅ Connection closed")
-
-
-if __name__ == "__main__":
-    try:
-        chat_loop()
-    except KeyboardInterrupt:
-        print("\n\n👋 Goodbye!")
-

chat_interface.py

@@ -13,6 +13,7 @@ import sys
 import json
 import time
 import glob
+from datetime import datetime
 from dotenv import load_dotenv
 from demo_builder_class import DemoBuilder
 from supabase_client import load_gradio_settings, get_admin_setting, inject_admin_settings_to_env
@@ -35,6 +36,53 @@ from llm_config import (
 load_dotenv(override=True)
 
 
+# ==========================================================================
+# TS ENVIRONMENT HELPERS
+# ==========================================================================
+
+def get_ts_environments() -> list:
+    """Return list of environment labels from TS_ENV_N_LABEL/URL/.env entries."""
+    envs = []
+    i = 1
+    while True:
+        label = os.getenv(f'TS_ENV_{i}_LABEL', '').strip()
+        url = os.getenv(f'TS_ENV_{i}_URL', '').strip()
+        if not label or not url:
+            break
+        envs.append(label)
+        i += 1
+    return envs or ['(no environments configured)']
+
+
+def get_ts_env_url(label: str) -> str:
+    """Return the URL for a given environment label."""
+    i = 1
+    while True:
+        env_label = os.getenv(f'TS_ENV_{i}_LABEL', '').strip()
+        if not env_label:
+            break
+        if env_label == label:
+            return os.getenv(f'TS_ENV_{i}_URL', '').strip().rstrip('/')
+        i += 1
+    return ''
+
+
+def get_ts_env_auth_key(label: str) -> str:
+    """Return the actual auth key value for a given environment label.
+    TS_ENV_N_KEY_VAR holds the name of the env var containing the token.
+    """
+    i = 1
+    while True:
+        env_label = os.getenv(f'TS_ENV_{i}_LABEL', '').strip()
+        if not env_label:
+            break
+        if env_label == label:
+            key_var = os.getenv(f'TS_ENV_{i}_KEY_VAR', '').strip()
+            return os.getenv(key_var, '').strip() if key_var else ''
+        i += 1
+    return ''
+
+
 # ==========================================================================
 # SETTINGS SCHEMA - Single source of truth for all settings
 # To add a new setting: add ONE entry here, then create the UI component
@@ -43,7 +91,7 @@ load_dotenv(override=True)
 SETTINGS_SCHEMA = [
     # App Settings - Left Column
     ('default_ai_model', 'default_llm', DEFAULT_LLM_MODEL, str),
-    ('default_company_url', 'default_company_url', '', str),
+    # default_company_url removed — company is chat-driven now
     ('default_use_case', 'default_use_case', 'Sales Analytics', str),
     ('liveboard_name', 'liveboard_name', '', str),
     ('tag_name', 'tag_name', '', str),
@@ -68,8 +116,10 @@ SETTINGS_SCHEMA = [
     ('default_warehouse', 'default_warehouse', 'COMPUTE_WH', str),
     ('default_database', 'default_database', 'DEMO_DB', str),
     ('default_schema', 'default_schema', 'PUBLIC', str),
-    ('ts_instance_url', 'thoughtspot_url', '', str),
+    # ts_instance_url removed — replaced by TS Environment dropdown on front page
     ('ts_username', 'thoughtspot_username', '', str),
+    ('data_adjuster_url', 'data_adjuster_url', '', str),
+    ('share_with', 'share_with', '', str),
     # Status (special - not saved, just displayed)
     ('settings_status', None, '', str),
 ]
@@ -144,10 +194,9 @@ def require_authenticated_email(request: gr.Request = None, user_email: str = No
 
 def build_initial_chat_message(company: str, use_case: str) -> str:
     """Build the pre-filled chat message from current settings."""
-    return (
-        "I am creating a demo for the company: "
-        f"{company} and for the use case: {use_case}"
-    )
+    if company and use_case:
+        return f"{company}, {use_case}"
+    return ""
 
 
 def safe_print(*args, **kwargs):
@@ -185,6 +234,9 @@ class ChatDemoInterface:
         self.live_progress_log = []  # Real-time deployment progress
         self.demo_pack_content = ""   # Generated demo pack markdown
         self.spotter_viz_story = ""   # Spotter Viz story (NL prompts for Spotter Viz agent)
+        # Per-session loggers (NOT module-level singletons — avoids cross-session contamination)
+        self._session_logger = None
+        self._prompt_logger = None
 
     def _get_effective_user_email(self) -> str:
         """Resolve and cache effective user identity for settings access."""
@@ -252,37 +304,23 @@ class ChatDemoInterface:
             
             return f"""👋 **Welcome to ThoughtSpot Demo Builder!**
 
-I need two things to get started:
+Tell me who you're demoing to and what they care about — I'll build the whole thing.
 
-**1. Company URL** (must be a real website)
-   - Examples: Nike.com, Target.com, Walmart.com
-   - I'll research their actual business model and data
+**Just say something like:**
+- *"Nike.com, Retail Sales"*
+- *"Salesforce.com, Software Sales"*
+- *"Target.com — supply chain analytics for their VP of Operations"*
+- *"Pfizer.com — analyzing clinical trial pipeline for a CMO persona"*
 
-**2. Use Case** (Vertical × Function, or any custom use case!)
-   
-   **Configured combinations** (with KPIs, outliers, and Spotter questions):
+**Pre-configured use cases** (KPIs, outliers, and Spotter questions ready to go):
 {uc_list}
    - ...and more!
-   
-   **Or create your own custom use case:**
-   - "analyzing customer churn patterns across regions"
-   - "tracking manufacturing defects by product line"
-   - Literally ANYTHING — AI will research and build it!
-
-**How to tell me (BOTH required):**
-
-```
-I'm creating a demo for company: Nike.com use case: Retail Sales
-```
 
-Or with custom use case:
-```
-I'm creating a demo for company: Target.com use case: analyzing seasonal inventory trends
-```
+**Or describe any custom use case** — AI will research and build it from scratch.
 
-⚠️ **Important:** You MUST provide BOTH company AND use case in one message!
+> 💡 **Tip:** Select your ThoughtSpot environment and AI model in the panel to the right before starting.
 
-*Ready? Tell me your company and use case!*"""
+*What company and use case are you working on?*"""
         else:
             return f"""👋 **Welcome to ThoughtSpot Demo Builder!**
 
@@ -335,10 +373,27 @@ What's your first step?"""
         Process user message and return updated chat history and state (with streaming)
         Returns: (chat_history, current_stage, current_model, company, use_case, next_textbox_value)
         """
-        
+        from session_logger import init_session_logger
+        from prompt_logger import reset_prompt_logger
+
+        # Init per-session loggers on first message (stored on self, not module singletons)
+        if self._session_logger is None:
+            session_id = datetime.now().strftime("%Y%m%d_%H%M%S")
+            self._session_logger = init_session_logger(session_id, user_email=getattr(self, 'user_email', None))
+            self._prompt_logger = reset_prompt_logger(session_id)
+        _slog = self._session_logger
+        _slog.log(current_stage or 'init', f"user message received: {message[:120]}")
+
         # Add user message to history
         chat_history.append((message, None))
-        
+
+        # If data_adjuster_url is saved in settings and we're at init, inject it as the message
+        # so the user lands directly in Data Adjuster without having to paste the URL manually
+        da_url = self.settings.get('data_adjuster_url', '').strip()
+        if da_url and current_stage == 'initialization' and 'pinboard/' in da_url:
+            message = da_url
+            chat_history[-1] = (da_url, None)
+
         # Validate required settings before proceeding
         missing = self.validate_required_settings()
         if missing and current_stage == 'initialization':
@@ -369,6 +424,70 @@ What's your first step?"""
         
         # Stage-based processing
         if current_stage == 'initialization':
+            # Check if user pasted a ThoughtSpot liveboard URL → jump straight to data adjuster
+            lb_guid_match = re.search(
+                r'pinboard/([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})',
+                message, re.I
+            )
+            if lb_guid_match:
+                liveboard_guid = lb_guid_match.group(1)
+                chat_history[-1] = (message, "🔍 **Resolving liveboard context...**")
+                yield chat_history, current_stage, current_model, company, use_case, ""
+
+                try:
+                    from smart_data_adjuster import SmartDataAdjuster, load_context_from_liveboard
+                    from thoughtspot_deployer import ThoughtSpotDeployer
+                    from supabase_client import get_admin_setting
+
+                    ts_url = (self.settings.get('thoughtspot_url') or '').strip() or get_admin_setting('THOUGHTSPOT_URL')
+                    ts_user = get_admin_setting('THOUGHTSPOT_ADMIN_USER')
+                    ts_secret = (self.settings.get('thoughtspot_trusted_auth_key') or '').strip() or get_admin_setting('THOUGHTSPOT_TRUSTED_AUTH_KEY')
+
+                    ts_client = ThoughtSpotDeployer(base_url=ts_url, username=ts_user, secret_key=ts_secret)
+                    ts_client.authenticate()
+
+                    ctx = load_context_from_liveboard(liveboard_guid, ts_client)
+
+                    llm_model = self.settings.get('model', DEFAULT_LLM_MODEL)
+                    adjuster = SmartDataAdjuster(
+                        database=ctx['database'],
+                        schema=ctx['schema'],
+                        liveboard_guid=liveboard_guid,
+                        llm_model=llm_model,
+                        ts_url=ts_url,
+                        ts_secret=ts_secret,
+                    )
+                    adjuster.connect()
+
+                    if not adjuster.load_liveboard_context():
+                        raise ValueError("Liveboard has no answer-based visualizations to adjust.")
+
+                    self._adjuster = adjuster
+                    current_stage = 'outlier_adjustment'
+
+                    viz_list = "\n".join(
+                        f"  [{i+1}] {v['name']}"
+                        for i, v in enumerate(adjuster.visualizations)
+                    )
+                    response = f"""✅ **Liveboard context loaded — ready for data adjustments**
+
+**Liveboard:** {ctx['liveboard_name']}
+**Model:** {ctx['model_name']}
+**Snowflake:** `{ctx['database']}`.`{ctx['schema']}`
+
+**Visualizations:**
+{viz_list}
+
+Tell me what to change — e.g. *"increase webcam revenue by 20%"* or *"make Acme Corp 50B"*.
+Type **done** when finished."""
+
+                except Exception as e:
+                    response = f"❌ **Could not load liveboard context**\n\n`{e}`"
+
+                chat_history[-1] = (message, response)
+                yield chat_history, current_stage, current_model, company, use_case, ""
+                return
+
             # Check if user just provided a standalone URL (e.g., "Comscore.com")
             standalone_url = re.search(r'^([a-zA-Z0-9-]+\.[a-zA-Z]{2,})$', cleaned_message.strip())
             if standalone_url:
@@ -540,7 +659,71 @@ This may take 1-2 minutes. Watch the AI Feedback tab for progress!"""
                     yield chat_history, current_stage, current_model, company, use_case, ""
                 
                 return
-        
+
+            # --- Catch-all: try to extract company + use case from any free-form message ---
+            # Handles: "Nike.com, Retail Sales" / "Salesforce - Software Sales" / etc.
+            extracted_company = self.extract_company_from_message(cleaned_message)
+            extracted_use_case = self.extract_use_case_from_message(cleaned_message)
+
+            if extracted_company and extracted_use_case:
+                company = extracted_company
+                self.vertical, self.function = parse_use_case(extracted_use_case)
+                self.use_case_config = get_use_case_config(
+                    self.vertical or "Generic", self.function or "Generic"
+                )
+                is_known = self.vertical and self.function and not self.use_case_config.get('is_generic')
+                use_case_display = self.use_case_config.get('use_case_name', extracted_use_case)
+                self.is_generic_use_case = not is_known
+                self.pending_generic_company = company
+                self.pending_generic_use_case = use_case_display
+
+                if is_known:
+                    note = f"\n\n*Matched: **{self.vertical}** × **{self.function}** — KPIs, outliers, and Spotter questions ready.*"
+                elif self.vertical or self.function:
+                    note = f"\n\n*Partial match: **{self.vertical or self.function}** recognized — AI will fill in the gaps.*"
+                else:
+                    note = "\n\n*Custom use case — AI will research and build from scratch.*"
+
+                context_prompt = f"""✅ **Demo Configuration**
+
+**Company:** {company}
+**Use Case:** {use_case_display}{note}
+
+**Want to add any requirements?** (or just say "proceed")
+- "Include a RETURNS table"
+- "Focus on enterprise accounts only"
+- "I need 2 fact tables: Sales and Inventory"
+
+*Type your requirements, or say **"proceed"** to use defaults.*"""
+
+                chat_history[-1] = (message, context_prompt)
+                current_stage = 'awaiting_context'
+                yield chat_history, current_stage, current_model, company, use_case_display, "proceed"
+                return
+
+            elif extracted_company and not extracted_use_case:
+                uc_opts = "\n".join([f"- {v} {f}" for v in list(VERTICALS.keys())[:3] for f in FUNCTIONS.keys()])
+                response = f"""Got it — **{extracted_company}**!
+
+What use case are we building? A few options:
+{uc_opts}
+- Or describe any custom scenario!"""
+                chat_history[-1] = (message, response)
+                yield chat_history, current_stage, current_model, company, use_case, ""
+                return
+
+            else:
+                # Nothing useful extracted — show a brief prompt
+                response = """I need a **company** and **use case** to get started.
+
+Try something like:
+- *"Nike.com, Retail Sales"*
+- *"Salesforce.com — Software pipeline analytics"*
+- *"Walmart.com, Supply Chain"*"""
+                chat_history[-1] = (message, response)
+                yield chat_history, current_stage, current_model, company, use_case, ""
+                return
+
         elif current_stage == 'awaiting_context':
             # User is providing context for ANY use case (both generic and established)
             if message_lower.strip() in ['proceed', 'continue', 'no', 'skip']:
@@ -618,6 +801,10 @@ Watch the AI Feedback tab for real-time progress!"""
                     for progress_update in self.run_deployment_streaming():
                         if isinstance(progress_update, tuple):
                             final_result = progress_update
+                        elif isinstance(progress_update, dict):
+                            current_stage = progress_update.get('stage', current_stage)
+                            chat_history[-1] = (message, progress_update['response'])
+                            yield chat_history, current_stage, current_model, company, use_case, ""
                         else:
                             deploy_update_count += 1
                             # Only yield every 5th deployment update
@@ -815,10 +1002,12 @@ This may take 1-2 minutes. Watch the AI Feedback tab for progress!"""
                     final_result = None
                     for progress_update in self.run_deployment_streaming():
                         if isinstance(progress_update, tuple):
-                            # Final result with next_msg
                             final_result = progress_update
+                        elif isinstance(progress_update, dict):
+                            current_stage = progress_update.get('stage', current_stage)
+                            chat_history[-1] = (message, f"**DDL Approved - Deploying...**\n\n{progress_update['response']}")
+                            yield chat_history, current_stage, current_model, company, use_case, ""
                         else:
-                            # Progress update - show in chat
                             chat_history[-1] = (message, f"**DDL Approved - Deploying...**\n\n{progress_update}")
                             yield chat_history, current_stage, current_model, company, use_case, ""
                     
@@ -881,10 +1070,12 @@ This may take 1-2 minutes. Watch the AI Feedback tab for progress!"""
                     final_result = None
                     for progress_update in self.run_deployment_streaming():
                         if isinstance(progress_update, tuple):
-                            # Final result with next_msg
                             final_result = progress_update
+                        elif isinstance(progress_update, dict):
+                            current_stage = progress_update.get('stage', current_stage)
+                            chat_history[-1] = (message, progress_update['response'])
+                            yield chat_history, current_stage, current_model, company, use_case, ""
                         else:
-                            # Progress update - show in chat
                             chat_history[-1] = (message, progress_update)
                             yield chat_history, current_stage, current_model, company, use_case, ""
                     
@@ -930,6 +1121,10 @@ This may take 1-2 minutes. Watch the AI Feedback tab for progress!"""
                         for progress_update in self.run_deployment_streaming():
                             if isinstance(progress_update, tuple):
                                 final_result = progress_update
+                            elif isinstance(progress_update, dict):
+                                current_stage = progress_update.get('stage', current_stage)
+                                chat_history.append((None, progress_update['response']))
+                                yield chat_history, current_stage, current_model, company, use_case, ""
                             else:
                                 chat_history.append((None, progress_update))
                                 yield chat_history, current_stage, current_model, company, use_case, ""
@@ -1191,7 +1386,7 @@ This may take 1-2 minutes. Watch the AI Feedback tab for progress!"""
                     return
                 return
         
-        elif current_stage == 'outliers':
+        elif current_stage == 'outlier_adjustment':
             # Handle outlier adjustment stage
             if 'done' in message_lower or 'finish' in message_lower or 'complete' in message_lower:
                 # Close adjuster connection
@@ -1316,40 +1511,49 @@ Try again or type **'done'** to finish."""
                     yield chat_history, current_stage, current_model, company, use_case, ""
                     return
                 
-                # Get current value
-                current_value = adjuster.get_current_value(
-                    match['entity_value'],
-                    match['metric_column']
-                )
-                
-                if current_value == 0:
-                    response = f"""❌ **No data found**
+                # Pick metric column (use hint from match if available)
+                metric_hint = match.get('metric_hint')
+                metric_column = adjuster._pick_metric_column(metric_hint)
+                if not metric_column:
+                    response = "❌ Could not identify a metric column in your schema. Try specifying the column name."
+                    chat_history[-1] = (message, response)
+                    yield chat_history, current_stage, current_model, company, use_case, ""
+                    return
 
-Could not find data for '{match['entity_value']}'.
+                # Get current value (new 4-tuple return: value, matched_name, dim_table, fact_table)
+                entity_type = match.get('entity_type')
+                current_value, matched_entity, dim_table, fact_table = adjuster.get_current_value(
+                    match['entity_value'], metric_column, entity_type
+                )
 
-Please check the spelling or try a different entity.
-Type **'done'** to finish."""
+                if current_value == 0 or matched_entity is None:
+                    response = (
+                        f"❌ **No data found** for `{match['entity_value']}`.\n\n"
+                        f"Check the spelling or try a different entity. Type **'done'** to finish."
+                    )
                     chat_history[-1] = (message, response)
                     yield chat_history, current_stage, current_model, company, use_case, ""
                     return
-                
-                # Calculate target if percentage
+
+                # Calculate target
                 target_value = match.get('target_value')
-                if match.get('is_percentage'):
-                    percentage = match.get('percentage', 0)
+                percentage = match.get('percentage') if match.get('is_percentage') else None
+                if percentage is not None:
                     target_value = current_value * (1 + percentage / 100)
                     match['target_value'] = target_value
-                
+
                 # Generate strategy
                 strategy = adjuster.generate_strategy(
                     match['entity_value'],
-                    match['metric_column'],
+                    metric_column,
                     current_value,
-                    target_value
+                    target_value=target_value,
+                    percentage=percentage,
+                    entity_type=entity_type,
                 )
-                
+
                 # Present smart confirmation
-                confirmation = adjuster.present_smart_confirmation(match, current_value, strategy)
+                confirmation = adjuster.present_smart_confirmation(match, current_value, strategy, metric_column)
                 
                 # Store for execution if user confirms
                 self._pending_adjustment = {
@@ -1414,7 +1618,12 @@ Try a different request or type **'done'** to finish."""
                 # Clean up trailing dots just in case
                 company = company.rstrip('.')
                 return company
-        
+
+        # Fallback: any bare domain.tld in the message (e.g. "Nike.com, Retail Sales")
+        bare_url = re.search(r'\b([a-zA-Z0-9-]+\.[a-zA-Z]{2,})\b', cleaned_message, re.IGNORECASE)
+        if bare_url:
+            return bare_url.group(1).rstrip('.')
+
         return None
     
     def extract_use_case_from_message(self, message):
@@ -1447,7 +1656,18 @@ Try a different request or type **'done'** to finish."""
                 if re.search(r'\.(com|org|net|io|co|ai)\b', use_case, re.IGNORECASE):
                     continue
                 return use_case
-        
+
+        # Fallback: text after a comma or dash following a domain.tld
+        # Handles: "Nike.com, Retail Sales" / "Nike.com - Supply Chain"
+        after_domain = re.search(
+            r'[a-zA-Z0-9-]+\.[a-zA-Z]{2,}[\s]*[,\-–—]\s*(.+)',
+            message, re.IGNORECASE
+        )
+        if after_domain:
+            use_case = after_domain.group(1).strip().rstrip('.')
+            if use_case and not re.search(r'\.(com|org|net|io|co|ai)\b', use_case, re.IGNORECASE):
+                return use_case
+
         return None
     
     def handle_override(self, message):
@@ -1505,12 +1725,15 @@ To change settings, use:
     
     def run_research_streaming(self, company, use_case, generic_context=""):
         """Run the research phase with streaming updates
-        
+
         Args:
             company: Company URL/name
             use_case: Use case name
             generic_context: Additional context provided by user for generic use cases
         """
+        _slog = self._session_logger
+        _t = _slog.log_start("research") if _slog else None
+
         print(f"\n\n[CACHE DEBUG] === run_research_streaming called ===")
         print(f"[CACHE DEBUG] company: {company}")
         print(f"[CACHE DEBUG] use_case: {use_case}\n\n")
@@ -1897,38 +2120,33 @@ To change settings, use:
         """Generate use-case specific Spotter questions based on the schema.
         
         Priority order:
-        1. OutlierPattern.spotter_questions from the vertical×function system
-        2. FUNCTIONS[fn].spotter_templates from the config  
+        1. liveboard_questions.spotter_qs from the vertical×function config
+        2. FUNCTIONS[fn].spotter_templates from the config
         3. Hardcoded fallbacks per use case
         4. Generic questions
         """
         import re
-        
-        # Priority 1: Try the outlier system for configured Spotter questions
+
+        # Priority 1: Use spotter_qs from liveboard_questions in use case config
         try:
-            from outlier_system import get_outliers_for_use_case
-            v = self.vertical or "Generic"
-            f = self.function or "Generic"
-            outlier_config = get_outliers_for_use_case(v, f)
-            
+            uc_config = self.use_case_config or get_use_case_config(
+                self.vertical or "Generic", self.function or "Generic"
+            )
+            lq = uc_config.get("liveboard_questions", [])
             configured_questions = []
-            for op in outlier_config.required + outlier_config.optional:
-                for sq in op.spotter_questions:
+            for q in lq:
+                for sq in q.get('spotter_qs', []):
                     configured_questions.append({
                         'question': sq,
-                        'purpose': f'Reveals {op.name} pattern'
-                    })
-                for fq in op.spotter_followups[:1]:  # 1 followup per pattern
-                    configured_questions.append({
-                        'question': fq,
-                        'purpose': f'Follow-up on {op.name}'
+                        'purpose': f'Reveals {q["title"]} pattern'
                     })
-            
             if configured_questions:
-                self.log_feedback(f"📋 Using {len(configured_questions)} Spotter questions from {v}×{f} outlier config")
-                return configured_questions[:8]  # Cap at 8
+                v = self.vertical or "Generic"
+                f = self.function or "Generic"
+                self.log_feedback(f"📋 Using {len(configured_questions)} Spotter questions from {v}×{f} config")
+                return configured_questions[:8]
         except Exception as e:
-            self.log_feedback(f"⚠️ Outlier Spotter questions not available: {e}")
+            self.log_feedback(f"⚠️ Spotter questions not available: {e}")
         
         # Priority 2: Try FUNCTIONS config for spotter_templates
         try:
@@ -2042,89 +2260,111 @@ To change settings, use:
 - **Ask questions**: Let the AI demonstrate natural language
 - **End with action**: Show how insights lead to decisions""")
     
-    def _generate_spotter_viz_story(self, company_name: str, use_case: str, 
+    def _generate_spotter_viz_story(self, company_name: str, use_case: str,
                                      model_name: str = None, liveboard_name: str = None) -> str:
         """Generate a Spotter Viz story — a conversational sequence of NL prompts
         that can be entered into ThoughtSpot's Spotter Viz agent to build a liveboard.
-        
-        Uses the build_prompt() system with stage="spotter_viz_story" + LLM call.
-        Falls back to a template-based story if LLM fails.
+
+        Returns two sections:
+        1. Persona-driven story built from liveboard_questions config (always present)
+        2. AI-generated story from LLM (appended after a divider)
         """
+        from demo_personas import parse_use_case, get_use_case_config
+
+        v, f = parse_use_case(use_case or '')
+        uc_cfg = get_use_case_config(v or "Generic", f or "Generic")
+        lq = uc_cfg.get("liveboard_questions", [])
+        data_source = model_name or f"{company_name} model"
+
+        # --- Section 1: Persona-driven story from liveboard_questions ---
+        header = f"""# Spotter Viz Story: {company_name}
+## {use_case}
+
+*Copy these prompts into ThoughtSpot Spotter Viz to build this liveboard interactively.*
+
+---
+
+## Part 1: Structured Demo Flow
+
+"""
+        steps = []
+        step_num = 1
+        for q in lq:
+            title = q['title']
+            viz_q = q['viz_question']
+            insight = q.get('insight', '')
+            spotter_qs = q.get('spotter_qs', [])
+
+            step = f"### Step {step_num}: {title}\n"
+            step += f'> "{viz_q}"\n\n'
+            if insight:
+                step += f"**What to look for:** {insight}\n\n"
+            if spotter_qs:
+                step += "**Follow-up Spotter questions:**\n"
+                for sq in spotter_qs[:2]:
+                    step += f'> "{sq}"\n\n'
+            steps.append(step)
+            step_num += 1
+
+        if steps:
+            persona_section = header + "\n".join(steps)
+        else:
+            persona_section = header + f'> "Build a {use_case} dashboard for {company_name} using {data_source}"\n\n'
+
+        persona_section += "\n---\n\n"
+
+        # --- Section 2: AI-generated story ---
         try:
             from prompts import build_prompt
-            from demo_personas import parse_use_case
-            
-            v, f = parse_use_case(use_case or '')
+
             vertical = v or "Generic"
             function = f or "Generic"
-            
-            # Build company context for the prompt
+
             company_context = f"Company: {company_name}\nUse Case: {use_case}"
             if model_name:
                 company_context += f"\nData Source/Model: {model_name}"
             if liveboard_name:
                 company_context += f"\nLiveboard Name: {liveboard_name}"
-            
-            # Add research context if available
             if hasattr(self, 'demo_builder') and self.demo_builder:
                 research = getattr(self.demo_builder, 'company_summary', '') or ''
                 if research:
                     company_context += f"\n\nCompany Research:\n{research[:1500]}"
-            
+
             prompt = build_prompt(
                 stage="spotter_viz_story",
                 vertical=vertical,
                 function=function,
                 company_context=company_context,
             )
-            
-            # Make LLM call
-            from prompt_logger import logged_completion
-            llm_model = self.settings.get('model', DEFAULT_LLM_MODEL)
-            self.log_feedback(f"🎬 Generating Spotter Viz story ({llm_model})...")
-            
-            response = logged_completion(
-                stage="spotter_viz_story",
-                model=llm_model,
-                messages=[{"role": "user", "content": prompt}],
-                max_tokens=2000,
-                temperature=0.7,
-            )
-            
-            story = response.choices[0].message.content.strip()
-            
-            # Add header
-            header = f"""# Spotter Viz Story: {company_name}
-## {use_case}
 
-*Copy these prompts into ThoughtSpot Spotter Viz to build this liveboard interactively.*
+            llm_model = self.settings.get('model', DEFAULT_LLM_MODEL)
+            self.log_feedback(f"🎬 Generating AI Spotter Viz story ({llm_model})...")
 
----
+            provider_name, model_name_str = map_llm_display_to_provider(llm_model)
+            researcher = MultiLLMResearcher(provider=provider_name, model=model_name_str)
+            ai_story = researcher.make_request(prompt, max_tokens=2000, temperature=0.7)
+            ai_section = "## Part 2: AI-Generated Story\n\n" + ai_story
 
-"""
-            return header + story
-            
         except Exception as e:
-            self.log_feedback(f"⚠️ Spotter Viz story generation failed: {e}")
-            # Fallback: build a basic template from what we know
-            return self._build_fallback_spotter_story(company_name, use_case, model_name)
+            self.log_feedback(f"⚠️ AI Spotter Viz story generation failed: {e}")
+            ai_section = "## Part 2: AI-Generated Story\n\n*(Generation failed — use the structured flow above.)*"
+
+        return persona_section + ai_section
     
     def _build_fallback_spotter_story(self, company_name: str, use_case: str, 
                                        model_name: str = None) -> str:
         """Build a basic Spotter Viz story without LLM, using available context."""
         data_source = model_name or f"{company_name} model"
         
-        # Get spotter questions from outlier system
+        # Get spotter questions from use case config
         spotter_qs = []
         try:
-            from demo_personas import parse_use_case
-            from outlier_system import get_outliers_for_use_case
+            from demo_personas import parse_use_case, get_use_case_config
             v, f = parse_use_case(use_case or '')
-            if v or f:
-                outlier_config = get_outliers_for_use_case(v or "Generic", f or "Generic")
-                for op in outlier_config.required:
-                    for sq in op.spotter_questions[:1]:
-                        spotter_qs.append(sq)
+            uc_cfg = get_use_case_config(v or "Generic", f or "Generic")
+            for q in uc_cfg.get("liveboard_questions", []):
+                if q.get("required") and q.get("spotter_qs"):
+                    spotter_qs.append(q["spotter_qs"][0])
         except:
             pass
         
@@ -2641,7 +2881,10 @@ Generate complete CREATE TABLE statements with proper Snowflake syntax and depen
                     # Generate domain-specific realistic data based on column name, then truncate to fit
                     base_value = None
                     if 'NAME' in col_name_upper and 'COMPANY' not in col_name_upper:
-                        if 'PRODUCT' in col_name_upper:
+                        # Check domain-specific name columns BEFORE falling back to fake.name()
+                        if 'DRUG' in col_name_upper or 'MEDICATION' in col_name_upper or 'THERAPEUTIC' in col_name_upper:
+                            base_value = "random.choice(['Lipitor', 'Humira', 'Eliquis', 'Keytruda', 'Revlimid', 'Opdivo', 'Ozempic', 'Dupixent', 'Trulicity', 'Entresto', 'Metformin', 'Atorvastatin', 'Lisinopril', 'Amlodipine', 'Metoprolol', 'Omeprazole', 'Simvastatin', 'Losartan', 'Albuterol', 'Gabapentin'])"
+                        elif 'PRODUCT' in col_name_upper:
                             base_value = "random.choice(['Laptop Pro 15', 'Wireless Mouse 2.4GHz', 'USB-C Cable 6ft', 'Monitor Stand Adjustable', 'Mechanical Keyboard RGB', 'Noise Canceling Headphones', '1080p Webcam', 'Portable SSD 1TB', 'Power Bank 20000mAh', 'Tablet 10 inch', 'Smart Watch', 'Bluetooth Speaker', 'Gaming Mouse Pad', 'Phone Case', 'Screen Protector', 'Charging Cable', 'Desk Lamp LED', 'Laptop Bag', 'Wireless Earbuds', 'USB Hub'])"
                         elif 'CUSTOMER' in col_name_upper or 'USER' in col_name_upper:
                             base_value = "fake.name()"
@@ -2996,16 +3239,19 @@ LegitData will generate realistic, AI-powered data.
     
     def run_deployment_streaming(self):
         """Run deployment to Snowflake using LegitData - yields progress updates"""
+        _slog = self._session_logger
+        _t_deploy = _slog.log_start("deploy") if _slog else None
+
         progress = ""
-        
+
         # Clear and initialize live progress for Snowflake deployment
         self.live_progress_log = ["=" * 60, "SNOWFLAKE DEPLOYMENT STARTING", "=" * 60, ""]
-        
+
         def log_progress(msg):
             """Log to both AI feedback and live progress"""
             self.log_feedback(msg)
             self.live_progress_log.append(msg)
-        
+
         try:
             # Ensure deploy-time modules that still use os.getenv() see Supabase admin settings.
             inject_admin_settings_to_env()
@@ -3409,10 +3655,11 @@ Cannot deploy to ThoughtSpot without tables.""",
             
             yield f"**Starting ThoughtSpot Deployment...**\n\nSchema verified: {database}.{schema_name}\nFound {len(tables)} tables\n\n"
             
-            # Create deployer with settings (fall back to .env if not in settings)
-            ts_url = get_admin_setting('THOUGHTSPOT_URL')
+            # Create deployer — prefer session-selected env (from TS env dropdown),
+            # fall back to admin settings
+            ts_url = (self.settings.get('thoughtspot_url') or '').strip() or get_admin_setting('THOUGHTSPOT_URL')
             ts_user = get_admin_setting('THOUGHTSPOT_ADMIN_USER')
-            ts_secret = get_admin_setting('THOUGHTSPOT_TRUSTED_AUTH_KEY')
+            ts_secret = (self.settings.get('thoughtspot_trusted_auth_key') or '').strip() or get_admin_setting('THOUGHTSPOT_TRUSTED_AUTH_KEY')
             
             deployer = ThoughtSpotDeployer(
                 base_url=ts_url,
@@ -3434,11 +3681,13 @@ Cannot deploy to ThoughtSpot without tables.""",
                 safe_print(msg, flush=True)
             
             # Show initial message
-            yield """**Starting ThoughtSpot Deployment...**
+            yield {
+                'stage': 'thoughtspot',
+                'response': """**Starting ThoughtSpot Deployment...**
 
 Authenticating with ThoughtSpot...
 
-**This takes 2-5 minutes.** 
+**This takes 2-5 minutes.**
 
 **Switch to the "Live Progress" tab** to watch real-time progress.
 
@@ -3449,6 +3698,7 @@ Steps:
 4. Liveboard creation
 
 This chat will update when complete."""
+            }
             
             safe_print("\n" + "="*60, flush=True)
             safe_print("THOUGHTSPOT DEPLOYMENT STARTING", flush=True)
@@ -3492,6 +3742,7 @@ This chat will update when complete."""
                         llm_model=llm_model,
                         tag_name=tag_name_value,
                         liveboard_method=liveboard_method,
+                        share_with=self.settings.get('share_with', '').strip() or None,
                         progress_callback=progress_callback
                     )
                 except Exception as e:
@@ -3660,17 +3911,21 @@ Ask these questions to showcase ThoughtSpot's AI capabilities:
                     try:
                         from smart_data_adjuster import SmartDataAdjuster
                         
-                        # Pass the selected LLM model to the adjuster
+                        # Pass selected LLM model + session-selected TS env to adjuster
                         llm_model = self.settings.get('model', DEFAULT_LLM_MODEL)
-                        adjuster = SmartDataAdjuster(database, schema_name, liveboard_guid, llm_model=llm_model)
+                        adjuster = SmartDataAdjuster(
+                            database, schema_name, liveboard_guid,
+                            llm_model=llm_model,
+                            ts_url=self.settings.get('thoughtspot_url') or None,
+                            ts_secret=self.settings.get('thoughtspot_trusted_auth_key') or None,
+                        )
                         adjuster.connect()
                         
                         if adjuster.load_liveboard_context():
                             self._adjuster = adjuster
                             
                             viz_list = "\n".join([
-                                f"  [{i+1}] {v['name']}\n      Columns: {', '.join(v['columns'][:5])}"
-                                + (f"... (+{len(v['columns'])-5} more)" if len(v['columns']) > 5 else "")
+                                f"  [{i+1}] {v['name']}"
                                 for i, v in enumerate(adjuster.visualizations)
                             ])
                             
@@ -3714,7 +3969,7 @@ I've loaded your liveboard context. Here are the visualizations:
 - Reference by viz number: "viz 3, increase laptop to 50B"
 
 **Try an adjustment now, or type 'done' to finish!**"""
-                            final_stage = 'outliers'
+                            final_stage = 'outlier_adjustment'
                         else:
                             ts_url = get_admin_setting('THOUGHTSPOT_URL', required=False).rstrip('/')
                             model_guid = results.get('model_guid') or ''
@@ -3765,27 +4020,27 @@ Your demo is ready!
 Note: Could not load liveboard context for adjustments: {str(e)}
 Type **'done'** to finish."""
                 else:
-                    # No liveboard_guid - this shouldn't happen on success, but handle gracefully
+                    # Deployed OK but no liveboard GUID returned — treat as partial success
                     ts_url = get_admin_setting('THOUGHTSPOT_URL', required=False).rstrip('/')
                     model_guid = results.get('model_guid') or ''
-                    lb_url = results.get('liveboard_url', '')
                     table_names = results.get('tables', [])
                     tables_list = ', '.join(table_names) if table_names else 'N/A'
-                    
-                    # Build URLs for easy Slack pasting
-                    model_url = f"{ts_url}/#/data/tables/{model_guid}" if model_guid and ts_url else None
-                    
-                    response = f"""**ThoughtSpot Deployment Complete**
+                    model_url = f"{ts_url}/#/data/tables/{model_guid}" if model_guid and ts_url else results.get('model', 'N/A')
+
+                    response = f"""⚠️ **Partial Success — Dataset & Model Created**
+
+Your Snowflake data and ThoughtSpot model were deployed successfully.
+The liveboard GUID couldn't be retrieved — it may still have been created in ThoughtSpot.
 
 **Created:**
 - Connection: {results.get('connection', 'N/A')}
 - Tables: {tables_list}
-- Model: {model_url if model_url else results.get('model', 'N/A')}
-- Liveboard: {lb_url if lb_url else liveboard_name_result}
-
-Your demo is ready!
+- Model: {model_url}
 
-You can now access it in ThoughtSpot.
+**What you can do:**
+- Check the **Spotter Viz Story** tab to recreate the liveboard manually
+- Log into ThoughtSpot to verify whether the liveboard was created
+- Type **'retry liveboard'** to try building it again
 
 Type **'done'** to finish."""
                 
@@ -3793,18 +4048,55 @@ Type **'done'** to finish."""
             else:
                 errors = results.get('errors', ['Unknown error'])
                 error_details = '\n'.join(errors)
-                
-                if 'schema validation' in error_details.lower() or 'schema' in error_details.lower():
-                    guidance = "**Root Cause:** The model TML has validation errors."
-                elif 'connection' in error_details.lower():
-                    guidance = "**Root Cause:** Connection issue with ThoughtSpot or Snowflake"
-                elif 'authenticate' in error_details.lower() or 'auth' in error_details.lower():
-                    guidance = "**Root Cause:** Authentication failed"
+
+                # Check for partial success: Snowflake + model deployed OK but liveboard failed
+                model_ok = bool(results.get('model_guid'))
+                liveboard_errors = [e for e in errors if 'liveboard' in e.lower()]
+                non_liveboard_errors = [e for e in errors if 'liveboard' not in e.lower()]
+
+                if model_ok and liveboard_errors and not non_liveboard_errors:
+                    # Partial success — dataset and model are live, only liveboard failed
+                    ts_url = get_admin_setting('THOUGHTSPOT_URL', required=False).rstrip('/')
+                    model_guid = results.get('model_guid') or ''
+                    table_names = results.get('tables', [])
+                    tables_list = ', '.join(table_names) if table_names else 'N/A'
+                    model_url = f"{ts_url}/#/data/tables/{model_guid}" if model_guid and ts_url else results.get('model', 'N/A')
+                    lb_error = liveboard_errors[0]
+
+                    yield {
+                        'response': f"""⚠️ **Partial Success — Dataset & Model Created**
+
+Your Snowflake data and ThoughtSpot model were deployed successfully.
+The liveboard couldn't be built automatically.
+
+**Created:**
+- Connection: {results.get('connection', 'N/A')}
+- Tables: {tables_list}
+- Model: {model_url}
+
+**Liveboard error:**
+```
+{lb_error}
+```
+
+**What you can do:**
+- Check the **Spotter Viz Story** tab — use that sequence to recreate the liveboard manually in Spotter Viz
+- Type **'retry liveboard'** to try building it again
+- Or continue in ThoughtSpot using your model directly""",
+                        'stage': 'deploy'
+                    }
                 else:
-                    guidance = "**Check the progress log above for details.**"
-                
-                yield {
-                    'response': f"""❌ **ThoughtSpot Deployment Failed**
+                    if 'schema validation' in error_details.lower() or 'schema' in error_details.lower():
+                        guidance = "**Root Cause:** The model TML has validation errors."
+                    elif 'connection' in error_details.lower():
+                        guidance = "**Root Cause:** Connection issue with ThoughtSpot or Snowflake"
+                    elif 'authenticate' in error_details.lower() or 'auth' in error_details.lower():
+                        guidance = "**Root Cause:** Authentication failed"
+                    else:
+                        guidance = "**Check the progress log above for details.**"
+
+                    yield {
+                        'response': f"""❌ **ThoughtSpot Deployment Failed**
 
 **Error Details:**
 ```
@@ -3820,8 +4112,8 @@ Type **'done'** to finish."""
 **Next Steps:**
 - Type **'retry'** to try again
 - Or fix the issues above first""",
-                    'stage': 'deploy'
-                }
+                        'stage': 'deploy'
+                    }
                 
         except Exception as e:
             import traceback
@@ -3971,6 +4263,7 @@ def create_chat_interface():
         current_model = gr.State(default_settings['model'])
         current_company = gr.State(default_settings['company'])
         current_usecase = gr.State(default_settings['use_case'])
+        current_liveboard_name = gr.State(default_settings.get('liveboard_name', ''))
         
         # Header
         gr.Markdown("""
@@ -4234,17 +4527,22 @@ def create_chat_interface():
                 gr.Markdown("### System-Wide Settings")
                 gr.Markdown("These settings apply to all users. Only admins can view and edit.")
                 
+                # Hidden fields — values still saved/loaded but not shown in UI
+                admin_ts_url = gr.Textbox(visible=False)
+                admin_ts_auth_key = gr.Textbox(visible=False)
+                admin_openai_key = gr.Textbox(visible=False)
+                admin_google_key = gr.Textbox(visible=False)
+
                 with gr.Row():
                     with gr.Column():
                         gr.Markdown("#### ThoughtSpot Connection")
-                        admin_ts_url = gr.Textbox(label="ThoughtSpot URL", placeholder="https://your-instance.thoughtspot.cloud")
                         admin_ts_user = gr.Textbox(label="ThoughtSpot Admin Username", placeholder="admin@company.com")
-                        admin_ts_auth_key = gr.Textbox(label="Trusted Auth Key", type="password")
-                        
-                        gr.Markdown("#### LLM API Keys (Legacy storage - runtime uses .env)")
-                        admin_openai_key = gr.Textbox(label="OpenAI API Key", type="password")
-                        admin_google_key = gr.Textbox(label="Google API Key", type="password")
-                    
+                        admin_share_with = gr.Textbox(
+                            label="Default Share With (User or Group)",
+                            placeholder="user@company.com or group-name",
+                            info="System-wide default: model + liveboard shared here after every build"
+                        )
+
                     with gr.Column():
                         gr.Markdown("#### Snowflake Connection")
                         admin_sf_account = gr.Textbox(label="Snowflake Account")
@@ -4268,14 +4566,16 @@ def create_chat_interface():
                     admin_sf_account, admin_sf_kp_user, admin_sf_kp_pk,
                     admin_sf_kp_pass, admin_sf_role, admin_sf_warehouse,
                     admin_sf_database, admin_sf_sso_user,
+                    admin_share_with,
                 ]
-                
+
                 admin_keys_order = [
                     "THOUGHTSPOT_URL", "THOUGHTSPOT_TRUSTED_AUTH_KEY", "THOUGHTSPOT_ADMIN_USER",
                     "OPENAI_API_KEY", "GOOGLE_API_KEY",
                     "SNOWFLAKE_ACCOUNT", "SNOWFLAKE_KP_USER", "SNOWFLAKE_KP_PK",
                     "SNOWFLAKE_KP_PASSPHRASE", "SNOWFLAKE_ROLE", "SNOWFLAKE_WAREHOUSE",
                     "SNOWFLAKE_DATABASE", "SNOWFLAKE_SSO_USER",
+                    "SHARE_WITH",
                 ]
                 
                 def load_admin_settings_handler():
@@ -4322,6 +4622,59 @@ def create_chat_interface():
                     outputs=admin_fields + [admin_settings_status]
                 )
 
+                # --- Session Log Viewer ---
+                gr.Markdown("---")
+                gr.Markdown("### 📋 Session Logs")
+
+                with gr.Row():
+                    log_user_filter = gr.Textbox(label="Filter by user (email, blank=all)", scale=2)
+                    log_limit = gr.Dropdown(label="Show", choices=["25", "50", "100"], value="50", scale=1)
+                    log_refresh_btn = gr.Button("🔄 Refresh", scale=1)
+
+                session_log_display = gr.Dataframe(
+                    headers=["Time", "User", "Stage", "Event", "Duration (ms)", "Error"],
+                    label="Recent Sessions",
+                    interactive=False,
+                    wrap=True,
+                )
+
+                def load_session_logs(user_filter, limit):
+                    """Load session logs from Supabase session_logs table."""
+                    try:
+                        from supabase_client import SupabaseSettings
+                        ss = SupabaseSettings()
+                        if not ss.is_enabled():
+                            return [["Supabase not configured", "", "", "", "", ""]]
+
+                        query = ss.client.table("session_logs").select(
+                            "ts,user_email,stage,event,duration_ms,error"
+                        ).order("ts", desc=True).limit(int(limit))
+
+                        if user_filter and user_filter.strip():
+                            query = query.ilike("user_email", f"%{user_filter.strip()}%")
+
+                        result = query.execute()
+                        rows = []
+                        for r in result.data:
+                            ts = r.get("ts", "")[:19].replace("T", " ")  # trim to seconds
+                            rows.append([
+                                ts,
+                                r.get("user_email", ""),
+                                r.get("stage", ""),
+                                r.get("event", ""),
+                                str(r.get("duration_ms", "") or ""),
+                                r.get("error", "") or "",
+                            ])
+                        return rows if rows else [["No logs found", "", "", "", "", ""]]
+                    except Exception as e:
+                        return [[f"Error: {e}", "", "", "", "", ""]]
+
+                log_refresh_btn.click(
+                    fn=load_session_logs,
+                    inputs=[log_user_filter, log_limit],
+                    outputs=[session_log_display]
+                )
+
                 # Check admin status and toggle admin-only settings visibility
         def check_admin_visibility(request: gr.Request):
             """Check if logged-in user is admin and toggle settings visibility."""
@@ -4411,6 +4764,7 @@ def create_chat_interface():
             company = (str(settings.get("default_company_url", "")).strip() or "Amazon.com")
             use_case = (str(settings.get("default_use_case", "")).strip() or "Sales Analytics")
             model = (str(settings.get("default_llm", "")).strip() or DEFAULT_LLM_MODEL)
+            liveboard_name = str(settings.get("liveboard_name", "")).strip()
             initial_message = build_initial_chat_message(company, use_case)
 
             return (
@@ -4418,10 +4772,12 @@ def create_chat_interface():
                 model,
                 company,
                 use_case,
+                liveboard_name,
                 gr.update(value=model),
+                gr.update(value=liveboard_name),
                 initial_message,
             )
-        
+
         # Wire up load handler - outputs follow SETTINGS_SCHEMA order
         interface.load(
             fn=load_settings_on_startup,
@@ -4437,7 +4793,9 @@ def create_chat_interface():
                 current_model,
                 current_company,
                 current_usecase,
+                current_liveboard_name,
                 chat_components["model_dropdown"],
+                chat_components["liveboard_name_input"],
                 chat_components["msg"],
             ]
         )
@@ -4473,14 +4831,9 @@ def create_chat_tab(chat_controller_state, settings, current_stage, current_mode
             )
             
             with gr.Row():
-                # Pre-populate with company and use case from settings
-                initial_message = build_initial_chat_message(settings['company'], settings['use_case'])
-                print(f"DEBUG: Initial message set to: {initial_message}")
-                print(f"DEBUG: Settings company: {settings['company']}, use_case: {settings['use_case']}")
-                
                 msg = gr.Textbox(
                     label="Your message",
-                    value=initial_message,
+                    value="",
                     placeholder="Type your message here or use /over to change settings...",
                     lines=1,
                     max_lines=1,
@@ -4498,17 +4851,16 @@ def create_chat_tab(chat_controller_state, settings, current_stage, current_mode
         
         # Right column - Status & Settings
         with gr.Column(scale=1):
-            gr.Markdown("### 📊 Current Status")
-            
-            # Stage display (read-only)
-            stage_display = gr.Textbox(
-                label="Stage",
-                value=settings['stage'].replace('_', ' ').title(),
-                interactive=False,
-                lines=1
+            # TS Environment selector
+            ts_env_choices = get_ts_environments()
+            ts_env_dropdown = gr.Dropdown(
+                label="TS Environment",
+                choices=ts_env_choices,
+                value=ts_env_choices[0] if ts_env_choices else None,
+                interactive=True,
             )
-            
-            # Model selector (editable)
+
+            # AI Model selector
             model_dropdown = gr.Dropdown(
                 label="AI Model",
                 choices=list(UI_MODEL_CHOICES),
@@ -4516,60 +4868,135 @@ def create_chat_tab(chat_controller_state, settings, current_stage, current_mode
                 interactive=True,
                 allow_custom_value=True
             )
-            
+
+            # Liveboard name (quick access — same setting as in Settings tab)
+            liveboard_name_input = gr.Textbox(
+                label="Liveboard Name",
+                placeholder="Auto-generated if blank",
+                value=settings.get('liveboard_name', ''),
+                lines=1,
+                interactive=True,
+            )
+
             gr.Markdown("### 📈 Progress")
-            
+
+            # Stage order used to determine done/current/upcoming
+            _STAGE_ORDER = [
+                'initialization', 'awaiting_context', 'research',
+                'create_ddl', 'deploy', 'populate',
+                'thoughtspot', 'outlier_adjustment', 'complete',
+            ]
+            # Each display step: (label, [stage keys that map to it])
+            _PROGRESS_STEPS = [
+                ('Init',         ['initialization']),
+                ('Research',     ['awaiting_context', 'research']),
+                ('DDL',          ['create_ddl']),
+                ('Data',         ['deploy', 'populate']),
+                ('ThoughtSpot',  ['thoughtspot']),
+                ('Data Adjuster',['outlier_adjustment']),
+                ('Complete',     ['complete']),
+            ]
+
             def get_progress_html(stage):
-                """Generate progress HTML - highlight current stage in blue"""
-                stages = [
-                    ('research', 'Research'),
-                    ('create_ddl', 'Create DDL'),
-                    ('deploy', 'Deploy'),
-                    ('thoughtspot', 'ThoughtSpot'),
-                    ('complete', 'Complete')
-                ]
-                
-                html = "<div style='padding: 10px; font-size: 14px;'>"
-                for stage_key, stage_name in stages:
-                    if stage == stage_key:
-                        # Current - blue
-                        html += f"<div style='margin: 5px 0; color: #3b82f6; font-weight: bold;'>🔵 {stage_name}</div>"
+                """Generate progress HTML showing done/current/upcoming states."""
+                try:
+                    current_idx = _STAGE_ORDER.index(stage)
+                except ValueError:
+                    current_idx = 0
+
+                # Find which display step is current
+                current_step = None
+                for step_label, step_keys in _PROGRESS_STEPS:
+                    for k in step_keys:
+                        if stage == k:
+                            current_step = step_label
+                            break
+
+                # Build ordered list of display steps that are reached
+                reached = set()
+                for step_label, step_keys in _PROGRESS_STEPS:
+                    for k in step_keys:
+                        try:
+                            if _STAGE_ORDER.index(k) <= current_idx:
+                                reached.add(step_label)
+                        except ValueError:
+                            pass
+
+                html = "<div style='padding:8px 4px; font-size:13px; line-height:1.8;'>"
+                for step_label, _ in _PROGRESS_STEPS:
+                    # Skip Data Adjuster unless it's active
+                    if step_label == 'Data Adjuster' and step_label not in reached:
+                        continue
+                    if step_label == current_step:
+                        html += (f"<div style='margin:3px 0; color:#3b82f6; font-weight:bold;'>"
+                                 f"▶ {step_label}</div>")
+                    elif step_label in reached:
+                        html += (f"<div style='margin:3px 0; color:#22c55e;'>"
+                                 f"✓ {step_label}</div>")
                     else:
-                        # Not current - gray
-                        html += f"<div style='margin: 5px 0; color: #6b7280;'>⚪ {stage_name}</div>"
-                
+                        html += (f"<div style='margin:3px 0; color:#9ca3af;'>"
+                                 f"○ {step_label}</div>")
                 html += "</div>"
                 return html
-            
+
             progress_html = gr.HTML(get_progress_html('initialization'))
         
         # Event handlers - each creates/uses session-specific controller
-        def send_message(controller, message, history, stage, model, company, usecase, request: gr.Request = None):
+        def send_message(controller, message, history, stage, model, company, usecase, env_label=None, liveboard_name_ui=None, request: gr.Request = None):
             """Handle sending a message - creates controller if needed"""
-            # Create controller for this session if it doesn't exist
+            import traceback
             username = getattr(request, 'username', None) if request else None
             if controller is None:
                 controller = ChatDemoInterface(user_email=username)
                 print(f"[SESSION] Created new ChatDemoInterface for {username or 'anonymous'}")
-            
-            # Process the message
-            for result in controller.process_chat_message(
-                message, history, stage, model, company, usecase
-            ):
-                # result = (chat_history, stage, model, company, usecase, msg_clear)
-                # Extract the new stage to update progress
-                new_stage = result[1] if len(result) > 1 else stage
-                progress = get_progress_html(new_stage)
-                # Yield: controller, chat_history, stage, model, company, usecase, msg_clear, progress_html
-                yield (controller,) + result + (progress,)
-        
-        def quick_action(controller, action_text, history, stage, model, company, usecase, request: gr.Request = None):
+                # Apply selected TS environment settings to the new controller
+                if env_label:
+                    _url = get_ts_env_url(env_label)
+                    _key_value = get_ts_env_auth_key(env_label)
+                    if _url:
+                        controller.settings['thoughtspot_url'] = _url
+                    if _key_value:
+                        controller.settings['thoughtspot_trusted_auth_key'] = _key_value
+            # Always use the current UI value — takes priority over DB-loaded default
+            if liveboard_name_ui is not None:
+                controller.settings['liveboard_name'] = liveboard_name_ui
+            try:
+                for result in controller.process_chat_message(
+                    message, history, stage, model, company, usecase
+                ):
+                    new_stage = result[1] if len(result) > 1 else stage
+                    progress = get_progress_html(new_stage)
+                    yield (controller,) + result + (progress,)
+            except Exception as e:
+                err_tb = traceback.format_exc()
+                print(f"[ERROR] send_message unhandled exception:\n{err_tb}")
+                err_msg = (
+                    f"❌ **An unexpected error occurred**\n\n"
+                    f"`{type(e).__name__}: {e}`\n\n"
+                    f"The pipeline has been interrupted. You can try again or start a new session."
+                )
+                history = history or []
+                history.append((message, err_msg))
+                yield (controller, history, stage, model, company, usecase, "", get_progress_html(stage))
+
+        def quick_action(controller, action_text, history, stage, model, company, usecase, env_label=None, liveboard_name_ui=None, request: gr.Request = None):
             """Handle quick action button clicks"""
             username = getattr(request, 'username', None) if request else None
             if controller is None:
                 controller = ChatDemoInterface(user_email=username)
                 print(f"[SESSION] Created new ChatDemoInterface for {username or 'anonymous'}")
-            
+                # Apply selected TS environment settings to the new controller
+                if env_label:
+                    _url = get_ts_env_url(env_label)
+                    _key_value = get_ts_env_auth_key(env_label)
+                    if _url:
+                        controller.settings['thoughtspot_url'] = _url
+                    if _key_value:
+                        controller.settings['thoughtspot_trusted_auth_key'] = _key_value
+            # Always use the current UI value — takes priority over DB-loaded default
+            if liveboard_name_ui is not None:
+                controller.settings['liveboard_name'] = liveboard_name_ui
+
             for result in controller.process_chat_message(
                 action_text, history, stage, model, company, usecase
             ):
@@ -4578,58 +5005,70 @@ def create_chat_tab(chat_controller_state, settings, current_stage, current_mode
                 yield (controller,) + result + (progress,)
         
         # Wire up send button and enter key
-        msg.submit(
-            fn=send_message,
-            inputs=[chat_controller_state, msg, chatbot, current_stage, current_model, current_company, current_usecase],
-            outputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, msg, progress_html]
-        )
-        
-        send_btn.click(
-            fn=send_message,
-            inputs=[chat_controller_state, msg, chatbot, current_stage, current_model, current_company, current_usecase],
-            outputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, msg, progress_html]
-        )
+        _send_inputs = [chat_controller_state, msg, chatbot, current_stage, current_model, current_company, current_usecase, ts_env_dropdown, liveboard_name_input]
+        _send_outputs = [chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, msg, progress_html]
+
+        msg.submit(fn=send_message, inputs=_send_inputs, outputs=_send_outputs)
+        send_btn.click(fn=send_message, inputs=_send_inputs, outputs=_send_outputs)
         
         # Quick action wrapper functions
-        def start_action(controller, history, stage, model, company, usecase):
-            yield from quick_action(controller, "Start research", history, stage, model, company, usecase)
-        
-        def configure_action(controller, history, stage, model, company, usecase):
-            yield from quick_action(controller, "Configure settings", history, stage, model, company, usecase)
-        
-        def help_action(controller, history, stage, model, company, usecase):
-            yield from quick_action(controller, "Help", history, stage, model, company, usecase)
-        
+        def start_action(controller, history, stage, model, company, usecase, env_label, liveboard_name_ui):
+            yield from quick_action(controller, "Start research", history, stage, model, company, usecase, env_label, liveboard_name_ui)
+
+        def configure_action(controller, history, stage, model, company, usecase, env_label, liveboard_name_ui):
+            yield from quick_action(controller, "Configure settings", history, stage, model, company, usecase, env_label, liveboard_name_ui)
+
+        def help_action(controller, history, stage, model, company, usecase, env_label, liveboard_name_ui):
+            yield from quick_action(controller, "Help", history, stage, model, company, usecase, env_label, liveboard_name_ui)
+
+        _action_inputs = [chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, ts_env_dropdown, liveboard_name_input]
+        _action_outputs = [chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, msg, progress_html]
+
         # Quick action buttons
-        start_btn.click(
-            fn=start_action,
-            inputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase],
-            outputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, msg, progress_html]
-        )
-        
-        configure_btn.click(
-            fn=configure_action,
-            inputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase],
-            outputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, msg, progress_html]
-        )
-        
-        help_btn.click(
-            fn=help_action,
-            inputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase],
-            outputs=[chat_controller_state, chatbot, current_stage, current_model, current_company, current_usecase, msg, progress_html]
-        )
+        start_btn.click(fn=start_action, inputs=_action_inputs, outputs=_action_outputs)
+        configure_btn.click(fn=configure_action, inputs=_action_inputs, outputs=_action_outputs)
+        help_btn.click(fn=help_action, inputs=_action_inputs, outputs=_action_outputs)
         
         # Model dropdown change
         def update_model(new_model, controller, history):
             if controller is not None:
                 controller.settings['model'] = new_model
             return new_model, history
-        
+
         model_dropdown.change(
             fn=update_model,
             inputs=[model_dropdown, chat_controller_state, chatbot],
             outputs=[current_model, chatbot]
         )
+
+        # Liveboard name change — update controller settings in real-time
+        def update_liveboard_name(name, controller):
+            if controller is not None:
+                controller.settings['liveboard_name'] = name
+            return name
+
+        liveboard_name_input.change(
+            fn=update_liveboard_name,
+            inputs=[liveboard_name_input, chat_controller_state],
+            outputs=[liveboard_name_input]
+        )
+
+        # TS environment change — update controller settings in real-time
+        def update_ts_env(label, controller):
+            url = get_ts_env_url(label)
+            auth_key_value = get_ts_env_auth_key(label)
+            if controller is not None:
+                if url:
+                    controller.settings['thoughtspot_url'] = url
+                if auth_key_value:
+                    controller.settings['thoughtspot_trusted_auth_key'] = auth_key_value
+            return label
+
+        ts_env_dropdown.change(
+            fn=update_ts_env,
+            inputs=[ts_env_dropdown, chat_controller_state],
+            outputs=[]
+        )
     
     # Return components for external access
     return {
@@ -4637,8 +5076,9 @@ def create_chat_tab(chat_controller_state, settings, current_stage, current_mode
         'msg': msg,
         'model_dropdown': model_dropdown,
         'send_btn': send_btn,
-        'send_btn_ref': send_btn,  # Reference for updates
-        'stage_display': stage_display,
+        'send_btn_ref': send_btn,
+        'ts_env_dropdown': ts_env_dropdown,
+        'liveboard_name_input': liveboard_name_input,
         'progress_html': progress_html
     }
 
@@ -4651,106 +5091,86 @@ def create_settings_tab():
     gr.Markdown("## ⚙️ Configuration Settings")
     gr.Markdown("Configure your demo builder preferences")
     
-    # Main settings - Most important at the top in 2 columns
-    gr.Markdown("### ⭐ App Settings")
-    
+    # Default Settings — the three fields that appear on the chat page too
+    gr.Markdown("### ⭐ Default Settings")
+    gr.Markdown("*These also appear on the chat page — set them once here as defaults.*")
+
+    # default_company_url removed — company is set via chat conversation
+    default_company_url = gr.Textbox(visible=False)
+
+    # Build use case choices from VERTICALS × FUNCTIONS matrix
+    use_case_choices = []
+    for v_name in VERTICALS:
+        for f_name in FUNCTIONS:
+            use_case_choices.append(f"{v_name} {f_name}")
+    use_case_choices.append("Custom (type in chat)")
+
     with gr.Row():
-        with gr.Column():
-            default_ai_model = gr.Dropdown(
-                label="Default AI Model",
-                choices=list(UI_MODEL_CHOICES),
-                value=DEFAULT_LLM_MODEL,
-                info="Primary model for demo generation",
-                allow_custom_value=True
-            )
-            
-            default_company_url = gr.Textbox(
-                label="Default Company URL",
-                placeholder="https://example.com",
-                value="",
-                info="Pre-fill company for faster starts"
-            )
-            
-            # Build use case choices from VERTICALS × FUNCTIONS matrix
-            use_case_choices = []
-            for v_name in VERTICALS:
-                for f_name in FUNCTIONS:
-                    use_case_choices.append(f"{v_name} {f_name}")
-            use_case_choices.append("Custom (type in chat)")
-            
-            default_use_case = gr.Dropdown(
-                label="Default Use Case",
-                choices=use_case_choices,
-                value="Retail Sales",
+        default_ai_model = gr.Dropdown(
+            label="Default AI Model",
+            choices=list(UI_MODEL_CHOICES),
+            value=DEFAULT_LLM_MODEL,
+            info="Primary model for demo generation",
+            allow_custom_value=True
+        )
+        default_use_case = gr.Dropdown(
+            label="Default Use Case",
+            choices=use_case_choices,
+            value="Retail Sales",
+            info="Vertical × Function combination, or type any custom use case"
+        )
+        liveboard_name = gr.Textbox(
+            label="Default Liveboard Name",
+            placeholder="My Demo Liveboard",
+            value="",
+            info="Default name for generated liveboards (overridable on chat page)"
+        )
 
-                info="Vertical × Function combination, or type any custom use case"
-            )
-        
-            liveboard_name = gr.Textbox(
-                label="Liveboard Name",
-                placeholder="My Demo Liveboard",
-                value="",
-                info="Default name for generated liveboards"
-            )
-            
+    gr.Markdown("---")
+    gr.Markdown("### 🔧 App Settings")
+
+    with gr.Row():
+        with gr.Column():
             tag_name = gr.Textbox(
                 label="Tag Name",
                 placeholder="e.g., 'Sales_Demo' or 'Q4_2024'",
                 value="",
                 info="Tag to apply to all ThoughtSpot objects (connection, tables, model, liveboard)"
             )
-        
-        with gr.Column():
+
             fact_table_size = gr.Dropdown(
                 label="Fact Table Size",
                 choices=["1000", "10000", "100000"],
                 value="1000",
                 info="Number of rows in fact table"
             )
-            
+
             dim_table_size = gr.Dropdown(
                 label="Dim Table Size",
                 choices=["50", "100", "1000"],
                 value="100",
                 info="Number of rows in dimension tables"
             )
-            
+
+        with gr.Column():
             object_naming_prefix = gr.Textbox(
                 label="Object Naming Prefix",
                 placeholder="e.g., 'ACME_' or 'DEMO_'",
                 value="",
                 info="Prefix for ThoughtSpot objects (for future use)"
             )
-            
+
             column_naming_style = gr.Dropdown(
                 label="Column Naming Style",
                 choices=["Regular Case", "snake_case", "camelCase", "PascalCase", "UPPER_CASE", "original"],
                 value="Regular Case",
                 info="Naming convention for ThoughtSpot model columns (Regular Case = State Id, Total Revenue)"
             )
-            
 
-    
-    # Existing Model Section
-    gr.Markdown("### 🔗 Use Existing Model")
-    gr.Markdown("*Skip table/model creation and create liveboard from an existing ThoughtSpot model*")
-    
-    with gr.Row():
-        with gr.Column():
-            use_existing_model = gr.Checkbox(
-                label="Use Existing Model",
-                value=False,
-                info="Enable to skip table creation and use an existing model"
-            )
-            
-            existing_model_guid = gr.Textbox(
-                label="Model GUID",
-                placeholder="e.g., ce2b12d9-07c0-4f38-9394-2cc1d5a5dc6f",
-                value="",
-                info="GUID of the existing ThoughtSpot model to use",
-                visible=True
-            )
-        
+    # Hidden — use_existing_model kept in schema for backward compat but not shown
+    use_existing_model = gr.Checkbox(visible=False, value=False)
+    existing_model_guid = gr.Textbox(visible=False, value="")
+
     gr.Markdown("---")
     gr.Markdown("### 🌍 Other Settings")
     
@@ -4814,8 +5234,8 @@ def create_settings_tab():
     admin_db_accordion = gr.Accordion("💾 Database Connections (Admin)", open=False, visible=True)
     with admin_db_accordion:
         gr.Markdown("""
-        **⚠️ Note:** These settings are **not currently used** by the application.  
-        Connection credentials are read from your `.env` file.  
+        **⚠️ Note:** These fields are **legacy placeholders** and are **not used by deploy runtime**.  
+        Active deployment credentials come from **Admin Settings** (system-wide `__admin__` values in Supabase).  
         This section is for reference/future use only.
         """)
         
@@ -4862,11 +5282,8 @@ def create_settings_tab():
             with gr.Column():
                 gr.Markdown("### 📊 ThoughtSpot Settings")
                 
-                ts_instance_url = gr.Textbox(
-                    label="ThoughtSpot URL",
-                    placeholder="https://your-instance.thoughtspot.cloud",
-                    info="Your ThoughtSpot instance"
-                )
+                # ts_instance_url removed — replaced by TS Environment dropdown on front page
+                ts_instance_url = gr.Textbox(visible=False)
                 
                 ts_username = gr.Textbox(
                     label="ThoughtSpot Username",
@@ -4882,15 +5299,31 @@ def create_settings_tab():
                 )
     
     gr.Markdown("---")
-    gr.Markdown("""
-    ### 🔐 Security Note
-    Settings are saved to your user profile. API keys remain in `.env` file.
-    
-    Required in `.env`:
-    - `SNOWFLAKE_PASSWORD` or `SNOWFLAKE_KP_PK` (for key-pair auth)
-    - `OPENAI_API_KEY` (required for Codex/OpenAI models)
-    """)
-    
+    gr.Markdown("### 🔧 Data Adjuster")
+    gr.Markdown("*Jump straight to data adjustment on an existing liveboard — skips the build pipeline entirely.*")
+
+    with gr.Row():
+        with gr.Column():
+            data_adjuster_url = gr.Textbox(
+                label="Liveboard URL",
+                placeholder="https://your-instance.thoughtspot.cloud/#/pinboard/guid",
+                value="",
+                info="Paste a ThoughtSpot liveboard URL to open it directly in Data Adjuster"
+            )
+
+    gr.Markdown("---")
+    gr.Markdown("### 🔗 Sharing")
+    gr.Markdown("*After building, model and liveboard are shared with this user or group (can edit).*")
+    with gr.Row():
+        with gr.Column():
+            share_with = gr.Textbox(
+                label="Share With (User or Group)",
+                placeholder="your.email@company.com or group-name",
+                value="",
+                info="User email (contains @) or group name. Leave blank to skip sharing."
+            )
+
+    gr.Markdown("---")
     with gr.Row():
         save_settings_btn = gr.Button("💾 Save Settings", variant="primary", size="lg")
         reset_settings_btn = gr.Button("🔄 Reset to Defaults", size="lg")
@@ -4956,6 +5389,8 @@ def create_settings_tab():
         'default_schema': default_schema,
         'ts_instance_url': ts_instance_url,
         'ts_username': ts_username,
+        'data_adjuster_url': data_adjuster_url,
+        'share_with': share_with,
         # Status
         'settings_status': settings_status,
         # Admin-only visibility toggles

conversational_data_adjuster.py

@@ -1,448 +0,0 @@
-"""
-Conversational Data Adjuster
-
-Allows natural language data adjustments with strategy selection:
-User: "Make 1080p webcam sales 50B"
-System: Analyzes data, presents options
-User: Picks strategy
-System: Executes SQL
-"""
-
-from typing import Dict, List, Optional
-from snowflake_auth import get_snowflake_connection
-import json
-from llm_config import DEFAULT_LLM_MODEL, build_openai_chat_token_kwargs
-from llm_client_factory import create_openai_client
-
-
-class ConversationalDataAdjuster:
-    """Interactive data adjustment with user choice of strategy"""
-    
-    def __init__(self, database: str, schema: str, model_id: str):
-        self.database = database
-        self.schema = schema
-        self.model_id = model_id
-        self.conn = None
-        self.openai_client = create_openai_client()
-        self.current_context = {}
-    
-    def connect(self):
-        """Connect to Snowflake"""
-        self.conn = get_snowflake_connection()
-        cursor = self.conn.cursor()
-        cursor.execute(f"USE DATABASE {self.database}")
-        cursor.execute(f'USE SCHEMA "{self.schema}"')  # Quote schema name (may start with number)
-        print(f"✅ Connected to {self.database}.{self.schema}")
-    
-    def parse_adjustment_request(self, request: str, available_tables: List[str]) -> Dict:
-        """
-        Parse natural language request to identify what to adjust
-        
-        Args:
-            request: e.g., "increase 1080p webcam sales to 50B"
-            available_tables: List of table names in schema
-        
-        Returns:
-            {
-                'table': 'SALES_TRANSACTIONS',
-                'entity_column': 'product_name',
-                'entity_value': '1080p webcam',
-                'metric_column': 'total_revenue',
-                'target_value': 50000000000,
-                'current_value': 30000000000  # if known
-            }
-        """
-        prompt = f"""Parse this data adjustment request.
-
-Request: "{request}"
-
-Available tables: {', '.join(available_tables)}
-
-Common columns:
-- SALES_TRANSACTIONS: PRODUCT_ID, CUSTOMER_ID, SELLER_ID, TOTAL_REVENUE, QUANTITY_SOLD, PROFIT_MARGIN, ORDER_DATE
-- PRODUCTS: PRODUCT_ID, PRODUCT_NAME, CATEGORY
-- CUSTOMERS: CUSTOMER_ID, CUSTOMER_SEGMENT
-
-IMPORTANT - Column Meanings:
-- TOTAL_REVENUE = dollar value of sales (e.g., $50B means fifty billion dollars)
-- QUANTITY_SOLD = number of units sold (e.g., 1000 units)
-
-When user says "sales", "revenue", or dollar amounts → use TOTAL_REVENUE
-When user says "quantity", "units", or "items sold" → use QUANTITY_SOLD
-
-Note: To filter by product name, you'll need to reference PRODUCTS table or use PRODUCT_ID directly
-
-Extract:
-1. table: Which table to modify (likely SALES_TRANSACTIONS for revenue/sales changes)
-2. entity_column: Column to filter by (e.g., product_name, customer_segment)
-3. entity_value: Specific value to filter (e.g., "1080p webcam", "Electronics")
-4. metric_column: Numeric column to change
-   - If request mentions "sales", "revenue", or dollar amounts → TOTAL_REVENUE
-   - If request mentions "quantity", "units", "items" → QUANTITY_SOLD
-   - If request mentions "profit margin" → PROFIT_MARGIN
-5. target_value: The target numeric value (convert B to billions, M to millions)
-
-Return ONLY valid JSON: {{"table": "...", "entity_column": "...", "entity_value": "...", "metric_column": "...", "target_value": 123}}
-
-Examples:
-- "increase 1080p webcam sales to 50B" → {{"table": "SALES_TRANSACTIONS", "entity_column": "PRODUCT_ID", "entity_value": "1080p Webcam", "metric_column": "TOTAL_REVENUE", "target_value": 50000000000, "needs_join": "PRODUCTS", "join_column": "PRODUCT_NAME"}}
-- "make tablet revenue 100 billion" → {{"table": "SALES_TRANSACTIONS", "entity_column": "PRODUCT_ID", "entity_value": "Tablet", "metric_column": "TOTAL_REVENUE", "target_value": 100000000000, "needs_join": "PRODUCTS", "join_column": "PRODUCT_NAME"}}
-- "increase laptop quantity to 50000 units" → {{"table": "SALES_TRANSACTIONS", "entity_column": "PRODUCT_ID", "entity_value": "Laptop", "metric_column": "QUANTITY_SOLD", "target_value": 50000, "needs_join": "PRODUCTS", "join_column": "PRODUCT_NAME"}}
-- "set profit margin to 25% for electronics" → {{"table": "SALES_TRANSACTIONS", "entity_column": "PRODUCT_ID", "entity_value": "electronics", "metric_column": "PROFIT_MARGIN", "target_value": 25, "needs_join": "PRODUCTS", "join_column": "CATEGORY"}}
-
-If the entity refers to a column not in the target table (e.g., product_name when modifying SALES_TRANSACTIONS),
-include "needs_join" with the table name and "join_column" with the column to match on.
-"""
-        
-        response = self.openai_client.chat.completions.create(
-            model=DEFAULT_LLM_MODEL,
-            messages=[{"role": "user", "content": prompt}],
-            temperature=0,
-            **build_openai_chat_token_kwargs(DEFAULT_LLM_MODEL, 2000),
-        )
-        
-        content = response.choices[0].message.content
-        
-        # Strip markdown code blocks if present
-        if content.startswith('```'):
-            lines = content.split('\n')
-            content = '\n'.join(lines[1:-1])  # Remove first and last line (``` markers)
-        
-        try:
-            result = json.loads(content)
-            print(f"✅ Parsed request: {result.get('entity_value')} - {result.get('metric_column')}")
-            return result
-        except json.JSONDecodeError as e:
-            print(f"❌ Failed to parse JSON: {e}")
-            print(f"Content was: {content}")
-            return {'error': f'Failed to parse request: {content}'}
-    
-    def analyze_current_data(self, adjustment: Dict) -> Dict:
-        """
-        Query current state of the data
-        
-        Returns:
-            {
-                'current_total': float,
-                'row_count': int,
-                'avg_value': float,
-                'min_value': float,
-                'max_value': float,
-                'gap': float  # target - current
-            }
-        """
-        cursor = self.conn.cursor()
-        
-        table = adjustment['table']
-        entity_col = adjustment['entity_column']
-        entity_val = adjustment['entity_value']
-        metric_col = adjustment['metric_column']
-        target = adjustment['target_value']
-        
-        # Build WHERE clause - handle joins if needed
-        if adjustment.get('needs_join'):
-            join_table = adjustment['needs_join']
-            join_col = adjustment['join_column']
-            where_clause = f"""WHERE {entity_col} IN (
-                SELECT PRODUCT_ID FROM {self.database}."{self.schema}".{join_table}
-                WHERE LOWER({join_col}) = LOWER('{entity_val}')
-            )"""
-        else:
-            where_clause = f"WHERE LOWER({entity_col}) = LOWER('{entity_val}')"
-        
-        # Query current state
-        query = f"""
-        SELECT 
-            SUM({metric_col}) as total,
-            COUNT(*) as row_count,
-            AVG({metric_col}) as avg_value,
-            MIN({metric_col}) as min_value,
-            MAX({metric_col}) as max_value
-        FROM {self.database}."{self.schema}".{table}
-        {where_clause}
-        """
-        
-        print(f"\n🔍 Analyzing current data...")
-        print(f"   Query: {query}")
-        
-        cursor.execute(query)
-        row = cursor.fetchone()
-        
-        current_total = float(row[0]) if row[0] else 0
-        row_count = int(row[1])
-        avg_value = float(row[2]) if row[2] else 0
-        min_value = float(row[3]) if row[3] else 0
-        max_value = float(row[4]) if row[4] else 0
-        
-        gap = target - current_total
-        
-        return {
-            'current_total': current_total,
-            'row_count': row_count,
-            'avg_value': avg_value,
-            'min_value': min_value,
-            'max_value': max_value,
-            'gap': gap
-        }
-    
-    def generate_strategy_options(self, adjustment: Dict, analysis: Dict) -> List[Dict]:
-        """
-        Generate 3 strategy options for achieving the target
-        
-        Returns list of strategies with details
-        """
-        table = adjustment['table']
-        entity_col = adjustment['entity_column']
-        entity_val = adjustment['entity_value']
-        metric_col = adjustment['metric_column']
-        target = adjustment['target_value']
-        
-        # Build WHERE clause - handle joins if needed
-        if adjustment.get('needs_join'):
-            join_table = adjustment['needs_join']
-            join_col = adjustment['join_column']
-            where_clause = f"""{entity_col} IN (
-                SELECT PRODUCT_ID FROM {self.database}."{self.schema}".{join_table}
-                WHERE LOWER({join_col}) = LOWER('{entity_val}')
-            )"""
-        else:
-            where_clause = f"LOWER({entity_col}) = LOWER('{entity_val}')"
-        
-        current = analysis['current_total']
-        gap = analysis['gap']
-        row_count = analysis['row_count']
-        
-        if gap <= 0:
-            return [{
-                'id': 'decrease',
-                'name': 'Decrease All',
-                'description': f"Current value ({current:,.0f}) already exceeds target ({target:,.0f})",
-                'sql': None
-            }]
-        
-        strategies = []
-        
-        # Strategy A: Distribute increase across all rows
-        multiplier = target / current if current > 0 else 1
-        percentage_increase = (multiplier - 1) * 100
-        
-        strategies.append({
-            'id': 'A',
-            'name': 'Distribute Across All Transactions',
-            'description': f"Increase all {row_count:,} existing transactions by {percentage_increase:.1f}%",
-            'details': {
-                'approach': 'Multiply all existing values',
-                'rows_affected': row_count,
-                'new_avg': analysis['avg_value'] * multiplier
-            },
-            'sql': f"""UPDATE {self.database}."{self.schema}".{table}
-SET {metric_col} = {metric_col} * {multiplier:.6f}
-WHERE {where_clause}"""
-        })
-        
-        # Strategy B: Add new large transactions
-        num_new_transactions = max(1, int(gap / (analysis['max_value'] * 2)))  # Add transactions 2x the current max
-        value_per_new = gap / num_new_transactions
-        
-        strategies.append({
-            'id': 'B',
-            'name': 'Add New Large Transactions',
-            'description': f"Insert {num_new_transactions} new transactions of ${value_per_new:,.0f} each",
-            'details': {
-                'approach': 'Create new outlier transactions',
-                'rows_to_add': num_new_transactions,
-                'value_each': value_per_new
-            },
-            'sql': f"""-- INSERT new transactions (requires full row data)
--- INSERT INTO {self.database}."{self.schema}".{table} ({entity_col}, {metric_col}, ...)
--- VALUES ('{entity_val}', {value_per_new}, ...)
--- NOTE: This requires knowing all required columns in the table"""
-        })
-        
-        # Strategy C: Boost top transactions
-        top_n = min(10, max(1, row_count // 10))  # Top 10% of transactions
-        boost_needed_per_row = gap / top_n
-        
-        strategies.append({
-            'id': 'C',
-            'name': 'Boost Top Transactions',
-            'description': f"Increase the top {top_n} transactions by ${boost_needed_per_row:,.0f} each",
-            'details': {
-                'approach': 'Create outliers from existing top transactions',
-                'rows_affected': top_n,
-                'boost_per_row': boost_needed_per_row
-            },
-            'sql': f"""WITH top_rows AS (
-  SELECT * FROM {self.database}."{self.schema}".{table}
-  WHERE {where_clause}
-  ORDER BY {metric_col} DESC
-  LIMIT {top_n}
-)
-UPDATE {self.database}."{self.schema}".{table} t
-SET {metric_col} = {metric_col} + {boost_needed_per_row:.2f}
-WHERE EXISTS (
-  SELECT 1 FROM top_rows
-  WHERE top_rows.rowid = t.rowid
-)"""
-        })
-        
-        return strategies
-    
-    def present_options(self, adjustment: Dict, analysis: Dict, strategies: List[Dict]) -> None:
-        """Display options to user in a friendly format"""
-        
-        print("\n" + "="*80)
-        print("📊 DATA ADJUSTMENT OPTIONS")
-        print("="*80)
-        
-        entity = f"{adjustment['entity_column']}='{adjustment['entity_value']}'"
-        metric = adjustment['metric_column']
-        
-        print(f"\n🎯 Goal: Adjust {metric} for {entity}")
-        print(f"   Current Total: ${analysis['current_total']:,.0f}")
-        print(f"   Target Total:  ${adjustment['target_value']:,.0f}")
-        print(f"   Gap to Fill:   ${analysis['gap']:,.0f} ({analysis['gap']/analysis['current_total']*100:.1f}% increase)")
-        
-        print(f"\n📈 Current Data:")
-        print(f"   Rows: {analysis['row_count']:,}")
-        print(f"   Average: ${analysis['avg_value']:,.0f}")
-        print(f"   Range: ${analysis['min_value']:,.0f} - ${analysis['max_value']:,.0f}")
-        
-        print(f"\n" + "="*80)
-        print("STRATEGY OPTIONS:")
-        print("="*80)
-        
-        for strategy in strategies:
-            print(f"\n[{strategy['id']}] {strategy['name']}")
-            print(f"    {strategy['description']}")
-            
-            if 'details' in strategy:
-                details = strategy['details']
-                print(f"    Details:")
-                for key, value in details.items():
-                    if isinstance(value, float):
-                        print(f"      - {key}: ${value:,.0f}")
-                    else:
-                        print(f"      - {key}: {value}")
-            
-            if strategy['sql']:
-                print(f"\n    SQL Preview:")
-                sql_preview = strategy['sql'].strip().split('\n')
-                for line in sql_preview[:3]:  # Show first 3 lines
-                    print(f"      {line}")
-                if len(sql_preview) > 3:
-                    print(f"      ... ({len(sql_preview)-3} more lines)")
-        
-        print("\n" + "="*80)
-    
-    def execute_strategy(self, strategy: Dict) -> Dict:
-        """Execute the chosen strategy"""
-        
-        if not strategy['sql']:
-            return {
-                'success': False,
-                'error': 'This strategy requires manual implementation (INSERT statements)'
-            }
-        
-        cursor = self.conn.cursor()
-        
-        print(f"\n⚙️  Executing strategy: {strategy['name']}")
-        print(f"   SQL: {strategy['sql'][:200]}...")
-        
-        try:
-            cursor.execute(strategy['sql'])
-            rows_affected = cursor.rowcount
-            self.conn.commit()
-            
-            return {
-                'success': True,
-                'message': f"✅ Updated {rows_affected} rows",
-                'rows_affected': rows_affected
-            }
-        except Exception as e:
-            self.conn.rollback()
-            return {
-                'success': False,
-                'error': str(e)
-            }
-    
-    def get_available_tables(self) -> List[str]:
-        """Get list of tables in the schema"""
-        cursor = self.conn.cursor()
-        cursor.execute(f"""
-            SELECT TABLE_NAME 
-            FROM {self.database}.INFORMATION_SCHEMA.TABLES 
-            WHERE TABLE_SCHEMA = '{self.schema}'
-        """)
-        tables = [row[0] for row in cursor.fetchall()]
-        return tables
-    
-    def close(self):
-        """Close connection"""
-        if self.conn:
-            self.conn.close()
-
-
-# Test/demo function
-def demo_conversation():
-    """Simulate the conversational flow"""
-    
-    print("""
-╔════════════════════════════════════════════════════════════╗
-║                                                            ║
-║     Conversational Data Adjuster Demo                     ║
-║                                                            ║
-╚════════════════════════════════════════════════════════════╝
-    """)
-    
-    # Setup from environment variables
-    from dotenv import load_dotenv
-    load_dotenv()
-    
-    adjuster = ConversationalDataAdjuster(
-        database=os.getenv('SNOWFLAKE_DATABASE'),
-        schema="20251116_140933_AMAZO_SAL",  # Schema from deployment
-        model_id="3c97b0d6-448b-440a-b628-bac1f3d73049"
-    )
-    
-    print(f"Using database: {os.getenv('SNOWFLAKE_DATABASE')}")
-    print(f"Using schema: 20251116_140933_AMAZO_SAL")
-    
-    adjuster.connect()
-    
-    # User request (using actual product from our data)
-    user_request = "increase 1080p Webcam sales to 50 billion"
-    print(f"\n💬 User: \"{user_request}\"")
-    print(f"   (Current: ~$17.6B, Target: $50B)")
-    
-    # Step 1: Parse request
-    tables = adjuster.get_available_tables()
-    adjustment = adjuster.parse_adjustment_request(user_request, tables)
-    
-    # Step 2: Analyze current data
-    analysis = adjuster.analyze_current_data(adjustment)
-    
-    # Step 3: Generate strategies
-    strategies = adjuster.generate_strategy_options(adjustment, analysis)
-    
-    # Step 4: Present options
-    adjuster.present_options(adjustment, analysis, strategies)
-    
-    # Step 5: User picks (simulated)
-    print("\n💬 User: \"Use strategy A\"")
-    chosen_strategy = strategies[0]  # Strategy A
-    
-    # Step 6: Execute
-    result = adjuster.execute_strategy(chosen_strategy)
-    
-    if result['success']:
-        print(f"\n{result['message']}")
-    else:
-        print(f"\n❌ Error: {result.get('error')}")
-    
-    adjuster.close()
-
-
-if __name__ == "__main__":
-    demo_conversation()
-

data_adjuster.py

@@ -1,213 +0,0 @@
-"""
-Data Adjustment Module for Liveboard Refinement
-
-Allows natural language adjustments to demo data:
-- "make Product A 55% higher"
-- "increase Customer B revenue by 20%"
-- "set profit margin to 15% for Segment C"
-"""
-
-import re
-from typing import Dict, Optional
-from snowflake_auth import get_snowflake_connection
-from llm_config import DEFAULT_LLM_MODEL, build_openai_chat_token_kwargs
-from llm_client_factory import create_openai_client
-
-
-class DataAdjuster:
-    """Adjust demo data based on natural language requests"""
-    
-    def __init__(self, database: str, schema: str):
-        self.database = database
-        self.schema = schema
-        self.conn = None
-        self.openai_client = create_openai_client()
-    
-    def connect(self):
-        """Connect to Snowflake"""
-        self.conn = get_snowflake_connection()
-        self.conn.cursor().execute(f"USE DATABASE {self.database}")
-        self.conn.cursor().execute(f"USE SCHEMA {self.schema}")
-        print(f"✅ Connected to {self.database}.{self.schema}")
-    
-    def parse_adjustment_request(self, request: str, available_columns: list) -> Dict:
-        """
-        Parse natural language adjustment request using AI
-        
-        Args:
-            request: e.g., "make Product A 55% higher" or "increase revenue for Customer B by 20%"
-            available_columns: List of column names in the data
-        
-        Returns:
-            {
-                'entity_column': 'product_name',
-                'entity_value': 'Product A',
-                'metric_column': 'total_revenue',
-                'adjustment_type': 'percentage_increase',
-                'adjustment_value': 55
-            }
-        """
-        prompt = f"""Parse this data adjustment request and extract structured information.
-
-Request: "{request}"
-
-Available columns in the dataset: {', '.join(available_columns)}
-
-Extract:
-1. entity_column: Which column identifies what to change (e.g., product_name, customer_segment)
-2. entity_value: The specific value to filter by (e.g., "Product A", "Electronics")
-3. metric_column: Which numeric column to adjust (e.g., total_revenue, profit_margin, quantity_sold)
-4. adjustment_type: One of: "percentage_increase", "percentage_decrease", "set_value", "add_value"
-5. adjustment_value: The numeric value (e.g., 55 for "55%", 1000 for "add 1000")
-
-Return ONLY a JSON object with these fields. If you can't parse it, return {{"error": "description"}}.
-
-Examples:
-- "make Product A 55% higher" → {{"entity_column": "product_name", "entity_value": "Product A", "metric_column": "total_revenue", "adjustment_type": "percentage_increase", "adjustment_value": 55}}
-- "set profit margin to 15% for Electronics" → {{"entity_column": "category", "entity_value": "Electronics", "metric_column": "profit_margin", "adjustment_type": "set_value", "adjustment_value": 15}}
-"""
-        
-        response = self.openai_client.chat.completions.create(
-            model=DEFAULT_LLM_MODEL,
-            messages=[{"role": "user", "content": prompt}],
-            temperature=0,
-            **build_openai_chat_token_kwargs(DEFAULT_LLM_MODEL, 2000),
-        )
-        
-        import json
-        result = json.loads(response.choices[0].message.content)
-        return result
-    
-    def get_available_columns(self, table_name: str) -> list:
-        """Get list of columns from a table"""
-        cursor = self.conn.cursor()
-        cursor.execute(f"""
-            SELECT COLUMN_NAME 
-            FROM {self.database}.INFORMATION_SCHEMA.COLUMNS 
-            WHERE TABLE_SCHEMA = '{self.schema}' 
-            AND TABLE_NAME = '{table_name.upper()}'
-        """)
-        columns = [row[0].lower() for row in cursor.fetchall()]
-        return columns
-    
-    def apply_adjustment(self, table_name: str, adjustment: Dict) -> Dict:
-        """
-        Apply the parsed adjustment to the database
-        
-        Returns:
-            {'success': bool, 'message': str, 'rows_affected': int}
-        """
-        if 'error' in adjustment:
-            return {'success': False, 'message': adjustment['error']}
-        
-        cursor = self.conn.cursor()
-        
-        # Build the UPDATE statement
-        entity_col = adjustment['entity_column']
-        entity_val = adjustment['entity_value']
-        metric_col = adjustment['metric_column']
-        adj_type = adjustment['adjustment_type']
-        adj_value = adjustment['adjustment_value']
-        
-        # Calculate new value based on adjustment type
-        if adj_type == 'percentage_increase':
-            new_value_expr = f"{metric_col} * (1 + {adj_value}/100.0)"
-        elif adj_type == 'percentage_decrease':
-            new_value_expr = f"{metric_col} * (1 - {adj_value}/100.0)"
-        elif adj_type == 'set_value':
-            new_value_expr = f"{adj_value}"
-        elif adj_type == 'add_value':
-            new_value_expr = f"{metric_col} + {adj_value}"
-        else:
-            return {'success': False, 'message': f"Unknown adjustment type: {adj_type}"}
-        
-        # Execute UPDATE
-        update_sql = f"""
-        UPDATE {self.database}.{self.schema}.{table_name}
-        SET {metric_col} = {new_value_expr}
-        WHERE LOWER({entity_col}) = LOWER('{entity_val}')
-        """
-        
-        print(f"\n🔧 Executing adjustment:")
-        print(f"   SQL: {update_sql}")
-        
-        try:
-            cursor.execute(update_sql)
-            rows_affected = cursor.rowcount
-            self.conn.commit()
-            
-            return {
-                'success': True,
-                'message': f"Updated {rows_affected} rows: {entity_col}='{entity_val}', adjusted {metric_col} by {adj_type}",
-                'rows_affected': rows_affected
-            }
-        except Exception as e:
-            return {
-                'success': False,
-                'message': f"Database error: {str(e)}"
-            }
-    
-    def adjust_data_for_liveboard(self, request: str, table_name: str) -> Dict:
-        """
-        Full workflow: parse request, update data
-        
-        Args:
-            request: Natural language request like "make Product A 55% higher"
-            table_name: Name of the table to update
-        
-        Returns:
-            Result dictionary with success status and details
-        """
-        if not self.conn:
-            self.connect()
-        
-        # Get available columns
-        columns = self.get_available_columns(table_name)
-        print(f"📋 Available columns: {', '.join(columns)}")
-        
-        # Parse the request
-        print(f"\n🤔 Parsing request: '{request}'")
-        adjustment = self.parse_adjustment_request(request, columns)
-        print(f"✅ Parsed: {adjustment}")
-        
-        if 'error' in adjustment:
-            return {'success': False, 'error': adjustment['error']}
-        
-        # Apply the adjustment
-        result = self.apply_adjustment(table_name, adjustment)
-        
-        return result
-    
-    def close(self):
-        """Close database connection"""
-        if self.conn:
-            self.conn.close()
-
-
-# Example usage function
-def test_data_adjustment():
-    """Test the data adjustment functionality"""
-    adjuster = DataAdjuster(
-        database="DEMO_DATABASE",
-        schema="DEMO_SCHEMA"
-    )
-    
-    # Example: "make Product A 55% higher"
-    result = adjuster.adjust_data_for_liveboard(
-        request="make Product A 55% higher",
-        table_name="FACT_SALES"
-    )
-    
-    print(f"\n{'='*60}")
-    if result['success']:
-        print(f"✅ SUCCESS: {result['message']}")
-        print(f"📊 Rows affected: {result['rows_affected']}")
-    else:
-        print(f"❌ FAILED: {result.get('error', result.get('message'))}")
-    
-    adjuster.close()
-
-
-if __name__ == "__main__":
-    test_data_adjustment()
-

demo_personas.py

@@ -3,6 +3,8 @@ Demo Personas and Use Case Configurations
 All persona data and prompt templates for use case-driven demo preparation
 """
 
+from copy import deepcopy
+
 from schema_utils import extract_key_business_terms
 
 # ============================================================================
@@ -12,27 +14,92 @@ from schema_utils import extract_key_business_terms
 # Keep USE_CASE_PERSONAS below for backward compatibility during transition
 # ============================================================================
 
+# Story controls are intentionally strict and deterministic for demo realism.
+DEFAULT_STORY_CONTROLS = {
+    "realism_mode": "strict",
+    "trend_style": "smooth",
+    "trend_noise_band_pct": 0.04,
+    "seasonal_strength": 0.10,
+    "reproducibility_seed_strategy": "company_use_case_hash",
+    "outlier_budget": {
+        "max_events": 2,
+        "max_points_pct": 0.02,
+        "event_multiplier_min": 1.8,
+        "event_multiplier_max": 3.2,
+    },
+    "value_guardrails": {
+        "allow_negative_revenue": False,
+        "max_mom_change_pct": 35,
+    },
+    "quality_targets": {
+        "min_semantic_pass_ratio": 0.98,
+    },
+}
+
+
+def _deep_merge_dict(base: dict, update: dict) -> dict:
+    """Deep merge dictionaries while preserving base keys."""
+    merged = deepcopy(base)
+    for key, val in (update or {}).items():
+        if isinstance(val, dict) and isinstance(merged.get(key), dict):
+            merged[key] = _deep_merge_dict(merged[key], val)
+        else:
+            merged[key] = deepcopy(val)
+    return merged
+
+
+def _merge_story_controls(*controls: dict) -> dict:
+    merged = deepcopy(DEFAULT_STORY_CONTROLS)
+    for control in controls:
+        merged = _deep_merge_dict(merged, control or {})
+    return merged
+
+
 # VERTICALS: Industry-specific context
 VERTICALS = {
     "Retail": {
         "typical_entities": ["Store", "Product", "Category", "Region", "Customer"],
         "industry_terms": ["SKU", "basket", "shrink", "markdown", "comp sales", "footfall"],
         "data_patterns": ["seasonality", "holiday_spikes", "weather_impact", "back_to_school"],
+        "story_controls": {
+            "seasonal_strength": 0.18,
+            "trend_noise_band_pct": 0.035,
+            "outlier_budget": {"max_events": 3},
+            "recommended_companies": ["Amazon.com", "Walmart.com", "Target.com"],
+        },
     },
     "Banking": {
         "typical_entities": ["Account", "Customer", "Branch", "Product", "Loan"],
         "industry_terms": ["AUM", "NIM", "deposits", "charge-off", "delinquency", "APR"],
         "data_patterns": ["month_end_spikes", "rate_sensitivity", "quarter_close"],
+        "story_controls": {
+            "seasonal_strength": 0.06,
+            "trend_noise_band_pct": 0.025,
+            "outlier_budget": {"max_events": 2},
+            "recommended_companies": ["Chase.com", "CapitalOne.com", "BankOfAmerica.com"],
+        },
     },
     "Software": {
         "typical_entities": ["Account", "User", "Subscription", "Feature", "License"],
         "industry_terms": ["ARR", "MRR", "churn", "NRR", "seats", "expansion"],
         "data_patterns": ["renewal_cycles", "usage_spikes", "trial_conversion"],
+        "story_controls": {
+            "seasonal_strength": 0.05,
+            "trend_noise_band_pct": 0.03,
+            "outlier_budget": {"max_events": 2},
+            "recommended_companies": ["Salesforce.com", "HubSpot.com", "Atlassian.com"],
+        },
     },
     "Manufacturing": {
         "typical_entities": ["Plant", "Line", "Product", "Supplier", "Shift"],
         "industry_terms": ["OEE", "yield", "scrap", "downtime", "throughput", "WIP"],
         "data_patterns": ["shift_patterns", "maintenance_cycles", "supply_disruptions"],
+        "story_controls": {
+            "seasonal_strength": 0.08,
+            "trend_noise_band_pct": 0.03,
+            "outlier_budget": {"max_events": 2},
+            "recommended_companies": ["Caterpillar.com", "GE.com", "3M.com"],
+        },
     },
 }
 
@@ -53,6 +120,18 @@ FUNCTIONS = {
             "Why did {kpi} drop last month?",
             "Compare {kpi} across {dimension}",
         ],
+        "liveboard_questions": [
+            {"title": "Revenue Trend", "viz_type": "LINE", "viz_question": "Dollar Sales trend by month", "required": True, "insight": "Monthly revenue trend reveals growth patterns and seasonality", "spotter_qs": ["What is the sales trend?", "Show me revenue by month", "When was our peak sales period?"]},
+            {"title": "Sales by Region", "viz_type": "COLUMN", "viz_question": "Dollar Sales by Region", "required": True, "insight": "Regional performance comparison reveals geographic variance", "spotter_qs": ["Which region has the highest sales?", "Compare regional performance", "Show me the top performing regions"]},
+            {"title": "ASP Trend", "viz_type": "KPI", "viz_question": "ASP weekly", "required": False, "insight": "Average selling price trend shows pricing health", "spotter_qs": ["Why did ASP change last month?", "Show me ASP by product", "Which products have the biggest discount?"]},
+            {"title": "Sales by Category", "viz_type": "COLUMN", "viz_question": "Dollar Sales by Category", "required": False, "insight": "Category mix reveals product performance drivers", "spotter_qs": ["Which category grew fastest?", "Compare top categories", "Show me category performance trend"]},
+        ],
+        "story_controls": {
+            "trend_style": "smooth",
+            "trend_noise_band_pct": 0.03,
+            "allowed_dimensions": ["Region", "Product", "Category", "Store", "Channel"],
+            "allowed_measures": ["Dollar Sales", "Unit Sales", "ASP"],
+        },
     },
     "Supply Chain": {
         "kpis": ["Avg Inventory", "OTIF", "Days on Hand", "Stockout Rate"],
@@ -69,6 +148,19 @@ FUNCTIONS = {
             "Show inventory levels by {dimension}",
             "Which suppliers have the longest lead times?",
         ],
+        "liveboard_questions": [
+            {"title": "Inventory Levels", "viz_type": "KPI", "viz_question": "Average inventory by month", "required": True, "insight": "Inventory trend over time", "spotter_qs": ["What is current inventory level?", "Show me inventory trend by month"]},
+            {"title": "Stockout Risk by Product", "viz_type": "COLUMN", "viz_question": "Stockout rate by product", "required": True, "insight": "Products at risk of going out of stock", "spotter_qs": ["Which products are at risk of stockout?", "Show stockout rate by category"]},
+            {"title": "OTIF Performance", "viz_type": "KPI", "viz_question": "OTIF rate by month", "required": False, "insight": "On-time in-full delivery performance", "spotter_qs": ["What is our OTIF rate?", "Which suppliers have the lowest OTIF?"]},
+            {"title": "Days on Hand by Supplier", "viz_type": "COLUMN", "viz_question": "Days on hand by supplier", "required": False, "insight": "Supplier inventory coverage comparison", "spotter_qs": ["Which supplier has the highest days on hand?", "Show me inventory coverage by supplier"]},
+        ],
+        "story_controls": {
+            "trend_style": "smooth",
+            "trend_noise_band_pct": 0.025,
+            "allowed_dimensions": ["Supplier", "Warehouse", "Region", "Product", "Category"],
+            "allowed_measures": ["Avg Inventory", "OTIF", "Days on Hand", "Stockout Rate"],
+            "outlier_budget": {"max_events": 2},
+        },
     },
     "Marketing": {
         "kpis": ["CTR", "Bounce Rate", "Fill Rate", "Approval Rate"],
@@ -85,6 +177,102 @@ FUNCTIONS = {
             "Show me the funnel for {campaign}",
             "Which channel has the highest CTR?",
         ],
+        "liveboard_questions": [
+            {"title": "Conversion Funnel", "viz_type": "COLUMN", "viz_question": "Conversion rate by funnel stage", "required": True, "insight": "Drop-off at each stage reveals where prospects are lost", "spotter_qs": ["Where is our biggest funnel drop-off?", "What is our application completion rate?"]},
+            {"title": "CTR by Channel", "viz_type": "COLUMN", "viz_question": "CTR by channel", "required": True, "insight": "Click-through rate comparison reveals best-performing channels", "spotter_qs": ["Which channel has the best CTR?", "Compare channel performance"]},
+            {"title": "Campaign ROI Trend", "viz_type": "KPI", "viz_question": "Campaign ROI by month", "required": False, "insight": "Return on marketing investment over time", "spotter_qs": ["Which campaigns have the best ROI?", "Show me ROI trend"]},
+            {"title": "Cost per Acquisition Trend", "viz_type": "LINE", "viz_question": "Cost per acquisition trend by month", "required": False, "insight": "CPA trend reveals acquisition efficiency", "spotter_qs": ["How is our CPA trending?", "Which channel has the lowest CPA?"]},
+        ],
+        "story_controls": {
+            "trend_style": "smooth",
+            "trend_noise_band_pct": 0.04,
+            "allowed_dimensions": ["Channel", "Campaign", "Segment", "Region"],
+            "allowed_measures": ["CTR", "Bounce Rate", "Fill Rate", "Approval Rate"],
+            "outlier_budget": {"max_events": 2},
+        },
+    },
+    "Finance": {
+        "kpis": [
+            "ARR",
+            "MRR",
+            "NRR",
+            "GRR",
+            "Net New ARR",
+            "Billings",
+            "Collections",
+            "Deferred Revenue",
+            "CAC",
+            "Gross Margin %",
+            "DSO",
+        ],
+        "kpi_definitions": {
+            "ARR": "Ending annual recurring revenue for the period",
+            "MRR": "Monthly recurring revenue",
+            "NRR": "(Starting ARR + Expansion - Contraction - Churn) ÷ Starting ARR",
+            "GRR": "(Starting ARR - Contraction - Churn) ÷ Starting ARR",
+            "Net New ARR": "New Logo ARR + Expansion ARR - Contraction ARR - Churned ARR",
+            "Billings": "Amount invoiced in the period",
+            "Collections": "Cash collected in the period",
+            "Deferred Revenue": "Billed but not yet recognized revenue balance",
+            "CAC": "Sales and marketing spend ÷ new customers acquired",
+            "Gross Margin %": "Gross Margin ÷ Revenue",
+            "DSO": "Average days sales outstanding",
+        },
+        "viz_types": [
+            "KPI_sparkline",
+            "arr_bridge",
+            "retention_trend",
+            "cash_flow",
+            "segment_comparison",
+            "by_region",
+        ],
+        "outlier_categories": [
+            "expansion_uplift",
+            "churn_spike",
+            "collections_lag",
+            "upsell_conversion",
+            "billing_prepay",
+        ],
+        "spotter_templates": [
+            "Why did ARR jump in Q4?",
+            "What is driving NRR by segment?",
+            "Show billings versus collections trend by month",
+            "Which region has the highest CAC?",
+        ],
+        "liveboard_questions": [
+            {"title": "ARR Trend", "viz_type": "KPI", "viz_question": "ARR by month", "required": True, "insight": "Annual recurring revenue growth trend", "spotter_qs": ["Why did ARR jump in Q4?", "Show me ARR trend by segment"]},
+            {"title": "ARR Bridge", "viz_type": "COLUMN", "viz_question": "New logo, expansion, contraction, and churn ARR by month", "required": True, "insight": "Waterfall breakdown of ARR movements", "spotter_qs": ["What is driving net new ARR?", "How much ARR did we lose to churn?"]},
+            {"title": "NRR by Segment", "viz_type": "COLUMN", "viz_question": "Net revenue retention by segment", "required": False, "insight": "Retention health across customer segments", "spotter_qs": ["What is driving NRR by segment?", "Which segment has the lowest retention?"]},
+            {"title": "CAC by Region", "viz_type": "COLUMN", "viz_question": "CAC by region", "required": False, "insight": "Customer acquisition cost comparison", "spotter_qs": ["Which region has the highest CAC?", "How is CAC trending?"]},
+        ],
+        "story_controls": {
+            "trend_style": "smooth",
+            "trend_noise_band_pct": 0.02,
+            "seasonal_strength": 0.04,
+            "allowed_dimensions": [
+                "Segment",
+                "Region",
+                "Product Family",
+                "Plan Tier",
+                "Contract Type",
+                "Customer Tier",
+            ],
+            "allowed_measures": [
+                "ARR",
+                "MRR",
+                "NRR",
+                "GRR",
+                "Net New ARR",
+                "Billings",
+                "Collections",
+                "Deferred Revenue",
+                "CAC",
+                "Gross Margin %",
+                "DSO",
+            ],
+            "outlier_budget": {"max_events": 2, "max_points_pct": 0.015},
+            "value_guardrails": {"max_mom_change_pct": 20},
+        },
     },
 }
 
@@ -100,6 +288,18 @@ MATRIX_OVERRIDES = {
         "add_viz": ["by_store", "by_category"],
         "target_persona": "VP Merchandising, Retail Sales Leader",
         "business_problem": "$1T lost annually to stockouts and overstock",
+        "liveboard_questions": [
+            {"title": "ASP Decline", "viz_type": "KPI", "viz_question": "average selling price by week", "required": True, "insight": "ASP dropped even though revenue is up — discounting too heavily", "spotter_qs": ["Why did ASP drop last month?", "Which products have the biggest discount?", "Show me ASP by region"]},
+            {"title": "Regional Variance", "viz_type": "COLUMN", "viz_question": "total sales revenue by region", "required": True, "insight": "West region outperforming by 40% — what are they doing differently?", "spotter_qs": ["Which region has the highest sales?", "Compare West to East performance", "What products are driving the top region?"]},
+            {"title": "Seasonal Trend", "viz_type": "LINE", "viz_question": "total sales revenue by category and month", "required": False, "insight": "Holiday surge 3x normal — were we prepared?", "spotter_qs": ["Show me sales trend for Q4", "When was our peak sales day?"]},
+            {"title": "Category Surge", "viz_type": "COLUMN", "viz_question": "total sales revenue by product category", "required": False, "insight": "Electronics up 60% YoY while Apparel flat", "spotter_qs": ["Which category grew fastest?", "Compare Electronics to Apparel"]},
+        ],
+        "story_controls": {
+            "seasonal_strength": 0.20,
+            "trend_noise_band_pct": 0.028,
+            "outlier_budget": {"max_events": 3, "max_points_pct": 0.025},
+            "allowed_dimensions": ["Store", "Category", "Region", "Product"],
+        },
     },
     ("Banking", "Marketing"): {
         "add_kpis": ["Application Fill Rate", "Cost per Acquisition"],
@@ -110,6 +310,18 @@ MATRIX_OVERRIDES = {
         "rename_kpis": {"CTR": "Click-through Rate"},
         "target_persona": "CMO, VP Digital Marketing",
         "business_problem": "High cost per acquisition, low funnel conversion",
+        "liveboard_questions": [
+            {"title": "Funnel Drop-off", "viz_type": "COLUMN", "viz_question": "Conversion rate by funnel stage", "required": True, "insight": "70% drop-off at application page — UX issue?", "spotter_qs": ["Where is our biggest funnel drop-off?", "What is our application completion rate?"]},
+            {"title": "Channel CTR", "viz_type": "COLUMN", "viz_question": "CTR by channel", "required": True, "insight": "Mobile CTR 2x desktop — shift budget?", "spotter_qs": ["Which channel has the best CTR?", "Compare mobile vs desktop performance"]},
+            {"title": "Cost per Acquisition Trend", "viz_type": "LINE", "viz_question": "Cost per acquisition by month", "required": False, "insight": "CPA trend over time reveals acquisition efficiency", "spotter_qs": ["How is CPA trending?", "Which channel has the lowest CPA?"]},
+            {"title": "Application Volume by Segment", "viz_type": "COLUMN", "viz_question": "Applications by customer segment", "required": False, "insight": "Which segments are converting on products?", "spotter_qs": ["Which segment submits the most applications?", "Compare approval rates by segment"]},
+        ],
+        "story_controls": {
+            "seasonal_strength": 0.07,
+            "trend_noise_band_pct": 0.03,
+            "outlier_budget": {"max_events": 2},
+            "allowed_dimensions": ["Channel", "Campaign", "Branch", "Segment"],
+        },
     },
     ("Software", "Sales"): {
         "add_kpis": ["ARR", "Net Revenue Retention", "Pipeline Coverage"],
@@ -120,7 +332,124 @@ MATRIX_OVERRIDES = {
         },
         "add_viz": ["by_segment", "by_rep"],
         "target_persona": "CRO, VP Sales",
+        "liveboard_questions": [
+            {"title": "ARR by Segment", "viz_type": "COLUMN", "viz_question": "ARR by segment", "required": True, "insight": "Revenue breakdown reveals which segments drive growth", "spotter_qs": ["Which segment has the highest ARR?", "Show ARR growth by segment"]},
+            {"title": "Pipeline Coverage", "viz_type": "KPI", "viz_question": "Pipeline coverage ratio by month", "required": True, "insight": "Pipeline vs quota ratio shows sales health", "spotter_qs": ["What is our pipeline coverage?", "Which rep has the lowest pipeline coverage?"]},
+            {"title": "Win Rate by Rep", "viz_type": "COLUMN", "viz_question": "Win rate by sales rep", "required": False, "insight": "Rep performance comparison reveals coaching opportunities", "spotter_qs": ["Which rep has the highest win rate?", "Compare win rates by region"]},
+            {"title": "Deal Velocity Trend", "viz_type": "LINE", "viz_question": "Average sales cycle length by month", "required": False, "insight": "Deal velocity trend shows if sales motion is tightening or slowing", "spotter_qs": ["How is deal cycle length trending?", "Which segment closes fastest?"]},
+        ],
+        "story_controls": {
+            "seasonal_strength": 0.05,
+            "trend_noise_band_pct": 0.025,
+            "outlier_budget": {"max_events": 2},
+            "allowed_dimensions": ["Segment", "Rep", "Region", "Product"],
+        },
     },
+    ("Software", "Finance"): {
+        "use_case_name": "SaaS Finance and Unit Economics",
+        "canonical_use_case": "SaaS Finance and Unit Economics",
+        "target_persona": "CFO, VP Finance, FP&A Leader",
+        "business_problem": (
+            "Recurring revenue growth can look healthy while retention, cash conversion, "
+            "and acquisition efficiency quietly deteriorate."
+        ),
+        "story_controls": {
+            "seasonal_strength": 0.03,
+            "trend_noise_band_pct": 0.015,
+            "outlier_budget": {"max_events": 2, "max_points_pct": 0.0125},
+            "value_guardrails": {"max_mom_change_pct": 20},
+            "allowed_dimensions": [
+                "Segment",
+                "Region",
+                "Product Family",
+                "Plan Tier",
+                "Contract Type",
+                "Customer Tier",
+                "Billing Cadence",
+            ],
+        },
+        "schema_contract": {
+            "mode": "saas_finance_gold",
+            "date_grain": "monthly",
+            "history_months": 24,
+            "required_dimensions": [
+                "DATES",
+                "CUSTOMERS",
+                "PRODUCTS",
+                "LOCATIONS",
+            ],
+            "required_facts": [
+                "SAAS_CUSTOMER_MONTHLY",
+                "SALES_MARKETING_SPEND_MONTHLY",
+            ],
+            "customer_month_fact": {
+                "table": "SAAS_CUSTOMER_MONTHLY",
+                "grain": "one row per customer per month",
+                "required_columns": [
+                    "MONTH_KEY",
+                    "CUSTOMER_KEY",
+                    "PRODUCT_KEY",
+                    "LOCATION_KEY",
+                    "SCENARIO",
+                    "STARTING_ARR_USD",
+                    "NEW_LOGO_ARR_USD",
+                    "EXPANSION_ARR_USD",
+                    "CONTRACTION_ARR_USD",
+                    "CHURNED_ARR_USD",
+                    "ENDING_ARR_USD",
+                    "MRR_USD",
+                    "BILLINGS_USD",
+                    "COLLECTIONS_USD",
+                    "DEFERRED_REVENUE_USD",
+                    "GROSS_MARGIN_USD",
+                    "GROSS_MARGIN_PCT",
+                    "SUPPORT_COST_USD",
+                    "CAC_ALLOCATED_USD",
+                    "DSO_DAYS",
+                    "SEAT_COUNT",
+                    "USAGE_UNITS",
+                    "CUSTOMER_ACQUIRED_FLAG",
+                    "CUSTOMER_CHURNED_FLAG",
+                ],
+            },
+            "spend_fact": {
+                "table": "SALES_MARKETING_SPEND_MONTHLY",
+                "grain": "one row per month per segment per region",
+                "required_columns": [
+                    "MONTH_KEY",
+                    "SEGMENT",
+                    "REGION",
+                    "SALES_SPEND_USD",
+                    "MARKETING_SPEND_USD",
+                    "TOTAL_S_AND_M_SPEND_USD",
+                    "NEW_CUSTOMERS_ACQUIRED",
+                    "NEW_LOGO_ARR_USD",
+                    "NET_NEW_ARR_USD",
+                    "NRR_PCT",
+                    "GRR_PCT",
+                    "CAC_USD",
+                ],
+            },
+            "identities": [
+                "ENDING_ARR_USD = STARTING_ARR_USD + NEW_LOGO_ARR_USD + EXPANSION_ARR_USD - CONTRACTION_ARR_USD - CHURNED_ARR_USD",
+                "MRR_USD = ENDING_ARR_USD / 12",
+                "TOTAL_S_AND_M_SPEND_USD = SALES_SPEND_USD + MARKETING_SPEND_USD",
+                "CAC_ALLOCATED_USD >= 0",
+                "COLLECTIONS_USD <= BILLINGS_USD + DEFERRED_REVENUE_USD",
+            ],
+        },
+    },
+}
+
+
+ROUTED_USE_CASES = {
+    "saas finance and unit economics": ("Software", "Finance"),
+    "saas finance": ("Software", "Finance"),
+    "software finance": ("Software", "Finance"),
+    "financial analytics": ("Software", "Finance"),
+    "finance analytics": ("Software", "Finance"),
+    "fp&a": ("Software", "Finance"),
+    "fpa": ("Software", "Finance"),
 }
 
 
@@ -142,6 +471,10 @@ def parse_use_case(user_input: str) -> tuple[str | None, str | None]:
         return (None, None)
     
     user_input_lower = user_input.strip().lower()
+
+    for phrase, routed in sorted(ROUTED_USE_CASES.items(), key=lambda item: len(item[0]), reverse=True):
+        if phrase in user_input_lower:
+            return routed
     
     # Try to find both vertical and function in the input
     found_vertical = None
@@ -211,6 +544,14 @@ def get_use_case_config(vertical: str, function: str) -> dict:
         "viz_types": f.get("viz_types", []).copy(),
         "outlier_categories": f.get("outlier_categories", []).copy(),
         "spotter_templates": f.get("spotter_templates", []).copy(),
+        "story_controls": _merge_story_controls(
+            v.get("story_controls", {}),
+            f.get("story_controls", {}),
+            override.get("story_controls", {}),
+        ),
+        "schema_contract": deepcopy(override.get("schema_contract", {})),
+        # Override takes precedence; else use function-level questions; else empty
+        "liveboard_questions": override.get("liveboard_questions") or f.get("liveboard_questions", []),
         
         # Flags
         "is_generic": False,
@@ -233,6 +574,22 @@ def get_use_case_config(vertical: str, function: str) -> dict:
         config["target_persona"] = override["target_persona"]
     if override.get("business_problem"):
         config["business_problem"] = override["business_problem"]
+    if override.get("use_case_name"):
+        config["use_case_name"] = override["use_case_name"]
+    if override.get("canonical_use_case"):
+        config["canonical_use_case"] = override["canonical_use_case"]
+
+    # Story guardrails derived from matrix controls
+    config["allowed_story_dimensions"] = (
+        config["story_controls"].get("allowed_dimensions")
+        or f.get("story_controls", {}).get("allowed_dimensions")
+        or config["entities"][:]
+    )
+    config["allowed_story_measures"] = (
+        config["story_controls"].get("allowed_measures")
+        or f.get("story_controls", {}).get("allowed_measures")
+        or config["kpis"][:]
+    )
     
     # Handle generic cases
     if not is_known_vertical and not is_known_function:
@@ -240,6 +597,13 @@ def get_use_case_config(vertical: str, function: str) -> dict:
         config["is_generic"] = True
         config["ai_should_determine"] = ["entities", "industry_terms", "kpis", "viz_types", "outliers"]
         config["prompt_user_for"] = ["key_metrics", "target_persona", "business_questions"]
+        config["story_controls"] = _deep_merge_dict(
+            config["story_controls"],
+            {
+                "outlier_budget": {"max_events": 1},
+                "trend_noise_band_pct": 0.03,
+            },
+        )
     elif not is_known_vertical:
         # Known function, unknown vertical
         config["is_generic"] = True
@@ -264,6 +628,8 @@ def get_use_case_config(vertical: str, function: str) -> dict:
         config["cost_impact"] = "Significant business impact through data-driven decisions"
     if "success_outcomes" not in config:
         config["success_outcomes"] = f"Improved {function.lower()} performance and faster insights"
+    if "recommended_companies" not in config:
+        config["recommended_companies"] = config["story_controls"].get("recommended_companies", [])
     
     return config
 

legitdata_project/README.md

@@ -0,0 +1,265 @@
+# LegitData
+
+> Generate realistic synthetic data for analytics warehouses - not dummy data.
+
+LegitData uses AI to generate contextually appropriate, realistic data for your dimension and fact tables. Instead of random strings and numbers from Faker, get real product names, actual company names, and coherent data that tells a story.
+
+## Features
+
+- **Smart Column Classification**: Automatically determines how to source data for each column
+  - `SEARCH_REAL`: Web search for real products, brands, companies
+  - `AI_GEN`: AI-generated contextual data (regions, categories, segments)
+  - `GENERIC`: Calculated values (dates, amounts, IDs)
+
+- **Company-Aware Context**: Scrapes your company URL to understand your business and generate appropriate data
+
+- **FK Integrity**: Maintains referential integrity with realistic distributions (Pareto for fact tables)
+
+- **Size Presets**: Small, Medium, Large, XL presets for different use cases
+
+- **Caching**: Caches search results and generated values for consistency and speed
+
+## Installation
+
+```bash
+# Basic installation
+pip install legitdata
+
+# With Snowflake support
+pip install legitdata[snowflake]
+
+# With AI support (recommended)
+pip install legitdata[ai]
+
+# Everything
+pip install legitdata[all]
+```
+
+## Quick Start
+
+### Python Library
+
+```python
+from legitdata import LegitGenerator
+from anthropic import Anthropic
+
+# Initialize with your context
+gen = LegitGenerator(
+    url="https://amazon.com",
+    use_case="Retail Analytics",
+    connection_string="snowflake://user:pass@account/database/schema",
+    anthropic_client=Anthropic()  # Uses ANTHROPIC_API_KEY env var
+)
+
+# Load your schema
+gen.load_ddl("schema.sql")
+
+# Generate and insert data
+results = gen.generate(size="medium")
+# {'CUSTOMERS': 100, 'PRODUCTS': 100, 'SELLERS': 100, 'SALES_TRANSACTIONS': 1000}
+```
+
+### CLI
+
+```bash
+# Generate data
+legitdata generate \
+  --ddl schema.sql \
+  --url https://amazon.com \
+  --use-case "Retail Analytics" \
+  --connection "snowflake://user:pass@account/db/schema" \
+  --size medium
+
+# Preview without writing
+legitdata preview \
+  --ddl schema.sql \
+  --url https://amazon.com \
+  --use-case "Retail Analytics" \
+  --rows 5
+
+# Manage cache
+legitdata cache --stats
+legitdata cache --clear
+```
+
+## Size Presets
+
+| Size | Fact Rows | Dim Rows | Use Case |
+|------|-----------|----------|----------|
+| small | 100 | 20 | Quick testing |
+| medium | 1,000 | 100 | Demo/dev |
+| large | 10,000 | 500 | Realistic workload |
+| xl | 100,000 | 500 | Performance testing |
+
+Or specify exact counts:
+
+```python
+gen.generate(row_counts={
+    "CUSTOMERS": 200,
+    "PRODUCTS": 50,
+    "SALES_TRANSACTIONS": 5000
+})
+```
+
+## Use Cases
+
+Built-in use cases that help guide data generation:
+
+- Sales Analytics
+- Supply Chain Analytics
+- Customer Analytics
+- Financial Analytics
+- Marketing Analytics
+- Retail Analytics
+
+Or use any custom use case:
+
+```python
+gen = LegitGenerator(
+    url="https://example.com",
+    use_case="Call Center Sentiment Analysis",  # Custom!
+    ...
+)
+```
+
+## How It Works
+
+### 1. Context Building
+LegitData scrapes your company URL to understand:
+- Company name and industry
+- Products/services offered
+- Target customers
+- Geographic focus
+
+### 2. Column Classification
+Each column is analyzed and classified:
+
+| Column Type | Classification | Example |
+|-------------|----------------|---------|
+| ProductName | SEARCH_REAL | Web search for real products |
+| Brand | SEARCH_REAL | Web search for real brands |
+| Region | AI_GEN | AI generates realistic regions |
+| CustomerSegment | AI_GEN | AI generates "Enterprise", "SMB", etc. |
+| OrderDate | GENERIC | Random date in range |
+| Revenue | GENERIC | Calculated decimal |
+| CustomerID | GENERIC | FK reference |
+
+### 3. Data Generation
+- **SEARCH_REAL**: Web search → AI extracts real values
+- **AI_GEN**: AI generates contextually appropriate values
+- **GENERIC**: Faker/calculated values
+
+### 4. FK Relationships
+Fact tables use Pareto distribution (80/20) for realistic patterns:
+- Top 20% of customers generate 80% of orders
+- Popular products appear more frequently
+
+## Connection Strings
+
+### Snowflake
+```
+snowflake://user:password@account/database/schema
+snowflake://user:password@account/database/schema?warehouse=WH&role=ROLE
+```
+
+## Caching
+
+LegitData caches:
+- Company context (extracted from URL)
+- Web search results
+- Column classifications
+- Generated values
+
+Cache location: `.legitdata_cache/`
+
+```python
+# Disable caching
+gen = LegitGenerator(..., cache_enabled=False)
+
+# Clear cache
+from legitdata.cache import FileCache
+cache = FileCache()
+cache.clear()
+```
+
+## Preview Mode
+
+Test generation without writing to database:
+
+```python
+# Preview 5 rows per table
+preview = gen.preview(num_rows=5)
+
+for table, rows in preview.items():
+    print(f"{table}: {len(rows)} rows")
+    print(rows[0])  # First row
+```
+
+Or use dry run:
+
+```python
+gen = LegitGenerator(..., dry_run=True)
+gen.generate()  # Prints operations but doesn't write
+```
+
+## API Reference
+
+### LegitGenerator
+
+```python
+LegitGenerator(
+    url: str,                    # Company website URL
+    use_case: str,               # Analytics use case
+    connection_string: str,      # Database connection
+    anthropic_client=None,       # Anthropic client for AI
+    web_search_fn=None,          # Custom web search function
+    cache_enabled=True,          # Enable caching
+    cache_dir=".legitdata_cache", # Cache directory
+    dry_run=False                # Don't write to DB
+)
+```
+
+### Methods
+
+- `load_ddl(ddl_or_path)` - Load DDL from string or file
+- `generate(size, row_counts, truncate_first)` - Generate and insert data
+- `preview(table_name, num_rows)` - Preview without inserting
+
+## Extending
+
+### Custom Web Search
+
+```python
+def my_search(query: str) -> list[dict]:
+    # Your search implementation
+    return [{"title": "...", "snippet": "...", "url": "..."}]
+
+gen = LegitGenerator(
+    ...,
+    web_search_fn=my_search
+)
+```
+
+### Custom Database Writer
+
+```python
+from legitdata.writers import BaseWriter
+
+class MyWriter(BaseWriter):
+    def connect(self): ...
+    def disconnect(self): ...
+    def insert_rows(self, table, columns, rows, batch_size): ...
+    def truncate_table(self, table): ...
+    def table_exists(self, table): ...
+    def get_table_columns(self, table): ...
+```
+
+## Requirements
+
+- Python 3.10+
+- Snowflake Connector (for Snowflake support)
+- Anthropic SDK (for AI features)
+
+## License
+
+MIT

legitdata_project/legitdata/__init__.py

@@ -18,6 +18,7 @@ from .config import SIZE_PRESETS, USE_CASES, GenerationConfig
 from .ddl import parse_ddl, parse_ddl_file, Schema, Table, Column, ColumnClassification
 from .analyzer import CompanyContext
 from .writers import SnowflakeWriter, DryRunWriter
+from .storyspec import StorySpec, build_story_spec, build_story_bundle
 
 __version__ = "0.1.0"
 
@@ -35,4 +36,7 @@ __all__ = [
     'CompanyContext',
     'SnowflakeWriter',
     'DryRunWriter',
+    'StorySpec',
+    'build_story_spec',
+    'build_story_bundle',
 ]

legitdata_project/legitdata/analyzer/column_classifier.py

@@ -2,11 +2,20 @@
 
 import json
 import re
-from datetime import datetime
+from datetime import datetime, timedelta
 from typing import Optional
 from ..ddl.models import Schema, Table, Column, ColumnClassification
 from ..config import COLUMN_HINTS
 from .context_builder import CompanyContext
+from ..domain import infer_semantic_type, is_business_categorical
+
+
+def _years_ago(dt: datetime, years: int) -> datetime:
+    """Return datetime shifted back by full years (leap-safe)."""
+    try:
+        return dt.replace(year=dt.year - years)
+    except ValueError:
+        return dt.replace(month=2, day=28, year=dt.year - years)
 
 
 class ColumnClassifier:
@@ -153,6 +162,7 @@ Return ONLY valid JSON, no other text."""
         """Classify a single column using heuristics."""
         col_name = column.name.lower()
         col_upper = column.name.upper()
+        semantic = infer_semantic_type(column.name, column.data_type, table.name, use_case)
         
         # Primary keys and identity columns
         if column.is_primary_key or column.is_identity:
@@ -161,6 +171,11 @@ Return ONLY valid JSON, no other text."""
         # Foreign keys
         if col_upper in fk_columns:
             return ColumnClassification.GENERIC, "foreign_key"
+
+        # Quality-first: keep business categorical columns contextual.
+        if is_business_categorical(semantic):
+            prompt = self._generate_ai_prompt(col_name, table.name, context, use_case)
+            return ColumnClassification.AI_GEN, prompt
         
         # Check patterns for SEARCH_REAL
         for pattern in COLUMN_HINTS["search_real_patterns"]:
@@ -220,7 +235,13 @@ Return ONLY valid JSON, no other text."""
         col_name = column.name.lower()
         data_type = column.data_type.upper()
         
-        # BOOLEAN COLUMNS - Check FIRST before phone (is_mobile should be boolean, not phone!)
+        # BOOLEAN COLUMNS - Check FIRST before phone/contact heuristics
+        # so names like MobileTransaction BOOLEAN don't get treated as
+        # phone numbers just because they contain "mobile".
+        if 'BOOLEAN' in data_type or data_type in ('BOOL', 'BIT'):
+            return "boolean:0.8"
+
+        # BOOLEAN NAME PATTERNS - still keep these ahead of contact fields.
         if col_name.startswith('is_') or col_name.startswith('has_') or col_name.endswith('_flag'):
             return "boolean:0.5"
         
@@ -231,14 +252,27 @@ Return ONLY valid JSON, no other text."""
         # Phone columns (but NOT is_mobile - that's boolean, checked above)
         if 'phone' in col_name or 'mobile' in col_name or 'tel' in col_name:
             return "phone"
+
+        # Identifier-like short codes should not be treated as monetary values.
+        if (
+            'tax_id' in col_name
+            or 'taxid' in col_name
+            or 'last4' in col_name
+            or col_name.endswith('_id_last4')
+        ):
+            if any(t in data_type for t in ('CHAR', 'TEXT', 'STRING', 'VARCHAR')):
+                return "random_string:4"
+            return "random_int:1000,9999"
         
         # Name columns
         if col_name in ('first_name', 'firstname', 'fname', 'given_name'):
             return "first_name"
         if col_name in ('last_name', 'lastname', 'lname', 'surname'):
             return "last_name"
-        if col_name in ('full_name', 'fullname', 'name', 'customer_name', 'user_name'):
+        if col_name in ('full_name', 'fullname', 'name', 'user_name'):
             return "name"
+        if col_name in ('customer_name', 'account_name', 'client_name', 'company_name', 'organization_name'):
+            return "company"
         
         # Address columns
         if 'address' in col_name and 'email' not in col_name:
@@ -252,9 +286,46 @@ Return ONLY valid JSON, no other text."""
         if 'zip' in col_name or 'postal' in col_name:
             return "zipcode"
         
-        # Date columns - use current date for end range
-        today = datetime.now().strftime('%Y-%m-%d')
-        if 'date' in col_name or data_type in ('DATE', 'TIMESTAMP', 'DATETIME'):
+        # Birth date columns - must be evaluated before generic date checks
+        # to avoid impossible ages from generic recent-date generation.
+        today_dt = datetime.now()
+        today = today_dt.strftime('%Y-%m-%d')
+        is_birth_col = (
+            'birthdate' in col_name or
+            'birth_date' in col_name or
+            'date_of_birth' in col_name or
+            col_name == 'dob' or
+            col_name.endswith('_dob')
+        )
+        if is_birth_col:
+            normalized_today = today_dt.replace(hour=0, minute=0, second=0, microsecond=0)
+            oldest = _years_ago(normalized_today, 95).strftime('%Y-%m-%d')
+            youngest = _years_ago(normalized_today, 18).strftime('%Y-%m-%d')
+            return f"date_between:{oldest},{youngest}"
+        
+        # Date columns - use current date for end range.
+        # Guard against financial names like *_AMOUNT_TO_DATE on numeric fields.
+        financial_to_date_name = (
+            ('to_date' in col_name or col_name.endswith('todate'))
+            and any(
+                token in col_name
+                for token in (
+                    'amount',
+                    'balance',
+                    'cost',
+                    'price',
+                    'revenue',
+                    'sales',
+                    'fee',
+                    'tax',
+                    'total',
+                    'recovery',
+                    'payment',
+                )
+            )
+        )
+        is_name_date = ('date' in col_name) and not financial_to_date_name
+        if is_name_date or data_type in ('DATE', 'TIMESTAMP', 'DATETIME'):
             if 'created' in col_name or 'registration' in col_name:
                 return f"date_between:2020-01-01,{today}"
             elif 'launch' in col_name or 'open' in col_name:
@@ -269,8 +340,16 @@ Return ONLY valid JSON, no other text."""
         # Numeric columns
         if 'quantity' in col_name:
             return "random_int:1,100"
-        elif 'revenue' in col_name or 'amount' in col_name:
+        elif 'discount' in col_name:
+            return "random_decimal:0.00,200.00"
+        elif 'shipping' in col_name:
+            return "random_decimal:2.00,150.00"
+        elif 'tax' in col_name:
+            return "random_decimal:1.00,250.00"
+        elif 'revenue' in col_name:
             return "random_decimal:10.00,10000.00"
+        elif 'amount' in col_name:
+            return "random_decimal:5.00,5000.00"
         elif 'cost' in col_name or 'fee' in col_name or 'price' in col_name:
             return "random_decimal:1.00,500.00"
         elif 'rating' in col_name:
@@ -381,15 +460,43 @@ Return ONLY valid JSON, no other text."""
         These are columns where Faker produces better results than AI generation.
         """
         col_lower = col_name.lower()
+        semantic = infer_semantic_type(col_name, data_type)
+
+        # Never force business categorical columns to generic;
+        # these need domain-aware contextual generation.
+        if is_business_categorical(semantic):
+            return None
         
         # DEBUG: Log all checks
         print(f"  [FORCE_CHECK] Column: '{col_name}' (lower: '{col_lower}')")
         
-        # BOOLEAN COLUMNS - Check FIRST before phone (is_mobile should be boolean, not phone)
+        # BOOLEAN COLUMNS - Check FIRST before phone/contact heuristics so
+        # boolean names containing "mobile" don't become phone numbers.
+        if data_type:
+            dt_upper = data_type.upper()
+            if 'BOOLEAN' in dt_upper or dt_upper in ('BOOL', 'BIT'):
+                print(f"    → FORCED to 'boolean:0.5' (boolean data type)")
+                return "boolean:0.5"
+
+        # BOOLEAN NAME PATTERNS
         if col_lower.startswith('is_') or col_lower.startswith('has_') or col_lower.endswith('_flag'):
             print(f"    → FORCED to 'boolean:0.5' (boolean pattern)")
             return "boolean:0.5"
         
+        # Birth date columns must always produce realistic adult ages.
+        is_birth_col = (
+            'birthdate' in col_lower or
+            'birth_date' in col_lower or
+            'date_of_birth' in col_lower or
+            col_lower == 'dob' or
+            col_lower.endswith('_dob')
+        )
+        if is_birth_col:
+            today_dt = datetime.now()
+            oldest = _years_ago(today_dt, 95).strftime('%Y-%m-%d')
+            youngest = _years_ago(today_dt, 18).strftime('%Y-%m-%d')
+            return f"date_between:{oldest},{youngest}"
+        
         # Check if data type is numeric
         is_numeric = False
         if data_type:
@@ -439,20 +546,27 @@ Return ONLY valid JSON, no other text."""
             return "last_name"
         
         # Full name - any column that's clearly a person's name
-        if col_lower in ('full_name', 'fullname', 'name', 'customer_name', 'user_name', 'username'):
+        if col_lower in ('full_name', 'fullname', 'name', 'user_name', 'username'):
             return "name"
+        if col_lower in ('customer_name', 'account_name', 'client_name', 'company_name', 'organization_name'):
+            return "company"
         
         # Person name columns that don't match exact list above
         # rep_name, manager_name, agent_name, employee_name, contact_name, etc.
         if col_lower.endswith('_name') or col_lower.endswith('name'):
             # Exclude non-person names (these are handled elsewhere as SEARCH_REAL or AI_GEN)
             non_person = ('product_name', 'productname', 'company_name', 'companyname',
+                         'account_name', 'accountname', 'customer_name', 'customername',
+                         'client_name', 'clientname', 'organization_name', 'organizationname',
                          'brand_name', 'brandname', 'store_name', 'storename',
                          'warehouse_name', 'warehousename', 'center_name', 'centername',
                          'campaign_name', 'campaignname', 'table_name', 'tablename',
                          'column_name', 'columnname', 'schema_name', 'schemaname',
                          'file_name', 'filename', 'host_name', 'hostname',
-                         'category_name', 'categoryname', 'holiday_name', 'holidayname')
+                         'category_name', 'categoryname', 'holiday_name', 'holidayname',
+                         'branch_name', 'branchname', 'department_name', 'departmentname',
+                         'region_name', 'regionname', 'city_name', 'cityname',
+                         'month_name', 'monthname', 'quarter_name', 'quartername', 'day_name', 'dayname')
             if col_lower not in non_person:
                 print(f"    → FORCED to 'name' (person name pattern: {col_lower})")
                 return "name"
@@ -481,7 +595,11 @@ Return ONLY valid JSON, no other text."""
             return "country"
         
         # URL/Website
-        if 'url' in col_lower or 'website' in col_lower:
+        # Avoid false positives like "hourly" containing "url".
+        if (
+            re.search(r'(^|[_\-])url([_\-]|$)', col_lower)
+            or 'website' in col_lower
+        ):
             return "url"
         
         return None

legitdata_project/legitdata/ddl/parser.py

@@ -63,6 +63,9 @@ class DDLParser:
                     table.columns.append(column)
                     if column.is_primary_key:
                         table.primary_key = column.name
+                    inline_fk = self._parse_inline_foreign_key(part, column.name)
+                    if inline_fk:
+                        table.foreign_keys.append(inline_fk)
         
         return table
     
@@ -116,14 +119,30 @@ class DDLParser:
             table.foreign_keys.append(fk)
             return
         
-        # Primary key constraint: PRIMARY KEY (col)
-        pk_match = re.search(r'PRIMARY\s+KEY\s*\((\w+)\)', part, re.IGNORECASE)
+        # Primary key constraint: PRIMARY KEY (col) or PRIMARY KEY (col1, col2, ...)
+        pk_match = re.search(r'PRIMARY\s+KEY\s*\((\w+)', part, re.IGNORECASE)
         if pk_match:
             table.primary_key = pk_match.group(1)
             # Mark the column as PK
             col = table.get_column(pk_match.group(1))
             if col:
                 col.is_primary_key = True
+
+    def _parse_inline_foreign_key(self, part: str, column_name: str) -> Optional[ForeignKey]:
+        """Parse inline FK syntax like `col INT REFERENCES parent(id)`."""
+        fk_match = re.search(
+            r'\bREFERENCES\s+(\w+)\s*\((\w+)\)',
+            part,
+            re.IGNORECASE
+        )
+        if not fk_match:
+            return None
+
+        return ForeignKey(
+            column_name=column_name,
+            references_table=fk_match.group(1),
+            references_column=fk_match.group(2)
+        )
     
     def _parse_column(self, part: str) -> Optional[Column]:
         """Parse a column definition."""

legitdata_project/legitdata/domain/__init__.py

@@ -0,0 +1,22 @@
+"""Domain-aware semantic typing and value packs."""
+
+from .semantic_types import (
+    SemanticType,
+    BUSINESS_CATEGORICAL_TYPES,
+    NUMERIC_TYPES,
+    infer_semantic_type,
+    is_business_categorical,
+)
+from .domain_packs import get_domain_values, infer_vertical_key, map_state_to_region
+
+__all__ = [
+    "SemanticType",
+    "BUSINESS_CATEGORICAL_TYPES",
+    "NUMERIC_TYPES",
+    "infer_semantic_type",
+    "is_business_categorical",
+    "get_domain_values",
+    "infer_vertical_key",
+    "map_state_to_region",
+]
+

legitdata_project/legitdata/domain/domain_packs.py

@@ -0,0 +1,543 @@
+"""Curated domain value packs for quality-first data generation."""
+
+from __future__ import annotations
+
+from .semantic_types import SemanticType
+
+
+BASE_DOMAIN_VALUES = {
+    SemanticType.REGION: [
+        "Northeast",
+        "Southeast",
+        "Midwest",
+        "Southwest",
+        "West",
+    ],
+    SemanticType.CATEGORY: [
+        "Electronics",
+        "Apparel",
+        "Home Goods",
+        "Sports & Outdoors",
+        "Health & Beauty",
+        "Office Supplies",
+        "Food & Grocery",
+    ],
+    SemanticType.SECTOR_NAME: [
+        "Technology",
+        "Healthcare",
+        "Consumer Discretionary",
+        "Consumer Staples",
+        "Industrials",
+        "Financial Services",
+        "Business Services",
+    ],
+    SemanticType.SECTOR_CATEGORY: ["Growth", "Defensive", "Cyclical", "Core"],
+    SemanticType.SEGMENT: ["Enterprise", "Mid-Market", "SMB", "Consumer", "Government"],
+    SemanticType.TIER: ["Platinum", "Gold", "Silver", "Bronze"],
+    SemanticType.STATUS: ["Active", "Pending", "Completed", "Cancelled", "On Hold"],
+    SemanticType.FUND_STRATEGY: [
+        "Buyout",
+        "Growth Equity",
+        "Venture Capital",
+        "Private Credit",
+        "Infrastructure",
+        "Secondaries",
+    ],
+    SemanticType.INVESTOR_TYPE: [
+        "Pension Fund",
+        "Endowment",
+        "Family Office",
+        "Sovereign Wealth",
+        "Insurance",
+    ],
+    SemanticType.INVESTMENT_STAGE: ["Active", "Realized", "Written Off"],
+    SemanticType.COVENANT_STATUS: ["Compliant", "Waived", "Breached"],
+    SemanticType.DEBT_PERFORMANCE_STATUS: ["Performing", "Watch List", "Non-Performing"],
+    SemanticType.CHANNEL: ["Online", "In-Store", "Mobile App", "Partner", "Marketplace"],
+    SemanticType.COUNTRY: ["United States", "Canada", "United Kingdom"],
+    SemanticType.STATE: [
+        "California",
+        "Texas",
+        "New York",
+        "Florida",
+        "Illinois",
+        "Washington",
+        "Georgia",
+        "Massachusetts",
+    ],
+    SemanticType.CITY: [
+        "Los Angeles",
+        "San Francisco",
+        "Dallas",
+        "Houston",
+        "New York City",
+        "Chicago",
+        "Seattle",
+        "Atlanta",
+        "Boston",
+        "Miami",
+    ],
+    SemanticType.POSTAL_CODE: [
+        "90001",
+        "94102",
+        "75201",
+        "77001",
+        "10001",
+        "60601",
+        "98101",
+        "30301",
+        "02101",
+        "33101",
+    ],
+    SemanticType.SEASON: ["Winter", "Spring", "Summer", "Fall"],
+    SemanticType.HOLIDAY_NAME: [
+        "New Year's Day",
+        "Memorial Day",
+        "Independence Day",
+        "Labor Day",
+        "Thanksgiving",
+        "Christmas Day",
+        "None",
+    ],
+    SemanticType.EVENT_NAME: [
+        "Regular Trading Day",
+        "Back to School",
+        "Holiday Promotions",
+        "Black Friday",
+        "Cyber Monday",
+        "Year End Campaign",
+    ],
+    SemanticType.PRODUCT_NAME: [
+        "Wireless Earbuds Pro",
+        "4K Smart TV 55in",
+        "Running Shoes Elite",
+        "Organic Cotton T-Shirt",
+        "Stainless Water Bottle 32oz",
+        "Laptop Backpack 15in",
+        "Bluetooth Soundbar",
+        "Air Fryer 6 Quart",
+        "Portable Charger 20000mAh",
+        "Smart Home Camera",
+    ],
+    SemanticType.BRAND_NAME: [
+        "Apex",
+        "Northline",
+        "EverPeak",
+        "BlueHarbor",
+        "Lumen",
+        "SummitCo",
+        "UrbanLeaf",
+        "PrimeWorks",
+    ],
+    SemanticType.BRANCH_NAME: [
+        "Downtown Branch",
+        "Midtown Branch",
+        "Northside Branch",
+        "Southside Branch",
+        "Riverside Branch",
+        "West End Branch",
+        "Lakeside Branch",
+        "Hillcrest Branch",
+    ],
+    SemanticType.ORG_NAME: [
+        "Apex Advisory Group",
+        "Northline Holdings",
+        "BlueHarbor Partners",
+        "PrimeWorks Consulting",
+        "Summit Operations Group",
+        "UrbanLeaf Services",
+        "Lumen Strategy Co",
+        "EverPeak Solutions",
+    ],
+    SemanticType.PERSON_NAME: [
+        "Olivia Carter",
+        "Liam Brooks",
+        "Ava Morgan",
+        "Noah Bennett",
+        "Emma Collins",
+        "Ethan Cooper",
+        "Sophia Reed",
+        "Mason Turner",
+        "Isabella Hayes",
+        "Lucas Ward",
+    ],
+    SemanticType.DEPARTMENT_NAME: [
+        "Operations",
+        "Finance",
+        "Legal",
+        "Sales",
+        "Marketing",
+        "Customer Success",
+        "Product",
+        "Engineering",
+    ],
+    SemanticType.COLOR_DESCRIPTION: [
+        "Black",
+        "White",
+        "Navy Blue",
+        "Charcoal Gray",
+        "Forest Green",
+        "Red",
+        "Beige",
+    ],
+    SemanticType.PACKAGING_SIZE: [
+        "Single Pack",
+        "2-Pack",
+        "4-Pack",
+        "8-Pack",
+        "16 oz",
+        "32 oz",
+        "64 oz",
+    ],
+}
+
+
+VERTICAL_OVERRIDES = {
+    "healthcare": {
+        SemanticType.CATEGORY: [
+            "Primary Care",
+            "Cardiology",
+            "Oncology",
+            "Orthopedics",
+            "Neurology",
+            "Emergency Services",
+            "Pediatrics",
+        ],
+        SemanticType.SEGMENT: ["Inpatient", "Outpatient", "Emergency", "Telehealth"],
+    },
+    "banking": {
+        SemanticType.CATEGORY: [
+            "Checking",
+            "Savings",
+            "Credit Card",
+            "Mortgage",
+            "Auto Loan",
+            "Investment",
+        ],
+        SemanticType.SEGMENT: ["Consumer", "Small Business", "Commercial", "Private Banking"],
+        SemanticType.CHANNEL: ["Branch", "Online Banking", "Mobile Banking", "Call Center", "ATM"],
+        SemanticType.BRANCH_NAME: [
+            "Main Street Branch",
+            "Riverside Branch",
+            "Oak Grove Branch",
+            "Lakeside Branch",
+            "Midtown Branch",
+            "Heritage Branch",
+        ],
+        SemanticType.DEPARTMENT_NAME: [
+            "Retail Banking",
+            "Consumer Lending",
+            "Mortgage Services",
+            "Wealth Management",
+            "Commercial Banking",
+            "Collections",
+        ],
+    },
+    "retail": {
+        SemanticType.CATEGORY: [
+            "Electronics",
+            "Apparel",
+            "Home & Kitchen",
+            "Beauty",
+            "Sports & Outdoors",
+            "Toys",
+            "Grocery",
+        ],
+        SemanticType.CHANNEL: ["E-commerce", "Store", "Marketplace", "Mobile App"],
+        SemanticType.BRAND_NAME: ["Nike", "Adidas", "Apple", "Samsung", "Sony", "Levi's", "KitchenAid", "Dyson"],
+        SemanticType.DEPARTMENT_NAME: ["Footwear", "Apparel", "Consumer Electronics", "Home Appliances", "Beauty", "Outdoor"],
+    },
+    "sportswear": {
+        SemanticType.CATEGORY: [
+            "Footwear",
+            "Apparel",
+            "Running",
+            "Training",
+            "Basketball",
+            "Soccer",
+            "Lifestyle",
+        ],
+        SemanticType.SEGMENT: [
+            "Performance Athlete",
+            "Everyday Active",
+            "Team Sports",
+            "Youth",
+            "Women",
+            "Men",
+        ],
+        SemanticType.CHANNEL: [
+            "Nike App",
+            "SNKRS",
+            "Nike.com",
+            "Brand Store",
+            "Outlet",
+            "Wholesale Partner",
+        ],
+        SemanticType.BRAND_NAME: [
+            "Nike",
+            "Jordan",
+            "Converse",
+            "Nike SB",
+            "ACG",
+            "Nike Pro",
+        ],
+        SemanticType.PRODUCT_NAME: [
+            "Air Zoom Pegasus",
+            "Air Force 1",
+            "Dunk Low",
+            "Metcon Trainer",
+            "Mercurial Vapor",
+            "Phantom GX",
+            "Dri-FIT Tee",
+            "Tech Fleece Hoodie",
+        ],
+        SemanticType.DEPARTMENT_NAME: [
+            "Running",
+            "Basketball",
+            "Training",
+            "Sportswear",
+            "Lifestyle",
+            "Accessories",
+        ],
+        SemanticType.BRANCH_NAME: [
+            "5th Ave Flagship",
+            "SoHo Experience Store",
+            "Melrose Brand House",
+            "Chicago Magnificent Mile",
+            "Miami Beach Store",
+            "Seattle Downtown Store",
+        ],
+    },
+    "pharma": {
+        SemanticType.CATEGORY: [
+            "Cardiometabolic",
+            "Oncology",
+            "Women's Health",
+            "Immunology",
+            "Primary Care",
+            "Rare Disease",
+            "Vaccine",
+        ],
+        SemanticType.SEGMENT: [
+            "High Prescriber",
+            "New Writer",
+            "Existing Writer",
+            "Hospital Network",
+            "Community Practice",
+            "Specialty Clinic",
+        ],
+        SemanticType.CHANNEL: [
+            "Field Sales",
+            "Digital HCP",
+            "Medical Affairs",
+            "Patient Support",
+            "Specialty Pharmacy",
+            "Wholesaler",
+        ],
+        SemanticType.BRAND_NAME: [
+            "Bayer",
+            "Xarelto",
+            "Eylea",
+            "Mirena",
+            "Yaz",
+            "Nexavar",
+        ],
+        SemanticType.PRODUCT_NAME: [
+            "Rivaroxaban 20mg",
+            "Aflibercept 2mg",
+            "Levonorgestrel IUS",
+            "Drospirenone Ethinyl Estradiol",
+            "Sorafenib 200mg",
+            "Aspirin 81mg",
+            "Canagliflozin 100mg",
+            "Rosuvastatin 20mg",
+        ],
+        SemanticType.DEPARTMENT_NAME: [
+            "Cardiology Franchise",
+            "Oncology Franchise",
+            "Women's Health Franchise",
+            "Medical Affairs",
+            "Market Access",
+            "Patient Services",
+        ],
+    },
+    "supply chain": {
+        SemanticType.CATEGORY: [
+            "Raw Materials",
+            "Finished Goods",
+            "Packaging",
+            "MRO",
+            "Transportation",
+        ],
+        SemanticType.STATUS: ["Planned", "In Transit", "Received", "Delayed", "Backordered"],
+        SemanticType.CHANNEL: ["Direct", "Distributor", "3PL", "Wholesale"],
+    },
+    "private_equity": {
+        SemanticType.SECTOR_NAME: [
+            "Industrials",
+            "Healthcare",
+            "Technology",
+            "Consumer",
+            "Financial Services",
+        ],
+        SemanticType.SECTOR_CATEGORY: ["Cyclical", "Defensive", "Growth"],
+        SemanticType.FUND_STRATEGY: ["Buyout", "Growth", "Venture"],
+        SemanticType.INVESTOR_TYPE: [
+            "Pension Fund",
+            "Endowment",
+            "Family Office",
+            "Sovereign Wealth",
+            "Insurance",
+        ],
+        SemanticType.INVESTMENT_STAGE: ["Active", "Realized", "Written Off"],
+        SemanticType.COVENANT_STATUS: ["Compliant", "Waived", "Breached"],
+        SemanticType.DEBT_PERFORMANCE_STATUS: ["Performing", "Watch List", "Non-Performing"],
+    },
+    "legal": {
+        SemanticType.CATEGORY: [
+            "Commercial",
+            "Litigation",
+            "Intellectual Property",
+            "Employment",
+            "Corporate Governance",
+            "Regulatory",
+            "Privacy",
+            "Compliance",
+        ],
+        SemanticType.DEPARTMENT_NAME: [
+            "Commercial Legal",
+            "Intellectual Property",
+            "Employment Law",
+            "Corporate Legal",
+            "Regulatory Affairs",
+            "Privacy & Compliance",
+            "Litigation",
+        ],
+        SemanticType.ORG_NAME: [
+            "Cooley LLP",
+            "Baker McKenzie",
+            "McDermott Will & Emery",
+            "Young Basile",
+            "Newman Du Wors",
+            "Morgan Lewis",
+            "Latham & Watkins",
+            "Wilson Sonsini",
+        ],
+    },
+    "saas": {
+        SemanticType.CATEGORY: [
+            "Core Platform",
+            "AI Analyst",
+            "Analytics",
+            "Security",
+            "Observability",
+            "Integrations",
+            "Embedded Analytics",
+            "Platform Add-ons",
+            "Support",
+        ],
+        SemanticType.SEGMENT: [
+            "Enterprise",
+            "Mid-Market",
+            "SMB",
+        ],
+        SemanticType.REGION: [
+            "North America",
+            "EMEA",
+            "APAC",
+        ],
+        SemanticType.CHANNEL: [
+            "Direct Sales",
+            "Partner",
+            "Digital Self-Serve",
+            "Expansion",
+        ],
+        SemanticType.DEPARTMENT_NAME: [
+            "Finance",
+            "FP&A",
+            "Revenue Operations",
+            "Customer Success",
+            "Sales",
+            "Product",
+            "Engineering",
+        ],
+        SemanticType.ORG_NAME: [
+            "Enterprise Customers",
+            "Mid-Market Customers",
+            "SMB Customers",
+            "Strategic Accounts",
+            "Channel Partners",
+            "Global Accounts",
+        ],
+    },
+}
+
+
+STATE_TO_REGION = {
+    "CALIFORNIA": "West",
+    "WASHINGTON": "West",
+    "OREGON": "West",
+    "TEXAS": "Southwest",
+    "ARIZONA": "Southwest",
+    "NEW MEXICO": "Southwest",
+    "ILLINOIS": "Midwest",
+    "OHIO": "Midwest",
+    "MICHIGAN": "Midwest",
+    "NEW YORK": "Northeast",
+    "MASSACHUSETTS": "Northeast",
+    "PENNSYLVANIA": "Northeast",
+    "FLORIDA": "Southeast",
+    "GEORGIA": "Southeast",
+    "NORTH CAROLINA": "Southeast",
+}
+
+
+def infer_vertical_key(use_case: str | None) -> str:
+    text = (use_case or "").lower()
+    if any(
+        k in text
+        for k in (
+            "private equity",
+            "lp reporting",
+            "tvpi",
+            "dpi",
+            "rvpi",
+            "buyout",
+            "growth equity",
+            "venture capital",
+            "fund strategy",
+        )
+    ):
+        return "private_equity"
+    if any(k in text for k in ("legal", "counsel", "litigation", "contract", "matter", "spend management", "gc", "attorney")):
+        return "legal"
+    if any(k in text for k in ("saas", "arr", "mrr", "nrr", "churn", "subscription", "fp&a")):
+        return "saas"
+    if any(k in text for k in ("pharma", "pharmaceutical", "prescription", "formulary", "payer", "drug", "hcp", "rx")):
+        return "pharma"
+    if any(k in text for k in ("nike", "athletic", "sportswear", "sneaker", "footwear", "running shoe")):
+        return "sportswear"
+    if "health" in text or "clinical" in text or "patient" in text:
+        return "healthcare"
+    if "bank" in text or "credit union" in text or "mortgage" in text or "loan" in text:
+        return "banking"
+    if "retail" in text or "ecommerce" in text or "commerce" in text or "merch" in text:
+        return "retail"
+    if "supply" in text or "inventory" in text or "logistics" in text:
+        return "supply chain"
+    return "default"
+
+
+def get_domain_values(semantic_type: SemanticType, use_case: str | None = None) -> list[str]:
+    values = list(BASE_DOMAIN_VALUES.get(semantic_type, []))
+    vertical = infer_vertical_key(use_case)
+    if vertical in VERTICAL_OVERRIDES and semantic_type in VERTICAL_OVERRIDES[vertical]:
+        values = list(VERTICAL_OVERRIDES[vertical][semantic_type])
+    return values
+
+
+def map_state_to_region(state_value: str) -> str | None:
+    if not state_value:
+        return None
+    return STATE_TO_REGION.get(str(state_value).strip().upper())
+

legitdata_project/legitdata/domain/semantic_types.py

@@ -0,0 +1,292 @@
+"""Semantic typing for schema columns."""
+
+from __future__ import annotations
+
+from enum import Enum
+
+
+class SemanticType(str, Enum):
+    UNKNOWN = "unknown"
+
+    REGION = "region"
+    CATEGORY = "category"
+    SECTOR_NAME = "sector_name"
+    SECTOR_CATEGORY = "sector_category"
+    SEGMENT = "segment"
+    TIER = "tier"
+    STATUS = "status"
+    FUND_STRATEGY = "fund_strategy"
+    INVESTOR_TYPE = "investor_type"
+    INVESTMENT_STAGE = "investment_stage"
+    COVENANT_STATUS = "covenant_status"
+    DEBT_PERFORMANCE_STATUS = "debt_performance_status"
+    CHANNEL = "channel"
+    COUNTRY = "country"
+    STATE = "state"
+    CITY = "city"
+    POSTAL_CODE = "postal_code"
+    SEASON = "season"
+    HOLIDAY_NAME = "holiday_name"
+    EVENT_NAME = "event_name"
+
+    PERSON_NAME = "person_name"
+    ORG_NAME = "org_name"
+    PRODUCT_NAME = "product_name"
+    BRAND_NAME = "brand_name"
+    BRANCH_NAME = "branch_name"
+    DEPARTMENT_NAME = "department_name"
+    COLOR_DESCRIPTION = "color_description"
+    PACKAGING_SIZE = "packaging_size"
+
+    MONEY = "money"
+    QUANTITY = "quantity"
+    COUNT = "count"
+    PERCENT = "percent"
+    INTEREST_RATE = "interest_rate"
+    RETURN_RATE = "return_rate"
+    RETURN_MULTIPLE = "return_multiple"
+    BASIS_POINTS = "basis_points"
+    LEVERAGE_RATIO = "leverage_ratio"
+    SCORE = "score"
+    DURATION_DAYS = "duration_days"
+
+    DATE_EVENT = "date_event"
+    DATE_START = "date_start"
+    DATE_END = "date_end"
+    DATE_BIRTH = "date_birth"
+
+    BOOLEAN_FLAG = "boolean_flag"
+
+
+BUSINESS_CATEGORICAL_TYPES = {
+    SemanticType.REGION,
+    SemanticType.CATEGORY,
+    SemanticType.SECTOR_NAME,
+    SemanticType.SECTOR_CATEGORY,
+    SemanticType.SEGMENT,
+    SemanticType.TIER,
+    SemanticType.STATUS,
+    SemanticType.FUND_STRATEGY,
+    SemanticType.INVESTOR_TYPE,
+    SemanticType.INVESTMENT_STAGE,
+    SemanticType.COVENANT_STATUS,
+    SemanticType.DEBT_PERFORMANCE_STATUS,
+    SemanticType.CHANNEL,
+    SemanticType.COUNTRY,
+    SemanticType.STATE,
+    SemanticType.CITY,
+    SemanticType.POSTAL_CODE,
+    SemanticType.PRODUCT_NAME,
+    SemanticType.BRAND_NAME,
+    SemanticType.BRANCH_NAME,
+    SemanticType.ORG_NAME,
+    SemanticType.DEPARTMENT_NAME,
+    SemanticType.COLOR_DESCRIPTION,
+    SemanticType.PACKAGING_SIZE,
+    SemanticType.SEASON,
+    SemanticType.HOLIDAY_NAME,
+    SemanticType.EVENT_NAME,
+}
+
+
+NUMERIC_TYPES = {
+    SemanticType.MONEY,
+    SemanticType.QUANTITY,
+    SemanticType.COUNT,
+    SemanticType.PERCENT,
+    SemanticType.INTEREST_RATE,
+    SemanticType.RETURN_RATE,
+    SemanticType.RETURN_MULTIPLE,
+    SemanticType.BASIS_POINTS,
+    SemanticType.LEVERAGE_RATIO,
+    SemanticType.SCORE,
+    SemanticType.DURATION_DAYS,
+}
+
+
+def _contains_any(text: str, tokens: tuple[str, ...]) -> bool:
+    return any(tok in text for tok in tokens)
+
+
+def infer_semantic_type(
+    column_name: str,
+    data_type: str | None = None,
+    table_name: str | None = None,
+    use_case: str | None = None,
+) -> SemanticType:
+    """Infer a coarse semantic type from schema metadata."""
+    col = (column_name or "").lower()
+    dt = (data_type or "").upper()
+    table = (table_name or "").lower()
+    _ = (use_case or "").lower()  # Reserved for future use-case specific typing.
+
+    if col.startswith("is_") or col.startswith("has_") or col.endswith("_flag") or "BOOL" in dt:
+        return SemanticType.BOOLEAN_FLAG
+
+    is_date_type = any(t in dt for t in ("DATE", "TIMESTAMP", "DATETIME"))
+    financial_to_date_name = (
+        ("to_date" in col or col.endswith("todate"))
+        and _contains_any(
+            col,
+            (
+                "amount",
+                "balance",
+                "cost",
+                "price",
+                "revenue",
+                "sales",
+                "fee",
+                "tax",
+                "total",
+                "recovery",
+                "payment",
+            ),
+        )
+    )
+    is_name_date = ("date" in col) and not financial_to_date_name
+    if _contains_any(col, ("birth", "dob", "date_of_birth")):
+        return SemanticType.DATE_BIRTH
+    if is_name_date or is_date_type:
+        if _contains_any(col, ("start", "begin", "open", "admission", "created")):
+            return SemanticType.DATE_START
+        if _contains_any(col, ("end", "close", "discharge", "resolved", "shipped")):
+            return SemanticType.DATE_END
+        return SemanticType.DATE_EVENT
+
+    # Identifier columns should remain code-like, not categorical labels.
+    if (
+        (col.endswith("_id") or col.endswith("id") or col.endswith("_key") or col.endswith("key"))
+        and not _contains_any(col, ("paid", "valid", "invalid"))
+    ):
+        return SemanticType.UNKNOWN
+
+    if _contains_any(col, ("region", "territory")):
+        return SemanticType.REGION
+    if "sector_category" in col:
+        return SemanticType.SECTOR_CATEGORY
+    if col == "sector" or "sector_name" in col or "sub_sector" in col:
+        return SemanticType.SECTOR_NAME
+    if ("month" in col and "name" in col) or ("quarter" in col and "name" in col):
+        # Calendar labels are handled by date-dimension generation rules.
+        return SemanticType.UNKNOWN
+    if _contains_any(col, ("category", "subcategory")):
+        return SemanticType.CATEGORY
+    if "segment" in col:
+        return SemanticType.SEGMENT
+    if _contains_any(col, ("tier", "level")):
+        return SemanticType.TIER
+    if "fund_strategy" in col or col == "strategy" or col.endswith("_strategy"):
+        return SemanticType.FUND_STRATEGY
+    if "investor_type" in col or "lp_type" in col:
+        return SemanticType.INVESTOR_TYPE
+    if "investment_stage" in col:
+        return SemanticType.INVESTMENT_STAGE
+    if "covenant_status" in col:
+        return SemanticType.COVENANT_STATUS
+    if "debt_performance_status" in col or "debt_status" in col:
+        return SemanticType.DEBT_PERFORMANCE_STATUS
+    if "status" in col:
+        return SemanticType.STATUS
+    if _contains_any(col, ("channel", "source")):
+        return SemanticType.CHANNEL
+    if "country" in col:
+        return SemanticType.COUNTRY
+    if _contains_any(col, ("state", "province")):
+        return SemanticType.STATE
+    if "city" in col:
+        return SemanticType.CITY
+    if _contains_any(col, ("zip", "postal")):
+        return SemanticType.POSTAL_CODE
+    if _contains_any(col, ("product_name", "item_name")):
+        return SemanticType.PRODUCT_NAME
+    if "brand" in col and "name" in col:
+        return SemanticType.BRAND_NAME
+    if _contains_any(col, ("branch_name", "branchname")) or col == "branch":
+        return SemanticType.BRANCH_NAME
+    if _contains_any(col, ("department_name", "dept_name", "department", "dept")):
+        return SemanticType.DEPARTMENT_NAME
+    if _contains_any(col, ("employee_name", "manager_name", "agent_name", "rep_name", "officer_name", "contact_name")):
+        return SemanticType.PERSON_NAME
+    if _contains_any(col, ("color_description", "colour_description", "color_desc")):
+        return SemanticType.COLOR_DESCRIPTION
+    if _contains_any(col, ("packaging_size", "package_size", "pack_size")):
+        return SemanticType.PACKAGING_SIZE
+    if "season" in col:
+        return SemanticType.SEASON
+    if "holiday" in col:
+        return SemanticType.HOLIDAY_NAME
+    if "event" in col:
+        return SemanticType.EVENT_NAME
+
+    if col.endswith("_name") or col == "name":
+        # Treat business-entity naming columns as organization labels first.
+        if _contains_any(
+            col,
+            (
+                "account",
+                "customer",
+                "client",
+                "company",
+                "organization",
+                "org",
+                "business",
+                "firm",
+                "vendor",
+                "supplier",
+                "partner",
+                "merchant",
+                "product",
+                "brand",
+                "store",
+                "warehouse",
+                "facility",
+                "branch",
+                "department",
+            ),
+        ):
+            return SemanticType.ORG_NAME
+        if _contains_any(table, ("customer", "employee", "staff", "rep", "agent", "user", "patient", "provider")):
+            return SemanticType.PERSON_NAME
+        return SemanticType.ORG_NAME
+
+    # SaaS/finance abbreviations frequently skip explicit "amount"/"revenue" suffixes.
+    if _contains_any(col, ("arr", "mrr", "cac", "ltv", "billings", "collections", "deferred_revenue")):
+        return SemanticType.MONEY
+    if _contains_any(col, ("net_new_arr", "churned_arr", "expansion_arr", "new_logo_arr", "contraction_arr")):
+        return SemanticType.MONEY
+    if "gross_margin" in col and _contains_any(col, ("_usd", "usd", "amount", "value")):
+        return SemanticType.MONEY
+    if "debt_to_ebitda" in col or "leverage_ratio" in col:
+        return SemanticType.LEVERAGE_RATIO
+    if "bps" in col:
+        return SemanticType.BASIS_POINTS
+    if "irr" in col:
+        return SemanticType.RETURN_RATE
+    if "multiple" in col:
+        return SemanticType.RETURN_MULTIPLE
+    if _contains_any(col, ("nrr", "grr", "gross_margin", "margin_pct", "churn_rate", "retention_rate")):
+        return SemanticType.PERCENT
+    if _contains_any(col, ("payback_months", "payback_period", "days_sales_outstanding", "dso")):
+        return SemanticType.DURATION_DAYS
+
+    if _contains_any(col, ("amount", "revenue", "sales", "cost", "price", "fee", "income", "profit", "total", "value", "balance", "commitment", "aum", "nav")):
+        return SemanticType.MONEY
+    if _contains_any(col, ("quantity", "qty", "units")):
+        return SemanticType.QUANTITY
+    if _contains_any(col, ("count", "num_", "number", "total_")):
+        return SemanticType.COUNT
+    if _contains_any(col, ("percent", "pct", "ratio", "margin")):
+        return SemanticType.PERCENT
+    if "interest_rate" in col or (("interest" in col or "apr" in col) and "rate" in col):
+        return SemanticType.INTEREST_RATE
+    if _contains_any(col, ("score", "rating", "grade")):
+        return SemanticType.SCORE
+    if _contains_any(col, ("days", "day_count", "length_of_stay", "duration")):
+        return SemanticType.DURATION_DAYS
+
+    return SemanticType.UNKNOWN
+
+
+def is_business_categorical(semantic_type: SemanticType) -> bool:
+    return semantic_type in BUSINESS_CATEGORICAL_TYPES
+

legitdata_project/legitdata/generator.py

@@ -1,9 +1,11 @@
 """Main LegitGenerator class - the primary interface for data generation."""
 
 import hashlib
+import math
 import random
+import re
 from typing import Any, Callable, Optional
-from datetime import datetime
+from datetime import datetime, date, timedelta
 
 from .ddl import parse_ddl, parse_ddl_file, Schema, Table, Column, ColumnClassification
 from .analyzer import ContextBuilder, CompanyContext, ColumnClassifier
@@ -12,11 +14,37 @@ from .relationships import FKManager
 from .writers import BaseWriter, SnowflakeWriter, DryRunWriter
 from .cache import FileCache
 from .config import SIZE_PRESETS, GenerationConfig, MAX_AI_ROWS_PER_CALL
+from .domain import (
+    SemanticType,
+    get_domain_values,
+    infer_semantic_type,
+    is_business_categorical,
+    map_state_to_region,
+)
+from .storyspec import StorySpec, build_story_spec
+
+try:
+    from demo_personas import parse_use_case, get_use_case_config
+except Exception:
+    parse_use_case = None
+    get_use_case_config = None
 
 
 class LegitGenerator:
     """Main class for generating realistic synthetic data."""
     
+    EXECUTABLE_GENERIC_STRATEGIES = {
+        "random_int", "random_decimal", "random_float", "random_string",
+        "choice", "weighted_choice", "uuid", "boolean", "date_between", "sequential",
+        "email", "phone", "name", "first_name", "last_name", "company",
+        "address", "city", "state", "country", "zipcode", "url", "text",
+        "paragraph", "sentence", "word", "lookup", "sku",
+        # aliases supported by GenericSourcer
+        "int", "integer", "decimal", "float", "number", "date", "datetime",
+        "bool", "flag", "guid", "id", "zip", "postal",
+    }
+    _GARBAGE_TOKEN_RE = re.compile(r'^[A-Z0-9]{8,}$')
+    
     def __init__(
         self,
         url: str,
@@ -42,8 +70,13 @@ class LegitGenerator:
         self.schema: Optional[Schema] = None
         self.context: Optional[CompanyContext] = None
         self.classifications: Optional[dict] = None
+        self.semantic_types: dict[str, dict[str, SemanticType]] = {}
+        self.story_spec: StorySpec | None = None
+        self.story_events: list[dict[str, Any]] = []
+        self._story_rng = random.Random()
         
         self._init_components()
+        self._refresh_story_spec()
     
     def _init_components(self) -> None:
         self.context_builder = ContextBuilder(
@@ -67,6 +100,43 @@ class LegitGenerator:
         self.generic_sourcer = GenericSourcer()
         self.fk_manager = FKManager()
     
+    def _resolve_use_case_config(self) -> dict[str, Any]:
+        """Resolve vertical/function matrix config when available."""
+        if not parse_use_case or not get_use_case_config:
+            return {}
+        try:
+            vertical, function = parse_use_case(self.use_case or "")
+            return get_use_case_config(vertical or "Generic", function or "Generic")
+        except Exception:
+            return {}
+
+    def _refresh_story_spec(self) -> None:
+        """Build per-run StorySpec and deterministic RNG seed."""
+        cfg = self._resolve_use_case_config()
+        company_name = None
+        if self.context and getattr(self.context, "company_name", None):
+            company_name = self.context.company_name
+
+        self.story_spec = build_story_spec(
+            company_url=self.url,
+            use_case=self.use_case,
+            company_name=company_name,
+            use_case_config=cfg,
+        )
+
+        # Stabilize pseudo-random behavior for reproducible runs.
+        self._story_rng = random.Random(self.story_spec.seed)
+        random.seed(self.story_spec.seed)
+
+    def _domain_use_case(self) -> str:
+        """Use company + use_case context for domain pack routing."""
+        company_hint = ""
+        if self.context and getattr(self.context, "company_name", None):
+            company_hint = str(self.context.company_name).strip()
+        elif self.url:
+            company_hint = str(self.url).strip()
+        return f"{company_hint} {self.use_case or ''}".strip()
+
     def load_ddl(self, ddl_or_path: str) -> Schema:
         if '\n' not in ddl_or_path and (
             ddl_or_path.endswith('.sql') or 
@@ -99,25 +169,42 @@ class LegitGenerator:
             row_counts=row_counts
         )
         
-        print("\n=== Step 1: Building Company Context ===")
-        self._build_context()
-        
-        print("\n=== Step 2: Classifying Columns ===")
-        self._classify_columns()
-        
+        is_saas_finance_gold = self._is_saas_finance_gold_schema()
+        if is_saas_finance_gold:
+            print("\n=== Step 1: Preparing Deterministic Finance Context ===")
+            self.context = None
+            self._refresh_story_spec()
+            print(f"  Story seed: {self.story_spec.seed}")
+        else:
+            print("\n=== Step 1: Building Company Context ===")
+            self._build_context()
+
+        if is_saas_finance_gold:
+            print("\n=== Step 2: Skipping Column Classification ===")
+            print("  Using deterministic semantic typing for SaaS finance gold-path schema...")
+            self._build_semantic_types()
+        else:
+            print("\n=== Step 2: Classifying Columns ===")
+            self._classify_columns()
+
         print("\n=== Step 3: Generating Data ===")
         generated_data = {}
-        
-        for table in self.schema.get_dependency_order():
-            num_rows = config.get_table_row_count(table.name, table.is_fact_table)
-            print(f"\nGenerating {num_rows} rows for {table.name}...")
-            
-            rows = self._generate_table_data(table, num_rows)
-            generated_data[table.name] = rows
-            
-            # Register PK values and full rows for FK references
-            self._register_pk_values(table, rows)
-            self.fk_manager.register_rows(table.name, rows)
+        self.story_events = []
+
+        if is_saas_finance_gold:
+            print("  Using deterministic SaaS finance gold-path generator...")
+            generated_data = self._generate_saas_finance_gold(config)
+        else:
+            for table in self.schema.get_dependency_order():
+                num_rows = config.get_table_row_count(table.name, table.is_fact_table)
+                print(f"\nGenerating {num_rows} rows for {table.name}...")
+
+                rows = self._generate_table_data(table, num_rows)
+                generated_data[table.name] = rows
+
+                # Register PK values and full rows for FK references
+                self._register_pk_values(table, rows)
+                self.fk_manager.register_rows(table.name, rows)
         
         print("\n=== Step 4: Writing to Database ===")
         results = self._write_to_database(generated_data, truncate_first)
@@ -127,28 +214,510 @@ class LegitGenerator:
         print(f"Total rows inserted: {total}")
         
         return results
+
+    def _is_saas_finance_gold_schema(self) -> bool:
+        if not self.schema:
+            return False
+        table_names = {table.name.upper() for table in self.schema.tables}
+        required = {"DATES", "CUSTOMERS", "PRODUCTS", "LOCATIONS", "SAAS_CUSTOMER_MONTHLY", "SALES_MARKETING_SPEND_MONTHLY"}
+        if required.issubset(table_names):
+            return True
+
+        cfg = self._resolve_use_case_config()
+        schema_contract = cfg.get("schema_contract", {}) if isinstance(cfg, dict) else {}
+        return schema_contract.get("mode") == "saas_finance_gold"
+
+    def _generate_saas_finance_gold(self, config: GenerationConfig) -> dict[str, list[dict]]:
+        """Generate a deterministic recurring-revenue SaaS finance dataset."""
+        rng = self._story_rng
+        today = datetime.now().date().replace(day=1)
+        months = []
+        for offset in range(23, -1, -1):
+            year = today.year
+            month = today.month - offset
+            while month <= 0:
+                month += 12
+                year -= 1
+            while month > 12:
+                month -= 12
+                year += 1
+            month_date = date(year, month, 1)
+            months.append(month_date)
+
+        def month_key(dt: date) -> int:
+            return int(dt.strftime("%Y%m01"))
+
+        dates_rows = []
+        for dt in months:
+            quarter = ((dt.month - 1) // 3) + 1
+            dates_rows.append(
+                {
+                    "MONTH_KEY": month_key(dt),
+                    "FULL_DATE": dt,
+                    "MONTH_NAME": dt.strftime("%B"),
+                    "QUARTER_NAME": f"Q{quarter}",
+                    "YEAR_NUM": dt.year,
+                    "IS_QUARTER_END": dt.month in {3, 6, 9, 12},
+                    "IS_YEAR_END": dt.month == 12,
+                }
+            )
+
+        segment_weights = [("Enterprise", 0.28), ("Mid-Market", 0.34), ("SMB", 0.38)]
+        region_arr_multipliers = {"North America": 1.28, "EMEA": 1.04, "APAC": 0.81}
+        segment_motion_profiles = {
+            "Enterprise": {"expansion": (0.026, 0.078), "contraction": (0.000, 0.008)},
+            "Mid-Market": {"expansion": (0.006, 0.028), "contraction": (0.006, 0.026)},
+            "SMB": {"expansion": (0.000, 0.010), "contraction": (0.022, 0.072)},
+        }
+        region_motion_bias = {
+            "North America": {"expansion": 0.014, "contraction": -0.004},
+            "EMEA": {"expansion": -0.012, "contraction": 0.016},
+            "APAC": {"expansion": 0.002, "contraction": 0.006},
+        }
+        region_defs = [
+            ("North America", "NA - West", "Americas", "US", "CA", "San Francisco"),
+            ("North America", "NA - Central", "Americas", "US", "TX", "Austin"),
+            ("North America", "NA - East", "Americas", "US", "NY", "New York"),
+            ("EMEA", "EMEA - UKI", "Europe", "GB", "", "London"),
+            ("EMEA", "EMEA - DACH", "Europe", "DE", "", "Berlin"),
+            ("EMEA", "EMEA - Southern Europe", "Europe", "FR", "", "Paris"),
+            ("APAC", "APAC - ANZ", "Asia Pacific", "AU", "", "Sydney"),
+            ("APAC", "APAC - Japan", "Asia Pacific", "JP", "", "Tokyo"),
+            ("APAC", "APAC - India", "Asia Pacific", "IN", "", "Bengaluru"),
+        ]
+        product_defs = [
+            ("ThoughtSpot Analytics Cloud", "Core Platform", "Enterprise", "Subscription", True, False, 84.0, 120000.0),
+            ("Agentic Analytics", "AI Analyst", "Enterprise", "Subscription", False, True, 87.0, 42000.0),
+            ("Embedded Analytics", "Embedded Analytics", "Scale", "Subscription", False, True, 82.0, 36000.0),
+            ("Semantic Modeling Studio", "Semantic Modeling", "Professional", "Subscription", True, False, 86.0, 26000.0),
+            ("Data Prep Studio", "Data Prep & Studio", "Professional", "Subscription", False, True, 80.0, 18000.0),
+            ("Security & Governance", "Platform Add-ons", "Enterprise", "Subscription", False, True, 88.0, 22000.0),
+            ("Observability Cloud", "Observability", "Professional", "Usage", False, True, 79.0, 24000.0),
+            ("Developer APIs", "Platform Add-ons", "Growth", "Usage", False, True, 90.0, 12000.0),
+            ("Consumption Credits", "Usage Services", "Growth", "Usage", False, True, 76.0, 9000.0),
+            ("Customer Success Premium", "Services", "Enterprise", "Services", False, True, 68.0, 15000.0),
+        ]
+
+        location_rows = []
+        for idx, (region, sub_region, geo, country, state_abbr, city) in enumerate(region_defs, start=1):
+            location_rows.append(
+                {
+                    "LOCATION_KEY": idx,
+                    "GEO": geo,
+                    "REGION": region,
+                    "SUB_REGION": sub_region,
+                    "COUNTRY": country,
+                    "STATE_ABBR": state_abbr,
+                    "CITY": city,
+                }
+            )
+
+        product_rows = []
+        for idx, product in enumerate(product_defs, start=1):
+            product_rows.append(
+                {
+                    "PRODUCT_KEY": idx,
+                    "PRODUCT_NAME": product[0],
+                    "PRODUCT_FAMILY": product[1],
+                    "PLAN_TIER": product[2],
+                    "PRICING_MODEL": product[3],
+                    "IS_CORE_PLATFORM": product[4],
+                    "IS_ADD_ON": product[5],
+                    "DEFAULT_GROSS_MARGIN_PCT": round(product[6], 2),
+                    "LIST_PRICE_USD": round(product[7], 2),
+                }
+            )
+
+        requested_customer_count = config.row_counts.get("CUSTOMERS") if config.row_counts else None
+        if requested_customer_count is None:
+            requested_customer_count = config.get_table_row_count("CUSTOMERS", False)
+        customer_count = max(72, min(108, int(requested_customer_count or 84)))
+        company_tokens = ["Cloud", "Data", "Ops", "Signal", "Metric", "Scale", "Insight", "Lake", "Grid", "Core", "Logic", "Flow"]
+        suffixes = ["Systems", "Labs", "Software", "Analytics", "Platforms", "Works", "Networks", "Cloud", "Dynamics"]
+
+        def weighted_choice(weighted_values: list[tuple[str, float]]) -> str:
+            roll = rng.random()
+            running = 0.0
+            for value, weight in weighted_values:
+                running += weight
+                if roll <= running:
+                    return value
+            return weighted_values[-1][0]
+
+        customers = []
+        customers_rows = []
+        for idx in range(1, customer_count + 1):
+            segment = weighted_choice(segment_weights)
+            region_row = rng.choice(location_rows)
+            product_row = rng.choice(product_rows[:8])
+            acquired_idx = rng.randint(0, 17)
+            acquired_month = months[acquired_idx]
+            billing_cadence = rng.choices(
+                ["Annual", "Quarterly", "Monthly"],
+                weights=[0.62, 0.18, 0.20] if segment != "SMB" else [0.18, 0.12, 0.70],
+                k=1,
+            )[0]
+            contract_type = rng.choices(
+                ["New Business", "Renewal", "Expansion"],
+                weights=[0.55, 0.25, 0.20],
+                k=1,
+            )[0]
+            risk_band = rng.choices(["Low", "Medium", "High"], weights=[0.55, 0.30, 0.15], k=1)[0]
+            owner = f"AE {idx:03d}"
+            token_a = company_tokens[idx % len(company_tokens)]
+            token_b = company_tokens[(idx * 3) % len(company_tokens)]
+            company_name = f"{token_a}{token_b} {suffixes[idx % len(suffixes)]}"
+            if segment == "Enterprise":
+                base_arr = rng.randint(90000, 340000)
+            elif segment == "Mid-Market":
+                base_arr = rng.randint(30000, 95000)
+            else:
+                base_arr = rng.randint(6000, 24000)
+            base_arr = int(
+                base_arr
+                * region_arr_multipliers[region_row["REGION"]]
+                * rng.uniform(0.88, 1.16)
+            )
+
+            customers.append(
+                {
+                    "customer_key": idx,
+                    "name": company_name,
+                    "segment": segment,
+                    "location_key": region_row["LOCATION_KEY"],
+                    "region": region_row["REGION"],
+                    "sub_region": region_row["SUB_REGION"],
+                    "product_key": product_row["PRODUCT_KEY"],
+                    "billing_cadence": billing_cadence,
+                    "contract_type": contract_type,
+                    "risk_band": risk_band,
+                    "owner": owner,
+                    "acquired_idx": acquired_idx,
+                    "base_arr": float(base_arr),
+                    "renewal_offset": acquired_idx % (12 if billing_cadence == "Annual" else 3 if billing_cadence == "Quarterly" else 1),
+                }
+            )
+
+            customers_rows.append(
+                {
+                    "CUSTOMER_KEY": idx,
+                    "CUSTOMER_NAME": company_name,
+                    "CUSTOMER_TIER": "Strategic" if segment == "Enterprise" else "Growth" if segment == "Mid-Market" else "Velocity",
+                    "SEGMENT": segment,
+                    "CONTRACT_TYPE": contract_type,
+                    "BILLING_CADENCE": billing_cadence,
+                    "ACQUISITION_CHANNEL": rng.choices(
+                        ["Direct Sales", "Partner", "Digital Self-Serve"],
+                        weights=[0.58, 0.20, 0.22],
+                        k=1,
+                    )[0],
+                    "CHURN_RISK_BAND": risk_band,
+                    "CUSTOMER_STATUS": "Active",
+                    "CUSTOMER_ACQUIRED_MONTH_KEY": month_key(acquired_month),
+                    "PRIMARY_REGION": region_row["REGION"],
+                    "ACCOUNT_OWNER": owner,
+                    "PRIMARY_USE_CASE": rng.choice(
+                        ["Executive Analytics", "Embedded Analytics", "FP&A", "Product Analytics", "Sales Operations"]
+                    ),
+                }
+            )
+
+        fact_rows: list[dict[str, Any]] = []
+        new_logo_lookup: dict[tuple[int, str, str], list[int]] = {}
+        spend_accumulator: dict[tuple[int, str, str], dict[str, float]] = {}
+        fact_key = 1
+
+        july_spike_key = month_key(next((m for m in months if m.month == 7), months[len(months) // 2]))
+        q4_keys = {month_key(m) for m in months if m.month in {10, 11, 12} and m.year == months[-1].year}
+        emea_lag_keys = {month_key(m) for m in months if m.year == months[-1].year and m.month in {5, 6, 7}}
+        upsell_keys = {month_key(m) for m in months if m.year == months[-1].year and m.month in {4, 5, 6}}
+
+        customer_status_by_key = {row["CUSTOMER_KEY"]: row for row in customers_rows}
+        deferred_balance_by_customer: dict[int, float] = {}
+
+        for customer in customers:
+            churn_probability = 0.05 if customer["segment"] == "Enterprise" else 0.14 if customer["segment"] == "Mid-Market" else 0.30
+            if customer["region"] == "EMEA":
+                churn_probability += 0.05
+            elif customer["region"] == "North America":
+                churn_probability -= 0.02
+            if customer["risk_band"] == "High":
+                churn_probability += 0.06
+            elif customer["risk_band"] == "Low":
+                churn_probability -= 0.025
+            churn_probability = max(0.02, min(0.42, churn_probability))
+            churn_idx = None
+            if rng.random() < churn_probability:
+                earliest = max(customer["acquired_idx"] + 6, 8)
+                if earliest < len(months):
+                    churn_idx = rng.randint(earliest, len(months) - 1)
+            prev_arr = 0.0
+            deferred_balance = 0.0
+            base_seats = max(5, int(customer["base_arr"] / (1200 if customer["segment"] == "Enterprise" else 600 if customer["segment"] == "Mid-Market" else 240)))
+
+            for month_index, month_start in enumerate(months):
+                if month_index < customer["acquired_idx"]:
+                    continue
+                if churn_idx is not None and month_index > churn_idx:
+                    continue
+
+                mk = month_key(month_start)
+                starting_arr = round(prev_arr, 2)
+                new_logo_arr = round(customer["base_arr"], 2) if month_index == customer["acquired_idx"] else 0.0
+
+                expansion_pct = 0.0
+                contraction_pct = 0.0
+                if starting_arr > 0:
+                    motion_profile = segment_motion_profiles[customer["segment"]]
+                    region_bias = region_motion_bias[customer["region"]]
+                    base_expansion = rng.uniform(*motion_profile["expansion"]) + region_bias["expansion"]
+                    base_contraction = rng.uniform(*motion_profile["contraction"]) + region_bias["contraction"]
+                    if customer["segment"] == "Enterprise" and mk in q4_keys:
+                        base_expansion += rng.uniform(0.05, 0.09)
+                    if customer["segment"] == "SMB" and mk == july_spike_key:
+                        base_contraction += rng.uniform(0.08, 0.14)
+                    if customer["segment"] == "Mid-Market" and mk in upsell_keys:
+                        base_expansion += rng.uniform(0.03, 0.06)
+                    if customer["segment"] == "SMB" and cadence == "Monthly" and month_start.month in {3, 6, 9, 12}:
+                        base_contraction += rng.uniform(0.015, 0.045)
+                    if customer["segment"] == "Enterprise" and customer["region"] == "North America" and month_start.month in {2, 5, 8, 11}:
+                        base_expansion += rng.uniform(0.015, 0.03)
+                    if customer["region"] == "EMEA" and month_start.month in {4, 5, 6, 7}:
+                        base_contraction += rng.uniform(0.012, 0.026)
+                    expansion_pct = max(0.0, min(0.18, base_expansion))
+                    contraction_pct = max(0.0, min(0.22, base_contraction))
+
+                expansion_arr = round(starting_arr * expansion_pct, 2)
+                contraction_arr = round(starting_arr * contraction_pct, 2)
+                churned_arr = 0.0
+                customer_churned_flag = False
+                if churn_idx is not None and month_index == churn_idx:
+                    churned_arr = round(max(starting_arr + new_logo_arr + expansion_arr - contraction_arr, 0.0), 2)
+                    customer_churned_flag = churned_arr > 0
+
+                ending_arr = round(max(starting_arr + new_logo_arr + expansion_arr - contraction_arr - churned_arr, 0.0), 2)
+                mrr_usd = round(ending_arr / 12.0, 2)
+
+                cadence = customer["billing_cadence"]
+                cadence_mod = 12 if cadence == "Annual" else 3 if cadence == "Quarterly" else 1
+                billing_due = ((month_index - customer["renewal_offset"]) % cadence_mod) == 0
+                billings_usd = 0.0
+                if cadence == "Monthly":
+                    billings_usd = round(mrr_usd * rng.uniform(0.98, 1.03), 2)
+                elif billing_due and cadence == "Quarterly":
+                    billings_usd = round(mrr_usd * 3.0 * rng.uniform(0.98, 1.04), 2)
+                elif billing_due and cadence == "Annual":
+                    billings_usd = round(max(ending_arr, 0.0) * rng.uniform(0.99, 1.05), 2)
+
+                if customer["segment"] == "Enterprise" and month_start.month == 11 and cadence == "Annual":
+                    billings_usd = round(billings_usd * 1.18, 2)
+
+                recognized_revenue = mrr_usd
+                lag_factor = 0.97
+                if customer["region"] == "EMEA":
+                    lag_factor = 0.88 if mk in emea_lag_keys else 0.93
+                elif customer["region"] == "APAC":
+                    lag_factor = 0.95
+                if customer["segment"] == "SMB":
+                    lag_factor -= 0.02
+                collections_usd = round(min(billings_usd, max(billings_usd * lag_factor, 0.0)), 2)
+                deferred_balance = round(max(deferred_balance + billings_usd - recognized_revenue, 0.0), 2)
+                dso_days = round(max(22.0, min(92.0, 30.0 + ((billings_usd - collections_usd) / billings_usd * 45.0 if billings_usd else 0.0) + (10.0 if mk in emea_lag_keys and customer["region"] == "EMEA" else 0.0))), 2)
+
+                product_row = product_rows[customer["product_key"] - 1]
+                gm_pct = float(product_row["DEFAULT_GROSS_MARGIN_PCT"])
+                if product_row["PRODUCT_FAMILY"] == "Services":
+                    gm_pct -= 12.0
+                if month_start.month in {5, 6} and product_row["PRODUCT_FAMILY"] == "Observability":
+                    gm_pct -= 6.0
+                gm_pct = round(max(45.0, min(92.0, gm_pct + rng.uniform(-1.5, 1.5))), 2)
+                gross_margin_usd = round(recognized_revenue * (gm_pct / 100.0), 2)
+                support_cost_usd = round(max(recognized_revenue - gross_margin_usd, recognized_revenue * 0.06), 2)
+                seat_count = int(max(1, round(base_seats * max(ending_arr, customer["base_arr"]) / max(customer["base_arr"], 1.0))))
+                usage_units = round(seat_count * rng.uniform(1.8, 4.6), 2)
+                customer_acquired_flag = month_index == customer["acquired_idx"]
+
+                fact_rows.append(
+                    {
+                        "SAAS_CUSTOMER_MONTHLY_KEY": fact_key,
+                        "MONTH_KEY": mk,
+                        "CUSTOMER_KEY": customer["customer_key"],
+                        "PRODUCT_KEY": customer["product_key"],
+                        "LOCATION_KEY": customer["location_key"],
+                        "SCENARIO": "Actual",
+                        "CONTRACT_TERM_MONTHS": 12 if cadence == "Annual" else 3 if cadence == "Quarterly" else 1,
+                        "STARTING_ARR_USD": starting_arr,
+                        "NEW_LOGO_ARR_USD": new_logo_arr,
+                        "EXPANSION_ARR_USD": expansion_arr,
+                        "CONTRACTION_ARR_USD": contraction_arr,
+                        "CHURNED_ARR_USD": churned_arr,
+                        "ENDING_ARR_USD": ending_arr,
+                        "MRR_USD": mrr_usd,
+                        "BILLINGS_USD": billings_usd,
+                        "COLLECTIONS_USD": collections_usd,
+                        "DEFERRED_REVENUE_USD": deferred_balance,
+                        "GROSS_MARGIN_USD": gross_margin_usd,
+                        "GROSS_MARGIN_PCT": gm_pct,
+                        "SUPPORT_COST_USD": support_cost_usd,
+                        "CAC_ALLOCATED_USD": 0.0,
+                        "DSO_DAYS": dso_days,
+                        "SEAT_COUNT": seat_count,
+                        "USAGE_UNITS": usage_units,
+                        "CUSTOMER_ACQUIRED_FLAG": customer_acquired_flag,
+                        "CUSTOMER_CHURNED_FLAG": customer_churned_flag,
+                    }
+                )
+                if customer_acquired_flag:
+                    lookup_key = (mk, customer["segment"], customer["region"])
+                    new_logo_lookup.setdefault(lookup_key, []).append(fact_key)
+
+                spend_bucket = spend_accumulator.setdefault(
+                    (mk, customer["segment"], customer["region"]),
+                    {
+                        "new_customers": 0.0,
+                        "new_logo_arr": 0.0,
+                        "starting_arr": 0.0,
+                        "expansion_arr": 0.0,
+                        "contraction_arr": 0.0,
+                        "churned_arr": 0.0,
+                    },
+                )
+                spend_bucket["starting_arr"] += starting_arr
+                spend_bucket["expansion_arr"] += expansion_arr
+                spend_bucket["contraction_arr"] += contraction_arr
+                spend_bucket["churned_arr"] += churned_arr
+                if customer_acquired_flag:
+                    spend_bucket["new_customers"] += 1.0
+                    spend_bucket["new_logo_arr"] += new_logo_arr
+
+                fact_key += 1
+                prev_arr = ending_arr
+                deferred_balance_by_customer[customer["customer_key"]] = deferred_balance
+
+            if churn_idx is not None:
+                customer_status_by_key[customer["customer_key"]]["CUSTOMER_STATUS"] = "Churned"
+
+        spend_rows: list[dict[str, Any]] = []
+        spend_key = 1
+        cac_targets = {
+            "Enterprise": 32000.0,
+            "Mid-Market": 14500.0,
+            "SMB": 4200.0,
+        }
+        region_multipliers = {"North America": 1.0, "EMEA": 1.12, "APAC": 0.94}
+
+        for month_start in months:
+            mk = month_key(month_start)
+            for segment in ("Enterprise", "Mid-Market", "SMB"):
+                for region in ("North America", "EMEA", "APAC"):
+                    bucket = spend_accumulator.get(
+                        (mk, segment, region),
+                        {
+                            "new_customers": 0.0,
+                            "new_logo_arr": 0.0,
+                            "starting_arr": 0.0,
+                            "expansion_arr": 0.0,
+                            "contraction_arr": 0.0,
+                            "churned_arr": 0.0,
+                        },
+                    )
+                    new_customers = int(bucket["new_customers"])
+                    baseline_new_customers = max(new_customers, 1 if segment != "Enterprise" else 0)
+                    target_cac = cac_targets[segment] * region_multipliers[region]
+                    if region == "EMEA" and month_start.month in {5, 6, 7}:
+                        target_cac *= 1.08
+                    total_spend = round(baseline_new_customers * target_cac, 2)
+                    sales_spend = round(total_spend * 0.64, 2)
+                    marketing_spend = round(total_spend - sales_spend, 2)
+                    net_new_arr = round(
+                        bucket["new_logo_arr"] + bucket["expansion_arr"] - bucket["contraction_arr"] - bucket["churned_arr"],
+                        2,
+                    )
+                    starting_arr = float(bucket["starting_arr"] or 0.0)
+                    nrr_pct = round(
+                        ((starting_arr + bucket["expansion_arr"] - bucket["contraction_arr"] - bucket["churned_arr"]) / starting_arr) * 100.0,
+                        4,
+                    ) if starting_arr > 0 else 100.0
+                    grr_pct = round(
+                        ((starting_arr - bucket["contraction_arr"] - bucket["churned_arr"]) / starting_arr) * 100.0,
+                        4,
+                    ) if starting_arr > 0 else 100.0
+                    cac_usd = round(total_spend / new_customers, 2) if new_customers > 0 else 0.0
+                    spend_rows.append(
+                        {
+                            "SALES_MARKETING_SPEND_MONTHLY_KEY": spend_key,
+                            "MONTH_KEY": mk,
+                            "SEGMENT": segment,
+                            "REGION": region,
+                            "SALES_SPEND_USD": sales_spend,
+                            "MARKETING_SPEND_USD": marketing_spend,
+                            "TOTAL_S_AND_M_SPEND_USD": round(sales_spend + marketing_spend, 2),
+                            "NEW_CUSTOMERS_ACQUIRED": new_customers,
+                            "NEW_LOGO_ARR_USD": round(bucket["new_logo_arr"], 2),
+                            "NET_NEW_ARR_USD": net_new_arr,
+                            "NRR_PCT": nrr_pct,
+                            "GRR_PCT": grr_pct,
+                            "CAC_USD": cac_usd,
+                        }
+                    )
+                    if new_customers > 0:
+                        per_customer_cac = cac_usd
+                        for fact_row_key in new_logo_lookup.get((mk, segment, region), []):
+                            fact_rows[fact_row_key - 1]["CAC_ALLOCATED_USD"] = per_customer_cac
+                    spend_key += 1
+
+        return {
+            "DATES": dates_rows,
+            "CUSTOMERS": customers_rows,
+            "PRODUCTS": product_rows,
+            "LOCATIONS": location_rows,
+            "SAAS_CUSTOMER_MONTHLY": fact_rows,
+            "SALES_MARKETING_SPEND_MONTHLY": spend_rows,
+        }
     
     def _register_pk_values(self, table: Table, rows: list[dict]) -> None:
-        """Register PK values for FK lookups. Generate fake IDs for identity columns."""
-        pk_col = table.primary_key
-        if not pk_col:
+        """Register key-like values for FK lookups.
+
+        Parser output is not always perfect with table-level PK constraints, so
+        we also register identity/id-style columns that are likely FK targets.
+        """
+        key_columns: list[str] = []
+        if table.primary_key:
+            key_columns.append(table.primary_key)
+
+        for col in table.columns:
+            if col.is_identity and col.name not in key_columns:
+                key_columns.append(col.name)
+
+        if not key_columns:
+            for col in table.columns:
+                c = col.name.upper()
+                if c == "ID" or c.endswith("_ID") or c.endswith("ID"):
+                    key_columns.append(col.name)
+                    break
+
+        if not key_columns:
             return
-        
-        # Get the PK column definition
-        pk_column = table.get_column(pk_col)
-        
-        pk_values = []
-        for i, row in enumerate(rows):
-            pk_val = row.get(pk_col)
-            
-            # If PK is None (identity column), generate a fake ID for FK purposes
-            if pk_val is None:
-                pk_val = i + 1  # Simple sequential ID
-                row[pk_col] = pk_val  # Update the row too
-            
-            pk_values.append(pk_val)
-        
-        self.fk_manager.register_pk_values(table.name, pk_col, pk_values)
+
+        for key_col in key_columns:
+            col_def = table.get_column(key_col)
+            if not col_def:
+                continue
+
+            key_values = []
+            for i, row in enumerate(rows):
+                key_val = row.get(key_col)
+
+                # Keep deterministic IDs available for FK assignment before DB insert.
+                if key_val is None and col_def.is_identity:
+                    key_val = i + 1
+                    row[key_col] = key_val
+
+                if key_val is not None:
+                    key_values.append(key_val)
+
+            if key_values:
+                self.fk_manager.register_pk_values(table.name, key_col, key_values)
     
     def _build_context(self) -> None:
         if self.cache_enabled and self.cache:
@@ -156,13 +725,18 @@ class LegitGenerator:
             if cached:
                 print(f"Using cached context for {self.url}")
                 self.context = CompanyContext.from_dict(cached)
+                self._refresh_story_spec()
+                print(f"  Story seed: {self.story_spec.seed}")
                 return
         
         self.context = self.context_builder.build_context(self.url, self.use_case)
         print(f"Built context for: {self.context.company_name}")
         print(f"  Industry: {self.context.industry}")
         print(f"  Focus: {self.context.geographic_focus}")
-        
+
+        self._refresh_story_spec()
+        print(f"  Story seed: {self.story_spec.seed}")
+
         if self.cache_enabled and self.cache:
             self.cache.set_context(self.url, self.context.to_dict())
     
@@ -177,6 +751,7 @@ class LegitGenerator:
             if cached:
                 print("Using cached column classifications")
                 self.classifications = cached
+                self._build_semantic_types()
                 return
         
         self.classifications = self.column_classifier.classify_schema(
@@ -194,13 +769,33 @@ class LegitGenerator:
             ai_gen = sum(1 for c in columns.values() if c['classification'] == 'AI_GEN')
             generic = sum(1 for c in columns.values() if c['classification'] == 'GENERIC')
             print(f"  {table_name}: {search_real} SEARCH_REAL, {ai_gen} AI_GEN, {generic} GENERIC")
+
+        self._build_semantic_types()
         
         if self.cache_enabled and self.cache:
             self.cache.set_classification(schema_hash, self.classifications)
+
+    def _build_semantic_types(self) -> None:
+        """Build semantic type map for all table columns."""
+        self.semantic_types = {}
+        if not self.schema:
+            return
+        for table in self.schema.tables:
+            table_semantics: dict[str, SemanticType] = {}
+            for column in table.columns:
+                table_semantics[column.name] = infer_semantic_type(
+                    column.name,
+                    column.data_type,
+                    table.name,
+                    self._domain_use_case(),
+                )
+            self.semantic_types[table.name] = table_semantics
     
     def _generate_table_data(self, table: Table, num_rows: int) -> list[dict]:
         rows = []
         table_class = self.classifications.get(table.name, {})
+        table_semantics = self.semantic_types.get(table.name, {})
+        date_dim_profile = self._build_date_dimension_profile(table, table_semantics, num_rows)
         
         # Build set of FK column names
         fk_columns = {fk.column_name.upper() for fk in table.foreign_keys}
@@ -213,6 +808,7 @@ class LegitGenerator:
             col_class = table_class.get(column.name, {})
             classification = col_class.get('classification', 'GENERIC').upper()
             strategy = col_class.get('strategy', '')
+            semantic_type = table_semantics.get(column.name, SemanticType.UNKNOWN)
             
             # Skip identity/PK/FK columns
             if column.is_identity or column.is_primary_key:
@@ -224,16 +820,31 @@ class LegitGenerator:
                 coherent_columns.append({
                     'name': column.name,
                     'classification': classification,
-                    'strategy': strategy
+                    'strategy': strategy,
+                    'semantic_type': semantic_type,
                 })
                 print(f"  [CLASSIFY] {column.name}: {classification} → coherent (AI-generated)")
+            elif is_business_categorical(semantic_type):
+                # Treat business categorical columns as coherent to avoid noisy per-row drift.
+                coherent_columns.append({
+                    'name': column.name,
+                    'classification': 'AI_GEN',
+                    'strategy': strategy or f"Generate realistic {column.name} values",
+                    'semantic_type': semantic_type,
+                })
+                print(f"  [CLASSIFY] {column.name}: semantic={semantic_type.value} → coherent (domain-aware)")
             else:
                 generic_columns.append(column)
                 print(f"  [CLASSIFY] {column.name}: {classification} strategy='{strategy}' → generic (Faker)")
         
         # For dimension tables with multiple coherent columns, generate as entities
         coherent_entities = []
-        if not table.is_fact_table and len(coherent_columns) >= 2:
+        domain_use_case = self._domain_use_case()
+        has_freeform_coherent = any(
+            not is_business_categorical(col.get("semantic_type", SemanticType.UNKNOWN))
+            for col in coherent_columns
+        )
+        if not table.is_fact_table and len(coherent_columns) >= 2 and has_freeform_coherent:
             print(f"  Generating {num_rows} coherent entities for {table.name}...")
             coherent_entities = self.ai_generator.generate_entity_batch(
                 table_name=table.name,
@@ -249,11 +860,12 @@ class LegitGenerator:
                 col_name = col_info['name']
                 classification = col_info['classification']
                 strategy = col_info['strategy']
+                semantic_type = col_info.get('semantic_type', SemanticType.UNKNOWN)
                 
                 if classification == 'SEARCH_REAL':
-                    values = self._get_search_real_values(col_name, strategy, num_rows)
+                    values = self._get_search_real_values(col_name, strategy, num_rows, semantic_type)
                 else:  # AI_GEN
-                    values = self._get_ai_gen_values(col_name, strategy, num_rows)
+                    values = self._get_ai_gen_values(col_name, strategy, num_rows, semantic_type)
                 column_values[col_name] = values
             
             # Convert to entity format
@@ -342,36 +954,56 @@ class LegitGenerator:
                 
                 # GENERIC columns - use classification strategy if set, otherwise infer
                 col_class = table_class.get(column.name, {})
-                strategy = col_class.get('strategy', '')
-                
-                # If strategy looks like a Faker type (not an AI prompt), use it
-                faker_strategies = ('email', 'first_name', 'last_name', 'name', 'phone',
-                                   'city', 'state', 'country', 'zipcode', 'address', 
-                                   'uuid', 'url', 'company')
-                
-                # Check if column is numeric - don't use faker for numeric columns!
-                data_type_upper = (column.data_type or '').upper()
-                is_numeric = any(t in data_type_upper for t in ('INT', 'NUMBER', 'NUMERIC', 'DECIMAL', 'BIGINT', 'SMALLINT', 'FLOAT'))
+                classification = col_class.get('classification', 'GENERIC').upper()
+                strategy = (col_class.get('strategy') or '').strip()
+                strategy_is_executable = self._is_executable_generic_strategy(strategy)
+                semantic_type = table_semantics.get(col_name, SemanticType.UNKNOWN)
                 
                 # DEBUG: Log first row only
                 if i == 0:
-                    print(f"    [GEN] {col_name}: strategy='{strategy}', is_faker={strategy in faker_strategies}, is_numeric={is_numeric}")
+                    print(f"    [GEN] {col_name}: class='{classification}', strategy='{strategy}', executable={strategy_is_executable}")
                 
-                if strategy and strategy in faker_strategies and not is_numeric:
-                    # Only use faker for non-numeric columns
-                    row[col_name] = self.generic_sourcer.generate_value(strategy)
+                # Quality-first: strong semantic types override generic model hints.
+                semantic_strategy = self.generic_sourcer.get_strategy_for_semantic(semantic_type, domain_use_case)
+                story_domain_values = self._get_story_domain_values(semantic_type)
+                if semantic_strategy and (is_business_categorical(semantic_type) or semantic_type in {SemanticType.ORG_NAME, SemanticType.PRODUCT_NAME, SemanticType.BRAND_NAME, SemanticType.PERSON_NAME}):
+                    if story_domain_values:
+                        row[col_name] = self._choose_story_value(story_domain_values, i)
+                    else:
+                        row[col_name] = self.generic_sourcer.generate_value(semantic_strategy, expected_type=column.data_type)
+                elif semantic_strategy:
+                    row[col_name] = self.generic_sourcer.generate_value(semantic_strategy, expected_type=column.data_type)
+                # Use explicit GENERIC strategy whenever possible.
+                elif classification == 'GENERIC' and strategy_is_executable:
+                    row[col_name] = self.generic_sourcer.generate_value(strategy, expected_type=column.data_type)
                 else:
-                    # Fall back to inferred strategy
-                    inferred = self._infer_strategy(column)
+                    # Fall back to semantic strategy first, then inferred strategy.
+                    inferred = self.generic_sourcer.get_strategy_for_semantic(semantic_type, domain_use_case)
+                    if not inferred:
+                        inferred = self._infer_strategy(column)
                     if i == 0:
                         print(f"    [GEN] {col_name}: using inferred='{inferred}'")
-                    row[col_name] = self.generic_sourcer.generate_value(inferred)
+                    row[col_name] = self.generic_sourcer.generate_value(inferred, expected_type=column.data_type)
+
+            # If this table looks like a date dimension, force coherent calendar fields.
+            if date_dim_profile:
+                self._apply_date_dimension_rules(row, i, date_dim_profile)
+            
+            # Apply row-level realism rules after base generation.
+            self._apply_row_business_rules(table, row)
             
             rows.append(row)
-        
+
+        self._apply_storyspec_time_series(table, rows, table_semantics)
         return rows
     
-    def _get_search_real_values(self, column_name: str, strategy: str, num_needed: int) -> list[str]:
+    def _get_search_real_values(
+        self,
+        column_name: str,
+        strategy: str,
+        num_needed: int,
+        semantic_type: SemanticType = SemanticType.UNKNOWN,
+    ) -> list[str]:
         cache_key = f"search:{column_name}:{strategy}"
         
         if self.cache_enabled and self.cache:
@@ -396,12 +1028,20 @@ class LegitGenerator:
                 use_case=self.use_case
             )
         
+        values = self._sanitize_generated_values(values, column_name, semantic_type, num_needed)
+
         if self.cache_enabled and self.cache and values:
             self.cache.set_generated_values(cache_key, values)
         
         return values
     
-    def _get_ai_gen_values(self, column_name: str, strategy: str, num_needed: int) -> list[str]:
+    def _get_ai_gen_values(
+        self,
+        column_name: str,
+        strategy: str,
+        num_needed: int,
+        semantic_type: SemanticType = SemanticType.UNKNOWN,
+    ) -> list[str]:
         cache_key = f"aigen:{column_name}:{strategy}:{self.context.company_name}"
         
         if self.cache_enabled and self.cache:
@@ -425,14 +1065,587 @@ class LegitGenerator:
             if not values:
                 break
         
+        all_values = self._sanitize_generated_values(all_values, column_name, semantic_type, num_needed)
+
         if self.cache_enabled and self.cache and all_values:
             self.cache.set_generated_values(cache_key, all_values)
         
         return all_values
+
+    def _sanitize_generated_values(
+        self,
+        values: list[Any],
+        column_name: str,
+        semantic_type: SemanticType,
+        num_needed: int,
+    ) -> list[Any]:
+        """Normalize generated values to avoid obvious categorical junk."""
+        if not values:
+            fallback = self._get_story_domain_values(semantic_type)
+            if fallback:
+                return [fallback[i % len(fallback)] for i in range(num_needed)]
+            if semantic_type == SemanticType.PERSON_NAME:
+                return [self.generic_sourcer.generate_value("name") for _ in range(num_needed)]
+            if semantic_type in {SemanticType.ORG_NAME, SemanticType.BRANCH_NAME}:
+                return [self.generic_sourcer.generate_value("company") for _ in range(num_needed)]
+            return values
+
+        domain_values = self._get_story_domain_values(semantic_type)
+        strict_domain_semantics = {
+            SemanticType.CATEGORY,
+            SemanticType.SECTOR_NAME,
+            SemanticType.SECTOR_CATEGORY,
+            SemanticType.SEGMENT,
+            SemanticType.CHANNEL,
+            SemanticType.FUND_STRATEGY,
+            SemanticType.INVESTOR_TYPE,
+            SemanticType.INVESTMENT_STAGE,
+            SemanticType.COVENANT_STATUS,
+            SemanticType.DEBT_PERFORMANCE_STATUS,
+            SemanticType.PRODUCT_NAME,
+            SemanticType.BRAND_NAME,
+            SemanticType.DEPARTMENT_NAME,
+            SemanticType.BRANCH_NAME,
+            SemanticType.ORG_NAME,
+        }
+
+        def _looks_like_person_name(value: str) -> bool:
+            parts = [p for p in value.replace(".", " ").split() if p]
+            if len(parts) < 2:
+                return False
+            return all(re.fullmatch(r"[A-Za-z][A-Za-z'\-]*", p) for p in parts[:2])
+
+        if domain_values:
+            allowed = {v.lower() for v in domain_values}
+            cleaned = []
+            for idx, value in enumerate(values):
+                sval = str(value).strip()
+                upper = sval.upper()
+                is_number_suffixed = any(ch.isdigit() for ch in sval) and sval.lower() not in allowed
+                looks_placeholder = (
+                    not sval
+                    or sval.lower().endswith("_value")
+                    or sval.lower() == f"{column_name.lower()}_value"
+                    or upper in {"UNKNOWN", "N/A", "NA", "NULL"}
+                    or bool(self._GARBAGE_TOKEN_RE.match(upper))
+                )
+                needs_domain_value = semantic_type in strict_domain_semantics and sval.lower() not in allowed
+                if semantic_type == SemanticType.PERSON_NAME and not _looks_like_person_name(sval):
+                    cleaned.append(self.generic_sourcer.generate_value("name"))
+                elif looks_placeholder or (is_business_categorical(semantic_type) and is_number_suffixed) or needs_domain_value:
+                    cleaned.append(domain_values[idx % len(domain_values)])
+                else:
+                    cleaned.append(sval)
+            values = cleaned
+        elif semantic_type == SemanticType.PERSON_NAME:
+            values = [
+                (str(v).strip() if _looks_like_person_name(str(v).strip()) else self.generic_sourcer.generate_value("name"))
+                for v in values
+            ]
+
+        if len(values) < num_needed:
+            # Repeat without appending numeric suffixes (prevents "Central 7" style artifacts).
+            values = [values[i % len(values)] for i in range(num_needed)]
+        return values[:num_needed]
+
+    def _get_story_domain_values(self, semantic_type: SemanticType) -> list[str]:
+        """Get domain values with StorySpec overrides when available."""
+        base_values = list(get_domain_values(semantic_type, self._domain_use_case()))
+        if not self.story_spec:
+            return base_values
+
+        override_key_map = {
+            SemanticType.CATEGORY: "category",
+            SemanticType.SEGMENT: "segment",
+            SemanticType.CHANNEL: "channel",
+            SemanticType.ORG_NAME: "org_name",
+            SemanticType.PRODUCT_NAME: "product_name",
+            SemanticType.BRAND_NAME: "brand_name",
+            SemanticType.COUNTRY: "country",
+            SemanticType.REGION: "region",
+            SemanticType.CITY: "city",
+            SemanticType.BRANCH_NAME: "org_name",
+            SemanticType.DEPARTMENT_NAME: "category",
+        }
+        override_key = override_key_map.get(semantic_type)
+        if not override_key:
+            return base_values
+
+        override_values = list((self.story_spec.domain_overrides or {}).get(override_key, []))
+        if not override_values:
+            return base_values
+
+        # Keep story overrides inside the curated domain when one exists.
+        # This prevents use-case drift like legal demos inheriting noisy geo/org
+        # values from broad research context.
+        if base_values:
+            base_set = {str(v).strip().lower() for v in base_values if str(v).strip()}
+            filtered_override_values = [
+                str(v).strip()
+                for v in override_values
+                if str(v).strip() and str(v).strip().lower() in base_set
+            ]
+            if filtered_override_values:
+                override_values = filtered_override_values
+            else:
+                override_values = []
+
+        # Keep deterministic order and avoid duplicates.
+        seen = set()
+        merged = []
+        for value in override_values + base_values:
+            v = str(value).strip()
+            if not v:
+                continue
+            if v.lower() in seen:
+                continue
+            seen.add(v.lower())
+            merged.append(v)
+        return merged
+
+    def _choose_story_value(self, values: list[str], row_index: int) -> str:
+        if not values:
+            return ""
+        if len(values) == 1:
+            return values[0]
+        idx = (row_index + self._story_rng.randint(0, max(len(values) - 1, 0))) % len(values)
+        return values[idx]
+
+    def _apply_storyspec_time_series(
+        self,
+        table: Table,
+        rows: list[dict[str, Any]],
+        table_semantics: dict[str, SemanticType],
+    ) -> None:
+        """Apply smooth, deterministic trend profile plus bounded outliers."""
+        if not rows or not table.is_fact_table or not self.story_spec:
+            return
+
+        date_cols = [
+            c for c, s in table_semantics.items()
+            if s in {SemanticType.DATE_EVENT, SemanticType.DATE_START, SemanticType.DATE_END}
+        ]
+        if not date_cols:
+            return
+
+        date_col = None
+        for candidate in date_cols:
+            if any(self._to_python_date(r.get(candidate)) for r in rows):
+                date_col = candidate
+                break
+        if not date_col:
+            return
+
+        measure_cols = []
+        for col_name, semantic in table_semantics.items():
+            if semantic in {SemanticType.MONEY, SemanticType.QUANTITY, SemanticType.COUNT}:
+                measure_cols.append(col_name)
+
+        if not measure_cols:
+            for col_name in rows[0].keys():
+                low = col_name.lower()
+                if any(tok in low for tok in ("revenue", "sales", "amount", "quantity", "units", "count", "trx")):
+                    measure_cols.append(col_name)
+        measure_cols = list(dict.fromkeys(measure_cols))
+        if not measure_cols:
+            return
+
+        indexed_dates = []
+        for idx, row in enumerate(rows):
+            dt = self._to_python_date(row.get(date_col))
+            if dt:
+                indexed_dates.append((idx, dt))
+        if len(indexed_dates) < 4:
+            return
+
+        indexed_dates.sort(key=lambda x: x[1])
+        n = len(indexed_dates)
+
+        controls = self.story_spec
+        noise_band = max(0.0, float(controls.trend_profile.noise_band_pct))
+        seasonal_strength = max(0.0, float(controls.trend_profile.seasonal_strength))
+
+        if controls.trend_profile.style == "smooth":
+            slope = 0.18
+        elif controls.trend_profile.style == "flat":
+            slope = 0.06
+        else:
+            slope = 0.24
+
+        signals: dict[int, float] = {}
+        for rank, (idx, dt) in enumerate(indexed_dates):
+            progress = rank / max(n - 1, 1)
+            trend_factor = 1.0 + slope * progress
+            seasonal_factor = 1.0 + seasonal_strength * math.sin((2.0 * math.pi * (dt.month - 1)) / 12.0)
+            noise_factor = 1.0 + self._story_rng.uniform(-noise_band, noise_band)
+            signal = max(0.2, trend_factor * seasonal_factor * noise_factor)
+            signals[idx] = signal
+
+        # Controlled outlier injection (sparse and explainable).
+        budget = controls.outlier_budget
+        max_points = max(0, int(round(n * float(budget.max_points_pct))))
+        outlier_events = min(max_points, int(budget.max_events))
+
+        if outlier_events > 0:
+            candidate_pos = list(range(max(2, int(n * 0.1)), max(3, int(n * 0.9))))
+            self._story_rng.shuffle(candidate_pos)
+            for pos in candidate_pos[:outlier_events]:
+                idx, dt = indexed_dates[pos]
+                multiplier = self._story_rng.uniform(float(budget.event_multiplier_min), float(budget.event_multiplier_max))
+                signals[idx] = max(0.2, signals[idx] * multiplier)
+                self.story_events.append({
+                    "table": table.name,
+                    "date": dt.isoformat(),
+                    "row_index": idx,
+                    "kind": "intentional_outlier",
+                    "multiplier": round(multiplier, 4),
+                })
+
+        for col_name in measure_cols:
+            numeric_points = []
+            for idx, _ in indexed_dates:
+                value = rows[idx].get(col_name)
+                if isinstance(value, (int, float)) and not isinstance(value, bool):
+                    numeric_points.append(float(value))
+            if not numeric_points:
+                continue
+
+            numeric_points_sorted = sorted(numeric_points)
+            baseline = numeric_points_sorted[len(numeric_points_sorted) // 2]
+
+            for idx, _ in indexed_dates:
+                raw = rows[idx].get(col_name)
+                if not isinstance(raw, (int, float)) or isinstance(raw, bool):
+                    continue
+
+                low_name = col_name.lower()
+                signal = signals.get(idx, 1.0)
+
+                if any(tok in low_name for tok in ("quantity", "units", "count", "trx", "orders")):
+                    signal = 1.0 + (signal - 1.0) * 0.85
+
+                blended = (0.65 * float(raw)) + (0.35 * baseline)
+                adjusted = blended * signal
+
+                if not self.story_spec.value_guardrails.get("allow_negative_revenue", False):
+                    if any(tok in low_name for tok in ("revenue", "sales", "amount", "gross", "net")):
+                        adjusted = max(0.0, adjusted)
+
+                rows[idx][col_name] = int(round(adjusted)) if isinstance(raw, int) else round(adjusted, 2)
+
+    def _build_date_dimension_profile(
+        self,
+        table: Table,
+        table_semantics: dict[str, SemanticType],
+        num_rows: int,
+    ) -> dict[str, Any] | None:
+        """
+        Detect date-dimension tables and prepare deterministic calendar generation.
+        """
+        if num_rows <= 0:
+            return None
+
+        table_name_upper = table.name.upper()
+        has_full_date_col = any("FULL_DATE" == c.name.upper() or c.name.upper().endswith("_DATE") for c in table.columns)
+        has_calendar_attrs = any(
+            any(tok in c.name.upper() for tok in ("WEEKEND", "HOLIDAY", "SEASON", "EVENT"))
+            for c in table.columns
+        )
+        is_date_dimension = ("DATE" in table_name_upper and has_full_date_col) or (has_full_date_col and has_calendar_attrs)
+        if not is_date_dimension:
+            return None
+
+        today = datetime.now().date()
+        start_date = today - timedelta(days=max(num_rows - 1, 0))
+        return {
+            "start_date": start_date,
+            "date_key_columns": [c.name for c in table.columns if c.name.upper().endswith("DATE_KEY")],
+            "full_date_columns": [c.name for c in table.columns if "DATE" in c.name.upper()],
+            "month_name_columns": [c.name for c in table.columns if "MONTH" in c.name.upper() and "NAME" in c.name.upper()],
+            "quarter_name_columns": [c.name for c in table.columns if "QUARTER" in c.name.upper() and "NAME" in c.name.upper()],
+            "year_columns": [
+                c.name
+                for c in table.columns
+                if "YEAR" in c.name.upper() and "NAME" not in c.name.upper()
+            ],
+            "is_weekend_columns": [col_name for col_name in table_semantics.keys() if "WEEKEND" in col_name.upper()],
+            "is_holiday_columns": [c.name for c in table.columns if "HOLIDAY" in c.name.upper() and c.name.upper().startswith("IS_")],
+            "holiday_name_columns": [col_name for col_name, s in table_semantics.items() if s == SemanticType.HOLIDAY_NAME],
+            "season_columns": [col_name for col_name, s in table_semantics.items() if s == SemanticType.SEASON],
+            "event_columns": [col_name for col_name, s in table_semantics.items() if s == SemanticType.EVENT_NAME],
+        }
+
+    def _apply_date_dimension_rules(self, row: dict[str, Any], row_index: int, profile: dict[str, Any]) -> None:
+        """
+        Force coherent calendar rows (full date, weekend/holiday/season/event).
+        """
+        base_date = profile["start_date"] + timedelta(days=row_index)
+        month = base_date.month
+        weekday = base_date.weekday()  # 0=Mon, 6=Sun
+        is_weekend = weekday >= 5
+
+        # Lightweight US-holiday approximation.
+        fixed_holidays = {
+            (1, 1): "New Year's Day",
+            (7, 4): "Independence Day",
+            (12, 25): "Christmas Day",
+        }
+        holiday_name = fixed_holidays.get((month, base_date.day), "None")
+        is_holiday = holiday_name != "None"
+
+        def season_for_month(m: int) -> str:
+            if m in (12, 1, 2):
+                return "Winter"
+            if m in (3, 4, 5):
+                return "Spring"
+            if m in (6, 7, 8):
+                return "Summer"
+            return "Fall"
+
+        season_name = season_for_month(month)
+        if month == 11 and base_date.day >= 20:
+            event_name = "Black Friday Campaign"
+        elif month == 12:
+            event_name = "Holiday Promotions"
+        elif month in (8, 9):
+            event_name = "Back to School"
+        else:
+            event_name = "Regular Trading Day"
+
+        for col in profile.get("full_date_columns", []):
+            col_upper = col.upper()
+            if col_upper.endswith("DATE_KEY"):
+                row[col] = int(base_date.strftime("%Y%m%d"))
+            elif col_upper.endswith("_DATE") or col_upper == "FULL_DATE":
+                row[col] = base_date
+        for col in profile.get("date_key_columns", []):
+            row[col] = int(base_date.strftime("%Y%m%d"))
+        for col in profile.get("month_name_columns", []):
+            col_upper = col.upper()
+            if "ABBR" in col_upper or "SHORT" in col_upper:
+                row[col] = base_date.strftime("%b")
+            else:
+                row[col] = base_date.strftime("%B")
+        for col in profile.get("quarter_name_columns", []):
+            quarter = ((month - 1) // 3) + 1
+            row[col] = f"Q{quarter}"
+        for col in profile.get("year_columns", []):
+            row[col] = base_date.year
+        for col in profile.get("is_weekend_columns", []):
+            row[col] = is_weekend
+        for col in profile.get("is_holiday_columns", []):
+            row[col] = is_holiday
+        for col in profile.get("holiday_name_columns", []):
+            row[col] = holiday_name
+        for col in profile.get("season_columns", []):
+            row[col] = season_name
+        for col in profile.get("event_columns", []):
+            row[col] = event_name
+    
+    def _is_executable_generic_strategy(self, strategy: str) -> bool:
+        """Return True if strategy can be executed by GenericSourcer directly."""
+        if not strategy or not isinstance(strategy, str):
+            return False
+        strategy_type = strategy.split(":", 1)[0].strip().lower()
+        return strategy_type in self.EXECUTABLE_GENERIC_STRATEGIES
+    
+    def _to_python_date(self, value: Any) -> date | None:
+        """Best-effort conversion of generated date values to date objects."""
+        if isinstance(value, datetime):
+            return value.date()
+        if isinstance(value, date):
+            return value
+        if isinstance(value, str):
+            try:
+                return datetime.strptime(value[:10], "%Y-%m-%d").date()
+            except ValueError:
+                return None
+        return None
+    
+    @staticmethod
+    def _years_ago(base_date: date, years: int) -> date:
+        """Return a date shifted back by full years (leap-safe)."""
+        try:
+            return base_date.replace(year=base_date.year - years)
+        except ValueError:
+            return base_date.replace(month=2, day=28, year=base_date.year - years)
+
+    def _apply_story_seasonality_scale(self, base_multiplier: float) -> float:
+        if not self.story_spec:
+            return base_multiplier
+        strength = max(0.0, min(0.35, float(self.story_spec.trend_profile.seasonal_strength)))
+        # 0.10 is the legacy baseline amplitude used by earlier heuristics.
+        scale = strength / 0.10 if 0.10 > 0 else 1.0
+        return 1.0 + ((base_multiplier - 1.0) * scale)
+
+    def _resolve_seasonality_multiplier(self, event_date: date) -> float:
+        """Use-case aware seasonality to keep trend lines jumpy but believable."""
+        month = event_date.month
+        use_case_lower = (self.use_case or "").lower()
+        
+        if any(k in use_case_lower for k in ("retail", "sales", "ecommerce", "merch")):
+            by_month = {
+                1: 0.86, 2: 0.90, 3: 0.95, 4: 0.98, 5: 1.00, 6: 1.03,
+                7: 1.05, 8: 1.07, 9: 1.02, 10: 1.08, 11: 1.28, 12: 1.40,
+            }
+            return self._apply_story_seasonality_scale(by_month.get(month, 1.0))
+        
+        if "marketing" in use_case_lower:
+            by_month = {
+                1: 0.92, 2: 0.95, 3: 0.98, 4: 1.00, 5: 1.04, 6: 1.07,
+                7: 1.03, 8: 1.01, 9: 1.05, 10: 1.12, 11: 1.26, 12: 1.34,
+            }
+            return self._apply_story_seasonality_scale(by_month.get(month, 1.0))
+        
+        if "supply" in use_case_lower or "inventory" in use_case_lower:
+            by_month = {
+                1: 0.96, 2: 0.97, 3: 0.99, 4: 1.00, 5: 1.01, 6: 1.03,
+                7: 1.04, 8: 1.02, 9: 1.01, 10: 1.06, 11: 1.14, 12: 1.18,
+            }
+            return self._apply_story_seasonality_scale(by_month.get(month, 1.0))
+        
+        # Conservative default with mild Q4 lift.
+        return self._apply_story_seasonality_scale({10: 1.05, 11: 1.10, 12: 1.14}.get(month, 1.0))
+    
+    def _apply_row_business_rules(self, table: Table, row: dict[str, Any]) -> None:
+        """
+        Apply lightweight realism constraints and correlations.
+        Keeps data plausible without overfitting to one schema.
+        """
+        if not row:
+            return
+        
+        today = datetime.now().date()
+        
+        # 1) Birthdate guardrails: enforce realistic adult ages (18-95).
+        for key, value in list(row.items()):
+            col_lower = key.lower()
+            is_birth_col = (
+                ("birth" in col_lower and "date" in col_lower) or
+                col_lower == "dob" or
+                col_lower.endswith("_dob")
+            )
+            if not is_birth_col or value is None:
+                continue
+            
+            dob = self._to_python_date(value)
+            if not dob:
+                continue
+            
+            min_dob = self._years_ago(today, 95)
+            max_dob = self._years_ago(today, 18)
+            fixed = min(max(dob, min_dob), max_dob)
+            row[key] = fixed
+        
+        # 2) Keep non-birth dates from drifting into the future.
+        for key, value in list(row.items()):
+            col_lower = key.lower()
+            if "date" not in col_lower or "birth" in col_lower or col_lower == "dob" or col_lower.endswith("_dob"):
+                continue
+            dt_val = self._to_python_date(value)
+            if dt_val and dt_val > today:
+                row[key] = today
+
+        # 2b) Enforce simple start/end date ordering and LOS coherence.
+        start_key = next((k for k in row if any(tok in k.lower() for tok in ("start_date", "admission_date", "created_date", "begin_date"))), None)
+        end_key = next((k for k in row if any(tok in k.lower() for tok in ("end_date", "discharge_date", "closed_date", "resolved_date"))), None)
+        los_key = next((k for k in row if any(tok in k.lower() for tok in ("length_of_stay", "los_days", "stay_days"))), None)
+        if start_key and end_key:
+            start_dt = self._to_python_date(row.get(start_key))
+            end_dt = self._to_python_date(row.get(end_key))
+            if start_dt and end_dt and end_dt < start_dt:
+                if los_key and isinstance(row.get(los_key), (int, float)):
+                    days = max(0, int(round(float(row[los_key]))))
+                    row[end_key] = start_dt + timedelta(days=days)
+                else:
+                    row[end_key] = start_dt
+        if start_key and los_key and end_key:
+            start_dt = self._to_python_date(row.get(start_key))
+            if start_dt and isinstance(row.get(los_key), (int, float)):
+                days = max(0, int(round(float(row[los_key]))))
+                row[end_key] = start_dt + timedelta(days=days)
+        
+        # 3) Seasonal adjustments for fact-table measures.
+        event_date = None
+        for key, value in row.items():
+            col_lower = key.lower()
+            if "date" in col_lower and "birth" not in col_lower and col_lower != "dob" and not col_lower.endswith("_dob"):
+                event_date = self._to_python_date(value)
+                if event_date:
+                    break
+        
+        if table.is_fact_table and event_date:
+            seasonal = self._resolve_seasonality_multiplier(event_date) * self._story_rng.uniform(0.92, 1.08)
+            for key, value in list(row.items()):
+                if isinstance(value, bool) or not isinstance(value, (int, float)):
+                    continue
+                col_lower = key.lower()
+                if any(tok in col_lower for tok in ("revenue", "sales", "amount", "total", "gross", "net")):
+                    adjusted = max(0.0, float(value) * seasonal)
+                    row[key] = int(round(adjusted)) if isinstance(value, int) else round(adjusted, 2)
+                elif any(tok in col_lower for tok in ("quantity", "qty", "units", "count")):
+                    adjusted = max(1.0, float(value) * seasonal * self._story_rng.uniform(0.95, 1.05))
+                    row[key] = int(round(adjusted))
+
+        # 3b) Normalize state/region coherence within row when both exist.
+        state_key = next((k for k in row if "state" in k.lower() or "province" in k.lower()), None)
+        region_key = next((k for k in row if "region" in k.lower() or "territory" in k.lower()), None)
+        if state_key and region_key and row.get(state_key):
+            mapped = map_state_to_region(str(row.get(state_key)))
+            if mapped:
+                row[region_key] = mapped
+        
+        # 4) Enforce simple arithmetic consistency when possible.
+        quantity_key = next((k for k in row if any(tok in k.lower() for tok in ("quantity", "qty", "units"))), None)
+        price_key = next((k for k in row if "unit_price" in k.lower()), None)
+        if price_key is None:
+            price_key = next((k for k in row if k.lower().endswith("price")), None)
+        revenue_key = next(
+            (
+                k for k in row
+                if any(tok in k.lower() for tok in ("total_revenue", "sales_amount", "gross_revenue", "net_revenue", "total_amount", "revenue"))
+            ),
+            None
+        )
+        discount_key = next((k for k in row if any(tok in k.lower() for tok in ("discount_pct", "discount_rate"))), None)
+        
+        if quantity_key and price_key and revenue_key:
+            qty = row.get(quantity_key)
+            price = row.get(price_key)
+            revenue_val = row.get(revenue_key)
+            if isinstance(qty, (int, float)) and isinstance(price, (int, float)) and isinstance(revenue_val, (int, float)):
+                discount = 0.0
+                if discount_key and isinstance(row.get(discount_key), (int, float)):
+                    raw_discount = float(row[discount_key])
+                    discount = raw_discount if raw_discount <= 1 else raw_discount / 100.0
+                    discount = min(max(discount, 0.0), 0.8)
+                
+                computed = max(0.0, float(qty) * float(price) * (1.0 - discount))
+                computed *= self._story_rng.uniform(0.97, 1.03)  # mild variance
+                row[revenue_key] = int(round(computed)) if isinstance(revenue_val, int) else round(computed, 2)
+
+        # 5) Interest/rate sanity.
+        for key, value in list(row.items()):
+            col_lower = key.lower()
+            if not isinstance(value, (int, float)) or isinstance(value, bool):
+                continue
+            if "interest_rate" in col_lower or ("interest" in col_lower and "rate" in col_lower):
+                row[key] = round(min(max(float(value), 0.0), 1.0), 4)
+            elif "percent" in col_lower or col_lower.endswith("_pct"):
+                pct = float(value)
+                if 0 <= pct <= 1:
+                    pct = pct * 100.0
+                row[key] = round(min(max(pct, 0.0), 100.0), 2)
     
     def _infer_strategy(self, column: Column) -> str:
         col_lower = column.name.lower()
         data_type = column.data_type.upper() if column.data_type else ''
+        domain_use_case = self._domain_use_case()
+        semantic = infer_semantic_type(column.name, column.data_type, use_case=domain_use_case)
+
+        semantic_strategy = self.generic_sourcer.get_strategy_for_semantic(semantic, domain_use_case)
+        if semantic_strategy:
+            return semantic_strategy
         
         # Check if data type is numeric
         is_numeric = any(t in data_type for t in ('INT', 'NUMBER', 'NUMERIC', 'DECIMAL', 'BIGINT', 'SMALLINT'))
@@ -443,13 +1656,23 @@ class LegitGenerator:
         
         # Date/timestamp columns - use current date for end range
         # Calculate dynamic date ranges relative to today
-        from datetime import timedelta
         today_dt = datetime.now()
         today = today_dt.strftime('%Y-%m-%d')
         three_years_ago = (today_dt - timedelta(days=3*365)).strftime('%Y-%m-%d')
         two_years_ago = (today_dt - timedelta(days=2*365)).strftime('%Y-%m-%d')
         five_years_ago = (today_dt - timedelta(days=5*365)).strftime('%Y-%m-%d')
         
+        # Birth dates must be handled before generic date matching.
+        is_birth_col = (
+            ('birth' in col_lower and 'date' in col_lower) or
+            col_lower == 'dob' or
+            col_lower.endswith('_dob')
+        )
+        if is_birth_col:
+            oldest = self._years_ago(today_dt.date(), 95).strftime('%Y-%m-%d')
+            youngest = self._years_ago(today_dt.date(), 18).strftime('%Y-%m-%d')
+            return f"date_between:{oldest},{youngest}"
+        
         if 'date' in col_lower or 'DATE' in data_type or 'TIMESTAMP' in data_type:
             if 'created' in col_lower:
                 return f"date_between:{three_years_ago},{today}"

legitdata_project/legitdata/quality/__init__.py

@@ -0,0 +1,22 @@
+"""Quality validation and repair modules."""
+
+from .quality_spec import (
+    QualityIssue,
+    QualityReport,
+    QualitySpec,
+    QualityThresholds,
+    default_quality_spec,
+)
+from .validator import DataQualityValidator
+from .repair import DataQualityRepairEngine
+
+__all__ = [
+    "QualityIssue",
+    "QualityReport",
+    "QualitySpec",
+    "QualityThresholds",
+    "default_quality_spec",
+    "DataQualityValidator",
+    "DataQualityRepairEngine",
+]
+

legitdata_project/legitdata/quality/quality_spec.py

@@ -0,0 +1,149 @@
+"""Quality thresholds and report models."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any
+
+
+@dataclass
+class QualityThresholds:
+    max_categorical_junk: int = 0
+    max_fk_orphans: int = 0
+    max_temporal_violations: int = 0
+    max_numeric_violations: int = 0
+    min_semantic_pass_ratio: float = 0.95
+
+    # Story-quality thresholds
+    max_volatility_breaches: int = 0
+    min_smoothness_score: float = 0.70
+    min_outlier_explainability: float = 0.70
+    min_kpi_consistency: float = 0.80
+
+
+@dataclass
+class QualitySpec:
+    thresholds: QualityThresholds = field(default_factory=QualityThresholds)
+    max_repair_attempts: int = 2
+    sample_limit_per_issue: int = 12
+
+
+@dataclass
+class QualityIssue:
+    issue_type: str
+    table: str
+    column: str
+    severity: str
+    count: int
+    message: str
+    samples: list[Any] = field(default_factory=list)
+
+
+@dataclass
+class QualityReport:
+    passed: bool
+    generated_at: str
+    summary: dict[str, Any]
+    issues: list[QualityIssue] = field(default_factory=list)
+    repair_actions: list[str] = field(default_factory=list)
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "passed": self.passed,
+            "generated_at": self.generated_at,
+            "summary": self.summary,
+            "issues": [
+                {
+                    "issue_type": issue.issue_type,
+                    "table": issue.table,
+                    "column": issue.column,
+                    "severity": issue.severity,
+                    "count": issue.count,
+                    "message": issue.message,
+                    "samples": issue.samples,
+                }
+                for issue in self.issues
+            ],
+            "repair_actions": self.repair_actions,
+        }
+
+    def to_markdown(self) -> str:
+        lines = [
+            "# Data Quality Report",
+            "",
+            f"- Generated: {self.generated_at}",
+            f"- Passed: {'YES' if self.passed else 'NO'}",
+            f"- Semantic pass ratio: {self.summary.get('semantic_pass_ratio', 0):.3f}",
+            f"- Categorical junk: {self.summary.get('categorical_junk_count', 0)}",
+            f"- FK orphans: {self.summary.get('fk_orphan_count', 0)}",
+            f"- Temporal violations: {self.summary.get('temporal_violations', 0)}",
+            f"- Numeric violations: {self.summary.get('numeric_violations', 0)}",
+            f"- Volatility breaches: {self.summary.get('volatility_breaches', 0)}",
+            f"- Smoothness score: {self.summary.get('smoothness_score', 0):.3f}",
+            f"- Outlier explainability: {self.summary.get('outlier_explainability', 0):.3f}",
+            f"- KPI consistency: {self.summary.get('kpi_consistency', 0):.3f}",
+            "",
+        ]
+        if self.repair_actions:
+            lines.append("## Repair Actions")
+            lines.extend([f"- {a}" for a in self.repair_actions])
+            lines.append("")
+        if self.issues:
+            lines.append("## Issues")
+            for issue in self.issues:
+                lines.append(
+                    f"- [{issue.severity}] {issue.issue_type} {issue.table}.{issue.column} "
+                    f"count={issue.count} - {issue.message}"
+                )
+            lines.append("")
+        return "\n".join(lines)
+
+
+
+def default_quality_spec(story_controls: dict[str, Any] | None = None) -> QualitySpec:
+    spec = QualitySpec()
+    controls = story_controls or {}
+
+    quality_targets = controls.get("quality_targets", {}) if isinstance(controls, dict) else {}
+    min_sem = quality_targets.get("min_semantic_pass_ratio")
+    if min_sem is not None:
+        try:
+            spec.thresholds.min_semantic_pass_ratio = float(min_sem)
+        except (TypeError, ValueError):
+            pass
+
+    guardrails = controls.get("value_guardrails", {}) if isinstance(controls, dict) else {}
+    max_mom = guardrails.get("max_mom_change_pct")
+    if max_mom is not None:
+        # Allow a small number of breaches for sparse datasets.
+        try:
+            max_mom_f = float(max_mom)
+            if max_mom_f <= 20:
+                spec.thresholds.max_volatility_breaches = 0
+            elif max_mom_f <= 35:
+                spec.thresholds.max_volatility_breaches = 2
+            else:
+                spec.thresholds.max_volatility_breaches = 4
+        except (TypeError, ValueError):
+            pass
+
+    outlier_budget = controls.get("outlier_budget", {}) if isinstance(controls, dict) else {}
+    max_events = outlier_budget.get("max_events")
+    if max_events is not None:
+        try:
+            max_events_i = int(max_events)
+            if max_events_i <= 1:
+                spec.thresholds.min_outlier_explainability = 0.85
+            elif max_events_i <= 3:
+                spec.thresholds.min_outlier_explainability = 0.75
+            else:
+                spec.thresholds.min_outlier_explainability = 0.65
+        except (TypeError, ValueError):
+            pass
+
+    return spec
+
+
+def now_utc_iso() -> str:
+    return datetime.now(timezone.utc).isoformat()

legitdata_project/legitdata/quality/repair.py

@@ -0,0 +1,612 @@
+"""Targeted repair passes for generated data."""
+
+from __future__ import annotations
+
+import random
+from datetime import date, datetime, timedelta
+from typing import Any
+
+from ..ddl.models import Schema
+from ..domain import SemanticType, get_domain_values, infer_semantic_type, map_state_to_region
+
+
+def _to_datetime(value: Any) -> datetime | None:
+    if isinstance(value, datetime):
+        return value
+    if isinstance(value, date):
+        # Normalize plain dates so temporal repair can handle generator date objects.
+        return datetime.combine(value, datetime.min.time())
+    if isinstance(value, str):
+        try:
+            return datetime.strptime(value[:19], "%Y-%m-%d %H:%M:%S")
+        except ValueError:
+            try:
+                return datetime.strptime(value[:10], "%Y-%m-%d")
+            except ValueError:
+                return None
+    return None
+
+
+def _semantic_map(schema: Schema, use_case: str | None) -> dict[str, dict[str, SemanticType]]:
+    mapping: dict[str, dict[str, SemanticType]] = {}
+    for table in schema.tables:
+        table_map: dict[str, SemanticType] = {}
+        for col in table.columns:
+            table_map[col.name] = infer_semantic_type(col.name, col.data_type, table.name, use_case)
+        mapping[table.name] = table_map
+    return mapping
+
+
+class DataQualityRepairEngine:
+    """Applies bounded, deterministic repair operations."""
+
+    def repair_dataset(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        use_case: str | None = None,
+        story_controls: dict[str, Any] | None = None,
+    ) -> tuple[dict[str, list[dict]], list[str]]:
+        semantic_map = _semantic_map(schema, use_case)
+        actions: list[str] = []
+
+        actions.extend(self._repair_categorical(schema, generated_data, semantic_map, use_case))
+        actions.extend(self._repair_numeric(schema, generated_data, semantic_map))
+        actions.extend(self._repair_temporal(schema, generated_data, semantic_map))
+        actions.extend(self._repair_geography(schema, generated_data, semantic_map))
+        actions.extend(self._repair_fk_orphans(schema, generated_data))
+        actions.extend(self._repair_series_volatility(schema, generated_data, semantic_map, story_controls or {}))
+        actions.extend(self._repair_kpi_consistency(schema, generated_data, semantic_map))
+        actions.extend(self._repair_private_equity_financials(schema, generated_data))
+        actions.extend(self._repair_saas_finance_contract(generated_data))
+
+        return generated_data, actions
+
+    def _repair_saas_finance_contract(
+        self,
+        generated_data: dict[str, list[dict]],
+    ) -> list[str]:
+        actions: list[str] = []
+        fact_rows = generated_data.get("SAAS_CUSTOMER_MONTHLY", [])
+        spend_rows = generated_data.get("SALES_MARKETING_SPEND_MONTHLY", [])
+        if not fact_rows and not spend_rows:
+            return actions
+
+        fact_changes = 0
+        for row in fact_rows:
+            starting_arr = float(row.get("STARTING_ARR_USD") or 0.0)
+            new_logo_arr = float(row.get("NEW_LOGO_ARR_USD") or 0.0)
+            expansion_arr = float(row.get("EXPANSION_ARR_USD") or 0.0)
+            contraction_arr = float(row.get("CONTRACTION_ARR_USD") or 0.0)
+            churned_arr = float(row.get("CHURNED_ARR_USD") or 0.0)
+            ending_arr = round(max(starting_arr + new_logo_arr + expansion_arr - contraction_arr - churned_arr, 0.0), 2)
+            mrr_usd = round(ending_arr / 12.0, 2)
+            gm_pct = round(max(45.0, min(92.0, float(row.get("GROSS_MARGIN_PCT") or 78.0))), 2)
+            gross_margin_usd = round(mrr_usd * (gm_pct / 100.0), 2)
+            support_cost_usd = round(max(mrr_usd - gross_margin_usd, mrr_usd * 0.06), 2)
+            acquired_flag = bool(row.get("CUSTOMER_ACQUIRED_FLAG")) and new_logo_arr > 0.0
+            churned_flag = bool(row.get("CUSTOMER_CHURNED_FLAG")) and ending_arr == 0.0 and churned_arr > 0.0
+
+            updates = {
+                "ENDING_ARR_USD": ending_arr,
+                "MRR_USD": mrr_usd,
+                "GROSS_MARGIN_PCT": gm_pct,
+                "GROSS_MARGIN_USD": gross_margin_usd,
+                "SUPPORT_COST_USD": support_cost_usd,
+                "CUSTOMER_ACQUIRED_FLAG": acquired_flag,
+                "CUSTOMER_CHURNED_FLAG": churned_flag,
+            }
+            for key, new_value in updates.items():
+                if row.get(key) != new_value:
+                    row[key] = new_value
+                    fact_changes += 1
+
+        spend_changes = 0
+        for row in spend_rows:
+            sales = round(float(row.get("SALES_SPEND_USD") or 0.0), 2)
+            marketing = round(float(row.get("MARKETING_SPEND_USD") or 0.0), 2)
+            total = round(sales + marketing, 2)
+            if row.get("TOTAL_S_AND_M_SPEND_USD") != total:
+                row["TOTAL_S_AND_M_SPEND_USD"] = total
+                spend_changes += 1
+            new_customers = int(row.get("NEW_CUSTOMERS_ACQUIRED") or 0)
+            if new_customers <= 0:
+                new_logo_arr = round(max(float(row.get("NEW_LOGO_ARR_USD") or 0.0), 0.0), 2)
+                if row.get("NEW_LOGO_ARR_USD") != new_logo_arr:
+                    row["NEW_LOGO_ARR_USD"] = new_logo_arr
+                    spend_changes += 1
+
+        if fact_changes:
+            actions.append(f"SAAS_CUSTOMER_MONTHLY: recalculated {fact_changes} SaaS finance metric values")
+        if spend_changes:
+            actions.append(f"SALES_MARKETING_SPEND_MONTHLY: repaired {spend_changes} spend identity values")
+        return actions
+
+    def _repair_categorical(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        semantic_map: dict[str, dict[str, SemanticType]],
+        use_case: str | None,
+    ) -> list[str]:
+        actions: list[str] = []
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+            for col_name, semantic in semantic_map.get(table.name, {}).items():
+                if semantic not in {
+                    SemanticType.REGION,
+                    SemanticType.CATEGORY,
+                    SemanticType.SECTOR_NAME,
+                    SemanticType.SECTOR_CATEGORY,
+                    SemanticType.SEGMENT,
+                    SemanticType.TIER,
+                    SemanticType.STATUS,
+                    SemanticType.FUND_STRATEGY,
+                    SemanticType.INVESTOR_TYPE,
+                    SemanticType.INVESTMENT_STAGE,
+                    SemanticType.COVENANT_STATUS,
+                    SemanticType.DEBT_PERFORMANCE_STATUS,
+                    SemanticType.CHANNEL,
+                    SemanticType.COUNTRY,
+                    SemanticType.STATE,
+                    SemanticType.CITY,
+                    SemanticType.POSTAL_CODE,
+                    SemanticType.PRODUCT_NAME,
+                    SemanticType.BRAND_NAME,
+                    SemanticType.DEPARTMENT_NAME,
+                    SemanticType.BRANCH_NAME,
+                    SemanticType.ORG_NAME,
+                    SemanticType.PERSON_NAME,
+                }:
+                    continue
+                domain_values = get_domain_values(semantic, use_case)
+                if not domain_values:
+                    continue
+                strict_domain_semantics = {
+                    SemanticType.CATEGORY,
+                    SemanticType.SECTOR_NAME,
+                    SemanticType.SECTOR_CATEGORY,
+                    SemanticType.SEGMENT,
+                    SemanticType.CHANNEL,
+                    SemanticType.FUND_STRATEGY,
+                    SemanticType.INVESTOR_TYPE,
+                    SemanticType.INVESTMENT_STAGE,
+                    SemanticType.COVENANT_STATUS,
+                    SemanticType.DEBT_PERFORMANCE_STATUS,
+                    SemanticType.PRODUCT_NAME,
+                    SemanticType.BRAND_NAME,
+                    SemanticType.DEPARTMENT_NAME,
+                    SemanticType.BRANCH_NAME,
+                    SemanticType.ORG_NAME,
+                }
+                allowed = {d.lower() for d in domain_values}
+                changed = 0
+                for idx, row in enumerate(rows):
+                    current = row.get(col_name)
+                    if current is None:
+                        row[col_name] = domain_values[idx % len(domain_values)]
+                        changed += 1
+                        continue
+                    sval = str(current).strip().lower()
+                    needs_repair = (
+                        (any(ch.isdigit() for ch in sval) and sval not in allowed)
+                        or (semantic in strict_domain_semantics and sval not in allowed)
+                    )
+                    if needs_repair:
+                        row[col_name] = domain_values[idx % len(domain_values)]
+                        changed += 1
+                if changed:
+                    actions.append(f"{table.name}.{col_name}: normalized {changed} categorical values")
+        return actions
+
+    def _repair_numeric(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        semantic_map: dict[str, dict[str, SemanticType]],
+    ) -> list[str]:
+        actions: list[str] = []
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+            for col_name, semantic in semantic_map.get(table.name, {}).items():
+                changed = 0
+                for row in rows:
+                    value = row.get(col_name)
+                    if isinstance(value, bool) or not isinstance(value, (int, float)):
+                        continue
+                    v = float(value)
+                    fixed = value
+                    if semantic == SemanticType.INTEREST_RATE:
+                        fixed = min(max(v, 0.0), 1.0)
+                    elif semantic == SemanticType.RETURN_RATE:
+                        fixed = min(max(v, -0.25), 0.75)
+                    elif semantic == SemanticType.RETURN_MULTIPLE:
+                        fixed = min(max(v, 0.0), 10.0)
+                    elif semantic == SemanticType.BASIS_POINTS:
+                        fixed = min(max(v, -1000.0), 5000.0)
+                    elif semantic == SemanticType.LEVERAGE_RATIO:
+                        fixed = min(max(v, 0.0), 20.0)
+                    elif semantic == SemanticType.PERCENT:
+                        if 0 <= v <= 1:
+                            fixed = v * 100.0
+                        fixed = min(max(float(fixed), 0.0), 100.0)
+                    elif semantic == SemanticType.DURATION_DAYS:
+                        fixed = min(max(v, 0.0), 3650.0)
+                    elif semantic == SemanticType.QUANTITY:
+                        fixed = min(max(v, 0.0), 1_000_000.0)
+                    elif semantic == SemanticType.COUNT:
+                        fixed = min(max(v, 0.0), 100_000_000.0)
+                    elif semantic == SemanticType.SCORE:
+                        fixed = min(max(v, 0.0), 1000.0)
+                    elif semantic == SemanticType.MONEY:
+                        fixed = min(max(v, -100_000_000.0), 100_000_000.0)
+
+                    if fixed != value:
+                        row[col_name] = int(round(fixed)) if isinstance(value, int) else round(float(fixed), 4)
+                        changed += 1
+                if changed:
+                    actions.append(f"{table.name}.{col_name}: clamped {changed} numeric values")
+        return actions
+
+    def _repair_temporal(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        semantic_map: dict[str, dict[str, SemanticType]],
+    ) -> list[str]:
+        actions: list[str] = []
+        today = datetime.now()
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+            table_sem = semantic_map.get(table.name, {})
+            date_starts = [c for c, s in table_sem.items() if s == SemanticType.DATE_START]
+            date_ends = [c for c, s in table_sem.items() if s == SemanticType.DATE_END]
+            date_births = [c for c, s in table_sem.items() if s == SemanticType.DATE_BIRTH]
+            event_dates = [c for c, s in table_sem.items() if s == SemanticType.DATE_EVENT]
+            changed = 0
+            for row in rows:
+                for col in date_births:
+                    dt = _to_datetime(row.get(col))
+                    if not dt:
+                        continue
+                    age = today.year - dt.year
+                    if dt > today or age < 18:
+                        row[col] = (today - timedelta(days=random.randint(18 * 365, 70 * 365))).date()
+                        changed += 1
+                for col in event_dates:
+                    dt = _to_datetime(row.get(col))
+                    if dt and dt > today:
+                        row[col] = today.date()
+                        changed += 1
+                for start_col in date_starts:
+                    start_dt = _to_datetime(row.get(start_col))
+                    if not start_dt:
+                        continue
+                    if start_dt > today:
+                        row[start_col] = today.date()
+                        start_dt = _to_datetime(row.get(start_col)) or start_dt
+                        changed += 1
+                    for end_col in date_ends:
+                        end_dt = _to_datetime(row.get(end_col))
+                        if not end_dt:
+                            continue
+                        if end_dt > today:
+                            row[end_col] = today.date()
+                            end_dt = _to_datetime(row.get(end_col)) or end_dt
+                            changed += 1
+                        if end_dt < start_dt:
+                            fixed_end = start_dt + timedelta(days=random.randint(0, 14))
+                            if fixed_end > today:
+                                fixed_end = today
+                            row[end_col] = fixed_end.date()
+                            changed += 1
+            if changed:
+                actions.append(f"{table.name}: repaired {changed} temporal violations")
+        return actions
+
+    def _repair_geography(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        semantic_map: dict[str, dict[str, SemanticType]],
+    ) -> list[str]:
+        actions: list[str] = []
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+            table_sem = semantic_map.get(table.name, {})
+            state_cols = [c for c, s in table_sem.items() if s == SemanticType.STATE]
+            region_cols = [c for c, s in table_sem.items() if s == SemanticType.REGION]
+            if not state_cols or not region_cols:
+                continue
+            changed = 0
+            for row in rows:
+                state_val = row.get(state_cols[0])
+                if not state_val:
+                    continue
+                expected_region = map_state_to_region(str(state_val))
+                if not expected_region:
+                    continue
+                for region_col in region_cols:
+                    if str(row.get(region_col, "")).strip().lower() != expected_region.lower():
+                        row[region_col] = expected_region
+                        changed += 1
+            if changed:
+                actions.append(f"{table.name}: fixed {changed} state/region mismatches")
+        return actions
+
+    def _repair_fk_orphans(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+    ) -> list[str]:
+        actions: list[str] = []
+        parent_values: dict[tuple[str, str], list[Any]] = {}
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            for col in table.columns:
+                vals = [row.get(col.name) for row in rows if row.get(col.name) is not None]
+                parent_values[(table.name.upper(), col.name.upper())] = vals
+
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+            for fk in table.foreign_keys:
+                allowed = parent_values.get((fk.references_table.upper(), fk.references_column.upper()), [])
+                if not allowed:
+                    continue
+                changed = 0
+                for row in rows:
+                    value = row.get(fk.column_name)
+                    if value is not None and value not in allowed:
+                        row[fk.column_name] = random.choice(allowed)
+                        changed += 1
+                if changed:
+                    actions.append(
+                        f"{table.name}.{fk.column_name}: remapped {changed} orphan FK values "
+                        f"to {fk.references_table}.{fk.references_column}"
+                    )
+        return actions
+
+    def _repair_series_volatility(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        semantic_map: dict[str, dict[str, SemanticType]],
+        story_controls: dict[str, Any],
+    ) -> list[str]:
+        actions: list[str] = []
+        guardrails = story_controls.get("value_guardrails", {}) if isinstance(story_controls, dict) else {}
+        max_mom = float(guardrails.get("max_mom_change_pct", 35.0)) / 100.0
+        # Smoothness scoring requires median change well below the raw guardrail.
+        # Use a tighter internal repair band so repaired series can pass both
+        # volatility and smoothness checks.
+        target_mom = max(0.05, max_mom * 0.35)
+
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+
+            table_sem = semantic_map.get(table.name, {})
+            date_cols = [c for c, s in table_sem.items() if s in {SemanticType.DATE_EVENT, SemanticType.DATE_START, SemanticType.DATE_END}]
+            measure_cols = [c for c, s in table_sem.items() if s in {SemanticType.MONEY, SemanticType.QUANTITY, SemanticType.COUNT}]
+            if not date_cols or not measure_cols:
+                continue
+
+            date_col = date_cols[0]
+            dated_rows = []
+            for row in rows:
+                dval = _to_datetime(row.get(date_col))
+                if dval:
+                    dated_rows.append((dval, row))
+            dated_rows.sort(key=lambda x: x[0])
+            if len(dated_rows) < 4:
+                continue
+
+            # Work at month granularity so repairs target trend volatility
+            # rather than row-level transaction noise.
+            month_rows: dict[str, list[dict[str, Any]]] = {}
+            for dval, row in dated_rows:
+                month_key = f"{dval.year:04d}-{dval.month:02d}"
+                month_rows.setdefault(month_key, []).append(row)
+
+            changed = 0
+            for col in measure_cols:
+                month_means: list[tuple[str, float]] = []
+                for month_key in sorted(month_rows.keys()):
+                    vals = [
+                        float(r[col])
+                        for r in month_rows[month_key]
+                        if isinstance(r.get(col), (int, float)) and not isinstance(r.get(col), bool)
+                    ]
+                    if vals:
+                        month_means.append((month_key, sum(vals) / len(vals)))
+                if len(month_means) < 2:
+                    continue
+
+                prev_target = None
+                for month_key, month_mean in month_means:
+                    target = month_mean
+                    if prev_target is not None and prev_target > 0:
+                        low = max(0.0, prev_target * (1.0 - target_mom))
+                        high = prev_target * (1.0 + target_mom)
+                        if target < low:
+                            target = low
+                        elif target > high:
+                            target = high
+
+                    if month_mean > 0 and abs(target - month_mean) > 1e-9:
+                        scale = target / month_mean
+                        for row in month_rows.get(month_key, []):
+                            value = row.get(col)
+                            if not isinstance(value, (int, float)) or isinstance(value, bool):
+                                continue
+                            new_value = float(value) * scale
+                            row[col] = int(round(new_value)) if isinstance(value, int) else round(new_value, 2)
+                            changed += 1
+                    prev_target = target
+
+            if changed:
+                actions.append(f"{table.name}: smoothed {changed} volatility points")
+
+        return actions
+
+    def _repair_kpi_consistency(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        semantic_map: dict[str, dict[str, SemanticType]],
+    ) -> list[str]:
+        actions: list[str] = []
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+
+            cols = list(semantic_map.get(table.name, {}).keys())
+            qty_key = next((k for k in cols if any(t in k.lower() for t in ("quantity", "qty", "units"))), None)
+            price_key = next((k for k in cols if "price" in k.lower()), None)
+            revenue_key = next((k for k in cols if any(t in k.lower() for t in ("revenue", "sales_amount", "total_amount", "gross", "net"))), None)
+            if not qty_key or not price_key or not revenue_key:
+                continue
+
+            changed = 0
+            for row in rows:
+                qty = row.get(qty_key)
+                price = row.get(price_key)
+                revenue = row.get(revenue_key)
+                if not isinstance(qty, (int, float)) or not isinstance(price, (int, float)) or not isinstance(revenue, (int, float)):
+                    continue
+                expected = float(qty) * float(price)
+                if expected <= 0:
+                    continue
+                rel_err = abs(float(revenue) - expected) / max(expected, 1.0)
+                if rel_err > 0.25:
+                    row[revenue_key] = int(round(expected)) if isinstance(revenue, int) else round(expected, 2)
+                    changed += 1
+
+            if changed:
+                actions.append(f"{table.name}: repaired {changed} KPI consistency rows")
+
+        return actions
+
+    def _repair_private_equity_financials(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+    ) -> list[str]:
+        actions: list[str] = []
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            if not rows:
+                continue
+
+            col_map = {col.name.lower(): col.name for col in table.columns}
+            total_value_col = col_map.get("total_value_usd") or col_map.get("total_value")
+            reported_value_col = col_map.get("reported_value_usd") or col_map.get("reported_value")
+            distributions_col = col_map.get("distributions_usd") or col_map.get("distributions")
+            gross_irr_col = col_map.get("gross_irr")
+            net_irr_col = col_map.get("net_irr")
+            gross_without_subline_col = col_map.get("gross_irr_without_sub_line")
+            irr_impact_bps_col = col_map.get("irr_sub_line_impact_bps")
+            total_return_multiple_col = col_map.get("total_return_multiple")
+            dpi_multiple_col = col_map.get("dpi_multiple")
+            rvpi_multiple_col = col_map.get("rvpi_multiple")
+            revenue_col = col_map.get("revenue_usd") or col_map.get("revenue")
+            ebitda_col = col_map.get("ebitda_usd") or col_map.get("ebitda")
+            ebitda_margin_col = col_map.get("ebitda_margin_pct")
+            net_debt_col = col_map.get("net_debt_usd") or col_map.get("net_debt")
+            debt_to_ebitda_col = col_map.get("debt_to_ebitda_ratio")
+
+            changed = 0
+            for row in rows:
+                if (
+                    total_value_col
+                    and reported_value_col
+                    and distributions_col
+                    and isinstance(row.get(reported_value_col), (int, float))
+                    and isinstance(row.get(distributions_col), (int, float))
+                ):
+                    computed_total = float(row[reported_value_col]) + float(row[distributions_col])
+                    current_total = row.get(total_value_col)
+                    if not isinstance(current_total, (int, float)) or abs(float(current_total) - computed_total) > 0.01:
+                        row[total_value_col] = round(computed_total, 2)
+                        changed += 1
+
+                if (
+                    gross_irr_col
+                    and net_irr_col
+                    and isinstance(row.get(gross_irr_col), (int, float))
+                    and isinstance(row.get(net_irr_col), (int, float))
+                    and float(row[net_irr_col]) > float(row[gross_irr_col])
+                ):
+                    row[net_irr_col] = round(float(row[gross_irr_col]) - 0.01, 4)
+                    changed += 1
+
+                if (
+                    gross_irr_col
+                    and gross_without_subline_col
+                    and irr_impact_bps_col
+                    and isinstance(row.get(gross_irr_col), (int, float))
+                    and isinstance(row.get(gross_without_subline_col), (int, float))
+                ):
+                    impact_bps = round((float(row[gross_irr_col]) - float(row[gross_without_subline_col])) * 10000, 0)
+                    current_bps = row.get(irr_impact_bps_col)
+                    if not isinstance(current_bps, (int, float)) or abs(float(current_bps) - impact_bps) > 1:
+                        row[irr_impact_bps_col] = int(impact_bps)
+                        changed += 1
+
+                if (
+                    total_return_multiple_col
+                    and dpi_multiple_col
+                    and rvpi_multiple_col
+                    and isinstance(row.get(dpi_multiple_col), (int, float))
+                    and isinstance(row.get(rvpi_multiple_col), (int, float))
+                ):
+                    tvpi = float(row[dpi_multiple_col]) + float(row[rvpi_multiple_col])
+                    current_tvpi = row.get(total_return_multiple_col)
+                    if not isinstance(current_tvpi, (int, float)) or abs(float(current_tvpi) - tvpi) > 0.01:
+                        row[total_return_multiple_col] = round(tvpi, 3)
+                        changed += 1
+
+                if (
+                    revenue_col
+                    and ebitda_col
+                    and ebitda_margin_col
+                    and isinstance(row.get(revenue_col), (int, float))
+                    and isinstance(row.get(ebitda_col), (int, float))
+                    and float(row[revenue_col]) > 0
+                ):
+                    margin_pct = (float(row[ebitda_col]) / float(row[revenue_col])) * 100.0
+                    current_margin = row.get(ebitda_margin_col)
+                    if not isinstance(current_margin, (int, float)) or abs(float(current_margin) - margin_pct) > 0.25:
+                        row[ebitda_margin_col] = round(margin_pct, 2)
+                        changed += 1
+
+                if (
+                    net_debt_col
+                    and ebitda_col
+                    and debt_to_ebitda_col
+                    and isinstance(row.get(net_debt_col), (int, float))
+                    and isinstance(row.get(ebitda_col), (int, float))
+                    and float(row[ebitda_col]) > 0
+                ):
+                    leverage = float(row[net_debt_col]) / float(row[ebitda_col])
+                    current_leverage = row.get(debt_to_ebitda_col)
+                    if not isinstance(current_leverage, (int, float)) or abs(float(current_leverage) - leverage) > 0.05:
+                        row[debt_to_ebitda_col] = round(leverage, 3)
+                        changed += 1
+
+            if changed:
+                actions.append(f"{table.name}: repaired {changed} private-equity financial identity values")
+
+        return actions

legitdata_project/legitdata/quality/validator.py

@@ -0,0 +1,935 @@
+"""Post-generation validator for demo data quality."""
+
+from __future__ import annotations
+
+import math
+import re
+from collections import defaultdict
+from datetime import date, datetime
+from typing import Any
+
+from ..ddl.models import Schema
+from ..domain import (
+    SemanticType,
+    get_domain_values,
+    infer_semantic_type,
+    is_business_categorical,
+    map_state_to_region,
+)
+from .quality_spec import QualityIssue, QualityReport, QualitySpec, now_utc_iso
+
+
+JUNK_PATTERN = re.compile(
+    r"^(?:region|category|segment|tier|status)\s*\d+$|^[A-Za-z]+\s+\d+$",
+    re.IGNORECASE,
+)
+PERSON_NAME_PATTERN = re.compile(r"^[A-Za-z][A-Za-z'\-]*\s+[A-Za-z][A-Za-z'\-]*(?:\s+[A-Za-z][A-Za-z'\-]*)*$")
+
+
+def _to_date(value: Any) -> date | None:
+    if isinstance(value, datetime):
+        return value.date()
+    if isinstance(value, date):
+        return value
+    if isinstance(value, str):
+        try:
+            return datetime.strptime(value[:10], "%Y-%m-%d").date()
+        except ValueError:
+            return None
+    return None
+
+
+def _is_numeric(value: Any) -> bool:
+    if isinstance(value, bool):
+        return False
+    return isinstance(value, (int, float))
+
+
+def _numeric_violation(semantic: SemanticType, value: Any) -> bool:
+    if not _is_numeric(value):
+        return False
+    v = float(value)
+    if math.isnan(v) or math.isinf(v):
+        return True
+    if semantic == SemanticType.INTEREST_RATE:
+        return v < 0 or v > 1.0
+    if semantic == SemanticType.RETURN_RATE:
+        return v < -0.25 or v > 0.75
+    if semantic == SemanticType.RETURN_MULTIPLE:
+        return v < 0 or v > 10.0
+    if semantic == SemanticType.BASIS_POINTS:
+        return v < -1000 or v > 5000
+    if semantic == SemanticType.LEVERAGE_RATIO:
+        return v < 0 or v > 20.0
+    if semantic == SemanticType.PERCENT:
+        if 0 <= v <= 1:
+            return False
+        return not (0 <= v <= 100)
+    if semantic == SemanticType.QUANTITY:
+        return v < 0 or v > 1_000_000
+    if semantic == SemanticType.COUNT:
+        return v < 0 or v > 100_000_000
+    if semantic == SemanticType.SCORE:
+        return v < 0 or v > 1000
+    if semantic == SemanticType.DURATION_DAYS:
+        return v < 0 or v > 3650
+    if semantic == SemanticType.MONEY:
+        return abs(v) > 100_000_000
+    return False
+
+
+def _find_pct_changes(values: list[float]) -> list[float]:
+    pct_changes: list[float] = []
+    for i in range(1, len(values)):
+        prev = values[i - 1]
+        cur = values[i]
+        if prev == 0:
+            continue
+        pct_changes.append((cur - prev) / abs(prev))
+    return pct_changes
+
+
+def _month_bucket(dval: date) -> str:
+    return f"{dval.year:04d}-{dval.month:02d}"
+
+
+def _series_by_month(dated_rows: list[tuple[date, dict[str, Any]]], col_name: str) -> list[float]:
+    buckets: dict[str, list[float]] = {}
+    for dval, row in dated_rows:
+        value = row.get(col_name)
+        if not _is_numeric(value):
+            continue
+        buckets.setdefault(_month_bucket(dval), []).append(float(value))
+    if not buckets:
+        return []
+    series: list[float] = []
+    for month in sorted(buckets.keys()):
+        vals = buckets[month]
+        if vals:
+            series.append(sum(vals) / len(vals))
+    return series
+
+
+class DataQualityValidator:
+    """Validates generated row data before warehouse write."""
+
+    def __init__(self, quality_spec: QualitySpec):
+        self.spec = quality_spec
+
+    def _semantic_map(self, schema: Schema, use_case: str | None) -> dict[str, dict[str, SemanticType]]:
+        mapping: dict[str, dict[str, SemanticType]] = {}
+        for table in schema.tables:
+            table_map: dict[str, SemanticType] = {}
+            for col in table.columns:
+                table_map[col.name] = infer_semantic_type(col.name, col.data_type, table.name, use_case)
+            mapping[table.name] = table_map
+        return mapping
+
+    def _validate_saas_finance_contract(
+        self,
+        generated_data: dict[str, list[dict]],
+    ) -> tuple[list[QualityIssue], int, int, int, dict[str, Any]]:
+        """Validate recurring-revenue accounting identities for the gold-path schema."""
+        fact_rows = generated_data.get("SAAS_CUSTOMER_MONTHLY", [])
+        spend_rows = generated_data.get("SALES_MARKETING_SPEND_MONTHLY", [])
+        if not fact_rows:
+            return [], 0, 0, 0, {}
+
+        issues: list[QualityIssue] = []
+        kpi_checks = 0
+        kpi_passes = 0
+        temporal_violations = 0
+
+        identity_failures = 0
+        continuity_failures = 0
+        retention_failures = 0
+        spend_failures = 0
+
+        customer_months: dict[Any, list[int]] = defaultdict(list)
+        retention_buckets: dict[tuple[int, str], dict[str, float]] = defaultdict(
+            lambda: {
+                "starting": 0.0,
+                "expansion": 0.0,
+                "contraction": 0.0,
+                "churned": 0.0,
+            }
+        )
+
+        for row in fact_rows:
+            month_key = int(row.get("MONTH_KEY") or 0)
+            customer_key = row.get("CUSTOMER_KEY")
+            segment = None
+            starting_arr = float(row.get("STARTING_ARR_USD") or 0.0)
+            new_logo_arr = float(row.get("NEW_LOGO_ARR_USD") or 0.0)
+            expansion_arr = float(row.get("EXPANSION_ARR_USD") or 0.0)
+            contraction_arr = float(row.get("CONTRACTION_ARR_USD") or 0.0)
+            churned_arr = float(row.get("CHURNED_ARR_USD") or 0.0)
+            ending_arr = float(row.get("ENDING_ARR_USD") or 0.0)
+            mrr_usd = float(row.get("MRR_USD") or 0.0)
+            gm_pct = float(row.get("GROSS_MARGIN_PCT") or 0.0)
+            gm_usd = float(row.get("GROSS_MARGIN_USD") or 0.0)
+            acquired_flag = bool(row.get("CUSTOMER_ACQUIRED_FLAG"))
+            churned_flag = bool(row.get("CUSTOMER_CHURNED_FLAG"))
+
+            kpi_checks += 1
+            expected_ending = starting_arr + new_logo_arr + expansion_arr - contraction_arr - churned_arr
+            if abs(ending_arr - expected_ending) <= 1.0:
+                kpi_passes += 1
+            else:
+                identity_failures += 1
+
+            kpi_checks += 1
+            if abs((mrr_usd * 12.0) - ending_arr) <= 12.0:
+                kpi_passes += 1
+            else:
+                identity_failures += 1
+
+            recognized_revenue = ending_arr / 12.0
+            kpi_checks += 1
+            if recognized_revenue <= 0 or abs(gm_usd - (recognized_revenue * (gm_pct / 100.0))) <= max(50.0, recognized_revenue * 0.15):
+                kpi_passes += 1
+            else:
+                identity_failures += 1
+
+            kpi_checks += 1
+            if (acquired_flag and new_logo_arr > 0) or (not acquired_flag):
+                kpi_passes += 1
+            else:
+                identity_failures += 1
+
+            kpi_checks += 1
+            if (churned_flag and ending_arr == 0.0) or (not churned_flag):
+                kpi_passes += 1
+            else:
+                identity_failures += 1
+
+            if customer_key is not None and month_key:
+                customer_months[customer_key].append(month_key)
+
+        customer_segments = {
+            row.get("CUSTOMER_KEY"): str(row.get("SEGMENT") or "")
+            for row in generated_data.get("CUSTOMERS", [])
+        }
+        for row in fact_rows:
+            month_key = int(row.get("MONTH_KEY") or 0)
+            customer_key = row.get("CUSTOMER_KEY")
+            segment = customer_segments.get(customer_key, "")
+            bucket = retention_buckets[(month_key, segment)]
+            bucket["starting"] += float(row.get("STARTING_ARR_USD") or 0.0)
+            bucket["expansion"] += float(row.get("EXPANSION_ARR_USD") or 0.0)
+            bucket["contraction"] += float(row.get("CONTRACTION_ARR_USD") or 0.0)
+            bucket["churned"] += float(row.get("CHURNED_ARR_USD") or 0.0)
+
+        for month_keys in customer_months.values():
+            ordered = sorted(set(month_keys))
+            for prev, cur in zip(ordered, ordered[1:]):
+                prev_year, prev_month = divmod(prev // 100, 100)
+                cur_year, cur_month = divmod(cur // 100, 100)
+                month_gap = (cur_year - prev_year) * 12 + (cur_month - prev_month)
+                if month_gap > 1:
+                    continuity_failures += 1
+                    temporal_violations += 1
+
+        for (month_key, segment), bucket in retention_buckets.items():
+            starting = bucket["starting"]
+            if starting <= 0:
+                continue
+            nrr = (starting + bucket["expansion"] - bucket["contraction"] - bucket["churned"]) / starting
+            grr = (starting - bucket["contraction"] - bucket["churned"]) / starting
+            kpi_checks += 2
+            if 0.70 <= nrr <= 1.30:
+                kpi_passes += 1
+            else:
+                retention_failures += 1
+            if 0.60 <= grr <= 1.05:
+                kpi_passes += 1
+            else:
+                retention_failures += 1
+
+        for row in spend_rows:
+            sales = float(row.get("SALES_SPEND_USD") or 0.0)
+            marketing = float(row.get("MARKETING_SPEND_USD") or 0.0)
+            total = float(row.get("TOTAL_S_AND_M_SPEND_USD") or 0.0)
+            new_customers = int(row.get("NEW_CUSTOMERS_ACQUIRED") or 0)
+            new_logo_arr = float(row.get("NEW_LOGO_ARR_USD") or 0.0)
+            nrr_pct = float(row.get("NRR_PCT") or 0.0)
+            grr_pct = float(row.get("GRR_PCT") or 0.0)
+            cac_usd = float(row.get("CAC_USD") or 0.0)
+
+            kpi_checks += 1
+            if abs(total - (sales + marketing)) <= 1.0:
+                kpi_passes += 1
+            else:
+                spend_failures += 1
+
+            kpi_checks += 1
+            if (new_customers == 0 and new_logo_arr >= 0.0) or (new_customers > 0 and new_logo_arr > 0.0):
+                kpi_passes += 1
+            else:
+                spend_failures += 1
+
+            kpi_checks += 2
+            if 70.0 <= nrr_pct <= 130.0:
+                kpi_passes += 1
+            else:
+                spend_failures += 1
+            if 60.0 <= grr_pct <= 105.0:
+                kpi_passes += 1
+            else:
+                spend_failures += 1
+
+            kpi_checks += 1
+            if (new_customers == 0 and cac_usd == 0.0) or (new_customers > 0 and cac_usd > 0.0):
+                kpi_passes += 1
+            else:
+                spend_failures += 1
+
+        if identity_failures:
+            issues.append(
+                QualityIssue(
+                    issue_type="saas_finance_identity",
+                    table="SAAS_CUSTOMER_MONTHLY",
+                    column="ENDING_ARR_USD",
+                    severity="high",
+                    count=identity_failures,
+                    message="Recurring-revenue identities do not reconcile",
+                    samples=[],
+                )
+            )
+        if continuity_failures:
+            issues.append(
+                QualityIssue(
+                    issue_type="saas_finance_continuity",
+                    table="SAAS_CUSTOMER_MONTHLY",
+                    column="MONTH_KEY",
+                    severity="high",
+                    count=continuity_failures,
+                    message="Detected missing customer-month continuity in recurring-revenue history",
+                    samples=[],
+                )
+            )
+        if retention_failures:
+            issues.append(
+                QualityIssue(
+                    issue_type="saas_finance_retention",
+                    table="SAAS_CUSTOMER_MONTHLY",
+                    column="NRR/GRR",
+                    severity="high",
+                    count=retention_failures,
+                    message="NRR or GRR fell outside the allowed range for one or more segment-months",
+                    samples=[],
+                )
+            )
+        if spend_failures:
+            issues.append(
+                QualityIssue(
+                    issue_type="saas_finance_spend",
+                    table="SALES_MARKETING_SPEND_MONTHLY",
+                    column="TOTAL_S_AND_M_SPEND_USD",
+                    severity="high",
+                    count=spend_failures,
+                    message="Sales and marketing spend rows failed CAC/spend integrity checks",
+                    samples=[],
+                )
+            )
+
+        summary = {
+            "saas_finance_identity_failures": identity_failures,
+            "saas_finance_continuity_failures": continuity_failures,
+            "saas_finance_retention_failures": retention_failures,
+            "saas_finance_spend_failures": spend_failures,
+        }
+        return issues, kpi_checks, kpi_passes, temporal_violations, summary
+
+    def validate(
+        self,
+        schema: Schema,
+        generated_data: dict[str, list[dict]],
+        use_case: str | None = None,
+        story_controls: dict[str, Any] | None = None,
+    ) -> QualityReport:
+        semantic_map = self._semantic_map(schema, use_case)
+        issues: list[QualityIssue] = []
+
+        categorical_junk_count = 0
+        fk_orphan_count = 0
+        temporal_violations = 0
+        numeric_violations = 0
+        semantic_checks = 0
+        semantic_passes = 0
+
+        # Story-quality metrics
+        volatility_breaches = 0
+        smoothness_scores: list[float] = []
+        outlier_candidates = 0
+        outlier_points_total = 0
+        kpi_checks = 0
+        kpi_passes = 0
+
+        controls = story_controls or {}
+        guardrails = controls.get("value_guardrails", {}) if isinstance(controls, dict) else {}
+        max_mom_change_pct = float(guardrails.get("max_mom_change_pct", 35))
+        outlier_budget = controls.get("outlier_budget", {}) if isinstance(controls, dict) else {}
+        max_outlier_points_pct = float(outlier_budget.get("max_points_pct", 0.02))
+        max_outlier_events = int(outlier_budget.get("max_events", 2))
+
+        today = datetime.now().date()
+
+        # Build parent value sets for FK integrity.
+        parent_values: dict[tuple[str, str], set[Any]] = {}
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            for col in table.columns:
+                values = {row.get(col.name) for row in rows if row.get(col.name) is not None}
+                parent_values[(table.name.upper(), col.name.upper())] = values
+
+        for table in schema.tables:
+            rows = generated_data.get(table.name, [])
+            table_semantics = semantic_map.get(table.name, {})
+
+            # Column-level checks.
+            for col in table.columns:
+                semantic = table_semantics.get(col.name, SemanticType.UNKNOWN)
+                values = [row.get(col.name) for row in rows if row.get(col.name) is not None]
+                if not values:
+                    continue
+
+                if is_business_categorical(semantic):
+                    semantic_checks += len(values)
+                    junk_values = []
+                    allowed = {v.lower() for v in get_domain_values(semantic, use_case)}
+                    strict_domain_semantics = {
+                        SemanticType.CATEGORY,
+                        SemanticType.SECTOR_NAME,
+                        SemanticType.SECTOR_CATEGORY,
+                        SemanticType.SEGMENT,
+                        SemanticType.CHANNEL,
+                        SemanticType.FUND_STRATEGY,
+                        SemanticType.INVESTOR_TYPE,
+                        SemanticType.INVESTMENT_STAGE,
+                        SemanticType.COVENANT_STATUS,
+                        SemanticType.DEBT_PERFORMANCE_STATUS,
+                        SemanticType.PRODUCT_NAME,
+                        SemanticType.BRAND_NAME,
+                        SemanticType.DEPARTMENT_NAME,
+                        SemanticType.BRANCH_NAME,
+                        SemanticType.ORG_NAME,
+                    }
+                    for value in values:
+                        sval = str(value).strip()
+                        is_junk = (
+                            (JUNK_PATTERN.match(sval) and sval.lower() not in allowed)
+                            or (semantic in strict_domain_semantics and allowed and sval.lower() not in allowed)
+                        )
+                        if is_junk:
+                            junk_values.append(sval)
+                        else:
+                            semantic_passes += 1
+                    if junk_values:
+                        categorical_junk_count += len(junk_values)
+                        issues.append(
+                            QualityIssue(
+                                issue_type="categorical_junk",
+                                table=table.name,
+                                column=col.name,
+                                severity="high",
+                                count=len(junk_values),
+                                message="Detected synthetic categorical values",
+                                samples=junk_values[: self.spec.sample_limit_per_issue],
+                            )
+                        )
+
+                if semantic == SemanticType.PERSON_NAME:
+                    semantic_checks += len(values)
+                    bad_person_names = []
+                    for value in values:
+                        sval = str(value).strip()
+                        if PERSON_NAME_PATTERN.match(sval):
+                            semantic_passes += 1
+                        else:
+                            bad_person_names.append(sval)
+                    if bad_person_names:
+                        categorical_junk_count += len(bad_person_names)
+                        issues.append(
+                            QualityIssue(
+                                issue_type="categorical_junk",
+                                table=table.name,
+                                column=col.name,
+                                severity="high",
+                                count=len(bad_person_names),
+                                message="Detected unrealistic person-name values",
+                                samples=bad_person_names[: self.spec.sample_limit_per_issue],
+                            )
+                        )
+
+                if semantic in {
+                    SemanticType.MONEY,
+                    SemanticType.QUANTITY,
+                    SemanticType.COUNT,
+                    SemanticType.PERCENT,
+                    SemanticType.INTEREST_RATE,
+                    SemanticType.RETURN_RATE,
+                    SemanticType.RETURN_MULTIPLE,
+                    SemanticType.BASIS_POINTS,
+                    SemanticType.LEVERAGE_RATIO,
+                    SemanticType.SCORE,
+                    SemanticType.DURATION_DAYS,
+                }:
+                    semantic_checks += len(values)
+                    bad = [v for v in values if _numeric_violation(semantic, v)]
+                    numeric_violations += len(bad)
+                    semantic_passes += len(values) - len(bad)
+                    if bad:
+                        issues.append(
+                            QualityIssue(
+                                issue_type="numeric_range",
+                                table=table.name,
+                                column=col.name,
+                                severity="high",
+                                count=len(bad),
+                                message=f"Values outside plausible range for {semantic.value}",
+                                samples=bad[: self.spec.sample_limit_per_issue],
+                            )
+                        )
+
+            # Row-level temporal and geo checks.
+            for row in rows:
+                start_dates = []
+                end_dates = []
+                state_value = None
+                region_value = None
+                reference_date = None
+                month_name_values: list[str] = []
+                quarter_name_values: list[str] = []
+                year_values: list[int] = []
+
+                for col in table.columns:
+                    value = row.get(col.name)
+                    if value is None:
+                        continue
+                    col_lower = col.name.lower()
+                    semantic = table_semantics.get(col.name, SemanticType.UNKNOWN)
+                    if semantic == SemanticType.DATE_START:
+                        dt = _to_date(value)
+                        if dt:
+                            start_dates.append((col.name, dt))
+                            if reference_date is None:
+                                reference_date = dt
+                            if dt > today:
+                                temporal_violations += 1
+                    elif semantic == SemanticType.DATE_END:
+                        dt = _to_date(value)
+                        if dt:
+                            end_dates.append((col.name, dt))
+                            if reference_date is None:
+                                reference_date = dt
+                            if dt > today:
+                                temporal_violations += 1
+                    elif semantic == SemanticType.DATE_EVENT:
+                        dt = _to_date(value)
+                        if dt:
+                            if reference_date is None:
+                                reference_date = dt
+                            if dt > today:
+                                temporal_violations += 1
+                    elif semantic == SemanticType.DATE_BIRTH:
+                        dt = _to_date(value)
+                        if dt and (dt > today or (today.year - dt.year) < 18):
+                            temporal_violations += 1
+                    elif semantic == SemanticType.STATE:
+                        state_value = value
+                    elif semantic == SemanticType.REGION:
+                        region_value = value
+
+                    if "month" in col_lower and "name" in col_lower:
+                        month_name_values.append(str(value).strip())
+                    elif "quarter" in col_lower and "name" in col_lower:
+                        quarter_name_values.append(str(value).strip())
+                    elif "year" in col_lower and "name" not in col_lower and _is_numeric(value):
+                        year_values.append(int(float(value)))
+
+                for _, start_dt in start_dates:
+                    for _, end_dt in end_dates:
+                        if end_dt < start_dt:
+                            temporal_violations += 1
+
+                if reference_date:
+                    expected_month_full = reference_date.strftime("%B").lower()
+                    expected_month_abbr = reference_date.strftime("%b").lower()
+                    expected_quarter = f"q{((reference_date.month - 1) // 3) + 1}"
+                    expected_year = reference_date.year
+
+                    for month_value in month_name_values:
+                        normalized = month_value.lower()
+                        if not (normalized.startswith(expected_month_full) or normalized.startswith(expected_month_abbr)):
+                            temporal_violations += 1
+
+                    for quarter_value in quarter_name_values:
+                        normalized = quarter_value.lower().replace(" ", "")
+                        if not (normalized.startswith(expected_quarter) or normalized == expected_quarter.replace("q", "")):
+                            temporal_violations += 1
+
+                    for year_value in year_values:
+                        if year_value != expected_year:
+                            temporal_violations += 1
+
+                if state_value and region_value:
+                    expected_region = map_state_to_region(str(state_value))
+                    if expected_region and str(region_value).strip().lower() != expected_region.lower():
+                        issues.append(
+                            QualityIssue(
+                                issue_type="geo_mismatch",
+                                table=table.name,
+                                column="region",
+                                severity="medium",
+                                count=1,
+                                message=f"State/region mismatch ({state_value} -> {region_value})",
+                                samples=[{"state": state_value, "region": region_value, "expected": expected_region}],
+                            )
+                        )
+
+            # FK integrity checks.
+            for fk in table.foreign_keys:
+                parent_key = (fk.references_table.upper(), fk.references_column.upper())
+                allowed = parent_values.get(parent_key, set())
+                if not allowed:
+                    continue
+                missing = 0
+                for row in rows:
+                    value = row.get(fk.column_name)
+                    if value is None:
+                        continue
+                    if value not in allowed:
+                        missing += 1
+                if missing:
+                    fk_orphan_count += missing
+                    issues.append(
+                        QualityIssue(
+                            issue_type="fk_orphan",
+                            table=table.name,
+                            column=fk.column_name,
+                            severity="critical",
+                            count=missing,
+                            message=f"Foreign key not found in {fk.references_table}.{fk.references_column}",
+                            samples=[],
+                        )
+                    )
+
+            # Story metrics: smoothness + volatility + KPI consistency.
+            date_cols = [c for c, s in table_semantics.items() if s in {SemanticType.DATE_EVENT, SemanticType.DATE_START, SemanticType.DATE_END}]
+            measure_cols = [c for c, s in table_semantics.items() if s in {SemanticType.MONEY, SemanticType.QUANTITY, SemanticType.COUNT}]
+
+            if date_cols and measure_cols and rows:
+                date_col = date_cols[0]
+                dated_rows = []
+                for row in rows:
+                    dval = _to_date(row.get(date_col))
+                    if dval:
+                        dated_rows.append((dval, row))
+                dated_rows.sort(key=lambda x: x[0])
+
+                if len(dated_rows) >= 4:
+                    for col_name in measure_cols:
+                        # Evaluate volatility/smoothness on monthly aggregates to avoid
+                        # false positives from row-level transaction jitter.
+                        series = _series_by_month(dated_rows, col_name)
+                        if len(series) < 4:
+                            continue
+                        pct_changes = _find_pct_changes(series)
+                        if not pct_changes:
+                            continue
+
+                        outlier_points_total += len(pct_changes)
+                        abs_changes = [abs(v) for v in pct_changes]
+
+                        breaches = sum(1 for v in abs_changes if (v * 100.0) > max_mom_change_pct)
+                        volatility_breaches += breaches
+
+                        median_abs = sorted(abs_changes)[len(abs_changes) // 2]
+                        max_band = max(max_mom_change_pct / 100.0, 0.01)
+                        smoothness = max(0.0, 1.0 - (median_abs / (max_band * 1.4)))
+                        smoothness_scores.append(smoothness)
+
+                        outlier_candidates += sum(1 for v in abs_changes if v > 0.80)
+
+            qty_key = next((k for k in table_semantics.keys() if any(t in k.lower() for t in ("quantity", "qty", "units"))), None)
+            price_key = next((k for k in table_semantics.keys() if "price" in k.lower()), None)
+            revenue_key = next((k for k in table_semantics.keys() if any(t in k.lower() for t in ("revenue", "sales_amount", "total_amount", "gross", "net"))), None)
+            if qty_key and price_key and revenue_key and rows:
+                bad_kpi = 0
+                for row in rows:
+                    qty = row.get(qty_key)
+                    price = row.get(price_key)
+                    revenue = row.get(revenue_key)
+                    if not (_is_numeric(qty) and _is_numeric(price) and _is_numeric(revenue)):
+                        continue
+                    expected = float(qty) * float(price)
+                    if expected <= 0:
+                        continue
+                    rel_err = abs(float(revenue) - expected) / max(expected, 1.0)
+                    kpi_checks += 1
+                    if rel_err <= 0.25:
+                        kpi_passes += 1
+                    else:
+                        bad_kpi += 1
+                if bad_kpi:
+                    issues.append(
+                        QualityIssue(
+                            issue_type="kpi_consistency",
+                            table=table.name,
+                            column=revenue_key,
+                            severity="medium",
+                            count=bad_kpi,
+                            message="Revenue deviates from quantity x price beyond tolerance",
+                            samples=[],
+                        )
+                    )
+
+            col_map = {col.name.lower(): col.name for col in table.columns}
+
+            total_value_key = col_map.get("total_value_usd") or col_map.get("total_value")
+            reported_value_key = col_map.get("reported_value_usd") or col_map.get("reported_value")
+            distributions_key = col_map.get("distributions_usd") or col_map.get("distributions")
+            gross_irr_key = col_map.get("gross_irr")
+            net_irr_key = col_map.get("net_irr")
+            gross_without_subline_key = col_map.get("gross_irr_without_sub_line")
+            irr_impact_bps_key = col_map.get("irr_sub_line_impact_bps")
+            total_return_multiple_key = col_map.get("total_return_multiple")
+            dpi_multiple_key = col_map.get("dpi_multiple")
+            rvpi_multiple_key = col_map.get("rvpi_multiple")
+            revenue_metric_key = col_map.get("revenue_usd") or col_map.get("revenue")
+            ebitda_key = col_map.get("ebitda_usd") or col_map.get("ebitda")
+            ebitda_margin_key = col_map.get("ebitda_margin_pct")
+            net_debt_key = col_map.get("net_debt_usd") or col_map.get("net_debt")
+            debt_to_ebitda_key = col_map.get("debt_to_ebitda_ratio")
+
+            pe_identity_failures: dict[str, int] = {
+                "total_value_identity": 0,
+                "net_vs_gross_irr": 0,
+                "irr_bps_identity": 0,
+                "tvpi_identity": 0,
+                "ebitda_margin_identity": 0,
+                "debt_to_ebitda_identity": 0,
+            }
+
+            for row in rows:
+                if (
+                    total_value_key
+                    and reported_value_key
+                    and distributions_key
+                    and _is_numeric(row.get(total_value_key))
+                    and _is_numeric(row.get(reported_value_key))
+                    and _is_numeric(row.get(distributions_key))
+                ):
+                    expected_total = float(row[reported_value_key]) + float(row[distributions_key])
+                    rel_err = abs(float(row[total_value_key]) - expected_total) / max(abs(expected_total), 1.0)
+                    kpi_checks += 1
+                    if rel_err <= 0.02:
+                        kpi_passes += 1
+                    else:
+                        pe_identity_failures["total_value_identity"] += 1
+
+                if gross_irr_key and net_irr_key and _is_numeric(row.get(gross_irr_key)) and _is_numeric(row.get(net_irr_key)):
+                    kpi_checks += 1
+                    if float(row[net_irr_key]) <= float(row[gross_irr_key]):
+                        kpi_passes += 1
+                    else:
+                        pe_identity_failures["net_vs_gross_irr"] += 1
+
+                if (
+                    gross_irr_key
+                    and gross_without_subline_key
+                    and irr_impact_bps_key
+                    and _is_numeric(row.get(gross_irr_key))
+                    and _is_numeric(row.get(gross_without_subline_key))
+                    and _is_numeric(row.get(irr_impact_bps_key))
+                ):
+                    expected_bps = (float(row[gross_irr_key]) - float(row[gross_without_subline_key])) * 10000
+                    kpi_checks += 1
+                    if abs(float(row[irr_impact_bps_key]) - expected_bps) <= 5.0:
+                        kpi_passes += 1
+                    else:
+                        pe_identity_failures["irr_bps_identity"] += 1
+
+                if (
+                    total_return_multiple_key
+                    and dpi_multiple_key
+                    and rvpi_multiple_key
+                    and _is_numeric(row.get(total_return_multiple_key))
+                    and _is_numeric(row.get(dpi_multiple_key))
+                    and _is_numeric(row.get(rvpi_multiple_key))
+                ):
+                    expected_tvpi = float(row[dpi_multiple_key]) + float(row[rvpi_multiple_key])
+                    kpi_checks += 1
+                    if abs(float(row[total_return_multiple_key]) - expected_tvpi) <= 0.05:
+                        kpi_passes += 1
+                    else:
+                        pe_identity_failures["tvpi_identity"] += 1
+
+                if (
+                    revenue_metric_key
+                    and ebitda_key
+                    and ebitda_margin_key
+                    and _is_numeric(row.get(revenue_metric_key))
+                    and _is_numeric(row.get(ebitda_key))
+                    and _is_numeric(row.get(ebitda_margin_key))
+                    and float(row[revenue_metric_key]) > 0
+                ):
+                    expected_margin = (float(row[ebitda_key]) / float(row[revenue_metric_key])) * 100.0
+                    kpi_checks += 1
+                    if abs(float(row[ebitda_margin_key]) - expected_margin) <= 0.5:
+                        kpi_passes += 1
+                    else:
+                        pe_identity_failures["ebitda_margin_identity"] += 1
+
+                if (
+                    net_debt_key
+                    and ebitda_key
+                    and debt_to_ebitda_key
+                    and _is_numeric(row.get(net_debt_key))
+                    and _is_numeric(row.get(ebitda_key))
+                    and _is_numeric(row.get(debt_to_ebitda_key))
+                    and float(row[ebitda_key]) > 0
+                ):
+                    expected_ratio = float(row[net_debt_key]) / float(row[ebitda_key])
+                    kpi_checks += 1
+                    if abs(float(row[debt_to_ebitda_key]) - expected_ratio) <= 0.1:
+                        kpi_passes += 1
+                    else:
+                        pe_identity_failures["debt_to_ebitda_identity"] += 1
+
+            pe_issue_messages = {
+                "total_value_identity": "Total value deviates from reported value plus distributions",
+                "net_vs_gross_irr": "Net IRR exceeds gross IRR",
+                "irr_bps_identity": "IRR basis-point impact does not match gross vs unsubsidized IRR delta",
+                "tvpi_identity": "Total return multiple deviates from DPI plus RVPI",
+                "ebitda_margin_identity": "EBITDA margin deviates from EBITDA divided by revenue",
+                "debt_to_ebitda_identity": "Debt-to-EBITDA ratio deviates from net debt divided by EBITDA",
+            }
+            pe_issue_columns = {
+                "total_value_identity": total_value_key or "*",
+                "net_vs_gross_irr": net_irr_key or "*",
+                "irr_bps_identity": irr_impact_bps_key or "*",
+                "tvpi_identity": total_return_multiple_key or "*",
+                "ebitda_margin_identity": ebitda_margin_key or "*",
+                "debt_to_ebitda_identity": debt_to_ebitda_key or "*",
+            }
+            for issue_name, failure_count in pe_identity_failures.items():
+                if failure_count:
+                    issues.append(
+                        QualityIssue(
+                            issue_type=issue_name,
+                            table=table.name,
+                            column=pe_issue_columns[issue_name],
+                            severity="high",
+                            count=failure_count,
+                            message=pe_issue_messages[issue_name],
+                            samples=[],
+                        )
+                    )
+
+        semantic_pass_ratio = (semantic_passes / semantic_checks) if semantic_checks else 1.0
+
+        finance_summary: dict[str, Any] = {}
+        if "SAAS_CUSTOMER_MONTHLY" in generated_data:
+            finance_issues, extra_kpi_checks, extra_kpi_passes, finance_temporal_violations, finance_summary = (
+                self._validate_saas_finance_contract(generated_data)
+            )
+            issues.extend(finance_issues)
+            kpi_checks += extra_kpi_checks
+            kpi_passes += extra_kpi_passes
+            temporal_violations += finance_temporal_violations
+
+        smoothness_score = (sum(smoothness_scores) / len(smoothness_scores)) if smoothness_scores else 1.0
+        allowed_outliers = int(outlier_points_total * max_outlier_points_pct) + max_outlier_events
+        if outlier_candidates <= allowed_outliers:
+            outlier_explainability = 1.0
+        else:
+            outlier_explainability = allowed_outliers / max(outlier_candidates, 1)
+        kpi_consistency = (kpi_passes / kpi_checks) if kpi_checks else 1.0
+
+        thresholds = self.spec.thresholds
+
+        if volatility_breaches > thresholds.max_volatility_breaches:
+            issues.append(
+                QualityIssue(
+                    issue_type="volatility",
+                    table="*",
+                    column="*",
+                    severity="high",
+                    count=volatility_breaches,
+                    message=f"Detected {volatility_breaches} period-over-period volatility breaches",
+                    samples=[],
+                )
+            )
+        if smoothness_score < thresholds.min_smoothness_score:
+            issues.append(
+                QualityIssue(
+                    issue_type="smoothness",
+                    table="*",
+                    column="*",
+                    severity="high",
+                    count=1,
+                    message=f"Smoothness score {smoothness_score:.3f} below threshold",
+                    samples=[],
+                )
+            )
+        if outlier_explainability < thresholds.min_outlier_explainability:
+            issues.append(
+                QualityIssue(
+                    issue_type="outlier_explainability",
+                    table="*",
+                    column="*",
+                    severity="medium",
+                    count=1,
+                    message=f"Outlier explainability {outlier_explainability:.3f} below threshold",
+                    samples=[],
+                )
+            )
+        if kpi_consistency < thresholds.min_kpi_consistency:
+            issues.append(
+                QualityIssue(
+                    issue_type="kpi_consistency",
+                    table="*",
+                    column="*",
+                    severity="medium",
+                    count=1,
+                    message=f"KPI consistency {kpi_consistency:.3f} below threshold",
+                    samples=[],
+                )
+            )
+
+        passed = (
+            categorical_junk_count <= thresholds.max_categorical_junk
+            and fk_orphan_count <= thresholds.max_fk_orphans
+            and temporal_violations <= thresholds.max_temporal_violations
+            and numeric_violations <= thresholds.max_numeric_violations
+            and semantic_pass_ratio >= thresholds.min_semantic_pass_ratio
+            and volatility_breaches <= thresholds.max_volatility_breaches
+            and smoothness_score >= thresholds.min_smoothness_score
+            and outlier_explainability >= thresholds.min_outlier_explainability
+            and kpi_consistency >= thresholds.min_kpi_consistency
+        )
+
+        summary = {
+            "categorical_junk_count": categorical_junk_count,
+            "fk_orphan_count": fk_orphan_count,
+            "temporal_violations": temporal_violations,
+            "numeric_violations": numeric_violations,
+            "semantic_checks": semantic_checks,
+            "semantic_passes": semantic_passes,
+            "semantic_pass_ratio": semantic_pass_ratio,
+            "volatility_breaches": volatility_breaches,
+            "smoothness_score": smoothness_score,
+            "outlier_candidates": outlier_candidates,
+            "outlier_explainability": outlier_explainability,
+            "kpi_checks": kpi_checks,
+            "kpi_consistency": kpi_consistency,
+        }
+        summary.update(finance_summary)
+        return QualityReport(
+            passed=passed,
+            generated_at=now_utc_iso(),
+            summary=summary,
+            issues=issues,
+        )

legitdata_project/legitdata/relationships/fk_manager.py

@@ -62,10 +62,9 @@ class FKManager:
         table_key = references_table.upper()
         col_key = references_column.upper()
         
-        if table_key not in self._pk_values:
-            return None
-        
-        if col_key not in self._pk_values[table_key]:
+        table_pk_map = self._pk_values.get(table_key, {})
+
+        if col_key not in table_pk_map:
             # Referenced column is not the PK - look it up from registered rows
             rows = self._table_rows.get(table_key, [])
             if rows:
@@ -81,12 +80,12 @@ class FKManager:
             
             # Fallback to PK only if it's the same type situation
             # (This is the old behavior - may return wrong type)
-            if self._pk_values[table_key]:
-                col_key = list(self._pk_values[table_key].keys())[0]
+            if table_pk_map:
+                col_key = list(table_pk_map.keys())[0]
             else:
                 return None
         
-        values = self._pk_values[table_key][col_key]
+        values = table_pk_map.get(col_key, [])
         if not values:
             return None
         

legitdata_project/legitdata/sourcer/ai_generator.py

@@ -4,6 +4,7 @@ import json
 import re
 from typing import Optional
 from ..analyzer.context_builder import CompanyContext
+from ..domain import get_domain_values, infer_semantic_type
 
 
 class AIGenerator:
@@ -167,37 +168,19 @@ Return ONLY a JSON array of strings. Example: ["value1", "value2", "value3"]"""
     
     def _generate_fallback(self, column_name: str, num_values: int) -> list[str]:
         """Generate fallback values when AI is unavailable."""
-        col_lower = column_name.lower()
-        
-        # Common fallback values based on column type
-        if 'region' in col_lower:
-            base = ["North", "South", "East", "West", "Central", "Northeast", "Southeast", "Northwest", "Southwest", "Midwest"]
-        elif 'country' in col_lower:
-            base = ["United States", "Canada", "United Kingdom", "Germany", "France", "Japan", "Australia", "Brazil", "India", "Mexico"]
-        elif 'status' in col_lower:
-            base = ["Active", "Inactive", "Pending", "Completed", "Cancelled"]
-        elif 'tier' in col_lower:
-            base = ["Premium", "Standard", "Basic", "Enterprise", "Starter"]
-        elif 'type' in col_lower:
-            base = ["Type A", "Type B", "Type C", "Standard", "Special"]
-        elif 'segment' in col_lower:
-            base = ["Enterprise", "SMB", "Consumer", "Government", "Education"]
-        elif 'category' in col_lower:
-            base = ["Category 1", "Category 2", "Category 3", "Category 4", "Category 5"]
-        elif 'group' in col_lower:
-            base = ["Group A", "Group B", "Group C", "Group D", "Group E"]
-        else:
-            base = [f"{column_name}_{i}" for i in range(1, num_values + 1)]
-        
-        # Extend if needed
-        result = base.copy()
-        while len(result) < num_values:
-            for i, v in enumerate(base):
-                result.append(f"{v} {len(result) // len(base) + 1}")
-                if len(result) >= num_values:
-                    break
-        
-        return result[:num_values]
+        semantic = infer_semantic_type(column_name)
+        base = get_domain_values(semantic)
+        if not base:
+            col_lower = column_name.lower()
+            if "type" in col_lower:
+                base = ["Standard", "Premium", "Basic"]
+            elif "group" in col_lower:
+                base = ["Group A", "Group B", "Group C"]
+            else:
+                base = [f"{column_name}_value"]
+
+        # Cycle deterministically without numeric suffix pollution.
+        return [base[i % len(base)] for i in range(num_values)]
     
     def clear_cache(self):
         """Clear the generation cache."""

legitdata_project/legitdata/sourcer/generic.py

@@ -4,15 +4,27 @@ FIXED VERSION - Proper fallbacks, no garbage concatenated words.
 """
 
 import random
+import re
+import string
 import uuid
 from datetime import datetime, timedelta
 from decimal import Decimal
 from typing import Any, Optional
 from faker import Faker
+from ..domain import SemanticType, get_domain_values, infer_semantic_type
 
 fake = Faker()
 
 
+def _years_ago(dt: datetime, years: int) -> datetime:
+    """Return datetime shifted back by full years (leap-safe)."""
+    try:
+        return dt.replace(year=dt.year - years)
+    except ValueError:
+        # Handle Feb 29 on non-leap years.
+        return dt.replace(month=2, day=28, year=dt.year - years)
+
+
 class GenericSourcer:
     """Generates generic/synthetic data based on column strategies."""
     
@@ -182,7 +194,7 @@ class GenericSourcer:
         
         return f"{prefix}_{self._key_counters[prefix]:05d}"
     
-    def generate_value(self, strategy: str) -> Any:
+    def generate_value(self, strategy: str, expected_type: Optional[str] = None) -> Any:
         """
         Generate a value based on the strategy string.
         
@@ -215,31 +227,95 @@ class GenericSourcer:
             return self._generate(strategy_type, params)
         except Exception as e:
             print(f"Warning: Strategy '{strategy}' failed: {e}, using safe default")
-            return self._gen_safe_default()
+            return self._gen_safe_default(expected_type)
+
+    def get_strategy_for_semantic(self, semantic_type: SemanticType, use_case: str | None = None) -> Optional[str]:
+        """Return deterministic strategy for a semantic type when possible."""
+        if semantic_type in {
+            SemanticType.REGION,
+            SemanticType.CATEGORY,
+            SemanticType.SECTOR_NAME,
+            SemanticType.SECTOR_CATEGORY,
+            SemanticType.SEGMENT,
+            SemanticType.TIER,
+            SemanticType.STATUS,
+            SemanticType.FUND_STRATEGY,
+            SemanticType.INVESTOR_TYPE,
+            SemanticType.INVESTMENT_STAGE,
+            SemanticType.COVENANT_STATUS,
+            SemanticType.DEBT_PERFORMANCE_STATUS,
+            SemanticType.CHANNEL,
+            SemanticType.COUNTRY,
+            SemanticType.STATE,
+            SemanticType.CITY,
+            SemanticType.POSTAL_CODE,
+            SemanticType.SEASON,
+            SemanticType.HOLIDAY_NAME,
+            SemanticType.EVENT_NAME,
+            SemanticType.PRODUCT_NAME,
+            SemanticType.BRAND_NAME,
+            SemanticType.BRANCH_NAME,
+            SemanticType.ORG_NAME,
+            SemanticType.DEPARTMENT_NAME,
+            SemanticType.COLOR_DESCRIPTION,
+            SemanticType.PACKAGING_SIZE,
+        }:
+            values = get_domain_values(semantic_type, use_case)
+            if values:
+                return f"choice:{','.join(values)}"
+        if semantic_type == SemanticType.PERSON_NAME:
+            return "name"
+        if semantic_type == SemanticType.INTEREST_RATE:
+            return "random_decimal:0.01,0.35,4"
+        if semantic_type == SemanticType.RETURN_RATE:
+            return "random_decimal:-0.10,0.35,4"
+        if semantic_type == SemanticType.RETURN_MULTIPLE:
+            return "random_decimal:0.50,4.50,3"
+        if semantic_type == SemanticType.BASIS_POINTS:
+            return "random_int:25,250"
+        if semantic_type == SemanticType.LEVERAGE_RATIO:
+            return "random_decimal:0.50,8.00,3"
+        if semantic_type == SemanticType.PERCENT:
+            return "random_decimal:0.00,100.00,2"
+        if semantic_type == SemanticType.DURATION_DAYS:
+            return "random_int:0,365"
+        if semantic_type == SemanticType.QUANTITY:
+            return "random_int:1,50"
+        if semantic_type == SemanticType.COUNT:
+            return "random_int:0,50000"
+        if semantic_type == SemanticType.SCORE:
+            return "random_decimal:1.0,10.0,1"
+        return None
     
     def _generate(self, strategy_type: str, params: list) -> Any:
         """Route to specific generator based on strategy type."""
         
         # Random integers
         if strategy_type == 'random_int':
-            min_val = int(params[0]) if params else 1
-            max_val = int(params[1]) if len(params) > 1 else 1000
+            min_val = self._extract_int(params[0], 1) if params else 1
+            max_val = self._extract_int(params[1], 1000) if len(params) > 1 else 1000
             return random.randint(min_val, max_val)
         
         # Random decimals
         elif strategy_type == 'random_decimal':
-            min_val = float(params[0]) if params else 0.0
-            max_val = float(params[1]) if len(params) > 1 else 100.0
-            precision = int(params[2]) if len(params) > 2 else 2
+            min_val = self._extract_float(params[0], 0.0) if params else 0.0
+            max_val = self._extract_float(params[1], 100.0) if len(params) > 1 else 100.0
+            precision = self._extract_int(params[2], 2) if len(params) > 2 else 2
             value = random.uniform(min_val, max_val)
             return Decimal(str(round(value, precision)))
         
         # Random floats
         elif strategy_type == 'random_float':
-            min_val = float(params[0]) if params else 0.0
-            max_val = float(params[1]) if len(params) > 1 else 100.0
+            min_val = self._extract_float(params[0], 0.0) if params else 0.0
+            max_val = self._extract_float(params[1], 100.0) if len(params) > 1 else 100.0
             return round(random.uniform(min_val, max_val), 2)
         
+        # Random strings (alphanumeric IDs/codes)
+        elif strategy_type == 'random_string':
+            length = self._extract_int(params[0], 10) if params else 10
+            alphabet = string.ascii_uppercase + string.digits
+            return ''.join(random.choices(alphabet, k=max(1, length)))
+        
         # Choice from list
         elif strategy_type == 'choice':
             if params:
@@ -251,7 +327,7 @@ class GenericSourcer:
             # Format: "weighted_choice:val1,weight1,val2,weight2,..."
             if len(params) >= 2:
                 values = params[0::2]
-                weights = [float(w) for w in params[1::2]]
+                weights = [self._extract_float(w, 1.0) for w in params[1::2]]
                 return random.choices(values, weights=weights)[0]
             return params[0] if params else "Unknown"
         
@@ -261,14 +337,14 @@ class GenericSourcer:
         
         # Boolean
         elif strategy_type == 'boolean':
-            true_pct = float(params[0]) if params else 0.5
+            true_pct = self._extract_float(params[0], 0.5) if params else 0.5
             return random.random() < true_pct
         
         # Dates
         elif strategy_type == 'date_between':
             if params and len(params) >= 2:
-                start = datetime.strptime(params[0], '%Y-%m-%d')
-                end = datetime.strptime(params[1], '%Y-%m-%d')
+                start = datetime.strptime(self._extract_date(params[0], '2024-01-01'), '%Y-%m-%d')
+                end = datetime.strptime(self._extract_date(params[1], datetime.now().strftime('%Y-%m-%d')), '%Y-%m-%d')
             else:
                 end = datetime.now()
                 start = end - timedelta(days=365*2)
@@ -280,7 +356,7 @@ class GenericSourcer:
         # Sequential (for IDs)
         elif strategy_type == 'sequential':
             # This should be handled by the generator with state
-            start = int(params[0]) if params else 1
+            start = self._extract_int(params[0], 1) if params else 1
             return start
         
         # Faker-based generators
@@ -353,13 +429,39 @@ class GenericSourcer:
         else:
             return self._gen_safe_default()
     
-    def _gen_safe_default(self) -> str:
+    def _gen_safe_default(self, expected_type: Optional[str] = None) -> Any:
         """
         Generate a safe default value - NOT garbage concatenated words.
         Returns a simple, realistic-looking value.
         """
+        if expected_type:
+            t = expected_type.upper()
+            if any(x in t for x in ('INT', 'BIGINT', 'SMALLINT')):
+                return 0
+            if any(x in t for x in ('NUMBER', 'NUMERIC', 'DECIMAL', 'FLOAT', 'DOUBLE', 'REAL')):
+                return Decimal("0.00")
+            if 'BOOL' in t or 'BOOLEAN' in t or 'BIT' in t:
+                return False
+            if 'DATE' in t:
+                return datetime.now()
+
         # Return a simple word, not concatenated garbage
         return self.fake.word().capitalize()
+
+    def _extract_float(self, raw: str, default: float) -> float:
+        cleaned = (raw or "").split("(", 1)[0].strip()
+        match = re.search(r"[-+]?\d*\.?\d+", cleaned)
+        return float(match.group(0)) if match else default
+
+    def _extract_int(self, raw: str, default: int) -> int:
+        cleaned = (raw or "").split("(", 1)[0].strip()
+        match = re.search(r"[-+]?\d+", cleaned)
+        return int(match.group(0)) if match else default
+
+    def _extract_date(self, raw: str, default: str) -> str:
+        cleaned = (raw or "").split("(", 1)[0].strip()
+        match = re.search(r"\d{4}-\d{2}-\d{2}", cleaned)
+        return match.group(0) if match else default
     
     def new_row(self):
         """
@@ -395,8 +497,25 @@ class GenericSourcer:
         Infer the best strategy for a column based on its name and type.
         This is called when no explicit strategy is provided.
         """
-        col_lower = col_name.lower().replace('_', '').replace('-', '')
+        col_raw_lower = col_name.lower()
+        col_lower = col_raw_lower.replace('_', '').replace('-', '')
         col_type_upper = col_type.upper() if col_type else 'VARCHAR'
+
+        # Prefer semantic strategy first for quality-critical columns.
+        semantic = infer_semantic_type(col_name, col_type)
+        semantic_strategy = self.get_strategy_for_semantic(semantic)
+        if semantic_strategy:
+            return semantic_strategy
+
+        # Identifier-like short codes (e.g., TAX_ID_LAST4) should remain code-like.
+        if (
+            'taxid' in col_lower
+            or 'last4' in col_lower
+            or col_lower.endswith('idlast4')
+        ):
+            if 'VARCHAR' in col_type_upper or 'CHAR' in col_type_upper or 'TEXT' in col_type_upper:
+                return 'random_string:4'
+            return 'random_int:1000,9999'
         
         # === ID COLUMNS ===
         if col_lower.endswith('id') or col_lower.endswith('key'):
@@ -415,6 +534,12 @@ class GenericSourcer:
             return 'random_int:100000,9999999'
         
         # === PRICE/MONEY COLUMNS ===
+        if 'discount' in col_lower:
+            return 'random_decimal:0.00,200.00'
+        
+        if any(x in col_lower for x in ['tax', 'shipping']):
+            return 'random_decimal:1.00,250.00'
+        
         if any(x in col_lower for x in ['price', 'cost', 'amount', 'total', 'subtotal']):
             return 'random_decimal:1.00,500.00'
         
@@ -530,12 +655,30 @@ class GenericSourcer:
             return 'lookup:quarter'
         
         # === DATE COLUMNS ===
-        today = datetime.now().strftime('%Y-%m-%d')
-        if any(x in col_lower for x in ['date', 'datetime', 'timestamp', 'createdat', 'updatedat']):
-            return f'date_between:2024-01-01,{today}'
-        
-        if 'birthdate' in col_lower or 'dob' in col_lower:
-            return 'date_between:1960-01-01,2005-12-31'
+        today = datetime.now()
+        today_str = today.strftime('%Y-%m-%d')
+        
+        # Birth dates MUST be handled before generic date matching to avoid toddler ages
+        is_birth_col = (
+            'birthdate' in col_lower or
+            'dateofbirth' in col_lower or
+            col_lower.endswith('dob') or
+            col_lower == 'dob'
+        )
+        if is_birth_col:
+            oldest = _years_ago(today, 95).strftime('%Y-%m-%d')
+            youngest = _years_ago(today, 18).strftime('%Y-%m-%d')
+            return f'date_between:{oldest},{youngest}'
+        
+        financial_to_date_name = (
+            ('todate' in col_lower or 'to_date' in col_lower)
+            and any(
+                x in col_lower
+                for x in ['amount', 'balance', 'cost', 'price', 'revenue', 'sales', 'fee', 'tax', 'total', 'recovery', 'payment']
+            )
+        )
+        if any(x in col_lower for x in ['date', 'datetime', 'timestamp', 'createdat', 'updatedat']) and not financial_to_date_name:
+            return f'date_between:2024-01-01,{today_str}'
         
         # === BOOLEAN COLUMNS - Check FIRST before contact columns! ===
         # is_mobile should be boolean, not phone number
@@ -552,8 +695,10 @@ class GenericSourcer:
             return 'boolean:0.5'
         
         # === NAME COLUMNS ===
-        if col_lower in ('name', 'fullname', 'customername', 'clientname'):
+        if col_lower in ('name', 'fullname'):
             return 'name'
+        if col_lower in ('customername', 'accountname', 'clientname', 'companyname', 'organizationname'):
+            return 'company'
         
         if col_lower in ('firstname', 'fname', 'givenname'):
             return 'first_name'
@@ -565,10 +710,12 @@ class GenericSourcer:
         # col_lower has underscores stripped, so check for 'name' suffix
         if col_lower.endswith('name'):
             # Exclude non-person names
-            non_person = ('productname', 'companyname', 'brandname', 'storename',
+            non_person = ('productname', 'companyname', 'accountname', 'customername',
+                         'clientname', 'organizationname', 'brandname', 'storename',
                          'warehousename', 'centername', 'campaignname', 'tablename',
                          'columnname', 'schemaname', 'filename', 'hostname',
-                         'categoryname', 'holidayname')
+                         'categoryname', 'holidayname', 'branchname', 'departmentname',
+                         'regionname', 'cityname')
             if col_lower not in non_person:
                 return 'name'
         
@@ -600,7 +747,20 @@ class GenericSourcer:
             return 'company'
         
         # === URL COLUMNS ===
-        if 'url' in col_lower or 'website' in col_lower or 'link' in col_lower:
+        has_url_token = (
+            col_raw_lower == 'url'
+            or col_raw_lower.startswith('url_')
+            or col_raw_lower.endswith('_url')
+            or '_url_' in col_raw_lower
+            or 'website' in col_raw_lower
+        )
+        has_link_token = (
+            col_raw_lower == 'link'
+            or col_raw_lower.startswith('link_')
+            or col_raw_lower.endswith('_link')
+            or '_link_' in col_raw_lower
+        )
+        if has_url_token or has_link_token:
             return 'url'
         
         # === DESCRIPTION/TEXT COLUMNS ===
@@ -625,8 +785,7 @@ class GenericSourcer:
             return 'boolean:0.5'
         
         if 'DATE' in col_type_upper or 'TIME' in col_type_upper:
-            today = datetime.now().strftime('%Y-%m-%d')
-            return f'date_between:2024-01-01,{today}'
+            return f'date_between:2024-01-01,{today_str}'
         
         # === FINAL FALLBACK ===
         # Return a single word, not garbage

legitdata_project/legitdata/storyspec.py

@@ -0,0 +1,220 @@
+"""Story-first generation contract for deterministic, realistic demo data."""
+
+from __future__ import annotations
+
+import hashlib
+from dataclasses import asdict, dataclass, field
+from typing import Any
+
+
+@dataclass
+class TrendProfile:
+    """Controls for synthetic trend shape and reproducibility."""
+
+    style: str = "smooth"
+    noise_band_pct: float = 0.04
+    seasonal_strength: float = 0.10
+
+
+@dataclass
+class OutlierBudget:
+    """Controls for intentional outlier placement."""
+
+    max_events: int = 2
+    max_points_pct: float = 0.02
+    event_multiplier_min: float = 1.8
+    event_multiplier_max: float = 3.2
+
+
+@dataclass
+class StorySpec:
+    """Canonical story specification used during generation."""
+
+    company_url: str
+    company_name: str
+    use_case: str
+    vertical: str
+    function: str
+    seed: int
+    realism_mode: str = "strict"
+    trend_profile: TrendProfile = field(default_factory=TrendProfile)
+    outlier_budget: OutlierBudget = field(default_factory=OutlierBudget)
+    kpis: list[str] = field(default_factory=list)
+    allowed_dimensions: list[str] = field(default_factory=list)
+    allowed_measures: list[str] = field(default_factory=list)
+    quality_targets: dict[str, Any] = field(default_factory=dict)
+    value_guardrails: dict[str, Any] = field(default_factory=dict)
+    domain_overrides: dict[str, list[str]] = field(default_factory=dict)
+    recommended_companies: list[str] = field(default_factory=list)
+
+    def to_dict(self) -> dict[str, Any]:
+        return asdict(self)
+
+
+def _stable_seed(company_url: str, use_case: str) -> int:
+    text = f"{(company_url or '').strip().lower()}|{(use_case or '').strip().lower()}"
+    digest = hashlib.sha256(text.encode("utf-8")).hexdigest()[:8]
+    return int(digest, 16)
+
+
+def _safe_float(value: Any, default: float) -> float:
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return default
+
+
+def _safe_int(value: Any, default: int) -> int:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return default
+
+
+def _build_domain_overrides(company_name: str, use_case: str) -> dict[str, list[str]]:
+    company_clean = (company_name or "Demo Company").strip().title()
+    text = f"{company_name} {use_case}".lower()
+    overrides: dict[str, list[str]] = {}
+
+    if "amazon" in text:
+        overrides["product_name"] = [
+            "Echo Dot (5th Gen)",
+            "Kindle Paperwhite",
+            "Fire TV Stick 4K",
+            "Amazon Basics USB-C Cable",
+            "Ring Video Doorbell",
+            "Blink Outdoor Camera",
+            "iRobot Roomba Combo",
+            "Instant Pot Duo",
+            "LEGO Star Wars Set",
+            "Ninja Air Fryer",
+        ]
+        overrides["category"] = [
+            "Electronics",
+            "Home & Kitchen",
+            "Books",
+            "Fashion",
+            "Grocery",
+            "Sports & Outdoors",
+            "Health & Personal Care",
+        ]
+        overrides["channel"] = ["Prime", "Marketplace", "Subscribe & Save", "Amazon Fresh", "Retail"]
+
+    if any(tok in text for tok in ("prescription", "formulary", "pharma", "drug", "payer")):
+        overrides["category"] = [
+            "Cardiometabolic",
+            "Oncology",
+            "Women's Health",
+            "Immunology",
+            "Rare Disease",
+        ]
+        overrides["segment"] = [
+            "High Prescriber",
+            "Growth Prescriber",
+            "Low Prescriber",
+            "New Writer",
+            "Existing Writer",
+        ]
+        overrides["channel"] = [
+            "Field Sales",
+            "Medical Affairs",
+            "Payer Team",
+            "Digital HCP",
+            "Patient Support",
+        ]
+        overrides["org_name"] = [
+            "AOK Bayern",
+            "Techniker Krankenkasse",
+            "Barmer",
+            "DAK-Gesundheit",
+            "IKK Classic",
+            "Bayer Field Team South",
+            "Bayer Field Team North",
+        ]
+        overrides["product_name"] = ["Drug X 5mg", "Drug X 10mg", "Drug X Starter Pack", "Drug X Maintenance"]
+        if "germany" in text:
+            overrides["country"] = ["Germany"]
+            overrides["region"] = ["Bavaria", "North Rhine-Westphalia", "Berlin", "Hesse", "Saxony"]
+            overrides["city"] = ["Berlin", "Munich", "Hamburg", "Cologne", "Frankfurt", "Stuttgart"]
+
+    if not overrides.get("org_name"):
+        overrides["org_name"] = [
+            f"{company_clean} North Division",
+            f"{company_clean} South Division",
+            f"{company_clean} Enterprise Team",
+            f"{company_clean} Commercial Operations",
+        ]
+
+    return overrides
+
+
+def build_story_spec(
+    company_url: str,
+    use_case: str,
+    company_name: str | None = None,
+    use_case_config: dict[str, Any] | None = None,
+) -> StorySpec:
+    """Build StorySpec from matrix controls and runtime context."""
+    config = use_case_config or {}
+    controls = config.get("story_controls", {}) if isinstance(config, dict) else {}
+    outlier_cfg = controls.get("outlier_budget", {})
+    quality_targets = controls.get("quality_targets", {}) or {}
+    value_guardrails = controls.get("value_guardrails", {}) or {}
+
+    seed = _stable_seed(company_url, use_case)
+    trend_profile = TrendProfile(
+        style=str(controls.get("trend_style", "smooth")),
+        noise_band_pct=_safe_float(controls.get("trend_noise_band_pct"), 0.04),
+        seasonal_strength=_safe_float(controls.get("seasonal_strength"), 0.10),
+    )
+    outlier_budget = OutlierBudget(
+        max_events=max(0, _safe_int(outlier_cfg.get("max_events"), 2)),
+        max_points_pct=max(0.0, _safe_float(outlier_cfg.get("max_points_pct"), 0.02)),
+        event_multiplier_min=max(1.0, _safe_float(outlier_cfg.get("event_multiplier_min"), 1.8)),
+        event_multiplier_max=max(1.0, _safe_float(outlier_cfg.get("event_multiplier_max"), 3.2)),
+    )
+
+    if outlier_budget.event_multiplier_max < outlier_budget.event_multiplier_min:
+        outlier_budget.event_multiplier_max = outlier_budget.event_multiplier_min
+
+    name = (company_name or company_url or "Demo Company").split(".")[0].title()
+    vertical = str(config.get("vertical") or "Generic")
+    function = str(config.get("function") or "Generic")
+
+    return StorySpec(
+        company_url=company_url,
+        company_name=name,
+        use_case=use_case,
+        vertical=vertical,
+        function=function,
+        seed=seed,
+        realism_mode=str(controls.get("realism_mode", "strict")),
+        trend_profile=trend_profile,
+        outlier_budget=outlier_budget,
+        kpis=list(config.get("kpis", []) or []),
+        allowed_dimensions=list(config.get("allowed_story_dimensions", []) or []),
+        allowed_measures=list(config.get("allowed_story_measures", []) or []),
+        quality_targets=quality_targets,
+        value_guardrails=value_guardrails,
+        domain_overrides=_build_domain_overrides(name, use_case),
+        recommended_companies=list(config.get("recommended_companies", []) or []),
+    )
+
+
+def build_story_bundle(spec: StorySpec) -> dict[str, Any]:
+    """Build compact payload used by downstream ThoughtSpot handoff."""
+    return {
+        "vertical": spec.vertical,
+        "function": spec.function,
+        "kpis": spec.kpis[:8],
+        "allowed_dimensions": spec.allowed_dimensions[:10],
+        "allowed_measures": spec.allowed_measures[:10],
+        "trend_style": spec.trend_profile.style,
+        "trend_noise_band_pct": spec.trend_profile.noise_band_pct,
+        "seasonal_strength": spec.trend_profile.seasonal_strength,
+        "outlier_budget": {
+            "max_events": spec.outlier_budget.max_events,
+            "max_points_pct": spec.outlier_budget.max_points_pct,
+        },
+        "quality_targets": spec.quality_targets,
+    }

legitdata_project/pyproject.toml

@@ -0,0 +1,67 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "legitdata"
+version = "0.1.0"
+description = "Generate realistic synthetic data for analytics warehouses"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+authors = [
+    {name = "Your Name", email = "your.email@example.com"}
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Database",
+    "Topic :: Software Development :: Testing",
+]
+keywords = ["synthetic data", "data generation", "analytics", "snowflake", "testing"]
+
+dependencies = []
+
+[project.optional-dependencies]
+snowflake = ["snowflake-connector-python>=3.0.0"]
+ai = ["anthropic>=0.18.0"]
+all = [
+    "snowflake-connector-python>=3.0.0",
+    "anthropic>=0.18.0",
+]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-cov>=4.0.0",
+    "black>=23.0.0",
+    "ruff>=0.1.0",
+]
+
+[project.scripts]
+legitdata = "legitdata.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/yourname/legitdata"
+Documentation = "https://github.com/yourname/legitdata#readme"
+Repository = "https://github.com/yourname/legitdata"
+Issues = "https://github.com/yourname/legitdata/issues"
+
+[tool.setuptools.packages.find]
+include = ["legitdata*"]
+
+[tool.black]
+line-length = 100
+target-version = ['py310']
+
+[tool.ruff]
+line-length = 100
+select = ["E", "F", "W", "I", "N"]
+ignore = ["E501"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]

legitdata_project/test_legitdata.py

@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+"""Test script for legitdata functionality."""
+
+import sys
+sys.path.insert(0, '/home/claude')
+
+from legitdata import (
+    LegitGenerator, 
+    parse_ddl_file, 
+    SIZE_PRESETS,
+    USE_CASES
+)
+from legitdata.analyzer import ContextBuilder, ColumnClassifier, CompanyContext
+from legitdata.sourcer import GenericSourcer
+from legitdata.relationships import FKManager
+
+
+def test_ddl_parsing():
+    """Test DDL parsing."""
+    print("=" * 60)
+    print("TEST: DDL Parsing")
+    print("=" * 60)
+    
+    schema = parse_ddl_file('legitdata/sample_schema.sql')
+    
+    print(f"\nParsed schema: {schema}")
+    print(f"  Dimension tables: {len(schema.dimension_tables)}")
+    print(f"  Fact tables: {len(schema.fact_tables)}")
+    
+    print("\nDependency order:")
+    for table in schema.get_dependency_order():
+        print(f"  {table}")
+        for fk in table.foreign_keys:
+            print(f"    - {fk}")
+    
+    # Verify SALES_TRANSACTIONS has FKs
+    sales = schema.get_table('SALES_TRANSACTIONS')
+    assert sales is not None, "SALES_TRANSACTIONS table not found"
+    assert sales.is_fact_table, "SALES_TRANSACTIONS should be a fact table"
+    assert len(sales.foreign_keys) == 4, f"Expected 4 FKs, got {len(sales.foreign_keys)}"
+    
+    print("\n✓ DDL parsing tests passed!")
+    return schema
+
+
+def test_context_builder():
+    """Test context building (without actual web scraping)."""
+    print("\n" + "=" * 60)
+    print("TEST: Context Builder")
+    print("=" * 60)
+    
+    builder = ContextBuilder()
+    
+    # Test minimal context creation
+    context = builder._create_minimal_context(
+        url="https://amazon.com",
+        use_case="Retail Analytics"
+    )
+    
+    print(f"\nMinimal context:")
+    print(f"  Company: {context.company_name}")
+    print(f"  Industry: {context.industry}")
+    print(f"  Use case prompt:\n{context.to_prompt()}")
+    
+    assert context.company_name, "Company name should not be empty"
+    assert context.industry == "Retail", "Industry should be Retail for Retail Analytics"
+    
+    print("\n✓ Context builder tests passed!")
+    return context
+
+
+def test_column_classifier(schema, context):
+    """Test column classification."""
+    print("\n" + "=" * 60)
+    print("TEST: Column Classifier")
+    print("=" * 60)
+    
+    classifier = ColumnClassifier()
+    
+    # Test heuristic classification
+    classifications = classifier._classify_with_heuristics(
+        schema, context, "Retail Analytics"
+    )
+    
+    print("\nClassifications:")
+    for table_name, columns in classifications.items():
+        print(f"\n{table_name}:")
+        for col_name, col_info in columns.items():
+            print(f"  {col_name}: {col_info['classification']} -> {col_info['strategy'][:50]}...")
+    
+    # Verify some expected classifications (case-insensitive)
+    products = classifications.get('PRODUCTS', {})
+    assert products.get('ProductName', {}).get('classification', '').upper() == 'SEARCH_REAL', \
+        "ProductName should be SEARCH_REAL"
+    assert products.get('ProductID', {}).get('classification', '').upper() == 'GENERIC', \
+        "ProductID should be GENERIC"
+    
+    print("\n✓ Column classifier tests passed!")
+    return classifications
+
+
+def test_generic_sourcer():
+    """Test generic data generation."""
+    print("\n" + "=" * 60)
+    print("TEST: Generic Sourcer")
+    print("=" * 60)
+    
+    sourcer = GenericSourcer()
+    
+    # Test various strategies
+    tests = [
+        ("random_int:1,100", "integer"),
+        ("random_decimal:10.00,500.00", "decimal"),
+        ("date_between:2023-01-01,2024-12-31", "date"),
+        ("boolean:0.7", "boolean"),
+        ("uuid", "uuid"),
+        ("choice:Gold|Silver|Bronze", "choice"),
+    ]
+    
+    print("\nGenerated values:")
+    for strategy, desc in tests:
+        value = sourcer.generate_value(strategy)
+        print(f"  {desc} ({strategy}): {value}")
+    
+    # Test key generation
+    key = sourcer.generate_key("CUS")
+    print(f"  business key: {key}")
+    
+    # Test amount generation
+    amount = sourcer.generate_amount(10, 1000)
+    print(f"  amount: {amount}")
+    
+    print("\n✓ Generic sourcer tests passed!")
+
+
+def test_fk_manager():
+    """Test FK relationship manager."""
+    print("\n" + "=" * 60)
+    print("TEST: FK Manager")
+    print("=" * 60)
+    
+    manager = FKManager()
+    
+    # Register some PK values
+    customer_ids = [1, 2, 3, 4, 5]
+    product_ids = [10, 20, 30, 40, 50]
+    
+    manager.register_pk_values("CUSTOMERS", "CustomerID", customer_ids)
+    manager.register_pk_values("PRODUCTS", "ProductID", product_ids)
+    
+    print("\nRegistered tables:", manager.get_available_tables())
+    
+    # Get FK values with different distributions
+    print("\nUniform distribution (10 values):")
+    uniform_values = manager.get_fk_values("CUSTOMERS", "CustomerID", 10, "uniform")
+    print(f"  {uniform_values}")
+    
+    print("\nPareto distribution (20 values):")
+    pareto_values = manager.get_fk_values("CUSTOMERS", "CustomerID", 20, "pareto")
+    print(f"  {pareto_values}")
+    
+    # Count distribution
+    from collections import Counter
+    counts = Counter(pareto_values)
+    print(f"  Distribution: {dict(counts)}")
+    
+    print("\n✓ FK manager tests passed!")
+
+
+def test_full_preview():
+    """Test full preview generation."""
+    print("\n" + "=" * 60)
+    print("TEST: Full Preview (Dry Run)")
+    print("=" * 60)
+    
+    # Create generator in dry run mode
+    gen = LegitGenerator(
+        url="https://amazon.com",
+        use_case="Retail Analytics",
+        connection_string="",
+        dry_run=True,
+        cache_enabled=False
+    )
+    
+    # Load DDL
+    gen.load_ddl('legitdata/sample_schema.sql')
+    
+    # Preview (this exercises the full pipeline except DB writes)
+    print("\nGenerating preview...")
+    preview = gen.preview(num_rows=3)
+    
+    print("\nPreview results:")
+    for table_name, rows in preview.items():
+        print(f"\n{table_name} ({len(rows)} rows):")
+        if rows:
+            # Show first row
+            row = rows[0]
+            for key, value in row.items():
+                if value is not None:
+                    print(f"  {key}: {value}")
+    
+    # Verify FK relationships in fact table
+    sales_rows = preview.get('SALES_TRANSACTIONS', [])
+    if sales_rows:
+        row = sales_rows[0]
+        print(f"\nFK integrity check:")
+        print(f"  CustomerID: {row.get('CustomerID')}")
+        print(f"  ProductID: {row.get('ProductID')}")
+        print(f"  SellerID: {row.get('SellerID')}")
+        print(f"  FulfillmentCenterID: {row.get('FulfillmentCenterID')}")
+    
+    print("\n✓ Full preview tests passed!")
+
+
+def test_size_presets():
+    """Test size presets."""
+    print("\n" + "=" * 60)
+    print("TEST: Size Presets")
+    print("=" * 60)
+    
+    print("\nAvailable presets:")
+    for name, preset in SIZE_PRESETS.items():
+        print(f"  {name}: {preset.fact_rows} facts, {preset.dim_rows} dims - {preset.description}")
+    
+    print(f"\nAvailable use cases:")
+    for uc in USE_CASES:
+        print(f"  - {uc}")
+    
+    print("\n✓ Size presets test passed!")
+
+
+def main():
+    """Run all tests."""
+    print("\n" + "=" * 60)
+    print("LEGITDATA TEST SUITE")
+    print("=" * 60)
+    
+    try:
+        # Run tests
+        schema = test_ddl_parsing()
+        context = test_context_builder()
+        test_column_classifier(schema, context)
+        test_generic_sourcer()
+        test_fk_manager()
+        test_size_presets()
+        test_full_preview()
+        
+        print("\n" + "=" * 60)
+        print("ALL TESTS PASSED!")
+        print("=" * 60)
+        return 0
+        
+    except Exception as e:
+        print(f"\n✗ TEST FAILED: {e}")
+        import traceback
+        traceback.print_exc()
+        return 1
+
+
+if __name__ == '__main__':
+    sys.exit(main())

legitdata_project/test_with_ai.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""Test legitdata with AI generation - larger dataset."""
+
+from dotenv import load_dotenv
+load_dotenv()
+
+from anthropic import Anthropic
+from legitdata import LegitGenerator
+
+client = Anthropic()
+
+def web_search(query: str) -> list[dict]:
+    """Use Claude to search the web."""
+    response = client.messages.create(
+        model="claude-sonnet-4-20250514",
+        max_tokens=1024,
+        tools=[{
+            "type": "web_search_20250305",
+            "name": "web_search"
+        }],
+        messages=[{"role": "user", "content": f"Search for: {query}. Return a list of results."}]
+    )
+    
+    results = []
+    for block in response.content:
+        if hasattr(block, 'text'):
+            results.append({"title": block.text[:100], "snippet": block.text})
+    
+    return results
+
+
+gen = LegitGenerator(
+    url="https://amazon.com",
+    use_case="Retail Analytics",
+    connection_string="",
+    anthropic_client=client,
+    web_search_fn=web_search,
+    dry_run=True
+)
+
+gen.load_ddl('legitdata_project/legitdata/sample_schema.sql')
+
+# Use "small" preset: 100 fact rows, 20 dim rows
+preview = gen.preview(num_rows=20)
+
+print("\n" + "="*60)
+print("DATASET SUMMARY")
+print("="*60)
+
+for table, rows in preview.items():
+    print(f"\n{table}: {len(rows)} rows")
+    
+    # Show first 3 rows
+    for i, row in enumerate(rows[:3]):
+        print(f"  Row {i+1}: {row}")
+    
+    if len(rows) > 3:
+        print(f"  ... and {len(rows) - 3} more rows")
+
+# Show FK distribution in fact table
+print("\n" + "="*60)
+print("FK DISTRIBUTION IN SALES_TRANSACTIONS")
+print("="*60)
+
+from collections import Counter
+sales = preview.get('SALES_TRANSACTIONS', [])
+if sales:
+    customer_dist = Counter(r['CustomerID'] for r in sales)
+    product_dist = Counter(r['ProductID'] for r in sales)
+    
+    print(f"\nCustomerID distribution: {dict(customer_dist)}")
+    print(f"ProductID distribution: {dict(product_dist)}")

liveboard_creator.py

@@ -158,13 +158,17 @@ def _get_answer_direct(question: str, model_id: str) -> dict:
     
     try:
         resp = session.post(url, json=payload, timeout=60)
-        
+
         if resp.status_code == 200:
             data = resp.json()
-            # Format response similar to MCP getAnswer
-            # IMPORTANT: MCP's createLiveboard requires 'question' field!
+            print(f"  [direct API] HTTP 200 — keys: {list(data.keys())}", flush=True)
+            print(f"  [direct API] session_identifier: {data.get('session_identifier')}", flush=True)
+            print(f"  [direct API] visualization_type: {data.get('visualization_type')}", flush=True)
+            print(f"  [direct API] message_type: {data.get('message_type')}", flush=True)
+            has_tokens = bool(data.get('tokens'))
+            print(f"  [direct API] tokens present: {has_tokens}", flush=True)
             return {
-                'question': question,  # MCP createLiveboard requires this!
+                'question': question,
                 'session_identifier': data.get('session_identifier'),
                 'tokens': data.get('tokens'),
                 'display_tokens': data.get('display_tokens'),
@@ -173,10 +177,10 @@ def _get_answer_direct(question: str, model_id: str) -> dict:
                 'message_type': data.get('message_type', 'TSAnswer')
             }
         else:
-            error_msg = resp.text[:200] if resp.text else f"Status {resp.status_code}"
-            print(f"  ⚠️  Direct API answer failed: {error_msg}")
+            error_msg = resp.text[:500] if resp.text else f"Status {resp.status_code}"
+            print(f"  ⚠️  Direct API answer FAILED — HTTP {resp.status_code}: {error_msg}", flush=True)
             return None
-            
+
     except Exception as e:
         print(f"  ⚠️  Direct API exception: {str(e)}")
         return None
@@ -378,6 +382,7 @@ def clean_viz_title(question: str) -> str:
         (r'^What is the ', ''),
         (r'^What are the ', ''),
         (r'^Show me the ', ''),
+        (r'^Show me ', ''),
         (r'^Show the ', ''),
         (r'^Show ', ''),
         (r'^Create a detailed table showing ', ''),
@@ -519,11 +524,18 @@ CRITICAL RULES:
 4. If query mentions "location", find the EXACT column name (e.g., "Storename", "locaLocationid")
 5. ThoughtSpot auto-aggregates - DO NOT use sum(), count(), avg() functions
 6. Use [column name] syntax with exact column name: [Productname] not [product]
+7. TIME GRANULARITY: NEVER use bare "weekly", "monthly", "daily" as standalone tokens.
+   Always attach granularity to the date column: [Date Column].weekly, [Date Column].monthly
+   Example: "sales weekly" → "[Sales Amount] [Order Date].weekly"
+   Example: "revenue by month" → "[Revenue] [Transaction Date].monthly"
+8. If no date column is needed, just use the measure and dimension columns.
 
 Examples:
 - "Show products where stock > 900" → "[Productname] [Stocklevel] [Stocklevel] > 900"
 - "Show customers where lifetime value > 50000" → "[Name] [Lifetimevalue] [Lifetimevalue] > 50000"
 - "Show sales by region" → "[Salesamount] [Region]"
+- "Show ASP weekly trend" → "[Asp Amount] [Order Date].weekly"
+- "Revenue by month last 12 months" → "[Revenue] [Date].monthly [Date].'last 12 months'"
 
 Convert the natural query above to ThoughtSpot search syntax using EXACT column names.
 Return ONLY the search query string, nothing else."""
@@ -1093,6 +1105,18 @@ Examples:
                     )
                     print(f"      Search: {search_query}")
 
+                    # Validate all [Column] tokens exist in the model — skip if any are missing
+                    import re as _re
+                    col_names_lower = {col.get('name', '').lower() for col in self.model_columns if col.get('name')}
+                    # Extract tokens (strip granularity suffix like .weekly)
+                    unknown_tokens = [
+                        t for t in _re.findall(r'\[([^\]]+)\]', search_query)
+                        if t.split('.')[0].strip().lower() not in col_names_lower
+                    ]
+                    if unknown_tokens:
+                        print(f"      ⚠️  Skipping '{outlier['title']}' — columns not in model: {unknown_tokens}")
+                        continue
+
                     chart_type = self.query_translator.infer_chart_type(
                         search_query,
                         self.model_columns,
@@ -1111,17 +1135,8 @@ Examples:
                     viz_counter += 1
 
             except Exception as e:
-                print(f"      ⚠️  Error: {e}")
-                # Fallback visualization
-                viz_configs.append({
-                    'id': f'Viz_{viz_counter}',
-                    'name': outlier['title'],
-                    'description': outlier['insight'],
-                    'chart_type': 'TABLE',
-                    'search_query_direct': '[data]',  # Minimal fallback
-                    'outlier_context': outlier
-                })
-                viz_counter += 1
+                print(f"      ⚠️  Skipping '{outlier.get('title', 'unknown')}' due to error: {e}")
+                # Don't add a broken viz — skip and continue
 
         print(f"   Generated {len(viz_configs)} visualizations total")
         return viz_configs
@@ -2714,14 +2729,12 @@ def _convert_outlier_to_mcp_question(outlier: Dict) -> str:
         # Clean up the query - remove quotes, extra formatting
         question = show_me.replace('"', '').replace("'", '').strip()
         
-        # If it doesn't start with question words, make it a question
-        if not any(question.lower().startswith(q) for q in ['what', 'which', 'how', 'show', 'who', 'when', 'where']):
-            question = f"Show me {question}"
-        
+        # Capitalize first letter if needed
+        question = question[0].upper() + question[1:] if question else question
         return question
-    
-    # Fallback: create question from title
-    return f"Show me {outlier.get('title', 'data')}"
+
+    # Fallback: use title directly
+    return outlier.get('title', 'data')
 
 
 def _create_kpi_question_from_outlier(outlier: Dict) -> Optional[str]:
@@ -2820,45 +2833,24 @@ For example, if measures are [IMPRESSIONS, CLICKS, CONVERSIONS], use those - NOT
 Company: {company_data.get('name', 'Unknown Company')}
 Use Case: {use_case}
 {column_context}
-Create questions that will produce high-quality visualizations. Each question MUST:
-- Use the ACTUAL column names from the data model above (NOT generic terms like "revenue" or "products")
-- Be specific and actionable for business users
-- Use "top N" with specific numbers (top 5, top 10, top 15)
-- Include time periods (last 12 months, this year, last quarter, past 18 months)
-- DO NOT include chart type in the question (no "bar chart", "line chart", etc.)
-
-REQUIRED FORMAT - Questions should be concise and business-focused. NO chart type hints!
-
-IMPORTANT: The FIRST TWO questions MUST be KPIs in this EXACT format:
-- Question 1: "[MEASURE] weekly" (exactly 2-3 words, creates KPI with WoW sparkline)
-- Question 2: "[MEASURE] monthly" (exactly 2-3 words, creates KPI with MoM sparkline)
-
-EXAMPLES OF CORRECT KPI QUESTIONS:
-- "spend weekly" ✅
-- "impressions monthly" ✅
-- "conversions weekly" ✅
-- "total revenue monthly" ✅
-
-REQUIRED QUESTIONS (adapt to actual columns above):
-1. KPI: "[PRIMARY_MEASURE] weekly" (2-3 words only!)
-2. KPI: "[SECONDARY_MEASURE] monthly" (2-3 words only!)
-3. "Top 10 [DIMENSION] by [MEASURE] last 12 months"
-4. "[MEASURE] by [DIMENSION] last 12 months" 
-5. "[MEASURE] by month last 18 months"
-6. "[MEASURE] by [DIMENSION1] vs [DIMENSION2] last 12 months"
-7. "Top 15 [ANOTHER_DIMENSION] by [MEASURE]"
-8. "[MEASURE] by [DIMENSION] last 2 years"
-
-TIME FILTER RULES (IMPORTANT):
-- Use "last 12 months" or "last 18 months" instead of "this year" (more data coverage)
-- Use "last 12 months" instead of "last 6 months" (avoids empty results)
+The liveboard already has KPI cards showing single metrics. These fill questions should surface INTERESTING BUSINESS INSIGHTS that a senior executive would find compelling — comparisons, rankings, and breakdowns that tell a story.
+
+WHAT MAKES A GREAT FILL QUESTION:
+- Reveals which dimension is driving performance: "Top 10 [DIMENSION] by [MEASURE] last 12 months"
+- Compares performance across a meaningful grouping: "[MEASURE] by [DIMENSION] last 12 months"
+- Shows how a breakdown has shifted: "[MEASURE] by [DIMENSION1] and [DIMENSION2] last 18 months"
+- Surfaces a ranking with business stakes: "Top 5 [DIMENSION] by [MEASURE] vs prior year"
+
+DO NOT generate simple single-metric questions like "[MEASURE] by month" — those become redundant KPI cards and the liveboard already has enough of those. Every question here should involve at least one dimension to compare against.
+
+TIME FILTER RULES:
+- Use "last 12 months" or "last 18 months" — never "this year" or "this quarter" (too restrictive with generated data)
 - Use "last 2 years" for broad comparisons
-- NEVER use "this year" or "this quarter" - too restrictive with generated data
 
-CRITICAL: 
-- Use the EXACT column names from the data model
-- DO NOT add "bar chart", "line chart", "stacked" to questions - ThoughtSpot picks chart type automatically
-- Include variety: rankings, breakdowns, trends, comparisons
+CRITICAL:
+- Use the EXACT column names from the data model above
+- Do NOT mention chart types — ThoughtSpot picks automatically
+- Each question should make a business leader say "that's interesting" — not just confirm a number
 
 Return ONLY a JSON object with this exact structure (no other text):
 {{
@@ -2947,18 +2939,18 @@ Return ONLY a JSON object with this exact structure (no other text):
                 m2 = measures[1].lower().replace('_', ' ') if len(measures) > 1 else m1
                 d1 = dimensions[0].lower().replace('_', ' ') if dimensions else 'category'
                 return [
-                    f"{m1} weekly",  # KPI with sparkline
-                    f"{m2} monthly",  # KPI with sparkline  
+                    f"{m1} by week",
+                    f"{m2} by month",
                     f"top 10 {d1} by {m1}",
                     f"{m1} by {d1}",
                     f"{m1} by month last 12 months",
                     f"{m2} by {d1}"
                 ][:num_questions]
-        
-        # Ultimate fallback - simple KPI format
+
+        # Ultimate fallback
         return [
-            "revenue weekly",  # Simple KPI format for sparkline
-            "sales monthly",   # Simple KPI format for sparkline
+            "revenue by week",
+            "sales by month",
             "top 10 products by revenue",
             "revenue by category",
             "revenue by month last 12 months",
@@ -3262,10 +3254,14 @@ def create_liveboard_from_model_mcp(
                     print(f"🎨 Creating liveboard: {final_liveboard_name}")
                     print(f"📊 Preparing to send {len(answers)} answers to createLiveboard")
                     
-                    # Debug: Log what we're sending
-                    print(f"🔍 Answer structure sample:")
-                    if answers:
-                        print(f"   Keys in first answer: {list(answers[0].keys())}")
+                    # Log what we're sending to createLiveboard
+                    print(f"📋 [createLiveboard] Answer summary ({len(answers)} total):", flush=True)
+                    for idx, ans in enumerate(answers):
+                        sid = ans.get('session_identifier', 'MISSING')
+                        viz = ans.get('visualization_type', 'MISSING')
+                        q = ans.get('question', 'MISSING')[:60]
+                        has_tok = bool(ans.get('tokens'))
+                        print(f"   [{idx+1}] session={sid} viz={viz} tokens={has_tok} q='{q}'", flush=True)
                     
                     try:
                         print(f"📦 [createLiveboard] Sending {len(answers)} answers...", flush=True)
@@ -3279,8 +3275,8 @@ def create_liveboard_from_model_mcp(
                         })
                         
                         result_text = liveboard_result.content[0].text
-                        print(f"   [createLiveboard] Response: {result_text[:200]}", flush=True)
-                        
+                        print(f"   [createLiveboard] FULL response: {result_text}", flush=True)
+
                         if not result_text or result_text.strip() == '':
                             raise ValueError("createLiveboard returned empty response")
                         
@@ -3688,6 +3684,12 @@ def enhance_mcp_liveboard(
         
         visualizations = liveboard_tml.get('liveboard', {}).get('visualizations', [])
         print(f"  Found {len(visualizations)} visualizations", flush=True)
+        for _v in visualizations:
+            _vid = _v.get('id', '?')
+            _is_note = 'note_tile' in _v
+            _chart_type = _v.get('answer', {}).get('chart', {}).get('type', 'N/A') if not _is_note else 'note'
+            _name = _v.get('answer', {}).get('name', '') if not _is_note else '(note tile)'
+            print(f"    viz {_vid}: type={_chart_type} note={_is_note} name='{_name}'", flush=True)
         
         # Step 2: Classify visualizations by type and purpose
         kpi_vizs = []      # KPI charts (big numbers, sparklines)
@@ -3735,7 +3737,9 @@ def enhance_mcp_liveboard(
         print(f"  Classification: {len(kpi_vizs)} KPIs, {len(trend_vizs)} trends, {len(bar_vizs)} bars, {len(table_vizs)} tables, {len(note_vizs)} notes", flush=True)
         
         # Step 2.5: Remove note tiles (MCP requires them but we don't want them in final liveboard)
-        if note_vizs:
+        # Safety: only remove note tiles if there are other vizzes to show — never leave liveboard empty
+        non_note_count = len(visualizations) - len(note_vizs)
+        if note_vizs and non_note_count > 0:
             print(f"  Removing {len(note_vizs)} note tile(s)...", flush=True)
             original_count = len(visualizations)
             liveboard_tml['liveboard']['visualizations'] = [
@@ -3744,6 +3748,8 @@ def enhance_mcp_liveboard(
             visualizations = liveboard_tml['liveboard']['visualizations']
             print(f"  [OK] Removed note tiles ({original_count} -> {len(visualizations)} visualizations)", flush=True)
             enhancements_applied.append(f"Removed {len(note_vizs)} note tile(s)")
+        elif note_vizs and non_note_count == 0:
+            print(f"  ⚠️  Only note tiles found ({len(note_vizs)}) — MCP may have returned no chart answers. Keeping note tiles to preserve liveboard content.", flush=True)
         
         # Step 3: Add Groups - simplified: just KPI section, rest ungrouped
         if add_groups:

outlier_system.py

@@ -1,769 +0,0 @@
-"""
-Outlier System - Unified SQL generation for data pattern injection.
-
-This system handles two use cases with the SAME core mechanism:
-1. Auto-Injection: AI identifies patterns during research → SQL UPDATEs after population
-2. Chat Adjustment: User requests changes via chat → SQL UPDATEs on demand
-
-The core insight: Both are SQL UPDATE statements that modify data to create patterns.
-
-Usage:
-    from outlier_system import OutlierGenerator, apply_outliers
-    
-    # Generate SQL from story descriptions
-    generator = OutlierGenerator(schema_info)
-    sql_updates = generator.generate_outlier_sql(
-        "Premium subscriptions spike 3x in Q4",
-        target_column="subscription_tier",
-        target_value="premium"
-    )
-    
-    # Apply multiple outliers after population
-    apply_outliers(snowflake_conn, outliers, schema_name)
-"""
-
-import re
-import os
-from typing import Dict, List, Optional, Tuple
-from dataclasses import dataclass, field
-from datetime import datetime
-
-
-# ============================================================================
-# Phase 1: New Structured Outlier System (February 2026 Sprint)
-# ============================================================================
-
-@dataclass
-class OutlierPattern:
-    """
-    Defines a single outlier pattern that serves three purposes:
-    1. Liveboard visualizations
-    2. Spotter questions
-    3. Demo talking points
-    """
-    # Identity
-    name: str                                    # "ASP Decline"
-    category: str                                # "pricing", "volume", "inventory"
-    
-    # For LIVEBOARD (visualization)
-    viz_type: str                                # "KPI", "COLUMN", "LINE"
-    viz_question: str                            # "ASP weekly"
-    viz_talking_point: str                       # "ASP dropped 12% — excessive discounting"
-    
-    # For SPOTTER (ad-hoc questions)
-    spotter_questions: List[str] = field(default_factory=list)
-    spotter_followups: List[str] = field(default_factory=list)
-    
-    # For DATA INJECTION (SQL generation)
-    sql_template: str = ""                       # "UPDATE {fact_table} SET {column} = ..."
-    affected_columns: List[str] = field(default_factory=list)
-    magnitude: str = ""                          # "15% below normal"
-    target_filter: str = ""                      # "WHERE REGION = 'West'"
-    
-    # For DEMO NOTES
-    demo_setup: str = ""                         # "Start by showing overall sales are UP"
-    demo_payoff: str = ""                        # "Then reveal ASP is DOWN — 'at what cost?'"
-
-
-@dataclass
-class OutlierConfig:
-    """
-    Configuration for outliers per use case.
-    Combines required patterns, optional patterns, and AI generation guidance.
-    """
-    required: List[OutlierPattern] = field(default_factory=list)    # Always include
-    optional: List[OutlierPattern] = field(default_factory=list)    # AI picks 1-2
-    allow_ai_generated: bool = True                                  # AI can create 1 custom
-    ai_guidance: str = ""                                            # Hint for AI generation
-
-
-OUTLIER_CONFIGS = {
-    ("Retail", "Sales"): OutlierConfig(
-        required=[
-            OutlierPattern(
-                name="ASP Decline",
-                category="pricing",
-                viz_type="KPI",
-                viz_question="ASP weekly",
-                viz_talking_point="ASP dropped 12% even though revenue is up — we're discounting too heavily",
-                spotter_questions=[
-                    "Why did ASP drop last month?",
-                    "Which products have the biggest discount?",
-                    "Show me ASP by region",
-                ],
-                spotter_followups=[
-                    "Compare to same period last year",
-                    "Which stores are discounting most?",
-                ],
-                sql_template="UPDATE {fact_table} SET UNIT_PRICE = UNIT_PRICE * 0.85 WHERE REGION = 'West' AND {date_column} > '{recent_date}'",
-                affected_columns=["UNIT_PRICE", "DISCOUNT_PCT"],
-                magnitude="15% below normal",
-                target_filter="WHERE REGION = 'West'",
-                demo_setup="Start by showing overall sales are UP — everything looks good",
-                demo_payoff="Then reveal ASP is DOWN — 'but at what cost?' moment",
-            ),
-            OutlierPattern(
-                name="Regional Variance",
-                category="geographic",
-                viz_type="COLUMN",
-                viz_question="Dollar Sales by Region",
-                viz_talking_point="West region outperforming by 40% — what are they doing differently?",
-                spotter_questions=[
-                    "Which region has the highest sales?",
-                    "Compare West to East performance",
-                ],
-                spotter_followups=[
-                    "What products are driving West?",
-                    "Show me the trend for West region",
-                ],
-                sql_template="UPDATE {fact_table} SET QUANTITY = QUANTITY * 1.4 WHERE REGION = 'West'",
-                affected_columns=["QUANTITY", "REVENUE"],
-                magnitude="40% above other regions",
-                target_filter="WHERE REGION = 'West'",
-                demo_setup="Show overall sales by region",
-                demo_payoff="West is crushing it — drill in to find out why",
-            ),
-        ],
-        optional=[
-            OutlierPattern(
-                name="Seasonal Spike",
-                category="temporal",
-                viz_type="LINE",
-                viz_question="Dollar Sales trend by month",
-                viz_talking_point="Holiday surge 3x normal — were we prepared?",
-                spotter_questions=["Show me sales trend for Q4", "When was our peak sales day?"],
-                spotter_followups=[],
-                sql_template="UPDATE {fact_table} SET QUANTITY = QUANTITY * 3 WHERE MONTH IN (11, 12)",
-                affected_columns=["QUANTITY", "REVENUE"],
-                magnitude="3x normal",
-                target_filter="WHERE MONTH IN (11, 12)",
-                demo_setup="",
-                demo_payoff="",
-            ),
-            OutlierPattern(
-                name="Category Surge",
-                category="product",
-                viz_type="COLUMN",
-                viz_question="Dollar Sales by Category",
-                viz_talking_point="Electronics up 60% YoY while Apparel flat",
-                spotter_questions=["Which category grew fastest?", "Compare Electronics to Apparel"],
-                spotter_followups=[],
-                sql_template="",
-                affected_columns=[],
-                magnitude="60% YoY",
-                target_filter="",
-                demo_setup="",
-                demo_payoff="",
-            ),
-        ],
-        allow_ai_generated=True,
-        ai_guidance="If company has sustainability initiatives, create outlier around eco-friendly product sales",
-    ),
-    
-    ("Banking", "Marketing"): OutlierConfig(
-        required=[
-            OutlierPattern(
-                name="Funnel Drop-off",
-                category="conversion",
-                viz_type="COLUMN",
-                viz_question="Conversion rate by funnel stage",
-                viz_talking_point="70% drop-off at application page — UX issue?",
-                spotter_questions=[
-                    "Where is our biggest funnel drop-off?",
-                    "What's our application completion rate?",
-                ],
-                spotter_followups=[],
-                sql_template="",
-                affected_columns=[],
-                magnitude="70% drop-off",
-                target_filter="",
-                demo_setup="Show the full funnel from impression to approval",
-                demo_payoff="The application page is killing conversions",
-            ),
-        ],
-        optional=[
-            OutlierPattern(
-                name="Channel Performance",
-                category="channel",
-                viz_type="COLUMN",
-                viz_question="CTR by channel",
-                viz_talking_point="Mobile CTR 2x desktop — shift budget?",
-                spotter_questions=["Which channel has the best CTR?"],
-                spotter_followups=[],
-                sql_template="",
-                affected_columns=[],
-                magnitude="2x desktop",
-                target_filter="",
-                demo_setup="",
-                demo_payoff="",
-            ),
-        ],
-        allow_ai_generated=True,
-        ai_guidance="Consider seasonal patterns in loan applications",
-    ),
-}
-
-
-def get_outliers_for_use_case(vertical: str, function: str) -> OutlierConfig:
-    """Get outlier configuration for a use case, with fallback to empty config."""
-    return OUTLIER_CONFIGS.get(
-        (vertical, function),
-        OutlierConfig(
-            required=[],
-            optional=[],
-            allow_ai_generated=True,
-            ai_guidance=f"Generate outliers appropriate for {vertical} {function}"
-        )
-    )
-
-
-# ============================================================================
-# Legacy Outlier System (existing code below)
-# ============================================================================
-
-@dataclass
-class LegacyOutlierPattern:
-    """Represents a data pattern to inject (legacy structure)."""
-    title: str
-    description: str
-    sql_update: str
-    affected_table: str
-    affected_column: str
-    target_value: Optional[str] = None
-    conditions: Optional[str] = None
-    impact_description: Optional[str] = None
-    spotter_question: Optional[str] = None
-    talking_point: Optional[str] = None
-
-
-class OutlierGenerator:
-    """
-    Generate SQL UPDATE statements from text descriptions.
-    
-    The generator understands common patterns like:
-    - Time-based: "spike in Q4", "drop in summer"
-    - Value-based: "premium users 3x higher", "decrease electronics by 10%"  
-    - Conditional: "when category is X, increase Y"
-    """
-    
-    def __init__(self, schema_info: Dict = None):
-        """
-        Initialize with schema information.
-        
-        Args:
-            schema_info: Dict with table and column information
-                {
-                    'tables': {'SUBSCRIPTIONS': {'columns': [...]}, ...},
-                    'fact_tables': ['SUBSCRIPTIONS', ...],
-                    'dimension_tables': ['USERS', ...]
-                }
-        """
-        self.schema_info = schema_info or {}
-        self.tables = schema_info.get('tables', {}) if schema_info else {}
-        
-    def generate_outlier_sql(
-        self,
-        pattern_description: str,
-        target_table: str = None,
-        target_column: str = None,
-        multiplier: float = None,
-        absolute_value: float = None,
-        conditions: Dict = None
-    ) -> OutlierPattern:
-        """
-        Generate SQL UPDATE from a pattern description.
-        
-        Args:
-            pattern_description: Natural language description
-                "Premium subscriptions spike 3x in Q4"
-                "Decrease electronics revenue by 10%"
-            target_table: Table to update (optional, inferred from description)
-            target_column: Column to update (optional, inferred)
-            multiplier: Factor to multiply by (e.g., 3.0 for "3x")
-            absolute_value: Exact value to set
-            conditions: WHERE clause conditions as dict
-                {'month': ['Oct', 'Nov', 'Dec'], 'category': 'Electronics'}
-        
-        Returns:
-            OutlierPattern with SQL and documentation
-        """
-        # Parse the description to extract intent
-        parsed = self._parse_pattern_description(pattern_description)
-        
-        # Use provided values or parsed ones
-        target_table = target_table or parsed.get('table')
-        target_column = target_column or parsed.get('column')
-        multiplier = multiplier or parsed.get('multiplier')
-        absolute_value = absolute_value if absolute_value is not None else parsed.get('absolute_value')
-        conditions = conditions or parsed.get('conditions', {})
-        
-        # Build the SQL UPDATE
-        sql = self._build_update_sql(
-            target_table, target_column, multiplier, absolute_value, conditions
-        )
-        
-        # Generate Spotter question
-        spotter_q = self._generate_spotter_question(
-            target_table, target_column, conditions, pattern_description
-        )
-        
-        return LegacyOutlierPattern(
-            title=parsed.get('title', pattern_description[:50]),
-            description=pattern_description,
-            sql_update=sql,
-            affected_table=target_table,
-            affected_column=target_column,
-            target_value=str(absolute_value) if absolute_value else f"{multiplier}x",
-            conditions=str(conditions) if conditions else None,
-            impact_description=parsed.get('impact'),
-            spotter_question=spotter_q,
-            talking_point=self._generate_talking_point(pattern_description)
-        )
-    
-    def _parse_pattern_description(self, description: str) -> Dict:
-        """
-        Parse natural language pattern description.
-        
-        Examples:
-            "Premium subscriptions spike 3x in Q4"
-            → {'multiplier': 3.0, 'conditions': {'quarter': 'Q4'}, 'column': 'subscription_tier'}
-            
-            "Decrease electronics by 10%"
-            → {'multiplier': 0.9, 'conditions': {'category': 'electronics'}}
-        """
-        result = {
-            'title': description[:50] + ('...' if len(description) > 50 else ''),
-            'conditions': {}
-        }
-        
-        desc_lower = description.lower()
-        
-        # Parse multiplier patterns
-        # "3x", "spike 3x", "increase 3x"
-        multiplier_match = re.search(r'(\d+(?:\.\d+)?)\s*x', desc_lower)
-        if multiplier_match:
-            result['multiplier'] = float(multiplier_match.group(1))
-        
-        # "increase by 20%", "decrease by 10%", "decrease revenue by 10%"
-        # More flexible pattern that allows words between action and amount
-        percent_match = re.search(r'(increase|decrease|drop|spike|rise)s?\s+.*?(?:by\s+)?(\d+(?:\.\d+)?)\s*%', desc_lower)
-        if percent_match:
-            action = percent_match.group(1)
-            pct = float(percent_match.group(2))
-            if action in ['decrease', 'drop']:
-                result['multiplier'] = 1 - (pct / 100)
-            else:
-                result['multiplier'] = 1 + (pct / 100)
-        
-        # "set to 50B", "make 40B"
-        absolute_match = re.search(r'(?:set|make|to)\s+(\d+(?:\.\d+)?)\s*(B|M|K)?', description, re.IGNORECASE)
-        if absolute_match:
-            value = float(absolute_match.group(1))
-            suffix = absolute_match.group(2)
-            if suffix:
-                suffix = suffix.upper()
-                if suffix == 'B':
-                    value *= 1_000_000_000
-                elif suffix == 'M':
-                    value *= 1_000_000
-                elif suffix == 'K':
-                    value *= 1_000
-            result['absolute_value'] = value
-        
-        # Parse time conditions
-        # Q1, Q2, Q3, Q4
-        quarter_match = re.search(r'Q([1-4])', description, re.IGNORECASE)
-        if quarter_match:
-            q = int(quarter_match.group(1))
-            quarter_months = {
-                1: ['Jan', 'Feb', 'Mar'],
-                2: ['Apr', 'May', 'Jun'],
-                3: ['Jul', 'Aug', 'Sep'],
-                4: ['Oct', 'Nov', 'Dec']
-            }
-            result['conditions']['month'] = quarter_months[q]
-        
-        # Specific months
-        months = ['january', 'february', 'march', 'april', 'may', 'june',
-                  'july', 'august', 'september', 'october', 'november', 'december']
-        for i, month in enumerate(months):
-            if month in desc_lower:
-                result['conditions']['month'] = month.capitalize()[:3]
-        
-        # "in summer", "in winter"
-        if 'summer' in desc_lower:
-            result['conditions']['month'] = ['Jun', 'Jul', 'Aug']
-        elif 'winter' in desc_lower:
-            result['conditions']['month'] = ['Dec', 'Jan', 'Feb']
-        elif 'spring' in desc_lower:
-            result['conditions']['month'] = ['Mar', 'Apr', 'May']
-        elif 'fall' in desc_lower or 'autumn' in desc_lower:
-            result['conditions']['month'] = ['Sep', 'Oct', 'Nov']
-        
-        # Parse category/product conditions
-        # "electronics", "premium", "gold tier"
-        tier_patterns = ['premium', 'gold', 'silver', 'bronze', 'free', 'basic', 'pro', 'enterprise']
-        for tier in tier_patterns:
-            if tier in desc_lower:
-                result['conditions']['tier'] = tier.capitalize()
-                result['target_value'] = tier.capitalize()
-        
-        # Try to infer table/column from description
-        if 'subscription' in desc_lower:
-            result['table'] = 'SUBSCRIPTIONS'
-            result['column'] = 'subscription_tier' if 'tier' in desc_lower or 'premium' in desc_lower else 'revenue'
-        elif 'revenue' in desc_lower or 'sales' in desc_lower:
-            result['table'] = 'SALES'
-            result['column'] = 'revenue' if 'revenue' in desc_lower else 'total_amount'
-        elif 'churn' in desc_lower:
-            result['table'] = 'SUBSCRIPTIONS'
-            result['column'] = 'churned'
-        
-        return result
-    
-    def _build_update_sql(
-        self,
-        table: str,
-        column: str,
-        multiplier: float = None,
-        absolute_value: float = None,
-        conditions: Dict = None
-    ) -> str:
-        """Build SQL UPDATE statement."""
-        if not table or not column:
-            return "-- Unable to generate SQL: missing table or column"
-        
-        # Build SET clause
-        if absolute_value is not None:
-            set_clause = f"{column} = {absolute_value}"
-        elif multiplier is not None:
-            set_clause = f"{column} = {column} * {multiplier}"
-        else:
-            return "-- Unable to generate SQL: no value modification specified"
-        
-        # Build WHERE clause
-        where_parts = []
-        if conditions:
-            for col, val in conditions.items():
-                if isinstance(val, list):
-                    # IN clause for multiple values
-                    values_str = ', '.join(f"'{v}'" for v in val)
-                    where_parts.append(f"{col} IN ({values_str})")
-                else:
-                    where_parts.append(f"{col} = '{val}'")
-        
-        where_clause = ' AND '.join(where_parts) if where_parts else '1=1'
-        
-        # Add randomization for realistic distribution (not all rows affected)
-        if multiplier and multiplier != 1.0:
-            # Apply to ~30% of matching rows for variation
-            where_clause += ' AND RANDOM() < 0.3'
-        
-        sql = f"""UPDATE {table}
-SET {set_clause}
-WHERE {where_clause};"""
-        
-        return sql
-    
-    def _generate_spotter_question(
-        self,
-        table: str,
-        column: str,
-        conditions: Dict,
-        description: str
-    ) -> str:
-        """Generate a Spotter question that reveals this outlier."""
-        # Base question on the pattern
-        if conditions and 'month' in conditions:
-            months = conditions['month']
-            if isinstance(months, list):
-                time_phrase = f"by month"
-            else:
-                time_phrase = f"in {months}"
-        else:
-            time_phrase = "over time"
-        
-        if 'revenue' in column.lower() or 'sales' in column.lower():
-            return f"What is the total {column} {time_phrase}?"
-        elif 'tier' in column.lower() or 'subscription' in column.lower():
-            return f"What is the breakdown of subscription tiers {time_phrase}?"
-        elif 'churn' in column.lower():
-            return f"What is the churn rate {time_phrase}?"
-        else:
-            return f"Show me {column} {time_phrase}"
-    
-    def _generate_talking_point(self, description: str) -> str:
-        """Generate a sales-ready talking point."""
-        return f"Notice how ThoughtSpot instantly surfaces this pattern: {description}. This is the kind of insight that would take hours to find manually."
-
-
-def parse_chat_adjustment(message: str) -> Dict:
-    """
-    Parse a chat adjustment request into structured form.
-    
-    Examples:
-        "decrease electronics by 10%" → 
-            {'action': 'decrease', 'entity': 'electronics', 'amount': 0.1, 'type': 'percent'}
-        
-        "make laptop 50B" → 
-            {'action': 'set', 'entity': 'laptop', 'amount': 50000000000, 'type': 'absolute'}
-        
-        "viz 3, increase premium by 20%" →
-            {'action': 'increase', 'entity': 'premium', 'amount': 0.2, 'type': 'percent', 'viz': 3}
-    """
-    result = {'raw_message': message}
-    msg_lower = message.lower()
-    
-    # Extract viz reference
-    viz_match = re.search(r'viz\s*(\d+)', msg_lower)
-    if viz_match:
-        result['viz'] = int(viz_match.group(1))
-    
-    # Extract action
-    if any(word in msg_lower for word in ['decrease', 'reduce', 'drop', 'lower']):
-        result['action'] = 'decrease'
-    elif any(word in msg_lower for word in ['increase', 'raise', 'boost', 'spike']):
-        result['action'] = 'increase'
-    elif any(word in msg_lower for word in ['set', 'make', 'change to']):
-        result['action'] = 'set'
-    
-    # Extract amount - percentage
-    pct_match = re.search(r'(\d+(?:\.\d+)?)\s*%', message)
-    if pct_match:
-        result['amount'] = float(pct_match.group(1)) / 100
-        result['type'] = 'percent'
-    
-    # Extract amount - absolute with suffix
-    abs_match = re.search(r'(\d+(?:\.\d+)?)\s*(B|M|K)(?:\s|$)', message, re.IGNORECASE)
-    if abs_match:
-        value = float(abs_match.group(1))
-        suffix = abs_match.group(2).upper()
-        if suffix == 'B':
-            value *= 1_000_000_000
-        elif suffix == 'M':
-            value *= 1_000_000
-        elif suffix == 'K':
-            value *= 1_000
-        result['amount'] = value
-        result['type'] = 'absolute'
-    
-    # Extract entity (product name, category, etc.)
-    # Remove action words and amounts to find entity
-    entity_text = msg_lower
-    for word in ['decrease', 'increase', 'reduce', 'raise', 'boost', 'spike', 
-                 'set', 'make', 'change', 'to', 'by', 'viz', '%']:
-        entity_text = entity_text.replace(word, ' ')
-    # Remove numbers
-    entity_text = re.sub(r'\d+(?:\.\d+)?(?:B|M|K)?', '', entity_text, flags=re.IGNORECASE)
-    # Clean up
-    entity_text = ' '.join(entity_text.split()).strip()
-    if entity_text:
-        result['entity'] = entity_text
-    
-    return result
-
-
-def generate_adjustment_sql(
-    adjustment: Dict,
-    schema_info: Dict,
-    schema_name: str
-) -> Tuple[str, str]:
-    """
-    Generate SQL UPDATE for a chat adjustment.
-    
-    Args:
-        adjustment: Parsed adjustment dict from parse_chat_adjustment()
-        schema_info: Schema information with tables/columns
-        schema_name: Snowflake schema name
-    
-    Returns:
-        Tuple of (sql_update, explanation)
-    """
-    entity = adjustment.get('entity', '')
-    action = adjustment.get('action', 'set')
-    amount = adjustment.get('amount', 0)
-    amount_type = adjustment.get('type', 'percent')
-    
-    # Determine which table/column to update based on entity
-    # For now, assume it's a product/category affecting revenue
-    # TODO: Make this smarter with schema analysis
-    
-    if amount_type == 'percent':
-        if action == 'decrease':
-            multiplier = 1 - amount
-        else:
-            multiplier = 1 + amount
-        set_clause = f"total_revenue = total_revenue * {multiplier}"
-    else:
-        set_clause = f"total_revenue = {amount}"
-    
-    # Build WHERE clause to find the entity
-    where_clause = f"LOWER(product_name) LIKE '%{entity.lower()}%'"
-    
-    sql = f"""UPDATE {schema_name}.SALES
-SET {set_clause}
-WHERE product_id IN (
-    SELECT product_id FROM {schema_name}.PRODUCTS
-    WHERE {where_clause}
-);"""
-    
-    explanation = f"{'Decreased' if action == 'decrease' else 'Increased'} {entity} by {amount * 100 if amount_type == 'percent' else amount}"
-    
-    return sql, explanation
-
-
-def apply_outliers(
-    snowflake_conn,
-    outliers: List[LegacyOutlierPattern],
-    schema_name: str,
-    dry_run: bool = False
-) -> List[Dict]:
-    """
-    Apply multiple outlier patterns to the database.
-    
-    Args:
-        snowflake_conn: Snowflake connection
-        outliers: List of OutlierPattern objects
-        schema_name: Schema to apply updates to
-        dry_run: If True, only print SQL without executing
-    
-    Returns:
-        List of results with success/failure for each outlier
-    """
-    results = []
-    
-    cursor = snowflake_conn.cursor()
-    
-    try:
-        cursor.execute(f"USE SCHEMA {schema_name}")
-        
-        for outlier in outliers:
-            result = {
-                'title': outlier.title,
-                'sql': outlier.sql_update,
-                'success': False,
-                'rows_affected': 0,
-                'error': None
-            }
-            
-            print(f"📊 Applying outlier: {outlier.title}", flush=True)
-            print(f"   SQL: {outlier.sql_update[:100]}...", flush=True)
-            
-            if dry_run:
-                print(f"   [DRY RUN - not executed]", flush=True)
-                result['success'] = True
-            else:
-                try:
-                    cursor.execute(outlier.sql_update)
-                    result['rows_affected'] = cursor.rowcount
-                    result['success'] = True
-                    print(f"   ✅ Affected {result['rows_affected']} rows", flush=True)
-                except Exception as e:
-                    result['error'] = str(e)
-                    print(f"   ❌ Error: {e}", flush=True)
-            
-            results.append(result)
-        
-        if not dry_run:
-            snowflake_conn.commit()
-            print(f"✅ All outliers applied and committed", flush=True)
-    
-    finally:
-        cursor.close()
-    
-    return results
-
-
-def generate_demo_pack(
-    outliers: List[LegacyOutlierPattern],
-    company_name: str,
-    use_case: str
-) -> str:
-    """
-    Generate a Demo Pack markdown document from outliers.
-    
-    Args:
-        outliers: List of OutlierPattern objects (applied or to be applied)
-        company_name: Company name for the demo
-        use_case: Use case description
-    
-    Returns:
-        Markdown string with demo documentation
-    """
-    today = datetime.now().strftime("%Y-%m-%d")
-    
-    md = f"""# {company_name} Demo Pack
-## {use_case}
-
-*Generated: {today}*
-
----
-
-## Overview
-
-This demo showcases ThoughtSpot's AI-powered analytics for {company_name}'s {use_case} use case. The data has been enhanced with realistic patterns that highlight ThoughtSpot's ability to surface business-critical insights.
-
----
-
-## Key Insights (Outliers)
-
-"""
-    
-    for i, outlier in enumerate(outliers, 1):
-        md += f"""### {i}. {outlier.title}
-
-**Pattern:** {outlier.description}
-
-**Spotter Question:** _{outlier.spotter_question or 'N/A'}_
-
-**Talking Point:** {outlier.talking_point or 'N/A'}
-
-**Technical Details:**
-- Table: `{outlier.affected_table}`
-- Column: `{outlier.affected_column}`
-- Conditions: {outlier.conditions or 'None'}
-
----
-
-"""
-    
-    md += """## Demo Flow
-
-1. **Start with overview liveboard** - Show the big picture
-2. **Ask Spotter questions** - Use the questions above to surface each insight
-3. **Drill into details** - Let ThoughtSpot guide exploration
-4. **Highlight AI capabilities** - Show how patterns were auto-detected
-
----
-
-## Notes
-
-- Data patterns are realistic but not actual customer data
-- Refresh liveboard after any data adjustments
-- Use Monitor feature to show proactive alerts
-
-"""
-    
-    return md
-
-
-# CLI interface for testing
-if __name__ == "__main__":
-    print("Outlier System - Testing")
-    print("=" * 40)
-    
-    generator = OutlierGenerator()
-    
-    # Test cases
-    test_patterns = [
-        "Premium subscriptions spike 3x in Q4",
-        "Decrease electronics revenue by 10%",
-        "Make laptop revenue 50B",
-        "Churn increases 20% in summer months"
-    ]
-    
-    for pattern in test_patterns:
-        print(f"\nPattern: {pattern}")
-        result = generator.generate_outlier_sql(pattern)
-        print(f"SQL:\n{result.sql_update}")
-        print(f"Spotter Q: {result.spotter_question}")

prompt_logger.py

@@ -11,7 +11,7 @@ Usage:
     # Option 1: Wrap litellm.completion() calls
     response = logged_completion(
         stage="ddl",
-        model="claude-sonnet-4.5",
+        model="gpt-4o",
         messages=[{"role": "user", "content": "..."}],
         max_tokens=4000
     )
@@ -20,7 +20,7 @@ Usage:
     logger = get_prompt_logger()
     logger.log_prompt(
         stage="research_company",
-        model="anthropic/claude-sonnet-4.5",
+        model="openai/gpt-4o",
         messages=[...],
         response_text="...",
         tokens_in=500,

prompts.py

@@ -438,19 +438,18 @@ def build_prompt(
         Complete prompt string ready for LLM
     """
     from demo_personas import get_use_case_config
-    from outlier_system import get_outliers_for_use_case
-    
+
     # Get merged configuration
     config = get_use_case_config(vertical, function)
-    outliers = get_outliers_for_use_case(vertical, function)
-    
+    lq = config.get("liveboard_questions", [])
+
     # Build sections
     sections = []
-    
+
     # Section A: Company Context
     sections.append(f"""## COMPANY CONTEXT
 {company_context}""")
-    
+
     # Section B: Use Case Framework
     persona = config.get("target_persona", "Business Leader")
     problem = config.get("business_problem", "Need for faster, data-driven decisions")
@@ -460,7 +459,7 @@ def build_prompt(
 - **Business Problem:** {problem}
 - **Industry Terms:** {', '.join(config.get('industry_terms', []))}
 - **Typical Entities:** {', '.join(config.get('entities', []))}""")
-    
+
     # Section C: Required KPIs and Visualizations
     kpi_text = "\n".join([f"- {kpi}: {config['kpi_definitions'].get(kpi, '')}" for kpi in config.get('kpis', [])])
     sections.append(f"""## REQUIRED KPIs
@@ -468,17 +467,18 @@ def build_prompt(
 
 ## REQUIRED VISUALIZATIONS
 {', '.join(config.get('viz_types', []))}""")
-    
-    # Section D: Outlier Patterns
-    if outliers.required:
-        outlier_text = "\n".join([f"- **{o.name}:** {o.viz_talking_point}" for o in outliers.required])
+
+    # Section D: Data stories from liveboard questions
+    required_qs = [q for q in lq if q.get("required")]
+    if required_qs:
+        outlier_text = "\n".join([f"- **{q['title']}:** {q.get('insight', '')}" for q in required_qs])
         sections.append(f"""## DATA STORIES TO CREATE
 {outlier_text}""")
-    
+
     # Section E: Spotter Questions
     spotter_qs = []
-    for o in outliers.required:
-        spotter_qs.extend(o.spotter_questions[:2])  # Top 2 from each required outlier
+    for q in required_qs:
+        spotter_qs.extend(q.get('spotter_qs', [])[:2])
     if spotter_qs:
         sections.append(f"""## SPOTTER QUESTIONS TO ENABLE
 {chr(10).join(['- ' + q for q in spotter_qs[:6]])}""")

session_logger.py

@@ -0,0 +1,204 @@
+"""
+Session Logger — Structured pipeline event logger.
+
+Writes events to both a local JSONL file and the Supabase `session_logs` table.
+Falls back gracefully to file-only logging if Supabase is unavailable.
+
+===========================================================================
+Supabase table setup (run once in Supabase SQL editor):
+===========================================================================
+
+    CREATE TABLE session_logs (
+        id BIGSERIAL PRIMARY KEY,
+        session_id TEXT NOT NULL,
+        user_email TEXT,
+        ts TIMESTAMPTZ DEFAULT NOW(),
+        stage TEXT,
+        event TEXT NOT NULL,
+        duration_ms INTEGER,
+        error TEXT,
+        meta JSONB
+    );
+    CREATE INDEX ON session_logs (session_id);
+    CREATE INDEX ON session_logs (user_email);
+    CREATE INDEX ON session_logs (ts DESC);
+
+===========================================================================
+
+Usage:
+    from session_logger import init_session_logger, get_session_logger
+
+    logger = init_session_logger("20260319_143000", user_email="alice@example.com")
+
+    t = logger.log_start("research")
+    # ... do work ...
+    logger.log_end("research", t, model="gpt-4o", rows=42)
+
+    logger.log("deploy", "Liveboard created", liveboard_id="abc-123")
+"""
+
+import json
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Optional
+
+
+# ---------------------------------------------------------------------------
+# SessionLogger class
+# ---------------------------------------------------------------------------
+
+class SessionLogger:
+    """Structured event logger that writes to file and Supabase session_logs."""
+
+    TABLE = "session_logs"
+
+    def __init__(self, session_id: str, user_email: str = None):
+        """
+        Initialize the session logger.
+
+        Args:
+            session_id: Unique ID for this build session (e.g. datetime string).
+            user_email: Email of the user running this session.
+        """
+        self.session_id = session_id
+        self.user_email = user_email
+
+        # File log path: logs/sessions/{session_id}.log (one JSON line per event)
+        script_dir = Path(__file__).parent
+        log_dir = script_dir / "logs" / "sessions"
+        try:
+            log_dir.mkdir(parents=True, exist_ok=True)
+        except Exception as e:
+            print(f"[SessionLogger] Could not create log directory {log_dir}: {e}", file=sys.stderr)
+        self._log_file = log_dir / f"{session_id}.log"
+
+        # Try to initialise Supabase — never raise
+        self._supabase_ok = False
+        self._client = None
+        self._init_supabase()
+
+    def _init_supabase(self):
+        """Attempt to connect to Supabase. Sets self._supabase_ok and self._client."""
+        try:
+            # Lazy import to avoid circular imports
+            from supabase_client import SupabaseSettings
+            ss = SupabaseSettings()
+            if ss.is_enabled():
+                self._client = ss.client
+                self._supabase_ok = True
+        except Exception as e:
+            print(f"[SessionLogger] Supabase unavailable, falling back to file-only logging: {e}",
+                  file=sys.stderr)
+
+    # ------------------------------------------------------------------
+    # Core log method
+    # ------------------------------------------------------------------
+
+    def log(self, stage: str, event: str, duration_ms: int = None,
+            error: str = None, **meta):
+        """
+        Log one pipeline event.
+
+        Args:
+            stage: Pipeline stage name (e.g. 'research', 'deploy').
+            event: Short description of what happened.
+            duration_ms: Optional elapsed time in milliseconds.
+            error: Optional error message if the event represents a failure.
+            **meta: Arbitrary key/value pairs stored in the meta JSONB column.
+        """
+        ts = datetime.now(timezone.utc).isoformat()
+        record = {
+            "session_id": self.session_id,
+            "user_email": self.user_email,
+            "ts": ts,
+            "stage": stage,
+            "event": event,
+            "duration_ms": duration_ms,
+            "error": error,
+            "meta": meta if meta else None,
+        }
+
+        # Always write to file
+        self._write_file(record)
+
+        # Write to Supabase if available
+        if self._supabase_ok:
+            self._write_supabase(record)
+
+    def _write_file(self, record: dict):
+        """Append one JSON line to the session log file. Never raises."""
+        try:
+            with open(self._log_file, "a", encoding="utf-8") as fh:
+                fh.write(json.dumps(record, default=str) + "\n")
+        except Exception as e:
+            print(f"[SessionLogger] File write failed: {e}", file=sys.stderr)
+
+    def _write_supabase(self, record: dict):
+        """Insert one row into session_logs. Never raises."""
+        try:
+            # Build the insert payload, omitting None values for cleanliness
+            payload = {k: v for k, v in record.items() if v is not None}
+            self._client.table(self.TABLE).insert(payload).execute()
+        except Exception as e:
+            # Supabase write failure is non-fatal — demote to stderr
+            print(f"[SessionLogger] Supabase write failed: {e}", file=sys.stderr)
+            # Mark Supabase as unavailable so we stop trying for this session
+            self._supabase_ok = False
+
+    # ------------------------------------------------------------------
+    # Convenience helpers
+    # ------------------------------------------------------------------
+
+    def log_start(self, stage: str) -> float:
+        """
+        Log that a pipeline stage has started.
+
+        Returns:
+            Monotonic start time (pass to log_end).
+        """
+        self.log(stage, f"{stage} started")
+        return time.monotonic()
+
+    def log_end(self, stage: str, start_time: float, error: str = None, **meta):
+        """
+        Log that a pipeline stage has ended, computing duration from start_time.
+
+        Args:
+            stage: Pipeline stage name (must match the one passed to log_start).
+            start_time: Value returned by the corresponding log_start call.
+            error: Optional error message if the stage failed.
+            **meta: Arbitrary key/value pairs stored in the meta column.
+        """
+        elapsed_ms = int((time.monotonic() - start_time) * 1000)
+        event = f"{stage} failed" if error else f"{stage} completed"
+        self.log(stage, event, duration_ms=elapsed_ms, error=error, **meta)
+
+
+# ---------------------------------------------------------------------------
+# Module-level singleton helpers
+# ---------------------------------------------------------------------------
+
+_current_logger: Optional[SessionLogger] = None
+
+
+def get_session_logger() -> Optional[SessionLogger]:
+    """Return the active SessionLogger, or None if not yet initialised."""
+    return _current_logger
+
+
+def init_session_logger(session_id: str, user_email: str = None) -> SessionLogger:
+    """
+    Create (or replace) the module-level SessionLogger.
+
+    Args:
+        session_id: Unique ID for this build session.
+        user_email: Email of the user running this session.
+
+    Returns:
+        The newly created SessionLogger instance.
+    """
+    global _current_logger
+    _current_logger = SessionLogger(session_id, user_email)
+    return _current_logger

smart_data_adjuster.py

@@ -1,78 +1,139 @@
 """
-Smart Conversational Data Adjuster
+Smart Data Adjuster
 
-Understands liveboard context and asks smart qualifying questions.
-Bundles confirmations into one step when confident.
+Understands liveboard and schema context; handles conversational, multi-turn
+data adjustment requests in natural language.
+
+Connects to:
+  - ThoughtSpot (to load liveboard viz context)
+  - Snowflake (to query and update data)
+
+Works with any configured LLM (Claude, GPT-4, etc.) via litellm.
+Schema is discovered dynamically — no hardcoded table names.
 """
 
 from typing import Dict, List, Optional, Tuple
 from snowflake_auth import get_snowflake_connection
 from thoughtspot_deployer import ThoughtSpotDeployer
 import json
-from llm_config import build_openai_chat_token_kwargs, is_openai_model_name, resolve_model_name
-from llm_client_factory import create_openai_client
+import re
+from llm_config import resolve_model_name
 
 
 class SmartDataAdjuster:
-    """Smart adjuster with liveboard context and conversational flow"""
-    
-    def __init__(self, database: str, schema: str, liveboard_guid: str, llm_model: str = None):
+    """
+    Conversational data adjuster with liveboard context and schema discovery.
+
+    Usage:
+        adjuster = SmartDataAdjuster(database, schema, liveboard_guid, llm_model)
+        adjuster.connect()
+        adjuster.load_liveboard_context()
+
+        # Per user message:
+        result = adjuster.handle_message("make webcam revenue 40B")
+        # result: {'type': 'confirmation', 'text': '...', 'pending': {...}}
+        #      or {'type': 'result', 'text': '...'}
+        #      or {'type': 'error', 'text': '...'}
+    """
+
+    def __init__(self, database: str, schema: str, liveboard_guid: str,
+                 llm_model: str = None, ts_url: str = None, ts_secret: str = None):
         self.database = database
         self.schema = schema
         self.liveboard_guid = liveboard_guid
-        self.conn = None
-        self.ts_client = None
-        
-        # LLM setup - require explicit model from settings/controller
+        self.ts_url = (ts_url or "").strip() or None
+        self.ts_secret = (ts_secret or "").strip() or None
+
         self.llm_model = (llm_model or "").strip()
         if not self.llm_model:
             raise ValueError("SmartDataAdjuster requires llm_model from settings.")
-        self._llm_client = None
-        
-        # Context about the liveboard
-        self.liveboard_name = None
-        self.visualizations = []  # List of viz metadata
-    
-    def _call_llm(self, prompt: str) -> str:
-        """Call the configured LLM (OpenAI GPT only)."""
-        target_model = resolve_model_name(self.llm_model)
-        if not is_openai_model_name(target_model):
-            raise ValueError(
-                f"SmartDataAdjuster only supports OpenAI GPT/Codex models. Received '{self.llm_model}'."
-            )
 
-        client = create_openai_client()
+        self.conn = None
+        self.ts_client = None
 
-        request_kwargs = {
-            "model": target_model,
-            "messages": [{"role": "user", "content": prompt}],
-            "temperature": 0,
-        }
-        request_kwargs.update(build_openai_chat_token_kwargs(target_model, 2000))
+        # Populated by load_liveboard_context()
+        self.liveboard_name: Optional[str] = None
+        self.visualizations: List[Dict] = []
+
+        # Populated by _discover_schema() in connect()
+        self.schema_tables: Dict[str, List[str]] = {}   # table → [col, ...]
+        self.fact_tables: List[str] = []                # tables likely to be updated
+        self.dimension_tables: Dict[str, str] = {}      # table → name_column
+
+    # ------------------------------------------------------------------
+    # Connection & schema discovery
+    # ------------------------------------------------------------------
 
-        response = client.chat.completions.create(**request_kwargs)
-        return response.choices[0].message.content
-    
     def connect(self):
-        """Connect to Snowflake and ThoughtSpot"""
-        # Snowflake
+        """Connect to Snowflake and ThoughtSpot, then discover schema."""
         self.conn = get_snowflake_connection()
         cursor = self.conn.cursor()
         cursor.execute(f"USE DATABASE {self.database}")
         cursor.execute(f'USE SCHEMA "{self.schema}"')
-        
-        # ThoughtSpot
-        self.ts_client = ThoughtSpotDeployer()
+
+        self.ts_client = ThoughtSpotDeployer(
+            base_url=self.ts_url or None,
+            secret_key=self.ts_secret or None,
+        )
         self.ts_client.authenticate()
-        
-        print(f"✅ Connected to {self.database}.{self.schema}")
-        print(f"✅ Connected to ThoughtSpot")
-    
-    def load_liveboard_context(self):
-        """Load liveboard metadata and visualization details"""
-        print(f"\n📊 Loading liveboard context...")
-        
-        # Get liveboard metadata
+
+        self._discover_schema()
+
+    def _discover_schema(self):
+        """Read actual table/column structure from INFORMATION_SCHEMA."""
+        cursor = self.conn.cursor()
+        cursor.execute(f"""
+            SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE
+            FROM {self.database}.INFORMATION_SCHEMA.COLUMNS
+            WHERE TABLE_SCHEMA = '{self.schema}'
+            ORDER BY TABLE_NAME, ORDINAL_POSITION
+        """)
+        raw: Dict[str, List[Dict]] = {}
+        for table, column, dtype in cursor.fetchall():
+            raw.setdefault(table, []).append({'name': column, 'type': dtype.upper()})
+
+        self.schema_tables = {t: [c['name'] for c in cols] for t, cols in raw.items()}
+
+        # Heuristic: dimension tables have a _NAME column; fact tables have date + numeric cols
+        for table, cols in raw.items():
+            col_names = [c['name'] for c in cols]
+            col_types = {c['name']: c['type'] for c in cols}
+
+            name_cols = [c for c in col_names if c.endswith('_NAME')]
+            num_cols = [c for c in col_names
+                        if any(t in col_types.get(c, '') for t in ('NUMBER', 'FLOAT', 'INT', 'DECIMAL', 'NUMERIC'))]
+            date_cols = [c for c in col_names
+                         if any(t in col_types.get(c, '') for t in ('DATE', 'TIME', 'TIMESTAMP'))]
+
+            if name_cols:
+                # Use the first _NAME column as the entity name column
+                self.dimension_tables[table] = name_cols[0]
+            if num_cols and date_cols:
+                self.fact_tables.append(table)
+
+        # If nothing looks like a fact table, fall back to largest table
+        if not self.fact_tables and self.schema_tables:
+            self.fact_tables = list(self.schema_tables.keys())
+
+    def _call_llm(self, prompt: str) -> str:
+        """Call the configured LLM via litellm (supports all providers)."""
+        from prompt_logger import logged_completion
+        model = resolve_model_name(self.llm_model)
+        response = logged_completion(
+            stage="data_adjuster",
+            model=model,
+            messages=[{"role": "user", "content": prompt}],
+            temperature=0,
+            max_tokens=1000,
+        )
+        return response.choices[0].message.content.strip()
+
+    # ------------------------------------------------------------------
+    # Liveboard context
+    # ------------------------------------------------------------------
+
+    def load_liveboard_context(self) -> bool:
+        """Load liveboard metadata and visualization list from ThoughtSpot."""
         response = self.ts_client.session.post(
             f"{self.ts_client.base_url}/api/rest/2.0/metadata/search",
             json={
@@ -80,544 +141,490 @@ class SmartDataAdjuster:
                 "include_visualization_headers": True
             }
         )
-        
         if response.status_code != 200:
-            print(f"❌ Failed to load liveboard")
             return False
-        
+
         data = response.json()[0]
         self.liveboard_name = data.get('metadata_name', 'Unknown Liveboard')
-        
-        viz_headers = data.get('visualization_headers', [])
-        
-        print(f"   Liveboard: {self.liveboard_name}")
-        print(f"   Visualizations: {len(viz_headers)}")
-        
-        # Extract viz details
-        for viz in viz_headers:
+
+        for viz in data.get('visualization_headers', []):
             name = viz.get('name', '')
-            viz_id = viz.get('id')
-            
-            # Skip note tiles
             if 'note-tile' in name.lower():
                 continue
-            
-            # Parse the name to extract columns used
-            # Names like "top 10 product_name by total revenue"
-            viz_info = {
-                'id': viz_id,
-                'name': name,
-                'columns': self._extract_columns_from_name(name)
-            }
-            
-            self.visualizations.append(viz_info)
-            print(f"   - {name}")
-        
-        return True
-    
-    def _extract_columns_from_name(self, name: str) -> List[str]:
-        """Extract column names from visualization name"""
-        # Simple heuristic: look for column-like words
-        # e.g., "top 10 product_name by total revenue" → [product_name, total_revenue]
-        
-        columns = []
-        name_lower = name.lower()
-        
-        # Common column patterns
-        if 'product_name' in name_lower:
-            columns.append('PRODUCT_NAME')
-        if 'total revenue' in name_lower or 'total_revenue' in name_lower:
-            columns.append('TOTAL_AMOUNT')
-        if 'quantity' in name_lower:
-            columns.append('QUANTITY_SOLD')
-        if 'profit margin' in name_lower or 'profit_margin' in name_lower:
-            columns.append('PROFIT_MARGIN')
-        if 'customer_segment' in name_lower:
-            columns.append('CUSTOMER_SEGMENT')
-        if 'category' in name_lower:
-            columns.append('CATEGORY')
-        if 'seller' in name_lower:
-            columns.append('SELLER_NAME')
-        
-        return columns
-    
-    def _simple_parse(self, message: str) -> Optional[Dict]:
-        """Simple regex-based parser for common patterns like 'decrease phone case by 10%' or 'decrease seller acme by 10%'"""
-        import re
-        
-        print(f"🔍 DEBUG _simple_parse: message='{message}'")
-        msg_lower = message.lower()
-        
-        # Detect if user specified a viz number
-        viz_match = re.search(r'(?:viz|visualization)\s+(\d+)', msg_lower)
-        viz_number = int(viz_match.group(1)) if viz_match else None
-        
-        # Detect entity type (product or seller)
-        # Check explicit "seller" keyword, or infer from viz number
-        is_seller = 'seller' in msg_lower
-        
-        # If viz number is specified, check if it's a seller viz
-        if viz_number and not is_seller and len(self.visualizations) >= viz_number:
-            viz = self.visualizations[viz_number - 1]
-            if 'seller' in viz['name'].lower():
-                is_seller = True
-        
-        entity_type = 'seller' if is_seller else 'product'
-        
-        # Extract entity name - try quotes first, then words after action verbs
-        entity_match = re.search(r'"([^"]+)"', message)
-        if not entity_match:
-            # Try to find entity name after action words, but stop before numbers
-            # Include "seller" keyword if present
-            if is_seller:
-                # Match: "decrease seller home depot by 20%"
-                action_pattern = r'(?:decrease|increase|make|set|adjust)\s+(?:the\s+)?(?:profit\s+margin\s+for\s+)?seller\s+([a-z\s]+?)(?:\s+\d|\s+by|\s+to|\s*$)'
-            else:
-                # Match: "decrease bluetooth speaker by 10%" OR "decrease the revenue for bluetooth speaker by 10%"
-                action_pattern = r'(?:decrease|increase|make|set|adjust)\s+(?:the\s+)?(?:revenue\s+for\s+|profit\s+margin\s+for\s+)?([a-z\s]+?)(?:\s+\d|\s+by|\s+to|\s*$)'
-            entity_match = re.search(action_pattern, msg_lower, re.I)
-        
-        if not entity_match:
+            self.visualizations.append({'id': viz.get('id'), 'name': name})
+
+        return bool(self.visualizations)
+
+    # ------------------------------------------------------------------
+    # Entity matching (schema-aware)
+    # ------------------------------------------------------------------
+
+    def _fuzzy_match(self, target: str, candidates: List[str]) -> Optional[str]:
+        """Return the best matching candidate for target, or None."""
+        def norm(s):
+            return s.lower().replace(' ', '').replace('-', '').replace('_', '')
+
+        t = norm(target)
+        t_lower = target.lower()
+
+        for c in candidates:
+            if c.lower() == t_lower:
+                return c
+        for c in candidates:
+            if norm(c) == t:
+                return c
+        for c in candidates:
+            if t_lower in c.lower() or c.lower() in t_lower:
+                return c
+        for c in candidates:
+            if t in norm(c) or norm(c) in t:
+                return c
+        return None
+
+    def _find_entity(self, entity_value: str, entity_type_hint: str = None) -> Tuple[Optional[str], Optional[str], Optional[str]]:
+        """
+        Find the closest matching entity name in any dimension table.
+
+        Returns: (matched_name, table_name, name_column) or (None, None, None)
+        """
+        cursor = self.conn.cursor()
+
+        # Sort dimension tables: prefer ones whose name matches entity_type_hint
+        tables_to_try = list(self.dimension_tables.items())
+        if entity_type_hint:
+            hint = entity_type_hint.lower()
+            tables_to_try.sort(key=lambda x: 0 if hint in x[0].lower() else 1)
+
+        for table, name_col in tables_to_try:
+            cursor.execute(f'SELECT DISTINCT {name_col} FROM {self.database}."{self.schema}".{table}')
+            candidates = [row[0] for row in cursor.fetchall() if row[0]]
+            match = self._fuzzy_match(entity_value, candidates)
+            if match:
+                return match, table, name_col
+
+        return None, None, None
+
+    def _find_fact_join(self, dim_table: str) -> Optional[Tuple[str, str, str]]:
+        """
+        Find a fact table that has an FK column referencing dim_table.
+        Returns: (fact_table, fact_fk_column, dim_pk_column) or None.
+        """
+        # Look for an ID column in dim_table
+        dim_cols = self.schema_tables.get(dim_table, [])
+        dim_id = next((c for c in dim_cols if c.endswith('_ID')), None)
+        if not dim_id:
             return None
-        
-        entity = entity_match.group(1).strip()
-        
-        # Find percentage or absolute value
-        is_percentage = False
+
+        # Look for that column in fact tables
+        for ft in self.fact_tables:
+            if ft == dim_table:
+                continue
+            ft_cols = self.schema_tables.get(ft, [])
+            if dim_id in ft_cols:
+                return ft, dim_id, dim_id
+
+        return None
+
+    # ------------------------------------------------------------------
+    # Request interpretation
+    # ------------------------------------------------------------------
+
+    def _parse_request_simple(self, message: str) -> Optional[Dict]:
+        """
+        Fast regex-based parser for common patterns:
+          "decrease webcam by 10%", "make laptop 50B", "increase revenue for acme by 20%"
+        Returns parsed dict or None if pattern not matched.
+        """
+        msg_lower = message.lower()
+
+        # Percentage match: "by 20%", "-10%"
+        pct_match = re.search(r'by\s+(-?\d+\.?\d*)%|(-?\d+\.?\d*)%', msg_lower)
+        # Absolute value: "50B", "50 billion", "1.5M", "1000000"
+        val_match = re.search(r'(\d+\.?\d*)\s*(b(?:illion)?|m(?:illion)?|k(?:thousand)?)\b', msg_lower)
+        bare_num = re.search(r'\b(\d{4,})\b', message)  # bare large integer
+
+        is_percentage = bool(pct_match)
         percentage = None
         target_value = None
-        
-        # Look for percentage like "by 10%" or "10%"
-        pct_match = re.search(r'by\s+(\d+\.?\d*)%|(\d+\.?\d*)%', msg_lower)
+
         if pct_match:
-            is_percentage = True
-            percentage = float(pct_match.group(1) or pct_match.group(2))
-            # Check if it's decrease or increase
-            if 'decrease' in msg_lower or 'reduce' in msg_lower or 'lower' in msg_lower:
-                percentage = -percentage
+            raw_pct = float(pct_match.group(1) or pct_match.group(2))
+            if any(w in msg_lower for w in ('decrease', 'reduce', 'lower', 'drop', 'cut')):
+                raw_pct = -abs(raw_pct)
+            percentage = raw_pct
+        elif val_match:
+            num = float(val_match.group(1))
+            unit = val_match.group(2)[0].lower()
+            multipliers = {'b': 1e9, 'm': 1e6, 'k': 1e3}
+            target_value = num * multipliers.get(unit, 1)
+        elif bare_num:
+            target_value = float(bare_num.group(1))
         else:
-            # Look for absolute value like "50B", "50 billion", "1000000"
-            val_match = re.search(r'(\d+\.?\d*)\s*([bBmMkK]|billion|million|thousand)?', message)
-            if val_match:
-                num = float(val_match.group(1))
-                unit = (val_match.group(2) or '').lower()
-                if unit in ['b', 'billion']:
-                    target_value = num * 1_000_000_000
-                elif unit in ['m', 'million']:
-                    target_value = num * 1_000_000
-                elif unit in ['k', 'thousand']:
-                    target_value = num * 1_000
-                else:
-                    target_value = num
-        
-        if not is_percentage and not target_value:
             return None
-        
-        # Find appropriate viz and determine metric column
-        # If user specified viz number, use it; otherwise search for matching viz
-        if viz_number:
-            viz_num = viz_number
-        elif is_seller:
-            # Look for seller-related viz
-            viz_num = 1  # Default
-            for i, viz in enumerate(self.visualizations, 1):
-                if 'seller' in viz['name'].lower():
-                    viz_num = i
-                    break
-        else:
-            # Look for product-related viz
-            viz_num = 1  # Default
-            for i, viz in enumerate(self.visualizations, 1):
-                if 'product' in viz['name'].lower():
-                    viz_num = i
-                    break
-        
-        # Determine metric based on entity type
-        if is_seller:
-            metric_column = 'PROFIT_MARGIN'  # Sellers typically use profit margin
+
+        # Extract entity name: quoted or after action verb
+        entity = None
+        quoted = re.search(r'"([^"]+)"', message)
+        if quoted:
+            entity = quoted.group(1).strip()
         else:
-            metric_column = 'TOTAL_AMOUNT'  # Products typically use revenue (column is TOTAL_AMOUNT)
-        
-        result = {
-            'viz_number': viz_num,
+            # "make/set/increase/decrease/adjust <entity> [by/to]"
+            action_pat = r'(?:make|set|increase|decrease|reduce|boost|lower|adjust|change)\s+(?:the\s+)?(?:\w+\s+(?:for|of)\s+)?([a-z0-9][\w\s-]*?)(?:\s+(?:by|to|revenue|sales|at)\b|\s+\d|$)'
+            am = re.search(action_pat, msg_lower, re.I)
+            if am:
+                entity = am.group(1).strip()
+
+        if not entity:
+            return None
+
+        # Detect entity type from keywords
+        entity_type = None
+        for kw in ('seller', 'vendor', 'customer', 'product', 'item', 'region', 'store'):
+            if kw in msg_lower:
+                entity_type = kw
+                break
+
+        return {
             'entity_value': entity,
             'entity_type': entity_type,
-            'metric_column': metric_column,
-            'target_value': target_value,
             'is_percentage': is_percentage,
             'percentage': percentage,
+            'target_value': target_value,
             'confidence': 'medium',
-            'reasoning': f'Simple {entity_type} parse'
         }
-        
-        print(f"🔍 DEBUG _simple_parse result: entity='{entity}', percentage={percentage}, metric={metric_column}, viz_num={viz_num}")
-        
-        if viz_num <= len(self.visualizations):
-            result['viz'] = self.visualizations[viz_num - 1]
-        
-        return result
-    
+
     def match_request_to_viz(self, user_request: str) -> Optional[Dict]:
         """
-        Use AI to match user request to specific visualization
-        
-        Returns:
-            {
-                'viz': {...},
-                'confidence': 'high'|'medium'|'low',
-                'entity_value': '1080p Webcam',
-                'metric_column': 'TOTAL_AMOUNT',
-                'target_value': 50000000000
-            }
+        Parse request and enrich with schema context.
+        Returns structured match dict or None.
         """
-        # Try simple parse first (faster, no AI needed)
-        simple_result = self._simple_parse(user_request)
-        if simple_result:
-            print(f"   ⚡ Quick parse: '{simple_result['entity_value']}' → {simple_result.get('percentage', simple_result.get('target_value'))}")
-            return simple_result
-        
-        viz_list = "\n".join([
-            f"{i+1}. {v['name']} (columns: {', '.join(v['columns'])})"
-            for i, v in enumerate(self.visualizations)
-        ])
-        
-        prompt = f"""User is looking at a ThoughtSpot liveboard and wants to adjust data.
-
-User request: "{user_request}"
-
-Available visualizations on the liveboard:
-{viz_list}
-
-Analyze the request and determine:
-1. Which visualization (by number) is the user referring to?
-2. What entity/product are they talking about? (e.g., "1080p Webcam")
-3. What metric should be adjusted? (TOTAL_AMOUNT, QUANTITY_SOLD, PROFIT_MARGIN)
-4. What's the target value? 
-   - If absolute value (e.g., "40B", "100M"): convert to number (40B = 40000000000)
-   - If percentage increase (e.g., "increase by 20%"): set is_percentage=true and percentage=20
-5. How confident are you? (high/medium/low)
-
-Return JSON:
-{{
-  "viz_number": 1,
-  "entity_value": "1080p Webcam",
-  "metric_column": "TOTAL_AMOUNT",
-  "target_value": 50000000000,
-  "is_percentage": false,
-  "percentage": null,
-  "confidence": "high",
-  "reasoning": "User mentioned product and the top 10 products viz uses PRODUCT_NAME and TOTAL_AMOUNT"
-}}
-
-OR for percentage increase:
+        result = self._parse_request_simple(user_request)
+
+        if not result:
+            # Fall back to LLM for complex requests
+            schema_summary = "\n".join(
+                f"  {t}: {', '.join(cols[:8])}" + (' ...' if len(cols) > 8 else '')
+                for t, cols in self.schema_tables.items()
+            )
+            viz_summary = "\n".join(f"  {i+1}. {v['name']}" for i, v in enumerate(self.visualizations))
+            prompt = f"""Parse this data adjustment request.
+
+Request: "{user_request}"
+
+Snowflake schema tables:
+{schema_summary}
+
+Liveboard visualizations:
+{viz_summary}
+
+Return JSON with these fields (numbers only, not strings):
 {{
-  "viz_number": 1,
-  "entity_value": "1080p Webcam",
-  "metric_column": "TOTAL_AMOUNT",
-  "target_value": null,
-  "is_percentage": true,
-  "percentage": 20,
-  "confidence": "high",
-  "reasoning": "User wants to increase revenue by 20%"
-}}
-
-CRITICAL: target_value and percentage must be numbers, never strings.
-If unsure about ANY field, set confidence to "low" or "medium".
-"""
-        
-        content = self._call_llm(prompt)
-        if content.startswith('```'):
-            lines = content.split('\n')
-            content = '\n'.join(lines[1:-1])
-        
-        try:
-            result = json.loads(content)
-            
-            # Add the actual viz object
-            viz_num = result.get('viz_number', 1)
-            if 1 <= viz_num <= len(self.visualizations):
-                result['viz'] = self.visualizations[viz_num - 1]
-            
-            return result
-        except:
-            return None
-    
-    def _find_closest_entity(self, entity_value: str, entity_type: str = 'product') -> Optional[str]:
-        """Find the closest matching entity name (product or seller) in the database"""
-        cursor = self.conn.cursor()
-        
-        # Get all entity names based on type
-        if entity_type == 'seller':
-            cursor.execute(f"""
-                SELECT DISTINCT SELLER_NAME 
-                FROM {self.database}."{self.schema}".SELLERS
-            """)
-        else:  # product
-            cursor.execute(f"""
-                SELECT DISTINCT PRODUCT_NAME 
-                FROM {self.database}."{self.schema}".PRODUCTS
-            """)
-        
-        entities = [row[0] for row in cursor.fetchall()]
-        
-        # Normalize: lowercase and remove spaces for comparison
-        def normalize(s):
-            return s.lower().replace(' ', '').replace('-', '').replace('_', '')
-        
-        entity_normalized = normalize(entity_value)
-        
-        # First try exact case-insensitive match
-        entity_lower = entity_value.lower()
-        for entity in entities:
-            if entity.lower() == entity_lower:
-                return entity
-        
-        # Try normalized match (ignoring spaces/dashes)
-        for entity in entities:
-            if normalize(entity) == entity_normalized:
-                return entity
-        
-        # Try partial match (contains)
-        for entity in entities:
-            if entity_lower in entity.lower() or entity.lower() in entity_lower:
-                return entity
-        
-        # Try normalized partial match
-        for entity in entities:
-            if entity_normalized in normalize(entity) or normalize(entity) in entity_normalized:
-                return entity
-        
-        return None
-    
-    def _find_closest_product(self, entity_value: str) -> Optional[str]:
-        """Backward compatibility wrapper"""
-        return self._find_closest_entity(entity_value, 'product')
-    
-    def get_current_value(self, entity_value: str, metric_column: str, entity_type: str = 'product') -> float:
-        """Query current value from Snowflake"""
+  "entity_value": "the entity name to adjust (e.g. '1080p Webcam')",
+  "entity_type": "product|seller|customer|region|null",
+  "is_percentage": true|false,
+  "percentage": <number or null>,
+  "target_value": <number or null>,
+  "metric_hint": "keyword like 'revenue', 'sales', 'profit_margin', or null",
+  "confidence": "high|medium|low"
+}}"""
+            try:
+                raw = self._call_llm(prompt)
+                if raw.startswith('```'):
+                    raw = '\n'.join(raw.split('\n')[1:-1])
+                result = json.loads(raw)
+            except Exception:
+                return None
+
+        return result if result else None
+
+    # ------------------------------------------------------------------
+    # Value retrieval & SQL generation
+    # ------------------------------------------------------------------
+
+    def get_current_value(self, entity_value: str, metric_column: str,
+                          entity_type: str = None) -> Tuple[float, Optional[str], Optional[str], Optional[str]]:
+        """
+        Query current aggregate value for entity from Snowflake.
+
+        Returns: (current_value, matched_entity, dim_table, fact_table)
+        """
+        matched, dim_table, name_col = self._find_entity(entity_value, entity_type)
+        if not matched:
+            return 0.0, None, None, None
+
+        join_info = self._find_fact_join(dim_table) if dim_table else None
+
         cursor = self.conn.cursor()
-        
-        # Find closest matching entity
-        matched_entity = self._find_closest_entity(entity_value, entity_type)
-        
-        if not matched_entity:
-            print(f"⚠️  Could not find {entity_type} matching '{entity_value}'")
-            return 0
-        
-        if matched_entity.lower() != entity_value.lower():
-            print(f"   📝 Using closest match: '{matched_entity}'")
-        
-        # Build query based on entity type
-        if entity_type == 'seller':
+        if join_info:
+            fact_table, fk_col, dim_pk_col = join_info
+            dim_cols = self.schema_tables.get(dim_table, [])
+            dim_pk = next((c for c in dim_cols if c.endswith('_ID')), dim_pk_col)
             query = f"""
-            SELECT AVG(st.{metric_column})
-            FROM {self.database}."{self.schema}".SALES_TRANSACTIONS st
-            JOIN {self.database}."{self.schema}".SELLERS s ON st.SELLER_ID = s.SELLER_ID
-            WHERE LOWER(s.SELLER_NAME) = LOWER('{matched_entity}')
+                SELECT SUM(f.{metric_column})
+                FROM {self.database}."{self.schema}".{fact_table} f
+                JOIN {self.database}."{self.schema}".{dim_table} d
+                  ON f.{fk_col} = d.{dim_pk}
+                WHERE LOWER(d.{name_col}) = LOWER('{matched}')
             """
-        else:  # product
+        else:
+            # entity is directly in the table with the metric
             query = f"""
-            SELECT SUM(st.{metric_column})
-            FROM {self.database}."{self.schema}".SALES_TRANSACTIONS st
-            JOIN {self.database}."{self.schema}".PRODUCTS p ON st.PRODUCT_ID = p.PRODUCT_ID
-            WHERE LOWER(p.PRODUCT_NAME) = LOWER('{matched_entity}')
+                SELECT SUM({metric_column})
+                FROM {self.database}."{self.schema}".{dim_table}
+                WHERE LOWER({name_col}) = LOWER('{matched}')
             """
-        
-        cursor.execute(query)
-        result = cursor.fetchone()
-        return float(result[0]) if result and result[0] else 0
-    
-    def generate_strategy(self, entity_value: str, metric_column: str, current_value: float, target_value: float = None, percentage: float = None, entity_type: str = 'product') -> Dict:
-        """Generate the best strategy (default to Strategy A for now)"""
-        
-        print(f"🔍 DEBUG generate_strategy: entity='{entity_value}', metric={metric_column}, percentage={percentage}, current={current_value}")
-        
-        # Find the actual entity name
-        matched_entity = self._find_closest_entity(entity_value, entity_type)
-        if not matched_entity:
-            matched_entity = entity_value  # Fallback
-        
-        print(f"🔍 DEBUG matched_entity: '{matched_entity}'")
-        
-        # Calculate multiplier
+
+        try:
+            cursor.execute(query)
+            row = cursor.fetchone()
+            value = float(row[0]) if row and row[0] is not None else 0.0
+            fact_table_used = join_info[0] if join_info else dim_table
+            return value, matched, dim_table, fact_table_used
+        except Exception as e:
+            print(f"[SmartDataAdjuster] get_current_value query failed: {e}")
+            return 0.0, None, None, None
+
+    def _pick_metric_column(self, metric_hint: str = None) -> Optional[str]:
+        """Choose the best metric column from fact tables based on hint."""
+        # Build a list of all numeric-looking columns across fact tables
+        candidates = []
+        for ft in self.fact_tables:
+            for col in self.schema_tables.get(ft, []):
+                if any(kw in col.upper() for kw in ('AMOUNT', 'REVENUE', 'TOTAL', 'SALES', 'VALUE', 'MARGIN', 'PROFIT', 'COST', 'PRICE')):
+                    candidates.append((ft, col))
+
+        if not candidates:
+            return None
+
+        if metric_hint:
+            hint_upper = metric_hint.upper()
+            for ft, col in candidates:
+                if hint_upper in col:
+                    return col
+
+        # Default: prefer TOTAL_AMOUNT, REVENUE, then first available
+        for preferred in ('TOTAL_AMOUNT', 'TOTAL_REVENUE', 'REVENUE', 'AMOUNT'):
+            for ft, col in candidates:
+                if col == preferred:
+                    return col
+
+        return candidates[0][1] if candidates else None
+
+    def generate_strategy(self, entity_value: str, metric_column: str,
+                          current_value: float, target_value: float = None,
+                          percentage: float = None, entity_type: str = None) -> Dict:
+        """Generate an UPDATE strategy based on the adjustment request."""
+        matched, dim_table, name_col = self._find_entity(entity_value, entity_type)
+        if not matched:
+            matched = entity_value
+
         if percentage is not None:
-            # Percentage-based: "decrease by 10%" means multiply by 0.9
             multiplier = 1 + (percentage / 100)
-            percentage_change = percentage
-            target_value = current_value * multiplier
+            pct_change = percentage
+            if target_value is None:
+                target_value = current_value * multiplier
+        elif target_value and current_value > 0:
+            multiplier = target_value / current_value
+            pct_change = (multiplier - 1) * 100
         else:
-            # Absolute target value
-            multiplier = target_value / current_value if current_value > 0 else 1
-            percentage_change = (multiplier - 1) * 100
-        
-        # Build SQL based on entity type
-        if entity_type == 'seller':
-            sql = f"""UPDATE {self.database}."{self.schema}".SALES_TRANSACTIONS
+            multiplier = 1.0
+            pct_change = 0.0
+
+        join_info = self._find_fact_join(dim_table) if dim_table else None
+
+        if join_info:
+            fact_table, fk_col, _ = join_info
+            dim_cols = self.schema_tables.get(dim_table, [])
+            dim_pk = next((c for c in dim_cols if c.endswith('_ID')), fk_col)
+            sql = f"""UPDATE {self.database}."{self.schema}".{fact_table}
 SET {metric_column} = {metric_column} * {multiplier:.6f}
-WHERE SELLER_ID IN (
-    SELECT SELLER_ID FROM {self.database}."{self.schema}".SELLERS
-    WHERE LOWER(SELLER_NAME) = LOWER('{matched_entity}')
+WHERE {fk_col} IN (
+    SELECT {dim_pk}
+    FROM {self.database}."{self.schema}".{dim_table}
+    WHERE LOWER({name_col}) = LOWER('{matched}')
 )"""
-        else:  # product
-            sql = f"""UPDATE {self.database}."{self.schema}".SALES_TRANSACTIONS
+        elif dim_table:
+            sql = f"""UPDATE {self.database}."{self.schema}".{dim_table}
 SET {metric_column} = {metric_column} * {multiplier:.6f}
-WHERE PRODUCT_ID IN (
-    SELECT PRODUCT_ID FROM {self.database}."{self.schema}".PRODUCTS
-    WHERE LOWER(PRODUCT_NAME) = LOWER('{matched_entity}')
-)"""
-        
-        print(f"🔍 DEBUG SQL generated:\n{sql}")
-        
+WHERE LOWER({name_col}) = LOWER('{matched}')"""
+        else:
+            sql = f"-- Could not determine table structure for '{entity_value}'"
+
         return {
             'id': 'A',
-            'name': 'Distribute Across All Transactions',
-            'description': f"Multiply all transactions by {multiplier:.2f}x ({percentage_change:+.1f}%)",
+            'name': 'Scale All Transactions',
+            'description': f"Multiply all rows for '{matched}' by {multiplier:.3f}x ({pct_change:+.1f}%)",
             'sql': sql,
-            'matched_product': matched_entity,  # Keep key name for compatibility
-            'target_value': target_value
+            'matched_entity': matched,
+            'target_value': target_value,
         }
-    
-    def present_smart_confirmation(self, match: Dict, current_value: float, strategy: Dict) -> str:
-        """Create a bundled confirmation prompt"""
-        
-        viz_name = match['viz']['name']
-        entity = match['entity_value']
-        matched_product = strategy.get('matched_product', entity)
-        metric = match['metric_column']
-        target = strategy.get('target_value', match.get('target_value'))  # Use calculated target from strategy
-        confidence = match['confidence']
-        
-        # Show if we fuzzy matched
-        entity_display = entity
-        if matched_product.lower() != entity.lower():
-            entity_display = f"{entity} → '{matched_product}'"
-        
-        confirmation = f"""
-{'='*80}
-📋 SMART CONFIRMATION
-{'='*80}
-
-Liveboard: {self.liveboard_name}
-Visualization: [{viz_name}]
-
-Adjustment:
-  Entity: {entity_display}
-  Metric: {metric}
-  Current Value: ${current_value:,.0f}
-  Target Value: ${target:,.0f}
-  Change: ${target - current_value:+,.0f} ({(target/current_value - 1)*100:+.1f}%)
-
-Strategy: {strategy['name']}
-  {strategy['description']}
-
-Confidence: {confidence.upper()}
-{match.get('reasoning', '')}
-
-SQL Preview:
-{strategy['sql'][:200]}...
 
-"""
-        
-        if confidence == 'low':
-            confirmation += "\n⚠️  Low confidence - please verify this is correct\n"
-        
-        confirmation += "\n" + "="*80 + "\n"
-        return confirmation
-    
+    def present_smart_confirmation(self, match: Dict, current_value: float,
+                                   strategy: Dict, metric_column: str) -> str:
+        """Format a human-readable confirmation message."""
+        entity = match.get('entity_value', '?')
+        matched = strategy.get('matched_entity', entity)
+        target = strategy.get('target_value', 0) or 0
+
+        if matched.lower() != entity.lower():
+            entity_display = f"{entity} → **{matched}**"
+        else:
+            entity_display = f"**{matched}**"
+
+        change = target - current_value
+        pct = (change / current_value * 100) if current_value else 0
+
+        lines = [
+            f"**Liveboard:** {self.liveboard_name}",
+            f"**Entity:** {entity_display}",
+            f"**Metric:** `{metric_column}`",
+            f"**Current:** {current_value:,.0f}",
+            f"**Target:** {target:,.0f}  ({change:+,.0f} / {pct:+.1f}%)",
+            f"**Strategy:** {strategy['description']}",
+            "",
+            f"```sql\n{strategy['sql']}\n```",
+        ]
+
+        if match.get('confidence') == 'low':
+            lines.append("\n⚠️ Low confidence — please verify before confirming.")
+
+        return "\n".join(lines)
+
+    # ------------------------------------------------------------------
+    # SQL execution
+    # ------------------------------------------------------------------
+
     def execute_sql(self, sql: str) -> Dict:
-        """Execute the SQL update"""
-        print(f"🔍 DEBUG execute_sql: About to execute SQL")
-        print(f"SQL:\n{sql}")
+        """Execute an UPDATE statement. Returns success/error dict."""
         cursor = self.conn.cursor()
-        
         try:
             cursor.execute(sql)
             rows_affected = cursor.rowcount
             self.conn.commit()
-            print(f"✅ SQL executed successfully, rows affected: {rows_affected}")
-            
-            return {
-                'success': True,
-                'rows_affected': rows_affected
-            }
+            return {'success': True, 'rows_affected': rows_affected}
         except Exception as e:
-            self.conn.rollback()
-            return {
-                'success': False,
-                'error': str(e)
-            }
-    
+            try:
+                self.conn.rollback()
+            except Exception:
+                pass
+            return {'success': False, 'error': str(e)}
+
+    # ------------------------------------------------------------------
+    # Teardown
+    # ------------------------------------------------------------------
+
     def close(self):
-        """Close connections"""
+        """Close Snowflake connection."""
         if self.conn:
-            self.conn.close()
-
-
-def test_smart_adjuster():
-    """Test the smart adjuster"""
-    from dotenv import load_dotenv
-    load_dotenv()
-    
-    print("""
-╔════════════════════════════════════════════════════════════╗
-║                                                            ║
-║          Smart Data Adjuster Test                         ║
-║                                                            ║
-╚═══════════════════════════════��════════════════════════════╝
-    """)
-    
-    adjuster = SmartDataAdjuster(
-        database=get_admin_setting('SNOWFLAKE_DATABASE'),
-        schema="20251116_140933_AMAZO_SAL",
-        liveboard_guid="9a30c9e4-efba-424a-8359-b16eb3a43ec3"
+            try:
+                self.conn.close()
+            except Exception:
+                pass
+
+
+# ---------------------------------------------------------------------------
+# Liveboard-first context loader
+# ---------------------------------------------------------------------------
+
+def load_context_from_liveboard(liveboard_guid: str, ts_client) -> dict:
+    """
+    Resolve Snowflake database/schema from a liveboard GUID.
+
+    Flow:
+      liveboard TML (export_fqn=True)
+        → model GUID from visualizations[n].answer.tables[0].fqn
+        → model TML
+        → database / schema from model.tables[0].table.{db, schema}
+
+    Args:
+        liveboard_guid: ThoughtSpot liveboard GUID
+        ts_client: Authenticated ThoughtSpotDeployer instance
+
+    Returns:
+        dict with keys: liveboard_name, model_guid, model_name, database, schema
+
+    Raises:
+        ValueError if any step fails to resolve.
+    """
+    import yaml
+
+    # Step 1: Export liveboard TML with FQNs
+    response = ts_client.session.post(
+        f"{ts_client.base_url}/api/rest/2.0/metadata/tml/export",
+        json={
+            "metadata": [{"identifier": liveboard_guid}],
+            "export_associated": False,
+            "export_fqn": True,
+            "format_type": "YAML",
+        }
     )
-    
-    adjuster.connect()
-    adjuster.load_liveboard_context()
-    
-    # Test request
-    user_request = "make 1080p Webcam 50 billion"
-    print(f"\n💬 User: \"{user_request}\"")
-    
-    # Match to viz
-    print(f"\n🤔 Analyzing request...")
-    match = adjuster.match_request_to_viz(user_request)
-    
-    if not match:
-        print("❌ Could not understand request")
-        return
-    
-    # Get current value
-    current = adjuster.get_current_value(match['entity_value'], match['metric_column'])
-    
-    # Generate strategy (handle both absolute and percentage)
-    strategy = adjuster.generate_strategy(
-        match['entity_value'],
-        match['metric_column'],
-        current,
-        target_value=match.get('target_value'),
-        percentage=match.get('percentage')
+    if response.status_code != 200:
+        raise ValueError(
+            f"Failed to export liveboard TML ({response.status_code}): {response.text[:300]}"
+        )
+
+    tml_data = response.json()
+    if not tml_data:
+        raise ValueError("Empty response from liveboard TML export")
+
+    lb_tml = yaml.safe_load(tml_data[0]['edoc'])
+    liveboard_name = lb_tml.get('liveboard', {}).get('name', 'Unknown Liveboard')
+
+    # Step 2: Find model GUID from first visualization with answer.tables[].fqn
+    model_guid = None
+    for viz in lb_tml.get('liveboard', {}).get('visualizations', []):
+        for t in viz.get('answer', {}).get('tables', []):
+            fqn = t.get('fqn')
+            if fqn:
+                model_guid = fqn
+                break
+        if model_guid:
+            break
+
+    if not model_guid:
+        raise ValueError(
+            "Could not find model GUID in liveboard TML — "
+            "make sure the liveboard has at least one answer-based visualization."
+        )
+
+    # Step 3: Export model TML to get database/schema
+    response = ts_client.session.post(
+        f"{ts_client.base_url}/api/rest/2.0/metadata/tml/export",
+        json={
+            "metadata": [{"identifier": model_guid, "type": "LOGICAL_TABLE"}],
+            "export_associated": False,
+            "export_fqn": True,
+            "format_type": "YAML",
+        }
     )
-    
-    # Present confirmation
-    confirmation = adjuster.present_smart_confirmation(match, current, strategy)
-    print(confirmation)
-    
-    # Ask for confirmation
-    response = input("Run SQL? [yes/no]: ").strip().lower()
-    
-    if response in ['yes', 'y']:
-        result = adjuster.execute_sql(strategy['sql'])
-        if result['success']:
-            print(f"\n✅ Success! Updated {result['rows_affected']} rows")
-        else:
-            print(f"\n❌ Failed: {result['error']}")
-    else:
-        print("\n❌ Cancelled")
-    
-    adjuster.close()
+    if response.status_code != 200:
+        raise ValueError(
+            f"Failed to export model TML ({response.status_code}): {response.text[:300]}"
+        )
+
+    tml_data = response.json()
+    model_tml = yaml.safe_load(tml_data[0]['edoc'])
+    model_name = model_tml.get('model', {}).get('name', 'Unknown Model')
 
+    # Step 4: Extract db/schema from first model table entry
+    tables = model_tml.get('model', {}).get('tables', [])
+    if not tables:
+        raise ValueError("No tables found in model TML")
 
-if __name__ == "__main__":
-    test_smart_adjuster()
+    first_table = tables[0].get('table', {})
+    database = first_table.get('db')
+    schema = first_table.get('schema')
+
+    if not database or not schema:
+        raise ValueError(
+            f"Could not resolve database/schema from model TML "
+            f"(db={database!r}, schema={schema!r})"
+        )
 
+    return {
+        'liveboard_name': liveboard_name,
+        'model_guid': model_guid,
+        'model_name': model_name,
+        'database': database,
+        'schema': schema,
+    }

snowflake_auth.py

@@ -6,14 +6,10 @@ Supports multiple private key formats for cloud deployment flexibility:
 - Base64-encoded PEM (single line, recommended for HF Spaces)
 - Newline-escaped PEM (\\n replaced with actual newlines)
 """
-import os
 import base64
-from dotenv import load_dotenv
 from cryptography.hazmat.primitives import serialization
 from cryptography.hazmat.primitives.serialization import load_pem_private_key
-
-# Load environment variables (no-op if not using .env file)
-load_dotenv()
+from supabase_client import get_admin_setting
 
 
 def _decode_private_key(raw_key: str) -> str:
@@ -79,35 +75,8 @@ def get_snowflake_connection_params():
     Returns:
         dict: Connection parameters for snowflake.connector.connect()
     """
-    # Get private key from environment
-    private_key_raw = os.getenv('SNOWFLAKE_KP_PK')
-    
-    # Fallback: try to read from .env file directly (for local development)
-    if not private_key_raw:
-        try:
-            env_path = '.env'
-            if os.path.exists(env_path):
-                with open(env_path, 'r') as f:
-                    content = f.read()
-                    # Find the private key section manually
-                    start_marker = 'SNOWFLAKE_KP_PK=-----BEGIN'
-                    
-                    if start_marker in content:
-                        start_idx = content.find('-----BEGIN', content.find(start_marker))
-                        end_idx = content.find('-----END', start_idx)
-                        if end_idx != -1:
-                            end_idx = content.find('-----', end_idx + 7) + 5
-                            private_key_raw = content[start_idx:end_idx].strip()
-                            print("✅ Loaded private key from .env file")
-        except Exception as e:
-            print(f"⚠️ Could not read .env file: {e}")
-    
-    if not private_key_raw:
-        raise ValueError(
-            "SNOWFLAKE_KP_PK environment variable not set or could not be parsed.\n"
-            "For HF Spaces: Add as a Secret in Settings > Repository Secrets\n"
-            "Supported formats: Direct PEM, Base64-encoded PEM, or escaped newlines"
-        )
+    # Source of truth: admin settings in Supabase
+    private_key_raw = get_admin_setting('SNOWFLAKE_KP_PK')
     
     # Decode the private key from various formats
     private_key_pem = _decode_private_key(private_key_raw)
@@ -116,7 +85,7 @@ def get_snowflake_connection_params():
     password = None
     if 'ENCRYPTED' in private_key_pem:
         # Try to get password from environment if key is encrypted
-        password = os.getenv('SNOWFLAKE_KP_PASSPHRASE')
+        password = get_admin_setting('SNOWFLAKE_KP_PASSPHRASE', required=False)
         if password:
             password = password.encode()
             print("✅ Using passphrase for encrypted private key")
@@ -130,8 +99,8 @@ def get_snowflake_connection_params():
     except Exception as e:
         if 'ENCRYPTED' in private_key_pem and not password:
             raise ValueError(
-                "Private key is encrypted but SNOWFLAKE_KP_PASSPHRASE not provided.\n"
-                "Add SNOWFLAKE_KP_PASSPHRASE as a Secret in HF Spaces settings."
+                "Private key is encrypted but SNOWFLAKE_KP_PASSPHRASE is missing. "
+                "Set it in Admin Settings."
             )
         else:
             raise ValueError(
@@ -148,14 +117,15 @@ def get_snowflake_connection_params():
     )
     
     # Return connection parameters
+    schema_name = get_admin_setting('SNOWFLAKE_SCHEMA', required=False) or 'PUBLIC'
     return {
-        'user': os.getenv('SNOWFLAKE_KP_USER'),
+        'user': get_admin_setting('SNOWFLAKE_KP_USER'),
         'private_key': private_key_bytes,
-        'account': os.getenv('SNOWFLAKE_ACCOUNT'),
-        'role': os.getenv('SNOWFLAKE_ROLE'),
-        'warehouse': os.getenv('SNOWFLAKE_WAREHOUSE'),
-        'database': os.getenv('SNOWFLAKE_DATABASE'),
-        'schema': os.getenv('SNOWFLAKE_SCHEMA', 'PUBLIC'),
+        'account': get_admin_setting('SNOWFLAKE_ACCOUNT'),
+        'role': get_admin_setting('SNOWFLAKE_ROLE', required=False),
+        'warehouse': get_admin_setting('SNOWFLAKE_WAREHOUSE'),
+        'database': get_admin_setting('SNOWFLAKE_DATABASE'),
+        'schema': schema_name,
     }
 
 def get_snowflake_connection():

sprint_2026_01.md

@@ -1,520 +0,0 @@
-# Sprint: January 2026 - Making the Demo Better
-
-## Core Mission 
-
-**"The goal of the demo builder is to tell a better story based on different personas, such as a line of business leader or data analyst, and to use a framework around the demo to effectively tell that story so that we can sell scloud software."**
-
-This framework influences the objects, visualizations, and data generated - including **outliers** that drive compelling demo narratives.
-
-**Terminology Clarification:**
-- **Outliers** = Creating interesting data points during data generation (story points for demos)
-- **Outliers Adjustment** = Modifying those outliers to make them stand out more
-
-See `dev_notes/USE_CASE_FLOW.md` for full documentation of the use case framework.
-
----
-
-## Sprint Objectives
-
-### 1. Generic Use Case Handling ✅ DONE
-
-**Problem:** Use cases not in the predefined list (like "subscription" for tinder.com) don't work as well as desired.
-
-**Solution:** Conversational flow for generic use cases - asks user for additional context when use case doesn't match packaged options.
-
-**Files:** `chat_interface.py` - use case validation and initialization flow
-
-**Also fixed:**
-- Company extraction now handles "with the use" format
-- Research continues even if website is inaccessible
-
----
-
-### 2. Unified Outlier System ✅ DONE
-
-**File:** `outlier_system.py`
-
-**Key Classes:**
-- `OutlierGenerator` - Generates SQL UPDATEs from text descriptions
-- `OutlierPattern` - Data class for outlier with SQL, documentation, Spotter question
-- `parse_chat_adjustment()` - Parse chat messages like "decrease electronics by 10%"
-- `apply_outliers()` - Execute multiple outlier SQL UPDATEs
-- `generate_demo_pack()` - Generate markdown demo documentation
-
-**Features:**
-- Parses natural language patterns ("Premium subscriptions spike 3x in Q4")
-- Extracts multipliers, percentages, absolute values
-- Understands time conditions (Q4, summer, specific months)
-- Generates Spotter questions automatically
-- Creates talking points for demos
-
-#### Two Entry Points, Same System
-
-| Entry Point | When | Who Decides | Example |
-|-------------|------|-------------|---------|
-| **Auto-Injection** | After population | AI (from research) | "Create compelling story patterns" |
-| **Chat Adjustment** | After deployment | User via chat | "decrease electronics by 10%" |
-
----
-
-### 3. Demo Pack Generation ✅ DONE
-
-**Function:** `generate_demo_pack()` in `outlier_system.py`
-
-**Output:** Markdown document with:
-- Outlier stories that were injected
-- Suggested Spotter questions for each story
-- Talking points
-- Technical details (table, column, conditions)
-- Demo flow guide
-
----
-
-### 4. Golden Demo with Hybrid MCP/TML Approach ✅ DONE
-
-**Goal:** Create the "perfect demo" liveboard using best of both methods.
-
-**Implementation Complete (Jan 18, 2026):**
-
-**Three-Method System:**
-- `TML` - Pure template-based approach, full control
-- `MCP` - Pure AI-driven approach, fast but basic
-- `HYBRID` - MCP creates + TML post-processing (recommended)
-
-**Configuration:**
-- Settings UI: Admin tab → "Liveboard Creation Method" dropdown
-- Environment variable: `LIVEBOARD_METHOD=TML|MCP|HYBRID`
-- Default: HYBRID
-
-**Files Modified:**
-- `thoughtspot_deployer.py` - Extended deploy_all() with liveboard_method parameter
-- `demo_prep.py` - Added liveboard_method_dropdown to Settings UI
-- `liveboard_creator.py` - Added `enhance_mcp_liveboard()` function
-- `chat_interface.py` - Passes liveboard_method from settings
-- `CLAUDE.md` - Updated documentation
-
-**enhance_mcp_liveboard() Function:**
-1. Exports MCP-created liveboard TML
-2. Classifies visualizations by type (KPI, trend, categorical)
-3. Adds Groups (tabs) for organization
-4. Fixes KPI sparklines and comparisons
-5. Applies brand colors (GBC_* for groups, TBC_* for tiles)
-6. Re-imports enhanced TML
-
----
-
-### 5. Justin Recommendations
-
-**a) snake_case Column Names in ThoughtSpot** ✅ DONE
-- Model column names now use snake_case (e.g., `shipping_mode`, `days_to_ship`)
-- Function: `_to_snake_case()` in `thoughtspot_deployer.py`
-- Updated `_resolve_column_name_conflict()` to use snake_case
-
-**b) ThoughtSpot Date Handling**
-- Proper date type configuration for time-series analysis
-- Calendar types, fiscal calendars, etc.
-- Affects: Model TML column properties
-
----
-
-### 6. Remove Chart Type Hints from Viz Titles ✅ DONE
-
-**Problem:** Viz titles include "(line chart)", "(bar chart)" etc. which look unprofessional.
-
-**Solution:** Added `_strip_chart_type_hints()` function in `liveboard_creator.py` that strips these hints before sending to MCP.
-
----
-
-### 7. Better Messaging / Progress Indication ✅ DONE
-
-**Problem:** After "Population Complete", UI appears frozen while processing continues in background. User thinks app is stuck.
-
-**Solution:** Added `flush=True` to critical print statements in:
-- `legitdata_bridge.py` - Population steps
-- `liveboard_creator.py` - MCP workflow
-- `chat_interface.py` - Deployment messages
-
----
-
-## Tasks
-
-### Done
-- [x] Documentation reorganization ✅
-- [x] LegitData integration ✅
-- [x] Chat interface working end-to-end ✅
-- [x] MCP liveboard creation working ✅
-- [x] Generic use case handling ✅
-- [x] Fix LegitData `generate_key` error ✅
-- [x] Full Tinder demo run successful! ✅
-
-### In Progress
-- [x] Debug MCP viz failures (2 vizs failed on Tinder demo) ✅ DONE
-- [x] **MCP on SEBE cluster** ✅ DONE - Bearer auth implementation fixed this (Feb 2)
-
----
-
-## Planned Features
-
-### Existing Model Selection (Future)
-
-**Goal:** Allow users to select an existing ThoughtSpot model and create liveboard / adjust data without creating new tables.
-
-**Questions to resolve:**
-1. **Model Selection** - Dropdown of models? Search by name? Filter by connection?
-2. **Use Case Matching** - How to validate model matches the use case? AI analyze columns vs keywords?
-3. **Database Access Check** - Verify connection is valid? Run test query? Check permissions?
-4. **Data Adjustment** - Outlier injection into existing data? Create views?
-5. **Entry Point** - "Create new demo" vs "Use existing model" option at start?
-6. **Cluster Scope** - Only models from cluster in .env
-
-**Implementation ideas:**
-- Use `/api/rest/2.0/metadata/search` to list models user has access to
-- Export model TML to get column names for use case matching
-- Check model's connection status via API
-
----
-
-### MCP Multi-Cluster Auth (Needed)
-
-**Problem:** MCP OAuth tokens are cached per-endpoint. When switching between SE and SEBE clusters, must manually clear `~/.mcp-auth/` and re-authenticate.
-
-**Goal:** Be smart about which cluster MCP is authenticated to and switch automatically based on .env settings.
-
-**Options explored:**
-1. **Bearer auth with `@thoughtspot/mcp-server`** - Use stdio transport with `TS_INSTANCE` and `TS_AUTH_TOKEN` env vars. Generate token via trusted auth from `THOUGHTSPOT_SECRET_KEY`.
-2. **Multiple cached tokens** - Store tokens by cluster URL, switch based on .env `THOUGHTSPOT_URL`
-3. **Manual for now** - Clear `~/.mcp-auth/` when switching clusters
-
-**Current workaround:** `rm -rf ~/.mcp-auth/` then re-auth when switching clusters.
-
-**Jan 31 findings:**
-- MCP works fine on SE cluster with same code
-- MCP fails on SEBE with "No answer found" errors
-- Not a code issue - same DONT_INDEX settings work on SE
-- Could be SEBE cluster config, Sage not enabled, or service issue
-
-### High Priority - Use Case Framework ✅ DONE
-- [x] **Complete 6 core use case configs** ✅ - Added all missing persona configs:
-  - Customer Analytics ✅
-  - Financial Analytics ✅
-  - Marketing Analytics ✅
-  - Retail Analytics ✅
-- [x] **Define outlier patterns for each use case** ✅ - 4-5 outlier patterns per use case
-- [x] **Use case aliases** ✅ - Flexible matching (e.g., "Sales" → "Sales Analytics")
-- [ ] **Generic use case improvements** - Better context gathering flow (future)
-- See `dev_notes/USE_CASE_FLOW.md` for documentation
-
-### To Do
-- [ ] Auto-injection step (hook outlier_system into population flow) - **DEFER: unclear what this is, revisit later**
-- [ ] Chat adjustment using outlier system - **DEFER to next sprint**
-- [x] ThoughtSpot date handling (Justin rec) ✅ DONE
-
-### UI Enhancements
-- [ ] 🧙 **Wizard Tab** - Old school step-by-step wizard (right of Chat, left of Settings)
-- [x] 📺 **Live Progress Tab** ✅ - Real-time deployment output in UI
-- [x] 📋 **Demo Pack Tab** ✅ - Auto-generated demo notes/talking points after deployment
-- [x] Removed Population Code tab ✅ (not needed with LegitData)
-
-### Done
-- [x] Outlier SQL generation core ✅ - `outlier_system.py` created
-- [x] Demo Pack generation ✅ - `generate_demo_pack()` function
-- [x] snake_case naming for ThoughtSpot objects ✅ - `_to_snake_case()` in deployer
-- [x] Fix UI feedback buffering issue ✅ - Added flush=True to critical print statements
-- [x] Remove "(line chart)" / "(bar chart)" from viz titles ✅ - `_strip_chart_type_hints()` 
-- [x] Fix date range mismatch - dates now generated up to current date ✅
-- [x] **MCP Bearer Auth** ✅ - Major weekend effort (Jan 31 - Feb 2). Bearer auth with trusted token ensures same org context. Works on both SE and SEBE clusters.
-- [x] Dynamic date generation ✅ - LegitData now generates dates relative to today
-- [x] Universal context prompt ✅ - All use cases (not just generic) now get context prompt
-- [x] ThoughtSpot date handling ✅
-
----
-
-## Notes
-
-### Jan 13, 2026 (Evening)
-**🎉 SUCCESSFUL FULL RUN: Tinder Subscription Demo**
-- Liveboard created: https://se-thoughtspot-cloud.thoughtspot.cloud/#/pinboard/e5c0eb54-ba91-4380-b7f4-353c134038a7
-- 5 of 6 vizs succeeded, 1 failed (KPI viz error)
-
-**Issues found:**
-1. **MCP Viz "No Data"** - 2 vizs showing no data due to date range mismatch:
-   - AI questions use "last 12 months" → filters to 2025-2026
-   - Generated data has dates from 2022-2024
-   - Fix: Either generate data with recent dates OR avoid relative time filters in questions
-2. **Viz Title Cleanup** - Remove "(line chart)" / "(bar chart)" from visualization titles
-3. **UI Feedback Buffering** - After "Population Complete", UI freezes while work continues in background. Terminal output is buffered, not real-time. User thinks app is stuck.
-4. **Progress Indication** - Need better way to show user that processing is still happening
-
-**Fixes today:**
-- Generic use case handling complete
-- Company/use case extraction improved
-- Research continues without website access
-- LegitData generate_key method added
-- Key realization: Outliers + Outliers Adjustment = same system
-- Unified model: SQL UPDATEs for pattern injection
-- Two entry points: AI-driven (auto) + User-driven (chat)
-
-### Jan 12, 2026
-- Returned to project after a month away
-- Reorganized documentation structure
-
----
-
-## Backlog (Future Sprints)
-
-- ~~**Concurrent demo builds**~~ ✅ FIXED - ChatDemoInterface now stored in `gr.State()` for per-session isolation
-- Site creator
-- Bot creator
-- Approval gates in chat workflow
-- LegitData integration strategy (pip package, submodule, etc.)
-- Real-time terminal output (fix buffering)
-
----
-
-*Sprint Started: January 12, 2026*
-
----
-
-## Interface Mode Refactor (Jan 26, 2026)
-
-### Status: Planning Complete
-
-**Goal:** Revive `demo_prep.py` (form-based interface) and add ability to switch between interfaces.
-
-### Findings
-
-#### 1. demo_prep.py Status: ✅ WORKS
-- Tested successfully on port 7870
-- Only issue was port conflict (7860 in use by chat interface)
-- All imports work, interface creates, Gradio launches
-
-#### 2. Code Duplication Analysis
-
-**Duplicated between demo_prep.py and chat_interface.py:**
-
-| Area | Notes |
-|------|-------|
-| Cache checking logic | ~60 lines each - nearly identical domain extraction, filename building, age check |
-| DDL generation prompt | ~80 lines each - same schema requirements, Snowflake syntax |
-| Research workflow | ~150 lines each - website extract → company analysis → industry research |
-| ThoughtSpot deployment | ~100 lines each - schema verification, deployer calls, progress callbacks |
-
-**Already Shared (chat_interface imports from demo_prep):**
-- `map_llm_display_to_provider()` - LLM provider mapping
-- `execute_population_script()` - Population execution
-- `generate_demo_base_name()` - Naming convention
-
-**Unique to demo_prep.py:**
-- `validate_ddl_syntax()` - DDL validation
-- `validate_python_syntax()` - Python code validation
-- `extract_outliers_from_population_script()` - Outlier extraction (see below)
-- `extract_python_code()` - Code block extraction
-- Expert/Fast mode toggle
-
-**Unique to chat_interface.py:**
-- `ChatDemoInterface` class - State management
-- Message parsing methods
-- Stage-based conversation flow
-- `SmartDataAdjuster` integration
-
-#### 3. Proposed Shared Class: `DemoWorkflowEngine`
-
-```python
-class DemoWorkflowEngine:
-    """Shared workflow logic for both form and chat interfaces"""
-    
-    def __init__(self, use_case: str, company_url: str, settings: dict = None)
-    
-    # Research
-    def check_cache(self, company, use_case) -> tuple[bool, str, dict]
-    def run_research(self, company, use_case, use_cache=True) -> Generator
-    
-    # DDL Generation
-    def generate_ddl(self) -> Generator
-    def validate_ddl(self, ddl: str) -> tuple[bool, str]
-    
-    # Population
-    def generate_population(self) -> Generator
-    def execute_population(self, schema_name) -> tuple[bool, str]
-    
-    # Deployment
-    def deploy_to_snowflake(self, schema_name) -> Generator
-    def deploy_to_thoughtspot(self, schema_name, ...) -> Generator
-    
-    # State
-    @property
-    def current_stage(self) -> str
-    def advance_stage(self)
-```
-
-#### 4. workflow_style Setting
-
-- **Setting name:** `workflow_style`
-- **Values:** `classic` (form) or `chat`
-- **Default:** `classic`
-- **Storage:** Supabase user settings
-- **Switch mechanism:** Requires app restart
-
----
-
-## Outlier Feature Comparison (Phase 2)
-
-### Two Different Approaches
-
-#### demo_prep.py - Pre-Planned Outlier Documentation
-
-**How it works:**
-1. LLM generates structured comments in population script:
-   ```python
-   # DEMO_OUTLIER: High-Value Customers at Risk
-   # INSIGHT: Top 5 customers (>$50K LTV) showing declining satisfaction
-   # SHOW_ME: "Show customers where lifetime_value > 50000 and satisfaction < 3"
-   # IMPACT: $250K annual revenue at risk
-   # TALKING_POINT: "Notice how ThoughtSpot surfaces your most valuable at-risk accounts"
-   ```
-2. `extract_outliers_from_population_script()` parses these comments
-3. Generates "Demo Notes & Presentation Guide" with talking points
-
-**Strength:** Creates pre-planned cheat sheet for demos
-
-#### chat_interface.py - Interactive Post-Deployment Adjustment
-
-**How it works:**
-1. Uses `SmartDataAdjuster` class after deployment
-2. User types natural language commands:
-   - "make 1080p webcam 40B"
-   - "increase smart watch by 20%"
-3. Executes SQL UPDATEs directly to Snowflake
-4. User refreshes liveboard to see changes
-
-**Strength:** Real-time customization before demos
-
-### Key Differences
-
-| Aspect | demo_prep.py | chat_interface.py |
-|--------|-------------|-------------------|
-| When | During data generation | After deployment |
-| What | Documents planned outliers | Modifies existing data |
-| Output | Presentation guide/cheat sheet | SQL updates |
-| Persistence | Baked into generated data | Changes live data |
-
-### Phase 2 Recommendation
-
-1. **Keep both approaches** - they serve different purposes
-2. **demo_prep.py outliers** should be migrated to shared engine (generates cheat sheet)
-3. **SmartDataAdjuster** already separate - can be used by both interfaces
-4. **Combine them:** Show pre-planned outliers AND allow interactive adjustment
-5. **chat_interface.py currently doesn't parse structured comments** - opportunity to add
-
-**The demo_prep.py approach is arguably MORE valuable** because:
-- Creates documentation that survives after the session
-- Gives salespeople a script to follow
-- Has richer metadata (viz types, KPIs, talking points)
-
----
-
-## Updated Tasks
-
-### Interface Refactor - Phase 1
-- [x] Test demo_prep.py - confirm it works ✅
-- [x] Map duplicated code sections ✅
-- [x] Design shared class API ✅
-- [ ] Create `demo_workflow_engine.py` with shared logic
-- [ ] Refactor demo_prep.py to use shared engine
-- [ ] Refactor chat_interface.py to use shared engine
-- [ ] Add `workflow_style` setting to Supabase
-- [ ] Create unified launcher
-
-### Outlier System - Phase 2
-- [ ] Port `extract_outliers_from_population_script()` to shared engine
-- [ ] Add outlier documentation parsing to chat_interface.py
-- [ ] Combine pre-planned outliers with interactive adjustment
-- [ ] Unified demo pack generation
-
----
-
-*Updated: January 26, 2026*
-
----
-
-## Bugs Found (Jan 26, 2026)
-
-### BUG: Liveboard link not clickable in chat output
-**Priority:** High (dev101)
-**Location:** `chat_interface.py` - deployment success message
-**Issue:** Liveboard URL renders as plain text, not a clickable markdown link
-**Expected:** `[Liveboard Name](https://url)` format that's clickable
-**Fix:** Check markdown link formatting in response string
-
-### BUG: TML Liveboard Quality - Comscore Demo
-**Priority:** High
-**Method:** TML (not MCP/HYBRID)
-**Company:** Comscore
-**Issues observed:**
-1. **No KPIs** - Missing headline KPI tiles with big numbers
-2. **Identical-looking vizs** - "Total Sales Performance" and "Sales Trend by Channel" both render as identical blue line charts
-3. **"Sales Trend by Channel"** - Shows single line, should show multiple lines (one per channel) or stacked area
-4. **Third viz failed** - Error icon, viz didn't render
-5. **No groups/tabs** - Flat layout instead of organized groups
-6. **No brand colors** - Everything default blue
-7. **Only 5 vizs** - Should have more variety
-
-**Root cause investigation needed:**
-- `liveboard_creator.py` - `create_visualization_tml()` not generating KPIs?
-- `_generate_smart_questions_with_ai()` generating boring/duplicate questions?
-- Group creation logic missing or not running?
-- Visualization type selection too conservative?
-
-**Reference:** Golden demo at `dev_notes/liveboard_demogold2/`
-
----
-
-## Fixes Applied (Jan 26, 2026 - Evening)
-
-### FIX: Liveboard link not clickable
-**File:** `chat_interface.py`
-**Issue:** Bold formatting inside link brackets `[**name**](url)` breaking Gradio markdown
-**Fix:** Changed to `[name](url)` format (4 places)
-
-### FIX: TML Liveboard Quality - No Groups/Tabs/Styling
-**File:** `liveboard_creator.py`
-**Issue:** `USE_ADVANCED_TML` defaulted to `'false'`, disabling groups/tabs/styling
-**Fix:** Changed default to `'true'` - now groups and brand colors enabled by default
-
-### FIX: TML Liveboard Quality - No KPIs, Boring Charts
-**File:** `liveboard_creator.py`
-**Issue:** AI-generated visualizations not enforced to have variety
-**Fix:** Added `_enforce_visualization_variety()` method that:
-- Ensures at least 2 KPIs are always created
-- Converts LINE charts without dimensions to KPIs if needed
-- Prevents more than 2 of the same chart type (diversifies to BAR, COLUMN, etc.)
-- Called after AI generates visualizations, before returning
-
-**Test needed:** Run a new demo to verify improvements
-
----
-
-## Carry Forward to February Sprint
-
-### Unsatisfied / Needs More Work
-- **Unified Outlier System** - Core done but needs refinement, not satisfied with output quality
-- **Demo Pack Generation** - Very unsatisfied, needs significant improvement
-- **Chart Titles** - Still not happy with viz titles, needs better naming
-
-### Needs Testing/Verification
-- **Existing Model Selection** - May be done but needs confirmation testing
-- **Universal Context Prompt** - Double-test this feature
-- **Self-join skip in models** - Verify this is working correctly
-
-### Deferred Features
-- **Chat adjustment using outlier system** - Never got to this
-- **Wizard Tab UI** - Not started
-- **Interface Mode Refactor** - `DemoWorkflowEngine` shared class concept
-
-### Unclear / Revisit Later
-- **Auto-injection step** - Unclear what this was supposed to be
-
----
-
-*Sprint Closed: February 2, 2026*
- t

sprint_2026_03.md

@@ -0,0 +1,227 @@
+# Sprint: March 2026
+
+*Started: March 16, 2026*
+*Planning doc: `dev_notes/plan_march_2026.md`*
+
+---
+
+## Context
+
+App is live. Small group launch early this week, broader rollout (4–5 people) within a couple weeks.
+This sprint covers hardening, settings, and new capabilities before that happens.
+
+---
+
+## Sprint Objectives
+
+### Before Small Group Launch (This Week)
+
+- [x] **TS Environment Dropdown** ✅ — ENV-based dropdown on front page; URL→key map hard-coded in app
+  - `TS_ENV_N_LABEL/URL` pattern in `.env`; `get_ts_environments()` reads all to build dropdown
+  - Dropdown in right panel alongside AI Model + Liveboard Name
+  - `update_ts_env()` resolves URL + auth key (via `os.getenv(key_name)`) into controller settings
+  - Bug fixed: was storing ENV var name instead of actual secret value
+  - Controller created on first message also receives current dropdown env selection
+- [x] **Front Page Redesign** ✅ — Right panel (where Stage + AI Model currently live):
+  - Add **TS Environment dropdown** here alongside AI Model
+  - Add **Liveboard Name** field here
+  - **Remove the Stage textbox** — replace with proper progress indicator (see below)
+  - Company/use case stays chat-driven only
+  - Remove `default_company_url` setting (replaced by chat-driven flow)
+- [x] **Progress Meter Fix** ✅ — current stage textbox is not great UX
+  - Add **Init** as the first stage in the progress sequence
+  - Replace the stage textbox with a visual progress indicator (step 1–N style)
+  - Stages: Init → Research → DDL → Data → Model → Liveboard (→ Data Adjuster when in that phase)
+- [x] **Chat Flow UX Improvements** ✅
+  - In-chat help text at session start: brief instructions message
+  - Clearer prompting back to user when use case is ambiguous
+  - Consider `?` tooltip near chat input
+- [x] **Error Handling Review** ✅
+  - **Liveboard partial success** ✅: Snowflake + model OK but liveboard fails → ⚠️ message with Spotter Viz Story tab pointer + 'retry liveboard' prompt
+  - **TML import errors**: parse TS `error_list`, show which viz failed specifically
+  - **MCP failures**: show which step failed, whether partial work should be kept
+  - **Top-level wrapper** ✅: `process_chat_message` wrapped in try/except → yield friendly error + log full traceback
+  - **Data Adjuster errors**: SQL execution failures shown clearly with context, not swallowed
+  - **Snowflake connection errors**: distinguish auth failure vs. query failure vs. timeout
+- [x] **Supabase Session Logging** ✅ — `session_logger.py`
+  - `SessionLogger` class: writes to `logs/sessions/{id}.log` + Supabase `session_logs` table
+  - Falls back silently to file-only if Supabase unavailable
+  - Initialized at first message in `process_chat_message`; logs research + deploy stage start/end
+  - `init_session_logger()` / `get_session_logger()` module-level helpers
+  - Table DDL in file docstring (run once in Supabase SQL editor)
+- [x] **Admin Log Viewer** ✅ — added to Admin Settings tab; email filter + row limit + Refresh button; queries `session_logs` via Supabase
+- [x] **Data Adjuster Cleanup & Controller Integration** ✅
+  - Existing files: `data_adjuster.py`, `smart_data_adjuster.py`, `conversational_data_adjuster.py`, `chat_data_adjuster.py`
+  - Currently wired post-liveboard but state lives on `self._adjuster` / `self._pending_adjustment` (instance vars — wrong)
+  - **Controller owns the adjuster phase**: adjuster state moves into the chat controller phase flow, not instance vars
+  - **Multi-turn**: controller stays in the adjuster phase across multiple messages; user can ask multiple questions and make multiple adjustments in one session
+  - **Smart adjustments**: LLM understands the request, maps to the right table/column, proposes the SQL, confirms with user, executes
+  - Consolidate: decide which of the 4 files survives (likely `smart_data_adjuster.py` as the engine, rest retired or merged)
+- [x] **State Isolation Audit** ✅ — audit complete; two HIGH issues identified
+  - **HIGH**: `session_logger.py` module-level `_current_logger` singleton — concurrent sessions overwrite each other's logger
+  - **HIGH**: `prompt_logger.py` module-level `_prompt_logger` singleton — all users' LLM prompts mix in same in-memory list
+  - **MEDIUM**: `inject_admin_settings_to_env()` writes to process-global `os.environ` — concurrent deployments could use wrong Snowflake account
+  - **MEDIUM**: Admin settings cache has no TTL — external Supabase edits invisible until restart
+  - Main `ChatDemoInterface` state IS isolated via `gr.State` — the pipeline itself is safe
+  - Fix (loggers + os.environ) tracked under "Before Broader Rollout → State Isolation Fix" below
+
+### Before Broader Rollout
+- [x] **Settings Audit & Cleanup** ✅
+  - `ts_instance_url` removed from SETTINGS_SCHEMA + hidden in Settings UI (replaced by env dropdown)
+  - `default_company_url` removed from SETTINGS_SCHEMA + hidden in Settings UI (chat-driven now)
+  - AI Model selection already on front page ✅
+- [ ] **demo_prep.py Refresh** — scope too large for this sprint, moved to Phase 2
+  - Audit done: ~2-3 day job (Spotter Viz tab, outlier integration, logged_completion, class refactor)
+- [x] **Session Persistence Verification** ✅ — verified working
+  - `ts_username` in SETTINGS_SCHEMA → pre-fills on load via `load_settings_on_startup`
+  - `liveboard_name`, `default_use_case`, `default_llm` all pre-fill on startup
+  - company no longer pre-filled (chat-driven) — working as intended
+- [x] **State Isolation Fix** ✅ — HIGH risk items resolved
+  - Session logger: stored on `self._session_logger` (per controller instance, not module singleton)
+  - Prompt logger: `reset_prompt_logger(session_id)` called at session start → fresh instance per session
+  - Both loggers initialized on first message of each session
+  - `inject_admin_settings_to_env()` still in use (deferred — requires cdw_connector refactor)
+
+---
+
+## Phase 2 (Next Sprint or Later)
+
+### demo_prep.py Refresh (from March Sprint)
+
+- [ ] **demo_prep.py Refresh** — sync with `chat_interface.py` improvements (~2-3 days)
+  - Add Spotter Viz Story tab + `_generate_spotter_viz_story()` (5h)
+  - Add Demo Pack tab with outlier-driven talking points + Spotter questions (4h)
+  - Replace all `researcher.make_request()` calls with `logged_completion()` wrapper (5h)
+  - Refactor to class-based pattern (like `ChatDemoInterface`) for state isolation (7h)
+  - Full outlier system integration (2.5h)
+  - Per-user session logging throughout (2h)
+
+---
+
+### Carry-forward from Sprint 2
+
+- [ ] **Unified Outlier System** — core done, not satisfied with output quality; needs refinement
+- [ ] **Demo Pack Generation** — very unsatisfied, needs significant improvement
+- [ ] **Chart Titles** — not happy with viz titles/naming; needs better approach
+- [ ] **Existing Model Selection + Self-Join Skip** — may be done; needs confirmation test + verify self-join skip is working correctly
+- [ ] **Universal Context Prompt** — double-test this feature end-to-end
+- [ ] **Chat Adjustment Using Outlier System** — never got to this
+- [ ] **Interface Mode Refactor** (`DemoWorkflowEngine` shared class concept)
+- [ ] **Wizard Tab UI** — not started
+- [ ] **Tag Assignment to Models** — returns 404 (works for tables, not models); needs investigation
+- [ ] **Spotter Viz Story Verification** — run end-to-end and verify story generation + blank viz (ASP, Total Sales Weekly) and brand colors rendering
+- [ ] **Fix Research Cache Not Loading** — relative path issue; fix was ready, needs test
+- [ ] **Fix DAYSONHAND Generation** — currently random; needs business logic (realistic 15–120 day distribution)
+- [ ] **Verify KPIs in Liveboard** — requires live deployment test
+- [ ] **Auto-injection step** — revisit what this was supposed to be
+- [ ] **Dead code cleanup: model TML generators** — `thoughtspot_deployer.py` has 3 model TML functions; only `_create_model_with_constraints` is called by `deploy_all`; remove `create_actual_model_tml` and `create_model_tml`
+
+### From March Plan
+
+- [x] **Data Adjuster — Liveboard-First Entry Point** ✅
+  - Paste any TS liveboard URL in the init stage → jumps straight to adjuster (skips build pipeline)
+  - `load_context_from_liveboard()` in `smart_data_adjuster.py`: liveboard TML (export_fqn) → model GUID → model TML → db/schema
+  - Detection in `chat_interface.py` init stage: regex on `pinboard/<guid>` pattern → auth TS client → load context → init SmartDataAdjuster → `outlier_adjustment` stage
+- [x] **Sharing** ✅ — model + liveboard shared (can_edit / MODIFY) after every build
+  - `share_objects()` method in `thoughtspot_deployer.py`: POST `/api/rest/2.0/security/metadata/share`
+  - Detects `@` in value → USER type, otherwise → USER_GROUP
+  - `share_with` in regular Settings (per-user); `SHARE_WITH` in Admin Settings (system-wide default)
+  - Per-user setting takes priority; falls back to admin setting if empty
+  - Model shared after creation, liveboard shared after creation
+- [x] **Sage Indexing Retry** ✅ — `_get_answer_direct` now retries once with 20s wait on 10004 "No answer found"; flag is module-level so the wait happens only once per build run, not per question
+- [x] **Fallback TML: Skip Invalid Column Refs** ✅ — after `convert_natural_to_search`, validates `[Column]` tokens against model columns; skips viz (instead of failing the whole liveboard) if any token is missing
+- [ ] **MCP 500 Retry Logic** — broader retry for other 5xx errors
+- [ ] **Model Generator: Chasm Trap Fix** — when two fact tables share a dimension, model generator must:
+  - Include ALL FK joins from each fact table to shared dimensions (e.g. `PRIOR_AUTHORIZATIONS.DRUG_NDC → DRUGS`)
+  - Set `is_attribution_dimension: false` on shared dimension tables so TS doesn't fold fact tables together
+  - Without this: queries fan out through a shared date dimension → every group gets the same average
+  - Fixed manually for Abarca: added `PRIOR_AUTHORIZATIONS → DRUGS` join + `DRUGS.is_attribution_dimension=false`
+- [ ] **Data Narrative Layer for Population** — LLM generates random/flat data because it doesn't know what story the KPIs should tell
+  - Root cause: population script gets DDL + company context but NOT the KPI formulas or desired metric distributions
+  - Fix: before data generation, build a "data narrative" spec from the vertical×function matrix: explicit per-column constraints ("IS_GENERIC: 93% for Medicaid rows, 80% for Commercial"), outlier targets, trend directions
+  - Pass this narrative spec as a required section in the population prompt
+  - Domain rules baked in: specialty/biologic drugs get low PA approval, GLP-1s face high scrutiny, Medicaid has highest GDR, etc.
+  - Goal: generated data tells the story on first run — no manual Snowflake fixups required
+- [x] **Fix Domain-Specific NAME Column Generation** ✅ — `DRUG_NAME` was falling through to `fake.name()` (person name) because `'NAME' in col_name_upper` matched first, before the drug-specific check; fixed by adding DRUG/MEDICATION check at the top of the NAME block in `chat_interface.py`
+- [x] **Abarca Demo Data — KPI Variation** ✅ — GDR and PA Approval Rate KPIs had flat sparklines
+  - Root cause: IS_GENERIC set by plan type only (uniform across months); PA rate set by therapeutic class only
+  - Fix: `scratch/fix_abarca_kpi_variation.py` — full per-month reset using plan-type + monthly adjustment
+  - GDR visible range: 84–93% with clear oscillations; PA visible range: 74–86%
+  - **⚠️ Re-run `fix_abarca_kpi_variation.py` if other data changes clobber monthly variation**
+  - PA by therapeutic class fix: `scratch/fix_abarca_pa_therapeutic_class.py` + `scratch/fix_abarca_pa_ts_table.py`
+  - Root cause of flat PA-by-class chart: no PRIOR_AUTHORIZATIONS→DRUGS join in TS model; added THERAPEUTIC_CLASS column directly to PRIOR_AUTHORIZATIONS table instead
+
+---
+
+## Phase 3 (Future)
+
+- [ ] **OAuth/SSO Login** — swap Gradio auth for proper OAuth flow
+- [ ] **Batch Runner Gradio Tab** — after CLI proves out, add Gradio tab for batch testing
+- [ ] **Batch Runner: Full Pipeline Stages** — add population, deploy_snowflake, deploy_thoughtspot, liveboard stages
+- [ ] **Request New Environment Form** — if/when needed
+- [ ] **Liveboard Question Column Mapping** — `liveboard_questions[].viz_question` uses natural language that may not match actual DDL column names; after model is built, map questions to real column names before sending to MCP. Currently worked around with generic NL ("average selling price by week" vs "ASP weekly") but proper runtime column substitution would be more reliable.
+
+---
+
+## Cancelled / Resolved
+
+- ~~**MCP Bearer Auth investigation**~~ — resolved; bearer auth working, no further action needed
+
+---
+
+## Done
+
+### Session: March 26, 2026 — Liveboard Name Fix + Settings Reorganization
+
+- [x] **Liveboard Name Bug Fixed** ✅ — UI field value now takes priority over DB-loaded default
+  - `send_message` and `quick_action` accept `liveboard_name_ui` param
+  - `liveboard_name_input` added to `_send_inputs` and `_action_inputs`
+  - Applied to `controller.settings['liveboard_name']` on every message — always uses current UI value
+- [x] **Settings UI Reorganized** ✅ — Split into "Default Settings" and "App Settings"
+  - **Default Settings**: AI Model, Default Use Case, Default Liveboard Name (3-up row)
+  - **App Settings**: Tag Name, Fact Table Size, Dim Table Size, Object Naming Prefix, Column Naming Style
+
+---
+
+### Session: March 26, 2026 — New Vision Merge + Pipeline Investigation
+
+- [x] **Spotter enable fix verified** ✅ — `spotter_config` placement confirmed correct (nested inside `model.properties`, not sibling). Tested on model `f40ff5bd` via `scratch/test_spotter_enable.py` — Spotter answered (HTTP 200).
+
+- [x] **Liveboard pipeline bugs documented** ✅ — Full trace written in `dev_notes/liveboard_flow_amazon_retail.md`
+  - **6 KPI root cause**: AI-generated fill questions (slots 5–8) are all time+metric → MCP creates them all as KPI. Fix: cap AI questions to max 2 single-metric.
+  - **"Show me" title bug**: `_convert_outlier_to_mcp_question` prepends "Show me"; `_clean_viz_title` strips "Show " leaving "me...". Fix: add `(r'^Show me ', '')` before `(r'^Show ', '')`.
+  - **Spotter Viz Story mismatch**: `_generate_spotter_viz_story` never sees actual viz names — generates independently. Fix: pass actual viz names post-build.
+  - OutlierPattern fields `sql_template`, `magnitude`, `affected_columns`, `target_filter`, `demo_setup`, `demo_payoff` are all dead — never read anywhere.
+
+- [x] **DemoPrep_new_vision2 merge completed** ✅ — New data generation engine with real outlier injection merged into current codebase:
+  - `demo_personas.py` — replaced with new version: `DEFAULT_STORY_CONTROLS`, `story_controls` on every vertical/function, Finance/SaaS overrides, `ROUTED_USE_CASES`, merged `get_use_case_config()`
+  - `legitdata_project/legitdata/generator.py` — replaced with 1,600-line version: `_refresh_story_spec()`, `_apply_storyspec_time_series()` (actual outlier injection with deterministic seed + trend/seasonal signals), `_generate_saas_finance_gold()`
+  - `legitdata_project/legitdata/storyspec.py` — new file: `StorySpec`, `TrendProfile`, `OutlierBudget`, `ValueGuardrails` dataclasses
+  - `legitdata_project/legitdata/domain/` — new package: `SemanticType` enum + domain value libraries
+  - `legitdata_project/legitdata/quality/` — new package: quality rules, validator, repair
+  - 5 updated source files: `column_classifier.py`, `ai_generator.py`, `generic.py`, `fk_manager.py`, `parser.py`
+  - `legitdata_project/legitdata/__init__.py` — updated with StorySpec exports
+  - Verified end-to-end: seed=1780963166 (deterministic from `amazon.com+Retail Sales`), 1 outlier injected Sept 9 2024 at 2.4x multiplier in SALES_TRANSACTIONS
+
+---
+
+## Notes
+
+### Vertical × Function Matrix System
+
+The matrix determines what gets built — KPIs, visualizations, outliers, target persona.
+See `dev_notes/plan_march_2026.md` appendix for full documentation.
+
+**Current coverage:**
+
+| Vertical | Sales | Supply Chain | Marketing |
+|----------|-------|-------------|-----------|
+| Retail | ✅ Override | Base merge | Base merge |
+| Banking | Base merge | Base merge | ✅ Override |
+| Software | ✅ Override | Base merge | Base merge |
+| Manufacturing | Base merge | Base merge | Base merge |
+| *other* | Generic | Generic | Generic |
+
+*Override = enriched with persona, extra KPIs, specific viz*
+*Base merge = Vertical + Function combined, no special override*
+*Generic = AI adapts from closest function match*

tests/__init__.py

@@ -0,0 +1 @@
+# DemoPrep test package

tests/newvision_sample_runner.py

@@ -0,0 +1,1246 @@
+#!/usr/bin/env python3
+"""Run the New Vision 4-case sample set without UI login flow.
+
+Modes:
+1) Full chat pipeline mode (uses configured default_llm and required settings)
+2) Offline DDL mode (deterministic schema template, still validates settings up front)
+
+Usage:
+  source ./demoprep/bin/activate
+  python tests/newvision_sample_runner.py
+  python tests/newvision_sample_runner.py --offline-ddl
+  python tests/newvision_sample_runner.py --skip-thoughtspot
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+PROJECT_ROOT = Path(__file__).parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+from dotenv import load_dotenv
+
+load_dotenv(PROJECT_ROOT / ".env")
+os.environ.setdefault("DEMOPREP_NO_AUTH", "true")
+
+# Pull admin settings into environment when available.
+try:
+    from supabase_client import inject_admin_settings_to_env
+
+    inject_admin_settings_to_env()
+except Exception as exc:  # noqa: BLE001
+    print(f"[newvision_runner] Admin setting injection unavailable: {exc}")
+
+OFFLINE_DEMO_DDL = """
+CREATE TABLE DIM_DATE (
+    DATE_KEY INT PRIMARY KEY,
+    ORDER_DATE DATE,
+    MONTH_NAME VARCHAR(30),
+    QUARTER_NAME VARCHAR(10),
+    YEAR_NUM INT,
+    IS_WEEKEND BOOLEAN
+);
+
+CREATE TABLE DIM_LOCATION (
+    LOCATION_KEY INT PRIMARY KEY,
+    COUNTRY VARCHAR(100),
+    REGION VARCHAR(100),
+    STATE VARCHAR(100),
+    CITY VARCHAR(100),
+    SALES_CHANNEL VARCHAR(100),
+    CUSTOMER_SEGMENT VARCHAR(100)
+);
+
+CREATE TABLE DIM_PRODUCT (
+    PRODUCT_KEY INT PRIMARY KEY,
+    PRODUCT_NAME VARCHAR(200),
+    BRAND_NAME VARCHAR(100),
+    CATEGORY VARCHAR(100),
+    SUB_CATEGORY VARCHAR(100),
+    PRODUCT_TIER VARCHAR(50),
+    UNIT_PRICE DECIMAL(12,2)
+);
+
+CREATE TABLE FACT_RETAIL_DAILY (
+    TRANSACTION_KEY INT PRIMARY KEY,
+    DATE_KEY INT,
+    LOCATION_KEY INT,
+    PRODUCT_KEY INT,
+    ORDER_DATE DATE,
+    ORDER_COUNT INT,
+    UNITS_SOLD INT,
+    UNIT_PRICE DECIMAL(12,2),
+    GROSS_REVENUE DECIMAL(14,2),
+    NET_REVENUE DECIMAL(14,2),
+    SALES_AMOUNT DECIMAL(14,2),
+    DISCOUNT_PCT DECIMAL(5,2),
+    INVENTORY_ON_HAND INT,
+    LOST_SALES_USD DECIMAL(14,2),
+    IS_OOS BOOLEAN,
+    FOREIGN KEY (DATE_KEY) REFERENCES DIM_DATE(DATE_KEY),
+    FOREIGN KEY (LOCATION_KEY) REFERENCES DIM_LOCATION(LOCATION_KEY),
+    FOREIGN KEY (PRODUCT_KEY) REFERENCES DIM_PRODUCT(PRODUCT_KEY)
+);
+""".strip()
+
+
+def _now_utc_iso() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+
+def _load_cases(cases_file: Path) -> list[dict[str, Any]]:
+    data = yaml.safe_load(cases_file.read_text(encoding="utf-8")) or {}
+    return list(data.get("test_cases", []))
+
+
+def _parse_quality_report_path(message: str) -> str | None:
+    for line in (message or "").splitlines():
+        if "Report:" in line:
+            return line.split("Report:", 1)[1].strip()
+        if "See report:" in line:
+            return line.split("See report:", 1)[1].strip()
+    return None
+
+
+def _load_quality_report(report_path: str | None) -> dict[str, Any]:
+    if not report_path:
+        return {}
+    path = Path(report_path)
+    json_path = path.with_suffix(".json")
+    if json_path.exists():
+        try:
+            return json.loads(json_path.read_text(encoding="utf-8"))
+        except Exception:  # noqa: BLE001
+            return {}
+    return {}
+
+
+def _build_quality_gate_stage(report_path: str | None) -> dict[str, Any]:
+    report = _load_quality_report(report_path)
+    summary = report.get("summary", {}) if isinstance(report, dict) else {}
+    passed = report.get("passed") if isinstance(report, dict) else None
+    return {
+        "ok": bool(passed) if passed is not None else False,
+        "report_path": report_path,
+        "passed": passed,
+        "summary": {
+            "semantic_pass_ratio": summary.get("semantic_pass_ratio"),
+            "categorical_junk_count": summary.get("categorical_junk_count"),
+            "fk_orphan_count": summary.get("fk_orphan_count"),
+            "temporal_violations": summary.get("temporal_violations"),
+            "numeric_violations": summary.get("numeric_violations"),
+            "volatility_breaches": summary.get("volatility_breaches"),
+            "smoothness_score": summary.get("smoothness_score"),
+            "outlier_explainability": summary.get("outlier_explainability"),
+            "kpi_consistency": summary.get("kpi_consistency"),
+        },
+    }
+
+
+def _resolve_runtime_settings() -> tuple[str, str]:
+    user_email = (
+        os.getenv("USER_EMAIL")
+        or os.getenv("INITIAL_USER")
+        or os.getenv("THOUGHTSPOT_ADMIN_USER")
+        or "default@user.com"
+    ).strip()
+    default_llm = (os.getenv("DEFAULT_LLM") or os.getenv("OPENAI_MODEL") or "").strip()
+    if not default_llm:
+        raise ValueError("Missing required env var: DEFAULT_LLM or OPENAI_MODEL")
+    return user_email, default_llm
+
+
+def _run_realism_sanity_checks(schema_name: str, case: dict[str, Any]) -> dict[str, Any]:
+    """Fast, opinionated sanity checks for demo realism.
+
+    These checks intentionally target user-visible demo breakages that can slip
+    through structural quality gates (e.g., null dimensions in top-N charts).
+    """
+    checks: list[dict[str, Any]] = []
+    failures: list[str] = []
+    if not schema_name:
+        return {"ok": False, "checks": checks, "failures": ["Missing schema name for sanity checks"]}
+
+    use_case = str(case.get("use_case", "") or "")
+    use_case_lower = (use_case or "").lower()
+    case_name = str(case.get("name", "") or "").lower()
+    is_legal = "legal" in use_case_lower
+    is_private_equity = any(
+        marker in use_case_lower
+        for marker in ("private equity", "lp reporting", "state street")
+    )
+    is_saas_finance = any(
+        marker in use_case_lower
+        for marker in ("saas finance", "unit economics", "financial analytics", "fp&a", "fpa")
+    )
+    if not is_legal and not is_private_equity and not is_saas_finance:
+        # Keep runtime fast by evaluating only scoped vertical checks.
+        return {"ok": True, "checks": checks, "failures": []}
+
+    from supabase_client import inject_admin_settings_to_env
+    from snowflake_auth import get_snowflake_connection
+
+    inject_admin_settings_to_env()
+    conn = None
+    cur = None
+    try:
+        db_name = (os.getenv("SNOWFLAKE_DATABASE") or "DEMOBUILD").strip()
+        safe_schema = schema_name.replace('"', "")
+        conn = get_snowflake_connection()
+        cur = conn.cursor()
+        cur.execute(f'USE DATABASE "{db_name}"')
+        cur.execute(f'USE SCHEMA "{safe_schema}"')
+
+        if is_legal:
+            cur.execute("SHOW TABLES")
+            legal_tables = {str(row[1]).upper() for row in cur.fetchall()}
+            has_split_legal = {"LEGAL_MATTERS", "OUTSIDE_COUNSEL_INVOICES", "ATTORNEYS", "MATTER_TYPES"}.issubset(legal_tables)
+            has_event_legal = "LEGAL_SPEND_EVENTS" in legal_tables
+
+            if has_split_legal:
+                # 1) Invoice -> matter -> attorney join coverage.
+                cur.execute(
+                    """
+                    SELECT
+                        COUNT(*) AS total_rows,
+                        COUNT_IF(a.ATTORNEY_NAME IS NULL) AS null_rows
+                    FROM OUTSIDE_COUNSEL_INVOICES oci
+                    LEFT JOIN LEGAL_MATTERS lm ON oci.MATTER_ID = lm.MATTER_ID
+                    LEFT JOIN ATTORNEYS a ON lm.ASSIGNED_ATTORNEY_ID = a.ATTORNEY_ID
+                    """
+                )
+                total_rows, null_rows = cur.fetchone()
+                null_pct = (float(null_rows) * 100.0 / float(total_rows)) if total_rows else 100.0
+                checks.append(
+                    {
+                        "name": "legal_attorney_join_null_pct",
+                        "value": round(null_pct, 2),
+                        "threshold": "<= 5.0",
+                        "ok": null_pct <= 5.0,
+                    }
+                )
+                if null_pct > 5.0:
+                    failures.append(
+                        f"Attorney join null rate too high: {null_pct:.2f}% (expected <= 5%)"
+                    )
+
+                # 1b) Invoice MATTER_ID linkage must be complete.
+                cur.execute(
+                    """
+                    SELECT
+                        COUNT(*) AS total_rows,
+                        COUNT_IF(MATTER_ID IS NULL) AS null_rows
+                    FROM OUTSIDE_COUNSEL_INVOICES
+                    """
+                )
+                total_rows, null_rows = cur.fetchone()
+                null_pct = (float(null_rows) * 100.0 / float(total_rows)) if total_rows else 100.0
+                checks.append(
+                    {
+                        "name": "legal_invoice_matter_id_null_pct",
+                        "value": round(null_pct, 2),
+                        "threshold": "== 0.0",
+                        "ok": null_pct == 0.0,
+                    }
+                )
+                if null_pct != 0.0:
+                    failures.append(
+                        f"Invoice MATTER_ID null rate is {null_pct:.2f}% (expected 0%)"
+                    )
+
+                # 2) Region cardinality should be compact for legal executive demos.
+                cur.execute("SELECT COUNT(DISTINCT REGION) FROM LEGAL_MATTERS WHERE REGION IS NOT NULL")
+                region_cardinality = int(cur.fetchone()[0] or 0)
+                checks.append(
+                    {
+                        "name": "legal_region_distinct_count",
+                        "value": region_cardinality,
+                        "threshold": "<= 6",
+                        "ok": region_cardinality <= 6,
+                    }
+                )
+                if region_cardinality > 6:
+                    failures.append(
+                        f"Region cardinality too high: {region_cardinality} distinct values (expected <= 6)"
+                    )
+
+                # 3) Firm names should not contain obvious cross-vertical banking/org jargon.
+                cur.execute(
+                    """
+                    SELECT COUNT(*)
+                    FROM OUTSIDE_COUNSEL
+                    WHERE REGEXP_LIKE(
+                        LOWER(FIRM_NAME),
+                        'retail banking|consumer lending|digital channels|enterprise operations|regional service'
+                    )
+                    """
+                )
+                bad_firm_count = int(cur.fetchone()[0] or 0)
+                checks.append(
+                    {
+                        "name": "legal_firm_name_cross_vertical_count",
+                        "value": bad_firm_count,
+                        "threshold": "== 0",
+                        "ok": bad_firm_count == 0,
+                    }
+                )
+                if bad_firm_count != 0:
+                    failures.append(
+                        f"Detected {bad_firm_count} cross-vertical/non-legal firm names"
+                    )
+
+                # 4) Matter type taxonomy should remain concise and demo-friendly.
+                cur.execute(
+                    """
+                    SELECT COUNT(DISTINCT mt.MATTER_TYPE_NAME)
+                    FROM LEGAL_MATTERS lm
+                    LEFT JOIN MATTER_TYPES mt ON lm.MATTER_TYPE_ID = mt.MATTER_TYPE_ID
+                    WHERE mt.MATTER_TYPE_NAME IS NOT NULL
+                    """
+                )
+                matter_type_cardinality = int(cur.fetchone()[0] or 0)
+                checks.append(
+                    {
+                        "name": "legal_matter_type_distinct_count",
+                        "value": matter_type_cardinality,
+                        "threshold": "<= 15",
+                        "ok": matter_type_cardinality <= 15,
+                    }
+                )
+                if matter_type_cardinality > 15:
+                    failures.append(
+                        f"Matter type cardinality too high: {matter_type_cardinality} distinct values (expected <= 15)"
+                    )
+            elif has_event_legal:
+                # 1) Attorney dimension join coverage (critical for "Top Attorney by Cost").
+                cur.execute(
+                    """
+                    SELECT
+                        COUNT(*) AS total_rows,
+                        COUNT_IF(a.ATTORNEY_NAME IS NULL) AS null_rows
+                    FROM LEGAL_SPEND_EVENTS lse
+                    LEFT JOIN ATTORNEYS a ON lse.ATTORNEY_ID = a.ATTORNEY_ID
+                    """
+                )
+                total_rows, null_rows = cur.fetchone()
+                null_pct = (float(null_rows) * 100.0 / float(total_rows)) if total_rows else 100.0
+                checks.append(
+                    {
+                        "name": "legal_attorney_join_null_pct",
+                        "value": round(null_pct, 2),
+                        "threshold": "<= 5.0",
+                        "ok": null_pct <= 5.0,
+                    }
+                )
+                if null_pct > 5.0:
+                    failures.append(
+                        f"Attorney join null rate too high: {null_pct:.2f}% (expected <= 5%)"
+                    )
+
+                # 2) Region cardinality should be compact for legal executive demos.
+                cur.execute("SELECT COUNT(DISTINCT REGION) FROM LEGAL_SPEND_EVENTS WHERE REGION IS NOT NULL")
+                region_cardinality = int(cur.fetchone()[0] or 0)
+                checks.append(
+                    {
+                        "name": "legal_region_distinct_count",
+                        "value": region_cardinality,
+                        "threshold": "<= 6",
+                        "ok": region_cardinality <= 6,
+                    }
+                )
+                if region_cardinality > 6:
+                    failures.append(
+                        f"Region cardinality too high: {region_cardinality} distinct values (expected <= 6)"
+                    )
+
+                # 3) Firm names should not contain obvious cross-vertical banking/org jargon.
+                cur.execute(
+                    """
+                    SELECT COUNT(*)
+                    FROM OUTSIDE_COUNSEL_FIRMS
+                    WHERE REGEXP_LIKE(
+                        LOWER(FIRM_NAME),
+                        'retail banking|consumer lending|digital channels|enterprise operations|regional service'
+                    )
+                    """
+                )
+                bad_firm_count = int(cur.fetchone()[0] or 0)
+                checks.append(
+                    {
+                        "name": "legal_firm_name_cross_vertical_count",
+                        "value": bad_firm_count,
+                        "threshold": "== 0",
+                        "ok": bad_firm_count == 0,
+                    }
+                )
+                if bad_firm_count != 0:
+                    failures.append(
+                        f"Detected {bad_firm_count} cross-vertical/non-legal firm names"
+                    )
+
+                # 4) Matter type taxonomy should remain concise and demo-friendly.
+                cur.execute(
+                    """
+                    SELECT COUNT(DISTINCT mt.MATTER_TYPE_NAME)
+                    FROM LEGAL_SPEND_EVENTS lse
+                    LEFT JOIN MATTER_TYPES mt ON lse.MATTER_TYPE_ID = mt.MATTER_TYPE_ID
+                    WHERE mt.MATTER_TYPE_NAME IS NOT NULL
+                    """
+                )
+                matter_type_cardinality = int(cur.fetchone()[0] or 0)
+                checks.append(
+                    {
+                        "name": "legal_matter_type_distinct_count",
+                        "value": matter_type_cardinality,
+                        "threshold": "<= 15",
+                        "ok": matter_type_cardinality <= 15,
+                    }
+                )
+                if matter_type_cardinality > 15:
+                    failures.append(
+                        f"Matter type cardinality too high: {matter_type_cardinality} distinct values (expected <= 15)"
+                    )
+            else:
+                failures.append("Could not find supported legal schema shape for realism checks")
+
+        if is_private_equity:
+            # Guard against semantic leakage where sector/strategy dimensions are
+            # accidentally populated with company names.
+            cur.execute(
+                """
+                WITH dim_companies AS (
+                    SELECT DISTINCT COMPANY_NAME
+                    FROM PORTFOLIO_COMPANIES
+                    WHERE COMPANY_NAME IS NOT NULL
+                ),
+                dim_sectors AS (
+                    SELECT DISTINCT SECTOR_NAME
+                    FROM SECTORS
+                    WHERE SECTOR_NAME IS NOT NULL
+                ),
+                dim_strategies AS (
+                    SELECT DISTINCT FUND_STRATEGY
+                    FROM FUNDS
+                    WHERE FUND_STRATEGY IS NOT NULL
+                )
+                SELECT
+                    (SELECT COUNT(*) FROM dim_sectors),
+                    (SELECT COUNT(*) FROM dim_strategies),
+                    (SELECT COUNT(*) FROM dim_sectors s JOIN dim_companies c ON s.SECTOR_NAME = c.COMPANY_NAME),
+                    (SELECT COUNT(*) FROM dim_strategies f JOIN dim_companies c ON f.FUND_STRATEGY = c.COMPANY_NAME)
+                """
+            )
+            sector_distinct, strategy_distinct, sector_overlap, strategy_overlap = cur.fetchone()
+            sector_distinct = int(sector_distinct or 0)
+            strategy_distinct = int(strategy_distinct or 0)
+            sector_overlap = int(sector_overlap or 0)
+            strategy_overlap = int(strategy_overlap or 0)
+
+            checks.append(
+                {
+                    "name": "pe_sector_name_company_overlap_count",
+                    "value": sector_overlap,
+                    "threshold": "== 0",
+                    "ok": sector_overlap == 0,
+                }
+            )
+            if sector_overlap != 0:
+                failures.append(
+                    f"Sector names overlap company names ({sector_overlap} overlaps); likely mislabeled dimensions"
+                )
+
+            checks.append(
+                {
+                    "name": "pe_fund_strategy_company_overlap_count",
+                    "value": strategy_overlap,
+                    "threshold": "== 0",
+                    "ok": strategy_overlap == 0,
+                }
+            )
+            if strategy_overlap != 0:
+                failures.append(
+                    f"Fund strategy values overlap company names ({strategy_overlap} overlaps); likely mislabeled dimensions"
+                )
+
+            checks.append(
+                {
+                    "name": "pe_sector_distinct_count",
+                    "value": sector_distinct,
+                    "threshold": ">= 4 and <= 20",
+                    "ok": 4 <= sector_distinct <= 20,
+                }
+            )
+            if not (4 <= sector_distinct <= 20):
+                failures.append(
+                    f"Sector distinct count out of expected demo range: {sector_distinct} (expected 4-20)"
+                )
+
+            checks.append(
+                {
+                    "name": "pe_fund_strategy_distinct_count",
+                    "value": strategy_distinct,
+                    "threshold": ">= 4 and <= 20",
+                    "ok": 4 <= strategy_distinct <= 20,
+                }
+            )
+            if not (4 <= strategy_distinct <= 20):
+                failures.append(
+                    f"Fund strategy distinct count out of expected demo range: {strategy_distinct} (expected 4-20)"
+                )
+
+            if case_name == "statestreet_private_equity_lp_reporting":
+                cur.execute(
+                    """
+                    SELECT
+                        COUNT(*) AS total_rows,
+                        COUNT_IF(ABS(TOTAL_VALUE_USD - (REPORTED_VALUE_USD + DISTRIBUTIONS_USD)) > 0.01) AS bad_rows
+                    FROM PORTFOLIO_PERFORMANCE
+                    """
+                )
+                total_rows, bad_rows = cur.fetchone()
+                total_rows = int(total_rows or 0)
+                bad_rows = int(bad_rows or 0)
+                identity_ok = total_rows > 0 and bad_rows == 0
+                checks.append(
+                    {
+                        "name": "pe_total_value_identity_bad_rows",
+                        "value": bad_rows,
+                        "threshold": "== 0",
+                        "ok": identity_ok,
+                    }
+                )
+                if not identity_ok:
+                    failures.append(
+                        f"Total value identity broken in {bad_rows} PE fact rows"
+                    )
+
+                cur.execute(
+                    """
+                    SELECT
+                        COUNT(*) AS total_rows,
+                        COUNT_IF(IRR_SUB_LINE_IMPACT_BPS BETWEEN 80 AND 210) AS in_band_rows,
+                        COUNT_IF(ABS(IRR_SUB_LINE_IMPACT_BPS - ((GROSS_IRR - GROSS_IRR_WITHOUT_SUB_LINE) * 10000)) <= 5) AS identity_rows
+                    FROM PORTFOLIO_PERFORMANCE
+                    """
+                )
+                total_rows, in_band_rows, identity_rows = cur.fetchone()
+                total_rows = int(total_rows or 0)
+                in_band_rows = int(in_band_rows or 0)
+                identity_rows = int(identity_rows or 0)
+                irr_band_ok = total_rows > 0 and in_band_rows == total_rows and identity_rows == total_rows
+                checks.append(
+                    {
+                        "name": "pe_subscription_line_impact_rows_valid",
+                        "value": {"total": total_rows, "in_band": in_band_rows, "identity": identity_rows},
+                        "threshold": "all rows in 80-210 bps band and identity holds",
+                        "ok": irr_band_ok,
+                    }
+                )
+                if not irr_band_ok:
+                    failures.append("Subscription line impact rows do not consistently satisfy PE IRR delta rules")
+
+                cur.execute(
+                    """
+                    SELECT
+                        COUNT(*) AS apex_rows,
+                        MAX(pp.IRR_SUB_LINE_IMPACT_BPS) AS apex_max_bps,
+                        (
+                            SELECT MAX(IRR_SUB_LINE_IMPACT_BPS)
+                            FROM PORTFOLIO_PERFORMANCE
+                        ) AS overall_max_bps
+                    FROM PORTFOLIO_PERFORMANCE pp
+                    JOIN PORTFOLIO_COMPANIES pc ON pp.COMPANY_ID = pc.COMPANY_ID
+                    WHERE LOWER(pc.COMPANY_NAME) = 'apex industrial solutions'
+                    """
+                )
+                apex_rows, apex_max_bps, overall_max_bps = cur.fetchone()
+                apex_ok = int(apex_rows or 0) > 0 and apex_max_bps is not None and abs(float(apex_max_bps) - 210.0) <= 1.0 and overall_max_bps is not None and abs(float(overall_max_bps) - 210.0) <= 1.0
+                checks.append(
+                    {
+                        "name": "pe_apex_subscription_line_outlier",
+                        "value": {"rows": int(apex_rows or 0), "apex_max_bps": apex_max_bps, "overall_max_bps": overall_max_bps},
+                        "threshold": "Apex exists and max impact == 210 bps",
+                        "ok": apex_ok,
+                    }
+                )
+                if not apex_ok:
+                    failures.append("Apex Industrial Solutions outlier is missing or not set to the expected 210 bps impact")
+
+                cur.execute(
+                    """
+                    WITH covenant_exceptions AS (
+                        SELECT
+                            LOWER(pc.COMPANY_NAME) AS company_name,
+                            LOWER(pp.COVENANT_STATUS) AS covenant_status,
+                            COUNT(*) AS row_count
+                        FROM PORTFOLIO_PERFORMANCE pp
+                        JOIN PORTFOLIO_COMPANIES pc ON pp.COMPANY_ID = pc.COMPANY_ID
+                        WHERE LOWER(pp.COVENANT_STATUS) <> 'compliant'
+                        GROUP BY 1, 2
+                    )
+                    SELECT
+                        COUNT_IF(company_name = 'meridian specialty chemicals' AND covenant_status = 'waived') AS meridian_waived_groups,
+                        COUNT_IF(company_name <> 'meridian specialty chemicals' OR covenant_status <> 'waived') AS invalid_groups
+                    FROM covenant_exceptions
+                    """
+                )
+                meridian_groups, invalid_groups = cur.fetchone()
+                meridian_ok = int(meridian_groups or 0) > 0 and int(invalid_groups or 0) == 0
+                checks.append(
+                    {
+                        "name": "pe_meridian_covenant_exception",
+                        "value": {"meridian_groups": int(meridian_groups or 0), "invalid_groups": int(invalid_groups or 0)},
+                        "threshold": "Meridian only, status waived",
+                        "ok": meridian_ok,
+                    }
+                )
+                if not meridian_ok:
+                    failures.append("Meridian Specialty Chemicals is not the sole waived/non-compliant covenant exception")
+
+                cur.execute(
+                    """
+                    WITH sector_perf AS (
+                        SELECT
+                            s.SECTOR_NAME,
+                            AVG(pp.ENTRY_EV_EBITDA_MULTIPLE) AS avg_entry_multiple,
+                            AVG(pp.TOTAL_RETURN_MULTIPLE) AS avg_tvpi
+                        FROM PORTFOLIO_PERFORMANCE pp
+                        JOIN PORTFOLIO_COMPANIES pc ON pp.COMPANY_ID = pc.COMPANY_ID
+                        JOIN SECTORS s ON pc.SECTOR_ID = s.SECTOR_ID
+                        GROUP BY 1
+                    )
+                    SELECT
+                        MAX(CASE WHEN LOWER(SECTOR_NAME) = 'technology' THEN avg_entry_multiple END) AS tech_entry,
+                        MAX(CASE WHEN LOWER(SECTOR_NAME) = 'technology' THEN avg_tvpi END) AS tech_tvpi,
+                        MAX(CASE WHEN LOWER(SECTOR_NAME) <> 'technology' THEN avg_entry_multiple END) AS other_entry_max,
+                        MAX(CASE WHEN LOWER(SECTOR_NAME) <> 'technology' THEN avg_tvpi END) AS other_tvpi_max
+                    FROM sector_perf
+                    """
+                )
+                tech_entry, tech_tvpi, other_entry_max, other_tvpi_max = cur.fetchone()
+                tech_sector_ok = (
+                    tech_entry is not None
+                    and tech_tvpi is not None
+                    and other_entry_max is not None
+                    and other_tvpi_max is not None
+                    and float(tech_entry) >= float(other_entry_max)
+                    and float(tech_tvpi) >= float(other_tvpi_max)
+                )
+                checks.append(
+                    {
+                        "name": "pe_technology_sector_leads_multiples",
+                        "value": {
+                            "tech_entry": tech_entry,
+                            "tech_tvpi": tech_tvpi,
+                            "other_entry_max": other_entry_max,
+                            "other_tvpi_max": other_tvpi_max,
+                        },
+                        "threshold": "Technology leads average entry and return multiples",
+                        "ok": tech_sector_ok,
+                    }
+                )
+                if not tech_sector_ok:
+                    failures.append("Technology sector does not lead entry and return multiples as required by the State Street narrative")
+
+                cur.execute(
+                    """
+                    WITH vintage_rank AS (
+                        SELECT
+                            VINTAGE_YEAR,
+                            SUM(REPORTED_VALUE_USD) AS total_reported_value,
+                            DENSE_RANK() OVER (ORDER BY SUM(REPORTED_VALUE_USD) DESC) AS value_rank
+                        FROM PORTFOLIO_PERFORMANCE
+                        GROUP BY 1
+                    )
+                    SELECT LISTAGG(TO_VARCHAR(VINTAGE_YEAR), ',') WITHIN GROUP (ORDER BY VINTAGE_YEAR)
+                    FROM vintage_rank
+                    WHERE value_rank <= 2
+                    """
+                )
+                top_vintages = cur.fetchone()[0] or ""
+                top_vintage_set = {part.strip() for part in str(top_vintages).split(",") if part.strip()}
+                vintage_ok = top_vintage_set == {"2021", "2022"}
+                checks.append(
+                    {
+                        "name": "pe_top_vintages_reported_value",
+                        "value": sorted(top_vintage_set),
+                        "threshold": "top 2 vintages are 2021 and 2022",
+                        "ok": vintage_ok,
+                    }
+                )
+                if not vintage_ok:
+                    failures.append("2021 and 2022 are not the top reported-value vintages")
+
+                cur.execute(
+                    """
+                    WITH healthcare_quarters AS (
+                        SELECT
+                            DATE_TRUNC('quarter', pp.FULL_DATE) AS quarter_start,
+                            AVG(pp.TOTAL_VALUE_USD) AS avg_total_value
+                        FROM PORTFOLIO_PERFORMANCE pp
+                        JOIN PORTFOLIO_COMPANIES pc ON pp.COMPANY_ID = pc.COMPANY_ID
+                        JOIN SECTORS s ON pc.SECTOR_ID = s.SECTOR_ID
+                        WHERE LOWER(s.SECTOR_NAME) = 'healthcare'
+                        GROUP BY 1
+                    )
+                    SELECT
+                        MAX(CASE WHEN quarter_start = DATE '2024-07-01' THEN avg_total_value END) AS q3_2024_value,
+                        MAX(CASE WHEN quarter_start = DATE '2024-10-01' THEN avg_total_value END) AS q4_2024_value
+                    FROM healthcare_quarters
+                    """
+                )
+                q3_2024_value, q4_2024_value = cur.fetchone()
+                healthcare_dip_ok = (
+                    q3_2024_value is not None
+                    and q4_2024_value is not None
+                    and float(q4_2024_value) < float(q3_2024_value)
+                )
+                checks.append(
+                    {
+                        "name": "pe_healthcare_q4_2024_dip",
+                        "value": {"q3_2024": q3_2024_value, "q4_2024": q4_2024_value},
+                        "threshold": "Q4 2024 healthcare total value lower than Q3 2024",
+                        "ok": healthcare_dip_ok,
+                    }
+                )
+                if not healthcare_dip_ok:
+                    failures.append("Healthcare Q4 2024 performance dip is missing")
+
+                cur.execute(
+                    """
+                    WITH company_trends AS (
+                        SELECT
+                            pc.COMPANY_NAME,
+                            FIRST_VALUE(pp.REVENUE_USD) OVER (PARTITION BY pc.COMPANY_NAME ORDER BY pp.FULL_DATE) AS first_revenue,
+                            LAST_VALUE(pp.REVENUE_USD) OVER (
+                                PARTITION BY pc.COMPANY_NAME ORDER BY pp.FULL_DATE
+                                ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
+                            ) AS last_revenue,
+                            FIRST_VALUE(pp.EBITDA_MARGIN_PCT) OVER (PARTITION BY pc.COMPANY_NAME ORDER BY pp.FULL_DATE) AS first_margin,
+                            LAST_VALUE(pp.EBITDA_MARGIN_PCT) OVER (
+                                PARTITION BY pc.COMPANY_NAME ORDER BY pp.FULL_DATE
+                                ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
+                            ) AS last_margin
+                        FROM PORTFOLIO_PERFORMANCE pp
+                        JOIN PORTFOLIO_COMPANIES pc ON pp.COMPANY_ID = pc.COMPANY_ID
+                    )
+                    SELECT COUNT(DISTINCT COMPANY_NAME)
+                    FROM company_trends
+                    WHERE last_revenue > first_revenue AND last_margin < first_margin
+                    """
+                )
+                trend_company_count = int(cur.fetchone()[0] or 0)
+                trend_ok = trend_company_count >= 1
+                checks.append(
+                    {
+                        "name": "pe_revenue_up_margin_down_company_count",
+                        "value": trend_company_count,
+                        "threshold": ">= 1",
+                        "ok": trend_ok,
+                    }
+                )
+                if not trend_ok:
+                    failures.append("No portfolio company shows the required revenue-up / EBITDA-margin-down trend")
+
+        if is_saas_finance:
+            cur.execute("SELECT COUNT(DISTINCT MONTH_KEY) FROM DATES")
+            month_count = int(cur.fetchone()[0] or 0)
+            checks.append(
+                {
+                    "name": "saas_month_count",
+                    "value": month_count,
+                    "threshold": ">= 24",
+                    "ok": month_count >= 24,
+                }
+            )
+            if month_count < 24:
+                failures.append(f"SaaS finance month count too low: {month_count} (expected >= 24)")
+
+            cur.execute("SELECT COUNT(DISTINCT SEGMENT) FROM CUSTOMERS WHERE SEGMENT IS NOT NULL")
+            segment_count = int(cur.fetchone()[0] or 0)
+            checks.append(
+                {
+                    "name": "saas_segment_distinct_count",
+                    "value": segment_count,
+                    "threshold": ">= 3",
+                    "ok": segment_count >= 3,
+                }
+            )
+            if segment_count < 3:
+                failures.append(f"SaaS finance segment count too low: {segment_count} (expected >= 3)")
+
+            cur.execute("SELECT COUNT(DISTINCT REGION) FROM LOCATIONS WHERE REGION IS NOT NULL")
+            region_count = int(cur.fetchone()[0] or 0)
+            checks.append(
+                {
+                    "name": "saas_region_distinct_count",
+                    "value": region_count,
+                    "threshold": ">= 3",
+                    "ok": region_count >= 3,
+                }
+            )
+            if region_count < 3:
+                failures.append(f"SaaS finance region count too low: {region_count} (expected >= 3)")
+
+            cur.execute(
+                """
+                SELECT
+                    COUNT(*) AS total_rows,
+                    COUNT_IF(
+                        ABS(
+                            ENDING_ARR_USD - (
+                                STARTING_ARR_USD + NEW_LOGO_ARR_USD + EXPANSION_ARR_USD
+                                - CONTRACTION_ARR_USD - CHURNED_ARR_USD
+                            )
+                        ) > 1.0
+                    ) AS bad_arr_rows,
+                    COUNT_IF(ABS((MRR_USD * 12.0) - ENDING_ARR_USD) > 12.0) AS bad_mrr_rows
+                FROM SAAS_CUSTOMER_MONTHLY
+                """
+            )
+            total_rows, bad_arr_rows, bad_mrr_rows = cur.fetchone()
+            total_rows = int(total_rows or 0)
+            bad_arr_rows = int(bad_arr_rows or 0)
+            bad_mrr_rows = int(bad_mrr_rows or 0)
+            arr_identity_ok = total_rows > 0 and bad_arr_rows == 0 and bad_mrr_rows == 0
+            checks.append(
+                {
+                    "name": "saas_arr_rollforward_bad_rows",
+                    "value": {"total": total_rows, "bad_arr": bad_arr_rows, "bad_mrr": bad_mrr_rows},
+                    "threshold": "all rows reconcile",
+                    "ok": arr_identity_ok,
+                }
+            )
+            if not arr_identity_ok:
+                failures.append(
+                    f"SaaS finance ARR identities broken (bad_arr={bad_arr_rows}, bad_mrr={bad_mrr_rows})"
+                )
+
+            cur.execute(
+                """
+                WITH month_counts AS (
+                    SELECT CUSTOMER_KEY, COUNT(DISTINCT MONTH_KEY) AS active_months
+                    FROM SAAS_CUSTOMER_MONTHLY
+                    GROUP BY 1
+                )
+                SELECT AVG(active_months), MIN(active_months), MAX(active_months)
+                FROM month_counts
+                """
+            )
+            avg_active_months, min_active_months, max_active_months = cur.fetchone()
+            avg_active_months = float(avg_active_months or 0.0)
+            min_active_months = int(min_active_months or 0)
+            max_active_months = int(max_active_months or 0)
+            density_ok = avg_active_months >= 12.0 and max_active_months >= 20
+            checks.append(
+                {
+                    "name": "saas_customer_month_density",
+                    "value": {
+                        "avg_active_months": round(avg_active_months, 2),
+                        "min_active_months": min_active_months,
+                        "max_active_months": max_active_months,
+                    },
+                    "threshold": "avg >= 12.0 and max >= 20",
+                    "ok": density_ok,
+                }
+            )
+            if not density_ok:
+                failures.append(
+                    f"SaaS finance customer-month density too sparse (avg={avg_active_months:.2f}, max={max_active_months})"
+                )
+
+            cur.execute(
+                """
+                SELECT
+                    COUNT(*) AS total_rows,
+                    COUNT_IF(ABS(TOTAL_S_AND_M_SPEND_USD - (SALES_SPEND_USD + MARKETING_SPEND_USD)) > 1.0) AS bad_rows
+                FROM SALES_MARKETING_SPEND_MONTHLY
+                """
+            )
+            spend_total_rows, bad_spend_rows = cur.fetchone()
+            spend_total_rows = int(spend_total_rows or 0)
+            bad_spend_rows = int(bad_spend_rows or 0)
+            spend_ok = spend_total_rows > 0 and bad_spend_rows == 0
+            checks.append(
+                {
+                    "name": "saas_spend_identity_bad_rows",
+                    "value": {"total": spend_total_rows, "bad_rows": bad_spend_rows},
+                    "threshold": "== 0",
+                    "ok": spend_ok,
+                }
+            )
+            if not spend_ok:
+                failures.append(f"SaaS finance spend identity broken in {bad_spend_rows} rows")
+
+    except Exception as exc:  # noqa: BLE001
+        failures.append(f"Realism sanity checks failed to execute: {exc}")
+    finally:
+        try:
+            if cur is not None:
+                cur.close()
+        except Exception:
+            pass
+        try:
+            if conn is not None:
+                conn.close()
+        except Exception:
+            pass
+
+    return {"ok": len(failures) == 0, "checks": checks, "failures": failures}
+
+
+def _run_case_chat(
+    case: dict[str, Any],
+    default_llm: str,
+    user_email: str,
+    skip_thoughtspot: bool = False,
+) -> dict[str, Any]:
+    from chat_interface import ChatDemoInterface
+    from demo_personas import get_use_case_config, parse_use_case
+
+    company = case["company"]
+    use_case = case["use_case"]
+    model = default_llm
+    context = case.get("context", "")
+
+    controller = ChatDemoInterface(user_email=user_email)
+    controller.settings["model"] = model
+    controller.vertical, controller.function = parse_use_case(use_case or "")
+    controller.use_case_config = get_use_case_config(
+        controller.vertical or "Generic",
+        controller.function or "Generic",
+    )
+
+    result: dict[str, Any] = {
+        "name": case.get("name") or f"{company}_{use_case}",
+        "company": company,
+        "use_case": use_case,
+        "mode": "chat",
+        "started_at": _now_utc_iso(),
+        "success": False,
+        "stages": {},
+    }
+
+    stage_start = datetime.now(timezone.utc)
+    last_research = None
+    for update in controller.run_research_streaming(company, use_case, generic_context=context):
+        last_research = update
+    result["stages"]["research"] = {
+        "ok": bool(controller.demo_builder and controller.demo_builder.company_analysis_results),
+        "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+        "preview": str(last_research)[:500] if last_research else "",
+    }
+
+    stage_start = datetime.now(timezone.utc)
+    ddl_text = (controller.demo_builder.schema_generation_results or "") if controller.demo_builder else ""
+    if not ddl_text:
+        _, ddl_text = controller.run_ddl_creation()
+    result["stages"]["ddl"] = {
+        "ok": bool(ddl_text and "CREATE TABLE" in ddl_text.upper()),
+        "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+        "ddl_length": len(ddl_text or ""),
+    }
+
+    if not result["stages"]["ddl"]["ok"]:
+        result["error"] = "DDL generation failed"
+        result["finished_at"] = _now_utc_iso()
+        return result
+
+    stage_start = datetime.now(timezone.utc)
+    deploy_error = None
+    try:
+        for _ in controller.run_deployment_streaming():
+            pass
+    except Exception as exc:  # noqa: BLE001
+        deploy_error = str(exc)
+    deployed_schema = getattr(controller, "_deployed_schema_name", None)
+    schema_candidate = deployed_schema or getattr(controller, "_last_schema_name", None)
+    result["stages"]["deploy_snowflake"] = {
+        "ok": bool(deployed_schema) and deploy_error is None,
+        "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+        "schema": schema_candidate,
+        "error": deploy_error,
+    }
+
+    if schema_candidate:
+        quality_report_path = getattr(controller, "_last_population_quality_report_path", None)
+        result["stages"]["quality_gate"] = _build_quality_gate_stage(quality_report_path)
+        if not result["stages"]["quality_gate"]["ok"]:
+            result["error"] = f"Quality gate failed: {quality_report_path or 'missing quality report'}"
+        elif deploy_error and not result.get("error"):
+            result["error"] = deploy_error
+
+        stage_start = datetime.now(timezone.utc)
+        if result["stages"]["quality_gate"]["ok"]:
+            sanity = _run_realism_sanity_checks(schema_candidate, case)
+            result["stages"]["realism_sanity"] = {
+                "ok": bool(sanity.get("ok")),
+                "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+                "checks": sanity.get("checks", []),
+                "failures": sanity.get("failures", []),
+            }
+            if not result["stages"]["realism_sanity"]["ok"] and not result.get("error"):
+                result["error"] = "Realism sanity checks failed before ThoughtSpot deployment"
+
+    if not skip_thoughtspot and deployed_schema and not result.get("error"):
+        stage_start = datetime.now(timezone.utc)
+        ts_ok = True
+        ts_last = None
+        try:
+            for ts_update in controller._run_thoughtspot_deployment(deployed_schema, company, use_case):
+                ts_last = ts_update
+        except Exception as exc:  # noqa: BLE001
+            ts_ok = False
+            ts_last = str(exc)
+        # Some deployment paths return a structured failure payload rather than
+        # raising; treat those as failures so pass/fail reporting is accurate.
+        ts_preview_text = str(ts_last) if ts_last is not None else ""
+        if ts_ok and (
+            "THOUGHTSPOT DEPLOYMENT FAILED" in ts_preview_text.upper()
+            or "MODEL VALIDATION FAILED" in ts_preview_text.upper()
+            or "LIVEBOARD CREATION FAILED" in ts_preview_text.upper()
+        ):
+            ts_ok = False
+        result["stages"]["deploy_thoughtspot"] = {
+            "ok": ts_ok,
+            "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+            "preview": ts_preview_text[:1000],
+        }
+
+    result["schema_name"] = schema_candidate
+    result["success"] = all(stage.get("ok") for stage in result["stages"].values())
+    result["finished_at"] = _now_utc_iso()
+    return result
+
+
+def _run_case_offline(
+    case: dict[str, Any],
+    default_llm: str,
+    user_email: str,
+    skip_thoughtspot: bool = False,
+) -> dict[str, Any]:
+    from cdw_connector import SnowflakeDeployer
+    from demo_prep import generate_demo_base_name
+    from legitdata_bridge import populate_demo_data
+    from thoughtspot_deployer import deploy_to_thoughtspot
+
+    company = case["company"]
+    use_case = case["use_case"]
+
+    result: dict[str, Any] = {
+        "name": case.get("name") or f"{company}_{use_case}",
+        "company": company,
+        "use_case": use_case,
+        "mode": "offline_ddl",
+        "started_at": _now_utc_iso(),
+        "success": False,
+        "stages": {},
+        "ddl_template": "offline_star_schema_v1",
+    }
+
+    deployer = SnowflakeDeployer()
+
+    # 1) Snowflake schema + DDL deploy
+    stage_start = datetime.now(timezone.utc)
+    ok, msg = deployer.connect()
+    if not ok:
+        result["stages"]["snowflake_connect"] = {
+            "ok": False,
+            "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+            "message": msg,
+        }
+        result["error"] = msg
+        result["finished_at"] = _now_utc_iso()
+        return result
+
+    base_name = generate_demo_base_name("", company)
+    ok, schema_name, ddl_msg = deployer.create_demo_schema_and_deploy(base_name, OFFLINE_DEMO_DDL)
+    result["stages"]["snowflake_ddl"] = {
+        "ok": ok,
+        "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+        "schema": schema_name,
+        "message": ddl_msg,
+    }
+    if not ok or not schema_name:
+        result["error"] = ddl_msg
+        result["finished_at"] = _now_utc_iso()
+        return result
+
+    # 2) Data population via LegitData
+    stage_start = datetime.now(timezone.utc)
+    pop_ok, pop_msg, pop_results = populate_demo_data(
+        ddl_content=OFFLINE_DEMO_DDL,
+        company_url=company,
+        use_case=use_case,
+        schema_name=schema_name,
+        llm_model=default_llm,
+        user_email=user_email,
+        size="medium",
+    )
+    result["stages"]["populate_data"] = {
+        "ok": pop_ok,
+        "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+        "rows": pop_results,
+        "quality_report": _parse_quality_report_path(pop_msg),
+    }
+    if not pop_ok:
+        result["error"] = pop_msg
+        result["finished_at"] = _now_utc_iso()
+        return result
+
+    quality_report_path = _parse_quality_report_path(pop_msg)
+    result["stages"]["quality_gate"] = _build_quality_gate_stage(quality_report_path)
+    if not result["stages"]["quality_gate"]["ok"]:
+        result["error"] = f"Quality gate failed: {quality_report_path or 'missing quality report'}"
+        result["schema_name"] = schema_name
+        result["finished_at"] = _now_utc_iso()
+        return result
+
+    stage_start = datetime.now(timezone.utc)
+    sanity = _run_realism_sanity_checks(schema_name, case)
+    result["stages"]["realism_sanity"] = {
+        "ok": bool(sanity.get("ok")),
+        "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+        "checks": sanity.get("checks", []),
+        "failures": sanity.get("failures", []),
+    }
+    if not result["stages"]["realism_sanity"]["ok"]:
+        result["error"] = "Realism sanity checks failed before ThoughtSpot deployment"
+        result["schema_name"] = schema_name
+        result["finished_at"] = _now_utc_iso()
+        return result
+
+    # 3) ThoughtSpot model + liveboard
+    if not skip_thoughtspot:
+        stage_start = datetime.now(timezone.utc)
+        ts_result = deploy_to_thoughtspot(
+            ddl=OFFLINE_DEMO_DDL,
+            database=os.getenv("SNOWFLAKE_DATABASE", "DEMOBUILD"),
+            schema=schema_name,
+            base_name=base_name,
+            connection_name=f"{base_name}_conn",
+            company_name=company,
+            use_case=use_case,
+            llm_model=default_llm,
+        )
+        result["stages"]["deploy_thoughtspot"] = {
+            "ok": bool(ts_result and not ts_result.get("errors")),
+            "duration_s": (datetime.now(timezone.utc) - stage_start).total_seconds(),
+            "result": ts_result,
+        }
+
+    result["schema_name"] = schema_name
+    result["success"] = all(stage.get("ok") for stage in result["stages"].values())
+    result["finished_at"] = _now_utc_iso()
+    return result
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Run New Vision sample set")
+    parser.add_argument(
+        "--cases-file",
+        default="tests/newvision_test_cases.yaml",
+        help="Path to YAML test case file",
+    )
+    parser.add_argument(
+        "--skip-thoughtspot",
+        action="store_true",
+        help="Run through data generation only and skip ThoughtSpot object creation",
+    )
+    parser.add_argument(
+        "--offline-ddl",
+        action="store_true",
+        help="Force offline DDL mode (no LLM dependency)",
+    )
+    args = parser.parse_args()
+    user_email, default_llm = _resolve_runtime_settings()
+    from startup_validation import validate_required_pipeline_settings_or_raise
+
+    validate_required_pipeline_settings_or_raise(
+        default_llm=default_llm,
+        require_thoughtspot=not args.skip_thoughtspot,
+        require_snowflake=True,
+    )
+
+    cases_file = (PROJECT_ROOT / args.cases_file).resolve()
+    cases = _load_cases(cases_file)
+
+    run_id = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
+    out_dir = PROJECT_ROOT / "results" / "newvision_samples" / run_id
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    use_offline = bool(args.offline_ddl)
+    print(f"Mode: {'offline_ddl' if use_offline else 'chat'}", flush=True)
+    print(f"default_llm: {default_llm}", flush=True)
+
+    results = []
+    for idx, case in enumerate(cases, start=1):
+        print(f"\n[{idx}/{len(cases)}] {case.get('name', case['company'])} -> {case['use_case']}", flush=True)
+        try:
+            if use_offline:
+                case_result = _run_case_offline(
+                    case,
+                    default_llm=default_llm,
+                    user_email=user_email,
+                    skip_thoughtspot=args.skip_thoughtspot,
+                )
+            else:
+                case_result = _run_case_chat(
+                    case,
+                    default_llm=default_llm,
+                    user_email=user_email,
+                    skip_thoughtspot=args.skip_thoughtspot,
+                )
+        except Exception as exc:  # noqa: BLE001
+            case_result = {
+                "name": case.get("name") or f"{case['company']}_{case['use_case']}",
+                "company": case["company"],
+                "use_case": case["use_case"],
+                "mode": "offline_ddl" if use_offline else "chat",
+                "started_at": _now_utc_iso(),
+                "finished_at": _now_utc_iso(),
+                "success": False,
+                "error": f"Runner exception: {exc}",
+                "stages": {
+                    "runner_exception": {
+                        "ok": False,
+                        "message": str(exc),
+                    }
+                },
+            }
+        results.append(case_result)
+        (out_dir / f"{case_result['name']}.json").write_text(
+            json.dumps(case_result, indent=2),
+            encoding="utf-8",
+        )
+        print(f"  success={case_result['success']} schema={case_result.get('schema_name')}", flush=True)
+
+    summary = {
+        "run_id": run_id,
+        "mode": "offline_ddl" if use_offline else "chat",
+        "cases_file": str(cases_file),
+        "total": len(results),
+        "passed": sum(1 for r in results if r.get("success")),
+        "failed": sum(1 for r in results if not r.get("success")),
+        "results": results,
+    }
+    (out_dir / "summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
+
+    print("\nSaved sample artifacts:", out_dir)
+    print(f"Passed: {summary['passed']} / {summary['total']}")
+
+    if summary["failed"]:
+        raise SystemExit(1)
+
+
+if __name__ == "__main__":
+    main()

tests/newvision_test_cases.yaml

@@ -0,0 +1,46 @@
+# New Vision 4-case review set (no-login runner)
+
+test_cases:
+  - name: "amazon_defined"
+    company: "amazon.com"
+    use_case: "Sales Analytics"
+    model: "claude-sonnet-4.5"
+    tags: [defined, retail, baseline]
+
+  - name: "chase_defined"
+    company: "chase.com"
+    use_case: "Financial Analytics"
+    model: "claude-sonnet-4.5"
+    tags: [defined, banking]
+
+  - name: "nike_defined"
+    company: "nike.com"
+    use_case: "Retail Analytics"
+    model: "claude-sonnet-4.5"
+    tags: [defined, retail]
+
+  - name: "bayer_generic"
+    company: "bayer.com"
+    use_case: "Prescription Performance and Sales Impact"
+    model: "claude-sonnet-4.5"
+    tags: [generic, pharma]
+    context: |
+      Primary business driver:
+      Prescription growth and revenue performance of a branded pharmaceutical product.
+
+      Frequent changes and initiatives:
+      - Price adjustments and contract negotiations
+      - Reimbursement and formulary access changes
+      - Sales force targeting and promotional activity
+      - Launch or modification of patient access programs
+
+      Highly segmented by:
+      - Geography and sales territory
+      - Payer and formulary status
+      - Prescriber specialty and prescribing volume
+      - Customer segment (high vs low prescribers, new vs existing writers)
+
+      Business questions:
+      1) How did new and total prescriptions for Drug X change in Germany after the price and formulary update, compared to the previous month?
+      2) Break this down by payer status and prescriber segment.
+      3) Did prescribing frequency or sales volume change among physicians who did not increase prescribing after the update?

tests/newvision_test_cases_2.yaml

@@ -0,0 +1,278 @@
+test_cases:
+  - name: "thoughtspot_legal_ops_spend"
+    company: "thoughtspot.com"
+    use_case: "Legal Operations and Spend Management"
+    model: "claude-sonnet-4.5"
+    tags: [legal, operations, spend, generic]
+    context: |
+      Target persona: General Counsel / VP of Legal Operations.
+
+      Build a schema with exactly 2 fact tables and 4 dimension tables.
+
+      Fact table 1: LEGAL_MATTERS (workload/matter management)
+      Columns:
+      - matter_id
+      - matter_key (example: LGL-6691)
+      - summary
+      - request_type (Sales, NDA, Procurement, Alliance, Marketing, Other)
+      - priority (P1, P2, P3)
+      - status (New, In Progress, Done)
+      - assigned_attorney_id (FK)
+      - reporter_name
+      - created_date
+      - updated_date
+      - resolved_date
+      - matter_type_id (FK to MATTER_TYPES)
+      - region (US East, US West, EMEA, APAC)
+      - is_customer_deal (boolean)
+      - sfdc_opportunity_value
+
+      Fact table 2: OUTSIDE_COUNSEL_INVOICES (spend management)
+      Columns:
+      - invoice_id
+      - outside_counsel_id (FK)
+      - matter_id (FK to LEGAL_MATTERS)
+      - matter_number (business reference only; do not use as the relationship key)
+      - description
+      - services_amount
+      - costs_amount
+      - total_amount
+      - invoice_date
+      - invoice_month
+      - payment_status (Paid, Pending, Overdue)
+
+      IMPORTANT RELATIONSHIP RULE:
+      - MATTER_TYPE belongs to LEGAL_MATTERS through matter_type_id.
+      - OUTSIDE_COUNSEL_INVOICES must link to LEGAL_MATTERS through matter_id.
+      - Do not make invoices carry an independent matter_type truth.
+
+      Dimension table 1: ATTORNEYS
+      - 100 in-house attorneys
+      - attorney_id, attorney_name, title (Associate GC, Senior Counsel, Paralegal, VP Legal, GC),
+        department (Commercial, IP, Employment, Corporate, Compliance), hire_date, manager_name
+
+      Dimension table 2: OUTSIDE_COUNSEL
+      - 5 firms: Cooley LLP, Young Basile, Baker McKenzie, McDermott Will & Emery, Newman Du Wors
+      - firm_id, firm_name, specialty (IP, Corporate, Employment, Commercial, Regulatory),
+        billing_model (Hourly, Flat Fee, Blended), avg_hourly_rate
+
+      Dimension table 3: MATTER_TYPES
+      - NDA, Sales Agreement, Procurement, Alliance, Marketing, Patent Filing, Trademark Application,
+        Litigation, Employment, Corporate Governance
+
+      Dimension table 4: DATE_DIM
+      - Standard date dimension
+
+      Outlier patterns to include:
+      - November spend spike about 40% higher than other months, driven by 4 new patent filings and litigation surge
+      - Travis Guerre spends disproportionate time on NDA reviews versus peers
+      - One attorney should have significantly longer completion times on Sales matters
+      - Cooley LLP should be highest-spend firm, followed by Baker McKenzie
+      - Q4 should show higher volume across all request types
+
+      Data volume:
+      - About 2,000 legal matters over 24 months
+      - About 500 invoices across the 5 firms
+
+      Key Spotter questions to support:
+      - Why was spend so high in November?
+      - What were the top 3 outside counsels by spend?
+      - What does Travis spend most of his time on?
+      - Average time to close NDA requests
+      - Which attorneys have the longest completion times for Sales matters?
+
+      QUALITY GUARDRAILS (required):
+      - Keep month-over-month change for core spend and volume measures within +/-20% in normal months.
+      - Allow large changes only for named events (November spend spike and Q4 lift), with at most 2 outlier periods.
+      - Enforce arithmetic consistency: total_amount = services_amount + costs_amount.
+      - Keep payment_status and request_type values strictly within listed domains.
+      - Keep trend lines smooth and realistic, not random jagged spikes.
+
+  - name: "thoughtspot_saas_finance_defined"
+    company: "thoughtspot.com"
+    use_case: "Financial Analytics"
+    model: "claude-sonnet-4.5"
+    tags: [saas, finance, arr, defined]
+    context: |
+      Build a SaaS finance story for a CFO/VP Finance persona.
+
+      Focus metrics:
+      - ARR, MRR, NRR, Gross Revenue Retention, churn_rate
+      - expansion_arr, contraction_arr, new_logo_arr
+      - CAC, LTV, payback_months
+      - billings, collections, deferred_revenue
+
+      Data expectations:
+      - 24 months of monthly history
+      - 1,500-2,500 customer-account records
+      - Enterprise, Mid-Market, SMB segments
+      - Regions: North America, EMEA, APAC
+      - Required dimensions: DATES, CUSTOMERS, PRODUCTS, LOCATIONS
+      - Required facts: SAAS_CUSTOMER_MONTHLY, SALES_MARKETING_SPEND_MONTHLY
+      - SAAS_CUSTOMER_MONTHLY grain must be one row per customer per month
+      - SALES_MARKETING_SPEND_MONTHLY grain must be one row per month per segment per region
+      - SALES_MARKETING_SPEND_MONTHLY must carry NRR_PCT, GRR_PCT, NET_NEW_ARR_USD, and CAC_USD
+
+      Outlier patterns to include:
+      - Q4 ARR uplift from enterprise expansion
+      - July SMB churn spike and temporary NRR dip
+      - EMEA collections lag increases DSO
+      - One segment with unusually strong upsell conversion
+
+      Key Spotter questions:
+      - Why did ARR jump in Q4?
+      - Which segment is driving the most churn?
+      - What are the top drivers of NRR change month over month?
+      - Where are collections delays impacting cash flow?
+
+      QUALITY GUARDRAILS (required):
+      - Keep ARR/MRR/NRR time series smooth with normal month-over-month movement within +/-20%.
+      - Restrict major variance to named events only (Q4 ARR uplift and July SMB churn spike), max 2 outlier periods.
+      - Preserve metric relationships: ending_arr approximately prior_arr + expansion_arr + new_logo_arr - churned_arr - contraction_arr.
+      - Preserve metric relationships: mrr approximately ending_arr / 12.
+      - Preserve spend relationships: total_s_and_m_spend = sales_spend + marketing_spend.
+      - Preserve derived KPI relationships: nrr and grr should reconcile to the monthly ARR movement components for each segment/region.
+      - Keep categorical fields in-domain only (segment and region lists above).
+      - Avoid synthetic or nonsense category values.
+
+  - name: "datadog_saas_finance_generic"
+    company: "datadog.com"
+    use_case: "SaaS Finance and Unit Economics"
+    model: "claude-sonnet-4.5"
+    tags: [saas, finance, unit_economics, generic]
+    context: |
+      Create a generic SaaS finance and unit economics demo for FP&A and Finance Ops.
+
+      Include realistic SaaS business entities:
+      - Customers/accounts, subscriptions, invoices, payments, usage
+      - Product lines (core platform, security, observability, add-ons)
+      - Segments, geo regions, and contract terms
+      - Required dimensions: DATES, CUSTOMERS, PRODUCTS, LOCATIONS
+      - Required facts: SAAS_CUSTOMER_MONTHLY, SALES_MARKETING_SPEND_MONTHLY
+      - SAAS_CUSTOMER_MONTHLY grain must be one row per customer per month
+      - SALES_MARKETING_SPEND_MONTHLY grain must be one row per month per segment per region
+      - SALES_MARKETING_SPEND_MONTHLY must carry NRR_PCT, GRR_PCT, NET_NEW_ARR_USD, and CAC_USD
+
+      Core metrics:
+      - ARR, MRR, NRR, GRR
+      - churned_arr, expansion_arr, net_new_arr
+      - CAC, LTV, gross_margin, payback_period
+      - billed_vs_collected, deferred_revenue
+
+      Outlier patterns to include:
+      - November billings spike due to annual prepay renewals
+      - One product line has margin compression for two quarters
+      - APAC has stronger net new ARR growth than other regions
+      - SMB churn rises while enterprise expansion offsets total ARR
+
+      Key Spotter questions:
+      - Why were billings unusually high in November?
+      - Which product line has the weakest margin trend?
+      - What is driving NRR by segment?
+      - Which region contributes most to net new ARR?
+
+      QUALITY GUARDRAILS (required):
+      - Keep monthly finance series smooth with normal month-over-month movement within +/-20%.
+      - Allow major movement only for named events (November billings spike, two-quarter margin compression), max 2 outlier periods.
+      - Use strict arithmetic consistency where relevant (for example billed_vs_collected and deferred revenue behavior over time).
+      - Preserve metric relationships: ending_arr approximately prior_arr + expansion_arr + new_logo_arr - churned_arr - contraction_arr.
+      - Preserve metric relationships: mrr approximately ending_arr / 12.
+      - Preserve spend relationships: total_s_and_m_spend = sales_spend + marketing_spend.
+      - Preserve derived KPI relationships: nrr and grr should reconcile to the monthly ARR movement components for each segment/region.
+      - Use only realistic categorical values for product lines, segments, and regions.
+      - For organization entities, prefer ACCOUNT_NAME or COMPANY_NAME style values; avoid person-like names in org columns.
+
+  - name: "statestreet_private_equity_lp_reporting"
+    company: "statestreet.com"
+    use_case: "Private Equity Portfolio Analytics for LP Reporting"
+    model: "claude-sonnet-4.5"
+    tags: [private_equity, lp_reporting, portfolio_analytics, statestreet]
+    context: |
+      Schema Requirements - 1 fact table and 5 dimension tables:
+
+      FACT TABLE: PORTFOLIO_PERFORMANCE (quarterly reporting data per portfolio company per quarter)
+      - record_id
+      - fund_id (FK)
+      - company_id (FK)
+      - quarter_date
+      - vintage_year
+      - invested_capital
+      - reported_value (NAV)
+      - distributions
+      - total_value (reported_value + distributions)
+      - gross_irr
+      - net_irr
+      - gross_irr_without_sub_line
+      - irr_sub_line_impact_bps
+      - total_return_multiple (TVPI)
+      - dpi_multiple
+      - rvpi_multiple
+      - revenue
+      - ebitda
+      - net_debt
+      - entry_ev_ebitda_multiple
+      - current_ev_ebitda_multiple
+      - revenue_growth_pct
+      - ebitda_margin_pct
+      - debt_to_ebitda_ratio
+      - covenant_status (Compliant, Waived, Breached)
+      - debt_performance_status (Performing, Watch List, Non-Performing)
+
+      DIM TABLE 1: FUNDS
+      - fund_id
+      - fund_name ("Alpha Private Equity Fund III, L.P." and 2-3 others)
+      - fund_manager
+      - fund_size
+      - fund_currency
+      - fund_strategy (Buyout, Growth, Venture)
+      - inception_date
+
+      DIM TABLE 2: PORTFOLIO_COMPANIES
+      - company_id
+      - company_name (10 realistic PE-backed companies like "Apex Industrial Solutions", "Meridian Specialty Chemicals", "NovaTech Data Systems")
+      - sector (Industrials, Healthcare, Technology, Consumer, Financial Services)
+      - sub_sector
+      - headquarters_state
+      - investment_date
+      - investment_stage (Active, Realized, Written Off)
+      - board_seats_held
+
+      DIM TABLE 3: SECTORS
+      - sector_id
+      - sector_name
+      - sector_category (Cyclical, Defensive, Growth)
+
+      DIM TABLE 4: LP_INVESTORS
+      - investor_id
+      - investor_name (pension funds, endowments, sovereign wealth)
+      - investor_type (Pension Fund, Endowment, Family Office, Sovereign Wealth, Insurance)
+      - commitment_amount
+      - aum_total
+
+      DIM TABLE 5: DATE_DIM
+      - standard quarterly date dimension
+
+      Outlier patterns to bake in (critical for demo narrative):
+      - Subscription line impact: all companies should have gross_irr_without_sub_line that is 80-210 bps lower than gross_irr. Apex Industrial Solutions should have the highest impact at exactly 210 bps.
+      - Meridian Specialty Chemicals should have covenant_status = 'Waived' as the only company with a non-compliant status.
+      - Technology sector investments made at higher entry multiples (12-15x EV/EBITDA) should have the highest total return multiples (2.5-3.8x TVPI).
+      - 2021 and 2022 vintage years should hold the most reported value.
+      - Q4 2024 should show a performance dip in Healthcare sector.
+      - One company should show revenue growing but EBITDA margin declining.
+
+      Data volume:
+      - 10 portfolio companies x 12 quarters (3 years) = ~120 rows in fact table
+      - Keep it small and realistic (private equity reporting model)
+      - 5 LP investors, 3 funds
+
+      Target persona:
+      - Portfolio Manager at a $50B Pension Fund (LP client using the embedded experience)
+
+      Key Spotter questions the demo MUST support (in this order):
+      - show me my total invested and reported value by sector
+      - now by vintage year
+      - compare gross irr with and without subscription facility for my active investments
+      - which company has the highest irr impact from the sub line?
+      - show me the trend of revenue vs ebitda for Apex Industrial Solutions since investment
+      - what is the covenant status for my active investments with debt status performing?
+      - show me the relationship between entry ev/ebitda multiple and total return multiple by sector

tests/test_mcp_liveboard.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+"""
+Quick test script for MCP liveboard creation
+"""
+from liveboard_creator import create_liveboard_from_model_mcp
+from thoughtspot_deployer import ThoughtSpotDeployer
+from dotenv import load_dotenv
+import os
+
+load_dotenv()
+
+# Create ThoughtSpot client
+ts_client = ThoughtSpotDeployer(
+    base_url=os.getenv('THOUGHTSPOT_URL'),
+    username=os.getenv('THOUGHTSPOT_USERNAME'),
+    password=os.getenv('THOUGHTSPOT_PASSWORD')
+)
+
+# Test with the model you specified
+model_name = "GEN_02031143_YWV_mdl"
+model_id = "36de83be-e284-4d76-98cf-330902ba1973"
+
+company_data = {
+    'name': 'Test Company',
+    'website': 'test.com'
+}
+
+print(f"Testing MCP liveboard creation with model: {model_name}")
+print(f"Model ID: {model_id}")
+print("-" * 60)
+
+result = create_liveboard_from_model_mcp(
+    ts_client=ts_client,
+    model_id=model_id,
+    model_name=model_name,
+    company_data=company_data,
+    use_case="Retail Sales",
+    num_visualizations=3,  # Just 3 to test quickly
+    liveboard_name="Test Liveboard MCP",
+    llm_model="claude-sonnet-4"
+)
+
+print("-" * 60)
+print(f"Result: {result}")
+
+if result.get('success'):
+    print(f"✅ SUCCESS!")
+    print(f"   Liveboard: {result.get('liveboard_name')}")
+    print(f"   GUID: {result.get('liveboard_guid')}")
+    print(f"   URL: {result.get('liveboard_url')}")
+else:
+    print(f"❌ FAILED: {result.get('error')}")
+
+

thoughtspot_deployer.py

@@ -749,7 +749,10 @@ class ThoughtSpotDeployer:
                 'columns': [],
                 'properties': {
                     'is_bypass_rls': False,
-                    'join_progressive': True
+                    'join_progressive': True,
+                    'spotter_config': {
+                        'is_spotter_enabled': True
+                    }
                 }
             }
         }
@@ -904,7 +907,10 @@ class ThoughtSpotDeployer:
                 'columns': [],
                 'properties': {
                     'is_bypass_rls': False,
-                    'join_progressive': True
+                    'join_progressive': True,
+                    'spotter_config': {
+                        'is_spotter_enabled': True
+                    }
                 }
             }
         }
@@ -1456,7 +1462,10 @@ class ThoughtSpotDeployer:
                 'worksheet_columns': [],  # Adding back - but with GUID references
                 'properties': {
                     'is_bypass_rls': False,
-                    'join_progressive': True
+                    'join_progressive': True,
+                    'spotter_config': {
+                        'is_spotter_enabled': True
+                    }
                 }
             }
         }
@@ -1731,6 +1740,49 @@ class ThoughtSpotDeployer:
             print(f"[ThoughtSpot]    ⚠️  Tag assignment error: {str(e)}", flush=True)
             return False
 
+    def share_objects(self, object_guids: List[str], object_type: str, share_with: str) -> bool:
+        """
+        Share ThoughtSpot objects with a user or group (can_edit / MODIFY).
+
+        Args:
+            object_guids: GUIDs to share
+            object_type: 'LOGICAL_TABLE' for models/tables, 'LIVEBOARD' for liveboards
+            share_with: user email (contains '@') or group name
+        """
+        if not share_with or not object_guids:
+            return True
+
+        principal_type = "USER" if '@' in share_with else "USER_GROUP"
+
+        try:
+            response = self.session.post(
+                f"{self.base_url}/api/rest/2.0/security/metadata/share",
+                json={
+                    "permissions": [
+                        {
+                            "principal": {
+                                "identifier": share_with,
+                                "type": principal_type
+                            },
+                            "share_mode": "MODIFY"
+                        }
+                    ],
+                    "metadata": [
+                        {"identifier": guid, "type": object_type}
+                        for guid in object_guids
+                    ]
+                }
+            )
+            if response.status_code in [200, 204]:
+                print(f"[ThoughtSpot]    ✅ Shared {len(object_guids)} {object_type} with {principal_type} '{share_with}'", flush=True)
+                return True
+            else:
+                print(f"[ThoughtSpot]    ⚠️  Share failed: {response.status_code} - {response.text[:200]}", flush=True)
+                return False
+        except Exception as e:
+            print(f"[ThoughtSpot]    ⚠️  Share error: {str(e)}", flush=True)
+            return False
+
     def _generate_demo_names(self, company_name: str = None, use_case: str = None):
         """Generate standardized demo names using DM convention"""
         from datetime import datetime
@@ -1764,10 +1816,10 @@ class ThoughtSpotDeployer:
         }
 
     def deploy_all(self, ddl: str, database: str, schema: str, base_name: str,
-                  connection_name: str = None, company_name: str = None, 
-                  use_case: str = None, liveboard_name: str = None, 
-                  llm_model: str = None, tag_name: str = None, 
-                  liveboard_method: str = None,
+                  connection_name: str = None, company_name: str = None,
+                  use_case: str = None, liveboard_name: str = None,
+                  llm_model: str = None, tag_name: str = None,
+                  liveboard_method: str = None, share_with: str = None,
                   progress_callback=None) -> Dict:
         """
         Deploy complete data model to ThoughtSpot
@@ -2169,18 +2221,48 @@ class ThoughtSpotDeployer:
                         log_progress(f"Assigning tag '{tag_name}' to model...")
                         self.assign_tags_to_objects([model_guid], 'LOGICAL_TABLE', tag_name)
 
-                        # Step 3.5: Enable Spotter on the model via API
+                        # Share model
+                        _effective_share = share_with or get_admin_setting('SHARE_WITH', required=False)
+                        if _effective_share:
+                            log_progress(f"Sharing model with '{_effective_share}'...")
+                            self.share_objects([model_guid], 'LOGICAL_TABLE', _effective_share)
+
+                        # Step 3.5: Enable Spotter via TML update
+                        # create_new=True import ignores spotter_config — must export then re-import to set it
                         try:
-                            enable_response = self.session.post(
-                                f"{self.base_url}/api/rest/2.0/metadata/sage/enable",
+                            export_resp = self.session.post(
+                                f"{self.base_url}/api/rest/2.0/metadata/tml/export",
                                 json={
-                                    "metadata_identifiers": [model_guid]
+                                    "metadata": [{"identifier": model_guid, "type": "LOGICAL_TABLE"}],
+                                    "export_associated": False,
+                                    "format_type": "YAML"
                                 }
                             )
-                            if enable_response.status_code == 200:
-                                log_progress(f"🤖 Spotter enabled")
+                            if export_resp.status_code == 200:
+                                export_data = export_resp.json()
+                                if export_data and 'edoc' in export_data[0]:
+                                    model_tml_dict = json.loads(export_data[0]['edoc'])
+                                    # Set spotter_config inside properties (correct location per golden demo TML)
+                                    model_tml_dict.setdefault('model', {}).setdefault('properties', {})['spotter_config'] = {'is_spotter_enabled': True}
+                                    updated_tml = yaml.dump(model_tml_dict, allow_unicode=True, sort_keys=False)
+                                    update_resp = self.session.post(
+                                        f"{self.base_url}/api/rest/2.0/metadata/tml/import",
+                                        json={
+                                            "metadata_tmls": [updated_tml],
+                                            "import_policy": "ALL_OR_NONE",
+                                            "create_new": False
+                                        }
+                                    )
+                                    if update_resp.status_code == 200:
+                                        log_progress(f"🤖 Spotter enabled on model")
+                                    else:
+                                        log_progress(f"🤖 Spotter enable update failed: HTTP {update_resp.status_code} — {update_resp.text[:200]}")
+                                else:
+                                    log_progress(f"🤖 Spotter enable: export returned no edoc")
+                            else:
+                                log_progress(f"🤖 Spotter enable: export failed HTTP {export_resp.status_code}")
                         except Exception as spotter_error:
-                            pass  # Not critical
+                            log_progress(f"🤖 Spotter enable exception: {spotter_error}")
 
                         # Step 4: Auto-create Liveboard from model
                         lb_start = time.time()
@@ -2216,40 +2298,37 @@ class ThoughtSpotDeployer:
                                         for col in columns_list:
                                             model_columns.append(col)
                                 
-                                # Get outlier patterns from the vertical×function system
+                                # Get liveboard questions from the vertical×function config
                                 outlier_dicts = []
                                 try:
-                                    from outlier_system import get_outliers_for_use_case
-                                    from demo_personas import parse_use_case
+                                    from demo_personas import parse_use_case, get_use_case_config
                                     uc_vertical, uc_function = parse_use_case(use_case or '')
-                                    if uc_vertical or uc_function:
-                                        outlier_config = get_outliers_for_use_case(
-                                            uc_vertical or "Generic", 
-                                            uc_function or "Generic"
-                                        )
-                                        # Convert OutlierPattern objects to dicts for MCP function
-                                        for op in outlier_config.required:
-                                            outlier_dicts.append({
-                                                'title': op.name,
-                                                'insight': op.viz_talking_point,
-                                                'viz_type': op.viz_type,
-                                                'show_me_query': op.viz_question,
-                                                'kpi_companion': True,
-                                                'spotter_questions': op.spotter_questions,
-                                            })
-                                        for op in outlier_config.optional[:2]:  # Include up to 2 optional
-                                            outlier_dicts.append({
-                                                'title': op.name,
-                                                'insight': op.viz_talking_point,
-                                                'viz_type': op.viz_type,
-                                                'show_me_query': op.viz_question,
-                                                'kpi_companion': False,
-                                                'spotter_questions': op.spotter_questions,
-                                            })
-                                        if outlier_dicts:
-                                            log_progress(f"   [MCP] Using {len(outlier_dicts)} outlier patterns from {uc_vertical}×{uc_function}")
+                                    uc_config = get_use_case_config(uc_vertical or "Generic", uc_function or "Generic")
+                                    lq = uc_config.get("liveboard_questions", [])
+                                    required_qs = [q for q in lq if q.get("required")]
+                                    optional_qs = [q for q in lq if not q.get("required")]
+                                    for q in required_qs:
+                                        outlier_dicts.append({
+                                            'title': q['title'],
+                                            'insight': q.get('insight', ''),
+                                            'viz_type': q['viz_type'],
+                                            'show_me_query': q['viz_question'],
+                                            'kpi_companion': True,
+                                            'spotter_questions': q.get('spotter_qs', []),
+                                        })
+                                    for q in optional_qs[:2]:
+                                        outlier_dicts.append({
+                                            'title': q['title'],
+                                            'insight': q.get('insight', ''),
+                                            'viz_type': q['viz_type'],
+                                            'show_me_query': q['viz_question'],
+                                            'kpi_companion': False,
+                                            'spotter_questions': q.get('spotter_qs', []),
+                                        })
+                                    if outlier_dicts:
+                                        log_progress(f"   [MCP] Using {len(outlier_dicts)} liveboard questions from {uc_vertical}×{uc_function}")
                                 except Exception as outlier_err:
-                                    log_progress(f"   [MCP] Outlier loading skipped: {outlier_err}")
+                                    log_progress(f"   [MCP] Liveboard questions loading skipped: {outlier_err}")
                                 
                                 # MCP creates liveboard
                                 if method == 'HYBRID':
@@ -2265,7 +2344,7 @@ class ThoughtSpotDeployer:
                                         model_name=model_name,
                                         company_data=company_data,
                                         use_case=use_case or 'General Analytics',
-                                        num_visualizations=8,
+                                        num_visualizations=10,
                                         liveboard_name=liveboard_name,
                                         outliers=outlier_dicts if outlier_dicts else None,
                                         llm_model=llm_model,
@@ -2290,7 +2369,7 @@ class ThoughtSpotDeployer:
                                             model_name=model_name,
                                             company_data=company_data,
                                             use_case=use_case or 'General Analytics',
-                                            num_visualizations=8,
+                                            num_visualizations=10,
                                             liveboard_name=liveboard_name,
                                             llm_model=llm_model,
                                             outliers=outlier_dicts if outlier_dicts else None,
@@ -2343,6 +2422,12 @@ class ThoughtSpotDeployer:
                                 if tag_name and liveboard_result.get('liveboard_guid'):
                                     log_progress(f"Assigning tag '{tag_name}' to liveboard...")
                                     self.assign_tags_to_objects([liveboard_result['liveboard_guid']], 'PINBOARD_ANSWER_BOOK', tag_name)
+
+                                # Share liveboard
+                                _effective_share = share_with or get_admin_setting('SHARE_WITH', required=False)
+                                if _effective_share and liveboard_result.get('liveboard_guid'):
+                                    log_progress(f"Sharing liveboard with '{_effective_share}'...")
+                                    self.share_objects([liveboard_result['liveboard_guid']], 'LIVEBOARD', _effective_share)
                             else:
                                 error = f"Liveboard creation failed: {liveboard_result.get('error', 'Unknown error')}"
                                 print(f"❌ DEBUG: Liveboard creation failed! Error: {error}")