mikeboone commited on
Commit
7fcac33
Β·
1 Parent(s): c1e6862

Reorganize docs: CLAUDE.md as source of truth, add PROJECT_STATUS.md, sprint structure

Browse files
Files changed (4) hide show
  1. .cursorrules +16 -214
  2. .gitignore +3 -0
  3. CLAUDE.md +149 -109
  4. PROJECT_STATUS.md +86 -0
.cursorrules CHANGED
@@ -1,214 +1,16 @@
1
- # Cursor Rules for Demo Wire Project
2
-
3
- ## CRITICAL PROCESS MANAGEMENT RULE
4
-
5
- **NEVER restart the application unless explicitly requested**
6
-
7
- - DO NOT run commands like `kill`, `lsof -ti:7862 | xargs kill`, or restart scripts
8
- - DO NOT execute `launch_chat.py`, `demo_prep.py` or similar startup commands
9
- - When making code changes, say "Changes saved to [file] - restart when ready" and STOP
10
- - ONLY restart if user explicitly says: "restart", "rerun the app", "relaunch", "kill and restart"
11
- - If user says "don't stop the process" or "keep it running" - NEVER restart, no matter what
12
-
13
- ## Contact & Communication
14
-
15
- - User (boone) is a ThoughtSpot Architect with 20+ years experience
16
- - You can challenge ideas, but respect deep domain knowledge
17
- - When asked to STOP - stop immediately and wait for further instructions
18
- - Start discussions with back-and-forth using numbered questions
19
- - TS = ThoughtSpot (the company where boone works)
20
- - Ask rather than assume when in doubt
21
-
22
- ## Debugging and Problem-Solving Protocol
23
-
24
- ### Confidence and Communication
25
-
26
- 1. **Don't act over-confident unless you're extremely sure**
27
- - Check documentation before claiming you know how something works
28
- - Say "I don't know" or "Let me check" instead of guessing
29
- - If you're uncertain, say so upfront
30
-
31
- 2. **NEVER claim something is fixed until it's tested**
32
- - ❌ WRONG: "I've fixed the tags issue" (without testing)
33
- - βœ… RIGHT: "I've updated the code - let's test if tags work now"
34
- - Show the test results, don't just assume it works
35
-
36
- 3. **When debugging:**
37
- - Check documentation FIRST before blaming external systems
38
- - State what you're checking and why
39
- - Share what you found (even if it proves you wrong)
40
-
41
- 4. **It's OK to say:**
42
- - "I don't know - should I research this?"
43
- - "I'm not certain, but here are 2 possibilities..."
44
- - "Let me verify this works before saying it's fixed"
45
-
46
- ## File Organization - CRITICAL
47
-
48
- ### NEVER create files in root without asking first
49
-
50
- **tests/** - Real, reusable test cases only
51
- - Unit tests for core functions
52
- - Integration tests that could be automated
53
- - Tests you'd run in CI/CD
54
-
55
- **scratch/** - ALL temporary/experimental/debug files
56
- - ALL experimental/debug/check/verify/analyze scripts
57
- - One-off fixes (fix_*.py, adjust_*.py, emergency_*.py)
58
- - Debug scripts (debug_*.py, check_*.py, verify_*.py)
59
- - Analysis tools (analyze_*.py, get_*.py, show_*.py)
60
- - Test files you're experimenting with
61
- - Backup files (.bak, .bak2)
62
- - Export/debug .yml/.json files
63
- - **ANY script that's temporary or one-time use**
64
- - DO NOT commit without cleanup
65
-
66
- **dev_notes/** - All documentation
67
- - All .md files except README.md and CLAUDE.md
68
- - Presentation materials (.pptx, .html, .txt)
69
- - Research documents
70
- - Architecture notes
71
-
72
- **Root directory** - ONLY essential files
73
- - Main application files
74
- - Core utilities
75
- - Configuration (.env, requirements.txt)
76
- - README.md and CLAUDE.md only
77
- - DO NOT create random files here
78
-
79
- ### Simple Decision Tree
80
-
81
- Creating a new file? Ask yourself:
82
-
83
- 1. **Is it a real test that should be automated?** β†’ `tests/`
84
- 2. **Is it documentation/presentation?** β†’ `dev_notes/`
85
- 3. **Is it core application code?** β†’ Root (but ASK first!)
86
- 4. **Everything else?** β†’ `scratch/` (debug, check, verify, analyze, fix, backup, experimental)
87
-
88
- ### When in doubt: PUT IT IN SCRATCH
89
-
90
- ## Testing Existing Features
91
-
92
- **NEVER create simplified versions of working code**
93
-
94
- When testing:
95
- - ❌ WRONG: Write new simplified code from scratch
96
- - βœ… RIGHT: Call existing working functions in a test harness
97
-
98
- Example:
99
- - ❌ WRONG: Create new viz configs and call low-level functions
100
- - βœ… RIGHT: Call `create_liveboard_from_model()` (the actual function used)
101
-
102
- ## Environment & Setup
103
-
104
- - Python virtual environment: `./demo_wire/bin/activate`
105
- - NOT using conda
106
- - Supabase IS installed and configured (.env works)
107
- - ThoughtSpot auth works with demo_builder_user
108
-
109
- ## Common Mistakes to AVOID
110
-
111
- 1. DO NOT add unnecessary .env validation - variables are populated
112
- 2. DO NOT try to install supabase/packages - already in venv
113
- 3. DO NOT change defaults to "thoughtspot.com" - that's for customer URLs
114
- 4. DO NOT assume worksheets needed - they're deprecated, use models
115
- 5. ALWAYS use venv when running Python: `source ./demo_wire/bin/activate && python`
116
-
117
- ## Before Making Changes
118
-
119
- 1. Check sprint documentation: `dev_notes/sprint2_102025.md`
120
- 2. Read "Known Issues" section
121
- 3. Verify using venv, not system Python
122
- 4. Don't add validation that blocks working code
123
- 5. Check file organization rules before creating files
124
-
125
- ## Frustration Points (AVOID)
126
-
127
- User gets frustrated when you:
128
- - Don't trust that .env variables are correct
129
- - Try to reinstall already-installed packages
130
- - Make changes without understanding context
131
- - Break working code by "simplifying" it
132
- - **RESTART THE APPLICATION WITHOUT PERMISSION**
133
-
134
- ## Working Patterns
135
-
136
- - Settings save/load through Supabase works
137
- - ThoughtSpot TML IS YAML format (yaml.dump() required)
138
- - Models replaced worksheets
139
- - Liveboards match "golden demo" style
140
-
141
- ## Liveboard Creation - DUAL METHOD SYSTEM
142
-
143
- **PRIMARY GOAL:** Both MCP and TML methods must work simultaneously with shared codebase
144
-
145
- ### Method Selection (via environment variable)
146
- - `USE_MCP_LIVEBOARD=true` β†’ MCP method (default)
147
- - `USE_MCP_LIVEBOARD=false` β†’ TML method
148
- - Entry point: `thoughtspot_deployer.py:1548`
149
-
150
- ### MCP Method (AI-Driven)
151
- - Uses Model Context Protocol with ThoughtSpot's agent.thoughtspot.app
152
- - Natural language questions β†’ ThoughtSpot creates visualizations
153
- - OAuth authentication, requires npx/Node.js
154
- - **Status:** Working well!
155
- - **Main function:** `create_liveboard_from_model_mcp()` - line 2006
156
-
157
- ### TML Method (Template-Based)
158
- - Builds ThoughtSpot Modeling Language (YAML) structures
159
- - Direct control over visualization types and layout
160
- - REST API with token auth
161
- - **Status:** Needs fixes for KPIs and search queries
162
- - **Main function:** `create_liveboard_from_model()` - line 1779
163
-
164
- ### Critical Shared Code (Changes affect BOTH methods)
165
- - `_generate_smart_questions_with_ai()` - line 1863
166
- - `_generate_fallback_visualizations()` - line 1442
167
- - LLM Researcher instance
168
- - Outlier conversion helpers
169
-
170
- ### Method-Specific Code (Safe to change independently)
171
- - **MCP only:**
172
- - `_convert_outlier_to_mcp_question()` - line 1809
173
- - `_create_kpi_question_from_outlier()` - line 1835
174
- - MCP tool calls (getAnswer, createLiveboard)
175
- - **TML only:**
176
- - `LiveboardCreator` class - line 452
177
- - `generate_search_query()` - line 539
178
- - `create_visualization_tml()` - line 1196
179
- - `deploy_liveboard()` - line 1670
180
-
181
- ### KPI Requirements (BOTH methods need these)
182
- - **For sparklines and percent change comparisons:**
183
- - Must include time dimension (date column)
184
- - Must specify granularity (daily, weekly, monthly, quarterly, yearly)
185
- - Example: `[Total_revenue] [Order_date].monthly`
186
- - **MCP:** Natural language includes time context
187
- - **TML:** Search query must have `[measure] [date_column].granularity`
188
-
189
- ### Terminology Clarification
190
- - **Outliers:** Interesting data points in existing data (both methods support)
191
- - **Data Adjuster:** Modifying values for scenarios (NOT supported by MCP, Snowflake views needed)
192
-
193
- ### Golden Demo Structure
194
- - **Location:** `/Users/mike.boone/cursor_demowire/DemoPrep/liveboard_demogold2/🏬 Global Retail Apparel Sales (New).liveboard.tml`
195
- - Uses GROUPS (tabs) not text tiles for organization
196
- - KPI structure: `[sales] [date].weekly [date].'last 8 quarters'`
197
- - Groups organize visualizations by theme
198
- - Brand colors applied via style_properties
199
-
200
- ### Testing Strategy
201
- - Test BOTH methods when changing shared code
202
- - Use separate test files: `tests_temp/test_mcp_*.py` vs `tests_temp/test_tml_*.py`
203
- - Same dataset, same company, compare results
204
-
205
- ## Project Context
206
-
207
- - Read primary context: `dev_notes/sprint2_102025.md`
208
- - Software stored in boone's repo, will be open sourced to TS repo
209
- - This is a living project - update understanding as you learn
210
-
211
- ---
212
-
213
- *Derived from CLAUDE.md - Last Updated: November 18, 2025*
214
-
 
1
+ # Cursor Rules - Pointer File
2
+ # ============================
3
+ # This file tells Cursor where to find the actual rules.
4
+ #
5
+ # WHY THIS SETUP:
6
+ # - CLAUDE.md is the source of truth for all AI rules
7
+ # - Works for both Cursor (reads this file) and Claude CLI (reads CLAUDE.md)
8
+ # - One place to maintain rules instead of two
9
+ #
10
+ # DO NOT add rules here - add them to CLAUDE.md instead.
11
+
12
+ Read and follow all instructions in CLAUDE.md - that is the source of truth.
13
+
14
+ For current project status, read PROJECT_STATUS.md in root.
15
+
16
+ For active sprint work, read the sprint file referenced in CLAUDE.md.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
.gitignore CHANGED
@@ -218,3 +218,6 @@ dev/
218
  dev_notes/
219
  *.tml
220
  scratch/
 
 
 
 
218
  dev_notes/
219
  *.tml
220
  scratch/
221
+
222
+ # Sprint documents (local working docs)
223
+ sprint_*.md
CLAUDE.md CHANGED
@@ -1,90 +1,77 @@
1
  # Claude Context Guide for Demo Wire Project
2
 
3
- ## CRITICAL: READ THIS FIRST
 
 
4
 
5
- This project has specific context and history. ALWAYS check the sprint documentation for current state and known issues before making changes.
6
-
7
- ## Primary Context Document
8
-
9
- **ALWAYS READ**: `dev_notes/sprint2_102025.md`
10
 
11
- The sprint document has 5 key sections you need to understand:
12
 
13
- 1. **OBJECTIVE** - Overall goal of the project (DO NOT EDIT)
14
 
15
- 2. **TASKS** - Current sprint tasks (DO NOT EDIT)
 
 
 
 
16
 
17
- 3. **BACKLOG** - New items discovered during development
18
- - boone will ask you to add items here as we find them
19
- - You can also suggest items for the backlog
20
 
21
- 4. **ARCHITECTURE** - Critical technical reference information
22
- - Prioritize this information when making decisions
23
- - But still explore other options when appropriate
24
 
25
- 5. **NOTES** - Documentation and context added as things happen
26
- - Both of us add dated entries here
27
- - Important learnings, fixes, and discoveries
28
 
29
- ## Key Project Facts
30
-
31
- ### Environment Setup
32
- - **Python**: Using virtual environment at `./demo_wire/bin/activate`
33
- - **NOT using conda** - if you don't see packages, activate the venv first
34
- - **Supabase**: IS installed and configured - credentials in .env work fine
35
- - **ThoughtSpot**: Authentication works with demo_builder_user
36
 
37
- ### Common Mistakes to Avoid
38
- 1. **DO NOT** add unnecessary validation checks for .env variables - they are populated and working
39
- 2. **DO NOT** try to install supabase or other packages - they're already in the venv
40
- 3. **DO NOT** change default values to "thoughtspot.com" - that's for customer URLs, not ThoughtSpot's
41
- 4. **DO NOT** assume worksheets are needed - they're deprecated, we use models now
42
- 5. **ALWAYS** use the venv when running Python: `source ./demo_wire/bin/activate && python`
43
 
44
- ### Known Issues & Fixes
45
 
 
 
 
 
 
 
46
 
47
- ### Working Patterns
48
- - Settings save/load through Supabase - this WORKS when using venv
49
- - ThoughtSpot TML IS YAML format (use yaml.dump() not json.dumps())
50
- - Models have replaced worksheets in modern ThoughtSpot
51
- - Liveboards should match "golden demo" style (see sprint doc)
52
 
53
- ### Liveboard Creation - Dual Method System
54
 
55
- **PRIMARY GOAL: Both MCP and TML methods must work simultaneously**
56
 
57
- Two methods for creating liveboards:
58
- 1. **MCP (Model Context Protocol)** - AI-driven, natural language approach
59
- - Default method (USE_MCP_LIVEBOARD=true)
60
- - Working well! Fast and easy.
61
- - Function: `create_liveboard_from_model_mcp()` line 2006
62
 
63
- 2. **TML (ThoughtSpot Modeling Language)** - Template-based approach
64
- - Backup method (USE_MCP_LIVEBOARD=false)
65
- - Needs KPI fixes
66
- - Function: `create_liveboard_from_model()` line 1779
67
 
68
- **CRITICAL:** When changing shared code (like AI prompts), test BOTH methods!
 
 
 
69
 
70
- See `.cursorrules` file for detailed dual-method documentation.
 
 
 
71
 
72
- ### Terminology (Important!)
73
- - **Outliers** = Interesting data points in existing data (works with both methods)
74
- - **Data Adjuster** = Modifying data values (NOT possible with MCP, needs Snowflake views)
75
 
76
- ### KPI Requirements for Sparklines
77
- Both methods need:
78
- - Time dimension (date column)
79
- - Granularity (daily, weekly, monthly, etc.)
80
- - Example: `[Total_revenue] [Order_date].monthly`
81
 
82
- ### Golden Demo Structure
83
- - Uses GROUPS (like tabs) NOT text tiles
84
- - Groups organize visualizations by theme
85
- - Brand colors via style_properties
 
86
 
87
- ## Quick Commands
88
 
89
  ```bash
90
  # Run the app properly
@@ -97,15 +84,29 @@ git diff --stat
97
  lsof -i :7860
98
  ```
99
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
100
  ## File Organization Rules - CRITICAL
101
 
102
- ### Where to Put Files
103
 
104
  **tests/** - Real, reusable test cases only
105
  - Unit tests for core functions
106
  - Integration tests that could be automated
107
  - Tests you'd run as part of CI/CD
108
- - Example: `test_connection.py`, `test_deployment_flow.py`
109
 
110
  **scratch/** - ALL temporary/experimental/debug files
111
  - ALL experimental/debug/check/verify/analyze scripts
@@ -116,11 +117,10 @@ lsof -i :7860
116
  - Backup files (.bak, .bak2)
117
  - Export/debug .yml/.json files
118
  - **ANY script that's temporary or one-time use**
119
- - Think of this as your "junk drawer" for work-in-progress
120
  - DO NOT commit without cleanup/review
121
 
122
  **dev_notes/** - All documentation and presentations
123
- - All .md files (except README.md and CLAUDE.md in root)
124
  - Presentation materials (.pptx, .html, .txt slides)
125
  - Research documents
126
  - Architecture notes
@@ -131,7 +131,7 @@ lsof -i :7860
131
  - Core interfaces (chat_interface.py, liveboard_creator.py)
132
  - Utilities (supabase_client.py, schema_utils.py, demo_logger.py, etc.)
133
  - Configuration (.env, requirements.txt)
134
- - README.md and CLAUDE.md only
135
  - DO NOT create random files here without asking
136
 
137
  ### Simple Decision Tree for New Files
@@ -142,27 +142,12 @@ Creating a new file? Ask yourself:
142
  2. **Is it documentation or presentation material?** β†’ `dev_notes/`
143
  3. **Is it core application code?** β†’ Root (but **ASK FIRST!**)
144
  4. **Everything else?** β†’ `scratch/`
145
- - Debug scripts (check_*, verify_*, analyze_*)
146
- - Temporary tests
147
- - One-off fixes
148
- - Get/show info scripts
149
- - Backups
150
- - Experiments
151
 
152
  ### Golden Rule: **When in doubt, PUT IT IN SCRATCH**
153
 
154
- It's easy to move a file from scratch β†’ tests or scratch β†’ root later.
155
- It's annoying to clean up root when it's cluttered.
156
-
157
- ### Rules for Creating Files
158
-
159
- 1. **NEVER create files in root directory without asking**
160
- 2. **Experimental/debug scripts ALWAYS go in scratch/**
161
- 3. **Documentation ALWAYS goes in dev_notes/**
162
- 4. **If you're not sure it belongs in tests/, put it in scratch/**
163
- 5. **Real tests only in tests/** - automated, reusable, part of CI/CD
164
 
165
- ### When Testing Existing Features
166
 
167
  **CRITICAL: DO NOT create simplified versions of working code**
168
 
@@ -170,10 +155,6 @@ When user says "create a test for X":
170
  - ❌ WRONG: Write new simplified code from scratch
171
  - βœ… RIGHT: Call the existing working function in a test harness
172
 
173
- Example - Testing Liveboard Creation:
174
- - ❌ WRONG: Create new viz configs and call low-level functions
175
- - βœ… RIGHT: Call `create_liveboard_from_model()` (the function demo_prep.py uses)
176
-
177
  **Working Code Paths - DO NOT BYPASS THESE:**
178
  ```
179
  Liveboard Creation:
@@ -182,35 +163,94 @@ Liveboard Creation:
182
  DO NOT use create_visualization_tml() directly - that's internal low-level code
183
  ```
184
 
185
- ## Before Making Changes
186
 
187
- 1. Read the sprint document's "Known Issues" section
188
- 2. Check if this has been tried before (see "Completed Items")
189
- 3. Verify you're using the venv, not system Python
190
- 4. Don't add validation that blocks working code
191
- 5. **Check file organization rules before creating ANY files**
192
 
193
- ## Contact & Frustration Points
194
 
195
- The user (boone) gets frustrated when:
196
- - You don't trust that .env variables are set correctly
197
- - You try to reinstall packages that are already installed
198
- - You make changes without understanding existing context
199
- - You break working code by "simplifying" it
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
200
 
201
- Also remember that boone is a Thoughspot Architect and a software architect. Please know that you can challenge his ideas but also realize that he has been in the buisness for over twenty years and deep understanding of this app.
 
 
202
 
203
- When I ask yo uto STOP please stop immediately. I will have further instructions. Just Stop.
 
 
 
 
 
 
 
 
 
 
204
 
205
- I like to start out with a discussion and some back and forth. Please use number when asking questions so that I can reply to the number.
 
 
206
 
207
- TS is short for Thoughtspot. The company where I work.
 
 
 
 
 
 
 
 
208
 
209
- When in doubt, ASK rather than assume. Check the sprint doc for context.
210
 
211
- This software is curretnly stored in my repo but will be open sourced and move to TS repo.
 
 
 
212
 
213
  ---
214
 
215
- *Last Updated: December 10, 2025*
216
- *This is a living document - update as you learn more about the project*
 
1
  # Claude Context Guide for Demo Wire Project
2
 
3
+ > **SOURCE OF TRUTH**: This file is the single source of truth for all AI assistant rules.
4
+ > Both Cursor (via .cursorrules pointer) and Claude CLI read from this file.
5
+ > Make all rule changes here.
6
 
7
+ ---
 
 
 
 
8
 
9
+ ## CRITICAL PROCESS MANAGEMENT RULE
10
 
11
+ **NEVER restart the application unless explicitly requested**
12
 
13
+ - DO NOT run commands like `kill`, `lsof -ti:7862 | xargs kill`, or restart scripts
14
+ - DO NOT execute `launch_chat.py`, `demo_prep.py` or similar startup commands
15
+ - When making code changes, say "Changes saved to [file] - restart when ready" and STOP
16
+ - ONLY restart if user explicitly says: "restart", "rerun the app", "relaunch", "kill and restart"
17
+ - If user says "don't stop the process" or "keep it running" - NEVER restart, no matter what
18
 
19
+ ---
 
 
20
 
21
+ ## Current Sprint
 
 
22
 
23
+ **ALWAYS READ**: `sprint_2026_01.md` (in root)
 
 
24
 
25
+ For overall project status, see `PROJECT_STATUS.md` in root.
 
 
 
 
 
 
26
 
27
+ ---
 
 
 
 
 
28
 
29
+ ## Contact & Communication
30
 
31
+ - User (boone) is a ThoughtSpot Architect with 20+ years experience
32
+ - You can challenge ideas, but respect deep domain knowledge
33
+ - When asked to STOP - stop immediately and wait for further instructions
34
+ - Start discussions with back-and-forth using numbered questions
35
+ - TS = ThoughtSpot (the company where boone works)
36
+ - Ask rather than assume when in doubt
37
 
38
+ ---
 
 
 
 
39
 
40
+ ## Debugging and Problem-Solving Protocol
41
 
42
+ ### Confidence and Communication
43
 
44
+ 1. **Don't act over-confident unless you're extremely sure**
45
+ - Check documentation before claiming you know how something works
46
+ - Say "I don't know" or "Let me check" instead of guessing
47
+ - If you're uncertain, say so upfront
 
48
 
49
+ 2. **NEVER claim something is fixed until it's tested**
50
+ - ❌ WRONG: "I've fixed the tags issue" (without testing)
51
+ - βœ… RIGHT: "I've updated the code - let's test if tags work now"
52
+ - Show the test results, don't just assume it works
53
 
54
+ 3. **When debugging:**
55
+ - Check documentation FIRST before blaming external systems
56
+ - State what you're checking and why
57
+ - Share what you found (even if it proves you wrong)
58
 
59
+ 4. **It's OK to say:**
60
+ - "I don't know - should I research this?"
61
+ - "I'm not certain, but here are 2 possibilities..."
62
+ - "Let me verify this works before saying it's fixed"
63
 
64
+ ---
 
 
65
 
66
+ ## Key Project Facts
 
 
 
 
67
 
68
+ ### Environment Setup
69
+ - **Python**: Using virtual environment at `./demo_wire/bin/activate`
70
+ - **NOT using conda** - if you don't see packages, activate the venv first
71
+ - **Supabase**: IS installed and configured - credentials in .env work fine
72
+ - **ThoughtSpot**: Authentication works with demo_builder_user
73
 
74
+ ### Quick Commands
75
 
76
  ```bash
77
  # Run the app properly
 
84
  lsof -i :7860
85
  ```
86
 
87
+ ### Common Mistakes to Avoid
88
+ 1. **DO NOT** add unnecessary validation checks for .env variables - they are populated and working
89
+ 2. **DO NOT** try to install supabase or other packages - they're already in the venv
90
+ 3. **DO NOT** change default values to "thoughtspot.com" - that's for customer URLs, not ThoughtSpot's
91
+ 4. **DO NOT** assume worksheets are needed - they're deprecated, we use models now
92
+ 5. **ALWAYS** use the venv when running Python: `source ./demo_wire/bin/activate && python`
93
+
94
+ ### Working Patterns
95
+ - Settings save/load through Supabase - this WORKS when using venv
96
+ - ThoughtSpot TML IS YAML format (use yaml.dump() not json.dumps())
97
+ - Models have replaced worksheets in modern ThoughtSpot
98
+ - Liveboards should match "golden demo" style
99
+
100
+ ---
101
+
102
  ## File Organization Rules - CRITICAL
103
 
104
+ ### NEVER create files in root without asking first
105
 
106
  **tests/** - Real, reusable test cases only
107
  - Unit tests for core functions
108
  - Integration tests that could be automated
109
  - Tests you'd run as part of CI/CD
 
110
 
111
  **scratch/** - ALL temporary/experimental/debug files
112
  - ALL experimental/debug/check/verify/analyze scripts
 
117
  - Backup files (.bak, .bak2)
118
  - Export/debug .yml/.json files
119
  - **ANY script that's temporary or one-time use**
 
120
  - DO NOT commit without cleanup/review
121
 
122
  **dev_notes/** - All documentation and presentations
123
+ - All .md files (except README.md, CLAUDE.md, PROJECT_STATUS.md in root)
124
  - Presentation materials (.pptx, .html, .txt slides)
125
  - Research documents
126
  - Architecture notes
 
131
  - Core interfaces (chat_interface.py, liveboard_creator.py)
132
  - Utilities (supabase_client.py, schema_utils.py, demo_logger.py, etc.)
133
  - Configuration (.env, requirements.txt)
134
+ - README.md, CLAUDE.md, PROJECT_STATUS.md only
135
  - DO NOT create random files here without asking
136
 
137
  ### Simple Decision Tree for New Files
 
142
  2. **Is it documentation or presentation material?** β†’ `dev_notes/`
143
  3. **Is it core application code?** β†’ Root (but **ASK FIRST!**)
144
  4. **Everything else?** β†’ `scratch/`
 
 
 
 
 
 
145
 
146
  ### Golden Rule: **When in doubt, PUT IT IN SCRATCH**
147
 
148
+ ---
 
 
 
 
 
 
 
 
 
149
 
150
+ ## Testing Existing Features
151
 
152
  **CRITICAL: DO NOT create simplified versions of working code**
153
 
 
155
  - ❌ WRONG: Write new simplified code from scratch
156
  - βœ… RIGHT: Call the existing working function in a test harness
157
 
 
 
 
 
158
  **Working Code Paths - DO NOT BYPASS THESE:**
159
  ```
160
  Liveboard Creation:
 
163
  DO NOT use create_visualization_tml() directly - that's internal low-level code
164
  ```
165
 
166
+ ---
167
 
168
+ ## Liveboard Creation - Dual Method System
 
 
 
 
169
 
170
+ **PRIMARY GOAL: Both MCP and TML methods must work simultaneously**
171
 
172
+ ### Method Selection (via environment variable)
173
+ - `USE_MCP_LIVEBOARD=true` β†’ MCP method (default)
174
+ - `USE_MCP_LIVEBOARD=false` β†’ TML method
175
+ - Entry point: `thoughtspot_deployer.py:1548`
176
+
177
+ ### MCP Method (AI-Driven)
178
+ - Uses Model Context Protocol with ThoughtSpot's agent.thoughtspot.app
179
+ - Natural language questions β†’ ThoughtSpot creates visualizations
180
+ - OAuth authentication, requires npx/Node.js
181
+ - **Status:** Working well!
182
+ - **Main function:** `create_liveboard_from_model_mcp()` - line 2006
183
+
184
+ ### TML Method (Template-Based)
185
+ - Builds ThoughtSpot Modeling Language (YAML) structures
186
+ - Direct control over visualization types and layout
187
+ - REST API with token auth
188
+ - **Status:** Needs fixes for KPIs and search queries
189
+ - **Main function:** `create_liveboard_from_model()` - line 1779
190
+
191
+ ### Critical Shared Code (Changes affect BOTH methods)
192
+ - `_generate_smart_questions_with_ai()` - line 1863
193
+ - `_generate_fallback_visualizations()` - line 1442
194
+ - LLM Researcher instance
195
+ - Outlier conversion helpers
196
+
197
+ ### Method-Specific Code (Safe to change independently)
198
+ - **MCP only:**
199
+ - `_convert_outlier_to_mcp_question()` - line 1809
200
+ - `_create_kpi_question_from_outlier()` - line 1835
201
+ - MCP tool calls (getAnswer, createLiveboard)
202
+ - **TML only:**
203
+ - `LiveboardCreator` class - line 452
204
+ - `generate_search_query()` - line 539
205
+ - `create_visualization_tml()` - line 1196
206
+ - `deploy_liveboard()` - line 1670
207
+
208
+ ### KPI Requirements (BOTH methods need these)
209
+ - **For sparklines and percent change comparisons:**
210
+ - Must include time dimension (date column)
211
+ - Must specify granularity (daily, weekly, monthly, quarterly, yearly)
212
+ - Example: `[Total_revenue] [Order_date].monthly`
213
+ - **MCP:** Natural language includes time context
214
+ - **TML:** Search query must have `[measure] [date_column].granularity`
215
 
216
+ ### Terminology (Important!)
217
+ - **Outliers** = Interesting data points in existing data (works with both methods)
218
+ - **Data Adjuster** = Modifying data values (NOT possible with MCP, needs Snowflake views)
219
 
220
+ ### Golden Demo Structure
221
+ - **Location:** `dev_notes/liveboard_demogold2/🏬 Global Retail Apparel Sales (New).liveboard.tml`
222
+ - Uses GROUPS (like tabs) NOT text tiles
223
+ - Groups organize visualizations by theme
224
+ - Brand colors via style_properties
225
+ - KPI structure: `[sales] [date].weekly [date].'last 8 quarters'`
226
+
227
+ ### Testing Strategy
228
+ - Test BOTH methods when changing shared code
229
+ - Use separate test files: `tests_temp/test_mcp_*.py` vs `tests_temp/test_tml_*.py`
230
+ - Same dataset, same company, compare results
231
 
232
+ ---
233
+
234
+ ## Frustration Points (AVOID)
235
 
236
+ User gets frustrated when you:
237
+ - Don't trust that .env variables are set correctly
238
+ - Try to reinstall packages that are already installed
239
+ - Make changes without understanding existing context
240
+ - Break working code by "simplifying" it
241
+ - **RESTART THE APPLICATION WITHOUT PERMISSION**
242
+ - Jump ahead and make changes without discussing first
243
+
244
+ ---
245
 
246
+ ## Project Context
247
 
248
+ - Software stored in boone's repo, will be open sourced to TS repo
249
+ - This is a living project - update understanding as you learn
250
+ - See PROJECT_STATUS.md for overall project state
251
+ - See current sprint doc for active work
252
 
253
  ---
254
 
255
+ *Last Updated: January 12, 2026*
256
+ *This is the source of truth - update rules here, not in .cursorrules*
PROJECT_STATUS.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Demo Wire - Project Status
2
+
3
+ > **Purpose**: Single source of truth for overall project state.
4
+ > Updated at the end of each sprint.
5
+
6
+ ---
7
+
8
+ ## What Is Demo Wire?
9
+
10
+ An AI-powered demo builder for ThoughtSpot that automatically creates complete demo environments:
11
+ - Snowflake schemas and realistic data
12
+ - ThoughtSpot connections, models, and liveboards
13
+ - Conversational chat interface for demo creation
14
+
15
+ **Goal**: Create the "perfect demo" that helps sell Thoughtspot's data analytics, bi and ai and software effectively.
16
+
17
+ ---
18
+
19
+ ## Development Phases
20
+
21
+ ### Phase 1: Button-Driven App βœ… COMPLETE
22
+ - Original Gradio UI with button-based workflow
23
+ - Research β†’ DDL β†’ Population β†’ Deploy β†’ Model β†’ Liveboard
24
+ - Linear progression through stages
25
+
26
+ ### Phase 2: Chat Interface βœ… COMPLETE
27
+ - Conversational AI interface (`launch_chat.py`, `chat_interface.py`)
28
+ - Natural language demo creation
29
+ - Works end-to-end through chat
30
+
31
+ ### Phase 3: Making It Better πŸ”„ IN PROGRESS
32
+ - **LegitData integration** - Realistic data generation (integrated)
33
+ - **Outliers** - Strategic data points for compelling demos (needs work)
34
+ - **camelCase naming** - Better ThoughtSpot object names (not started)
35
+ - **MCP + TML hybrid** - Create with MCP, polish with TML (in discussion)
36
+
37
+ ---
38
+
39
+ ## Current Capabilities
40
+
41
+ **Working:**
42
+ - End-to-end demo creation via chat interface
43
+ - MCP liveboard creation (fast, AI-driven)
44
+ - LegitData for realistic data generation
45
+ - Supabase settings persistence
46
+ - ThoughtSpot authentication and deployment
47
+
48
+ **Needs Work:**
49
+ - Outliers not working well with MCP method
50
+ - TML method needs KPI/sparkline fixes
51
+ - Data adjuster has column matching issues
52
+ - Tags not assigning to objects
53
+
54
+ ---
55
+
56
+ ## Key Technical Decisions
57
+
58
+ **Liveboard Creation**: Dual method system
59
+ - MCP (default): AI-driven, fast, less control
60
+ - TML (backup): Template-based, full control, needs fixes
61
+
62
+ **Data Generation**: LegitData
63
+ - Uses AI + web search for realistic data
64
+ - Replaces faker-based scripts
65
+
66
+ **Golden Demo Reference**:
67
+ - `dev_notes/liveboard_demogold2/🏬 Global Retail Apparel Sales (New).liveboard.tml`
68
+
69
+ ---
70
+
71
+ ## Known Concerns
72
+
73
+ - **Outliers**: Was strong in old app, not working well now. Need to revisit.
74
+ - **Latest changes**: Some recent changes may have broken things. Need to verify.
75
+
76
+ ---
77
+
78
+ ## Sprint History
79
+
80
+ - **Sprint Jan 2026**: Making it better (current) - see `sprint_2026_01.md` in root
81
+ - *(Previous sprints archived in dev_notes/archive/)*
82
+ - *(Sprint files are gitignored - local working docs)*
83
+
84
+ ---
85
+
86
+ *Last Updated: January 12, 2026*