Spaces:
Running
Running
Reorganize docs: CLAUDE.md as source of truth, add PROJECT_STATUS.md, sprint structure
Browse files- .cursorrules +16 -214
- .gitignore +3 -0
- CLAUDE.md +149 -109
- PROJECT_STATUS.md +86 -0
.cursorrules
CHANGED
|
@@ -1,214 +1,16 @@
|
|
| 1 |
-
# Cursor Rules
|
| 2 |
-
|
| 3 |
-
#
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
-
|
| 8 |
-
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
- When asked to STOP - stop immediately and wait for further instructions
|
| 18 |
-
- Start discussions with back-and-forth using numbered questions
|
| 19 |
-
- TS = ThoughtSpot (the company where boone works)
|
| 20 |
-
- Ask rather than assume when in doubt
|
| 21 |
-
|
| 22 |
-
## Debugging and Problem-Solving Protocol
|
| 23 |
-
|
| 24 |
-
### Confidence and Communication
|
| 25 |
-
|
| 26 |
-
1. **Don't act over-confident unless you're extremely sure**
|
| 27 |
-
- Check documentation before claiming you know how something works
|
| 28 |
-
- Say "I don't know" or "Let me check" instead of guessing
|
| 29 |
-
- If you're uncertain, say so upfront
|
| 30 |
-
|
| 31 |
-
2. **NEVER claim something is fixed until it's tested**
|
| 32 |
-
- β WRONG: "I've fixed the tags issue" (without testing)
|
| 33 |
-
- β
RIGHT: "I've updated the code - let's test if tags work now"
|
| 34 |
-
- Show the test results, don't just assume it works
|
| 35 |
-
|
| 36 |
-
3. **When debugging:**
|
| 37 |
-
- Check documentation FIRST before blaming external systems
|
| 38 |
-
- State what you're checking and why
|
| 39 |
-
- Share what you found (even if it proves you wrong)
|
| 40 |
-
|
| 41 |
-
4. **It's OK to say:**
|
| 42 |
-
- "I don't know - should I research this?"
|
| 43 |
-
- "I'm not certain, but here are 2 possibilities..."
|
| 44 |
-
- "Let me verify this works before saying it's fixed"
|
| 45 |
-
|
| 46 |
-
## File Organization - CRITICAL
|
| 47 |
-
|
| 48 |
-
### NEVER create files in root without asking first
|
| 49 |
-
|
| 50 |
-
**tests/** - Real, reusable test cases only
|
| 51 |
-
- Unit tests for core functions
|
| 52 |
-
- Integration tests that could be automated
|
| 53 |
-
- Tests you'd run in CI/CD
|
| 54 |
-
|
| 55 |
-
**scratch/** - ALL temporary/experimental/debug files
|
| 56 |
-
- ALL experimental/debug/check/verify/analyze scripts
|
| 57 |
-
- One-off fixes (fix_*.py, adjust_*.py, emergency_*.py)
|
| 58 |
-
- Debug scripts (debug_*.py, check_*.py, verify_*.py)
|
| 59 |
-
- Analysis tools (analyze_*.py, get_*.py, show_*.py)
|
| 60 |
-
- Test files you're experimenting with
|
| 61 |
-
- Backup files (.bak, .bak2)
|
| 62 |
-
- Export/debug .yml/.json files
|
| 63 |
-
- **ANY script that's temporary or one-time use**
|
| 64 |
-
- DO NOT commit without cleanup
|
| 65 |
-
|
| 66 |
-
**dev_notes/** - All documentation
|
| 67 |
-
- All .md files except README.md and CLAUDE.md
|
| 68 |
-
- Presentation materials (.pptx, .html, .txt)
|
| 69 |
-
- Research documents
|
| 70 |
-
- Architecture notes
|
| 71 |
-
|
| 72 |
-
**Root directory** - ONLY essential files
|
| 73 |
-
- Main application files
|
| 74 |
-
- Core utilities
|
| 75 |
-
- Configuration (.env, requirements.txt)
|
| 76 |
-
- README.md and CLAUDE.md only
|
| 77 |
-
- DO NOT create random files here
|
| 78 |
-
|
| 79 |
-
### Simple Decision Tree
|
| 80 |
-
|
| 81 |
-
Creating a new file? Ask yourself:
|
| 82 |
-
|
| 83 |
-
1. **Is it a real test that should be automated?** β `tests/`
|
| 84 |
-
2. **Is it documentation/presentation?** β `dev_notes/`
|
| 85 |
-
3. **Is it core application code?** β Root (but ASK first!)
|
| 86 |
-
4. **Everything else?** β `scratch/` (debug, check, verify, analyze, fix, backup, experimental)
|
| 87 |
-
|
| 88 |
-
### When in doubt: PUT IT IN SCRATCH
|
| 89 |
-
|
| 90 |
-
## Testing Existing Features
|
| 91 |
-
|
| 92 |
-
**NEVER create simplified versions of working code**
|
| 93 |
-
|
| 94 |
-
When testing:
|
| 95 |
-
- β WRONG: Write new simplified code from scratch
|
| 96 |
-
- β
RIGHT: Call existing working functions in a test harness
|
| 97 |
-
|
| 98 |
-
Example:
|
| 99 |
-
- β WRONG: Create new viz configs and call low-level functions
|
| 100 |
-
- β
RIGHT: Call `create_liveboard_from_model()` (the actual function used)
|
| 101 |
-
|
| 102 |
-
## Environment & Setup
|
| 103 |
-
|
| 104 |
-
- Python virtual environment: `./demo_wire/bin/activate`
|
| 105 |
-
- NOT using conda
|
| 106 |
-
- Supabase IS installed and configured (.env works)
|
| 107 |
-
- ThoughtSpot auth works with demo_builder_user
|
| 108 |
-
|
| 109 |
-
## Common Mistakes to AVOID
|
| 110 |
-
|
| 111 |
-
1. DO NOT add unnecessary .env validation - variables are populated
|
| 112 |
-
2. DO NOT try to install supabase/packages - already in venv
|
| 113 |
-
3. DO NOT change defaults to "thoughtspot.com" - that's for customer URLs
|
| 114 |
-
4. DO NOT assume worksheets needed - they're deprecated, use models
|
| 115 |
-
5. ALWAYS use venv when running Python: `source ./demo_wire/bin/activate && python`
|
| 116 |
-
|
| 117 |
-
## Before Making Changes
|
| 118 |
-
|
| 119 |
-
1. Check sprint documentation: `dev_notes/sprint2_102025.md`
|
| 120 |
-
2. Read "Known Issues" section
|
| 121 |
-
3. Verify using venv, not system Python
|
| 122 |
-
4. Don't add validation that blocks working code
|
| 123 |
-
5. Check file organization rules before creating files
|
| 124 |
-
|
| 125 |
-
## Frustration Points (AVOID)
|
| 126 |
-
|
| 127 |
-
User gets frustrated when you:
|
| 128 |
-
- Don't trust that .env variables are correct
|
| 129 |
-
- Try to reinstall already-installed packages
|
| 130 |
-
- Make changes without understanding context
|
| 131 |
-
- Break working code by "simplifying" it
|
| 132 |
-
- **RESTART THE APPLICATION WITHOUT PERMISSION**
|
| 133 |
-
|
| 134 |
-
## Working Patterns
|
| 135 |
-
|
| 136 |
-
- Settings save/load through Supabase works
|
| 137 |
-
- ThoughtSpot TML IS YAML format (yaml.dump() required)
|
| 138 |
-
- Models replaced worksheets
|
| 139 |
-
- Liveboards match "golden demo" style
|
| 140 |
-
|
| 141 |
-
## Liveboard Creation - DUAL METHOD SYSTEM
|
| 142 |
-
|
| 143 |
-
**PRIMARY GOAL:** Both MCP and TML methods must work simultaneously with shared codebase
|
| 144 |
-
|
| 145 |
-
### Method Selection (via environment variable)
|
| 146 |
-
- `USE_MCP_LIVEBOARD=true` β MCP method (default)
|
| 147 |
-
- `USE_MCP_LIVEBOARD=false` β TML method
|
| 148 |
-
- Entry point: `thoughtspot_deployer.py:1548`
|
| 149 |
-
|
| 150 |
-
### MCP Method (AI-Driven)
|
| 151 |
-
- Uses Model Context Protocol with ThoughtSpot's agent.thoughtspot.app
|
| 152 |
-
- Natural language questions β ThoughtSpot creates visualizations
|
| 153 |
-
- OAuth authentication, requires npx/Node.js
|
| 154 |
-
- **Status:** Working well!
|
| 155 |
-
- **Main function:** `create_liveboard_from_model_mcp()` - line 2006
|
| 156 |
-
|
| 157 |
-
### TML Method (Template-Based)
|
| 158 |
-
- Builds ThoughtSpot Modeling Language (YAML) structures
|
| 159 |
-
- Direct control over visualization types and layout
|
| 160 |
-
- REST API with token auth
|
| 161 |
-
- **Status:** Needs fixes for KPIs and search queries
|
| 162 |
-
- **Main function:** `create_liveboard_from_model()` - line 1779
|
| 163 |
-
|
| 164 |
-
### Critical Shared Code (Changes affect BOTH methods)
|
| 165 |
-
- `_generate_smart_questions_with_ai()` - line 1863
|
| 166 |
-
- `_generate_fallback_visualizations()` - line 1442
|
| 167 |
-
- LLM Researcher instance
|
| 168 |
-
- Outlier conversion helpers
|
| 169 |
-
|
| 170 |
-
### Method-Specific Code (Safe to change independently)
|
| 171 |
-
- **MCP only:**
|
| 172 |
-
- `_convert_outlier_to_mcp_question()` - line 1809
|
| 173 |
-
- `_create_kpi_question_from_outlier()` - line 1835
|
| 174 |
-
- MCP tool calls (getAnswer, createLiveboard)
|
| 175 |
-
- **TML only:**
|
| 176 |
-
- `LiveboardCreator` class - line 452
|
| 177 |
-
- `generate_search_query()` - line 539
|
| 178 |
-
- `create_visualization_tml()` - line 1196
|
| 179 |
-
- `deploy_liveboard()` - line 1670
|
| 180 |
-
|
| 181 |
-
### KPI Requirements (BOTH methods need these)
|
| 182 |
-
- **For sparklines and percent change comparisons:**
|
| 183 |
-
- Must include time dimension (date column)
|
| 184 |
-
- Must specify granularity (daily, weekly, monthly, quarterly, yearly)
|
| 185 |
-
- Example: `[Total_revenue] [Order_date].monthly`
|
| 186 |
-
- **MCP:** Natural language includes time context
|
| 187 |
-
- **TML:** Search query must have `[measure] [date_column].granularity`
|
| 188 |
-
|
| 189 |
-
### Terminology Clarification
|
| 190 |
-
- **Outliers:** Interesting data points in existing data (both methods support)
|
| 191 |
-
- **Data Adjuster:** Modifying values for scenarios (NOT supported by MCP, Snowflake views needed)
|
| 192 |
-
|
| 193 |
-
### Golden Demo Structure
|
| 194 |
-
- **Location:** `/Users/mike.boone/cursor_demowire/DemoPrep/liveboard_demogold2/π¬ Global Retail Apparel Sales (New).liveboard.tml`
|
| 195 |
-
- Uses GROUPS (tabs) not text tiles for organization
|
| 196 |
-
- KPI structure: `[sales] [date].weekly [date].'last 8 quarters'`
|
| 197 |
-
- Groups organize visualizations by theme
|
| 198 |
-
- Brand colors applied via style_properties
|
| 199 |
-
|
| 200 |
-
### Testing Strategy
|
| 201 |
-
- Test BOTH methods when changing shared code
|
| 202 |
-
- Use separate test files: `tests_temp/test_mcp_*.py` vs `tests_temp/test_tml_*.py`
|
| 203 |
-
- Same dataset, same company, compare results
|
| 204 |
-
|
| 205 |
-
## Project Context
|
| 206 |
-
|
| 207 |
-
- Read primary context: `dev_notes/sprint2_102025.md`
|
| 208 |
-
- Software stored in boone's repo, will be open sourced to TS repo
|
| 209 |
-
- This is a living project - update understanding as you learn
|
| 210 |
-
|
| 211 |
-
---
|
| 212 |
-
|
| 213 |
-
*Derived from CLAUDE.md - Last Updated: November 18, 2025*
|
| 214 |
-
|
|
|
|
| 1 |
+
# Cursor Rules - Pointer File
|
| 2 |
+
# ============================
|
| 3 |
+
# This file tells Cursor where to find the actual rules.
|
| 4 |
+
#
|
| 5 |
+
# WHY THIS SETUP:
|
| 6 |
+
# - CLAUDE.md is the source of truth for all AI rules
|
| 7 |
+
# - Works for both Cursor (reads this file) and Claude CLI (reads CLAUDE.md)
|
| 8 |
+
# - One place to maintain rules instead of two
|
| 9 |
+
#
|
| 10 |
+
# DO NOT add rules here - add them to CLAUDE.md instead.
|
| 11 |
+
|
| 12 |
+
Read and follow all instructions in CLAUDE.md - that is the source of truth.
|
| 13 |
+
|
| 14 |
+
For current project status, read PROJECT_STATUS.md in root.
|
| 15 |
+
|
| 16 |
+
For active sprint work, read the sprint file referenced in CLAUDE.md.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.gitignore
CHANGED
|
@@ -218,3 +218,6 @@ dev/
|
|
| 218 |
dev_notes/
|
| 219 |
*.tml
|
| 220 |
scratch/
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
dev_notes/
|
| 219 |
*.tml
|
| 220 |
scratch/
|
| 221 |
+
|
| 222 |
+
# Sprint documents (local working docs)
|
| 223 |
+
sprint_*.md
|
CLAUDE.md
CHANGED
|
@@ -1,90 +1,77 @@
|
|
| 1 |
# Claude Context Guide for Demo Wire Project
|
| 2 |
|
| 3 |
-
|
|
|
|
|
|
|
| 4 |
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
## Primary Context Document
|
| 8 |
-
|
| 9 |
-
**ALWAYS READ**: `dev_notes/sprint2_102025.md`
|
| 10 |
|
| 11 |
-
|
| 12 |
|
| 13 |
-
|
| 14 |
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
|
| 17 |
-
|
| 18 |
-
- boone will ask you to add items here as we find them
|
| 19 |
-
- You can also suggest items for the backlog
|
| 20 |
|
| 21 |
-
|
| 22 |
-
- Prioritize this information when making decisions
|
| 23 |
-
- But still explore other options when appropriate
|
| 24 |
|
| 25 |
-
|
| 26 |
-
- Both of us add dated entries here
|
| 27 |
-
- Important learnings, fixes, and discoveries
|
| 28 |
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
### Environment Setup
|
| 32 |
-
- **Python**: Using virtual environment at `./demo_wire/bin/activate`
|
| 33 |
-
- **NOT using conda** - if you don't see packages, activate the venv first
|
| 34 |
-
- **Supabase**: IS installed and configured - credentials in .env work fine
|
| 35 |
-
- **ThoughtSpot**: Authentication works with demo_builder_user
|
| 36 |
|
| 37 |
-
|
| 38 |
-
1. **DO NOT** add unnecessary validation checks for .env variables - they are populated and working
|
| 39 |
-
2. **DO NOT** try to install supabase or other packages - they're already in the venv
|
| 40 |
-
3. **DO NOT** change default values to "thoughtspot.com" - that's for customer URLs, not ThoughtSpot's
|
| 41 |
-
4. **DO NOT** assume worksheets are needed - they're deprecated, we use models now
|
| 42 |
-
5. **ALWAYS** use the venv when running Python: `source ./demo_wire/bin/activate && python`
|
| 43 |
|
| 44 |
-
##
|
| 45 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
-
|
| 48 |
-
- Settings save/load through Supabase - this WORKS when using venv
|
| 49 |
-
- ThoughtSpot TML IS YAML format (use yaml.dump() not json.dumps())
|
| 50 |
-
- Models have replaced worksheets in modern ThoughtSpot
|
| 51 |
-
- Liveboards should match "golden demo" style (see sprint doc)
|
| 52 |
|
| 53 |
-
##
|
| 54 |
|
| 55 |
-
|
| 56 |
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
-
|
| 60 |
-
-
|
| 61 |
-
- Function: `create_liveboard_from_model_mcp()` line 2006
|
| 62 |
|
| 63 |
-
2. **
|
| 64 |
-
-
|
| 65 |
-
-
|
| 66 |
-
-
|
| 67 |
|
| 68 |
-
**
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
-
|
|
|
|
|
|
|
|
|
|
| 71 |
|
| 72 |
-
|
| 73 |
-
- **Outliers** = Interesting data points in existing data (works with both methods)
|
| 74 |
-
- **Data Adjuster** = Modifying data values (NOT possible with MCP, needs Snowflake views)
|
| 75 |
|
| 76 |
-
##
|
| 77 |
-
Both methods need:
|
| 78 |
-
- Time dimension (date column)
|
| 79 |
-
- Granularity (daily, weekly, monthly, etc.)
|
| 80 |
-
- Example: `[Total_revenue] [Order_date].monthly`
|
| 81 |
|
| 82 |
-
###
|
| 83 |
-
-
|
| 84 |
-
-
|
| 85 |
-
-
|
|
|
|
| 86 |
|
| 87 |
-
## Quick Commands
|
| 88 |
|
| 89 |
```bash
|
| 90 |
# Run the app properly
|
|
@@ -97,15 +84,29 @@ git diff --stat
|
|
| 97 |
lsof -i :7860
|
| 98 |
```
|
| 99 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
## File Organization Rules - CRITICAL
|
| 101 |
|
| 102 |
-
###
|
| 103 |
|
| 104 |
**tests/** - Real, reusable test cases only
|
| 105 |
- Unit tests for core functions
|
| 106 |
- Integration tests that could be automated
|
| 107 |
- Tests you'd run as part of CI/CD
|
| 108 |
-
- Example: `test_connection.py`, `test_deployment_flow.py`
|
| 109 |
|
| 110 |
**scratch/** - ALL temporary/experimental/debug files
|
| 111 |
- ALL experimental/debug/check/verify/analyze scripts
|
|
@@ -116,11 +117,10 @@ lsof -i :7860
|
|
| 116 |
- Backup files (.bak, .bak2)
|
| 117 |
- Export/debug .yml/.json files
|
| 118 |
- **ANY script that's temporary or one-time use**
|
| 119 |
-
- Think of this as your "junk drawer" for work-in-progress
|
| 120 |
- DO NOT commit without cleanup/review
|
| 121 |
|
| 122 |
**dev_notes/** - All documentation and presentations
|
| 123 |
-
- All .md files (except README.md
|
| 124 |
- Presentation materials (.pptx, .html, .txt slides)
|
| 125 |
- Research documents
|
| 126 |
- Architecture notes
|
|
@@ -131,7 +131,7 @@ lsof -i :7860
|
|
| 131 |
- Core interfaces (chat_interface.py, liveboard_creator.py)
|
| 132 |
- Utilities (supabase_client.py, schema_utils.py, demo_logger.py, etc.)
|
| 133 |
- Configuration (.env, requirements.txt)
|
| 134 |
-
- README.md
|
| 135 |
- DO NOT create random files here without asking
|
| 136 |
|
| 137 |
### Simple Decision Tree for New Files
|
|
@@ -142,27 +142,12 @@ Creating a new file? Ask yourself:
|
|
| 142 |
2. **Is it documentation or presentation material?** β `dev_notes/`
|
| 143 |
3. **Is it core application code?** β Root (but **ASK FIRST!**)
|
| 144 |
4. **Everything else?** β `scratch/`
|
| 145 |
-
- Debug scripts (check_*, verify_*, analyze_*)
|
| 146 |
-
- Temporary tests
|
| 147 |
-
- One-off fixes
|
| 148 |
-
- Get/show info scripts
|
| 149 |
-
- Backups
|
| 150 |
-
- Experiments
|
| 151 |
|
| 152 |
### Golden Rule: **When in doubt, PUT IT IN SCRATCH**
|
| 153 |
|
| 154 |
-
|
| 155 |
-
It's annoying to clean up root when it's cluttered.
|
| 156 |
-
|
| 157 |
-
### Rules for Creating Files
|
| 158 |
-
|
| 159 |
-
1. **NEVER create files in root directory without asking**
|
| 160 |
-
2. **Experimental/debug scripts ALWAYS go in scratch/**
|
| 161 |
-
3. **Documentation ALWAYS goes in dev_notes/**
|
| 162 |
-
4. **If you're not sure it belongs in tests/, put it in scratch/**
|
| 163 |
-
5. **Real tests only in tests/** - automated, reusable, part of CI/CD
|
| 164 |
|
| 165 |
-
##
|
| 166 |
|
| 167 |
**CRITICAL: DO NOT create simplified versions of working code**
|
| 168 |
|
|
@@ -170,10 +155,6 @@ When user says "create a test for X":
|
|
| 170 |
- β WRONG: Write new simplified code from scratch
|
| 171 |
- β
RIGHT: Call the existing working function in a test harness
|
| 172 |
|
| 173 |
-
Example - Testing Liveboard Creation:
|
| 174 |
-
- β WRONG: Create new viz configs and call low-level functions
|
| 175 |
-
- β
RIGHT: Call `create_liveboard_from_model()` (the function demo_prep.py uses)
|
| 176 |
-
|
| 177 |
**Working Code Paths - DO NOT BYPASS THESE:**
|
| 178 |
```
|
| 179 |
Liveboard Creation:
|
|
@@ -182,35 +163,94 @@ Liveboard Creation:
|
|
| 182 |
DO NOT use create_visualization_tml() directly - that's internal low-level code
|
| 183 |
```
|
| 184 |
|
| 185 |
-
|
| 186 |
|
| 187 |
-
|
| 188 |
-
2. Check if this has been tried before (see "Completed Items")
|
| 189 |
-
3. Verify you're using the venv, not system Python
|
| 190 |
-
4. Don't add validation that blocks working code
|
| 191 |
-
5. **Check file organization rules before creating ANY files**
|
| 192 |
|
| 193 |
-
|
| 194 |
|
| 195 |
-
|
| 196 |
-
-
|
| 197 |
-
-
|
| 198 |
-
-
|
| 199 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
|
| 201 |
-
|
|
|
|
|
|
|
| 202 |
|
| 203 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 204 |
|
| 205 |
-
|
|
|
|
|
|
|
| 206 |
|
| 207 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 208 |
|
| 209 |
-
|
| 210 |
|
| 211 |
-
|
|
|
|
|
|
|
|
|
|
| 212 |
|
| 213 |
---
|
| 214 |
|
| 215 |
-
*Last Updated:
|
| 216 |
-
*This is
|
|
|
|
| 1 |
# Claude Context Guide for Demo Wire Project
|
| 2 |
|
| 3 |
+
> **SOURCE OF TRUTH**: This file is the single source of truth for all AI assistant rules.
|
| 4 |
+
> Both Cursor (via .cursorrules pointer) and Claude CLI read from this file.
|
| 5 |
+
> Make all rule changes here.
|
| 6 |
|
| 7 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
+
## CRITICAL PROCESS MANAGEMENT RULE
|
| 10 |
|
| 11 |
+
**NEVER restart the application unless explicitly requested**
|
| 12 |
|
| 13 |
+
- DO NOT run commands like `kill`, `lsof -ti:7862 | xargs kill`, or restart scripts
|
| 14 |
+
- DO NOT execute `launch_chat.py`, `demo_prep.py` or similar startup commands
|
| 15 |
+
- When making code changes, say "Changes saved to [file] - restart when ready" and STOP
|
| 16 |
+
- ONLY restart if user explicitly says: "restart", "rerun the app", "relaunch", "kill and restart"
|
| 17 |
+
- If user says "don't stop the process" or "keep it running" - NEVER restart, no matter what
|
| 18 |
|
| 19 |
+
---
|
|
|
|
|
|
|
| 20 |
|
| 21 |
+
## Current Sprint
|
|
|
|
|
|
|
| 22 |
|
| 23 |
+
**ALWAYS READ**: `sprint_2026_01.md` (in root)
|
|
|
|
|
|
|
| 24 |
|
| 25 |
+
For overall project status, see `PROJECT_STATUS.md` in root.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
+
## Contact & Communication
|
| 30 |
|
| 31 |
+
- User (boone) is a ThoughtSpot Architect with 20+ years experience
|
| 32 |
+
- You can challenge ideas, but respect deep domain knowledge
|
| 33 |
+
- When asked to STOP - stop immediately and wait for further instructions
|
| 34 |
+
- Start discussions with back-and-forth using numbered questions
|
| 35 |
+
- TS = ThoughtSpot (the company where boone works)
|
| 36 |
+
- Ask rather than assume when in doubt
|
| 37 |
|
| 38 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
+
## Debugging and Problem-Solving Protocol
|
| 41 |
|
| 42 |
+
### Confidence and Communication
|
| 43 |
|
| 44 |
+
1. **Don't act over-confident unless you're extremely sure**
|
| 45 |
+
- Check documentation before claiming you know how something works
|
| 46 |
+
- Say "I don't know" or "Let me check" instead of guessing
|
| 47 |
+
- If you're uncertain, say so upfront
|
|
|
|
| 48 |
|
| 49 |
+
2. **NEVER claim something is fixed until it's tested**
|
| 50 |
+
- β WRONG: "I've fixed the tags issue" (without testing)
|
| 51 |
+
- β
RIGHT: "I've updated the code - let's test if tags work now"
|
| 52 |
+
- Show the test results, don't just assume it works
|
| 53 |
|
| 54 |
+
3. **When debugging:**
|
| 55 |
+
- Check documentation FIRST before blaming external systems
|
| 56 |
+
- State what you're checking and why
|
| 57 |
+
- Share what you found (even if it proves you wrong)
|
| 58 |
|
| 59 |
+
4. **It's OK to say:**
|
| 60 |
+
- "I don't know - should I research this?"
|
| 61 |
+
- "I'm not certain, but here are 2 possibilities..."
|
| 62 |
+
- "Let me verify this works before saying it's fixed"
|
| 63 |
|
| 64 |
+
---
|
|
|
|
|
|
|
| 65 |
|
| 66 |
+
## Key Project Facts
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
|
| 68 |
+
### Environment Setup
|
| 69 |
+
- **Python**: Using virtual environment at `./demo_wire/bin/activate`
|
| 70 |
+
- **NOT using conda** - if you don't see packages, activate the venv first
|
| 71 |
+
- **Supabase**: IS installed and configured - credentials in .env work fine
|
| 72 |
+
- **ThoughtSpot**: Authentication works with demo_builder_user
|
| 73 |
|
| 74 |
+
### Quick Commands
|
| 75 |
|
| 76 |
```bash
|
| 77 |
# Run the app properly
|
|
|
|
| 84 |
lsof -i :7860
|
| 85 |
```
|
| 86 |
|
| 87 |
+
### Common Mistakes to Avoid
|
| 88 |
+
1. **DO NOT** add unnecessary validation checks for .env variables - they are populated and working
|
| 89 |
+
2. **DO NOT** try to install supabase or other packages - they're already in the venv
|
| 90 |
+
3. **DO NOT** change default values to "thoughtspot.com" - that's for customer URLs, not ThoughtSpot's
|
| 91 |
+
4. **DO NOT** assume worksheets are needed - they're deprecated, we use models now
|
| 92 |
+
5. **ALWAYS** use the venv when running Python: `source ./demo_wire/bin/activate && python`
|
| 93 |
+
|
| 94 |
+
### Working Patterns
|
| 95 |
+
- Settings save/load through Supabase - this WORKS when using venv
|
| 96 |
+
- ThoughtSpot TML IS YAML format (use yaml.dump() not json.dumps())
|
| 97 |
+
- Models have replaced worksheets in modern ThoughtSpot
|
| 98 |
+
- Liveboards should match "golden demo" style
|
| 99 |
+
|
| 100 |
+
---
|
| 101 |
+
|
| 102 |
## File Organization Rules - CRITICAL
|
| 103 |
|
| 104 |
+
### NEVER create files in root without asking first
|
| 105 |
|
| 106 |
**tests/** - Real, reusable test cases only
|
| 107 |
- Unit tests for core functions
|
| 108 |
- Integration tests that could be automated
|
| 109 |
- Tests you'd run as part of CI/CD
|
|
|
|
| 110 |
|
| 111 |
**scratch/** - ALL temporary/experimental/debug files
|
| 112 |
- ALL experimental/debug/check/verify/analyze scripts
|
|
|
|
| 117 |
- Backup files (.bak, .bak2)
|
| 118 |
- Export/debug .yml/.json files
|
| 119 |
- **ANY script that's temporary or one-time use**
|
|
|
|
| 120 |
- DO NOT commit without cleanup/review
|
| 121 |
|
| 122 |
**dev_notes/** - All documentation and presentations
|
| 123 |
+
- All .md files (except README.md, CLAUDE.md, PROJECT_STATUS.md in root)
|
| 124 |
- Presentation materials (.pptx, .html, .txt slides)
|
| 125 |
- Research documents
|
| 126 |
- Architecture notes
|
|
|
|
| 131 |
- Core interfaces (chat_interface.py, liveboard_creator.py)
|
| 132 |
- Utilities (supabase_client.py, schema_utils.py, demo_logger.py, etc.)
|
| 133 |
- Configuration (.env, requirements.txt)
|
| 134 |
+
- README.md, CLAUDE.md, PROJECT_STATUS.md only
|
| 135 |
- DO NOT create random files here without asking
|
| 136 |
|
| 137 |
### Simple Decision Tree for New Files
|
|
|
|
| 142 |
2. **Is it documentation or presentation material?** β `dev_notes/`
|
| 143 |
3. **Is it core application code?** β Root (but **ASK FIRST!**)
|
| 144 |
4. **Everything else?** β `scratch/`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
|
| 146 |
### Golden Rule: **When in doubt, PUT IT IN SCRATCH**
|
| 147 |
|
| 148 |
+
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
|
| 150 |
+
## Testing Existing Features
|
| 151 |
|
| 152 |
**CRITICAL: DO NOT create simplified versions of working code**
|
| 153 |
|
|
|
|
| 155 |
- β WRONG: Write new simplified code from scratch
|
| 156 |
- β
RIGHT: Call the existing working function in a test harness
|
| 157 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 158 |
**Working Code Paths - DO NOT BYPASS THESE:**
|
| 159 |
```
|
| 160 |
Liveboard Creation:
|
|
|
|
| 163 |
DO NOT use create_visualization_tml() directly - that's internal low-level code
|
| 164 |
```
|
| 165 |
|
| 166 |
+
---
|
| 167 |
|
| 168 |
+
## Liveboard Creation - Dual Method System
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
|
| 170 |
+
**PRIMARY GOAL: Both MCP and TML methods must work simultaneously**
|
| 171 |
|
| 172 |
+
### Method Selection (via environment variable)
|
| 173 |
+
- `USE_MCP_LIVEBOARD=true` β MCP method (default)
|
| 174 |
+
- `USE_MCP_LIVEBOARD=false` β TML method
|
| 175 |
+
- Entry point: `thoughtspot_deployer.py:1548`
|
| 176 |
+
|
| 177 |
+
### MCP Method (AI-Driven)
|
| 178 |
+
- Uses Model Context Protocol with ThoughtSpot's agent.thoughtspot.app
|
| 179 |
+
- Natural language questions β ThoughtSpot creates visualizations
|
| 180 |
+
- OAuth authentication, requires npx/Node.js
|
| 181 |
+
- **Status:** Working well!
|
| 182 |
+
- **Main function:** `create_liveboard_from_model_mcp()` - line 2006
|
| 183 |
+
|
| 184 |
+
### TML Method (Template-Based)
|
| 185 |
+
- Builds ThoughtSpot Modeling Language (YAML) structures
|
| 186 |
+
- Direct control over visualization types and layout
|
| 187 |
+
- REST API with token auth
|
| 188 |
+
- **Status:** Needs fixes for KPIs and search queries
|
| 189 |
+
- **Main function:** `create_liveboard_from_model()` - line 1779
|
| 190 |
+
|
| 191 |
+
### Critical Shared Code (Changes affect BOTH methods)
|
| 192 |
+
- `_generate_smart_questions_with_ai()` - line 1863
|
| 193 |
+
- `_generate_fallback_visualizations()` - line 1442
|
| 194 |
+
- LLM Researcher instance
|
| 195 |
+
- Outlier conversion helpers
|
| 196 |
+
|
| 197 |
+
### Method-Specific Code (Safe to change independently)
|
| 198 |
+
- **MCP only:**
|
| 199 |
+
- `_convert_outlier_to_mcp_question()` - line 1809
|
| 200 |
+
- `_create_kpi_question_from_outlier()` - line 1835
|
| 201 |
+
- MCP tool calls (getAnswer, createLiveboard)
|
| 202 |
+
- **TML only:**
|
| 203 |
+
- `LiveboardCreator` class - line 452
|
| 204 |
+
- `generate_search_query()` - line 539
|
| 205 |
+
- `create_visualization_tml()` - line 1196
|
| 206 |
+
- `deploy_liveboard()` - line 1670
|
| 207 |
+
|
| 208 |
+
### KPI Requirements (BOTH methods need these)
|
| 209 |
+
- **For sparklines and percent change comparisons:**
|
| 210 |
+
- Must include time dimension (date column)
|
| 211 |
+
- Must specify granularity (daily, weekly, monthly, quarterly, yearly)
|
| 212 |
+
- Example: `[Total_revenue] [Order_date].monthly`
|
| 213 |
+
- **MCP:** Natural language includes time context
|
| 214 |
+
- **TML:** Search query must have `[measure] [date_column].granularity`
|
| 215 |
|
| 216 |
+
### Terminology (Important!)
|
| 217 |
+
- **Outliers** = Interesting data points in existing data (works with both methods)
|
| 218 |
+
- **Data Adjuster** = Modifying data values (NOT possible with MCP, needs Snowflake views)
|
| 219 |
|
| 220 |
+
### Golden Demo Structure
|
| 221 |
+
- **Location:** `dev_notes/liveboard_demogold2/π¬ Global Retail Apparel Sales (New).liveboard.tml`
|
| 222 |
+
- Uses GROUPS (like tabs) NOT text tiles
|
| 223 |
+
- Groups organize visualizations by theme
|
| 224 |
+
- Brand colors via style_properties
|
| 225 |
+
- KPI structure: `[sales] [date].weekly [date].'last 8 quarters'`
|
| 226 |
+
|
| 227 |
+
### Testing Strategy
|
| 228 |
+
- Test BOTH methods when changing shared code
|
| 229 |
+
- Use separate test files: `tests_temp/test_mcp_*.py` vs `tests_temp/test_tml_*.py`
|
| 230 |
+
- Same dataset, same company, compare results
|
| 231 |
|
| 232 |
+
---
|
| 233 |
+
|
| 234 |
+
## Frustration Points (AVOID)
|
| 235 |
|
| 236 |
+
User gets frustrated when you:
|
| 237 |
+
- Don't trust that .env variables are set correctly
|
| 238 |
+
- Try to reinstall packages that are already installed
|
| 239 |
+
- Make changes without understanding existing context
|
| 240 |
+
- Break working code by "simplifying" it
|
| 241 |
+
- **RESTART THE APPLICATION WITHOUT PERMISSION**
|
| 242 |
+
- Jump ahead and make changes without discussing first
|
| 243 |
+
|
| 244 |
+
---
|
| 245 |
|
| 246 |
+
## Project Context
|
| 247 |
|
| 248 |
+
- Software stored in boone's repo, will be open sourced to TS repo
|
| 249 |
+
- This is a living project - update understanding as you learn
|
| 250 |
+
- See PROJECT_STATUS.md for overall project state
|
| 251 |
+
- See current sprint doc for active work
|
| 252 |
|
| 253 |
---
|
| 254 |
|
| 255 |
+
*Last Updated: January 12, 2026*
|
| 256 |
+
*This is the source of truth - update rules here, not in .cursorrules*
|
PROJECT_STATUS.md
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Demo Wire - Project Status
|
| 2 |
+
|
| 3 |
+
> **Purpose**: Single source of truth for overall project state.
|
| 4 |
+
> Updated at the end of each sprint.
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
## What Is Demo Wire?
|
| 9 |
+
|
| 10 |
+
An AI-powered demo builder for ThoughtSpot that automatically creates complete demo environments:
|
| 11 |
+
- Snowflake schemas and realistic data
|
| 12 |
+
- ThoughtSpot connections, models, and liveboards
|
| 13 |
+
- Conversational chat interface for demo creation
|
| 14 |
+
|
| 15 |
+
**Goal**: Create the "perfect demo" that helps sell Thoughtspot's data analytics, bi and ai and software effectively.
|
| 16 |
+
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
## Development Phases
|
| 20 |
+
|
| 21 |
+
### Phase 1: Button-Driven App β
COMPLETE
|
| 22 |
+
- Original Gradio UI with button-based workflow
|
| 23 |
+
- Research β DDL β Population β Deploy β Model β Liveboard
|
| 24 |
+
- Linear progression through stages
|
| 25 |
+
|
| 26 |
+
### Phase 2: Chat Interface β
COMPLETE
|
| 27 |
+
- Conversational AI interface (`launch_chat.py`, `chat_interface.py`)
|
| 28 |
+
- Natural language demo creation
|
| 29 |
+
- Works end-to-end through chat
|
| 30 |
+
|
| 31 |
+
### Phase 3: Making It Better π IN PROGRESS
|
| 32 |
+
- **LegitData integration** - Realistic data generation (integrated)
|
| 33 |
+
- **Outliers** - Strategic data points for compelling demos (needs work)
|
| 34 |
+
- **camelCase naming** - Better ThoughtSpot object names (not started)
|
| 35 |
+
- **MCP + TML hybrid** - Create with MCP, polish with TML (in discussion)
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## Current Capabilities
|
| 40 |
+
|
| 41 |
+
**Working:**
|
| 42 |
+
- End-to-end demo creation via chat interface
|
| 43 |
+
- MCP liveboard creation (fast, AI-driven)
|
| 44 |
+
- LegitData for realistic data generation
|
| 45 |
+
- Supabase settings persistence
|
| 46 |
+
- ThoughtSpot authentication and deployment
|
| 47 |
+
|
| 48 |
+
**Needs Work:**
|
| 49 |
+
- Outliers not working well with MCP method
|
| 50 |
+
- TML method needs KPI/sparkline fixes
|
| 51 |
+
- Data adjuster has column matching issues
|
| 52 |
+
- Tags not assigning to objects
|
| 53 |
+
|
| 54 |
+
---
|
| 55 |
+
|
| 56 |
+
## Key Technical Decisions
|
| 57 |
+
|
| 58 |
+
**Liveboard Creation**: Dual method system
|
| 59 |
+
- MCP (default): AI-driven, fast, less control
|
| 60 |
+
- TML (backup): Template-based, full control, needs fixes
|
| 61 |
+
|
| 62 |
+
**Data Generation**: LegitData
|
| 63 |
+
- Uses AI + web search for realistic data
|
| 64 |
+
- Replaces faker-based scripts
|
| 65 |
+
|
| 66 |
+
**Golden Demo Reference**:
|
| 67 |
+
- `dev_notes/liveboard_demogold2/π¬ Global Retail Apparel Sales (New).liveboard.tml`
|
| 68 |
+
|
| 69 |
+
---
|
| 70 |
+
|
| 71 |
+
## Known Concerns
|
| 72 |
+
|
| 73 |
+
- **Outliers**: Was strong in old app, not working well now. Need to revisit.
|
| 74 |
+
- **Latest changes**: Some recent changes may have broken things. Need to verify.
|
| 75 |
+
|
| 76 |
+
---
|
| 77 |
+
|
| 78 |
+
## Sprint History
|
| 79 |
+
|
| 80 |
+
- **Sprint Jan 2026**: Making it better (current) - see `sprint_2026_01.md` in root
|
| 81 |
+
- *(Previous sprints archived in dev_notes/archive/)*
|
| 82 |
+
- *(Sprint files are gitignored - local working docs)*
|
| 83 |
+
|
| 84 |
+
---
|
| 85 |
+
|
| 86 |
+
*Last Updated: January 12, 2026*
|