Spaces:
Sleeping
Sleeping
| title: Braindump Sanctuary | |
| emoji: 🚀 | |
| colorFrom: purple | |
| colorTo: indigo | |
| sdk: docker | |
| app_port: 8501 | |
| tags: | |
| - streamlit | |
| pinned: false | |
| short_description: ✨ Transform scattered thoughts into insights. | |
| license: mit | |
| # 🧠 Brain Dump Sanctuary | |
| > **Capture, cluster, and reflect on your thoughts in real-time.** | |
| A full-stack NLP application that transforms scattered thoughts into organized, semantically meaningful clusters with AI-powered insights. Built with Streamlit, LangGraph agents, and Neo4j. | |
| --- | |
| ## ✨ Features | |
| ### Core Capabilities | |
| - **💭 Brain Dump Capture**: Write down thoughts, ideas, and questions instantly | |
| - **🔗 Semantic Clustering**: Automatically group related thoughts using embeddings (sentence-transformers) and HDBSCAN | |
| - **📊 Interactive Knowledge Graph**: Visualize all thoughts as an interactive Plotly graph with color-coded clusters | |
| - **🏷️ AI-Generated Labels**: Auto-label clusters using Google Gemini LLM (e.g., "Personal Growth", "Technical Concepts") | |
| - **🤖 Multi-Agent Intelligence**: | |
| - **SearchAgent**: Generate contextualized summaries with optional web search (Tavily) | |
| - **QuestionAgent**: Generate Socratic questions for deeper reflection | |
| - **GenerationAgent**: Synthesize insights across related thoughts | |
| - **FeedAgent**: Curate a blog-style feed of your 5 most recent thoughts with agent insights | |
| - **📝 Persistent Storage**: All thoughts stored in Neo4j graph database with full history | |
| - **🔄 Real-time Updates**: Refresh embeddings and recalculate clusters on demand | |
| ### Two-View Interface | |
| 1. **Home Tab** 🏠: Semantic cluster map + comprehensive table of all thoughts | |
| 2. **Feed Tab** 📰: Blog-style cards with summaries and reflection questions | |
| --- | |
| ## 🏗️ Architecture | |
| ### Tech Stack | |
| ``` | |
| Frontend: Streamlit (single-page Python web app) | |
| Backend: Python 3.10+ with LangChain/LangGraph | |
| Database: Neo4j Aura (graph database) | |
| NLP Pipeline: | |
| - Embeddings: sentence-transformers (all-MiniLM-L6-v2) | |
| - Clustering: HDBSCAN + UMAP dimensionality reduction | |
| - LLM: Google Gemini 2.5 Flash (via LangChain) | |
| Search: Tavily API (optional, with mock fallback) | |
| ``` | |
| ### System Architecture | |
| ``` | |
| ┌─────────────────────────────────────────────────────────┐ | |
| │ STREAMLIT UI │ | |
| │ (app.py - Home Tab | Feed Tab | Sidebar Controls) │ | |
| └─────────────────┬───────────────────────────────────────┘ | |
| │ | |
| ┌─────────┴─────────┐ | |
| │ │ | |
| ┌───▼────┐ ┌────▼──────┐ | |
| │ AGENTS │ │ EMBEDDINGS │ | |
| │ (Day2) │ │ & CLUSTERS │ | |
| └───┬────┘ │ (Day 1) │ | |
| │ └────┬───────┘ | |
| │ │ | |
| ┌───┴───────────────────▼────┐ | |
| │ BRAINDUMP_CORE.PY │ | |
| │ - BrainDumpDB (Neo4j) │ | |
| │ - EmbeddingEngine │ | |
| │ - ClusterEngine │ | |
| │ - ClusterLabelEngine │ | |
| └───┬────────────────────────┘ | |
| │ | |
| ┌───▼──────────────────┐ | |
| │ NEO4J GRAPH DB │ | |
| │ (Aura Cloud) │ | |
| │ - Dump Nodes │ | |
| │ - Cluster Nodes │ | |
| │ - Relationships │ | |
| └──────────────────────┘ | |
| ``` | |
| ### Data Flow | |
| **Input → Processing → Storage → Visualization** | |
| 1. **User enters brain dump** → `app.py` sidebar | |
| 2. **Generate embedding** → `EmbeddingEngine` (sentence-transformer) | |
| 3. **Store in Neo4j** → `BrainDumpDB.add_dump()` | |
| 4. **Re-cluster on demand** → `ClusterEngine.cluster()` + `ClusterLabelEngine.label_clusters()` | |
| 5. **Visualize clusters** → Plotly interactive graph (Home tab) | |
| 6. **Enrich with AI** → Agents analyze for Feed tab | |
| ### Neo4j Graph Schema | |
| ```cypher | |
| Node: Dump | |
| ├── id: UUID | |
| ├── text: String (the brain dump) | |
| ├── embedding: Vector[384] (from sentence-transformer) | |
| ├── created_at: DateTime | |
| └── cluster_id: Reference to Cluster | |
| Node: Cluster | |
| ├── id: UUID | |
| ├── label: String (e.g., "Personal Growth") | |
| ├── description: String (optional) | |
| ├── created_at: DateTime | |
| Relationships: | |
| ├── Dump -[:IN_CLUSTER]-> Cluster | |
| └── Dump -[:SIMILAR_TO {weight: 0.0-1.0}]-> Dump | |
| ``` | |
| ### Component Breakdown | |
| | File | Purpose | Key Classes | | |
| |------|---------|------------| | |
| | `app.py` | Streamlit UI & session management | Main app logic, page layout | | |
| | `braindump_core.py` | NLP pipeline & database | `BrainDumpDB`, `EmbeddingEngine`, `ClusterEngine`, `ClusterLabelEngine` | | |
| | `agents.py` | AI agents for insights | `QuestionAgent`, `SearchAgent`, `GenerationAgent`, `FeedAgent` | | |
| | `neo4j_maintenance.py` | Database utilities | Neo4j query helpers, debugging | | |
| | `cleanup_db.py` | Reset database | Wipe all data (for testing) | | |
| --- | |
| ## 🚀 Quick Start | |
| ### Prerequisites | |
| - Python 3.10+ | |
| - Neo4j Aura account (free tier available) | |
| - API keys for Google Gemini (free) and optionally Tavily | |
| ### 1. Setup Neo4j (Required) | |
| Create a free Neo4j Aura instance: | |
| 1. Go to https://console.neo4j.io/ | |
| 2. Sign up for a free account | |
| 3. Create a new "Free" Aura instance | |
| 4. Copy your connection details: | |
| - **URI**: `neo4j+ssc://xxxxx.databases.neo4j.io` | |
| - **Username**: `neo4j` | |
| - **Password**: (set during creation) | |
| ### 2. Installation | |
| ```bash | |
| # Clone repository | |
| git clone <repo-url> | |
| cd braindump-sanctuary | |
| # Install dependencies | |
| pip install -r requirements.txt | |
| # Create environment file | |
| cp .env.example .env # or create manually | |
| # Edit .env with your credentials | |
| ``` | |
| **.env Template:** | |
| ```env | |
| # Neo4j (Required) | |
| NEO4J_URI=neo4j+ssc://your-instance-id.databases.neo4j.io | |
| NEO4J_USER=neo4j | |
| NEO4J_PASSWORD=your-password-here | |
| # Google Gemini (Required for AI features) | |
| GOOGLE_API_KEY=your_google_api_key_here | |
| # Web Search (Optional - uses mock data if not provided) | |
| TAVILY_API_KEY=your_tavily_api_key_here | |
| # Alternative: Perplexity (Optional) | |
| PERPLEXITY_API_KEY=your_perplexity_key_here | |
| ``` | |
| **Get API Keys:** | |
| - 🔑 **Google Gemini** (free): https://aistudio.google.com/app/apikey | |
| - 🔍 **Tavily Search** (optional): https://app.tavily.com | |
| - 🧠 **Perplexity** (optional): https://www.perplexity.ai/ | |
| ### 3. Run the App | |
| ```bash | |
| streamlit run app.py | |
| ``` | |
| App opens at `http://localhost:8501` | |
| --- | |
| ## 📱 User Interface Guide | |
| ### Sidebar Controls | |
| **Left Sidebar** - Main interaction hub: | |
| - 🗺️ **Navigation**: Radio buttons to toggle between **Home** and **Feed** tabs | |
| - 📝 **Text Area**: Input field for new brain dumps | |
| - ✅ **Add to Sanctuary**: Save the thought to Neo4j | |
| - 🗑️ **Clear**: Empty the input field | |
| - 🔄 **Refresh Clusters**: Recalculate all embeddings and clusters (computationally intensive, 10-30 seconds) | |
| - ⚠️ **Clear All Dumps**: Permanently delete all thoughts from the database | |
| - 📊 **Total Brain Dumps**: Counter showing total stored thoughts | |
| ### Home Tab 🏠 - Knowledge Graph View | |
| **Top Section: Interactive Cluster Map** | |
| - **Visualization**: 2D interactive Plotly graph | |
| - **Nodes**: Each point = one brain dump | |
| - **Colors**: Each color = a semantic cluster (similar ideas grouped together) | |
| - **Layout**: UMAP dimensionality reduction for spatial meaning | |
| - **Interactions**: Hover to see full text, zoom/pan to explore | |
| - **Relationships**: Edges show semantic similarity between thoughts | |
| **Bottom Section: Comprehensive Table** | |
| - **Columns**: Brain Dump | Cluster Label | Created At | |
| - **Sorting**: Most recent first (newest at top) | |
| - **Format**: Plain text, easy to copy | |
| **Actions**: | |
| - Click "🔄 **Refresh Clusters**" to recalculate (when you add many new thoughts) | |
| - Spinner animations show progress of embedding generation and labeling | |
| ### Feed Tab 📰 - Blog-Style Feed | |
| **What You See**: | |
| 1. **Up to 5 Recent Cards** in reverse chronological order | |
| 2. **Each Card Contains**: | |
| - 💭 **Title**: Your full brain dump text | |
| - 🏷️ **Cluster Badge**: Category label (generated by LLM) | |
| - 📄 **Summary**: AI-generated context | |
| - If Tavily API set: Real web search + synthesis | |
| - If not: Mock contextual data (demo mode) | |
| - ❓ **Reflection Questions**: 5 Socratic questions (from QuestionAgent) | |
| **Why Use Feed?** | |
| - Quick morning review without table scrolling | |
| - AI-generated insights help you think deeper | |
| - Guided reflection via questions | |
| - Patterns become visible across multiple cards | |
| - Perfect for journaling workflow | |
| --- | |
| ## 🔄 Typical Workflows | |
| ### Workflow 1: Daily Brain Dumping + Review | |
| ``` | |
| Morning: | |
| 1. Open app at http://localhost:8501 | |
| 2. Sidebar: Type 3-5 quick thoughts | |
| 3. Click "Add to Sanctuary" each time | |
| 4. Switch to Feed tab | |
| 5. Read summaries and reflect on questions | |
| 6. (Takes 5-10 minutes) | |
| Evening: | |
| 1. Go to Home tab | |
| 2. Click "Refresh Clusters" | |
| 3. Observe how new thoughts clustered with existing ones | |
| 4. Spot emerging themes and connections | |
| ``` | |
| ### Workflow 2: Deep Dive Analysis | |
| ``` | |
| 1. Accumulate 20+ thoughts over several days | |
| 2. Go to Home tab | |
| 3. Click "Refresh Clusters" | |
| 4. Examine the visualization: | |
| - Which clusters are densest? | |
| - What themes emerge? | |
| - Any surprising connections? | |
| 5. Switch to Feed tab | |
| 6. Read the agent-generated insights | |
| 7. Use for next creative session | |
| ``` | |
| ### Workflow 3: Topic-Specific Exploration | |
| ``` | |
| 1. Add 10+ thoughts about a specific topic | |
| 2. They should cluster together automatically | |
| 3. Hover over the cluster in Home tab | |
| 4. Read the AI-generated cluster label | |
| 5. Check Feed tab for synthesis across related thoughts | |
| 6. Use the questions to drill deeper | |
| ``` | |
| --- | |
| ## 🤖 Agent Capabilities | |
| ### QuestionAgent | |
| **Role**: Socratic questioning for deeper reflection | |
| **Input**: Brain dump text | |
| **Output**: 5 open-ended questions | |
| **Example**: | |
| ``` | |
| Brain Dump: "Why do I procrastinate on important tasks?" | |
| Generated Questions: | |
| 1. What task are you procrastinating on right now? | |
| 2. What emotion arises when you think about starting it? | |
| 3. What would you do instead if you didn't procrastinate? | |
| 4. What is the smallest first step you could take? | |
| 5. What would happen if you started today? | |
| ``` | |
| ### SearchAgent | |
| **Role**: Web search + synthesis or mock data generation | |
| **Input**: Brain dump text | |
| **Output**: Relevant context or summary | |
| **With Tavily API**: | |
| - Searches the web for related information | |
| - Synthesizes findings with LLM | |
| - Provides citations and context | |
| **Without Tavily API** (Demo Mode): | |
| - Generates plausible contextual information | |
| - Maintains consistency with the thought | |
| - Allows testing without API key | |
| ### GenerationAgent | |
| **Role**: Cross-thought synthesis | |
| **Input**: Multiple related brain dumps (from same cluster) | |
| **Output**: Synthesized insight combining themes | |
| **Example**: | |
| ``` | |
| Related Dumps: | |
| - "Neural networks learn patterns" | |
| - "Our brains also learn from patterns" | |
| - "What if consciousness is pattern recognition?" | |
| Synthesis: These thoughts suggest that consciousness might emerge | |
| from the brain's pattern recognition capabilities... | |
| ``` | |
| ### FeedAgent | |
| **Role**: Orchestrates other agents for feed generation | |
| **Process**: | |
| 1. Selects 5 most recent thoughts | |
| 2. For each thought: | |
| - Gets cluster assignment | |
| - Calls SearchAgent for summary | |
| - Calls QuestionAgent for reflection questions | |
| 3. Formats as blog-style cards | |
| --- | |
| ## ⚙️ Configuration & Customization | |
| ### Environment Variables (.env) | |
| Create `.env` file in project root: | |
| ```bash | |
| # ============ REQUIRED ============ | |
| # Neo4j Aura Connection | |
| NEO4J_URI=neo4j+ssc://your-instance-id.databases.neo4j.io | |
| NEO4J_USER=neo4j | |
| NEO4J_PASSWORD=your-super-secure-password | |
| # Google Gemini LLM API | |
| GOOGLE_API_KEY=AIzaSyDxxxxxxxxxxxxxxxxxxxxx | |
| # ============ OPTIONAL ============ | |
| # Tavily Web Search (for better summaries) | |
| TAVILY_API_KEY=tvly-xxxxxxxxxxxxxxxxx | |
| # Perplexity API (alternative search) | |
| PERPLEXITY_API_KEY=pplx-xxxxxxxxxxxxxxxxx | |
| ``` | |
| ### Optional: Streamlit Configuration | |
| Create `.streamlit/config.toml`: | |
| ```toml | |
| [theme] | |
| primaryColor = "#4ECDC4" | |
| backgroundColor = "#0d1117" | |
| secondaryBackgroundColor = "#161b22" | |
| textColor = "#c9d1d9" | |
| [client] | |
| showErrorDetails = true | |
| [logger] | |
| level = "info" | |
| [server] | |
| port = 8501 | |
| ``` | |
| --- | |
| ## 🔧 Development & Debugging | |
| ### Running Locally | |
| ```bash | |
| streamlit run app.py | |
| ``` | |
| ### Clearing Database | |
| ```bash | |
| python cleanup_db.py | |
| ``` | |
| ### Neo4j Maintenance | |
| ```bash | |
| python neo4j_maintenance.py | |
| ``` | |
| ### Viewing Logs | |
| ```bash | |
| # Terminal shows Streamlit logs | |
| # Check sidebar for spinner status during refresh | |
| ``` | |
| ### Common Issues | |
| **Issue**: "Failed to connect to Neo4j" | |
| - Verify `.env` has correct URI, username, password | |
| - Check Neo4j Aura instance is running (console.neo4j.io) | |
| - Firewall: Neo4j needs outbound HTTPS (port 7687) | |
| **Issue**: "GOOGLE_API_KEY not found" | |
| - Ensure key is in `.env` file | |
| - Restart streamlit: `streamlit run app.py` | |
| **Issue**: Clusters not showing | |
| - Click "Refresh Clusters" in sidebar | |
| - Wait for spinner to finish (10-30 seconds) | |
| - Needs at least 2-3 thoughts for clustering | |
| **Issue**: Tavily search not working | |
| - Mock data is used if API key missing (expected) | |
| - Optional feature; not required for core functionality | |
| --- | |
| ## 📊 Performance Notes | |
| ### Computational Complexity | |
| - **Adding 1 thought**: ~1 second | |
| - **Clustering N thoughts**: ~O(N) to O(N log N) depending on N | |
| - 10 thoughts: ~2 seconds | |
| - 100 thoughts: ~5-10 seconds | |
| - 1000 thoughts: ~30-60 seconds | |
| - **LLM labeling**: ~2-5 seconds per cluster (depends on Gemini API latency) | |
| ### Storage | |
| - Neo4j Free Tier: ~5 GB storage | |
| - Typical thought: ~500 bytes | |
| - Can store ~10 million thoughts theoretically | |
| ### Scaling Recommendations | |
| - **Local testing**: ✅ Recommended | |
| - **Shared team use**: Consider dedicated Neo4j instance | |
| - **Large scale (10k+ thoughts)**: May need performance tuning (indexing, batching) | |
| --- | |
| ## 🎓 Learning Resources | |
| ### Concepts Explained | |
| **Semantic Embeddings** | |
| - sentence-transformers model converts text to 384-dimensional vectors | |
| - Similar texts → similar vectors → can cluster together | |
| - Distance in vector space ≈ semantic similarity | |
| **HDBSCAN Clustering** | |
| - Density-based clustering (unlike K-means which needs K) | |
| - Automatically finds clusters of any shape | |
| - Robust to outliers (marks them as "noise") | |
| **Neo4j Graph Database** | |
| - Stores relationships as first-class citizens | |
| - Fast for relationship queries (unlike SQL) | |
| - Perfect for knowledge graphs and recommendations | |
| **LLMs for Labeling** | |
| - Google Gemini generates cluster labels from examples | |
| - LangChain chains handle prompt + LLM + parsing | |
| - Enables semantic understanding of cluster themes | |
| --- | |
| ## 📝 License | |
| See `LICENSE` file for details. | |
| --- | |
| ## 🤝 Contributing | |
| Found a bug or have a feature idea? | |
| 1. Check existing issues | |
| 2. Create new issue with clear description | |
| 3. (Optional) Submit PR with fix | |
| --- | |
| ## ❓ FAQ | |
| **Q**: Can I use this with a local Neo4j instance? | |
| **A**: Yes! Update `NEO4J_URI` to `neo4j://localhost:7687` in `.env` | |
| **Q**: What if I don't have a Tavily API key? | |
| **A**: That's fine! SearchAgent will generate mock data (demo mode) | |
| **Q**: Can I export all my thoughts? | |
| **A**: Use the table in Home tab or query Neo4j directly. Export feature coming soon. | |
| **Q**: How often should I click "Refresh Clusters"? | |
| **A**: After adding multiple new thoughts (5+), or when you want to see updated organization. | |
| **Q**: Is my data secure? | |
| **A**: Only you have your Neo4j password. Data is encrypted in transit and at rest on Neo4j Aura. | |
| --- | |
| ## 🚀 Roadmap | |
| - [ ] Export functionality (CSV, JSON, markdown) | |
| - [ ] Thought search & filtering | |
| - [ ] Collaborative mode (multiple users) | |
| - [ ] Email digest of weekly summaries | |
| - [ ] Browser extension for quick capture | |
| - [ ] Mobile app | |
| - [ ] Advanced analytics dashboard | |
| [client] | |
| showErrorDetails = true | |
| ``` | |
| --- | |
| ## 🔧 Database Management | |
| ### Re-calculate Clusters | |
| 1. Go to Home tab | |
| 2. Click 🔄 **Refresh** button | |
| 3. Wait for spinner to complete | |
| ### Clear Everything and Start Fresh | |
| 1. Click 🗑️ **Clear All Dumps** in sidebar | |
| 2. Confirm action | |
| 3. Add new thoughts | |
| ### Neo4j Maintenance | |
| Use the `neo4j_maintenance.py` script: | |
| ```bash | |
| # Show database statistics | |
| python neo4j_maintenance.py stats | |
| # Remove duplicate brain dumps | |
| python neo4j_maintenance.py dedup | |
| # Clear all data (WARNING: Irreversible) | |
| python neo4j_maintenance.py clear | |
| ``` | |
| --- | |
| ## 📊 Performance Tips | |
| ### Clustering | |
| - **Fast**: 3-20 dumps (< 5 seconds) | |
| - **Moderate**: 20-100 dumps (5-30 seconds) | |
| - **Slow**: 100+ dumps (> 30 seconds) | |
| - Use caching to avoid recomputing | |
| ### API Usage | |
| - **Google Gemini**: Free tier allows ~60 requests/min | |
| - **Tavily Search**: Free tier allows ~100 searches/month | |
| - Set environment variables to enable features | |
| ### Neo4j Query Performance | |
| - Dump and cluster lookups are indexed for fast retrieval | |
| - Embedding vectors cached in graph nodes | |
| - Knowledge graph relationships enable fast similarity searches | |
| --- | |
| ## 🐛 Troubleshooting | |
| ### Neo4j Connection Error | |
| → Verify your Aura instance is running: https://console.neo4j.io/ | |
| → Check `NEO4J_URI`, `NEO4J_USER`, `NEO4J_PASSWORD` in `.env` | |
| → Ensure your IP is allowlisted in Aura instance settings | |
| ### "API Key not found" error | |
| → Make sure `.env` file exists with `GOOGLE_API_KEY` | |
| → Restart streamlit: `streamlit run app.py` | |
| ### Clustering takes forever | |
| → You might have 100+ dumps | |
| → Only happens on first run or "Refresh" | |
| → Consider batching cluster operations for large datasets | |
| ### Feed cards showing mock data | |
| → Your `TAVILY_API_KEY` isn't set | |
| → Add it to `.env` and restart | |
| ### "ModuleNotFoundError" for imports | |
| → Missing dependencies: `pip install -r requirements.txt` | |
| → Wrong Python version? Try `python3 -m pip install ...` | |
| --- | |
| ## 📚 Architecture Overview | |
| ``` | |
| braindump_core.py | |
| ├── BrainDumpDB (Neo4j Graph Database) | |
| ├── EmbeddingEngine (sentence-transformers) | |
| ├── ClusterEngine (HDBSCAN + UMAP) | |
| └── Visualization (Plotly) | |
| agents.py | |
| ├── QuestionAgent (Socratic questions) | |
| └── SearchAgent (Web search + synthesis) | |
| app.py | |
| ├── Sidebar (Input + Navigation) | |
| ├── Home Tab (Knowledge Graph + Table) | |
| ├── Feed Tab (Blog-style cards) | |
| └── Helper Functions (Rendering) | |
| neo4j_maintenance.py | |
| ├── Database statistics | |
| ├── Deduplication | |
| └── Data cleanup | |
| ``` | |
| --- | |
| ## 🎯 Next Steps | |
| 1. **Set up Neo4j Aura** instance at https://console.neo4j.io/ | |
| 2. **Add your first thought** via sidebar | |
| 3. **Switch between Home and Feed** to see different views | |
| 4. **Check back daily** to build your idea garden | |
| 5. **Monitor performance** using `neo4j_maintenance.py stats` | |
| --- | |
| ## 💡 Tips | |
| - **Best for**: Capturing fleeting thoughts, finding patterns, exploring ideas | |
| - **Start small**: 5-10 thoughts to see clustering in action | |
| - **Use specificity**: "Why do dreams fade?" works better than "dreams" | |
| - **Review regularly**: Feed tab is great for morning/evening review | |
| - **Share clusters**: Screenshot Home tab to share your thinking | |
| --- | |
| ## 📝 Notes | |
| - All data stored locally in `braindump.db` (no cloud sync) | |
| - Embeddings cached in database (fast retrieval) | |
| - Cluster labels generated once and reused | |
| - Each brain dump timestamped automatically | |
| --- | |
| **Ready to catch your thoughts before they slip away!** 🧠✨ | |
| For issues or questions, check REFACTOR_SUMMARY.md or ARCHITECTURE.md |