app_requirements/6_feature_Streamlit.txt

Feature 6: Lightweight Streamlit MCP Monitor & Query Tester
6. Feature: Streamlit MCP Monitor & Query Tester
6.1 Story: As a developer, I need a lightweight Streamlit app to monitor MCP connections and test the agentic query engine.

6.1.1 Task: Create /streamlit/requirements.txt with streamlit==1.28.0, requests==2.31.0, pandas==2.1.0, python-dotenv==1.0.0 - NO neo4j driver as all Neo4j access MUST go through MCP server
6.1.2 Task: Create /streamlit/app.py that ONLY communicates with MCP server at http://mcp:8000/mcp, never directly to Neo4j - use st.set_page_config(page_title="MCP Monitor", layout="wide"), create two tabs using st.tabs(["🔌 Connection Status", "🤖 Query Tester"])
6.1.3 Task: Create /streamlit/Dockerfile with FROM python:3.11-slim, WORKDIR /app, install requirements, expose port 8501, CMD ["streamlit", "run", "app.py", "--server.address=0.0.0.0"]
6.1.4 Task: Add streamlit service to docker-compose.yml with build: ./streamlit, ports: 8501:8501, depends_on: mcp, environment: MCP_URL=http://mcp:8000/mcp, MCP_API_KEY=dev-key-123, explicitly NOT including NEO4J_BOLT_URL as direct access is forbidden

6.2 Story: As a user, I need to monitor the health and performance of all MCP connections in real-time.

6.2.1 Task: Create connection test function that calls MCP tools ONLY - test Neo4j via call_mcp("get_schema"), test PostgreSQL via call_mcp("query_postgres", {"query": "SELECT 1"}), never use direct database connections
6.2.2 Task: Display connection status in 3-column layout using st.columns(), show Neo4j (via MCP), PostgreSQL (via MCP), MCP Server status with st.metric(label, value="Online"/"Offline", delta=f"{response_ms}ms")
6.2.3 Task: Implement auto-refresh using st.empty() placeholder with while True loop, time.sleep(5), st.rerun() to update every 5 seconds, show "Last checked: {timestamp}" with st.caption()
6.2.4 Task: Add manual refresh button with st.button("🔄 Refresh Now") that immediately re-runs all MCP-based connection tests and updates metrics
6.2.5 Task: Query performance stats via MCP call_mcp("query_graph", {"query": "MATCH (l:Log) WHERE l.timestamp > datetime() - duration('PT1H') RETURN count(l) as count"}), display with st.info() - emphasize this is the ONLY way to query Neo4j

6.3 Story: As a user, I need to test natural language queries through the agentic engine without using the main chat interface.

6.3.1 Task: Create query input section with st.text_area("Enter your question:", height=100), st.button("🚀 Execute Query", type="primary") to trigger workflow creation via MCP server only
6.3.2 Task: On execute, create workflow via call_mcp("write_graph", {action: "create_node", label: "Workflow", properties: {...}}), then create instructions via MCP write_graph, store workflow_id in st.session_state - all graph writes MUST use MCP
6.3.3 Task: Poll workflow status via call_mcp("query_graph", {"query": "MATCH (i:Instruction) WHERE i.workflow_id=$id RETURN i.status", "parameters": {"id": workflow_id}}) every 2 seconds, update st.progress() based on results
6.3.4 Task: Fetch results via call_mcp("query_graph", {"query": "MATCH (e:Execution) WHERE e.workflow_id=$id RETURN e.result", "parameters": {"id": workflow_id}}), display SQL in st.code() and data in st.dataframe()
6.3.5 Task: Add "Clear Results" button that resets st.session_state.workflow_id and clears displayed results, ready for next query

6.4 Story: As a developer, I need to examine the agentic process flow to understand how answers are derived.

6.4.1 Task: Fetch execution trace via call_mcp("query_graph", {"query": "MATCH (w:Workflow {id: $id})-[:HAS_INSTRUCTION]->(i)-[:EXECUTED_AS]->(e) RETURN i, e ORDER BY i.sequence", "parameters": {"id": workflow_id}}) - this is the ONLY way to get execution data
6.4.2 Task: Display each step in expandable sections using st.expander(f"Step {i.sequence}: {i.type}"), show instruction parameters, execution times, and status from MCP query results
6.4.3 Task: For SQL generation steps, query schema context via call_mcp("query_graph", {"query": "MATCH (t:Table)-[:HAS_COLUMN]->(c:Column) RETURN t.name, collect(c.name)"}), display in st.code() to show what LLM received
6.4.4 Task: Show execution timeline by calculating time differences from execution nodes returned by MCP query_graph, display as: "Schema Discovery (5s) → [Pause 30s] → SQL Generation (3s) → Results (1s)"
6.4.5 Task: Add "View Raw Execution Data" toggle that shows full JSON response from call_mcp("query_graph", {"query": "MATCH (e:Execution {workflow_id: $id}) RETURN e"}), displayed with st.json()

6.5 Story: As a developer, I need the Streamlit app to handle errors gracefully and provide useful debugging information.

6.5.1 Task: Wrap all call_mcp() invocations in try/except blocks, on exception show st.error(f"MCP Server Error: {str(e)}") emphasizing no direct database access is possible
6.5.2 Task: Implement retry logic for failed MCP calls with 3 attempts and exponential backoff, show st.warning("Retrying MCP connection...") during retries, cache last successful response
6.5.3 Task: Add debug panel with st.expander("🔧 Debug Information") showing last 5 MCP requests/responses from st.session_state.debug_log, emphasize all database operations go through MCP
6.5.4 Task: On workflow execution failure, query error details via call_mcp("query_graph", {"query": "MATCH (e:Execution {status: 'failed'}) RETURN e.error"}), display with suggestions for common issues
6.5.5 Task: Create MCP diagnostics on startup - if call_mcp("get_schema") fails, show st.error("Cannot reach Neo4j through MCP server. The app cannot directly connect to Neo4j - all access must go through MCP at {MCP_URL}")


Critical Implementation Notes
The Streamlit app MUST:

NEVER import or use neo4j Python driver
NEVER import or use psycopg2 directly
ONLY communicate with databases through MCP server endpoints
ALWAYS use call_mcp() for any data retrieval or storage
EXPLICITLY show in error messages that direct database access is not permitted

Example of CORRECT implementation:
python# ✅ CORRECT - All Neo4j access through MCP
def get_workflow_status(workflow_id):
    result, _ = call_mcp("query_graph", {
        "query": "MATCH (w:Workflow {id: $id}) RETURN w.status",
        "parameters": {"id": workflow_id}
    })
    return result['data'][0]['status'] if result else None
Example of INCORRECT implementation:
python# ❌ WRONG - Direct Neo4j access is forbidden
from neo4j import GraphDatabase
driver = GraphDatabase.driver("bolt://neo4j:7687")  # NEVER DO THIS
This ensures the Streamlit app respects the architecture principle that all Neo4j access MUST go through the MCP server gateway, maintaining the single point of control and audit trail.