commit step 10

ops/scripts/acceptance_test.md

@@ -0,0 +1,145 @@
+# MVP Acceptance Test Checklist
+
+## Pre-requisites
+- [ ] Docker and Docker Compose installed
+- [ ] LLM API key configured in `.env`
+- [ ] All containers running (`docker-compose up -d`)
+
+## Test Scenarios
+
+### Scenario 1: Basic Query Flow
+1. [ ] Open http://localhost:3000
+2. [ ] Type: "Show all customers"
+3. [ ] Verify workflow created message appears
+4. [ ] Verify graph visualization updates
+5. [ ] Wait for results (may take 5+ minutes due to pauses)
+6. [ ] Verify table with customer data appears
+7. [ ] Verify SQL query is displayed
+
+### Scenario 2: Human Intervention
+1. [ ] Start new query: "Count all orders"
+2. [ ] During the 5-minute pause (watch agent logs)
+3. [ ] Open Neo4j Browser: http://localhost:7474
+4. [ ] Run query: `MATCH (i:Instruction {status: 'pending', type: 'generate_sql'}) RETURN i`
+5. [ ] Edit instruction: `SET i.parameters = '{"question": "Count orders with status completed"}'`
+6. [ ] Verify agent uses modified query after pause
+7. [ ] Verify results reflect the change
+
+### Scenario 3: Stop Workflow
+1. [ ] Start query: "Find expensive orders"
+2. [ ] Click STOP button during execution
+3. [ ] Verify workflow stops
+4. [ ] Check Neo4j: `MATCH (w:Workflow) RETURN w.status` shows 'stopped'
+
+### Scenario 4: Graph Visualization
+1. [ ] Start any query
+2. [ ] Verify workflow node appears (blue/gray)
+3. [ ] Verify instruction nodes appear
+4. [ ] Verify colors change:
+   - Gray = pending
+   - Yellow = executing  
+   - Green = complete
+5. [ ] Click on node shows properties
+
+### Scenario 5: Audit Trail
+1. [ ] After running queries, open Neo4j Browser
+2. [ ] Run: `MATCH (l:Log) RETURN l ORDER BY l.timestamp DESC LIMIT 10`
+3. [ ] Verify all MCP operations are logged
+4. [ ] Run: `MATCH (e:Execution) RETURN e`
+5. [ ] Verify all executions have results
+
+## Performance Checks
+- [ ] Agent processes instructions within 30 seconds of becoming available
+- [ ] Frontend responds within 1 second
+- [ ] Graph updates within 10 seconds
+- [ ] No memory leaks after 10+ queries
+
+## Data Validation
+Run these queries in Neo4j Browser:
+
+```cypher
+// Check schema discovered
+MATCH (t:Table) RETURN t.name
+
+// Check instructions executed
+MATCH (i:Instruction)-[:EXECUTED_AS]->(e:Execution)
+RETURN i.type, i.status, e.completed_at
+
+// Check successful SQL generation
+MATCH (qt:QueryTemplate)
+RETURN qt.query, qt.created_at
+ORDER BY qt.created_at DESC
+LIMIT 5
+
+// Check workflow completion rate
+MATCH (w:Workflow)
+RETURN w.status, count(w) as count
+```
+
+## Sign-off
+- [ ] All test scenarios pass
+- [ ] No errors in container logs
+- [ ] System recovers from agent restart
+- [ ] Clean startup from `docker-compose down && docker-compose up -d`
+
+**Tester:** _________________  
+**Date:** _________________  
+**Version:** MVP 1.0
+
+---
+
+## Final validation commands:
+
+```bash
+# Complete test sequence
+docker-compose down
+docker-compose up -d
+sleep 10
+
+# Check health
+curl -s http://localhost:8000/health && echo "MCP: OK"
+curl -s http://localhost:3000 > /dev/null && echo "Frontend: OK"
+
+# Seed data
+docker-compose exec mcp python /app/ops/scripts/seed.py
+
+# Run automated validation
+docker-compose exec mcp python /app/ops/scripts/validate.py
+
+# Run demo
+bash ops/scripts/demo.sh
+
+# In another terminal, watch all logs
+docker-compose logs -f
+
+# Manual test in another terminal
+curl -X POST http://localhost:8000/mcp \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: dev-key-123" \
+  -d '{
+    "tool": "write_graph",
+    "params": {
+      "action": "create_node",
+      "label": "Instruction",
+      "properties": {
+        "id": "final-test",
+        "type": "generate_sql",
+        "status": "pending",
+        "sequence": 999,
+        "pause_duration": 30,
+        "parameters": "{\"question\": \"What is the total revenue from all orders?\"}"
+      }
+    }
+  }'
+
+# Check execution after ~30 seconds
+curl -X POST http://localhost:8000/mcp \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: dev-key-123" \
+  -d '{
+    "tool": "query_graph",
+    "params": {
+      "query": "MATCH (i:Instruction {id: \"final-test\"})-[:EXECUTED_AS]->(e:Execution) RETURN e.result"
+    }
+  }'
+```

ops/scripts/demo.ps1

@@ -0,0 +1,108 @@
+# Graph-Driven Agent Demo Script for PowerShell
+# Run this with: powershell -ExecutionPolicy Bypass -File ops/scripts/demo.ps1
+
+Write-Host "🚀 Graph-Driven Agent Demo" -ForegroundColor Green
+Write-Host "=========================" -ForegroundColor Green
+Write-Host ""
+
+Write-Host "Step 1: Starting services..." -ForegroundColor Blue
+docker-compose up -d
+Start-Sleep 10
+
+Write-Host "Step 2: Checking health..." -ForegroundColor Blue
+
+Write-Host "Checking Neo4j..." -ForegroundColor Cyan
+docker-compose exec neo4j cypher-shell -u neo4j -p password "MATCH (n) RETURN count(n) LIMIT 1" 2>$null
+if ($LASTEXITCODE -eq 0) { 
+    Write-Host "✅ Neo4j: Healthy" -ForegroundColor Green 
+} else { 
+    Write-Host "❌ Neo4j: Unhealthy" -ForegroundColor Red 
+}
+
+Write-Host "Checking PostgreSQL..." -ForegroundColor Cyan
+docker-compose exec postgres pg_isready -U postgres 2>$null
+if ($LASTEXITCODE -eq 0) { 
+    Write-Host "✅ PostgreSQL: Healthy" -ForegroundColor Green 
+} else { 
+    Write-Host "❌ PostgreSQL: Unhealthy" -ForegroundColor Red 
+}
+
+Write-Host "Checking MCP Server..." -ForegroundColor Cyan
+try {
+    $response = Invoke-WebRequest -Uri "http://localhost:8000/health" -UseBasicParsing -TimeoutSec 5
+    if ($response.StatusCode -eq 200) { 
+        Write-Host "✅ MCP Server: Healthy" -ForegroundColor Green 
+    } else { 
+        Write-Host "❌ MCP Server: Unhealthy" -ForegroundColor Red 
+    }
+} catch {
+    Write-Host "❌ MCP Server: Unhealthy" -ForegroundColor Red
+}
+
+Write-Host "Checking Frontend..." -ForegroundColor Cyan
+try {
+    $response = Invoke-WebRequest -Uri "http://localhost:3000" -UseBasicParsing -TimeoutSec 5
+    if ($response.StatusCode -eq 200) { 
+        Write-Host "✅ Frontend: Healthy" -ForegroundColor Green 
+    } else { 
+        Write-Host "❌ Frontend: Unhealthy" -ForegroundColor Red 
+    }
+} catch {
+    Write-Host "❌ Frontend: Unhealthy" -ForegroundColor Red
+}
+
+Write-Host "Checking Agent..." -ForegroundColor Cyan
+$agentStatus = docker-compose ps agent | Select-String "Up"
+if ($agentStatus) { 
+    Write-Host "✅ Agent: Running" -ForegroundColor Green 
+} else { 
+    Write-Host "❌ Agent: Not running" -ForegroundColor Red 
+}
+
+Write-Host ""
+Write-Host "Step 3: Seeding demo workflow..." -ForegroundColor Blue
+docker-compose exec mcp python /app/ops/scripts/seed.py
+
+Write-Host ""
+Write-Host "✅ Demo environment ready!" -ForegroundColor Green
+Write-Host ""
+Write-Host "📝 Demo Instructions:" -ForegroundColor Yellow
+Write-Host "-------------------" -ForegroundColor Yellow
+Write-Host "1. Open http://localhost:3000 in your browser" -ForegroundColor White
+Write-Host ""
+Write-Host "2. In the chat, type: 'Show me customers who have ordered more than `$100'" -ForegroundColor White
+Write-Host ""
+Write-Host "3. You'll see:" -ForegroundColor White
+Write-Host "   - 'Workflow created!' message" -ForegroundColor Gray
+Write-Host "   - Graph visualization showing workflow nodes" -ForegroundColor Gray
+Write-Host "   - Status showing 'Agent thinking...'" -ForegroundColor Gray
+Write-Host ""
+Write-Host "4. The agent will:" -ForegroundColor White
+Write-Host "   - Pause for 5 minutes (or configured duration)" -ForegroundColor Gray
+Write-Host "   - Discover PostgreSQL schema" -ForegroundColor Gray
+Write-Host "   - Generate SQL query" -ForegroundColor Gray
+Write-Host "   - Execute and return results" -ForegroundColor Gray
+Write-Host ""
+Write-Host "5. During the pause, you can:" -ForegroundColor White
+Write-Host "   - Open Neo4j Browser: http://localhost:7474" -ForegroundColor Gray
+Write-Host "   - Login: neo4j/password" -ForegroundColor Gray
+Write-Host "   - Run: MATCH (i:Instruction {status: 'pending'}) RETURN i" -ForegroundColor Gray
+Write-Host "   - Edit parameters to change the query" -ForegroundColor Gray
+Write-Host ""
+Write-Host "6. After execution completes:" -ForegroundColor White
+Write-Host "   - Results appear in a table" -ForegroundColor Gray
+Write-Host "   - Generated SQL is shown" -ForegroundColor Gray
+Write-Host "   - Graph shows all nodes as green (complete)" -ForegroundColor Gray
+Write-Host ""
+Write-Host "Optional: Edit instruction during pause" -ForegroundColor Yellow
+Write-Host "docker-compose exec neo4j cypher-shell -u neo4j -p password \"
+Write-Host "  \"MATCH (i:Instruction {type: 'generate_sql', status: 'pending'}) \"
+Write-Host "   SET i.parameters = '{\"question\": \"Count all orders\"}'\""
+Write-Host ""
+Write-Host "🎉 DEMO IS RUNNING!" -ForegroundColor Green
+Write-Host "Press Ctrl+C to stop monitoring logs." -ForegroundColor Yellow
+Write-Host ""
+
+# Keep running and show agent logs
+Write-Host "Monitoring agent logs..." -ForegroundColor Blue
+docker-compose logs -f agent

ops/scripts/demo.sh

@@ -0,0 +1,74 @@
+#!/bin/bash
+
+echo "🚀 Graph-Driven Agent Demo"
+echo "========================="
+echo ""
+
+# Colors for output
+GREEN='\033[0;32m'
+BLUE='\033[0;34m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+echo -e "${BLUE}Step 1: Starting services...${NC}"
+docker-compose up -d
+sleep 10
+
+echo -e "${BLUE}Step 2: Checking health...${NC}"
+echo "Checking Neo4j..."
+docker-compose exec neo4j cypher-shell -u neo4j -p password "MATCH (n) RETURN count(n) LIMIT 1" > /dev/null 2>&1 && echo "✅ Neo4j: Healthy" || echo "❌ Neo4j: Unhealthy"
+
+echo "Checking PostgreSQL..."
+docker-compose exec postgres pg_isready -U postgres > /dev/null 2>&1 && echo "✅ PostgreSQL: Healthy" || echo "❌ PostgreSQL: Unhealthy"
+
+echo "Checking MCP Server..."
+curl -s http://localhost:8000/health > /dev/null && echo "✅ MCP Server: Healthy" || echo "❌ MCP Server: Unhealthy"
+
+echo "Checking Frontend..."
+curl -s http://localhost:3000 > /dev/null && echo "✅ Frontend: Healthy" || echo "❌ Frontend: Unhealthy"
+
+echo "Checking Agent..."
+docker-compose ps agent | grep -q "Up" && echo "✅ Agent: Running" || echo "❌ Agent: Not running"
+
+echo -e "${BLUE}Step 3: Seeding demo workflow...${NC}"
+docker-compose exec mcp python /app/ops/scripts/seed.py
+
+echo -e "${GREEN}✅ Demo environment ready!${NC}"
+echo ""
+echo "📝 Demo Script:"
+echo "---------------"
+echo "1. Open http://localhost:3000 in your browser"
+echo ""
+echo "2. In the chat, type: 'Show me customers who have ordered more than \$100'"
+echo ""
+echo "3. You'll see:"
+echo "   - 'Workflow created!' message"
+echo "   - Graph visualization showing workflow nodes"
+echo "   - Status showing 'Agent thinking...'"
+echo ""
+echo "4. The agent will:"
+echo "   - Pause for 5 minutes (or configured duration)"
+echo "   - Discover PostgreSQL schema"
+echo "   - Generate SQL query"
+echo "   - Execute and return results"
+echo ""
+echo "5. During the pause, you can:"
+echo "   - Open Neo4j Browser: http://localhost:7474"
+echo "   - Login: neo4j/password"
+echo "   - Run: MATCH (i:Instruction {status: 'pending'}) RETURN i"
+echo "   - Edit parameters to change the query"
+echo ""
+echo "6. After execution completes:"
+echo "   - Results appear in a table"
+echo "   - Generated SQL is shown"
+echo "   - Graph shows all nodes as green (complete)"
+echo ""
+echo -e "${YELLOW}Optional: Edit instruction during pause${NC}"
+echo "docker-compose exec neo4j cypher-shell -u neo4j -p password \\"
+echo "  \"MATCH (i:Instruction {type: 'generate_sql', status: 'pending'}) \\"
+echo "   SET i.parameters = '{\\\"question\\\": \\\"Count all orders\\\"}'\""
+echo ""
+echo -e "${GREEN}Demo is running! Press Ctrl+C to stop.${NC}"
+
+# Keep running
+docker-compose logs -f agent

ops/scripts/validate.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+"""Validate that the MVP system is working correctly"""
+
+import requests
+import time
+import json
+import sys
+
+MCP_URL = "http://localhost:8000/mcp"
+API_KEY = "dev-key-123"
+
+def call_mcp(tool, params=None):
+    response = requests.post(
+        MCP_URL,
+        headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
+        json={"tool": tool, "params": params or {}}
+    )
+    return response.json()
+
+def validate_step(step_name, validation_func):
+    """Run a validation step and report results"""
+    print(f"Validating: {step_name}...", end=" ")
+    try:
+        result = validation_func()
+        if result:
+            print("✅ PASSED")
+            return True
+        else:
+            print("❌ FAILED")
+            return False
+    except Exception as e:
+        print(f"❌ ERROR: {e}")
+        return False
+
+def check_neo4j():
+    """Check Neo4j is accessible via MCP"""
+    result = call_mcp("get_schema")
+    return "labels" in result
+
+def check_postgres():
+    """Check PostgreSQL has sample data"""
+    result = call_mcp("query_postgres", {"query": "SELECT COUNT(*) as count FROM customers"})
+    return result.get("data", [{}])[0].get("count", 0) > 0
+
+def check_schema_discovery():
+    """Check schema can be discovered"""
+    result = call_mcp("discover_postgres_schema")
+    schema = result.get("schema", {})
+    return "customers" in schema and "orders" in schema
+
+def create_test_workflow():
+    """Create a test workflow"""
+    # Create workflow
+    workflow = call_mcp("write_graph", {
+        "action": "create_node",
+        "label": "Workflow",
+        "properties": {
+            "id": "validation-workflow",
+            "name": "Validation Test",
+            "status": "active"
+        }
+    })
+    
+    # Create test instruction
+    instruction = call_mcp("write_graph", {
+        "action": "create_node",
+        "label": "Instruction",
+        "properties": {
+            "id": "validation-inst-1",
+            "type": "generate_sql",
+            "sequence": 1,
+            "status": "pending",
+            "pause_duration": 5,  # Short pause for testing
+            "parameters": json.dumps({"question": "Count all customers"})
+        }
+    })
+    
+    # Link instruction to workflow
+    call_mcp("query_graph", {
+        "query": """
+            MATCH (w:Workflow {id: 'validation-workflow'}), 
+                  (i:Instruction {id: 'validation-inst-1'})
+            CREATE (w)-[:HAS_INSTRUCTION]->(i)
+        """
+    })
+    
+    return workflow.get("created") is not None
+
+def check_instruction_execution():
+    """Check if instruction gets executed"""
+    # Wait for agent to pick it up (max 60 seconds)
+    for _ in range(12):
+        result = call_mcp("query_graph", {
+            "query": """
+                MATCH (i:Instruction {id: 'validation-inst-1'})
+                RETURN i.status as status
+            """
+        })
+        
+        status = result["data"][0]["status"] if result.get("data") else "unknown"
+        
+        if status in ["executing", "complete"]:
+            return True
+        
+        time.sleep(5)
+    
+    return False
+
+def check_execution_logged():
+    """Check if execution was logged"""
+    result = call_mcp("query_graph", {
+        "query": """
+            MATCH (i:Instruction {id: 'validation-inst-1'})-[:EXECUTED_AS]->(e:Execution)
+            RETURN e
+        """
+    })
+    
+    return len(result.get("data", [])) > 0
+
+def check_frontend():
+    """Check if frontend is accessible"""
+    try:
+        response = requests.get("http://localhost:3000")
+        return response.status_code == 200
+    except:
+        return False
+
+def main():
+    print("=" * 50)
+    print("MVP SYSTEM VALIDATION")
+    print("=" * 50)
+    print()
+    
+    # Track results
+    all_passed = True
+    
+    # Core services
+    all_passed &= validate_step("Neo4j accessible via MCP", check_neo4j)
+    all_passed &= validate_step("PostgreSQL has sample data", check_postgres)
+    all_passed &= validate_step("Frontend is accessible", check_frontend)
+    
+    # Functionality
+    all_passed &= validate_step("Schema discovery works", check_schema_discovery)
+    all_passed &= validate_step("Workflow creation works", create_test_workflow)
+    
+    print()
+    print("Waiting for agent to process instruction...")
+    all_passed &= validate_step("Instruction gets executed", check_instruction_execution)
+    all_passed &= validate_step("Execution is logged", check_execution_logged)
+    
+    # Final verification
+    print()
+    print("=" * 50)
+    if all_passed:
+        print("✅ ALL VALIDATIONS PASSED!")
+        print("The MVP system is working correctly.")
+        
+        # Show sample query result
+        result = call_mcp("query_graph", {
+            "query": """
+                MATCH (n)
+                RETURN labels(n)[0] as label, count(n) as count
+                ORDER BY count DESC
+            """
+        })
+        
+        print()
+        print("Graph Statistics:")
+        for item in result.get("data", []):
+            print(f"  - {item['label']}: {item['count']} nodes")
+            
+    else:
+        print("❌ SOME VALIDATIONS FAILED")
+        print("Please check the logs and fix issues.")
+        sys.exit(1)
+    
+    print("=" * 50)
+
+if __name__ == "__main__":
+    main()