MVP requirements

app_requirements/1_feature_KG_backend.txt

@@ -1,29 +1,14 @@
-1. Feature: Neo4j Knowledge Graph Core
-1.1 Story: As a developer, I need a flexible Neo4j deployment that serves as the central nervous system for all data and metadata.
-
-1.1.1 Task: Create Dockerfile for Neo4j Community Edition with APOC plugins
-1.1.2 Task: Configure environment variables for deployment modes (Docker/Enterprise/Aura)
-1.1.3 Task: Set up persistent volumes for graph data and backups
-1.1.4 Task: Implement connection pooling and retry logic
-1.1.5 Task: Create migration scripts from Community → Enterprise → Aura
-1.1.6 Task: Configure Neo4j for vector similarity search support
-
-1.2 Story: As a system, I need a comprehensive graph schema that models workflows, source systems, and their relationships.
-
-1.2.1 Task: Create operational nodes: Workflow, Phase, Instruction, Execution, Checkpoint, HumanIntervention, MonitoringQA
-1.2.2 Task: Create metadata nodes: SourceSystem, Database, Schema, Table, Column, DataType
-1.2.3 Task: Create knowledge nodes: CrossReference, SchemaVersion, SchemaChange, DataQuality, QueryTemplate
-1.2.4 Task: Implement all relationships with cardinality constraints
-1.2.5 Task: Add vector embedding properties for similarity search
-1.2.6 Task: Create composite indexes for query performance
-
-1.3 Story: As a system, I need automatic schema introspection and documentation generation.
-
-1.3.1 Task: Build meta-queries that extract complete graph structure
-1.3.2 Task: Generate JSON Schema from Neo4j model for API contracts
-1.3.3 Task: Create GraphQL schema from Neo4j structure
-1.3.4 Task: Auto-generate API documentation with example queries
-1.3.5 Task: Implement schema versioning with migration tracking
-1.3.6 Task: Cache schema with intelligent invalidation
-
+1. Feature: Neo4j Knowledge Graph Core1.1 Story: As a developer, I need a Neo4j instance to store workflows and metadata.
+
+1.1.1 Task: Create Dockerfile for Neo4j Community Edition v5.x with APOC plugins enabled, exposing port 7474 (browser) and 7687 (bolt)
+1.1.2 Task: Configure persistent volume mount at /data for database files, ensuring data survives container restarts
+1.1.3 Task: Set default credentials via environment variables (NEO4J_AUTH=neo4j/password) and document in .env.example
+1.1.4 Task: Write health check script that attempts bolt connection and runs "MATCH (n) RETURN count(n) LIMIT 1" to verify database is responsive
+1.2 Story: As a system, I need the essential graph schema for agentic workflows.
+
+1.2.1 Task: Create bootstrap script that uses Neo4j MCP server to create nodes: Workflow (properties: id, name, status, max_iterations, current_iteration), Instruction (id, sequence, type, parameters, status, pause_duration), Execution (id, started_at, completed_at, result, error)
+1.2.2 Task: Via MCP server, create PostgreSQL metadata nodes: SourceSystem (id, name, connection_string_ref), Table (name, schema, row_count), Column (name, data_type, nullable, is_primary_key)
+1.2.3 Task: Via MCP server, create relationships: (Workflow)-[:HAS_INSTRUCTION]->(Instruction), (Instruction)-[:NEXT_INSTRUCTION]->(Instruction), (Instruction)-[:EXECUTED_AS]->(Execution), (Table)-[:HAS_COLUMN]->(Column)
+1.2.4 Task: Via MCP server, add status properties with constraints: status IN ['pending', 'executing', 'paused', 'complete', 'failed'], pause_duration integer (seconds), sequence integer (execution order)
+1.2.5 Task: Via MCP server, create composite index on (status, sequence) for efficient "next instruction" queries, and single indexes on Workflow.id and Instruction.id
 

app_requirements/2_feature_API_integration.txt

@@ -1,28 +1,12 @@
-2. Feature: Unified MCP Server Hub
-2.1 Story: As a system, I need a central MCP server that orchestrates all interactions between agents, Neo4j, and external sources.
-
-2.1.1 Task: Define core MCP tools: get_schema, query_graph, write_graph, run_workflow
-2.1.2 Task: Add orchestration tools: get_next_instruction, update_instruction, checkpoint_workflow
-2.1.3 Task: Add source tools: discover_sources, query_source, refresh_schema, get_lineage
-2.1.4 Task: Implement authentication layers (JWT internal, API key external)
-2.1.5 Task: Create permission matrix for tool access by caller type
-2.1.6 Task: Build request router that directs calls to appropriate handlers
-
-2.2 Story: As an external consumer, I need safe, governed access to the knowledge graph and connected sources.
-
-2.2.1 Task: Implement query sanitization and parameterization
-2.2.2 Task: Add query cost estimation and limits
-2.2.3 Task: Create result pagination for large datasets
-2.2.4 Task: Build response caching with smart invalidation
-2.2.5 Task: Implement field-level access controls
-2.2.6 Task: Generate audit trail for all external access
-
-2.3 Story: As a developer, I need comprehensive observability across all MCP operations.
-
-2.3.1 Task: Create MCP_Log nodes with full request/response capture
-2.3.2 Task: Link logs to workflows, sources, and users
-2.3.3 Task: Track metrics: latency, data volume, token usage, error rates
-2.3.4 Task: Build real-time monitoring dashboard
-2.3.5 Task: Implement alerting for anomalies and failures
-2.3.6 Task: Create performance optimization recommendations
-
+2. Feature: Neo4j MCP Server (Single Gateway)2.1 Story: As a system, I need an MCP server as the sole interface to Neo4j.
+
+2.1.1 Task: Implement get_schema tool that executes "CALL db.schema.visualization()" and returns JSON with node labels, relationship types, and property keys - no direct Cypher access allowed outside MCP
+2.1.2 Task: Implement query_graph tool that accepts parameterized Cypher (e.g., "MATCH (w:Workflow {id: $id})" with params: {id: "123"}), validates against injection, executes via bolt driver, and returns JSON results
+2.1.3 Task: Implement write_graph tool for CREATE/MERGE operations with transaction support, accepting structured input like {action: "create_node", label: "Instruction", properties: {...}} rather than raw Cypher
+2.1.4 Task: Implement get_next_instruction tool that internally runs "MATCH (i:Instruction) WHERE i.status = 'pending' RETURN i ORDER BY i.sequence LIMIT 1" and returns the instruction or null
+2.1.5 Task: Create API key authentication middleware that checks X-API-Key header against environment variable MCP_API_KEYS (comma-separated list) before allowing any tool execution
+2.2 Story: As an agent, I need to log all operations in Neo4j for auditability.
+
+2.2.1 Task: After each MCP operation, use write_graph to create Log node with properties: timestamp, operation_type, parameters, duration_ms, success, error_message
+2.2.2 Task: Create relationships (Log)-[:OPERATED_ON]->(Node) linking logs to affected Workflow/Instruction nodes using their IDs
+2.2.3 Task: Implement log retention by adding created_at timestamp and a cleanup tool that deletes logs older than 30 days (configurable)

app_requirements/3_feature_agentic_reasoning_loop.txt

@@ -1,38 +1,29 @@
-3. Feature: Intelligent Agent Orchestration Layer
-3.1 Story: As an agent, I need to operate entirely from graph-stored instructions for full auditability.
+3. Feature: Graph-Driven Agent Execution
+3.1 Story: As an agent, I must read all instructions from Neo4j nodes.
 
-3.1.1 Task: Implement instruction fetcher that queries Neo4j for next task
-3.1.2 Task: Load instruction context including parameters and dependencies
-3.1.3 Task: Check for human interventions that modify instructions
-3.1.4 Task: Update instruction status atomically with optimistic locking
-3.1.5 Task: Implement instruction timeout and retry logic
-3.1.6 Task: Validate workflow iteration limits before proceeding
+3.1.1 Task: Agent main loop calls MCP server's get_next_instruction tool every 30 seconds to fetch pending instructions - agent NEVER connects directly to Neo4j
+3.1.2 Task: Parse returned instruction node to extract type ('query_postgres', 'analyze_schema', 'generate_sql') and parameters JSON object
+3.1.3 Task: Before execution, call MCP write_graph to update instruction status to 'executing' with current timestamp
+3.1.4 Task: After execution, call MCP write_graph to create Execution node with result data and create EXECUTED_AS relationship to instruction
+3.1.5 Task: Call MCP write_graph to update instruction status to 'complete' or 'failed' based on execution outcome, including error details if failed
 
-3.2 Story: As an agent, I need to execute complex multi-phase workflows with continuous learning.
+3.2 Story: As an agent, I need to connect to PostgreSQL and store its schema in Neo4j.
 
-3.2.1 Task: Initialize workflows from templates or custom definitions
-3.2.2 Task: Generate requirement nodes by analyzing data sources
-3.2.3 Task: Create implementation plans based on available MCP tools
-3.2.4 Task: Execute code/queries and store results as Execution nodes
-3.2.5 Task: Run QA validations with configurable success criteria
-3.2.6 Task: Generate refinement instructions when QA fails
-3.2.7 Task: Update QueryTemplate nodes with successful patterns
-3.2.8 Task: Create checkpoints for workflow state recovery
+3.2.1 Task: Read PostgreSQL connection string from POSTGRES_CONNECTION environment variable (format: postgresql://user:pass@host:5432/dbname)
+3.2.2 Task: When instruction type is 'discover_schema', query PostgreSQL information_schema.tables and information_schema.columns to get full schema
+3.2.3 Task: For each discovered table, call MCP write_graph to create Table node, then for each column create Column node and HAS_COLUMN relationship
+3.2.4 Task: Generate 3 example SQL queries per table and store as Instruction nodes with type='query_template' linked to table via QUERIES relationship
 
-3.3 Story: As an operations team, I need human-in-the-loop controls for oversight and guidance.
+3.3 Story: As an operator, I need a 5-minute pause between instructions for human review.
 
-3.3.1 Task: Implement configurable pause points between phases
-3.3.2 Task: Create approval workflow for high-risk operations
-3.3.3 Task: Build real-time notification system for required approvals
-3.3.4 Task: Store all human edits as HumanIntervention nodes
-3.3.5 Task: Implement emergency stop with graceful state preservation
-3.3.6 Task: Add scheduled review points for long-running workflows
+3.3.1 Task: Read pause_duration from Instruction node (default 300 seconds) before starting execution, log "Pausing for X seconds for human review"
+3.3.2 Task: Implement interruptible sleep that checks every 10 seconds for a 'stop' flag in the Workflow node (allows emergency stop)
+3.3.3 Task: Before and after pause, call MCP write_graph to create Log nodes with pause_started_at and pause_ended_at timestamps
+3.3.4 Task: Document that operators can use Neo4j Browser during pause to modify instruction parameters via Cypher: "MATCH (i:Instruction {id: 'X'}) SET i.parameters = '{...}'"
 
-3.4 Story: As an agent, I need LLM integration for reasoning, embedding generation, and natural language processing.
+3.4 Story: As an agent, I need LLM access for natural language to SQL translation.
 
-3.4.1 Task: Create LLM abstraction layer supporting multiple providers
-3.4.2 Task: Implement secure credential management (vault/environment)
-3.4.3 Task: Generate and store embeddings for semantic search
-3.4.4 Task: Build similarity graph with SIMILAR_TO relationships
-3.4.5 Task: Track token usage and costs per workflow
-3.4.6 Task: Implement fallback strategies for LLM failures
+3.4.1 Task: Configure LLM_API_KEY and LLM_MODEL (gpt-4 or claude-3) via environment variables, validate on startup
+3.4.2 Task: When instruction type is 'generate_sql', fetch schema context via MCP query_graph, format as "Tables: [list], Question: [user question]"
+3.4.3 Task: Send prompt to LLM: "Given this PostgreSQL schema: [schema], generate SQL for: [question]. Return only valid SQL, no explanation."
+3.4.4 Task: Execute generated SQL against PostgreSQL, store both query and results in Execution node via MCP write_graph with execution_time_ms

app_requirements/4_feature_UI.txt

@@ -0,0 +1,23 @@
+4. Feature: Minimal Web Interface
+4.1 Story: As a user, I need a chat interface to interact with the system.
+
+4.1.1 Task: Create Next.js app with single page at /chat, using App Router, TypeScript, and Tailwind CSS for styling
+4.1.2 Task: Implement chat UI with message history stored in React state, input field with enter-to-send, and auto-scroll to bottom
+4.1.3 Task: On message submit, call MCP server via POST /api/mcp with {tool: "write_graph", params: {action: "create_workflow", user_question: "..."}}, which triggers agent
+4.1.4 Task: Show pulsing "Agent thinking..." indicator by polling MCP get_next_instruction every 2 seconds while workflow is active
+4.1.5 Task: When execution completes, fetch results via MCP query_graph and display in HTML table with column headers and zebra striping
+
+4.2 Story: As a user, I need to see what the agent is doing.
+
+4.2.1 Task: Add status panel that polls MCP query_graph every 5 seconds for active workflow and current instruction, displaying name and status
+4.2.2 Task: During pause, show countdown timer (setInterval every second) with "Human review window: 4:32 remaining" and orange background
+4.2.3 Task: Query and display last 5 instructions via MCP: "MATCH (i:Instruction)-[:EXECUTED_AS]->(e:Execution) RETURN i, e ORDER BY e.completed_at DESC LIMIT 5"
+4.2.4 Task: Implement red STOP button that calls MCP write_graph to set workflow status='stopped', which agent checks during pause loop
+
+4.3 Story: As a user, I need basic graph visualization.
+
+4.3.1 Task: Add Cytoscape.js component that calls MCP query_graph to fetch active workflow nodes and relationships, rendering as directed graph
+4.3.2 Task: Apply status-based styling: pending=gray (#9CA3AF), executing=yellow (#FCD34D), complete=green (#10B981), failed=red (#EF4444)
+4.3.3 Task: On node click, display properties panel showing all node properties formatted as key-value pairs in monospace font
+4.3.4 Task: Implement auto-refresh every 10 seconds using setInterval, with smooth transitions to avoid jarring updates
+

app_requirements/4_feature_source_system_repo.txt

@@ -1,46 +0,0 @@
-4. Feature: Source System Integration & Schema Repository
-4.1 Story: As a system, I need to connect to and catalog all available data sources through MCP.
-
-4.1.1 Task: Implement MCP client for PostgreSQL with full introspection
-4.1.2 Task: Implement MCP client for MySQL/MariaDB
-4.1.3 Task: Implement MCP client for MongoDB with schema inference
-4.1.4 Task: Implement MCP client for S3/filesystem with format detection
-4.1.5 Task: Implement MCP client for REST APIs with OpenAPI import
-4.1.6 Task: Create SourceSystem nodes with connection metadata
-
-4.2 Story: As an agent, I need to automatically discover and map data across all sources.
-
-4.2.1 Task: Run initial discovery to catalog all tables/collections/endpoints
-4.2.2 Task: Extract column-level metadata (types, constraints, statistics)
-4.2.3 Task: Identify primary/foreign keys and relationships
-4.2.4 Task: Sample data for profiling and example generation
-4.2.5 Task: Detect potential cross-source join keys
-4.2.6 Task: Generate and store example queries for each source
-
-4.3 Story: As an agent, I need to continuously monitor sources for changes.
-
-4.3.1 Task: Implement scheduled schema comparison workflows
-4.3.2 Task: Run lightweight heartbeat queries to detect changes
-4.3.3 Task: Create SchemaChange nodes when differences found
-4.3.4 Task: Assess impact of changes on existing workflows
-4.3.5 Task: Alert on breaking changes requiring attention
-4.3.6 Task: Update statistics and samples periodically
-
-4.4 Story: As an agent, I need to intelligently route queries to appropriate sources.
-
-4.4.1 Task: Parse user questions for entity and domain references
-4.4.2 Task: Match entities to source tables using schema repository
-4.4.3 Task: Generate source-specific queries via MCP
-4.4.4 Task: Create QueryPlan nodes showing execution strategy
-4.4.5 Task: Execute parallel queries when multiple sources needed
-4.4.6 Task: Merge and reconcile results from multiple sources
-
-4.5 Story: As a system, I need to track data lineage and dependencies.
-
-4.5.1 Task: Create lineage relationships between source and derived data
-4.5.2 Task: Store transformation logic as nodes
-4.5.3 Task: Build impact analysis queries
-4.5.4 Task: Generate data flow documentation
-4.5.5 Task: Identify redundant or conflicting data sources
-4.5.6 Task: Recommend source consolidation opportunities
-

app_requirements/5_feature_Docker_Deployment.txt

@@ -0,0 +1,16 @@
+5. Feature: Docker Deployment
+5.1 Story: As a developer, I need everything to run with docker-compose up.
+
+5.1.1 Task: Create docker-compose.yml with services: neo4j (image: neo4j:5), mcp-server (build: ./mcp), agent (build: ./agent), frontend (build: ./frontend), postgres (image: postgres:15)
+5.1.2 Task: Define service dependencies: mcp-server depends_on neo4j, agent depends_on mcp-server and postgres, frontend depends_on mcp-server
+5.1.3 Task: Create .env.example with all required variables: NEO4J_AUTH, POSTGRES_CONNECTION, LLM_API_KEY, MCP_API_KEYS, annotated with descriptions
+5.1.4 Task: Configure volume mounts: ./neo4j/data:/data for Neo4j, ./postgres/data:/var/lib/postgresql/data for PostgreSQL persistence
+5.1.5 Task: Add health checks: Neo4j bolt port 7687, PostgreSQL port 5432, MCP server /health endpoint, with restart policies on failure
+
+5.2 Story: As a developer, I need a test workflow to validate the system.
+
+5.2.1 Task: Create init.cypher script that MCP server runs on startup to create "Entity Resolution Demo" workflow with 3 pre-configured instructions
+5.2.2 Task: Via MCP write_graph, create instructions: (1) discover_schema with postgres target, (2) find_duplicates with similarity threshold 0.8, (3) merge_entities with merge strategy
+5.2.3 Task: Run test by sending "Find duplicate customers" to chat, verify agent executes instruction #1, then pauses for exactly 5 minutes (check logs)
+5.2.4 Task: During pause, use Neo4j Browser to edit instruction #2 parameters, changing threshold to 0.9, verify agent uses updated value
+5.2.5 Task: After workflow completes, use MCP query_graph to verify all executions logged: "MATCH (e:Execution) RETURN count(e)" should equal 3

app_requirements/5_feature_UI.txt

@@ -1,45 +0,0 @@
-5. Feature: Next.js Intelligent Frontend
-5.1 Story: As a user, I need a modern web interface to interact with the system.
-
-5.1.1 Task: Setup Next.js with TypeScript, Tailwind, and shadcn/ui
-5.1.2 Task: Implement tRPC for type-safe API communication
-5.1.3 Task: Add WebSocket support for real-time updates
-5.1.4 Task: Create Zustand stores for state management
-5.1.5 Task: Implement NextAuth with role-based access
-5.1.6 Task: Build responsive layout with dark mode
-
-5.2 Story: As a user, I need natural language interaction with intelligent query routing.
-
-5.2.1 Task: Create chat interface with context awareness
-5.2.2 Task: Display query routing decisions and source selection
-5.2.3 Task: Show real-time execution progress through sources
-5.2.4 Task: Present unified results with source attribution
-5.2.5 Task: Highlight confidence scores and data quality
-5.2.6 Task: Implement follow-up question suggestions
-
-5.3 Story: As a user, I need to visualize and explore the knowledge graph and data relationships.
-
-5.3.1 Task: Integrate Cytoscape.js for large graph exploration
-5.3.2 Task: Implement React Flow for workflow building
-5.3.3 Task: Create schema browser with source system navigation
-5.3.4 Task: Build lineage visualization showing data flow
-5.3.5 Task: Add search and filter capabilities
-5.3.6 Task: Implement node/edge inspection panels
-
-5.4 Story: As a user, I need to monitor and control workflow execution.
-
-5.4.1 Task: Create workflow dashboard with status overview
-5.4.2 Task: Build approval queue for pending instructions
-5.4.3 Task: Implement instruction editor with validation
-5.4.4 Task: Add execution timeline with phase progress
-5.4.5 Task: Create audit trail viewer
-5.4.6 Task: Build performance analytics dashboard
-
-5.5 Story: As a user, I need to manage data sources and their schemas.
-
-5.5.1 Task: Create source system configuration interface
-5.5.2 Task: Build schema change notification center
-5.5.3 Task: Implement data quality monitoring dashboard
-5.5.4 Task: Add query performance analytics by source
-5.5.5 Task: Create cross-source entity mapping tool
-5.5.6 Task: Build source health status monitor

app_requirements/6_feature_QA.txt

@@ -1,27 +0,0 @@
-6. Feature: Testing, Quality & Learning
-6.1 Story: As a developer, I need comprehensive testing across all system layers.
-
-6.1.1 Task: Unit tests for MCP server and source clients
-6.1.2 Task: Integration tests for Neo4j operations
-6.1.3 Task: End-to-end tests for complete workflows
-6.1.4 Task: Test schema change detection and handling
-6.1.5 Task: Validate cross-source query execution
-6.1.6 Task: Load test concurrent workflow execution
-
-6.2 Story: As a system, I need to continuously improve through learning from operations.
-
-6.2.1 Task: Analyze query patterns to optimize routing
-6.2.2 Task: Learn entity relationships from successful joins
-6.2.3 Task: Identify and cache frequently accessed data
-6.2.4 Task: Generate new workflow templates from patterns
-6.2.5 Task: Recommend schema optimizations
-6.2.6 Task: Build anomaly detection for data quality
-
-6.3 Story: As an operations team, I need monitoring of system health and effectiveness.
-
-6.3.1 Task: Track QA pass rates by workflow type
-6.3.2 Task: Monitor source system response times
-6.3.3 Task: Measure human intervention frequency
-6.3.4 Task: Analyze workflow completion rates
-6.3.5 Task: Create SLA compliance reports
-6.3.6 Task: Generate daily operations summary

app_requirements/6_feature_parking_lot_items.txt

@@ -0,0 +1,32 @@
+Parking Lot (Post-MVP)
+Immediate Next (v1.1)
+
+Multiple data sources (S3, MySQL, MongoDB)
+Approval workflows (not just pauses)
+Better graph visualization (React Flow)
+Schema change detection
+Human intervention tracking nodes
+
+Soon After (v1.2)
+
+External MCP access for other teams
+Workflow templates library
+QA validation loops
+Performance monitoring dashboard
+Vector embeddings for similarity
+
+Later (v2.0)
+
+Full Next.js frontend with auth
+Cross-source query federation
+Automated learning from patterns
+Kubernetes deployment
+RBAC and compliance
+
+Future Vision
+
+Self-improving workflows
+Anomaly detection
+Data lineage tracking
+Blue-green deployments
+Multi-tenant isolation

app_requirements/7_feature_deployment.md

@@ -1,27 +0,0 @@
-7. Feature: Deployment & Operations
-7.1 Story: As a developer, I need containerized deployment with production readiness.
-
-7.1.1 Task: Create multi-stage Docker builds for all services
-7.1.2 Task: Write Docker Compose for local development
-7.1.3 Task: Create Kubernetes manifests for production
-7.1.4 Task: Implement health checks and readiness probes
-7.1.5 Task: Configure resource limits and auto-scaling
-7.1.6 Task: Set up distributed tracing with OpenTelemetry
-
-7.2 Story: As an operations team, I need security and compliance controls.
-
-7.2.1 Task: Implement RBAC with fine-grained permissions
-7.2.2 Task: Add data encryption at rest and in transit
-7.2.3 Task: Create data masking for sensitive fields
-7.2.4 Task: Build compliance audit reports
-7.2.5 Task: Implement secret rotation for credentials
-7.2.6 Task: Add penetration testing to CI/CD
-
-7.3 Story: As a developer, I need automated CI/CD with quality gates.
-
-7.3.1 Task: Setup GitHub Actions for automated testing
-7.3.2 Task: Add static analysis and security scanning
-7.3.3 Task: Implement database migration automation
-7.3.4 Task: Create blue-green deployment strategy
-7.3.5 Task: Add automated rollback on failures
-7.3.6 Task: Setup monitoring with Prometheus/Grafana