Docs/ARCHITECTURE.md

Complete System Architecture

Document purpose: System-wide architecture covering backend, frontend, database, deployment, and all component interactions. Read this to understand how all pieces fit together.

---

Table of Contents

1. High-Level System Diagram
2. Backend Architecture (src/)
3. Frontend Architecture (frontend/)
4. API Contract
5. Data Flow
6. Deployment Architecture

---

High-Level System Diagram

┌────────────────────────────────────────────────────────────────────────────────┐
│                           EXTERNAL DATA SOURCES                                │
│  ┌──────────┐ ┌────────────┐ ┌────────┐ ┌──────┐ ┌──────┐ ┌──────────────┐   │
│  │ Notion   │ │ Confluence │ │ GitHub │ │ Slack │ │ Jira  │ │URLs + Firecrawl│   │
│  └──────────┘ └────────────┘ └────────┘ └──────┘ └──────┘ └──────────────┘   │
└────────────────────┬───────────────────────────────────────────────────────────┘
                     │ (Webhooks + Polling)
                     ▼
┌────────────────────────────────────────────────────────────────────────────────┐
│                            BACKEND (Python/FastAPI)                            │
│  ┌────────────────┐  ┌──────────────────┐  ┌──────────────────────────────┐   │
│  │  Data Ingestion │  │ RAG + Retrieval  │  │  Analytics & Intelligence   │   │
│  │  ├─ Adapters    │  │ ├─ Hybrid search │  │  ├─ Query events           │   │
│  │  ├─ Docling     │  │ ├─ BGE-M3        │  │  ├─ Knowledge graph        │   │
│  │  ├─ GLiNER PII  │  │ ├─ Qdrant        │  │  └─ Anomaly detection      │   │
│  │  └─ Chunking    │  │ └─ LLM agents    │  └──────────────────────────────┘   │
│  └────────────────┘  └──────────────────┘                                      │
│         ▲                      ▲                          ▲                     │
│         │                      │                          │                     │
│  ┌──────┴──────────────────────┴──────────────────────────┴────────────┐      │
│  │              FastAPI Backend (Uvicorn)                             │      │
│  │              ├─ /api/query/* (search + follow-up)                 │      │
│  │              ├─ /api/analytics/* (dashboards)                    │      │
│  │              ├─ /api/admin/* (data source management)            │      │
│  │              └─ /ws (WebSocket for real-time alerts)             │      │
│  └──────┬──────────────────────────────────────────────────────────┘      │
│         │                                                                   │
│  ┌──────▼────────────────────────────────────────────────────────┐        │
│  │     Data Layer (PostgreSQL, Qdrant, Neo4j, Redis, S3)         │        │
│  │  ├─ PostgreSQL: Metadata, RBAC, audit trails, queries        │        │
│  │  ├─ Qdrant: Vector embeddings (dense + sparse)              │        │
│  │  ├─ Neo4j: Knowledge graph (Service/Library/Incident/Team)  │        │
│  │  ├─ Redis: Cache, session state, pub/sub, task queues       │        │
│  │  └─ S3: PDFs, user uploads, exports                          │        │
│  └──────┬────────────────────────────────────────────────────────┘        │
└─────────┼────────────────────────────────────────────────────────────────────┘
          │
          │ (REST API + WebSocket)
          ▼
┌────────────────────────────────────────────────────────────────────────────────┐
│                        FRONTEND (React/TypeScript)                             │
│  ┌───────────────────────┐  ┌──────────────────┐  ┌────────────────────────┐  │
│  │  Query Interface      │  │  Dashboards      │  │  Admin UI              │  │
│  │  ├─ Search box        │  │  ├─ Query trends │  │  ├─ Data source mgmt   │  │
│  │  ├─ Results display   │  │  ├─ Knowledge    │  │  ├─ User management    │  │
│  │  ├─ Citations         │  │  │   health      │  │  ├─ RBAC editor        │  │
│  │  ├─ Follow-ups        │  │  ├─ Dependencies │  │  ├─ API keys           │  │
│  │  └─ Knowledge graph   │  │  └─ Alerts       │  │  └─ System health      │  │
│  └───────────────────────┘  └──────────────────┘  └────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────────────────────────┐  │
│  │  Component Layer (shadcn/ui + Tailwind)                                 │  │
│  │  ├─ Query & Search components                                           │  │
│  │  ├─ Chart & data table components (Recharts, TanStack Table)           │  │
│  │  ├─ Knowledge graph visualizer (Force-Graph)                           │  │
│  │  ├─ Authentication flow (JWT)                                          │  │
│  │  └─ Real-time notifications (WebSocket)                               │  │
│  └──────────────────────────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────────────────────────┐  │
│  │  State Management (TanStack Query + Zustand)                            │  │
│  │  ├─ Server state: Queries, analytics, user data (TanStack Query)      │  │
│  │  └─ Client state: UI state, theme, filters (Zustand)                  │  │
│  └──────────────────────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────────────────────┘

---

Backend Architecture (src/) — Agent-Based Design

Core Principle: Per-Source Agents

Rather than generic adapters flowing through a single pipeline, each data source is an independent agent with:

• Source-specific authentication & adapters
• Source-optimized chunking (preserves context like Confluence breadcrumbs, Jira comment threading)
• Independent Celery tasks (different polling cadences, priorities)
• Independent FastAPI routers (explicit webhooks like /webhooks/jira)
• Self-contained testing (test_run.py per agent)

This design ensures scalability by source, operational clarity, and production-grade maintainability.

Directory Structure

Note: The actual repo layout diverges from early plans. The implemented structure is below. src/query_engine/ and src/retrieval/ referenced in earlier design docs do not exist — that logic lives in agent/. Graph endpoints live in graph_store/, not src/api/graph.py.

agent/                          # LangGraph multi-agent query engine (IMPLEMENTED)
├── api.py                      # POST /agent/query — SSE streaming endpoint
├── graph.py                    # LangGraph build: planner → [doc_search|ticket_lookup|live_docs|sql_query] → join → synthesiser → guardrail
├── models.py                   # KnowledgeGraphState, QueryInput, ExecutionPlan, AgentResult, RetrievedChunk
├── config.py                   # LLM + agent config
├── prompts.py                  # Prompt templates
├── agents/
│   ├── planner.py              # Breaks query into AgentTask list
│   ├── synthesiser.py          # Streams answer tokens from top chunks
│   ├── guardrail.py            # Validates answer against sources; sets escalate flag
│   └── _gemini.py              # Gemini client helper (used in planner/synthesiser)
└── tools/
    ├── doc_search.py           # Qdrant hybrid dense+sparse search
    ├── ticket_lookup.py        # Jira-specific retrieval
    ├── live_docs.py            # Firecrawl real-time doc fetching
    ├── sql_query.py            # NL-to-SQL: translates query → validated SELECT → asyncpg execution
    └── summariser.py           # Context compression before synthesis

graph_store/                    # Neo4j knowledge graph (IMPLEMENTED)
├── api.py                      # GET /graph/nodes, POST /graph/ingest, GET /graph/traverse
├── stream.py                   # WS /graph/stream — streams nodes+edges with 50ms delay
├── extractor.py                # Gemini 2.5 Pro entity+relationship extraction (4 types, whitelist rels)
├── writer.py                   # Async Neo4j MERGE upserts, index creation
├── reader.py                   # Cypher traversal: incident→service→library→chunks
├── models.py                   # ExtractedEntity, ExtractedRelationship, ExtractionResult
└── config.py                   # Neo4j connection settings

src/
├── agents_app.py               # Combined FastAPI app: all agent routers + Qdrant/Redis init
│
├── jira_agent/                 # JIRA ingestion agent (IMPLEMENTED)
│   ├── __init__.py
│   ├── config.py               # JiraAgentConfig — JIRA_BASE_URL, JIRA_EMAIL, JIRA_API_TOKEN,
│   │                           #   JIRA_PROJECT_KEYS (csv), JIRA_WEBHOOK_SECRET, TEAM_ID
│   ├── adapter.py              # JiraAdapter — fetch_issue, fetch_all (JQL), fetch_incremental
│   │                           #   Basic auth (base64 email:api_token), ADF text extraction
│   ├── chunker.py              # chunk_jira_issue → chunk 0: issue body, chunks 1..N: comments
│   │                           #   Preserves thread structure for relation extraction
│   ├── pipeline.py             # ingest_issue / ingest_project → chunk → PII mask → embed → Qdrant
│   │                           #   Returns entity graph nodes for real-time streaming
│   ├── tasks.py                # Celery: jira_process_issue (queue=critical), 
│   │                           #   jira_sync_project (queue=polling)
│   ├── router.py               # FastAPI: POST /webhooks/jira, POST /jira/sync/{project_key}
│   └── test_run.py             # Mock + real runthrough; works without credentials
│
├── confluence_agent/           # Confluence ingestion agent (IMPLEMENTED)
│   ├── __init__.py
│   ├── config.py               # ConfluenceAgentConfig — BASE_URL, TOKEN, EMAIL,
│   │                           #   CONFLUENCE_SPACES (csv), CONFLUENCE_WEBHOOK_SECRET, TEAM_ID
│   ├── adapter.py              # ConfluenceAdapter — fetch_page, fetch_space, fetch_incremental (CQL)
│   │                           #   REST v2 API with pagination
│   ├── chunker.py              # chunk_confluence_page — BeautifulSoup heading-split + breadcrumbs
│   │                           #   [Space > Ancestor > Page] prefix on every chunk; tables = 1 chunk each
│   │                           #   Preserves hierarchy for entity linking
│   ├── pipeline.py             # ingest_page / ingest_space → chunk → PII mask → embed → Qdrant
│   │                           #   Returns entity graph nodes
│   ├── tasks.py                # Celery: confluence_process_page (queue=critical), 
│   │                           #   confluence_sync_space (queue=polling),
│   │                           #   confluence_periodic_sync (beat, 60 min incremental sync)
│   ├── router.py               # FastAPI: POST /webhooks/confluence, POST /confluence/sync/{space_key}
│   │                           #   POST /confluence/search (for admin dashboard)
│   └── test_run.py             # Mock + real runthrough; works without credentials
│
├── file_agent/                 # File ingestion agent (IMPLEMENTED)
│   ├── __init__.py
│   ├── config.py               # FileAgentConfig — UPLOAD_DIR, MAX_FILE_SIZE, ALLOWED_TYPES
│   ├── adapter.py              # FileAdapter — handle PDFs, DOCX, PPTX, TXT
│   │                           #   Uses docling for multi-format parsing
│   ├── chunker.py              # chunk_file_document — respects document structure (sections, pages)
│   ├── pipeline.py             # ingest_file → chunk → PII mask → embed → Qdrant
│   ├── tasks.py                # Celery: file_process_upload (queue=critical)
│   ├── router.py               # FastAPI: POST /files/upload, GET /files/{file_id}
│   └── test_run.py
│
├── shared/                     # Shared utilities (used by all agents)
│   ├── __init__.py
│   ├── pii_masker.py           # GLiNER-based PII detection (local, zero egress)
│   ├── embedder.py             # BGE-M3 embeddings (local inference)
│   ├── qdrant_client.py        # Qdrant connection + upsert helpers
│   ├── entity_extractor.py     # Extract entities/relationships from chunks (used per-agent)
│   ├── models.py               # Pydantic models (RawDocument, ChunkedDocument, Entity, Graph)
│   └── config.py               # Shared config (QDRANT_URL, REDIS_URL, etc.)
│
├── retrieval/                  # T1, T2, T3 retrieval layers (shared across queries)
│   ├── __init__.py
│   ├── hybrid_search.py        # T1: Dense + Sparse (RRF fusion) — queries Qdrant
│   ├── reranker.py             # BGE-reranker-v2-m3 integration
│   ├── context_compressor.py   # Compress top-5 into LLM context
│   ├── cag_agent.py            # T2: Cache-Augmented Generation (recent syncs)
│   ├── live_doc_agent.py       # T3: Real-time doc fetching (Firecrawl)
│   └── models.py               # Pydantic models for retrieval
│
├── query_engine/               # Query execution (LangGraph-based)
│   ├── __init__.py
│   ├── generator_agent.py      # Generator LLM agent (creates answer from context)
│   ├── critic_agent.py         # Critic LLM agent (validates against sources)
│   ├── orchestrator.py         # LangGraph: routes query through retrieval → generation → validation
│   ├── streaming.py            # Stream answer chunks + citations + graph to frontend
│   └── models.py               # Pydantic models for query responses
│
├── redis/                      # Redis utilities (shared)
│   ├── __init__.py
│   ├── cache.py                # Caching layer (with TTL)
│   ├── queues.py               # Task queues (per-agent ingestion, webhook events)
│   ├── session_state.py        # Query session state
│   ├── locks.py                # Distributed locks (prevent concurrent agent syncs)
│   └── pubsub.py               # Pub/sub for real-time graph updates to frontend (query_id → node)
│
├── api/                        # FastAPI main app + shared endpoints
│   ├── __init__.py
│   ├── auth.py                 # POST /auth/login, /auth/logout, /auth/refresh
│   ├── query.py                # POST /api/query (streaming), /api/query/{id}/follow-up
│   ├── workspace.py            # GET/POST /api/workspace/queries, /saved
│   ├── admin.py                # GET /api/admin/agents (show all agent statuses)
│   └── graph.py                # GET /api/graph/entities, /api/graph/query/{query_id}
│
├── db/                         # Database models & utilities
│   ├── __init__.py
│   ├── models.py               # SQLAlchemy models (User, Query, Document, Entity, Graph)
│   ├── session.py              # Database session management
│   └── init_db.py              # Schema initialization
│
├── auth/                       # Authentication & authorization
│   ├── __init__.py
│   ├── jwt_handler.py          # JWT encode/decode, token refresh
│   ├── oauth.py                # OAuth2 + SSO integration (phase 2)
│   ├── rbac.py                 # Role-based access control decorator
│   ├── permissions.py          # Permission checks
│   └── models.py               # User, Role, Permission models
│
├── utils/                      # Shared utilities
│   ├── __init__.py
│   ├── logger.py               # Structured logging (JSON)
│   ├── metrics.py              # Prometheus metrics
│   ├── telemetry.py            # OpenTelemetry (phase 2)
│   └── exceptions.py           # Custom exceptions
│
└── tests/                      # Comprehensive test suite
    ├── __init__.py
    ├── agents/                 # Per-agent tests (JIRA, Confluence, File)
    ├── retrieval/              # Retrieval pipeline tests
    ├── query_engine/           # Query generation + validation tests
    ├── fixtures/               # Pytest fixtures (mock data)
    └── integration/            # End-to-end scenarios

Key Backend Design Decisions

1. Per-Source Agents: Each source (Jira, Confluence, File) is an independent module with its own adapter, chunker, pipeline, and Celery tasks. This enables source-specific optimization and independent scaling.

2. Source-Optimized Chunking:

   • Confluence: Preserves [Space > Ancestor > Page] hierarchy for entity linking
   • Jira: Preserves comment threading for relation extraction
   • File: Respects document structure (sections, pages)
   • Each source extracts its own entity relationships

3. Independent Celery Scheduling:

   • jira_sync_project → configurable interval (often 1 hour)
   • confluence_periodic_sync → beat scheduler (60 min incremental)
   • file_process_upload → immediate (queue=critical)
   • Each agent controls its own cadence

4. PII Masking First: GLiNER runs in shared/pii_masker.py — local, zero-egress, runs before Qdrant indexing.

5. Entity Extraction Per-Agent: Each pipeline returns a graph of entities + relationships (e.g., Jira: issue→linked_issue, Confluence: page→linked_page). Frontend streams these nodes as they're extracted.

6. Real-Time Graph Streaming: Via Redis pub/sub (query_id → {nodes, edges}) — frontend doesn't wait for full completion.

7. Redis Everywhere: Cache, queues, session state, distributed locks, and pub/sub all via Redis.

8. Hybrid Retrieval (T1): Dense (BGE-M3) + Sparse (BM25) via RRF — queries Qdrant.

---

Frontend Architecture (frontend/)

Directory Structure

frontend/
├── index.html                   # Entry HTML (Vite serves this)
├── vite.config.ts              # Vite build config
├── tsconfig.json               # TypeScript config
├── tailwind.config.ts          # Tailwind design tokens + dark mode
├── postcss.config.js           # PostCSS + Tailwind plugins
├── package.json                # Dependencies + scripts
├── .env.example                # Required environment variables
│
├── src/
│   ├── main.tsx                # React app entry point
│   ├── App.tsx                 # Root component + routing
│   │
│   ├── components/
│   │   ├── common/             # Reusable components
│   │   │   ├── Header.tsx      # Top nav bar
│   │   │   ├── Sidebar.tsx     # Left navigation
│   │   │   ├── Footer.tsx      # Footer
│   │   │   ├── Button.tsx      # Button variants (from shadcn)
│   │   │   ├── Input.tsx       # Text input (from shadcn)
│   │   │   ├── Card.tsx        # Card container
│   │   │   ├── Modal.tsx       # Modal/dialog
│   │   │   ├── Badge.tsx       # Status badges
│   │   │   ├── Tooltip.tsx     # Tooltips
│   │   │   ├── Toast.tsx       # Toast notifications
│   │   │   └── Loading.tsx     # Loading skeleton
│   │   │
│   │   ├── query/              # Query interface (Engineer primary)
│   │   │   ├── SearchBox.tsx   # Main search input (Cmd+K support)
│   │   │   ├── QueryModal.tsx  # Modal for new query
│   │   │   ├── QueryHistory.tsx # Query history panel
│   │   │   ├── SuggestedTopics.tsx # Related queries
│   │   │   └── QueryFeedback.tsx # Thumbs up/down
│   │   │
│   │   ├── results/            # Results display + knowledge graph
│   │   │   ├── ResultsPage.tsx # Main results container
│   │   │   ├── Answer.tsx      # Generated answer with citations
│   │   │   ├── Citations.tsx   # Cited source chunks
│   │   │   ├── FollowUp.tsx    # Follow-up prompt
│   │   │   ├── KnowledgeGraph.tsx # Knowledge graph visualization
│   │   │   ├── GraphNode.tsx   # Individual node component
│   │   │   ├── RelatedDocs.tsx # Related document snippets
│   │   │   └── ShareResults.tsx # Share/export options
│   │   │
│   │   ├── analytics/          # Dashboards (Manager primary)
│   │   │   ├── AnalyticsDashboard.tsx # Main analytics page
│   │   │   ├── QueryTrendChart.tsx # Line chart for query volume
│   │   │   ├── TopicsChart.tsx # Bar chart for topics
│   │   │   ├── SuccessRateGauge.tsx # Gauge chart
│   │   │   ├── KnowledgeHealthDashboard.tsx # Health metrics
│   │   │   ├── DependencyTracker.tsx # Breaking changes table
│   │   │   ├── EscalationTable.tsx # Unresolved queries
│   │   │   ├── TeamSettings.tsx # Team configuration
│   │   │   └── AnalyticsExport.tsx # Export reports
│   │   │
│   │   ├── admin/              # Admin UI (Admin primary)
│   │   │   ├── AdminDashboard.tsx # Main admin page
│   │   │   ├── SystemHealth.tsx # Health status cards
│   │   │   ├── DataSourceManager.tsx # Add/edit sources
│   │   │   ├── DataSourceForm.tsx # Source configuration wizard
│   │   │   ├── UserManager.tsx # User list + invite
│   │   │   ├── RBACEditor.tsx  # RBAC policy editor
│   │   │   ├── APIKeyManager.tsx # Generate/revoke keys
│   │   │   └── SystemLogs.tsx  # View logs + alerts
│   │   │
│   │   └── auth/               # Authentication UI
│   │       ├── LoginPage.tsx   # Login form (SSO + fallback)
│   │       ├── SSORedirect.tsx # OAuth callback handler
│   │       └── ProtectedRoute.tsx # Route guard
│   │
│   ├── pages/                  # Route pages (using TanStack Router)
│   │   ├── Home.tsx            # Dashboard home
│   │   ├── QueryPage.tsx       # Query results page
│   │   ├── AnalyticsPage.tsx   # Analytics dashboards
│   │   ├── AdminPage.tsx       # Admin dashboards
│   │   ├── WorkspacePage.tsx   # Personal/team workspace
│   │   ├── NotFoundPage.tsx    # 404 page
│   │   └── ErrorPage.tsx       # Error boundary
│   │
│   ├── hooks/                  # Custom React hooks
│   │   ├── useSSEStream.ts     # SSE consumer for POST /agent/query — manages fetch + ReadableStream parsing
│   │   ├── useGraphStream.ts   # WebSocket consumer for WS /graph/stream — feeds Force-Graph 2D progressively
│   │   ├── useNotifications.ts # WebSocket consumer for WS /ws system notifications (future)
│   │   ├── useAnalytics.ts     # Fetch analytics data
│   │   ├── useAuth.ts          # Authentication state
│   │   ├── useTheme.ts         # Dark mode toggle
│   │   ├── useLocalStorage.ts  # Persist state to localStorage
│   │   ├── usePagination.ts    # Pagination logic
│   │   └── useDebounce.ts      # Debounce search input
│   │
│   ├── stores/                 # Zustand state management
│   │   ├── authStore.ts        # User + auth state
│   │   ├── uiStore.ts          # UI state (theme, sidebar open, etc.)
│   │   ├── filterStore.ts      # Dashboard filters
│   │   └── workspaceStore.ts   # Workspace selections
│   │
│   ├── lib/
│   │   ├── api.ts              # TanStack Query setup + HTTP client
│   │   ├── http.ts             # httpx client wrapper (JWT refresh)
│   │   ├── auth.ts             # JWT helpers, localStorage auth
│   │   ├── websocket.ts        # WebSocket manager for alerts
│   │   ├── utils.ts            # General utilities (debounce, etc.)
│   │   ├── validators.ts       # Input validation (Zod)
│   │   ├── constants.ts        # App-wide constants
│   │   ├── error-handler.ts    # Centralized error handling
│   │   └── date.ts             # Date formatting helpers
│   │
│   ├── types/
│   │   ├── index.ts            # Re-export all types
│   │   ├── api.ts              # API response types
│   │   ├── user.ts             # User + auth types
│   │   ├── query.ts            # Query + results types
│   │   ├── analytics.ts        # Analytics types
│   │   ├── components.ts       # Component prop types
│   │   └── errors.ts           # Error types
│   │
│   ├── styles/
│   │   ├── globals.css         # Global styles + Tailwind imports
│   │   ├── design-tokens.css   # Design tokens (terracotta, white, dark mode)
│   │   ├── animations.css      # Custom animations (optional)
│   │   └── responsive.css      # Responsive utility classes
│   │
│   └── config/
│       ├── routes.ts           # TanStack Router configuration
│       ├── env.ts              # Environment variables + validation
│       └── queryClient.ts      # TanStack Query client config
│
├── public/                     # Static assets
│   ├── logo.svg                # Logo
│   ├── favicon.ico             # Favicon
│   └── assets/                 # Images, icons
│
├── tests/
│   ├── __mocks__/              # Mock data + API responses
│   ├── components/             # Component tests (Vitest + RTL)
│   ├── hooks/                  # Hook tests
│   ├── utils/                  # Utility tests
│   └── setup.ts                # Vitest + RTL setup
│
├── .eslintrc.json              # ESLint config
├── .prettierrc                 # Prettier config
└── README.md                   # Frontend development guide

Frontend Design Decisions

1. Vite + React 18: Fast dev, instant HMR, minimal config. No SSR needed for SPA.
2. TanStack Router: Fully typed routing; better DX than React Router v6.
3. TanStack Query: Server state management with automatic caching/refetching.
4. Zustand: Lightweight client state (theme, UI, filters); no Redux boilerplate.
5. shadcn/ui + Tailwind: Copy-paste components, full control, design tokens system.
6. Responsive Design: Mobile (320px), Tablet (768px), Desktop (1024px+).
7. WebSocket: Native API for real-time alerts; no Socket.io overhead.
8. JWT + httpOnly Cookies: Secure auth; backend validates on every request.

---

API Contract

Authentication

POST /api/auth/login
├─ Request: { email, password } or { sso_provider, sso_token }
├─ Response: { access_token, refresh_token, user: { id, email, role, team_id } }
├─ Sets httpOnly cookie: __auth_token
└─ Bearer token in Authorization header for all subsequent requests

POST /api/auth/refresh
├─ Request: { refresh_token }
├─ Response: { access_token }
└─ Auto-called by frontend before token expires

POST /api/auth/logout
├─ Clears httpOnly cookie
└─ Backend invalidates refresh token in Redis

Agent Webhook Endpoints (Per-Source)

POST /webhooks/jira
├─ Validates Jira webhook signature (X-Atlassian-Webhook-Signature)
├─ Extracts issue_created, issue_updated, comment_created events
├─ Routes to jira_process_issue Celery task (queue=critical)
└─ Returns immediately (202 Accepted)

POST /webhooks/confluence
├─ Validates Confluence webhook signature
├─ Extracts page_created, page_updated, page_trashed events
├─ Routes to confluence_process_page Celery task (queue=critical)
└─ Returns immediately (202 Accepted)

POST /files/upload
├─ Accepts multipart/form-data with file + team_id
├─ Routes to file_process_upload Celery task (queue=critical)
├─ Returns file_id immediately; processing async
└─ Frontend polls /files/{file_id} for status

POST /jira/sync/{project_key}
├─ Manual trigger; requires admin role
├─ Routes to jira_sync_project Celery task (queue=polling)
└─ Returns job_id for polling

POST /confluence/sync/{space_key}
├─ Manual trigger; requires admin role
├─ Routes to confluence_sync_space Celery task (queue=polling)
└─ Returns job_id for polling

Query API (Streaming SSE)

POST /agent/query
├─ Request: { query: string, team_id: string, session_id: string }
├─ Response: Content-Type: text/event-stream
│  ├─ event: plan_ready        → { tasks: [AgentTask], reasoning: string }
│  ├─ event: agent_started     → { agent: "doc_search"|"ticket_lookup"|"live_docs"|"sql_query"|"summariser" }
│  ├─ event: agent_done        → { agent: string, chunks: [RetrievedChunk], confidence: "high"|"medium"|"low" }
│  ├─ event: synthesis_started → {}
│  ├─ event: answer_chunk      → { chunk: string }   (repeats, one per token)
│  ├─ event: guardrail_result  → { score: float, escalate: bool }
│  ├─ event: done              → {}
│  └─ event: error             → { message: string }
└─ Headers: Cache-Control: no-cache, X-Accel-Buffering: no

POST /api/query/{query_id}/feedback
├─ Request: { sentiment: "helpful"|"not_helpful"|"hallucinated", text?: string }
└─ Response: { success: true }

Knowledge Graph API

GET /graph/nodes?limit=50
└─ Response: { count: int, nodes: [{ label: string, name: string }] }
   (excludes Chunk and Document nodes — returns Service/Library/Incident/Team only)

POST /graph/ingest
├─ Request: { chunk_ids: [string], team_id: string }
└─ Response: { ingested: int }
   (fetches chunks from Supabase, runs Gemini extraction, upserts to Neo4j)

GET /graph/traverse?type=incident|service|library&name=string&team_id=string
└─ Response: { type, name, team_id, chunks: [string] }
   (multi-hop Cypher traversal — returns text chunks for context augmentation)

WS /graph/stream
└─ Streams: node events, edge events, then done event (see Real-Time API above)

Analytics API

GET /api/analytics/queries?date_range=30d&team_id=...
├─ Response: {
│    query_count: 1243,
│    unique_users: 243,
│    avg_response_time_ms: 1200,
│    success_rate: 0.76,
│    trend: { data: [{date, count}] }
│  }

GET /api/analytics/knowledge-health
├─ Response: {
│    overall_score: 7.2,
│    coverage: 0.68,
│    freshness: 0.82,
│    accuracy: 0.76,
│    accessibility: 0.71,
│    gaps: [{ topic: "ORM patterns", queries: 12, solutions: 0 }]
│  }

GET /api/analytics/dependencies
├─ Response: {
│    dependencies: [{name, current_version, latest_version, breaking_changes}],
│    alerts: 3
│  }

Admin API

POST /api/admin/sources
├─ Request: { type, config, rbac_level }
├─ Response: { id, status, test_result }
└─ Triggers background sync

GET /api/admin/sources
├─ Response: [{ id, type, status, last_sync, record_count }]

PATCH /api/admin/sources/{id}
├─ Request: { name, config, rbac_level }
├─ Response: { updated_source }

DELETE /api/admin/sources/{id}
├─ Soft delete; preserves audit trail

---

POST /api/admin/users/invite
├─ Request: { emails: ["alice@..."], role, team_id }
├─ Response: { invitations: [{ email, invitation_id, expires_at }] }
└─ Sends email invite

GET /api/admin/users
├─ Response: [{ id, email, role, team_id, status, created_at }]

DELETE /api/admin/users/{user_id}
├─ Deactivates user (no hard delete for compliance)

---

POST /api/admin/rbac
├─ Request: { name, description, teams, sources, filters }
├─ Response: { id, policy }
└─ Returns doc count matching policy

GET /api/admin/rbac
├─ Response: [{ id, name, doc_count }]

PATCH /api/admin/rbac/{id}
├─ Update existing policy

---

POST /api/admin/api-keys
├─ Request: { name, permissions, rate_limits, expiry }
├─ Response: { key: "sk_...", created_at }
└─ Only returned once

GET /api/admin/api-keys
├─ Response: [{ name, created_at, last_used, permissions }]

Bash Development Testing

Use these instead of Swagger UI when you need to test streaming behaviour from the terminal.

Test SSE query stream (replaces Swagger — Swagger can't stream SSE):

#!/usr/bin/env bash
# test_query.sh — streams the SSE response token-by-token to stdout

BASE_URL="${GODSPEED_API:-http://localhost:8000}"

curl -N -s \
  -X POST "${BASE_URL}/agent/query" \
  -H "Content-Type: application/json" \
  -d '{"query":"What is the auth service?","team_id":"team-1","session_id":"test-001"}' \
| while IFS= read -r line; do
    echo "$line"
  done

Test graph REST endpoints:

BASE_URL="${GODSPEED_API:-http://localhost:8000}"

# List all graph nodes
curl -s "${BASE_URL}/graph/nodes?limit=20" | python3 -m json.tool

# Traverse from a service
curl -s "${BASE_URL}/graph/traverse?type=service&name=auth-service&team_id=team-1" \
  | python3 -m json.tool

# Ingest chunks into graph
curl -s -X POST "${BASE_URL}/graph/ingest" \
  -H "Content-Type: application/json" \
  -d '{"chunk_ids":["chunk-abc123"],"team_id":"team-1"}' \
  | python3 -m json.tool

Test WebSocket graph stream (requires wscat — install with npm i -g wscat):

BASE_URL="${GODSPEED_WS:-ws://localhost:8000}"
wscat -c "${BASE_URL}/graph/stream"
# Prints node/edge/done events as they arrive

Test Jira webhook signature (bash + openssl):

BASE_URL="${GODSPEED_API:-http://localhost:8000}"
BODY='{"webhookEvent":"jira:issue_created","issue":{"id":"TEST-1","fields":{"summary":"Auth service down"}}}'
SECRET="your_jira_webhook_secret"
SIG="sha256=$(echo -n "${BODY}" | openssl dgst -sha256 -hmac "${SECRET}" | awk '{print $2}')"

curl -s -X POST "${BASE_URL}/webhooks/jira" \
  -H "Content-Type: application/json" \
  -H "X-Atlassian-Webhook-Signature: ${SIG}" \
  -d "${BODY}"

Test file upload:

BASE_URL="${GODSPEED_API:-http://localhost:8000}"
curl -s -X POST "${BASE_URL}/files/upload" \
  -F "file=@/path/to/doc.pdf" \
  -F "team_id=team-1"

---

Real-Time API

There are two distinct real-time channels — do not conflate them:

Channel 1: Query streaming (SSE)

POST /agent/query   →   Content-Type: text/event-stream

Emits events in order:
  event: plan_ready        data: { tasks: [...], reasoning: "..." }
  event: agent_started     data: { agent: "doc_search" }
  event: agent_done        data: { agent: "doc_search", chunks: [...], confidence: "high" }
  event: synthesis_started data: {}
  event: answer_chunk      data: { chunk: "token text" }   ← repeats per token
  event: guardrail_result  data: { score: 0.92, escalate: false }
  event: done              data: {}
  event: error             data: { message: "..." }        ← on failure

Request body: { query: string, team_id: string, session_id: string }

Channel 2: Knowledge graph visualization (WebSocket)

WS /graph/stream

Emits in order (50ms delay between each):
  { event: "node", id: "...", label: "Service", name: "auth-service" }
  { event: "edge", from: "...", to: "...", rel: "DEPENDS_ON" }
  ...
  { event: "done", nodes_count: 42, edges_count: 87 }

Channel 3: System notifications (WebSocket)

WS /ws   (future — not yet implemented)

Will emit:
  event: "query_answered"  → { query_id, new_docs_count }
  event: "escalation_spike" → { topic, spike_rate }         (manager-only)
  event: "breaking_change"  → { dependency, version, url }  (admin-only)
  event: "data_sync_failed" → { source, error }             (admin-only)
  event: "knowledge_gap"    → { topic, query_count }        (all users)

---

Data Flow

Flow 1: Engineer Query → Answer

1. Engineer types query in SearchBox
   ├─ frontend sends POST /agent/query { query, team_id, session_id }
   └─ frontend simultaneously opens WS /graph/stream for parallel graph rendering

2. Backend receives query via SSE stream
   ├─ LangGraph planner breaks query into AgentTask list → emits plan_ready
   ├─ Each agent runs (doc_search / ticket_lookup / live_docs) → emits agent_started + agent_done
   ├─ doc_search: BGE-M3 embed → Qdrant hybrid search (dense+sparse RRF) → top 50 → BGE reranker → top 5
   ├─ Synthesiser streams answer tokens → emits answer_chunk per token
   └─ Guardrail validates answer against source chunks → emits guardrail_result

3. Guardrail result
   ├─ guardrail_passed=true → done event
   ├─ guardrail_passed=false + escalate=true → warning banner shown in frontend
   └─ Citations come from agent_done chunks (already streamed in step 2)

4. Frontend connects to graph stream (parallel to query SSE)
   ├─ WS /graph/stream streams the pre-built Neo4j graph (query-scoped subgraph)
   ├─ Nodes arrive one-by-one with 50ms delays: { event:"node", label, name }
   ├─ Edges arrive after nodes: { event:"edge", from, to, rel }
   └─ { event:"done" } signals completion
   Note: The knowledge graph is pre-built at ingestion time by Gemini 2.5 Pro
   (graph_store/extractor.py), not extracted from the answer at query time.

5. Frontend receives stream
   ├─ Displays answer immediately (no waiting)
   ├─ Renders citations as they arrive
   ├─ Knowledge graph appears once first connection established
   ├─ Related docs populate as backend fetches
   └─ Full page interactive once final "done" event received

6. Feedback recorded
   ├─ Engineer clicks thumbs up/down
   ├─ Frontend POSTs /api/query/{id}/feedback
   ├─ Backend records sentiment + triggers analytics update
   └─ Feedback visible in query history + aggregated for managers

Flow 2: Data Ingestion (Daily/Polling)

1. Ingestion task triggered
   ├─ Webhook from source (e.g., Notion) OR Celery periodic task

2. Fetch stage
   ├─ Adapter queries source API
   ├─ Detects new/updated items (via timestamps or ETags)
   ├─ Downloads content

3. Normalize stage (Docling)
   ├─ Converts PDF/HTML/markdown to clean markdown
   ├─ Extracts tables as markdown tables
   ├─ Detects code blocks + language

4. PII Mask stage (GLiNER, local)
   ├─ Scans text for PII (names, emails, IDs, etc.)
   ├─ Replaces PII with placeholders (e.g., [REDACTED_EMAIL])
   ├─ Logs redaction for audit trail

5. Chunk stage (Semantic)
   ├─ Splits by paragraph/sentence boundaries
   ├─ Never splits code blocks or lists
   ├─ 15% overlap between chunks
   ├─ 256–512 tokens per chunk

6. Tag stage (Metadata)
   ├─ Adds source_uri, source_type, ingested_at
   ├─ Adds RBAC tag (public / team / restricted)
   ├─ Computes content_hash (for change detection)
   ├─ Detects doc_type (SOP, API doc, PR, etc.)

7. Embed stage
   ├─ Sends chunks to BGE-M3
   ├─ Gets 384-dim dense vectors
   ├─ Extracts sparse BM25-like vectors

8. Index stage
   ├─ Uploads dense vectors to Qdrant HNSW index
   ├─ Uploads sparse vectors to Qdrant sparse index
   ├─ Upserts metadata (PostgreSQL)
   ├─ Updates Redis cache (last_sync_timestamp)

9. Complete
   ├─ Backend records sync success in PostgreSQL
   ├─ Triggers webhook for frontend real-time update
   └─ Notifies admins if errors

Flow 3: Manager Views Analytics

1. Manager navigates to /analytics

2. Frontend loads analytics data
   ├─ POST /api/analytics/queries?date_range=30d
   ├─ POST /api/analytics/knowledge-health
   ├─ POST /api/analytics/dependencies

3. Backend aggregates from event logs
   ├─ Queries PostgreSQL (query_events table)
   ├─ Aggregates by date, team, topic
   ├─ Computes trends, success rates
   ├─ Identifies gaps from failed queries

4. Frontend renders dashboards
   ├─ Query trends (Recharts line chart)
   ├─ Topics (bar chart)
   ├─ Success rate (gauge)
   ├─ Escalations table
   ├─ Knowledge health heatmap

5. Optional: Manager exports report
   ├─ Frontend POSTs /api/analytics/export?format=pdf
   ├─ Backend generates PDF via ReportLab
   ├─ Streams PDF download to browser

---

Deployment Architecture

Development (docker-compose)

services:
  postgres:
    image: postgres:15
    volumes: [./data/postgres:/var/lib/postgresql/data]
    ports: [5432:5432]

  redis:
    image: redis:7-alpine
    ports: [6379:6379]

  qdrant:
    image: qdrant/qdrant:latest
    volumes: [./data/qdrant:/qdrant/storage]
    ports: [6333:6333]

  backend:
    build: ./backend
    ports: [8000:8000]
    depends_on: [postgres, redis, qdrant]
    environment:
      SQLALCHEMY_DATABASE_URL: postgresql://user:pass@postgres:5432/godspeed
      REDIS_URL: redis://redis:6379
      QDRANT_URL: http://qdrant:6333

  frontend:
    build: ./frontend
    ports: [3000:3000]
    depends_on: [backend]
    environment:
      VITE_API_BASE_URL: http://localhost:8000

  neo4j:
    image: neo4j:5
    ports: ["7474:7474", "7687:7687"]
    volumes: [./data/neo4j:/data]
    environment:
      NEO4J_AUTH: neo4j/godspeed_dev
      NEO4J_PLUGINS: '["apoc"]'

  celery:
    build: ./backend
    command: celery -A src.celery_app worker -Q critical,default,polling -l info
    depends_on: [postgres, redis, qdrant, neo4j]
    environment:
      SQLALCHEMY_DATABASE_URL: postgresql://user:pass@postgres:5432/godspeed
      REDIS_URL: redis://redis:6379
      NEO4J_URI: bolt://neo4j:7687
      NEO4J_USERNAME: neo4j
      NEO4J_PASSWORD: godspeed_dev

Production (Kubernetes)

# Deployments
- backend (FastAPI, 3 replicas, HPA)
- frontend (Nginx, 2 replicas, CDN)
- celery-worker (5 replicas, autoscaling on queue depth)

# StatefulSets
- postgres (with backup via S3)
- redis (cluster mode)
- qdrant (with persistence)

# Services
- backend-svc (ClusterIP)
- frontend-svc (LoadBalancer)
- postgres-svc (ClusterIP)
- redis-svc (ClusterIP)
- qdrant-svc (ClusterIP)

# ConfigMaps & Secrets
- app-config (env vars)
- api-keys (AWS S3, Notion OAuth, etc.)
- tls-certs (HTTPS)

# Ingress
- Routes /api/* to backend
- Routes /* to frontend
- TLS termination

Self-Hosted (Single Server)

nginx (reverse proxy, static frontend)
  ├─ localhost:8000 (FastAPI backend)
  ├─ localhost:5432 (PostgreSQL)
  ├─ localhost:6379 (Redis)
  └─ localhost:6333 (Qdrant)

All services in systemd or Docker containers
Automated backups via Cron + S3
Monitoring via Prometheus + Grafana (optional)

---

Key Architectural Principles

1. Separation of Concerns: Each layer (adapter, ingestion, retrieval, agent, API) has one responsibility.
2. Stateless Backend: FastAPI scales horizontally; state lives in PostgreSQL/Redis.
3. Async Everywhere: Celery for long-running tasks; FastAPI with asyncio for I/O.
4. RBAC First: All queries filtered by user's team/permissions at retrieval time.
5. Streaming Results: Don't wait for complete answer; stream chunks to frontend progressively.
6. Local PII: GLiNER runs on-premises; zero data egress for compliance.
7. Cacheable at Every Layer: Embeddings cached, searches cached, answers cached (with refresh policy).
8. Observable: Structured logging, metrics, traces (OpenTelemetry phase 2).