# CognifyAI — System Architecture

CognifyAI is a multimodal adaptive learning platform. It ingests raw knowledge (PDFs, web data), dynamically generates personalized learning materials across Visual, Textual, and Auditory modalities, and orchestrates complex RAG workflows backed by a LangGraph agent engine.

---

## System Overview

```mermaid
graph TD
    User([Learner]) --> FE[Next.js Frontend]

    FE -- REST / SSE --> API[FastAPI Backend]
    FE -- WebRTC --> LK[LiveKit Voice Server]
    FE -- Auth --> SB_AUTH[Supabase Auth]

    API --> AgentSvc[Agent Service\nLangGraph StateGraph]
    API --> RAGSvc[RAG Pipeline]
    API --> LearnSvc[Learning Services]
    API --> DB[(Supabase\nPostgreSQL + pgvector)]

    AgentSvc --> Router[Intent Router\n9 intents]
    Router --> Tools[Specialized Tool Nodes]
    Tools --> RAGSvc
    Tools --> LLMSvc[Gemini LLM Service]
    Tools --> WebSearch[DuckDuckGo / Brave]

    RAGSvc --> EmbedMS[HuggingFace\nEmbedding Microservice\nBAAI/bge-base-en-v1.5]
    RAGSvc --> DB

    LearnSvc --> ProfileSvc[Learner Profile\nEMA Engine]
    LearnSvc --> LessonSvc[Lesson Planner]
    LearnSvc --> ContentEng[Content Engine]
```

---

## 1. Enterprise RAG Module

**What it does:** Ingests documents, retrieves highly relevant context via hybrid search, and validates AI answers to prevent hallucinations.

### Contains

| File | Role |
|------|------|
| `services/chunking_service.py` | Hierarchical AST-based PDF chunking with breadcrumb enrichment; preserves heading parent-child relationships |
| `services/embedding_service.py` | Remote client to HuggingFace embedding microservice — no local torch/transformers |
| `services/rag_service.py` | Hybrid Retrieval (dense + sparse) merged via Reciprocal Rank Fusion (RRF) |
| `services/rerank_service.py` | Cross-attention reranking, filters chunks below 0.55 relevance threshold |
| `services/validation_service.py` | Self-correcting layer — cross-references LLM output against retrieved chunks to detect/fix hallucinations |
| `agents/chat_tools/retrieval.py` | Intent-aware top_k adjustment and document retrieval orchestration |
| `agents/chat_tools/query_planner.py` | Transforms user messages into optimised retrieval queries for doc + web search |
| `agents/note_composer_agent.py` | Consumes RAG context, emits structured multimodal study notes (text + diagrams + images) with citations |
| `agents/research_note_graph.py` | LangGraph topology: classifies topic sensitivity (medical/legal/finance) and triggers external web search when RAG confidence is low |
| `embedding-space/cognify-embedding/app.py` | Standalone HuggingFace Space — BAAI/bge-base-en-v1.5, 768-dim embeddings |

### Pipeline Flow

```mermaid
sequenceDiagram
    participant U as User Query
    participant QP as query_planner
    participant R as rag_service
    participant ES as Embedding MS
    participant DB as pgvector DB
    participant RR as rerank_service
    participant NC as note_composer_agent
    participant V as validation_service

    U->>QP: Raw user message
    QP-->>R: Optimised retrieval query
    R->>ES: Embed query (remote HF Space)
    ES-->>R: 768-dim dense vector
    R->>DB: Dense similarity search (pgvector)
    R->>DB: Sparse keyword search (tsvector)
    DB-->>R: Top-N chunks (both methods)
    R->>R: Reciprocal Rank Fusion merge
    R->>RR: Re-rank merged chunks
    RR-->>R: Top-K high-fidelity chunks (≥0.55)
    R->>NC: Chunks + topic metadata
    NC->>V: Draft multimodal notes
    V-->>U: Hallucination-checked, cited output
```

### Document Ingestion Flow

```mermaid
flowchart LR
    PDF[PDF Upload] --> OCR[Text Extraction]
    OCR --> Chunk[chunking_service\nAST Hierarchical Split]
    Chunk --> Breadcrumb[Breadcrumb Enrichment\nHeading Path]
    Breadcrumb --> Embed[embedding_service\nRemote HF Space]
    Embed --> Store[(pgvector\n768-dim chunks)]
```

### Research Note Graph (Sensitivity-Aware)

```mermaid
flowchart TD
    Topic[Topic Input] --> Classify{research_note_graph\nSensitivity Classifier}
    Classify -- low / general_education --> RAGOnly[RAG Context Only]
    Classify -- medium / high\nmedical, legal, finance --> WebTrigger[External Search\nDuckDuckGo + Wikipedia]
    WebTrigger --> Merge[Merge RAG + Web Context]
    RAGOnly --> Compose[note_composer_agent]
    Merge --> Compose
    Compose --> Notes[Structured Study Notes\nwith Citations]
```

---

## 2. LangGraph Agent Orchestration Module

**What it does:** Routes every user message to the correct specialised agent pipeline via a LangGraph StateGraph with 9 classified intents.

### Contains

| File | Role |
|------|------|
| `services/agent_service.py` | Root LangGraph StateGraph — intent detection, node routing, state management |
| `agents/chat_tools/router.py` | Classifies user messages into 9 intents |
| `agents/chat_tools/normal.py` | Handles greetings, small-talk, app help — no RAG involved |
| `agents/chat_tools/tutoring.py` | Educational explanations with grounded RAG context |
| `agents/chat_tools/quiz.py` | Quiz generation with calibrated difficulty |
| `agents/chat_tools/visual.py` | Diagram and visual explanation generation |
| `agents/chat_tools/web_search.py` | DuckDuckGo external knowledge search |
| `agents/chat_tools/image_search.py` | Image retrieval for visual references |
| `agents/chat_tools/mcp.py` | MCP connectors — calendar, Drive, Docs, email |
| `agents/chat_tools/analysis.py` | Misconception analysis and targeted feedback on wrong quiz answers |

### Agent Routing Flow

```mermaid
flowchart TD
    Msg[User Message] --> Router[router.py\nIntent Classifier]

    Router -- normal --> Normal[normal.py\nGreetings / Help]
    Router -- study --> Tutor[tutoring.py\nRAG-grounded Explanation]
    Router -- quiz --> Quiz[quiz.py\nAdaptive Quiz Generation]
    Router -- analyze --> Analyze[analysis.py\nMisconception Feedback]
    Router -- visual --> Visual[visual.py\nDiagram Generation]
    Router -- web_search --> WebS[web_search.py\nDuckDuckGo]
    Router -- image_search --> ImgS[image_search.py\nImage Retrieval]
    Router -- mcp_action --> MCP[mcp.py\nCalendar / Drive / Docs]
    Router -- multi_tool --> Multi[Multi-node\nComposed Pipeline]

    Tutor --> RAG[rag_service]
    Quiz --> RAG
    Analyze --> RAG
    Visual --> VisualPipeline[visual_tools pipeline]

    RAG --> LLM[llm_service\nGemini]
    VisualPipeline --> LLM
    Normal --> LLM
```

---

## 3. Voice & Auditory Support Module

**What it does:** Provides real-time conversational learning via WebRTC, text-to-speech for study notes, and low-latency voice chat with a streaming AI tutor.

### Contains

| File | Role |
|------|------|
| `api/endpoints/audio.py` | `/audio/query` and `/audio/query/stream` — voice-optimised endpoints with short, no-markdown, conversational responses |
| `core/config.py` | LiveKit URL, API key, and API secret configuration |
| `dashboard/auditory/page.tsx` | Frontend voice tutor UI — microphone input, real-time AI response, playback |
| Browser `window.speechSynthesis` | Native TTS — no backend ML models required; keeps backend serverless-compatible |
| LiveKit WebRTC | Ultra-low-latency real-time audio streaming infrastructure |

### Voice Architecture

```mermaid
sequenceDiagram
    participant L as Learner (Browser)
    participant FE as auditory/page.tsx
    participant LK as LiveKit WebRTC
    participant API as audio.py
    participant LLM as llm_service (Gemini)
    participant TTS as window.speechSynthesis

    L->>FE: Speaks (microphone)
    FE->>LK: WebRTC audio stream
    LK->>API: Transcribed text
    API->>LLM: Auditory system prompt\n(short, no markdown, conversational)
    LLM-->>API: Streaming text response
    API-->>FE: SSE token stream
    FE->>TTS: Text chunks
    TTS-->>L: Spoken audio response
```

### Why No Backend TTS/STT Models

```mermaid
flowchart LR
    subgraph Removed
        Torch[PyTorch / Audio8\nHeavy ML Models]
    end
    subgraph Used
        LiveKit[LiveKit WebRTC\nLow-latency streaming]
        BrowserTTS[Browser speechSynthesis\nNative, zero-latency]
    end
    Removed -. replaced by .-> Used
```

> **Design decision:** Heavy local ML models (torch, transformers) were removed to keep the backend serverless-friendly and horizontally scalable. The browser's native `speechSynthesis` API provides resilient, zero-dependency TTS.

---

## 4. Visual & Diagram Support Module

**What it does:** Translates text concepts into interactive visual structures — mind maps, flowcharts, and canvases — rendered live in the browser.

### Contains

| File | Role |
|------|------|
| `agents/visual_tools/planner.py` | Plans diagram structure and node hierarchy from a text description |
| `agents/visual_tools/validator.py` | Validates diagram AST; provides fallback plan if LLM output is malformed |
| `agents/visual_tools/layout.py` | Force-directed Dagre layout — positions nodes and edges on canvas |
| `agents/visual_tools/pipeline.py` | Orchestrates: plan → validate → layout in sequence |
| `agents/visual_tools/schemas.py` | Pydantic models for nodes, edges, and styles |
| `agents/visual_tools/icon_manifest.py` | Shape and icon definitions for rendering |
| `services/layout_solver.py` | Dagre graph layout solver (backend wrapper) |
| `api/endpoints/visual.py` | `/visual/explain` and `/visual/diagram` REST endpoints |
| `components/DiagramRenderer.tsx` | Mermaid.js React wrapper — renders AST as interactive canvas |
| `dashboard/visual/page.tsx` | Visual learner dashboard UI |

### Visual Generation Pipeline

```mermaid
flowchart LR
    Q[User Query] --> Planner[visual_tools/planner\nDiagram Plan]
    Planner --> Validator{visual_tools/validator\nAST Valid?}
    Validator -- yes --> Layout[visual_tools/layout\nDagre Positioning]
    Validator -- no --> Fallback[Fallback Plan]
    Fallback --> Layout
    Layout --> Schema[schemas.py\nNodes + Edges + Styles]
    Schema --> FE[DiagramRenderer.tsx\nMermaid.js Canvas]
    FE --> User([Interactive Diagram])
```

---

## 5. Adaptive Learning Services Module

**What it does:** Tracks each learner's evolving preferences across 10 dimensions, plans personalised lessons, and generates calibrated multimodal content.

### Contains

| File | Role |
|------|------|
| `services/learner_profile_service.py` | EMA-based dynamic learner model — 10 preference dimensions updated after each session |
| `services/lesson_planner_service.py` | Transforms learner profile into a concrete lesson plan (modality mix + difficulty + quiz count) |
| `services/content_engine_service.py` | Generates the full adaptive lesson payload: text, diagrams, examples, analogies, MCQs |
| `services/context_service.py` | Aggregates context from RAG output, learner history, and topic metadata |
| `api/endpoints/learning.py` | `/learning/profile`, `/learning/onboarding/*`, `/learning/lesson/*`, `/learning/generate-quiz`, `/learning/analyze-answer` |
| `agents/quiz_tools/validator.py` | Validates quiz question format and answer correctness |

### Adaptive Learning Loop

```mermaid
flowchart TD
    OB[Onboarding\nModality Detection] --> Profile[learner_profile_service\n10-dim EMA Model]

    Profile --> LP[lesson_planner_service\nModality Mix + Difficulty]
    LP --> CE[content_engine_service]

    CE --> TXT[Text Explanation\nGemini LLM]
    CE --> DGM[Diagram Section\nVisual Pipeline]
    CE --> EX[Examples + Analogies]
    CE --> QZ[Quiz MCQs\nCalibrated Difficulty]

    TXT & DGM & EX & QZ --> Lesson[Multimodal Lesson\nDelivered via SSE]

    Lesson --> Learner([Learner])
    Learner -- Score + Response Time --> Update[Profile EMA Update\nnew = old×0.7 + signal×0.3]
    Update --> Profile
```

### Learner Profile Dimensions

```mermaid
mindmap
  root((Learner Profile))
    Modality
      visual
      textual
      auditory
      code
    Content Style
      examples
      analogies
      repetition
    Difficulty
      skill_level
      difficulty_preference
    Pace
      pace_classification
        slow
        medium
        fast
```

---

## 6. Frontend Application Modules

**What it does:** Next.js 15 app partitioned into distinct learning modalities, real-time streaming views, and utility pages.

### Page Map

```mermaid
flowchart TD
    Landing[app/page.tsx\nMarketing / Landing]
    Landing --> Login[auth/login\nSupabase Auth]
    Login --> DB[dashboard/layout.tsx\nSidebar + Providers]

    DB --> Learn[/learn\nRAG Notes + SSE Stream + Citations]
    DB --> Chat[/chat\nDocument Chat Interface]
    DB --> Visual[/visual\nDiagram Canvas]
    DB --> Auditory[/auditory\nVoice Tutor + WebRTC]
    DB --> Quiz[/quiz\nAdaptive Testing]
    DB --> Upload[/upload\nPDF Ingestion]
    DB --> Analytics[/analytics\nProgress + Modality Stats]
    DB --> Profile[/profile\nLearner Preferences]
    DB --> Onboarding[/onboarding\nModality Setup Flow]
    DB --> Search[/search\nGlobal Document Search]
```

### Data Flow: Learn Page (SSE Streaming)

```mermaid
sequenceDiagram
    participant FE as /dashboard/learn
    participant API as FastAPI SSE
    participant Agent as note_composer_agent
    participant RAG as rag_service

    FE->>API: POST /learning/lesson (topic)
    API->>RAG: Retrieve context chunks
    RAG-->>Agent: High-fidelity chunks
    Agent-->>API: Structured sections (text/diagram/image)
    API-->>FE: SSE token stream
    FE->>FE: Render sections live\n(text, Mermaid diagrams, cited refs)
```

---

## 7. Backend Infrastructure & Deployment

**What it does:** Stateless FastAPI backend deployable to Vercel or AWS Lambda — no local ML models, all state in Supabase.

### Contains

| File | Role |
|------|------|
| `main.py` | FastAPI app init — CORS, router registration, `/health` endpoint |
| `core/config.py` | Pydantic Settings — Gemini, Supabase, LiveKit, HF embedding service, Brave API |
| `core/auth.py` | JWT validation and Supabase token → user ID extraction |
| `core/prompts.py` | Shared LLM system prompts across the platform |
| `db/database.py` | SQLAlchemy session management + Supabase connection |
| `models/db_models.py` | ORM models: Document (768-dim pgvector), User, QuizSession, QuizAttempt |
| `services/llm_service.py` | Gemini API wrapper — streaming, JSON parsing, multi-turn history |
| `Dockerfile` | Container image for FastAPI + uvicorn |
| `supabase_migration.sql` / `_v2.sql` | Database schema + pgvector extension setup |

### Infrastructure Stack

```mermaid
flowchart LR
    subgraph Frontend
        Next[Next.js 15\nVercel Edge]
        Tailwind[TailwindCSS]
        Mermaid[Mermaid.js]
    end

    subgraph Backend
        FAPI[FastAPI\nuvicorn]
        LangG[LangGraph\nStateGraph]
        Gemini[Gemini LLM\ncloud API]
    end

    subgraph Data
        SB[(Supabase\nPostgreSQL)]
        PGV[pgvector\n768-dim]
        SBAuth[Supabase Auth\nJWT]
    end

    subgraph External
        HF[HuggingFace Space\nBAAI/bge-base-en-v1.5]
        LK[LiveKit\nWebRTC]
        DDG[DuckDuckGo\nBrave Search]
    end

    Next -- REST/SSE --> FAPI
    Next -- WebRTC --> LK
    FAPI --> LangG
    FAPI --> SB
    FAPI --> Gemini
    FAPI --> HF
    LangG --> DDG
```

### Stateless Scaling Model

```mermaid
flowchart LR
    LB[Load Balancer] --> I1[FastAPI Instance 1]
    LB --> I2[FastAPI Instance 2]
    LB --> I3[FastAPI Instance N]

    I1 & I2 & I3 --> SB[(Supabase\nAll State Stored Here)]
    I1 & I2 & I3 --> HF[HF Embedding Space]
    I1 & I2 & I3 --> Gemini[Gemini API]
```

> All conversation history, document states, learner profiles, and vector embeddings live in Supabase. FastAPI instances carry zero local state, enabling seamless horizontal scaling.

---

## Full End-to-End Request Flow

```mermaid
sequenceDiagram
    participant U as Learner
    participant FE as Next.js
    participant API as FastAPI
    participant Router as Intent Router
    participant RAG as RAG Pipeline
    participant LLM as Gemini
    participant DB as Supabase

    U->>FE: Sends message / voice input
    FE->>API: POST /agent/message (JWT auth)
    API->>Router: Classify intent (9 intents)
    Router->>RAG: Retrieve relevant chunks
    RAG->>DB: Hybrid vector + keyword search
    DB-->>RAG: Top-N chunks
    RAG->>RAG: RRF merge + rerank (≥0.55)
    RAG-->>LLM: Chunks + system prompt
    LLM-->>API: Streaming response tokens
    API-->>FE: SSE stream
    FE-->>U: Live-rendered response\n(text, diagrams, citations)
    API->>DB: Persist session + update learner profile
```