ARCHITECTURE.md

CognifyAI — System Architecture

CognifyAI is a multimodal adaptive learning platform. It ingests raw knowledge (PDFs, web data), dynamically generates personalized learning materials across Visual, Textual, and Auditory modalities, and orchestrates complex RAG workflows backed by a LangGraph agent engine.

---

System Overview

graph TD
    User([Learner]) --> FE[Next.js Frontend]

    FE -- REST / SSE --> API[FastAPI Backend]
    FE -- WebRTC --> LK[LiveKit Voice Server]
    FE -- Auth --> SB_AUTH[Supabase Auth]

    API --> AgentSvc[Agent Service\nLangGraph StateGraph]
    API --> RAGSvc[RAG Pipeline]
    API --> LearnSvc[Learning Services]
    API --> DB[(Supabase\nPostgreSQL + pgvector)]

    AgentSvc --> Router[Intent Router\n9 intents]
    Router --> Tools[Specialized Tool Nodes]
    Tools --> RAGSvc
    Tools --> LLMSvc[Gemini LLM Service]
    Tools --> WebSearch[DuckDuckGo / Brave]

    RAGSvc --> EmbedMS[HuggingFace\nEmbedding Microservice\nBAAI/bge-base-en-v1.5]
    RAGSvc --> DB

    LearnSvc --> ProfileSvc[Learner Profile\nEMA Engine]
    LearnSvc --> LessonSvc[Lesson Planner]
    LearnSvc --> ContentEng[Content Engine]

---

1. Enterprise RAG Module

What it does: Ingests documents, retrieves highly relevant context via hybrid search, and validates AI answers to prevent hallucinations.

Contains

File	Role
services/chunking_service.py	Hierarchical AST-based PDF chunking with breadcrumb enrichment; preserves heading parent-child relationships
services/embedding_service.py	Remote client to HuggingFace embedding microservice — no local torch/transformers
services/rag_service.py	Hybrid Retrieval (dense + sparse) merged via Reciprocal Rank Fusion (RRF)
services/rerank_service.py	Cross-attention reranking, filters chunks below 0.55 relevance threshold
services/validation_service.py	Self-correcting layer — cross-references LLM output against retrieved chunks to detect/fix hallucinations
agents/chat_tools/retrieval.py	Intent-aware top_k adjustment and document retrieval orchestration
agents/chat_tools/query_planner.py	Transforms user messages into optimised retrieval queries for doc + web search
agents/note_composer_agent.py	Consumes RAG context, emits structured multimodal study notes (text + diagrams + images) with citations
agents/research_note_graph.py	LangGraph topology: classifies topic sensitivity (medical/legal/finance) and triggers external web search when RAG confidence is low
embedding-space/cognify-embedding/app.py	Standalone HuggingFace Space — BAAI/bge-base-en-v1.5, 768-dim embeddings

Pipeline Flow

sequenceDiagram
    participant U as User Query
    participant QP as query_planner
    participant R as rag_service
    participant ES as Embedding MS
    participant DB as pgvector DB
    participant RR as rerank_service
    participant NC as note_composer_agent
    participant V as validation_service

    U->>QP: Raw user message
    QP-->>R: Optimised retrieval query
    R->>ES: Embed query (remote HF Space)
    ES-->>R: 768-dim dense vector
    R->>DB: Dense similarity search (pgvector)
    R->>DB: Sparse keyword search (tsvector)
    DB-->>R: Top-N chunks (both methods)
    R->>R: Reciprocal Rank Fusion merge
    R->>RR: Re-rank merged chunks
    RR-->>R: Top-K high-fidelity chunks (≥0.55)
    R->>NC: Chunks + topic metadata
    NC->>V: Draft multimodal notes
    V-->>U: Hallucination-checked, cited output

Document Ingestion Flow

flowchart LR
    PDF[PDF Upload] --> OCR[Text Extraction]
    OCR --> Chunk[chunking_service\nAST Hierarchical Split]
    Chunk --> Breadcrumb[Breadcrumb Enrichment\nHeading Path]
    Breadcrumb --> Embed[embedding_service\nRemote HF Space]
    Embed --> Store[(pgvector\n768-dim chunks)]

Research Note Graph (Sensitivity-Aware)

flowchart TD
    Topic[Topic Input] --> Classify{research_note_graph\nSensitivity Classifier}
    Classify -- low / general_education --> RAGOnly[RAG Context Only]
    Classify -- medium / high\nmedical, legal, finance --> WebTrigger[External Search\nDuckDuckGo + Wikipedia]
    WebTrigger --> Merge[Merge RAG + Web Context]
    RAGOnly --> Compose[note_composer_agent]
    Merge --> Compose
    Compose --> Notes[Structured Study Notes\nwith Citations]

---

2. LangGraph Agent Orchestration Module

What it does: Routes every user message to the correct specialised agent pipeline via a LangGraph StateGraph with 9 classified intents.

Contains

File	Role
services/agent_service.py	Root LangGraph StateGraph — intent detection, node routing, state management
agents/chat_tools/router.py	Classifies user messages into 9 intents
agents/chat_tools/normal.py	Handles greetings, small-talk, app help — no RAG involved
agents/chat_tools/tutoring.py	Educational explanations with grounded RAG context
agents/chat_tools/quiz.py	Quiz generation with calibrated difficulty
agents/chat_tools/visual.py	Diagram and visual explanation generation
agents/chat_tools/web_search.py	DuckDuckGo external knowledge search
agents/chat_tools/image_search.py	Image retrieval for visual references
agents/chat_tools/mcp.py	MCP connectors — calendar, Drive, Docs, email
agents/chat_tools/analysis.py	Misconception analysis and targeted feedback on wrong quiz answers

Agent Routing Flow

flowchart TD
    Msg[User Message] --> Router[router.py\nIntent Classifier]

    Router -- normal --> Normal[normal.py\nGreetings / Help]
    Router -- study --> Tutor[tutoring.py\nRAG-grounded Explanation]
    Router -- quiz --> Quiz[quiz.py\nAdaptive Quiz Generation]
    Router -- analyze --> Analyze[analysis.py\nMisconception Feedback]
    Router -- visual --> Visual[visual.py\nDiagram Generation]
    Router -- web_search --> WebS[web_search.py\nDuckDuckGo]
    Router -- image_search --> ImgS[image_search.py\nImage Retrieval]
    Router -- mcp_action --> MCP[mcp.py\nCalendar / Drive / Docs]
    Router -- multi_tool --> Multi[Multi-node\nComposed Pipeline]

    Tutor --> RAG[rag_service]
    Quiz --> RAG
    Analyze --> RAG
    Visual --> VisualPipeline[visual_tools pipeline]

    RAG --> LLM[llm_service\nGemini]
    VisualPipeline --> LLM
    Normal --> LLM

---

3. Voice & Auditory Support Module

What it does: Provides real-time conversational learning via WebRTC, text-to-speech for study notes, and low-latency voice chat with a streaming AI tutor.

Contains

File	Role
api/endpoints/audio.py	/audio/query and /audio/query/stream — voice-optimised endpoints with short, no-markdown, conversational responses
core/config.py	LiveKit URL, API key, and API secret configuration
dashboard/auditory/page.tsx	Frontend voice tutor UI — microphone input, real-time AI response, playback
Browser window.speechSynthesis	Native TTS — no backend ML models required; keeps backend serverless-compatible
LiveKit WebRTC	Ultra-low-latency real-time audio streaming infrastructure

Voice Architecture

sequenceDiagram
    participant L as Learner (Browser)
    participant FE as auditory/page.tsx
    participant LK as LiveKit WebRTC
    participant API as audio.py
    participant LLM as llm_service (Gemini)
    participant TTS as window.speechSynthesis

    L->>FE: Speaks (microphone)
    FE->>LK: WebRTC audio stream
    LK->>API: Transcribed text
    API->>LLM: Auditory system prompt\n(short, no markdown, conversational)
    LLM-->>API: Streaming text response
    API-->>FE: SSE token stream
    FE->>TTS: Text chunks
    TTS-->>L: Spoken audio response

Why No Backend TTS/STT Models

flowchart LR
    subgraph Removed
        Torch[PyTorch / Audio8\nHeavy ML Models]
    end
    subgraph Used
        LiveKit[LiveKit WebRTC\nLow-latency streaming]
        BrowserTTS[Browser speechSynthesis\nNative, zero-latency]
    end
    Removed -. replaced by .-> Used

Design decision: Heavy local ML models (torch, transformers) were removed to keep the backend serverless-friendly and horizontally scalable. The browser's native speechSynthesis API provides resilient, zero-dependency TTS.

---

4. Visual & Diagram Support Module

What it does: Translates text concepts into interactive visual structures — mind maps, flowcharts, and canvases — rendered live in the browser.

Contains

File	Role
agents/visual_tools/planner.py	Plans diagram structure and node hierarchy from a text description
agents/visual_tools/validator.py	Validates diagram AST; provides fallback plan if LLM output is malformed
agents/visual_tools/layout.py	Force-directed Dagre layout — positions nodes and edges on canvas
agents/visual_tools/pipeline.py	Orchestrates: plan → validate → layout in sequence
agents/visual_tools/schemas.py	Pydantic models for nodes, edges, and styles
agents/visual_tools/icon_manifest.py	Shape and icon definitions for rendering
services/layout_solver.py	Dagre graph layout solver (backend wrapper)
api/endpoints/visual.py	/visual/explain and /visual/diagram REST endpoints
components/DiagramRenderer.tsx	Mermaid.js React wrapper — renders AST as interactive canvas
dashboard/visual/page.tsx	Visual learner dashboard UI

Visual Generation Pipeline

flowchart LR
    Q[User Query] --> Planner[visual_tools/planner\nDiagram Plan]
    Planner --> Validator{visual_tools/validator\nAST Valid?}
    Validator -- yes --> Layout[visual_tools/layout\nDagre Positioning]
    Validator -- no --> Fallback[Fallback Plan]
    Fallback --> Layout
    Layout --> Schema[schemas.py\nNodes + Edges + Styles]
    Schema --> FE[DiagramRenderer.tsx\nMermaid.js Canvas]
    FE --> User([Interactive Diagram])

---

5. Adaptive Learning Services Module

What it does: Tracks each learner's evolving preferences across 10 dimensions, plans personalised lessons, and generates calibrated multimodal content.

Contains

File	Role
services/learner_profile_service.py	EMA-based dynamic learner model — 10 preference dimensions updated after each session
services/lesson_planner_service.py	Transforms learner profile into a concrete lesson plan (modality mix + difficulty + quiz count)
services/content_engine_service.py	Generates the full adaptive lesson payload: text, diagrams, examples, analogies, MCQs
services/context_service.py	Aggregates context from RAG output, learner history, and topic metadata
api/endpoints/learning.py	/learning/profile, /learning/onboarding/*, /learning/lesson/*, /learning/generate-quiz, /learning/analyze-answer
agents/quiz_tools/validator.py	Validates quiz question format and answer correctness

Adaptive Learning Loop

flowchart TD
    OB[Onboarding\nModality Detection] --> Profile[learner_profile_service\n10-dim EMA Model]

    Profile --> LP[lesson_planner_service\nModality Mix + Difficulty]
    LP --> CE[content_engine_service]

    CE --> TXT[Text Explanation\nGemini LLM]
    CE --> DGM[Diagram Section\nVisual Pipeline]
    CE --> EX[Examples + Analogies]
    CE --> QZ[Quiz MCQs\nCalibrated Difficulty]

    TXT & DGM & EX & QZ --> Lesson[Multimodal Lesson\nDelivered via SSE]

    Lesson --> Learner([Learner])
    Learner -- Score + Response Time --> Update[Profile EMA Update\nnew = old×0.7 + signal×0.3]
    Update --> Profile

Learner Profile Dimensions

mindmap
  root((Learner Profile))
    Modality
      visual
      textual
      auditory
      code
    Content Style
      examples
      analogies
      repetition
    Difficulty
      skill_level
      difficulty_preference
    Pace
      pace_classification
        slow
        medium
        fast

---

6. Frontend Application Modules

What it does: Next.js 15 app partitioned into distinct learning modalities, real-time streaming views, and utility pages.

Page Map

flowchart TD
    Landing[app/page.tsx\nMarketing / Landing]
    Landing --> Login[auth/login\nSupabase Auth]
    Login --> DB[dashboard/layout.tsx\nSidebar + Providers]

    DB --> Learn[/learn\nRAG Notes + SSE Stream + Citations]
    DB --> Chat[/chat\nDocument Chat Interface]
    DB --> Visual[/visual\nDiagram Canvas]
    DB --> Auditory[/auditory\nVoice Tutor + WebRTC]
    DB --> Quiz[/quiz\nAdaptive Testing]
    DB --> Upload[/upload\nPDF Ingestion]
    DB --> Analytics[/analytics\nProgress + Modality Stats]
    DB --> Profile[/profile\nLearner Preferences]
    DB --> Onboarding[/onboarding\nModality Setup Flow]
    DB --> Search[/search\nGlobal Document Search]

Data Flow: Learn Page (SSE Streaming)

sequenceDiagram
    participant FE as /dashboard/learn
    participant API as FastAPI SSE
    participant Agent as note_composer_agent
    participant RAG as rag_service

    FE->>API: POST /learning/lesson (topic)
    API->>RAG: Retrieve context chunks
    RAG-->>Agent: High-fidelity chunks
    Agent-->>API: Structured sections (text/diagram/image)
    API-->>FE: SSE token stream
    FE->>FE: Render sections live\n(text, Mermaid diagrams, cited refs)

---

7. Backend Infrastructure & Deployment

What it does: Stateless FastAPI backend deployable to Vercel or AWS Lambda — no local ML models, all state in Supabase.

Contains

File	Role
main.py	FastAPI app init — CORS, router registration, /health endpoint
core/config.py	Pydantic Settings — Gemini, Supabase, LiveKit, HF embedding service, Brave API
core/auth.py	JWT validation and Supabase token → user ID extraction
core/prompts.py	Shared LLM system prompts across the platform
db/database.py	SQLAlchemy session management + Supabase connection
models/db_models.py	ORM models: Document (768-dim pgvector), User, QuizSession, QuizAttempt
services/llm_service.py	Gemini API wrapper — streaming, JSON parsing, multi-turn history
Dockerfile	Container image for FastAPI + uvicorn
supabase_migration.sql / _v2.sql	Database schema + pgvector extension setup

Infrastructure Stack

flowchart LR
    subgraph Frontend
        Next[Next.js 15\nVercel Edge]
        Tailwind[TailwindCSS]
        Mermaid[Mermaid.js]
    end

    subgraph Backend
        FAPI[FastAPI\nuvicorn]
        LangG[LangGraph\nStateGraph]
        Gemini[Gemini LLM\ncloud API]
    end

    subgraph Data
        SB[(Supabase\nPostgreSQL)]
        PGV[pgvector\n768-dim]
        SBAuth[Supabase Auth\nJWT]
    end

    subgraph External
        HF[HuggingFace Space\nBAAI/bge-base-en-v1.5]
        LK[LiveKit\nWebRTC]
        DDG[DuckDuckGo\nBrave Search]
    end

    Next -- REST/SSE --> FAPI
    Next -- WebRTC --> LK
    FAPI --> LangG
    FAPI --> SB
    FAPI --> Gemini
    FAPI --> HF
    LangG --> DDG

Stateless Scaling Model

flowchart LR
    LB[Load Balancer] --> I1[FastAPI Instance 1]
    LB --> I2[FastAPI Instance 2]
    LB --> I3[FastAPI Instance N]

    I1 & I2 & I3 --> SB[(Supabase\nAll State Stored Here)]
    I1 & I2 & I3 --> HF[HF Embedding Space]
    I1 & I2 & I3 --> Gemini[Gemini API]

All conversation history, document states, learner profiles, and vector embeddings live in Supabase. FastAPI instances carry zero local state, enabling seamless horizontal scaling.

---

Full End-to-End Request Flow

sequenceDiagram
    participant U as Learner
    participant FE as Next.js
    participant API as FastAPI
    participant Router as Intent Router
    participant RAG as RAG Pipeline
    participant LLM as Gemini
    participant DB as Supabase

    U->>FE: Sends message / voice input
    FE->>API: POST /agent/message (JWT auth)
    API->>Router: Classify intent (9 intents)
    Router->>RAG: Retrieve relevant chunks
    RAG->>DB: Hybrid vector + keyword search
    DB-->>RAG: Top-N chunks
    RAG->>RAG: RRF merge + rerank (≥0.55)
    RAG-->>LLM: Chunks + system prompt
    LLM-->>API: Streaming response tokens
    API-->>FE: SSE stream
    FE-->>U: Live-rendered response\n(text, diagrams, citations)
    API->>DB: Persist session + update learner profile