ARCHITECTURE.md

# ContextFlow Architecture: Complete System Overview

## Table of Contents
1. [System Vision](#1-system-vision)
2. [High-Level Architecture](#2-high-level-architecture)
3. [Frontend Layer](#3-frontend-layer)
4. [Backend Layer](#4-backend-layer)
5. [Agent Network](#5-agent-network)
6. [Reinforcement Learning Pipeline](#6-reinforcement-learning-pipeline)
7. [Data Flow](#7-data-flow)
8. [API Design](#8-api-design)
9. [Multi-Modal Detection](#9-multi-modal-detection)
10. [Privacy & Security](#10-privacy--security)
11. [Deployment Architecture](#11-deployment-architecture)

---

## 1. System Vision

**ContextFlow** is an AI-powered learning intelligence engine that predicts when learners will get confused BEFORE it happens, enabling proactive intervention in educational settings.

### Core Problem Solved
- Traditional learning systems are **reactive** - they respond after confusion occurs
- ContextFlow is **proactive** - it predicts confusion and intervenes before disengagement

### Key Innovations
1. **Predictive AI** - RL-based doubt prediction
2. **Gesture Control** - Hands-free learning assistance
3. **Multi-Agent Orchestration** - 9 specialized agents working in concert
4. **Privacy-First** - Face blur for classroom deployment

---

## 2. High-Level Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                          USERS                                         │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │
│  │  Students   │  │  Teachers   │  │  Researchers │               │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘               │
└─────────┼─────────────────┼─────────────────┼─────────────────────────┘
          │                 │                 │
          ▼                 ▼                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     PRESENTATION LAYER                                │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    React Frontend (Vite)                       │   │
│  │  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐       │   │
│  │  │  Learn  │ │ LLMFlow │ │Gestures │ │ Predict │ ...     │   │
│  │  │   Tab    │ │  Tab    │ │   Tab   │ │   Tab   │         │   │
│  │  └─────────┘ └─────────┘ └─────────┘ └─────────┘         │   │
│  │                                                              │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │         MediaPipe Camera Feed (Gesture + Face)       │    │   │
│  │  │    ┌──────────┐              ┌──────────┐          │    │   │
│  │  │    │ Hand     │              │ Face     │          │    │   │
│  │  │    │ Detection │              │ Blur     │          │    │   │
│  │  │    └──────────┘              └──────────┘          │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘
                                    │
                                    │ REST API (JSON)
                                    │ WebSocket (Optional)
                                    ▼
┌─────────────────────────────────────────────────────────────────────┐
│                       BACKEND LAYER (Flask)                          │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    API Gateway (Flask Blueprints)               │   │
│  │   /api/session/*  /api/predict/*  /api/gesture/*  /api/*     │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                    │                                  │
│                                    ▼                                  │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │              STUDY ORCHESTRATOR (Central Coordinator)           │   │
│  │   ┌────────────────────────────────────────────────────┐      │   │
│  │   │                   Agent Registry                    │      │   │
│  │   │  DoubtPredictor │ Behavioral │ Gesture │ Recall  │      │   │
│  │   │  KnowledgeGraph │ PeerLearn │ LLMOrch │ Prompt │      │   │
│  │   └────────────────────────────────────────────────────┘      │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                    │                                  │
│    ┌───────────────┬─────────────┼─────────────┬───────────────┐  │
│    ▼               ▼             ▼             ▼               ▼      │
│  ┌─────┐       ┌─────┐      ┌─────┐      ┌─────┐      ┌─────┐   │
│  │ Q-  │       │Behavioral│   │Gesture│    │Recall│     │LLM  │   │
│  │Network│     │Agent   │    │Agent │     │Agent │     │Orch │   │
│  └─────┘       └─────┘      └─────┘      └─────┘      └─────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         DATA LAYER                                    │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────────┐ │
│  │  Checkpoint │  │  Session   │  │  Knowledge │  │   Real     │ │
│  │  (RL Model) │  │  State     │  │  Graph     │  │   Data     │ │
│  │   .pkl      │  │  JSON      │  │  NetworkX  │  │  Collection│ │
│  └────────────┘  └────────────┘  └────────────┘  └────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 3. Frontend Layer

### 3.1 Technology Stack

| Component | Technology | Purpose |
|-----------|------------|---------|
| Framework | React 18 | UI Components |
| Build Tool | Vite | Fast development |
| Styling | Tailwind CSS | Responsive design |
| Icons | Lucide React | Consistent icons |
| Camera | MediaPipe | Hand/Face detection |

### 3.2 Application Structure

```
frontend/src/
├── App.jsx              # Main application (9 tabs)
├── main.jsx             # Entry point
├── index.css            # Global styles
├── BrowserLLMLauncher.js  # AI chat launcher
└── MediaPipeProcessor.js # Camera + gesture processing
```

### 3.3 Tab Interface

| Tab | Purpose |
|-----|---------|
| **Learn** | Dashboard with predictions, reviews, gamification |
| **LLM Flow** | Browser-based AI launcher (no API keys) |
| **Gestures** | Train custom hand gestures |
| **Predict** | RL doubt prediction visualization |
| **Behavior** | Behavioral signal tracking |
| **Peer** | Social learning insights |
| **Stats** | Learning statistics |
| **Gamify** | Fish/XP rewards system |
| **Settings** | AI provider configuration |

### 3.4 BrowserLLMLauncher.js

Opens AI chats directly in browser without API keys:

```javascript
// Opens chat.openai.com with pre-filled context
openAIChat(context, model = 'gpt-4') {
  const url = `https://chat.openai.com/?q=${encodeURIComponent(context)}`;
  window.open(url, '_blank');
}
```

### 3.5 MediaPipeProcessor.js

Handles real-time camera processing:

```
┌─────────────────┐
│   Camera Feed   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐    ┌─────────────────┐
│  Hand Landmark  │    │   Face Mesh     │
│   Detection     │    │   Detection     │
│  (21 points)   │    │  (468 points)  │
└────────┬────────┘    └────────┬────────┘
         │                       │
         ▼                       ▼
┌─────────────────┐    ┌─────────────────┐
│ Gesture         │    │ Face Blur       │
│ Recognition     │───▶│ (Privacy)       │
└────────┬────────┘    └─────────────────┘
         │
         ▼
┌─────────────────┐
│  Backend API    │
│  /api/gesture/  │
└─────────────────┘
```

---

## 4. Backend Layer

### 4.1 Technology Stack

| Component | Technology | Purpose |
|-----------|------------|---------|
| Framework | Flask | REST API |
| Async | asyncio | Non-blocking I/O |
| ML | PyTorch | RL model |
| Data | NumPy | Feature extraction |
| Graphs | NetworkX | Knowledge graphs |
| Storage | JSON/SQLite | Session persistence |

### 4.2 Flask Application Structure

```
backend/
├── run.py                    # Application entry point
├── app/
│   ├── __init__.py          # Flask app factory
│   ├── config.py            # Configuration
│   ├── api/
│   │   ├── __init__.py
│   │   └── main.py          # All API routes (889 lines)
│   └── agents/
│       ├── __init__.py
│       ├── study_orchestrator.py    # Central coordinator
│       ├── doubt_predictor.py       # RL prediction
│       ├── behavioral_agent.py      # Signal processing
│       ├── hand_gesture_agent.py    # MediaPipe integration
│       ├── recall_agent.py          # Spaced repetition
│       ├── knowledge_graph_agent.py # Concept mapping
│       ├── peer_learning_agent.py    # Social learning
│       ├── llm_orchestrator_agent.py # Multi-AI
│       ├── gesture_action_agent.py  # Gesture→Action
│       └── prompt_agent.py          # Prompt templates
```

### 4.3 Flask App Factory

```python
def create_app():
    app = Flask(__name__)
    
    # Load config
    app.config.from_object('app.config.Config')
    
    # Register blueprints
    from app.api.main import api
    app.register_blueprint(api, url_prefix='/api')
    
    # Initialize agents
    init_agents()
    
    return app
```

---

## 5. Agent Network

### 5.1 Agent Overview

```
┌─────────────────────────────────────────────────────────────┐
│                  STUDY ORCHESTRATOR                          │
│              (Central Coordinator)                             │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │   Doubt     │  │ Behavioral  │  │   Hand     │        │
│  │  Predictor  │◀─│   Agent     │─▶│  Gesture   │        │
│  │   Agent    │  │             │  │   Agent    │        │
│  └──────┬──────┘  └─────────────┘  └──────┬──────┘        │
│         │                                  │                │
│         ▼                                  ▼                │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │ Knowledge  │  │   Recall    │  │    LLM      │        │
│  │   Graph    │◀─│   Agent     │─▶│ Orchestrator│        │
│  │   Agent    │  │             │  │             │        │
│  └─────────────┘  └─────────────┘  └─────────────┘        │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐                           │
│  │    Peer    │  │   Gesture  │                           │
│  │  Learning  │  │   Action    │                           │
│  │   Agent    │  │   Mapper    │                           │
│  └─────────────┘  └─────────────┘                           │
└─────────────────────────────────────────────────────────────┘
```

### 5.2 StudyOrchestrator (Central Coordinator)

The orchestrator manages the learning lifecycle:

```python
class StudyOrchestrator:
    def __init__(self, user_id: str):
        self.user_id = user_id
        
        # Initialize all agents
        self.doubt_predictor = DoubtPredictorAgent(user_id)
        self.behavioral_agent = BehavioralAgent(user_id)
        self.gesture_agent = HandGestureAgent(user_id)
        self.recall_agent = RecallAgent(user_id)
        self.knowledge_graph = KnowledgeGraphAgent(user_id)
        self.peer_agent = PeerLearningAgent(user_id)
        
        # State management
        self.state = OrchestratorState()
```

**Session Lifecycle:**
1. **PRE_LEARNING** - Load predictions, check recalls, get peer insights
2. **ACTIVE_LEARNING** - Monitor signals, update predictions, capture doubts
3. **REVIEW** - Trigger spaced repetition, update knowledge graph
4. **POST_LEARNING** - Sync data, update gamification, generate summary

### 5.3 DoubtPredictorAgent (RL Core)

Predicts confusion before it happens:

```python
class DoubtPredictorAgent:
    def __init__(self, user_id: str, config: dict = None):
        self.user_id = user_id
        self.model = self._load_checkpoint()
        self.feature_extractor = FeatureExtractor()
    
    def predict_doubts(self, context: dict, top_k: int = 5):
        # 1. Extract 64-dim state vector
        state = self.feature_extractor.extract_state(context)
        
        # 2. Get Q-values from RL model
        q_values = self.model.predict(state)
        
        # 3. Return top-k predictions
        return self._format_predictions(q_values, top_k)
```

### 5.4 BehavioralAgent

Processes raw behavioral signals:

```python
class BehavioralSignal:
    mouse_hesitation: float      # Pause frequency
    scroll_reversals: int       # Back-and-forth
    time_on_page: float         # Seconds
    eye_tracking: Tuple[float, float]
    click_frequency: int
        
    def calculate_confusion_score(self) -> float:
        # Weighted average of signals
        weights = {
            'hesitation': 0.3,
            'reversals': 0.25,
            'time_on_page': 0.2,
            'tab_switches': 0.15,
            'back_button': 0.1
        }
        return weighted_sum(signals, weights)
```

### 5.5 HandGestureAgent

MediaPipe integration for gesture recognition:

```
Camera Frame
    │
    ▼
┌─────────────────┐
│ MediaPipe Hands │
│  (21 landmarks) │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Gesture Template│
│   Matching      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Confidence    │──▶ Recognized Gesture
│   Score (0-1)   │
└─────────────────┘
```

**Pre-built Gestures:**
| Gesture | Description |
|---------|-------------|
| pinch | Thumb + Index |
| swipe_up | 2-finger up |
| swipe_down | 2-finger down |
| swipe_right | 2-finger right |
| swipe_left | 2-finger left |
| point | Index extended |
| wave | Open palm wave |
| thumbs_up | 👍 confirmation |
| thumbs_down | 👎 rejection |
| fist | Closed hand |

### 5.6 RecallAgent

SM-2 based spaced repetition:

```python
class RecallCard:
    front: str          # Question
    back: str           # Answer
    interval: int       # Days until review
    ease_factor: float  # Difficulty (default 2.5)
    repetitions: int   # Successful reviews

def schedule_review(card: RecallCard, quality: int):
    if quality >= 3:  # Correct
        if card.repetitions == 0:
            card.interval = 1
        elif card.repetitions == 1:
            card.interval = 6
        else:
            card.interval *= card.ease_factor
        card.repetitions += 1
    else:  # Incorrect
        card.repetitions = 0
        card.interval = 1
    
    # Update ease factor
    card.ease_factor += (0.1 - (5 - quality) * (0.08 + (5 - quality) * 0.02))
    card.ease_factor = max(1.3, card.ease_factor)
```

### 5.7 KnowledgeGraphAgent

Concept mapping with NetworkX:

```python
class KnowledgeGraphAgent:
    def __init__(self, user_id: str):
        self.graph = nx.MultiDiGraph()
    
    def add_doubt_to_graph(self, doubt: dict):
        # Create node
        self.graph.add_node(
            doubt['concept'],
            type='concept',
            topic=doubt['topic'],
            timestamp=datetime.now()
        )
        
        # Connect to prerequisites
        for prereq in doubt.get('prerequisites', []):
            self.graph.add_edge(prereq, doubt['concept'], type='prerequisite')
        
        # Connect to related concepts
        for related in doubt.get('related', []):
            self.graph.add_edge(doubt['concept'], related, type='related')
    
    def find_learning_path(self, from_topic: str, to_topic: str):
        try:
            return nx.shortest_path(self.graph, from_topic, to_topic)
        except nx.NetworkXNoPath:
            return []
```

### 5.8 LLMOrchestrator

Multi-provider AI integration:

```python
class LLMOrchestrator:
    SUPPORTED_PROVIDERS = {
        'chatgpt': LLMProvider.CHATGPT,
        'gemini': LLMProvider.GEMINI,
        'claude': LLMProvider.CLAUDE,
        'deepseek': LLMProvider.DEEPSEEK,
        'ollama': LLMProvider.OLLAMA,
        'groq': LLMProvider.GROQ
    }
    
    async def query_parallel(self, request: LLMRequest):
        tasks = []
        for provider in request.providers:
            task = self._query_provider(provider, request)
            tasks.append(task)
        
        # Execute all queries concurrently
        responses = await asyncio.gather(*tasks, return_exceptions=True)
        return [r for r in responses if not isinstance(r, Exception)]
```

### 5.9 GestureActionMapper

Maps gestures to system actions:

```python
class GestureAction(Enum):
    QUERY_MULTI_LLM = "query_multi_llm"
    QUERY_CHATGPT = "query_chatgpt"
    QUERY_GEMINI = "query_gemini"
    TRIGGER_RL_LOOP = "trigger_rl_loop"
    CAPTURE_CONTENT = "capture_content"
    PAUSE_SESSION = "pause_session"
    RESUME_SESSION = "resume_session"

class GestureActionMapper:
    def __init__(self):
        self.action_rules = {
            GestureAction.QUERY_MULTI_LLM: {
                "trigger": {"finger_count": 2, "swipe": "right"}
            },
            GestureAction.PAUSE_SESSION: {
                "trigger": {"gesture": "open_palm"}
            },
            GestureAction.RESUME_SESSION: {
                "trigger": {"gesture": "thumbs_up"}
            }
        }
```

### 5.10 PeerLearningAgent

Social learning insights:

```python
class PeerLearningAgent:
    def get_peer_insights(self, topic: str):
        # Aggregate insights from "similar" students
        insights = []
        
        # Find students who learned this topic
        similar_students = self._find_similar_students(topic)
        
        for student in similar_students:
            # What confused them?
            insights.extend(student.difficult_concepts)
        
        # Return aggregated insights
        return self._aggregate_insights(insights)
```

---

## 6. Reinforcement Learning Pipeline

### 6.1 Problem Formulation

**State Space (64 dimensions):**
```
┌────────────────────────────────────────────────────────────────┐
│  Topic Embedding (32)  │ Progress │ Confusion (16) │ Gesture (14) │ Time │
│  TF-IDF of topic       │ 0.0-1.0 │ Behavioral    │ Hand        │ 0-1 │
│                        │          │ signals       │ signals     │      │
└────────────────────────────────────────────────────────────────┘
```

**Action Space (10 doubt types):**
1. `what_is_backpropagation`
2. `why_gradient_descent`
3. `how_overfitting_works`
4. `explain_regularization`
5. `what_loss_function`
6. `how_optimization_works`
7. `explain_learning_rate`
8. `what_regularization`
9. `how_batch_norm_works`
10. `explain_softmax`

**Reward Function:**
| Event | Reward |
|-------|--------|
| Correct prediction | +1.0 |
| Helpful explanation | +0.5 |
| Engagement maintained | +0.3 |
| False positive | -0.5 |
| Missed confusion | -1.0 |

### 6.2 Q-Network Architecture

```python
class QNetwork(nn.Module):
    def __init__(self, state_dim=64, action_dim=10, hidden_dim=128):
        super().__init__()
        self.fc1 = nn.Linear(state_dim, hidden_dim)  # 64 → 128
        self.fc2 = nn.Linear(hidden_dim, hidden_dim) # 128 → 128
        self.fc3 = nn.Linear(hidden_dim, action_dim) # 128 → 10
    
    def forward(self, x):
        x = F.relu(self.fc1(x))  # ReLU activation
        x = F.relu(self.fc2(x))
        return self.fc3(x)  # Q-values for each action
```

### 6.3 Training Algorithm (GRPO)

```python
class DoubtPredictionRL:
    def train(self, epochs=10, batch_size=32):
        for epoch in range(epochs):
            for batch in self.dataloader:
                # 1. Get current Q-values
                q_values = self.q_network(batch.states)
                
                # 2. Compute targets (GRPO-style)
                with torch.no_grad():
                    next_q = self.target_network(batch.next_states).max(1)[0]
                    targets = batch.rewards + self.gamma * next_q * (~batch.dones)
                
                # 3. Compute loss and update
                loss = self.loss_fn(q_values.gather(1, batch.actions), targets)
                loss.backward()
                self.optimizer.step()
            
            # 4. Update target network
            self.update_target_network()
            
            # 5. Decay epsilon (exploration)
            self.epsilon *= self.epsilon_decay
```

### 6.4 Feature Extraction

```python
class FeatureExtractor:
    STATE_DIM = 64
    
    def extract_state(self, context: dict) -> np.ndarray:
        # Topic embedding (32 dims)
        topic_emb = self._extract_topic_embedding(context['topic'])
        
        # Progress (1 dim)
        progress = np.array([context['progress']])
        
        # Confusion signals (16 dims)
        confusion = self._extract_confusion_signals(context['confusion_signals'])
        
        # Gesture signals (14 dims)
        gestures = self._extract_gesture_signals(context['gesture_signals'])
        
        # Time spent (1 dim)
        time_spent = np.array([context['time_spent'] / 1800])
        
        # Concatenate
        return np.concatenate([topic_emb, progress, confusion, gestures, time_spent])
```

---

## 7. Data Flow

### 7.1 Learning Session Flow

```
┌─────────────────────────────────────────────────────────────────┐
│                        USER STARTS SESSION                         │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                     ORCHESTRATOR.START_SESSION()                   │
│  1. Create new LearningSession                                    │
│  2. Load RL model checkpoint                                      │
│  3. Build learning context                                       │
└─────────────────────────────────────────────────────────────────┘
                                │
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
        ┌───────────┐   ┌───────────┐   ┌───────────┐
        │   Doubt   │   │ Behavioral│   │   Peer    │
        │ Predictor │   │   Agent   │   │  Learning │
        │           │   │           │   │   Agent   │
        │  Predict  │   │  Analyze  │   │   Get     │
        │  doubts   │   │  signals  │   │  insights │
        └─────┬─────┘   └─────┬─────┘   └─────┬─────┘
              │               │               │
              └───────────────┼───────────────┘
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                  RETURN INITIAL PREDICTIONS                        │
│  - Top 5 predicted doubts                                         │
│  - Pending reviews                                               │
│  - Peer insights                                                 │
└─────────────────────────────────────────────────────────────────┘
```

### 7.2 Behavioral Signal Flow

```
┌─────────────────────────────────────────────────────────────────┐
│                      REAL-TIME SIGNALS                           │
│                                                                  │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐             │
│  │  Mouse  │  │ Scroll  │  │Gesture  │  │  Time   │             │
│  │Movement │  │ Pattern │  │Camera   │  │  On     │             │
│  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘             │
└───────┼───────────┼───────────┼───────────┼───────────────────────┘
        │           │           │           │
        └───────────┴─────┬─────┴───────────┘
                          ▼
              ┌───────────────────────┐
              │   BEHAVIORAL AGENT     │
              │                       │
              │ calculate_confusion_   │
              │ score(signals)        │
              │                       │
              │ Returns: 0.0 - 1.0   │
              └───────────┬───────────┘
                          │
                          ▼
              ┌───────────────────────┐
              │   DOUBT PREDICTOR     │
              │                       │
              │ If score > 0.5:       │
              │   Re-predict doubts    │
              │   Trigger intervention│
              │                       │
              └───────────────────────┘
```

### 7.3 Gesture-to-Action Flow

```
┌─────────────────────────────────────────────────────────────────┐
│                         CAMERA FRAME                               │
└─────────────────────────────┬───────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      MEDIAPIPE PROCESSING                         │
│                                                                  │
│  ┌──────────────────────┐     ┌──────────────────────┐          │
│  │   Hand Landmark      │     │    Face Mesh         │          │
│  │   Detection         │     │    (468 points)      │          │
│  │   (21 points)       │     │                       │          │
│  └──────────┬─────────┘     └──────────┬───────────┘          │
└─────────────┼───────────────────────────┼───────────────────────┘
              │                           │
              ▼                           ▼
┌──────────────────────┐     ┌──────────────────────┐
│  GESTURE TEMPLATE   │     │    FACE BLUR         │
│  MATCHING           │     │    (Privacy)          │
│                     │     │                       │
│  Compare landmarks  │     │  Blur regions with    │
│  to known gestures  │     │  facial keypoints     │
└──────────┬─────────┘     └───────────────────────┘
           │
           ▼
┌──────────────────────┐
│  GESTURE RECOGNIZED  │──▶ Backend /api/gesture/recognize
│                      │
│  {                   │
│    "gesture": "pinch",│
│    "confidence": 0.92│
│  }                   │
└──────────────────────┘
           │
           ▼
┌──────────────────────┐
│ GESTURE ACTION MAPPER │
│                      │
│ pinch ──────────────▶│ TRIGGER_AI_HELP
│ swipe_right ────────▶│ LAUNCH_BROWSER_CHAT
│ open_palm ──────────▶│ PAUSE_SESSION
│ thumbs_up ──────────▶│ MARK_UNDERSTOOD
└──────────────────────┘
```

---

## 8. API Design

### 8.1 API Structure

| Category | Endpoints |
|----------|-----------|
| Session | `/session/start`, `/session/update`, `/session/end`, `/session/insights` |
| Prediction | `/predict/doubts`, `/recommendations` |
| Behavior | `/behavior/track`, `/behavior/heatmap` |
| Graph | `/graph/add`, `/graph/query`, `/graph/path` |
| Review | `/review/due`, `/review/complete`, `/review/stats` |
| Peer | `/peer/insights`, `/peer/doubts`, `/peer/trending` |
| Gesture | `/gesture/list`, `/gesture/recognize`, `/gesture/training/*` |
| LLM | `/llm/query`, `/llm/gesture-action`, `/llm/rl/*` |

### 8.2 Session API

```python
# POST /api/session/start
{
    "user_id": "student123",
    "topic": "Machine Learning",
    "subtopic": "Neural Networks"
}

# Response
{
    "session_id": "session_1699999999.123",
    "topic": "Machine Learning",
    "predictions": [
        {
            "doubt": "how_overfitting_works",
            "confidence": 0.85,
            "explanation": "Student showing signs of confusion...",
            "priority": 1
        }
    ],
    "pending_reviews": 5,
    "peer_insights_count": 3
}
```

### 8.3 Doubt Prediction API

```python
# POST /api/predict/doubts
{
    "context": {
        "topic": "Neural Networks",
        "progress": 0.5,
        "confusion_signals": 0.7
    }
}

# Response
{
    "predictions": [
        {
            "doubt": "how_overfitting_works",
            "confidence": 0.85,
            "explanation": "...",
            "priority": 1,
            "estimated_time": "10 min",
            "prerequisites": ["regularization", "bias-variance"]
        }
    ]
}
```

---

## 9. Multi-Modal Detection

### 9.1 Supported Modalities

```
┌─────────────────────────────────────────────────────────────────┐
│                    MULTI-MODAL FUSION                             │
│                                                                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐             │
│  │   Audio     │  │  Biometric  │  │ Behavioral  │             │
│  │             │  │             │  │             │             │
│  │ Speech rate │  │ Heart rate  │  │ Mouse moves │             │
│  │ Hesitations │  │ GSR         │  │ Scroll      │             │
│  │ Pauses      │  │ Eye tracking│  │ Key presses │             │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘             │
│         │                │                │                     │
│         └────────────────┼────────────────┘                     │
│                          ▼                                      │
│              ┌─────────────────────────┐                       │
│              │   WEIGHTED FUSION       │                       │
│              │                         │                       │
│              │ audio_weight:    0.2    │                       │
│              │ biometric_weight: 0.3   │                       │
│              │ behavioral_weight: 0.5  │                       │
│              └───────────┬─────────────┘                       │
│                          │                                      │
│                          ▼                                      │
│              ┌─────────────────────────┐                       │
│              │  UNIFIED CONFUSION     │                       │
│              │       SCORE            │                       │
│              │       0.0 - 1.0       │                       │
│              └─────────────────────────┘                       │
└─────────────────────────────────────────────────────────────────┘
```

### 9.2 Feature Extraction by Modality

**Audio (7 features):**
- Speech rate (WPM)
- Pause frequency
- Pause duration
- Pitch variation
- Volume level
- Hesitation count
- Question markers

**Biometric (6 features):**
- Heart rate (BPM)
- Heart rate variability
- Skin conductance (GSR)
- Skin temperature
- Eye blink rate
- Eye open duration

**Behavioral (8 features):**
- Mouse hesitation
- Scroll reversals
- Time on page
- Click frequency
- Back button usage
- Tab switches
- Copy attempts
- Search usage

---

## 10. Privacy & Security

### 10.1 Face Blur Implementation

```python
class FaceBlurProcessor:
    def __init__(self):
        self.face_mesh = mp_face_mesh.FaceMesh(
            static_image_mode=False,
            max_num_faces=1,
            refine_landmarks=True
        )
    
    def blur_face(self, frame):
        # Detect face landmarks
        results = self.face_mesh.process(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
        
        if results.multi_face_landmarks:
            # Get face region
            face_region = self._get_face_region(frame, results)
            
            # Apply Gaussian blur
            blurred = cv2.GaussianBlur(face_region, (51, 51), 0)
            
            # Replace face region
            frame = self._replace_region(frame, blurred, results)
        
        return frame
```

### 10.2 Data Privacy

| Data Type | Storage | Privacy |
|-----------|---------|---------|
| Video frames | None | Processed in-memory only |
| Face images | None | Auto-blurred |
| Hand landmarks | Optional | Anonymized |
| Session data | Local JSON | User-owned |
| Model weights | HuggingFace | Open |

---

## 11. Deployment Architecture

### 11.1 Development Setup

```
┌─────────────────────────────────────────────────────────────────┐
│                      DEVELOPMENT                                  │
│                                                                  │
│  Terminal 1:                 Terminal 2:                        │
│  ┌─────────────────┐         ┌─────────────────┐               │
│  │ cd backend      │         │ cd frontend     │               │
│  │ python run.py   │         │ npm run dev     │               │
│  │                 │         │                 │               │
│  │ Flask :5001     │         │ Vite    :5173   │               │
│  └────────┬────────┘         └────────┬────────┘               │
└───────────┼───────────────────────────┼─────────────────────────┘
            │                           │
            │           ┌───────────────┘
            │           │
            ▼           ▼
┌─────────────────────────────────────────────────────────────────┐
│                    BROWSER (localhost)                            │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐  │
│  │  Frontend (:5173) <─────── Proxy ───────> Backend (:5001)│  │
│  └─────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
```

### 11.2 Production Setup

```
                          ┌─────────────────┐
                          │   Load Balancer  │
                          └────────┬────────┘
                                   │
              ┌────────────────────┼────────────────────┐
              │                    │                    │
              ▼                    ▼                    ▼
      ┌───────────────┐    ┌───────────────┐    ┌───────────────┐
      │  Flask Worker │    │  Flask Worker │    │  Flask Worker │
      │    (:5001)    │    │    (:5001)    │    │    (:5001)    │
      └───────────────┘    └───────────────┘    └───────────────┘
              │                    │                    │
              └────────────────────┼────────────────────┘
                                   │
                                   ▼
                          ┌─────────────────┐
                          │   Redis Cache   │
                          └────────┬────────┘
                                   │
                                   ▼
                          ┌─────────────────┐
                          │   PostgreSQL    │
                          └─────────────────┘
```

### 11.3 HuggingFace Model Hosting

```
┌─────────────────────────────────────────────────────────────────┐
│                     HuggingFace Hub                              │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              namish10/contextflow-rl                     │   │
│  │                                                          │   │
│  │  checkpoint.pkl        ← Trained RL model            │   │
│  │  train_rl.py            ← Training script            │   │
│  │  feature_extractor.py    ← State extraction           │   │
│  │  online_learning.py      ← Continuous learning        │   │
│  │  data_collector.py      ← Real data collection        │   │
│  │  multimodal_detection.py ← Audio/biometric fusion      │   │
│  │  demo.ipynb             ← Interactive demo            │   │
│  │  RESEARCH_PAPER.md      ← Full documentation          │   │
│  │                                                          │   │
│  │  app/ (9 agents + API)                                │   │
│  │  frontend/ (React UI)                                  │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
```

---

## Summary

ContextFlow is a comprehensive system combining:

1. **Predictive AI** - RL-based doubt prediction before confusion occurs
2. **Multi-Agent Architecture** - 9 specialized agents coordinated by orchestrator
3. **Gesture Recognition** - Privacy-first MediaPipe hand detection
4. **Multi-Modal Sensing** - Audio + Biometric + Behavioral fusion
5. **Browser-Based AI** - Direct AI chat launching without API keys
6. **Continuous Learning** - Online learning from user feedback

The system is production-ready with all 9 API endpoints working, complete agent network, and trained RL model available on HuggingFace.