docs/architecture.md

EMOTIA Architecture

System Overview

EMOTIA is a multi-modal AI system that analyzes video calls to infer emotional state, conversational intent, engagement, and confidence using facial expressions, vocal tone, spoken language, and temporal context.

Architecture Diagram

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Video Input   │    │  Audio Input    │    │   Text Input    │
│   (25-30 FPS)   │    │  (16kHz WAV)    │    │  (ASR Trans.)   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                        │                        │
         ▼                        ▼                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ Vision Branch   │    │ Audio Branch    │    │ Text Branch     │
│ • ViT-Base      │    │ • CNN + Trans.  │    │ • BERT Encoder  │
│ • Face Detect   │    │ • Wav2Vec2      │    │ • Intent Detect │
│ • Emotion Class │    │ • Prosody       │    │ • Sentiment     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                        │                        │
         └────────────────────────┼────────────────────────┘
                                  ▼
                   ┌─────────────────────────────┐
                   │   Cross-Modal Fusion        │
                   │ • Attention Mechanism       │
                   │ • Dynamic Weighting         │
                   │ • Temporal Transformer      │
                   │ • Modality Contributions    │
                   └─────────────────────────────┘
                                  │
                                  ▼
                   ┌─────────────────────────────┐
                   │   Multi-Task Outputs        │
                   │ • Emotion Classification    │
                   │ • Intent Classification     │
                   │ • Engagement Regression     │
                   │ • Confidence Estimation     │
                   └─────────────────────────────┘

Component Details

Vision Branch

• Input: RGB video frames (224x224)
• Face Detection: OpenCV Haar cascades
• Feature Extraction: Vision Transformer (ViT-Base)
• Fine-tuning: FER-2013, AffectNet, RAF-DB datasets
• Output: Emotion logits (7 classes), confidence score

Audio Branch

• Input: Audio waveforms (16kHz, 3-second windows)
• Preprocessing: Mel-spectrogram extraction
• Feature Extraction: Wav2Vec2 + CNN layers
• Prosody Analysis: Pitch, rhythm, energy features
• Output: Emotion logits, stress/confidence score

Text Branch

• Input: Transcribed speech text
• Preprocessing: Tokenization, cleaning
• Feature Extraction: BERT-base for intent/sentiment
• Intent Detection: Hesitation phrases, confidence markers
• Output: Intent logits (5 classes), sentiment logits

Fusion Network

• Modality Projection: Linear layers to common embedding space (256D)
• Cross-Attention: Multi-head attention between modalities
• Temporal Modeling: Transformer encoder for sequence processing
• Dynamic Weighting: Learned modality importance scores
• Outputs: Fused predictions with contribution weights

Data Flow

1. Input Processing: Video frames, audio chunks, ASR text
2. Sliding Windows: 5-10 second temporal windows
3. Feature Extraction: Parallel processing per modality
4. Fusion: Cross-modal attention and temporal aggregation
5. Prediction: Multi-task classification/regression
6. Explainability: Modality contribution scores

Deployment Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Client Application                       │
│  ┌─────────────────────────────────────────────────────┐    │
│  │                WebRTC Video Stream                 │    │
│  │  • Camera Access                                  │    │
│  │  • Audio Capture                                  │    │
│  │  • Real-time Streaming                             │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────┐
│                     FastAPI Backend                         │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              Inference Pipeline                    │    │
│  │  • Model Loading                                  │    │
│  │  • Preprocessing                                  │    │
│  │  • GPU Inference                                  │    │
│  │  • Post-processing                                │    │
│  └─────────────────────────────────────────────────────┘    │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              Real-time Processing                  │    │
│  │  • Sliding Window Buffering                       │    │
│  │  • Asynchronous Processing                        │    │
│  │  • Streaming Responses                            │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────┐
│                    Response Formatting                      │
│  • JSON API Responses                                     │
│  • Real-time WebSocket Updates                            │
│  • Batch Processing for Post-call Analysis                │
└─────────────────────────────────────────────────────────────┘

Performance Requirements

• Latency: <200ms end-to-end
• Throughput: 25-30 FPS video processing
• Accuracy: F1 > 0.80 for emotion classification
• Scalability: Horizontal scaling with load balancer
• Reliability: 99.9% uptime, graceful degradation

Security Considerations

• Data Privacy: No biometric storage by default
• Encryption: TLS 1.3 for all communications
• Access Control: API key authentication
• Audit Logging: All inference requests logged
• Compliance: GDPR, CCPA compliance features