prd.md

# EMOTIA Product Requirements Document

## 1. Product Overview

### Problem
Video calls remove many human signals. Recruiters, educators, sales teams, and therapists lack objective insights into:
- Emotional state
- Engagement
- Confidence
- Intent (confusion, agreement, hesitation)

Manual observation is subjective, inconsistent, and non-scalable.

### Solution
A real-time multi-modal AI system that analyzes:
- Facial expressions (video)
- Vocal tone (audio)
- Spoken language (text)
- Temporal behavior (over time)

…and produces interpretable, ethical, probabilistic insights.

### Target Users
- Recruiters & hiring platforms
- EdTech platforms
- Sales & customer success teams
- Remote therapy & coaching platforms
- Product teams analyzing user calls

## 2. Core Features

### 2.1 Live Video Call Analysis
- Real-time emotion detection
- Engagement tracking
- Confidence & stress indicators
- Timeline-based emotion shifts

### 2.2 Post-Call Analytics Dashboard
- Emotion timeline
- Intent heatmap
- Modality influence breakdown
- Key moments (confusion spikes, stress peaks)

### 2.3 Multi-Modal Explainability
Why a prediction was made:
- Face vs voice vs text contribution
- Visual overlays (heatmaps)
- Confidence intervals (not hard labels)

### 2.4 Ethics & Bias Controls
- Bias evaluation toggle
- Per-modality opt-out
- Clear disclaimers (non-diagnostic, assistive AI)

## 3. UI / UX Vision

### 3.1 UI Style
- Dark mode only
- Glassmorphism cards
- Neon accent colors (cyan / violet / lime)
- Smooth micro-animations
- Real-time waveform + emotion graphs

### 3.2 Main Dashboard

#### Left Panel
- Live video feed
- Face bounding box
- Micro-expression indicators

#### Center
- Emotion timeline (animated)
- Engagement meter (0–100)
- Confidence score

#### Right Panel
- Intent probabilities
- Stress indicators
- Modality contribution bars

### 3.3 Post-Call Report UI
- Scrollable emotion timeline
- Clickable "critical moments"
- Modality dominance chart
- Exportable report (PDF)

### 3.4 UI Components (Must-Have)
- Animated confidence rings
- Temporal scrubber
- Heatmap overlays
- Tooltips explaining AI decisions

## 4. Technical Architecture

### 4.1 Input Pipeline
- Webcam video (25–30 FPS)
- Microphone audio
- Real-time ASR
- Sliding temporal windows (5–10 sec)

### 4.2 Model Architecture (Production-Grade)

#### 🔹 Visual Branch
- Vision Transformer (ViT) fine-tuned for facial expressions
- Face detection + alignment
- Temporal pooling

#### 🔹 Audio Branch
- Audio → Mel-spectrogram
- CNN + Transformer
- Prosody, pitch, rhythm modeling

#### 🔹 Text Branch
- Transformer-based language model
- Fine-tuned for intent & sentiment
- Confidence / hesitation phrase detection

#### 🔹 Fusion Network (KEY DIFFERENTIATOR)
- Cross-modal attention
- Dynamic modality weighting
- Temporal transformer for sequence learning

#### 🔹 Output Heads
- Emotion classification
- Intent classification
- Engagement regression
- Confidence regression

## 5. Models to Use (Strong + Realistic)

### Visual
- ViT-Base / EfficientNet
- Pretrained on face emotion datasets

### Audio
- Wav2Vec-style embeddings
- CNN-Transformer hybrid

### Text
- Transformer encoder (fine-tuned)
- Focus on conversational intent

### Fusion
- Custom attention-based multi-head network
- (this is your original contribution)

## 6. Datasets (CV-Worthy)

### Facial Emotion
- FER-2013
- AffectNet
- RAF-DB

### Audio Emotion
- RAVDESS
- CREMA-D

### Speech + Intent
- IEMOCAP
- MELD (multi-party dialogue)

### Strategy
- Pretrain each modality separately
- Fine-tune jointly
- Align timestamps across modalities

## 7. Training & Evaluation

### Training
- Multi-task learning
- Weighted losses per output
- Curriculum learning (single → multi-modal)

### Metrics
- F1-score per emotion
- Concordance correlation (regression)
- Confusion matrices
- Per-modality ablation

## 8. Deployment

### Backend
- FastAPI
- GPU inference support
- Streaming inference pipeline

### Frontend
- Next.js / React
- WebRTC video
- Web Audio API
- WebGL visualizations

### Infrastructure
- Dockerized services
- Modular microservices
- Model versioning

## 9. Non-Functional Requirements
- Real-time latency < 200ms
- Modular model replacement
- Privacy-first design
- No biometric storage by default