# FaceDetection System — Architecture Documentation

> **Last Updated:** June 9, 2026  
> **Stack:** Python · FastAPI · OpenCV · ONNX · SQLite · React

---

## Table of Contents

1. [Current Architecture](#1-current-architecture)
2. [Logic Flow (Step-by-Step)](#2-logic-flow-step-by-step)
3. [Core Models](#3-core-models)
4. [Liveness Detection](#4-liveness-detection)
5. [Attendance Logging](#5-attendance-logging)
6. [Proposed Architecture (Alternative Plan)](#6-proposed-architecture-alternative-plan)
7. [Side-by-Side Comparison](#7-side-by-side-comparison)
8. [What's Already Done vs. What Can Be Improved](#8-whats-already-done-vs-what-can-be-improved)
9. [Recommended Next Steps](#9-recommended-next-steps)

---

## 1. Current Architecture

```
┌──────────────────────────────────────────────────────────────────────┐
│                         CAPTURE LAYER                                │
│   VideoCamera (Singleton)  ·  640×480  ·  ~30 FPS cap               │
│   OpenCV VideoCapture  →  Horizontal Flip (mirror mode)             │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                       DETECTION LAYER                                │
│   Model : YuNet  (face_detection_yunet_2023mar.onnx)                │
│   Format: ONNX  ·  Input: dynamic (set to frame size per call)      │
│   Output: bounding box [x, y, w, h] + 5-point facial landmarks      │
│           + detection confidence score                               │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                       LIVENESS LAYER                                 │
│   Method : Landmark Standard-Deviation Analysis                      │
│   Buffer : Last 15 frames of 5-point landmark coordinates            │
│   Logic  : If avg. std-dev of all coordinates < 0.7 px → SPOOF      │
│            (A real face has micro-tremors; a printed photo is rigid) │
│   Output : is_live = True / False + liveness label                  │
└─────────────────────────────┬────────────────────────────────────────┘
                              │  (only if is_live == True)
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                      EMBEDDING LAYER                                 │
│   Model : SFace  (face_recognition_sface_2021dec.onnx)              │
│   Format: ONNX  ·  ~38 MB on disk                                   │
│   Steps : 1. alignCrop(frame, face_landmarks)  → aligned 112×112    │
│            2. recognizer.feature(aligned_face) → 128-dim vector     │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     VERIFICATION LAYER                               │
│   Method    : Cosine Similarity  (cv2.FaceRecognizerSF_FR_COSINE)   │
│   Threshold : 0.363  (calibrated by OpenCV SFace research paper)    │
│   Strategy  : Best-of-N — takes highest score per employee across   │
│               all stored face templates                              │
│   Output    : (employee_id, employee_name, similarity_score) | None │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                    ATTENDANCE LOGGER                                 │
│   Debounce  : 2-minute cooldown per employee_id (in-memory dict)    │
│   Logic     : check_in → check_out → check_in (toggle per day)     │
│   DB        : SQLite  (attendance.db)                               │
│   Alerts    : Thread-safe Queue → SSE stream → React frontend       │
└──────────────────────────────────────────────────────────────────────┘
```

---

## 2. Logic Flow (Step-by-Step)

```
Camera Frame
    │
    ├─► [YuNet Detector]
    │        │
    │        ├── No faces found  ──► Clear liveness buffer, draw scan-line animation
    │        │
    │        └── Faces found (1..N) ──► For each face:
    │                │
    │                ├─► Parse bbox [x, y, w, h] + 5 landmarks + score
    │                │
    │                ├─► Update landmark buffer (last 15 frames)
    │                │
    │                ├─► [Liveness Check]
    │                │       ├── Buffer < 8 frames  ──► "PENDING..."
    │                │       ├── std_dev < 0.7 px   ──► "SPOOF: STATIC PHOTO" → push SSE alert
    │                │       └── std_dev ≥ 0.7 px   ──► "LIVENESS: VERIFIED"
    │                │
    │                ├─► [SFace Embedding]  (only if live)
    │                │       └── alignCrop → feature() → 128-d vector
    │                │
    │                ├─► [Cosine Match]
    │                │       ├── score ≥ 0.363  ──► (emp_id, name, score)
    │                │       └── score < 0.363  ──► Unknown
    │                │
    │                └─► [Attendance Trigger]  (only if emp_id found)
    │                        ├── In cooldown?  ──► Skip
    │                        ├── Determine event_type (check_in / check_out)
    │                        ├── Log to SQLite
    │                        └── Push notification to SSE Queue
    │
    └─► [HUD Renderer]
            ├── Neon bounding box (color: green=match, red=unknown, crimson=spoof)
            ├── 5-point landmark dots
            ├── Label banner (name, score %, liveness status)
            └── Corner tick HUD overlay + scan-line animation
```

---

## 3. Core Models

| Property | YuNet (Detector) | SFace (Recognizer) |
|---|---|---|
| **File** | `face_detection_yunet_2023mar.onnx` | `face_recognition_sface_2021dec.onnx` |
| **Size** | ~232 KB | ~38 MB |
| **Format** | ONNX | ONNX |
| **Input** | Dynamic (frame resolution) | 112×112 aligned crop |
| **Output** | bbox + 5 landmarks + score | 128-dimensional embedding vector |
| **Runtime** | OpenCV DNN (`cv2.FaceDetectorYN`) | OpenCV DNN (`cv2.FaceRecognizerSF`) |
| **Source** | [HuggingFace: opencv/face_detection_yunet](https://huggingface.co/opencv/face_detection_yunet) | [HuggingFace: opencv/face_recognition_sface](https://huggingface.co/opencv/face_recognition_sface) |

---

## 4. Liveness Detection

**Goal:** Prevent "proxy attendance" using a printed photo or screen-displayed image.

**Method: Landmark Standard-Deviation Analysis**

```python
# Collect last 15 frames of 5-point landmarks
pts = np.array(landmark_buffer)        # shape: (15, 5, 2)
std_coords = np.std(pts, axis=0)       # shape: (5, 2)
avg_std = np.mean(std_coords)

if avg_std < 0.7:
    is_live = False   # Coordinates are rigid → static photo / spoof
else:
    is_live = True    # Natural micro-tremors detected → real face
```

**Why it works:** A real human face held in front of a camera has natural micro-tremors, breathing movements, and small head shifts. A photo or screen holds perfectly still. The 0.7-pixel threshold captures this difference reliably.

**Current Limitation:** This method only catches **perfectly static** spoofs. It won't catch a video of a face played on a phone, or a photo shaken manually.

---

## 5. Attendance Logging

- **Toggle Logic:** The system alternates `check_in` → `check_out` → `check_in` based on the last event stored in the DB for that employee on the current day.
- **Debounce:** A 2-minute in-memory cooldown per `employee_id` prevents duplicate logs from continuous recognition.
- **Real-Time Alerts:** Every attendance event is pushed into a thread-safe `queue.Queue` which is streamed to the React frontend via **Server-Sent Events (SSE)**.
- **Spoof Alerts:** Spoof detection events also push to the SSE queue with a 10-second debounce to avoid flooding the UI.

---

## 6. Proposed Architecture (Alternative Plan)

The following is an alternative architecture plan that was evaluated for this system:

```
Capture Layer
    └─► BlazeFace (initial detection)
            └─► Correlation Tracker (tracks bounding box between detections)
                    └─► FaceNet (128-d or 512-d embeddings)
                            └─► Cosine Similarity (threshold: 0.7)
                                    └─► Attendance Logger
```

**Additional tips proposed:**
- Model Quantization (ONNX / TFLite)
- Frame Skipping (process every other frame)
- Lower detection resolution (320×320)
- Liveness via blink detection / MediaPipe landmarks

---

## 7. Side-by-Side Comparison

| Dimension | Current Implementation | Proposed Plan |
|---|---|---|
| **Detector** | YuNet (ONNX, ~232 KB) | BlazeFace |
| **Tracking** | Per-frame YuNet detection | BlazeFace → Correlation Tracker handoff |
| **Recognizer** | SFace (ONNX, 128-d) via OpenCV | FaceNet (PyTorch / TFLite, 128-d or 512-d) |
| **Similarity** | Cosine via `cv2.FaceRecognizerSF` | Manual cosine similarity |
| **Threshold** | **0.363** (SFace-calibrated) | **0.7** *(incompatible with SFace scale)* |
| **Model Format** | ✅ Already ONNX | Needs conversion to ONNX/TFLite |
| **Liveness** | ✅ Landmark std-dev (static photo guard) | Blink detection (EAR via MediaPipe) |
| **Frame Rate** | ~30 FPS cap (sleep 0.033s) | 15–20 FPS target |
| **Complexity** | Low — single pipeline | Higher — 2-stage detect+track pipeline |
| **Dependencies** | OpenCV only (no heavy ML framework) | Requires PyTorch or TFLite runtime |
| **Status** | ✅ Production-ready | 🔧 Would require significant re-engineering |

### Key Conflicts to Note

> ⚠️ **Threshold Incompatibility:** The proposed threshold of `0.7` is designed for FaceNet's cosine similarity scale. SFace cosine scores use a different calibration where `0.363` is the verified boundary. Applying `0.7` to the current SFace engine would reject almost all legitimate matches.

> ℹ️ **Detector Redundancy:** YuNet already operates at a low-resolution input size and is fast enough for real-time CPU use. Adding a Correlation Tracker on top introduces state complexity with marginal speed gain for an attendance use case (stationary camera, controlled environment).

> ✅ **ONNX is already done:** Both YuNet and SFace are `.onnx` format. The quantization recommendation is already fulfilled.

---

## 8. What's Already Done vs. What Can Be Improved

### ✅ Already Implemented

- [x] ONNX model inference (YuNet + SFace)
- [x] Real-time face detection at 640×480
- [x] 128-d embedding extraction with face alignment
- [x] Cosine similarity matching with calibrated threshold
- [x] In-memory embedding cache (reload on new employee registration)
- [x] Liveness detection (static photo / spoof guard)
- [x] Attendance toggle logic (check-in / check-out)
- [x] 2-minute debounce cooldown per employee
- [x] Real-time SSE alerts to frontend
- [x] Spoof alerts with 10-second debounce
- [x] Premium HUD overlay (neon bounding boxes, corner ticks, scan-line animation)
- [x] Thread-safe singleton camera instance

### 🔧 Improvements Worth Adding

| Priority | Improvement | Effort | Impact |
|---|---|---|---|
| ⚡ High | **Frame Skipping** — process every 2nd frame | Low | 30–50% CPU reduction |
| ⚡ High | **Downscale for detection** — run YuNet on 320×240 copy, scale landmarks back to 640×480 for display | Low | Faster detection pass |
| 👁️ Medium | **Blink Detection (EAR)** — upgrade to MediaPipe 468-point face mesh for eye-aspect-ratio liveness | Medium | Catches video/moving-photo spoofs |
| 📁 Medium | **Multi-angle enrollment** — store 3–5 embeddings per employee from different angles | Low | Improves recognition rate |
| 🚀 Low | **Async embedding extraction** — move embedding+matching to a worker thread, keep capture loop purely for frames | Medium | Eliminates blocking in capture loop |

---

## 9. Recommended Next Steps

```
Priority 1 (Quick Wins — 1-2 hours):
  ├── Add frame_count % 2 skip in _capture_loop()
  └── Add downscale copy for YuNet detect call

Priority 2 (Accuracy — 1 day):
  └── Multi-angle face enrollment UI (capture 3-5 samples per person)

Priority 3 (Liveness Upgrade — 2-3 days):
  └── Integrate MediaPipe FaceMesh for blink-based EAR liveness
```

---

*Generated by Antigravity AI · FaceDetection Project · June 2026*