ARCHITECTURE.md

FaceDetection System — Architecture Documentation

Last Updated: June 9, 2026
Stack: Python · FastAPI · OpenCV · ONNX · SQLite · React

---

Table of Contents

1. Current Architecture
2. Logic Flow (Step-by-Step)
3. Core Models
4. Liveness Detection
5. Attendance Logging
6. Proposed Architecture (Alternative Plan)
7. Side-by-Side Comparison
8. What's Already Done vs. What Can Be Improved
9. Recommended Next Steps

---

1. Current Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                         CAPTURE LAYER                                │
│   VideoCamera (Singleton)  ·  640×480  ·  ~30 FPS cap               │
│   OpenCV VideoCapture  →  Horizontal Flip (mirror mode)             │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                       DETECTION LAYER                                │
│   Model : YuNet  (face_detection_yunet_2023mar.onnx)                │
│   Format: ONNX  ·  Input: dynamic (set to frame size per call)      │
│   Output: bounding box [x, y, w, h] + 5-point facial landmarks      │
│           + detection confidence score                               │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                       LIVENESS LAYER                                 │
│   Method : Landmark Standard-Deviation Analysis                      │
│   Buffer : Last 15 frames of 5-point landmark coordinates            │
│   Logic  : If avg. std-dev of all coordinates < 0.7 px → SPOOF      │
│            (A real face has micro-tremors; a printed photo is rigid) │
│   Output : is_live = True / False + liveness label                  │
└─────────────────────────────┬────────────────────────────────────────┘
                              │  (only if is_live == True)
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                      EMBEDDING LAYER                                 │
│   Model : SFace  (face_recognition_sface_2021dec.onnx)              │
│   Format: ONNX  ·  ~38 MB on disk                                   │
│   Steps : 1. alignCrop(frame, face_landmarks)  → aligned 112×112    │
│            2. recognizer.feature(aligned_face) → 128-dim vector     │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     VERIFICATION LAYER                               │
│   Method    : Cosine Similarity  (cv2.FaceRecognizerSF_FR_COSINE)   │
│   Threshold : 0.363  (calibrated by OpenCV SFace research paper)    │
│   Strategy  : Best-of-N — takes highest score per employee across   │
│               all stored face templates                              │
│   Output    : (employee_id, employee_name, similarity_score) | None │
└─────────────────────────────┬────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────────┐
│                    ATTENDANCE LOGGER                                 │
│   Debounce  : 2-minute cooldown per employee_id (in-memory dict)    │
│   Logic     : check_in → check_out → check_in (toggle per day)     │
│   DB        : SQLite  (attendance.db)                               │
│   Alerts    : Thread-safe Queue → SSE stream → React frontend       │
└──────────────────────────────────────────────────────────────────────┘

---

2. Logic Flow (Step-by-Step)

Camera Frame
    │
    ├─► [YuNet Detector]
    │        │
    │        ├── No faces found  ──► Clear liveness buffer, draw scan-line animation
    │        │
    │        └── Faces found (1..N) ──► For each face:
    │                │
    │                ├─► Parse bbox [x, y, w, h] + 5 landmarks + score
    │                │
    │                ├─► Update landmark buffer (last 15 frames)
    │                │
    │                ├─► [Liveness Check]
    │                │       ├── Buffer < 8 frames  ──► "PENDING..."
    │                │       ├── std_dev < 0.7 px   ──► "SPOOF: STATIC PHOTO" → push SSE alert
    │                │       └── std_dev ≥ 0.7 px   ──► "LIVENESS: VERIFIED"
    │                │
    │                ├─► [SFace Embedding]  (only if live)
    │                │       └── alignCrop → feature() → 128-d vector
    │                │
    │                ├─► [Cosine Match]
    │                │       ├── score ≥ 0.363  ──► (emp_id, name, score)
    │                │       └── score < 0.363  ──► Unknown
    │                │
    │                └─► [Attendance Trigger]  (only if emp_id found)
    │                        ├── In cooldown?  ──► Skip
    │                        ├── Determine event_type (check_in / check_out)
    │                        ├── Log to SQLite
    │                        └── Push notification to SSE Queue
    │
    └─► [HUD Renderer]
            ├── Neon bounding box (color: green=match, red=unknown, crimson=spoof)
            ├── 5-point landmark dots
            ├── Label banner (name, score %, liveness status)
            └── Corner tick HUD overlay + scan-line animation

---

3. Core Models

Property	YuNet (Detector)	SFace (Recognizer)
File	face_detection_yunet_2023mar.onnx	face_recognition_sface_2021dec.onnx
Size	~232 KB	~38 MB
Format	ONNX	ONNX
Input	Dynamic (frame resolution)	112×112 aligned crop
Output	bbox + 5 landmarks + score	128-dimensional embedding vector
Runtime	OpenCV DNN (cv2.FaceDetectorYN)	OpenCV DNN (cv2.FaceRecognizerSF)
Source	HuggingFace: opencv/face_detection_yunet	HuggingFace: opencv/face_recognition_sface

---

4. Liveness Detection

Goal: Prevent "proxy attendance" using a printed photo or screen-displayed image.

Method: Landmark Standard-Deviation Analysis

# Collect last 15 frames of 5-point landmarks
pts = np.array(landmark_buffer)        # shape: (15, 5, 2)
std_coords = np.std(pts, axis=0)       # shape: (5, 2)
avg_std = np.mean(std_coords)

if avg_std < 0.7:
    is_live = False   # Coordinates are rigid → static photo / spoof
else:
    is_live = True    # Natural micro-tremors detected → real face

Why it works: A real human face held in front of a camera has natural micro-tremors, breathing movements, and small head shifts. A photo or screen holds perfectly still. The 0.7-pixel threshold captures this difference reliably.

Current Limitation: This method only catches perfectly static spoofs. It won't catch a video of a face played on a phone, or a photo shaken manually.

---

5. Attendance Logging

• Toggle Logic: The system alternates check_in → check_out → check_in based on the last event stored in the DB for that employee on the current day.
• Debounce: A 2-minute in-memory cooldown per employee_id prevents duplicate logs from continuous recognition.
• Real-Time Alerts: Every attendance event is pushed into a thread-safe queue.Queue which is streamed to the React frontend via Server-Sent Events (SSE).
• Spoof Alerts: Spoof detection events also push to the SSE queue with a 10-second debounce to avoid flooding the UI.

---

6. Proposed Architecture (Alternative Plan)

The following is an alternative architecture plan that was evaluated for this system:

Capture Layer
    └─► BlazeFace (initial detection)
            └─► Correlation Tracker (tracks bounding box between detections)
                    └─► FaceNet (128-d or 512-d embeddings)
                            └─► Cosine Similarity (threshold: 0.7)
                                    └─► Attendance Logger

Additional tips proposed:

• Model Quantization (ONNX / TFLite)
• Frame Skipping (process every other frame)
• Lower detection resolution (320×320)
• Liveness via blink detection / MediaPipe landmarks

---

7. Side-by-Side Comparison

Dimension	Current Implementation	Proposed Plan
Detector	YuNet (ONNX, ~232 KB)	BlazeFace
Tracking	Per-frame YuNet detection	BlazeFace → Correlation Tracker handoff
Recognizer	SFace (ONNX, 128-d) via OpenCV	FaceNet (PyTorch / TFLite, 128-d or 512-d)
Similarity	Cosine via cv2.FaceRecognizerSF	Manual cosine similarity
Threshold	0.363 (SFace-calibrated)	0.7 (incompatible with SFace scale)
Model Format	✅ Already ONNX	Needs conversion to ONNX/TFLite
Liveness	✅ Landmark std-dev (static photo guard)	Blink detection (EAR via MediaPipe)
Frame Rate	~30 FPS cap (sleep 0.033s)	15–20 FPS target
Complexity	Low — single pipeline	Higher — 2-stage detect+track pipeline
Dependencies	OpenCV only (no heavy ML framework)	Requires PyTorch or TFLite runtime
Status	✅ Production-ready	🔧 Would require significant re-engineering

Key Conflicts to Note

⚠️ Threshold Incompatibility: The proposed threshold of 0.7 is designed for FaceNet's cosine similarity scale. SFace cosine scores use a different calibration where 0.363 is the verified boundary. Applying 0.7 to the current SFace engine would reject almost all legitimate matches.

ℹ️ Detector Redundancy: YuNet already operates at a low-resolution input size and is fast enough for real-time CPU use. Adding a Correlation Tracker on top introduces state complexity with marginal speed gain for an attendance use case (stationary camera, controlled environment).

✅ ONNX is already done: Both YuNet and SFace are .onnx format. The quantization recommendation is already fulfilled.

---

8. What's Already Done vs. What Can Be Improved

✅ Already Implemented

• ONNX model inference (YuNet + SFace)
• Real-time face detection at 640×480
• 128-d embedding extraction with face alignment
• Cosine similarity matching with calibrated threshold
• In-memory embedding cache (reload on new employee registration)
• Liveness detection (static photo / spoof guard)
• Attendance toggle logic (check-in / check-out)
• 2-minute debounce cooldown per employee
• Real-time SSE alerts to frontend
• Spoof alerts with 10-second debounce
• Premium HUD overlay (neon bounding boxes, corner ticks, scan-line animation)
• Thread-safe singleton camera instance

🔧 Improvements Worth Adding

Priority	Improvement	Effort	Impact
⚡ High	Frame Skipping — process every 2nd frame	Low	30–50% CPU reduction
⚡ High	Downscale for detection — run YuNet on 320×240 copy, scale landmarks back to 640×480 for display	Low	Faster detection pass
👁️ Medium	Blink Detection (EAR) — upgrade to MediaPipe 468-point face mesh for eye-aspect-ratio liveness	Medium	Catches video/moving-photo spoofs
📁 Medium	Multi-angle enrollment — store 3–5 embeddings per employee from different angles	Low	Improves recognition rate
🚀 Low	Async embedding extraction — move embedding+matching to a worker thread, keep capture loop purely for frames	Medium	Eliminates blocking in capture loop

---

9. Recommended Next Steps

Priority 1 (Quick Wins — 1-2 hours):
  ├── Add frame_count % 2 skip in _capture_loop()
  └── Add downscale copy for YuNet detect call

Priority 2 (Accuracy — 1 day):
  └── Multi-angle face enrollment UI (capture 3-5 samples per person)

Priority 3 (Liveness Upgrade — 2-3 days):
  └── Integrate MediaPipe FaceMesh for blink-based EAR liveness

---

Generated by Antigravity AI · FaceDetection Project · June 2026