Spaces:
Sleeping
Sleeping
| # FaceDetection System — Architecture Documentation | |
| > **Last Updated:** June 9, 2026 | |
| > **Stack:** Python · FastAPI · OpenCV · ONNX · SQLite · React | |
| --- | |
| ## Table of Contents | |
| 1. [Current Architecture](#1-current-architecture) | |
| 2. [Logic Flow (Step-by-Step)](#2-logic-flow-step-by-step) | |
| 3. [Core Models](#3-core-models) | |
| 4. [Liveness Detection](#4-liveness-detection) | |
| 5. [Attendance Logging](#5-attendance-logging) | |
| 6. [Proposed Architecture (Alternative Plan)](#6-proposed-architecture-alternative-plan) | |
| 7. [Side-by-Side Comparison](#7-side-by-side-comparison) | |
| 8. [What's Already Done vs. What Can Be Improved](#8-whats-already-done-vs-what-can-be-improved) | |
| 9. [Recommended Next Steps](#9-recommended-next-steps) | |
| --- | |
| ## 1. Current Architecture | |
| ``` | |
| ┌──────────────────────────────────────────────────────────────────────┐ | |
| │ CAPTURE LAYER │ | |
| │ VideoCamera (Singleton) · 640×480 · ~30 FPS cap │ | |
| │ OpenCV VideoCapture → Horizontal Flip (mirror mode) │ | |
| └─────────────────────────────┬────────────────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌──────────────────────────────────────────────────────────────────────┐ | |
| │ DETECTION LAYER │ | |
| │ Model : YuNet (face_detection_yunet_2023mar.onnx) │ | |
| │ Format: ONNX · Input: dynamic (set to frame size per call) │ | |
| │ Output: bounding box [x, y, w, h] + 5-point facial landmarks │ | |
| │ + detection confidence score │ | |
| └─────────────────────────────┬────────────────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌──────────────────────────────────────────────────────────────────────┐ | |
| │ LIVENESS LAYER │ | |
| │ Method : Landmark Standard-Deviation Analysis │ | |
| │ Buffer : Last 15 frames of 5-point landmark coordinates │ | |
| │ Logic : If avg. std-dev of all coordinates < 0.7 px → SPOOF │ | |
| │ (A real face has micro-tremors; a printed photo is rigid) │ | |
| │ Output : is_live = True / False + liveness label │ | |
| └─────────────────────────────┬────────────────────────────────────────┘ | |
| │ (only if is_live == True) | |
| ▼ | |
| ┌──────────────────────────────────────────────────────────────────────┐ | |
| │ EMBEDDING LAYER │ | |
| │ Model : SFace (face_recognition_sface_2021dec.onnx) │ | |
| │ Format: ONNX · ~38 MB on disk │ | |
| │ Steps : 1. alignCrop(frame, face_landmarks) → aligned 112×112 │ | |
| │ 2. recognizer.feature(aligned_face) → 128-dim vector │ | |
| └─────────────────────────────┬────────────────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌──────────────────────────────────────────────────────────────────────┐ | |
| │ VERIFICATION LAYER │ | |
| │ Method : Cosine Similarity (cv2.FaceRecognizerSF_FR_COSINE) │ | |
| │ Threshold : 0.363 (calibrated by OpenCV SFace research paper) │ | |
| │ Strategy : Best-of-N — takes highest score per employee across │ | |
| │ all stored face templates │ | |
| │ Output : (employee_id, employee_name, similarity_score) | None │ | |
| └─────────────────────────────┬────────────────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌──────────────────────────────────────────────────────────────────────┐ | |
| │ ATTENDANCE LOGGER │ | |
| │ Debounce : 2-minute cooldown per employee_id (in-memory dict) │ | |
| │ Logic : check_in → check_out → check_in (toggle per day) │ | |
| │ DB : SQLite (attendance.db) │ | |
| │ Alerts : Thread-safe Queue → SSE stream → React frontend │ | |
| └──────────────────────────────────────────────────────────────────────┘ | |
| ``` | |
| --- | |
| ## 2. Logic Flow (Step-by-Step) | |
| ``` | |
| Camera Frame | |
| │ | |
| ├─► [YuNet Detector] | |
| │ │ | |
| │ ├── No faces found ──► Clear liveness buffer, draw scan-line animation | |
| │ │ | |
| │ └── Faces found (1..N) ──► For each face: | |
| │ │ | |
| │ ├─► Parse bbox [x, y, w, h] + 5 landmarks + score | |
| │ │ | |
| │ ├─► Update landmark buffer (last 15 frames) | |
| │ │ | |
| │ ├─► [Liveness Check] | |
| │ │ ├── Buffer < 8 frames ──► "PENDING..." | |
| │ │ ├── std_dev < 0.7 px ──► "SPOOF: STATIC PHOTO" → push SSE alert | |
| │ │ └── std_dev ≥ 0.7 px ──► "LIVENESS: VERIFIED" | |
| │ │ | |
| │ ├─► [SFace Embedding] (only if live) | |
| │ │ └── alignCrop → feature() → 128-d vector | |
| │ │ | |
| │ ├─► [Cosine Match] | |
| │ │ ├── score ≥ 0.363 ──► (emp_id, name, score) | |
| │ │ └── score < 0.363 ──► Unknown | |
| │ │ | |
| │ └─► [Attendance Trigger] (only if emp_id found) | |
| │ ├── In cooldown? ──► Skip | |
| │ ├── Determine event_type (check_in / check_out) | |
| │ ├── Log to SQLite | |
| │ └── Push notification to SSE Queue | |
| │ | |
| └─► [HUD Renderer] | |
| ├── Neon bounding box (color: green=match, red=unknown, crimson=spoof) | |
| ├── 5-point landmark dots | |
| ├── Label banner (name, score %, liveness status) | |
| └── Corner tick HUD overlay + scan-line animation | |
| ``` | |
| --- | |
| ## 3. Core Models | |
| | Property | YuNet (Detector) | SFace (Recognizer) | | |
| |---|---|---| | |
| | **File** | `face_detection_yunet_2023mar.onnx` | `face_recognition_sface_2021dec.onnx` | | |
| | **Size** | ~232 KB | ~38 MB | | |
| | **Format** | ONNX | ONNX | | |
| | **Input** | Dynamic (frame resolution) | 112×112 aligned crop | | |
| | **Output** | bbox + 5 landmarks + score | 128-dimensional embedding vector | | |
| | **Runtime** | OpenCV DNN (`cv2.FaceDetectorYN`) | OpenCV DNN (`cv2.FaceRecognizerSF`) | | |
| | **Source** | [HuggingFace: opencv/face_detection_yunet](https://huggingface.co/opencv/face_detection_yunet) | [HuggingFace: opencv/face_recognition_sface](https://huggingface.co/opencv/face_recognition_sface) | | |
| --- | |
| ## 4. Liveness Detection | |
| **Goal:** Prevent "proxy attendance" using a printed photo or screen-displayed image. | |
| **Method: Landmark Standard-Deviation Analysis** | |
| ```python | |
| # Collect last 15 frames of 5-point landmarks | |
| pts = np.array(landmark_buffer) # shape: (15, 5, 2) | |
| std_coords = np.std(pts, axis=0) # shape: (5, 2) | |
| avg_std = np.mean(std_coords) | |
| if avg_std < 0.7: | |
| is_live = False # Coordinates are rigid → static photo / spoof | |
| else: | |
| is_live = True # Natural micro-tremors detected → real face | |
| ``` | |
| **Why it works:** A real human face held in front of a camera has natural micro-tremors, breathing movements, and small head shifts. A photo or screen holds perfectly still. The 0.7-pixel threshold captures this difference reliably. | |
| **Current Limitation:** This method only catches **perfectly static** spoofs. It won't catch a video of a face played on a phone, or a photo shaken manually. | |
| --- | |
| ## 5. Attendance Logging | |
| - **Toggle Logic:** The system alternates `check_in` → `check_out` → `check_in` based on the last event stored in the DB for that employee on the current day. | |
| - **Debounce:** A 2-minute in-memory cooldown per `employee_id` prevents duplicate logs from continuous recognition. | |
| - **Real-Time Alerts:** Every attendance event is pushed into a thread-safe `queue.Queue` which is streamed to the React frontend via **Server-Sent Events (SSE)**. | |
| - **Spoof Alerts:** Spoof detection events also push to the SSE queue with a 10-second debounce to avoid flooding the UI. | |
| --- | |
| ## 6. Proposed Architecture (Alternative Plan) | |
| The following is an alternative architecture plan that was evaluated for this system: | |
| ``` | |
| Capture Layer | |
| └─► BlazeFace (initial detection) | |
| └─► Correlation Tracker (tracks bounding box between detections) | |
| └─► FaceNet (128-d or 512-d embeddings) | |
| └─► Cosine Similarity (threshold: 0.7) | |
| └─► Attendance Logger | |
| ``` | |
| **Additional tips proposed:** | |
| - Model Quantization (ONNX / TFLite) | |
| - Frame Skipping (process every other frame) | |
| - Lower detection resolution (320×320) | |
| - Liveness via blink detection / MediaPipe landmarks | |
| --- | |
| ## 7. Side-by-Side Comparison | |
| | Dimension | Current Implementation | Proposed Plan | | |
| |---|---|---| | |
| | **Detector** | YuNet (ONNX, ~232 KB) | BlazeFace | | |
| | **Tracking** | Per-frame YuNet detection | BlazeFace → Correlation Tracker handoff | | |
| | **Recognizer** | SFace (ONNX, 128-d) via OpenCV | FaceNet (PyTorch / TFLite, 128-d or 512-d) | | |
| | **Similarity** | Cosine via `cv2.FaceRecognizerSF` | Manual cosine similarity | | |
| | **Threshold** | **0.363** (SFace-calibrated) | **0.7** *(incompatible with SFace scale)* | | |
| | **Model Format** | ✅ Already ONNX | Needs conversion to ONNX/TFLite | | |
| | **Liveness** | ✅ Landmark std-dev (static photo guard) | Blink detection (EAR via MediaPipe) | | |
| | **Frame Rate** | ~30 FPS cap (sleep 0.033s) | 15–20 FPS target | | |
| | **Complexity** | Low — single pipeline | Higher — 2-stage detect+track pipeline | | |
| | **Dependencies** | OpenCV only (no heavy ML framework) | Requires PyTorch or TFLite runtime | | |
| | **Status** | ✅ Production-ready | 🔧 Would require significant re-engineering | | |
| ### Key Conflicts to Note | |
| > ⚠️ **Threshold Incompatibility:** The proposed threshold of `0.7` is designed for FaceNet's cosine similarity scale. SFace cosine scores use a different calibration where `0.363` is the verified boundary. Applying `0.7` to the current SFace engine would reject almost all legitimate matches. | |
| > ℹ️ **Detector Redundancy:** YuNet already operates at a low-resolution input size and is fast enough for real-time CPU use. Adding a Correlation Tracker on top introduces state complexity with marginal speed gain for an attendance use case (stationary camera, controlled environment). | |
| > ✅ **ONNX is already done:** Both YuNet and SFace are `.onnx` format. The quantization recommendation is already fulfilled. | |
| --- | |
| ## 8. What's Already Done vs. What Can Be Improved | |
| ### ✅ Already Implemented | |
| - [x] ONNX model inference (YuNet + SFace) | |
| - [x] Real-time face detection at 640×480 | |
| - [x] 128-d embedding extraction with face alignment | |
| - [x] Cosine similarity matching with calibrated threshold | |
| - [x] In-memory embedding cache (reload on new employee registration) | |
| - [x] Liveness detection (static photo / spoof guard) | |
| - [x] Attendance toggle logic (check-in / check-out) | |
| - [x] 2-minute debounce cooldown per employee | |
| - [x] Real-time SSE alerts to frontend | |
| - [x] Spoof alerts with 10-second debounce | |
| - [x] Premium HUD overlay (neon bounding boxes, corner ticks, scan-line animation) | |
| - [x] Thread-safe singleton camera instance | |
| ### 🔧 Improvements Worth Adding | |
| | Priority | Improvement | Effort | Impact | | |
| |---|---|---|---| | |
| | ⚡ High | **Frame Skipping** — process every 2nd frame | Low | 30–50% CPU reduction | | |
| | ⚡ High | **Downscale for detection** — run YuNet on 320×240 copy, scale landmarks back to 640×480 for display | Low | Faster detection pass | | |
| | 👁️ Medium | **Blink Detection (EAR)** — upgrade to MediaPipe 468-point face mesh for eye-aspect-ratio liveness | Medium | Catches video/moving-photo spoofs | | |
| | 📁 Medium | **Multi-angle enrollment** — store 3–5 embeddings per employee from different angles | Low | Improves recognition rate | | |
| | 🚀 Low | **Async embedding extraction** — move embedding+matching to a worker thread, keep capture loop purely for frames | Medium | Eliminates blocking in capture loop | | |
| --- | |
| ## 9. Recommended Next Steps | |
| ``` | |
| Priority 1 (Quick Wins — 1-2 hours): | |
| ├── Add frame_count % 2 skip in _capture_loop() | |
| └── Add downscale copy for YuNet detect call | |
| Priority 2 (Accuracy — 1 day): | |
| └── Multi-angle face enrollment UI (capture 3-5 samples per person) | |
| Priority 3 (Liveness Upgrade — 2-3 days): | |
| └── Integrate MediaPipe FaceMesh for blink-based EAR liveness | |
| ``` | |
| --- | |
| *Generated by Antigravity AI · FaceDetection Project · June 2026* | |