| # LIPE V2: SYSTEM WORKFLOW SPECIFICATION (INTEGRATED-HARDWARE COMPLIANT) |
|
|
| ## 1. System Overview & Target Benchmarks |
| * **System Name:** LIPE V2 (Landmark-guided Image Patch Embedder V2). |
| * **Domain:** Driver Monitoring Systems (DMS) / Real-time Edge Gaze Estimation. |
| * **Core Philosophy:** Input-driven model compression via asymmetric spatio-temporal modeling. Optimized specifically for commodity host systems lacking discrete deployment accelerators. |
| * **Target Optimization Pareto Frontier (Integrated Hardware):** |
| * Computational Cost: $< 0.12$ GFLOPs. |
| * RAM Consumption: $< 45$ MB (Well within the 16GB system boundary). |
| * Inference Speed: $\ge 30$ FPS running purely on host CPU / Integrated GPU (iGPU). |
| * Accuracy: Mean 3D Angular Error $\mathcal{E}_{angular} < 4.2^{\circ}$. |
| |
| --- |
| |
| ## 2. Mathematical Framework & Optimization Core |
| * **Asymmetric Input Resolution:** Image domain is restricted to $K=4$ localized patches of size $8 \times 8 \times 1$ pixels centered around the pupil coordinate $P_{pupil}$. |
| * **Loss Function:** **Adaptive Wing Loss (AW Loss)** instead of MSE/L1 to resolve optical aliasing under low-resolution conditions: |
| $$\mathcal{L}_{AW}(y,\hat{y})=\begin{cases}\omega~ln(1+|\frac{y-\hat{y}}{\epsilon}|^{n}) & \text{if } |y-\hat{y}|<\theta \\ A|y-\hat{y}|-B & \text{if } |y-\hat{y}|\ge\theta\end{cases}$$ |
| * *High-gradient penalty mechanism:* Exponentially spikes the gradient when errors fall below 1 pixel, forcing the Fine Regressor to converge accurately at the pupil center without requiring dense multi-layer CNN parameters. |
| |
| --- |
| |
| ## 3. End-to-End Operational Pipeline (30 FPS Stream) |
| |
| ```text |
| [Camera Raw Input (30 FPS)] |
| │ |
| ▼ |
| [MediaPipe Face Mesh] ──> Extract Geometry Vector (V_t) [Executed via CPU/iGPU OpenCL] |
| │ |
| ▼ |
| [Affine Scale Normalization] ──> Standardized Coordinate Vector (V_t^norm) |
| │ |
| ├───► [Condition: t mod 3 == 0] (State A - 10 FPS) |
| │ │ |
| │ ├──► Landmark-Guided Patch Division (4x 8x8 Patches) |
| │ ├──► Mini Conv-Embedder (Shallow 2-layer CNN Feature Extraction) |
| │ ├──► Token-pair Feature Injection (Geometry + Appearance Concatenation) |
| │ └──► Fine Regressor (MLP) ──► G_t^fine |
| │ |
| └───► [Condition: t mod 3 != 0] (State B - 20 FPS) |
| │ |
| ├──► Freeze Image Memory I/O (Maintains Low RAM & CPU Overhead) |
| ├──► Coarse Regressor (MLP Using V_t^norm Only) ──► G_t^coarse |
| └──► Gaze Interpolation Stage (Dynamic Blending with G_t-1^fine) ──► G_t |
| │ |
| ▼ |
| [Lightweight Temporal Fusion Filter] (EMA / One Dollar Filter) |
| │ |
| ▼ |
| [Final Gaze Vector Output] ──► Pitch & Yaw (Radians) |
