File size: 2,920 Bytes
a10ba7f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | # LIPE V2: SYSTEM WORKFLOW SPECIFICATION (INTEGRATED-HARDWARE COMPLIANT)
## 1. System Overview & Target Benchmarks
* **System Name:** LIPE V2 (Landmark-guided Image Patch Embedder V2).
* **Domain:** Driver Monitoring Systems (DMS) / Real-time Edge Gaze Estimation.
* **Core Philosophy:** Input-driven model compression via asymmetric spatio-temporal modeling. Optimized specifically for commodity host systems lacking discrete deployment accelerators.
* **Target Optimization Pareto Frontier (Integrated Hardware):**
* Computational Cost: $< 0.12$ GFLOPs.
* RAM Consumption: $< 45$ MB (Well within the 16GB system boundary).
* Inference Speed: $\ge 30$ FPS running purely on host CPU / Integrated GPU (iGPU).
* Accuracy: Mean 3D Angular Error $\mathcal{E}_{angular} < 4.2^{\circ}$.
---
## 2. Mathematical Framework & Optimization Core
* **Asymmetric Input Resolution:** Image domain is restricted to $K=4$ localized patches of size $8 \times 8 \times 1$ pixels centered around the pupil coordinate $P_{pupil}$.
* **Loss Function:** **Adaptive Wing Loss (AW Loss)** instead of MSE/L1 to resolve optical aliasing under low-resolution conditions:
$$\mathcal{L}_{AW}(y,\hat{y})=\begin{cases}\omega~ln(1+|\frac{y-\hat{y}}{\epsilon}|^{n}) & \text{if } |y-\hat{y}|<\theta \\ A|y-\hat{y}|-B & \text{if } |y-\hat{y}|\ge\theta\end{cases}$$
* *High-gradient penalty mechanism:* Exponentially spikes the gradient when errors fall below 1 pixel, forcing the Fine Regressor to converge accurately at the pupil center without requiring dense multi-layer CNN parameters.
---
## 3. End-to-End Operational Pipeline (30 FPS Stream)
```text
[Camera Raw Input (30 FPS)]
│
▼
[MediaPipe Face Mesh] ──> Extract Geometry Vector (V_t) [Executed via CPU/iGPU OpenCL]
│
▼
[Affine Scale Normalization] ──> Standardized Coordinate Vector (V_t^norm)
│
├───► [Condition: t mod 3 == 0] (State A - 10 FPS)
│ │
│ ├──► Landmark-Guided Patch Division (4x 8x8 Patches)
│ ├──► Mini Conv-Embedder (Shallow 2-layer CNN Feature Extraction)
│ ├──► Token-pair Feature Injection (Geometry + Appearance Concatenation)
│ └──► Fine Regressor (MLP) ──► G_t^fine
│
└───► [Condition: t mod 3 != 0] (State B - 20 FPS)
│
├──► Freeze Image Memory I/O (Maintains Low RAM & CPU Overhead)
├──► Coarse Regressor (MLP Using V_t^norm Only) ──► G_t^coarse
└──► Gaze Interpolation Stage (Dynamic Blending with G_t-1^fine) ──► G_t
│
▼
[Lightweight Temporal Fusion Filter] (EMA / One Dollar Filter)
│
▼
[Final Gaze Vector Output] ──► Pitch & Yaw (Radians) |