docs/AGENT-READABLE.md

LIPE V2: SYSTEM WORKFLOW SPECIFICATION (INTEGRATED-HARDWARE COMPLIANT)

1. System Overview & Target Benchmarks

• System Name: LIPE V2 (Landmark-guided Image Patch Embedder V2).
• Domain: Driver Monitoring Systems (DMS) / Real-time Edge Gaze Estimation.
• Core Philosophy: Input-driven model compression via asymmetric spatio-temporal modeling. Optimized specifically for commodity host systems lacking discrete deployment accelerators.
• Target Optimization Pareto Frontier (Integrated Hardware):
  • Computational Cost: $< 0.12$ GFLOPs.
  • RAM Consumption: $< 45$ MB (Well within the 16GB system boundary).
  • Inference Speed: $\ge 30$ FPS running purely on host CPU / Integrated GPU (iGPU).
  • Accuracy: Mean 3D Angular Error $\mathcal{E}_{angular} < 4.2^{\circ}$.

---

2. Mathematical Framework & Optimization Core

• Asymmetric Input Resolution: Image domain is restricted to $K=4$ localized patches of size $8 \times 8 \times 1$ pixels centered around the pupil coordinate $P_{pupil}$.
• Loss Function: Adaptive Wing Loss (AW Loss) instead of MSE/L1 to resolve optical aliasing under low-resolution conditions: $$\mathcal{L}_{AW}(y,\hat{y})=\begin{cases}\omega~ln(1+|\frac{y-\hat{y}}{\epsilon}|^{n}) & \text{if } |y-\hat{y}|<\theta \ A|y-\hat{y}|-B & \text{if } |y-\hat{y}|\ge\theta\end{cases}$$
  • High-gradient penalty mechanism: Exponentially spikes the gradient when errors fall below 1 pixel, forcing the Fine Regressor to converge accurately at the pupil center without requiring dense multi-layer CNN parameters.

---

3. End-to-End Operational Pipeline (30 FPS Stream)

[Camera Raw Input (30 FPS)]
         │
         ▼
[MediaPipe Face Mesh] ──> Extract Geometry Vector (V_t) [Executed via CPU/iGPU OpenCL]
         │
         ▼
[Affine Scale Normalization] ──> Standardized Coordinate Vector (V_t^norm)
         │
         ├───► [Condition: t mod 3 == 0] (State A - 10 FPS)
         │         │
         │         ├──► Landmark-Guided Patch Division (4x 8x8 Patches)
         │         ├──► Mini Conv-Embedder (Shallow 2-layer CNN Feature Extraction)
         │         ├──► Token-pair Feature Injection (Geometry + Appearance Concatenation)
         │         └──► Fine Regressor (MLP) ──► G_t^fine
         │
         └───► [Condition: t mod 3 != 0] (State B - 20 FPS)
                   │
                   ├──► Freeze Image Memory I/O (Maintains Low RAM & CPU Overhead)
                   ├──► Coarse Regressor (MLP Using V_t^norm Only) ──► G_t^coarse
                   └──► Gaze Interpolation Stage (Dynamic Blending with G_t-1^fine) ──► G_t
         │
         ▼
[Lightweight Temporal Fusion Filter] (EMA / One Dollar Filter)
         │
         ▼
[Final Gaze Vector Output] ──► Pitch & Yaw (Radians)