| Step 1: Pre-processing & Scale Stabilization (Executes at 30 FPS) |
|
|
| Capture local eye landmark coordinates using an ultra-lightweight geometric extractor (e.g., MediaPipe Face Mesh): |
|
|
| $$\zeta_{t}=\{P_{inner},P_{outer},P_{pupil},P_{eyelid}\}\in R^{4\times2}$$ |
| |
| |
| |
| Apply an adaptive affine scaling matrix $\mathcal{T}_{scale}$ to normalize distance/scale variations: |
|
|
|
|
| $$V_{t}^{norm}=\mathcal{T}_{scale}(V_{t})$$ |
| |
| |
| |
| --- |
| |
| Step 2: Asynchronous Multi-rate Branch Routing |
| |
| The system splits operations based on the current frame index $t$: |
| |
| CASE I: State A (Active Appearance Frame) — Triggered at $t \pmod 3 = 0$ (10 FPS) |
| |
| * |
| **Landmark-Guided Patch Division:** Isolate the eye socket and extract $K=4$ sub-patches $I_{k}\in R^{8\times8\times1}$ using the localized pupil center $P_{pupil}$. |
| |
| |
| * |
| **Mini Conv-Embedder:** Pass each patch through a shallow 3-layer CNN to obtain appearance tokens: |
| |
| |
| |
| $$f_{k}^{app}=\text{Flatten}(\text{MaxPool}(\text{Conv2D}(I_{k})))\in R^{D}$$ |
| |
| |
| |
| |
| |
| * |
| **Token-pair Feature Injection:** Project appearance and geometric markers onto a shared latent space, then concatenate into joint structural-appearance tokens: |
| |
| |
| |
| $$T_{k}=[\mathcal{W}_{app}f_{k}^{app}||\mathcal{W}_{geo}P_{k}]$$ |
|
|
|
|
|
|
|
|
|
|
| * |
| **Fine Regressor:** Feed $T=\text{Concat}(T_{1},...,T_{K})$ into a flat MLP block to infer fine-grained gaze vectors: |
|
|
|
|
|
|
| $$G_{t}^{fine}=(\theta_{fine},\phi_{fine})$$ |
| |
| |
| |
| |
| |
| |
| CASE II: State B (Missing Appearance Frame) — Triggered at $t \pmod 3 \neq 0$ (20 FPS) |
| |
| * |
| **CPU Duty Cycle Optimization:** Power down the entire CNN inference branch and freeze memory image I/O operations. |
| |
| |
| * |
| **Coarse Regressor:** Pass the normalized geometric vector $V_{t}^{norm}$ directly through an independent lightweight MLP: |
|
|
|
|
|
|
| $$G_{t}^{coarse}=\mathcal{M}\mathcal{L}\mathcal{P}_{coarse}(V_{t}^{norm})$$ |
| |
| |
| |
| |
| |
| * |
| **Gaze Interpolation:** Compute the final coordinates by dynamically blending the current coarse prediction with the cached fine prediction from the last active frame: |
| |
| |
| |
| $$G_{t}=\alpha\cdot G_{t}^{coarse}+(1-\alpha)\cdot G_{t-1}^{fine}$$ |
|
|
|
|
|
|
|
|
|
|
|
|
| (where $\alpha \in [0, 1]$ is dynamically tuned based on eye landmark movement velocity ). |
|
|
|
|
|
|
| --- |
|
|
| ### Step 3: Lightweight Temporal Fusion (Post-Processing) |
|
|
| Pass the output vector $G_t$ through an Exponential Moving Average (EMA) or a One Dollar Filter to suppress tracking jitter and prevent spatial lag across state-switching intervals: |
| |
| |
| $$G_{smooth} = \beta \cdot G_t + (1 - \beta) \cdot G_{smooth, t-1}$$ |
|
|
| --- |
|
|
| ## 4. Hardware and Deployment Optimization Roadmap |
|
|
| ### A. Training & Knowledge Distillation (KD) Strategy |
|
|
| * **Hardware Configuration:** NVIDIA GeForce RTX 3060 Laptop GPU. |
| * **Framework:** PyTorch 2.x with CUDA acceleration. |
| * **Knowledge Distillation Pipeline:** |
| * **Teacher Network:** SOTA Appearance-based model (e.g., L2CS-Net with a ResNet-50 backbone) operating on high-resolution $224 \times 224$ inputs. |
| * **Student Network:** LIPE V2 shallow MLP architecture. |
| * **Mechanism:** Train LIPE V2 to replicate the soft output distributions and logit mappings of L2CS-Net, retaining deep geometric representations inside a highly compact model. |
