# LIPE V2 Student Model Specification & Tracking ## 1. Architecture Overview The Student model is an asymmetric dual-branch network designed for high-efficiency gaze estimation on CPU/iGPU. ### Branch A: Mini Conv-Embedder (Appearance) * **Purpose:** Extract appearance features from eye patches. * **Input:** `(Batch, 4, 8, 8)` - 4 grayscale patches of 8x8 pixels. * **Layers:** * Shared CNN Backbone: * Conv2d(1 -> 16, k=3, p=0) + ReLU (Out: 6x6) * Conv2d(16 -> 32, k=3, p=0) + ReLU (Out: 4x4) * Conv2d(32 -> 64, k=3, p=0) + ReLU (Out: 2x2) * Global Average Pooling (GAP) (Out: 64) * Flatten & Reshape: $4 \times 64 = 256$ features. ### Branch B: Geometric MLP (Coarse) * **Purpose:** Extract geometric features from facial landmarks. * **Input:** `(Batch, 956)` - 478 landmarks (x, y) flattened and Zero-Centered. * **Layers:** * Linear(956 -> 256) + LayerNorm + ReLU + Dropout(0.05) * Linear(256 -> 256) + ReLU ### Fusion & Regression Heads * **Fusion:** Residual Addition (Appearance [256] + Geometry [256] = 256). * **Pitch Head:** Linear(256 -> 64) -> ReLU -> Linear(64 -> 1) * **Yaw Head:** Linear(256 -> 64) -> ReLU -> Linear(64 -> 1) --- ## 2. Technical Targets | Metric | Target Value | Current Status | | :--- | :--- | :--- | | **Computational Cost** | < 0.12 GFLOPs | ~0.001 GFLOPs (Verified) | | **RAM Usage** | < 45 MB | ~1.5 MB (Weights only) | | **Inference Speed** | >= 30 FPS (CPU) | TBD (Estimated >> 100 FPS) | | **Angular Error** | < 4.2° | TBD | --- ## 3. Implementation Checklist - [x] Define `LIPEV2Student` class in `src/models/student.py`. - [x] Implement `Mini Conv-Embedder` with shared weights. - [x] Implement `Geometric MLP` branch. - [x] Implement `forward` method with State A/B switching logic (Residual Addition). - [x] Implement `AdaptiveWingLoss` in `src/models/loss.py`. - [x] Weight initialization (Kaiming/Xavier). - [x] Verify forward pass with dummy tensors. - [ ] Estimate GFLOPs using `thop` or manual calculation. --- ## 4. Input/Output Specs * **Input Patches:** `torch.Tensor` of shape `(N, 4, 8, 8)`, dtype `float32`, range `[0, 1]`. * **Input Landmarks:** `torch.Tensor` of shape `(N, 956)`, dtype `float32`, Zero-Centered. * **Output:** `torch.Tensor` of shape `(N, 2)` representing `[Pitch, Yaw]` in Radians.