File size: 2,365 Bytes
a10ba7f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | # LIPE V2 Student Model Specification & Tracking
## 1. Architecture Overview
The Student model is an asymmetric dual-branch network designed for high-efficiency gaze estimation on CPU/iGPU.
### Branch A: Mini Conv-Embedder (Appearance)
* **Purpose:** Extract appearance features from eye patches.
* **Input:** `(Batch, 4, 8, 8)` - 4 grayscale patches of 8x8 pixels.
* **Layers:**
* Shared CNN Backbone:
* Conv2d(1 -> 16, k=3, p=0) + ReLU (Out: 6x6)
* Conv2d(16 -> 32, k=3, p=0) + ReLU (Out: 4x4)
* Conv2d(32 -> 64, k=3, p=0) + ReLU (Out: 2x2)
* Global Average Pooling (GAP) (Out: 64)
* Flatten & Reshape: $4 \times 64 = 256$ features.
### Branch B: Geometric MLP (Coarse)
* **Purpose:** Extract geometric features from facial landmarks.
* **Input:** `(Batch, 956)` - 478 landmarks (x, y) flattened and Zero-Centered.
* **Layers:**
* Linear(956 -> 256) + LayerNorm + ReLU + Dropout(0.05)
* Linear(256 -> 256) + ReLU
### Fusion & Regression Heads
* **Fusion:** Residual Addition (Appearance [256] + Geometry [256] = 256).
* **Pitch Head:** Linear(256 -> 64) -> ReLU -> Linear(64 -> 1)
* **Yaw Head:** Linear(256 -> 64) -> ReLU -> Linear(64 -> 1)
---
## 2. Technical Targets
| Metric | Target Value | Current Status |
| :--- | :--- | :--- |
| **Computational Cost** | < 0.12 GFLOPs | ~0.001 GFLOPs (Verified) |
| **RAM Usage** | < 45 MB | ~1.5 MB (Weights only) |
| **Inference Speed** | >= 30 FPS (CPU) | TBD (Estimated >> 100 FPS) |
| **Angular Error** | < 4.2° | TBD |
---
## 3. Implementation Checklist
- [x] Define `LIPEV2Student` class in `src/models/student.py`.
- [x] Implement `Mini Conv-Embedder` with shared weights.
- [x] Implement `Geometric MLP` branch.
- [x] Implement `forward` method with State A/B switching logic (Residual Addition).
- [x] Implement `AdaptiveWingLoss` in `src/models/loss.py`.
- [x] Weight initialization (Kaiming/Xavier).
- [x] Verify forward pass with dummy tensors.
- [ ] Estimate GFLOPs using `thop` or manual calculation.
---
## 4. Input/Output Specs
* **Input Patches:** `torch.Tensor` of shape `(N, 4, 8, 8)`, dtype `float32`, range `[0, 1]`.
* **Input Landmarks:** `torch.Tensor` of shape `(N, 956)`, dtype `float32`, Zero-Centered.
* **Output:** `torch.Tensor` of shape `(N, 2)` representing `[Pitch, Yaw]` in Radians.
|