File size: 6,641 Bytes
a10ba7f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | # Master Plan: LIPE V2 Gaze Estimation
## 1. Project Objective
Deliver a production-grade, real-time gaze estimation system (LIPE V2) capable of $\ge 30$ FPS on integrated CPU/iGPU hardware with $< 4.2^\circ$ angular error.
## 2. Technical Roadmap
### Phase 1: Canonical Geometric Normalization & Dataset Preparation
* **Goal:** Extract pose-invariant patches and prepare training data in HDF5 format.
* **Tasks:**
1. [x] Integrate **MediaPipe Face Mesh** (via Tasks API) for 478-landmark extraction (incl. Iris).
2. [x] Implement **Unified Affine Transformation** in `src/utils/preprocess.py`.
3. [~] **MPIIGaze HDF5 Streaming Pipeline (`src/data/preprocess_mpii.py`):** (IN PROGRESS - Background Execution)
* [x] Landmark Consistency: Use MediaPipe landmarks.
* [x] Mapping & Gaze Rotation: Align 3D labels with rotated patches.
* [x] Zero-Centering: Pose-relative geometric features.
* [ ] Serialization: Streaming to `data/processed/pXX.h5`.
4. [x] **Quality Control & Verification:**
* [x] Visual verification script (`verify_processed.py`).
* [x] Verify affine stability and label alignment.
### Phase 2: Model Architecture Implementation
* **Goal:** Build the asymmetric dual-branch inference engine adaptable to multiple resolutions.
* **Tasks:**
1. [x] **State A (Fine Branch):** Implement **Mini Conv-Embedder** with Valid Padding (p=0).
2. [x] **State B (Coarse Branch):** Implement **Geometric MLP** with LayerNorm & Dropout.
3. [x] **Switching Logic:** Implement **Residual Addition Fusion** for State A/B routing.
4. [x] **HDF5 DataLoader:** Implement `GazeDataset` for efficient training.
5. [x] **Architecture Verification:** Confirm < 0.12 GFLOPs and tensor shape stability.
### Phase 3: Training & Knowledge Distillation (KD)
* **Goal:** Transfer deep features from a SOTA teacher to the LIPE V2 student.
* **Tasks:**
1. [x] Set up the **Teacher Model (L2CS-Net)** with ResNet-50 weights.
2. [x] Implement **Adaptive Wing Loss (AW Loss)** and **KD Loss** in `src/models/loss.py`.
3. [~] **Distillation Pipeline:** Execute training (LOPO strategy).
* [x] Offline Label Caching script (`src/data/generate_teacher_labels.py`).
* [x] Training loop with Dynamic Weighting and WeightedRandomSampler (`src/train.py`).
* [~] Batch Label Generation (IN PROGRESS - Background).
4. [ ] Validate against standard benchmarks (MPIIGaze).
### Phase 4: Post-Processing & Hardware Optimization
* **Goal:** Ensure real-time stability and meet hardware resource constraints (RAM/Compute).
* **Tasks:**
1. **Temporal Smoothing (Stabilization):**
* [ ] Implement **One Euro Filter ($1€ Filter$)** in `src/utils/filters.py` for adaptive jitter suppression.
* [ ] Tune $\beta$ (speed) and $d_{cutoff}$ parameters to balance prediction latency and visual smoothness.
2. **Resource & Inference Optimization:**
* [ ] **Branch-Specific Logic:** Implement logic to freeze CNN weights and bypass Image I/O during State B ($t \pmod 3 \neq 0$) to hit $< 45$ MB RAM target.
* [ ] **Quantization:** Explore FP16 or INT8 Post-Training Quantization (PTQ) to reduce model size to $< 2$ MB and accelerate CPU inference.
* [x] **Inference Wrapper:** Build a high-level API to handle the dual-state switching transparently (`src/inference_pipeline.py`).
3. **Benchmarking & Final Validation:**
* [x] **Metric Implementation:** Implement 3D Angular Error formula for scientific validation (`src/utils/metrics.py`).
* [x] **Hardware Profiling:** Measured 1.93 MFLOPs and ~0.8ms latency per frame (Verified).
* [ ] **Final Benchmark:** Run cross-subject evaluation on all 15 MPIIGaze folds and report final mean $\mathcal{E}_{angular}$.
4. **Real-Time Demo Integration:**
* [x] Build `src/demo_live.py`: Multi-threaded Camera I/O + MediaPipe + LIPE V2 Student integration (Ready).
## 3. Immediate Next Steps
1. [x] Initialize project directory structure (`src/models`, `src/utils`, `data/`).
2. [ ] Update `requirements.txt` with `scipy` and `h5py`.
3. [ ] Implement the **HDF5 Preprocessing Pipeline** for MPIIGaze.
4. [ ] Perform a **Random Sampling Validation** on the generated HDF5 files.
5. [ ] Implement the Student Model (Mini Conv-Embedder).
---
## 5. Experimental Backlog (Ablation Studies)
*These experiments are queued to find the absolute optimal configuration after the baseline Phase 3 is completed.*
* **A-1: Input Resolution Study:**
* Compare **16x16** vs **8x8** patch sizes.
* Requires updating `src/data/preprocess_mpii.py` to generate parallel HDF5 versions (e.g., `pXX_v16.h5`).
* **A-2: Fusion Mechanism Benchmarking:**
* **Concatenation** vs **Residual Addition** vs **Gated Fusion**.
* Goal: Determine which method best preserves appearance tokens without bloating the MLP.
* **A-3: Hyperparameter Search (KD Tuning):**
* Test Distillation Temperature $T \in [1, 3, 5]$.
* Evaluate impact of $w_{KD}$ weighting on convergence speed.
* **A-4: Architecture Scaling:**
* Hidden layer scaling: Compare **128**, **256**, and **512** neurons in the Geometric MLP.
## 6. Phase 4: Post-Processing & Integration (Live Pipeline)
* **Status**: Initialized.
* **Objective**: Convert model predictions into a stable, high-frequency gaze stream.
* **Key Deliverables**:
* `src/inference_pipeline.py` with temporal filtering (One Dollar/EMA).
* State-switching logic (State A vs. State B).
* Hardware profiling report.
## 7. Phase 5: Gold Standard Refinement (The Final Step)
* **Status**: Planning.
* **Objective**: Reach the absolute limit of accuracy (Target < 4.2°).
* **Key Deliverables**:
* **Feature Fusion**: Combining Appearance and Geometry.
* **Adaptive Wing Loss**: Deep convergence for low-resolution patches.
* **The "Slow and Steady" Run**: 15-subject LOPO with high patience.
* **Comparative Archiving**: Baseline vs. Refined performance analysis.
## 8. Future Extensions (Proposal Research Only)
*These tasks are currently outside the primary implementation scope and serve as future optimization targets.*
* **Quantization-Aware Training (QAT) INT8:**
* Integrate fake quantization modules during Phase 3 training.
* Evaluate potential for further reducing inference latency on specific hardware (AVX-512/OpenVINO).
* Analyze impact on angular error $\mathcal{E}_{angular}$ under extreme bit-width reduction.
|