Master Plan: LIPE V2 Gaze Estimation

1. Project Objective

Deliver a production-grade, real-time gaze estimation system (LIPE V2) capable of $\ge 30$ FPS on integrated CPU/iGPU hardware with $< 4.2^\circ$ angular error.

2. Technical Roadmap

Phase 1: Canonical Geometric Normalization & Dataset Preparation

Goal: Extract pose-invariant patches and prepare training data in HDF5 format.
Tasks:
1. Integrate MediaPipe Face Mesh (via Tasks API) for 478-landmark extraction (incl. Iris).
2. Implement Unified Affine Transformation in src/utils/preprocess.py.
3. [~] MPIIGaze HDF5 Streaming Pipeline (src/data/preprocess_mpii.py): (IN PROGRESS - Background Execution)
  - Landmark Consistency: Use MediaPipe landmarks.
  - Mapping & Gaze Rotation: Align 3D labels with rotated patches.
  - Zero-Centering: Pose-relative geometric features.
  - Serialization: Streaming to data/processed/pXX.h5.
4. Quality Control & Verification:
  - Visual verification script (verify_processed.py).
  - Verify affine stability and label alignment.

Phase 2: Model Architecture Implementation

Goal: Build the asymmetric dual-branch inference engine adaptable to multiple resolutions.
Tasks:
1. State A (Fine Branch): Implement Mini Conv-Embedder with Valid Padding (p=0).
2. State B (Coarse Branch): Implement Geometric MLP with LayerNorm & Dropout.
3. Switching Logic: Implement Residual Addition Fusion for State A/B routing.
4. HDF5 DataLoader: Implement GazeDataset for efficient training.
5. Architecture Verification: Confirm < 0.12 GFLOPs and tensor shape stability.

Phase 3: Training & Knowledge Distillation (KD)

Goal: Transfer deep features from a SOTA teacher to the LIPE V2 student.
Tasks:
1. Set up the Teacher Model (L2CS-Net) with ResNet-50 weights.
2. Implement Adaptive Wing Loss (AW Loss) and KD Loss in src/models/loss.py.
3. [~] Distillation Pipeline: Execute training (LOPO strategy).
  - Offline Label Caching script (src/data/generate_teacher_labels.py).
  - Training loop with Dynamic Weighting and WeightedRandomSampler (src/train.py).
  - [~] Batch Label Generation (IN PROGRESS - Background).
4. Validate against standard benchmarks (MPIIGaze).

Phase 4: Post-Processing & Hardware Optimization

Goal: Ensure real-time stability and meet hardware resource constraints (RAM/Compute).
Tasks:
1. Temporal Smoothing (Stabilization):
  - Implement One Euro Filter ($1€ Filter$) in src/utils/filters.py for adaptive jitter suppression.
  - Tune $\beta$ (speed) and $d_{cutoff}$ parameters to balance prediction latency and visual smoothness.
2. Resource & Inference Optimization:
  - Branch-Specific Logic: Implement logic to freeze CNN weights and bypass Image I/O during State B ($t \pmod 3 \neq 0$) to hit $< 45$ MB RAM target.
  - Quantization: Explore FP16 or INT8 Post-Training Quantization (PTQ) to reduce model size to $< 2$ MB and accelerate CPU inference.
  - Inference Wrapper: Build a high-level API to handle the dual-state switching transparently (src/inference_pipeline.py).
3. Benchmarking & Final Validation:
  - Metric Implementation: Implement 3D Angular Error formula for scientific validation (src/utils/metrics.py).
  - Hardware Profiling: Measured 1.93 MFLOPs and ~0.8ms latency per frame (Verified).
  - Final Benchmark: Run cross-subject evaluation on all 15 MPIIGaze folds and report final mean $\mathcal{E}_{angular}$.
4. Real-Time Demo Integration:
  - Build src/demo_live.py: Multi-threaded Camera I/O + MediaPipe + LIPE V2 Student integration (Ready).

3. Immediate Next Steps

Initialize project directory structure (src/models, src/utils, data/).
Update requirements.txt with scipy and h5py.
Implement the HDF5 Preprocessing Pipeline for MPIIGaze.
Perform a Random Sampling Validation on the generated HDF5 files.
Implement the Student Model (Mini Conv-Embedder).

5. Experimental Backlog (Ablation Studies)

These experiments are queued to find the absolute optimal configuration after the baseline Phase 3 is completed.

A-1: Input Resolution Study:
- Compare 16x16 vs 8x8 patch sizes.
- Requires updating src/data/preprocess_mpii.py to generate parallel HDF5 versions (e.g., pXX_v16.h5).
A-2: Fusion Mechanism Benchmarking:
- Concatenation vs Residual Addition vs Gated Fusion.
- Goal: Determine which method best preserves appearance tokens without bloating the MLP.
A-3: Hyperparameter Search (KD Tuning):
- Test Distillation Temperature $T \in [1, 3, 5]$.
- Evaluate impact of $w_{KD}$ weighting on convergence speed.
A-4: Architecture Scaling:
- Hidden layer scaling: Compare 128, 256, and 512 neurons in the Geometric MLP.

6. Phase 4: Post-Processing & Integration (Live Pipeline)

Status: Initialized.
Objective: Convert model predictions into a stable, high-frequency gaze stream.
Key Deliverables:
- src/inference_pipeline.py with temporal filtering (One Dollar/EMA).
- State-switching logic (State A vs. State B).
- Hardware profiling report.

7. Phase 5: Gold Standard Refinement (The Final Step)

Status: Planning.
Objective: Reach the absolute limit of accuracy (Target < 4.2°).
Key Deliverables:
- Feature Fusion: Combining Appearance and Geometry.
- Adaptive Wing Loss: Deep convergence for low-resolution patches.
- The "Slow and Steady" Run: 15-subject LOPO with high patience.
- Comparative Archiving: Baseline vs. Refined performance analysis.

8. Future Extensions (Proposal Research Only)

These tasks are currently outside the primary implementation scope and serve as future optimization targets.

Quantization-Aware Training (QAT) INT8:
- Integrate fake quantization modules during Phase 3 training.
- Evaluate potential for further reducing inference latency on specific hardware (AVX-512/OpenVINO).
- Analyze impact on angular error $\mathcal{E}_{angular}$ under extreme bit-width reduction.