docs/MASTER_PLAN.md

# Master Plan: LIPE V2 Gaze Estimation

## 1. Project Objective
Deliver a production-grade, real-time gaze estimation system (LIPE V2) capable of $\ge 30$ FPS on integrated CPU/iGPU hardware with $< 4.2^\circ$ angular error.

## 2. Technical Roadmap

### Phase 1: Canonical Geometric Normalization & Dataset Preparation
*   **Goal:** Extract pose-invariant patches and prepare training data in HDF5 format.
*   **Tasks:**
    1.  [x] Integrate **MediaPipe Face Mesh** (via Tasks API) for 478-landmark extraction (incl. Iris).
    2.  [x] Implement **Unified Affine Transformation** in `src/utils/preprocess.py`.
    3.  [~] **MPIIGaze HDF5 Streaming Pipeline (`src/data/preprocess_mpii.py`):** (IN PROGRESS - Background Execution)
        *   [x] Landmark Consistency: Use MediaPipe landmarks.
        *   [x] Mapping & Gaze Rotation: Align 3D labels with rotated patches.
        *   [x] Zero-Centering: Pose-relative geometric features.
        *   [ ] Serialization: Streaming to `data/processed/pXX.h5`.
    4.  [x] **Quality Control & Verification:**
        *   [x] Visual verification script (`verify_processed.py`).
        *   [x] Verify affine stability and label alignment.

### Phase 2: Model Architecture Implementation
*   **Goal:** Build the asymmetric dual-branch inference engine adaptable to multiple resolutions.
*   **Tasks:**
    1.  [x] **State A (Fine Branch):** Implement **Mini Conv-Embedder** with Valid Padding (p=0).
    2.  [x] **State B (Coarse Branch):** Implement **Geometric MLP** with LayerNorm & Dropout.
    3.  [x] **Switching Logic:** Implement **Residual Addition Fusion** for State A/B routing.
    4.  [x] **HDF5 DataLoader:** Implement `GazeDataset` for efficient training.
    5.  [x] **Architecture Verification:** Confirm < 0.12 GFLOPs and tensor shape stability.

### Phase 3: Training & Knowledge Distillation (KD)
*   **Goal:** Transfer deep features from a SOTA teacher to the LIPE V2 student.
*   **Tasks:**
    1.  [x] Set up the **Teacher Model (L2CS-Net)** with ResNet-50 weights.
    2.  [x] Implement **Adaptive Wing Loss (AW Loss)** and **KD Loss** in `src/models/loss.py`.
    3.  [~] **Distillation Pipeline:** Execute training (LOPO strategy).
        *   [x] Offline Label Caching script (`src/data/generate_teacher_labels.py`).
        *   [x] Training loop with Dynamic Weighting and WeightedRandomSampler (`src/train.py`).
        *   [~] Batch Label Generation (IN PROGRESS - Background).
    4.  [ ] Validate against standard benchmarks (MPIIGaze).

### Phase 4: Post-Processing & Hardware Optimization
*   **Goal:** Ensure real-time stability and meet hardware resource constraints (RAM/Compute).
*   **Tasks:**
    1.  **Temporal Smoothing (Stabilization):**
        *   [ ] Implement **One Euro Filter ($1€ Filter$)** in `src/utils/filters.py` for adaptive jitter suppression.
        *   [ ] Tune $\beta$ (speed) and $d_{cutoff}$ parameters to balance prediction latency and visual smoothness.
    2.  **Resource & Inference Optimization:**
        *   [ ] **Branch-Specific Logic:** Implement logic to freeze CNN weights and bypass Image I/O during State B ($t \pmod 3 \neq 0$) to hit $< 45$ MB RAM target.
        *   [ ] **Quantization:** Explore FP16 or INT8 Post-Training Quantization (PTQ) to reduce model size to $< 2$ MB and accelerate CPU inference.
        *   [x] **Inference Wrapper:** Build a high-level API to handle the dual-state switching transparently (`src/inference_pipeline.py`).
    3.  **Benchmarking & Final Validation:**
        *   [x] **Metric Implementation:** Implement 3D Angular Error formula for scientific validation (`src/utils/metrics.py`).
        *   [x] **Hardware Profiling:** Measured 1.93 MFLOPs and ~0.8ms latency per frame (Verified).
        *   [ ] **Final Benchmark:** Run cross-subject evaluation on all 15 MPIIGaze folds and report final mean $\mathcal{E}_{angular}$.
    4.  **Real-Time Demo Integration:**
        *   [x] Build `src/demo_live.py`: Multi-threaded Camera I/O + MediaPipe + LIPE V2 Student integration (Ready).

## 3. Immediate Next Steps
1.  [x] Initialize project directory structure (`src/models`, `src/utils`, `data/`).
2.  [ ] Update `requirements.txt` with `scipy` and `h5py`.
3.  [ ] Implement the **HDF5 Preprocessing Pipeline** for MPIIGaze.
4.  [ ] Perform a **Random Sampling Validation** on the generated HDF5 files.
5.  [ ] Implement the Student Model (Mini Conv-Embedder).

---

## 5. Experimental Backlog (Ablation Studies)
*These experiments are queued to find the absolute optimal configuration after the baseline Phase 3 is completed.*

*   **A-1: Input Resolution Study:**
    *   Compare **16x16** vs **8x8** patch sizes.
    *   Requires updating `src/data/preprocess_mpii.py` to generate parallel HDF5 versions (e.g., `pXX_v16.h5`).
*   **A-2: Fusion Mechanism Benchmarking:**
    *   **Concatenation** vs **Residual Addition** vs **Gated Fusion**.
    *   Goal: Determine which method best preserves appearance tokens without bloating the MLP.
*   **A-3: Hyperparameter Search (KD Tuning):**
    *   Test Distillation Temperature $T \in [1, 3, 5]$.
    *   Evaluate impact of $w_{KD}$ weighting on convergence speed.
*   **A-4: Architecture Scaling:**
    *   Hidden layer scaling: Compare **128**, **256**, and **512** neurons in the Geometric MLP.

## 6. Phase 4: Post-Processing & Integration (Live Pipeline)
*   **Status**: Initialized.
*   **Objective**: Convert model predictions into a stable, high-frequency gaze stream.
*   **Key Deliverables**:
    *   `src/inference_pipeline.py` with temporal filtering (One Dollar/EMA).
    *   State-switching logic (State A vs. State B).
    *   Hardware profiling report.

## 7. Phase 5: Gold Standard Refinement (The Final Step)
*   **Status**: Planning.
*   **Objective**: Reach the absolute limit of accuracy (Target < 4.2°).
*   **Key Deliverables**:
    *   **Feature Fusion**: Combining Appearance and Geometry.
    *   **Adaptive Wing Loss**: Deep convergence for low-resolution patches.
    *   **The "Slow and Steady" Run**: 15-subject LOPO with high patience.
    *   **Comparative Archiving**: Baseline vs. Refined performance analysis.

## 8. Future Extensions (Proposal Research Only)
*These tasks are currently outside the primary implementation scope and serve as future optimization targets.*

*   **Quantization-Aware Training (QAT) INT8:**
    *   Integrate fake quantization modules during Phase 3 training.
    *   Evaluate potential for further reducing inference latency on specific hardware (AVX-512/OpenVINO).
    *   Analyze impact on angular error $\mathcal{E}_{angular}$ under extreme bit-width reduction.