docs/memory.md

# Project Memory: LIPE V2 Gaze Estimation

## 1. Accomplishments
### Architectural Design
- Defined the **Dual-State Asymmetric Pipeline** (State A: 10 FPS Appearance, State B: 20 FPS Geometry).
- Adopted **Knowledge Distillation** strategy using **L2CS-Net** as the teacher and **LIPE V2** as the student.
- Integrated **Adaptive Wing Loss (AW Loss)** to handle low-resolution optical aliasing.

### Implementation - Core Components
- **Teacher Model (`src/models/teacher.py`)**: Robust wrapper for L2CS-Net (ResNet-18/34/50) with separate Pitch/Yaw branches.
- **Preprocessing Utility (`src/utils/preprocess.py`)**: 
    - Integrated MediaPipe Tasks API (`face_landmarker.task`) with 478 points.
    - Implemented Unified Affine Transformation with standardized corner selection (viewer's left/right).
    - Added 3D gaze rotation logic to match rotated patches.
- **MPIIGaze Preprocessing (`src/data/preprocess_mpii.py`)**:
    - Developed an HDF5 streaming pipeline with GZIP compression.
    - Implemented Landmark Zero-Centering (pose-relative features).
- **Student Model (`src/models/student.py`)**:
    - Implemented `LIPEV2Student` with Shared CNN Backbone and Valid Padding (p=0).
    - Integrated LayerNorm in Geometric MLP for stability.
    - Achieved **~0.001 GFLOPs** (Verified).
- **HDF5 DataLoader (`src/data/dataset.py`)**: Efficient `GazeDataset` with lazy-handle management.

### Integration & Verification (2026-05-30)
- **End-to-End Demo (`demo_integration.py`)**: Successfully integrated Phase 1 (Data) and Phase 2 (Model).
    - **Data Integrity**: Verified `p03.h5` samples load with correct shapes: Patches `(4, 8, 8)` and Landmarks `(956)`.
    - **Model Compatibility**: `LIPEV2Student` performed successful inference on real processed data.
    - **Visual Check**: `demo_patch_p03.png` confirmed pupil-centered alignment after affine transform.
    - **I/O Stability**: Resolved HDF5 file locking issues.
- **Phase 3 Initialization**:
    - **Teacher Setup**: Secured pre-trained `resnet50.pt` weights and verified inference accuracy.
    - **Offline Distillation**: Launched background batch label generation (`src/data/batch_generate_labels.py`) to pre-calculate teacher logits for all 15 participants.
    - **Training Loop**: Implemented `src/train.py` with Subject-level WeightedRandomSampler and dynamic weighting (AW Loss + KL Divergence).
    - **LOPO Completion**: Finished full 15-subject LOPO training. Mean MAE: 5.12°. Best fold (p11): 4.19°.
- **Study A-1 (Resolution Boost)**:
    - **Objective**: Reach < 4.2° MAE by increasing patch size to 16x16.
    - **Status**: IN PROGRESS. `src/data/prepare_study_a1.py` launched in background to generate `_v16.h5` files with teacher labels.
    - **Pipeline Updates**: `src/data/dataset.py` and `src/train.py` updated to support `--version_suffix`.
- **Phase 4 Integration**:
    - **Smoothing**: Implemented **One Euro Filter ($1€ Filter$)** in `src/utils/filters.py` for real-time jitter suppression.
    - **Benchmarking**: Implemented 3D **Angular Error** formula in `src/utils/metrics.py`.
    - **Profiling**: Verified **1.93 MFLOPs** and **~0.8ms latency** per frame on CPU (State A).
    - **Live Demo**: Created `src/demo_live.py` with multi-threaded potential and dual-state switching.
    - **Refinement**: Modified model and preprocessing to support variable resolutions (e.g., 16x16 study).
- **Cross-Dataset Validation (2026-06-03)**:
    - **Strategy**: Established "Test first, Fine-tune later" approach for open datasets (Gaze360, GazeCapture).
    - **Verification**: Successfully tested MediaPipe landmark recognition on "in-the-wild" samples (`test_gaze.jpg`) with 100% success rate.
    - **Preprocessing**: Verified that the Affine Normalization and Patch Extraction pipeline is robust to non-laboratory samples.
- **Gaze360 Integrity & Baseline Reset (2026-06-04)**:
    - **Subject Splitting**: Implemented strict isolation using `person_identity`. Group A (Train: 0, 1, 17, 25, 49, 61) vs Group B (Test: 60, 62).
    - **Data Isolation**: Updated `src/data/preprocess_gaze360_robust.py` to output separate `_train_A.h5` and `_test_B.h5` files.
    - **Official Baseline**: Established **10.69° MAE** on clean Group B subset (Frontal +/- 45°) using coordinate correction (-1, -1).
    - **Fair Comparison**: All future domain adaptation results will be measured against this 10.69° reference point.

## 2. Technical Decisions
- **MediaPipe Tasks API**: Chosen for modern compatibility and 478-point landmark support.
- **Grayscale Normalization**: Minimizes computational load (< 45 MB target).
- **Residual Addition Fusion**: Replaced concatenation with feature addition (`App + Geo`) to ensure tensor shape stability across State A/B.
- **Zero-Centering Geometry**: Encourages the Coarse MLP to learn eye-shape rather than absolute positioning.
- **Adaptive Wing Loss**: Chosen to resolve optical aliasing in low-resolution (8x8) conditions.
- **Offline Label Caching**: Decided to store Teacher logits in HDF5 files during preprocessing to eliminate Teacher inference overhead during Student training.
- **Resolution Agnostic Architecture**: Updated `LIPEV2Student` to handle varying patch sizes (8x8, 16x16) automatically via Adaptive Pooling.
- **Gaze360 Evaluation Strategy**: Focus on "Frontal 20" subset to ensure fair comparison with MPIIGaze and minimize "Out-of-Distribution" noise.

## 3. Pending Tasks
- **Teacher Label Generation**: Wait for background process (p07-p14) to complete.
- **Phase 3 Training**: Execute full LOPO training on the complete distilled dataset.
- **Hardware Optimization**: Implement cycle-specific weight freezing to hit the 45MB RAM target.
- **Gaze360 Subset Acquisition**: Find a manageable mirror for Gaze360 metadata and images (avoiding 28GB overhead).
- **Zero-shot Cross-Dataset Test**: Evaluate LIPE V2 on Gaze360 without re-training to benchmark generalization.

## 4. Current State
- **Phase 1 (Preprocessing)**: COMPLETED.
- **Phase 2 (Model Implementation)**: COMPLETED & VERIFIED.
- **Phase 3 (Training)**: IN PROGRESS (Label generation background task).
- **Phase 4 (Post-Processing)**: INITIALIZED (Filters and metrics implemented).