Project Memory: LIPE V2 Gaze Estimation

1. Accomplishments

Architectural Design

Defined the Dual-State Asymmetric Pipeline (State A: 10 FPS Appearance, State B: 20 FPS Geometry).
Adopted Knowledge Distillation strategy using L2CS-Net as the teacher and LIPE V2 as the student.
Integrated Adaptive Wing Loss (AW Loss) to handle low-resolution optical aliasing.

Implementation - Core Components

Teacher Model (src/models/teacher.py): Robust wrapper for L2CS-Net (ResNet-18/34/50) with separate Pitch/Yaw branches.
Preprocessing Utility (src/utils/preprocess.py):
- Integrated MediaPipe Tasks API (face_landmarker.task) with 478 points.
- Implemented Unified Affine Transformation with standardized corner selection (viewer's left/right).
- Added 3D gaze rotation logic to match rotated patches.
MPIIGaze Preprocessing (src/data/preprocess_mpii.py):
- Developed an HDF5 streaming pipeline with GZIP compression.
- Implemented Landmark Zero-Centering (pose-relative features).
Student Model (src/models/student.py):
- Implemented LIPEV2Student with Shared CNN Backbone and Valid Padding (p=0).
- Integrated LayerNorm in Geometric MLP for stability.
- Achieved ~0.001 GFLOPs (Verified).
HDF5 DataLoader (src/data/dataset.py): Efficient GazeDataset with lazy-handle management.

Integration & Verification (2026-05-30)

End-to-End Demo (demo_integration.py): Successfully integrated Phase 1 (Data) and Phase 2 (Model).
- Data Integrity: Verified p03.h5 samples load with correct shapes: Patches (4, 8, 8) and Landmarks (956).
- Model Compatibility: LIPEV2Student performed successful inference on real processed data.
- Visual Check: demo_patch_p03.png confirmed pupil-centered alignment after affine transform.
- I/O Stability: Resolved HDF5 file locking issues.
Phase 3 Initialization:
- Teacher Setup: Secured pre-trained resnet50.pt weights and verified inference accuracy.
- Offline Distillation: Launched background batch label generation (src/data/batch_generate_labels.py) to pre-calculate teacher logits for all 15 participants.
- Training Loop: Implemented src/train.py with Subject-level WeightedRandomSampler and dynamic weighting (AW Loss + KL Divergence).
- LOPO Completion: Finished full 15-subject LOPO training. Mean MAE: 5.12°. Best fold (p11): 4.19°.
Study A-1 (Resolution Boost):
- Objective: Reach < 4.2° MAE by increasing patch size to 16x16.
- Status: IN PROGRESS. src/data/prepare_study_a1.py launched in background to generate _v16.h5 files with teacher labels.
- Pipeline Updates: src/data/dataset.py and src/train.py updated to support --version_suffix.
Phase 4 Integration:
- Smoothing: Implemented One Euro Filter ($1€ Filter$) in src/utils/filters.py for real-time jitter suppression.
- Benchmarking: Implemented 3D Angular Error formula in src/utils/metrics.py.
- Profiling: Verified 1.93 MFLOPs and ~0.8ms latency per frame on CPU (State A).
- Live Demo: Created src/demo_live.py with multi-threaded potential and dual-state switching.
- Refinement: Modified model and preprocessing to support variable resolutions (e.g., 16x16 study).
Cross-Dataset Validation (2026-06-03):
- Strategy: Established "Test first, Fine-tune later" approach for open datasets (Gaze360, GazeCapture).
- Verification: Successfully tested MediaPipe landmark recognition on "in-the-wild" samples (test_gaze.jpg) with 100% success rate.
- Preprocessing: Verified that the Affine Normalization and Patch Extraction pipeline is robust to non-laboratory samples.
Gaze360 Integrity & Baseline Reset (2026-06-04):
- Subject Splitting: Implemented strict isolation using person_identity. Group A (Train: 0, 1, 17, 25, 49, 61) vs Group B (Test: 60, 62).
- Data Isolation: Updated src/data/preprocess_gaze360_robust.py to output separate _train_A.h5 and _test_B.h5 files.
- Official Baseline: Established 10.69° MAE on clean Group B subset (Frontal +/- 45°) using coordinate correction (-1, -1).
- Fair Comparison: All future domain adaptation results will be measured against this 10.69° reference point.

2. Technical Decisions

MediaPipe Tasks API: Chosen for modern compatibility and 478-point landmark support.
Grayscale Normalization: Minimizes computational load (< 45 MB target).
Residual Addition Fusion: Replaced concatenation with feature addition (App + Geo) to ensure tensor shape stability across State A/B.
Zero-Centering Geometry: Encourages the Coarse MLP to learn eye-shape rather than absolute positioning.
Adaptive Wing Loss: Chosen to resolve optical aliasing in low-resolution (8x8) conditions.
Offline Label Caching: Decided to store Teacher logits in HDF5 files during preprocessing to eliminate Teacher inference overhead during Student training.
Resolution Agnostic Architecture: Updated LIPEV2Student to handle varying patch sizes (8x8, 16x16) automatically via Adaptive Pooling.
Gaze360 Evaluation Strategy: Focus on "Frontal 20" subset to ensure fair comparison with MPIIGaze and minimize "Out-of-Distribution" noise.

3. Pending Tasks

Teacher Label Generation: Wait for background process (p07-p14) to complete.
Phase 3 Training: Execute full LOPO training on the complete distilled dataset.
Hardware Optimization: Implement cycle-specific weight freezing to hit the 45MB RAM target.
Gaze360 Subset Acquisition: Find a manageable mirror for Gaze360 metadata and images (avoiding 28GB overhead).
Zero-shot Cross-Dataset Test: Evaluate LIPE V2 on Gaze360 without re-training to benchmark generalization.

4. Current State

Phase 1 (Preprocessing): COMPLETED.
Phase 2 (Model Implementation): COMPLETED & VERIFIED.
Phase 3 (Training): IN PROGRESS (Label generation background task).
Phase 4 (Post-Processing): INITIALIZED (Filters and metrics implemented).