docs/PHASE3_PLAN.md

Phase 3 Master Plan: Knowledge Distillation (KD)

1. Executive Summary

Phase 3 focuses on "distilling" the spatial awareness and high-accuracy gaze features of a State-of-the-Art Teacher (L2CS-Net) into our lightweight Student (LIPE V2). By training the Student to replicate the Teacher's output distribution, we aim to achieve professional-grade accuracy (< 4.2° error) at a fraction of the computational cost (~0.001 GFLOPs).

2. Technical Architecture

A. The Teacher (Feature Provider)

• Model: L2CS-Net with ResNet backbone.
• Primary Role: Generate "Soft Labels" for every sample.
• Optimization: Offline Distillation. Labels are pre-computed to eliminate inference latency during the main training loop.

B. The Student (Feature Trainee)

• Model: LIPEV2Student (Refined Architecture).
• Primary Role: Learn to predict gaze using only 8x8 patches and zero-centered landmarks.

C. Hybrid Loss Strategy ($L_{total}$)

$L_{total} = w_1 \cdot L_{AW} + w_2 \cdot L_{KD}$

Dynamic Weighting Schedule (Refined with Gradient Smoothing)

To prevent Gradient Shock, transitions between stages use Linear Annealing over a 10-epoch window:

• Stage 0: Sanity Check (Hard-Threshold): Fixed $w_1=1.0, w_2=0.5$ (Epochs 1-5).
• Stage 1: Teacher Warm-up: $w_1=0.2, w_2=1.0$ (Epochs 6-30).
• Transition Phase: Linearly interpolate $w_1 \to 1.0$ and $w_2 \to 0.1$ (Epochs 31-40).
• Stage 2: GT Fine-tuning: Fixed $w_1=1.0, w_2=0.1$ (Epochs 41-100).

3. Advanced Training Techniques

A. Performance Optimization: Offline Distillation

To eliminate On-the-fly Latency, we shift from real-time Teacher inference to Offline Label Caching:

• A pre-processing script will run the Teacher model once on the entire dataset.
• Teacher "Soft Labels" (Pitch/Yaw logits) will be stored in the HDF5 files.

B. Fairness: Subject-Level Weighted Sampling

To mitigate Subject Imbalance (e.g., p03 vs p10):

• Implement a WeightedRandomSampler where each sample weight is $1/N_{participant_samples}$.
• This ensures each participant contributes equally to the gradient updates.

C. Gradient Control with AW Loss

We will tune the AW Loss parameters ($\alpha, \omega, \epsilon$) specifically for 8x8 patches to handle optical aliasing and prevent gradient explosion.

4. Validation Strategy: LOPO (Leave-One-Person-Out)

15-fold cross-validation to ensure subject-independent performance.

5. Execution Roadmap

Step 1: Teacher Integration & Offline Caching

• Load pre-trained L2CS-Net weights.
• New Task: Develop src/data/generate_teacher_labels.py.
• Pre-compute and save soft labels into data/processed/pXX.h5.

Step 2: Training Loop Implementation (src/train.py)

• Data Pipeline: Connect GazeDataset with WeightedRandomSampler.
• Granular Logging: Separate tracking for AW Loss, KD Loss, and MAE.
• Scheduler Integration: Implement the Linear Annealing weight transition.
• Checkpointing: Save best .pt weights based on Angular Error.
• Early Stopping: Implemented patience-based stopping to optimize LOPO duration.

Step 3: Intermediate Verification & Quality Control

• Teacher Label Audit: Verified keys in HDF5 files.
• Metric Validation: Unit tested in previous sessions.
• Convergence Monitoring: Verified smooth transition and improvement during epochs 31-40.

Step 4: Benchmarking & Finalization

• LOPO Automation: Created src/lopo_train.py.
• Compute mean 3D Angular Error ($\mathcal{E}_{angular}$) across all 15 LOPO folds.
• Final weight export (Target size < 2 MB).
• Hardware Profiling: Verify inference speed (FPS) and peak RAM on a standard CPU.