# Phase 3 Master Plan: Knowledge Distillation (KD)

## 1. Executive Summary
Phase 3 focuses on "distilling" the spatial awareness and high-accuracy gaze features of a State-of-the-Art **Teacher (L2CS-Net)** into our lightweight **Student (LIPE V2)**. By training the Student to replicate the Teacher's output distribution, we aim to achieve professional-grade accuracy (< 4.2° error) at a fraction of the computational cost (~0.001 GFLOPs).

## 2. Technical Architecture

### A. The Teacher (Feature Provider)
*   **Model:** L2CS-Net with ResNet backbone.
*   **Primary Role:** Generate "Soft Labels" for every sample. 
*   **Optimization:** **Offline Distillation.** Labels are pre-computed to eliminate inference latency during the main training loop.

### B. The Student (Feature Trainee)
*   **Model:** `LIPEV2Student` (Refined Architecture).
*   **Primary Role:** Learn to predict gaze using only 8x8 patches and zero-centered landmarks.

### C. Hybrid Loss Strategy ($L_{total}$)
$L_{total} = w_1 \cdot L_{AW} + w_2 \cdot L_{KD}$

#### Dynamic Weighting Schedule (Refined with Gradient Smoothing)
To prevent **Gradient Shock**, transitions between stages use Linear Annealing over a 10-epoch window:
*   **Stage 0: Sanity Check (Hard-Threshold):** Fixed $w_1=1.0, w_2=0.5$ (Epochs 1-5).
*   **Stage 1: Teacher Warm-up:** $w_1=0.2, w_2=1.0$ (Epochs 6-30).
*   **Transition Phase:** Linearly interpolate $w_1 \to 1.0$ and $w_2 \to 0.1$ (Epochs 31-40).
*   **Stage 2: GT Fine-tuning:** Fixed $w_1=1.0, w_2=0.1$ (Epochs 41-100).

## 3. Advanced Training Techniques

### A. Performance Optimization: Offline Distillation
To eliminate **On-the-fly Latency**, we shift from real-time Teacher inference to **Offline Label Caching**:
*   A pre-processing script will run the Teacher model once on the entire dataset.
*   Teacher "Soft Labels" (Pitch/Yaw logits) will be stored in the HDF5 files.

### B. Fairness: Subject-Level Weighted Sampling
To mitigate **Subject Imbalance** (e.g., p03 vs p10):
*   Implement a `WeightedRandomSampler` where each sample weight is $1/N_{participant\_samples}$.
*   This ensures each participant contributes equally to the gradient updates.

### C. Gradient Control with AW Loss
We will tune the AW Loss parameters ($\alpha, \omega, \epsilon$) specifically for 8x8 patches to handle optical aliasing and prevent gradient explosion.

## 4. Validation Strategy: LOPO (Leave-One-Person-Out)
15-fold cross-validation to ensure subject-independent performance.

## 5. Execution Roadmap

### Step 1: Teacher Integration & Offline Caching
- [x] Load pre-trained L2CS-Net weights.
- [x] **New Task:** Develop `src/data/generate_teacher_labels.py`.
- [x] Pre-compute and save soft labels into `data/processed/pXX.h5`.

### Step 2: Training Loop Implementation (`src/train.py`)
- [x] **Data Pipeline:** Connect `GazeDataset` with `WeightedRandomSampler`.
- [x] **Granular Logging:** Separate tracking for AW Loss, KD Loss, and MAE.
- [x] **Scheduler Integration:** Implement the Linear Annealing weight transition.
- [x] **Checkpointing:** Save best `.pt` weights based on Angular Error.
- [x] **Early Stopping:** Implemented patience-based stopping to optimize LOPO duration.

### Step 3: Intermediate Verification & Quality Control
- [x] **Teacher Label Audit:** Verified keys in HDF5 files.
- [x] **Metric Validation:** Unit tested in previous sessions.
- [x] **Convergence Monitoring:** Verified smooth transition and improvement during epochs 31-40.

### Step 4: Benchmarking & Finalization
- [ ] **LOPO Automation:** Created `src/lopo_train.py`.
- [ ] Compute mean 3D **Angular Error** ($\mathcal{E}_{angular}$) across all 15 LOPO folds.
- [ ] Final weight export (Target size < 2 MB).
- [ ] **Hardware Profiling:** Verify inference speed (FPS) and peak RAM on a standard CPU.