todo.md

# Gradient Clipping Experiment

## Objective
Demonstrate how gradient clipping stabilizes training by preventing sudden large weight updates caused by rare, high-loss data points.

## Task Breakdown

- [ ] Step 1: Implement simple PyTorch model (Embedding + Linear)
- [ ] Step 2: Create imbalanced synthetic dataset (990 'A', 10 'B' targets)
- [ ] Step 3: Training loop WITHOUT gradient clipping - record metrics
- [ ] Step 4: Training loop WITH gradient clipping (threshold=1.0) - record metrics
- [ ] Step 5: Generate comparison plots
- [ ] Step 6: Write summary report with findings

## Key Metrics to Track
1. Training loss per step
2. L2 norm of gradients (before clipping)
3. L2 norm of model weights

## Expected Outcome
- Without clipping: Spiky gradient norms when encountering rare 'B' samples
- With clipping: Bounded gradient norms, more stable training