# Optimizer Options in Unified Training Script

## Overview
The unified training script now supports multiple optimizers. Adam remains the default for backward compatibility with existing configurations.

## Available Optimizers

### 1. Adam (Default)
Standard Adam optimizer, good general-purpose choice.

```yaml
optimizer_type: 'Adam'  # This is the default if not specified
weight_decay: 0.0       # L2 regularization
betas: [0.9, 0.999]     # Optional: Adam beta parameters
eps: 0.00000001         # Optional: Adam epsilon for numerical stability
```

### 2. AdamW
Adam with decoupled weight decay, often better for transformer models and when using weight decay.

```yaml
optimizer_type: 'AdamW'
weight_decay: 0.01      # Decoupled weight decay (common values: 0.01-0.1)
betas: [0.9, 0.999]     # Optional: AdamW beta parameters
eps: 0.00000001         # Optional: AdamW epsilon
```

### 3. SGD
Stochastic Gradient Descent with momentum, sometimes better for ConvNets.

```yaml
optimizer_type: 'SGD'
weight_decay: 0.0001    # L2 regularization
momentum: 0.9           # Optional: SGD momentum (default: 0.9)
nesterov: True          # Optional: Use Nesterov momentum (default: True)
```

## Command Line Override

You can override optimizer settings via command line:

```bash
# Use AdamW
python train_network_unified.py \
    --config your_config.yaml \
    --opts \
    optimizer_type AdamW \
    weight_decay 0.01

# Use SGD with specific momentum
python train_network_unified.py \
    --config your_config.yaml \
    --opts \
    optimizer_type SGD \
    momentum 0.95 \
    weight_decay 0.0001
```

## Recommendations

### For LLaVA/Transformer Models
- **Optimizer**: AdamW
- **Weight decay**: 0.01-0.05
- **Learning rate**: 1e-4 to 5e-4
- **Warmup**: 2000-5000 steps

```yaml
optimizer_type: 'AdamW'
weight_decay: 0.01
base_lr: 0.0001
warmup_steps: 2000
```

### For ConvNet Backbones
- **Optimizer**: Adam or SGD
- **Weight decay**: 0.0 (Adam) or 0.0001 (SGD)
- **Learning rate**: 1e-4 to 1e-3

```yaml
# Option 1: Adam
optimizer_type: 'Adam'
weight_decay: 0.0
base_lr: 0.0001

# Option 2: SGD
optimizer_type: 'SGD'
weight_decay: 0.0001
base_lr: 0.001
momentum: 0.9
```

### Fine-tuning Pre-trained Models
- **Optimizer**: AdamW (preferred) or Adam
- **Weight decay**: Lower values (0.0-0.01)
- **Learning rate**: Lower than training from scratch
- **Warmup**: Short warmup (500-1000 steps)

```yaml
optimizer_type: 'AdamW'
weight_decay: 0.01
base_lr: 0.00005
warmup_steps: 1000
```

## Monitoring

The optimizer type is logged to WandB along with other hyperparameters. Monitor:
- Learning rate curves
- Training/validation loss
- Gradient norms (if logged)
- Weight decay effectiveness