docs/optimizer_options.md

Optimizer Options in Unified Training Script

Overview

The unified training script now supports multiple optimizers. Adam remains the default for backward compatibility with existing configurations.

Available Optimizers

1. Adam (Default)

Standard Adam optimizer, good general-purpose choice.

optimizer_type: 'Adam'  # This is the default if not specified
weight_decay: 0.0       # L2 regularization
betas: [0.9, 0.999]     # Optional: Adam beta parameters
eps: 0.00000001         # Optional: Adam epsilon for numerical stability

2. AdamW

Adam with decoupled weight decay, often better for transformer models and when using weight decay.

optimizer_type: 'AdamW'
weight_decay: 0.01      # Decoupled weight decay (common values: 0.01-0.1)
betas: [0.9, 0.999]     # Optional: AdamW beta parameters
eps: 0.00000001         # Optional: AdamW epsilon

3. SGD

Stochastic Gradient Descent with momentum, sometimes better for ConvNets.

optimizer_type: 'SGD'
weight_decay: 0.0001    # L2 regularization
momentum: 0.9           # Optional: SGD momentum (default: 0.9)
nesterov: True          # Optional: Use Nesterov momentum (default: True)

Command Line Override

You can override optimizer settings via command line:

# Use AdamW
python train_network_unified.py \
    --config your_config.yaml \
    --opts \
    optimizer_type AdamW \
    weight_decay 0.01

# Use SGD with specific momentum
python train_network_unified.py \
    --config your_config.yaml \
    --opts \
    optimizer_type SGD \
    momentum 0.95 \
    weight_decay 0.0001

Recommendations

For LLaVA/Transformer Models

• Optimizer: AdamW
• Weight decay: 0.01-0.05
• Learning rate: 1e-4 to 5e-4
• Warmup: 2000-5000 steps

optimizer_type: 'AdamW'
weight_decay: 0.01
base_lr: 0.0001
warmup_steps: 2000

For ConvNet Backbones

• Optimizer: Adam or SGD
• Weight decay: 0.0 (Adam) or 0.0001 (SGD)
• Learning rate: 1e-4 to 1e-3

# Option 1: Adam
optimizer_type: 'Adam'
weight_decay: 0.0
base_lr: 0.0001

# Option 2: SGD
optimizer_type: 'SGD'
weight_decay: 0.0001
base_lr: 0.001
momentum: 0.9

Fine-tuning Pre-trained Models

• Optimizer: AdamW (preferred) or Adam
• Weight decay: Lower values (0.0-0.01)
• Learning rate: Lower than training from scratch
• Warmup: Short warmup (500-1000 steps)

optimizer_type: 'AdamW'
weight_decay: 0.01
base_lr: 0.00005
warmup_steps: 1000

Monitoring

The optimizer type is logged to WandB along with other hyperparameters. Monitor:

• Learning rate curves
• Training/validation loss
• Gradient norms (if logged)
• Weight decay effectiveness