docs/API_OPTIMIZATIONS_WIRED.md

API Endpoints - Optimization Parameters Wired Up

All optimization parameters are now exposed through the API endpoints.

✅ Updated Endpoints

1. /train/start (Fine-tuning)

Request Model: TrainRequest

New Optimization Parameters:

• gradient_accumulation_steps (int, default: 1) - Gradient accumulation
• use_amp (bool, default: True) - Mixed precision training
• warmup_steps (int, default: 0) - Learning rate warmup
• num_workers (Optional[int], default: None) - Data loading workers
• resume_from_checkpoint (Optional[str], default: None) - Resume training
• use_ema (bool, default: False) - Exponential Moving Average
• ema_decay (float, default: 0.9999) - EMA decay factor
• use_onecycle (bool, default: False) - OneCycleLR scheduler
• use_gradient_checkpointing (bool, default: False) - Memory-efficient training
• compile_model (bool, default: True) - Torch.compile optimization

Example Request:

{
  "training_data_dir": "data/training",
  "epochs": 10,
  "lr": 1e-5,
  "batch_size": 1,
  "use_amp": true,
  "gradient_accumulation_steps": 4,
  "use_ema": true,
  "use_onecycle": true,
  "compile_model": true
}

2. /train/pretrain (Pre-training)

Request Model: PretrainRequest

New Optimization Parameters:

• All the same as /train/start plus:
• cache_dir (Optional[str], default: None) - BA result caching directory

Example Request:

{
  "arkit_sequences_dir": "data/arkit_sequences",
  "epochs": 10,
  "lr": 1e-4,
  "use_amp": true,
  "use_ema": true,
  "use_onecycle": true,
  "cache_dir": "cache/ba_results",
  "compile_model": true
}

3. /dataset/build (Dataset Building)

Request Model: BuildDatasetRequest

New Optimization Parameters:

• use_batched_inference (bool, default: False) - Batch multiple sequences
• inference_batch_size (int, default: 4) - Batch size for inference
• use_inference_cache (bool, default: False) - Cache inference results
• cache_dir (Optional[str], default: None) - Inference cache directory
• compile_model (bool, default: True) - Torch.compile for inference

Example Request:

{
  "sequences_dir": "data/sequences",
  "output_dir": "data/training",
  "use_batched_inference": true,
  "inference_batch_size": 4,
  "use_inference_cache": true,
  "cache_dir": "cache/inference",
  "compile_model": true
}

🔄 Data Flow

API Request (JSON)
    ↓
Request Model (Pydantic validation)
    ↓
Router Endpoint (training.py)
    ↓
CLI Function (cli.py) - passes through all params
    ↓
Service Function (fine_tune.py / pretrain.py / data_pipeline.py)
    ↓
Optimized Training/Inference

📝 Files Updated

1. ylff/models/api_models.py

   • Added optimization fields to TrainRequest
   • Added optimization fields to PretrainRequest
   • Added optimization fields to BuildDatasetRequest

2. ylff/routers/training.py

   • Updated /train/start to pass optimization params
   • Updated /train/pretrain to pass optimization params
   • Updated /dataset/build to pass optimization params

3. ylff/cli.py

   • Updated train() CLI function to accept optimization params
   • Updated pretrain() CLI function to accept optimization params
   • Updated build_dataset() CLI function to accept optimization params
   • All params are passed through to service functions

🎯 Usage Examples

Fast Training via API

curl -X POST "http://localhost:8000/api/v1/train/start" \
  -H "Content-Type: application/json" \
  -d '{
    "training_data_dir": "data/training",
    "epochs": 10,
    "use_amp": true,
    "gradient_accumulation_steps": 4,
    "use_ema": true,
    "use_onecycle": true,
    "compile_model": true
  }'

Optimized Dataset Building

curl -X POST "http://localhost:8000/api/v1/dataset/build" \
  -H "Content-Type: application/json" \
  -d '{
    "sequences_dir": "data/sequences",
    "use_batched_inference": true,
    "inference_batch_size": 4,
    "use_inference_cache": true,
    "cache_dir": "cache/inference"
  }'

✅ Status

All optimization parameters are:

• ✅ Defined in API request models
• ✅ Validated by Pydantic
• ✅ Passed through router endpoints
• ✅ Accepted by CLI functions
• ✅ Forwarded to service functions
• ✅ Documented with descriptions and examples

The API is fully wired up to use all optimization capabilities! 🚀