| # TraceOpt |
|
|
| **Runtime observability and failure attribution for PyTorch training: step-aware, low-overhead, and always-on.** |
|
|
| TraceOpt builds **TraceML**, a lightweight runtime observability layer that makes PyTorch training behavior visible *while it runs*. |
|
|
| TraceML helps answer questions that are hard to debug with infrastructure metrics or heavyweight profilers: |
|
|
| - Which layer caused a CUDA OOM? |
| - Why did this training step suddenly slow down? |
| - Is the bottleneck data loading, forward, backward, or the optimizer? |
| - Where did a memory spike actually occur? |
|
|
| ## What TraceML focuses on |
|
|
| TraceML provides **semantic, step-level signals** that bridge the gap between system metrics and model-level behavior: |
|
|
| - Step-level timing (dataloader → forward → backward → optimizer) |
| - Step-level GPU memory tracking with peak attribution |
| - Optional deep-dive per-layer memory and compute diagnostics |
| - Designed to run continuously with low overhead during real training jobs |
|
|
| ## What TraceML is (and is not) |
|
|
| **What it is** |
| - Always-on runtime observability |
| - Step-aware attribution, not post-hoc profiling |
| - Focused on training-time behavior |
|
|
| **What it is not** |
| - Not a replacement for PyTorch Profiler or Nsight |
| - Not a training framework |
| - Not an orchestration or deployment tool |
|
|
| ## Project status |
|
|
| - Actively developed |
| - Single-GPU PyTorch training supported |
| - Multi-GPU (DDP / FSDP) support in progress |
| - APIs may evolve as abstractions are validated |
|
|
| ## Links |
|
|
| - GitHub: https://github.com/traceopt-ai/traceml |
| - PyPI: https://pypi.org/project/traceml-ai/ |
| - Website: https://traceopt.ai |
|
|
| We’re especially interested in feedback from people training real models hitting performance or memory pathologies. |
|
|
