# TraceOpt

**Runtime observability and failure attribution for PyTorch training: step-aware, low-overhead, and always-on.**

TraceOpt builds **TraceML**, a lightweight runtime observability layer that makes PyTorch training behavior visible *while it runs*.

TraceML helps answer questions that are hard to debug with infrastructure metrics or heavyweight profilers:

- Which layer caused a CUDA OOM?
- Why did this training step suddenly slow down?
- Is the bottleneck data loading, forward, backward, or the optimizer?
- Where did a memory spike actually occur?

## What TraceML focuses on

TraceML provides **semantic, step-level signals** that bridge the gap between system metrics and model-level behavior:

- Step-level timing (dataloader → forward → backward → optimizer)
- Step-level GPU memory tracking with peak attribution
- Optional deep-dive per-layer memory and compute diagnostics
- Designed to run continuously with low overhead during real training jobs

## What TraceML is (and is not)

**What it is**
- Always-on runtime observability
- Step-aware attribution, not post-hoc profiling
- Focused on training-time behavior

**What it is not**
- Not a replacement for PyTorch Profiler or Nsight
- Not a training framework
- Not an orchestration or deployment tool

## Project status

- Actively developed
- Single-GPU PyTorch training supported
- Multi-GPU (DDP / FSDP) support in progress
- APIs may evolve as abstractions are validated

## Links

- GitHub: https://github.com/traceopt-ai/traceml
- PyPI: https://pypi.org/project/traceml-ai/
- Website: https://traceopt.ai

We’re especially interested in feedback from people training real models hitting performance or memory pathologies.