docs/PIPELINE_TIMING_AND_REALTIME.md

# Pipeline timing and realtime suitability

Last updated: 2026-05-12

## Measurement scope

The timing benchmark in `docs/benchmark-subprocesses.json` measures synthetic drum fixtures with:

- `stem=all`, so Demucs is bypassed.
- Warm process after import/model-library initialization.
- Synthetic rock/funk/halftime fixtures generated by `synth_generator.py`.
- `scripts/benchmark_subprocesses.py` as the benchmark driver.

This isolates the sample-extraction subprocesses from source-separation noise. Demucs timing depends heavily on model, hardware, track length, first-run downloads, and CPU/GPU availability, so it is analyzed separately.

## Significant subprocesses

| Subprocess | Current implementation | Timing behavior | Realtime suitability |
|---|---|---|---|
| Source load / stem extraction | `extract_stem`; full mix via `librosa`, stems via Demucs | Full mix is usually small; Demucs dominates full jobs | Full mix: near-realtime. Demucs: no. |
| BPM detection | `detect_bpm` using onset envelope and beat tracking | Usually sub-second for short fixtures | Near-realtime with buffering; not critical path. |
| Onset detection + slicing | `detect_onsets` multi-band SuperFlux-style envelope | Often the largest pure-DSP stage | Near-realtime with bounded lookahead. |
| Classification | Rule-based spectral analysis per hit | Fast relative to onset/clustering | Near-realtime. |
| Batch clustering | Mel fingerprints + transient NCC + agglomerative clustering | Pairwise/batch; scales poorly with many hits | Not realtime. Final-quality batch mode. |
| Online clustering | Prototype assignment per hit | Scales with hit count × cluster count | Near-realtime preview path. |
| Representative selection | Scores each candidate hit | Moderate for many clusters/hits | Near-realtime for moderate hit counts. |
| Synthesis | Weighted aligned average per multi-hit cluster | Usually small | Near-realtime for moderate clusters. |
| Export/package | WAV/MIDI/render/ZIP writes | Disk-bound; ZIP is batch finalization | Not meaningful as realtime; finalization step. |

## Current benchmark summary

The checked-in benchmark files were refreshed on 2026-05-12 with synthetic 2-bar fixtures and Demucs bypassed:

- `docs/benchmark-subprocesses.json`: `batch_quality` clustering.
- `docs/benchmark-online-preview.json`: `online_preview` clustering.

| Stage | Batch quality mean | Online preview mean |
|---|---:|---:|
| source load | 0.010 s | 0.010 s |
| BPM detection | 0.155 s | 0.126 s |
| onset detection + slicing | 1.964 s | 1.763 s |
| classification | 0.042 s | 0.041 s |
| clustering | 0.046 s | 0.037 s |
| representative selection | 0.177 s | 0.158 s |
| synthesis | 0.001 s | 0.001 s |
| export/package | 0.158 s | 0.291 s |

On these small fixtures, `online_preview` reduced clustering time compared with `batch_quality`, while export time increased because this pass now writes every accepted hit as a review WAV under `review/hits/`. The total run is still dominated by onset detection, so the next realtime optimization target is streaming/incremental onset analysis rather than only clustering.

First cold runs can be much slower because imports and library initialization are paid up front.

## Batch quality versus online preview clustering

### `batch_quality`

Current final-quality clustering path:

1. Compute mel fingerprint for each hit.
2. Compute pairwise mel cosine prefilter.
3. Compute transient NCC only for candidate pairs.
4. Build distance matrix.
5. Run agglomerative clustering.
6. Optionally merge singleton clusters.

This gives better global grouping, but it is fundamentally batch-oriented because it wants the full similarity matrix.

### `online_preview`

Current near-realtime-oriented clustering path:

1. Process hits in onset order.
2. Compute one mel fingerprint and one transient per hit.
3. Compare the new hit against existing cluster prototypes.
4. Assign it to the best prototype or create a new cluster until the target cap is reached.
5. Update prototype fingerprints/transients using energy-weighted rolling averages.

Complexity is roughly `O(number_of_hits × number_of_clusters)`, not `O(number_of_hits²)`, and does not require future hits before producing a current assignment. It is suitable for progressive preview and fast iteration, but it is not guaranteed to match the global batch clustering result.

## What can run in or near realtime

These can be performed progressively with small buffers:

- Source decode for already-separated/full-mix audio.
- Onset envelope computation.
- Peak picking with bounded lookahead.
- Hit slicing once enough tail audio is buffered.
- Rule-based classification.
- Mel fingerprint extraction.
- Online prototype clustering.
- Representative preview selection.
- Basic reconstruction preview.

## What should stay offline/batch

- Demucs source separation.
- All-pairs transient NCC for large hit sets.
- Agglomerative clustering.
- Final ZIP packaging.
- Full high-quality rerender/export.

## Recommended runtime strategy

| Phase | Mode | Purpose |
|---|---|---|
| Upload / first pass | `stem=all`, `clustering_mode=online_preview` | Fast inspection and parameter tuning. |
| Final extraction from full mix/stem | `stem=all`, `clustering_mode=batch_quality` | Better grouping without source separation. |
| Final extraction from full song | `stem=drums`, `clustering_mode=batch_quality`, disk cache on | Best quality with offline Demucs cost paid once. |

## Disk cache impact

Disk cache now stores decoded full mix or Demucs stem output under `.cache/stems/`, keyed by:

- Source SHA-256.
- Stem name.
- Demucs model.
- Demucs shifts.
- Demucs overlap.
- Device/decode mode.

This does not make Demucs realtime, but it prevents repeated source separation work when retuning onset/clustering parameters for the same source and stem settings.

## Remaining realtime work

The current `online_preview` mode is invoked by the batch job API after onset detection. To make the application genuinely realtime/progressive, add:

1. A streaming/ranged audio analysis API.
2. Incremental onset detector state.
3. Incremental hit artifact writing.
4. UI that appends hits/clusters as they arrive instead of waiting for the completed manifest.
5. Optional final `batch_quality` consolidation pass.