docs/SPLEETER_AND_SEPARATION_BACKENDS.md

# Spleeter and separation backends

Last updated: 2026-05-12

## Decision

The application now defaults to Spleeter:

```json
{
  "separation_backend": "spleeter",
  "spleeter_model": "spleeter:4stems",
  "stem": "drums",
  "allow_backend_fallback": true
}
```

Spleeter is treated as the normal first-pass separation backend because it is much lighter for the common UX: drop a track, get drum-card candidates quickly, and only escalate to heavier processing when necessary.

Demucs remains available as a higher-cost quality backend:

```json
{"separation_backend":"demucs","demucs_model":"htdemucs_ft"}
```

Full-mix preview remains available for the fastest possible iteration:

```json
{"separation_backend":"none","stem":"all","clustering_mode":"online_preview"}
```

## Supported engines

| Backend | Status | Use |
|---|---:|---|
| `spleeter` | Default | Lightweight drum/source separation for the common automatic workflow. |
| `demucs` | Supported | Explicit higher-cost quality backend when Spleeter/full-mix results are insufficient. |
| `none` | Supported | Bypass source separation and process the full mix. Best for quick UI/debug iteration. |

## Spleeter models

| Model | Stems exposed |
|---|---|
| `spleeter:2stems` | `vocals`, `accompaniment`, `all` |
| `spleeter:4stems` | `vocals`, `drums`, `bass`, `other`, `all` |
| `spleeter:5stems` | `vocals`, `drums`, `bass`, `piano`, `other`, `all` |

## Installation

Spleeter is optional and intentionally not installed by the main `requirements.txt`, because TensorFlow/Spleeter compatibility can be environment-sensitive.

Install it only when needed:

```bash
pip install -r requirements-spleeter.txt
```

For the normal local app, leave `allow_backend_fallback=true`. If Spleeter is unavailable or fails, the job falls back to full-mix processing and logs that fallback in the stage details. Demucs is no longer used as a silent fallback for Spleeter because full-track Demucs is too slow and surprising for the default automatic UI. Disable fallback only when actively debugging Spleeter.

## Caching

The disk cache key includes:

- source digest,
- selected stem,
- separation backend,
- Spleeter model,
- Demucs model,
- Demucs shifts/overlap.

This avoids accidentally reusing stems from a different engine or model.

## Progress behavior

Demucs exposes chunk progress through the existing extraction callback, so the waveform can advance during stem separation when chunk data is available.

Spleeter does not expose reliable per-chunk progress through the current backend path. The app therefore reports only real start/completion boundaries for Spleeter. It does not interpolate fake progress.

## Future research: localized separation

The recommended next architecture is not “run Demucs on the whole track after Spleeter.” The better path is:

1. Use Spleeter or full-mix onset detection to find candidate hit regions.
2. Expand each candidate region with context padding.
3. Run heavier separation only on those short windows.
4. Stitch or use the refined region only for the card/export preview.

That could make Demucs feasible as a local refinement step instead of an expensive full-track prerequisite.

## Future research: overlapping samples

AudioSep-like text/query-guided separation may be useful for overlaps where source classes matter, for example “kick drum”, “closed hi-hat”, or “snare transient”. It should be investigated as an optional refinement tool, not as the default first-pass extractor.

USEF-TSE and target-speaker-extraction style systems are mostly speech-targeted. They are not a good near-term default for drum sample extraction, but the conditioning pattern is relevant if the app later supports “extract more sounds like this selected example.”

Audio inpainting is more promising for cleaning card tails/gaps and removing overlap residue after a hit has already been localized than for first-pass sample discovery.