docs/FEATURES.md

# Feature inventory

Last updated: 2026-05-12

## Product goal

Turn an input audio file into a practical drum sample pack: detected hits, grouped sample classes, representative WAVs, optional synthesized alternates, MIDI reconstruction, target-stem reconstruction, full-context reproduced audio, and an inspectable manifest.

## Implemented features

| Area | Feature | Status | Notes |
|---|---|---:|---|
| UI | Custom browser frontend | Implemented | `web/index.html`, `web/styles.css`, `web/app.js`; no Gradio dependency in active app. |
| UI | Explicit upload button | Implemented | Top bar contains a visible `Upload audio` control. |
| UI | Whole-app drag/drop audio upload | Implemented | Dropping files anywhere on the app selects the file and shows a drop overlay during drag. |
| UI | Clean fixed non-scrolling workstation layout | Implemented | Body is viewport-locked; primary workflow stays visible; secondary tools live in collapsed left/right sidebars and a compact bottom dock; long content scrolls inside panels only. |
| UI | Minimal custom transport | Implemented | One play/time/progress row can audition Source, Stem, or Reproduced previews; completed runs default to Reproduced. |
| UI | Common vs advanced parameters | Implemented | Default view shows stem, sensitivity, group count, and two presets; helper copy and advanced model/DSP/export controls stay hidden until the user opens the relevant panel. |
| UI | Streaming progress | Implemented | Uses `EventSource` over `GET /api/jobs/{id}/events`, with polling fallback. |
| UI | Visible request/runtime errors | Implemented | Failed API requests and pipeline errors appear in a global error banner and are also written to summary/log panels. |
| UI | Waveform/onset overview | Implemented | Canvas envelope plus clickable onset markers from `manifest.json`. |
| UI | Result downloads | Implemented | ZIP, MIDI, stem WAV, reconstruction WAV, individual sample WAVs, and per-hit review WAVs. |
| UI | Run history browser | Implemented | Lists completed `.runs/*/output/manifest.json` entries and reloads results. |
| UI | Hit and sample audition | Implemented | Click-to-audition via hidden audio elements keeps the default screen clean. |
| API | Health/config | Implemented | `GET /api/health`, `GET /api/config`. |
| API | Job creation/status | Implemented | `POST /api/jobs`, `GET /api/jobs/{id}`. Browser-style string params are coerced before validation. |
| API | SSE job events | Implemented | `GET /api/jobs/{id}/events` streams job snapshots until complete/error. |
| API | Run listing | Implemented | `GET /api/jobs` returns active and completed runs. |
| API | Safe artifact serving | Implemented | Path traversal is blocked by resolved output-root checks. |
| API | Cache clear | Implemented | Clears in-memory DSP cache and disk stem/source cache. |
| Pipeline | Demucs stem extraction | Implemented | Offline/batch stage; not advertised as realtime. |
| Pipeline | Stem/full-mix disk cache | Implemented | Keyed by source SHA-256 plus stem/model/shifts/overlap/device. |
| Pipeline | BPM detection | Implemented | `librosa` onset/beat based estimate. |
| Pipeline | SuperFlux-style onset detection | Implemented | Multi-band auto mode plus percussive/harmonic/broadband modes. |
| Pipeline | Hit classification | Implemented | Rule-based spectral class labels. |
| Pipeline | Batch quality clustering | Implemented | Mel prefilter + transient NCC + agglomerative clustering. |
| Pipeline | Online preview clustering | Implemented | Prototype-based incremental assignment for near-realtime feedback. |
| Pipeline | Representative selection | Implemented | Quality score picks best hit per cluster. |
| Pipeline | Optional synthesis | Implemented | Weighted aligned average for multi-hit clusters. |
| Pipeline | MIDI export | Implemented | Quantized or unquantized reconstruction MIDI. |
| Pipeline | Target reconstruction render | Implemented | Renders the selected sample representatives from MIDI/onset timing and matches RMS to the target stem. |
| Pipeline | Full-context reproduced mix | Implemented | Writes `reconstruction.wav` as non-target context bed plus target reconstruction, so separated-stem runs incorporate all other stems. |
| Pipeline | Per-hit review export | Implemented | Writes every accepted detected hit to `review/hits/*.wav` and records rows in the manifest. |
| Pipeline | Sample pack ZIP | Implemented | Includes WAVs, index JSON, MIDI, full-context reproduced mix, and target-stem reconstruction. |
| Supervision | Edited artifact re-export | Implemented | `supervised_export.py` writes edited samples, MIDI, reconstruction, ZIP, and `supervised/manifest.json`. |
| Supervision | Force-onset from waveform | Implemented | Adds user-forced hit slices from cached `stem.wav`; UI add-onset mode posts to `/hits/force-onset`. |
| Supervision | Suppressed-hit restore | Implemented | Restore endpoint and UI button reverse suppression without undoing unrelated edits. |
| Supervision | Suggestion diff previews | Implemented | Open suggestions include exact hit/cluster before-after previews and a UI `Diff` button. |
| Docs | Project review | Implemented | `docs/PROJECT_REVIEW.md`. |
| Docs | Timing/realtime analysis | Implemented | `docs/PIPELINE_TIMING_AND_REALTIME.md`. |
| Docs | API docs | Implemented | `docs/API.md`. |
| Docs | UI replacement docs | Implemented | `docs/UI_REPLACEMENT.md`. |
| Docs | Feature/task/progress tracking | Implemented | This file, `TASKS.md`, `PROGRESS.md`. |
| Docs | Hit review and streaming docs | Implemented | `docs/HIT_REVIEW_AND_STREAMING.md`. |

## Partially implemented features

| Area | Feature | Current state | Needed to call it complete |
|---|---|---|---|
| Progress | Stage progress | SSE streams stage boundaries and logs | Add lower-level progress inside Demucs and clustering. |
| Realtime | Online clustering | Implemented as batch-invoked prototype assignment | Add streaming/incremental audio analysis API for true realtime preview. |
| Run history | Manifest browser | Lists and reloads completed runs | Add side-by-side comparison and filtering/search. |
| Editing | Review workflow | Click-to-audition for hits and samples is implemented | Add onset editing, cluster merge/split, label reassignment. |
| Frontend quality | No-build JavaScript UI | Good enough for local app | Convert to TypeScript once interaction model stabilizes. |

## Explicit non-goals for this pass

- Realtime Demucs. It is not realistic for this use-case and should remain offline/cached.
- Perfect source separation. Stem quality depends on model choice and input material.
- Full DAW/sample-editor UX. This pass creates the workstation foundation; detailed editing is next.

## Interactive supervised UX features

| Area | Feature | Status | Notes |
|---|---|---|---|
| Supervision | Supplied UX docs embedded | Implemented | Added and aligned under `docs/interactive-ux/`. |
| Supervision | Persistent semantic state | Implemented | `supervision_state.json` is created beside each run manifest. |
| Supervision | Hit/cluster state model | Implemented | State tracks current cluster assignment, confidence, suppression, favorite/review flags, and representatives. |
| Supervision | Constraint store | Implemented | Stores `force-cluster`, `must-link`, `cannot-link`, `lock-cluster`, `suppress-pattern`, and `pin-representative`. |
| Supervision | Event log | Implemented | State changes append replay/audit events. |
| Supervision | Undo stack | Implemented | Last semantic edit can be undone. |
| Supervision | Confidence scoring | Partial | Heuristic and deterministic; does not yet use cached mel/transient feature margins. |
| Supervision | Outlier-first review queue | Implemented | UI prioritizes low-confidence/singleton/unstable hits. |
| Supervision | Move hit to cluster | Implemented | Creates supervision constraints and may produce suggestions. |
| Supervision | Pull hit into new cluster | Implemented | Creates a user cluster and cannot-link/force-cluster constraints. |
| Supervision | Lock cluster | Implemented | Lock state persists and updates confidence/UI. |
| Supervision | Suppress hit as bleed | Implemented | Marks hit suppressed, stores suppress-pattern, may suggest similar suppressions. |
| Supervision | Favorite representative | Implemented | Pins semantic representative and supervised export honors it before quality scoring. |
| Supervision | Suggestion inbox | Implemented | Move/split/suppress suggestions can be accepted/rejected and inspected with exact diff previews. |
| Supervision | Cluster explanation | Implemented | Backend and UI show confidence reasons, label distribution, outliers, and constraints. |
| Supervision | Edited artifact re-export | Implemented | Exports edited state into `supervised/` without mutating original batch artifacts. |
| Supervision | Force-onset from waveform | Implemented | Add-onset mode turns waveform clicks into forced hit slices from `stem.wav`. |


## Visual/UI experience

Status: implemented.

- Light, minimal, waveform-first interface closely aligned with the supplied reference composition.
- Compact top file selector plus one primary purple extract action.
- Quiet main waveform card with colored onset markers and click-to-select / force-onset behavior.
- One custom preview transport row instead of multiple native audio players.
- Right-side core-control card now exposes only stem, sensitivity, cluster count, and export samples in the default view.
- Fast modes, DSP/model parameters, pipeline logs, history, supervision, and raw tables are hidden behind `Advanced` / `Review & edit`.
- Extracted samples render as auditionable cards with waveform thumbnails and minimal labels.


## Clean default UI status

Status: implemented.

- Secondary panels are collapsed by default so the first screen is no longer dominated by logs, history, exports, or review/edit tools.
- The bottom dock stays as a small tab bar until a tool is opened.
- The top bar, sidebars, common controls, transport, and sample cards were tightened to give the waveform/sample workspace more room.
- No feature was removed; advanced and supervised tools remain available through expandable panels.


## Immediate waveform and guided flow

- Uploading or dropping a file now renders the source waveform in the browser before extraction starts.
- The left sidebar contains a compact `Start here` sequence: Load audio, set controls, extract, review/export.
- The waveform HUD states the current step and current progress status.
- The Source transport is available immediately after upload.

## Real progress visualization

- Job payloads and SSE updates include a top-level `progress` object.
- The waveform is tinted in two colors during active extraction: accent for completed work, neutral for remaining work.
- Demucs stem extraction reports exact completed split chunks when running separated-stem extraction.
- Stages that do not expose internal chunk progress update only at exact stage boundaries.
- The UI does not synthesize estimated progress or ETA.

## Pass: automatic card-flow UX

| Area | Feature | Status | Notes |
|---|---|---:|---|
| UI | Drop/upload starts processing automatically | Implemented | File selection calls the same job API without requiring the user to find a separate start button. |
| UI | Progressive sample cards | Implemented | Backend emits `sample` progress events and active jobs expose `partial_samples`. |
| UI | Grouped sample columns | Implemented | Cards are grouped by kick/snare/hihat/cymbal/tom/perc/other. |
| UI | Draw another candidate | Implemented | Column-level `Draw` action picks another detected hit candidate of that type. |
| UI | Dismiss card | Implemented | Hides the card locally and attempts to suppress its representative hit in supervised state. |
| UI | Trim/extend card | Implemented | Simple card controls adjust proposed timing and can save a forced hit. |
| UI | Zoomable/pannable waveform | Implemented | Pinch/ctrl-wheel zoom and horizontal/shift-wheel pan. |
| Pipeline | Automatic tuning | Implemented | `auto_tune=true` chooses onset sensitivity and grouping bounds from the separated target stem. |
| Export | MIDI fallback | Implemented | Missing `pretty_midi` no longer breaks extraction; a compact fallback MIDI writer is used. |


## Reference-style visual workflow

- The default web UI is now a reference-style sample extractor workspace: compact top bar, large waveform, persistent settings panel, grouped sample columns, and bottom selection bar.
- Users can still just drop audio anywhere; waveform rendering and extraction begin automatically.
- Expert parameters and semantic editing tools are available without cluttering the default path.

## Selected cards and backend simplification update

Implemented after the reference-image UI pass:

| Area | Feature | Status | Notes |
|---|---|---:|---|
| Separation | Spleeter backend | Implemented | Default backend, with `spleeter:4stems` selected by default. |
| Separation | Demucs backend | Implemented | Explicit higher-cost backend and automatic fallback when enabled. |
| Separation | No-separation backend | Implemented | Full-mix preview path for fastest iteration. |
| Export | Per-card selection | Implemented | Cards have real checkbox state; selected count drives `Export Selected`. |
| Export | Selected-only export | Implemented | `POST /api/jobs/{job_id}/export-selected` writes `selected/sample-pack.zip`. |
| Cards | Draw another | Implemented | Persists the next representative hit as a semantic override. |
| Cards | Trim/extend preview | Implemented | Rewrites a playable preview WAV immediately under `overrides/hits/`. |
| Docs | Separation backend docs | Implemented | See `docs/SPLEETER_AND_SEPARATION_BACKENDS.md`. |
| Docs | Card action docs | Implemented | See `docs/CARD_SELECTION_EXPORT_AND_EDITING.md`. |

## Upload/runtime fallback update (2026-05-12)

- Added a visible top-bar `Choose audio` affordance in addition to whole-app drag/drop.
- Fixed the default hidden state of the error banner so placeholder errors are not shown on page load.
- API errors now surface request path/status/detail in the visible banner and pipeline logs.
- `/api/config` now includes runtime diagnostics for optional separation backends.
- If Spleeter is unavailable, the simple UI keeps the app usable by switching to full-mix mode; backend fallback also uses full-mix rather than silently launching Demucs.