docs/FEATURES.md

Feature inventory

Last updated: 2026-05-12

Product goal

Turn an input audio file into a practical drum sample pack: detected hits, grouped sample classes, representative WAVs, optional synthesized alternates, MIDI reconstruction, target-stem reconstruction, full-context reproduced audio, and an inspectable manifest.

Implemented features

Area	Feature	Status	Notes
UI	Custom browser frontend	Implemented	web/index.html, web/styles.css, web/app.js; no Gradio dependency in active app.
UI	Explicit upload button	Implemented	Top bar contains a visible Upload audio control.
UI	Whole-app drag/drop audio upload	Implemented	Dropping files anywhere on the app selects the file and shows a drop overlay during drag.
UI	Clean fixed non-scrolling workstation layout	Implemented	Body is viewport-locked; primary workflow stays visible; secondary tools live in collapsed left/right sidebars and a compact bottom dock; long content scrolls inside panels only.
UI	Minimal custom transport	Implemented	One play/time/progress row can audition Source, Stem, or Reproduced previews; completed runs default to Reproduced.
UI	Common vs advanced parameters	Implemented	Default view shows stem, sensitivity, group count, and two presets; helper copy and advanced model/DSP/export controls stay hidden until the user opens the relevant panel.
UI	Streaming progress	Implemented	Uses EventSource over GET /api/jobs/{id}/events, with polling fallback.
UI	Visible request/runtime errors	Implemented	Failed API requests and pipeline errors appear in a global error banner and are also written to summary/log panels.
UI	Waveform/onset overview	Implemented	Canvas envelope plus clickable onset markers from manifest.json.
UI	Result downloads	Implemented	ZIP, MIDI, stem WAV, reconstruction WAV, individual sample WAVs, and per-hit review WAVs.
UI	Run history browser	Implemented	Lists completed .runs/*/output/manifest.json entries and reloads results.
UI	Hit and sample audition	Implemented	Click-to-audition via hidden audio elements keeps the default screen clean.
API	Health/config	Implemented	GET /api/health, GET /api/config.
API	Job creation/status	Implemented	POST /api/jobs, GET /api/jobs/{id}. Browser-style string params are coerced before validation.
API	SSE job events	Implemented	GET /api/jobs/{id}/events streams job snapshots until complete/error.
API	Run listing	Implemented	GET /api/jobs returns active and completed runs.
API	Safe artifact serving	Implemented	Path traversal is blocked by resolved output-root checks.
API	Cache clear	Implemented	Clears in-memory DSP cache and disk stem/source cache.
Pipeline	Demucs stem extraction	Implemented	Offline/batch stage; not advertised as realtime.
Pipeline	Stem/full-mix disk cache	Implemented	Keyed by source SHA-256 plus stem/model/shifts/overlap/device.
Pipeline	BPM detection	Implemented	librosa onset/beat based estimate.
Pipeline	SuperFlux-style onset detection	Implemented	Multi-band auto mode plus percussive/harmonic/broadband modes.
Pipeline	Hit classification	Implemented	Rule-based spectral class labels.
Pipeline	Batch quality clustering	Implemented	Mel prefilter + transient NCC + agglomerative clustering.
Pipeline	Online preview clustering	Implemented	Prototype-based incremental assignment for near-realtime feedback.
Pipeline	Representative selection	Implemented	Quality score picks best hit per cluster.
Pipeline	Optional synthesis	Implemented	Weighted aligned average for multi-hit clusters.
Pipeline	MIDI export	Implemented	Quantized or unquantized reconstruction MIDI.
Pipeline	Target reconstruction render	Implemented	Renders the selected sample representatives from MIDI/onset timing and matches RMS to the target stem.
Pipeline	Full-context reproduced mix	Implemented	Writes reconstruction.wav as non-target context bed plus target reconstruction, so separated-stem runs incorporate all other stems.
Pipeline	Per-hit review export	Implemented	Writes every accepted detected hit to review/hits/*.wav and records rows in the manifest.
Pipeline	Sample pack ZIP	Implemented	Includes WAVs, index JSON, MIDI, full-context reproduced mix, and target-stem reconstruction.
Supervision	Edited artifact re-export	Implemented	supervised_export.py writes edited samples, MIDI, reconstruction, ZIP, and supervised/manifest.json.
Supervision	Force-onset from waveform	Implemented	Adds user-forced hit slices from cached stem.wav; UI add-onset mode posts to /hits/force-onset.
Supervision	Suppressed-hit restore	Implemented	Restore endpoint and UI button reverse suppression without undoing unrelated edits.
Supervision	Suggestion diff previews	Implemented	Open suggestions include exact hit/cluster before-after previews and a UI Diff button.
Docs	Project review	Implemented	docs/PROJECT_REVIEW.md.
Docs	Timing/realtime analysis	Implemented	docs/PIPELINE_TIMING_AND_REALTIME.md.
Docs	API docs	Implemented	docs/API.md.
Docs	UI replacement docs	Implemented	docs/UI_REPLACEMENT.md.
Docs	Feature/task/progress tracking	Implemented	This file, TASKS.md, PROGRESS.md.
Docs	Hit review and streaming docs	Implemented	docs/HIT_REVIEW_AND_STREAMING.md.

Partially implemented features

Area	Feature	Current state	Needed to call it complete
Progress	Stage progress	SSE streams stage boundaries and logs	Add lower-level progress inside Demucs and clustering.
Realtime	Online clustering	Implemented as batch-invoked prototype assignment	Add streaming/incremental audio analysis API for true realtime preview.
Run history	Manifest browser	Lists and reloads completed runs	Add side-by-side comparison and filtering/search.
Editing	Review workflow	Click-to-audition for hits and samples is implemented	Add onset editing, cluster merge/split, label reassignment.
Frontend quality	No-build JavaScript UI	Good enough for local app	Convert to TypeScript once interaction model stabilizes.

Explicit non-goals for this pass

• Realtime Demucs. It is not realistic for this use-case and should remain offline/cached.
• Perfect source separation. Stem quality depends on model choice and input material.
• Full DAW/sample-editor UX. This pass creates the workstation foundation; detailed editing is next.

Interactive supervised UX features

Area	Feature	Status	Notes
Supervision	Supplied UX docs embedded	Implemented	Added and aligned under docs/interactive-ux/.
Supervision	Persistent semantic state	Implemented	supervision_state.json is created beside each run manifest.
Supervision	Hit/cluster state model	Implemented	State tracks current cluster assignment, confidence, suppression, favorite/review flags, and representatives.
Supervision	Constraint store	Implemented	Stores force-cluster, must-link, cannot-link, lock-cluster, suppress-pattern, and pin-representative.
Supervision	Event log	Implemented	State changes append replay/audit events.
Supervision	Undo stack	Implemented	Last semantic edit can be undone.
Supervision	Confidence scoring	Partial	Heuristic and deterministic; does not yet use cached mel/transient feature margins.
Supervision	Outlier-first review queue	Implemented	UI prioritizes low-confidence/singleton/unstable hits.
Supervision	Move hit to cluster	Implemented	Creates supervision constraints and may produce suggestions.
Supervision	Pull hit into new cluster	Implemented	Creates a user cluster and cannot-link/force-cluster constraints.
Supervision	Lock cluster	Implemented	Lock state persists and updates confidence/UI.
Supervision	Suppress hit as bleed	Implemented	Marks hit suppressed, stores suppress-pattern, may suggest similar suppressions.
Supervision	Favorite representative	Implemented	Pins semantic representative and supervised export honors it before quality scoring.
Supervision	Suggestion inbox	Implemented	Move/split/suppress suggestions can be accepted/rejected and inspected with exact diff previews.
Supervision	Cluster explanation	Implemented	Backend and UI show confidence reasons, label distribution, outliers, and constraints.
Supervision	Edited artifact re-export	Implemented	Exports edited state into supervised/ without mutating original batch artifacts.
Supervision	Force-onset from waveform	Implemented	Add-onset mode turns waveform clicks into forced hit slices from stem.wav.

Visual/UI experience

Status: implemented.

• Light, minimal, waveform-first interface closely aligned with the supplied reference composition.
• Compact top file selector plus one primary purple extract action.
• Quiet main waveform card with colored onset markers and click-to-select / force-onset behavior.
• One custom preview transport row instead of multiple native audio players.
• Right-side core-control card now exposes only stem, sensitivity, cluster count, and export samples in the default view.
• Fast modes, DSP/model parameters, pipeline logs, history, supervision, and raw tables are hidden behind Advanced / Review & edit.
• Extracted samples render as auditionable cards with waveform thumbnails and minimal labels.

Clean default UI status

Status: implemented.

• Secondary panels are collapsed by default so the first screen is no longer dominated by logs, history, exports, or review/edit tools.
• The bottom dock stays as a small tab bar until a tool is opened.
• The top bar, sidebars, common controls, transport, and sample cards were tightened to give the waveform/sample workspace more room.
• No feature was removed; advanced and supervised tools remain available through expandable panels.

Immediate waveform and guided flow

• Uploading or dropping a file now renders the source waveform in the browser before extraction starts.
• The left sidebar contains a compact Start here sequence: Load audio, set controls, extract, review/export.
• The waveform HUD states the current step and current progress status.
• The Source transport is available immediately after upload.

Real progress visualization

• Job payloads and SSE updates include a top-level progress object.
• The waveform is tinted in two colors during active extraction: accent for completed work, neutral for remaining work.
• Demucs stem extraction reports exact completed split chunks when running separated-stem extraction.
• Stages that do not expose internal chunk progress update only at exact stage boundaries.
• The UI does not synthesize estimated progress or ETA.

Pass: automatic card-flow UX

Area	Feature	Status	Notes
UI	Drop/upload starts processing automatically	Implemented	File selection calls the same job API without requiring the user to find a separate start button.
UI	Progressive sample cards	Implemented	Backend emits sample progress events and active jobs expose partial_samples.
UI	Grouped sample columns	Implemented	Cards are grouped by kick/snare/hihat/cymbal/tom/perc/other.
UI	Draw another candidate	Implemented	Column-level Draw action picks another detected hit candidate of that type.
UI	Dismiss card	Implemented	Hides the card locally and attempts to suppress its representative hit in supervised state.
UI	Trim/extend card	Implemented	Simple card controls adjust proposed timing and can save a forced hit.
UI	Zoomable/pannable waveform	Implemented	Pinch/ctrl-wheel zoom and horizontal/shift-wheel pan.
Pipeline	Automatic tuning	Implemented	auto_tune=true chooses onset sensitivity and grouping bounds from the separated target stem.
Export	MIDI fallback	Implemented	Missing pretty_midi no longer breaks extraction; a compact fallback MIDI writer is used.

Reference-style visual workflow

• The default web UI is now a reference-style sample extractor workspace: compact top bar, large waveform, persistent settings panel, grouped sample columns, and bottom selection bar.
• Users can still just drop audio anywhere; waveform rendering and extraction begin automatically.
• Expert parameters and semantic editing tools are available without cluttering the default path.

Selected cards and backend simplification update

Implemented after the reference-image UI pass:

Area	Feature	Status	Notes
Separation	Spleeter backend	Implemented	Default backend, with spleeter:4stems selected by default.
Separation	Demucs backend	Implemented	Explicit higher-cost backend and automatic fallback when enabled.
Separation	No-separation backend	Implemented	Full-mix preview path for fastest iteration.
Export	Per-card selection	Implemented	Cards have real checkbox state; selected count drives Export Selected.
Export	Selected-only export	Implemented	POST /api/jobs/{job_id}/export-selected writes selected/sample-pack.zip.
Cards	Draw another	Implemented	Persists the next representative hit as a semantic override.
Cards	Trim/extend preview	Implemented	Rewrites a playable preview WAV immediately under overrides/hits/.
Docs	Separation backend docs	Implemented	See docs/SPLEETER_AND_SEPARATION_BACKENDS.md.
Docs	Card action docs	Implemented	See docs/CARD_SELECTION_EXPORT_AND_EDITING.md.

Upload/runtime fallback update (2026-05-12)

• Added a visible top-bar Choose audio affordance in addition to whole-app drag/drop.
• Fixed the default hidden state of the error banner so placeholder errors are not shown on page load.
• API errors now surface request path/status/detail in the visible banner and pipeline logs.
• /api/config now includes runtime diagnostics for optional separation backends.
• If Spleeter is unavailable, the simple UI keeps the app usable by switching to full-mix mode; backend fallback also uses full-mix rather than silently launching Demucs.