docs/interactive-ux/PROGRESS.md

Interactive UX progress

Last updated

2026-05-12

Current phase

Implementation is now in Phase 1–3 foundation: persistent state, constraints, events, confidence/review queue, and supervised cluster interactions are implemented at the semantic-state layer.

Completed in this pass

Item	Status	Notes
Add supplied docs to docs/interactive-ux/	done	All supplied Markdown docs were copied into this directory and aligned with the implemented project.
Persistent job state	done	supervision_state.json is created beside manifest.json for each completed run.
Hit/cluster state schema	done	Implemented in supervised_state.py using JSON-serializable dictionaries.
Event log	done	State mutations append job.state.created, constraint.created, hit.moved, hit.pulled_out, cluster.locked, hit.suppressed, suggestion.created, etc.
Constraint store	done	Supports force-cluster, must-link, cannot-link, lock-cluster, suppress-pattern, and pin-representative.
Confidence scoring	partial	Heuristic scores based on cluster size, label agreement, energy rank, representative/favorite/explicit state, suppression, and lock state. Feature-vector margin scoring is not implemented yet.
Outlier-first review queue	done	Backend computes review_queue; UI renders it and lets the user jump to the selected hit.
Move hit to cluster	done	Endpoint creates constraints and updates state; it also proposes similar move suggestions.
Pull hit into new cluster	done	Endpoint creates cannot-link and force-cluster constraints and a user cluster.
Lock cluster	done	Endpoint toggles lock state and records a lock constraint.
Suppress hit as bleed/noise	done	Endpoint creates suppress-pattern, marks the hit suppressed, and proposes similar suppressions.
Favorite sample / pin representative	partial	Endpoint supports review status favorite, records pin-representative, and updates the representative hit in semantic state. Audio artifact selection is not re-exported yet.
Suggestion inbox	partial	Open suggestions render in the UI and can be accepted/rejected. Suggestion generation is heuristic and limited to move/split/suppress patterns.
Cluster explanation drawer	done	Endpoint and UI show representative, confidence reasons, outliers, relevant constraints, and label distribution.
Undo	done	Last semantic edit can be restored using an undo snapshot stack.
Validation script	done	Added scripts/test_interactive_supervision.py.

Not yet implemented

• Real cached feature-vector store for local reclustering.
• Artifact re-export after semantic edits.
• Waveform click-to-add missed onset.
• Restore suppressed hit/batch restore.
• Real local neighborhood reclustering that changes assignments beyond explicit move/suggestion acceptance.
• Constraint violation detection and reporting.
• Predictive diff preview before accepting suggestions.
• Reconstruction-error-driven correction.
• Multi-resolution/hierarchical clusters.
• User correction profiles / teach mode across songs.
• Frontend TypeScript migration and browser automation tests.

Current risks

Risk	Impact	Mitigation
Semantic edits do not rewrite exports yet	User may expect moved/suppressed hits to affect ZIP/MIDI immediately	Next task should be edited-state export.
Confidence scores are heuristic	Review queue may sometimes prioritize the wrong hits	Add cached mel/transient features and margin-to-next-cluster scoring.
Suggestions are simple	May over-suggest or under-suggest	Keep them previewable, explicit, and undoable; never silently apply.
Locks are semantic only	Batch reruns do not yet replay constraints	Add deterministic replay/local recluster using constraints.
No browser tests	UI regressions are easy	Add Playwright or lightweight DOM tests.

Next implementation milestone

Milestone: edited-state export and force-onset correction.

Minimum deliverables:

1. Add POST /api/jobs/{job_id}/export/supervised that creates a ZIP/MIDI/manifest from supervision_state.json.
2. Exclude suppressed hits/clusters from the supervised export.
3. Honor favorite/pinned representatives in exported samples.
4. Add force-onset endpoint that slices a new hit from cached stem.wav.
5. Add waveform shift-click or add-onset mode in the UI.
6. Add tests proving semantic edits change the supervised export without rerunning stem extraction.

Definition of done for the current foundation

This loop now works:

analyze audio
→ inspect clusters
→ load semantic state
→ move one wrong hit
→ store constraints/events
→ see confidence/review queue update
→ lock corrected cluster
→ suppress bleed
→ inspect explanations and suggestions
→ undo semantic edits
→ reload the job and preserve explicit decisions

The remaining missing piece is that edited semantic state is not yet reflected in a regenerated sample pack.

Pass 5 alignment

The interactive UX docs are now aligned with the implemented semantic edit/export loop. The project supports move, pull-out, suppress, restore, favorite, lock, force-onset, suggestion diff preview, undo, and edited artifact export. The current boundary is no longer “semantic only”; edits can now produce separate supervised WAV/MIDI/reconstruction/ZIP artifacts while original batch outputs remain immutable.

Remaining UX work is concentrated around cluster-level editing and comparison: merge/relabel/split, feature-vector local reclustering, edited-vs-original diff views, and browser tests.

2026-05-12 automatic card-flow update

The default UX now follows the interactive-doc direction more closely by hiding most expert controls and making the user action model concrete:

• drop/upload starts processing automatically;
• waveform appears before backend processing finishes;
• real backend progress tints the waveform;
• sample candidates appear as cards grouped by type;
• dismissing a card becomes a supervised suppression when state is available;
• drawing another candidate models the "draw a card" interaction for missing samples;
• trim/extend controls can save adjusted timing as a forced hit;
• waveform zoom/pan supports close inspection without leaving the main flow.

The remaining mismatch is that drawn candidate cards are still frontend candidate previews, not persisted representative-selection constraints. That should be promoted into the semantic state model next.

Pass 14: selected cards and Spleeter backend

Completed in this pass:

1. Added spleeter as the default separation backend, with selectable spleeter:2stems, spleeter:4stems, and spleeter:5stems profiles.
2. Kept demucs as a quality/fallback backend and none as the full-mix preview backend.
3. Added optional requirements-spleeter.txt instead of forcing TensorFlow/Spleeter into the base install.
4. Added per-card checkbox state with manual select-all/clear behavior.
5. Added selected-only backend export via POST /api/jobs/{job_id}/export-selected.
6. Made draw another persist the chosen representative in semantic state.
7. Made trim/extend rewrite playable preview audio immediately under overrides/hits/.
8. Added scripts/test_selected_export_card_actions.py.

Outcome:

The default app now behaves more like a card review tool: drop audio, let Spleeter/fallback separation run, review grouped cards, select/dismiss/draw/trim, and export only the selected pack.