# Architecture notes for supervised interactive extraction ## Required shift Original extraction flow: ```text audio → pipeline → result artifacts ``` Current interactive foundation: ```text audio/cache → immutable manifest/artifacts → supervision_state.json → reactive UI → user constraints/events/suggestions ``` The current implementation keeps the batch extraction artifacts immutable. Interactive edits mutate `supervision_state.json`, then `supervised_export.py` can render those edits into a separate `supervised/` artifact tree. The original `manifest.json`, original sample WAVs, original MIDI/reconstruction, and original ZIP remain untouched. ## Implemented modules | Module/file | Responsibility | |---|---| | `pipeline_runner.py` | Batch extraction, timing, manifests, review-hit WAV exports | | `sample_extractor.py` | Audio analysis, classification, batch/online clustering, export helpers | | `supervised_state.py` | Persistent semantic job state, constraints, events, confidence, suggestions, force-onset, restore, undo | | `app.py` | FastAPI endpoints for batch jobs, supervised state mutations, force-onset, restore, and edited export | | `supervised_export.py` | Converts semantic state into edited WAV/MIDI/reconstruction/ZIP artifacts under `supervised/` | | `web/app.js` | Browser state rendering, review queue, cluster board, suggestions, actions | | `web/index.html` | Workstation layout and interactive supervision panel | | `web/styles.css` | Visual treatment for low confidence, suppression, locks, panels | ## Core entities as implemented `supervised_state.py` stores JSON dictionaries equivalent to these shapes: ```ts type Hit = { id: string; index: number; label: string; cluster_id: string; original_cluster_id: string; cluster_label: string; onset_sec: number; duration_ms: number; rms_energy: number; spectral_centroid_hz: number; file: string; is_representative: boolean; source: "detected" | "forced"; suppressed: boolean; favorite: boolean; review_status: "unreviewed" | "accepted" | "favorite" | "suppressed"; confidence: number; confidence_reasons: string[]; explicit: boolean; }; type Cluster = { id: string; label: string; classification: string; hit_ids: string[]; representative_hit_id: string | null; locked: boolean; user_named: boolean; confidence: number; confidence_reasons: string[]; suppressed_count: number; original_id: string | null; }; type Constraint = | { id: string; type: "must-link"; a: string; b: string; source: string } | { id: string; type: "cannot-link"; a: string; b: string; source: string } | { id: string; type: "force-cluster"; hit_id: string; cluster_id: string; source: string } | { id: string; type: "lock-cluster"; cluster_id: string; locked: boolean; source: string } | { id: string; type: "suppress-pattern"; example_hit_id: string; reason: string; source: string } | { id: string; type: "pin-representative"; hit_id: string; cluster_id: string; source: string }; ``` ## Current state file Each completed run now gets: ```text .runs//output/manifest.json .runs//output/supervision_state.json .runs//output/supervised/manifest.json # after edited export .runs//output/supervised/sample-pack.zip # after edited export ``` `manifest.json` is the immutable batch result. `supervision_state.json` is the mutable, replayable semantic state. ## Implemented API additions ```text GET /api/jobs/{job_id}/state POST /api/jobs/{job_id}/hits/{hit_id}/move POST /api/jobs/{job_id}/hits/{hit_id}/pull-out POST /api/jobs/{job_id}/hits/{hit_id}/suppress POST /api/jobs/{job_id}/hits/{hit_id}/review POST /api/jobs/{job_id}/hits/{hit_id}/restore POST /api/jobs/{job_id}/hits/force-onset POST /api/jobs/{job_id}/export POST /api/jobs/{job_id}/clusters/{cluster_id}/lock GET /api/jobs/{job_id}/suggestions POST /api/jobs/{job_id}/suggestions/{suggestion_id}/accept POST /api/jobs/{job_id}/suggestions/{suggestion_id}/reject GET /api/jobs/{job_id}/explain/cluster/{cluster_id} POST /api/jobs/{job_id}/undo ``` ## Current confidence scoring Initial confidence is heuristic and deterministic. It combines: - cluster size, - cluster label purity, - representative presence, - lock state, - hit label agreement with cluster label, - energy rank, - rough duration reasonableness, - representative/favorite/explicit assignment state, - suppression state. This is good enough to drive the review queue but should be replaced or supplemented by cached feature-vector margins. ## Current suggestion engine Implemented suggestion types: ```ts type Suggestion = | { type: "move-hits"; hit_ids: string[]; target_cluster_id: string; confidence: number; reason: string } | { type: "split-hits"; hit_ids: string[]; source_cluster_id: string; target_cluster_id: string; confidence: number; reason: string } | { type: "suppress-hits"; hit_ids: string[]; confidence: number; reason: string }; ``` Suggestion generation currently uses label, spectral centroid, and RMS-energy similarity. Open suggestions include exact `diff` previews showing affected hits and cluster counts before/after. Accepted suggestions become explicit constraints/examples. ## Event log Examples now emitted: ```text job.state.created constraint.created hit.moved hit.pulled_out cluster.locked cluster.unlocked hit.suppressed hit.reviewed suggestion.created suggestion.accepted suggestion.rejected state.undo hit.force_onset hit.restored supervised.exported ``` The UI renders recent events and constraints in the supervision panel. ## Local recomputation boundary Implemented now: ```text semantic edit → update hit/cluster membership in supervision_state.json → append constraints/events → generate heuristic suggestions → recompute hit/cluster confidence and review queue ``` Still not implemented: ```text semantic edit → load cached feature vectors → choose affected neighborhood → run constrained local reclustering → update suggestion/recluster preview from real feature margins ``` Implemented now: ```text semantic edit → export supervised state → write edited WAV/MIDI/reconstruction/ZIP under supervised/ ``` ## UI state implications Implemented panels: - waveform + onset audition, - representative samples, - detected hit review, - outlier-first review queue, - cluster board, - suggestion inbox, - constraint/history inspector, - cluster explanation drawer, - export/download panel for the original batch run. Still missing: - cluster merge/relabel/split controls, - edited-vs-original comparison view, - cached feature-neighborhood local reclustering. ## Implementation warning Automatic propagation must stay conservative. The current implementation follows this by creating suggestions rather than silently moving/suppressing batches. Every semantic edit is undoable.