docs/interactive-ux/ARCHITECTURE_NOTES.md

Architecture notes for supervised interactive extraction

Required shift

Original extraction flow:

audio → pipeline → result artifacts

Current interactive foundation:

audio/cache → immutable manifest/artifacts → supervision_state.json → reactive UI → user constraints/events/suggestions

The current implementation keeps the batch extraction artifacts immutable. Interactive edits mutate supervision_state.json, then supervised_export.py can render those edits into a separate supervised/ artifact tree. The original manifest.json, original sample WAVs, original MIDI/reconstruction, and original ZIP remain untouched.

Implemented modules

Module/file	Responsibility
pipeline_runner.py	Batch extraction, timing, manifests, review-hit WAV exports
sample_extractor.py	Audio analysis, classification, batch/online clustering, export helpers
supervised_state.py	Persistent semantic job state, constraints, events, confidence, suggestions, force-onset, restore, undo
app.py	FastAPI endpoints for batch jobs, supervised state mutations, force-onset, restore, and edited export
supervised_export.py	Converts semantic state into edited WAV/MIDI/reconstruction/ZIP artifacts under supervised/
web/app.js	Browser state rendering, review queue, cluster board, suggestions, actions
web/index.html	Workstation layout and interactive supervision panel
web/styles.css	Visual treatment for low confidence, suppression, locks, panels

Core entities as implemented

supervised_state.py stores JSON dictionaries equivalent to these shapes:

type Hit = {
  id: string;
  index: number;
  label: string;
  cluster_id: string;
  original_cluster_id: string;
  cluster_label: string;
  onset_sec: number;
  duration_ms: number;
  rms_energy: number;
  spectral_centroid_hz: number;
  file: string;
  is_representative: boolean;
  source: "detected" | "forced";
  suppressed: boolean;
  favorite: boolean;
  review_status: "unreviewed" | "accepted" | "favorite" | "suppressed";
  confidence: number;
  confidence_reasons: string[];
  explicit: boolean;
};

type Cluster = {
  id: string;
  label: string;
  classification: string;
  hit_ids: string[];
  representative_hit_id: string | null;
  locked: boolean;
  user_named: boolean;
  confidence: number;
  confidence_reasons: string[];
  suppressed_count: number;
  original_id: string | null;
};

type Constraint =
  | { id: string; type: "must-link"; a: string; b: string; source: string }
  | { id: string; type: "cannot-link"; a: string; b: string; source: string }
  | { id: string; type: "force-cluster"; hit_id: string; cluster_id: string; source: string }
  | { id: string; type: "lock-cluster"; cluster_id: string; locked: boolean; source: string }
  | { id: string; type: "suppress-pattern"; example_hit_id: string; reason: string; source: string }
  | { id: string; type: "pin-representative"; hit_id: string; cluster_id: string; source: string };

Current state file

Each completed run now gets:

.runs/<job_id>/output/manifest.json
.runs/<job_id>/output/supervision_state.json
.runs/<job_id>/output/supervised/manifest.json        # after edited export
.runs/<job_id>/output/supervised/sample-pack.zip      # after edited export

manifest.json is the immutable batch result. supervision_state.json is the mutable, replayable semantic state.

Implemented API additions

GET    /api/jobs/{job_id}/state
POST   /api/jobs/{job_id}/hits/{hit_id}/move
POST   /api/jobs/{job_id}/hits/{hit_id}/pull-out
POST   /api/jobs/{job_id}/hits/{hit_id}/suppress
POST   /api/jobs/{job_id}/hits/{hit_id}/review
POST   /api/jobs/{job_id}/hits/{hit_id}/restore
POST   /api/jobs/{job_id}/hits/force-onset
POST   /api/jobs/{job_id}/export
POST   /api/jobs/{job_id}/clusters/{cluster_id}/lock
GET    /api/jobs/{job_id}/suggestions
POST   /api/jobs/{job_id}/suggestions/{suggestion_id}/accept
POST   /api/jobs/{job_id}/suggestions/{suggestion_id}/reject
GET    /api/jobs/{job_id}/explain/cluster/{cluster_id}
POST   /api/jobs/{job_id}/undo

Current confidence scoring

Initial confidence is heuristic and deterministic. It combines:

• cluster size,
• cluster label purity,
• representative presence,
• lock state,
• hit label agreement with cluster label,
• energy rank,
• rough duration reasonableness,
• representative/favorite/explicit assignment state,
• suppression state.

This is good enough to drive the review queue but should be replaced or supplemented by cached feature-vector margins.

Current suggestion engine

Implemented suggestion types:

type Suggestion =
  | { type: "move-hits"; hit_ids: string[]; target_cluster_id: string; confidence: number; reason: string }
  | { type: "split-hits"; hit_ids: string[]; source_cluster_id: string; target_cluster_id: string; confidence: number; reason: string }
  | { type: "suppress-hits"; hit_ids: string[]; confidence: number; reason: string };

Suggestion generation currently uses label, spectral centroid, and RMS-energy similarity. Open suggestions include exact diff previews showing affected hits and cluster counts before/after. Accepted suggestions become explicit constraints/examples.

Event log

Examples now emitted:

job.state.created
constraint.created
hit.moved
hit.pulled_out
cluster.locked
cluster.unlocked
hit.suppressed
hit.reviewed
suggestion.created
suggestion.accepted
suggestion.rejected
state.undo
hit.force_onset
hit.restored
supervised.exported

The UI renders recent events and constraints in the supervision panel.

Local recomputation boundary

Implemented now:

semantic edit
→ update hit/cluster membership in supervision_state.json
→ append constraints/events
→ generate heuristic suggestions
→ recompute hit/cluster confidence and review queue

Still not implemented:

semantic edit
→ load cached feature vectors
→ choose affected neighborhood
→ run constrained local reclustering
→ update suggestion/recluster preview from real feature margins

Implemented now:

semantic edit
→ export supervised state
→ write edited WAV/MIDI/reconstruction/ZIP under supervised/

UI state implications

Implemented panels:

• waveform + onset audition,
• representative samples,
• detected hit review,
• outlier-first review queue,
• cluster board,
• suggestion inbox,
• constraint/history inspector,
• cluster explanation drawer,
• export/download panel for the original batch run.

Still missing:

• cluster merge/relabel/split controls,
• edited-vs-original comparison view,
• cached feature-neighborhood local reclustering.

Implementation warning

Automatic propagation must stay conservative. The current implementation follows this by creating suggestions rather than silently moving/suppressing batches. Every semantic edit is undoable.