Spaces:

Ashiedu
/

Synesthesia

Runtime error

App Files Files Community

Synesthesia / github-issues /T-002.md

Ashiedu

Sync unified workbench

0490201 verified about 1 month ago

preview code

raw

history blame contribute delete

38.4 kB

A newer version of the Gradio SDK is available: 6.14.0

Upgrade

T-002: IPC Command Registry — All Typed Serde Structs & TypeScript Mirror Types

Type: Task
Phase: 0 — Foundation
Autonomy: agent:review-required — Agent writes all types and stub handlers. Human reviews PR before merge. A type error here cascades to all 148 remaining tasks.
Stack: stack:rust stack:typescript
Version: v0.1
Iteration: iter-1
Effort: M (1–2 days)

⚠️ Agent Scope: Write the type system and stub command registry only. Every command handler returns IpcError::NotImplemented. No audio, no models, no cameras, no controllers. Logic arrives in T-003 through T-150. Your deliverable is a compile-clean, fully-typed contract.

⚠️ Contract Rule: Once merged, no type may be renamed, removed, or have a field removed without a new migration sub-issue. Adding Option<T> fields to existing structs is safe without a migration.

⚠️ Naming: The Tauri backend directory is kansas/ (not src-tauri/). All paths below reflect this. See AMENDMENT-001.

Context

Tauri IPC is the nervous system of Synesthesia. Every subsystem communicates with the Rust backend through typed invoke() calls and typed emit() events. If Rust and TypeScript types diverge, the mismatch surfaces as a silent runtime undefined or a JSON parse panic — not a compile error.

This task eliminates that class of bug by defining the complete type surface in one place on both sides, enforcing serde(rename_all = "camelCase") so the JSON wire format matches TypeScript conventions, and providing typed wrappers for every command and event in the full v1 system. Many commands will have stub handlers for months. That is correct. The stubs are compile-time documentation.

Prerequisites

T-001 merged — kansas/, src/, workspace Cargo.toml with edition = "2024" and wildcard deps exist
kansas/src/commands/mod.rs contains only ping() from T-001
cargo check --workspace passes on T-001 output before starting

Acceptance Criteria

Rust

cargo check --workspace — zero errors
cargo clippy --workspace -- -D warnings — zero warnings
Every public type derives Debug, Clone, Serialize, Deserialize
Every struct/enum has // [T-NNN] doc comment for the implementing task
All serde types use #[serde(rename_all = "camelCase")]
Option<T> fields use #[serde(skip_serializing_if = "Option::is_none")]
All stubs return Result<_, IpcError> — no unwrap!, panic!, todo!()
ipc module declared in kansas/src/lib.rs

TypeScript

pnpm typecheck — zero errors
src/ipc/types.ts — all interfaces match Rust counterparts field-for-field
src/ipc/commands.ts — typed invoke() wrapper for every command
src/ipc/events.ts — typed listen() wrapper for every event
No any types in the IPC layer

Registration

All commands in the Commands Catalog registered in tauri::generate_handler![]
All events in the Events Catalog have a TypeScript listen() wrapper
ping from T-001 preserved and moved into the new module structure

Capability

kansas/capabilities/default.json lists every new command
No command callable without an explicit capability entry

Serialization Contract

Rule: Rust is snake_case internally. JSON wire format and TypeScript are camelCase.

#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ExamplePayload {
    pub some_field: f32,           // → "someField"
    #[serde(skip_serializing_if = "Option::is_none")]
    pub optional_field: Option<u32>,
}

// Unit enum → tagged string
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum TransportState { Playing, Paused, Stopped }
// → { "kind": "playing" }

// Data-carrying enum → adjacent tagged
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "type", content = "data")]
pub enum HapticKind {
    BeatPulse { intensity: f32 },
    ChunkBoundary,
}
// → { "type": "beatPulse", "data": { "intensity": 0.8 } }

File Structure After This Task

kansas/src/
├── main.rs              ← registers all commands
├── lib.rs               ← declares ipc + commands modules
├── ipc/
│   ├── mod.rs           ← re-exports types, error
│   ├── types.rs         ← ALL serde structs and enums
│   └── error.rs         ← IpcError enum
└── commands/
    ├── mod.rs           ← re-exports all submodules
    ├── ping.rs          ← moved from T-001
    ├── transport.rs     ← stub → T-003
    ├── audio.rs         ← stub → T-004
    ├── models.rs        ← stub → T-046+
    ├── camera.rs        ← stub → T-031+
    ├── controllers.rs   ← stub → T-016+
    ├── spatial.rs       ← stub → T-096+
    ├── devices.rs       ← stub → T-013
    └── app_state.rs     ← stub → T-048

src/ipc/
├── types.ts             ← ALL TypeScript mirror interfaces
├── commands.ts          ← typed invoke() wrappers
└── events.ts            ← typed listen() wrappers

IPC Error Type

// kansas/src/ipc/error.rs
use serde::Serialize;
use thiserror::Error;

#[derive(Debug, Error, Serialize)]
#[serde(rename_all = "camelCase", tag = "kind", content = "message")]
pub enum IpcError {
    #[error("Not yet implemented: {0}")]
    NotImplemented(String),
    #[error("Invalid parameter: {0}")]
    InvalidParam(String),
    #[error("Device not found: {0}")]
    DeviceNotFound(String),
    #[error("Model not loaded: {0}")]
    ModelNotLoaded(String),
    #[error("Audio error: {0}")]
    AudioError(String),
    #[error("Internal error: {0}")]
    Internal(String),
}

impl From<IpcError> for String {
    fn from(e: IpcError) -> Self {
        serde_json::to_string(&e).unwrap_or_else(|_| e.to_string())
    }
}

// src/ipc/types.ts (add at top)
export type IpcErrorKind =
  | 'notImplemented' | 'invalidParam' | 'deviceNotFound'
  | 'modelNotLoaded' | 'audioError' | 'internal';

export interface IpcError { kind: IpcErrorKind; message: string; }

export function parseIpcError(raw: unknown): IpcError {
  if (typeof raw === 'string') {
    try { return JSON.parse(raw) as IpcError; } catch { /* fall through */ }
  }
  return { kind: 'internal', message: String(raw) };
}

Complete Types — Rust (`kansas/src/ipc/types.rs`)

use serde::{Deserialize, Serialize};

// ─── Primitives ───────────────────────────────────────────────────────────────

/// 3D vector; units depend on context (cm for head pos, meters for spatial audio)
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Vec3 { pub x: f32, pub y: f32, pub z: f32 }

/// Unit quaternion for orientation
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Quaternion { pub x: f32, pub y: f32, pub z: f32, pub w: f32 }
impl Default for Quaternion {
    fn default() -> Self { Self { x: 0.0, y: 0.0, z: 0.0, w: 1.0 } }
}

/// RGB color 0–255
#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Rgb { pub r: u8, pub g: u8, pub b: u8 }

// ─── Transport [T-003] ────────────────────────────────────────────────────────

/// [T-003] Partial update — only set fields that should change
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct TransportSetCmd {
    #[serde(skip_serializing_if = "Option::is_none")]
    pub state: Option<TransportState>,
    /// BPM 20.0–300.0
    #[serde(skip_serializing_if = "Option::is_none")]
    pub bpm: Option<f32>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub time_sig_numerator: Option<u8>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub time_sig_denominator: Option<u8>,
}

/// [T-003]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum TransportState { #[default] Stopped, Playing, Paused }

/// [T-003] Full snapshot returned by transport_get
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct TransportStatus {
    pub state: TransportState,
    pub bpm: f32,
    pub time_sig_numerator: u8,
    pub time_sig_denominator: u8,
    pub bar: u32,
    pub beat: u8,   // 1-indexed, 1..=time_sig_numerator
    pub tick: u32,  // 0..=95 (96 PPQN)
    pub elapsed_ms: u64,
}

/// [T-003] Emitted every tick by transport thread
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct TransportTickEvent {
    pub bar: u32,
    pub beat: u8,
    pub tick: u32,
    pub bpm: f32,
    pub elapsed_ms: u64,
    pub is_downbeat: bool,
}

// ─── Parameter Bus [T-024] ────────────────────────────────────────────────────

/// [T-024] Set any named parameter. value always 0.0–1.0 normalized.
/// param_id dot-notation: "perfrnn.lead.temperature", "magenta_rt.style_weight.0"
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ParamSetCmd {
    pub param_id: String,
    pub value: f32,
    pub source: ParamSource,
}

/// [T-024]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum ParamSource {
    Ui,
    Controller { id: ControllerId },
    Gesture,
    HeadTracking,
    Internal,
}

/// [T-024] Broadcast when any parameter changes
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ParamChangedEvent {
    pub param_id: String,
    pub value: f32,
    pub source: ParamSource,
    pub timestamp_ms: u64,
}

// ─── Model Inference [T-046+] ─────────────────────────────────────────────────

/// [T-046]
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum ModelId {
    MusicVaeEncoder, MusicVaeDecoder,
    PerfRnnStep,
    SpectroStreamEncode, SpectroStreamDecode,
    MusicCocaText, MusicCocaAudio,
    MagnetaRtGenerate,
    DdspEncode, DdspDecode,
    GemmaInference,
}

/// [T-046] input is model-specific JSON; handler validates + deserializes
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ModelInferCmd {
    pub model: ModelId,
    pub input: serde_json::Value,
}

/// [T-046]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ModelInferResult {
    pub model: ModelId,
    pub output: serde_json::Value,
    pub latency_ms: f32,
}

/// [T-127] Hot-swap ONNX file without restart
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ModelReloadCmd {
    pub model: ModelId,
    pub path: String,
}

/// [T-046] Emitted after inference completes (Rerun logging hook)
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ModelInferenceCompleteEvent {
    pub model: ModelId,
    pub latency_ms: f32,
    pub timestamp_ms: u64,
}

// ─── Devices [T-013] ──────────────────────────────────────────────────────────

/// [T-013]
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum DeviceKind { AudioInput, AudioOutput, Camera }

/// [T-013]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct DeviceInfo {
    pub id: u32,
    pub name: String,
    pub kind: DeviceKind,
    pub is_default: bool,
}

/// [T-013]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct DeviceListResult { pub devices: Vec<DeviceInfo> }

// ─── Camera [T-031–T-034] ─────────────────────────────────────────────────────

/// [T-034]
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum CameraRole {
    Pose,  // Primary: 30 FPS, MediaPipe + Gemma-3N
    Head,  // Secondary: 120 FPS stereo, head tracking only
}

/// [T-034]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CameraAssignCmd { pub role: CameraRole, pub device_id: u32 }

/// [T-031]
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CameraAssignment {
    #[serde(skip_serializing_if = "Option::is_none")]
    pub pose_device_id: Option<u32>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub head_device_id: Option<u32>,
}

// ─── Skeleton Pose [T-035–T-038] ─────────────────────────────────────────────

/// [T-038] Single MediaPipe BlazePose landmark (full topology: developers.google.com/mediapipe/solutions/vision/pose_landmarker)
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct PoseLandmark {
    pub x: f32,           // normalized 0.0 (left) – 1.0 (right)
    pub y: f32,           // normalized 0.0 (top) – 1.0 (bottom)
    pub z: f32,           // depth relative to hips, normalized
    pub visibility: f32,  // confidence 0.0–1.0
}

/// [T-038] 33-landmark skeleton; landmarks[0]=nose, [11]=L shoulder, [16]=R wrist
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct SkeletonPoseEvent {
    pub landmarks: Vec<PoseLandmark>,  // always 33 elements
    pub timestamp_ms: u64,
    pub detected: bool,
}

// ─── Head Tracking [T-039–T-041] ─────────────────────────────────────────────

/// [T-041] 6-DOF head pose from secondary 120 FPS stereo camera, broadcast at 120 Hz
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct HeadPoseEvent {
    pub position: Vec3,       // cm from camera origin
    pub orientation: Quaternion,
    pub yaw_deg: f32,
    pub pitch_deg: f32,
    pub roll_deg: f32,
    pub timestamp_ms: u64,
    pub confidence: f32,      // 0.0–1.0; below 0.5 treat as unreliable
}

// ─── Controllers [T-016–T-029] ────────────────────────────────────────────────

/// [T-016]
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum ControllerId { DualSense, JoyConLeft, JoyConRight }

/// [T-022] State snapshot broadcast at 60 Hz per connected controller
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ControllerStateEvent {
    pub controller: ControllerId,
    pub axes: Vec<f32>,      // –1.0–1.0, layout controller-specific (T-017, T-018)
    pub buttons: Vec<bool>,  // layout controller-specific
    #[serde(skip_serializing_if = "Option::is_none")]
    pub gyro: Option<Vec3>,  // rad/s
    #[serde(skip_serializing_if = "Option::is_none")]
    pub accel: Option<Vec3>, // m/s²
    #[serde(skip_serializing_if = "Option::is_none")]
    pub left_trigger: Option<f32>,   // DualSense only, 0.0–1.0
    #[serde(skip_serializing_if = "Option::is_none")]
    pub right_trigger: Option<f32>,  // DualSense only, 0.0–1.0
    #[serde(skip_serializing_if = "Option::is_none")]
    pub touchpad: Option<[f32; 2]>,  // DualSense only, normalized
    pub timestamp_ms: u64,
}

/// [T-019–T-021] Send haptic or LED command to a controller
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct HapticCmd { pub controller: ControllerId, pub kind: HapticKind }

/// [T-019–T-021]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "type", content = "data")]
pub enum HapticKind {
    BeatPulse { intensity: f32 },
    NoteOnset { note: u8, velocity: u8 },
    ChunkBoundary,
    MoodChange,
    TriggerResistance { trigger: TriggerSide, force: f32, start_pos: f32 },
    LedColor { r: u8, g: u8, b: u8 },
    RumbleLf { intensity: f32, duration_ms: u32 },
}

/// [T-019]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum TriggerSide { Left, Right }

/// [T-016] Emitted on connect/disconnect
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ControllerConnectionEvent {
    pub controller: ControllerId,
    pub connected: bool,
    pub timestamp_ms: u64,
}

// ─── Spatial Audio [T-096–T-102] ─────────────────────────────────────────────

/// [T-099] Position a named audio source for Steam Audio.
/// source_id: "perfrnn.lead" | "perfrnn.pad" | "perfrnn.bass" |
///            "magenta_rt" | "ddsp" | "pitch_class.{0..11}"
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct SourceSetPositionCmd {
    pub source_id: String,
    pub position: Vec3,  // meters, right-hand, origin at listener default
    pub gain: f32,       // 0.0–1.0
}

/// [T-101] Initialize/update Steam Audio room model
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct RoomModelCmd {
    pub width: f32,
    pub height: f32,
    pub depth: f32,
    pub absorption: f32,
}

/// [T-133] Load SOFA HRTF personalization file
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct HrtfLoadCmd { pub path: String }

// ─── Gemma-3N / Mood [T-049–T-052] ───────────────────────────────────────────

/// [T-052] Output of Gemma-3N visual analysis, broadcast at 0.5 Hz
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct MoodState {
    pub mood: String,       // e.g. "mysterious"
    pub energy: f32,        // 0.0–1.0
    pub texture: String,    // e.g. "dense"
    pub timestamp_ms: u64,
}

// ─── Magenta RT Style [T-083–T-086] ──────────────────────────────────────────

/// [T-084] Macro knobs 1–6 → MusicCoCa embedding blend weights. 0.0–1.0 each.
/// Need not sum to 1.0; backend normalizes before applying.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct StyleBlendSetCmd { pub weights: [f32; 6] }

/// [T-088] Emitted after each 2-second Magenta RT generation chunk
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct GenerationChunkEvent {
    pub source: GenerationSource,
    pub duration_ms: u32,
    pub timestamp_ms: u64,
}

/// [T-088]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind", content = "data")]
pub enum GenerationSource {
    MagnetaRt,
    PerfRnn { channel: RnnChannel },
    Ddsp,
}

/// [T-070]
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", tag = "kind")]
pub enum RnnChannel { Lead, Pad, Bass }

// ─── Application State [T-048+] ───────────────────────────────────────────────

/// [T-048] Full snapshot returned by get_app_state
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct AppState {
    pub transport: TransportStatus,
    pub webgpu_active: bool,
    pub controllers_connected: Vec<ControllerId>,
    pub camera_assignment: CameraAssignment,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub current_mood: Option<MoodState>,
    pub style_weights: [f32; 6],
}

Stub Command Pattern

Every unimplemented handler follows this exactly:

// kansas/src/commands/transport.rs
use crate::ipc::{error::IpcError, types::*};

/// [T-003] Set transport state and/or BPM
#[tauri::command]
pub fn transport_set(cmd: TransportSetCmd) -> Result<TransportStatus, IpcError> {
    Err(IpcError::NotImplemented("T-003".into()))
}

/// [T-003] Get current transport status
#[tauri::command]
pub fn transport_get() -> Result<TransportStatus, IpcError> {
    Err(IpcError::NotImplemented("T-003".into()))
}

Repeat this pattern for every command below. The T-NNN string in NotImplemented must match the task that implements it.

Commands Catalog

Command	Input	Return	Implements
`ping`	—	`&'static str`	T-001 ✅
`transport_set`	`TransportSetCmd`	`TransportStatus`	T-003
`transport_get`	—	`TransportStatus`	T-003
`model_infer`	`ModelInferCmd`	`ModelInferResult`	T-046
`model_reload`	`ModelReloadCmd`	`()`	T-127
`camera_assign`	`CameraAssignCmd`	`CameraAssignment`	T-034
`param_set`	`ParamSetCmd`	`()`	T-024
`haptic_send`	`HapticCmd`	`()`	T-019
`source_set_position`	`SourceSetPositionCmd`	`()`	T-099
`room_model_set`	`RoomModelCmd`	`()`	T-101
`hrtf_load`	`HrtfLoadCmd`	`()`	T-133
`style_blend_set`	`StyleBlendSetCmd`	`()`	T-084
`device_list`	—	`DeviceListResult`	T-013
`get_app_state`	—	`AppState`	T-048

Events Catalog

Fire-and-forget from Rust backend via app_handle.emit(name, payload).

Event	Payload	Emitted by
`transport_tick`	`TransportTickEvent`	T-003
`param_changed`	`ParamChangedEvent`	T-024
`skeleton_pose`	`SkeletonPoseEvent`	T-038
`head_pose`	`HeadPoseEvent`	T-041
`controller_state`	`ControllerStateEvent`	T-022
`controller_connection`	`ControllerConnectionEvent`	T-016
`mood_changed`	`MoodState`	T-052
`generation_chunk`	`GenerationChunkEvent`	T-088
`model_inference_complete`	`ModelInferenceCompleteEvent`	T-046
`device_list_updated`	`DeviceListResult`	T-013

TypeScript Mirror Types (`src/ipc/types.ts`)

// ─── Primitives ───────────────────────────────────────────────────────────────
export interface Vec3 { x: number; y: number; z: number }
export interface Quaternion { x: number; y: number; z: number; w: number }
export interface Rgb { r: number; g: number; b: number }
export type IpcErrorKind = 'notImplemented'|'invalidParam'|'deviceNotFound'|'modelNotLoaded'|'audioError'|'internal'
export interface IpcError { kind: IpcErrorKind; message: string }
export function parseIpcError(raw: unknown): IpcError {
  if (typeof raw === 'string') { try { return JSON.parse(raw) as IpcError } catch {} }
  return { kind: 'internal', message: String(raw) }
}

// ─── Transport ────────────────────────────────────────────────────────────────
export type TransportStateKind = 'stopped' | 'playing' | 'paused'
export interface TransportState { kind: TransportStateKind }
export interface TransportSetCmd {
  state?: TransportState; bpm?: number
  timeSigNumerator?: number; timeSigDenominator?: number
}
export interface TransportStatus {
  state: TransportState; bpm: number
  timeSigNumerator: number; timeSigDenominator: number
  bar: number; beat: number; tick: number; elapsedMs: number
}
export interface TransportTickEvent {
  bar: number; beat: number; tick: number
  bpm: number; elapsedMs: number; isDownbeat: boolean
}

// ─── Parameter Bus ────────────────────────────────────────────────────────────
export type ParamSourceKind = 'ui' | 'controller' | 'gesture' | 'headTracking' | 'internal'
export interface ParamSource { kind: ParamSourceKind; id?: ControllerId }
export interface ParamSetCmd { paramId: string; value: number; source: ParamSource }
export interface ParamChangedEvent { paramId: string; value: number; source: ParamSource; timestampMs: number }

// ─── Models ───────────────────────────────────────────────────────────────────
export type ModelIdKind =
  | 'musicVaeEncoder' | 'musicVaeDecoder' | 'perfRnnStep'
  | 'spectroStreamEncode' | 'spectroStreamDecode'
  | 'musicCocaText' | 'musicCocaAudio' | 'magnetaRtGenerate'
  | 'ddspEncode' | 'ddspDecode' | 'gemmaInference'
export interface ModelId { kind: ModelIdKind }
export interface ModelInferCmd { model: ModelId; input: unknown }
export interface ModelInferResult { model: ModelId; output: unknown; latencyMs: number }
export interface ModelReloadCmd { model: ModelId; path: string }
export interface ModelInferenceCompleteEvent { model: ModelId; latencyMs: number; timestampMs: number }

// ─── Devices ──────────────────────────────────────────────────────────────────
export type DeviceKind = 'audioInput' | 'audioOutput' | 'camera'
export interface DeviceInfo { id: number; name: string; kind: DeviceKind; isDefault: boolean }
export interface DeviceListResult { devices: DeviceInfo[] }

// ─── Camera ───────────────────────────────────────────────────────────────────
export type CameraRoleKind = 'pose' | 'head'
export interface CameraRole { kind: CameraRoleKind }
export interface CameraAssignCmd { role: CameraRole; deviceId: number }
export interface CameraAssignment { poseDeviceId?: number; headDeviceId?: number }

// ─── Skeleton Pose ────────────────────────────────────────────────────────────
export interface PoseLandmark { x: number; y: number; z: number; visibility: number }
export interface SkeletonPoseEvent {
  landmarks: PoseLandmark[]  // always 33
  timestampMs: number; detected: boolean
}

// ─── Head Tracking ────────────────────────────────────────────────────────────
export interface HeadPoseEvent {
  position: Vec3; orientation: Quaternion
  yawDeg: number; pitchDeg: number; rollDeg: number
  timestampMs: number; confidence: number
}

// ─── Controllers ──────────────────────────────────────────────────────────────
export type ControllerIdKind = 'dualSense' | 'joyConLeft' | 'joyConRight'
export interface ControllerId { kind: ControllerIdKind }
export interface ControllerStateEvent {
  controller: ControllerId; axes: number[]; buttons: boolean[]
  gyro?: Vec3; accel?: Vec3
  leftTrigger?: number; rightTrigger?: number
  touchpad?: [number, number]; timestampMs: number
}
export type TriggerSideKind = 'left' | 'right'
export type HapticKindType =
  | { type: 'beatPulse';          data: { intensity: number } }
  | { type: 'noteOnset';          data: { note: number; velocity: number } }
  | { type: 'chunkBoundary';      data: Record<string, never> }
  | { type: 'moodChange';         data: Record<string, never> }
  | { type: 'triggerResistance';  data: { trigger: { kind: TriggerSideKind }; force: number; startPos: number } }
  | { type: 'ledColor';           data: { r: number; g: number; b: number } }
  | { type: 'rumbleLf';           data: { intensity: number; durationMs: number } }
export interface HapticCmd { controller: ControllerId; kind: HapticKindType }
export interface ControllerConnectionEvent { controller: ControllerId; connected: boolean; timestampMs: number }

// ─── Spatial Audio ────────────────────────────────────────────────────────────
export interface SourceSetPositionCmd { sourceId: string; position: Vec3; gain: number }
export interface RoomModelCmd { width: number; height: number; depth: number; absorption: number }
export interface HrtfLoadCmd { path: string }

// ─── Mood ─────────────────────────────────────────────────────────────────────
export interface MoodState { mood: string; energy: number; texture: string; timestampMs: number }

// ─── Magenta RT ───────────────────────────────────────────────────────────────
export interface StyleBlendSetCmd { weights: [number,number,number,number,number,number] }
export type RnnChannelKind = 'lead' | 'pad' | 'bass'
export interface RnnChannel { kind: RnnChannelKind }
export type GenerationSource =
  | { kind: 'magnetaRt' }
  | { kind: 'perfRnn'; data: { channel: RnnChannel } }
  | { kind: 'ddsp' }
export interface GenerationChunkEvent { source: GenerationSource; durationMs: number; timestampMs: number }

// ─── App State ────────────────────────────────────────────────────────────────
export interface AppState {
  transport: TransportStatus; webgpuActive: boolean
  controllersConnected: ControllerId[]; cameraAssignment: CameraAssignment
  currentMood?: MoodState
  styleWeights: [number,number,number,number,number,number]
}

TypeScript Wrappers

`src/ipc/commands.ts`

import { invoke } from '@tauri-apps/api/core'
import type {
  TransportSetCmd, TransportStatus,
  ModelInferCmd, ModelInferResult, ModelReloadCmd,
  CameraAssignCmd, CameraAssignment,
  ParamSetCmd, HapticCmd,
  SourceSetPositionCmd, RoomModelCmd, HrtfLoadCmd,
  StyleBlendSetCmd, DeviceListResult, AppState,
} from './types'

export const ping               = ():                                  Promise<string>          => invoke('ping')
export const transportSet       = (cmd: TransportSetCmd):              Promise<TransportStatus> => invoke('transport_set',       { cmd })
export const transportGet       = ():                                  Promise<TransportStatus> => invoke('transport_get')
export const modelInfer         = (cmd: ModelInferCmd):                Promise<ModelInferResult>=> invoke('model_infer',         { cmd })
export const modelReload        = (cmd: ModelReloadCmd):               Promise<void>            => invoke('model_reload',        { cmd })
export const cameraAssign       = (cmd: CameraAssignCmd):              Promise<CameraAssignment>=> invoke('camera_assign',       { cmd })
export const paramSet           = (cmd: ParamSetCmd):                  Promise<void>            => invoke('param_set',           { cmd })
export const hapticSend         = (cmd: HapticCmd):                    Promise<void>            => invoke('haptic_send',         { cmd })
export const sourceSetPosition  = (cmd: SourceSetPositionCmd):         Promise<void>            => invoke('source_set_position', { cmd })
export const roomModelSet       = (cmd: RoomModelCmd):                 Promise<void>            => invoke('room_model_set',      { cmd })
export const hrtfLoad           = (cmd: HrtfLoadCmd):                  Promise<void>            => invoke('hrtf_load',           { cmd })
export const styleBlendSet      = (cmd: StyleBlendSetCmd):             Promise<void>            => invoke('style_blend_set',     { cmd })
export const deviceList         = ():                                  Promise<DeviceListResult>=> invoke('device_list')
export const getAppState        = ():                                  Promise<AppState>         => invoke('get_app_state')

`src/ipc/events.ts`

import { listen, type UnlistenFn } from '@tauri-apps/api/event'
import type {
  TransportTickEvent, ParamChangedEvent,
  SkeletonPoseEvent, HeadPoseEvent,
  ControllerStateEvent, ControllerConnectionEvent,
  MoodState, GenerationChunkEvent,
  ModelInferenceCompleteEvent, DeviceListResult,
} from './types'

// IMPORTANT: Every returned UnlistenFn MUST be called on component unmount.
// Dangling listeners accumulate and will OOM the WebView over long sessions.

const on = <T>(event: string, cb: (p: T) => void): Promise<UnlistenFn> =>
  listen<T>(event, (e) => cb(e.payload))

export const onTransportTick          = (cb: (e: TransportTickEvent)          => void) => on<TransportTickEvent>         ('transport_tick',          cb)
export const onParamChanged           = (cb: (e: ParamChangedEvent)           => void) => on<ParamChangedEvent>          ('param_changed',           cb)
export const onSkeletonPose           = (cb: (e: SkeletonPoseEvent)           => void) => on<SkeletonPoseEvent>          ('skeleton_pose',           cb)
export const onHeadPose               = (cb: (e: HeadPoseEvent)               => void) => on<HeadPoseEvent>              ('head_pose',               cb)
export const onControllerState        = (cb: (e: ControllerStateEvent)        => void) => on<ControllerStateEvent>       ('controller_state',        cb)
export const onControllerConnection   = (cb: (e: ControllerConnectionEvent)   => void) => on<ControllerConnectionEvent>  ('controller_connection',   cb)
export const onMoodChanged            = (cb: (e: MoodState)                   => void) => on<MoodState>                  ('mood_changed',            cb)
export const onGenerationChunk        = (cb: (e: GenerationChunkEvent)        => void) => on<GenerationChunkEvent>       ('generation_chunk',        cb)
export const onModelInferenceComplete = (cb: (e: ModelInferenceCompleteEvent) => void) => on<ModelInferenceCompleteEvent>('model_inference_complete', cb)
export const onDeviceListUpdated      = (cb: (e: DeviceListResult)            => void) => on<DeviceListResult>           ('device_list_updated',     cb)

Capability Configuration (`kansas/capabilities/default.json`)

{
  "$schema": "../gen/schemas/desktop-schema.json",
  "identifier": "default",
  "description": "Default capabilities for Synesthesia",
  "windows": ["main"],
  "permissions": [
    "core:default",
    "core:window:allow-start-dragging",
    "core:window:allow-close",
    "core:window:allow-minimize",
    "core:window:allow-toggle-maximize",
    "synesthesia:allow-ping",
    "synesthesia:allow-transport-set",
    "synesthesia:allow-transport-get",
    "synesthesia:allow-model-infer",
    "synesthesia:allow-model-reload",
    "synesthesia:allow-camera-assign",
    "synesthesia:allow-param-set",
    "synesthesia:allow-haptic-send",
    "synesthesia:allow-source-set-position",
    "synesthesia:allow-room-model-set",
    "synesthesia:allow-hrtf-load",
    "synesthesia:allow-style-blend-set",
    "synesthesia:allow-device-list",
    "synesthesia:allow-get-app-state"
  ]
}

Testing

Automated

cargo check --workspace — zero errors
cargo clippy --workspace -- -D warnings — zero warnings
pnpm typecheck — zero errors

Manual

cargo tauri dev → DevTools console:

await window.__TAURI__.core.invoke('transport_set', { cmd: { bpm: 140.0 } })
// Expected rejection: { "kind": "notImplemented", "message": "T-003" }

Call every command from the catalog — each must reject with structured IpcError, not crash.
await window.__TAURI__.core.invoke('ping') → "pong" ✓

Do Not

Implement any command logic — NotImplemented only except ping
Add AppHandle state — T-003+ own Tauri managed state
Use serde_json::Value for any input except ModelInferCmd
Use #[allow(dead_code)] — fix the registration if warnings appear
Use TypeScript class — interface and type only
Use as unknown as T casts in event listeners

Iteration Protocol

PR required before closing. Human reviews before merge.

## Review Blocker — [date]
**Type affected:** [TypeName]
**Problem:** [mismatch or design flaw]
**Downstream impact:** [which T-NNN tasks already use this]
**Proposed fix:** [recommendation]

GitHub CLI

gh issue create \
  --title "T-002: IPC command registry — typed serde structs & TypeScript mirror types" \
  --label "type:task,stack:rust,stack:typescript,agent:review-required,priority:critical,status:ready,day:1" \
  --body-file T-002.md

Parent: #GENESIS
Blocks: T-003–T-005, T-010, T-013–T-014, T-016–T-029, T-031–T-041, T-046–T-052, T-062–T-090, T-096–T-102, T-121–T-135
Blocked By: T-001
Version: v0.1 · Iteration: iter-1 · Effort: M (1–2 days)