docs/RUNTIME_API_MUSIC.md

🏗️ Runtime API: Music (Jazz)

The Runtime API defines the interface between the Orchestration layer (Genkit) and the Music Engines (Resonance).

🎛️ Music Steering Spec

All generation functions accept a MusicSteeringSpec as their primary steering input.

1. Symbolic Music (Path A)

generate_midi(steering: MusicSteeringSpec) -> MidiBundle

• Purpose: Generates a multi-track symbolic (MIDI) representation.
• Engines: Performance RNN, MusicVAE.
• Output: MidiBundle (includes tracks, metadata, and optional DDSP URL).

2. Audio Music (Path B)

generate_audio(steering: MusicSteeringSpec, midi?: MidiBundle) -> AudioStems

• Purpose: Generates a full audio stream or multi-track audio stems.
• Engines: Magenta RT, DDSP (if midi is provided).
• Output: AudioStems (includes audio streams and control curves).

---

🧭 Mental Model Routing

• Lyria Camera: Focuses on generate_audio (Magenta RT) for reactive audio clips.
• Infinite Crate: Focuses on generate_midi (MusicVAE) for slow evolution + generate_audio (DDSP) for timbre synthesis.
• Lyria Studio: Uses both paths extensively, often starting with generate_midi for composition followed by generate_audio for stems.

🛠️ Wiring Examples

1. TypeScript Orchestrator (Genkit)

const spec: MusicSteeringSpec = {
  style: { genre: "jazz", era: "bebop" },
  emotion: { energy: 0.8, tension: 0.4, warmth: 0.6, darkness: 0.2 },
  intentions: "Generate a bebop-style jazz session with high energy.",
};

const midiBundle = await generate_midi(spec);
const audioStems = await generate_audio(spec, midiBundle);

2. C++ Native Core (Resonance)

• Receives JSON-serialized MusicSteeringSpec.
• Loads corresponding ONNX models via WinML/DirectML.
• Executes inference and returns MidiBundle or AudioStems.