docs/01-vision.md

01 — Vision

The problem

Music generators and practice tools don't act like a bandmate. You sit down, start playing, and the software asks you to describe what you want in its terms — or it produces a finished "song" you have no part in. There's no tool that just listens to what you're playing and joins in.

AI music tools compound the problem:

• They generate full mixes at a fixed tempo, not parts that lock to your tempo.
• They're visual toy apps or opaque audio generators with no semantic structure, so blind and visually impaired producers can't use them at all.

The product

Jam Buddy is a call-and-response practice companion. The interaction is the product: you start playing, it joins in at your tempo, in the instrument you pick.

1. You load a MIDI take (controller) or audio take (mic/interface).
2. The buddy detects your tempo (exact from MIDI note times; tempo-range prior on audio) and matches its response length to your take.
3. You pick the instrument the buddy should play (bass / lead / rhythm / synth / drums) plus a genre and mood.
4. SA3 generates the response — small-music for melodic parts, small-sfx for clean isolated drums — at your tempo.
5. PLAY BOTH plays your take and the buddy's response together, in tempo.
6. Every response is saved to generations/ so you can keep and A/B it.

The UI is a hardware-sampler-styled rack, and it's screenreader-compatible by design: labelled range-input knobs, accessible button names, aria-live status. Voice-first is accessibility — we build the accessible version first, not as an afterthought.

Why this fits the Stability AI challenge

The brief asks for "a publicly available and accessible tool for music producers using the power of the Stable Audio 3 models, encouraging open development and showing the strengths of local open models."

• SA3 powers the response — generated locally from the open small-music / small-sfx weights, no black-box API.
• Two real input modes: MIDI (tempo + length lock) and audio (audio-to-audio — the buddy genuinely hears and responds to your groove).
• Accessibility as load-bearing, not bolted on.
• Open + local: the whole pipeline runs on consumer hardware.

Who this is for

Primary:

• Musicians who want a practice partner that locks to their tempo.
• Producers exploring complementary parts without leaving their flow.
• Blind / visually impaired producers — the rack is fully keyboard + screenreader operable.

Secondary:

• Any jam context — one player's take, a generated counterpart in the room.

What success looks like at the hackathon

Demo (3–4 minutes):

1. Elevator pitch: "a practice partner that hears you start playing and joins in at your tempo, in your instrument."
2. Load a MIDI take from a real controller.
3. Pick an instrument, hit JOIN IN — the buddy responds at your tempo.
4. PLAY BOTH — you + the buddy together, the co-play moment.
5. Toggle the keyboard/screenreader path to show accessibility.
6. Point at the generations/ files — everything it produced is kept.

Deliverables:

• Working web app (apps/web) + the tools/jam_buddy.py SA3 pipeline.
• Open-source MIT repo.
• Screenreader-tested UI (NVDA + VoiceOver).

Stretch / not claimed:

• A LoRA fine-tune for "your style" is a real path (via underfit on a GPU) but is NOT shipped — we demo the open local model, honestly.

What we explicitly are not

• Not a DAW. Reaper / the DAW does that.
• Not a full song generator. We produce one complementary part at your tempo.
• Not a VST first. The web rack is the primary surface; plugin wrap is stretch.

North star

A musician plays a 20-second idea, picks "lead guitar," and within a minute the buddy has responded with a lead part at the same tempo — and both play together. That's the co-play moment we're building toward.