docs/HANDOFF.md

HANDOFF — stable-jam resume guide

This file exists so a fresh session can cd D:/CODE/stable-jam and pick up exactly where the previous one left off — no loss of context. Read this first, then the linked docs.

Product: Jam Buddy (consolidated here as stable-jam). "You start playing, it joins in." A call-and-response AI music companion using Stable Audio 3, built for the Stability AI Challenge at Music Hackspace Montreal (Aug 22–23 2026).

---

What this repo is (and is not)

stable-jam/ is the source + docs + tests for the Jam Buddy product. It does NOT contain the heavy vendored model repos — those stay in the previous location and are referenced by env vars (below). Don't expect to git clone and get a GPU model; the SA3 weights are a separate download.

Copied from D:/CODE/unstable-drums/ (the previous working dir, still intact if you need to recover anything). This dir was created to give the hackathon a clean, product-accurate home.

Product state (what actually works, verified this session)

• Web app (apps/web): hardware-sampler rack UI. Instrument pads (bass/lead/ rhythm/synth/drums), Genre/Mood/Tempo knobs, a "Your take" section with BOTH a MIDI file input and an audio file input, JOIN IN + PLAY BOTH buttons, status panel (aria-live) showing "Buddy tempo: N BPM", and an audio player. Every response is saved to generations/ at the repo root.
• SA3 pipeline (tools/jam_buddy.py): the CLI that does the generation. Two input modes:
  • --midi take.mid — detects tempo (exact) + response length from the take, generates a complementary melodic part at that tempo. Ignores the groove (SA3 can't parse MIDI notes).
  • --wav take.wav — TRUE audio-to-audio via init_audio + init_noise_level (default 0.4). The buddy actually HEARS the groove and responds rhythmically.
  • --bpm N / --model small-music|small-sfx / --genre / --duration.
  • Drums force small-sfx (clean isolated hits); everything else small-music.
• API route (apps/web/app/api/jambuddy/route.ts): POST /api/jambuddy with {knobs, bpm?, midi?, audio?, duration?}, shells to the python, returns the WAV + X-Jam-Buddy-BPM header. Writes output to <repo>/generations/.
• Prompt builder (apps/web/lib/jambuddy/prompt.ts): pure function building the SA3 prompt from the knobs per the official SA3 prompting guide (TrackType: Instrument, instrument, genre, mood, BPM, studio recording). Negative prompt steers away from a full mix. MODEL_FOR_INSTRUMENT maps drums→small-sfx.
• Player (apps/web/lib/jambuddy/player.ts): browser Web Audio. playTogether renders the MIDI take as oscillators and plays it with the buddy WAV on the same clock. Note: this was "banked" — the user reported PLAY BOTH only played the generated part, not the MIDI synth. It's parked, not debugged.

Verified this session (proof)

• Web: pnpm run test -> 31/31 pass (prompt, player, engine, conversation); tsc --noEmit clean; page serves the rack.
• CLI: jam_buddy.py --midi <file> -> detected 158 BPM, 24.3s response matching the take. Audio-to-audio on a 5s take -> ~10s wall, valid WAV (peak 0.43, rms 0.08). A ~378s take times out on CPU — keep audio takes short.
• The generations dir fills with timestamped WAVs.

Key environment / paths

SA3 (the engine) now lives IN this repo at stable-audio-3/. The venv's editable-install .pth was repointed to this location (was D:/CODE/unstable-drums/...). The /api/jambuddy route auto-resolves it via join(repoRoot, "stable-audio-3", ...).

Var	Value
JAM_BUDDY_ROOT	D:/CODE/stable-jam (repo root; route walks up to find tools/jam_buddy.py)
JAM_BUDDY_PYTHON	D:/CODE/stable-jam/stable-audio-3/.venv/Scripts/python.exe (the SA3 venv)
SA3 weights	cached in stable-audio-3/.venv + HF cache on G:/AI/models/huggingface
HF_TOKEN	in old repo's .env (needed for gated SA3 model access) — copy if regenerating weights

The SA3 venv was extended with mido and librosa (uv pip install --python .../stable-audio-3/.venv/Scripts/python.exe mido librosa). A fresh session must use that venv or recreate it.

How to run (fresh session)

cd D:/CODE/stable-jam
# 1. install web deps (node_modules was NOT copied)
cd apps/web && pnpm install && cd ../..

# 2. run tests + typecheck
cd apps/web && npx vitest run && npx tsc --noEmit

# 3. dev server
cd apps/web && pnpm dev   # -> http://localhost:3000

# 4. python pipeline (SA3 venv is in THIS repo)
./stable-audio-3/.venv/Scripts/python.exe tools/jam_buddy.py --midi take.mid --instrument bass --out out.wav

Docs (all in docs/)

• 01-vision.md — Jam Buddy vision (rewritten from the old PatternTalk framing).
• 02-architecture.md — note: may still describe the old service layout; the real arch is: Next.js web + tools/jam_buddy.py + SA3 venv.
• 05-accessibility.md — the a11y strategy (still relevant; the product is screenreader-compatible by design).
• 06-stable-audio-integration.md — SA3 setup, audio-to-audio, Vega notes.
• HARNESS.md — agent harness commands + test contract (may still say "PatternTalk" in places; the real product is Jam Buddy).

What's deliberately NOT in this repo (and why)

• text2midi/ (2.8G vendored model) — not part of the Jam Buddy product; it stays in the previous working dir if needed.
• The SA3 weights / .venv are inside stable-audio-3/ here but gitignored (large, regenerable, license-gated). See .gitignore.
• node_modules/, .next/ — regenerable, not committed.
• generations/*.wav — gitignored (regenerable output).
• Source song/GP files + scratch WAVs in the old tools/ — not product source.

Pending / next steps (from the last session)

1. PLAY BOTH — unbank / fix: the MIDI synth part wasn't audible. Likely the object URL or the oscillator gain. That's the "co-play" wow moment.
2. Metadata sidecar — add a <file>.json next to each generations/*.wav with prompt / negative / model / BPM / duration / source. Answers "what did it go off on."
3. Re-render the docs 02-architecture.md, 03-data-model.md, 04-ux-voice-first.md, 07-reaper-integration.md to match Jam Buddy (they still describe the old PatternTalk drum-machine in places).
4. LoRA / underfit (style trainer) is a real path but needs a GPU — not shipped.
5. Demo script — the judge's-eye review (session bg_161745) recommended a 3-min demo script; see docs/01-vision.md "What success looks like".

The judge's review (session bg_161745 — hackathon judge design review)

Key finding: docs and product were telling different stories (PatternTalk vs Jam Buddy). That's why this repo was consolidated to stable-jam. Scoring rubric + recs live in that session. The #1 fix was "kill the PatternTalk framing" — largely done in README + 01-vision; the deeper docs still need the same pass.

The SA3 tempo/time-signature reality (don't re-learn it)

Stable Audio 3 has NO tempo or time-signature conditioning channel. "BPM" is a weak semantic hint in the prompt. So "make SA3 follow a time signature" is not possible with the stock model. The answer is MIDI-first (timing from MIDI/audio detection) + SA3 as the timbre generator. Full reasoning is in the skill ref sa3-no-tempo-control.md. The LoRA style path is separate.