Spaces:
Paused
Rebuild CODA on Stable Audio 3 Small Music
Browse filesReplace the MusicGen multi-pass chaining with SA3's native audio
inpainting: one diffusion call continues the clip into a finished
44.1kHz stereo track, deleting the windowed passes, energy guards and
re-roll logic entirely.
- engine.py: SA3 continuation core. preload() to CPU at boot, fp16 on
GPU inside the @spaces.GPU window; inpaint mask = [source_end, total];
source clip cast to model dtype; Windows-safe seed; over-long clips
rejected so the mask can't invert.
- stitch.py: stereo-native splice. Loudness-match, short equal-power
crossfade at the seam, cos^2 closing fade, peak guard, all at 44.1k.
- verify.py: dev-only math QA — duration, silence-collapse, clipping,
seam/loudness/tempo/key continuity, spectral rolloff, stereo width,
with a dark diagnostic plot.
- app.py: dark "DAW console" UI. Total-length slider (30-120s), optional
vibe prompt, instant on-upload analysis, staged progress, output player
built visible. Over-long uploads blocked with a clear message.
- requirements.txt mirrors Stability's ZeroGPU stack: torch 2.7.1/cu128,
stable-audio-tools from upstream, pytorch_lightning; Demucs dropped.
- Retarget tests to the new modules; drop the MusicGen build (preserved
on the musicgen-fallback branch) and stale research notes.
Verified end-to-end on Blackwell: 8-step generation in ~1s, all verify
gates pass on the default 60s continuation; full UI flow tested.
- .gitignore +5 -0
- README.md +65 -47
- RESEARCH_music_continuation_2026-06-10.md +0 -163
- app.py +158 -94
- continue_music.py +0 -499
- engine.py +213 -0
- requirements.txt +26 -11
- stitch.py +93 -0
- test_app_logic.py +82 -80
- test_engine_logic.py +69 -0
- test_stitch_logic.py +100 -267
- test_verify_logic.py +96 -0
- verify.py +287 -0
|
@@ -32,3 +32,8 @@ Thumbs.db
|
|
| 32 |
*.pdf
|
| 33 |
umbra-ai-architecture.md
|
| 34 |
memory-audit-*.md
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
*.pdf
|
| 33 |
umbra-ai-architecture.md
|
| 34 |
memory-audit-*.md
|
| 35 |
+
|
| 36 |
+
# local planning / scratch / research artifacts — never ship to the public Space
|
| 37 |
+
PLAN_*.md
|
| 38 |
+
RESEARCH_*.md
|
| 39 |
+
*coda_commits*
|
|
@@ -1,7 +1,7 @@
|
|
| 1 |
---
|
| 2 |
title: CODA
|
| 3 |
emoji: 🎵
|
| 4 |
-
colorFrom:
|
| 5 |
colorTo: yellow
|
| 6 |
sdk: gradio
|
| 7 |
sdk_version: 6.16.0
|
|
@@ -11,7 +11,7 @@ pinned: false
|
|
| 11 |
license: mit
|
| 12 |
short_description: AI that finishes the song you quit on.
|
| 13 |
models:
|
| 14 |
-
-
|
| 15 |
---
|
| 16 |
|
| 17 |
# CODA — the songs you quit on, finished.
|
|
@@ -23,83 +23,101 @@ key, same tempo, same groove — then splices the new part onto your original so
|
|
| 23 |
cleanly you have to hunt for the seam.
|
| 24 |
|
| 25 |
One job, done well. No lyric bot, no cover-art printer — just a real, listenable
|
| 26 |
-
continuation of *your* clip.
|
| 27 |
|
| 28 |
## How it works
|
| 29 |
|
| 30 |
1. **Listen.** librosa detects the key, tempo, and meter — pure DSP, no ML. If
|
| 31 |
the recording is lo-fi (a voice memo, a phone capture, an old MP3 rip), CODA
|
| 32 |
cleans a *copy* first — rumble filter, spectral noise gate, level — and
|
| 33 |
-
|
| 34 |
original still goes into the final track (tick **remaster my part** to apply
|
| 35 |
the same cleanup to your section too).
|
| 36 |
-
2. **
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
to a confident level before the model sees it.** A quiet lo-fi prompt is what
|
| 50 |
-
makes MusicGen drift into silence and "ghost noise"; hand it a full-level
|
| 51 |
-
prompt and the continuation stays sustained and full. A per-pass energy guard
|
| 52 |
-
holds the level and re-rolls any pass that collapses.
|
| 53 |
-
4. **Stitch.** The continuation is loudness-matched to your recording and the
|
| 54 |
-
splice lands inside the codec-aligned overlap with an **equal-gain** crossfade
|
| 55 |
-
(equal-power would swell +3dB on correlated audio — an audible seam). The
|
| 56 |
-
track then fades out to a clean close instead of cutting off mid-phrase.
|
| 57 |
|
| 58 |
Progress streams through every stage, so you never stare at a dead screen.
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
## The stack — small on purpose
|
| 61 |
|
| 62 |
| Component | Size | Job |
|
| 63 |
|---|---|---|
|
| 64 |
-
| [
|
| 65 |
-
|
|
| 66 |
| librosa + SciPy | 0 params | key/tempo/meter detection, lo-fi cleanup |
|
| 67 |
|
| 68 |
-
Well under the 32B cap, runs entirely
|
| 69 |
-
|
| 70 |
-
back as dense or denser at half the latency — which on ZeroGPU buys more
|
| 71 |
-
generation inside the window. Small collapses to near-silence on dense lo-fi
|
| 72 |
-
prompts.
|
| 73 |
|
| 74 |
## What it won't pretend to do
|
| 75 |
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
## Using it
|
| 82 |
|
| 83 |
-
Upload a WAV, MP3, or FLAC (a
|
| 84 |
-
demo — PUSHBACK**.
|
| 85 |
-
part*, and press **Finish this song**. You'll see the detected
|
| 86 |
-
moment a clip loads, watch the stages stream, and get
|
| 87 |
-
or download.
|
| 88 |
|
| 89 |
-
Run it locally
|
|
|
|
| 90 |
|
| 91 |
```bash
|
| 92 |
pip install -r requirements.txt
|
| 93 |
python app.py
|
| 94 |
```
|
| 95 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
## Notes for the curious
|
| 97 |
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
|
| 104 |
**Demo clip credit:** `examples/pushback_demo.mp3` is a clip by the band
|
| 105 |
**PUSHBACK**, sourced from TikTok, included solely as a sample to demonstrate the
|
|
|
|
| 1 |
---
|
| 2 |
title: CODA
|
| 3 |
emoji: 🎵
|
| 4 |
+
colorFrom: indigo
|
| 5 |
colorTo: yellow
|
| 6 |
sdk: gradio
|
| 7 |
sdk_version: 6.16.0
|
|
|
|
| 11 |
license: mit
|
| 12 |
short_description: AI that finishes the song you quit on.
|
| 13 |
models:
|
| 14 |
+
- stabilityai/stable-audio-3-small-music
|
| 15 |
---
|
| 16 |
|
| 17 |
# CODA — the songs you quit on, finished.
|
|
|
|
| 23 |
cleanly you have to hunt for the seam.
|
| 24 |
|
| 25 |
One job, done well. No lyric bot, no cover-art printer — just a real, listenable
|
| 26 |
+
continuation of *your* clip, in 44.1 kHz stereo.
|
| 27 |
|
| 28 |
## How it works
|
| 29 |
|
| 30 |
1. **Listen.** librosa detects the key, tempo, and meter — pure DSP, no ML. If
|
| 31 |
the recording is lo-fi (a voice memo, a phone capture, an old MP3 rip), CODA
|
| 32 |
cleans a *copy* first — rumble filter, spectral noise gate, level — and
|
| 33 |
+
conditions on that, so it follows the *song* and not the hiss. Your real
|
| 34 |
original still goes into the final track (tick **remaster my part** to apply
|
| 35 |
the same cleanup to your section too).
|
| 36 |
+
2. **Continue.** [Stable Audio 3 Small Music](https://huggingface.co/stabilityai/stable-audio-3-small-music)
|
| 37 |
+
does the rest in a **single call**. SA3 is a latent-diffusion model with
|
| 38 |
+
native audio *inpainting*: CODA places your clip at the front of the buffer
|
| 39 |
+
and masks the region after it, and the model fills that region — conditioned
|
| 40 |
+
on the whole clip — with a coherent continuation. No 30-second windows, no
|
| 41 |
+
multi-pass chaining, no energy guards, no re-rolls. Eight diffusion steps with
|
| 42 |
+
the pingpong sampler, a couple of seconds on a GPU. Leave the vibe box empty
|
| 43 |
+
for a pure audio-led continuation that holds your key and tempo; type a vibe
|
| 44 |
+
to creatively steer the new section.
|
| 45 |
+
3. **Stitch.** The continuation is loudness-matched to your recording, joined at
|
| 46 |
+
the seam with a short equal-power crossfade, and faded out to a clean close.
|
| 47 |
+
Everything is 44.1 kHz stereo — your original is resampled up to meet the SA3
|
| 48 |
+
tail, never the other way around.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
Progress streams through every stage, so you never stare at a dead screen.
|
| 51 |
|
| 52 |
+
## Why Stable Audio 3 (and not MusicGen)
|
| 53 |
+
|
| 54 |
+
CODA's earlier builds fought MusicGen's limits: a 30 s trained window forcing
|
| 55 |
+
fragile multi-pass chaining, 32 kHz mono output, and a tendency to drift into
|
| 56 |
+
silence on quiet prompts. SA3 Small Music removes all of it at once:
|
| 57 |
+
|
| 58 |
+
| | MusicGen-medium | SA3 Small Music |
|
| 59 |
+
|---|---|---|
|
| 60 |
+
| Continuation | chain 12 s-ctx / 18 s-new passes, compounding drift | one native inpaint call over the whole clip |
|
| 61 |
+
| Output | 32 kHz **mono** | **44.1 kHz stereo** |
|
| 62 |
+
| Speed | multiple passes per track | ~8 steps, a couple of seconds |
|
| 63 |
+
| License | CC-BY-NC (non-commercial) | Stability Community (commercial < $1M revenue) |
|
| 64 |
+
|
| 65 |
+
The hardest, most fragile part of the old build — the windowed chaining with its
|
| 66 |
+
energy guards and collapse re-rolls — is simply deleted.
|
| 67 |
+
|
| 68 |
## The stack — small on purpose
|
| 69 |
|
| 70 |
| Component | Size | Job |
|
| 71 |
|---|---|---|
|
| 72 |
+
| [Stable Audio 3 Small Music](https://huggingface.co/stabilityai/stable-audio-3-small-music) | ~0.6B | native audio-inpaint continuation |
|
| 73 |
+
| T5Gemma (bundled with SA3) | ~0.5B | optional text conditioning for the vibe box |
|
| 74 |
| librosa + SciPy | 0 params | key/tempo/meter detection, lo-fi cleanup |
|
| 75 |
|
| 76 |
+
Well under the 32B cap, runs entirely inside a single ZeroGPU window (generation
|
| 77 |
+
is seconds, not the whole budget), no cloud APIs.
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
## What it won't pretend to do
|
| 80 |
|
| 81 |
+
SA3 is a *music* model — the continuation is instrumental-leaning, and CODA
|
| 82 |
+
doesn't fake vocals it can't generate. The honest design choice: **your original
|
| 83 |
+
recording plays untouched up to the seam**, vocals and all, and the generated
|
| 84 |
+
section carries the music on from there. The band plays on; you write the next
|
| 85 |
+
verse. A typed vibe steers the new section creatively but can pull it away from
|
| 86 |
+
the original's exact key and tempo — that's the trade you're choosing when you
|
| 87 |
+
use it; leave it empty for a faithful continuation.
|
| 88 |
|
| 89 |
## Using it
|
| 90 |
|
| 91 |
+
Upload a WAV, MP3, or FLAC (a 15–30 second clip works best), or hit **try the
|
| 92 |
+
demo — PUSHBACK**. Set the finished length, optionally describe a vibe or tick
|
| 93 |
+
*remaster my part*, and press **Finish this song**. You'll see the detected
|
| 94 |
+
key/tempo the moment a clip loads, watch the stages stream, and get a 44.1 kHz
|
| 95 |
+
stereo track to play or download.
|
| 96 |
|
| 97 |
+
Run it locally (needs Python 3.10 and a CUDA GPU; the SA3 weights are gated, so
|
| 98 |
+
accept the licence on the model page and `huggingface-cli login` first):
|
| 99 |
|
| 100 |
```bash
|
| 101 |
pip install -r requirements.txt
|
| 102 |
python app.py
|
| 103 |
```
|
| 104 |
|
| 105 |
+
Deploying as a Space: because the SA3 weights are gated, add an **`HF_TOKEN`**
|
| 106 |
+
secret (from an account that has accepted the licence) in the Space settings, or
|
| 107 |
+
the model download 401s at startup. The bundled demo clip lives outside git (see
|
| 108 |
+
`.gitignore`) — upload `examples/pushback_demo.mp3` to the Space directly.
|
| 109 |
+
|
| 110 |
## Notes for the curious
|
| 111 |
|
| 112 |
+
`verify.py` is a dev-only QA harness: it measures the finished track against the
|
| 113 |
+
original for silence-collapse, clipping, seam continuity, tempo/key continuity,
|
| 114 |
+
spectral rolloff, and stereo width, and writes a diagnostic plot. It is not in
|
| 115 |
+
the app's runtime path — it's how regressions get caught between ear-checks.
|
| 116 |
+
|
| 117 |
+
Code is MIT. SA3 weights are under the
|
| 118 |
+
[Stability AI Community License](https://stability.ai/license) (free for
|
| 119 |
+
commercial use under $1M annual revenue) and bundle a T5Gemma encoder under the
|
| 120 |
+
Gemma Terms of Use — worth knowing if you fork it.
|
| 121 |
|
| 122 |
**Demo clip credit:** `examples/pushback_demo.mp3` is a clip by the band
|
| 123 |
**PUSHBACK**, sourced from TikTok, included solely as a sample to demonstrate the
|
|
@@ -1,163 +0,0 @@
|
|
| 1 |
-
# CODA Music Continuation — Root-Cause Research & Recommendations
|
| 2 |
-
|
| 3 |
-
**Date:** 2026-06-10
|
| 4 |
-
**Scope:** Research only — no code changed. Sources: `transformers` source (modeling_musicgen.py, logits_process.py, feature_extraction_encodec.py), MusicGen paper (arXiv 2306.05284), audiocraft repo (model card, musicgen.py, genmodel.py, official demo notebook), facebook/musicgen-large generation_config.json, community issue threads, and June-2026 survey of alternative models.
|
| 5 |
-
|
| 6 |
-
---
|
| 7 |
-
|
| 8 |
-
## TL;DR
|
| 9 |
-
|
| 10 |
-
**The "weird funk pop" output is not a mystery — it's the predictable result of two settings interacting:**
|
| 11 |
-
|
| 12 |
-
1. **`guidance_scale=7.0` amplifies ONLY the text prompt, never the audio prompt.** Verified in the transformers source: classifier-free guidance duplicates the audio-prompt tokens into *both* the conditional and unconditional branches, so they cancel out of the CFG formula `uncond + scale * (cond - uncond)`. Only the text-encoder states are nulled in the unconditional branch. The "style adherence" slider is therefore a **text-adherence** knob — at 7 it drags generation *away* from what the audio implies and *toward* the text, at more than double Meta's trained value (3.0).
|
| 13 |
-
|
| 14 |
-
2. **The text prompt is functionally an empty caption, and MusicGen's prior for empty captions IS funk-pop stock music.** Training captions are stock-library metadata ("90s rock song with a guitar riff", genre/BPM/mood tags). A meta-instruction like *"continuation of the same song, identical instruments, identical timbre, same production and mixing, professional studio recording"* describes a *relationship to other audio*, which never appears in training captions — T5 embeds it as near-noise. The training corpus is overwhelmingly ShutterStock/Pond5 production-library music, so a vague caption regresses to generic Western stock instrumental. The per-pass section hints ("chorus, fuller arrangement, more energy") push the same direction.
|
| 15 |
-
|
| 16 |
-
So the current code cranks a 7× amplifier on a caption that means "generic stock music." That is precisely "weird funk pop bullshit."
|
| 17 |
-
|
| 18 |
-
**Also fundamental:** MusicGen **cannot continue vocals, ever.** Vocals were stripped from its training data twice (tag filtering, then HT-Demucs separation — stated verbatim in Meta's model card: "Vocals have been removed from the data source… The model is not able to generate realistic vocals"). The README's promise ("Vocals, instruments, lyrics — it picks up all of it") is not deliverable with MusicGen as the backbone.
|
| 19 |
-
|
| 20 |
-
---
|
| 21 |
-
|
| 22 |
-
## 1. Is MusicGen capable of what we want?
|
| 23 |
-
|
| 24 |
-
**Partially. Set expectations correctly:**
|
| 25 |
-
|
| 26 |
-
| Goal | Achievable? |
|
| 27 |
-
|---|---|
|
| 28 |
-
| Continue tempo, key, groove, rough genre | **Yes** — this is what the autoregressive token prefix carries |
|
| 29 |
-
| Same instrumentation, broadly ("a rock band stays a rock band") | **Mostly**, with correct settings (below) |
|
| 30 |
-
| *Identical* timbre/production ("same recording, same take") | **No.** Hard ceiling: everything passes through EnCodec at 2.2 kbps (4 codebooks × 11 bits × 50 Hz), 32 kHz mono. Real guitar/drums come back with smeared cymbals, softened transients, and the "AI sheen" before the LM even contributes |
|
| 31 |
-
| Continue vocals | **Never.** Vocal-free training data, by design |
|
| 32 |
-
|
| 33 |
-
Continuation was never a trained or paper-evaluated task — it's an emergent property of the autoregressive LM that audiocraft exposes as `generate_continuation()`. Meta's own demos prompt with only **0.5–2 seconds** of audio. The paper's limitations section concedes: "Our simple generation method does not allow us to have fine-grained control over adherence of the generation to the conditioning."
|
| 34 |
-
|
| 35 |
-
**Verdict:** with the fixes below, MusicGen-large can deliver "plausibly the same band keeps playing" for *instrumental* material. It cannot deliver "the same song with the same instruments" to a discerning ear, and it cannot touch vocals.
|
| 36 |
-
|
| 37 |
-
---
|
| 38 |
-
|
| 39 |
-
## 2. How audio conditioning actually works (verified in transformers source)
|
| 40 |
-
|
| 41 |
-
Trace through `MusicgenForConditionalGeneration.generate()` (modeling_musicgen.py):
|
| 42 |
-
|
| 43 |
-
1. `input_values` (the waveform from the processor) → `_prepare_audio_encoder_kwargs_for_generation()` → `audio_encoder.encode()` → EnCodec RVQ codes → reshaped into `decoder_input_ids` (lines ~1744–1818). The audio prompt becomes a **frozen decoder token prefix** via `build_delay_pattern_mask`; generation only fills the empty slots after it.
|
| 44 |
-
2. **The output of `generate()` includes the prompt**, re-decoded through EnCodec (lossy). There is no slicing of the prefix. The current CODA code already accounts for this correctly (crossfading inside the overlap region) — keep that.
|
| 45 |
-
3. **CFG mechanics — the smoking gun.** `_prepare_text_encoder_kwargs_for_generation` (lines ~1732–1738): the unconditional branch is built by concatenating `torch.zeros_like(last_hidden_state)` — *text states only*. `prepare_inputs_for_generation` (lines ~1626–1631): `decoder_input_ids = decoder_input_ids.repeat((2, 1))` — *audio prompt copied into both branches*. `ClassifierFreeGuidanceLogitsProcessor`: `scores = uncond + (cond - uncond) * guidance_scale`. Since the audio prefix conditions both branches identically, **`guidance_scale` is purely a text knob. There is no knob in this implementation that increases audio-prompt fidelity.** Faithfulness to the audio comes from the prefix alone, and is *diluted* by strong text guidance.
|
| 46 |
-
4. **Defaults from facebook/musicgen-large `generation_config.json`:** `guidance_scale: 3.0`, `max_length: 1500` (= 30.0 s at 50 tokens/s), `do_sample: true`.
|
| 47 |
-
5. **Window math:** total sequence = prompt tokens + `max_new_tokens`. Nothing enforces the 1500-token (30 s) trained window when `max_new_tokens` is set — sinusoidal positional embeddings silently extrapolate with degraded output past it. Current CODA math (20 s prompt + 10 s new = exactly 1500) sits at the edge; correct, no headroom.
|
| 48 |
-
6. **Resampling:** the EnCodec feature extractor does **no resampling**; it hard-errors if `sampling_rate != 32000` is passed (good — CODA passes it) and silently produces pitch-shifted garbage if omitted. CODA's librosa resample-to-32k + `sampling_rate=MUSICGEN_SR` is correct.
|
| 49 |
-
7. **Shapes/padding:** mono 1-D float32 is the canonical input (a `(1, N)` array raises). `padding=True` with a single clip adds zero padding — safe. (Batching unequal-length prompts would inject silence tokens into the conditioning — avoid, but CODA doesn't batch.)
|
| 50 |
-
8. **fp32:** correct and necessary. The precision-sensitive part is EnCodec encode/decode and the T5 conditioning; community/literature confirm fp16 there produces the degraded "bitcrushed" output — this was the 8-bit failure mode, and the current fp32 fix is right. (LM-only bf16 would be a safe future speed optimization; full-fp16 is not.)
|
| 51 |
-
|
| 52 |
-
---
|
| 53 |
-
|
| 54 |
-
## 3. What's wrong with the current `continue_music.py` — specifically
|
| 55 |
-
|
| 56 |
-
| # | Problem | Where | Severity |
|
| 57 |
-
|---|---|---|---|
|
| 58 |
-
| 1 | `guidance_scale=7.0` default — 2.3× Meta's trained value, amplifies text only, actively fights audio fidelity, and degrades audio quality (documented HF caveat) | `continue_track()` signature; `app.py` slider (min 3 / default 7 / max 10) | **Critical — primary cause of "funk pop"** |
|
| 59 |
-
| 2 | Meta-instruction text prompt ("identical instruments, identical timbre…") is out-of-distribution → behaves as a vague caption → stock-music prior | `base` string in `continue_track()` | **Critical — the other half of the same bug** |
|
| 60 |
-
| 3 | Per-pass section hints ("chorus, fuller arrangement, more energy", "big chorus, peak energy") are genre-generic captions injected at guidance 7 — each chained pass gets pulled further toward stock pop | `SECTION_ARC` / `_section_plan()` | **High** |
|
| 61 |
-
| 4 | The "style adherence" slider label is inverted from reality: raising it makes output *more generic*, not more faithful | `app.py` | High (UX/diagnosis trap) |
|
| 62 |
-
| 5 | 6 chained passes for 60 s (10 s new per pass) — each pass re-encodes generated audio and re-rolls the dice; drift compounds per pass. audiocraft's own extend loop uses `extend_stride=18` (12 s context, 18 s new) = ~3 passes for 60 s | `per_pass` math in `continue_track()` | Medium |
|
| 63 |
-
| 6 | Vocal input goes straight into the prompt. MusicGen has never seen a voice; it will drop, garble, or replace it | `_load_tail()` / pipeline design | **Critical for any vocal demo** |
|
| 64 |
-
| 7 | No timbre/EQ post-matching — generated 32 kHz-band-limited mono is spliced against a 44.1 kHz master with only an RMS gain match, so even a good continuation reads as "different record" | `stitch_with_crossfade()` | Medium |
|
| 65 |
-
|
| 66 |
-
What's already **correct** (don't touch): fp32 + TF32, librosa resample to 32 kHz with `sampling_rate=` passed, mono 1-D float32 input, crossfade inside the codec-roundtripped overlap, equal-gain seam for correlated material, resampling generated audio *up* to the original's rate, window math 20 s + 10 s = 1500 tokens.
|
| 67 |
-
|
| 68 |
-
---
|
| 69 |
-
|
| 70 |
-
## 4. The fix — exact changes
|
| 71 |
-
|
| 72 |
-
### Fix 1 (do first): guidance + text — `continue_music.py`
|
| 73 |
-
|
| 74 |
-
**Two supported modes; implement (a) as default, (b) as the option:**
|
| 75 |
-
|
| 76 |
-
**(a) Pure continuation — no text, no CFG.** This is the canonical mode: Meta's MusicGen-Style page describes continuation as "given the conditioning, MusicGen continues it **without any textual description**", and audiocraft's `generate_continuation(descriptions=None)` is the reference implementation. Training used 20% condition dropout, so unconditional is in-distribution.
|
| 77 |
-
|
| 78 |
-
```python
|
| 79 |
-
# default path: audio prompt only
|
| 80 |
-
inputs = processor(
|
| 81 |
-
audio=prompt_audio,
|
| 82 |
-
sampling_rate=MUSICGEN_SR,
|
| 83 |
-
return_tensors="pt",
|
| 84 |
-
).to(model.device)
|
| 85 |
-
# merge in null text conditioning so the seq2seq generate path is well-formed
|
| 86 |
-
uncond = model.get_unconditional_inputs(num_samples=1)
|
| 87 |
-
output = model.generate(
|
| 88 |
-
**inputs,
|
| 89 |
-
encoder_outputs=None, # see note below
|
| 90 |
-
do_sample=True,
|
| 91 |
-
guidance_scale=1.0, # CFG off — nothing for it to amplify anyway
|
| 92 |
-
temperature=1.0,
|
| 93 |
-
top_k=250, # audiocraft defaults
|
| 94 |
-
max_new_tokens=max_new_tokens,
|
| 95 |
-
)
|
| 96 |
-
```
|
| 97 |
-
(Implementation note: the clean way in transformers is `model.generate(input_values=inputs.input_values, padding_mask=inputs.padding_mask, **vars-from-get_unconditional_inputs-minus-duplicates, guidance_scale=1.0, ...)` — or simply keep passing `text=[""]` through the processor with `guidance_scale=1.0`, which is equivalent in effect since CFG is off. Verify both on a test clip; keep whichever is stable on the installed transformers version. **Pin transformers < 4.57** if you ever touch the melody variant — melody conditioning silently broke after 4.48, GitHub issue #45647.)
|
| 98 |
-
|
| 99 |
-
**(b) Described continuation — user-supplied caption, guidance 3.0.** If the user types a real description ("garage rock, distorted electric guitar, live drums, male vocals removed, 120 bpm, E minor"), that caption is in-distribution and CFG at Meta's trained value 3.0 helps hold the genre. Auto-append detected key/BPM as now. **Never auto-generate meta-instructions.**
|
| 100 |
-
|
| 101 |
-
- Change `continue_track()` default `guidance_scale=7.0` → `1.0`; treat >1 as only valid when a user caption exists.
|
| 102 |
-
- Delete the `base` meta-instruction string.
|
| 103 |
-
- **Delete `SECTION_ARC` and `_section_plan()`** (or reduce to a single final-pass "outro, song ending" hint *only* in described mode). The audio chaining carries continuity; the hints only inject stock-pop pull.
|
| 104 |
-
|
| 105 |
-
### Fix 2: UI — `app.py`
|
| 106 |
-
|
| 107 |
-
- Replace the "style adherence" slider (currently min 3, so even its minimum is text-amplifying) with:
|
| 108 |
-
- a free-text **"describe your song (optional)"** box → mode (b) when filled, mode (a) when empty;
|
| 109 |
-
- if a guidance slider is kept at all: range 1.0–5.0, default 1.0 (no text) / 3.0 (with text), labeled **"text influence"**.
|
| 110 |
-
|
| 111 |
-
### Fix 3: vocals — Demucs gate (Demucs is already in requirements.txt)
|
| 112 |
-
|
| 113 |
-
Before building the prompt: run HT-Demucs on the tail, measure vocal-stem energy. If vocals are present:
|
| 114 |
-
1. Continue **the instrumental mix only** (this matches how the training data itself was prepared — HT-Demucs-separated instrumentals are exactly in-distribution).
|
| 115 |
-
2. Keep the user's original (with vocals) untouched up to the seam; the continuation is instrumental.
|
| 116 |
-
3. Update the README honestly: lyrics from Qwen3 are a *songwriting aid* deliverable, not synthesized vocals. (No open model on a ZeroGPU budget will clone-and-continue the user's voice well — YuE-extend is the closest and it's slow and finicky.)
|
| 117 |
-
|
| 118 |
-
### Fix 4: chaining — match audiocraft's reference loop
|
| 119 |
-
|
| 120 |
-
Keep first-pass prompt at 15–20 s (community sweet spot for fidelity). For subsequent passes use **12 s of context / 18 s new** (audiocraft's `extend_stride=18` default): 60 s of new audio in 3 passes instead of 6 — half the drift opportunities, half the GPU time. Keep total tokens ≤ 1500 per pass.
|
| 121 |
-
|
| 122 |
-
### Fix 5: post-match the seam — matchering
|
| 123 |
-
|
| 124 |
-
Add [`matchering`](https://github.com/sergree/matchering) (pure-Python, lightweight): REFERENCE = the original clip, TARGET = the generated continuation, before `stitch_with_crossfade()`. It matches RMS, spectral envelope, and peak — this is the single cheapest improvement to "sounds like the same record," and it's exactly what an audio engineer would do at a session splice: match the master, then crossfade on a musical boundary. (Optional polish: use the detected BPM to snap the seam to a downbeat.) Note it cannot invent content above 16 kHz — the 32 kHz EnCodec ceiling stands.
|
| 125 |
-
|
| 126 |
-
### Expected outcome after Fixes 1–5
|
| 127 |
-
|
| 128 |
-
Tempo/key/groove continuity, broadly matching instrumentation, no genre teleport, production glued by EQ-match. Still mono-derived, still slightly "AI-sheened," still instrumental-only. That is MusicGen's ceiling.
|
| 129 |
-
|
| 130 |
-
---
|
| 131 |
-
|
| 132 |
-
## 5. If the ceiling isn't good enough: the 2026 alternatives
|
| 133 |
-
|
| 134 |
-
Ranked for CODA's use case (faithful extension of arbitrary user audio, single ZeroGPU slice):
|
| 135 |
-
|
| 136 |
-
1. **ACE-Step 1.5** (Jan 2026; Apache/MIT-style, 8–16 GB VRAM) — the only open model where "**extend** my actual audio, keep timbre" is a designed first-class feature (`extend`, `repaint`, `audio_cover_strength`), **with vocal support in 50+ languages**. Best single-model replacement candidate; chain repaints for arbitrary length. Caveat: limited independent benchmarking yet.
|
| 137 |
-
2. **Stable Audio 3 Medium** (May 2026; 1.4B, ~6 GB, Stability Community License) — documented causal continuation/inpainting mode, 44.1 kHz **stereo** (fixes the mono+bandwidth ceiling). Unproven on vocals. Worth an A/B against ACE-Step on instrumental material.
|
| 138 |
-
3. **YuE-extend** — true continuation with voice cloning from separated stems; best vocal continuity in open weights, but slow (minutes/song) and ≤30 s prompt practical limit.
|
| 139 |
-
4. **MusicGen-Style** (facebook/musicgen-style) — conditions on a 1.5–4.5 s excerpt for timbre/style match; combining it with a continuation prefix is theoretically possible in audiocraft but undocumented/unvalidated. Research-grade only.
|
| 140 |
-
5. **Per-stem MusicGen continuation: don't.** Isolated stems are out-of-distribution for MusicGen (literature: models "produce nonsensical outputs when fed isolated vocals" — SingSong); only stem-native models (MusicGen-Stem, StemGen) do this, and their weights are research releases.
|
| 141 |
-
|
| 142 |
-
**License flag:** MusicGen weights are **CC-BY-NC** (non-commercial). Fine for the hackathon; a problem if CODA ever ships commercially. ACE-Step and Stable Audio 3 (under $1M revenue) are both more permissive.
|
| 143 |
-
|
| 144 |
-
**Recommended path:** ship the hackathon on MusicGen with Fixes 1–5 (small, low-risk diffs), and prototype ACE-Step 1.5 `extend` as CODA v2's backbone — it is designed to do exactly what CODA promises, including vocals.
|
| 145 |
-
|
| 146 |
-
---
|
| 147 |
-
|
| 148 |
-
## 6. Failure modes — explained
|
| 149 |
-
|
| 150 |
-
| Symptom | Cause | Status |
|
| 151 |
-
|---|---|---|
|
| 152 |
-
| 8-bit / chiptune output | fp16 EnCodec/T5 path (precision-sensitive); confirmed pattern in community + literature (TinyMusician keeps EnCodec fp32 for this reason) | **Fixed** by the fp32 change — keep it |
|
| 153 |
-
| Random new instruments | Text guidance pulling toward caption prior + per-pass re-rolls in 6-pass chaining | Fixes 1, 4 |
|
| 154 |
-
| Generic MusicGen-default sound | Vague/meta caption → stock-library prior (training data literally is stock music) | Fixes 1, 2 |
|
| 155 |
-
| "Weird funk pop" | guidance 7 × meaningless caption × "more energy" section hints = maximum pull toward the stock-pop prior | Fixes 1, 2 (delete SECTION_ARC) |
|
| 156 |
-
| ~14 s output | Deployment issue, already fixed | — |
|
| 157 |
-
| Vocals garbled/vanish | Vocal-free training data — model-level impossibility | Fix 3 (Demucs gate) |
|
| 158 |
-
|
| 159 |
-
---
|
| 160 |
-
|
| 161 |
-
## 7. What a professional audio engineer would say
|
| 162 |
-
|
| 163 |
-
A session engineer asked to "extend this demo" would: comp from the same takes, or re-amp/re-track matching the original chain, then **match the master** (EQ/loudness/width to the reference) and **edit on the grid** (splice at a downbeat, short crossfade). The transferable lessons for CODA: (1) never expect a different "performer" (the model) to nail the take — minimize how much it has to invent (longer audio context, no contradicting text); (2) the *master-matching* step is half of what makes a splice invisible — that's the matchering pass; (3) cut on musical boundaries — use the BPM grid CODA already detects to place the seam on a downbeat; (4) when a part can't be reproduced (vocals), you don't fake it — you arrange around it (instrumental continuation).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -1,13 +1,13 @@
|
|
| 1 |
"""CODA — finish the song you quit on.
|
| 2 |
|
| 3 |
-
Upload a short, unfinished music clip
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
level-matched crossfade and a clean closing fade.
|
| 8 |
|
| 9 |
-
Deliberately one job, done well. No lyrics generator, no cover-art printer —
|
| 10 |
-
|
| 11 |
"""
|
| 12 |
import os
|
| 13 |
import tempfile
|
|
@@ -17,12 +17,13 @@ import librosa
|
|
| 17 |
import numpy as np
|
| 18 |
import soundfile as sf
|
| 19 |
|
|
|
|
|
|
|
| 20 |
from analyze import fingerprint
|
| 21 |
from enhance import enhance_audio, enhance_to_tempfile, input_quality
|
| 22 |
|
| 23 |
# ZeroGPU shim: on a Space `spaces` exists and `@spaces.GPU` attaches a GPU for
|
| 24 |
-
# the duration of the call. Locally
|
| 25 |
-
# app still runs.
|
| 26 |
try:
|
| 27 |
import spaces
|
| 28 |
except ImportError:
|
|
@@ -31,27 +32,23 @@ except ImportError:
|
|
| 31 |
return fn if fn else (lambda f: f)
|
| 32 |
spaces = _FakeSpaces()
|
| 33 |
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
# the bundled demo: a low-quality phone capture of an unfinished song by the
|
| 37 |
-
# band PUSHBACK (shared via TikTok). lo-fi in, finished-sounding out.
|
| 38 |
PUSHBACK_DEMO = os.path.join(os.path.dirname(__file__),
|
| 39 |
"examples", "pushback_demo.mp3")
|
| 40 |
PUSHBACK_CREDIT = ("Demo clip: **PUSHBACK** (via TikTok), used with thanks. "
|
| 41 |
"Bring your own clip to finish your own song.")
|
| 42 |
|
| 43 |
-
#
|
| 44 |
-
#
|
| 45 |
-
|
| 46 |
-
MIN_ADD, MAX_ADD, DEFAULT_ADD = 12, 40, 30
|
| 47 |
|
| 48 |
-
# On a Space, pull
|
| 49 |
-
#
|
| 50 |
-
#
|
| 51 |
if os.environ.get("SPACE_ID"):
|
| 52 |
try:
|
| 53 |
-
|
| 54 |
-
_preload()
|
| 55 |
except Exception as _e:
|
| 56 |
print(f"[coda] preload failed ({_e}); will lazy-load", flush=True)
|
| 57 |
|
|
@@ -59,30 +56,37 @@ if os.environ.get("SPACE_ID"):
|
|
| 59 |
def _fmt_info(info, quality):
|
| 60 |
"""human-readable summary of what CODA heard."""
|
| 61 |
lines = [
|
| 62 |
-
f"**
|
| 63 |
-
f"**
|
| 64 |
-
f"**
|
| 65 |
-
f"**
|
| 66 |
]
|
| 67 |
if quality and quality.get("lofi"):
|
| 68 |
lines.append(
|
| 69 |
-
f"**
|
| 70 |
-
f"
|
| 71 |
-
f"
|
| 72 |
return " \n".join(lines)
|
| 73 |
|
| 74 |
|
| 75 |
def analyze_on_upload(audio_path):
|
| 76 |
-
"""Fast
|
| 77 |
-
|
| 78 |
if not audio_path:
|
| 79 |
return gr.update(value="", visible=False), gr.update(interactive=False)
|
| 80 |
try:
|
| 81 |
-
# listen to a cleaned copy — analysis should track the music, not the
|
| 82 |
-
# recording's noise floor
|
| 83 |
listen_path = enhance_to_tempfile(audio_path)
|
| 84 |
info = fingerprint(listen_path)
|
| 85 |
quality = input_quality(audio_path)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
md = "### CODA heard\n" + _fmt_info(info, quality)
|
| 87 |
return gr.update(value=md, visible=True), gr.update(interactive=True)
|
| 88 |
except Exception as e:
|
|
@@ -92,61 +96,62 @@ def analyze_on_upload(audio_path):
|
|
| 92 |
|
| 93 |
|
| 94 |
@spaces.GPU(duration=240)
|
| 95 |
-
def finish_song(audio_path,
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
if not audio_path:
|
| 100 |
raise gr.Error("Upload a clip (or load the PUSHBACK demo) first.")
|
| 101 |
|
| 102 |
-
|
| 103 |
progress(0.05, desc="Listening to your clip…")
|
| 104 |
|
| 105 |
-
# analyze + build the
|
| 106 |
listen_path = enhance_to_tempfile(audio_path)
|
| 107 |
info = fingerprint(listen_path)
|
| 108 |
-
quality = input_quality(audio_path)
|
| 109 |
|
| 110 |
-
# the original
|
| 111 |
-
# listener actually hears for the first stretch of the finished track
|
| 112 |
original, sr = librosa.load(audio_path, sr=None, mono=False)
|
| 113 |
if remaster:
|
| 114 |
progress(0.12, desc="Remastering your part…")
|
| 115 |
original = enhance_audio(original, sr)
|
| 116 |
|
| 117 |
-
#
|
| 118 |
-
def
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
|
|
|
|
|
|
|
|
|
| 135 |
|
| 136 |
out_path = os.path.join(tempfile.mkdtemp(), "coda_finished.wav")
|
| 137 |
-
sf.write(out_path, out.T
|
| 138 |
|
| 139 |
progress(1.0, desc="Done.")
|
| 140 |
-
total = out.shape[-1] /
|
|
|
|
|
|
|
| 141 |
summary = (
|
| 142 |
f"### Finished — {total:.0f}s\n"
|
| 143 |
-
f"Your **{
|
| 144 |
-
f"{info['bpm']} BPM
|
| 145 |
-
f"
|
| 146 |
-
f"
|
| 147 |
-
f"
|
| 148 |
-
f"instrumental music, so CODA strips the vocals before it continues "
|
| 149 |
-
f"and lets your original vocals carry the front of the track.*"
|
| 150 |
)
|
| 151 |
return out_path, summary
|
| 152 |
|
|
@@ -156,31 +161,85 @@ def load_demo():
|
|
| 156 |
return PUSHBACK_DEMO
|
| 157 |
|
| 158 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
CSS = """
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
"""
|
| 165 |
|
|
|
|
|
|
|
| 166 |
with gr.Blocks(title="CODA") as app:
|
| 167 |
-
gr.
|
| 168 |
-
|
|
|
|
|
|
|
|
|
|
| 169 |
gr.Markdown(
|
| 170 |
-
"Upload a short, unfinished music clip. CODA
|
| 171 |
-
"groove, then continues it into a
|
| 172 |
-
"same feel —
|
| 173 |
-
|
| 174 |
|
| 175 |
-
with gr.Row():
|
| 176 |
with gr.Column(scale=1):
|
| 177 |
audio_input = gr.Audio(
|
| 178 |
label="Your unfinished clip", type="filepath", sources=["upload"])
|
| 179 |
demo_btn = gr.Button("🎧 Try the demo — PUSHBACK (via TikTok)",
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
label="
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 184 |
remaster = gr.Checkbox(
|
| 185 |
value=False, label="Remaster my part too",
|
| 186 |
info="Apply the same lo-fi cleanup to your original section so "
|
|
@@ -190,15 +249,18 @@ with gr.Blocks(title="CODA") as app:
|
|
| 190 |
|
| 191 |
with gr.Column(scale=1):
|
| 192 |
info_md = gr.Markdown(visible=False, elem_classes="coda-card")
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
|
|
|
|
|
|
|
|
|
|
| 202 |
demo_btn.click(fn=load_demo, inputs=[], outputs=[audio_input])
|
| 203 |
|
| 204 |
def _show_summary():
|
|
@@ -206,10 +268,12 @@ with gr.Blocks(title="CODA") as app:
|
|
| 206 |
|
| 207 |
finish_btn.click(
|
| 208 |
fn=finish_song,
|
| 209 |
-
inputs=[audio_input,
|
| 210 |
outputs=[output_audio, summary_md]).then(
|
| 211 |
fn=_show_summary, inputs=[], outputs=[summary_md])
|
| 212 |
|
| 213 |
|
| 214 |
if __name__ == "__main__":
|
| 215 |
-
|
|
|
|
|
|
|
|
|
| 1 |
"""CODA — finish the song you quit on.
|
| 2 |
|
| 3 |
+
Upload a short, unfinished music clip. CODA listens to it (key, tempo, meter),
|
| 4 |
+
then continues it into a longer, finished-sounding track in the same feel using
|
| 5 |
+
Stable Audio 3 Small Music — a single native audio-continuation call, 44.1 kHz
|
| 6 |
+
stereo — and splices the new part seamlessly onto your pristine original with a
|
| 7 |
level-matched crossfade and a clean closing fade.
|
| 8 |
|
| 9 |
+
Deliberately one job, done well. No lyrics generator, no cover-art printer — just
|
| 10 |
+
a real, listenable continuation of your clip.
|
| 11 |
"""
|
| 12 |
import os
|
| 13 |
import tempfile
|
|
|
|
| 17 |
import numpy as np
|
| 18 |
import soundfile as sf
|
| 19 |
|
| 20 |
+
import engine
|
| 21 |
+
import stitch
|
| 22 |
from analyze import fingerprint
|
| 23 |
from enhance import enhance_audio, enhance_to_tempfile, input_quality
|
| 24 |
|
| 25 |
# ZeroGPU shim: on a Space `spaces` exists and `@spaces.GPU` attaches a GPU for
|
| 26 |
+
# the duration of the call. Locally we no-op so the app still runs.
|
|
|
|
| 27 |
try:
|
| 28 |
import spaces
|
| 29 |
except ImportError:
|
|
|
|
| 32 |
return fn if fn else (lambda f: f)
|
| 33 |
spaces = _FakeSpaces()
|
| 34 |
|
| 35 |
+
# bundled demo: a low-quality phone capture of an unfinished song by the band
|
| 36 |
+
# PUSHBACK (shared via TikTok). lo-fi in, finished-sounding out.
|
|
|
|
|
|
|
| 37 |
PUSHBACK_DEMO = os.path.join(os.path.dirname(__file__),
|
| 38 |
"examples", "pushback_demo.mp3")
|
| 39 |
PUSHBACK_CREDIT = ("Demo clip: **PUSHBACK** (via TikTok), used with thanks. "
|
| 40 |
"Bring your own clip to finish your own song.")
|
| 41 |
|
| 42 |
+
# total finished length. SA3 Small generates up to 120 s in one call, so this is
|
| 43 |
+
# a *total length* control (clip + continuation), not a "seconds to add" knob.
|
| 44 |
+
MIN_TOTAL, MAX_TOTAL, DEFAULT_TOTAL = 30, 120, 60
|
|
|
|
| 45 |
|
| 46 |
+
# On a Space, pull weights into CPU RAM at boot so the GPU window is spent
|
| 47 |
+
# generating, not reading 3 GB off disk. `spaces` defers CUDA placement until
|
| 48 |
+
# the first @spaces.GPU call.
|
| 49 |
if os.environ.get("SPACE_ID"):
|
| 50 |
try:
|
| 51 |
+
engine.preload()
|
|
|
|
| 52 |
except Exception as _e:
|
| 53 |
print(f"[coda] preload failed ({_e}); will lazy-load", flush=True)
|
| 54 |
|
|
|
|
| 56 |
def _fmt_info(info, quality):
|
| 57 |
"""human-readable summary of what CODA heard."""
|
| 58 |
lines = [
|
| 59 |
+
f"**KEY** `{info['key']}`",
|
| 60 |
+
f"**TEMPO** `{info['bpm']} BPM`",
|
| 61 |
+
f"**METER** `{info['time_signature']}`",
|
| 62 |
+
f"**CLIP** `{info['duration']}s`",
|
| 63 |
]
|
| 64 |
if quality and quality.get("lofi"):
|
| 65 |
lines.append(
|
| 66 |
+
f"**SOURCE** `lo-fi ~{quality['bandwidth_hz']/1000:.0f}kHz` "
|
| 67 |
+
f"— CODA cleans a copy before it listens, so it follows the *song*, "
|
| 68 |
+
f"not the hiss")
|
| 69 |
return " \n".join(lines)
|
| 70 |
|
| 71 |
|
| 72 |
def analyze_on_upload(audio_path):
|
| 73 |
+
"""Fast CPU-only pass the instant a clip loads, so the user sees what CODA
|
| 74 |
+
heard immediately instead of a dead screen."""
|
| 75 |
if not audio_path:
|
| 76 |
return gr.update(value="", visible=False), gr.update(interactive=False)
|
| 77 |
try:
|
|
|
|
|
|
|
| 78 |
listen_path = enhance_to_tempfile(audio_path)
|
| 79 |
info = fingerprint(listen_path)
|
| 80 |
quality = input_quality(audio_path)
|
| 81 |
+
# SA3 continues up to a 120s total, so a clip needs headroom for at
|
| 82 |
+
# least MIN_NEW seconds of new audio. Block over-long clips here, with a
|
| 83 |
+
# clear message, instead of failing at generation time.
|
| 84 |
+
if info["duration"] > engine.MAX_SOURCE_SECONDS:
|
| 85 |
+
msg = (f"### Clip too long\nThat clip is **{info['duration']:.0f}s**. "
|
| 86 |
+
f"CODA continues clips up to **{engine.MAX_SOURCE_SECONDS:.0f}s** "
|
| 87 |
+
f"(Stable Audio 3's {engine.MAX_TOTAL_SECONDS:.0f}s total cap). "
|
| 88 |
+
f"Trim it shorter and re-upload.")
|
| 89 |
+
return gr.update(value=msg, visible=True), gr.update(interactive=False)
|
| 90 |
md = "### CODA heard\n" + _fmt_info(info, quality)
|
| 91 |
return gr.update(value=md, visible=True), gr.update(interactive=True)
|
| 92 |
except Exception as e:
|
|
|
|
| 96 |
|
| 97 |
|
| 98 |
@spaces.GPU(duration=240)
|
| 99 |
+
def finish_song(audio_path, total_seconds, vibe, remaster,
|
| 100 |
+
progress=gr.Progress()):
|
| 101 |
+
"""The whole job inside one GPU window: listen -> continue (SA3) -> splice.
|
| 102 |
+
Returns (output_wav_path, summary_markdown)."""
|
| 103 |
if not audio_path:
|
| 104 |
raise gr.Error("Upload a clip (or load the PUSHBACK demo) first.")
|
| 105 |
|
| 106 |
+
total_seconds = int(total_seconds)
|
| 107 |
progress(0.05, desc="Listening to your clip…")
|
| 108 |
|
| 109 |
+
# analyze + build the conditioning feed from a cleaned copy
|
| 110 |
listen_path = enhance_to_tempfile(audio_path)
|
| 111 |
info = fingerprint(listen_path)
|
|
|
|
| 112 |
|
| 113 |
+
# the pristine original — what the listener hears for the first stretch
|
|
|
|
| 114 |
original, sr = librosa.load(audio_path, sr=None, mono=False)
|
| 115 |
if remaster:
|
| 116 |
progress(0.12, desc="Remastering your part…")
|
| 117 |
original = enhance_audio(original, sr)
|
| 118 |
|
| 119 |
+
# SA3 is a single call; map its stages onto the bar
|
| 120 |
+
def _on_stage(stage):
|
| 121 |
+
marks = {
|
| 122 |
+
"reading": (0.22, "Reading key, tempo & groove…"),
|
| 123 |
+
"composing": (0.45, "Composing the continuation…"),
|
| 124 |
+
"finalizing": (0.82, "Rendering 44.1 kHz stereo…"),
|
| 125 |
+
}
|
| 126 |
+
if stage in marks:
|
| 127 |
+
frac, desc = marks[stage]
|
| 128 |
+
progress(frac, desc=desc)
|
| 129 |
+
|
| 130 |
+
try:
|
| 131 |
+
new_tail, source_seconds, SR = engine.continue_audio(
|
| 132 |
+
listen_path, total_seconds=total_seconds,
|
| 133 |
+
prompt=(vibe or "").strip(), progress=_on_stage)
|
| 134 |
+
except ValueError as e:
|
| 135 |
+
# e.g. the clip is too long to continue under the 120s cap
|
| 136 |
+
raise gr.Error(str(e))
|
| 137 |
+
|
| 138 |
+
progress(0.9, desc="Splicing onto your original…")
|
| 139 |
+
out = stitch.stitch(original, sr, new_tail, source_seconds)
|
| 140 |
|
| 141 |
out_path = os.path.join(tempfile.mkdtemp(), "coda_finished.wav")
|
| 142 |
+
sf.write(out_path, out.T, SR, subtype="PCM_16")
|
| 143 |
|
| 144 |
progress(1.0, desc="Done.")
|
| 145 |
+
total = out.shape[-1] / SR
|
| 146 |
+
added = total - source_seconds
|
| 147 |
+
vibe_note = f" guided by *“{vibe.strip()}”*" if (vibe or "").strip() else ""
|
| 148 |
summary = (
|
| 149 |
f"### Finished — {total:.0f}s\n"
|
| 150 |
+
f"Your **{source_seconds:.0f}s** clip in **{info['key']}** at "
|
| 151 |
+
f"**{info['bpm']} BPM** continued for **~{added:.0f}s** more{vibe_note}, "
|
| 152 |
+
f"then crossfaded onto your original and faded to a clean close.\n\n"
|
| 153 |
+
f"*Stable Audio 3 generated the continuation as 44.1 kHz stereo in a "
|
| 154 |
+
f"single pass; your original recording plays untouched up to the seam.*"
|
|
|
|
|
|
|
| 155 |
)
|
| 156 |
return out_path, summary
|
| 157 |
|
|
|
|
| 161 |
return PUSHBACK_DEMO
|
| 162 |
|
| 163 |
|
| 164 |
+
# --- dark "DAW console" theme -------------------------------------------------
|
| 165 |
+
THEME = gr.themes.Base(
|
| 166 |
+
primary_hue=gr.themes.colors.cyan,
|
| 167 |
+
secondary_hue=gr.themes.colors.orange,
|
| 168 |
+
neutral_hue=gr.themes.colors.slate,
|
| 169 |
+
font=[gr.themes.GoogleFont("Inter"), "ui-sans-serif", "system-ui"],
|
| 170 |
+
font_mono=[gr.themes.GoogleFont("JetBrains Mono"), "ui-monospace", "monospace"],
|
| 171 |
+
).set(
|
| 172 |
+
body_background_fill="#0b0d10",
|
| 173 |
+
body_background_fill_dark="#0b0d10",
|
| 174 |
+
background_fill_primary="#14171c",
|
| 175 |
+
background_fill_secondary="#14171c",
|
| 176 |
+
block_background_fill="#14171c",
|
| 177 |
+
block_border_color="#222831",
|
| 178 |
+
block_label_background_fill="#14171c",
|
| 179 |
+
block_title_text_color="#9aa4b2",
|
| 180 |
+
body_text_color="#e6e9ef",
|
| 181 |
+
body_text_color_subdued="#9aa4b2",
|
| 182 |
+
button_primary_background_fill="#39d0d8",
|
| 183 |
+
button_primary_background_fill_hover="#4fe0e8",
|
| 184 |
+
button_primary_text_color="#06181a",
|
| 185 |
+
color_accent_soft="#1a2026",
|
| 186 |
+
border_color_accent="#39d0d8",
|
| 187 |
+
input_background_fill="#0e1115",
|
| 188 |
+
slider_color="#ffb347",
|
| 189 |
+
)
|
| 190 |
+
|
| 191 |
CSS = """
|
| 192 |
+
:root { --coda-accent:#39d0d8; --coda-amber:#ffb347; }
|
| 193 |
+
.gradio-container { max-width: 1060px !important; margin: 0 auto !important; }
|
| 194 |
+
#coda-head { text-align:center; padding: 8px 0 2px; }
|
| 195 |
+
#coda-title { font-weight:800; letter-spacing:.5px; font-size:2.5rem; margin:0;
|
| 196 |
+
background:linear-gradient(90deg,#39d0d8,#ffb347);
|
| 197 |
+
-webkit-background-clip:text; background-clip:text; -webkit-text-fill-color:transparent; }
|
| 198 |
+
#coda-tagline { color:#9aa4b2; margin:.1rem 0 0; font-size:1.02rem; letter-spacing:.3px; }
|
| 199 |
+
#coda-rule { height:2px; border:0; margin:10px auto 16px; max-width:240px;
|
| 200 |
+
background:linear-gradient(90deg,transparent,#39d0d8,#ffb347,transparent); opacity:.8; }
|
| 201 |
+
#coda-eq { display:flex; gap:4px; justify-content:center; align-items:flex-end; height:26px; margin-top:8px; }
|
| 202 |
+
#coda-eq span { width:5px; background:linear-gradient(180deg,#39d0d8,#1b6e72); border-radius:2px;
|
| 203 |
+
animation: codaeq 1.1s ease-in-out infinite; }
|
| 204 |
+
#coda-eq span:nth-child(2){animation-delay:.15s} #coda-eq span:nth-child(3){animation-delay:.30s}
|
| 205 |
+
#coda-eq span:nth-child(4){animation-delay:.45s} #coda-eq span:nth-child(5){animation-delay:.6s}
|
| 206 |
+
#coda-eq span:nth-child(6){animation-delay:.3s} #coda-eq span:nth-child(7){animation-delay:.1s}
|
| 207 |
+
@keyframes codaeq { 0%,100%{height:7px;opacity:.55} 50%{height:24px;opacity:1} }
|
| 208 |
+
.coda-card { border:1px solid #222831; border-radius:14px; padding:16px 18px;
|
| 209 |
+
background:#14171c; box-shadow: inset 0 1px 0 #1c222b; }
|
| 210 |
+
.coda-card h3 { color:var(--coda-accent); text-transform:uppercase; letter-spacing:1.5px;
|
| 211 |
+
font-size:.8rem; margin:.1rem 0 .7rem; }
|
| 212 |
+
#coda-foot { text-align:center; color:#6b7480; font-size:.85rem; margin-top:14px; }
|
| 213 |
"""
|
| 214 |
|
| 215 |
+
EQ_BARS = "<div id='coda-eq'>" + "".join("<span></span>" for _ in range(7)) + "</div>"
|
| 216 |
+
|
| 217 |
with gr.Blocks(title="CODA") as app:
|
| 218 |
+
with gr.Column(elem_id="coda-head"):
|
| 219 |
+
gr.HTML("<h1 id='coda-title'>🎵 CODA</h1>"
|
| 220 |
+
"<p id='coda-tagline'>the songs you quit on, finished.</p>"
|
| 221 |
+
+ EQ_BARS)
|
| 222 |
+
gr.HTML("<hr id='coda-rule'/>")
|
| 223 |
gr.Markdown(
|
| 224 |
+
"Upload a short, unfinished music clip. CODA reads its key, tempo and "
|
| 225 |
+
"groove, then **Stable Audio 3** continues it into a finished-sounding "
|
| 226 |
+
"track in the same feel — 44.1 kHz stereo, spliced seamlessly onto your "
|
| 227 |
+
"original.")
|
| 228 |
|
| 229 |
+
with gr.Row(equal_height=False):
|
| 230 |
with gr.Column(scale=1):
|
| 231 |
audio_input = gr.Audio(
|
| 232 |
label="Your unfinished clip", type="filepath", sources=["upload"])
|
| 233 |
demo_btn = gr.Button("🎧 Try the demo — PUSHBACK (via TikTok)",
|
| 234 |
+
size="sm")
|
| 235 |
+
total_slider = gr.Slider(
|
| 236 |
+
MIN_TOTAL, MAX_TOTAL, value=DEFAULT_TOTAL, step=1,
|
| 237 |
+
label="Finished length (seconds)",
|
| 238 |
+
info="Total length of the finished track. Longer = a bit slower.")
|
| 239 |
+
vibe = gr.Textbox(
|
| 240 |
+
label="Describe the vibe (optional)", lines=1,
|
| 241 |
+
placeholder="e.g. warm lo-fi, vinyl crackle, mellow piano",
|
| 242 |
+
info="Leave empty for pure audio-led continuation.")
|
| 243 |
remaster = gr.Checkbox(
|
| 244 |
value=False, label="Remaster my part too",
|
| 245 |
info="Apply the same lo-fi cleanup to your original section so "
|
|
|
|
| 249 |
|
| 250 |
with gr.Column(scale=1):
|
| 251 |
info_md = gr.Markdown(visible=False, elem_classes="coda-card")
|
| 252 |
+
# built visible inside the result group: Gradio drops visible=False
|
| 253 |
+
# audio players at build time, so we never toggle player visibility.
|
| 254 |
+
with gr.Group(elem_classes="coda-card"):
|
| 255 |
+
gr.Markdown("### Your finished song")
|
| 256 |
+
output_audio = gr.Audio(label="", type="filepath",
|
| 257 |
+
interactive=False)
|
| 258 |
+
summary_md = gr.Markdown(visible=False)
|
| 259 |
+
|
| 260 |
+
gr.Markdown(PUSHBACK_CREDIT, elem_id="coda-foot")
|
| 261 |
+
|
| 262 |
+
audio_input.change(fn=analyze_on_upload, inputs=[audio_input],
|
| 263 |
+
outputs=[info_md, finish_btn])
|
| 264 |
demo_btn.click(fn=load_demo, inputs=[], outputs=[audio_input])
|
| 265 |
|
| 266 |
def _show_summary():
|
|
|
|
| 268 |
|
| 269 |
finish_btn.click(
|
| 270 |
fn=finish_song,
|
| 271 |
+
inputs=[audio_input, total_slider, vibe, remaster],
|
| 272 |
outputs=[output_audio, summary_md]).then(
|
| 273 |
fn=_show_summary, inputs=[], outputs=[summary_md])
|
| 274 |
|
| 275 |
|
| 276 |
if __name__ == "__main__":
|
| 277 |
+
# Gradio 6 moved theme/css to launch(); pass them here (and they remain on
|
| 278 |
+
# Blocks above) so the dark DAW theme applies however the Space serves it.
|
| 279 |
+
app.queue().launch(theme=THEME, css=CSS)
|
|
@@ -1,499 +0,0 @@
|
|
| 1 |
-
import math
|
| 2 |
-
|
| 3 |
-
import librosa
|
| 4 |
-
import numpy as np
|
| 5 |
-
import torch
|
| 6 |
-
from transformers import AutoProcessor, MusicgenForConditionalGeneration
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
# transformers-native MusicGen. the audiocraft package is abandoned and
|
| 10 |
-
# hard-pins torch==2.1.0 / xformers<0.0.23, which breaks the HF Space build
|
| 11 |
-
# against gradio 6.x. transformers supports audio-prompted continuation
|
| 12 |
-
# directly, so we don't need audiocraft at all.
|
| 13 |
-
|
| 14 |
-
# medium, not large: a/b rolls on the same prompts showed equal-or-denser
|
| 15 |
-
# continuations from medium at half the latency and a third of the ram —
|
| 16 |
-
# which on zerogpu means more passes inside the 240s window. (small is not
|
| 17 |
-
# usable: it collapses to near-silence on dense lo-fi prompts.)
|
| 18 |
-
MODEL_ID = "facebook/musicgen-medium"
|
| 19 |
-
MUSICGEN_SR = 32000
|
| 20 |
-
FRAME_RATE = 50 # musicgen decoder tokens per second
|
| 21 |
-
# musicgen was trained on 30s windows; prompt + new audio per pass must fit.
|
| 22 |
-
# this is a hard model limit — the audio prompt can never exceed ~30s.
|
| 23 |
-
MAX_WINDOW_SECONDS = 30
|
| 24 |
-
# audio context / new-audio split per pass, matching audiocraft's reference
|
| 25 |
-
# extend loop (extend_stride=18): 12s of context + 18s new = exactly the
|
| 26 |
-
# 1500-token trained window. fewer, longer passes = fewer EnCodec re-encodes
|
| 27 |
-
# and re-rolls, so less drift across a long extension.
|
| 28 |
-
CONTEXT_SECONDS = 12
|
| 29 |
-
NEW_SECONDS_PER_PASS = 18
|
| 30 |
-
# the FIRST pass listens to more of the user's clip — community sweet spot
|
| 31 |
-
# for style fidelity is 15-20s — and generates correspondingly less, so the
|
| 32 |
-
# window stays at the trained 30s. later passes re-prompt on model audio,
|
| 33 |
-
# where 12s of context is enough to hold the groove.
|
| 34 |
-
DEFAULT_PROMPT_SECONDS = 20
|
| 35 |
-
# each generated chunk must stay near the prompt's level. unconditional
|
| 36 |
-
# continuation drifts — some rolls decay toward silence, some blow up — and
|
| 37 |
-
# chained passes compound whatever they inherit. the guard rescales a chunk
|
| 38 |
-
# whose rms ratio leaves this band back to the boundary, preserving musical
|
| 39 |
-
# dynamics inside it.
|
| 40 |
-
ENERGY_BAND = (0.7, 1.4)
|
| 41 |
-
# a chunk this far below the prompt level is a failed roll (near-silence /
|
| 42 |
-
# wisps), not a quiet section — regenerate it once rather than amplify mush
|
| 43 |
-
COLLAPSE_RATIO = 0.30
|
| 44 |
-
MAX_RETRIES_PER_TRACK = 2
|
| 45 |
-
# the single biggest quality lever. musicgen was trained on commercial-loudness
|
| 46 |
-
# audio; prompt it with a quiet lo-fi clip (rms ~0.06) and it loses the thread
|
| 47 |
-
# and drifts toward silence within a few seconds — the "ghost noise / wisps"
|
| 48 |
-
# failure. RMS-normalizing every prompt to a confident level before the model
|
| 49 |
-
# sees it keeps the continuation sustained and full (verified: a 16s roll went
|
| 50 |
-
# from tail/head energy 0.28 to 0.63 with this alone). NOT peak normalization —
|
| 51 |
-
# that hands the model loud transients over a quiet body and it collapses even
|
| 52 |
-
# harder (tail/head 0.12). the generated audio comes back at this level too and
|
| 53 |
-
# is matched back down to the original's level at the final stitch.
|
| 54 |
-
PROMPT_TARGET_RMS = 0.12
|
| 55 |
-
# guidance_scale in transformers' musicgen CFG amplifies the TEXT condition
|
| 56 |
-
# only — the audio-prompt tokens are copied into both CFG branches and cancel
|
| 57 |
-
# out of the formula. with no caption there is nothing for it to amplify, so
|
| 58 |
-
# pure continuation runs with CFG off (1.0). when the user supplies a real
|
| 59 |
-
# caption we use meta's trained value.
|
| 60 |
-
GUIDANCE_NO_TEXT = 1.0
|
| 61 |
-
GUIDANCE_WITH_TEXT = 3.0
|
| 62 |
-
# vocal stem rms / mix rms above this = the clip has real vocals
|
| 63 |
-
VOCAL_RMS_THRESHOLD = 0.1
|
| 64 |
-
|
| 65 |
-
_model = None
|
| 66 |
-
_processor = None
|
| 67 |
-
_demucs = None
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
def _load_model():
|
| 71 |
-
global _model, _processor
|
| 72 |
-
if _model is None:
|
| 73 |
-
_processor = AutoProcessor.from_pretrained(MODEL_ID)
|
| 74 |
-
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 75 |
-
# fp32, NOT fp16: musicgen's text encoder is T5, which overflows in
|
| 76 |
-
# fp16 (garbage conditioning), and EnCodec audio tokenization audibly
|
| 77 |
-
# degrades in half precision — together they made the model ignore the
|
| 78 |
-
# clip and emit generic synth output. audiocraft's reference impl also
|
| 79 |
-
# keeps EnCodec in fp32. TF32 matmuls recover most of the speed on
|
| 80 |
-
# ampere+ GPUs without the precision cliff.
|
| 81 |
-
if device == "cuda":
|
| 82 |
-
torch.backends.cuda.matmul.allow_tf32 = True
|
| 83 |
-
torch.backends.cudnn.allow_tf32 = True
|
| 84 |
-
_model = MusicgenForConditionalGeneration.from_pretrained(
|
| 85 |
-
MODEL_ID, torch_dtype=torch.float32
|
| 86 |
-
).to(device)
|
| 87 |
-
_model.eval()
|
| 88 |
-
return _model, _processor
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
def _load_demucs():
|
| 92 |
-
global _demucs
|
| 93 |
-
if _demucs is None:
|
| 94 |
-
from demucs.pretrained import get_model
|
| 95 |
-
_demucs = get_model("htdemucs")
|
| 96 |
-
_demucs.eval()
|
| 97 |
-
return _demucs
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
def preload():
|
| 101 |
-
"""load musicgen + demucs at process start. on zerogpu the per-call gpu
|
| 102 |
-
window is the scarce resource — weights must come off disk at boot, not
|
| 103 |
-
inside the window. spaces patches .to('cuda') so placement is deferred
|
| 104 |
-
until a @spaces.GPU call actually attaches a gpu."""
|
| 105 |
-
_load_model()
|
| 106 |
-
_load_demucs()
|
| 107 |
-
print("[coda] preload: musicgen + demucs resident", flush=True)
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
def _load_tail(path, prompt_duration):
|
| 111 |
-
"""load audio, return mono 32kHz numpy tail of `prompt_duration` seconds."""
|
| 112 |
-
# librosa (soundfile backend) instead of torchaudio.load — torchaudio 2.9+
|
| 113 |
-
# delegates decoding to torchcodec, which isn't installed on the Space.
|
| 114 |
-
track, _ = librosa.load(path, sr=MUSICGEN_SR, mono=True)
|
| 115 |
-
|
| 116 |
-
tail_samples = int(prompt_duration * MUSICGEN_SR)
|
| 117 |
-
if track.shape[0] > tail_samples:
|
| 118 |
-
track = track[-tail_samples:]
|
| 119 |
-
|
| 120 |
-
return track.astype(np.float32)
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
def _vocal_gate(path, prompt_duration):
|
| 124 |
-
"""
|
| 125 |
-
musicgen's training data had vocals stripped (twice — tag filtering, then
|
| 126 |
-
HT-Demucs separation), so a vocal prompt is out-of-distribution: the model
|
| 127 |
-
drops, garbles, or replaces the voice. this gate runs HT-Demucs on the
|
| 128 |
-
tail and, if it carries real vocal energy, returns the INSTRUMENTAL tail
|
| 129 |
-
(the sum of the non-vocal stems — matching how the training data was
|
| 130 |
-
prepared, and dropping the noisy residual demucs can't attribute) as
|
| 131 |
-
mono float32 @32kHz. returns None when the clip is already instrumental
|
| 132 |
-
or demucs is unavailable, meaning: use the plain tail.
|
| 133 |
-
"""
|
| 134 |
-
try:
|
| 135 |
-
from demucs.apply import apply_model
|
| 136 |
-
except Exception as e:
|
| 137 |
-
print(f"[coda] demucs unavailable ({e}); skipping vocal gate", flush=True)
|
| 138 |
-
return None
|
| 139 |
-
|
| 140 |
-
try:
|
| 141 |
-
demucs = _load_demucs()
|
| 142 |
-
sep_sr = demucs.samplerate
|
| 143 |
-
|
| 144 |
-
wav, _ = librosa.load(path, sr=sep_sr, mono=False)
|
| 145 |
-
if wav.ndim == 1:
|
| 146 |
-
wav = np.stack([wav, wav])
|
| 147 |
-
tail = wav[:, -int(prompt_duration * sep_sr):].astype(np.float32)
|
| 148 |
-
|
| 149 |
-
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 150 |
-
mix = torch.from_numpy(np.ascontiguousarray(tail))
|
| 151 |
-
# demucs expects the mix standardized (same as its own separate.py)
|
| 152 |
-
ref = mix.mean(0)
|
| 153 |
-
std = float(ref.std()) + 1e-8
|
| 154 |
-
with torch.no_grad():
|
| 155 |
-
sources = apply_model(
|
| 156 |
-
demucs.to(device), ((mix - ref.mean()) / std)[None].to(device),
|
| 157 |
-
device=device, split=True, overlap=0.25, progress=False,
|
| 158 |
-
)[0]
|
| 159 |
-
sources = sources.cpu() * std + ref.mean()
|
| 160 |
-
vocals = sources[demucs.sources.index("vocals")].numpy()
|
| 161 |
-
|
| 162 |
-
vocal_rms = float(np.sqrt(np.mean(vocals ** 2)))
|
| 163 |
-
mix_rms = float(np.sqrt(np.mean(tail ** 2)) + 1e-9)
|
| 164 |
-
ratio = vocal_rms / mix_rms
|
| 165 |
-
if ratio < VOCAL_RMS_THRESHOLD:
|
| 166 |
-
print(f"[coda] vocal gate: instrumental clip "
|
| 167 |
-
f"(vocal/mix rms {ratio:.3f}), using original tail", flush=True)
|
| 168 |
-
return None
|
| 169 |
-
|
| 170 |
-
print(f"[coda] vocal gate: vocals detected (vocal/mix rms {ratio:.3f}) "
|
| 171 |
-
f"— prompting with the instrumental mix", flush=True)
|
| 172 |
-
# sum of the non-vocal stems, NOT mix-minus-vocals: demucs stems
|
| 173 |
-
# don't sum exactly to the mix, and the residual is everything it
|
| 174 |
-
# couldn't attribute to an instrument — tape hiss, crackle, phasey
|
| 175 |
-
# vocal ghosts. mix-minus keeps all of that in the prompt and the
|
| 176 |
-
# model continues the noise; the stem sum is the band, clean.
|
| 177 |
-
inst = np.stack([
|
| 178 |
-
sources[demucs.sources.index(s)].numpy()
|
| 179 |
-
for s in demucs.sources if s != "vocals"
|
| 180 |
-
]).sum(axis=0).mean(axis=0)
|
| 181 |
-
inst = librosa.resample(inst, orig_sr=sep_sr, target_sr=MUSICGEN_SR,
|
| 182 |
-
res_type="soxr_hq")
|
| 183 |
-
return inst.astype(np.float32)
|
| 184 |
-
except Exception as e:
|
| 185 |
-
print(f"[coda] vocal gate failed ({e}); using original tail", flush=True)
|
| 186 |
-
return None
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
def _generate_pass(model, processor, prompt_audio, desc, new_seconds,
|
| 190 |
-
guidance_scale):
|
| 191 |
-
"""
|
| 192 |
-
one musicgen pass conditioned on `prompt_audio` (mono float32 @32kHz).
|
| 193 |
-
returns the full decoded output: a re-encoded copy of the prompt followed
|
| 194 |
-
by ~`new_seconds` of new material, 1-D float32 @32kHz.
|
| 195 |
-
|
| 196 |
-
the prompt is RMS-normalized to PROMPT_TARGET_RMS first — a quiet prompt is
|
| 197 |
-
what makes musicgen drift to silence, so every pass gets a confident-level
|
| 198 |
-
prompt regardless of where the source or the previous pass sat.
|
| 199 |
-
"""
|
| 200 |
-
prompt_audio = _normalize_rms(prompt_audio, PROMPT_TARGET_RMS)
|
| 201 |
-
inputs = processor(
|
| 202 |
-
audio=prompt_audio,
|
| 203 |
-
sampling_rate=MUSICGEN_SR,
|
| 204 |
-
text=[desc],
|
| 205 |
-
padding=True,
|
| 206 |
-
return_tensors="pt",
|
| 207 |
-
).to(model.device)
|
| 208 |
-
|
| 209 |
-
max_new_tokens = int(new_seconds * FRAME_RATE)
|
| 210 |
-
prompt_secs = len(prompt_audio) / MUSICGEN_SR
|
| 211 |
-
print(f"[coda] generate(): max_new_tokens={max_new_tokens} "
|
| 212 |
-
f"({new_seconds:.1f}s @ {FRAME_RATE} tok/s), "
|
| 213 |
-
f"guidance_scale={guidance_scale}, audio_prompt={prompt_secs:.1f}s, "
|
| 214 |
-
f"text={desc!r}", flush=True)
|
| 215 |
-
|
| 216 |
-
with torch.no_grad():
|
| 217 |
-
output = model.generate(
|
| 218 |
-
**inputs,
|
| 219 |
-
do_sample=True,
|
| 220 |
-
guidance_scale=guidance_scale,
|
| 221 |
-
max_new_tokens=max_new_tokens,
|
| 222 |
-
)
|
| 223 |
-
|
| 224 |
-
audio = output[0, 0].float().cpu().numpy()
|
| 225 |
-
print(f"[coda] generate() returned {audio.shape[0]} samples "
|
| 226 |
-
f"({audio.shape[0] / MUSICGEN_SR:.1f}s incl. re-encoded prompt)",
|
| 227 |
-
flush=True)
|
| 228 |
-
return audio
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
def _rms(x):
|
| 232 |
-
return float(np.sqrt(np.mean(np.asarray(x, dtype=np.float64) ** 2)))
|
| 233 |
-
|
| 234 |
-
|
| 235 |
-
def _normalize_rms(x, target, max_gain=12.0, peak_ceiling=0.97):
|
| 236 |
-
"""scale `x` to `target` rms, capping the gain (so we don't blow up a
|
| 237 |
-
near-silent buffer into noise) and the resulting peak (so transients don't
|
| 238 |
-
clip the encoder). returns float32."""
|
| 239 |
-
x = np.asarray(x, dtype=np.float32)
|
| 240 |
-
cur = _rms(x)
|
| 241 |
-
if cur < 1e-6:
|
| 242 |
-
return x
|
| 243 |
-
gain = min(target / cur, max_gain)
|
| 244 |
-
out = x * gain
|
| 245 |
-
peak = float(np.abs(out).max())
|
| 246 |
-
if peak > peak_ceiling:
|
| 247 |
-
out = out * (peak_ceiling / peak)
|
| 248 |
-
return out.astype(np.float32)
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
def _energy_guard(out, prompt_samples, anchor_rms=None):
|
| 252 |
-
"""
|
| 253 |
-
bound the generated region's level relative to the level of the USER'S
|
| 254 |
-
clip (`anchor_rms`), not the level of whatever the previous pass left.
|
| 255 |
-
unconditional musicgen rolls drift — some decay toward silence, some blow
|
| 256 |
-
up past full scale — and an anchor that re-bases every pass would let
|
| 257 |
-
that drift compound (0.7 of 0.7 of 0.7 across a chain). a chunk whose
|
| 258 |
-
rms leaves ENERGY_BAND is rescaled to the boundary (gain capped at 3x so
|
| 259 |
-
a failed roll doesn't become amplified mush), with a short ramp so the
|
| 260 |
-
correction never lands as a step. only the NEW region is touched: the
|
| 261 |
-
re-encoded prompt region must stay honest — the stitch stage rms-matches
|
| 262 |
-
against it. returns (out, raw_ratio).
|
| 263 |
-
"""
|
| 264 |
-
prompt_rms = anchor_rms or _rms(out[:prompt_samples])
|
| 265 |
-
new = out[prompt_samples:]
|
| 266 |
-
if prompt_rms < 1e-6 or len(new) == 0:
|
| 267 |
-
return out, 1.0
|
| 268 |
-
ratio = _rms(new) / prompt_rms
|
| 269 |
-
lo, hi = ENERGY_BAND
|
| 270 |
-
gain = 1.0
|
| 271 |
-
if ratio < lo:
|
| 272 |
-
gain = min(lo / max(ratio, 1e-6), 3.0)
|
| 273 |
-
elif ratio > hi:
|
| 274 |
-
gain = hi / ratio
|
| 275 |
-
if gain != 1.0:
|
| 276 |
-
ramp_n = min(int(0.5 * MUSICGEN_SR), len(new))
|
| 277 |
-
g = np.full(len(new), gain, dtype=np.float32)
|
| 278 |
-
g[:ramp_n] = np.linspace(1.0, gain, ramp_n, dtype=np.float32)
|
| 279 |
-
out = np.concatenate([out[:prompt_samples],
|
| 280 |
-
new * g]).astype(np.float32)
|
| 281 |
-
print(f"[coda] energy guard: gen/prompt rms {ratio:.2f} -> "
|
| 282 |
-
f"rescaled x{gain:.2f}", flush=True)
|
| 283 |
-
return out, ratio
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
def _aligned_xfade(a_tail, b_head):
|
| 287 |
-
"""
|
| 288 |
-
blend two time-aligned versions of the SAME content with an equal-gain
|
| 289 |
-
(linear) curve. equal-power curves are for uncorrelated material — on
|
| 290 |
-
correlated signals they swell up to +3dB mid-fade, which is exactly the
|
| 291 |
-
audible volume bump/dip at the seam. for aligned content, gains summing
|
| 292 |
-
to 1 keep the level dead flat through the blend.
|
| 293 |
-
"""
|
| 294 |
-
n = min(len(a_tail), len(b_head))
|
| 295 |
-
g = np.linspace(0.0, 1.0, n, dtype=np.float32)
|
| 296 |
-
return a_tail[:n] * (1.0 - g) + b_head[:n] * g
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
def _merge_aligned(prev, new_out, prompt_samples, fade_samples):
|
| 300 |
-
"""
|
| 301 |
-
join `prev` with a generation pass output whose first `prompt_samples`
|
| 302 |
-
are a re-encode of prev's last `prompt_samples` (time-aligned).
|
| 303 |
-
crossfades INSIDE that overlap region so the fade completes exactly where
|
| 304 |
-
the newly generated material begins — the handoff into new content then
|
| 305 |
-
happens entirely within model audio and is seamless.
|
| 306 |
-
"""
|
| 307 |
-
prompt_samples = min(prompt_samples, len(prev), len(new_out))
|
| 308 |
-
fade = min(fade_samples, prompt_samples)
|
| 309 |
-
if fade <= 0:
|
| 310 |
-
return np.concatenate([prev, new_out[prompt_samples:]]).astype(np.float32)
|
| 311 |
-
|
| 312 |
-
seam = _aligned_xfade(prev[-fade:], new_out[prompt_samples - fade:prompt_samples])
|
| 313 |
-
return np.concatenate([prev[:-fade], seam, new_out[prompt_samples:]]).astype(np.float32)
|
| 314 |
-
|
| 315 |
-
|
| 316 |
-
def continue_track(path, prompt_duration=DEFAULT_PROMPT_SECONDS, gen_duration=60,
|
| 317 |
-
key=None, bpm=None, fade_seconds=6.0, caption="",
|
| 318 |
-
guidance_scale=None, progress=None):
|
| 319 |
-
"""
|
| 320 |
-
generates ~`gen_duration` seconds of new audio continuing the input track,
|
| 321 |
-
chaining multiple musicgen passes when more than one 30s window is needed
|
| 322 |
-
(each pass is re-prompted with the last CONTEXT_SECONDS of the accumulated
|
| 323 |
-
output, so timbre carries through the whole extension).
|
| 324 |
-
|
| 325 |
-
two modes:
|
| 326 |
-
- caption empty (default): pure audio-conditioned continuation, no text,
|
| 327 |
-
CFG off (guidance 1.0). this is the canonical continuation mode —
|
| 328 |
-
audiocraft's generate_continuation(descriptions=None).
|
| 329 |
-
- caption given: the user's own description (in-distribution for the
|
| 330 |
-
text encoder), guidance 3.0 (meta's trained value), with detected
|
| 331 |
-
key/bpm appended.
|
| 332 |
-
|
| 333 |
-
returns (extended_tail, prompt_samples, MUSICGEN_SR) where extended_tail
|
| 334 |
-
is 1-D float32 @32kHz whose first `prompt_samples` samples are a
|
| 335 |
-
re-encoded, time-aligned copy of the original's last `prompt_duration`
|
| 336 |
-
seconds — the caller crossfades the original into that region.
|
| 337 |
-
"""
|
| 338 |
-
# `progress`, when given, receives ("separating",) before the vocal gate
|
| 339 |
-
# and ("pass", i, n_passes, audio_so_far) after each generation pass —
|
| 340 |
-
# purely observational, used by the ui to paint the waveform live.
|
| 341 |
-
def _notify(*event):
|
| 342 |
-
if progress is not None:
|
| 343 |
-
try:
|
| 344 |
-
progress(*event)
|
| 345 |
-
except Exception as e:
|
| 346 |
-
print(f"[coda] progress callback failed ({e})", flush=True)
|
| 347 |
-
|
| 348 |
-
model, processor = _load_model()
|
| 349 |
-
|
| 350 |
-
tail = _load_tail(path, prompt_duration)
|
| 351 |
-
# vocal prompts are out-of-distribution for musicgen — swap in the
|
| 352 |
-
# demucs-separated instrumental when the tail carries vocals
|
| 353 |
-
_notify("separating")
|
| 354 |
-
instrumental = _vocal_gate(path, prompt_duration)
|
| 355 |
-
if instrumental is not None:
|
| 356 |
-
tail = instrumental
|
| 357 |
-
prompt_samples = tail.shape[0]
|
| 358 |
-
prompt_secs = prompt_samples / MUSICGEN_SR
|
| 359 |
-
|
| 360 |
-
caption = (caption or "").strip()
|
| 361 |
-
if caption:
|
| 362 |
-
desc = caption
|
| 363 |
-
if key and bpm:
|
| 364 |
-
desc += f", in {key} at {round(bpm)} bpm"
|
| 365 |
-
elif key:
|
| 366 |
-
desc += f", in {key}"
|
| 367 |
-
guidance = guidance_scale if guidance_scale else GUIDANCE_WITH_TEXT
|
| 368 |
-
else:
|
| 369 |
-
# no text condition at all — with guidance 1.0, CFG is off and the
|
| 370 |
-
# empty caption's encoder states carry no stock-music pull. the audio
|
| 371 |
-
# prefix alone steers the generation.
|
| 372 |
-
desc = ""
|
| 373 |
-
guidance = guidance_scale if guidance_scale else GUIDANCE_NO_TEXT
|
| 374 |
-
|
| 375 |
-
# window-safe pass plan. every pass must fit musicgen's trained 30s
|
| 376 |
-
# window: PROMPT + NEW <= 30s, where the first prompt is long (up to 20s
|
| 377 |
-
# of the user's clip — fidelity comes from context) and later passes
|
| 378 |
-
# re-prompt on 12s of model audio and generate 18s. exceeding the window
|
| 379 |
-
# makes the sinusoidal positions extrapolate and the output degrades
|
| 380 |
-
# audibly (verified: a 20s+18s pass came back clipping with tempo drift).
|
| 381 |
-
ctx_samples = int(CONTEXT_SECONDS * MUSICGEN_SR)
|
| 382 |
-
fade = int(fade_seconds * MUSICGEN_SR)
|
| 383 |
-
|
| 384 |
-
first_new = max(1.0, min(NEW_SECONDS_PER_PASS, gen_duration,
|
| 385 |
-
MAX_WINDOW_SECONDS - prompt_secs))
|
| 386 |
-
# mirrors the loop below, including the dropped sub-3s remainder, so the
|
| 387 |
-
# "pass i of n" the ui shows is the count that actually happens
|
| 388 |
-
n_passes = 1 + max(0, math.ceil(
|
| 389 |
-
(gen_duration - first_new - 3.0) / NEW_SECONDS_PER_PASS))
|
| 390 |
-
|
| 391 |
-
print(f"[coda] continuation plan: ~{gen_duration:.0f}s of new audio in "
|
| 392 |
-
f"{n_passes} pass(es) (first: {prompt_secs:.1f}s prompt + "
|
| 393 |
-
f"{first_new:.1f}s new, then {CONTEXT_SECONDS}s ctx + "
|
| 394 |
-
f"{NEW_SECONDS_PER_PASS}s new), guidance {guidance} "
|
| 395 |
-
f"({'caption' if caption else 'pure audio'} mode)", flush=True)
|
| 396 |
-
|
| 397 |
-
acc = None
|
| 398 |
-
remaining = float(gen_duration)
|
| 399 |
-
retries_left = MAX_RETRIES_PER_TRACK
|
| 400 |
-
# every generated chunk is held against the level the model is prompted at
|
| 401 |
-
# (PROMPT_TARGET_RMS), not the previous pass — so drift can't compound and
|
| 402 |
-
# the guard's band is measured against the same level the model sees.
|
| 403 |
-
anchor_rms = PROMPT_TARGET_RMS
|
| 404 |
-
i = 0
|
| 405 |
-
# a sub-3s remainder isn't worth a model call (~8s of overhead for 2s
|
| 406 |
-
# of audio nobody will miss next to the closing fade) — drop it
|
| 407 |
-
while remaining > 3.0 or acc is None:
|
| 408 |
-
prompt = tail if acc is None else acc[-ctx_samples:]
|
| 409 |
-
p_secs = len(prompt) / MUSICGEN_SR
|
| 410 |
-
p_samples = len(prompt)
|
| 411 |
-
new_secs = max(1.0, min(NEW_SECONDS_PER_PASS, remaining,
|
| 412 |
-
MAX_WINDOW_SECONDS - p_secs))
|
| 413 |
-
out = _generate_pass(model, processor, prompt, desc, new_secs,
|
| 414 |
-
guidance)
|
| 415 |
-
out, ratio = _energy_guard(out, p_samples, anchor_rms)
|
| 416 |
-
if ratio < COLLAPSE_RATIO and retries_left > 0:
|
| 417 |
-
# a failed roll (near-silence, wisps) — re-roll the pass once
|
| 418 |
-
# instead of amplifying mush. the prompt is unchanged; sampling
|
| 419 |
-
# gives a fresh trajectory.
|
| 420 |
-
retries_left -= 1
|
| 421 |
-
print(f"[coda] pass collapsed (gen/prompt rms {ratio:.2f}) — "
|
| 422 |
-
f"re-rolling ({retries_left} retries left)", flush=True)
|
| 423 |
-
continue
|
| 424 |
-
i += 1
|
| 425 |
-
if acc is None:
|
| 426 |
-
acc = out
|
| 427 |
-
else:
|
| 428 |
-
acc = _merge_aligned(acc, out, p_samples, fade)
|
| 429 |
-
remaining -= new_secs
|
| 430 |
-
_notify("pass", i, n_passes, acc)
|
| 431 |
-
|
| 432 |
-
return acc, prompt_samples, MUSICGEN_SR
|
| 433 |
-
|
| 434 |
-
|
| 435 |
-
def stitch_with_crossfade(original, extended_tail, prompt_samples_gen, original_sr,
|
| 436 |
-
fade_seconds=6.0, end_fade_seconds=4.0):
|
| 437 |
-
"""
|
| 438 |
-
splice the generated continuation onto the original at the original's
|
| 439 |
-
native sample rate and channel count, and close the track with a fade-out
|
| 440 |
-
so it ends like a finished song instead of cutting mid-phrase.
|
| 441 |
-
|
| 442 |
-
original: float32, 1-D (mono) or 2-D (channels, samples) @original_sr
|
| 443 |
-
extended_tail: 1-D float32 @32kHz from continue_track(); its first
|
| 444 |
-
`prompt_samples_gen` samples mirror the end of `original`
|
| 445 |
-
returns the full track, same shape convention as `original`.
|
| 446 |
-
"""
|
| 447 |
-
# resample the generated audio UP to the original's rate before stitching —
|
| 448 |
-
# never resample the original down.
|
| 449 |
-
if original_sr != MUSICGEN_SR:
|
| 450 |
-
extended_tail = librosa.resample(
|
| 451 |
-
extended_tail, orig_sr=MUSICGEN_SR, target_sr=original_sr,
|
| 452 |
-
res_type="soxr_hq",
|
| 453 |
-
)
|
| 454 |
-
prompt_samples = int(round(prompt_samples_gen * original_sr / MUSICGEN_SR))
|
| 455 |
-
prompt_samples = min(prompt_samples, original.shape[-1], extended_tail.shape[0])
|
| 456 |
-
|
| 457 |
-
# loudness-match the generation to the original over the shared prompt
|
| 458 |
-
# region so the crossfade doesn't pump
|
| 459 |
-
orig_tail = original[..., -prompt_samples:]
|
| 460 |
-
if orig_tail.ndim == 2:
|
| 461 |
-
orig_tail = orig_tail.mean(axis=0)
|
| 462 |
-
ref_rms = float(np.sqrt(np.mean(orig_tail ** 2)) + 1e-9)
|
| 463 |
-
gen_rms = float(np.sqrt(np.mean(extended_tail[:prompt_samples] ** 2)) + 1e-9)
|
| 464 |
-
extended_tail = extended_tail * float(np.clip(ref_rms / gen_rms, 0.5, 2.0))
|
| 465 |
-
|
| 466 |
-
# match channel layout (musicgen is mono; duplicate across channels)
|
| 467 |
-
if original.ndim == 2:
|
| 468 |
-
extended_tail = np.tile(extended_tail, (original.shape[0], 1))
|
| 469 |
-
|
| 470 |
-
fade = min(int(fade_seconds * original_sr), prompt_samples)
|
| 471 |
-
if fade <= 0:
|
| 472 |
-
out = np.concatenate(
|
| 473 |
-
[original, extended_tail[..., prompt_samples:]], axis=-1)
|
| 474 |
-
else:
|
| 475 |
-
# the blend ends exactly where the new material begins: original hands
|
| 476 |
-
# off over the last `fade` samples to the time-aligned re-encoded
|
| 477 |
-
# prompt, so the entry into generated content happens entirely inside
|
| 478 |
-
# model audio. equal-GAIN (linear) curve, not equal-power: the two
|
| 479 |
-
# signals are aligned copies of the same content, and equal-power on
|
| 480 |
-
# correlated material swells +3dB mid-fade — an audible seam artifact.
|
| 481 |
-
g = np.linspace(0.0, 1.0, fade, dtype=np.float32)
|
| 482 |
-
seam = (original[..., -fade:] * (1.0 - g)
|
| 483 |
-
+ extended_tail[..., prompt_samples - fade:prompt_samples] * g)
|
| 484 |
-
out = np.concatenate(
|
| 485 |
-
[original[..., :-fade], seam, extended_tail[..., prompt_samples:]],
|
| 486 |
-
axis=-1)
|
| 487 |
-
|
| 488 |
-
out = out.astype(np.float32)
|
| 489 |
-
|
| 490 |
-
# smooth fade to silence at the very end so the song closes cleanly
|
| 491 |
-
end_fade = min(int(end_fade_seconds * original_sr), out.shape[-1])
|
| 492 |
-
if end_fade > 0:
|
| 493 |
-
curve = np.cos(np.linspace(0.0, np.pi / 2, end_fade, dtype=np.float32)) ** 2
|
| 494 |
-
out[..., -end_fade:] *= curve
|
| 495 |
-
|
| 496 |
-
peak = np.abs(out).max()
|
| 497 |
-
if peak > 1.0:
|
| 498 |
-
out = out / peak
|
| 499 |
-
return out
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -0,0 +1,213 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""engine.py — Stable Audio 3 Small Music continuation core.
|
| 2 |
+
|
| 3 |
+
CODA's job is one thing done well: take a short, unfinished clip and continue it
|
| 4 |
+
into a finished-sounding track in the same key, tempo and feel. SA3 does that in
|
| 5 |
+
a SINGLE call. Its `generate_diffusion_cond_inpaint` is a native audio-inpainting
|
| 6 |
+
diffusion sampler: place the user's clip at the front of the buffer, mask the
|
| 7 |
+
region after it, and the model fills the masked region conditioned on the kept
|
| 8 |
+
audio — true long-form continuation, 44.1 kHz stereo, no multi-pass chaining,
|
| 9 |
+
no energy guards, no re-roll logic.
|
| 10 |
+
|
| 11 |
+
This module is the whole generation core. It returns ONLY the newly generated
|
| 12 |
+
tail (the model's [source_end, total] region) plus the source length in seconds;
|
| 13 |
+
`stitch.py` joins that tail onto the user's *pristine* original so the real
|
| 14 |
+
recording (and any vocals) plays untouched up to the seam.
|
| 15 |
+
|
| 16 |
+
Mask convention (verified against the installed library source):
|
| 17 |
+
inpaint_mask = ones(buffer); inpaint_mask[start:end] = 0
|
| 18 |
+
-> 1 = keep the input audio, 0 = generate. So masking [L_src, L_total] keeps
|
| 19 |
+
the source in [0, L_src] and generates everything after it.
|
| 20 |
+
"""
|
| 21 |
+
import numpy as np
|
| 22 |
+
import torch
|
| 23 |
+
|
| 24 |
+
MODEL_ID = "stabilityai/stable-audio-3-small-music"
|
| 25 |
+
|
| 26 |
+
SR = 44100 # SA3 native sample rate (model_config: sample_rate)
|
| 27 |
+
STEPS = 8 # SA3 Small is an 8-step adversarially-distilled model
|
| 28 |
+
SAMPLER = "pingpong" # the sampler the distilled model was tuned for
|
| 29 |
+
DEFAULT_CFG = 1.0 # distilled-model guidance; the prompt still conditions
|
| 30 |
+
# at 1.0 (CFG amplification off, conditional path on)
|
| 31 |
+
MAX_TOTAL_SECONDS = 120 # SA3 Small duration cap (sample_size / sample_rate)
|
| 32 |
+
MIN_NEW_SECONDS = 5 # below this a "continuation" isn't worth a GPU call
|
| 33 |
+
|
| 34 |
+
_model = None
|
| 35 |
+
_model_config = None
|
| 36 |
+
_sample_size = None
|
| 37 |
+
_on_device = None # which device the weights currently live on
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def _device():
|
| 41 |
+
return "cuda" if torch.cuda.is_available() else "cpu"
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def preload():
|
| 45 |
+
"""Load model + autoencoder + T5Gemma conditioner into CPU RAM at process
|
| 46 |
+
start. On ZeroGPU the per-call GPU window is the scarce resource, so weights
|
| 47 |
+
must come off disk at boot, not inside the window. The CUDA placement + fp16
|
| 48 |
+
cast is deferred to the first `continue_audio` call (i.e. the @spaces.GPU
|
| 49 |
+
window), matching how Stability's own Space defers it."""
|
| 50 |
+
global _model, _model_config, _sample_size
|
| 51 |
+
if _model is None:
|
| 52 |
+
from stable_audio_tools import get_pretrained_model
|
| 53 |
+
_model, _model_config = get_pretrained_model(MODEL_ID)
|
| 54 |
+
_sample_size = int(_model_config["sample_size"])
|
| 55 |
+
_model.eval()
|
| 56 |
+
print(f"[coda] preload: SA3 resident "
|
| 57 |
+
f"(sr={_model_config['sample_rate']}, "
|
| 58 |
+
f"sample_size={_sample_size} = "
|
| 59 |
+
f"{_sample_size / int(_model_config['sample_rate']):.0f}s)",
|
| 60 |
+
flush=True)
|
| 61 |
+
return _model, _model_config
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def _ensure_on_device():
|
| 65 |
+
"""Ensure weights are on the GPU in fp16. Called inside the @spaces.GPU
|
| 66 |
+
window on every generation. `.to()` is a cheap no-op when the model is
|
| 67 |
+
already placed, so we re-ensure each call rather than caching device state —
|
| 68 |
+
that stays correct even if ZeroGPU detaches the GPU between calls. fp16 is
|
| 69 |
+
only valid on CUDA; on CPU the model stays fp32."""
|
| 70 |
+
global _model, _on_device
|
| 71 |
+
dev = _device()
|
| 72 |
+
_model = _model.to(dev)
|
| 73 |
+
if dev == "cuda":
|
| 74 |
+
_model = _model.to(torch.float16)
|
| 75 |
+
_on_device = dev
|
| 76 |
+
return _model
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def _load_source(clip_path):
|
| 80 |
+
"""Load the clip as stereo float32 @44.1k as a (2, N) tensor. SA3's
|
| 81 |
+
autoencoder is stereo; `prepare_audio` inside the sampler will pad/crop to
|
| 82 |
+
the buffer length and place this at the FRONT (PadCrop, randomize=False)."""
|
| 83 |
+
import librosa
|
| 84 |
+
y, _ = librosa.load(clip_path, sr=SR, mono=False)
|
| 85 |
+
y = np.asarray(y, dtype=np.float32)
|
| 86 |
+
if y.ndim == 1:
|
| 87 |
+
y = np.stack([y, y]) # mono -> stereo
|
| 88 |
+
elif y.shape[0] > 2:
|
| 89 |
+
y = y[:2]
|
| 90 |
+
return torch.from_numpy(np.ascontiguousarray(y))
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
#: longest clip that still leaves room for MIN_NEW of continuation under the cap
|
| 94 |
+
MAX_SOURCE_SECONDS = MAX_TOTAL_SECONDS - MIN_NEW_SECONDS
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def plan_continuation(source_seconds, total_seconds):
|
| 98 |
+
"""Pure helper (unit-testable, no model): clamp the request to SA3's limits
|
| 99 |
+
and return (total_seconds, new_seconds, mask_start, mask_end).
|
| 100 |
+
|
| 101 |
+
- the mask runs from where the source ends to the total length: that masked
|
| 102 |
+
region is what SA3 generates, so mask_end MUST exceed mask_start.
|
| 103 |
+
- total is capped at MAX_TOTAL_SECONDS and floored so at least
|
| 104 |
+
MIN_NEW_SECONDS of new audio is generated.
|
| 105 |
+
- raises ValueError when the source is already so long there's no room to
|
| 106 |
+
continue under the cap (otherwise the mask would invert and SA3 would
|
| 107 |
+
silently generate nothing).
|
| 108 |
+
"""
|
| 109 |
+
source_seconds = float(source_seconds)
|
| 110 |
+
total_seconds = float(total_seconds)
|
| 111 |
+
if source_seconds > MAX_SOURCE_SECONDS:
|
| 112 |
+
raise ValueError(
|
| 113 |
+
f"clip is {source_seconds:.0f}s — too long to continue under SA3's "
|
| 114 |
+
f"{MAX_TOTAL_SECONDS:.0f}s cap (need room for at least "
|
| 115 |
+
f"{MIN_NEW_SECONDS:.0f}s of new audio); trim it under "
|
| 116 |
+
f"{MAX_SOURCE_SECONDS:.0f}s.")
|
| 117 |
+
# source <= MAX_SOURCE_SECONDS, so source + MIN_NEW <= MAX_TOTAL: the floor
|
| 118 |
+
# never pushes total past the cap, and mask_end (total) > mask_start (source).
|
| 119 |
+
total_seconds = min(total_seconds, MAX_TOTAL_SECONDS)
|
| 120 |
+
total_seconds = max(total_seconds, source_seconds + MIN_NEW_SECONDS)
|
| 121 |
+
new_seconds = total_seconds - source_seconds
|
| 122 |
+
return total_seconds, new_seconds, source_seconds, total_seconds
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def continue_audio(clip_path, total_seconds, prompt="", cfg_scale=DEFAULT_CFG,
|
| 126 |
+
seed=-1, progress=None):
|
| 127 |
+
"""Continue `clip_path` up to `total_seconds` in one SA3 inpaint call.
|
| 128 |
+
|
| 129 |
+
Returns (new_tail, source_seconds, SR) where:
|
| 130 |
+
new_tail : (2, M) float32 @44.1k — ONLY the generated region
|
| 131 |
+
[source_end, total]. Peak-normalized to <= 1.0.
|
| 132 |
+
source_seconds : the clip's true length (the splice boundary, in seconds)
|
| 133 |
+
SR : 44100
|
| 134 |
+
|
| 135 |
+
`progress(stage_name)` is called (best-effort) at each stage so the UI can
|
| 136 |
+
paint a live status. SA3 is one diffusion call, so progress is stage-based.
|
| 137 |
+
"""
|
| 138 |
+
from einops import rearrange
|
| 139 |
+
from stable_audio_tools.inference.generation import (
|
| 140 |
+
generate_diffusion_cond_inpaint)
|
| 141 |
+
|
| 142 |
+
def _notify(stage):
|
| 143 |
+
if progress is not None:
|
| 144 |
+
try:
|
| 145 |
+
progress(stage)
|
| 146 |
+
except Exception as e:
|
| 147 |
+
print(f"[coda] progress callback failed ({e})", flush=True)
|
| 148 |
+
|
| 149 |
+
preload()
|
| 150 |
+
model = _ensure_on_device()
|
| 151 |
+
dev = _device()
|
| 152 |
+
|
| 153 |
+
# The library does `np.random.randint(0, 2**32-1)` when seed == -1, which
|
| 154 |
+
# overflows int32 on Windows/numpy<2. Draw a safe seed ourselves so the
|
| 155 |
+
# default path works everywhere, not just on the Linux Space.
|
| 156 |
+
if seed is None or seed < 0:
|
| 157 |
+
seed = int(np.random.randint(0, 2 ** 31 - 1))
|
| 158 |
+
|
| 159 |
+
_notify("reading")
|
| 160 |
+
source = _load_source(clip_path)
|
| 161 |
+
source_seconds = source.shape[-1] / SR
|
| 162 |
+
# the autoencoder runs in the model's dtype (fp16 on CUDA); the conditioning
|
| 163 |
+
# audio must match it or the encoder's conv1d rejects the input dtype.
|
| 164 |
+
model_dtype = next(model.model.parameters()).dtype
|
| 165 |
+
source = source.to(model_dtype)
|
| 166 |
+
|
| 167 |
+
total_seconds, new_seconds, mask_start, mask_end = plan_continuation(
|
| 168 |
+
source_seconds, total_seconds)
|
| 169 |
+
prompt = (prompt or "").strip()
|
| 170 |
+
|
| 171 |
+
print(f"[coda] continuation: source={source_seconds:.1f}s -> "
|
| 172 |
+
f"total={total_seconds:.1f}s (+{new_seconds:.1f}s new), "
|
| 173 |
+
f"mask=[{mask_start:.1f}s, {mask_end:.1f}s], steps={STEPS}, "
|
| 174 |
+
f"cfg={cfg_scale}, prompt={prompt!r}", flush=True)
|
| 175 |
+
|
| 176 |
+
_notify("composing")
|
| 177 |
+
with torch.no_grad():
|
| 178 |
+
output = generate_diffusion_cond_inpaint(
|
| 179 |
+
model,
|
| 180 |
+
steps=STEPS,
|
| 181 |
+
cfg_scale=cfg_scale,
|
| 182 |
+
conditioning=[{"prompt": prompt, "seconds_total": total_seconds}],
|
| 183 |
+
sample_size=_sample_size,
|
| 184 |
+
sampler_type=SAMPLER,
|
| 185 |
+
inpaint_audio=(SR, source),
|
| 186 |
+
inpaint_mask_start_seconds=mask_start,
|
| 187 |
+
inpaint_mask_end_seconds=mask_end,
|
| 188 |
+
seed=seed,
|
| 189 |
+
device=dev,
|
| 190 |
+
)
|
| 191 |
+
|
| 192 |
+
_notify("finalizing")
|
| 193 |
+
# (b, d, n) -> (d, b*n); peak-normalize like Stability's reference Space
|
| 194 |
+
output = rearrange(output, "b d n -> d (b n)")
|
| 195 |
+
audio = output.to(torch.float32).cpu().numpy()
|
| 196 |
+
peak = float(np.abs(audio).max())
|
| 197 |
+
if peak > 1e-9:
|
| 198 |
+
audio = audio / peak
|
| 199 |
+
|
| 200 |
+
if audio.shape[0] == 1: # safety: ensure stereo
|
| 201 |
+
audio = np.repeat(audio, 2, axis=0)
|
| 202 |
+
|
| 203 |
+
boundary = int(round(source_seconds * SR))
|
| 204 |
+
end = int(round(total_seconds * SR))
|
| 205 |
+
end = min(end, audio.shape[-1])
|
| 206 |
+
new_tail = audio[:, boundary:end]
|
| 207 |
+
new_tail = np.ascontiguousarray(new_tail.astype(np.float32))
|
| 208 |
+
|
| 209 |
+
print(f"[coda] generated tail: shape={new_tail.shape} "
|
| 210 |
+
f"({new_tail.shape[-1] / SR:.1f}s), peak after norm "
|
| 211 |
+
f"{float(np.abs(new_tail).max()):.3f}, "
|
| 212 |
+
f"rms {float(np.sqrt(np.mean(new_tail ** 2))):.3f}", flush=True)
|
| 213 |
+
return new_tail, source_seconds, SR
|
|
@@ -1,14 +1,29 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
# pinned to the
|
| 4 |
-
#
|
| 5 |
-
#
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
librosa>=0.10.2
|
| 10 |
-
numpy>=1.24.0,<2.0
|
| 11 |
-
scipy>=1.10.0
|
| 12 |
soundfile>=0.12.1
|
| 13 |
-
|
|
|
|
| 14 |
spaces
|
|
|
|
| 1 |
+
# CODA runs Stable Audio 3 Small Music through the stable-audio-tools library.
|
| 2 |
+
# This dependency set mirrors Stability's own ZeroGPU Space (the proven-green
|
| 3 |
+
# build for this model) and is pinned to the exact torch that was verified
|
| 4 |
+
# end-to-end locally: torch 2.7.1 on the cu128 wheel index, which carries kernels
|
| 5 |
+
# for both the dev GPU (Blackwell sm_120) and ZeroGPU's H200 (sm_90).
|
| 6 |
+
#
|
| 7 |
+
# gradio is provided by the Space SDK (see README front-matter), so it is NOT
|
| 8 |
+
# pinned here. Demucs is intentionally absent: CODA splices the user's pristine
|
| 9 |
+
# original (vocals and all) up to the seam, so no source separation is needed.
|
| 10 |
+
--extra-index-url https://download.pytorch.org/whl/cu128
|
| 11 |
+
torch==2.7.1
|
| 12 |
+
torchaudio==2.7.1
|
| 13 |
+
|
| 14 |
+
# The SA3 inference library. 0.0.20 (upstream main) is the first version with the
|
| 15 |
+
# SA3 architecture + native inpainting; PyPI only publishes up to 0.0.19, so this
|
| 16 |
+
# installs from the repo. It pulls its own pinned tree (vector-quantize-pytorch,
|
| 17 |
+
# v-diffusion-pytorch, alias-free-torch, torchsde, nnAudio, einops, …).
|
| 18 |
+
stable-audio-tools @ git+https://github.com/Stability-AI/stable-audio-tools.git
|
| 19 |
+
|
| 20 |
+
# stable-audio-tools imports pytorch_lightning (its LoRA callbacks) without
|
| 21 |
+
# declaring it; the official SA3 Space lists it explicitly, so CODA does too.
|
| 22 |
+
pytorch_lightning
|
| 23 |
+
|
| 24 |
+
# CODA's own DSP + ZeroGPU glue
|
| 25 |
librosa>=0.10.2
|
|
|
|
|
|
|
| 26 |
soundfile>=0.12.1
|
| 27 |
+
scipy>=1.10.0
|
| 28 |
+
numpy<2
|
| 29 |
spaces
|
|
@@ -0,0 +1,93 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""stitch.py — splice the SA3 continuation onto the user's pristine original.
|
| 2 |
+
|
| 3 |
+
CODA's identity: your real recording plays untouched up to the seam, then the
|
| 4 |
+
generated tail takes over. Everything here is 44.1 kHz stereo — SA3's native
|
| 5 |
+
format and the deliverable's — so the original is resampled up to 44.1k and made
|
| 6 |
+
stereo, and the engine's tail is already there.
|
| 7 |
+
|
| 8 |
+
Unlike the old MusicGen path, the SA3 tail does NOT contain a re-encoded copy of
|
| 9 |
+
the source; it is fresh audio that begins exactly where the source ends. The seam
|
| 10 |
+
is therefore a join between sequential content, so:
|
| 11 |
+
- level: the tail is loudness-matched to the original's tail RMS so the seam
|
| 12 |
+
doesn't pump (gain bounded so a quiet lo-fi clip can't crush the tail);
|
| 13 |
+
- click: a short equal-power crossfade smooths the join;
|
| 14 |
+
- close: a cos^2 fade to true silence ends the track like a finished song;
|
| 15 |
+
- then one peak-normalize lifts the whole (now level-consistent) track to a
|
| 16 |
+
confident listening level with -1 dBFS of headroom.
|
| 17 |
+
"""
|
| 18 |
+
import librosa
|
| 19 |
+
import numpy as np
|
| 20 |
+
|
| 21 |
+
SR = 44100
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def to_stereo_44k(audio, sr):
|
| 25 |
+
"""(channels, N) or (N,) @sr -> (2, N') float32 @44.1k stereo."""
|
| 26 |
+
a = np.asarray(audio, dtype=np.float32)
|
| 27 |
+
if a.ndim == 1:
|
| 28 |
+
a = a[None, :]
|
| 29 |
+
if sr != SR:
|
| 30 |
+
a = np.stack([
|
| 31 |
+
librosa.resample(ch, orig_sr=sr, target_sr=SR, res_type="soxr_hq")
|
| 32 |
+
for ch in a
|
| 33 |
+
])
|
| 34 |
+
if a.shape[0] == 1:
|
| 35 |
+
a = np.repeat(a, 2, axis=0)
|
| 36 |
+
elif a.shape[0] > 2:
|
| 37 |
+
a = a[:2]
|
| 38 |
+
return np.ascontiguousarray(a.astype(np.float32))
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def _rms(x):
|
| 42 |
+
return float(np.sqrt(np.mean(np.asarray(x, dtype=np.float64) ** 2)) + 1e-12)
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def stitch(original, original_sr, new_tail, source_seconds,
|
| 46 |
+
crossfade_seconds=0.10, end_fade_seconds=4.0, match_seconds=2.0,
|
| 47 |
+
peak_ceiling=0.891):
|
| 48 |
+
"""Join the user's original to the SA3-generated tail.
|
| 49 |
+
|
| 50 |
+
original : (channels, N) or (N,) float32 @original_sr — pristine clip
|
| 51 |
+
new_tail : (2, M) float32 @44.1k from engine.continue_audio
|
| 52 |
+
source_seconds : the splice boundary (clip length) in seconds
|
| 53 |
+
returns : (2, T) float32 @44.1k, peak == peak_ceiling (-1 dBFS)
|
| 54 |
+
"""
|
| 55 |
+
orig = to_stereo_44k(original, original_sr)
|
| 56 |
+
tail = to_stereo_44k(new_tail, SR)
|
| 57 |
+
|
| 58 |
+
boundary = min(int(round(source_seconds * SR)), orig.shape[-1])
|
| 59 |
+
orig = orig[:, :boundary] # only the real recording up to the seam
|
| 60 |
+
|
| 61 |
+
# loudness-match the tail to the original's level at the seam (continuity).
|
| 62 |
+
# bound the gain: a very quiet lo-fi clip shouldn't drag the full-bodied
|
| 63 |
+
# tail down to a whisper, and we never amplify the tail wildly either.
|
| 64 |
+
m = min(int(match_seconds * SR), orig.shape[-1], tail.shape[-1])
|
| 65 |
+
if m > 0:
|
| 66 |
+
gain = float(np.clip(_rms(orig[:, -m:]) / _rms(tail[:, :m]), 0.4, 2.5))
|
| 67 |
+
tail = tail * gain
|
| 68 |
+
|
| 69 |
+
# short equal-power crossfade across the join. the two sides are sequential
|
| 70 |
+
# (not time-aligned copies), so equal-power — not equal-gain — keeps the
|
| 71 |
+
# energy flat through the blend.
|
| 72 |
+
fade = min(int(crossfade_seconds * SR), orig.shape[-1], tail.shape[-1])
|
| 73 |
+
if fade > 0:
|
| 74 |
+
t = np.linspace(0.0, 1.0, fade, dtype=np.float32)
|
| 75 |
+
fout, fin = np.cos(t * np.pi / 2), np.sin(t * np.pi / 2)
|
| 76 |
+
seam = orig[:, -fade:] * fout + tail[:, :fade] * fin
|
| 77 |
+
out = np.concatenate([orig[:, :-fade], seam, tail[:, fade:]], axis=-1)
|
| 78 |
+
else:
|
| 79 |
+
out = np.concatenate([orig, tail], axis=-1)
|
| 80 |
+
out = out.astype(np.float32)
|
| 81 |
+
|
| 82 |
+
# closing fade to true silence so the song ends instead of cutting off
|
| 83 |
+
end_fade = min(int(end_fade_seconds * SR), out.shape[-1])
|
| 84 |
+
if end_fade > 0:
|
| 85 |
+
curve = np.cos(np.linspace(0.0, np.pi / 2, end_fade,
|
| 86 |
+
dtype=np.float32)) ** 2
|
| 87 |
+
out[:, -end_fade:] *= curve
|
| 88 |
+
|
| 89 |
+
# lift the whole, now level-consistent, track to a confident level
|
| 90 |
+
peak = float(np.abs(out).max())
|
| 91 |
+
if peak > 1e-9:
|
| 92 |
+
out = out * (peak_ceiling / peak)
|
| 93 |
+
return out.astype(np.float32)
|
|
@@ -1,68 +1,24 @@
|
|
| 1 |
-
"""
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
import os
|
| 6 |
-
import sys
|
| 7 |
|
| 8 |
import numpy as np
|
|
|
|
| 9 |
import soundfile as sf
|
| 10 |
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
def check(name, cond, detail=""):
|
| 15 |
-
print(f"[{'PASS' if cond else 'FAIL'}] {name} {detail}")
|
| 16 |
-
if not cond:
|
| 17 |
-
FAILURES.append(name)
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
import app
|
| 21 |
-
from continue_music import MUSICGEN_SR
|
| 22 |
|
| 23 |
DEMO = app.PUSHBACK_DEMO
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
# ---- 1. load_demo just hands back the demo path ----
|
| 27 |
-
check("load_demo returns the demo clip", app.load_demo() == DEMO)
|
| 28 |
-
|
| 29 |
-
# ---- 2. on-upload analysis: instant, CPU-only, enables the button ----
|
| 30 |
-
info_update, btn_update = app.analyze_on_upload(DEMO)
|
| 31 |
-
md = info_update["value"]
|
| 32 |
-
check("analysis reports a key", "Key" in md and "·" in md, md[:60])
|
| 33 |
-
check("analysis reports tempo", "BPM" in md)
|
| 34 |
-
check("analysis makes the panel visible", info_update["visible"] is True)
|
| 35 |
-
check("analysis enables Finish button", btn_update["interactive"] is True)
|
| 36 |
-
|
| 37 |
-
# empty input keeps the button disabled
|
| 38 |
-
empty_info, empty_btn = app.analyze_on_upload(None)
|
| 39 |
-
check("no clip -> button stays disabled", empty_btn["interactive"] is False)
|
| 40 |
-
|
| 41 |
-
# ---- 3. finish_song pipeline with the model stubbed ----
|
| 42 |
-
# replace the heavy continuation with a fast fake that mimics its contract:
|
| 43 |
-
# returns (tail, prompt_samples, sr) where tail[:prompt_samples] mirrors the
|
| 44 |
-
# end of the (cleaned) input so the stitch can crossfade into it.
|
| 45 |
-
import librosa
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
def fake_continue_track(path, gen_duration=30, progress=None):
|
| 49 |
-
if progress:
|
| 50 |
-
progress("separating")
|
| 51 |
-
progress("pass", 1, 1, np.zeros(10))
|
| 52 |
-
src, _ = librosa.load(path, sr=MUSICGEN_SR, mono=True)
|
| 53 |
-
prompt_samples = min(len(src), 20 * MUSICGEN_SR)
|
| 54 |
-
prompt = src[-prompt_samples:]
|
| 55 |
-
new = 0.1 * np.sin(2 * np.pi * 220 *
|
| 56 |
-
np.arange(int(gen_duration * MUSICGEN_SR)) / MUSICGEN_SR)
|
| 57 |
-
return (np.concatenate([prompt, new]).astype(np.float32),
|
| 58 |
-
prompt_samples, MUSICGEN_SR)
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
app.continue_track = fake_continue_track
|
| 62 |
|
| 63 |
|
| 64 |
class _Prog:
|
| 65 |
-
"""stand-in for gr.Progress —
|
| 66 |
def __init__(self):
|
| 67 |
self.calls = []
|
| 68 |
|
|
@@ -70,28 +26,74 @@ class _Prog:
|
|
| 70 |
self.calls.append((frac, desc))
|
| 71 |
|
| 72 |
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tests for app.py wiring: on-upload analysis, demo loading, and the full
|
| 2 |
+
finish_song pipeline with the SA3 engine stubbed (no GPU, no weights). Verifies
|
| 3 |
+
the real DSP (analyze, enhance, stitch) runs end-to-end on the bundled demo and
|
| 4 |
+
produces a valid 44.1 kHz stereo file. Skips if gradio isn't installed."""
|
| 5 |
import os
|
|
|
|
| 6 |
|
| 7 |
import numpy as np
|
| 8 |
+
import pytest
|
| 9 |
import soundfile as sf
|
| 10 |
|
| 11 |
+
pytest.importorskip("gradio")
|
| 12 |
+
import app # noqa: E402
|
| 13 |
+
import engine # noqa: E402
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
DEMO = app.PUSHBACK_DEMO
|
| 16 |
+
_HAVE_DEMO = os.path.exists(DEMO)
|
| 17 |
+
need_demo = pytest.mark.skipif(not _HAVE_DEMO, reason="bundled demo clip absent")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
|
| 20 |
class _Prog:
|
| 21 |
+
"""stand-in for gr.Progress — records the (frac, desc) stages it's told."""
|
| 22 |
def __init__(self):
|
| 23 |
self.calls = []
|
| 24 |
|
|
|
|
| 26 |
self.calls.append((frac, desc))
|
| 27 |
|
| 28 |
|
| 29 |
+
@need_demo
|
| 30 |
+
def test_load_demo_returns_clip():
|
| 31 |
+
assert app.load_demo() == DEMO
|
| 32 |
+
assert os.path.exists(DEMO)
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
@need_demo
|
| 36 |
+
def test_analyze_on_upload_reports_and_enables():
|
| 37 |
+
info_update, btn_update = app.analyze_on_upload(DEMO)
|
| 38 |
+
md = info_update["value"]
|
| 39 |
+
assert "KEY" in md and "TEMPO" in md and "BPM" in md
|
| 40 |
+
assert info_update["visible"] is True
|
| 41 |
+
assert btn_update["interactive"] is True
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def test_analyze_on_upload_empty_disables_button():
|
| 45 |
+
_, btn_update = app.analyze_on_upload(None)
|
| 46 |
+
assert btn_update["interactive"] is False
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
@need_demo
|
| 50 |
+
def test_finish_song_pipeline_with_stubbed_engine(monkeypatch):
|
| 51 |
+
"""stub engine.continue_audio with a fast synthetic tail; the rest of the
|
| 52 |
+
pipeline (enhance, analyze, stitch, write) runs for real."""
|
| 53 |
+
def fake_continue_audio(clip_path, total_seconds, prompt="", progress=None,
|
| 54 |
+
**kw):
|
| 55 |
+
if progress:
|
| 56 |
+
progress("reading")
|
| 57 |
+
progress("composing")
|
| 58 |
+
progress("finalizing")
|
| 59 |
+
new_secs = total_seconds - 29.53 # demo is ~29.53s
|
| 60 |
+
n = int(max(new_secs, 5) * engine.SR)
|
| 61 |
+
t = np.arange(n) / engine.SR
|
| 62 |
+
wave = 0.2 * np.sin(2 * np.pi * 220 * t).astype(np.float32)
|
| 63 |
+
tail = np.stack([wave, wave])
|
| 64 |
+
return tail, 29.53, engine.SR
|
| 65 |
+
|
| 66 |
+
monkeypatch.setattr(engine, "continue_audio", fake_continue_audio)
|
| 67 |
+
|
| 68 |
+
prog = _Prog()
|
| 69 |
+
out_path, summary = app.finish_song(DEMO, 60, "", False, progress=prog)
|
| 70 |
+
|
| 71 |
+
assert os.path.exists(out_path)
|
| 72 |
+
y, sr = sf.read(out_path)
|
| 73 |
+
assert sr == 44100
|
| 74 |
+
assert y.ndim == 2 and y.shape[1] == 2 # 44.1k stereo
|
| 75 |
+
assert len(y) / sr > 50 # ~60s finished
|
| 76 |
+
assert float(np.abs(y).max()) <= 1.0
|
| 77 |
+
assert float(np.sqrt(np.mean(y ** 2))) > 1e-3 # not silent
|
| 78 |
+
assert "BPM" in summary and "Stable Audio 3" in summary
|
| 79 |
+
# progress streamed several stages and ended at 100%
|
| 80 |
+
assert len(prog.calls) >= 4
|
| 81 |
+
assert prog.calls[-1][0] == 1.0
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
@need_demo
|
| 85 |
+
def test_finish_song_remaster_path(monkeypatch):
|
| 86 |
+
def fake_continue_audio(clip_path, total_seconds, prompt="", progress=None,
|
| 87 |
+
**kw):
|
| 88 |
+
n = int(15 * engine.SR)
|
| 89 |
+
wave = 0.2 * np.sin(2 * np.pi * 220 * np.arange(n) / engine.SR)
|
| 90 |
+
return np.stack([wave, wave]).astype(np.float32), 29.53, engine.SR
|
| 91 |
+
|
| 92 |
+
monkeypatch.setattr(engine, "continue_audio", fake_continue_audio)
|
| 93 |
+
out_path, _ = app.finish_song(DEMO, 45, "", True, progress=_Prog())
|
| 94 |
+
assert os.path.exists(out_path)
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def test_finish_song_rejects_empty_input():
|
| 98 |
+
with pytest.raises(Exception):
|
| 99 |
+
app.finish_song(None, 60, "", False, progress=_Prog())
|
|
@@ -0,0 +1,69 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Unit tests for engine.py's pure continuation planning — the source-length ->
|
| 2 |
+
mask-bounds + total-length-cap math. No model, no GPU: `plan_continuation` is a
|
| 3 |
+
pure function, so these run anywhere torch+numpy import."""
|
| 4 |
+
import pytest
|
| 5 |
+
|
| 6 |
+
import engine
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def test_normal_request_maps_to_tail_mask():
|
| 10 |
+
total, new, mstart, mend = engine.plan_continuation(30, 60)
|
| 11 |
+
assert total == 60
|
| 12 |
+
assert new == 30
|
| 13 |
+
# the mask runs from where the source ends to the total length
|
| 14 |
+
assert mstart == 30
|
| 15 |
+
assert mend == 60
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def test_total_capped_at_120():
|
| 19 |
+
total, new, mstart, mend = engine.plan_continuation(30, 200)
|
| 20 |
+
assert total == engine.MAX_TOTAL_SECONDS == 120
|
| 21 |
+
assert new == 90
|
| 22 |
+
assert mstart == 30 and mend == 120
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def test_min_new_floor_enforced():
|
| 26 |
+
# asking for barely-longer-than-source still generates at least MIN_NEW
|
| 27 |
+
total, new, mstart, mend = engine.plan_continuation(30, 31)
|
| 28 |
+
assert new == engine.MIN_NEW_SECONDS == 5
|
| 29 |
+
assert total == 35
|
| 30 |
+
assert mstart == 30 and mend == 35
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def test_mask_always_brackets_the_new_region():
|
| 34 |
+
for src, req in [(15, 40), (29.5, 60), (50, 90), (10, 120)]:
|
| 35 |
+
total, new, mstart, mend = engine.plan_continuation(src, req)
|
| 36 |
+
assert mstart == src # mask starts at the seam
|
| 37 |
+
assert mend == total # …and runs to the end
|
| 38 |
+
assert abs((mend - mstart) - new) < 1e-6 # masked span == new audio
|
| 39 |
+
assert total <= engine.MAX_TOTAL_SECONDS
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def test_source_at_max_keeps_min_new():
|
| 43 |
+
# the longest allowed clip still gets exactly MIN_NEW of continuation
|
| 44 |
+
total, new, mstart, mend = engine.plan_continuation(engine.MAX_SOURCE_SECONDS, 130)
|
| 45 |
+
assert total == engine.MAX_TOTAL_SECONDS == 120
|
| 46 |
+
assert new == engine.MIN_NEW_SECONDS == 5
|
| 47 |
+
assert mend > mstart # mask never inverts
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def test_overlong_source_raises_not_inverts():
|
| 51 |
+
# the old bug: source >= cap produced an inverted mask and a silent
|
| 52 |
+
# no-op continuation. now it must raise instead.
|
| 53 |
+
for src in (116, 120, 125, 200):
|
| 54 |
+
with pytest.raises(ValueError):
|
| 55 |
+
engine.plan_continuation(src, 60)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def test_no_plan_ever_inverts_the_mask():
|
| 59 |
+
for src in [1, 15, 29.5, 60, 90, 110, 115]:
|
| 60 |
+
total, new, mstart, mend = engine.plan_continuation(src, 60)
|
| 61 |
+
assert mend > mstart # mask_end strictly after mask_start
|
| 62 |
+
assert new >= engine.MIN_NEW_SECONDS - 1e-6
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def test_constants_match_sa3_contract():
|
| 66 |
+
assert engine.SR == 44100
|
| 67 |
+
assert engine.STEPS == 8
|
| 68 |
+
assert engine.SAMPLER == "pingpong"
|
| 69 |
+
assert engine.MAX_TOTAL_SECONDS == 120
|
|
@@ -1,273 +1,106 @@
|
|
| 1 |
-
"""
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
fade."""
|
| 5 |
-
import sys
|
| 6 |
-
import types
|
| 7 |
import numpy as np
|
|
|
|
| 8 |
|
| 9 |
-
|
| 10 |
-
torch_stub = types.ModuleType("torch")
|
| 11 |
-
torch_stub.float16 = "float16"
|
| 12 |
-
torch_stub.float32 = "float32"
|
| 13 |
-
torch_stub.no_grad = lambda: types.SimpleNamespace(__enter__=lambda s: None, __exit__=lambda s, *a: False)
|
| 14 |
-
torch_stub.cuda = types.SimpleNamespace(is_available=lambda: False)
|
| 15 |
-
torch_stub.Tensor = type("Tensor", (), {}) # scipy array-api probes this
|
| 16 |
-
torch_stub.backends = types.SimpleNamespace(
|
| 17 |
-
cuda=types.SimpleNamespace(matmul=types.SimpleNamespace(allow_tf32=False)),
|
| 18 |
-
cudnn=types.SimpleNamespace(allow_tf32=False),
|
| 19 |
-
)
|
| 20 |
-
sys.modules["torch"] = torch_stub
|
| 21 |
-
tf_stub = types.ModuleType("transformers")
|
| 22 |
-
tf_stub.AutoProcessor = object
|
| 23 |
-
tf_stub.MusicgenForConditionalGeneration = object
|
| 24 |
-
sys.modules["transformers"] = tf_stub
|
| 25 |
|
| 26 |
-
|
| 27 |
|
| 28 |
-
SR = cm.MUSICGEN_SR
|
| 29 |
-
CTX = cm.CONTEXT_SECONDS
|
| 30 |
-
FAILURES = []
|
| 31 |
|
| 32 |
-
|
| 33 |
-
def check(name, cond, detail=""):
|
| 34 |
-
status = "PASS" if cond else "FAIL"
|
| 35 |
-
print(f"[{status}] {name} {detail}")
|
| 36 |
-
if not cond:
|
| 37 |
-
FAILURES.append(name)
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
def tone(freq, seconds, sr, phase=0.0):
|
| 41 |
t = np.arange(int(seconds * sr)) / sr
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
normed = cm._normalize_rms(quiet, cm.PROMPT_TARGET_RMS)
|
| 136 |
-
check("normalize: quiet prompt lifted to target rms",
|
| 137 |
-
abs(cm._rms(normed) / cm.PROMPT_TARGET_RMS - 1) < 0.02,
|
| 138 |
-
f"rms_out={cm._rms(normed):.3f} target={cm.PROMPT_TARGET_RMS}")
|
| 139 |
-
check("normalize: gain capped so silence isn't blown into noise",
|
| 140 |
-
cm._rms(cm._normalize_rms(tone(220, 4, SR) * 1e-5, cm.PROMPT_TARGET_RMS))
|
| 141 |
-
< cm.PROMPT_TARGET_RMS,
|
| 142 |
-
"near-silent buffer hits the 12x gain cap, not the target")
|
| 143 |
-
loud = tone(220, 4, SR) * 1.5
|
| 144 |
-
check("normalize: peak ceiling prevents encoder clipping",
|
| 145 |
-
float(np.abs(cm._normalize_rms(loud, cm.PROMPT_TARGET_RMS)).max()) <= 0.971)
|
| 146 |
-
check("normalize: silent buffer passes through untouched",
|
| 147 |
-
np.array_equal(cm._normalize_rms(np.zeros(1000, np.float32),
|
| 148 |
-
cm.PROMPT_TARGET_RMS),
|
| 149 |
-
np.zeros(1000, np.float32)))
|
| 150 |
-
|
| 151 |
-
# ---- 6. stitching at native 44.1kHz stereo ----
|
| 152 |
-
orig = np.stack([tone(220, 35, 44100), tone(220, 35, 44100, phase=0.1)])
|
| 153 |
-
full = cm.stitch_with_crossfade(orig, ext, prompt_samples, 44100,
|
| 154 |
-
fade_seconds=3.0, end_fade_seconds=4.0)
|
| 155 |
-
|
| 156 |
-
check("output stereo", full.ndim == 2 and full.shape[0] == 2, f"shape={full.shape}")
|
| 157 |
-
expected_len = orig.shape[-1] + int(round((ext.shape[0] - prompt_samples) * 44100 / SR))
|
| 158 |
-
check("output length = original + upsampled new content",
|
| 159 |
-
abs(full.shape[-1] - expected_len) <= 2,
|
| 160 |
-
f"got {full.shape[-1]}, expected ~{expected_len}")
|
| 161 |
-
check("original preserved before fade",
|
| 162 |
-
np.allclose(full[:, :orig.shape[-1] - 3 * 44100], orig[:, :-3 * 44100], atol=1e-6))
|
| 163 |
-
check("no clipping", np.abs(full).max() <= 1.0, f"peak={np.abs(full).max():.3f}")
|
| 164 |
-
|
| 165 |
-
# seam continuity: no sample-to-sample jump bigger than the waveform itself allows
|
| 166 |
-
seam_region = full[0, orig.shape[-1] - 3 * 44100 - 100: orig.shape[-1] + 44100]
|
| 167 |
-
max_jump = np.abs(np.diff(seam_region)).max()
|
| 168 |
-
ref_jump = np.abs(np.diff(full[0, :44100])).max()
|
| 169 |
-
check("no click at seam", max_jump < ref_jump * 1.5,
|
| 170 |
-
f"max_jump={max_jump:.4f} vs ref={ref_jump:.4f}")
|
| 171 |
-
|
| 172 |
-
# level stays flat through the blend (equal-gain on correlated content —
|
| 173 |
-
# equal-power would swell ~+3dB at the midpoint here)
|
| 174 |
-
mid = orig.shape[-1] - int(1.5 * 44100) # middle of the 3s blend
|
| 175 |
-
mid_rms = np.sqrt(np.mean(full[0, mid - 11025: mid + 11025] ** 2))
|
| 176 |
-
pre_rms = np.sqrt(np.mean(full[0, mid - 6 * 44100: mid - 5 * 44100] ** 2))
|
| 177 |
-
check("no volume swell mid-blend", abs(mid_rms / pre_rms - 1) < 0.1,
|
| 178 |
-
f"mid={mid_rms:.4f} pre={pre_rms:.4f} ratio={mid_rms / pre_rms:.3f}")
|
| 179 |
-
|
| 180 |
-
# closing fade: track ends at silence
|
| 181 |
-
check("ends at silence", np.abs(full[:, -100:]).max() < 1e-3,
|
| 182 |
-
f"end peak={np.abs(full[:, -100:]).max():.5f}")
|
| 183 |
-
check("fade region is monotone-ish quieter",
|
| 184 |
-
np.abs(full[0, -4410:]).max() < np.abs(full[0, -4 * 44100:-3 * 44100]).max())
|
| 185 |
-
|
| 186 |
-
# ---- 7. rms matching: quiet generation gets boosted toward original level ----
|
| 187 |
-
quiet_ext = ext * 0.6
|
| 188 |
-
full_q = cm.stitch_with_crossfade(orig, quiet_ext, prompt_samples, 44100,
|
| 189 |
-
fade_seconds=3.0, end_fade_seconds=0.0)
|
| 190 |
-
tail_rms = np.sqrt(np.mean(full_q[:, -44100 * 5:] ** 2))
|
| 191 |
-
orig_rms = np.sqrt(np.mean(orig[:, -44100 * 5:] ** 2))
|
| 192 |
-
check("rms matched within 25%", abs(tail_rms / orig_rms - 1) < 0.25,
|
| 193 |
-
f"tail={tail_rms:.3f} orig={orig_rms:.3f}")
|
| 194 |
-
|
| 195 |
-
# ---- 8. mono original passes through as 1-D ----
|
| 196 |
-
full_m = cm.stitch_with_crossfade(orig[0], ext, prompt_samples, 44100, fade_seconds=3.0)
|
| 197 |
-
check("mono stays 1-D", full_m.ndim == 1)
|
| 198 |
-
|
| 199 |
-
# ---- 9. progress callback: fires per pass, never breaks generation ----
|
| 200 |
-
events = []
|
| 201 |
-
cm.continue_track("fake.wav", gen_duration=36,
|
| 202 |
-
progress=lambda *ev: events.append(ev))
|
| 203 |
-
kinds = [e[0] for e in events]
|
| 204 |
-
check("progress: separating fires first", kinds[0] == "separating", f"{kinds}")
|
| 205 |
-
# 36s = 10 + 18 + 8
|
| 206 |
-
check("progress: one event per pass", kinds.count("pass") == 3, f"{kinds}")
|
| 207 |
-
check("progress: pass carries (i, n, audio)",
|
| 208 |
-
events[1][1] == 1 and events[1][2] == 3 and hasattr(events[1][3], "shape"))
|
| 209 |
-
check("progress: audio grows between passes",
|
| 210 |
-
events[2][3].shape[0] > events[1][3].shape[0])
|
| 211 |
-
# a broken callback must not kill the generation
|
| 212 |
-
ext_cb, _, _ = cm.continue_track(
|
| 213 |
-
"fake.wav", gen_duration=18,
|
| 214 |
-
progress=lambda *ev: (_ for _ in ()).throw(ValueError("boom")))
|
| 215 |
-
check("progress: callback exceptions are swallowed", ext_cb is not None)
|
| 216 |
-
|
| 217 |
-
# ---- 10. _merge_aligned keeps alignment between chained passes ----
|
| 218 |
-
prev = tone(220, 30, SR)
|
| 219 |
-
nxt = np.concatenate([prev[-CTX * SR:] * 0.95, tone(220, 18, SR)])
|
| 220 |
-
merged = cm._merge_aligned(prev, nxt, CTX * SR, 6 * SR)
|
| 221 |
-
check("merge length = prev + new", merged.shape[0] == prev.shape[0] + 18 * SR,
|
| 222 |
-
f"got {merged.shape[0]}")
|
| 223 |
-
|
| 224 |
-
# ---- 11. energy guard: holds the level against the anchor ----
|
| 225 |
-
p = tone(220, 12, SR)
|
| 226 |
-
quiet = np.concatenate([p, tone(220, 18, SR) * 0.5]) # 0.5x the anchor level
|
| 227 |
-
anchor = float(np.sqrt(np.mean(p.astype(np.float64) ** 2)))
|
| 228 |
-
guarded, ratio = cm._energy_guard(quiet.copy(), len(p), anchor)
|
| 229 |
-
g_rms = np.sqrt(np.mean(guarded[len(p) + SR:] ** 2)) # past the 0.5s ramp
|
| 230 |
-
check("guard: quiet chunk boosted to band floor",
|
| 231 |
-
abs(g_rms / anchor - cm.ENERGY_BAND[0]) < 0.05,
|
| 232 |
-
f"ratio_in={ratio:.2f} rms_out/anchor={g_rms / anchor:.2f}")
|
| 233 |
-
dead = np.concatenate([p, tone(220, 18, SR) * 0.05]) # failed roll: 0.05x
|
| 234 |
-
guarded_d, ratio_d = cm._energy_guard(dead.copy(), len(p), anchor)
|
| 235 |
-
gd_rms = np.sqrt(np.mean(guarded_d[len(p) + SR:] ** 2))
|
| 236 |
-
check("guard: gain capped at 3x on dead rolls",
|
| 237 |
-
abs(gd_rms / anchor - 0.15) < 0.03,
|
| 238 |
-
f"ratio_in={ratio_d:.2f} rms_out/anchor={gd_rms / anchor:.2f}")
|
| 239 |
-
check("guard: prompt region untouched",
|
| 240 |
-
np.array_equal(guarded[:len(p)], quiet[:len(p)]))
|
| 241 |
-
loud = np.concatenate([p, tone(220, 18, SR) * 2.0]) # 2.0 = 6.7x anchor
|
| 242 |
-
guarded_l, _ = cm._energy_guard(loud.copy(), len(p), anchor)
|
| 243 |
-
gl_rms = np.sqrt(np.mean(guarded_l[len(p) + SR:] ** 2))
|
| 244 |
-
check("guard: hot chunk pulled to band ceiling",
|
| 245 |
-
abs(gl_rms / anchor - cm.ENERGY_BAND[1]) < 0.1,
|
| 246 |
-
f"rms_out/anchor={gl_rms / anchor:.2f}")
|
| 247 |
-
inband = np.concatenate([p, tone(220, 18, SR)])
|
| 248 |
-
guarded_i, ratio_i = cm._energy_guard(inband.copy(), len(p), anchor)
|
| 249 |
-
check("guard: in-band chunk passes untouched",
|
| 250 |
-
np.array_equal(guarded_i, inband), f"ratio={ratio_i:.2f}")
|
| 251 |
-
|
| 252 |
-
# ---- 12. collapse re-roll: a near-silent pass regenerates ----
|
| 253 |
-
roll_calls = []
|
| 254 |
-
|
| 255 |
-
def collapsing_pass(model, processor, prompt_audio, desc, new_seconds, guidance_scale):
|
| 256 |
-
roll_calls.append(new_seconds)
|
| 257 |
-
level = 0.01 if len(roll_calls) == 1 else 1.0 # first roll is dead air
|
| 258 |
-
return np.concatenate([prompt_audio * 0.95,
|
| 259 |
-
tone(220, new_seconds, SR) * level])
|
| 260 |
-
|
| 261 |
-
orig_pass = cm._generate_pass
|
| 262 |
-
cm._generate_pass = collapsing_pass
|
| 263 |
-
ext_r, _, _ = cm.continue_track("fake.wav", gen_duration=10)
|
| 264 |
-
cm._generate_pass = orig_pass
|
| 265 |
-
check("collapsed pass re-rolls once", len(roll_calls) == 2,
|
| 266 |
-
f"calls={roll_calls}")
|
| 267 |
-
cm._generate_pass = fake_generate_pass
|
| 268 |
-
|
| 269 |
-
print()
|
| 270 |
-
if FAILURES:
|
| 271 |
-
print(f"{len(FAILURES)} FAILURES: {FAILURES}")
|
| 272 |
-
sys.exit(1)
|
| 273 |
-
print("all checks passed")
|
|
|
|
| 1 |
+
"""Unit tests for stitch.py — the stereo-native splice/crossfade math. No model,
|
| 2 |
+
no GPU: stitch is pure librosa+numpy. Covers channel/rate normalization, seam
|
| 3 |
+
length, original preservation, loudness match, closing fade and the peak guard."""
|
|
|
|
|
|
|
|
|
|
| 4 |
import numpy as np
|
| 5 |
+
import pytest
|
| 6 |
|
| 7 |
+
import stitch
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
+
SR = stitch.SR
|
| 10 |
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
+
def tone(freq, seconds, sr=SR, amp=0.3, phase=0.0, channels=2):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
t = np.arange(int(seconds * sr)) / sr
|
| 14 |
+
y = (amp * np.sin(2 * np.pi * freq * t + phase)).astype(np.float32)
|
| 15 |
+
if channels == 1:
|
| 16 |
+
return y
|
| 17 |
+
return np.stack([y, np.sin(2 * np.pi * freq * t + phase + 0.1) * amp]).astype(np.float32)
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def test_to_stereo_44k_from_mono_22k():
|
| 21 |
+
mono = tone(220, 1.0, sr=22050, channels=1)
|
| 22 |
+
out = stitch.to_stereo_44k(mono, 22050)
|
| 23 |
+
assert out.shape[0] == 2 # mono -> stereo
|
| 24 |
+
assert abs(out.shape[1] - 44100) <= 2 # 22050 -> 44100 (1s)
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def test_to_stereo_44k_truncates_extra_channels():
|
| 28 |
+
five = np.zeros((5, 1000), dtype=np.float32)
|
| 29 |
+
out = stitch.to_stereo_44k(five, SR)
|
| 30 |
+
assert out.shape[0] == 2
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def test_output_is_stereo_44k_and_unclipped():
|
| 34 |
+
orig = tone(220, 30, channels=2)
|
| 35 |
+
tail = tone(220, 30, channels=2)
|
| 36 |
+
out = stitch.stitch(orig, SR, tail, source_seconds=30)
|
| 37 |
+
assert out.ndim == 2 and out.shape[0] == 2
|
| 38 |
+
peak = float(np.abs(out).max())
|
| 39 |
+
assert peak <= 1.0
|
| 40 |
+
assert peak == pytest.approx(0.891, abs=1e-3) # lifted to -1 dBFS
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def test_length_is_original_plus_tail_minus_crossfade():
|
| 44 |
+
orig = tone(220, 30, channels=2)
|
| 45 |
+
tail = tone(330, 25, channels=2)
|
| 46 |
+
xfade = 0.10
|
| 47 |
+
out = stitch.stitch(orig, SR, tail, source_seconds=30,
|
| 48 |
+
crossfade_seconds=xfade)
|
| 49 |
+
expected = orig.shape[-1] + tail.shape[-1] - int(xfade * SR)
|
| 50 |
+
assert abs(out.shape[-1] - expected) <= 2
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def test_original_preserved_before_seam():
|
| 54 |
+
orig = tone(220, 10, channels=2)
|
| 55 |
+
tail = tone(440, 10, channels=2)
|
| 56 |
+
out = stitch.stitch(orig, SR, tail, source_seconds=10,
|
| 57 |
+
crossfade_seconds=0.1, end_fade_seconds=0.0,
|
| 58 |
+
peak_ceiling=1.0)
|
| 59 |
+
boundary = int(10 * SR)
|
| 60 |
+
# before the crossfade region the original should match up to the global
|
| 61 |
+
# gain stitch applies (compare shapes/correlation, not exact equality)
|
| 62 |
+
pre = out[:, : boundary - int(0.2 * SR)]
|
| 63 |
+
ref = orig[:, : pre.shape[-1]]
|
| 64 |
+
# high correlation == the original content is intact (only scaled)
|
| 65 |
+
corr = np.corrcoef(pre[0], ref[0])[0, 1]
|
| 66 |
+
assert corr > 0.999
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def test_closing_fade_ends_in_silence():
|
| 70 |
+
orig = tone(220, 5, channels=2)
|
| 71 |
+
tail = tone(220, 10, channels=2)
|
| 72 |
+
out = stitch.stitch(orig, SR, tail, source_seconds=5, end_fade_seconds=4.0)
|
| 73 |
+
tail_peak = float(np.abs(out[:, -200:]).max())
|
| 74 |
+
assert tail_peak < 1e-3 # true silence at the end
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def test_quiet_tail_loudness_matched_up():
|
| 78 |
+
# a tail within the gain clamp (2x quieter) should be matched to the
|
| 79 |
+
# original's level at the seam; extreme ratios are deliberately only
|
| 80 |
+
# partially corrected (see the [0.4, 2.5] clamp in stitch.stitch).
|
| 81 |
+
orig = tone(220, 10, amp=0.4, channels=2)
|
| 82 |
+
quiet_tail = tone(220, 10, amp=0.2, channels=2) # 2x quieter — inside clamp
|
| 83 |
+
out = stitch.stitch(orig, SR, quiet_tail, source_seconds=10,
|
| 84 |
+
end_fade_seconds=0.0, peak_ceiling=1.0)
|
| 85 |
+
seam = int(10 * SR)
|
| 86 |
+
orig_rms = np.sqrt(np.mean(out[:, seam - SR: seam] ** 2))
|
| 87 |
+
tail_rms = np.sqrt(np.mean(out[:, seam + SR: seam + 3 * SR] ** 2))
|
| 88 |
+
assert tail_rms / orig_rms > 0.8
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def test_mono_original_becomes_stereo():
|
| 92 |
+
orig = tone(220, 5, channels=1)
|
| 93 |
+
tail = tone(220, 5, channels=2)
|
| 94 |
+
out = stitch.stitch(orig, SR, tail, source_seconds=5)
|
| 95 |
+
assert out.shape[0] == 2 # output is always stereo
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def test_no_seam_click():
|
| 99 |
+
orig = tone(220, 8, channels=2)
|
| 100 |
+
tail = tone(220, 8, channels=2, phase=1.3) # different phase at join
|
| 101 |
+
out = stitch.stitch(orig, SR, tail, source_seconds=8,
|
| 102 |
+
crossfade_seconds=0.1, end_fade_seconds=0.0)
|
| 103 |
+
boundary = int(8 * SR)
|
| 104 |
+
seam = out[0, boundary - 1000: boundary + 1000]
|
| 105 |
+
ref = np.abs(np.diff(out[0, :SR])).max()
|
| 106 |
+
assert np.abs(np.diff(seam)).max() < ref * 3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Unit tests for verify.py — the math QA metrics. Builds synthetic finished
|
| 2 |
+
tracks (good, silence-collapsed, clipped) as temp WAVs and checks the right
|
| 3 |
+
metric catches each fault. No model needed."""
|
| 4 |
+
import numpy as np
|
| 5 |
+
import soundfile as sf
|
| 6 |
+
|
| 7 |
+
import verify
|
| 8 |
+
|
| 9 |
+
SR = verify.SR
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def _stereo(freqs, seconds, amp=0.3):
|
| 13 |
+
"""sum of sine partials -> a richer (non-pure-tone) stereo signal."""
|
| 14 |
+
t = np.arange(int(seconds * SR)) / SR
|
| 15 |
+
y = sum(np.sin(2 * np.pi * f * t) for f in freqs) / len(freqs)
|
| 16 |
+
L = (amp * y).astype(np.float32)
|
| 17 |
+
R = (amp * np.roll(y, 13)).astype(np.float32) # slight decorrelation
|
| 18 |
+
return np.stack([L, R])
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def _write(path, audio):
|
| 22 |
+
sf.write(str(path), audio.T, SR, subtype="PCM_16")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def test_relative_keys():
|
| 26 |
+
assert "A minor" in verify._relative_keys("C major")
|
| 27 |
+
assert "C major" in verify._relative_keys("A minor")
|
| 28 |
+
assert "C major" in verify._relative_keys("C major")
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def test_good_continuation_passes_health_metrics(tmp_path):
|
| 32 |
+
chord = [220, 277, 330] # A major-ish triad
|
| 33 |
+
orig = _stereo(chord, 10)
|
| 34 |
+
finished = _stereo(chord, 40) # continuous, same content
|
| 35 |
+
op, fp = tmp_path / "o.wav", tmp_path / "f.wav"
|
| 36 |
+
_write(op, orig)
|
| 37 |
+
_write(fp, finished)
|
| 38 |
+
rep = verify.verify(str(op), str(fp), source_seconds=10)
|
| 39 |
+
m = rep["metrics"]
|
| 40 |
+
assert m["no_silence_collapse"]["pass"]
|
| 41 |
+
assert m["no_clipping"]["pass"]
|
| 42 |
+
assert m["duration"]["pass"]
|
| 43 |
+
assert m["loudness_continuity"]["pass"]
|
| 44 |
+
assert m["spectral_rolloff"]["pass"]
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def test_silence_collapse_is_caught(tmp_path):
|
| 48 |
+
chord = [220, 277, 330]
|
| 49 |
+
orig = _stereo(chord, 10)
|
| 50 |
+
new = _stereo(chord, 30)
|
| 51 |
+
new[:, 5 * SR:15 * SR] = 0.0 # a dead 10s stretch mid-tail
|
| 52 |
+
finished = np.concatenate([_stereo(chord, 10), new], axis=-1)
|
| 53 |
+
op, fp = tmp_path / "o.wav", tmp_path / "f.wav"
|
| 54 |
+
_write(op, orig)
|
| 55 |
+
_write(fp, finished)
|
| 56 |
+
rep = verify.verify(str(op), str(fp), source_seconds=10)
|
| 57 |
+
assert not rep["metrics"]["no_silence_collapse"]["pass"]
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def test_clipping_is_caught(tmp_path):
|
| 61 |
+
chord = [220, 277, 330]
|
| 62 |
+
orig = _stereo(chord, 10)
|
| 63 |
+
finished = _stereo(chord, 30) * 4.0 # drive it hard
|
| 64 |
+
finished = np.clip(finished, -1.0, 1.0) # …into hard clipping
|
| 65 |
+
op, fp = tmp_path / "o.wav", tmp_path / "f.wav"
|
| 66 |
+
_write(op, orig)
|
| 67 |
+
_write(fp, finished)
|
| 68 |
+
rep = verify.verify(str(op), str(fp), source_seconds=10)
|
| 69 |
+
assert not rep["metrics"]["no_clipping"]["pass"]
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def test_closing_fade_not_flagged_as_collapse(tmp_path):
|
| 73 |
+
"""a real cos^2 closing fade must NOT trip the silence-collapse metric."""
|
| 74 |
+
chord = [220, 277, 330]
|
| 75 |
+
finished = _stereo(chord, 40)
|
| 76 |
+
fade = int(4.0 * SR)
|
| 77 |
+
curve = np.cos(np.linspace(0, np.pi / 2, fade)) ** 2
|
| 78 |
+
finished[:, -fade:] *= curve
|
| 79 |
+
orig = _stereo(chord, 10)
|
| 80 |
+
op, fp = tmp_path / "o.wav", tmp_path / "f.wav"
|
| 81 |
+
_write(op, orig)
|
| 82 |
+
_write(fp, finished)
|
| 83 |
+
rep = verify.verify(str(op), str(fp), source_seconds=10, fade_seconds=4.0)
|
| 84 |
+
assert rep["metrics"]["no_silence_collapse"]["pass"]
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def test_report_shape(tmp_path):
|
| 88 |
+
chord = [220, 277, 330]
|
| 89 |
+
op, fp = tmp_path / "o.wav", tmp_path / "f.wav"
|
| 90 |
+
_write(op, _stereo(chord, 10))
|
| 91 |
+
_write(fp, _stereo(chord, 30))
|
| 92 |
+
rep = verify.verify(str(op), str(fp), source_seconds=10)
|
| 93 |
+
assert set(["metrics", "passed", "finished", "boundary"]).issubset(rep)
|
| 94 |
+
assert isinstance(rep["passed"], bool)
|
| 95 |
+
for name, mtr in rep["metrics"].items():
|
| 96 |
+
assert set(["value", "pass", "reason"]).issubset(mtr)
|
|
@@ -0,0 +1,287 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""verify.py — mathematical audio QA for CODA continuations (dev/test tool).
|
| 2 |
+
|
| 3 |
+
Not imported by the app at runtime. Given the user's original clip, the finished
|
| 4 |
+
track, and the splice boundary, it measures whether the generated region is real,
|
| 5 |
+
continuous music — the things an ear-check can miss between sessions:
|
| 6 |
+
|
| 7 |
+
duration · silence-collapse · clipping · loudness continuity · seam click ·
|
| 8 |
+
tempo continuity · key continuity · spectral rolloff · stereo width
|
| 9 |
+
|
| 10 |
+
Each metric returns a value, a pass/fail against a threshold, and a short reason.
|
| 11 |
+
`verify(...)` returns a structured report; `plot_report(...)` writes a diagnostic
|
| 12 |
+
PNG (waveform + mel-spectrogram + seam-RMS overlay + a pass/fail table).
|
| 13 |
+
|
| 14 |
+
CLI: python verify.py <original> <finished> <source_seconds> [out.png]
|
| 15 |
+
"""
|
| 16 |
+
import sys
|
| 17 |
+
|
| 18 |
+
import librosa
|
| 19 |
+
import numpy as np
|
| 20 |
+
|
| 21 |
+
from analyze import _key_from_audio, _scalar_tempo
|
| 22 |
+
|
| 23 |
+
SR = 44100
|
| 24 |
+
|
| 25 |
+
# thresholds (plan §4)
|
| 26 |
+
SILENCE_WINDOW_S = 2.0
|
| 27 |
+
SILENCE_MIN_RATIO = 0.30 # every 2s window > 0.3x source RMS
|
| 28 |
+
CLIP_MAX_FRACTION = 1e-4 # < 0.01% of samples at full scale
|
| 29 |
+
LOUDNESS_TOL_DB = 6.0 # seam RMS step within +/- 6 dB
|
| 30 |
+
SEAM_JUMP_FACTOR = 6.0 # seam jump < 6x the 99.9th-pct local delta
|
| 31 |
+
TEMPO_TOL = 0.08 # generated tempo within +/-8% of source
|
| 32 |
+
ROLLOFF_MIN_RATIO = 0.70 # new rolloff >= 0.7x source
|
| 33 |
+
STEREO_CORR_RANGE = (0.10, 0.98) # not mono-collapsed, not decorrelated noise
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _mono(y):
|
| 37 |
+
return y.mean(axis=0) if y.ndim == 2 else y
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def _rms(x):
|
| 41 |
+
x = np.asarray(x, dtype=np.float64)
|
| 42 |
+
return float(np.sqrt(np.mean(x ** 2)) + 1e-12)
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def _db(ratio):
|
| 46 |
+
return 20.0 * np.log10(max(ratio, 1e-12))
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def _relative_keys(key):
|
| 50 |
+
"""key string + its relative major/minor, for continuity matching."""
|
| 51 |
+
names = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B']
|
| 52 |
+
try:
|
| 53 |
+
root, mode = key.split()
|
| 54 |
+
except ValueError:
|
| 55 |
+
return {key}
|
| 56 |
+
i = names.index(root)
|
| 57 |
+
out = {key}
|
| 58 |
+
if mode == 'major':
|
| 59 |
+
out.add(f'{names[(i + 9) % 12]} minor') # relative minor
|
| 60 |
+
else:
|
| 61 |
+
out.add(f'{names[(i + 3) % 12]} major') # relative major
|
| 62 |
+
return out
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def verify(original_path, finished_path, source_seconds, fade_seconds=4.0):
|
| 66 |
+
"""Return a report dict: {metrics: {name: {value, pass, reason}}, passed:bool,
|
| 67 |
+
and arrays for plotting}. `fade_seconds` is the stitch closing fade, excluded
|
| 68 |
+
from the silence-collapse scan so the intentional ending isn't flagged."""
|
| 69 |
+
fin, _ = librosa.load(finished_path, sr=SR, mono=False)
|
| 70 |
+
if fin.ndim == 1:
|
| 71 |
+
fin = np.stack([fin, fin])
|
| 72 |
+
fin_m = _mono(fin)
|
| 73 |
+
total_s = fin.shape[-1] / SR
|
| 74 |
+
|
| 75 |
+
boundary = int(round(source_seconds * SR))
|
| 76 |
+
boundary = max(0, min(boundary, fin.shape[-1] - 1))
|
| 77 |
+
src_region = fin_m[:boundary]
|
| 78 |
+
new_region = fin_m[boundary:]
|
| 79 |
+
|
| 80 |
+
src_rms = _rms(src_region) if len(src_region) else _rms(fin_m)
|
| 81 |
+
metrics = {}
|
| 82 |
+
|
| 83 |
+
# 1. duration sane
|
| 84 |
+
metrics['duration'] = {
|
| 85 |
+
'value': round(total_s, 2),
|
| 86 |
+
'pass': total_s > source_seconds + 1.0,
|
| 87 |
+
'reason': f'{total_s:.1f}s total, source {source_seconds:.1f}s',
|
| 88 |
+
}
|
| 89 |
+
|
| 90 |
+
# 2. silence collapse — every window in the new region carries energy.
|
| 91 |
+
# exclude the intentional closing fade (last fade_seconds) so a real ending
|
| 92 |
+
# isn't read as a collapse.
|
| 93 |
+
win = int(SILENCE_WINDOW_S * SR)
|
| 94 |
+
body = new_region[:max(0, len(new_region) - int((fade_seconds + 1.0) * SR))]
|
| 95 |
+
worst = 1.0
|
| 96 |
+
if len(body) >= win:
|
| 97 |
+
ratios = [_rms(body[i:i + win]) / src_rms
|
| 98 |
+
for i in range(0, len(body) - win + 1, win)]
|
| 99 |
+
worst = min(ratios) if ratios else 1.0
|
| 100 |
+
metrics['no_silence_collapse'] = {
|
| 101 |
+
'value': round(worst, 3),
|
| 102 |
+
'pass': worst > SILENCE_MIN_RATIO,
|
| 103 |
+
'reason': f'quietest 2s window = {worst:.2f}x source RMS '
|
| 104 |
+
f'(need > {SILENCE_MIN_RATIO})',
|
| 105 |
+
}
|
| 106 |
+
|
| 107 |
+
# 3. clipping
|
| 108 |
+
clip_frac = float(np.mean(np.abs(fin) >= 0.999))
|
| 109 |
+
metrics['no_clipping'] = {
|
| 110 |
+
'value': clip_frac,
|
| 111 |
+
'pass': clip_frac < CLIP_MAX_FRACTION,
|
| 112 |
+
'reason': f'{clip_frac*100:.4f}% samples at full scale',
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
# 4. loudness continuity across the seam
|
| 116 |
+
w = int(3.0 * SR)
|
| 117 |
+
pre = fin_m[max(0, boundary - w):boundary]
|
| 118 |
+
post = fin_m[boundary:boundary + w]
|
| 119 |
+
step_db = _db(_rms(post) / _rms(pre)) if len(pre) and len(post) else 0.0
|
| 120 |
+
metrics['loudness_continuity'] = {
|
| 121 |
+
'value': round(step_db, 2),
|
| 122 |
+
'pass': abs(step_db) < LOUDNESS_TOL_DB,
|
| 123 |
+
'reason': f'seam RMS step {step_db:+.1f} dB '
|
| 124 |
+
f'(tol +/-{LOUDNESS_TOL_DB:.0f})',
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
# 5. seam discontinuity — no audible click at the splice point. reference
|
| 128 |
+
# the seam's largest sample step against a HIGH QUANTILE of the surrounding
|
| 129 |
+
# |delta| distribution, not its std: on percussive/bright music the biggest
|
| 130 |
+
# single delta is heavy-tailed and dwarfs the std with no click present, so
|
| 131 |
+
# an std-based gate false-positives on drums/electronic continuations.
|
| 132 |
+
g = int(0.05 * SR)
|
| 133 |
+
seg = fin_m[max(0, boundary - g):boundary + g]
|
| 134 |
+
if len(seg) > 4:
|
| 135 |
+
diffs = np.abs(np.diff(seg))
|
| 136 |
+
local = np.abs(np.diff(fin_m[max(0, boundary - SR):boundary + SR]))
|
| 137 |
+
local_ref = float(np.quantile(local, 0.999) + 1e-9) if len(local) else 1.0
|
| 138 |
+
jump = float(diffs.max()) / local_ref
|
| 139 |
+
else:
|
| 140 |
+
jump = 0.0
|
| 141 |
+
metrics['no_seam_click'] = {
|
| 142 |
+
'value': round(jump, 2),
|
| 143 |
+
'pass': jump < SEAM_JUMP_FACTOR,
|
| 144 |
+
'reason': f'max seam jump {jump:.1f}x the 99.9th-pct local delta '
|
| 145 |
+
f'(need < {SEAM_JUMP_FACTOR})',
|
| 146 |
+
}
|
| 147 |
+
|
| 148 |
+
# 6/7. tempo + key continuity (source region vs new region)
|
| 149 |
+
src_for_analysis = src_region if len(src_region) > SR else fin_m
|
| 150 |
+
src_tempo = _scalar_tempo(librosa.beat.beat_track(y=src_for_analysis, sr=SR)[0])
|
| 151 |
+
new_tempo = _scalar_tempo(librosa.beat.beat_track(y=new_region, sr=SR)[0]) \
|
| 152 |
+
if len(new_region) > SR else src_tempo
|
| 153 |
+
tempo_dev = abs(new_tempo - src_tempo) / max(src_tempo, 1e-6)
|
| 154 |
+
# allow half/double-time relationship (common + musically valid)
|
| 155 |
+
half_double = min(tempo_dev,
|
| 156 |
+
abs(new_tempo - 2 * src_tempo) / max(2 * src_tempo, 1e-6),
|
| 157 |
+
abs(new_tempo - 0.5 * src_tempo) / max(0.5 * src_tempo, 1e-6))
|
| 158 |
+
metrics['tempo_continuity'] = {
|
| 159 |
+
'value': f'{src_tempo:.0f}->{new_tempo:.0f} bpm',
|
| 160 |
+
'pass': half_double < TEMPO_TOL,
|
| 161 |
+
'reason': f'{half_double*100:.1f}% deviation (tol {TEMPO_TOL*100:.0f}%, '
|
| 162 |
+
f'half/double allowed)',
|
| 163 |
+
}
|
| 164 |
+
|
| 165 |
+
src_key = _key_from_audio(src_for_analysis, SR)
|
| 166 |
+
new_key = _key_from_audio(new_region, SR) if len(new_region) > SR else src_key
|
| 167 |
+
metrics['key_continuity'] = {
|
| 168 |
+
'value': f'{src_key} -> {new_key}',
|
| 169 |
+
'pass': new_key in _relative_keys(src_key),
|
| 170 |
+
'reason': f'generated key {new_key} vs source {src_key} '
|
| 171 |
+
f'(relative maj/min ok)',
|
| 172 |
+
}
|
| 173 |
+
|
| 174 |
+
# 8. spectral rolloff — confirm the tail isn't band-limited
|
| 175 |
+
src_roll = float(np.mean(librosa.feature.spectral_rolloff(
|
| 176 |
+
y=src_for_analysis, sr=SR))) + 1e-6
|
| 177 |
+
new_roll = float(np.mean(librosa.feature.spectral_rolloff(
|
| 178 |
+
y=new_region, sr=SR))) if len(new_region) > SR else src_roll
|
| 179 |
+
metrics['spectral_rolloff'] = {
|
| 180 |
+
'value': f'{src_roll/1000:.1f}->{new_roll/1000:.1f} kHz',
|
| 181 |
+
'pass': new_roll >= ROLLOFF_MIN_RATIO * src_roll,
|
| 182 |
+
'reason': f'new {new_roll/1000:.1f}kHz vs source {src_roll/1000:.1f}kHz '
|
| 183 |
+
f'(need >= {ROLLOFF_MIN_RATIO:.0%})',
|
| 184 |
+
}
|
| 185 |
+
|
| 186 |
+
# 9. stereo width in the new region
|
| 187 |
+
if fin.shape[0] == 2 and fin.shape[-1] - boundary > SR:
|
| 188 |
+
L, R = fin[0, boundary:], fin[1, boundary:]
|
| 189 |
+
if np.std(L) > 1e-6 and np.std(R) > 1e-6:
|
| 190 |
+
corr = float(np.corrcoef(L, R)[0, 1])
|
| 191 |
+
else:
|
| 192 |
+
corr = 1.0
|
| 193 |
+
else:
|
| 194 |
+
corr = 1.0
|
| 195 |
+
lo, hi = STEREO_CORR_RANGE
|
| 196 |
+
metrics['stereo_width'] = {
|
| 197 |
+
'value': round(corr, 3),
|
| 198 |
+
'pass': lo <= corr <= hi or corr == 1.0,
|
| 199 |
+
'reason': f'L/R correlation {corr:.2f} (target {lo}-{hi}; '
|
| 200 |
+
f'1.0 = intentional mono)',
|
| 201 |
+
}
|
| 202 |
+
|
| 203 |
+
passed = all(m['pass'] for m in metrics.values())
|
| 204 |
+
return {
|
| 205 |
+
'metrics': metrics,
|
| 206 |
+
'passed': passed,
|
| 207 |
+
'finished': fin,
|
| 208 |
+
'boundary': boundary,
|
| 209 |
+
'source_seconds': source_seconds,
|
| 210 |
+
'total_seconds': total_s,
|
| 211 |
+
}
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
def print_report(report):
|
| 215 |
+
print("\n==== CODA VERIFY ====")
|
| 216 |
+
width = max(len(k) for k in report['metrics'])
|
| 217 |
+
for name, m in report['metrics'].items():
|
| 218 |
+
tag = 'PASS' if m['pass'] else 'FAIL'
|
| 219 |
+
print(f" [{tag}] {name:<{width}} {m['reason']}")
|
| 220 |
+
print(f" ----\n OVERALL: {'PASS' if report['passed'] else 'FAIL'}\n")
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
def plot_report(report, out_png):
|
| 224 |
+
import matplotlib
|
| 225 |
+
matplotlib.use("Agg")
|
| 226 |
+
import matplotlib.pyplot as plt
|
| 227 |
+
import librosa.display # submodule isn't pulled in by `import librosa`
|
| 228 |
+
|
| 229 |
+
fin = report['finished']
|
| 230 |
+
fin_m = _mono(fin)
|
| 231 |
+
b = report['boundary']
|
| 232 |
+
sr = SR
|
| 233 |
+
t = np.arange(fin_m.shape[-1]) / sr
|
| 234 |
+
|
| 235 |
+
fig, axes = plt.subplots(3, 1, figsize=(12, 9), facecolor="#0b0d10")
|
| 236 |
+
for ax in axes:
|
| 237 |
+
ax.set_facecolor("#14171c")
|
| 238 |
+
ax.tick_params(colors="#9aa4b2")
|
| 239 |
+
for s in ax.spines.values():
|
| 240 |
+
s.set_color("#2a2f37")
|
| 241 |
+
|
| 242 |
+
axes[0].plot(t, fin_m, color="#39d0d8", lw=0.4)
|
| 243 |
+
axes[0].axvline(b / sr, color="#ffb347", lw=1.5, label="seam")
|
| 244 |
+
axes[0].set_title("Finished waveform (seam marked)", color="#e6e9ef")
|
| 245 |
+
axes[0].legend(facecolor="#14171c", labelcolor="#e6e9ef")
|
| 246 |
+
|
| 247 |
+
S = librosa.feature.melspectrogram(y=fin_m, sr=sr, n_mels=128)
|
| 248 |
+
Sdb = librosa.power_to_db(S, ref=np.max)
|
| 249 |
+
librosa.display.specshow(Sdb, sr=sr, x_axis="time", y_axis="mel", ax=axes[1])
|
| 250 |
+
axes[1].axvline(b / sr, color="#ffb347", lw=1.5)
|
| 251 |
+
axes[1].set_title("Mel-spectrogram", color="#e6e9ef")
|
| 252 |
+
|
| 253 |
+
rows = [[k, ("PASS" if m['pass'] else "FAIL"), str(m['value'])]
|
| 254 |
+
for k, m in report['metrics'].items()]
|
| 255 |
+
axes[2].axis("off")
|
| 256 |
+
tbl = axes[2].table(cellText=rows,
|
| 257 |
+
colLabels=["metric", "result", "value"],
|
| 258 |
+
loc="center", cellLoc="left")
|
| 259 |
+
tbl.auto_set_font_size(False)
|
| 260 |
+
tbl.set_fontsize(9)
|
| 261 |
+
for (r, c), cell in tbl.get_celld().items():
|
| 262 |
+
cell.set_edgecolor("#2a2f37")
|
| 263 |
+
if r == 0:
|
| 264 |
+
cell.set_facecolor("#22272e")
|
| 265 |
+
cell.set_text_props(color="#e6e9ef")
|
| 266 |
+
else:
|
| 267 |
+
ok = report['metrics'][rows[r - 1][0]]['pass']
|
| 268 |
+
cell.set_facecolor("#14171c")
|
| 269 |
+
cell.set_text_props(color="#5fd38d" if ok else "#ff6b6b")
|
| 270 |
+
fig.suptitle(f"CODA verify — {'PASS' if report['passed'] else 'FAIL'}",
|
| 271 |
+
color="#e6e9ef", fontsize=14)
|
| 272 |
+
fig.tight_layout()
|
| 273 |
+
fig.savefig(out_png, dpi=110, facecolor="#0b0d10")
|
| 274 |
+
plt.close(fig)
|
| 275 |
+
print(f" wrote {out_png}")
|
| 276 |
+
|
| 277 |
+
|
| 278 |
+
if __name__ == "__main__":
|
| 279 |
+
if len(sys.argv) < 4:
|
| 280 |
+
print("usage: python verify.py <original> <finished> "
|
| 281 |
+
"<source_seconds> [out.png]")
|
| 282 |
+
sys.exit(1)
|
| 283 |
+
rep = verify(sys.argv[1], sys.argv[2], float(sys.argv[3]))
|
| 284 |
+
print_report(rep)
|
| 285 |
+
if len(sys.argv) > 4:
|
| 286 |
+
plot_report(rep, sys.argv[4])
|
| 287 |
+
sys.exit(0 if rep['passed'] else 2)
|