Spaces:
Running on Zero
Full pipeline overhaul: lyrics, poster, meter detection, staged UI
Browse files- analyze.py: time-signature estimation (beat-accent grid scoring, 4/4
bias), single-decode fingerprint()
- stems.py: shared in-process HT-Demucs vocal isolation
- transcribe.py: drop torchaudio/torchcodec path and demucs subprocess;
demucs-isolated vocals -> whisper large-v3 long-form (chunked), with a
hallucination filter on the output
- write_lyrics.py: qwen3 thinking mode off, think/scaffold scrubbing,
recommended non-thinking sampling (t=0.7, top_p=0.8, top_k=20)
- poster.py: vinyl-sleeve 1080x1080 poster with two-tone waveform
(original cream / continuation amber), started+finished dates,
key/bpm/meter, "completed by CODA" branding; upload waveform renderer
- app.py: wire the whole pipeline behind one button with a streamed
4-stage progress tracker; mastering+stitch moved off the GPU window;
separate ZeroGPU calls for music (300s) and lyrics (180s); waveform
panel, meter card, lyrics panels, poster output, original-date input;
gradio 6 Image buttons API; flatten group .styler backgrounds
- README: honest scope (instrumental continuation, lyrics as songwriting
aid), model stack table, frontmatter model links, research notes
- test_app_logic.py: 35 offline checks (meter, fingerprint, waveform,
poster, hallucination filter, qwen cleaner, stage tracker)
- README.md +82 -5
- RESEARCH_music_continuation_2026-06-10.md +163 -0
- analyze.py +69 -24
- app.py +309 -76
- poster.py +162 -65
- stems.py +65 -0
- test_app_logic.py +137 -0
- transcribe.py +97 -71
- write_lyrics.py +66 -33
|
@@ -10,16 +10,93 @@ app_file: app.py
|
|
| 10 |
pinned: false
|
| 11 |
license: mit
|
| 12 |
short_description: Local AI that finishes the songs you quit on.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
---
|
| 14 |
|
| 15 |
-
# CODA
|
| 16 |
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
-
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
---
|
| 24 |
|
| 25 |
-
Built by Tony Winslow for the Build Small Hackathon 2026.
|
|
|
|
| 10 |
pinned: false
|
| 11 |
license: mit
|
| 12 |
short_description: Local AI that finishes the songs you quit on.
|
| 13 |
+
models:
|
| 14 |
+
- facebook/musicgen-large
|
| 15 |
+
- openai/whisper-large-v3
|
| 16 |
+
- Qwen/Qwen3-8B
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# CODA — the songs you quit on, finished.
|
| 20 |
|
| 21 |
+
Everyone who's ever made music has one: the song that stops at 0:47. You ran out
|
| 22 |
+
of ideas at the bridge, the voice memo got buried, life happened. CODA takes that
|
| 23 |
+
unfinished clip and continues it into a full song — same tempo, same key, same
|
| 24 |
+
groove — then prints a vinyl-sleeve poster stamped with the date you started it
|
| 25 |
+
and the date it got finished.
|
| 26 |
|
| 27 |
+
*"AI gave me back the song I quit in 2018."*
|
| 28 |
|
| 29 |
+
## How it works
|
| 30 |
+
|
| 31 |
+
1. **Listen.** librosa detects your key, tempo, and time signature — pure DSP,
|
| 32 |
+
no ML.
|
| 33 |
+
2. **Separate.** HT-Demucs splits vocals from instrumentals. The instrumental
|
| 34 |
+
feeds MusicGen (vocal prompts are out-of-distribution for it — its training
|
| 35 |
+
data was Demucs-separated, so this matches how the model learned); the vocal
|
| 36 |
+
stem feeds Whisper.
|
| 37 |
+
3. **Continue.** MusicGen Large extends the music from the cutoff point, chained
|
| 38 |
+
in 12s-context / 18s-new passes inside its trained 30s window. By default
|
| 39 |
+
there's **no text prompt at all** — classifier-free guidance is off
|
| 40 |
+
(`guidance_scale=1.0`) and the audio prefix alone steers the generation.
|
| 41 |
+
Type a description and it switches to text-guided mode at Meta's trained
|
| 42 |
+
guidance of 3.0.
|
| 43 |
+
4. **Master & stitch.** matchering matches the continuation's loudness and EQ to
|
| 44 |
+
your recording, then the splice lands inside the codec-aligned overlap with
|
| 45 |
+
an equal-gain crossfade — no volume bump at the seam, and the track closes
|
| 46 |
+
with a proper fade-out.
|
| 47 |
+
5. **Write.** Whisper Large v3 transcribes whatever you sang; Qwen3 continues the
|
| 48 |
+
lyrics in the same voice — your next verse, on paper.
|
| 49 |
+
6. **Press.** A song poster: the full waveform (your part in cream, the
|
| 50 |
+
continuation in amber), key, BPM, meter, started/finished dates. That's the
|
| 51 |
+
thing you screenshot.
|
| 52 |
+
|
| 53 |
+
## The stack — small on purpose
|
| 54 |
+
|
| 55 |
+
| Model | Size | Job |
|
| 56 |
+
|---|---|---|
|
| 57 |
+
| [MusicGen Large](https://huggingface.co/facebook/musicgen-large) | 3.3B | music continuation, audio-prompted |
|
| 58 |
+
| [Whisper Large v3](https://huggingface.co/openai/whisper-large-v3) | 1.5B | lyric transcription from the vocal stem |
|
| 59 |
+
| [Qwen3 8B](https://huggingface.co/Qwen/Qwen3-8B) | 8.2B | lyric continuation in your style |
|
| 60 |
+
| HT-Demucs | tiny | vocal/instrumental separation, used twice |
|
| 61 |
+
| librosa | 0 params | key, tempo, meter detection |
|
| 62 |
+
|
| 63 |
+
~13B parameters total — well under the 32B Build Small cap, and every weight
|
| 64 |
+
runs locally. No cloud APIs, no subscriptions, nothing leaves the machine.
|
| 65 |
+
|
| 66 |
+
## What it won't pretend to do
|
| 67 |
+
|
| 68 |
+
MusicGen cannot generate vocals — Meta stripped voices from its training data
|
| 69 |
+
twice over, by design. CODA doesn't fake it: the musical continuation is
|
| 70 |
+
instrumental (your original vocals stay untouched up to the seam), and the
|
| 71 |
+
lyric continuation is delivered as **words on the page** — a songwriting aid,
|
| 72 |
+
not a cloned voice. The band plays on; you sing the next verse.
|
| 73 |
+
|
| 74 |
+
## Using it
|
| 75 |
+
|
| 76 |
+
Upload a WAV, MP3, or FLAC (a 20–60 second clip works best). You'll see the
|
| 77 |
+
detected key, tempo, and meter plus the waveform immediately. Pick how much new
|
| 78 |
+
music you want (15–120s), optionally describe the sound, optionally tell it when
|
| 79 |
+
you started the song, and hit **Finish this song**. The pipeline streams its
|
| 80 |
+
progress stage by stage; on ZeroGPU the music stage is the long one.
|
| 81 |
+
|
| 82 |
+
Run it locally:
|
| 83 |
+
|
| 84 |
+
```bash
|
| 85 |
+
pip install -r requirements.txt
|
| 86 |
+
python app.py
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
## Notes for the curious
|
| 90 |
+
|
| 91 |
+
The continuation settings come out of a deep dive into the transformers MusicGen
|
| 92 |
+
source: `guidance_scale` only amplifies the *text* condition (the audio prompt is
|
| 93 |
+
copied into both CFG branches and cancels out), so cranking "style adherence"
|
| 94 |
+
actually drags generation *away* from your audio and toward stock-music captions.
|
| 95 |
+
Full write-up in [`RESEARCH_music_continuation_2026-06-10.md`](RESEARCH_music_continuation_2026-06-10.md).
|
| 96 |
+
|
| 97 |
+
Code is MIT. MusicGen weights are CC-BY-NC (non-commercial) — fine here, worth
|
| 98 |
+
knowing if you fork this for production.
|
| 99 |
|
| 100 |
---
|
| 101 |
|
| 102 |
+
Built by Tony Winslow — Black Box Analytics — for the Build Small Hackathon 2026.
|
|
@@ -0,0 +1,163 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# CODA Music Continuation — Root-Cause Research & Recommendations
|
| 2 |
+
|
| 3 |
+
**Date:** 2026-06-10
|
| 4 |
+
**Scope:** Research only — no code changed. Sources: `transformers` source (modeling_musicgen.py, logits_process.py, feature_extraction_encodec.py), MusicGen paper (arXiv 2306.05284), audiocraft repo (model card, musicgen.py, genmodel.py, official demo notebook), facebook/musicgen-large generation_config.json, community issue threads, and June-2026 survey of alternative models.
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
## TL;DR
|
| 9 |
+
|
| 10 |
+
**The "weird funk pop" output is not a mystery — it's the predictable result of two settings interacting:**
|
| 11 |
+
|
| 12 |
+
1. **`guidance_scale=7.0` amplifies ONLY the text prompt, never the audio prompt.** Verified in the transformers source: classifier-free guidance duplicates the audio-prompt tokens into *both* the conditional and unconditional branches, so they cancel out of the CFG formula `uncond + scale * (cond - uncond)`. Only the text-encoder states are nulled in the unconditional branch. The "style adherence" slider is therefore a **text-adherence** knob — at 7 it drags generation *away* from what the audio implies and *toward* the text, at more than double Meta's trained value (3.0).
|
| 13 |
+
|
| 14 |
+
2. **The text prompt is functionally an empty caption, and MusicGen's prior for empty captions IS funk-pop stock music.** Training captions are stock-library metadata ("90s rock song with a guitar riff", genre/BPM/mood tags). A meta-instruction like *"continuation of the same song, identical instruments, identical timbre, same production and mixing, professional studio recording"* describes a *relationship to other audio*, which never appears in training captions — T5 embeds it as near-noise. The training corpus is overwhelmingly ShutterStock/Pond5 production-library music, so a vague caption regresses to generic Western stock instrumental. The per-pass section hints ("chorus, fuller arrangement, more energy") push the same direction.
|
| 15 |
+
|
| 16 |
+
So the current code cranks a 7× amplifier on a caption that means "generic stock music." That is precisely "weird funk pop bullshit."
|
| 17 |
+
|
| 18 |
+
**Also fundamental:** MusicGen **cannot continue vocals, ever.** Vocals were stripped from its training data twice (tag filtering, then HT-Demucs separation — stated verbatim in Meta's model card: "Vocals have been removed from the data source… The model is not able to generate realistic vocals"). The README's promise ("Vocals, instruments, lyrics — it picks up all of it") is not deliverable with MusicGen as the backbone.
|
| 19 |
+
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
## 1. Is MusicGen capable of what we want?
|
| 23 |
+
|
| 24 |
+
**Partially. Set expectations correctly:**
|
| 25 |
+
|
| 26 |
+
| Goal | Achievable? |
|
| 27 |
+
|---|---|
|
| 28 |
+
| Continue tempo, key, groove, rough genre | **Yes** — this is what the autoregressive token prefix carries |
|
| 29 |
+
| Same instrumentation, broadly ("a rock band stays a rock band") | **Mostly**, with correct settings (below) |
|
| 30 |
+
| *Identical* timbre/production ("same recording, same take") | **No.** Hard ceiling: everything passes through EnCodec at 2.2 kbps (4 codebooks × 11 bits × 50 Hz), 32 kHz mono. Real guitar/drums come back with smeared cymbals, softened transients, and the "AI sheen" before the LM even contributes |
|
| 31 |
+
| Continue vocals | **Never.** Vocal-free training data, by design |
|
| 32 |
+
|
| 33 |
+
Continuation was never a trained or paper-evaluated task — it's an emergent property of the autoregressive LM that audiocraft exposes as `generate_continuation()`. Meta's own demos prompt with only **0.5–2 seconds** of audio. The paper's limitations section concedes: "Our simple generation method does not allow us to have fine-grained control over adherence of the generation to the conditioning."
|
| 34 |
+
|
| 35 |
+
**Verdict:** with the fixes below, MusicGen-large can deliver "plausibly the same band keeps playing" for *instrumental* material. It cannot deliver "the same song with the same instruments" to a discerning ear, and it cannot touch vocals.
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## 2. How audio conditioning actually works (verified in transformers source)
|
| 40 |
+
|
| 41 |
+
Trace through `MusicgenForConditionalGeneration.generate()` (modeling_musicgen.py):
|
| 42 |
+
|
| 43 |
+
1. `input_values` (the waveform from the processor) → `_prepare_audio_encoder_kwargs_for_generation()` → `audio_encoder.encode()` → EnCodec RVQ codes → reshaped into `decoder_input_ids` (lines ~1744–1818). The audio prompt becomes a **frozen decoder token prefix** via `build_delay_pattern_mask`; generation only fills the empty slots after it.
|
| 44 |
+
2. **The output of `generate()` includes the prompt**, re-decoded through EnCodec (lossy). There is no slicing of the prefix. The current CODA code already accounts for this correctly (crossfading inside the overlap region) — keep that.
|
| 45 |
+
3. **CFG mechanics — the smoking gun.** `_prepare_text_encoder_kwargs_for_generation` (lines ~1732–1738): the unconditional branch is built by concatenating `torch.zeros_like(last_hidden_state)` — *text states only*. `prepare_inputs_for_generation` (lines ~1626–1631): `decoder_input_ids = decoder_input_ids.repeat((2, 1))` — *audio prompt copied into both branches*. `ClassifierFreeGuidanceLogitsProcessor`: `scores = uncond + (cond - uncond) * guidance_scale`. Since the audio prefix conditions both branches identically, **`guidance_scale` is purely a text knob. There is no knob in this implementation that increases audio-prompt fidelity.** Faithfulness to the audio comes from the prefix alone, and is *diluted* by strong text guidance.
|
| 46 |
+
4. **Defaults from facebook/musicgen-large `generation_config.json`:** `guidance_scale: 3.0`, `max_length: 1500` (= 30.0 s at 50 tokens/s), `do_sample: true`.
|
| 47 |
+
5. **Window math:** total sequence = prompt tokens + `max_new_tokens`. Nothing enforces the 1500-token (30 s) trained window when `max_new_tokens` is set — sinusoidal positional embeddings silently extrapolate with degraded output past it. Current CODA math (20 s prompt + 10 s new = exactly 1500) sits at the edge; correct, no headroom.
|
| 48 |
+
6. **Resampling:** the EnCodec feature extractor does **no resampling**; it hard-errors if `sampling_rate != 32000` is passed (good — CODA passes it) and silently produces pitch-shifted garbage if omitted. CODA's librosa resample-to-32k + `sampling_rate=MUSICGEN_SR` is correct.
|
| 49 |
+
7. **Shapes/padding:** mono 1-D float32 is the canonical input (a `(1, N)` array raises). `padding=True` with a single clip adds zero padding — safe. (Batching unequal-length prompts would inject silence tokens into the conditioning — avoid, but CODA doesn't batch.)
|
| 50 |
+
8. **fp32:** correct and necessary. The precision-sensitive part is EnCodec encode/decode and the T5 conditioning; community/literature confirm fp16 there produces the degraded "bitcrushed" output — this was the 8-bit failure mode, and the current fp32 fix is right. (LM-only bf16 would be a safe future speed optimization; full-fp16 is not.)
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## 3. What's wrong with the current `continue_music.py` — specifically
|
| 55 |
+
|
| 56 |
+
| # | Problem | Where | Severity |
|
| 57 |
+
|---|---|---|---|
|
| 58 |
+
| 1 | `guidance_scale=7.0` default — 2.3× Meta's trained value, amplifies text only, actively fights audio fidelity, and degrades audio quality (documented HF caveat) | `continue_track()` signature; `app.py` slider (min 3 / default 7 / max 10) | **Critical — primary cause of "funk pop"** |
|
| 59 |
+
| 2 | Meta-instruction text prompt ("identical instruments, identical timbre…") is out-of-distribution → behaves as a vague caption → stock-music prior | `base` string in `continue_track()` | **Critical — the other half of the same bug** |
|
| 60 |
+
| 3 | Per-pass section hints ("chorus, fuller arrangement, more energy", "big chorus, peak energy") are genre-generic captions injected at guidance 7 — each chained pass gets pulled further toward stock pop | `SECTION_ARC` / `_section_plan()` | **High** |
|
| 61 |
+
| 4 | The "style adherence" slider label is inverted from reality: raising it makes output *more generic*, not more faithful | `app.py` | High (UX/diagnosis trap) |
|
| 62 |
+
| 5 | 6 chained passes for 60 s (10 s new per pass) — each pass re-encodes generated audio and re-rolls the dice; drift compounds per pass. audiocraft's own extend loop uses `extend_stride=18` (12 s context, 18 s new) = ~3 passes for 60 s | `per_pass` math in `continue_track()` | Medium |
|
| 63 |
+
| 6 | Vocal input goes straight into the prompt. MusicGen has never seen a voice; it will drop, garble, or replace it | `_load_tail()` / pipeline design | **Critical for any vocal demo** |
|
| 64 |
+
| 7 | No timbre/EQ post-matching — generated 32 kHz-band-limited mono is spliced against a 44.1 kHz master with only an RMS gain match, so even a good continuation reads as "different record" | `stitch_with_crossfade()` | Medium |
|
| 65 |
+
|
| 66 |
+
What's already **correct** (don't touch): fp32 + TF32, librosa resample to 32 kHz with `sampling_rate=` passed, mono 1-D float32 input, crossfade inside the codec-roundtripped overlap, equal-gain seam for correlated material, resampling generated audio *up* to the original's rate, window math 20 s + 10 s = 1500 tokens.
|
| 67 |
+
|
| 68 |
+
---
|
| 69 |
+
|
| 70 |
+
## 4. The fix — exact changes
|
| 71 |
+
|
| 72 |
+
### Fix 1 (do first): guidance + text — `continue_music.py`
|
| 73 |
+
|
| 74 |
+
**Two supported modes; implement (a) as default, (b) as the option:**
|
| 75 |
+
|
| 76 |
+
**(a) Pure continuation — no text, no CFG.** This is the canonical mode: Meta's MusicGen-Style page describes continuation as "given the conditioning, MusicGen continues it **without any textual description**", and audiocraft's `generate_continuation(descriptions=None)` is the reference implementation. Training used 20% condition dropout, so unconditional is in-distribution.
|
| 77 |
+
|
| 78 |
+
```python
|
| 79 |
+
# default path: audio prompt only
|
| 80 |
+
inputs = processor(
|
| 81 |
+
audio=prompt_audio,
|
| 82 |
+
sampling_rate=MUSICGEN_SR,
|
| 83 |
+
return_tensors="pt",
|
| 84 |
+
).to(model.device)
|
| 85 |
+
# merge in null text conditioning so the seq2seq generate path is well-formed
|
| 86 |
+
uncond = model.get_unconditional_inputs(num_samples=1)
|
| 87 |
+
output = model.generate(
|
| 88 |
+
**inputs,
|
| 89 |
+
encoder_outputs=None, # see note below
|
| 90 |
+
do_sample=True,
|
| 91 |
+
guidance_scale=1.0, # CFG off — nothing for it to amplify anyway
|
| 92 |
+
temperature=1.0,
|
| 93 |
+
top_k=250, # audiocraft defaults
|
| 94 |
+
max_new_tokens=max_new_tokens,
|
| 95 |
+
)
|
| 96 |
+
```
|
| 97 |
+
(Implementation note: the clean way in transformers is `model.generate(input_values=inputs.input_values, padding_mask=inputs.padding_mask, **vars-from-get_unconditional_inputs-minus-duplicates, guidance_scale=1.0, ...)` — or simply keep passing `text=[""]` through the processor with `guidance_scale=1.0`, which is equivalent in effect since CFG is off. Verify both on a test clip; keep whichever is stable on the installed transformers version. **Pin transformers < 4.57** if you ever touch the melody variant — melody conditioning silently broke after 4.48, GitHub issue #45647.)
|
| 98 |
+
|
| 99 |
+
**(b) Described continuation — user-supplied caption, guidance 3.0.** If the user types a real description ("garage rock, distorted electric guitar, live drums, male vocals removed, 120 bpm, E minor"), that caption is in-distribution and CFG at Meta's trained value 3.0 helps hold the genre. Auto-append detected key/BPM as now. **Never auto-generate meta-instructions.**
|
| 100 |
+
|
| 101 |
+
- Change `continue_track()` default `guidance_scale=7.0` → `1.0`; treat >1 as only valid when a user caption exists.
|
| 102 |
+
- Delete the `base` meta-instruction string.
|
| 103 |
+
- **Delete `SECTION_ARC` and `_section_plan()`** (or reduce to a single final-pass "outro, song ending" hint *only* in described mode). The audio chaining carries continuity; the hints only inject stock-pop pull.
|
| 104 |
+
|
| 105 |
+
### Fix 2: UI — `app.py`
|
| 106 |
+
|
| 107 |
+
- Replace the "style adherence" slider (currently min 3, so even its minimum is text-amplifying) with:
|
| 108 |
+
- a free-text **"describe your song (optional)"** box → mode (b) when filled, mode (a) when empty;
|
| 109 |
+
- if a guidance slider is kept at all: range 1.0–5.0, default 1.0 (no text) / 3.0 (with text), labeled **"text influence"**.
|
| 110 |
+
|
| 111 |
+
### Fix 3: vocals — Demucs gate (Demucs is already in requirements.txt)
|
| 112 |
+
|
| 113 |
+
Before building the prompt: run HT-Demucs on the tail, measure vocal-stem energy. If vocals are present:
|
| 114 |
+
1. Continue **the instrumental mix only** (this matches how the training data itself was prepared — HT-Demucs-separated instrumentals are exactly in-distribution).
|
| 115 |
+
2. Keep the user's original (with vocals) untouched up to the seam; the continuation is instrumental.
|
| 116 |
+
3. Update the README honestly: lyrics from Qwen3 are a *songwriting aid* deliverable, not synthesized vocals. (No open model on a ZeroGPU budget will clone-and-continue the user's voice well — YuE-extend is the closest and it's slow and finicky.)
|
| 117 |
+
|
| 118 |
+
### Fix 4: chaining — match audiocraft's reference loop
|
| 119 |
+
|
| 120 |
+
Keep first-pass prompt at 15–20 s (community sweet spot for fidelity). For subsequent passes use **12 s of context / 18 s new** (audiocraft's `extend_stride=18` default): 60 s of new audio in 3 passes instead of 6 — half the drift opportunities, half the GPU time. Keep total tokens ≤ 1500 per pass.
|
| 121 |
+
|
| 122 |
+
### Fix 5: post-match the seam — matchering
|
| 123 |
+
|
| 124 |
+
Add [`matchering`](https://github.com/sergree/matchering) (pure-Python, lightweight): REFERENCE = the original clip, TARGET = the generated continuation, before `stitch_with_crossfade()`. It matches RMS, spectral envelope, and peak — this is the single cheapest improvement to "sounds like the same record," and it's exactly what an audio engineer would do at a session splice: match the master, then crossfade on a musical boundary. (Optional polish: use the detected BPM to snap the seam to a downbeat.) Note it cannot invent content above 16 kHz — the 32 kHz EnCodec ceiling stands.
|
| 125 |
+
|
| 126 |
+
### Expected outcome after Fixes 1–5
|
| 127 |
+
|
| 128 |
+
Tempo/key/groove continuity, broadly matching instrumentation, no genre teleport, production glued by EQ-match. Still mono-derived, still slightly "AI-sheened," still instrumental-only. That is MusicGen's ceiling.
|
| 129 |
+
|
| 130 |
+
---
|
| 131 |
+
|
| 132 |
+
## 5. If the ceiling isn't good enough: the 2026 alternatives
|
| 133 |
+
|
| 134 |
+
Ranked for CODA's use case (faithful extension of arbitrary user audio, single ZeroGPU slice):
|
| 135 |
+
|
| 136 |
+
1. **ACE-Step 1.5** (Jan 2026; Apache/MIT-style, 8–16 GB VRAM) — the only open model where "**extend** my actual audio, keep timbre" is a designed first-class feature (`extend`, `repaint`, `audio_cover_strength`), **with vocal support in 50+ languages**. Best single-model replacement candidate; chain repaints for arbitrary length. Caveat: limited independent benchmarking yet.
|
| 137 |
+
2. **Stable Audio 3 Medium** (May 2026; 1.4B, ~6 GB, Stability Community License) — documented causal continuation/inpainting mode, 44.1 kHz **stereo** (fixes the mono+bandwidth ceiling). Unproven on vocals. Worth an A/B against ACE-Step on instrumental material.
|
| 138 |
+
3. **YuE-extend** — true continuation with voice cloning from separated stems; best vocal continuity in open weights, but slow (minutes/song) and ≤30 s prompt practical limit.
|
| 139 |
+
4. **MusicGen-Style** (facebook/musicgen-style) — conditions on a 1.5–4.5 s excerpt for timbre/style match; combining it with a continuation prefix is theoretically possible in audiocraft but undocumented/unvalidated. Research-grade only.
|
| 140 |
+
5. **Per-stem MusicGen continuation: don't.** Isolated stems are out-of-distribution for MusicGen (literature: models "produce nonsensical outputs when fed isolated vocals" — SingSong); only stem-native models (MusicGen-Stem, StemGen) do this, and their weights are research releases.
|
| 141 |
+
|
| 142 |
+
**License flag:** MusicGen weights are **CC-BY-NC** (non-commercial). Fine for the hackathon; a problem if CODA ever ships commercially. ACE-Step and Stable Audio 3 (under $1M revenue) are both more permissive.
|
| 143 |
+
|
| 144 |
+
**Recommended path:** ship the hackathon on MusicGen with Fixes 1–5 (small, low-risk diffs), and prototype ACE-Step 1.5 `extend` as CODA v2's backbone — it is designed to do exactly what CODA promises, including vocals.
|
| 145 |
+
|
| 146 |
+
---
|
| 147 |
+
|
| 148 |
+
## 6. Failure modes — explained
|
| 149 |
+
|
| 150 |
+
| Symptom | Cause | Status |
|
| 151 |
+
|---|---|---|
|
| 152 |
+
| 8-bit / chiptune output | fp16 EnCodec/T5 path (precision-sensitive); confirmed pattern in community + literature (TinyMusician keeps EnCodec fp32 for this reason) | **Fixed** by the fp32 change — keep it |
|
| 153 |
+
| Random new instruments | Text guidance pulling toward caption prior + per-pass re-rolls in 6-pass chaining | Fixes 1, 4 |
|
| 154 |
+
| Generic MusicGen-default sound | Vague/meta caption → stock-library prior (training data literally is stock music) | Fixes 1, 2 |
|
| 155 |
+
| "Weird funk pop" | guidance 7 × meaningless caption × "more energy" section hints = maximum pull toward the stock-pop prior | Fixes 1, 2 (delete SECTION_ARC) |
|
| 156 |
+
| ~14 s output | Deployment issue, already fixed | — |
|
| 157 |
+
| Vocals garbled/vanish | Vocal-free training data — model-level impossibility | Fix 3 (Demucs gate) |
|
| 158 |
+
|
| 159 |
+
---
|
| 160 |
+
|
| 161 |
+
## 7. What a professional audio engineer would say
|
| 162 |
+
|
| 163 |
+
A session engineer asked to "extend this demo" would: comp from the same takes, or re-amp/re-track matching the original chain, then **match the master** (EQ/loudness/width to the reference) and **edit on the grid** (splice at a downbeat, short crossfade). The transferable lessons for CODA: (1) never expect a different "performer" (the model) to nail the take — minimize how much it has to invent (longer audio context, no contradicting text); (2) the *master-matching* step is half of what makes a splice invisible — that's the matchering pass; (3) cut on musical boundaries — use the BPM grid CODA already detects to place the seam on a downbeat; (4) when a part can't be reproduced (vocals), you don't fake it — you arrange around it (instrumental continuation).
|
|
@@ -1,6 +1,7 @@
|
|
|
|
|
| 1 |
import librosa
|
| 2 |
import numpy as np
|
| 3 |
-
|
| 4 |
|
| 5 |
# krumhansl-schmuckler key profiles
|
| 6 |
# major and minor correlation vectors for pitch class distribution
|
|
@@ -13,12 +14,12 @@ MINOR_PROFILE = np.array([6.33, 2.68, 3.52, 5.38, 2.60, 3.53,
|
|
| 13 |
PITCH_CLASSES = ['C', 'C#', 'D', 'D#', 'E', 'F',
|
| 14 |
'F#', 'G', 'G#', 'A', 'A#', 'B']
|
| 15 |
|
|
|
|
|
|
|
| 16 |
|
| 17 |
-
def find_key(path):
|
| 18 |
-
"""chroma-based key detection using krumhansl-schmuckler profiles"""
|
| 19 |
-
track, sr = librosa.load(path, sr=None)
|
| 20 |
|
| 21 |
-
|
|
|
|
| 22 |
chroma = librosa.feature.chroma_cqt(y=track, sr=sr)
|
| 23 |
pitch_dist = np.mean(chroma, axis=1)
|
| 24 |
|
|
@@ -48,40 +49,83 @@ def find_key(path):
|
|
| 48 |
return best_key
|
| 49 |
|
| 50 |
|
| 51 |
-
def
|
| 52 |
-
track, sr = librosa.load(path, sr=None)
|
| 53 |
-
tempo, _ = librosa.beat.beat_track(y=track, sr=sr)
|
| 54 |
# librosa sometimes returns an array
|
| 55 |
if hasattr(tempo, '__len__'):
|
| 56 |
return float(tempo[0])
|
| 57 |
return float(tempo)
|
| 58 |
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
def get_duration(path):
|
| 61 |
return librosa.get_duration(path=path)
|
| 62 |
|
| 63 |
|
| 64 |
def fingerprint(path):
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
channels = 1 if track.ndim == 1 else track.shape[0]
|
| 68 |
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
mono = librosa.to_mono(track)
|
| 72 |
-
else:
|
| 73 |
-
mono = track
|
| 74 |
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
return {
|
| 80 |
-
'key':
|
| 81 |
-
'bpm': round(
|
| 82 |
-
'
|
| 83 |
-
'
|
| 84 |
-
'
|
|
|
|
| 85 |
}
|
| 86 |
|
| 87 |
|
|
@@ -94,6 +138,7 @@ if __name__ == '__main__':
|
|
| 94 |
info = fingerprint(sys.argv[1])
|
| 95 |
print(f"key: {info['key']}")
|
| 96 |
print(f"bpm: {info['bpm']}")
|
|
|
|
| 97 |
print(f"duration: {info['duration']}s")
|
| 98 |
print(f"sample rate: {info['sample_rate']}Hz")
|
| 99 |
print(f"channels: {info['channels']}")
|
|
|
|
| 1 |
+
"""key / tempo / meter detection. pure librosa dsp — no ML."""
|
| 2 |
import librosa
|
| 3 |
import numpy as np
|
| 4 |
+
import soundfile as sf
|
| 5 |
|
| 6 |
# krumhansl-schmuckler key profiles
|
| 7 |
# major and minor correlation vectors for pitch class distribution
|
|
|
|
| 14 |
PITCH_CLASSES = ['C', 'C#', 'D', 'D#', 'E', 'F',
|
| 15 |
'F#', 'G', 'G#', 'A', 'A#', 'B']
|
| 16 |
|
| 17 |
+
# all the analysis here works on envelopes and chroma; 22k mono is plenty
|
| 18 |
+
ANALYSIS_SR = 22050
|
| 19 |
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
+
def _key_from_audio(track, sr):
|
| 22 |
+
"""chroma-based key detection using krumhansl-schmuckler profiles"""
|
| 23 |
chroma = librosa.feature.chroma_cqt(y=track, sr=sr)
|
| 24 |
pitch_dist = np.mean(chroma, axis=1)
|
| 25 |
|
|
|
|
| 49 |
return best_key
|
| 50 |
|
| 51 |
|
| 52 |
+
def _scalar_tempo(tempo):
|
|
|
|
|
|
|
| 53 |
# librosa sometimes returns an array
|
| 54 |
if hasattr(tempo, '__len__'):
|
| 55 |
return float(tempo[0])
|
| 56 |
return float(tempo)
|
| 57 |
|
| 58 |
|
| 59 |
+
def _time_signature_from_beats(onset_env, beats):
|
| 60 |
+
"""
|
| 61 |
+
estimate the meter from where the accents land: take the onset strength
|
| 62 |
+
at each tracked beat and score how well a "downbeat every N beats" grid
|
| 63 |
+
lines up with the loud ones, for N = 3 and 4. waltz time has to win
|
| 64 |
+
clearly — ambiguous material is called 4/4, like most music.
|
| 65 |
+
"""
|
| 66 |
+
if len(beats) < 12:
|
| 67 |
+
return "4/4"
|
| 68 |
+
|
| 69 |
+
strengths = onset_env[beats].astype(float)
|
| 70 |
+
spread = strengths.std()
|
| 71 |
+
if spread < 1e-8:
|
| 72 |
+
return "4/4"
|
| 73 |
+
strengths = (strengths - strengths.mean()) / spread
|
| 74 |
+
|
| 75 |
+
def accent_score(meter):
|
| 76 |
+
# best phase alignment of the downbeat grid
|
| 77 |
+
return max(float(np.mean(strengths[offset::meter]))
|
| 78 |
+
for offset in range(meter))
|
| 79 |
+
|
| 80 |
+
three, four = accent_score(3), accent_score(4)
|
| 81 |
+
if three > 0 and three > four * 1.25:
|
| 82 |
+
return "3/4"
|
| 83 |
+
return "4/4"
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def find_key(path):
|
| 87 |
+
track, sr = librosa.load(path, sr=ANALYSIS_SR, mono=True)
|
| 88 |
+
return _key_from_audio(track, sr)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def get_tempo(path):
|
| 92 |
+
track, sr = librosa.load(path, sr=ANALYSIS_SR, mono=True)
|
| 93 |
+
tempo, _ = librosa.beat.beat_track(y=track, sr=sr)
|
| 94 |
+
return _scalar_tempo(tempo)
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def get_time_signature(path):
|
| 98 |
+
track, sr = librosa.load(path, sr=ANALYSIS_SR, mono=True)
|
| 99 |
+
onset_env = librosa.onset.onset_strength(y=track, sr=sr)
|
| 100 |
+
_, beats = librosa.beat.beat_track(onset_envelope=onset_env, sr=sr)
|
| 101 |
+
return _time_signature_from_beats(onset_env, beats)
|
| 102 |
+
|
| 103 |
+
|
| 104 |
def get_duration(path):
|
| 105 |
return librosa.get_duration(path=path)
|
| 106 |
|
| 107 |
|
| 108 |
def fingerprint(path):
|
| 109 |
+
"""one-stop analysis: key, bpm, meter, duration — single decode pass."""
|
| 110 |
+
track, sr = librosa.load(path, sr=ANALYSIS_SR, mono=True)
|
|
|
|
| 111 |
|
| 112 |
+
onset_env = librosa.onset.onset_strength(y=track, sr=sr)
|
| 113 |
+
tempo, beats = librosa.beat.beat_track(onset_envelope=onset_env, sr=sr)
|
|
|
|
|
|
|
|
|
|
| 114 |
|
| 115 |
+
# source-file metadata without a second full decode
|
| 116 |
+
try:
|
| 117 |
+
meta = sf.info(path)
|
| 118 |
+
native_sr, channels = meta.samplerate, meta.channels
|
| 119 |
+
except Exception:
|
| 120 |
+
native_sr, channels = sr, 1
|
| 121 |
|
| 122 |
return {
|
| 123 |
+
'key': _key_from_audio(track, sr),
|
| 124 |
+
'bpm': round(_scalar_tempo(tempo), 1),
|
| 125 |
+
'time_signature': _time_signature_from_beats(onset_env, beats),
|
| 126 |
+
'duration': round(len(track) / sr, 2),
|
| 127 |
+
'sample_rate': native_sr,
|
| 128 |
+
'channels': channels,
|
| 129 |
}
|
| 130 |
|
| 131 |
|
|
|
|
| 138 |
info = fingerprint(sys.argv[1])
|
| 139 |
print(f"key: {info['key']}")
|
| 140 |
print(f"bpm: {info['bpm']}")
|
| 141 |
+
print(f"time signature: {info['time_signature']} (estimated)")
|
| 142 |
print(f"duration: {info['duration']}s")
|
| 143 |
print(f"sample rate: {info['sample_rate']}Hz")
|
| 144 |
print(f"channels: {info['channels']}")
|
|
@@ -1,9 +1,12 @@
|
|
| 1 |
-
import gradio as gr
|
| 2 |
import os
|
| 3 |
import tempfile
|
| 4 |
-
|
|
|
|
|
|
|
| 5 |
import soundfile as sf
|
| 6 |
-
|
|
|
|
|
|
|
| 7 |
|
| 8 |
try:
|
| 9 |
import spaces
|
|
@@ -69,6 +72,14 @@ html, body, gradio-app, .gradio-container {
|
|
| 69 |
}
|
| 70 |
|
| 71 |
/* ---- panels ---- */
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
.upload-panel, .results-panel, .continue-panel {
|
| 73 |
background: var(--coda-surface) !important;
|
| 74 |
border: 1px solid var(--coda-border) !important;
|
|
@@ -136,6 +147,59 @@ html, body, gradio-app, .gradio-container {
|
|
| 136 |
line-height: 1.1 !important;
|
| 137 |
}
|
| 138 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
/* ---- buttons ---- */
|
| 140 |
.gradio-container button.primary {
|
| 141 |
background: var(--coda-accent) !important;
|
|
@@ -221,26 +285,26 @@ def make_stat_html(label, value):
|
|
| 221 |
)
|
| 222 |
|
| 223 |
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
gr.update(visible=False),
|
| 231 |
-
None,
|
| 232 |
-
None
|
| 233 |
-
)
|
| 234 |
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 244 |
|
| 245 |
|
| 246 |
def _output_subtype(audio_path):
|
|
@@ -258,59 +322,195 @@ def _output_subtype(audio_path):
|
|
| 258 |
|
| 259 |
|
| 260 |
@spaces.GPU(duration=300)
|
| 261 |
-
def
|
| 262 |
-
|
| 263 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 264 |
|
| 265 |
-
try:
|
| 266 |
-
# lazy import so the UI boots fast and torch only loads on demand
|
| 267 |
-
import librosa
|
| 268 |
-
from continue_music import (continue_track, master_match,
|
| 269 |
-
stitch_with_crossfade)
|
| 270 |
-
|
| 271 |
-
key = info.get("key") if info else None
|
| 272 |
-
bpm = info.get("bpm") if info else None
|
| 273 |
-
|
| 274 |
-
# generate at musicgen's native 32kHz, chaining passes for long
|
| 275 |
-
# extensions. no caption = pure audio continuation (CFG off); a user
|
| 276 |
-
# caption switches to described mode at guidance 3.0.
|
| 277 |
-
extended_tail, prompt_samples, gen_sr = continue_track(
|
| 278 |
-
audio_path,
|
| 279 |
-
gen_duration=float(gen_seconds),
|
| 280 |
-
key=key,
|
| 281 |
-
bpm=bpm,
|
| 282 |
-
fade_seconds=float(fade_seconds),
|
| 283 |
-
caption=caption or "",
|
| 284 |
-
)
|
| 285 |
|
| 286 |
-
|
| 287 |
-
|
| 288 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 289 |
|
| 290 |
-
|
| 291 |
-
|
| 292 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 293 |
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 297 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 298 |
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
|
| 306 |
-
|
| 307 |
-
|
| 308 |
-
|
| 309 |
-
|
| 310 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 311 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
except Exception as e:
|
| 313 |
-
|
|
|
|
| 314 |
|
| 315 |
|
| 316 |
HEADER_HTML = (
|
|
@@ -349,7 +549,12 @@ with gr.Blocks(title="CODA") as app:
|
|
| 349 |
with gr.Row():
|
| 350 |
key_display = gr.HTML(make_stat_html("key", "---"))
|
| 351 |
bpm_display = gr.HTML(make_stat_html("tempo", "---"))
|
| 352 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 353 |
|
| 354 |
with gr.Group(elem_classes="continue-panel", visible=False) as continue_section:
|
| 355 |
gr.HTML('<div class="section-label">Continue</div>')
|
|
@@ -369,20 +574,49 @@ with gr.Blocks(title="CODA") as app:
|
|
| 369 |
placeholder="leave empty to continue from the audio alone — or e.g. \"garage rock, distorted electric guitar, live drums\"",
|
| 370 |
lines=1,
|
| 371 |
)
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 375 |
|
| 376 |
audio_input.change(
|
| 377 |
fn=analyze_track,
|
| 378 |
inputs=[audio_input],
|
| 379 |
-
outputs=[key_display, bpm_display,
|
|
|
|
|
|
|
| 380 |
)
|
| 381 |
|
| 382 |
continue_btn.click(
|
| 383 |
-
fn=
|
| 384 |
-
inputs=[current_track, track_info, gen_length, fade_length,
|
| 385 |
-
|
|
|
|
|
|
|
|
|
|
| 386 |
)
|
| 387 |
|
| 388 |
gr.HTML(FOOTER_HTML)
|
|
@@ -390,4 +624,3 @@ with gr.Blocks(title="CODA") as app:
|
|
| 390 |
|
| 391 |
if __name__ == "__main__":
|
| 392 |
app.launch(theme=gr.themes.Base(), css=CUSTOM_CSS)
|
| 393 |
-
|
|
|
|
|
|
|
| 1 |
import os
|
| 2 |
import tempfile
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
|
| 5 |
+
import gradio as gr
|
| 6 |
import soundfile as sf
|
| 7 |
+
|
| 8 |
+
from analyze import fingerprint
|
| 9 |
+
from poster import make_poster, render_waveform_png
|
| 10 |
|
| 11 |
try:
|
| 12 |
import spaces
|
|
|
|
| 72 |
}
|
| 73 |
|
| 74 |
/* ---- panels ---- */
|
| 75 |
+
/* gradio 6 wraps group children in a .styler div with a zinc background;
|
| 76 |
+
flatten it (and row blocks / forms) so the panel surface reads as one */
|
| 77 |
+
.gradio-container .styler,
|
| 78 |
+
.gradio-container .row .block,
|
| 79 |
+
.gradio-container .form {
|
| 80 |
+
background: transparent !important;
|
| 81 |
+
border: none !important;
|
| 82 |
+
}
|
| 83 |
.upload-panel, .results-panel, .continue-panel {
|
| 84 |
background: var(--coda-surface) !important;
|
| 85 |
border: 1px solid var(--coda-border) !important;
|
|
|
|
| 147 |
line-height: 1.1 !important;
|
| 148 |
}
|
| 149 |
|
| 150 |
+
/* ---- waveform & poster images ---- */
|
| 151 |
+
.waveform-img { margin-top: 14px; }
|
| 152 |
+
.waveform-img img { border-radius: 12px; width: 100%; }
|
| 153 |
+
.poster-img img { border-radius: 12px; }
|
| 154 |
+
.gradio-container .image-container {
|
| 155 |
+
background: var(--coda-surface-2) !important;
|
| 156 |
+
border-color: var(--coda-border) !important;
|
| 157 |
+
}
|
| 158 |
+
|
| 159 |
+
/* ---- stage tracker ---- */
|
| 160 |
+
.stage-list {
|
| 161 |
+
display: flex;
|
| 162 |
+
flex-direction: column;
|
| 163 |
+
gap: 9px;
|
| 164 |
+
padding: 14px 4px 4px 4px;
|
| 165 |
+
}
|
| 166 |
+
.stage {
|
| 167 |
+
display: flex;
|
| 168 |
+
align-items: center;
|
| 169 |
+
gap: 11px;
|
| 170 |
+
color: var(--coda-muted) !important;
|
| 171 |
+
font-size: 0.88em;
|
| 172 |
+
letter-spacing: 0.01em;
|
| 173 |
+
}
|
| 174 |
+
.stage span { color: inherit !important; }
|
| 175 |
+
.stage .dot {
|
| 176 |
+
width: 9px; height: 9px;
|
| 177 |
+
border-radius: 50%;
|
| 178 |
+
background: #3a4150;
|
| 179 |
+
flex: none;
|
| 180 |
+
transition: background 0.2s ease;
|
| 181 |
+
}
|
| 182 |
+
.stage.active { color: var(--coda-text) !important; }
|
| 183 |
+
.stage.active .dot {
|
| 184 |
+
background: var(--coda-accent);
|
| 185 |
+
animation: coda-pulse 1.2s ease-in-out infinite;
|
| 186 |
+
}
|
| 187 |
+
.stage.done .dot { background: var(--coda-accent); }
|
| 188 |
+
.stage.skip { opacity: 0.55; }
|
| 189 |
+
.stage.error { color: #e07a6a !important; }
|
| 190 |
+
.stage.error .dot { background: #e07a6a; }
|
| 191 |
+
@keyframes coda-pulse {
|
| 192 |
+
0%, 100% { box-shadow: 0 0 0 0 rgba(226,168,92,0.45); }
|
| 193 |
+
50% { box-shadow: 0 0 0 6px rgba(226,168,92,0); }
|
| 194 |
+
}
|
| 195 |
+
.stage-note {
|
| 196 |
+
color: var(--coda-muted) !important;
|
| 197 |
+
font-size: 0.8em;
|
| 198 |
+
padding: 10px 4px 0 4px;
|
| 199 |
+
line-height: 1.5;
|
| 200 |
+
}
|
| 201 |
+
.stage-note span { color: inherit !important; }
|
| 202 |
+
|
| 203 |
/* ---- buttons ---- */
|
| 204 |
.gradio-container button.primary {
|
| 205 |
background: var(--coda-accent) !important;
|
|
|
|
| 285 |
)
|
| 286 |
|
| 287 |
|
| 288 |
+
PIPELINE_STAGES = (
|
| 289 |
+
"separate & continue the music",
|
| 290 |
+
"master & stitch the seam",
|
| 291 |
+
"transcribe & extend the lyrics",
|
| 292 |
+
"press the poster",
|
| 293 |
+
)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 294 |
|
| 295 |
+
|
| 296 |
+
def stages_html(states, note="", labels=None):
|
| 297 |
+
"""render the pipeline progress tracker. states are per-stage:
|
| 298 |
+
pending | active | done | skip | error."""
|
| 299 |
+
labels = labels or PIPELINE_STAGES
|
| 300 |
+
rows = []
|
| 301 |
+
for label, state in zip(labels, states):
|
| 302 |
+
rows.append(
|
| 303 |
+
f'<div class="stage {state}"><span class="dot"></span>'
|
| 304 |
+
f'<span>{label}</span></div>'
|
| 305 |
+
)
|
| 306 |
+
note_html = f'<div class="stage-note">{note}</div>' if note else ""
|
| 307 |
+
return f'<div class="stage-list">{"".join(rows)}</div>{note_html}'
|
| 308 |
|
| 309 |
|
| 310 |
def _output_subtype(audio_path):
|
|
|
|
| 322 |
|
| 323 |
|
| 324 |
@spaces.GPU(duration=300)
|
| 325 |
+
def generate_continuation(audio_path, key, bpm, gen_seconds, fade_seconds, caption):
|
| 326 |
+
"""gpu stage 1: demucs vocal gate + chained musicgen passes.
|
| 327 |
+
mastering and stitching are pure cpu work and run outside the gpu window.
|
| 328 |
+
no caption = pure audio continuation (CFG off at guidance 1.0); a user
|
| 329 |
+
caption switches to described mode at meta's trained guidance 3.0."""
|
| 330 |
+
# lazy import so the UI boots fast and torch only loads on demand
|
| 331 |
+
from continue_music import continue_track
|
| 332 |
+
return continue_track(
|
| 333 |
+
audio_path,
|
| 334 |
+
gen_duration=float(gen_seconds),
|
| 335 |
+
key=key,
|
| 336 |
+
bpm=bpm,
|
| 337 |
+
fade_seconds=float(fade_seconds),
|
| 338 |
+
caption=caption or "",
|
| 339 |
+
)
|
| 340 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 341 |
|
| 342 |
+
@spaces.GPU(duration=180)
|
| 343 |
+
def generate_lyrics(audio_path, key, bpm):
|
| 344 |
+
"""gpu stage 2: whisper transcribes the demucs-isolated vocals, qwen3
|
| 345 |
+
writes the next lines. returns (original, continuation) or (None, None)
|
| 346 |
+
when the clip is instrumental."""
|
| 347 |
+
from transcribe import transcribe_vocals
|
| 348 |
+
original = transcribe_vocals(audio_path)
|
| 349 |
+
if not original:
|
| 350 |
+
return None, None
|
| 351 |
+
from write_lyrics import continue_lyrics
|
| 352 |
+
return original, continue_lyrics(original, key=key, bpm=bpm)
|
| 353 |
+
|
| 354 |
+
|
| 355 |
+
def master_and_stitch(audio_path, extended_tail, prompt_samples, fade_seconds):
|
| 356 |
+
"""cpu stage: matchering mastering pass + native-rate splice + file write."""
|
| 357 |
+
import librosa
|
| 358 |
+
from continue_music import master_match, stitch_with_crossfade
|
| 359 |
+
|
| 360 |
+
# keep the original at its native sample rate and channel count —
|
| 361 |
+
# the generated audio gets resampled up to match, never the reverse
|
| 362 |
+
original, orig_sr = librosa.load(audio_path, sr=None, mono=False)
|
| 363 |
+
|
| 364 |
+
# mastering pass: match the generation's loudness and eq to the
|
| 365 |
+
# original before splicing so both halves read as the same record
|
| 366 |
+
extended_tail = master_match(extended_tail, original, orig_sr)
|
| 367 |
+
|
| 368 |
+
full = stitch_with_crossfade(
|
| 369 |
+
original, extended_tail, prompt_samples, orig_sr,
|
| 370 |
+
fade_seconds=float(fade_seconds),
|
| 371 |
+
)
|
| 372 |
+
|
| 373 |
+
out_path = os.path.join(tempfile.mkdtemp(), "coda_continuation.wav")
|
| 374 |
+
# soundfile expects (samples, channels)
|
| 375 |
+
sf.write(out_path, full.T if full.ndim == 2 else full, orig_sr,
|
| 376 |
+
subtype=_output_subtype(audio_path))
|
| 377 |
+
|
| 378 |
+
added = (full.shape[-1] - original.shape[-1]) / orig_sr
|
| 379 |
+
msg = (
|
| 380 |
+
f"added {added:.1f}s of new audio at {orig_sr} hz "
|
| 381 |
+
f"({_output_subtype(audio_path).replace('_', ' ').lower()}). "
|
| 382 |
+
f"blend completes at {original.shape[-1] / orig_sr:.1f}s, "
|
| 383 |
+
f"outro fades to silence."
|
| 384 |
+
)
|
| 385 |
+
return out_path, msg
|
| 386 |
+
|
| 387 |
|
| 388 |
+
def analyze_track(audio):
|
| 389 |
+
hide = gr.update(visible=False)
|
| 390 |
+
blank = (
|
| 391 |
+
make_stat_html("key", "---"),
|
| 392 |
+
make_stat_html("tempo", "---"),
|
| 393 |
+
make_stat_html("meter", "---"),
|
| 394 |
+
make_stat_html("length", "---"),
|
| 395 |
+
hide, # waveform
|
| 396 |
+
hide, # continue section
|
| 397 |
+
None, None, # states
|
| 398 |
+
gr.update(value="", visible=False), # status tracker
|
| 399 |
+
hide, hide, hide, # result audio / lyrics / poster panels
|
| 400 |
+
)
|
| 401 |
+
if audio is None:
|
| 402 |
+
return blank
|
| 403 |
|
| 404 |
+
try:
|
| 405 |
+
info = fingerprint(audio)
|
| 406 |
+
wave_png = render_waveform_png(audio)
|
| 407 |
+
return (
|
| 408 |
+
make_stat_html("key", info["key"]),
|
| 409 |
+
make_stat_html("tempo", str(info["bpm"]) + " bpm"),
|
| 410 |
+
make_stat_html("meter", info["time_signature"]),
|
| 411 |
+
make_stat_html("length", str(info["duration"]) + "s"),
|
| 412 |
+
gr.update(value=wave_png, visible=True),
|
| 413 |
+
gr.update(visible=True),
|
| 414 |
+
audio, info,
|
| 415 |
+
gr.update(value="", visible=False),
|
| 416 |
+
hide, hide, hide,
|
| 417 |
)
|
| 418 |
+
except Exception as e:
|
| 419 |
+
out = list(blank)
|
| 420 |
+
out[0] = make_stat_html("error", str(e)[:50])
|
| 421 |
+
return tuple(out)
|
| 422 |
+
|
| 423 |
+
|
| 424 |
+
def run_pipeline(audio_path, info, gen_seconds, fade_seconds, caption,
|
| 425 |
+
original_date):
|
| 426 |
+
"""generator: walks the four pipeline stages, streaming tracker updates
|
| 427 |
+
and revealing each result as it lands. outputs:
|
| 428 |
+
(status, audio_panel, audio, lyrics_panel, lyrics_orig, lyrics_cont,
|
| 429 |
+
poster_panel, poster)"""
|
| 430 |
+
skip = gr.skip()
|
| 431 |
+
|
| 432 |
+
def frame(states, note="", labels=None, audio_vis=skip, audio=skip,
|
| 433 |
+
lyrics_vis=skip, lyr_o=skip, lyr_c=skip, poster_vis=skip,
|
| 434 |
+
poster=skip):
|
| 435 |
+
return (gr.update(value=stages_html(states, note, labels), visible=True),
|
| 436 |
+
audio_vis, audio, lyrics_vis, lyr_o, lyr_c, poster_vis, poster)
|
| 437 |
+
|
| 438 |
+
states = ["pending"] * len(PIPELINE_STAGES)
|
| 439 |
+
labels = list(PIPELINE_STAGES)
|
| 440 |
+
|
| 441 |
+
if not audio_path or not info:
|
| 442 |
+
states[0] = "error"
|
| 443 |
+
yield frame(states, "upload a track first.")
|
| 444 |
+
return
|
| 445 |
+
|
| 446 |
+
states[0] = "active"
|
| 447 |
+
yield frame(states,
|
| 448 |
+
"musicgen large is listening to your track and continuing it "
|
| 449 |
+
"— this is the long stage.",
|
| 450 |
+
audio_vis=gr.update(visible=False),
|
| 451 |
+
lyrics_vis=gr.update(visible=False),
|
| 452 |
+
poster_vis=gr.update(visible=False))
|
| 453 |
+
|
| 454 |
+
try:
|
| 455 |
+
extended_tail, prompt_samples, _ = generate_continuation(
|
| 456 |
+
audio_path, info.get("key"), info.get("bpm"),
|
| 457 |
+
gen_seconds, fade_seconds, caption)
|
| 458 |
+
states[0] = "done"
|
| 459 |
+
states[1] = "active"
|
| 460 |
+
yield frame(states, "matching loudness and eq to your recording, "
|
| 461 |
+
"then splicing at the seam.")
|
| 462 |
+
out_path, stitch_msg = master_and_stitch(
|
| 463 |
+
audio_path, extended_tail, prompt_samples, fade_seconds)
|
| 464 |
+
states[1] = "done"
|
| 465 |
+
except Exception as e:
|
| 466 |
+
states[states.index("active")] = "error"
|
| 467 |
+
yield frame(states, f"continuation failed: {str(e)[:200]}")
|
| 468 |
+
return
|
| 469 |
|
| 470 |
+
states[2] = "active"
|
| 471 |
+
yield frame(states, stitch_msg,
|
| 472 |
+
audio_vis=gr.update(visible=True),
|
| 473 |
+
audio=gr.update(value=out_path))
|
| 474 |
+
|
| 475 |
+
lyr_o = lyr_c = None
|
| 476 |
+
try:
|
| 477 |
+
lyr_o, lyr_c = generate_lyrics(
|
| 478 |
+
audio_path, info.get("key"), info.get("bpm"))
|
| 479 |
+
except Exception as e:
|
| 480 |
+
print(f"[coda] lyric stage failed: {e}", flush=True)
|
| 481 |
+
|
| 482 |
+
if lyr_o:
|
| 483 |
+
states[2] = "done"
|
| 484 |
+
yield frame(states, stitch_msg,
|
| 485 |
+
lyrics_vis=gr.update(visible=True),
|
| 486 |
+
lyr_o=lyr_o, lyr_c=lyr_c or "")
|
| 487 |
+
else:
|
| 488 |
+
states[2] = "skip"
|
| 489 |
+
labels[2] = "transcribe & extend the lyrics — no vocals found, skipped"
|
| 490 |
+
yield frame(states, stitch_msg, labels=labels)
|
| 491 |
+
|
| 492 |
+
states[3] = "active"
|
| 493 |
+
yield frame(states, stitch_msg, labels=labels)
|
| 494 |
+
try:
|
| 495 |
+
title = Path(audio_path).stem.replace("_", " ").replace("-", " ").strip()
|
| 496 |
+
poster_path = make_poster(
|
| 497 |
+
out_path,
|
| 498 |
+
title=title or "untitled",
|
| 499 |
+
key_sig=info.get("key", "?"),
|
| 500 |
+
bpm=info.get("bpm", 0) or 0,
|
| 501 |
+
time_sig=info.get("time_signature", "4/4"),
|
| 502 |
+
split_seconds=float(info.get("duration", 0) or 0),
|
| 503 |
+
original_date=original_date or "",
|
| 504 |
)
|
| 505 |
+
states[3] = "done"
|
| 506 |
+
yield frame(states,
|
| 507 |
+
"done. your song is finished — save the poster.",
|
| 508 |
+
labels=labels,
|
| 509 |
+
poster_vis=gr.update(visible=True),
|
| 510 |
+
poster=gr.update(value=poster_path))
|
| 511 |
except Exception as e:
|
| 512 |
+
states[3] = "error"
|
| 513 |
+
yield frame(states, f"poster failed: {str(e)[:160]}", labels=labels)
|
| 514 |
|
| 515 |
|
| 516 |
HEADER_HTML = (
|
|
|
|
| 549 |
with gr.Row():
|
| 550 |
key_display = gr.HTML(make_stat_html("key", "---"))
|
| 551 |
bpm_display = gr.HTML(make_stat_html("tempo", "---"))
|
| 552 |
+
sig_display = gr.HTML(make_stat_html("meter", "---"))
|
| 553 |
+
dur_display = gr.HTML(make_stat_html("length", "---"))
|
| 554 |
+
waveform_display = gr.Image(
|
| 555 |
+
visible=False, show_label=False, container=False,
|
| 556 |
+
interactive=False, buttons=[], elem_classes="waveform-img",
|
| 557 |
+
)
|
| 558 |
|
| 559 |
with gr.Group(elem_classes="continue-panel", visible=False) as continue_section:
|
| 560 |
gr.HTML('<div class="section-label">Continue</div>')
|
|
|
|
| 574 |
placeholder="leave empty to continue from the audio alone — or e.g. \"garage rock, distorted electric guitar, live drums\"",
|
| 575 |
lines=1,
|
| 576 |
)
|
| 577 |
+
date_box = gr.Textbox(
|
| 578 |
+
label="when did you start this song? (optional)",
|
| 579 |
+
placeholder="e.g. \"2018\" or \"march 2019\" — printed on the poster",
|
| 580 |
+
lines=1,
|
| 581 |
+
)
|
| 582 |
+
continue_btn = gr.Button("Finish this song", variant="primary")
|
| 583 |
+
status_html = gr.HTML(visible=False)
|
| 584 |
+
|
| 585 |
+
with gr.Group(elem_classes="results-panel", visible=False) as result_section:
|
| 586 |
+
gr.HTML('<div class="section-label">The finished song</div>')
|
| 587 |
+
result_audio = gr.Audio(label="finished track", type="filepath",
|
| 588 |
+
interactive=False)
|
| 589 |
+
|
| 590 |
+
with gr.Group(elem_classes="results-panel", visible=False) as lyrics_section:
|
| 591 |
+
gr.HTML('<div class="section-label">Lyrics</div>')
|
| 592 |
+
with gr.Row():
|
| 593 |
+
lyrics_orig = gr.Textbox(label="what you had", lines=8,
|
| 594 |
+
interactive=False)
|
| 595 |
+
lyrics_cont = gr.Textbox(label="where it goes next", lines=8,
|
| 596 |
+
interactive=False)
|
| 597 |
+
|
| 598 |
+
with gr.Group(elem_classes="results-panel", visible=False) as poster_section:
|
| 599 |
+
gr.HTML('<div class="section-label">Song poster</div>')
|
| 600 |
+
poster_img = gr.Image(label="poster", type="filepath",
|
| 601 |
+
interactive=False, show_label=False,
|
| 602 |
+
buttons=["download", "fullscreen"],
|
| 603 |
+
elem_classes="poster-img")
|
| 604 |
|
| 605 |
audio_input.change(
|
| 606 |
fn=analyze_track,
|
| 607 |
inputs=[audio_input],
|
| 608 |
+
outputs=[key_display, bpm_display, sig_display, dur_display,
|
| 609 |
+
waveform_display, continue_section, current_track, track_info,
|
| 610 |
+
status_html, result_section, lyrics_section, poster_section],
|
| 611 |
)
|
| 612 |
|
| 613 |
continue_btn.click(
|
| 614 |
+
fn=run_pipeline,
|
| 615 |
+
inputs=[current_track, track_info, gen_length, fade_length,
|
| 616 |
+
caption_box, date_box],
|
| 617 |
+
outputs=[status_html, result_section, result_audio,
|
| 618 |
+
lyrics_section, lyrics_orig, lyrics_cont,
|
| 619 |
+
poster_section, poster_img],
|
| 620 |
)
|
| 621 |
|
| 622 |
gr.HTML(FOOTER_HTML)
|
|
|
|
| 624 |
|
| 625 |
if __name__ == "__main__":
|
| 626 |
app.launch(theme=gr.themes.Base(), css=CUSTOM_CSS)
|
|
|
|
@@ -1,79 +1,176 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
import os
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
|
| 4 |
-
#
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
|
| 11 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
"""
|
| 13 |
-
|
| 14 |
-
|
|
|
|
| 15 |
"""
|
| 16 |
-
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
draw = ImageDraw.Draw(img)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
-
#
|
| 21 |
-
cx, cy = w // 2,
|
| 22 |
-
radius
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
# inner circle (label area)
|
| 31 |
-
inner_r = 80
|
| 32 |
-
draw.ellipse(
|
| 33 |
-
[cx - inner_r, cy - inner_r, cx + inner_r, cy + inner_r],
|
| 34 |
-
fill=BG_COLOR,
|
| 35 |
-
outline=AMBER,
|
| 36 |
-
width=2
|
| 37 |
-
)
|
| 38 |
-
|
| 39 |
-
# spindle dot
|
| 40 |
draw.ellipse([cx - 6, cy - 6, cx + 6, cy + 6], fill=AMBER)
|
|
|
|
| 41 |
|
| 42 |
-
#
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
width=1
|
| 48 |
-
)
|
| 49 |
|
| 50 |
-
#
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
|
|
|
|
|
|
|
|
|
| 77 |
|
|
|
|
|
|
|
| 78 |
img.save(out_path)
|
| 79 |
return out_path
|
|
|
|
| 1 |
+
"""song poster + waveform rendering. pure pillow — no ML.
|
| 2 |
+
|
| 3 |
+
the poster is the shareable artifact: a vinyl-sleeve card carrying the full
|
| 4 |
+
song's waveform (original in cream, AI continuation in amber), the date the
|
| 5 |
+
song was started and the date coda finished it, and the detected
|
| 6 |
+
key / tempo / meter.
|
| 7 |
+
"""
|
| 8 |
import os
|
| 9 |
+
import tempfile
|
| 10 |
+
from datetime import date
|
| 11 |
+
|
| 12 |
+
import numpy as np
|
| 13 |
+
from PIL import Image, ImageDraw, ImageFont
|
| 14 |
|
| 15 |
+
# palette mirrors the app css
|
| 16 |
+
BG = (11, 12, 15)
|
| 17 |
+
SURFACE = (20, 22, 27)
|
| 18 |
+
SURFACE_2 = (26, 29, 36)
|
| 19 |
+
BORDER = (38, 42, 51)
|
| 20 |
+
GROOVE = (30, 33, 40)
|
| 21 |
+
TEXT = (237, 238, 242)
|
| 22 |
+
MUTED = (138, 144, 156)
|
| 23 |
+
AMBER = (226, 168, 92)
|
| 24 |
+
|
| 25 |
+
# debian (hf spaces) ships dejavu; windows dev boxes get georgia/arial
|
| 26 |
+
_FONT_PATHS = {
|
| 27 |
+
"serif": ["/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf",
|
| 28 |
+
"C:/Windows/Fonts/georgia.ttf",
|
| 29 |
+
"C:/Windows/Fonts/times.ttf"],
|
| 30 |
+
"serif_italic": ["/usr/share/fonts/truetype/dejavu/DejaVuSerif-Italic.ttf",
|
| 31 |
+
"C:/Windows/Fonts/georgiai.ttf",
|
| 32 |
+
"C:/Windows/Fonts/timesi.ttf"],
|
| 33 |
+
"sans": ["/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
|
| 34 |
+
"C:/Windows/Fonts/arial.ttf"],
|
| 35 |
+
"sans_bold": ["/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf",
|
| 36 |
+
"C:/Windows/Fonts/arialbd.ttf"],
|
| 37 |
+
}
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def _font(kind, size):
|
| 41 |
+
for path in _FONT_PATHS[kind]:
|
| 42 |
+
try:
|
| 43 |
+
return ImageFont.truetype(path, size)
|
| 44 |
+
except OSError:
|
| 45 |
+
continue
|
| 46 |
+
try:
|
| 47 |
+
return ImageFont.load_default(size)
|
| 48 |
+
except TypeError: # older pillow without size kwarg
|
| 49 |
+
return ImageFont.load_default()
|
| 50 |
|
| 51 |
|
| 52 |
+
def _center_text(draw, text, cx, top, font, fill):
|
| 53 |
+
bbox = draw.textbbox((0, 0), text, font=font)
|
| 54 |
+
draw.text((cx - (bbox[2] - bbox[0]) / 2, top), text, font=font, fill=fill)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def _envelope(samples, columns):
|
| 58 |
+
"""peak amplitude per column, normalized to [0, 1]."""
|
| 59 |
+
if len(samples) == 0:
|
| 60 |
+
return np.zeros(columns)
|
| 61 |
+
chunks = np.array_split(np.abs(samples), columns)
|
| 62 |
+
env = np.array([float(c.max()) if len(c) else 0.0 for c in chunks])
|
| 63 |
+
peak = env.max()
|
| 64 |
+
return env / peak if peak > 0 else env
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def _draw_waveform(draw, samples, box, color_a, color_b=None,
|
| 68 |
+
split_frac=None, bar=4, gap=3):
|
| 69 |
"""
|
| 70 |
+
bar-style waveform inside box=(x0, y0, x1, y1). columns left of
|
| 71 |
+
`split_frac` (fraction of the timeline) use color_a, the rest color_b —
|
| 72 |
+
that's how the poster shows where the original ends and coda begins.
|
| 73 |
"""
|
| 74 |
+
x0, y0, x1, y1 = box
|
| 75 |
+
step = bar + gap
|
| 76 |
+
columns = max(1, (x1 - x0) // step)
|
| 77 |
+
env = _envelope(samples, columns)
|
| 78 |
+
mid = (y0 + y1) / 2
|
| 79 |
+
half = (y1 - y0) / 2 - 1
|
| 80 |
+
split_col = columns + 1 if split_frac is None else int(columns * split_frac)
|
| 81 |
+
for i, e in enumerate(env):
|
| 82 |
+
h = max(2.0, e * half)
|
| 83 |
+
x = x0 + i * step
|
| 84 |
+
color = color_a if i < split_col else (color_b or color_a)
|
| 85 |
+
draw.rounded_rectangle([x, mid - h, x + bar, mid + h],
|
| 86 |
+
radius=bar / 2, fill=color)
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def render_waveform_png(audio_path, out_path=None, width=1280, height=200):
|
| 90 |
+
"""standalone waveform card for the ui, shown right after upload."""
|
| 91 |
+
import librosa
|
| 92 |
+
samples, _ = librosa.load(audio_path, sr=8000, mono=True)
|
| 93 |
+
img = Image.new("RGB", (width, height), SURFACE)
|
| 94 |
draw = ImageDraw.Draw(img)
|
| 95 |
+
_draw_waveform(draw, samples, (24, 22, width - 24, height - 22), AMBER)
|
| 96 |
+
if out_path is None:
|
| 97 |
+
out_path = os.path.join(tempfile.mkdtemp(), "waveform.png")
|
| 98 |
+
img.save(out_path)
|
| 99 |
+
return out_path
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def make_poster(audio_path, title, key_sig, bpm, time_sig, split_seconds,
|
| 103 |
+
original_date="", completed=None, out_path=None):
|
| 104 |
+
"""
|
| 105 |
+
1080x1080 vinyl-sleeve poster for the finished song.
|
| 106 |
+
|
| 107 |
+
audio_path: the FULL stitched song
|
| 108 |
+
split_seconds: where the original ends and the continuation begins
|
| 109 |
+
original_date: free text from the user ("2018", "march 2019", ...)
|
| 110 |
+
"""
|
| 111 |
+
import librosa
|
| 112 |
+
samples, sr = librosa.load(audio_path, sr=8000, mono=True)
|
| 113 |
+
total = len(samples) / sr if len(samples) else 1.0
|
| 114 |
+
split_frac = min(1.0, max(0.0, float(split_seconds) / total))
|
| 115 |
+
|
| 116 |
+
w = h = 1080
|
| 117 |
+
img = Image.new("RGB", (w, h), BG)
|
| 118 |
+
draw = ImageDraw.Draw(img)
|
| 119 |
+
|
| 120 |
+
# sleeve frame
|
| 121 |
+
draw.rectangle([24, 24, w - 25, h - 25], outline=BORDER, width=2)
|
| 122 |
|
| 123 |
+
# --- vinyl ---
|
| 124 |
+
cx, cy, radius = w // 2, 360, 250
|
| 125 |
+
draw.ellipse([cx - radius, cy - radius, cx + radius, cy + radius],
|
| 126 |
+
fill=SURFACE_2, outline=AMBER, width=3)
|
| 127 |
+
for r in range(112, radius - 12, 13):
|
| 128 |
+
draw.ellipse([cx - r, cy - r, cx + r, cy + r], outline=GROOVE, width=1)
|
| 129 |
+
label_r = 94
|
| 130 |
+
draw.ellipse([cx - label_r, cy - label_r, cx + label_r, cy + label_r],
|
| 131 |
+
fill=BG, outline=AMBER, width=2)
|
| 132 |
+
_center_text(draw, "CODA", cx, cy - 58, _font("sans_bold", 28), AMBER)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
draw.ellipse([cx - 6, cy - 6, cx + 6, cy + 6], fill=AMBER)
|
| 134 |
+
_center_text(draw, "SIDE B", cx, cy + 32, _font("sans", 17), MUTED)
|
| 135 |
|
| 136 |
+
# --- title ---
|
| 137 |
+
title = (title or "untitled").strip()
|
| 138 |
+
if len(title) > 32:
|
| 139 |
+
title = title[:29] + "..."
|
| 140 |
+
_center_text(draw, title, cx, 650, _font("serif", 58), TEXT)
|
|
|
|
|
|
|
| 141 |
|
| 142 |
+
# --- key / tempo / meter ---
|
| 143 |
+
info_line = f"{key_sig} · {round(float(bpm))} bpm · {time_sig}"
|
| 144 |
+
_center_text(draw, info_line, cx, 738, _font("sans", 27), AMBER)
|
| 145 |
+
|
| 146 |
+
# --- waveform: original (cream) vs continuation (amber) ---
|
| 147 |
+
_draw_waveform(draw, samples, (100, 798, 980, 902), TEXT, AMBER, split_frac)
|
| 148 |
+
|
| 149 |
+
# legend
|
| 150 |
+
leg_font = _font("sans", 19)
|
| 151 |
+
items = [("original", TEXT), ("continuation", AMBER)]
|
| 152 |
+
widths = [16 + 8 + draw.textbbox((0, 0), t, font=leg_font)[2] for t, _ in items]
|
| 153 |
+
lx = cx - (sum(widths) + 36) / 2
|
| 154 |
+
for (label, color), iw in zip(items, widths):
|
| 155 |
+
draw.rounded_rectangle([lx, 928, lx + 16, 944], radius=4, fill=color)
|
| 156 |
+
draw.text((lx + 24, 924), label, font=leg_font, fill=MUTED)
|
| 157 |
+
lx += iw + 36
|
| 158 |
+
|
| 159 |
+
# --- dates ---
|
| 160 |
+
completed = completed or date.today()
|
| 161 |
+
completed_text = f"{completed:%B} {completed.day}, {completed.year}"
|
| 162 |
+
original_date = (original_date or "").strip()
|
| 163 |
+
if original_date:
|
| 164 |
+
dates = f"started {original_date} — finished {completed_text}"
|
| 165 |
+
else:
|
| 166 |
+
dates = f"finished {completed_text}"
|
| 167 |
+
_center_text(draw, dates, cx, 974, _font("serif_italic", 25), MUTED)
|
| 168 |
+
|
| 169 |
+
# --- branding ---
|
| 170 |
+
_center_text(draw, "C O M P L E T E D B Y C O D A", cx, 1024,
|
| 171 |
+
_font("sans_bold", 17), AMBER)
|
| 172 |
|
| 173 |
+
if out_path is None:
|
| 174 |
+
out_path = os.path.join(tempfile.mkdtemp(), "coda_poster.png")
|
| 175 |
img.save(out_path)
|
| 176 |
return out_path
|
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""shared HT-Demucs stem separation.
|
| 2 |
+
|
| 3 |
+
the pipeline separates vocals twice with the same model: continue_music's
|
| 4 |
+
vocal gate prompts musicgen with the instrumental (vocal prompts are
|
| 5 |
+
out-of-distribution — musicgen's training data was demucs-separated), and
|
| 6 |
+
transcribe.py feeds the isolated vocal stem to whisper so it doesn't
|
| 7 |
+
hallucinate on drums and guitars.
|
| 8 |
+
"""
|
| 9 |
+
import librosa
|
| 10 |
+
import numpy as np
|
| 11 |
+
|
| 12 |
+
_demucs = None
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def _load_demucs():
|
| 16 |
+
global _demucs
|
| 17 |
+
if _demucs is None:
|
| 18 |
+
from demucs.pretrained import get_model
|
| 19 |
+
_demucs = get_model("htdemucs")
|
| 20 |
+
_demucs.eval()
|
| 21 |
+
return _demucs
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def isolate_vocals(path, max_seconds=None):
|
| 25 |
+
"""
|
| 26 |
+
separate `path` into stems and return the vocal stem.
|
| 27 |
+
|
| 28 |
+
returns (vocals, sr, vocal_ratio):
|
| 29 |
+
vocals float32 (2, N) at the demucs sample rate
|
| 30 |
+
sr demucs sample rate (44100)
|
| 31 |
+
vocal_ratio vocal-stem rms / mix rms — how vocal the clip is
|
| 32 |
+
or (None, None, 0.0) when demucs isn't available.
|
| 33 |
+
"""
|
| 34 |
+
try:
|
| 35 |
+
import torch
|
| 36 |
+
from demucs.apply import apply_model
|
| 37 |
+
model = _load_demucs()
|
| 38 |
+
except Exception as e:
|
| 39 |
+
print(f"[coda] demucs unavailable ({e})", flush=True)
|
| 40 |
+
return None, None, 0.0
|
| 41 |
+
|
| 42 |
+
sr = model.samplerate
|
| 43 |
+
wav, _ = librosa.load(path, sr=sr, mono=False)
|
| 44 |
+
if wav.ndim == 1:
|
| 45 |
+
wav = np.stack([wav, wav])
|
| 46 |
+
if max_seconds:
|
| 47 |
+
wav = wav[:, -int(max_seconds * sr):]
|
| 48 |
+
wav = wav.astype(np.float32)
|
| 49 |
+
|
| 50 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 51 |
+
mix = torch.from_numpy(np.ascontiguousarray(wav))
|
| 52 |
+
# demucs expects the mix standardized (same as its own separate.py)
|
| 53 |
+
ref = mix.mean(0)
|
| 54 |
+
std = float(ref.std()) + 1e-8
|
| 55 |
+
with torch.no_grad():
|
| 56 |
+
sources = apply_model(
|
| 57 |
+
model.to(device), ((mix - ref.mean()) / std)[None].to(device),
|
| 58 |
+
device=device, split=True, overlap=0.25, progress=False,
|
| 59 |
+
)[0]
|
| 60 |
+
sources = sources.cpu() * std + ref.mean()
|
| 61 |
+
vocals = sources[model.sources.index("vocals")].numpy().astype(np.float32)
|
| 62 |
+
|
| 63 |
+
vocal_rms = float(np.sqrt(np.mean(vocals ** 2)))
|
| 64 |
+
mix_rms = float(np.sqrt(np.mean(wav ** 2)) + 1e-9)
|
| 65 |
+
return vocals, sr, vocal_rms / mix_rms
|
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""offline tests for the non-GPU pipeline pieces: analysis (key/tempo/meter),
|
| 2 |
+
waveform + poster rendering, whisper hallucination filter, qwen output
|
| 3 |
+
cleaner, and the ui stage tracker. no torch required."""
|
| 4 |
+
import os
|
| 5 |
+
import sys
|
| 6 |
+
import tempfile
|
| 7 |
+
from datetime import date
|
| 8 |
+
|
| 9 |
+
import numpy as np
|
| 10 |
+
import soundfile as sf
|
| 11 |
+
|
| 12 |
+
FAILURES = []
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def check(name, cond, detail=""):
|
| 16 |
+
status = "PASS" if cond else "FAIL"
|
| 17 |
+
print(f"[{status}] {name} {detail}")
|
| 18 |
+
if not cond:
|
| 19 |
+
FAILURES.append(name)
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def click_track(meter, bpm=120, bars=16, sr=22050):
|
| 23 |
+
"""synthetic drum-machine loop: an accented hit on beat 1 of every bar."""
|
| 24 |
+
beat_len = int(sr * 60 / bpm)
|
| 25 |
+
total = beat_len * meter * bars
|
| 26 |
+
y = np.zeros(total, dtype=np.float32)
|
| 27 |
+
rng = np.random.default_rng(7)
|
| 28 |
+
burst = (rng.standard_normal(600) * np.exp(-np.linspace(0, 8, 600))).astype(np.float32)
|
| 29 |
+
for b in range(meter * bars):
|
| 30 |
+
amp = 1.0 if b % meter == 0 else 0.3
|
| 31 |
+
start = b * beat_len
|
| 32 |
+
y[start:start + 600] += amp * burst
|
| 33 |
+
return y, sr
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def write_tmp(y, sr):
|
| 37 |
+
path = os.path.join(tempfile.mkdtemp(), "t.wav")
|
| 38 |
+
sf.write(path, y, sr)
|
| 39 |
+
return path
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
# ---- 1. time signature estimation ----
|
| 43 |
+
from analyze import fingerprint, get_time_signature
|
| 44 |
+
|
| 45 |
+
y4, sr = click_track(4)
|
| 46 |
+
y3, _ = click_track(3)
|
| 47 |
+
check("4/4 click track detected as 4/4", get_time_signature(write_tmp(y4, sr)) == "4/4")
|
| 48 |
+
check("3/4 click track detected as 3/4", get_time_signature(write_tmp(y3, sr)) == "3/4")
|
| 49 |
+
|
| 50 |
+
# too short / ambiguous material falls back to 4/4, never crashes
|
| 51 |
+
y_short, _ = click_track(4, bars=1)
|
| 52 |
+
check("short clip falls back to 4/4", get_time_signature(write_tmp(y_short, sr)) == "4/4")
|
| 53 |
+
|
| 54 |
+
# ---- 2. fingerprint: one call, all fields ----
|
| 55 |
+
info = fingerprint(write_tmp(y4, sr))
|
| 56 |
+
for field in ("key", "bpm", "time_signature", "duration", "sample_rate", "channels"):
|
| 57 |
+
check(f"fingerprint has {field}", field in info)
|
| 58 |
+
check("fingerprint bpm ~120", 110 <= info["bpm"] <= 130, f"bpm={info['bpm']}")
|
| 59 |
+
check("fingerprint meter 4/4", info["time_signature"] == "4/4")
|
| 60 |
+
|
| 61 |
+
# ---- 3. waveform render ----
|
| 62 |
+
from poster import render_waveform_png, make_poster, _envelope
|
| 63 |
+
|
| 64 |
+
t = np.linspace(0, 5, 5 * 22050).astype(np.float32)
|
| 65 |
+
tone = (0.5 * np.sin(2 * np.pi * 220 * t) * np.linspace(0, 1, len(t))).astype(np.float32)
|
| 66 |
+
tone_path = write_tmp(tone, 22050)
|
| 67 |
+
|
| 68 |
+
wave_png = render_waveform_png(tone_path)
|
| 69 |
+
check("waveform png written", os.path.exists(wave_png) and os.path.getsize(wave_png) > 1000)
|
| 70 |
+
|
| 71 |
+
env = _envelope(tone, 100)
|
| 72 |
+
check("envelope normalized to [0,1]", 0.99 <= env.max() <= 1.0 and env.min() >= 0)
|
| 73 |
+
check("ramped tone envelope rises", env[-10:].mean() > env[:10].mean())
|
| 74 |
+
check("empty audio envelope is safe", np.all(_envelope(np.array([]), 50) == 0))
|
| 75 |
+
|
| 76 |
+
# ---- 4. poster ----
|
| 77 |
+
poster_path = make_poster(
|
| 78 |
+
tone_path, title="garage demo final FINAL v2 (real one this time)",
|
| 79 |
+
key_sig="F# minor", bpm=124.0, time_sig="4/4", split_seconds=2.5,
|
| 80 |
+
original_date="2018", completed=date(2026, 6, 12),
|
| 81 |
+
)
|
| 82 |
+
check("poster png written", os.path.exists(poster_path) and os.path.getsize(poster_path) > 5000)
|
| 83 |
+
from PIL import Image
|
| 84 |
+
pimg = Image.open(poster_path)
|
| 85 |
+
check("poster is 1080x1080", pimg.size == (1080, 1080), f"size={pimg.size}")
|
| 86 |
+
|
| 87 |
+
# poster without an original date still renders
|
| 88 |
+
p2 = make_poster(tone_path, title="x", key_sig="C major", bpm=90,
|
| 89 |
+
time_sig="3/4", split_seconds=0.0)
|
| 90 |
+
check("poster works without original date", os.path.exists(p2))
|
| 91 |
+
|
| 92 |
+
# ---- 5. whisper hallucination filter ----
|
| 93 |
+
from transcribe import looks_like_lyrics
|
| 94 |
+
|
| 95 |
+
check("real lyrics pass", looks_like_lyrics("I left my heart out on the wire\nwaiting for a sign"))
|
| 96 |
+
check("'thank you.' rejected", not looks_like_lyrics("Thank you."))
|
| 97 |
+
check("'thanks for watching' rejected", not looks_like_lyrics("Thanks for watching!"))
|
| 98 |
+
check("empty rejected", not looks_like_lyrics(""))
|
| 99 |
+
check("None rejected", not looks_like_lyrics(None))
|
| 100 |
+
check("two-word loop rejected", not looks_like_lyrics("la la la la la la la la la"))
|
| 101 |
+
check("short fragment rejected", not looks_like_lyrics("oh yeah"))
|
| 102 |
+
|
| 103 |
+
# ---- 6. qwen output cleaner ----
|
| 104 |
+
from write_lyrics import _clean
|
| 105 |
+
|
| 106 |
+
raw = """<think>
|
| 107 |
+
The user wants lyrics matching the tone...
|
| 108 |
+
</think>
|
| 109 |
+
Here are the lyrics:
|
| 110 |
+
the ashes settle where we used to stand
|
| 111 |
+
I trace the smoke back to your hand
|
| 112 |
+
Existing line here
|
| 113 |
+
and every echo learns my name
|
| 114 |
+
"""
|
| 115 |
+
cleaned = _clean(raw, existing_lines={"Existing line here"}, num_lines=8)
|
| 116 |
+
check("think block stripped", "<think>" not in cleaned and "tone" not in cleaned)
|
| 117 |
+
check("scaffold line dropped", "Here are the lyrics" not in cleaned)
|
| 118 |
+
check("existing line not repeated", "Existing line here" not in cleaned)
|
| 119 |
+
check("real lines kept", cleaned.startswith("the ashes settle"), f"{cleaned!r}")
|
| 120 |
+
check("line cap respected", len(_clean("a\nb\nc\nd\ne", set(), 3).split("\n")) == 3)
|
| 121 |
+
|
| 122 |
+
# ---- 7. stage tracker html ----
|
| 123 |
+
from app import stages_html, PIPELINE_STAGES, make_stat_html
|
| 124 |
+
|
| 125 |
+
html = stages_html(["done", "active", "pending", "pending"], note="working")
|
| 126 |
+
check("tracker renders all stages", all(s in html for s in PIPELINE_STAGES))
|
| 127 |
+
check("tracker carries states", 'stage done' in html and 'stage active' in html)
|
| 128 |
+
check("tracker note rendered", 'working' in html)
|
| 129 |
+
custom = stages_html(["skip"] * 4, labels=["a", "b", "c", "d"])
|
| 130 |
+
check("tracker custom labels", ">a<" in custom and 'stage skip' in custom)
|
| 131 |
+
check("stat card renders", 'stat-value' in make_stat_html("key", "F# minor"))
|
| 132 |
+
|
| 133 |
+
print()
|
| 134 |
+
if FAILURES:
|
| 135 |
+
print(f"{len(FAILURES)} FAILURES: {FAILURES}")
|
| 136 |
+
sys.exit(1)
|
| 137 |
+
print("all checks passed")
|
|
@@ -1,73 +1,99 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
"""
|
| 20 |
-
|
| 21 |
-
|
| 22 |
"""
|
| 23 |
-
import
|
| 24 |
-
import
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
"""
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
processor, model = _load_whisper()
|
| 56 |
-
|
| 57 |
-
track, sr = torchaudio.load(vocal_path)
|
| 58 |
-
|
| 59 |
-
# whisper wants 16kHz mono
|
| 60 |
-
if sr != 16000:
|
| 61 |
-
track = torchaudio.transforms.Resample(sr, 16000)(track)
|
| 62 |
-
if track.shape[0] > 1:
|
| 63 |
-
track = track.mean(dim=0, keepdim=True)
|
| 64 |
-
|
| 65 |
-
track = track.squeeze()
|
| 66 |
-
|
| 67 |
-
inputs = processor(track.numpy(), sampling_rate=16000, return_tensors="pt")
|
| 68 |
-
|
| 69 |
-
with torch.no_grad():
|
| 70 |
-
predicted_ids = model.generate(inputs.input_features)
|
| 71 |
-
|
| 72 |
-
lyrics = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
|
| 73 |
-
return lyrics.strip()
|
|
|
|
| 1 |
+
"""lyric extraction: demucs-isolated vocals -> whisper large-v3.
|
| 2 |
+
|
| 3 |
+
vocals are pulled out with HT-Demucs before transcription for two reasons:
|
| 4 |
+
whisper hallucinates on full mixes (drums decode to "thanks for watching"),
|
| 5 |
+
and it keeps this stage consistent with continue_music's vocal gate — both
|
| 6 |
+
agree on what counts as a vocal.
|
| 7 |
+
|
| 8 |
+
torchaudio.load is deliberately avoided here: torchaudio 2.9+ delegates
|
| 9 |
+
decoding to torchcodec, which isn't installed on the Space. librosa's
|
| 10 |
+
soundfile backend handles everything we accept.
|
| 11 |
+
"""
|
| 12 |
+
import numpy as np
|
| 13 |
+
|
| 14 |
+
WHISPER_ID = "openai/whisper-large-v3"
|
| 15 |
+
WHISPER_SR = 16000
|
| 16 |
+
# same bar as continue_music's vocal gate
|
| 17 |
+
VOCAL_RMS_THRESHOLD = 0.1
|
| 18 |
+
|
| 19 |
+
_asr = None
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _load_asr():
|
| 23 |
+
global _asr
|
| 24 |
+
if _asr is None:
|
| 25 |
+
import torch
|
| 26 |
+
from transformers import pipeline
|
| 27 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
| 28 |
+
_asr = pipeline(
|
| 29 |
+
"automatic-speech-recognition",
|
| 30 |
+
model=WHISPER_ID,
|
| 31 |
+
torch_dtype=torch.float16 if device == "cuda" else torch.float32,
|
| 32 |
+
device=device,
|
| 33 |
+
chunk_length_s=30, # long-form: clips aren't capped at whisper's 30s window
|
| 34 |
+
)
|
| 35 |
+
return _asr
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
# whisper's classic non-speech hallucinations — silence and instrument bleed
|
| 39 |
+
# decode to these with high confidence
|
| 40 |
+
_HALLUCINATION_LINES = {
|
| 41 |
+
"you", "thank you", "thanks for watching", "thank you for watching",
|
| 42 |
+
"bye", "music", "[music]", "(music)", "subscribe",
|
| 43 |
+
"please subscribe", "see you next time",
|
| 44 |
+
}
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def looks_like_lyrics(text):
|
| 48 |
+
"""filter whisper output that is clearly hallucinated, not sung words."""
|
| 49 |
+
text = (text or "").strip()
|
| 50 |
+
if not text:
|
| 51 |
+
return False
|
| 52 |
+
words = text.split()
|
| 53 |
+
if len(words) < 3:
|
| 54 |
+
return False
|
| 55 |
+
if text.lower().strip(" .!?") in _HALLUCINATION_LINES:
|
| 56 |
+
return False
|
| 57 |
+
# one short phrase looped over and over is the other hallucination shape
|
| 58 |
+
unique = {w.lower().strip(",.!?") for w in words}
|
| 59 |
+
if len(words) >= 8 and len(unique) <= 2:
|
| 60 |
+
return False
|
| 61 |
+
return True
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def transcribe_vocals(path):
|
| 65 |
"""
|
| 66 |
+
isolate vocals and transcribe them. returns the lyric text, or None when
|
| 67 |
+
the clip is instrumental (or separation finds no real vocal energy).
|
| 68 |
"""
|
| 69 |
+
import librosa
|
| 70 |
+
from stems import isolate_vocals
|
| 71 |
+
|
| 72 |
+
vocals, sep_sr, ratio = isolate_vocals(path)
|
| 73 |
+
if vocals is None:
|
| 74 |
+
print("[coda] transcribe: demucs unavailable, skipping lyrics", flush=True)
|
| 75 |
+
return None
|
| 76 |
+
if ratio < VOCAL_RMS_THRESHOLD:
|
| 77 |
+
print(f"[coda] transcribe: instrumental clip "
|
| 78 |
+
f"(vocal/mix rms {ratio:.3f}), no lyrics to pull", flush=True)
|
| 79 |
+
return None
|
| 80 |
+
|
| 81 |
+
mono = vocals.mean(axis=0) if vocals.ndim == 2 else vocals
|
| 82 |
+
wav = librosa.resample(mono.astype(np.float32), orig_sr=sep_sr,
|
| 83 |
+
target_sr=WHISPER_SR, res_type="soxr_hq")
|
| 84 |
+
|
| 85 |
+
asr = _load_asr()
|
| 86 |
+
out = asr(
|
| 87 |
+
{"array": wav, "sampling_rate": WHISPER_SR},
|
| 88 |
+
return_timestamps=True, # required for inputs longer than 30s
|
| 89 |
+
generate_kwargs={"task": "transcribe"},
|
| 90 |
+
)
|
| 91 |
+
text = (out.get("text") or "").strip()
|
| 92 |
+
|
| 93 |
+
if not looks_like_lyrics(text):
|
| 94 |
+
print(f"[coda] transcribe: discarding likely hallucination {text!r}",
|
| 95 |
+
flush=True)
|
| 96 |
+
return None
|
| 97 |
+
print(f"[coda] transcribe: pulled {len(text.split())} words of lyrics",
|
| 98 |
+
flush=True)
|
| 99 |
+
return text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -1,5 +1,12 @@
|
|
| 1 |
-
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
|
| 4 |
_model = None
|
| 5 |
_tokenizer = None
|
|
@@ -8,59 +15,85 @@ _tokenizer = None
|
|
| 8 |
def _load_qwen():
|
| 9 |
global _model, _tokenizer
|
| 10 |
if _model is None:
|
| 11 |
-
|
| 12 |
-
|
|
|
|
| 13 |
_model = AutoModelForCausalLM.from_pretrained(
|
| 14 |
-
|
| 15 |
-
torch_dtype="auto",
|
| 16 |
-
device_map="auto"
|
| 17 |
)
|
| 18 |
return _model, _tokenizer
|
| 19 |
|
| 20 |
|
| 21 |
-
def
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
model, tokenizer = _load_qwen()
|
| 27 |
|
| 28 |
context_parts = []
|
| 29 |
if key:
|
| 30 |
-
context_parts.append(f"
|
| 31 |
if bpm:
|
| 32 |
-
context_parts.append(f"at {bpm} BPM")
|
| 33 |
if style_hint:
|
| 34 |
context_parts.append(f"with a {style_hint} feel")
|
|
|
|
|
|
|
| 35 |
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
Do not repeat existing lines. Do not add commentary or explanations.
|
| 41 |
{context}
|
| 42 |
|
| 43 |
Existing lyrics:
|
| 44 |
-
{existing_lyrics}
|
| 45 |
-
|
| 46 |
-
Continuation:"""
|
| 47 |
|
| 48 |
messages = [{"role": "user", "content": prompt}]
|
| 49 |
-
text = tokenizer.apply_chat_template(
|
| 50 |
-
|
|
|
|
|
|
|
|
|
|
| 51 |
inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
generated = output[0][inputs.input_ids.shape[1]:]
|
| 62 |
result = tokenizer.decode(generated, skip_special_tokens=True)
|
| 63 |
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
return '\n'.join(lines[:num_lines])
|
|
|
|
| 1 |
+
"""lyric continuation with qwen3-8b.
|
| 2 |
|
| 3 |
+
this is a songwriting aid, not vocal synthesis: musicgen can't generate
|
| 4 |
+
voices (vocals were stripped from its training data by design), so coda
|
| 5 |
+
hands you the words for the new section instead of pretending to sing them.
|
| 6 |
+
"""
|
| 7 |
+
import re
|
| 8 |
+
|
| 9 |
+
MODEL_ID = "Qwen/Qwen3-8B"
|
| 10 |
|
| 11 |
_model = None
|
| 12 |
_tokenizer = None
|
|
|
|
| 15 |
def _load_qwen():
|
| 16 |
global _model, _tokenizer
|
| 17 |
if _model is None:
|
| 18 |
+
import torch
|
| 19 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 20 |
+
_tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
| 21 |
_model = AutoModelForCausalLM.from_pretrained(
|
| 22 |
+
MODEL_ID,
|
| 23 |
+
torch_dtype=torch.bfloat16 if torch.cuda.is_available() else "auto",
|
| 24 |
+
device_map="auto",
|
| 25 |
)
|
| 26 |
return _model, _tokenizer
|
| 27 |
|
| 28 |
|
| 29 |
+
def _clean(result, existing_lines, num_lines):
|
| 30 |
+
# qwen3 occasionally leaks reasoning even with thinking disabled
|
| 31 |
+
result = re.sub(r"<think>.*?</think>", "", result, flags=re.S)
|
| 32 |
+
result = result.replace("<think>", "").replace("</think>", "")
|
| 33 |
+
|
| 34 |
+
lines = []
|
| 35 |
+
for line in result.strip().split("\n"):
|
| 36 |
+
line = line.strip().strip('"')
|
| 37 |
+
if not line:
|
| 38 |
+
continue
|
| 39 |
+
# drop scaffolding the model sometimes adds despite instructions
|
| 40 |
+
if line.lower().rstrip(":") in (
|
| 41 |
+
"continuation", "lyrics", "new lyrics", "here are the lyrics",
|
| 42 |
+
"here is the continuation"):
|
| 43 |
+
continue
|
| 44 |
+
if line in existing_lines:
|
| 45 |
+
continue
|
| 46 |
+
lines.append(line)
|
| 47 |
+
return "\n".join(lines[:num_lines])
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def continue_lyrics(existing_lyrics, key=None, bpm=None, style_hint=None,
|
| 51 |
+
num_lines=8):
|
| 52 |
+
"""write `num_lines` new lines in the same voice as `existing_lyrics`."""
|
| 53 |
+
import torch
|
| 54 |
model, tokenizer = _load_qwen()
|
| 55 |
|
| 56 |
context_parts = []
|
| 57 |
if key:
|
| 58 |
+
context_parts.append(f"the song is in {key}")
|
| 59 |
if bpm:
|
| 60 |
+
context_parts.append(f"at {round(float(bpm))} BPM")
|
| 61 |
if style_hint:
|
| 62 |
context_parts.append(f"with a {style_hint} feel")
|
| 63 |
+
context = ("Musical context: " + ", ".join(context_parts) + "."
|
| 64 |
+
if context_parts else "")
|
| 65 |
|
| 66 |
+
prompt = f"""You are a songwriter finishing a song that was abandoned mid-write.
|
| 67 |
+
Continue these lyrics naturally — match the tone, vocabulary, rhyme feel, and imagery.
|
| 68 |
+
Write exactly {num_lines} new lines. Output only the lyric lines: no titles,
|
| 69 |
+
no section labels, no commentary, and do not repeat existing lines.
|
|
|
|
| 70 |
{context}
|
| 71 |
|
| 72 |
Existing lyrics:
|
| 73 |
+
{existing_lyrics}"""
|
|
|
|
|
|
|
| 74 |
|
| 75 |
messages = [{"role": "user", "content": prompt}]
|
| 76 |
+
text = tokenizer.apply_chat_template(
|
| 77 |
+
messages, tokenize=False, add_generation_prompt=True,
|
| 78 |
+
# hybrid thinking off — we want lyric lines, not <think> blocks
|
| 79 |
+
enable_thinking=False,
|
| 80 |
+
)
|
| 81 |
inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
| 82 |
|
| 83 |
+
with torch.no_grad():
|
| 84 |
+
output = model.generate(
|
| 85 |
+
**inputs,
|
| 86 |
+
max_new_tokens=320,
|
| 87 |
+
do_sample=True,
|
| 88 |
+
# qwen3's recommended non-thinking sampling
|
| 89 |
+
temperature=0.7,
|
| 90 |
+
top_p=0.8,
|
| 91 |
+
top_k=20,
|
| 92 |
+
pad_token_id=tokenizer.eos_token_id,
|
| 93 |
+
)
|
| 94 |
|
| 95 |
generated = output[0][inputs.input_ids.shape[1]:]
|
| 96 |
result = tokenizer.decode(generated, skip_special_tokens=True)
|
| 97 |
|
| 98 |
+
existing = {l.strip() for l in existing_lyrics.split("\n")}
|
| 99 |
+
return _clean(result, existing, num_lines)
|
|
|