| --- |
| license: apache-2.0 |
| base_model: MiniMaxAI/MiniMax-Music3 |
| tags: |
| - music |
| - audio |
| - music-generation |
| - minimax-music3 |
| - research |
| - toolkit |
| --- |
| |
| # Music3Lab |
|
|
| **Open research toolkit extending [MiniMax-Music3](https://huggingface.co/MiniMaxAI/MiniMax-Music3) |
| with arbitrary-audio encoding, continuation, inpainting, prepend generation, |
| prompt-free generation, and automated objective evaluation.** |
|
|
| Music3Lab is a reproducible, evidence-first toolkit built *around* the released |
| MiniMax-Music3 weights. It does not modify or redistribute those weights. Every |
| capability below was gated on preregistered objective metrics, and **the |
| failures are published alongside the successes** — they are the more useful part |
| of the research. |
|
|
| > **Honesty note.** This is a research toolkit, not a finished product. Several |
| > headline goals (native WAV→token encoding, true arbitrary-audio history-aware |
| > continuation/inpainting/prepend, direct long-form reference conditioning) were |
| > attempted and **did not pass their gates**. Those negative results, their code, |
| > configs, and exact metrics are all here on purpose. |
|
|
| --- |
|
|
| ## What it does |
|
|
| | Capability | Status | Notes / measured result | |
| |---|---|---| |
| | **Checkpoint audit** of the released weights | ✅ available | Every released tensor classified; proves there is **no** native RVQ waveform tokenizer in `dav.pth`. See [FINDINGS.md](FINDINGS.md). | |
| | **Continuous WAV → Flow-latent encoder** | ✅ available | One-pass ≈1.34 ms (base pilot). External real-music fine-tune improved held-out audio ruler 74.9% and SI-SDR 2.03→8.24 dB, but is a **rejected specialist** (protected teacher latent regressed +13.2%), not a promoted champion. | |
| | **Latent inversion** (research/oracle mode) | ✅ available | Iterative; a 1 s external clip reached 22.2 dB SI-SDR / 0.997 correlation. Slow (~208 s per 1 s) — a teacher, not a real-time encoder. | |
| | **Masked-Flow inpainting** (captured Music3 conditions) | ✅ available | +30.8% latent NMSE, +20.1% hole audio-ruler vs zero-adapter. Captured-condition only. | |
| | **Captured-state style continuation** | ✅ available | 12 s → 16 s, four candidates, objective style/seam ranking. Captured-state only, deterministic-from-frame-0. | |
| | **Full-state resume** | ✅ available | Serializes KV cache + CUDA RNG; reproduces frames + Flow chunks exactly across processes (deterministic backend). | |
| | **Reference-guided append** (CPU) | ✅ available | Appends a chosen reference-style candidate with **bit-exact** source preservation outside the crossfade. | |
| | **Prompt-free generation** | ✅ available | No user text; internal MIR/planning→text bridge. Best-of-N up to 90 s (max policy: 3/8 eligible full-length). | |
| | **Reference-style generation** | 🟡 partial | 8 s, ranked by direct continuous-latent + MIR similarity. Not direct model conditioning or full style transfer. | |
| | **Objective evaluation suite** | 🟡 partial | Integrity, reconstruction, SI-SDR/SNR, correlation, loudness/stereo, anti-copy. No learned musicality/aesthetic judges. | |
| | **Native WAV → Music3 RVQ tokens** | ❌ blocked | Released `dav.pth` has no quantizer/codebooks. (`62000_generator` is a PyTorch **ZIP folder name**, not a component.) | |
| | **Arbitrary-WAV continuation** | ❌ failed | Learned conditioner lost to repeat/roll baselines. | |
| | **Two-sided acoustic FIM (arbitrary WAV)** | ❌ failed | +5.9% ruler vs required +10%; boundaries worse than interpolation. | |
| | **Arbitrary-WAV / waveform prepend** | ❌ failed | Failed seam / anti-copy gates. | |
| | **Native-state Stage-1 residual prediction** | ❌ failed | Near-modal; mean CE 6.834, exact full token rows 0. | |
| | **Direct long-form reference conditioning** | ❌ failed | Tempo drift / early-EOS; no eligible 60 s candidate. | |
| | **Enforceable negative prompts (e.g. "no vocals")** | ❌ not enforceable | Reported honestly as `NOT_ENFORCEABLE`. | |
|
|
| `music3lab status --json` is the authoritative machine-readable capability |
| matrix. Full write-ups are in [`reports/`](reports/) and |
| [`reports/FINAL_RESULTS.md`](reports/FINAL_RESULTS.md). |
|
|
| --- |
|
|
| ## What it is NOT |
|
|
| - ❌ It does **not** include MiniMax-Music3 weights. You download those yourself. |
| - ❌ It does **not** redistribute any audio — not LAION/YouTube audio, not the |
| author's own songs. Only dataset *metadata* (IDs, hashes, splits) is included. |
| - ❌ It is **not** a native audio tokenizer for Music3. That does not exist in |
| the public release (see [FINDINGS.md](FINDINGS.md)). |
|
|
| --- |
|
|
| ## Quickstart |
|
|
| ```bash |
| # 1. Environment (Python 3.10; exact pins in requirements.lock) |
| python -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate |
| pip install -e . # core (CPU inspection/eval) |
| pip install -e ".[capture]" # + diffusers/transformers for generation (GPU) |
| |
| # 2. Get the base model yourself (NOT bundled). See REPRODUCING.md. |
| hf download MiniMaxAI/MiniMax-Music3 --local-dir ./models/minimax-music3 |
| |
| # 3. Inspect the released checkpoint (CPU, no GPU, no weights modified) |
| python inspect_dav.py ./models/minimax-music3/dav.pth --json --sha256 |
| |
| # 4. Machine-readable capability matrix |
| music3lab status --json |
| ``` |
|
|
| Full setup, model download, one-command demo, and benchmark commands: |
| [REPRODUCING.md](REPRODUCING.md). |
|
|
| --- |
|
|
| ## Repository layout |
|
|
| ``` |
| music3lab/ |
| ├── src/music3lab/ # the installable package (tested; layout preserved) |
| │ ├── codec/ # encoders, external fine-tune, native-state experiments |
| │ ├── editing/ # continuation / inpaint / prepend / append |
| │ ├── autonomous/ # champion/challenger promotion + rollback |
| │ ├── inversion*.py # latent inversion (research mode) |
| │ ├── eval.py # objective evaluation |
| │ └── ... # baseline capture, checkpoint audit, pipeline, release |
| ├── configs/ # frozen experiment/training configs (34) |
| ├── scripts/ # runnable training / experiment / benchmark scripts (38) |
| │ └── data/ # LAION downloader (laion_ingest.py, laion_freeze_interim.py) |
| ├── tests/ # focused + adversarial suites (58) |
| ├── reports/ # per-capability write-ups + FINAL_RESULTS.md |
| ├── data/laion_disco/ # dataset METADATA only (IDs, hashes, splits) — no audio |
| ├── checkpoints/ # POINTERS to trained adapters (no weights) — see README there |
| ├── examples/ # how to reproduce demo outputs (no bundled audio) |
| ├── requirements.lock # exact pinned environment |
| ├── LICENSE NOTICE THIRD_PARTY.md MODEL_CARD.md DATA.md TRAINING.md REPRODUCING.md |
| ``` |
|
|
| **Training.** Every trainable component ships its training script + config + |
| tests. See [TRAINING.md](TRAINING.md) for the full table and how to push the |
| open problems (native tokenization, arbitrary-audio editing). |
|
|
| **Note on structure.** The conceptual grouping (encoder / continuation / inpaint |
| / prepend / eval) is preserved *thematically* via the `codec/` and `editing/` |
| subpackages and this map, rather than by physically splitting `src/` — that keeps |
| the 48-test suite green and the package importable for a reproducible v0.1.0. A |
| physical refactor into top-level `encoder/continuation/...` packages is a |
| possible later, separately-tested change. |
|
|
| --- |
|
|
| ## The core finding |
|
|
| The released `dav.pth` is a **continuous** DAV analysis encoder + Gaussian |
| posterior heads + flow model + waveform decoder. It contains **no** RVQ/VQ |
| quantizer, **no** acoustic codebooks, and **no** `generator`/`62000_generator` |
| tensors. The string `62000_generator` is only the root folder name inside the |
| PyTorch ZIP archive — not a model component. Music3's eight-stream token space |
| therefore cannot be produced from an arbitrary WAV with the released weights. |
| Everything Music3Lab does works either in the continuous Flow-latent space or |
| from *captured* generation state. Details and reproduction: [FINDINGS.md](FINDINGS.md). |
|
|
| --- |
|
|
| ## Trained checkpoints & data |
|
|
| - **Adapters/encoders** are released separately (Hugging Face) — see |
| [checkpoints/README.md](checkpoints/README.md) and [MODEL_CARD.md](MODEL_CARD.md). |
| ⚠️ They are derivatives of MiniMax-Music3 and may be governed by its license; |
| confirm before redistributing. |
| - **Dataset**: only LAION-DISCO-12M metadata + a downloader are shipped. No audio. |
| See [DATA.md](DATA.md). |
|
|
| --- |
|
|
| ## License |
|
|
| Original Music3Lab code: **Apache-2.0** ([LICENSE](LICENSE), [NOTICE](NOTICE)). |
| Third-party components and their separate licenses: [THIRD_PARTY.md](THIRD_PARTY.md). |
|
|
