--- license: apache-2.0 base_model: MiniMaxAI/MiniMax-Music3 tags: - music - audio - music-generation - minimax-music3 - research - toolkit --- # Music3Lab **Open research toolkit extending [MiniMax-Music3](https://huggingface.co/MiniMaxAI/MiniMax-Music3) with arbitrary-audio encoding, continuation, inpainting, prepend generation, prompt-free generation, and automated objective evaluation.** Music3Lab is a reproducible, evidence-first toolkit built *around* the released MiniMax-Music3 weights. It does not modify or redistribute those weights. Every capability below was gated on preregistered objective metrics, and **the failures are published alongside the successes** — they are the more useful part of the research. > **Honesty note.** This is a research toolkit, not a finished product. Several > headline goals (native WAV→token encoding, true arbitrary-audio history-aware > continuation/inpainting/prepend, direct long-form reference conditioning) were > attempted and **did not pass their gates**. Those negative results, their code, > configs, and exact metrics are all here on purpose. --- ## What it does | Capability | Status | Notes / measured result | |---|---|---| | **Checkpoint audit** of the released weights | ✅ available | Every released tensor classified; proves there is **no** native RVQ waveform tokenizer in `dav.pth`. See [FINDINGS.md](FINDINGS.md). | | **Continuous WAV → Flow-latent encoder** | ✅ available | One-pass ≈1.34 ms (base pilot). External real-music fine-tune improved held-out audio ruler 74.9% and SI-SDR 2.03→8.24 dB, but is a **rejected specialist** (protected teacher latent regressed +13.2%), not a promoted champion. | | **Latent inversion** (research/oracle mode) | ✅ available | Iterative; a 1 s external clip reached 22.2 dB SI-SDR / 0.997 correlation. Slow (~208 s per 1 s) — a teacher, not a real-time encoder. | | **Masked-Flow inpainting** (captured Music3 conditions) | ✅ available | +30.8% latent NMSE, +20.1% hole audio-ruler vs zero-adapter. Captured-condition only. | | **Captured-state style continuation** | ✅ available | 12 s → 16 s, four candidates, objective style/seam ranking. Captured-state only, deterministic-from-frame-0. | | **Full-state resume** | ✅ available | Serializes KV cache + CUDA RNG; reproduces frames + Flow chunks exactly across processes (deterministic backend). | | **Reference-guided append** (CPU) | ✅ available | Appends a chosen reference-style candidate with **bit-exact** source preservation outside the crossfade. | | **Prompt-free generation** | ✅ available | No user text; internal MIR/planning→text bridge. Best-of-N up to 90 s (max policy: 3/8 eligible full-length). | | **Reference-style generation** | 🟡 partial | 8 s, ranked by direct continuous-latent + MIR similarity. Not direct model conditioning or full style transfer. | | **Objective evaluation suite** | 🟡 partial | Integrity, reconstruction, SI-SDR/SNR, correlation, loudness/stereo, anti-copy. No learned musicality/aesthetic judges. | | **Native WAV → Music3 RVQ tokens** | ❌ blocked | Released `dav.pth` has no quantizer/codebooks. (`62000_generator` is a PyTorch **ZIP folder name**, not a component.) | | **Arbitrary-WAV continuation** | ❌ failed | Learned conditioner lost to repeat/roll baselines. | | **Two-sided acoustic FIM (arbitrary WAV)** | ❌ failed | +5.9% ruler vs required +10%; boundaries worse than interpolation. | | **Arbitrary-WAV / waveform prepend** | ❌ failed | Failed seam / anti-copy gates. | | **Native-state Stage-1 residual prediction** | ❌ failed | Near-modal; mean CE 6.834, exact full token rows 0. | | **Direct long-form reference conditioning** | ❌ failed | Tempo drift / early-EOS; no eligible 60 s candidate. | | **Enforceable negative prompts (e.g. "no vocals")** | ❌ not enforceable | Reported honestly as `NOT_ENFORCEABLE`. | `music3lab status --json` is the authoritative machine-readable capability matrix. Full write-ups are in [`reports/`](reports/) and [`reports/FINAL_RESULTS.md`](reports/FINAL_RESULTS.md). --- ## What it is NOT - ❌ It does **not** include MiniMax-Music3 weights. You download those yourself. - ❌ It does **not** redistribute any audio — not LAION/YouTube audio, not the author's own songs. Only dataset *metadata* (IDs, hashes, splits) is included. - ❌ It is **not** a native audio tokenizer for Music3. That does not exist in the public release (see [FINDINGS.md](FINDINGS.md)). --- ## Quickstart ```bash # 1. Environment (Python 3.10; exact pins in requirements.lock) python -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate pip install -e . # core (CPU inspection/eval) pip install -e ".[capture]" # + diffusers/transformers for generation (GPU) # 2. Get the base model yourself (NOT bundled). See REPRODUCING.md. hf download MiniMaxAI/MiniMax-Music3 --local-dir ./models/minimax-music3 # 3. Inspect the released checkpoint (CPU, no GPU, no weights modified) python inspect_dav.py ./models/minimax-music3/dav.pth --json --sha256 # 4. Machine-readable capability matrix music3lab status --json ``` Full setup, model download, one-command demo, and benchmark commands: [REPRODUCING.md](REPRODUCING.md). --- ## Repository layout ``` music3lab/ ├── src/music3lab/ # the installable package (tested; layout preserved) │ ├── codec/ # encoders, external fine-tune, native-state experiments │ ├── editing/ # continuation / inpaint / prepend / append │ ├── autonomous/ # champion/challenger promotion + rollback │ ├── inversion*.py # latent inversion (research mode) │ ├── eval.py # objective evaluation │ └── ... # baseline capture, checkpoint audit, pipeline, release ├── configs/ # frozen experiment/training configs (34) ├── scripts/ # runnable training / experiment / benchmark scripts (38) │ └── data/ # LAION downloader (laion_ingest.py, laion_freeze_interim.py) ├── tests/ # focused + adversarial suites (58) ├── reports/ # per-capability write-ups + FINAL_RESULTS.md ├── data/laion_disco/ # dataset METADATA only (IDs, hashes, splits) — no audio ├── checkpoints/ # POINTERS to trained adapters (no weights) — see README there ├── examples/ # how to reproduce demo outputs (no bundled audio) ├── requirements.lock # exact pinned environment ├── LICENSE NOTICE THIRD_PARTY.md MODEL_CARD.md DATA.md TRAINING.md REPRODUCING.md ``` **Training.** Every trainable component ships its training script + config + tests. See [TRAINING.md](TRAINING.md) for the full table and how to push the open problems (native tokenization, arbitrary-audio editing). **Note on structure.** The conceptual grouping (encoder / continuation / inpaint / prepend / eval) is preserved *thematically* via the `codec/` and `editing/` subpackages and this map, rather than by physically splitting `src/` — that keeps the 48-test suite green and the package importable for a reproducible v0.1.0. A physical refactor into top-level `encoder/continuation/...` packages is a possible later, separately-tested change. --- ## The core finding The released `dav.pth` is a **continuous** DAV analysis encoder + Gaussian posterior heads + flow model + waveform decoder. It contains **no** RVQ/VQ quantizer, **no** acoustic codebooks, and **no** `generator`/`62000_generator` tensors. The string `62000_generator` is only the root folder name inside the PyTorch ZIP archive — not a model component. Music3's eight-stream token space therefore cannot be produced from an arbitrary WAV with the released weights. Everything Music3Lab does works either in the continuous Flow-latent space or from *captured* generation state. Details and reproduction: [FINDINGS.md](FINDINGS.md). --- ## Trained checkpoints & data - **Adapters/encoders** are released separately (Hugging Face) — see [checkpoints/README.md](checkpoints/README.md) and [MODEL_CARD.md](MODEL_CARD.md). ⚠️ They are derivatives of MiniMax-Music3 and may be governed by its license; confirm before redistributing. - **Dataset**: only LAION-DISCO-12M metadata + a downloader are shipped. No audio. See [DATA.md](DATA.md). --- ## License Original Music3Lab code: **Apache-2.0** ([LICENSE](LICENSE), [NOTICE](NOTICE)). Third-party components and their separate licenses: [THIRD_PARTY.md](THIRD_PARTY.md).