Music3Lab

Open research toolkit extending MiniMax-Music3 with arbitrary-audio encoding, continuation, inpainting, prepend generation, prompt-free generation, and automated objective evaluation.

Music3Lab is a reproducible, evidence-first toolkit built around the released MiniMax-Music3 weights. It does not modify or redistribute those weights. Every capability below was gated on preregistered objective metrics, and the failures are published alongside the successes — they are the more useful part of the research.

Honesty note. This is a research toolkit, not a finished product. Several headline goals (native WAV→token encoding, true arbitrary-audio history-aware continuation/inpainting/prepend, direct long-form reference conditioning) were attempted and did not pass their gates. Those negative results, their code, configs, and exact metrics are all here on purpose.

---

What it does

Capability	Status	Notes / measured result
Checkpoint audit of the released weights	✅ available	Every released tensor classified; proves there is no native RVQ waveform tokenizer in dav.pth. See FINDINGS.md.
Continuous WAV → Flow-latent encoder	✅ available	One-pass ≈1.34 ms (base pilot). External real-music fine-tune improved held-out audio ruler 74.9% and SI-SDR 2.03→8.24 dB, but is a rejected specialist (protected teacher latent regressed +13.2%), not a promoted champion.
Latent inversion (research/oracle mode)	✅ available	Iterative; a 1 s external clip reached 22.2 dB SI-SDR / 0.997 correlation. Slow (~208 s per 1 s) — a teacher, not a real-time encoder.
Masked-Flow inpainting (captured Music3 conditions)	✅ available	+30.8% latent NMSE, +20.1% hole audio-ruler vs zero-adapter. Captured-condition only.
Captured-state style continuation	✅ available	12 s → 16 s, four candidates, objective style/seam ranking. Captured-state only, deterministic-from-frame-0.
Full-state resume	✅ available	Serializes KV cache + CUDA RNG; reproduces frames + Flow chunks exactly across processes (deterministic backend).
Reference-guided append (CPU)	✅ available	Appends a chosen reference-style candidate with bit-exact source preservation outside the crossfade.
Prompt-free generation	✅ available	No user text; internal MIR/planning→text bridge. Best-of-N up to 90 s (max policy: 3/8 eligible full-length).
Reference-style generation	🟡 partial	8 s, ranked by direct continuous-latent + MIR similarity. Not direct model conditioning or full style transfer.
Objective evaluation suite	🟡 partial	Integrity, reconstruction, SI-SDR/SNR, correlation, loudness/stereo, anti-copy. No learned musicality/aesthetic judges.
Native WAV → Music3 RVQ tokens	❌ blocked	Released dav.pth has no quantizer/codebooks. (62000_generator is a PyTorch ZIP folder name, not a component.)
Arbitrary-WAV continuation	❌ failed	Learned conditioner lost to repeat/roll baselines.
Two-sided acoustic FIM (arbitrary WAV)	❌ failed	+5.9% ruler vs required +10%; boundaries worse than interpolation.
Arbitrary-WAV / waveform prepend	❌ failed	Failed seam / anti-copy gates.
Native-state Stage-1 residual prediction	❌ failed	Near-modal; mean CE 6.834, exact full token rows 0.
Direct long-form reference conditioning	❌ failed	Tempo drift / early-EOS; no eligible 60 s candidate.
Enforceable negative prompts (e.g. "no vocals")	❌ not enforceable	Reported honestly as NOT_ENFORCEABLE.

music3lab status --json is the authoritative machine-readable capability matrix. Full write-ups are in reports/ and reports/FINAL_RESULTS.md.

---

What it is NOT

• ❌ It does not include MiniMax-Music3 weights. You download those yourself.
• ❌ It does not redistribute any audio — not LAION/YouTube audio, not the author's own songs. Only dataset metadata (IDs, hashes, splits) is included.
• ❌ It is not a native audio tokenizer for Music3. That does not exist in the public release (see FINDINGS.md).

---

Quickstart

# 1. Environment (Python 3.10; exact pins in requirements.lock)
python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -e .                                  # core (CPU inspection/eval)
pip install -e ".[capture]"                       # + diffusers/transformers for generation (GPU)

# 2. Get the base model yourself (NOT bundled). See REPRODUCING.md.
hf download MiniMaxAI/MiniMax-Music3 --local-dir ./models/minimax-music3

# 3. Inspect the released checkpoint (CPU, no GPU, no weights modified)
python inspect_dav.py ./models/minimax-music3/dav.pth --json --sha256

# 4. Machine-readable capability matrix
music3lab status --json

Full setup, model download, one-command demo, and benchmark commands: REPRODUCING.md.

---

Repository layout

music3lab/
├── src/music3lab/        # the installable package (tested; layout preserved)
│   ├── codec/            # encoders, external fine-tune, native-state experiments
│   ├── editing/          # continuation / inpaint / prepend / append
│   ├── autonomous/       # champion/challenger promotion + rollback
│   ├── inversion*.py     # latent inversion (research mode)
│   ├── eval.py           # objective evaluation
│   └── ...               # baseline capture, checkpoint audit, pipeline, release
├── configs/              # frozen experiment/training configs (34)
├── scripts/              # runnable training / experiment / benchmark scripts (38)
│   └── data/             # LAION downloader (laion_ingest.py, laion_freeze_interim.py)
├── tests/                # focused + adversarial suites (58)
├── reports/              # per-capability write-ups + FINAL_RESULTS.md
├── data/laion_disco/     # dataset METADATA only (IDs, hashes, splits) — no audio
├── checkpoints/          # POINTERS to trained adapters (no weights) — see README there
├── examples/             # how to reproduce demo outputs (no bundled audio)
├── requirements.lock     # exact pinned environment
├── LICENSE  NOTICE  THIRD_PARTY.md  MODEL_CARD.md  DATA.md  TRAINING.md  REPRODUCING.md

Training. Every trainable component ships its training script + config + tests. See TRAINING.md for the full table and how to push the open problems (native tokenization, arbitrary-audio editing).

Note on structure. The conceptual grouping (encoder / continuation / inpaint / prepend / eval) is preserved thematically via the codec/ and editing/ subpackages and this map, rather than by physically splitting src/ — that keeps the 48-test suite green and the package importable for a reproducible v0.1.0. A physical refactor into top-level encoder/continuation/... packages is a possible later, separately-tested change.

---

The core finding

The released dav.pth is a continuous DAV analysis encoder + Gaussian posterior heads + flow model + waveform decoder. It contains no RVQ/VQ quantizer, no acoustic codebooks, and no generator/62000_generator tensors. The string 62000_generator is only the root folder name inside the PyTorch ZIP archive — not a model component. Music3's eight-stream token space therefore cannot be produced from an arbitrary WAV with the released weights. Everything Music3Lab does works either in the continuous Flow-latent space or from captured generation state. Details and reproduction: FINDINGS.md.

---

Trained checkpoints & data

• Adapters/encoders are released separately (Hugging Face) — see checkpoints/README.md and MODEL_CARD.md. ⚠️ They are derivatives of MiniMax-Music3 and may be governed by its license; confirm before redistributing.
• Dataset: only LAION-DISCO-12M metadata + a downloader are shipped. No audio. See DATA.md.

---

License

Original Music3Lab code: Apache-2.0 (LICENSE, NOTICE). Third-party components and their separate licenses: THIRD_PARTY.md.