music3lab / DATA.md
coolpoodle's picture
code and training scripts
90884df verified
|
Raw
History Blame Contribute Delete
3.02 kB

Data

Music3Lab ships dataset metadata only. No audio of any kind is included or redistributed.

LAION-DISCO-12M (external real-music corpus)

LAION-DISCO-12M is an index dataset: it contains YouTube video IDs and tags, licensed Apache-2.0 for the metadata, and explicitly contains no audio samples. Audio referenced by those IDs is not part of the dataset and is not redistributed here.

What is shipped in this repo (data/laion_disco/)

  • interim_tranche_678/manifest.jsonl — the 678 accepted tracks as records of: YouTube source_id/source_url, LAION source_dataset_revision, canonical audio content hashes (canonical_sha256, pcm_sha256), sample rate / channels / frame counts, and split assignment. No audio.
  • interim_tranche_678/splits.json — deterministic, source-level train/val/heldout split (algorithm, seed, assignment, counts).
  • interim_tranche_678/summary.json, candidates.summary.json, tranche.stopped.json, environment.json — provenance and stop-reason records.
  • UPSTREAM_README.md — the corpus builder's own notes.

What is NOT shipped

  • ❌ Any downloaded audio (files/*.wav, canonical/).
  • ❌ The full 20k candidate list is included only as a summary; the raw candidates.jsonl and the private author-song exclusion hashes (known_user22_*) are not published.

Reproducing the corpus

scripts/data/laion_ingest.py deterministically selects candidates from the pinned LAION revision (6e7bf3758a77301e46a715af894fefd79bb1da53), then uses yt-dlp + ffmpeg to fetch and canonicalize audio on your own machine. scripts/data/laion_freeze_interim.py freezes an immutable, hash-verified tranche and split. Both scripts contain instance-specific absolute paths from the original run and need their ROOT/paths adjusted before use.

⚠️ You are responsible for complying with YouTube's Terms of Service and applicable copyright when downloading any audio. Music3Lab neither hosts nor distributes that audio. The original run stopped at 678/2,000 tracks when YouTube returned bot-verification challenges — expect the same and throttle accordingly.

The author's own songs (private)

The project also evaluated on the author's own tracks (8, then 22 after deduplication). These are private by default and are not in this repo, not in the LAION splits, and were never used for training or checkpoint selection — only as out-of-distribution evaluation. Only their content hashes were used, to guarantee they never leaked into training. If you want a public demo, publish only short excerpts of audio you own and are licensed to share.

MiniMax-Music3 teacher data

Some experiments use WAV/latent/token pairs captured from MiniMax-Music3 generations. These captures are derivatives of the base model and are not shipped. scripts/ can regenerate them from the model you download yourself.