Duplicate from OpenMuQ/MuQ-MuLan-large

Co-authored-by: Haina Zhu <juhayna@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,111 @@
+---
+license: cc-by-nc-4.0
+language:
+- en
+- zh
+pipeline_tag: audio-classification
+tags:
+- music
+---
+
+# MuQ & MuQ-MuLan
+
+<div>
+  <a href='#'><img alt="Static Badge" src="https://img.shields.io/badge/Python-3.8%2B-blue?logo=python&logoColor=white"></a>
+  <a href='https://arxiv.org/abs/2501.01108'><img alt="Static Badge" src="https://img.shields.io/badge/arXiv-2501.01108-%23b31b1b?logo=arxiv&link=https%3A%2F%2Farxiv.org%2F"></a>
+  <a href='https://huggingface.co/OpenMuQ'><img alt="Static Badge" src="https://img.shields.io/badge/huggingface-OpenMuQ-%23FFD21E?logo=huggingface&link=https%3A%2F%2Fhuggingface.co%2FOpenMuQ"></a>
+  <a href='https://pytorch.org/'><img alt="Static Badge" src="https://img.shields.io/badge/framework-PyTorch-%23EE4C2C?logo=pytorch"></a>
+  <a href='https://pypi.org/project/muq'><img alt="Static Badge" src="https://img.shields.io/badge/pip%20install-muq-green?logo=PyPI&logoColor=white&link=https%3A%2F%2Fpypi.org%2Fproject%2Fmuq"></a>
+</div>
+
+
+This is the official repository for the paper *"**MuQ**: Self-Supervised **Mu**sic Representation Learning
+ with Mel Residual Vector **Q**uantization"*. For more detailed information, we strongly recommend referring to https://github.com/tencent-ailab/MuQ and the [paper]((https://arxiv.org/abs/2501.01108)). 
+
+In this repo, the following models are released:
+
+- **MuQ**(see [this link](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter)): A large music foundation model pre-trained via Self-Supervised Learning (SSL), achieving SOTA in various MIR tasks.
+- **MuQ-MuLan**(see [this link](https://huggingface.co/OpenMuQ/MuQ-MuLan-large)): A music-text joint embedding model trained via contrastive learning, supporting both English and Chinese texts.
+
+
+## Usage
+
+To begin with, please use pip to install the official `muq` lib, and ensure that your `python>=3.8`:
+```bash
+pip3 install muq
+```
+
+
+Using **MuQ-MuLan** to extract the music and text embeddings and calculate the similarity:
+```python
+import torch, librosa
+from muq import MuQMuLan
+
+# This will automatically fetch checkpoints from huggingface
+device = 'cuda'
+mulan = MuQMuLan.from_pretrained("OpenMuQ/MuQ-MuLan-large")
+mulan = mulan.to(device).eval()
+
+# Extract music embeddings
+wav, sr = librosa.load("path/to/music_audio.wav", sr = 24000)
+wavs = torch.tensor(wav).unsqueeze(0).to(device) 
+with torch.no_grad():
+    audio_embeds = mulan(wavs = wavs) 
+
+# Extract text embeddings (texts can be in English or Chinese)
+texts = ["classical genres, hopeful mood, piano.", "一首适合海边风景的小提琴曲，节奏欢快"]
+with torch.no_grad():
+    text_embeds = mulan(texts = texts)
+
+# Calculate dot product similarity
+sim = mulan.calc_similarity(audio_embeds, text_embeds)
+print(sim)
+```
+
+
+To extract music audio features using **MuQ**:
+```python
+import torch, librosa
+from muq import MuQ
+
+device = 'cuda'
+wav, sr = librosa.load("path/to/music_audio.wav", sr = 24000)
+wavs = torch.tensor(wav).unsqueeze(0).to(device) 
+
+# This will automatically fetch the checkpoint from huggingface
+muq = MuQ.from_pretrained("OpenMuQ/MuQ-large-msd-iter")
+muq = muq.to(device).eval()
+
+with torch.no_grad():
+    output = muq(wavs, output_hidden_states=True)
+
+print('Total number of layers: ', len(output.hidden_states))
+print('Feature shape: ', output.last_hidden_state.shape)
+
+```
+
+## Model Checkpoints
+
+| Model Name | Parameters | Data | HuggingFace🤗 |
+| ----------- | --- | ---  | ----------- |
+| MuQ    | ~300M  | MSD dataset | [OpenMuQ/MuQ-large-msd-iter](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter)       |
+| MuQ-MuLan  | ~700M | music-text pairs | [OpenMuQ/MuQ-MuLan-large](https://huggingface.co/OpenMuQ/MuQ-MuLan-large)       |
+
+**Note**: Please note that the open-sourced MuQ was trained on the Million Song Dataset. Due to differences in dataset size, the open-sourced model may not achieve the same level of performance as reported in the paper.
+
+## License
+
+The code is released under the MIT license.
+
+The model weights (MuQ-large-msd-iter, MuQ-MuLan-large) are released under the CC-BY-NC 4.0 license. 
+
+## Citation
+
+```
+@article{zhu2025muq,
+      title={MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization}, 
+      author={Haina Zhu and Yizhi Zhou and Hangting Chen and Jianwei Yu and Ziyang Ma and Rongzhi Gu and Yi Luo and Wei Tan and Xie Chen},
+      journal={arXiv preprint arXiv:2501.01108},
+      year={2025}
+} 
+```

config.json

@@ -0,0 +1,41 @@
+{
+  "mulan": {
+    "sr": 24000,
+    "clip_secs": 10,
+    "dim_latent": 512,
+    "decoupled_contrastive_learning": true,
+    "hierarchical_contrastive_loss": false,
+    "hierarchical_contrastive_loss_layers": null,
+    "sigmoid_contrastive_loss": false,
+    "rank_contrast": true
+  },
+  "audio_model": {
+    "name": "OpenMuQ/MuQ-large-msd-iter",
+    "model_dim": 1024,
+    "use_layer_idx": -1
+  },
+  "text_model": {
+    "name": "xlm-roberta-base",
+    "model_dim": null,
+    "use_layer_idx": -1
+  },
+  "audio_transformer": {
+    "dim": 768,
+    "tf_depth": 0,
+    "heads": 8,
+    "dim_head": 64,
+    "attn_dropout": 0,
+    "ff_dropout": 0,
+    "ff_mult": 4
+  },
+  "text_transformer": {
+    "dim": 768,
+    "tf_depth": 8,
+    "max_seq_len": 1024,
+    "dim_head": 64,
+    "heads": 8,
+    "attn_dropout": 0,
+    "ff_dropout": 0,
+    "ff_mult": 4
+  }
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d42ae3f7cb9b66759ee0089ddc70e2f28b130c2d8ba621457358272d32dd0444
+size 2653954401