Improve model card and add sample usage
#1
by
nielsr
HF Staff
- opened
README.md
CHANGED
|
@@ -1,48 +1,67 @@
|
|
| 1 |
---
|
| 2 |
-
license: cc-by-nc-4.0
|
| 3 |
language:
|
| 4 |
- zh
|
| 5 |
- en
|
| 6 |
- ja
|
| 7 |
- ko
|
| 8 |
- es
|
|
|
|
| 9 |
pipeline_tag: text-to-audio
|
| 10 |
tags:
|
| 11 |
- music
|
| 12 |
- art
|
| 13 |
---
|
| 14 |
|
| 15 |
-
# HeartMuLa
|
| 16 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
## Model Details
|
| 19 |
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
| 25 |
-
- **License:** [CC-BY-NC 4.0]
|
| 26 |
|
| 27 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
-
|
| 30 |
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
- **Demo:** https://heartmula.github.io/
|
| 34 |
-
- **HeartMuLa-oss-3B:** https://huggingface.co/HeartMuLa/HeartMuLa-oss-3B
|
| 35 |
-
- **HeartCodec-oss:** https://huggingface.co/HeartMuLa/HeartCodec-oss
|
| 36 |
-
- **HeartTranscriptor-oss:** https://huggingface.co/HeartMuLa/HeartTranscriptor-oss
|
| 37 |
|
|
|
|
|
|
|
| 38 |
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
## Citation
|
| 44 |
|
| 45 |
-
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
| 46 |
If you find HeartMuLa useful, please cite:
|
| 47 |
|
| 48 |
```bibtex
|
|
@@ -57,6 +76,5 @@ If you find HeartMuLa useful, please cite:
|
|
| 57 |
}
|
| 58 |
```
|
| 59 |
|
| 60 |
-
|
| 61 |
## Contact
|
| 62 |
If you are interested in HeartMuLa, feel free to reach us at heartmula.ai@gmail.com
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
language:
|
| 3 |
- zh
|
| 4 |
- en
|
| 5 |
- ja
|
| 6 |
- ko
|
| 7 |
- es
|
| 8 |
+
license: cc-by-nc-4.0
|
| 9 |
pipeline_tag: text-to-audio
|
| 10 |
tags:
|
| 11 |
- music
|
| 12 |
- art
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# HeartMuLa: A Family of Open Sourced Music Foundation Models
|
| 16 |
|
| 17 |
+
HeartMuLa is a family of open-source music foundation models designed to advance large-scale music understanding and generation across diverse tasks and modalities. It is an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio).
|
| 18 |
+
|
| 19 |
+
- **Project Page:** [https://heartmula.github.io/](https://heartmula.github.io/)
|
| 20 |
+
- **Repository:** [https://github.com/HeartMuLa/heartlib](https://github.com/HeartMuLa/heartlib)
|
| 21 |
+
- **Paper:** [HeartMuLa: A Family of Open Sourced Music Foundation Models](https://arxiv.org/abs/2601.10547)
|
| 22 |
+
- **Demo:** [https://heartmula.github.io/](https://heartmula.github.io/)
|
| 23 |
|
| 24 |
## Model Details
|
| 25 |
|
| 26 |
+
The HeartMuLa framework consists of four major components:
|
| 27 |
+
1. **HeartMuLa**: A music language model that generates music conditioned on lyrics and tags with multilingual support.
|
| 28 |
+
2. **HeartCodec**: A low-frame-rate (12.5 Hz), high-fidelity music codec tokenizer that captures long-range musical structure.
|
| 29 |
+
3. **HeartTranscriptor**: A robust lyric recognition model optimized for real-world music scenarios.
|
| 30 |
+
4. **HeartCLAP**: An audio-text alignment model for music descriptions and cross-modal retrieval.
|
| 31 |
|
| 32 |
+
## Installation
|
| 33 |
|
| 34 |
+
We recommend using `python=3.10` for local deployment. Clone the repository and install locally:
|
|
|
|
| 35 |
|
| 36 |
+
```bash
|
| 37 |
+
git clone https://github.com/HeartMuLa/heartlib.git
|
| 38 |
+
cd heartlib
|
| 39 |
+
pip install -e .
|
| 40 |
+
```
|
| 41 |
|
| 42 |
+
## Sample Usage
|
| 43 |
|
| 44 |
+
### Download Checkpoints
|
| 45 |
+
First, download the pretrained checkpoints into a `./ckpt` folder as described in the [GitHub README](https://github.com/HeartMuLa/heartlib).
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
+
### Inference
|
| 48 |
+
To generate music conditioned on lyrics and tags, run the following command:
|
| 49 |
|
| 50 |
+
```bash
|
| 51 |
+
python ./examples/run_music_generation.py --model_path=./ckpt --version="3B"
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
By default, this command generates a piece of music based on the lyrics and tags provided in the `./assets` folder. The output will be saved as `./assets/output.mp3`.
|
| 55 |
|
| 56 |
+
**Key Parameters:**
|
| 57 |
+
- `--model_path`: Path to the pretrained model checkpoint.
|
| 58 |
+
- `--lyrics`: Path to the lyrics file (e.g., `./assets/lyrics.txt`).
|
| 59 |
+
- `--tags`: Path to the tags file (e.g., `./assets/tags.txt`).
|
| 60 |
+
- `--save_path`: Output audio file path.
|
| 61 |
+
- `--version`: The version of HeartMuLa (choose `3B`).
|
| 62 |
|
| 63 |
## Citation
|
| 64 |
|
|
|
|
| 65 |
If you find HeartMuLa useful, please cite:
|
| 66 |
|
| 67 |
```bibtex
|
|
|
|
| 76 |
}
|
| 77 |
```
|
| 78 |
|
|
|
|
| 79 |
## Contact
|
| 80 |
If you are interested in HeartMuLa, feel free to reach us at heartmula.ai@gmail.com
|