Mirror iflytek/AudioFly from ModelScope

README.md

@@ -0,0 +1,74 @@
+---
+license: apache-2.0
+---
+
+> Mirror of [iflytek/AudioFly](https://www.modelscope.cn/models/iflytek/AudioFly) from ModelScope. All credit to the original authors. Not an official release from them. License unchanged (apache-2.0).
+
+# Model Card for AudioFly
+
+## Model Introduction
+AudioFly is an audio generation model. It synthesizes sound effects based on textual descriptions. The model can produce high-quality audio at a sampling rate of 44.1 kHz. The generated audio shows strong alignment with the prompt text.
+
+AudioFly adopts the Latent Diffusion Model architecture. The model has 1 billion parameters and is trained on a large and diverse corpus. The training data include open-source datasets, such as AudioSet, AudioCaps, and TUT, as well as proprietary internal data. The model performs well in both single-event and multi-event scenarios. In these cases, the generated audio accurately reflects the described content. AudioFly achieves superior performance compared to previous audio generation models on the AudioCaps dataset.
+
+
+## Evaluation Results
+
+The experimental results are reported on the AudioCaps dataset. For the baseline models, we  reused the evaluation results from [STABLE AUDIO OPEN](https://arxiv.org/pdf/2407.14358). We followed the same evaluation methodology to ensure consistency. The evaluation results are shown as follows.
+
+
+| Model               | FD ↓     | KL ↓        | CLAP ↑    | 
+|--------------------|----------|------------|---------|
+| AudioLDM2-48kHz    | 101.11   | 2.04       | 0.37    | 
+| AudioGen-medium    | 186.53   | 1.42       | **0.45**    | 
+| Stable Audio 1.0   | 103.66   | 2.89       | 0.24    | 
+| Stable Audio 2.0   | 110.62   | 2.70       | 0.23    | 
+| Stable Audio Open  | 78.24    | 2.14       | 0.29    | 
+| AudioFly              | **40.1**    | **1.35**       | **0.45**    | 
+
+
+
+
+
+
+## Usage
+
+**Requirements**
+
+We recommend setting up the runtime environment using the provided configuration file by running: 
+```bash
+pip install -r requirements.txt 
+# make sure to set the PYTHONPATH to include the AudioFly project root
+export PYTHONPATH=/path/to/AudioFly:$PYTHONPATH
+cd /path/to/AudioFly
+```
+
+**Quickstart**
+
+
+
+```python
+import yaml
+import torch
+from ldm.utils.util import instantiate_from_config
+
+
+configs = yaml.load(open('./config/config.yaml', "r"), Loader=yaml.FullLoader)
+model = instantiate_from_config(configs["model"])
+
+checkpoint = torch.load('./models/ldm/model.ckpt')
+model.load_state_dict(checkpoint, strict=False)
+model.eval()
+model = model.cuda()
+text = 'Fierce winds howl through the valley' 
+name = 'result'
+savedir = './result'
+model.generate_sample(
+        textlist=[text],
+        name=name,
+        cfg=3.5,# Guidance scale (controls how strongly generation follows the text prompt）; not recommended to change
+        ddim_steps=200,  # Number of denoising steps in the diffusion process; not recommended to change
+        outputdir=f"{savedir}")
+```
+## License
+AudioFly is licensed under Apache 2.0.