| --- |
| license: apache-2.0 |
| --- |
| |
| > Mirror of [iflytek/AudioFly](https://www.modelscope.cn/models/iflytek/AudioFly) from ModelScope. All credit to the original authors. Not an official release from them. License unchanged (apache-2.0). |
|
|
| # Model Card for AudioFly |
|
|
| ## Model Introduction |
| AudioFly is an audio generation model. It synthesizes sound effects based on textual descriptions. The model can produce high-quality audio at a sampling rate of 44.1 kHz. The generated audio shows strong alignment with the prompt text. |
|
|
| AudioFly adopts the Latent Diffusion Model architecture. The model has 1 billion parameters and is trained on a large and diverse corpus. The training data include open-source datasets, such as AudioSet, AudioCaps, and TUT, as well as proprietary internal data. The model performs well in both single-event and multi-event scenarios. In these cases, the generated audio accurately reflects the described content. AudioFly achieves superior performance compared to previous audio generation models on the AudioCaps dataset. |
|
|
|
|
| ## Evaluation Results |
|
|
| The experimental results are reported on the AudioCaps dataset. For the baseline models, we reused the evaluation results from [STABLE AUDIO OPEN](https://arxiv.org/pdf/2407.14358). We followed the same evaluation methodology to ensure consistency. The evaluation results are shown as follows. |
|
|
|
|
| | Model | FD ↓ | KL ↓ | CLAP ↑ | |
| |--------------------|----------|------------|---------| |
| | AudioLDM2-48kHz | 101.11 | 2.04 | 0.37 | |
| | AudioGen-medium | 186.53 | 1.42 | **0.45** | |
| | Stable Audio 1.0 | 103.66 | 2.89 | 0.24 | |
| | Stable Audio 2.0 | 110.62 | 2.70 | 0.23 | |
| | Stable Audio Open | 78.24 | 2.14 | 0.29 | |
| | AudioFly | **40.1** | **1.35** | **0.45** | |
|
|
|
|
|
|
|
|
|
|
|
|
| ## Usage |
|
|
| **Requirements** |
|
|
| We recommend setting up the runtime environment using the provided configuration file by running: |
| ```bash |
| pip install -r requirements.txt |
| # make sure to set the PYTHONPATH to include the AudioFly project root |
| export PYTHONPATH=/path/to/AudioFly:$PYTHONPATH |
| cd /path/to/AudioFly |
| ``` |
|
|
| **Quickstart** |
|
|
|
|
|
|
| ```python |
| import yaml |
| import torch |
| from ldm.utils.util import instantiate_from_config |
| |
| |
| configs = yaml.load(open('./config/config.yaml', "r"), Loader=yaml.FullLoader) |
| model = instantiate_from_config(configs["model"]) |
| |
| checkpoint = torch.load('./models/ldm/model.ckpt') |
| model.load_state_dict(checkpoint, strict=False) |
| model.eval() |
| model = model.cuda() |
| text = 'Fierce winds howl through the valley' |
| name = 'result' |
| savedir = './result' |
| model.generate_sample( |
| textlist=[text], |
| name=name, |
| cfg=3.5,# Guidance scale (controls how strongly generation follows the text prompt）; not recommended to change |
| ddim_steps=200, # Number of denoising steps in the diffusion process; not recommended to change |
| outputdir=f"{savedir}") |
| ``` |
| ## License |
| AudioFly is licensed under Apache 2.0. |
|
|
