README.md

---
license: apache-2.0
---
<div align="center">
<h1>FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications</h1>
</div>

#### 👉🏻 [FireRedTTS Paper](https://arxiv.org/abs/2409.03283) 👈🏻

#### 👉🏻 [FireRedTTS Demos](https://fireredteam.github.io/demos/firered_tts/) 👈🏻

#### 👉🏻 [FireRedTTS Space (Interactive Demo)](https://huggingface.co/spaces/fireredteam/FireRedTTS) 👈🏻

## News

- [2024/10/17] 🔥 We release [new rich-punctuation model](https://huggingface.co/fireredteam/FireRedTTS/tree/main), offering expanded punctuation coverage and enhanced audio production consistency. In addition, we have strengthened the capabilities of the text front-end and enhanced the stability of synthesis.
- [2024/09/26] 🔥 Our model is already available on [huggingface space](https://huggingface.co/spaces/fireredteam/FireRedTTS)，try it through the interactive interface.
- [2024/09/20] 🔥 We release the pre-trained checkpoints and inference code.
- [2024/09/06] 🔥 We release the [technical report](https://arxiv.org/abs/2409.03283) and [project page](https://fireredteam.github.io/demos/firered_tts/)

## Roadmap

- [ ] 2024/09
  - [x] Release the pre-trained checkpoints and inference code.
  - [ ] Release testing set.

- [ ] 2024/10
  - [x] Release rich punctuation version.
  - [ ] Release finetuned checkpoints for controllable human-like speech generation.

## Usage

#### Clone and install

- Clone the repo

```shell
https://github.com/FireRedTeam/FireRedTTS.git
cd FireRedTTS
```

- Create conda env

```shell
# step1.create env
conda create --name redtts python=3.10

# stpe2.install torch （pytorch should match the cuda-version on your machine）
# CUDA 11.8
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia
# CUDA 12.1
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia

# step3.install fireredtts form source
pip install -e . 

# step4.install other requirements
pip install -r requirements.txt
```

#### Download models

Download the required model files from [**Model_Lists**](https://huggingface.co/fireredteam/FireRedTTS/tree/main) and place them in the folder `pretrained_models`

#### Basic Usage

```python
import os
import torchaudio
from fireredtts.fireredtts import FireRedTTS

tts = FireRedTTS(
    config_path="configs/config_24k.json",
    pretrained_path=<pretrained_models_dir>,
)

#same language
rec_wavs = tts.synthesize(
        prompt_wav="examples/prompt_1.wav",
        text="小红书，是中国大陆的网络购物和社交平台，成立于二零一三年六月。",
        lang="zh",
)

rec_wavs = rec_wavs.detach().cpu()
out_wav_path = os.path.join("./example.wav")
torchaudio.save(out_wav_path, rec_wavs, 24000)

```

## Tips

- Removing the long silence (>1s) in the middle of prompt_wav may bring better stability. If there are too many long silences in your prompt_wav and it causes stability problems, it is recommended to use our tool(`tools/process_prompts.py`) to remove the silence.

## Acknowledgements

- [**Tortoise-tts**](https://github.com/neonbjb/tortoise-tts) and [**XTTS-v2**](https://github.com/coqui-ai/TTS) offer invaluable insights for constructing an autoregressive-style system.
- [**Matcha-TTS**](https://github.com/shivammehta25/Matcha-TTS) and [**CosyVoice**](https://github.com/FunAudioLLM/CosyVoice) demonstrate the excellent ability of flow-matching in converting audio code to mel.
  
- [**BigVGAN-v2**](https://github.com/NVIDIA/BigVGAN), utilized for vocoding.
- We referred to [**whisper’s**](https://github.com/openai/whisper) text tokenizer solution.