StyleTTS 2 (code, datasets, models, paper)

.gitattributes

@@ -33,3 +33,13 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/ca.json filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/el.json filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/en-xl.json filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/fa.json filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/pt.json filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/ru.json filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/uk.json filter=lfs diff=lfs merge=lfs -text
+datasets/multilingual-phonemes-10k-alpha/zh.json filter=lfs diff=lfs merge=lfs -text
+models/ar/StyleTTS2-LibriTTS-arabic/synthesized_audio.wav filter=lfs diff=lfs merge=lfs -text
+StyleTTS[[:space:]]2.[[:space:]]Towards[[:space:]]Human-Level[[:space:]]Text-to-Speech[[:space:]]through[[:space:]]Style[[:space:]]Diffusion[[:space:]]and[[:space:]]Adversarial[[:space:]]Training[[:space:]]with[[:space:]]Large[[:space:]]Speech[[:space:]]Language[[:space:]]Models.pdf filter=lfs diff=lfs merge=lfs -text

StyleTTS 2. Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f528cab389ea8af17cfcc95ab5847975bb79b00dd46424d6b2d44a1e44017c55
+size 4082571

code/StyleTTS-VC.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d26c819eea0d52ba571d8f7dc69ccb8acb3db568ad3983ef0270929406975bf8
+size 215127843

code/StyleTTS.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bb6571ecb71baf369e4a04f9b92c7d56b82af8d71c0f2ec69786de0064ddb1a
+size 235881010

code/StyleTTS2.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdb682c2d4bb88dbd0556de55b3893d5cb96c72bdd4323841ec2296e999c9897
+size 280792923

code/stylish-tts.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f941ff08b1be23b6531b127bd41149478facfbf728f0e574434e23fd3c8e7bdc
+size 2739847

datasets/multilingual-phonemes-10k-alpha/.gitattributes

@@ -0,0 +1,56 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.lz4 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+# Audio files - uncompressed
+*.pcm filter=lfs diff=lfs merge=lfs -text
+*.sam filter=lfs diff=lfs merge=lfs -text
+*.raw filter=lfs diff=lfs merge=lfs -text
+# Audio files - compressed
+*.aac filter=lfs diff=lfs merge=lfs -text
+*.flac filter=lfs diff=lfs merge=lfs -text
+*.mp3 filter=lfs diff=lfs merge=lfs -text
+*.ogg filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+# Image files - uncompressed
+*.bmp filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.tiff filter=lfs diff=lfs merge=lfs -text
+# Image files - compressed
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text

datasets/multilingual-phonemes-10k-alpha/README.md

@@ -0,0 +1,102 @@
+---
+license: cc-by-sa-3.0
+license_name: cc-by-sa
+configs:
+- config_name: en
+  data_files: en.json
+  default: true
+- config_name: en-xl
+  data_files: en-xl.json
+- config_name: ca
+  data_files: ca.json
+- config_name: de
+  data_files: de.json
+- config_name: es
+  data_files: es.json
+- config_name: el
+  data_files: el.json
+- config_name: fa
+  data_files: fa.json
+- config_name: fi
+  data_files: fi.json
+- config_name: fr
+  data_files: fr.json
+- config_name: it
+  data_files: it.json
+- config_name: pl
+  data_files: pl.json
+- config_name: pt
+  data_files: pt.json
+- config_name: ru
+  data_files: ru.json
+- config_name: sv
+  data_files: sv.json
+- config_name: uk
+  data_files: uk.json
+- config_name: zh
+  data_files: zh.json
+language:
+- en
+- ca
+- de
+- es
+- el
+- fa
+- fi
+- fr
+- it
+- pl
+- pt
+- ru
+- sv
+- uk
+- zh
+tags:
+- synthetic
+---
+
+# Multilingual Phonemes 10K Alpha
+
+
+This dataset contains approximately 10,000 pairs of text and phonemes from each supported language. We support 15 languages in this dataset, so we have a total of ~150K pairs. This does not include the English-XL dataset, which includes another 100K unique rows.
+
+## Languages
+
+We support 15 languages, which means we have around 150,000 pairs of text and phonemes in multiple languages. This excludes the English-XL dataset, which has 100K unique (not included in any other split) additional phonemized pairs.
+
+* English (en)
+* English-XL (en-xl): ~100K phonemized pairs, English-only
+* Catalan (ca)
+* German (de)
+* Spanish (es)
+* Greek (el)
+* Persian (fa): Requested by [@Respair](https://huggingface.co/Respair)
+* Finnish (fi)
+* French (fr)
+* Italian (it)
+* Polish (pl)
+* Portuguese (pt)
+* Russian (ru)
+* Swedish (sw)
+* Ukrainian (uk)
+* Chinese (zh): Thank you to [@eugenepentland](https://huggingface.co/eugenepentland) for assistance in processing this text, as East-Asian languages are the most compute-intensive!
+
+## License + Credits
+
+Source data comes from [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and is licensed under CC-BY-SA 3.0. This dataset is licensed under CC-BY-SA 3.0.
+
+## Processing
+
+We utilized the following process to preprocess the dataset:
+
+1. Download data from [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) by language, selecting only the first Parquet file and naming it with the language code
+2. Process using [Data Preprocessing Scripts (StyleTTS 2 Community members only)](https://huggingface.co/styletts2-community/data-preprocessing-scripts) and modify the code to work with the language
+3. Script: Clean the text
+4. Script: Remove ultra-short phrases
+5. Script: Phonemize
+6. Script: Save JSON
+7. Upload dataset
+
+## Note
+
+East-Asian languages are experimental. We do not distinguish between Traditional and Simplified Chinese. The dataset consists mainly of Simplified Chinese in the `zh` split. We recommend converting characters to Simplified Chinese during inference, using a library such as `hanziconv` or `chinese-converter`.

datasets/multilingual-phonemes-10k-alpha/ca.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:993ac7b508efb60c009c97861941676699549a82828898d214e40b0d43c459ab
+size 10888235

datasets/multilingual-phonemes-10k-alpha/el.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8525fbdbfce41ad9bd29b390c49854edcdd942a35e70631bae52e3210cb49a5
+size 12178762

datasets/multilingual-phonemes-10k-alpha/en-xl.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56856fb3ef4e8eb9d6335c8199d51bb40eee4affbb8d344cc76e317fc72b8d45
+size 84063713

datasets/multilingual-phonemes-10k-alpha/fa.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f33e05a4a5e3ee0104e0599c4830d4ff505debc5d77eaab5b989e9d8d1eda23e
+size 18121401

datasets/multilingual-phonemes-10k-alpha/languages.txt

@@ -0,0 +1,15 @@
+English
+Catalan
+German
+Spanish
+Greek
+Persian
+Finnish
+French
+Italian
+Polish
+Portuguese
+Russian
+Swedish
+Ukrainian
+Chinese

datasets/multilingual-phonemes-10k-alpha/pt.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:458b4ec76cb331fbcdd1da6f83ccf0277fed2b0866a6c0ac25ccc47f1f4c9fac
+size 11076509

datasets/multilingual-phonemes-10k-alpha/ru.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29cea02458a6e82bcfe35a30b12ebf29031200c26b2c4c621b76d975e3cfd27e
+size 15753792

datasets/multilingual-phonemes-10k-alpha/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/datasets/styletts2-community/multilingual-phonemes-10k-alpha

datasets/multilingual-phonemes-10k-alpha/uk.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4965c576559b6542a8b4127b1b13e81913b736f4eb6852b494dfbf7010866287
+size 13127182

datasets/multilingual-phonemes-10k-alpha/zh.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6f6acdd4b4e05158361e2ddc1621172355a8b9b330f06a3c0873052d3ca44fc
+size 20594478

models/ar/StyleTTS2-LibriTTS-arabic/.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+synthesized_audio.wav filter=lfs diff=lfs merge=lfs -text

models/ar/StyleTTS2-LibriTTS-arabic/README.md

@@ -0,0 +1,142 @@
+---
+language: ar
+tags:
+- text-to-speech
+- tts
+- arabic
+- styletts2
+- pl-bert
+license: mit
+hardware: H100
+---
+
+# Model Card for Arabic StyleTTS2
+
+This is an Arabic text-to-speech model based on StyleTTS2 architecture, specifically adapted for Arabic language synthesis. The model achieves good quality Arabic speech synthesis, though not yet state-of-the-art, and further experimentation is needed to optimize performance for Arabic language specifically. All training objectives from the original StyleTTS2 were maintained, except for the WavLM objectives which were removed as they were primarily designed for English speech.
+
+## Example
+
+Here is an example output from the model:
+
+#### Sample 1
+<audio controls>
+  <source src="https://huggingface.co/fadi77/StyleTTS2-LibriTTS-arabic/resolve/main/synthesized_audio.wav" type="audio/wav">
+  Your browser does not support the audio element.
+</audio>
+
+## Efficiency and Performance
+
+A key strength of this model lies in its efficiency and performance characteristics:
+
+- **Compact Architecture**: Achieves impressive quality with <100M parameters
+- **Limited Training Data**: Trained on only 22 hours of single-speaker audio
+- **Transfer Learning**: Successfully fine-tuned from LibriTTS multi-speaker model to single-speaker Arabic
+- **Resource Efficient**: Good quality achieved despite limited computational resources
+
+Note: According to the StyleTTS2 authors, performance should improve further when training a single-speaker model from scratch rather than fine-tuning. This wasn't attempted in our case due to computational resource constraints, suggesting potential for even better results with more extensive training.
+
+
+## Model Details
+
+### Model Description
+
+This model is a modified version of StyleTTS2, specifically adapted for Arabic text-to-speech synthesis. It incorporates a custom-trained PL-BERT model for Arabic language understanding and removes the WavLM adversarial training component (which was primarily designed for English).
+
+- **Developed by:** Fadi (GitHub: Fadi987)
+- **Model type:** Text-to-Speech (StyleTTS2 architecture)
+- **Language(s):** Arabic
+- **Finetuned from model:** [yl4579/StyleTTS2-LibriTTS](https://huggingface.co/yl4579/StyleTTS2-LibriTTS)
+
+### Model Sources
+
+- **Repository:** [Fadi987/StyleTTS2](https://github.com/Fadi987/StyleTTS2)
+- **Paper:** [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://arxiv.org/abs/2306.07691)
+- **PL-BERT Model:** [fadi77/pl-bert](https://huggingface.co/fadi77/pl-bert)
+
+## Uses
+
+### Direct Use
+
+The model can be used for generating Arabic speech from text. To use the model:
+
+1. Clone the StyleTTS2 repository:
+```bash
+git clone https://github.com/Fadi987/StyleTTS2
+cd StyleTTS2
+```
+
+2. Install `espeak-ng` for phonemization backend:
+```bash
+# For macOS
+brew install espeak-ng
+
+# For Ubuntu/Debian
+sudo apt-get install espeak-ng
+
+# For Windows
+# Download and install espeak-ng from: https://github.com/espeak-ng/espeak-ng/releases
+```
+
+3. Install Python dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+4. Download the `model.pth` and `config.yml` files from this repository
+
+5. Run inference using:
+```bash
+python inference.py --config config.yml --model model.pth --text "الإِتْقَانُ يَحْتَاجُ إِلَى الْعَمَلِ وَالْمُثَابَرَة"
+```
+
+Make sure use properly diacritized Arabic text for best results
+
+### Out-of-Scope Use
+
+The model is specifically designed for Arabic text-to-speech synthesis and may not perform well for:
+- Other languages
+- Heavy dialect variations
+- Non-diacritized Arabic text
+
+## Training Details
+
+### Training Data
+
+- Training was performed on approximately 22 hours of Arabic audiobook data
+- Dataset: [fadi77/arabic-audiobook-dataset-24khz](https://huggingface.co/datasets/fadi77/arabic-audiobook-dataset-24khz)
+- The PL-BERT component was trained on fully diacritized Wikipedia Arabic text
+
+### Training Hyperparameters
+
+- **Number of epochs:** 20
+- **Diffusion training:** Started from epoch 5
+
+### Objectives
+- **Training objectives:** All original StyleTTS2 objectives maintained, except WavLM adversarial training
+- **Validation objectives:** Identical to original StyleTTS2 validation process
+
+### Compute Infrastructure
+- **Hardware Type:** NVIDIA H100 GPU
+
+### Notable Modifications from Original StyleTTS2 in Architecture and Objectives
+The architecture of the model follows that of StyleTTS2 with the following exceptions:
+  - Removed WavLM adversarial training component
+  - Custom PL-BERT trained for Arabic language
+
+
+## Citation
+
+**BibTeX:**
+```bibtex
+@article{styletts2,
+  title={StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models},
+  author={Liu, Yinghao Aaron and Chen, Tao and Ping, Wei and Wu, Xiaoliang and Wang, Dongchao and Duan, Yuxuan and Li, Xiaodi and Li, Chong and Liang, Xuchen and Liu, Qiong and others},
+  journal={arXiv preprint arXiv:2306.07691},
+  year={2023}
+}
+```
+
+## Model Card Contact
+
+GitHub: [@Fadi987](https://github.com/Fadi987)
+Hugging Face: [@fadi77](https://huggingface.co/fadi77)

models/ar/StyleTTS2-LibriTTS-arabic/config.yml

@@ -0,0 +1,114 @@
+log_dir: "/style_tts2/Models/FineTune.AudioBook"
+log_interval: 10
+device: "cuda"
+epochs: 25 # number of finetuning epoch
+batch_size: 6
+max_len: 300 # maximum number of frames
+pretrained_model_repo: "yl4579/StyleTTS2-LibriTTS"
+pretrained_model_filename: "Models/LibriTTS/epochs_2nd_00020.pth"
+second_stage_load_pretrained: true # set to true if the pre-trained model is for 2nd stage
+load_only_params: true # set to true if do not want to load epoch numbers and optimizer parameters
+
+F0_path: "/root/Utils/JDC/bst.t7"
+ASR_config: "/root/Utils/ASR/config.yml"
+ASR_path: "/root/Utils/ASR/epoch_00080.pth"
+PLBERT_repo_id: "fadi77/pl-bert"
+PLBERT_dirname: "models/mlm_only_with_diacritics"
+
+data_params:
+  train_data: "Data/youtube_train_list.txt"
+  val_data: "Data/youtube_val_list.txt"
+  root_path: "Youtube/wavs"
+  OOD_data: "Data/youtube_train_list.txt"
+  min_length: 50 # sample until texts with this size are obtained for OOD texts
+
+preprocess_params:
+  sr: 24000
+  spect_params:
+    n_fft: 2048
+    win_length: 1200
+    hop_length: 300
+
+model_params:
+  multispeaker: false
+
+  dim_in: 64 
+  hidden_dim: 512
+  max_conv_dim: 512
+  n_layer: 3
+  n_mels: 80
+
+  n_token: 178 # number of phoneme tokens
+  max_dur: 50 # maximum duration of a single phoneme
+  style_dim: 128 # style vector size
+  
+  dropout: 0.2
+
+  # config for decoder
+  decoder: 
+      type: 'hifigan' # either hifigan or istftnet
+      resblock_kernel_sizes: [3,7,11]
+      upsample_rates :  [10,5,3,2]
+      upsample_initial_channel: 512
+      resblock_dilation_sizes: [[1,3,5], [1,3,5], [1,3,5]]
+      upsample_kernel_sizes: [20,10,6,4]
+      
+  # speech language model config
+  slm:
+      model: 'microsoft/wavlm-base-plus'
+      sr: 16000 # sampling rate of SLM
+      hidden: 768 # hidden size of SLM
+      nlayers: 13 # number of layers of SLM
+      initial_channel: 64 # initial channels of SLM discriminator head
+  
+  # style diffusion model config
+  diffusion:
+    embedding_mask_proba: 0.1
+    # transformer config
+    transformer:
+      num_layers: 3
+      num_heads: 8
+      head_features: 64
+      multiplier: 2
+
+    # diffusion distribution config
+    dist:
+      sigma_data: 0.2 # placeholder for estimate_sigma_data set to false
+      estimate_sigma_data: true # estimate sigma_data from the current batch if set to true
+      mean: -3.0
+      std: 1.0
+  
+loss_params:
+    lambda_mel: 5. # mel reconstruction loss
+    lambda_gen: 1. # generator loss
+    lambda_slm: 1. # slm feature matching loss
+    
+    lambda_mono: 1. # monotonic alignment loss (TMA)
+    lambda_s2s: 1. # sequence-to-sequence loss (TMA)
+
+    lambda_F0: 1. # F0 reconstruction loss
+    lambda_norm: 1. # norm reconstruction loss
+    lambda_dur: 1. # duration loss
+    lambda_ce: 20. # duration predictor probability output CE loss
+    lambda_sty: 1. # style reconstruction loss
+    lambda_diff: 1. # score matching loss
+
+    # Note: Current values for training are only adequate for second stage finetuning.
+    diffusion_training_epoch: 5
+    joint_training_epoch: 100
+
+# Note: Current values for learnings rates are very low. This is only adequate for second stage finetuning.
+optimizer_params:
+  lr: 0.0001 # general learning rate
+  bert_lr: 0.00001 # learning rate for PLBERT
+  ft_lr: 0.0001 # learning rate for acoustic modules
+  
+slmadv_params:
+  min_len: 400 # minimum length of samples
+  max_len: 500 # maximum length of samples
+  batch_percentage: 0.5 # to prevent out of memory, only use half of the original batch size
+  skip_update: 10 # update the discriminator every this iterations of generator update
+  thresh: 5 # gradient norm above which the gradient is scaled
+  scale: 0.01 # gradient scaling factor for predictors from SLM discriminators
+  sig: 1.5 # sigma for differentiable duration modeling
+  

models/ar/StyleTTS2-LibriTTS-arabic/model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59d2323412f0c55c774b5675b45e5c12659c0d9e0f9e7012eecc6b7dd845b132
+size 2201968238

models/ar/StyleTTS2-LibriTTS-arabic/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/fadi77/StyleTTS2-LibriTTS-arabic

models/ar/StyleTTS2-LibriTTS-arabic/synthesized_audio.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f60e90523d734eff1b9f4b95cca49f22277df5cb4acd0bd347fde18f1c3b0469
+size 1795058

models/en/StyleTTS2-LibriTTS/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/en/StyleTTS2-LibriTTS/Models/config.yml

@@ -0,0 +1,21 @@
+{ASR_config: Utils/ASR/config.yml, ASR_path: Utils/ASR/epoch_00080.pth, F0_path: Utils/JDC/bst.t7,
+  PLBERT_dir: Utils/PLBERT/, batch_size: 8, data_params: {OOD_data: Data/OOD_texts.txt,
+    min_length: 50, root_path: '', train_data: Data/train_list.txt, val_data: Data/val_list.txt},
+  device: cuda, epochs_1st: 40, epochs_2nd: 25, first_stage_path: first_stage.pth,
+  load_only_params: false, log_dir: Models/LibriTTS, log_interval: 10, loss_params: {
+    TMA_epoch: 4, diff_epoch: 0, joint_epoch: 0, lambda_F0: 1.0, lambda_ce: 20.0,
+    lambda_diff: 1.0, lambda_dur: 1.0, lambda_gen: 1.0, lambda_mel: 5.0, lambda_mono: 1.0,
+    lambda_norm: 1.0, lambda_s2s: 1.0, lambda_slm: 1.0, lambda_sty: 1.0}, max_len: 300,
+  model_params: {decoder: {resblock_dilation_sizes: [[1, 3, 5], [1, 3, 5], [1, 3,
+          5]], resblock_kernel_sizes: [3, 7, 11], type: hifigan, upsample_initial_channel: 512,
+      upsample_kernel_sizes: [20, 10, 6, 4], upsample_rates: [10, 5, 3, 2]}, diffusion: {
+      dist: {estimate_sigma_data: true, mean: -3.0, sigma_data: 0.19926648961191362,
+        std: 1.0}, embedding_mask_proba: 0.1, transformer: {head_features: 64, multiplier: 2,
+        num_heads: 8, num_layers: 3}}, dim_in: 64, dropout: 0.2, hidden_dim: 512,
+    max_conv_dim: 512, max_dur: 50, multispeaker: true, n_layer: 3, n_mels: 80, n_token: 178,
+    slm: {hidden: 768, initial_channel: 64, model: microsoft/wavlm-base-plus, nlayers: 13,
+      sr: 16000}, style_dim: 128}, optimizer_params: {bert_lr: 1.0e-05, ft_lr: 1.0e-05,
+    lr: 0.0001}, preprocess_params: {spect_params: {hop_length: 300, n_fft: 2048,
+      win_length: 1200}, sr: 24000}, pretrained_model: Models/LibriTTS/epoch_2nd_00002.pth,
+  save_freq: 1, second_stage_load_pretrained: true, slmadv_params: {batch_percentage: 0.5,
+    iter: 20, max_len: 500, min_len: 400, scale: 0.01, sig: 1.5, thresh: 5}}

models/en/StyleTTS2-LibriTTS/Models/epochs_2nd_00020.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1164ffe19a17449d2c722234cecaf2836b35a698fb8ffd42562d2663657dca0a
+size 771390526

models/en/StyleTTS2-LibriTTS/README.md

@@ -0,0 +1,100 @@
+---
+language:
+- en
+- multilingual
+tags:
+- text-to-speech
+- speech-synthesis
+- pytorch
+- styletts2
+- speaches
+- neural-tts
+- voice-cloning
+pipeline_tag: text-to-speech
+library_name: pytorch
+license: mit
+datasets:
+- LibriTTS
+metrics:
+- naturalness
+- similarity
+widget:
+- text: "Hello, this is a sample of StyleTTS2 speech synthesis."
+  example_title: "English Sample"
+- text: "StyleTTS2 can synthesize high-quality speech with style control."
+  example_title: "Style Control Sample"
+---
+
+# StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training
+
+StyleTTS 2 is a text-to-speech model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level text-to-speech synthesis. This model builds upon the original StyleTTS with significant improvements in naturalness and similarity.
+
+## Model Description
+
+- **Model Type**: Neural Text-to-Speech (TTS)
+- **Language(s)**: English (primary), with support for 18+ languages
+- **License**: MIT
+- **Paper**: [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training](https://arxiv.org/abs/2306.07691)
+- **Sample Rate**: 24,000 Hz
+- **Architecture**: Style diffusion with adversarial training
+
+## Features
+
+- **High-Quality Synthesis**: Achieves human-level naturalness in speech synthesis
+- **Style Control**: Advanced style transfer and voice cloning capabilities
+- **Multi-Language Support**: Primary English model with support for 18+ additional languages
+- **Voice Cloning**: Can clone voices from reference audio samples
+- **Diffusion-Based**: Uses diffusion models for high-quality audio generation
+
+## Usage
+
+This model is designed for text-to-speech synthesis with the following capabilities:
+
+1. **Multi-Voice Synthesis**: Generate speech using preset voice styles
+2. **Voice Cloning**: Clone voices from reference audio samples
+3. **Style Control**: Fine-tune synthesis parameters for different styles
+4. **Multi-Language**: Support for various languages with English-accented pronunciation
+
+### Parameters
+
+- `alpha` (0.0-1.0): Style blending factor (default: 0.3)
+- `beta` (0.0-1.0): Style mixing factor (default: 0.7)
+- `diffusion_steps` (3-20): Number of diffusion steps for quality (default: 5)
+- `embedding_scale` (1.0-10.0): Embedding scale factor (default: 1.0)
+
+## Training Data
+
+- **Primary Dataset**: LibriTTS
+- **Languages**: English (primary) + 18 additional languages
+- **Training Approach**: Style diffusion with adversarial training using large speech language models
+
+## Performance
+
+StyleTTS 2 achieves human-level performance in:
+- **Naturalness**: Comparable to human speech in listening tests
+- **Similarity**: High fidelity voice cloning and style transfer
+- **Quality**: Superior audio quality compared to previous TTS models
+
+## Limitations
+
+- **Compute Requirements**: Requires significant computational resources for inference
+- **English-First**: Optimized for English, other languages may have accented pronunciation
+- **Context Dependency**: Performance varies with input text length and complexity
+
+## Citation
+
+```bibtex
+@article{li2024styletts2,
+  title={StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models},
+  author={Li, Yinghao Aaron and Han, Cong and Mesgarani, Nima},
+  journal={arXiv preprint arXiv:2306.07691},
+  year={2024}
+}
+```
+
+## Links
+
+- Paper: [https://arxiv.org/abs/2306.07691](https://arxiv.org/abs/2306.07691)
+- Samples: [https://styletts2.github.io/](https://styletts2.github.io/)
+- Code: [https://github.com/yl4579/StyleTTS2](https://github.com/yl4579/StyleTTS2)
+- License: MIT License

models/en/StyleTTS2-LibriTTS/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/jakezp/StyleTTS2-LibriTTS