Add StyleTTS2+Vocos with AIHUB dataset model

README.md

@@ -1,4 +1,12 @@
-# Vocos LibriTTS Model
+# StyleTTS2 + Vocos with LibriTTS Dataset
+
+```
+StyleTTS2
+├── README.md
+└── Vocos
+    └── LibriTTS
+        └── [checkpoint files]
+```
 
 This model was trained using the train-clean-100 and train-clean-360 subsets of the LibriTTS dataset.
 
@@ -10,15 +18,43 @@ This model was trained using the train-clean-100 and train-clean-360 subsets of
 
 The training and inference code can be found at: [StyleTTS2-Vocos](https://github.com/5Hyeons/StyleTTS2-Vocos)
 
-### Folder Structure
+## License
+
+This model is released under the MIT License. This is one of the most permissive open-source licenses, allowing for both commercial and non-commercial use, modification, and distribution.
+
+---
+
+# StyleTTS2 + Vocos with AIHUB Dataset
+
 ```
 StyleTTS2
 ├── README.md
 └── Vocos
-    └── LibriTTS
+    └── AIHUB
         └── [checkpoint files]
 ```
 
+This model was trained using multiple datasets from AIHUB:
+
+1. **Korean Data** (~1000 hours)
+   - Source: [감성 및 발화스타일 동시 고려 음성합성 데이터](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&dataSetSn=71349)
+
+2. **English & Japanese Data** (~1000 hours)
+   - Source: [다국어 통·번역 낭독체 데이터](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&dataSetSn=71524)
+
+3. **Chinese Data** (~500 hours)
+   - Source: [한-영 및 한-중 음성발화 데이터](https://www.aihub.or.kr/aihubdata/data/view.do?currMenu=&topMenu=&aihubDataSe=data&dataSetSn=71261)
+
+Total samples: ~1.4M
+
+## Model Information
+
+- **Model Architecture**: StyleTTS2 + Vocos
+- **Training Data**: AIHUB Multilingual Dataset
+- **License**: CC BY-NC 4.0
+
+The training and inference code can be found at: [StyleTTS2-Vocos](https://github.com/5Hyeons/StyleTTS2-Vocos)
+
 ## License
 
-This model is released under the MIT License. This is one of the most permissive open-source licenses, allowing for both commercial and non-commercial use, modification, and distribution.
+This model is released under the CC BY-NC 4.0 License. This license allows for non-commercial use, modification, and distribution, as long as appropriate credit is given.

Vocos/AIHUB_ML/Data/OOD_texts_en_jp_ko_zh.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5887482e060d92e1bcb291b17a54e027df41e8269e53df3a47eaa6c0384f60d
+size 41221638

Vocos/AIHUB_ML/Data/train_multi_lingual_en_jp_ko_zh_filelist.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c4245cc79786a292baa4dbd8b2302da554cd1e07c7027967b25c5931ff6b6b8
+size 324929081

Vocos/AIHUB_ML/Data/valid_multi_lingual_en_jp_ko_zh_500_filelist.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd5c9878539bdf9fc4c12606a389c54ad940e86eeb29691461ae77e818b6f868
+size 113157

Vocos/AIHUB_ML/config_aihub_multi_lingual_en_jp_ko_zh_vocos.yml

@@ -0,0 +1,113 @@
+log_dir: "Models/aihub_multi_lingual_en_jp_ko_zh_vocos"
+first_stage_path: "aihub_multi_lingual_en_jp_ko_zh_vocos_first_stage.pth"
+save_freq: 1
+log_interval: 10
+device: "cuda"
+epochs_1st: 10 # number of epochs for first stage training (pre-training)
+epochs_2nd: 8 # number of peochs for second stage training (joint training)
+batch_size: 8
+max_len: 300 # maximum number of frames
+pretrained_model: "Models/aihub_multi_lingual_en_jp_ko_zh_vocos/epoch_1st_00007_batch_08_step_107999.pth"
+second_stage_load_pretrained: false # set to true if the pre-trained model is for 2nd stage
+load_only_params: true # set to true if do not want to load epoch numbers and optimizer parameters
+
+F0_path: "Utils/JDC/bst.t7"
+ASR_config: "Utils/ASR/config_aihub_multi_lingual_en_jp_ko_zh.yml"
+ASR_path: "Utils/ASR/aihub_multi_lingual_en_jp_ko_zh_epoch_00011.pth"
+PLBERT_dir: 'Utils/PLBERT/'
+
+data_params:
+  # train_data: "Data/multi_lingual_train_filelist.txt.cleaned"
+  train_data: "Data/train_multi_lingual_en_jp_ko_zh_filelist.txt.cleaned.valid_word.after_asr"
+  val_data: "Data/valid_multi_lingual_en_jp_ko_zh_500_filelist.txt.cleaned.removed"
+  root_path: "wavs/"
+  OOD_data: "Data/OOD_texts_en_jp_ko_zh.txt"
+  min_length: 50 # sample until texts with this size are obtained for OOD texts
+
+preprocess_params:
+  sr: 24000
+  spect_params:
+    n_fft: 2048
+    win_length: 1200
+    hop_length: 300
+
+model_params:
+  multispeaker: true
+
+  dim_in: 64 
+  hidden_dim: 512
+  max_conv_dim: 512
+  n_layer: 3
+  n_mels: 80
+
+  n_token: 498 # number of phoneme tokens
+  max_dur: 50 # maximum duration of a single phoneme
+  style_dim: 128 # style vector size
+  
+  dropout: 0.2
+
+  # config for decoder
+  decoder: 
+      type: 'vocos' # either hifigan or istftnet or vocos
+      intermediate_dim: 1536
+      num_layers: 8
+      gen_istft_n_fft: 1200
+      gen_istft_hop_size: 300
+      
+  # speech language model config
+  slm:
+      model: 'microsoft/wavlm-base-plus'
+      sr: 16000 # sampling rate of SLM
+      hidden: 768 # hidden size of SLM
+      nlayers: 13 # number of layers of SLM
+      initial_channel: 64 # initial channels of SLM discriminator head
+  
+  # style diffusion model config
+  diffusion:
+    embedding_mask_proba: 0.1
+    # transformer config
+    transformer:
+      num_layers: 3
+      num_heads: 8
+      head_features: 64
+      multiplier: 2
+
+    # diffusion distribution config
+    dist:
+      sigma_data: 0.2 # placeholder for estimate_sigma_data set to false
+      estimate_sigma_data: true # estimate sigma_data from the current batch if set to true
+      mean: -3.0
+      std: 1.0
+  
+loss_params:
+    lambda_mel: 5. # mel reconstruction loss
+    lambda_gen: 1. # generator loss
+    lambda_slm: 1. # slm feature matching loss
+    
+    lambda_mono: 1. # monotonic alignment loss (1st stage, TMA)
+    lambda_s2s: 1. # sequence-to-sequence loss (1st stage, TMA)
+    TMA_epoch: 0 # TMA starting epoch (1st stage)
+
+    lambda_F0: 1. # F0 reconstruction loss (2nd stage)
+    lambda_norm: 1. # norm reconstruction loss (2nd stage)
+    lambda_dur: 1. # duration loss (2nd stage)
+    lambda_ce: 20. # duration predictor probability output CE loss (2nd stage)
+    lambda_sty: 1. # style reconstruction loss (2nd stage)
+    lambda_diff: 1. # score matching loss (2nd stage)
+    
+    diff_epoch: 1 # style diffusion starting epoch (2nd stage)
+    joint_epoch: 2 # joint training starting epoch (2nd stage)
+
+optimizer_params:
+  lr: 0.0001 # general learning rate
+  bert_lr: 0.00001 # learning rate for PLBERT
+  ft_lr: 0.00001 # learning rate for acoustic modules
+  
+slmadv_params:
+  min_len: 400 # minimum length of samples
+  max_len: 500 # maximum length of samples
+  batch_percentage: 0.5 # to prevent out of memory, only use half of the original batch size
+  iter: 20 # update the discriminator every this iterations of generator update
+  thresh: 5 # gradient norm above which the gradient is scaled
+  scale: 0.01 # gradient scaling factor for predictors from SLM discriminators
+  sig: 1.5 # sigma for differentiable duration modeling

Vocos/AIHUB_ML/epoch_1st_00014.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ae9fd3788d9ed6e92228a298b4921d6cb4abf23a6a6644e70e4fb60d4b78654
+size 2169017196

Vocos/AIHUB_ML/epoch_2nd_00006.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62bffd7fbe35142f23c9628b952e1e88918edc0ec3cd067454be4c70d3d1560a
+size 2484562307