initial commit

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.yaml filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,102 @@
+---
+language:
+- en
+tags:
+- speech
+license:
+- cc-by-sa-3.0
+- cc-by-4.0
+---
+
+# Wave-Trainer-Fit | Neural vocoder from SSL features
+
+[[Code of Wave-Trainer-Fit](https://github.com/line/WaveTrainerFit)][[audio samples](https://i17oonaka-h.github.io/projects/research_topics/wave_trainer_fit/)]
+
+>**Abstract:**<br>
+We propose WaveTrainerFit, a neural vocoder that performs high-quality waveform generation from data-driven features such as SSL features. WaveTrainerFit builds upon the WaveFit vocoder, which integrates diffusion model and generative adversarial network. Furthermore, the proposed method incorporates the following key improvements: 1. By introducing trainable priors, the inference process starts from noise close to the target speech instead of Gaussian noise. 2. Reference-aware gain adjustment is performed by imposing constraints on the trainable prior to matching the speech energy. These improvements are expected to reduce the complexity of waveform modeling from data-driven features, enabling high-quality waveform generation with fewer inference steps. Through experiments, we showed that WaveTrainerFit can generate highly natural waveforms with improved speaker similarity from data-driven features, while requiring fewer iterations than WaveFit. Moreover, we showed that the proposed method works robustly with respect to the depth at which SSL features are extracted.
+
+![concept.png](./assets/concept.png)
+
+This repository provides pre-trained models and their optimizers. 
+The models were pre-trained on [LibriTTS-R](https://www.openslr.org/141/) (train-clean-360).
+Also, these models operate to reconstruct 24kHz audio by taking SSL features from 16kHz audio as input.
+
+## Pre-trained model list
+> [!IMPORTANT]
+⚠️ **License Notice**: The model weights provided in this repository are licensed under different terms. The `xlsr*` and `whisper*` models are licensed differently from the `wavlm*` models. Please refer to the [License](#license) section for details.
+
+The list of available models is as follows:
+
+| Model-name | Conditional features | Layer num | #iters of model |
+|:------| :---------: | :---: | :---: | 
+| wavlm2_wavetrainerfit5 | [WavLM-large](https://huggingface.co/microsoft/wavlm-large) | 2 | 5 |
+| wavlm2_wavefit5 | [WavLM-large](https://huggingface.co/microsoft/wavlm-large) | 2 | 5 |
+| wavlm8_wavetrainerfit5 | [WavLM-large](https://huggingface.co/microsoft/wavlm-large) | 8| 5 |
+| wavlm8_wavefit5 | [WavLM-large](https://huggingface.co/microsoft/wavlm-large) | 8| 5 |
+| wavlm24_wavetrainerfit5 | [WavLM-large](https://huggingface.co/microsoft/wavlm-large) | 24| 5 |
+| wavlm24_wavefit5 | [WavLM-large](https://huggingface.co/microsoft/wavlm-large) | 24| 5 |
+| xlsr8_wavetrainerfit5 | [XLS-R-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) | 8| 5 |
+| xlsr8_wavefit5 | [XLS-R-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) | 8| 5 |
+| whisper8_wavetrainerfit5 | ※ [Whisper-medium](https://huggingface.co/openai/whisper-medium) | 8| 5 |
+| whisper8_wavefit5 | ※ [Whisper-medium](https://huggingface.co/openai/whisper-medium) | 8| 5 |
+
+※ As a result of our verification, we found that amplitude decay occurs in Whisper features after about 2.0 seconds. 
+During evaluation, our model processed inputs by dividing them into `2.0-second segments → extracting features with the Whisper encoder → recombining → resynthesizing`. 
+If you use this model in your application, the upstream feature extraction must also follow this flow.
+
+## Usage
+Please refer to [our GitHub repository](https://github.com/line/WaveTrainerFit) for instructions on how to use the models provided here.
+The following is reproduced from the GitHub repository:
+```python
+import torchaudio
+import torch
+from wavetrainerfit import load_pretrained_vocoder
+from transformers import WavLMModel, AutoFeatureExtractor
+
+ssl_preprocessor = AutoFeatureExtractor.from_pretrained('microsoft/wavlm-large')
+ssl_model: WavLMModel = WavLMModel.from_pretrained('microsoft/wavlm-large')
+
+layer = 2
+ssl_vocoder, cfg = load_pretrained_vocoder(f'wavlm{layer}_wavetrainerfit5')
+waveform, sr = torchaudio.load('./assets/ljspeech-samples/LJ037-0171.wav')
+if sr != 16000:
+    waveform = torchaudio.transforms.Resample(
+        orig_freq=sr, 
+        new_freq=16000
+    )(waveform)
+inputs = ssl_preprocessor(
+    waveform[0].numpy(), 
+    sampling_rate=16000, 
+    return_tensors="pt"
+)
+
+with torch.no_grad():
+    inputs = ssl_model(**inputs, output_hidden_states=True)
+    inputs = inputs.hidden_states[layer]  # (Batch, Timeframe, Featuredim)
+    generated_waveform = ssl_vocoder.pred(
+        conditional_feature=inputs, # (Batch, Timeframe, Featuredim)
+        T_=5 # num of iteration
+    )
+
+torchaudio.save(
+    './assets/ljspeech-samples/LJ037-0171-reconstructed.wav', 
+    generated_waveform[-1][:, 0].cpu(), 24000
+)
+```
+
+## License
+
+### Model Weights
+
+**`xlsr*` and `whisper*` models**: Licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
+- Based on XLS-R by facebook (Apache 2.0) - https://huggingface.co/facebook/wav2vec2-xls-r-300m
+- Based on Whisper by OpenAI (Apache 2.0) - https://huggingface.co/openai/whisper-medium
+
+**`wavlm*` models**: Licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/)
+- Based on WavLM by Microsoft Corporation ([CC BY-SA 3.0](https://github.com/microsoft/UniSpeech/blob/main/LICENSE)) - https://huggingface.co/microsoft/wavlm-large
+- ⚠️ Derivative works must also use CC BY-SA 3.0
+
+**Training data**: LibriTTS-R (CC BY 4.0) - https://www.openslr.org/141/
+
+When using these models, you must comply with both our license and the original upstream model licenses.
+

wavlm24_wavefit5/config.yaml

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d586588d05a9b642ac80abbb32f01c3191425dc448bcda40bbea6f517e0a0ff
+size 3677

wavlm24_wavefit5/discriminator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6aee01df939ed25c322d28762afc88674afe2d33b4a48f541446c54de758542b
+size 67725739

wavlm24_wavefit5/generator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e1b80a25af03d99141090b152db8810239d07573d86c1ef1633655a9e985b47
+size 70381754

wavlm24_wavefit5/opt_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a5c03b48398fc3105ccd5bb3ef77f963c37244caaf9912ab31d5523b39cbc9a
+size 135436790

wavlm24_wavefit5/opt_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d09a1159985eda4d9c2d1823970b3ee4d98c5981ca27c51809bcab5960e4268e
+size 140717450

wavlm24_wavefit5/sche_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5ccff4d21f36dc59881448522898302255373b0fd88643993f30c892d74c3c3
+size 1052

wavlm24_wavefit5/sche_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac5e39de608aa344d36de8c5b797e21a0469120a8430cd24e6dc3c0b9bd28c37
+size 1052

wavlm24_wavetrainerfit5/config.yaml

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a701b56b2f165cac78fbea42ba31c232c770fffafa80b306ac665a2d17df54e7
+size 3912

wavlm24_wavetrainerfit5/discriminator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d347f5a97a9fd57dcafdcef74ff39b26f49128c0718ecf4c7c75543a48f161d1
+size 67725739

wavlm24_wavetrainerfit5/generator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:468c8db191e4a20882255daf2d2fa2b1362f62d823968d546f9a2d5e9d962a84
+size 91213626

wavlm24_wavetrainerfit5/opt_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:707cb0713154b1476a0accccbd8252912ba8ee46d10c133382aa7439f5cad606
+size 135436790

wavlm24_wavetrainerfit5/opt_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fc004ee2bcc1a461804565790849da25f044df55bd7da857e4dfbc963446b57
+size 182384602

wavlm24_wavetrainerfit5/sche_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5ccff4d21f36dc59881448522898302255373b0fd88643993f30c892d74c3c3
+size 1052

wavlm24_wavetrainerfit5/sche_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac5e39de608aa344d36de8c5b797e21a0469120a8430cd24e6dc3c0b9bd28c37
+size 1052

wavlm2_wavefit5/config.yaml

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e85396f18f084867faa6fdb1354e4e3bc35a7c48add5e95025bb6d411c38dcd
+size 3673

wavlm2_wavefit5/discriminator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e6aa5baf6c74fc46750287a29a23fa4546ff2730e7864ab2055d79abe370818
+size 67725739

wavlm2_wavefit5/generator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52de0f0156bd59aeac61ae07f5ddb016f82064b22fe047c607ebbd6426113e3d
+size 70381754

wavlm2_wavefit5/opt_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37abef47b4ed918754329e0e846712de923f322ae955a972b4d5563fce612428
+size 135436790

wavlm2_wavefit5/opt_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05c3f532b8bf92fd7c6fadb246ac59bd5d8fc20d353b4abcd4ea9824fe1946d1
+size 140717450

wavlm2_wavefit5/sche_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5ccff4d21f36dc59881448522898302255373b0fd88643993f30c892d74c3c3
+size 1052

wavlm2_wavefit5/sche_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac5e39de608aa344d36de8c5b797e21a0469120a8430cd24e6dc3c0b9bd28c37
+size 1052

wavlm2_wavetrainerfit5/config.yaml

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acf522aa9c5767f128ff303572d342185a4202068ef7776b3a32e0c5b40a7181
+size 3908

wavlm2_wavetrainerfit5/discriminator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a36ffaf39439d1658a480c832992679fd6d4689089bf2a3b220919b179d4c8d
+size 67725739

wavlm2_wavetrainerfit5/generator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:877098b6e68cbd5f3aebab53eace69ac9da2e19c3427715e59c41db9307fac5e
+size 91213626

wavlm2_wavetrainerfit5/opt_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc2b82acce5d21d8cc2080f312e63aa4892db1cd5d3f82ea5c49f9d57ac6d99f
+size 135436790

wavlm2_wavetrainerfit5/opt_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abdd25c65f8e667081218e455ed112508afd71fae9ce43111104fab1f1fc7593
+size 182384602

wavlm2_wavetrainerfit5/sche_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fdb702a665e174fdb6a3143aa3339da61f0029022afa7569c641098aa643475
+size 1052

wavlm2_wavetrainerfit5/sche_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fac4aba57e808c8fcf7e561404c578a6c2a5134d9ce4b6ec40e0ceb5e3436e73
+size 1052

wavlm8_wavefit5/config.yaml

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82e4e1d6a28bc0de9adcaa0e8d77e2732482b78dd4c1783ddba5ea7f4c990fe1
+size 3673

wavlm8_wavefit5/discriminator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83eb614cd4c210680945a5770159f0993d19baec822214366f4ab7174e9384f4
+size 67725739

wavlm8_wavefit5/generator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c9d1c589c7d18956f81a66dbf00836e417960a6e542f424381d305ebbe1811f
+size 70381754

wavlm8_wavefit5/opt_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65b8c76e303322ab79e1f63ff225b5d6e24eeecc0f5bbb5ec3fa785a42b3096a
+size 135436790

wavlm8_wavefit5/opt_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b02919a98e60f811e09df28b96555443e79f502f9aa60f07761233b9e0ea9b59
+size 140717450

wavlm8_wavefit5/sche_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5ccff4d21f36dc59881448522898302255373b0fd88643993f30c892d74c3c3
+size 1052

wavlm8_wavefit5/sche_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac5e39de608aa344d36de8c5b797e21a0469120a8430cd24e6dc3c0b9bd28c37
+size 1052

wavlm8_wavetrainerfit5/config.yaml

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5eeca4b9b0ca8714fda83c131b9afa4c23645fedda1d2f1c39195a1b8aa7b8f0
+size 3908

wavlm8_wavetrainerfit5/discriminator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7246427cdaa1eb4195f1e6654dcb429c39ac18be704170efab70c04e443527a
+size 67725739

wavlm8_wavetrainerfit5/generator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19ab744075817741909515bb7158baf61aa2086ff3c031b74d5c888109cb8a04
+size 91213626

wavlm8_wavetrainerfit5/opt_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fee63b67aac8f4def14d7fd4c2ebe151eafb49093020d1f70bb245fc8b5f8ba
+size 135436790

wavlm8_wavetrainerfit5/opt_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bee39ba66414fb4566eb0cc9ce84d1fd71af048d2185e04b15a81a58088d3db
+size 182384602

wavlm8_wavetrainerfit5/sche_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5ccff4d21f36dc59881448522898302255373b0fd88643993f30c892d74c3c3
+size 1052

wavlm8_wavetrainerfit5/sche_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac5e39de608aa344d36de8c5b797e21a0469120a8430cd24e6dc3c0b9bd28c37
+size 1052

whisper8_wavefit5/config.yaml

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd1c5e6bf31cb139aca11dc672dd0f1dd24e84870059308b7a551f2639c0fd26
+size 3681

whisper8_wavefit5/discriminator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6e8fafbf5a9d606065279ba4c919018c652f4d4ace7564e793b1ab24a0c847a
+size 67725739

whisper8_wavefit5/generator.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:862254c7c6fee39023ee20950d7e843d154eadb258893680b3824d48a9262ff0
+size 70381754

whisper8_wavefit5/opt_d.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9734c216ba0bc72314d3910b764b4284dd9e51fcabe3d81885cc8b8ac35b6d90
+size 135436790

whisper8_wavefit5/opt_g.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f18eb7c1927042d49bfa2c2afb46784bb92d5bb21a0163ad5821ca02d26ab8c
+size 140717130