Hugging Face's logo

proxectonos
/
Nos_StyleTTS2-Brais-GL

Text-to-Speech
Galician
tts
styletts2
galician
style-tts2
phoneme-level
proxecto-nos
Model card Files
xet
 Community
Nos_StyleTTS2-Brais-GL / README.md
cmagui's picture
cmagui
Initial commit: full repository with code, configs and weights
62c8ef8
preview code
|
Raw
History Blame Contribute Delete
14.7 kB
---
language:
- gl
license: apache-2.0
tags:
- text-to-speech
- tts
- styletts2
- galician
- style-tts2
- phoneme-level
- proxecto-nos
---
# Nos_StyleTTS2-Brais-GL: Galician Male TTS Model
## Overview
- [Model Description](#model-description)
- [System Architecture and Components](#system-architecture-and-components)
- [Intended Uses and Limitations](#intended-uses-and-limitations)
- [Prerequisites and Environment Setup](#prerequisites-and-environment-setup)
- [Auxiliary Components](#auxiliary-components)
- [Model Training](#model-training)
- [Inference](#inference)
- [Evaluation](#evaluation)
- [Results](#results)
- [Citation](#citation)
- [References](#references)
- [Additional Information](#additional-information)
- [Funding and Acknowledgements](#funding-and-acknowledgements)
---
## Model Description
**Nos_StyleTTS2-Brais-GL** is a high-quality text-to-speech (TTS) model for Galician based on the [StyleTTS2](https://github.com/yl4579/StyleTTS2) architecture. It was trained on the single-speaker male voice "Brais" ([`proxectonos/Nos_Brais-GL`](https://huggingface.co/proxectonos/Nos_Brais-GL)), developed as part of [Proxecto Nós](https://nos.gal/gl/proxecto-nos).
This model integrates a native Galician processing pipeline, utilizing [Cotovía](https://github.com/proxectonos/cotovia) for grapheme-to-phoneme (G2P) conversion and [PL-ModernBERT-gl](https://huggingface.co/proxectonos/PL-ModernBERT-gl) as its phoneme-level language encoder, providing rich contextualized phoneme embeddings.
### Key Features:
* **Architecture:** StyleTTS2 with iSTFTNet decoder, diffusion-based style modeling, and Speech Language Model (SLM) adversarial training via HuBERT.
* **Phoneme Encoder:** Powered by [`proxectonos/PL-ModernBERT-gl`](https://huggingface.co/proxectonos/PL-ModernBERT-gl) (ModernBERT trained on Galician phonemes).
* **G2P & Normalization:** Native Galician text processing using [Cotovía](https://github.com/proxectonos/cotovia).
* **Voice Profile:** Male Galician voice ("Brais") based on [`proxectonos/Nos_Brais-GL`](https://huggingface.co/proxectonos/Nos_Brais-GL).
---
## System Architecture and Components
The synthesis pipeline relies on four main modules:
1. **Phoneme Encoder (PL-BERT):** Uses `proxectonos/PL-ModernBERT-gl` trained with Cotovía's phoneme dictionary to supply contextual embeddings.
2. **Auxiliary ASR Aligner:** Text-audio alignment trained using a modified version of [AuxiliaryASR](https://github.com/yl4579/AuxiliaryASR) adapted for Cotovía phonemes.
3. **Pitch Extractor (JDC):** Pre-trained Joint Detection and Classification (JDC) network for F0 pitch extraction.
4. **SLM Discriminator:** Uses `BSC-LT/hubert-base-los-2k` to calculate Speech Language Model losses during second-stage adversarial training.
---
## Intended Uses and Limitations
### Intended Uses
* High-quality Galician speech synthesis for a wide range of applications (e.g., accessibility tools, virtual assistants, conversational agents, entertainment).
* Research on prosody modeling and speech synthesis in low-resource and regional language contexts.
### Limitations
* Single-speaker model: voice characteristics are fixed to the "Brais" male voice profile.
* Requires the **Cotovía** phonemizer for G2P conversion.
---
## Prerequisites and Environment Setup
1. **Clone this repository:**
```bash
git clone [https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL](https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL)
cd Nos_StyleTTS2-Brais-GL
```
2. **Set up the virtual environment and dependencies:**
```bash
conda create -n StyleTTS2 python=3.10 -y
conda activate StyleTTS2
pip install -r requirements.txt
```
3. **Install Cotovía:**
Cotovía is required for grapheme-to-phoneme (G2P) conversion. Follow the installation steps provided in `Utils/cotovia/README.md`.
## Dataset Preparation
Training data can be downloaded using the provided download script. You can specify the primary training dataset and the Out-Of-Domain (OOD) dataset in `Configs/download_data.yml`.
To download the dataset via the script:
```bash
python scripts/download_data.py --config Configs/download_data.yml --download_data
```
> **Note:** If you wish to switch the primary dataset, keep the existing audio folders, delete the `.txt` annotation files, and rerun the script without the `--download_data` flag:
```bash
python scripts/download_data.py --config Configs/download_data.yml
```
## Auxiliary Components
To ensure optimal synthesis, StyleTTS2 relies on three auxiliary models aligned with the same phonemizer (Cotovía):
### 1. Phoneme Encoder (PL-BERT)
StyleTTS2 requires a PL-BERT model trained on Galician phonemes generated by Cotovía. The training code and utilities for this component are maintained in the [`proxectonos/PL-ModernBERT-gl`](https://huggingface.co/proxectonos/PL-ModernBERT-gl) repository.
### 2. Text-Audio Aligner (Auxiliary ASR)
An Auxiliary ASR model aligns phoneme sequences with acoustic features. Follow the instructions in `Utils/ASR/AuxiliaryASR` (a modified version of AuxiliaryASR adapted for Cotovía). You can reuse the dataset downloaded in the previous steps for retraining.
### 3. Pitch Extractor (JDC)
The `Utils/JDC` directory contains a pre-trained Joint Detection and Classification (JDC) model for pitch (F0) extraction. Although pre-trained on the English LibriTTS corpus, pitch dynamics are language-independent, making retraining unnecessary.
## Model Training
### Configuration Parameters
#### Paths & Checkpoints
| Parameter | Value / Description |
| :--- | :--- |
| `first_stage_path` | `path_to_first_stage/model.pth` |
| `F0_path` | `Utils/JDC/bst.t7` |
| `ASR_config` | `Utils/ASR/config.yml` |
| `ASR_path` | `path_to_ASR/model.pth` |
| `PLBERT_dir` | Path to trained PL-BERT directory |
> **Important:** For `PLBERT_dir`, specify only the directory containing the `config.yml` configuration file, which indicates the path to the BERT model.
#### Main Training Parameters
| Parameter | Value | Parameter | Value |
| :--- | :--- | :--- | :--- |
| `epochs_1st` | 100 | `preprocess.sr` | 24,000 Hz |
| `epochs_2nd` | 100 | `preprocess.n_fft` | 2,048 |
| `batch_size` | 4 | `preprocess.win_length` | 1,200 |
| `grad_accumulation` | 1 | `preprocess.hop_length` | 300 |
| `max_len` | 500 | `model.n_mels` | 80 |
| `load_only_params` | `false` | `model.n_token` | 69 |
| `model.hidden_dim` | 512 | `model.style_dim` | 128 |
| `model.multispeaker` | `false` | `model.dropout` | 0.2 |
#### Decoder, SLM, and Diffusion Pipeline
| Component | Parameter | Value |
| :--- | :--- | :--- |
| **Decoder** | `type` | `istftnet` |
| | `resblock_kernel_sizes` | `[3, 7, 11]` |
| | `upsample_rates` | `[10, 6]` |
| | `upsample_initial_channel` | 512 |
| **SLM Discriminator** | `model` | `BSC-LT/hubert-base-los-2k` |
| | `sr` | 16,000 Hz |
| | `hidden` | 768 |
| | `nlayers` | 13 |
| **Diffusion** | `embedding_mask_proba` | 0.1 |
| | `transformer.num_layers` | 3 |
| | `transformer.num_heads` | 8 |
<!-- Extended Configuration Parameters (Full view from config.yml)
#### Main Training Parameters (Extended)
| Parameter | Value |
| :--- | :--- |
| `epochs_1st` | 100 |
| `epochs_2nd` | 100 |
| `batch_size` | 4 |
| `grad_accumulation` | 1 |
| `max_len` | 500 |
| `load_only_params` | false |
| `preprocess.sr` | 24000 |
| `preprocess.n_fft` | 2048 |
| `preprocess.win_length` | 1200 |
| `preprocess.hop_length` | 300 |
| `model.multispeaker` | false |
| `model.dim_in` | 64 |
| `model.hidden_dim` | 512 |
| `model.max_conv_dim` | 512 |
| `model.n_layer` | 3 |
| `model.n_mels` | 80 |
| `model.n_token` | 69 |
| `model.max_dur` | 50 |
| `model.style_dim` | 128 |
| `model.dropout` | 0.2 |
#### Decoder, SLM, and Diffusion (Extended)
| Parameter | Value |
| :--- | :--- |
| `decoder.type` | istftnet |
| `decoder.resblock_kernel_sizes` | [3,7,11] |
| `decoder.upsample_rates` | [10,6] |
| `decoder.upsample_initial_channel` | 512 |
| `decoder.resblock_dilation_sizes` | [[1,3,5],[1,3,5],[1,3,5]] |
| `decoder.upsample_kernel_sizes` | [20,12] |
| `decoder.gen_istft_n_fft` | 20 |
| `decoder.gen_istft_hop_size` | 5 |
| `slm.model` | BSC-LT/hubert-base-los-2k |
| `slm.sr` | 16000 |
| `slm.hidden` | 768 |
| `slm.nlayers` | 13 |
| `slm.initial_channel` | 64 |
| `diffusion.embedding_mask_proba` | 0.1 |
| `diffusion.transformer.num_layers` | 3 |
| `diffusion.transformer.num_heads` | 8 |
| `diffusion.transformer.head_features` | 64 |
| `diffusion.transformer.multiplier` | 2 |
-->
---
### Executing Training
Training is executed in two consecutive stages:
1. **First Stage Training (Acoustic Base):**
```bash
accelerate launch train_first.py --config_path ./Configs/config.yml
```
2. **Second Stage Training (Diffusion & SLM Refinement):**
```bash
python train_second.py --config_path ./Configs/config.yml
```
> **Note:** Multi-GPU training is **not supported** in the second stage.
Checkpoints are saved in `log_dir` using the naming conventions `epoch_1st_%05d.pth` and `epoch_2nd_%05d.pth`.
---
## Inference
This repository includes an inference script to synthesize audio using a pre-trained StyleTTS2-GL model. It accepts raw Galician text strings or input text files and supports configurable parameters to control the generation process.
```bash
python inference.py --config Configs/inference_config.yml --text "Texto en galego para sintetizar." --device 0
```
### Command-Line Arguments
| Argument | Description | Default / Example |
| :--- | :--- | :--- |
| `--config` | Path to the YAML inference configuration file | `Configs/inference_config.yml` |
| `--text` | Text string to synthesize | `"Texto a xerar"` |
| `--file` | Path to a file containing text samples to synthesize | `path/to/file.txt` |
| `--device` | Target execution device (`cpu` or GPU index) | `0` |
| `--output_dir` | Directory where output files will be saved | `./results` |
| `--output_file` | Filename for the generated audio file | `output.wav` |
| `--diffussion_steps` | Number of diffusion sampling steps | `30` |
| `--embedding_scale` | Scaling factor for the generated style vector | `1.0` |
| `--alpha` | Weight factor for the reference style embedding | `0.3` |
| `--beta` | Weight factor for the generated style embedding | `0.7` |
| `--t` | Interpolation factor with the previous style embedding | `0.0` |
| `--evaluate` | Enables automated MOS evaluation on generated audio | *Flag* |
### Recommended Inference Settings
The following parameters are recommended for optimal audio synthesis using the Brais StyleTTS2 model:
| Parameter | Argument | Recommended Value |
| :--- | :--- | :--- |
| $\alpha$ | `--alpha` | `0.6` |
| $\beta$ | `--beta` | `1.0` |
| $\theta$ | `--t` | `0.6` |
| Diffusion Steps | `--diffusion_steps` | `10` |
| Embedding Scale | `--embedding_scale` | `1.0` |
---
## Evaluation
An evaluation script utilizing the `speechmos` library is included to assess audio quality and verify model performance via MOS (Mean Opinion Score).
To run the evaluation on generated audio files:
```bash
python scripts/eval.py --audios audio1.wav audio2.wav --output output/mos_results.txt
```
Depending on the number of audio files provided, the script performs the following calculations:
* **MOS Calculation:** Performed for each individual audio file listed.
* **CMOS (Comparative MOS):** Automatically calculated when exactly two audio files are passed to compare performance against a reference model.
Results will be printed to the stdout console and saved in the designated `--output` file.
## Results
Results were obtained using the `speechmos` library to measure predicted **DNSMOS (OVRL)** quality and calculate **CMOS** gains.
### Text Samples
Models were evaluated using three different text lengths containing a mix of neutral, interrogative, and exclamatory sentences:
* **Short:** ~10 seconds of generated audio.
* **Medium:** ~30 seconds of generated audio.
* **Long:** >60 seconds of generated audio.
Synthesized audio samples were benchmarked against original corpus recordings and baseline VITS models from [Proxecto Nós](https://tts.nos.gal/).
### Original Corpus Reference Score
| Dataset / Voice Corpus | DNSMOS Metric (OVRL) |
| :--- | :---: |
| **Nos_Brais-GL (Original)** | **3.308** |
### Synthesis Quality Comparison (Brais Voice)
| Text Length | VITS Model (MOS) | StyleTTS2 Model (MOS) | Net Gain (CMOS) |
| :--- | :---: | :---: | :---: |
| **Short** (~10s) | 3.275 | **3.426** | **+0.151** |
| **Medium** (~30s) | 3.257 | **3.442** | **+0.185** |
| **Long** (>60s) | 3.241 | **3.397** | **+0.156** |
## Citation
If this model contributes to your research, please cite it as follows:
```bibtex
@misc{proxectonos/Nos_StyleTTS2-Brais-GL,
author = {{Proxecto Nós}},
title = {{Nos_StyleTTS2-Brais-GL: Galician Male TTS Model}},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{[https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL](https://huggingface.co/proxectonos/Nos_StyleTTS2-Brais-GL)}},
}
```
## References
### Models
- [AuxiliaryASR](https://github.com/yl4579/AuxiliaryASR)
- [PL-Bert](https://github.com/yl4579/PL-BERT)
- [StyleTTS2](https://github.com/yl4579/StyleTTS2)
### Datasets
- [Galician AI Language Resources (Zenodo)](https://zenodo.org/communities/proxecto-nos/records?q=&l=list&p=1&s=10&sort=newest)
- [TTS datasets - Proxecto Nós (Hugging Face)](https://huggingface.co/collections/proxectonos/tts-datasets)
- [ASR datasets - Proxecto Nós (Hugging Face)](https://huggingface.co/collections/proxectonos/asr-datasets)
## Additional Information
### Licensing
This model is licensed under the **Apache License 2.0**.
### Authors and Credits
* **Project Oversight:** [Proxecto Nós](https://nos.gal/gl/proxecto-nos)
* **Technical Development:** [Gradiant](https://www.gradiant.org/) (Centro Tecnolóxico de Telecomunicacións de Galicia)
## Funding and Acknowledgements
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA.
We would like to express our gratitude to the engineering and research teams at **Gradiant** for the technical development of this model, as well as to the **Aholab Signal Processing Laboratory (HiTZ)** and the **Language Technologies Laboratory (BSC)** for their technical support and collaboration.