Text-to-Speech
Catalan
catalan
tts
audio
flashsr
cleanunet
zipvoice
ebellob commited on
Commit
e895347
·
verified ·
1 Parent(s): c62de57

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +88 -41
README.md CHANGED
@@ -16,42 +16,58 @@ tags:
16
  pipeline_tag: text-to-speech
17
  ---
18
 
19
- # ZipVoice-CA: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching (now in Catalan!)
20
 
21
- This repository contains the checkpoint of the fine-tuned ZipVoice-CA model, able to synthesise speech in catalan with great quality. Its metrics follow below.
22
- For more information regarding the training and evaluation of the model please refer to its repository on https://github.com/ErikUPV/ZipVoice-CA.
23
- Please also visit the repository of the original [ZipVoice](https://github.com/k2-fsa/ZipVoice) model.
24
 
25
- To get a feel of the model, click [here](https://erikupv.github.io/zipvoice-samples/) to listen to some samples.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
  ## Performance Metrics
28
 
29
- | Dataset | WER (%) ↓ | CER (%) ↓ | SIM-o ↑ | UTMOS ↑ |
30
- | :--- | :---: | :---: | :---: | :---: |
31
- | **Common Voice 17** | 10.96 | 3.00 | 0.68 | 3.17 |
32
- | **Festcat** | 7.31 | 2.56 | 0.65 | 3.46 |
33
- | **LaFrescat** | 7.61 | 2.56 | 0.67 | 3.54 |
 
 
34
 
35
  ---
36
 
37
  ## Installation
38
 
39
- ### 1. Clone the repository
 
40
  ```bash
41
- git clone https://github.com/erikupv/zipvoice-ca
42
- cd zipvoice-ca
43
  ```
44
 
45
- ### 2. Environment Setup
46
- We recommend using **Conda** to manage your dependencies and ensure a clean environment:
47
  ```bash
48
  conda create -n ZipVoice python=3.11
49
  conda activate ZipVoice
50
  pip install -r requirements_zipvoice.txt
51
  ```
52
 
53
- ### 3. Download the Catalan Model
54
- Use the Hugging Face CLI to download the fine-tuned checkpoint directly into your local models directory:
55
  ```bash
56
  # pip install huggingface_hub
57
  huggingface-cli download \
@@ -64,41 +80,72 @@ huggingface-cli download \
64
 
65
  ## Inference
66
 
67
- To generate speech from a test.tsv file using the Catalan model, use the command below.
68
 
69
  ```bash
70
  python3 -m zipvoice.bin.infer_zipvoice \
71
- --model-name zipvoice \
72
- --model-dir ./models \
73
- --checkpoint-name zipvoice_ca.pt \
74
- --tokenizer "espeak" \
75
- --lang "ca" \
76
- --test-list data_cat/raw/test.tsv \
77
- --res-dir results/ \
78
- --guidance-scale 1.0 \
79
- --num-step 16
80
  ```
81
 
82
- For single file inference
83
 
84
  ```bash
85
  python3 -m zipvoice.bin.infer_zipvoice \
86
- --model-name zipvoice \
87
- --prompt-wav prompt.wav \
88
- --prompt-text "I am the transcription of the prompt wav." \
89
- --text "I am the text to be synthesized." \
90
- --res-wav-path result.wav
91
- --model-dir ./models \
92
- --checkpoint-name zipvoice_ca.pt \
93
- --tokenizer "espeak" \
94
- --lang "ca" \
95
- --guidance-scale 1.0 \
96
- --num-step 16
97
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
98
  ---
99
 
100
  ## Acknowledgments
101
- This work is a fine-tuned version of the [ZipVoice](https://github.com/k2-fsa/ZipVoice) project.
 
 
 
102
 
103
  ## License
104
- This project is licensed under the **Apache-2.0 License**.
 
 
16
  pipeline_tag: text-to-speech
17
  ---
18
 
19
+ # ZipVoice-CA: Catalan Zero-Shot Text-to-Speech with ZipVoice
20
 
21
+ Catalan fine-tune of [ZipVoice](https://github.com/k2-fsa/ZipVoice), a fast zero-shot text-to-speech model based on flow matching.
 
 
22
 
23
+ <p align="center">
24
+ <a href="https://erikupv.github.io/zipvoice-samples/">
25
+ <img src="https://img.shields.io/badge/🔊%20Listen-Samples-green" alt="Listen to samples">
26
+ </a>
27
+ <a href="https://github.com/ErikUPV/ZipVoice-CA">
28
+ <img src="https://img.shields.io/badge/GitHub-ZipVoice--CA-orange?logo=github&logoColor=white" alt="GitHub repository">
29
+ </a>
30
+ <a href="https://github.com/k2-fsa/ZipVoice">
31
+ <img src="https://img.shields.io/badge/Base%20Model-ZipVoice-blue" alt="Base ZipVoice repository">
32
+ </a>
33
+ </p>
34
+ This repository contains the fine-tuned ZipVoice-CA checkpoint for Catalan speech synthesis. For the full training, preprocessing, inference, and evaluation recipe, see the [GitHub repository](https://github.com/ErikUPV/ZipVoice-CA).
35
+
36
+ The metrics below are intended as indicative benchmarks under this repository's evaluation setup, not as definitive state-of-the-art claims.
37
+
38
+ ---
39
 
40
  ## Performance Metrics
41
 
42
+ | Dataset | WER (%) ↓ | CER (%) ↓ | SIM-o ↑ | UTMOS ↑ |
43
+ | --------------- | --------: | --------: | ------: | ------: |
44
+ | Common Voice 17 | 10.96 | 3.00 | 0.68 | 3.17 |
45
+ | FestCat | 7.31 | 2.56 | 0.65 | 3.46 |
46
+ | LaFrescat | 7.61 | 2.56 | 0.67 | 3.54 |
47
+
48
+ Evaluation uses generated samples from the ZipVoice-CA recipe with `guidance_scale=1.0` and `num_step=25`.
49
 
50
  ---
51
 
52
  ## Installation
53
 
54
+ ### 1. Clone the recipe repository
55
+
56
  ```bash
57
+ git clone https://github.com/ErikUPV/ZipVoice-CA.git
58
+ cd ZipVoice-CA
59
  ```
60
 
61
+ ### 2. Create the environment
62
+
63
  ```bash
64
  conda create -n ZipVoice python=3.11
65
  conda activate ZipVoice
66
  pip install -r requirements_zipvoice.txt
67
  ```
68
 
69
+ ### 3. Download the Catalan checkpoint
70
+
71
  ```bash
72
  # pip install huggingface_hub
73
  huggingface-cli download \
 
80
 
81
  ## Inference
82
 
83
+ ### Batch inference from a `test.tsv` file
84
 
85
  ```bash
86
  python3 -m zipvoice.bin.infer_zipvoice \
87
+ --model-name zipvoice \
88
+ --model-dir ./models \
89
+ --checkpoint-name zipvoice_ca.pt \
90
+ --tokenizer espeak \
91
+ --lang ca \
92
+ --test-list data_cat/raw/test.tsv \
93
+ --res-dir results/ \
94
+ --guidance-scale 1.0 \
95
+ --num-step 25
96
  ```
97
 
98
+ ### Single-sample inference
99
 
100
  ```bash
101
  python3 -m zipvoice.bin.infer_zipvoice \
102
+ --model-name zipvoice \
103
+ --prompt-wav prompt.wav \
104
+ --prompt-text "I am the transcription of the prompt wav." \
105
+ --text "I am the text to be synthesized." \
106
+ --res-wav-path result.wav \
107
+ --model-dir ./models \
108
+ --checkpoint-name zipvoice_ca.pt \
109
+ --tokenizer espeak \
110
+ --lang ca \
111
+ --guidance-scale 1.0 \
112
+ --num-step 25
113
  ```
114
+
115
+ The prompt audio should contain the reference speaker voice, and `--prompt-text` should match the transcription of that prompt audio.
116
+
117
+ ---
118
+
119
+ ## Evaluation Setup
120
+
121
+ The reported metrics are computed on generated samples from three Catalan evaluation sources:
122
+
123
+ * held-out Common Voice 17 Catalan samples,
124
+ * FestCat prompts,
125
+ * LaFrescat prompts.
126
+
127
+ Metrics:
128
+
129
+ * **WER / CER**: ASR-based intelligibility metrics.
130
+ * **SIM-o**: speaker similarity between prompt and generated speech.
131
+ * **UTMOS**: automatic MOS-style naturalness estimate.
132
+
133
+ ---
134
+
135
+ ## Limitations
136
+
137
+ This model is intended for Catalan text-to-speech research and experimentation. Quality may vary depending on prompt quality, prompt duration, speaker characteristics, text normalization, and out-of-domain inputs.
138
+
139
+ As with any zero-shot TTS model, users should avoid generating speech that impersonates real people without consent.
140
+
141
  ---
142
 
143
  ## Acknowledgments
144
+
145
+ This model is a fine-tuned version of [ZipVoice](https://github.com/k2-fsa/ZipVoice), using the pretrained checkpoint released by the original authors.
146
+
147
+ ---
148
 
149
  ## License
150
+
151
+ This model is released under the Apache-2.0 License.