Instructions to use Joeyfully/Voice-Generation with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Joeyfully/Voice-Generation with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-to-speech", model="Joeyfully/Voice-Generation")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForTextToSpectrogram processor = AutoProcessor.from_pretrained("Joeyfully/Voice-Generation") model = AutoModelForTextToSpectrogram.from_pretrained("Joeyfully/Voice-Generation", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Upload 5 files
Browse files- README.md +102 -3
- dataset_README.md +52 -0
- eval.py +220 -0
- train.py +260 -0
- tts_common.py +283 -0
README.md
CHANGED
|
@@ -1,3 +1,102 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Radio TTS Quick Start
|
| 2 |
+
|
| 3 |
+
This workspace fine-tunes a text-to-speech model on the dataset in `data/` and writes all outputs to `output/`.
|
| 4 |
+
|
| 5 |
+
## What you get
|
| 6 |
+
|
| 7 |
+
- `train.py` for training and automatic checkpoint saving
|
| 8 |
+
- `eval.py` for synthesizing one WAV file from text
|
| 9 |
+
- `output/model/` for saved model checkpoints
|
| 10 |
+
- `output/rollout/` for per-epoch audio samples
|
| 11 |
+
- `loss.csv` and `loss.png` in the project root for loss tracking
|
| 12 |
+
|
| 13 |
+
## Requirements
|
| 14 |
+
|
| 15 |
+
You need Python with these packages available:
|
| 16 |
+
|
| 17 |
+
- `torch`
|
| 18 |
+
- `torchaudio`
|
| 19 |
+
- `transformers`
|
| 20 |
+
- `datasets`
|
| 21 |
+
- `pandas`
|
| 22 |
+
- `soundfile`
|
| 23 |
+
- `matplotlib`
|
| 24 |
+
|
| 25 |
+
The first run will also download the pretrained SpeechT5 model and vocoder from Hugging Face.
|
| 26 |
+
|
| 27 |
+
## Data layout
|
| 28 |
+
|
| 29 |
+
The dataset should stay in the provided structure:
|
| 30 |
+
|
| 31 |
+
```text
|
| 32 |
+
data/
|
| 33 |
+
train-00000-of-00010.parquet
|
| 34 |
+
train-00001-of-00010.parquet
|
| 35 |
+
...
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
The dataset metadata is described in `dataset_README.md`.
|
| 39 |
+
|
| 40 |
+
## Train
|
| 41 |
+
|
| 42 |
+
Start training with the default settings:
|
| 43 |
+
|
| 44 |
+
```bash
|
| 45 |
+
python train.py
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
Recommended first run on Windows:
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
python train.py --epochs 1 --batch-size 2
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
Useful options:
|
| 55 |
+
|
| 56 |
+
- `--max-examples` limits the dataset for a quick smoke test
|
| 57 |
+
- `--sample-text` adds custom rollout prompts and may be passed multiple times
|
| 58 |
+
- `--output-dir` changes where checkpoints and samples are written
|
| 59 |
+
|
| 60 |
+
## Resume training
|
| 61 |
+
|
| 62 |
+
Training resumes automatically from the latest checkpoint in `output/model/` when one exists.
|
| 63 |
+
|
| 64 |
+
To force a fresh run, use:
|
| 65 |
+
|
| 66 |
+
```bash
|
| 67 |
+
python train.py --no-resume
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
## Outputs after each epoch
|
| 71 |
+
|
| 72 |
+
Each epoch writes:
|
| 73 |
+
|
| 74 |
+
- `output/model/epoch_XXXX/` for the checkpoint
|
| 75 |
+
- `output/rollout/epoch_XXXX/` for generated audio samples
|
| 76 |
+
- `loss.csv` and `loss.png` updated in the project root
|
| 77 |
+
- `output/model/last_checkpoint.txt` pointing to the most recent checkpoint
|
| 78 |
+
|
| 79 |
+
## Evaluate
|
| 80 |
+
|
| 81 |
+
Generate a WAV file from text:
|
| 82 |
+
|
| 83 |
+
```bash
|
| 84 |
+
python eval.py --text "Hello, this is a test."
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
The output is written to `output/eval/`.
|
| 88 |
+
|
| 89 |
+
## Long text
|
| 90 |
+
|
| 91 |
+
`eval.py` supports long text by splitting the input into chunks, re-encoding each chunk through the tokenizer, generating audio chunk by chunk, and stitching the result into one WAV file.
|
| 92 |
+
|
| 93 |
+
If you want to tune long-form generation, try:
|
| 94 |
+
|
| 95 |
+
```bash
|
| 96 |
+
python eval.py --text "your long text" --chunk-token-limit 480 --chunk-gap-seconds 0.05 --maxlenratio 60
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
## Notes
|
| 100 |
+
|
| 101 |
+
- The scripts use the pretrained `microsoft/speecht5_tts` model and `microsoft/speecht5_hifigan` vocoder by default.
|
| 102 |
+
- The dataset README says attribution is required when using the generated voice in interfaces that generate audio in response to user action. Refer to the voice as Jenny, and where practical, Jenny (Dioco).
|
dataset_README.md
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
dataset_info:
|
| 3 |
+
features:
|
| 4 |
+
- name: file_name
|
| 5 |
+
dtype: string
|
| 6 |
+
- name: transcription
|
| 7 |
+
dtype: string
|
| 8 |
+
- name: transcription_normalised
|
| 9 |
+
dtype: string
|
| 10 |
+
- name: audio
|
| 11 |
+
dtype: audio
|
| 12 |
+
splits:
|
| 13 |
+
- name: train
|
| 14 |
+
num_bytes: 4983072167.73
|
| 15 |
+
num_examples: 20978
|
| 16 |
+
download_size: 3741291896
|
| 17 |
+
dataset_size: 4983072167.73
|
| 18 |
+
configs:
|
| 19 |
+
- config_name: default
|
| 20 |
+
data_files:
|
| 21 |
+
- split: train
|
| 22 |
+
path: data/train-*
|
| 23 |
+
---
|
| 24 |
+
|
| 25 |
+
# Jenny TTS Dataset
|
| 26 |
+
|
| 27 |
+
A high-quality, varied ~30hr voice dataset suitable for training a TTS model.
|
| 28 |
+
|
| 29 |
+
Voice is recorded by Jenny. She's Irish.
|
| 30 |
+
|
| 31 |
+
Material read include:
|
| 32 |
+
- Newspaper headlines
|
| 33 |
+
- Transcripts of various Youtube videos
|
| 34 |
+
- About 2/3 of the book '1984'
|
| 35 |
+
- Some of the book 'Little Women'
|
| 36 |
+
- Wikipedia articles, different topics (philosophy, history, science)
|
| 37 |
+
- Recipes
|
| 38 |
+
- Reddit comments
|
| 39 |
+
- Song lyrics, including rap lyrics
|
| 40 |
+
- Transcripts to the show 'Friends'
|
| 41 |
+
|
| 42 |
+
Audio files are 48khz, 16-bit PCM files, 2 Channels (a single microphone was used.. hmm).
|
| 43 |
+
|
| 44 |
+
Some light preprocessing was done when the text was taken from the raw sources. A breakdown of where different material starts and ends can be reconstructed. Further information to follow.
|
| 45 |
+
|
| 46 |
+
# Important
|
| 47 |
+
|
| 48 |
+
The audiofiles are raw from the microphone, not trimmed. In some cases there are a few seconds of silence, sometimes a light 'knock' is audible at the beginning of the clip, where Jenny was hitting the start key. These issues will need to be addressed before training a TTS model. I'm a bit short on time these days, help welcome.
|
| 49 |
+
|
| 50 |
+
License - Attribution is required in software/websites/projects/interfaces (including voice interfaces) that generate audio in response to user action using this dataset. Atribution means: the voice must be referred to as "Jenny", and where at all practical, "Jenny (Dioco)". Attribution is not required when distributing the generated clips (although welcome). Commercial use is permitted. Don't do unfair things like claim the dataset is your own. No further restrictions apply.
|
| 51 |
+
|
| 52 |
+
Jenny is available to produce further recordings for your own use. Mail dioco@dioco.io
|
eval.py
ADDED
|
@@ -0,0 +1,220 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
import json
|
| 5 |
+
from datetime import datetime
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
from typing import Iterable
|
| 8 |
+
|
| 9 |
+
import numpy as np
|
| 10 |
+
import soundfile as sf
|
| 11 |
+
import torch
|
| 12 |
+
from transformers import SpeechT5ForTextToSpeech
|
| 13 |
+
|
| 14 |
+
from tts_common import (
|
| 15 |
+
DEFAULT_BASE_MODEL,
|
| 16 |
+
TARGET_SAMPLE_RATE,
|
| 17 |
+
ensure_directory,
|
| 18 |
+
find_latest_checkpoint,
|
| 19 |
+
load_processor,
|
| 20 |
+
load_vocoder,
|
| 21 |
+
normalize_text,
|
| 22 |
+
sanitize_filename,
|
| 23 |
+
)
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def parse_args() -> argparse.Namespace:
|
| 27 |
+
parser = argparse.ArgumentParser(description="Generate speech from text using the latest trained checkpoint.")
|
| 28 |
+
parser.add_argument("--text", type=str, default="While this kind of generalization has always been thought of as a key strength of robotic foundation models, actual models demonstrated to date have not shown the kind of broad compositional generalization that we’ve seen, for example, from LLMs.", help="Text to synthesize. If omitted, you will be prompted.")
|
| 29 |
+
parser.add_argument("--checkpoint", type=Path, default=Path("output/model"), help="Checkpoint directory or model root.")
|
| 30 |
+
parser.add_argument("--output-dir", type=Path, default=Path("output/eval"), help="Directory for synthesized audio.")
|
| 31 |
+
parser.add_argument("--base-model", type=str, default=DEFAULT_BASE_MODEL, help="Fallback base checkpoint if no trained checkpoint is found.")
|
| 32 |
+
parser.add_argument("--maxlenratio", type=float, default=60.0, help="Maximum generation length ratio. Larger values allow longer output.")
|
| 33 |
+
parser.add_argument("--threshold", type=float, default=0.5, help="Stop threshold used by SpeechT5 generation.")
|
| 34 |
+
parser.add_argument("--chunk-token-limit", type=int, default=480, help="Maximum tokens per generated chunk. Keep below the model limit for long text.")
|
| 35 |
+
parser.add_argument("--chunk-gap-seconds", type=float, default=0.05, help="Silence inserted between generated chunks when stitching long text.")
|
| 36 |
+
parser.add_argument("--device", type=str, default=None, help="Optional device override, for example cpu or cuda.")
|
| 37 |
+
return parser.parse_args()
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def resolve_checkpoint(path: Path, base_model: str) -> Path | str:
|
| 41 |
+
if path.is_dir():
|
| 42 |
+
try:
|
| 43 |
+
return find_latest_checkpoint(path)
|
| 44 |
+
except FileNotFoundError:
|
| 45 |
+
return path if any(path.iterdir()) else base_model
|
| 46 |
+
return path
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def compute_safe_chunk_token_limit(
|
| 50 |
+
model,
|
| 51 |
+
tokenizer,
|
| 52 |
+
requested_limit: int,
|
| 53 |
+
maxlenratio: float,
|
| 54 |
+
) -> int:
|
| 55 |
+
max_text_positions = int(getattr(model.config, "max_text_positions", getattr(tokenizer, "model_max_length", requested_limit)))
|
| 56 |
+
max_speech_positions = int(getattr(model.config, "max_speech_positions", 0))
|
| 57 |
+
reduction_factor = int(getattr(model.config, "reduction_factor", 1))
|
| 58 |
+
|
| 59 |
+
safe_limit = requested_limit
|
| 60 |
+
if max_speech_positions > 0 and maxlenratio > 0:
|
| 61 |
+
safe_limit = min(safe_limit, max(1, int(((max_speech_positions - 1) * reduction_factor) / maxlenratio)))
|
| 62 |
+
|
| 63 |
+
safe_limit = min(safe_limit, max_text_positions)
|
| 64 |
+
tokenizer_limit = int(getattr(tokenizer, "model_max_length", safe_limit))
|
| 65 |
+
if tokenizer_limit > 0:
|
| 66 |
+
safe_limit = min(safe_limit, tokenizer_limit)
|
| 67 |
+
|
| 68 |
+
return max(1, safe_limit)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def split_text_into_token_chunks(text: str, tokenizer, max_tokens: int) -> list[str]:
|
| 72 |
+
normalized = normalize_text(text)
|
| 73 |
+
if not normalized:
|
| 74 |
+
return []
|
| 75 |
+
|
| 76 |
+
token_ids = tokenizer(normalized, add_special_tokens=False).input_ids
|
| 77 |
+
if not token_ids:
|
| 78 |
+
return []
|
| 79 |
+
|
| 80 |
+
chunks: list[str] = []
|
| 81 |
+
for start in range(0, len(token_ids), max_tokens):
|
| 82 |
+
chunk_ids = token_ids[start : start + max_tokens]
|
| 83 |
+
if chunk_ids:
|
| 84 |
+
chunk_text = tokenizer.decode(chunk_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False)
|
| 85 |
+
chunk_text = normalize_text(chunk_text)
|
| 86 |
+
if chunk_text:
|
| 87 |
+
chunks.append(chunk_text)
|
| 88 |
+
return chunks
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def generate_waveform_chunks(
|
| 92 |
+
model,
|
| 93 |
+
tokenizer,
|
| 94 |
+
vocoder,
|
| 95 |
+
text: str,
|
| 96 |
+
device: torch.device,
|
| 97 |
+
threshold: float,
|
| 98 |
+
maxlenratio: float,
|
| 99 |
+
chunk_token_limit: int,
|
| 100 |
+
) -> list[np.ndarray]:
|
| 101 |
+
chunk_texts = split_text_into_token_chunks(text, tokenizer, chunk_token_limit)
|
| 102 |
+
if not chunk_texts:
|
| 103 |
+
return []
|
| 104 |
+
|
| 105 |
+
speaker_embeddings = torch.zeros((1, model.config.speaker_embedding_dim), device=device)
|
| 106 |
+
waveforms: list[np.ndarray] = []
|
| 107 |
+
|
| 108 |
+
with torch.inference_mode():
|
| 109 |
+
for chunk_text in chunk_texts:
|
| 110 |
+
chunk_tokens = tokenizer(chunk_text, return_tensors="pt")
|
| 111 |
+
chunk_tokens = {name: tensor.to(device) for name, tensor in chunk_tokens.items()}
|
| 112 |
+
waveform = model.generate_speech(
|
| 113 |
+
input_ids=chunk_tokens["input_ids"],
|
| 114 |
+
attention_mask=chunk_tokens.get("attention_mask"),
|
| 115 |
+
speaker_embeddings=speaker_embeddings,
|
| 116 |
+
threshold=threshold,
|
| 117 |
+
maxlenratio=maxlenratio,
|
| 118 |
+
vocoder=vocoder,
|
| 119 |
+
)
|
| 120 |
+
|
| 121 |
+
if isinstance(waveform, tuple):
|
| 122 |
+
waveform = waveform[0]
|
| 123 |
+
if waveform.ndim > 1:
|
| 124 |
+
waveform = waveform.squeeze(0)
|
| 125 |
+
|
| 126 |
+
waveforms.append(waveform.detach().cpu().numpy().astype(np.float32))
|
| 127 |
+
|
| 128 |
+
return waveforms
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def stitch_waveforms(waveforms: Iterable[np.ndarray], gap_seconds: float) -> np.ndarray:
|
| 132 |
+
chunks = [np.asarray(chunk, dtype=np.float32).reshape(-1) for chunk in waveforms if chunk is not None and len(chunk) > 0]
|
| 133 |
+
if not chunks:
|
| 134 |
+
return np.zeros(0, dtype=np.float32)
|
| 135 |
+
|
| 136 |
+
gap = np.zeros(int(TARGET_SAMPLE_RATE * max(gap_seconds, 0.0)), dtype=np.float32)
|
| 137 |
+
stitched: list[np.ndarray] = []
|
| 138 |
+
for index, chunk in enumerate(chunks):
|
| 139 |
+
if index > 0 and gap.size > 0:
|
| 140 |
+
stitched.append(gap)
|
| 141 |
+
stitched.append(chunk)
|
| 142 |
+
return np.concatenate(stitched) if stitched else np.zeros(0, dtype=np.float32)
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def main() -> None:
|
| 146 |
+
args = parse_args()
|
| 147 |
+
|
| 148 |
+
if args.text is None:
|
| 149 |
+
text = input("Enter text to synthesize: ").strip()
|
| 150 |
+
else:
|
| 151 |
+
text = args.text.strip()
|
| 152 |
+
|
| 153 |
+
if not text:
|
| 154 |
+
raise ValueError("Text input cannot be empty.")
|
| 155 |
+
|
| 156 |
+
device_name = args.device or ("cuda" if torch.cuda.is_available() else "cpu")
|
| 157 |
+
device = torch.device(device_name)
|
| 158 |
+
|
| 159 |
+
checkpoint = resolve_checkpoint(args.checkpoint, args.base_model)
|
| 160 |
+
processor = load_processor(str(checkpoint))
|
| 161 |
+
model = SpeechT5ForTextToSpeech.from_pretrained(str(checkpoint))
|
| 162 |
+
vocoder = load_vocoder()
|
| 163 |
+
|
| 164 |
+
model.to(device)
|
| 165 |
+
vocoder.to(device)
|
| 166 |
+
model.eval()
|
| 167 |
+
vocoder.eval()
|
| 168 |
+
|
| 169 |
+
waveform_chunks = generate_waveform_chunks(
|
| 170 |
+
model=model,
|
| 171 |
+
tokenizer=processor.tokenizer,
|
| 172 |
+
vocoder=vocoder,
|
| 173 |
+
text=text,
|
| 174 |
+
device=device,
|
| 175 |
+
threshold=args.threshold,
|
| 176 |
+
maxlenratio=args.maxlenratio,
|
| 177 |
+
chunk_token_limit=compute_safe_chunk_token_limit(
|
| 178 |
+
model=model,
|
| 179 |
+
tokenizer=processor.tokenizer,
|
| 180 |
+
requested_limit=args.chunk_token_limit,
|
| 181 |
+
maxlenratio=args.maxlenratio,
|
| 182 |
+
),
|
| 183 |
+
)
|
| 184 |
+
waveform = stitch_waveforms(waveform_chunks, args.chunk_gap_seconds)
|
| 185 |
+
|
| 186 |
+
output_dir = ensure_directory(args.output_dir)
|
| 187 |
+
stamp = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
|
| 188 |
+
stem = f"{stamp}_{sanitize_filename(text)}"
|
| 189 |
+
wav_path = output_dir / f"{stem}.wav"
|
| 190 |
+
txt_path = output_dir / f"{stem}.txt"
|
| 191 |
+
meta_path = output_dir / f"{stem}.json"
|
| 192 |
+
|
| 193 |
+
sf.write(wav_path, waveform.astype(np.float32), TARGET_SAMPLE_RATE)
|
| 194 |
+
txt_path.write_text(text, encoding="utf-8")
|
| 195 |
+
meta_path.write_text(
|
| 196 |
+
json.dumps(
|
| 197 |
+
{
|
| 198 |
+
"chunk_count": len(waveform_chunks),
|
| 199 |
+
"chunk_token_limit": compute_safe_chunk_token_limit(
|
| 200 |
+
model=model,
|
| 201 |
+
tokenizer=processor.tokenizer,
|
| 202 |
+
requested_limit=args.chunk_token_limit,
|
| 203 |
+
maxlenratio=args.maxlenratio,
|
| 204 |
+
),
|
| 205 |
+
"chunk_gap_seconds": args.chunk_gap_seconds,
|
| 206 |
+
"checkpoint": str(checkpoint),
|
| 207 |
+
},
|
| 208 |
+
indent=2,
|
| 209 |
+
ensure_ascii=False,
|
| 210 |
+
),
|
| 211 |
+
encoding="utf-8",
|
| 212 |
+
)
|
| 213 |
+
|
| 214 |
+
print(f"Saved audio to {wav_path}")
|
| 215 |
+
print(f"Text: {text}")
|
| 216 |
+
print(f"Checkpoint: {checkpoint}")
|
| 217 |
+
|
| 218 |
+
|
| 219 |
+
if __name__ == "__main__":
|
| 220 |
+
main()
|
train.py
ADDED
|
@@ -0,0 +1,260 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
import json
|
| 5 |
+
import math
|
| 6 |
+
from datetime import datetime
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
import random
|
| 9 |
+
|
| 10 |
+
import pandas as pd
|
| 11 |
+
import torch
|
| 12 |
+
from torch.utils.data import DataLoader
|
| 13 |
+
from transformers import SpeechT5ForTextToSpeech, get_linear_schedule_with_warmup
|
| 14 |
+
|
| 15 |
+
from tts_common import (
|
| 16 |
+
DEFAULT_BASE_MODEL,
|
| 17 |
+
ensure_directory,
|
| 18 |
+
build_default_sample_texts,
|
| 19 |
+
find_latest_checkpoint,
|
| 20 |
+
load_processor,
|
| 21 |
+
load_vocoder,
|
| 22 |
+
save_checkpoint_marker,
|
| 23 |
+
save_rollout_samples,
|
| 24 |
+
SpeechDataset,
|
| 25 |
+
TTSCollator,
|
| 26 |
+
write_loss_artifacts,
|
| 27 |
+
)
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def parse_args() -> argparse.Namespace:
|
| 31 |
+
parser = argparse.ArgumentParser(description="Train a text-to-speech model on the Radio dataset.")
|
| 32 |
+
parser.add_argument("--data-dir", type=Path, default=Path("data"), help="Directory containing train-*.parquet shards.")
|
| 33 |
+
parser.add_argument("--output-dir", type=Path, default=Path("output"), help="Directory for checkpoints and rollouts.")
|
| 34 |
+
parser.add_argument("--base-model", type=str, default=DEFAULT_BASE_MODEL, help="Base SpeechT5 checkpoint to fine-tune.")
|
| 35 |
+
parser.add_argument("--epochs", type=int, default=10, help="Number of training epochs.")
|
| 36 |
+
parser.add_argument("--batch-size", type=int, default=4, help="Training batch size.")
|
| 37 |
+
parser.add_argument("--learning-rate", type=float, default=1e-6, help="AdamW learning rate.")
|
| 38 |
+
parser.add_argument("--weight-decay", type=float, default=0.01, help="AdamW weight decay.")
|
| 39 |
+
parser.add_argument("--gradient-accumulation-steps", type=int, default=1, help="Gradient accumulation steps.")
|
| 40 |
+
parser.add_argument("--max-text-length", type=int, default=512, help="Maximum number of text tokens per sample.")
|
| 41 |
+
parser.add_argument("--max-examples", type=int, default=None, help="Optional cap on the number of dataset examples.")
|
| 42 |
+
parser.add_argument("--seed", type=int, default=random.randint(0, 2**32 - 1), help="Random seed.")
|
| 43 |
+
parser.add_argument("--num-workers", type=int, default=0, help="DataLoader workers. Keep 0 for maximum compatibility.")
|
| 44 |
+
parser.add_argument("--rollout-maxlenratio", type=float, default=20.0, help="Maximum output length ratio used for rollout samples.")
|
| 45 |
+
parser.add_argument("--rollout-threshold", type=float, default=0.5, help="Stop threshold used for rollout samples.")
|
| 46 |
+
parser.add_argument("--sample-text", action="append", default=None, help="Optional rollout text prompt. May be passed multiple times.")
|
| 47 |
+
parser.add_argument("--no-resume", dest="resume", action="store_false", help="Start a fresh run instead of resuming from the latest checkpoint.")
|
| 48 |
+
parser.set_defaults(resume=True)
|
| 49 |
+
return parser.parse_args()
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def set_seed(seed: int) -> None:
|
| 53 |
+
torch.manual_seed(seed)
|
| 54 |
+
torch.cuda.manual_seed_all(seed)
|
| 55 |
+
import random
|
| 56 |
+
import numpy as np
|
| 57 |
+
|
| 58 |
+
random.seed(seed)
|
| 59 |
+
np.random.seed(seed)
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def load_loss_history(loss_csv_path: Path) -> list[dict]:
|
| 63 |
+
if not loss_csv_path.exists():
|
| 64 |
+
return []
|
| 65 |
+
|
| 66 |
+
frame = pd.read_csv(loss_csv_path)
|
| 67 |
+
if frame.empty:
|
| 68 |
+
return []
|
| 69 |
+
|
| 70 |
+
records: list[dict] = []
|
| 71 |
+
for row in frame.itertuples(index=False):
|
| 72 |
+
records.append({"epoch": int(row.epoch), "loss": float(row.loss)})
|
| 73 |
+
return records
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def main() -> None:
|
| 77 |
+
args = parse_args()
|
| 78 |
+
set_seed(args.seed)
|
| 79 |
+
|
| 80 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 81 |
+
if device.type == "cuda":
|
| 82 |
+
torch.backends.cuda.matmul.allow_tf32 = True
|
| 83 |
+
torch.set_float32_matmul_precision("high")
|
| 84 |
+
|
| 85 |
+
root_dir = args.output_dir
|
| 86 |
+
model_dir = ensure_directory(root_dir / "model")
|
| 87 |
+
rollout_root = ensure_directory(root_dir / "rollout")
|
| 88 |
+
ensure_directory(root_dir)
|
| 89 |
+
|
| 90 |
+
loss_csv_path = root_dir / "loss.csv"
|
| 91 |
+
loss_plot_path = root_dir / "loss.png"
|
| 92 |
+
records: list[dict] = []
|
| 93 |
+
|
| 94 |
+
resume_checkpoint = None
|
| 95 |
+
resume_state = None
|
| 96 |
+
start_epoch = 1
|
| 97 |
+
|
| 98 |
+
if args.resume:
|
| 99 |
+
try:
|
| 100 |
+
resume_checkpoint = find_latest_checkpoint(model_dir)
|
| 101 |
+
state_path = resume_checkpoint / "training_state.pt"
|
| 102 |
+
if state_path.exists():
|
| 103 |
+
resume_state = torch.load(state_path, map_location="cpu")
|
| 104 |
+
start_epoch = int(resume_state.get("epoch", 0)) + 1
|
| 105 |
+
except FileNotFoundError:
|
| 106 |
+
resume_checkpoint = None
|
| 107 |
+
resume_state = None
|
| 108 |
+
|
| 109 |
+
if resume_state is not None:
|
| 110 |
+
records = [record for record in load_loss_history(loss_csv_path) if int(record["epoch"]) < start_epoch]
|
| 111 |
+
|
| 112 |
+
if args.resume and resume_checkpoint is not None:
|
| 113 |
+
print(f"Resuming from checkpoint: {resume_checkpoint}")
|
| 114 |
+
processor = load_processor(str(resume_checkpoint))
|
| 115 |
+
model = SpeechT5ForTextToSpeech.from_pretrained(str(resume_checkpoint))
|
| 116 |
+
else:
|
| 117 |
+
processor = load_processor(args.base_model)
|
| 118 |
+
model = SpeechT5ForTextToSpeech.from_pretrained(args.base_model)
|
| 119 |
+
|
| 120 |
+
vocoder = load_vocoder()
|
| 121 |
+
|
| 122 |
+
model.to(device)
|
| 123 |
+
vocoder.to(device)
|
| 124 |
+
|
| 125 |
+
dataset = SpeechDataset(args.data_dir, seed=args.seed, max_examples=args.max_examples)
|
| 126 |
+
collator = TTSCollator(
|
| 127 |
+
tokenizer=processor.tokenizer,
|
| 128 |
+
feature_extractor=processor.feature_extractor,
|
| 129 |
+
speaker_embedding_dim=model.config.speaker_embedding_dim,
|
| 130 |
+
reduction_factor=model.config.reduction_factor,
|
| 131 |
+
max_text_length=args.max_text_length,
|
| 132 |
+
)
|
| 133 |
+
|
| 134 |
+
train_loader = DataLoader(
|
| 135 |
+
dataset,
|
| 136 |
+
batch_size=args.batch_size,
|
| 137 |
+
shuffle=True,
|
| 138 |
+
num_workers=args.num_workers,
|
| 139 |
+
pin_memory=device.type == "cuda",
|
| 140 |
+
collate_fn=collator,
|
| 141 |
+
)
|
| 142 |
+
|
| 143 |
+
planned_update_steps = max(1, math.ceil(len(train_loader) / args.gradient_accumulation_steps) * args.epochs)
|
| 144 |
+
total_update_steps = planned_update_steps
|
| 145 |
+
|
| 146 |
+
optimizer = torch.optim.AdamW(
|
| 147 |
+
model.parameters(),
|
| 148 |
+
lr=args.learning_rate,
|
| 149 |
+
weight_decay=args.weight_decay,
|
| 150 |
+
)
|
| 151 |
+
|
| 152 |
+
warmup_steps = max(1, int(total_update_steps * 0.05))
|
| 153 |
+
scheduler = get_linear_schedule_with_warmup(
|
| 154 |
+
optimizer,
|
| 155 |
+
num_warmup_steps=warmup_steps,
|
| 156 |
+
num_training_steps=total_update_steps,
|
| 157 |
+
)
|
| 158 |
+
|
| 159 |
+
use_amp = device.type == "cuda"
|
| 160 |
+
scaler = torch.cuda.amp.GradScaler(enabled=use_amp)
|
| 161 |
+
|
| 162 |
+
sample_texts = args.sample_text if args.sample_text else build_default_sample_texts()
|
| 163 |
+
started_at = datetime.utcnow().isoformat(timespec="seconds") + "Z"
|
| 164 |
+
|
| 165 |
+
end_epoch = start_epoch + args.epochs - 1
|
| 166 |
+
print(f"Training on {len(dataset)} examples for epochs {start_epoch}..{end_epoch}.")
|
| 167 |
+
print(f"Device: {device}")
|
| 168 |
+
print(f"Output root: {root_dir.resolve()}")
|
| 169 |
+
print(f"Started: {started_at}")
|
| 170 |
+
|
| 171 |
+
for epoch in range(start_epoch, end_epoch + 1):
|
| 172 |
+
model.train()
|
| 173 |
+
running_loss = 0.0
|
| 174 |
+
batch_count = 0
|
| 175 |
+
|
| 176 |
+
optimizer.zero_grad(set_to_none=True)
|
| 177 |
+
|
| 178 |
+
for step, batch in enumerate(train_loader, start=1):
|
| 179 |
+
batch_count += 1
|
| 180 |
+
batch = {name: tensor.to(device) for name, tensor in batch.items()}
|
| 181 |
+
|
| 182 |
+
with torch.autocast(device_type=device.type, dtype=torch.float16, enabled=use_amp):
|
| 183 |
+
outputs = model(**batch)
|
| 184 |
+
loss = outputs.loss
|
| 185 |
+
if loss is None:
|
| 186 |
+
raise RuntimeError("SpeechT5 did not return a loss. Check the batch preparation.")
|
| 187 |
+
scaled_loss = loss / args.gradient_accumulation_steps
|
| 188 |
+
|
| 189 |
+
scaler.scale(scaled_loss).backward()
|
| 190 |
+
running_loss += float(loss.detach().cpu())
|
| 191 |
+
|
| 192 |
+
if step % args.gradient_accumulation_steps == 0 or step == len(train_loader):
|
| 193 |
+
scaler.step(optimizer)
|
| 194 |
+
scaler.update()
|
| 195 |
+
optimizer.zero_grad(set_to_none=True)
|
| 196 |
+
scheduler.step()
|
| 197 |
+
|
| 198 |
+
if step % 25 == 0 or step == len(train_loader):
|
| 199 |
+
print(f"Epoch {epoch}/{args.epochs} Step {step}/{len(train_loader)} Loss {running_loss / batch_count:.4f}")
|
| 200 |
+
|
| 201 |
+
avg_loss = running_loss / max(1, batch_count)
|
| 202 |
+
records.append({"epoch": epoch, "loss": avg_loss})
|
| 203 |
+
|
| 204 |
+
checkpoint_dir = ensure_directory(model_dir / f"epoch_{epoch:04d}")
|
| 205 |
+
model.save_pretrained(checkpoint_dir)
|
| 206 |
+
processor.save_pretrained(checkpoint_dir)
|
| 207 |
+
torch.save(
|
| 208 |
+
{
|
| 209 |
+
"epoch": epoch,
|
| 210 |
+
"loss": avg_loss,
|
| 211 |
+
"total_update_steps": total_update_steps,
|
| 212 |
+
"optimizer_state_dict": optimizer.state_dict(),
|
| 213 |
+
"scheduler_state_dict": scheduler.state_dict(),
|
| 214 |
+
"scaler_state_dict": scaler.state_dict(),
|
| 215 |
+
"base_model": args.base_model,
|
| 216 |
+
"timestamp": datetime.utcnow().isoformat(timespec="seconds") + "Z",
|
| 217 |
+
},
|
| 218 |
+
checkpoint_dir / "training_state.pt",
|
| 219 |
+
)
|
| 220 |
+
save_checkpoint_marker(model_dir, checkpoint_dir)
|
| 221 |
+
|
| 222 |
+
rollout_dir = ensure_directory(rollout_root / f"epoch_{epoch:04d}")
|
| 223 |
+
save_rollout_samples(
|
| 224 |
+
model=model,
|
| 225 |
+
processor=processor,
|
| 226 |
+
vocoder=vocoder,
|
| 227 |
+
sample_texts=sample_texts,
|
| 228 |
+
output_dir=rollout_dir,
|
| 229 |
+
device=device,
|
| 230 |
+
maxlenratio=args.rollout_maxlenratio,
|
| 231 |
+
threshold=args.rollout_threshold,
|
| 232 |
+
)
|
| 233 |
+
|
| 234 |
+
write_loss_artifacts(records, loss_csv_path, loss_plot_path)
|
| 235 |
+
|
| 236 |
+
print(f"Epoch {epoch} complete. Avg loss: {avg_loss:.4f}")
|
| 237 |
+
print(f"Saved checkpoint: {checkpoint_dir}")
|
| 238 |
+
print(f"Saved rollout: {rollout_dir}")
|
| 239 |
+
|
| 240 |
+
metadata = {
|
| 241 |
+
"base_model": args.base_model,
|
| 242 |
+
"epochs": args.epochs,
|
| 243 |
+
"resume": args.resume,
|
| 244 |
+
"resumed_from": str(resume_checkpoint) if resume_checkpoint is not None else None,
|
| 245 |
+
"start_epoch": start_epoch,
|
| 246 |
+
"end_epoch": end_epoch,
|
| 247 |
+
"batch_size": args.batch_size,
|
| 248 |
+
"learning_rate": args.learning_rate,
|
| 249 |
+
"weight_decay": args.weight_decay,
|
| 250 |
+
"gradient_accumulation_steps": args.gradient_accumulation_steps,
|
| 251 |
+
"max_text_length": args.max_text_length,
|
| 252 |
+
"max_examples": args.max_examples,
|
| 253 |
+
"seed": args.seed,
|
| 254 |
+
"started_at": started_at,
|
| 255 |
+
}
|
| 256 |
+
(root_dir / "training_run.json").write_text(json.dumps(metadata, indent=2), encoding="utf-8")
|
| 257 |
+
|
| 258 |
+
|
| 259 |
+
if __name__ == "__main__":
|
| 260 |
+
main()
|
tts_common.py
ADDED
|
@@ -0,0 +1,283 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import io
|
| 4 |
+
import json
|
| 5 |
+
import re
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
from typing import Sequence
|
| 8 |
+
|
| 9 |
+
import matplotlib
|
| 10 |
+
|
| 11 |
+
matplotlib.use("Agg")
|
| 12 |
+
|
| 13 |
+
import matplotlib.pyplot as plt
|
| 14 |
+
import numpy as np
|
| 15 |
+
import pandas as pd
|
| 16 |
+
import soundfile as sf
|
| 17 |
+
import torch
|
| 18 |
+
import torchaudio
|
| 19 |
+
from transformers import SpeechT5FeatureExtractor, SpeechT5HifiGan, SpeechT5Processor, SpeechT5Tokenizer
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
TARGET_SAMPLE_RATE = 16000
|
| 23 |
+
DEFAULT_BASE_MODEL = "microsoft/speecht5_tts"
|
| 24 |
+
DEFAULT_VOCODER = "microsoft/speecht5_hifigan"
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def ensure_directory(path: Path) -> Path:
|
| 28 |
+
path.mkdir(parents=True, exist_ok=True)
|
| 29 |
+
return path
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def normalize_text(text: str) -> str:
|
| 33 |
+
return re.sub(r"\s+", " ", str(text)).strip()
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def sanitize_filename(text: str, max_length: int = 72) -> str:
|
| 37 |
+
cleaned = normalize_text(text)
|
| 38 |
+
cleaned = re.sub(r"[^A-Za-z0-9._-]+", "_", cleaned)
|
| 39 |
+
cleaned = re.sub(r"_+", "_", cleaned).strip("._-")
|
| 40 |
+
if not cleaned:
|
| 41 |
+
cleaned = "sample"
|
| 42 |
+
return cleaned[:max_length]
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def load_speech_dataset(data_dir: Path, seed: int = 42, max_examples: int | None = None):
|
| 46 |
+
parquet_files = sorted(str(path) for path in data_dir.glob("train-*.parquet"))
|
| 47 |
+
if not parquet_files:
|
| 48 |
+
raise FileNotFoundError(f"No parquet shards found in {data_dir}")
|
| 49 |
+
|
| 50 |
+
frames = [pd.read_parquet(path, columns=["file_name", "transcription", "transcription_normalised", "audio"]) for path in parquet_files]
|
| 51 |
+
frame = pd.concat(frames, ignore_index=True)
|
| 52 |
+
|
| 53 |
+
if max_examples is not None:
|
| 54 |
+
max_examples = min(max_examples, len(frame))
|
| 55 |
+
frame = frame.iloc[:max_examples].copy()
|
| 56 |
+
|
| 57 |
+
frame = frame.sample(frac=1.0, random_state=seed).reset_index(drop=True)
|
| 58 |
+
return frame
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def decode_audio_bytes(audio_dict: dict, target_sample_rate: int = TARGET_SAMPLE_RATE) -> np.ndarray:
|
| 62 |
+
raw_bytes = audio_dict["bytes"]
|
| 63 |
+
waveform, sample_rate = sf.read(io.BytesIO(raw_bytes), dtype="float32", always_2d=False)
|
| 64 |
+
|
| 65 |
+
if waveform.ndim == 2:
|
| 66 |
+
waveform = waveform.mean(axis=1)
|
| 67 |
+
|
| 68 |
+
waveform = np.asarray(waveform, dtype=np.float32)
|
| 69 |
+
if sample_rate != target_sample_rate:
|
| 70 |
+
tensor_waveform = torch.from_numpy(waveform)
|
| 71 |
+
waveform = torchaudio.functional.resample(tensor_waveform, sample_rate, target_sample_rate).numpy()
|
| 72 |
+
|
| 73 |
+
return trim_silence(waveform)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def trim_silence(waveform: np.ndarray, threshold: float = 0.01, pad_seconds: float = 0.05) -> np.ndarray:
|
| 77 |
+
if waveform.size == 0:
|
| 78 |
+
return waveform.astype(np.float32, copy=False)
|
| 79 |
+
|
| 80 |
+
peak = float(np.max(np.abs(waveform)))
|
| 81 |
+
if peak <= 0.0:
|
| 82 |
+
return waveform.astype(np.float32, copy=False)
|
| 83 |
+
|
| 84 |
+
active = np.flatnonzero(np.abs(waveform) >= peak * threshold)
|
| 85 |
+
if active.size == 0:
|
| 86 |
+
return waveform.astype(np.float32, copy=False)
|
| 87 |
+
|
| 88 |
+
pad = int(TARGET_SAMPLE_RATE * pad_seconds)
|
| 89 |
+
start = max(int(active[0]) - pad, 0)
|
| 90 |
+
end = min(int(active[-1]) + pad + 1, waveform.shape[0])
|
| 91 |
+
return np.asarray(waveform[start:end], dtype=np.float32)
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
class SpeechDataset(torch.utils.data.Dataset):
|
| 95 |
+
def __init__(self, data_dir: Path, seed: int = 42, max_examples: int | None = None):
|
| 96 |
+
self.dataset = load_speech_dataset(data_dir=data_dir, seed=seed, max_examples=max_examples)
|
| 97 |
+
|
| 98 |
+
def __len__(self) -> int:
|
| 99 |
+
return len(self.dataset)
|
| 100 |
+
|
| 101 |
+
def __getitem__(self, index: int) -> dict:
|
| 102 |
+
row = self.dataset.iloc[int(index)]
|
| 103 |
+
text = row.get("transcription_normalised") or row.get("transcription") or ""
|
| 104 |
+
return {
|
| 105 |
+
"text": normalize_text(text),
|
| 106 |
+
"audio": row["audio"],
|
| 107 |
+
"file_name": row.get("file_name", ""),
|
| 108 |
+
}
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
class TTSCollator:
|
| 112 |
+
def __init__(
|
| 113 |
+
self,
|
| 114 |
+
tokenizer: SpeechT5Tokenizer,
|
| 115 |
+
feature_extractor: SpeechT5FeatureExtractor,
|
| 116 |
+
speaker_embedding_dim: int,
|
| 117 |
+
reduction_factor: int = 1,
|
| 118 |
+
max_text_length: int = 512,
|
| 119 |
+
):
|
| 120 |
+
self.tokenizer = tokenizer
|
| 121 |
+
self.feature_extractor = feature_extractor
|
| 122 |
+
self.speaker_embedding_dim = speaker_embedding_dim
|
| 123 |
+
self.reduction_factor = max(1, int(reduction_factor))
|
| 124 |
+
self.max_text_length = max_text_length
|
| 125 |
+
|
| 126 |
+
def _pad_spectrograms_for_reduction_factor(
|
| 127 |
+
self,
|
| 128 |
+
labels: torch.Tensor,
|
| 129 |
+
attention_mask: torch.Tensor,
|
| 130 |
+
) -> tuple[torch.Tensor, torch.Tensor]:
|
| 131 |
+
current_length = labels.shape[1]
|
| 132 |
+
pad_frames = (-current_length) % self.reduction_factor
|
| 133 |
+
if pad_frames == 0:
|
| 134 |
+
return labels, attention_mask
|
| 135 |
+
|
| 136 |
+
pad_labels = labels.new_full((labels.shape[0], pad_frames, labels.shape[2]), -100.0)
|
| 137 |
+
pad_mask = attention_mask.new_zeros((attention_mask.shape[0], pad_frames))
|
| 138 |
+
labels = torch.cat([labels, pad_labels], dim=1)
|
| 139 |
+
attention_mask = torch.cat([attention_mask, pad_mask], dim=1)
|
| 140 |
+
return labels, attention_mask
|
| 141 |
+
|
| 142 |
+
def __call__(self, batch: Sequence[dict]) -> dict[str, torch.Tensor]:
|
| 143 |
+
texts = [item["text"] for item in batch]
|
| 144 |
+
waveforms = [decode_audio_bytes(item["audio"]) for item in batch]
|
| 145 |
+
|
| 146 |
+
tokenized = self.tokenizer(
|
| 147 |
+
texts,
|
| 148 |
+
padding=True,
|
| 149 |
+
truncation=True,
|
| 150 |
+
max_length=self.max_text_length,
|
| 151 |
+
return_tensors="pt",
|
| 152 |
+
)
|
| 153 |
+
|
| 154 |
+
spectrograms = self.feature_extractor(
|
| 155 |
+
audio_target=waveforms,
|
| 156 |
+
sampling_rate=TARGET_SAMPLE_RATE,
|
| 157 |
+
padding=True,
|
| 158 |
+
return_tensors="pt",
|
| 159 |
+
)
|
| 160 |
+
|
| 161 |
+
labels = spectrograms["input_values"].to(torch.float32)
|
| 162 |
+
attention_mask = spectrograms["attention_mask"].to(torch.long)
|
| 163 |
+
labels, attention_mask = self._pad_spectrograms_for_reduction_factor(labels, attention_mask)
|
| 164 |
+
labels = labels.masked_fill(attention_mask.unsqueeze(-1) == 0, -100.0)
|
| 165 |
+
|
| 166 |
+
speaker_embeddings = torch.zeros((len(batch), self.speaker_embedding_dim), dtype=torch.float32)
|
| 167 |
+
|
| 168 |
+
return {
|
| 169 |
+
"input_ids": tokenized["input_ids"],
|
| 170 |
+
"attention_mask": tokenized["attention_mask"],
|
| 171 |
+
"labels": labels,
|
| 172 |
+
"speaker_embeddings": speaker_embeddings,
|
| 173 |
+
}
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
def load_processor(model_name: str = DEFAULT_BASE_MODEL) -> SpeechT5Processor:
|
| 177 |
+
return SpeechT5Processor.from_pretrained(model_name)
|
| 178 |
+
|
| 179 |
+
|
| 180 |
+
def load_tokenizer(model_name: str = DEFAULT_BASE_MODEL) -> SpeechT5Tokenizer:
|
| 181 |
+
return SpeechT5Tokenizer.from_pretrained(model_name)
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
def load_vocoder(model_name: str = DEFAULT_VOCODER) -> SpeechT5HifiGan:
|
| 185 |
+
return SpeechT5HifiGan.from_pretrained(model_name)
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def write_loss_artifacts(records: list[dict], csv_path: Path, plot_path: Path) -> None:
|
| 189 |
+
ensure_directory(csv_path.parent)
|
| 190 |
+
ensure_directory(plot_path.parent)
|
| 191 |
+
|
| 192 |
+
frame = pd.DataFrame(records)
|
| 193 |
+
frame.to_csv(csv_path, index=False)
|
| 194 |
+
|
| 195 |
+
plt.figure(figsize=(8, 5))
|
| 196 |
+
plt.plot(frame["epoch"], frame["loss"], linewidth=2.0)
|
| 197 |
+
plt.xlabel("Epoch")
|
| 198 |
+
plt.ylabel("Loss")
|
| 199 |
+
plt.title("Training Loss")
|
| 200 |
+
plt.grid(True, alpha=0.25)
|
| 201 |
+
plt.tight_layout()
|
| 202 |
+
plt.savefig(plot_path, dpi=160)
|
| 203 |
+
plt.close()
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
def find_latest_checkpoint(model_root: Path) -> Path:
|
| 207 |
+
latest_marker = model_root / "last_checkpoint.txt"
|
| 208 |
+
if latest_marker.exists():
|
| 209 |
+
stored = Path(latest_marker.read_text(encoding="utf-8").strip())
|
| 210 |
+
if stored.exists():
|
| 211 |
+
return stored
|
| 212 |
+
|
| 213 |
+
candidates = []
|
| 214 |
+
for path in model_root.glob("epoch_*"):
|
| 215 |
+
if path.is_dir():
|
| 216 |
+
match = re.search(r"epoch_(\d+)", path.name)
|
| 217 |
+
if match:
|
| 218 |
+
candidates.append((int(match.group(1)), path))
|
| 219 |
+
|
| 220 |
+
if not candidates:
|
| 221 |
+
raise FileNotFoundError(f"No checkpoints found in {model_root}")
|
| 222 |
+
|
| 223 |
+
candidates.sort(key=lambda item: item[0])
|
| 224 |
+
return candidates[-1][1]
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
def save_checkpoint_marker(model_root: Path, checkpoint_path: Path) -> None:
|
| 228 |
+
ensure_directory(model_root)
|
| 229 |
+
marker = model_root / "last_checkpoint.txt"
|
| 230 |
+
marker.write_text(str(checkpoint_path.resolve()), encoding="utf-8")
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
def save_rollout_samples(
|
| 234 |
+
model,
|
| 235 |
+
processor: SpeechT5Processor,
|
| 236 |
+
vocoder,
|
| 237 |
+
sample_texts: Sequence[str],
|
| 238 |
+
output_dir: Path,
|
| 239 |
+
device: torch.device,
|
| 240 |
+
maxlenratio: float,
|
| 241 |
+
threshold: float,
|
| 242 |
+
) -> None:
|
| 243 |
+
ensure_directory(output_dir)
|
| 244 |
+
model.eval()
|
| 245 |
+
vocoder.eval()
|
| 246 |
+
|
| 247 |
+
metadata = []
|
| 248 |
+
with torch.inference_mode():
|
| 249 |
+
for index, text in enumerate(sample_texts, start=1):
|
| 250 |
+
prompt = normalize_text(text)
|
| 251 |
+
inputs = processor.tokenizer(prompt, return_tensors="pt")
|
| 252 |
+
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}
|
| 253 |
+
speaker_embeddings = torch.zeros((1, model.config.speaker_embedding_dim), device=device)
|
| 254 |
+
|
| 255 |
+
waveform = model.generate_speech(
|
| 256 |
+
input_ids=inputs["input_ids"],
|
| 257 |
+
attention_mask=inputs.get("attention_mask"),
|
| 258 |
+
speaker_embeddings=speaker_embeddings,
|
| 259 |
+
threshold=threshold,
|
| 260 |
+
maxlenratio=maxlenratio,
|
| 261 |
+
vocoder=vocoder,
|
| 262 |
+
)
|
| 263 |
+
|
| 264 |
+
if isinstance(waveform, tuple):
|
| 265 |
+
waveform = waveform[0]
|
| 266 |
+
if waveform.ndim > 1:
|
| 267 |
+
waveform = waveform.squeeze(0)
|
| 268 |
+
|
| 269 |
+
sample_name = f"sample_{index:02d}.wav"
|
| 270 |
+
sample_path = output_dir / sample_name
|
| 271 |
+
sf.write(sample_path, waveform.detach().cpu().numpy().astype(np.float32), TARGET_SAMPLE_RATE)
|
| 272 |
+
(output_dir / f"sample_{index:02d}.txt").write_text(prompt, encoding="utf-8")
|
| 273 |
+
metadata.append({"file": sample_name, "text": prompt})
|
| 274 |
+
|
| 275 |
+
(output_dir / "samples.json").write_text(json.dumps(metadata, ensure_ascii=False, indent=2), encoding="utf-8")
|
| 276 |
+
|
| 277 |
+
|
| 278 |
+
def build_default_sample_texts() -> list[str]:
|
| 279 |
+
return [
|
| 280 |
+
"Hello, this is an epoch sample.",
|
| 281 |
+
"The model should become clearer and more stable as training continues.",
|
| 282 |
+
"This longer sentence checks whether the synthesizer can keep going without cutting off too early.",
|
| 283 |
+
]
|