speecht5_tts-pld-ceb-v2

sapinsapin/speecht5_tts-pld-ceb continue-finetuned on every usable Cebuano clip in sapinsapin/pld — 14,007 of them, against the ~1,800 the published checkpoint saw.

It was never a shortage of data. PLD holds ~50,900 Cebuano rows; 15,163 pass the TTS filter (read speech, not a prompt, ≥3 words) and 14,007 survive the digit and length caps — 19 hours. The original run stopped at 1,847 because --max-samples 3000 was a default nobody revisited. This gives the same model the rest of its own language.

"Continue", not "resume": the published repo ships model.safetensors and training_args.bin and nothing else — no optimizer moments, no scheduler state — so this warm-starts from the released weights with a fresh optimizer and a fresh linear schedule.

training clips	13,807 train + 200 eval (14,007 total)
steps	4,000 (≈9 epochs)
batch	8 × 4 accumulation
learning rate	1e-05
precision	fp32 + gradient checkpointing (fp16 NaNs SpeechT5's mel loss)
eval loss	0.3687, against 0.4070 for the base checkpoint
hardware	one free-tier Colab T4, ~3 h

What it bought

There is no way to measure a synthetic voice directly, so the test is indirect: play each clip to a speech recognizer that never saw the original sentence, and count how much of the text it fails to recover. Character error rate (CER) is the share of characters it gets wrong. Lower is better. This measures whether speech is intelligible to a machine listener — not whether it sounds natural or authentically Cebuano, which still needs human ears.

Ten held-out pld test lines, transcribed by whisper-large-v3-turbo:

	base	v2
CER, arctic:slt	0.141	0.093
WER, arctic:slt	0.497	0.388
CER, native pld:ceb:12	0.210	0.148
WER, native pld:ceb:12	0.637	0.524

A third of the character errors, gone. At 0.093 this scores below mms-tts-ceb on the same lines (0.153).

Four listeners, not one

A single judge is a single opinion, and every recognizer is deaf in its own way. Whisper has no Cebuano in its training data at all, so some share of every score above is the judge's failure rather than the model's — a real Cebuano speaker reading these same lines scores 0.081, not 0. Rescoring the same audio with four different recognizers separates the two:

judge	base	v2	what it is
sapinsapin/whisper-small-pld-ceb	0.139	0.065	small Whisper, finetuned on Cebuano
whisper-large-v3-turbo	0.146	0.123	large, zero-shot, no Cebuano
whisper-turbo + Filipino LoRA	0.126	0.092	large, nudged toward Philippine speech
facebook/mms-1b-all (ceb adapter)	0.125	0.067	CTC model, 491k hours, ceb head

CER, arctic:slt, ten lines. This is a different bench run from the table above — which is why turbo reads 0.123 here and 0.093 there, with no change to the weights. See the determinism note at the bottom; it is the same lesson.

The pattern is the finding. The judge that actually knows Cebuano sees the retrain cut errors by 53%; the judge that does not sees 16%. Zero-shot Whisper is measuring partly its own ignorance, and that noise compresses the distance between a good model and a poor one. Read down a column, never across: a CTC model and a sequence-to-sequence model make different kinds of mistakes, so their absolute numbers are not on the same scale.

One caveat on the in-domain judge: whisper-small-pld-ceb was finetuned on the same corpus this model trained on. It is the most informed listener available, and also the most likely to reward speech that sounds specifically like PLD. Treat it as the sharpest instrument, not the neutral one.

What it did not fix

Speaker conditioning. Handed a native Cebuano x-vector instead of the American arctic:slt, the decoder still collapses on most voices: 3 of 16 native candidates survive, against 1 of 16 for the base checkpoint. Tripled, still 13 of 16 broken — one renders a 4-second line as 17 seconds at rms 0.003.

Seven-fold data was not the fix, because the bottleneck is not clip count but clip distribution: those 19 hours are 139 speakers, and the best-covered has 15.5 minutes. PLD is broad and shallow — built for ASR coverage, not TTS depth. Averaging harder over 139 people does not produce one person.

If you need a single reliable voice, use Splintir/speecht5_tts-pld-ceb-solo, which continue-trains this checkpoint on one speaker alone.

Usage

SpeechT5 holds no voice of its own — every call needs a 512-d x-vector. None ships with the original PLD checkpoints, which is why they are usually run with an American speaker from the HuggingFace tutorial. speaker.npy in this repo is a real ceb speaker from the training data, so you can skip that.

import numpy as np, torch, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import SpeechT5ForTextToSpeech, SpeechT5HifiGan, SpeechT5Processor

REPO = "Splintir/speecht5_tts-pld-ceb-v2"

processor = SpeechT5Processor.from_pretrained(REPO)
model = SpeechT5ForTextToSpeech.from_pretrained(REPO).eval()
vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").eval()

# speaker.npy is stored as (512,); the model wants (1, 512). Without the
# unsqueeze this raises "The first dimension of speaker_embeddings must be
# either 1 or the same as batch size".
speaker = torch.from_numpy(np.load(hf_hub_download(REPO, "speaker.npy"))).float().unsqueeze(0)

ids = processor(text="Maayong buntag sa imong tanan.", return_tensors="pt")["input_ids"]
with torch.inference_mode():
    speech = model.generate_speech(ids, speaker, vocoder=vocoder)

sf.write("out.wav", speech.numpy(), 16000)   # 16 kHz mono

# In a notebook (Colab, Jupyter), play it inline instead of saving:
from IPython.display import Audio, display
display(Audio(speech.numpy(), rate=16000))

Runs unmodified on a stock Colab CPU runtime — every dependency is preinstalled, including the sentencepiece the tokenizer needs — at roughly 1.8× real time. For GPU, .to("cuda") the model, the vocoder and ids/speaker, then .cpu() the result before sf.write.

The display(Audio(...)) line renders a play button in a notebook cell. It produces no sound over a terminal or SSH session; there, write the wav and fetch it (from google.colab import files; files.download("out.wav")).

Spell numbers out: the tokenizer is character-level Latin and drops digits silently.

Two things to know before trusting a number

Generation is not deterministic. SpeechT5's decoder prenet keeps dropout active during inference, by design, as Tacotron2's does. Five renders of one line from one checkpoint gave five different durations (3.17–3.30 s) and rms 0.062–0.074. This checkpoint scored CER 0.093 in one bench run and 0.123 in another with nothing changed — so compare systems within a run, never across runs. In production, cache audio by hash(text + voice) to freeze one known-good render.

An out-of-distribution x-vector produces 12 seconds of quiet mumbling, not an error. If output is long and near-silent, the speaker embedding is the suspect, not the text.

Trained with scripts/train_tts.py; preprocessing matches finetune_tts.py from the halohalo pipeline — one x-vector per clip, never averaged.