Add model card

README.md

@@ -0,0 +1,176 @@
+---
+license: mit
+language:
+  - nan
+  - zh
+  - en
+library_name: GPT-SoVITS
+tags:
+  - text-to-speech
+  - tts
+  - voice-cloning
+  - taiwanese
+  - hokkien
+  - poj
+  - peh-oe-ji
+  - gpt-sovits
+pipeline_tag: text-to-speech
+base_model:
+  - lj1995/GPT-SoVITS
+---
+
+# GPT-SoVITS Taiwanese (Hokkien) — Trilingual S1 + r4 e15 S2
+
+Pre-trained weights for the Taiwanese (Hokkien / Pe̍h-ōe-jī) fork of
+[GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS). The S1 is
+trilingual (TW + ZH + weak EN) thanks to embedding transplant; the S2
+is a v2ProTw vocoder finetuned on Taiwanese audio.
+
+**Inference code, sandhi preprocessor, training recipe, and Traditional
+Chinese documentation:**
+[github.com/KaedeTai/GPT-SoVITS](https://github.com/KaedeTai/GPT-SoVITS) ·
+[TAIWANESE.md](https://github.com/KaedeTai/GPT-SoVITS/blob/main/TAIWANESE.md) ·
+[TAIWANESE.zh-tw.md](https://github.com/KaedeTai/GPT-SoVITS/blob/main/TAIWANESE.zh-tw.md)
+
+## Files
+
+| File | Size | What |
+|------|------|------|
+| `s1_trilingual.ckpt` | 156 MB | S1 GPT — TW (sandhi-trained, e15) + transplanted base ZH/EN embeddings |
+| `s2_r4_e15.pth` | 952 MB | S2 SoVITS v2ProTw — full-state ckpt at epoch 15 of finetune run r4 |
+
+## Quick start
+
+```bash
+git clone https://github.com/KaedeTai/GPT-SoVITS.git
+cd GPT-SoVITS
+python3.11 -m venv .venv && source .venv/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt
+./download_pretrained.sh   # upstream base pretraineds (BERT, hubert, etc.)
+
+# Pull these weights
+hf download KaedeTai/gpt-sovits-tw s1_trilingual.ckpt --local-dir ./models
+hf download KaedeTai/gpt-sovits-tw s2_r4_e15.pth      --local-dir ./models
+
+# One-line synthesis (POJ-with-diacritics in, mp3 out)
+python -m tw_inference.tts_cli "Lí hó, sè-kài!" -o hello.mp3
+```
+
+Or the local web UI:
+
+```bash
+python -m tw_inference.webui   # → http://127.0.0.1:5557/
+```
+
+## Quality
+
+| Language | Fluency | Pronunciation | Notes |
+|----------|---------|---------------|-------|
+| Taiwanese (POJ) | ~80 / 100 | ~75 / 100 | Single trained speaker; long sentences (>60 syllables) occasionally drift. |
+| Mandarin (中文) | usable | usable | Preserved via embedding transplant from `s1v3` base. |
+| English | weak | weak | Base never had real English; included for completeness only. |
+
+Code-switching within one utterance is **not supported** — use
+`{tw:...}` / `{zh:...}` markup per segment.
+
+## Architecture
+
+Two-stage TTS:
+
+- **S1 (GPT)** — autoregressive token model mapping POJ phoneme tokens
+  → SoVITS semantic codes. Vocabulary expanded from 732 → 1033 (301
+  Taiwanese `tw_*` tokens added on top of the upstream Mandarin
+  vocabulary). The trilingual variant preserves Mandarin by
+  transplanting rows 0..731 of the embedding table from a clean
+  `s1v3` checkpoint back into the TW-finetuned ckpt.
+- **S2 (SoVITS v2Pro / v2ProTw)** — non-autoregressive vocoder; takes
+  semantic codes + a speaker embedding (cnhubert + sv) and produces
+  32 kHz mono waveform.
+- **Sandhi preprocessor** — applies standard Taiwanese tone-sandhi
+  rules to citation-tone POJ before tokenization, so the model sees
+  the tone sequence speakers actually produce. 13 flags; defaults
+  match the eval configuration that produced our best reported CER.
+
+## Training data
+
+- **MoE Tâi-uân-gí 教育部臺灣閩南語常用詞辭典** example sentences
+  (majority of the corpus).
+- **Common Voice `nan-tw`** validated split.
+- Multi-speaker. Per-segment 3-12 s, 32 kHz mono, loudness normalised.
+- Labels: POJ with diacritics, pre-processed with the sandhi
+  preprocessor so the written form matches the audio realisation.
+
+Total: roughly 15-25 hours of paired audio + POJ.
+
+## Evaluation
+
+Reported quality is from human listening; ASR-based CER was used for
+ablations but flattens out at the top of the quality curve.
+
+| Test set | Stack | Mean POJ-CER (BreezeASR-26-derived) |
+|----------|-------|--------------------------------------|
+| Canonical 5-sentence | S1 trilingual e15 + S2 r4 e15 + sandhi v1 | **4.44%** |
+| 13-sentence long content | same | ~15% |
+
+Per-sentence breakdown for the 5-sentence set is in
+[`tw_samples/eval_summary.json`](https://github.com/KaedeTai/GPT-SoVITS/blob/main/tw_samples/eval_summary.json)
+in the GitHub repo. Demo mp3s are in
+[`tw_samples/demo_*.mp3`](https://github.com/KaedeTai/GPT-SoVITS/tree/main/tw_samples).
+
+## Known limitations
+
+- **English is weak.** Don't ship this for English use cases.
+- **Long sentences drift** past ~60 syllables. The inference pipeline
+  splits at punctuation to mitigate but doesn't eliminate this.
+- **Code-switching not supported** within a single utterance.
+- **Single training speaker fidelity** is capped by the multi-speaker
+  corpus heterogeneity; with a single-speaker corpus we'd expect
+  higher voice consistency but narrower coverage.
+- **POJ input only.** No built-in Han-character → POJ pipeline.
+- **MPS nondeterminism.** Same seed + same machine still produces
+  audibly different output across runs (5-10% spread).
+
+## How this was built (short version)
+
+The long version with lessons learned and what we'd do differently is
+in [TAIWANESE.md](https://github.com/KaedeTai/GPT-SoVITS/blob/main/TAIWANESE.md#lessons-learned).
+Short version:
+
+1. **S2 first** (~24 h on M1 Max): full SoVITS v2ProTw finetune from
+   `s2Gv2Pro.pth`. 15 epochs.
+2. **S1 next** (~12-30 h): `s1_train_mps_arpa_freeze.py` from
+   `s1v3.ckpt`, ARPA-row freeze, warmup → cosine LR (peak 1e-2, end
+   1e-4, 2000-step warmup, 40k-step decay). Critical patch: upstream
+   `lr_schedulers.py` had a hardcode locking every run to LR=0.002
+   regardless of yaml; that's now removed.
+3. **Sandhi-aligned labels are non-negotiable.** Training on
+   citation-tone POJ when the recordings have natural sandhi produces
+   a systematically mispronouncing model.
+4. **Embedding transplant** for the trilingual variant: copy rows
+   0..731 from a clean `s1v3` back into the TW-finetuned ckpt.
+   Restores Mandarin without touching the trained TW rows.
+
+## License & credits
+
+- License: **MIT** (matches upstream GPT-SoVITS).
+- Upstream: [RVC-Boss/GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS).
+- TW adaptation: [KaedeTai](https://github.com/KaedeTai).
+- Acknowledgments: MoE 教育部臺灣閩南語常用詞辭典 example sentence
+  corpus, Common Voice `nan-tw` (Mozilla), BreezeASR-26 (MediaTek)
+  for TW ASR eval, linshoufan/whisper-small-nan-tw-pinyin for POJ
+  ASR.
+
+## Citation
+
+If you find this useful in academic work, please cite the upstream
+GPT-SoVITS and this fork:
+
+```bibtex
+@misc{gpt-sovits-tw-2026,
+  title  = {GPT-SoVITS Taiwanese (Hokkien) trilingual fork},
+  author = {KaedeTai},
+  year   = {2026},
+  howpublished = {\url{https://huggingface.co/KaedeTai/gpt-sovits-tw}}
+}
+```