--- language: - eu license: apache-2.0 base_model: Audio8/Audio8-TTS-Preview-0.6b tags: - text-to-speech - tts - basque - euskara - audio - audio8 - fine-tuned library_name: transformers pipeline_tag: text-to-speech datasets: - mozilla-foundation/common_voice_17_0 - language-and-voice-lab/hitz_aholab_eu_tts --- # zortzi-tts **Basque (`eu`) fine-tune of [Audio8-TTS-Preview-0.6b](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b)** — a 0.6B DualAR text-to-speech model. This checkpoint speaks **only Basque** with two voices: **Maider** (female) and **Antton** (male), anchored to the [HiTZ-Aholab](https://zenodo.org/records/17952596) speakers. > 🏗️ **Source code & training recipe:** [github.com/itzune/zortzi-tts](https://github.com/itzune/zortzi-tts) --- ## Model details | | | |---|---| | **Architecture** | DualAR TTS (Qwen2-style decoder + codec) | | **Parameters** | 601M (AR) + codec | | **Dtype** | BF16 | | **Vocab size** | 155,776 (BPE, Qwen2-style) | | **Codebooks** | 10 × 4096 entries | | **Audio** | 44.1 kHz, ~21.5 frames/s | | **Max sequence length** | 2048 (~95s hard ceiling; keep clips ≤20s for stable training) | | **Base model** | [`Audio8/Audio8-TTS-Preview-0.6b`](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b) (Apache-2.0) | | **License** | Apache-2.0 (inherited from base) | ### Voices Only **two voices** are available. They are carried by reference audio clips (their codec codes are the primary voice carrier; the speaker token is always `<|speaker:0|>`). | Voice | Gender | Reference clip | Reference text | |-------|--------|----------------|----------------| | **Maider** | Female | `voices/maider.wav` | *Aurrelaria prest dago jokatzeko.* | | **Antton** | Male | `voices/antton.wav` | *Inguru hura gerrillarien esku izan da denbora luzean.* | These reference clips are included in this repository under `voices/`. They were selected with `find_neutral_ref.py` (pitch-range/median ratio < 0.5) so that the reference does not bleed expressive style into the output. --- ## Quick start (GPU — PyTorch) This is the highest-fidelity path: full BF16, no quantization, KV-cache on device. ### Install ```bash # Clone the Audio8_TTS runtime (provides audio8_tts_infer.py + custom modeling code) git clone https://github.com/Audio8-AI/Audio8_TTS.git cd Audio8_TTS uv venv --python 3.12 .venv uv pip install --python .venv/bin/python --torch-backend=cu124 \ transformers soundfile torch ``` ### Synthesize ```bash .venv/bin/python audio8_tts_infer.py \ --model itzune/zortzi-tts \ --text "Kaixo mundua! Nire izena Maider da." \ --reference-audio voices/maider.wav \ --reference-text "Aurrelaria prest dago jokatzeko." \ --output out.wav \ --temperature 0.8 --top-p 0.95 ``` > ⚠️ **Sampling is required.** Do **not** pass `--greedy`. Greedy decoding > causes repetition loops that never reach EOS. Always use > `--temperature 0.8 --top-p 0.95`. ### Switch voices Change the `--reference-audio` and `--reference-text` to the Antton clip: ```bash .venv/bin/python audio8_tts_infer.py \ --model itzune/zortzi-tts \ --text "Kaixo mundua! Nire izena Antton da." \ --reference-audio voices/antton.wav \ --reference-text "Inguru hura gerrillarien esku izan da denbora luzean." \ --output out.wav \ --temperature 0.8 --top-p 0.95 ``` ### Programmatic usage ```python import torch from transformers import AutoModel, AutoProcessor model = AutoModel.from_pretrained( "itzune/zortzi-tts", torch_dtype=torch.bfloat16, trust_remote_code=True, ).cuda() processor = AutoProcessor.from_pretrained("itzune/zortzi-tts", trust_remote_code=True) ``` See the upstream [`audio8_tts_infer.py`](https://github.com/Audio8-AI/Audio8_TTS/blob/main/audio8_tts_infer.py) for the full generation loop. --- ## ONNX deployment (CPU) For CPU deployment without PyTorch/GPU, see the companion model card: **[`itzune/zortzi-tts-onnx`](https://huggingface.co/itzune/zortzi-tts-onnx)** — INT4-quantized ONNX models (575 MB total) that run on the Audio8 ONNX runtime. --- ## Training summary A **two-phase curriculum** with full fine-tuning (not LoRA) on an NVIDIA L40 (48 GB). Full details in the [source repo](https://github.com/itzune/zortzi-tts). ### Phase 1 — phonotactics base (Common Voice 26.0) Learned the text→semantic (phonotactics) mapping for Basque across many speakers, with **no reference audio**. - **Data:** Mozilla Common Voice 26.0 Basque, 134,531 validated clips (CC0) *(v26.0 was downloaded directly from the [Common Voice website](https://commonvoice.mozilla.org/); the closest HuggingFace mirror is [`common_voice_17_0`](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0))* - **Config:** LR 1e-5 cosine, batch 4×4, max_steps 6000 - **Runtime:** ~64 min - **Key insight:** stopped at step 6000 (the "Goldilocks" zone) to avoid prosodic hysteresis — CV is flat read-speech, and training too long entrenches a neutral prosody prior that later phases cannot fully reverse ### Phase 2 — voice anchor + prosody polish (HiTZ, merged) Anchored Maider/Antton voice identity **and** restored expressive prosody (especially wh-question intonation) in a single pass. - **Data:** HiTZ-Aholab Basque TTS (Maider + Antton, 27,000 clips, CC BY 4.0), **upsampled** to 41,300 clips: declarative ×1, interrogative ×3, exclamative ×3 (functionally equivalent to sequence-level loss weighting) - **References:** cross-clip self-conditioning (ref = a *different* clip from the same speaker — teaches the zero-shot voice-cloning path) - **Config:** LR 5e-6 linear decay, batch 4×4, 3 epochs, warmup 0.03 - **Started from:** Phase 1 step-6000 checkpoint - **Final:** 7,746 steps, train_loss=29.96, slow_accuracy=30.9% ### Key decisions | Decision | Rationale | |----------|-----------| | Full fine-tune over LoRA | New-language phonotactics require full weight updates | | Two-phase curriculum | Phase 1 (CV) learns phonotactics; Phase 2 (HiTZ) anchors voice + prosody | | Stop Phase 1 at step 6000 | Avoid prosodic hysteresis from flat CV read-speech | | Upsample interrogatives/exclamatives 3× | Sequence-level loss weighting for prosody | | Cross-clip self-conditioning | Teaches zero-shot voice-cloning path | | Sampling (temp 0.8, top-p 0.95) | Greedy causes infinite repetition loops | --- ## Limitations & known issues - **Numerals are not pronounced in Basque.** Even with text normalization (digits expanded to Basque words), the model pronounces numbers in a mix of languages rather than Basque. Spell out numbers manually if correct pronunciation is required. - **No built-in text normalization at inference.** Audio8_TTS performs no language-specific G2P, number-to-words, or abbreviation expansion. Text is BPE-tokenized verbatim. Acronyms like "TTS" should be expanded to "te te ese" upstream of the model. - **Two voices only.** This checkpoint speaks only Basque with Maider and Antton. No other voices are registered. - **Sampling required.** Greedy decoding produces repetition loops. --- ## Datasets, licenses & acknowledgements This model is a fine-tuned derivative of the Audio8 TTS preview model. The fine-tuned weights are released under **Apache-2.0**, inherited from the upstream model. The training data carries its own obligations: | Dataset | Role | License | Clips | |---|---|---|---| | [Audio8-TTS-Preview-0.6b](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b) | Base model | Apache-2.0 | — | | [Mozilla Common Voice 26.0](https://commonvoice.mozilla.org/) — Basque ¹ | Phase 1: phonotactics | **CC0 1.0** | 134,531 | | [HiTZ-Aholab Basque TTS](https://zenodo.org/records/17952596) (Maider + Antton) | Phase 2: voice + prosody | **CC BY 4.0** | 27,000 | > ¹ Common Voice 26.0 was obtained directly from the Common Voice website. > The closest HuggingFace dataset mirror is > [`mozilla-foundation/common_voice_17_0`](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0). > **Redistribution note.** Because the CC BY 4.0 HiTZ-Aholab data conditions > the voice of this model (Maider, Antton), any redistribution of the > fine-tuned weights or generated audio must carry the HiTZ attribution and > citation below. ### Funding acknowledgements The HiTZ-Aholab speech synthesis dataset was developed with funding from: > The Ministerio para la Transformación Digital y de la Función Pública and > Plan de Recuperación, Transformación y Resiliencia — Funded by EU — > NextGenerationEU within the framework of the project ILENIA > (ref. 2022/TL22/00215335), and by a grant from the Department of Culture > and Language Policy of the Basque Government (IKER-GAITU project). The `zortzi-tts` fine-tuning work itself is independent and not directly funded by the above grants; they are acknowledged here solely because the voice-anchor dataset carries the obligation. ## Citation If you use this model or build on its training recipe, please cite the base model, the datasets, and this work: ```bibtex @dataset{itzune_zortzi_tts, author = {{itzune}}, title = {{zortzi-tts: Basque fine-tune of Audio8-TTS-Preview-0.6b}}, year = 2025, publisher = {Hugging Face}, url = {https://huggingface.co/itzune/zortzi-tts} } @inproceedings{ardila-etal-2020-common, title = "Common Voice: A Massively-Multilingual Speech Corpus", author = "Ardila, Rosana and Branson, Megan and Davis, Kelly and Kohler, Michael and Meyer, Josh and Henretty, Michael and Morais, Reuben and Saunders, Lindsay and Tyers, Francis and Weber, Gregor", booktitle = "Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC)", month = may, year = "2020", address = "Marseille, France", publisher = "European Language Resources Association", pages = "4218--4222", url = "https://aclanthology.org/2020.lrec-1.520/" } @dataset{navas_hernaez_2025_17952596, author = {Navas, Eva and Hernaez Rioja, Inmaculada and Saratxaga, Ibon and Sanchez, Jon and García Romillo, Víctor and Flores Ríos, Mariana and Bellanco, Aitor}, title = {{HiTZ-Aholab speech synthesis dataset in Basque}}, month = dec, year = 2025, publisher = {Zenodo}, version = {1.0}, doi = {10.5281/zenodo.17952596}, url = {https://doi.org/10.5281/zenodo.17952596} } ```