kale-tts-1-base / README.md
NERDDISCO's picture
kale-tts-1
98f4baa
|
Raw History Blame Contribute Delete
5.75 kB
---
# DRAFT, NOT PUBLISHED. Credit: based on Maya (Maya-1).
license: apache-2.0
language: [en]
pipeline_tag: text-to-speech
library_name: kokoro
tags: [text-to-speech, tts, kokoro, synthetic-voices, based-on-maya]
base_model: hexgrad/Kokoro-82M
base_model_relation: finetune
datasets: [KaleComputer/kale-tts-1-data]
---
![kale-tts-1-base](diagrams/header-base.png)
# kale-tts-1-base
kale-tts-1-base is [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) with its text side and its timing-and-melody planner
fine-tuned on our 10 voices designed with [Maya-1](https://huggingface.co/maya-research/maya1); its sound decoder is stock Kokoro-82M.
It's the teacher of [kale-tts-1-tiny](https://huggingface.co/KaleComputer/kale-tts-1-tiny) and
[kale-tts-1-nano](https://huggingface.co/KaleComputer/kale-tts-1-nano): both learned their timing and melody from it and have their own small
sound generators, so they run on a low-end tablet. kale-tts-1-base itself is too big and slow for that; use it on a desktop or GPU, or to
train and distill your own small voices.
## Distilled into
| model | parameters | CPU file (int8) | WebGPU file (float32) | Galaxy Tab S10 Lite | unclear words vs Maya-1 |
|---|---|---|---|---|---|
| [kale-tts-1-tiny](https://huggingface.co/KaleComputer/kale-tts-1-tiny) | 45.7M | 79.9 MB | 183.2 MB | 24x real time | +1.4 points |
| [kale-tts-1-nano](https://huggingface.co/KaleComputer/kale-tts-1-nano) | 7.5M | 14.8 MB | 30.6 MB | 44x real time | +1.6 points |
tiny keeps this model's timing-and-melody planner (frozen) and adds the Ettin word reader; nano is a smaller student distilled from it.
Both have their own small sound generator and run on a low-end tablet.
| measure | value |
|---|---|
| parameters | 81.8M (28.5M fine-tuned on our voices: text side + planner; the rest is stock Kokoro-82M's sound decoder) |
| size | 327 MB (`kale-tts-1-base.pth`, PyTorch) |
| voices | 10, as voice packs in `voices/` |
| trained on | [kale-tts-1-data](https://huggingface.co/datasets/KaleComputer/kale-tts-1-data): 1,300 takes per voice, 26.9 hours |
| clarity and melody (our release bench, 10 voices, 18 Nov) | unclear words +3.9 points vs Maya-1's own recordings (Maya-1: 5.82 %; tiny +1.4, nano +1.6); melody gap 12.2 (tiny 7.48, nano 7.46). Not built for the tablet |
## Voices
| name | voice | pronunciation |
|---|---|---|
| Kenzie | esports hype woman | US English |
| Simone | deep and confident woman | US English |
| Amy | crisp news-style woman | British English |
| Frankie | hard-boiled detective's partner | US English |
| Darlene | rough low-voiced old trucker woman | US English |
| Owen | bright young host | US English |
| Toby | mischievous prankster | US English |
| Jack | crisp news-style man | British English |
| Earl | African American jazz granddad | US English |
| Magnus | deep dramatic trailer narrator | US English |
Every voice was designed from a one-line text description; none is a real person.
## Use it
```bash
pip install kokoro soundfile
pip install -U huggingface_hub
hf download KaleComputer/kale-tts-1-base --local-dir kale-tts-1-base
cd kale-tts-1-base
```
Save as `speak.py` in that folder, then run `python speak.py`:
```python
import json, torch, soundfile as sf
from kokoro import KModel, KPipeline
model = KModel(config="config.json", model="kale-tts-1-base.pth").eval()
voices = {v["name"]: v for v in json.load(open("voices.json"))}
# American voices use lang_code "a", the two British ones (Amy, Jack) "b"
v = voices["Kenzie"]
pipe = KPipeline(lang_code=v["lang_code"], model=model)
pack = torch.load(v["file"], weights_only=True)
audio = torch.cat([a for _, _, a in pipe("Hello there! A volcano is a place where very hot, melted rock comes up out of the ground.", voice=pack)])
sf.write("hello.wav", audio.numpy(), 24000)
```
`KPipeline` turns text into sounds with Kokoro's own text front end (misaki). Our checked dictionary and text rules are in the
kale-tts-1-tiny and kale-tts-1-nano repos (`frontend/`).
## How it was made
![How we picked the base model, at selection](diagrams/birth.png)
1. **Maya-1** designs each of the 10 voices from a voice prompt and reads thousands of sentences with it; a filter keeps the clean takes
([kale-tts-1-data](https://huggingface.co/datasets/KaleComputer/kale-tts-1-data)).
2. We trained our own aligner on 1,300 takes per voice (26.9 hours, 30 epochs on an NVIDIA L40S), then fine-tuned Kokoro-82M on them
in two stages on an NVIDIA H100 (three epochs, then one more with the adversarial stage).
3. The voice packs come from the fine-tuned style encoder, one per voice.
4. kale-tts-1-tiny keeps this planner (frozen) and adds the Ettin word reader; kale-tts-1-nano is a smaller student distilled from it.
Both learn their sound straight from Maya-1's recordings.
## Limits
- English only. The two British voices read best with `lang_code="b"`.
- Too big and slow for a low-end tablet: that's what kale-tts-1-tiny and kale-tts-1-nano are for.
- Not for impersonating real people or passing the output off as human speech.
## Licence and credits
Apache-2.0.
AI-generated speech: everything this model says is synthetic speech made by an AI model, in voices that belong to no real person. Tell
listeners it is AI-generated, never present it as a human recording, and never use it to imitate a real person.
Credits: based on Maya-1: all training audio is original speech from Maya-1 (Maya Research, Apache-2.0), decoded with SNAC (MIT). Fine-tuned
from Kokoro-82M (hexgrad, Apache-2.0), whose sound decoder this model keeps unchanged, with StyleTTS2 (MIT) and kikiri-tts (Apache-2.0)
code. Training texts: public-domain books made available by Project Gutenberg. Dataset: kale-tts-1-data (CC BY 4.0).