Download GUIDE.md from KaleComputer/kale-tts-1-tiny: direct link, hf CLI and curl.
- Browser
- Download file 17.9 kB
-
https://huggingface.co/KaleComputer/kale-tts-1-tiny/resolve/main/GUIDE.md
- Command line
-
hf download hf://KaleComputer/kale-tts-1-tiny/GUIDE.md
-
curl -L -o GUIDE.md https://huggingface.co/KaleComputer/kale-tts-1-tiny/resolve/main/GUIDE.md
Build your own voice with kale-tts-1
This is the full recipe behind kale-tts-1-nano (~10 MB, one ONNX file, several voices, runs on a phone-class CPU). Use it to make a small, fast model in any voice you can describe in one sentence. Every number below was measured on our bench; failures are included where they cost us time, so you can skip them.
The pipeline:
- design a voice in Maya-1 (text description, no recordings),
- render ~25 hours of it,
- filter hard,
- train a teacher and a small sound side,
- distil the text side to nano size,
- export to ONNX,
- run it anywhere onnxruntime runs.
0. What you need
- GPUs for rendering, filtering and training (we used H100s; one voice end to end needs a few GPU-hours, see costs at each step).
- Maya-1 (Apache-2.0) with SNAC 24 kHz (MIT), vLLM, Whisper large-v3-turbo (MIT), a speaker-embedding model for voice checks, Kokoro-82M (Apache-2.0) as the teacher's starting point, kikiri-tts / StyleTTS2 training code (Apache-2.0 / MIT).
- This folder:
frontend/(text front end),python/kale_tts.py,js/kale-tts.mjs,server/server.mjs,runtimes/. recipe/(the scripts that built the release; each step's README lists its inputs and outputs; app and volume names inrecipe/config/modal.json):render/(text pool, voice descriptions inrender/config/voices.json, Maya-1 render),filter/,teacher/,train/,distil/,bench/(incl.melody_gap.py),export/(training exports -> the release files: one-file graphs, exact rewrites, end clamp, fp32 web graph, export check;build_release.sh).
1. Design the voice (Maya-1)
Write one description in Maya-1's own format: identity, then delivery. Example (Simone, one of the release voices):
Realistic female voice in the 40s age with a african american accent. Low pitch, deep timbre, conversational pacing, confident tone delivery at medium intensity, conversational domain, narrator role, natural delivery
What we learned designing ~500 candidates:
- The accent slot sets identity most. By speaker embedding (ECAPA), two designs with the same accent are +0.28 more alike; the same pitch adds +0.08; age adds almost nothing. For variety inside one language, use the accent slot (e.g. "southern american", "new york", "midwestern") and distinct character roles (person + situation + delivery). Light regional hints in free text barely move it.
- The pacing word sets the pace. On the same voice: "brisk"
226 words/min, "conversational" ~201, "moderate" ~200. A "slow pacing" design stays slow; change the word, don't speed the audio up. Lines made of many short sentences also read slow (150 wpm): use flowing clauses in your test line. - Check distinctness with ECAPA, not WavLM speaker verification. WavLM-sv rates most designed voices >= 0.965 alike. With ECAPA, a take to its own voice's centre is ~0.94; voices listeners heard as different sat at 0.75-0.78; "too similar" at 0.86-0.89. We keep a new voice below 0.80 to every voice already in the model.
- Render 12-16 takes of a test line per design and keep a design only if at least 2 takes are clean (see step 3) and the pace lands where you want it (we aim at 160-200 wpm).
- Whisper-style delivery works; laughs and giggles are unreliable (Maya often skips them), so a small model can't learn them cleanly.
- Ages and accents the model can't do convincingly come out generic or put-on. Listen before you train; don't trust an accent classifier.
- Adults only, designed only: never describe or imitate a real person.
2. Render (recipe/render/)
- Maya-1 on vLLM, bf16. Sampling: temperature 0.4, top-p 0.9, repetition penalty 1.1, max tokens min(2048, 220 + 60 x words), one seed per take. Decode SNAC codes to 24 kHz audio. Keep the original audio, no processing.
- Throughput: 46,000 takes (118 h of audio) in ~17 minutes on 4 H100s.
- Text: public-domain Project Gutenberg books (boilerplate removed;
recipe/render/make_text.py). Mix single sentences and passages of 2-5 consecutive sentences, by item count 40 / 20 / 20 / 12 / 8 %, each 1-20 s (Maya-1 stops around 25 s). Never glue unrelated sentences together. - Render more than you keep: ~21k items for a female voice, ~50k for a male voice (male takes fail the ending check more often).
- Bug to avoid: with a warm vLLM engine, outputs may not come back in prompt order. Match each output to its job by the prompt tokens, or takes land on the wrong text and voice.
3. Filter (recipe/filter/: stream_filter.py + keep.py)
Run every check on each chunk as it renders, in the same warm GPU container (Maya, SNAC, Whisper, quality and speaker models loaded once). A batch pass after the render took us hours; streaming screened ~100 voices end to end in ~5 minutes.
Keep a take only if:
| check | rule |
|---|---|
| complete | Maya's end token is present |
| read-back | Whisper large-v3-turbo reads the text back with no missed and no extra words |
| edges | 20 ms-0.8 s of silence before, at most 0.8 s after; not cut off; no full-scale sample |
| pace | 100-230 spoken words/min (raise the ceiling for a voice whose natural pace is faster) |
| pitch | within 15 % of the voice's median |
| same voice | ECAPA-TDNN cosine >= 0.46 to the voice's own centre (mean of its closest 60 % of takes); our release pass used WavLM-sv at 0.88, replaced because its pretraining data is non-commercial (recipe/filter/keep.py) |
| endings | a rough-ending (creak / cut-off) detector on EVERY sentence end, found by forced alignment, < 100 ms rough; pitch floor 70 Hz for male voices, 110 Hz for female |
| passages | one rough inner sentence end drops the whole take |
What we learned:
- Check every sentence end, not just the last one, and not by pauses. A pause-based pass let creaky inner endings through.
- A creak detector tuned on female voices deletes male voices. A stricter detector flagged nearly every normal low male ending, including clean single sentences: it dropped 93 % of one male voice's takes (the standard detector with a 70 Hz floor kept 81-90 %).
- Expect 44-63 % of female takes and fewer male takes to survive.
- More hours of a creaky voice don't fix creak; filtering does.
How many hours (same voice trained on 5 h, 25 h and all of its kept takes, everything else equal; 192-line bench, 3 seeds):
| voice | 5 h unclear % | 25 h | all | voice's own Maya takes |
|---|---|---|---|---|
| deep-voiced woman (34 h total) | 5.36 | 3.97 | 3.81 | 3.19 |
| calm older man (59 h total) | 5.08 | 4.79 | 4.52 | 4.04 |
- Aim for ~25 hours kept per voice. 5 h works but is less clear and reads slow; more than 25 h adds 0.2-0.3 points.
- Sound quality is already at its level with 5 h. Melody keeps improving with hours only for lively voices.
- The duration model needs at least ~1.5 h of takes inside the voice's pace window (step 4), or the voice reads slow.
4. Train
4a. Teacher text side (recipe/teacher/)
- Start from stock Kokoro-82M. Fine-tune it on your voices only with StyleTTS2 training: train your own text aligner on the voices first (10 epochs, ~30 min), then stage 1 (3 epochs) and stage 2 for one epoch only (later stage-2 epochs cost clarity and melody in our runs). One GPU: multi-GPU stage 1 ran out of memory in every rank for us.
- Loading older checkpoints: weight-norm key names differ between code versions, and a non-strict load silently leaves the pitch branch random. Rename the keys, fail on any missing tensor, and check that the predicted pitch is in a human range.
- Voice packs (style vectors) come from the teacher's style encoder over ~200 of the voice's own takes. These barely separate voices (packs are 0.96-0.99 alike), so each voice also gets learned rows and offsets (below).
- Open: the symbols the text side is trained on matter for melody. A Kokoro text side trained on Kokoro's own symbol conventions kept a much more natural melody than the same model trained on espeak-style IPA (melody gap 1.6-1.8 vs 4.5 on our scale, same clarity). We are retraining our teacher on Kokoro-native symbols; check the release notes for the result before you train.
4b. Sound side (recipe/train/)
- A 3.65 M-parameter decoder: AdaIN trunk, a 64-harmonic sine source plus noise, and an iSTFT head straight to 24 kHz (no separate vocoder). One learned 16-number style row per voice.
- From scratch: 50k steps (25k spectral, 25k adversarial), then continue to ~80k steps (50k -> 80k: unclear 4.71 -> 4.50 %, weakest-tenth sound 4.16 -> 4.38). Don't restart the learning-rate schedule to go further: it cost clarity at 100k-150k.
- An 8.4 M decoder tied the 3.65 M one; it isn't worth the size.
- Several voices in one model: balance batches (repeat small voices). A new voice added to a trained decoder reaches its own timbre after ~15 minutes of fine-tuning on one GPU; a new style vector alone keeps the old voice. One voice per model was 0.4-1.0 points clearer than a shared 8-voice model on the same data: sharing is a trade.
- The decoder always learns from real Maya audio, never from a teacher's renders.
4c. Pitch, loudness and duration heads
- First the pitch/loudness branch plus a learned 128-number offset per voice (3,000 steps, lr 2e-4), then the duration head with the offsets frozen (4,000 steps, lr 2e-4). Let the duration path see the offset detached: with one shared offset, a voice that needed a large pitch shift read 19 % too fast.
- Pace comes from the data. The duration head learns only from takes inside the voice's natural-pace window (we centre it where the voice's book-text pace maps to its natural pace on a test line). The model copies the pace of those takes: one window gave 204 wpm, a slower one 194.
- Never use a speed multiplier, a per-voice duration scale or stretched durations. A model is clearest at its own natural pace: speeding up or slowing down both cost clarity, and models trained on durations scaled x1.175 were less clear (6.3 %) than the same recipe at the natural pace (5.5 %).
- Train on whole passages, speak sentence by sentence. Passage training moved the sentence-to-sentence melody a lot (pitch-step error 0.85 -> 0.55 semitones). Rendering whole lines at once didn't help.
4d. Creaky sentence ends
In our models creak at sentence ends comes from the predicted pitch dropping below the voicing floor, not from the decoder. A render-time clamp fixes it without training: hold the last ~0.45 s of voiced pitch at or above max(85 Hz, sentence median - 6 semitones). On two deep male voices creak went 63 -> 13 % and 76 -> 4 % of sentence ends, with the same clarity. Training the decoder only on clean endings, or a lower pitch floor, did not help.
5. Distil to nano (recipe/distil/)
- Text side (~3 M parameters, ALBERT and predictor at about a third of the width): trained on text only to copy the teacher's per-sound durations, pitch and loudness for every voice. Because the teacher is deterministic per voice, copying it doesn't flatten the melody. 98 % of sounds land within one frame of the teacher after ~2,000 steps.
- Teach the student's timing from each voice's pace head (4c), not the teacher's raw timing: two voices went from 8.1 to 5.2 % and from 11.1 to 9.4 % unclear, and each voice kept its own pace.
- Sound side (~4.1 M including its own small text encoder): trained on the real audio, to ~80k steps. A voice with only ~2 h of data got rougher past ~70k steps.
- Distilled checkpoints swing in quality while the loss falls smoothly; pick by the bench, not by the last step.
- Result: 7.16 M parameters, 10.1 MB (int8 decoder, fp16 text side).
6. Export (recipe/export/)
- One ONNX graph (opset 17, batch 1) with a voice id input. Keep the sine source's phase path in float32 (some web builds can't cast to float64) and store 2πk in float32: an fp16 constant put the top harmonics up to ~0.5 rad off.
- Profile before optimising. In nano the decoder is ~80 % of the time; in the 61 MB model the text side is ~58 %.
- Two exact rewrites (same maths, no retraining): the 64-harmonic sine sum in closed form, sin(Kt/2) sin((K+1)t/2) / sin(t/2), and the six short-time Fourier transforms as one batched factorised DFT (600 = 25 x 24). On a Galaxy Tab S10 Lite (4 threads) nano went from 30x to 42x real time; the 8-voice bench didn't move (7.88 -> 7.85 % unclear).
- On nano, running 3 of the student ALBERT's 6 passes (no retraining) gave 46x real time and 0.10 s to the first sentence, with the bench unchanged or slightly better (7.88 -> 7.62 %). On the 61 MB model, cutting passes without retraining broke clarity and melody.
- int8: an int8 decoder is the default file. Plain per-row int8 of a text side held every measure, but on onnxruntime 1.32 set
session.disable_quant_qdq=1or you lose the speed gain. Dynamic int8 on ALBERT/LSTMs didn't help. - Splitting the first sentence at a comma halves the time to first audio but breaks the melody (the first half ends like a sentence).
7. Run it
- Python:
python/kale_tts.py(onnxruntime + numpy, front end in-process). Node and browser:js/kale-tts.mjs. HTTP:server/server.mjs(OpenAI-stylePOST /v1/audio/speech, WAV or streamed PCM). More engines and serving stacks:runtimes/RUNTIMES.md. - CPU is the right target. One server core runs ~30 nano streams in real time. GPUs don't speed up a batch-1 graph this small (a T4 beat an H100). vLLM and SGLang are built for autoregressive models and don't fit.
- Browser WebGPU: keep the six small LSTM layers on the CPU (
forceCpuNodeNames; about 2x faster for nano), use nano's fp32 graph (tiny: its int8 graph).js/kale-tts.mjsdoes both by default. - Small serverless functions: set onnxruntime's intra-op threads to 1 on a 1-vCPU function (the default halves speed) and ship only your platform's onnxruntime binaries.
- Mac: plain onnxruntime on the CPU (4 threads) beat CoreML by 2-4x.
Measured speed of the 8-voice nano file (16 Nov 2026):
| where | x real time | first audio |
|---|---|---|
| Galaxy Tab S10 Lite, 4 threads, release graph | 29-30x | 0.15-0.18 s |
| same, with the rewrites and 3 ALBERT passes (step 6) | 46x | 0.10 s |
| Linux server CPU, 4 threads | 37x (int8) / 46x (fp32) | 0.09 s |
| Mac laptop CPU (Apple M5 Max), 4 threads | 178x | 0.017 s |
| A phone-sized browser window on a laptop is not a phone: one real phone ran an older model of ours ~3x slower than our laptop | ||
| phone-sized harness. Use real devices for phone numbers. |
8. Measure
recipe/bench/: 192 lines, rendered sentence by sentence as an app would, 3 noise seeds (one render alone can be off by ~0.5
points), one loudness gain per model with zero clipped samples.
- Unclear words: each word's sounds checked against Maya's own takes of the same line. The score depends on the reference takes, so score each voice against its own Maya takes and print Maya's own score next to it. On easier references a model looked 2.4 points clearer than it was.
- Weakest tenth: mean UTMOS of the worst 10 % of lines. UTMOS likes large smooth decoders, sometimes more than the original audio.
- Melody gap to Maya (
melody_gap.py): 0 = the voice's own Maya takes, lower is better; an averaged, flat melody can't score well. We dropped "pitch-step error" as the headline: a flat average contour scored better than Maya herself. - Creak: share of sentence ends with >= 100 ms of rough sound; compare with the voice's own Maya takes (some deep voices creak in the source too).
- Pace: words / full clip seconds of one fixed test line at speed 1.0.
- Listen last, every time. Several of our models passed every metric and still sounded wrong.
8-voice nano (one 10 MB file), each voice against its own Maya takes, 3 seeds: unclear words 5.5-11.1 % in the first build and 4.4-9.4 % after the timing fix of step 5 (the voices' own Maya takes: 3.2-6.3 %), weakest tenth 4.13-4.44, 173-195 wpm. Per-voice tables: MODEL_CARD.md.
9. What cost us the most time
- Rendering and filtering in batches instead of streaming; waiting for every voice before training.
- A creak detector that deleted male voices; checking only the last sentence end.
- Speed multipliers to hit a pace (the pace has to come from the data).
- Reading a melody number that rewards flat speech; comparing clarity on different reference sets.
- An American pronunciation front end on British voices (19.8 % unclear; a British front end plus a short fine-tune brought two British voices from 8.4 to 7.7 % and 9.6 to 8.5 %).
- Many melody fixes on the pitch predictor (plain L1 fine-tunes, discriminators, contour codes, label heads, sampled pitch): none moved the melody gap the way the text side's symbol set did.
10. Costs we measured
- Data for 8 voices (render + filter + strict checks): ~$200 on rented GPUs, most of it the filter. Whole night including training and benches for both model sizes: ~$290.
- A two-voice run of a smaller recipe (render, filter, train one 21 MB model): ~$37.
11. Licences and credits
- Based on Maya (Maya-1, Maya Research, Apache-2.0); every training clip is synthetic, made with Maya-1 from a text description.
- Text side derived from Kokoro-82M (Apache-2.0, StyleTTS2 lineage, MIT). Front end: CMUdict (BSD-2 style notice), Britfone (MIT) for British English, number/date rules adapted from dizione (Apache-2.0); no espeak, no GPL.
- Our weights and code: Apache-2.0. Our dataset: CC BY 4.0.
- All voices are synthetic. Don't use them to pass speech off as a real person's or as human-recorded.