|
Download GUIDE.md from KaleComputer/kale-tts-1-tiny: direct link, hf CLI and curl.
- Browser
- Download file 17.9 kB
-
https://huggingface.co/KaleComputer/kale-tts-1-tiny/resolve/main/GUIDE.md
- Command line
-
hf download hf://KaleComputer/kale-tts-1-tiny/GUIDE.md
-
curl -L -o GUIDE.md https://huggingface.co/KaleComputer/kale-tts-1-tiny/resolve/main/GUIDE.md
17.9 kB
| <!-- DRAFT, NOT PUBLISHED. Nothing here goes public until the go. Built 16 Nov 2026 from our research notes; no product voice, no internal | |
| names, no private data, no secrets. Licences: weights and code Apache-2.0, dataset CC BY 4.0 (final, 16 Nov). --> | |
| # Build your own voice with kale-tts-1 | |
| This is the full recipe behind kale-tts-1-nano (~10 MB, one ONNX file, several voices, runs on a phone-class CPU). Use it to make a small, | |
| fast model in any voice you can describe in one sentence. Every number below was measured on our bench; failures are included where they | |
| cost us time, so you can skip them. | |
| The pipeline: | |
| 1. design a voice in Maya-1 (text description, no recordings), | |
| 2. render ~25 hours of it, | |
| 3. filter hard, | |
| 4. train a teacher and a small sound side, | |
| 5. distil the text side to nano size, | |
| 6. export to ONNX, | |
| 7. run it anywhere onnxruntime runs. | |
| ## 0. What you need | |
| - GPUs for rendering, filtering and training (we used H100s; one voice end to end needs a few GPU-hours, see costs at each step). | |
| - Maya-1 (Apache-2.0) with SNAC 24 kHz (MIT), vLLM, Whisper large-v3-turbo (MIT), a speaker-embedding model for voice checks, Kokoro-82M | |
| (Apache-2.0) as the teacher's starting point, kikiri-tts / StyleTTS2 training code (Apache-2.0 / MIT). | |
| - This folder: `frontend/` (text front end), `python/kale_tts.py`, `js/kale-tts.mjs`, `server/server.mjs`, `runtimes/`. | |
| - `recipe/` (the scripts that built the release; each step's README lists its inputs and outputs; app and volume names in | |
| `recipe/config/modal.json`): `render/` (text pool, voice descriptions in `render/config/voices.json`, Maya-1 render), `filter/`, | |
| `teacher/`, `train/`, `distil/`, `bench/` (incl. `melody_gap.py`), `export/` (training exports -> the release files: one-file graphs, | |
| exact rewrites, end clamp, fp32 web graph, export check; `build_release.sh`). | |
| ## 1. Design the voice (Maya-1) | |
| Write one description in Maya-1's own format: identity, then delivery. Example (Simone, one of the release voices): | |
| > Realistic female voice in the 40s age with a african american accent. Low pitch, deep timbre, conversational pacing, confident tone delivery at medium intensity, conversational domain, narrator role, natural delivery | |
| What we learned designing ~500 candidates: | |
| - **The accent slot sets identity most.** By speaker embedding (ECAPA), two designs with the same accent are +0.28 more alike; the same | |
| pitch adds +0.08; age adds almost nothing. For variety inside one language, use the accent slot (e.g. "southern american", | |
| "new york", "midwestern") and distinct character roles (person + situation + delivery). Light regional hints in free text barely move it. | |
| - **The pacing word sets the pace.** On the same voice: "brisk" ~226 words/min, "conversational" ~201, "moderate" ~200. A "slow pacing" | |
| design stays slow; change the word, don't speed the audio up. Lines made of many short sentences also read slow (~150 wpm): use flowing | |
| clauses in your test line. | |
| - **Check distinctness with ECAPA, not WavLM speaker verification.** WavLM-sv rates most designed voices >= 0.965 alike. With ECAPA, a | |
| take to its own voice's centre is ~0.94; voices listeners heard as different sat at 0.75-0.78; "too similar" at 0.86-0.89. We keep a | |
| new voice below 0.80 to every voice already in the model. | |
| - Render 12-16 takes of a test line per design and keep a design only if at least 2 takes are clean (see step 3) and the pace lands | |
| where you want it (we aim at 160-200 wpm). | |
| - Whisper-style delivery works; laughs and giggles are unreliable (Maya often skips them), so a small model can't learn them cleanly. | |
| - Ages and accents the model can't do convincingly come out generic or put-on. Listen before you train; don't trust an accent classifier. | |
| - Adults only, designed only: never describe or imitate a real person. | |
| ## 2. Render (`recipe/render/`) | |
| - Maya-1 on vLLM, bf16. Sampling: temperature 0.4, top-p 0.9, repetition penalty 1.1, max tokens min(2048, 220 + 60 x words), one seed per | |
| take. Decode SNAC codes to 24 kHz audio. Keep the original audio, no processing. | |
| - Throughput: 46,000 takes (118 h of audio) in ~17 minutes on 4 H100s. | |
| - Text: public-domain Project Gutenberg books (boilerplate removed; `recipe/render/make_text.py`). Mix single sentences and passages of 2-5 consecutive | |
| sentences, by item count 40 / 20 / 20 / 12 / 8 %, each 1-20 s (Maya-1 stops around 25 s). Never glue unrelated sentences together. | |
| - Render more than you keep: ~21k items for a female voice, ~50k for a male voice (male takes fail the ending check more often). | |
| - **Bug to avoid:** with a warm vLLM engine, outputs may not come back in prompt order. Match each output to its job by the prompt tokens, | |
| or takes land on the wrong text and voice. | |
| ## 3. Filter (`recipe/filter/`: `stream_filter.py` + `keep.py`) | |
| Run every check on each chunk as it renders, in the same warm GPU container (Maya, SNAC, Whisper, quality and speaker models loaded once). | |
| A batch pass after the render took us hours; streaming screened ~100 voices end to end in ~5 minutes. | |
| Keep a take only if: | |
| | check | rule | | |
| |---|---| | |
| | complete | Maya's end token is present | | |
| | read-back | Whisper large-v3-turbo reads the text back with no missed and no extra words | | |
| | edges | 20 ms-0.8 s of silence before, at most 0.8 s after; not cut off; no full-scale sample | | |
| | pace | 100-230 spoken words/min (raise the ceiling for a voice whose natural pace is faster) | | |
| | pitch | within 15 % of the voice's median | | |
| | same voice | ECAPA-TDNN cosine >= 0.46 to the voice's own centre (mean of its closest 60 % of takes); our release pass used WavLM-sv at 0.88, replaced because its pretraining data is non-commercial (`recipe/filter/keep.py`) | | |
| | endings | a rough-ending (creak / cut-off) detector on EVERY sentence end, found by forced alignment, < 100 ms rough; pitch floor 70 Hz for male voices, 110 Hz for female | | |
| | passages | one rough inner sentence end drops the whole take | | |
| What we learned: | |
| - **Check every sentence end, not just the last one, and not by pauses.** A pause-based pass let creaky inner endings through. | |
| - **A creak detector tuned on female voices deletes male voices.** A stricter detector flagged nearly every normal low male ending, | |
| including clean single sentences: it dropped 93 % of one male voice's takes (the standard detector with a 70 Hz floor kept 81-90 %). | |
| - Expect 44-63 % of female takes and fewer male takes to survive. | |
| - More hours of a creaky voice don't fix creak; filtering does. | |
| **How many hours** (same voice trained on 5 h, 25 h and all of its kept takes, everything else equal; 192-line bench, 3 seeds): | |
| | voice | 5 h unclear % | 25 h | all | voice's own Maya takes | | |
| |---|---|---|---|---| | |
| | deep-voiced woman (34 h total) | 5.36 | 3.97 | 3.81 | 3.19 | | |
| | calm older man (59 h total) | 5.08 | 4.79 | 4.52 | 4.04 | | |
| - Aim for **~25 hours kept per voice**. 5 h works but is less clear and reads slow; more than 25 h adds 0.2-0.3 points. | |
| - Sound quality is already at its level with 5 h. Melody keeps improving with hours only for lively voices. | |
| - The duration model needs **at least ~1.5 h of takes inside the voice's pace window** (step 4), or the voice reads slow. | |
| ## 4. Train | |
| ### 4a. Teacher text side (`recipe/teacher/`) | |
| - Start from stock Kokoro-82M. Fine-tune it on your voices only with StyleTTS2 training: train your own text aligner on the voices first | |
| (10 epochs, ~30 min), then stage 1 (3 epochs) and stage 2 for **one epoch only** (later stage-2 epochs cost clarity and melody in our runs). | |
| One GPU: multi-GPU stage 1 ran out of memory in every rank for us. | |
| - Loading older checkpoints: weight-norm key names differ between code versions, and a non-strict load silently leaves the pitch branch | |
| random. Rename the keys, fail on any missing tensor, and check that the predicted pitch is in a human range. | |
| - Voice packs (style vectors) come from the teacher's style encoder over ~200 of the voice's own takes. These barely separate voices | |
| (packs are 0.96-0.99 alike), so each voice also gets learned rows and offsets (below). | |
| - **Open:** the symbols the text side is trained on matter for melody. A Kokoro text side trained on Kokoro's own symbol conventions | |
| kept a much more natural melody than the same model trained on espeak-style IPA (melody gap 1.6-1.8 vs 4.5 on our scale, same clarity). | |
| We are retraining our teacher on Kokoro-native symbols; check the release notes for the result before you train. | |
| ### 4b. Sound side (`recipe/train/`) | |
| - A 3.65 M-parameter decoder: AdaIN trunk, a 64-harmonic sine source plus noise, and an iSTFT head straight to 24 kHz (no separate vocoder). | |
| One learned 16-number style row per voice. | |
| - From scratch: 50k steps (25k spectral, 25k adversarial), then continue to ~80k steps (50k -> 80k: unclear 4.71 -> 4.50 %, weakest-tenth | |
| sound 4.16 -> 4.38). Don't restart the learning-rate schedule to go further: it cost clarity at 100k-150k. | |
| - An 8.4 M decoder tied the 3.65 M one; it isn't worth the size. | |
| - Several voices in one model: balance batches (repeat small voices). A new voice added to a trained decoder reaches its own timbre after | |
| ~15 minutes of fine-tuning on one GPU; a new style vector alone keeps the old voice. One voice per model was 0.4-1.0 points clearer than | |
| a shared 8-voice model on the same data: sharing is a trade. | |
| - The decoder always learns from real Maya audio, never from a teacher's renders. | |
| ### 4c. Pitch, loudness and duration heads | |
| - First the pitch/loudness branch plus a learned 128-number offset per voice (3,000 steps, lr 2e-4), then the duration head with the | |
| offsets frozen (4,000 steps, lr 2e-4). Let the duration path see the offset detached: with one shared offset, a voice that needed a large | |
| pitch shift read 19 % too fast. | |
| - **Pace comes from the data.** The duration head learns only from takes inside the voice's natural-pace window (we centre it where the | |
| voice's book-text pace maps to its natural pace on a test line). The model copies the pace of those takes: one window gave 204 wpm, a | |
| slower one 194. | |
| - **Never use a speed multiplier, a per-voice duration scale or stretched durations.** A model is clearest at its own natural pace: | |
| speeding up or slowing down both cost clarity, and models trained on durations scaled x1.175 were less clear (6.3 %) than the same recipe | |
| at the natural pace (5.5 %). | |
| - **Train on whole passages, speak sentence by sentence.** Passage training moved the sentence-to-sentence melody a lot (pitch-step error | |
| 0.85 -> 0.55 semitones). Rendering whole lines at once didn't help. | |
| ### 4d. Creaky sentence ends | |
| In our models creak at sentence ends comes from the predicted pitch dropping below the voicing floor, not from the decoder. A render-time | |
| clamp fixes it without training: hold the last ~0.45 s of voiced pitch at or above max(85 Hz, sentence median - 6 semitones). On two deep | |
| male voices creak went 63 -> 13 % and 76 -> 4 % of sentence ends, with the same clarity. Training the decoder only on clean endings, or a | |
| lower pitch floor, did not help. | |
| ## 5. Distil to nano (`recipe/distil/`) | |
| - **Text side** (~3 M parameters, ALBERT and predictor at about a third of the width): trained on text only to copy the teacher's per-sound | |
| durations, pitch and loudness for every voice. Because the teacher is deterministic per voice, copying it doesn't flatten the melody. | |
| 98 % of sounds land within one frame of the teacher after ~2,000 steps. | |
| - Teach the student's timing from each voice's pace head (4c), not the teacher's raw timing: two voices went from 8.1 to 5.2 % and from | |
| 11.1 to 9.4 % unclear, and each voice kept its own pace. | |
| - **Sound side** (~4.1 M including its own small text encoder): trained on the real audio, to ~80k steps. A voice with only ~2 h of data | |
| got rougher past ~70k steps. | |
| - Distilled checkpoints swing in quality while the loss falls smoothly; pick by the bench, not by the last step. | |
| - Result: 7.16 M parameters, 10.1 MB (int8 decoder, fp16 text side). | |
| ## 6. Export (`recipe/export/`) | |
| - One ONNX graph (opset 17, batch 1) with a voice id input. Keep the sine source's phase path in float32 (some web builds can't cast to | |
| float64) and store 2πk in float32: an fp16 constant put the top harmonics up to ~0.5 rad off. | |
| - **Profile before optimising.** In nano the decoder is ~80 % of the time; in the 61 MB model the text side is ~58 %. | |
| - **Two exact rewrites** (same maths, no retraining): the 64-harmonic sine sum in closed form, sin(Kt/2) sin((K+1)t/2) / sin(t/2), and the | |
| six short-time Fourier transforms as one batched factorised DFT (600 = 25 x 24). On a Galaxy Tab S10 Lite (4 threads) nano went from | |
| 30x to 42x real time; the 8-voice bench didn't move (7.88 -> 7.85 % unclear). | |
| - On nano, running 3 of the student ALBERT's 6 passes (no retraining) gave 46x real time and 0.10 s to the first sentence, with the bench | |
| unchanged or slightly better (7.88 -> 7.62 %). On the 61 MB model, cutting passes without retraining broke clarity and melody. | |
| - int8: an int8 decoder is the default file. Plain per-row int8 of a text side held every measure, but on onnxruntime 1.32 set | |
| `session.disable_quant_qdq=1` or you lose the speed gain. Dynamic int8 on ALBERT/LSTMs didn't help. | |
| - Splitting the first sentence at a comma halves the time to first audio but breaks the melody (the first half ends like a sentence). | |
| ## 7. Run it | |
| - Python: `python/kale_tts.py` (onnxruntime + numpy, front end in-process). Node and browser: `js/kale-tts.mjs`. HTTP: | |
| `server/server.mjs` (OpenAI-style `POST /v1/audio/speech`, WAV or streamed PCM). More engines and serving stacks: `runtimes/RUNTIMES.md`. | |
| - CPU is the right target. One server core runs ~30 nano streams in real time. GPUs don't speed up a batch-1 graph this small (a T4 beat an | |
| H100). vLLM and SGLang are built for autoregressive models and don't fit. | |
| - Browser WebGPU: keep the six small LSTM layers on the CPU (`forceCpuNodeNames`; about 2x faster for nano), use nano's fp32 graph | |
| (tiny: its int8 graph). `js/kale-tts.mjs` does both by default. | |
| - Small serverless functions: set onnxruntime's intra-op threads to 1 on a 1-vCPU function (the default halves speed) and ship only your | |
| platform's onnxruntime binaries. | |
| - Mac: plain onnxruntime on the CPU (4 threads) beat CoreML by 2-4x. | |
| Measured speed of the 8-voice nano file (16 Nov 2026): | |
| | where | x real time | first audio | | |
| |---|---|---| | |
| | Galaxy Tab S10 Lite, 4 threads, release graph | 29-30x | 0.15-0.18 s | | |
| | same, with the rewrites and 3 ALBERT passes (step 6) | 46x | 0.10 s | | |
| | Linux server CPU, 4 threads | 37x (int8) / 46x (fp32) | 0.09 s | | |
| | Mac laptop CPU (Apple M5 Max), 4 threads | 178x | 0.017 s | | |
| A phone-sized browser window on a laptop is not a phone: one real phone ran an older model of ours ~3x slower than our laptop | |
| phone-sized harness. Use real devices for phone numbers. | |
| ## 8. Measure | |
| `recipe/bench/`: 192 lines, rendered sentence by sentence as an app would, 3 noise seeds (one render alone can be off by ~0.5 | |
| points), one loudness gain per model with zero clipped samples. | |
| - **Unclear words**: each word's sounds checked against Maya's own takes of the same line. The score depends on the reference takes, so | |
| score each voice against its own Maya takes and print Maya's own score next to it. On easier references a model looked 2.4 points | |
| clearer than it was. | |
| - **Weakest tenth**: mean UTMOS of the worst 10 % of lines. UTMOS likes large smooth decoders, sometimes more than the original audio. | |
| - **Melody gap to Maya** (`melody_gap.py`): 0 = the voice's own Maya takes, lower is better; an averaged, flat melody can't score well. | |
| We dropped "pitch-step error" as the headline: a flat average contour scored better than Maya herself. | |
| - **Creak**: share of sentence ends with >= 100 ms of rough sound; compare with the voice's own Maya takes (some deep voices creak in the | |
| source too). | |
| - **Pace**: words / full clip seconds of one fixed test line at speed 1.0. | |
| - Listen last, every time. Several of our models passed every metric and still sounded wrong. | |
| 8-voice nano (one 10 MB file), each voice against its own Maya takes, 3 seeds: unclear words 5.5-11.1 % in the first build and 4.4-9.4 % | |
| after the timing fix of step 5 (the voices' own Maya takes: 3.2-6.3 %), weakest tenth 4.13-4.44, 173-195 wpm. Per-voice tables: MODEL_CARD.md. | |
| ## 9. What cost us the most time | |
| - Rendering and filtering in batches instead of streaming; waiting for every voice before training. | |
| - A creak detector that deleted male voices; checking only the last sentence end. | |
| - Speed multipliers to hit a pace (the pace has to come from the data). | |
| - Reading a melody number that rewards flat speech; comparing clarity on different reference sets. | |
| - An American pronunciation front end on British voices (19.8 % unclear; a British front end plus a short fine-tune brought two British | |
| voices from 8.4 to 7.7 % and 9.6 to 8.5 %). | |
| - Many melody fixes on the pitch predictor (plain L1 fine-tunes, discriminators, contour codes, label heads, sampled pitch): none moved the | |
| melody gap the way the text side's symbol set did. | |
| ## 10. Costs we measured | |
| - Data for 8 voices (render + filter + strict checks): ~$200 on rented GPUs, most of it the filter. Whole night including training and | |
| benches for both model sizes: ~$290. | |
| - A two-voice run of a smaller recipe (render, filter, train one 21 MB model): ~$37. | |
| ## 11. Licences and credits | |
| - Based on Maya (Maya-1, Maya Research, Apache-2.0); every training clip is synthetic, made with Maya-1 from a text description. | |
| - Text side derived from Kokoro-82M (Apache-2.0, StyleTTS2 lineage, MIT). Front end: CMUdict (BSD-2 style notice), Britfone (MIT) for | |
| British English, number/date rules adapted from dizione (Apache-2.0); no espeak, no GPL. | |
| - Our weights and code: Apache-2.0. Our dataset: CC BY 4.0. | |
| - All voices are synthetic. Don't use them to pass speech off as a real person's or as human-recorded. | |