File size: 17,885 Bytes
f21d6eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
<!-- DRAFT, NOT PUBLISHED. Nothing here goes public until the go. Built 16 Nov 2026 from our research notes; no product voice, no internal
     names, no private data, no secrets. Licences: weights and code Apache-2.0, dataset CC BY 4.0 (final, 16 Nov). -->

# Build your own voice with kale-tts-1

This is the full recipe behind kale-tts-1-nano (~10 MB, one ONNX file, several voices, runs on a phone-class CPU). Use it to make a small,
fast model in any voice you can describe in one sentence. Every number below was measured on our bench; failures are included where they
cost us time, so you can skip them.

The pipeline:
1. design a voice in Maya-1 (text description, no recordings),
2. render ~25 hours of it,
3. filter hard,
4. train a teacher and a small sound side,
5. distil the text side to nano size,
6. export to ONNX,
7. run it anywhere onnxruntime runs.

## 0. What you need
- GPUs for rendering, filtering and training (we used H100s; one voice end to end needs a few GPU-hours, see costs at each step).
- Maya-1 (Apache-2.0) with SNAC 24 kHz (MIT), vLLM, Whisper large-v3-turbo (MIT), a speaker-embedding model for voice checks, Kokoro-82M
  (Apache-2.0) as the teacher's starting point, kikiri-tts / StyleTTS2 training code (Apache-2.0 / MIT).
- This folder: `frontend/` (text front end), `python/kale_tts.py`, `js/kale-tts.mjs`, `server/server.mjs`, `runtimes/`.
- `recipe/` (the scripts that built the release; each step's README lists its inputs and outputs; app and volume names in
  `recipe/config/modal.json`): `render/` (text pool, voice descriptions in `render/config/voices.json`, Maya-1 render), `filter/`,
  `teacher/`, `train/`, `distil/`, `bench/` (incl. `melody_gap.py`), `export/` (training exports -> the release files: one-file graphs,
  exact rewrites, end clamp, fp32 web graph, export check; `build_release.sh`).

## 1. Design the voice (Maya-1)
Write one description in Maya-1's own format: identity, then delivery. Example (Simone, one of the release voices):
> Realistic female voice in the 40s age with a african american accent. Low pitch, deep timbre, conversational pacing, confident tone delivery at medium intensity, conversational domain, narrator role, natural delivery

What we learned designing ~500 candidates:
- **The accent slot sets identity most.** By speaker embedding (ECAPA), two designs with the same accent are +0.28 more alike; the same
  pitch adds +0.08; age adds almost nothing. For variety inside one language, use the accent slot (e.g. "southern american",
  "new york", "midwestern") and distinct character roles (person + situation + delivery). Light regional hints in free text barely move it.
- **The pacing word sets the pace.** On the same voice: "brisk" ~226 words/min, "conversational" ~201, "moderate" ~200. A "slow pacing"
  design stays slow; change the word, don't speed the audio up. Lines made of many short sentences also read slow (~150 wpm): use flowing
  clauses in your test line.
- **Check distinctness with ECAPA, not WavLM speaker verification.** WavLM-sv rates most designed voices >= 0.965 alike. With ECAPA, a
  take to its own voice's centre is ~0.94; voices listeners heard as different sat at 0.75-0.78; "too similar" at 0.86-0.89. We keep a
  new voice below 0.80 to every voice already in the model.
- Render 12-16 takes of a test line per design and keep a design only if at least 2 takes are clean (see step 3) and the pace lands
  where you want it (we aim at 160-200 wpm).
- Whisper-style delivery works; laughs and giggles are unreliable (Maya often skips them), so a small model can't learn them cleanly.
- Ages and accents the model can't do convincingly come out generic or put-on. Listen before you train; don't trust an accent classifier.
- Adults only, designed only: never describe or imitate a real person.

## 2. Render (`recipe/render/`)
- Maya-1 on vLLM, bf16. Sampling: temperature 0.4, top-p 0.9, repetition penalty 1.1, max tokens min(2048, 220 + 60 x words), one seed per
  take. Decode SNAC codes to 24 kHz audio. Keep the original audio, no processing.
- Throughput: 46,000 takes (118 h of audio) in ~17 minutes on 4 H100s.
- Text: public-domain Project Gutenberg books (boilerplate removed; `recipe/render/make_text.py`). Mix single sentences and passages of 2-5 consecutive
  sentences, by item count 40 / 20 / 20 / 12 / 8 %, each 1-20 s (Maya-1 stops around 25 s). Never glue unrelated sentences together.
- Render more than you keep: ~21k items for a female voice, ~50k for a male voice (male takes fail the ending check more often).
- **Bug to avoid:** with a warm vLLM engine, outputs may not come back in prompt order. Match each output to its job by the prompt tokens,
  or takes land on the wrong text and voice.

## 3. Filter (`recipe/filter/`: `stream_filter.py` + `keep.py`)
Run every check on each chunk as it renders, in the same warm GPU container (Maya, SNAC, Whisper, quality and speaker models loaded once).
A batch pass after the render took us hours; streaming screened ~100 voices end to end in ~5 minutes.

Keep a take only if:
| check | rule |
|---|---|
| complete | Maya's end token is present |
| read-back | Whisper large-v3-turbo reads the text back with no missed and no extra words |
| edges | 20 ms-0.8 s of silence before, at most 0.8 s after; not cut off; no full-scale sample |
| pace | 100-230 spoken words/min (raise the ceiling for a voice whose natural pace is faster) |
| pitch | within 15 % of the voice's median |
| same voice | ECAPA-TDNN cosine >= 0.46 to the voice's own centre (mean of its closest 60 % of takes); our release pass used WavLM-sv at 0.88, replaced because its pretraining data is non-commercial (`recipe/filter/keep.py`) |
| endings | a rough-ending (creak / cut-off) detector on EVERY sentence end, found by forced alignment, < 100 ms rough; pitch floor 70 Hz for male voices, 110 Hz for female |
| passages | one rough inner sentence end drops the whole take |

What we learned:
- **Check every sentence end, not just the last one, and not by pauses.** A pause-based pass let creaky inner endings through.
- **A creak detector tuned on female voices deletes male voices.** A stricter detector flagged nearly every normal low male ending,
  including clean single sentences: it dropped 93 % of one male voice's takes (the standard detector with a 70 Hz floor kept 81-90 %).
- Expect 44-63 % of female takes and fewer male takes to survive.
- More hours of a creaky voice don't fix creak; filtering does.

**How many hours** (same voice trained on 5 h, 25 h and all of its kept takes, everything else equal; 192-line bench, 3 seeds):
| voice | 5 h unclear % | 25 h | all | voice's own Maya takes |
|---|---|---|---|---|
| deep-voiced woman (34 h total) | 5.36 | 3.97 | 3.81 | 3.19 |
| calm older man (59 h total) | 5.08 | 4.79 | 4.52 | 4.04 |
- Aim for **~25 hours kept per voice**. 5 h works but is less clear and reads slow; more than 25 h adds 0.2-0.3 points.
- Sound quality is already at its level with 5 h. Melody keeps improving with hours only for lively voices.
- The duration model needs **at least ~1.5 h of takes inside the voice's pace window** (step 4), or the voice reads slow.

## 4. Train
### 4a. Teacher text side (`recipe/teacher/`)
- Start from stock Kokoro-82M. Fine-tune it on your voices only with StyleTTS2 training: train your own text aligner on the voices first
  (10 epochs, ~30 min), then stage 1 (3 epochs) and stage 2 for **one epoch only** (later stage-2 epochs cost clarity and melody in our runs).
  One GPU: multi-GPU stage 1 ran out of memory in every rank for us.
- Loading older checkpoints: weight-norm key names differ between code versions, and a non-strict load silently leaves the pitch branch
  random. Rename the keys, fail on any missing tensor, and check that the predicted pitch is in a human range.
- Voice packs (style vectors) come from the teacher's style encoder over ~200 of the voice's own takes. These barely separate voices
  (packs are 0.96-0.99 alike), so each voice also gets learned rows and offsets (below).
- **Open:** the symbols the text side is trained on matter for melody. A Kokoro text side trained on Kokoro's own symbol conventions
  kept a much more natural melody than the same model trained on espeak-style IPA (melody gap 1.6-1.8 vs 4.5 on our scale, same clarity).
  We are retraining our teacher on Kokoro-native symbols; check the release notes for the result before you train.

### 4b. Sound side (`recipe/train/`)
- A 3.65 M-parameter decoder: AdaIN trunk, a 64-harmonic sine source plus noise, and an iSTFT head straight to 24 kHz (no separate vocoder).
  One learned 16-number style row per voice.
- From scratch: 50k steps (25k spectral, 25k adversarial), then continue to ~80k steps (50k -> 80k: unclear 4.71 -> 4.50 %, weakest-tenth
  sound 4.16 -> 4.38). Don't restart the learning-rate schedule to go further: it cost clarity at 100k-150k.
- An 8.4 M decoder tied the 3.65 M one; it isn't worth the size.
- Several voices in one model: balance batches (repeat small voices). A new voice added to a trained decoder reaches its own timbre after
  ~15 minutes of fine-tuning on one GPU; a new style vector alone keeps the old voice. One voice per model was 0.4-1.0 points clearer than
  a shared 8-voice model on the same data: sharing is a trade.
- The decoder always learns from real Maya audio, never from a teacher's renders.

### 4c. Pitch, loudness and duration heads
- First the pitch/loudness branch plus a learned 128-number offset per voice (3,000 steps, lr 2e-4), then the duration head with the
  offsets frozen (4,000 steps, lr 2e-4). Let the duration path see the offset detached: with one shared offset, a voice that needed a large
  pitch shift read 19 % too fast.
- **Pace comes from the data.** The duration head learns only from takes inside the voice's natural-pace window (we centre it where the
  voice's book-text pace maps to its natural pace on a test line). The model copies the pace of those takes: one window gave 204 wpm, a
  slower one 194.
- **Never use a speed multiplier, a per-voice duration scale or stretched durations.** A model is clearest at its own natural pace:
  speeding up or slowing down both cost clarity, and models trained on durations scaled x1.175 were less clear (6.3 %) than the same recipe
  at the natural pace (5.5 %).
- **Train on whole passages, speak sentence by sentence.** Passage training moved the sentence-to-sentence melody a lot (pitch-step error
  0.85 -> 0.55 semitones). Rendering whole lines at once didn't help.

### 4d. Creaky sentence ends
In our models creak at sentence ends comes from the predicted pitch dropping below the voicing floor, not from the decoder. A render-time
clamp fixes it without training: hold the last ~0.45 s of voiced pitch at or above max(85 Hz, sentence median - 6 semitones). On two deep
male voices creak went 63 -> 13 % and 76 -> 4 % of sentence ends, with the same clarity. Training the decoder only on clean endings, or a
lower pitch floor, did not help.

## 5. Distil to nano (`recipe/distil/`)
- **Text side** (~3 M parameters, ALBERT and predictor at about a third of the width): trained on text only to copy the teacher's per-sound
  durations, pitch and loudness for every voice. Because the teacher is deterministic per voice, copying it doesn't flatten the melody.
  98 % of sounds land within one frame of the teacher after ~2,000 steps.
- Teach the student's timing from each voice's pace head (4c), not the teacher's raw timing: two voices went from 8.1 to 5.2 % and from
  11.1 to 9.4 % unclear, and each voice kept its own pace.
- **Sound side** (~4.1 M including its own small text encoder): trained on the real audio, to ~80k steps. A voice with only ~2 h of data
  got rougher past ~70k steps.
- Distilled checkpoints swing in quality while the loss falls smoothly; pick by the bench, not by the last step.
- Result: 7.16 M parameters, 10.1 MB (int8 decoder, fp16 text side).

## 6. Export (`recipe/export/`)
- One ONNX graph (opset 17, batch 1) with a voice id input. Keep the sine source's phase path in float32 (some web builds can't cast to
  float64) and store 2πk in float32: an fp16 constant put the top harmonics up to ~0.5 rad off.
- **Profile before optimising.** In nano the decoder is ~80 % of the time; in the 61 MB model the text side is ~58 %.
- **Two exact rewrites** (same maths, no retraining): the 64-harmonic sine sum in closed form, sin(Kt/2) sin((K+1)t/2) / sin(t/2), and the
  six short-time Fourier transforms as one batched factorised DFT (600 = 25 x 24). On a Galaxy Tab S10 Lite (4 threads) nano went from
  30x to 42x real time; the 8-voice bench didn't move (7.88 -> 7.85 % unclear).
- On nano, running 3 of the student ALBERT's 6 passes (no retraining) gave 46x real time and 0.10 s to the first sentence, with the bench
  unchanged or slightly better (7.88 -> 7.62 %). On the 61 MB model, cutting passes without retraining broke clarity and melody.
- int8: an int8 decoder is the default file. Plain per-row int8 of a text side held every measure, but on onnxruntime 1.32 set
  `session.disable_quant_qdq=1` or you lose the speed gain. Dynamic int8 on ALBERT/LSTMs didn't help.
- Splitting the first sentence at a comma halves the time to first audio but breaks the melody (the first half ends like a sentence).

## 7. Run it
- Python: `python/kale_tts.py` (onnxruntime + numpy, front end in-process). Node and browser: `js/kale-tts.mjs`. HTTP:
  `server/server.mjs` (OpenAI-style `POST /v1/audio/speech`, WAV or streamed PCM). More engines and serving stacks: `runtimes/RUNTIMES.md`.
- CPU is the right target. One server core runs ~30 nano streams in real time. GPUs don't speed up a batch-1 graph this small (a T4 beat an
  H100). vLLM and SGLang are built for autoregressive models and don't fit.
- Browser WebGPU: keep the six small LSTM layers on the CPU (`forceCpuNodeNames`; about 2x faster for nano), use nano's fp32 graph
  (tiny: its int8 graph). `js/kale-tts.mjs` does both by default.
- Small serverless functions: set onnxruntime's intra-op threads to 1 on a 1-vCPU function (the default halves speed) and ship only your
  platform's onnxruntime binaries.
- Mac: plain onnxruntime on the CPU (4 threads) beat CoreML by 2-4x.

Measured speed of the 8-voice nano file (16 Nov 2026):
| where | x real time | first audio |
|---|---|---|
| Galaxy Tab S10 Lite, 4 threads, release graph | 29-30x | 0.15-0.18 s |
| same, with the rewrites and 3 ALBERT passes (step 6) | 46x | 0.10 s |
| Linux server CPU, 4 threads | 37x (int8) / 46x (fp32) | 0.09 s |
| Mac laptop CPU (Apple M5 Max), 4 threads | 178x | 0.017 s |
A phone-sized browser window on a laptop is not a phone: one real phone ran an older model of ours ~3x slower than our laptop
phone-sized harness. Use real devices for phone numbers.

## 8. Measure
`recipe/bench/`: 192 lines, rendered sentence by sentence as an app would, 3 noise seeds (one render alone can be off by ~0.5
points), one loudness gain per model with zero clipped samples.
- **Unclear words**: each word's sounds checked against Maya's own takes of the same line. The score depends on the reference takes, so
  score each voice against its own Maya takes and print Maya's own score next to it. On easier references a model looked 2.4 points
  clearer than it was.
- **Weakest tenth**: mean UTMOS of the worst 10 % of lines. UTMOS likes large smooth decoders, sometimes more than the original audio.
- **Melody gap to Maya** (`melody_gap.py`): 0 = the voice's own Maya takes, lower is better; an averaged, flat melody can't score well.
  We dropped "pitch-step error" as the headline: a flat average contour scored better than Maya herself.
- **Creak**: share of sentence ends with >= 100 ms of rough sound; compare with the voice's own Maya takes (some deep voices creak in the
  source too).
- **Pace**: words / full clip seconds of one fixed test line at speed 1.0.
- Listen last, every time. Several of our models passed every metric and still sounded wrong.

8-voice nano (one 10 MB file), each voice against its own Maya takes, 3 seeds: unclear words 5.5-11.1 % in the first build and 4.4-9.4 %
after the timing fix of step 5 (the voices' own Maya takes: 3.2-6.3 %), weakest tenth 4.13-4.44, 173-195 wpm. Per-voice tables: MODEL_CARD.md.

## 9. What cost us the most time
- Rendering and filtering in batches instead of streaming; waiting for every voice before training.
- A creak detector that deleted male voices; checking only the last sentence end.
- Speed multipliers to hit a pace (the pace has to come from the data).
- Reading a melody number that rewards flat speech; comparing clarity on different reference sets.
- An American pronunciation front end on British voices (19.8 % unclear; a British front end plus a short fine-tune brought two British
  voices from 8.4 to 7.7 % and 9.6 to 8.5 %).
- Many melody fixes on the pitch predictor (plain L1 fine-tunes, discriminators, contour codes, label heads, sampled pitch): none moved the
  melody gap the way the text side's symbol set did.

## 10. Costs we measured
- Data for 8 voices (render + filter + strict checks): ~$200 on rented GPUs, most of it the filter. Whole night including training and
  benches for both model sizes: ~$290.
- A two-voice run of a smaller recipe (render, filter, train one 21 MB model): ~$37.

## 11. Licences and credits
- Based on Maya (Maya-1, Maya Research, Apache-2.0); every training clip is synthetic, made with Maya-1 from a text description.
- Text side derived from Kokoro-82M (Apache-2.0, StyleTTS2 lineage, MIT). Front end: CMUdict (BSD-2 style notice), Britfone (MIT) for
  British English, number/date rules adapted from dizione (Apache-2.0); no espeak, no GPL.
- Our weights and code: Apache-2.0. Our dataset: CC BY 4.0.
- All voices are synthetic. Don't use them to pass speech off as a real person's or as human-recorded.