kale-tts-1-tiny / README.md
NERDDISCO's picture
Clear file names: kale-tts-1-tiny.onnx at the root, PyTorch weights in pytorch/
6901100 verified
|
Raw History Blame Contribute Delete
24.6 kB
metadata
license: apache-2.0
language:
  - en
pipeline_tag: text-to-speech
library_name: onnx
tags:
  - text-to-speech
  - tts
  - small
  - cpu
  - on-device
  - synthetic-voices
  - based-on-maya
base_model: KaleComputer/kale-tts-1-base
base_model_relation: finetune
datasets:
  - KaleComputer/kale-tts-1-data

kale-tts-1-tiny

kale-tts-1-tiny

kale-tts-1-tiny is a small English text-to-speech model with 10 voices designed with Maya-1, that runs on CPU or GPU on low-end devices. It stays as close as possible to Maya-1's melody, which already sounds great, while running on weak hardware.

measure value
size 79.9 MB on the CPU (int8, kale-tts-1-tiny.onnx); 183.2 MB for WebGPU in the browser (float32, kale-tts-1-tiny.fp32.onnx)
parameters 45.7M (16.8M of them the frozen Ettin word encoder)
unclear words +1.4 points vs Maya-1's own recordings (Maya-1: 5.82 %)
melody gap +7.48 points vs Maya-1's own recordings (Maya-1: 0)
speed, Samsung Galaxy Tab S10 Lite CPU 24x real time, first sentence after 0.16 s
speed, Mac browser 87x (WebGPU), 48x (WASM)

Unclear words: a speech recognizer listens word by word, and "unclear" is the share of words it isn't sure about; Maya-1's own recordings score 5.82 % on the same lines. Melody gap: how far the voice's pitch melody is from Maya-1's own recordings of the same lines, in percentile points; Maya-1 scores 0. Lower is better for both.

Want something smaller and faster, with slightly less quality? Try kale-tts-1-nano.

Trained on kale-tts-1-data: 204.7 hours (99,783 takes) of the 10 voices' speech with its text, CC BY 4.0, open for anyone to train their own.

Voices

name voice pronunciation
Kenzie esports hype woman US English
Simone deep and confident woman US English
Amy crisp news-style woman British English
Frankie hard-boiled detective's partner US English
Darlene rough low-voiced old trucker woman US English
Owen bright young host US English
Toby mischievous prankster US English
Jack crisp news-style man British English
Earl African American jazz granddad US English
Magnus deep dramatic trailer narrator US English

Every voice was designed from a one-line text description; none is a real person.

Listen

The kale-tts-1 demo: type a line, pick a voice, press play

Every voice, in your browser: the kale-tts-1 demo.

Why we made it

We build a privacy-focused computer for kids. Our goal is to run all the AI locally on the device, so privacy is as high as it can be. Audio models were the quick first win, because they're easy to do with AI right now.

We made our own voice for Kale (Hazel), and we're keeping that one. But we share the whole concept, so you can recreate any voice from Maya-1. And here are 10 voices to try right away.

How it works

What it's made of

What kale-tts-1-tiny is made of

45.7M parameters in one 79.9 MB file (the CPU file; 183.2 MB for WebGPU): the dictionary in front, the Ettin word reader (16.8M, frozen), the part that reads the text and plans timing and melody (24.8M: a 3.0M phoneme reader + the 21.8M planner of kale-tts-1-base, frozen), a sound generator (4.1M), and a voice pack per voice.

How it was made

How kale-tts-1-tiny was made

  1. Maya-1 designs the 10 voices from a line of text and records their training audio: 204.7 hours.
  2. kale-tts-1-base, our Kokoro-82M fine-tuned on our 10 voices (made with Maya-1), is the teacher: its timing-and-melody planner taught this model; it's too big and slow for the tablet. kale-tts-1-tiny is distilled from it; the teacher is its own model repo.
  3. kale-tts-1-tiny keeps the timing-and-melody planner of kale-tts-1-base (21.8M, frozen) and puts a small phoneme reader (3.0M) and the frozen Ettin word reader (16.8M), which understands whole sentences, in front of it. The part that makes the sound (4.1M, a small harmonic-plus-noise generator) learns straight from Maya-1's audio. kale-tts-1-nano is a smaller student distilled from the same teacher.
  4. In front of it all, the dictionary turns words into sounds; every word is checked.

No flow matching and no diffusion anywhere: one pass per sentence.

Turns words into sounds

Text to phonemes

Tech terms first ("Node.js" -> "Node J S"), then numbers, dates, money and units as words (dizione rules), then every word through a CMUdict-based dictionary, a hand-checked override list and a small guesser for unknown words. No espeak. Files: frontend/.

A checked dictionary

Small TTS models live or die by their front end: the wrong sounds make the best voice say the wrong word. We checked every word the front end can meet (334,000 words, American and British) and fixed 3,800+ readings in frontend/words/override.public.tsv. Example: "details" came out of the base dictionary as dih-TAILS (dɪtˈeɪlz); one line in the override list makes it DEE-tails (dˈiːtˌeɪlz).

Add a word: one line in frontend/words/override.public.tsv (British: frontend/words/en-gb/override.public.tsv): the word in lower case, a tab, its IPA (stress mark right before the vowel). It wins over the dictionary and the guesser. Prompt for a coding agent:

In this repository, add the word "<word>" to frontend/words/override.public.tsv with the pronunciation <IPA> (one line: word, a tab,
the IPA in the same style as the other lines). Check that the word is not a person's name and is not offensive; if it is, stop. Run the
front end on a sentence with the word (python/kale_tts.py: QuickJSFrontEnd) and confirm the new IPA. Then render it with kale-tts-1-tiny and listen.

Reads the text, plans timing and melody

A small phoneme reader (3.0M) and the frozen Ettin word reader (jhu-clsp/ettin-encoder-17m, MIT, 16.8M, inside the graph), which understands whole sentences, feed the teacher's own timing-and-melody planner (21.8M, kept frozen). The word inputs (ettin_ids, ettin_mask, wavg, widx) come from frontend/ettin.js + ettin-tokenizer.json.

Makes the sound

A small generator (4.1M): a harmonic source plus noise through a learned filter, then an inverse STFT to 24 kHz. No separate vocoder, no flow matching: one pass per sentence. At the end of a sentence a per-voice pitch floor keeps the last words from creaking.

Voices

Voices

Each voice starts as a one-line description given to Maya-1. Maya-1 reads thousands of sentences in that voice; a filter keeps the takes that read back right, match the voice and end cleanly. Each voice ships as a style table and a 16-number sound row (voices/).

Training data

Training data

99,783 original Maya-1 takes, 204.7 hours, 10 voices, texts from public-domain books: kale-tts-1-data (CC BY 4.0). About 25 hours of clean takes per voice is where clarity levels off; 5 hours already sounds right but is less clear.

How we picked the base model

How we picked the base model, at selection

We tried open-weight models on the same 192 test lines against Maya-1's own recordings. Fine-tuned on Maya-1's audio, Kokoro-82M was the clearest and the closest to Maya-1's melody, but it ran at about real time on the tablet, far too slow, so it became the teacher: kale-tts-1-base. The published kale-tts-1-base keeps that fine-tune's text side and planner with stock Kokoro-82M's sound decoder; on the same bench it is less clear (+3.9 points) than tiny and nano.

How we measure

How we measure

192 English lines (stories, explanations, numbers, questions), all 10 voices, sentence by sentence as an app speaks; each voice against Maya-1's own recordings of the same lines. Unclear words: word spans from CTC forced alignment (wav2vec2-large-960h-lv60-self); kale-tts-1-tiny reads 7.23 % unclear, Maya-1's recordings 5.82 %. Per voice: results/bench/bench-10voices.json. Export check: every release file against a float32 build of the same weights on 200 sentences; the same length on each (int8: at most one sentence one frame off) and a log-mel distance of at most 0.20. tiny: the same length on all 200 sentences, log-mel distance 0.077; Whisper word errors 0.49 % on the 10-voice bench.

Extend it

Extend it

  • A word: one line in the override list (above).
  • A voice: a one-line description, a few thousand Maya-1 takes (recipe/render/, recipe/filter/), then teacher, heads and a new sound row (recipe/train/, recipe/export/).
  • A language: its own dictionary, guesser and number rules, takes in the same voices, then the text reader and planner fine-tuned on both languages.

Where it runs

where runs speed and notes
Browser, WebGPU yes 87x; downloads the 183.2 MB float32 file (onnxruntime-web 1.31 or newer)
Browser, WASM yes 48x with 4 threads (the 79.9 MB CPU file; threads need COOP/COEP headers)
Android (onnxruntime) yes 24x on the Galaxy Tab S10 Lite CPU, first sentence after 0.16 s
iOS not tested onnxruntime runs there; we have no device numbers
Python yes 114x (python/kale_tts.py; needs pip install quickjs)
Node.js yes 117x (js/kale-tts.mjs)
HTTP server (server/server.mjs) yes 82x for whole lines, text in, WAV out (one process; WAV or streamed)
transformers.js yes in Node, with our text-prep helper for the inputs (see Use it)
sherpa-onnx, kokoro-onnx, plain C / Go / Rust / Swift no these pass phoneme ids only; tiny also needs word inputs from its Ettin encoder, which our JavaScript and Python code builds from the text

Speed = times faster than real time, median over 70 test lines in Kenzie's voice (18 Nov): the model one sentence at a time; the server row whole lines, text in, WAV out. Machines: Apple MacBook Pro, M5 Max (18 CPU cores, 128 GB RAM) for the browser, Python, Node and server rows; Samsung Galaxy Tab S10 Lite (Exynos 1380, 8 CPU cores: 4 × Cortex-A78 + 4 × Cortex-A55) for Android (10x on 1 thread). Speed was measured on a 73.9 MB variant of this graph (its phoneme encoder in fp16); the 79.9 MB file keeps that encoder in float32 and may be slightly slower.

Use it

transformers.js

npm i @huggingface/transformers
npm pkg set type=module

# our text-prep helper: the js/ and frontend/ folders of this repo
pip install -U huggingface_hub
hf download KaleComputer/kale-tts-1-tiny --local-dir . --include "js/*" --include "frontend/*"

Save as tts.mjs, then run node tts.mjs:

import { writeFileSync } from "node:fs";
import { StyleTextToSpeech2Model, Tensor } from "@huggingface/transformers";
import { KaleTTS, toWav } from "./js/kale-tts.mjs";

const model = await StyleTextToSpeech2Model.from_pretrained("KaleComputer/kale-tts-1-tiny", { subfolder: "", dtype: "fp32" });

// text -> the model's inputs, sentence by sentence
const prep = await KaleTTS.prepare("https://huggingface.co/KaleComputer/kale-tts-1-tiny/resolve/main/");

const parts = [];
for await (const { inputs } of prep.inputs("Hello there! It is 9:30 on March 3.", { voice: "kenzie" })) {
  const feeds = {};
  for (const [name, t] of Object.entries(inputs)) feeds[name] = new Tensor(t.type, t.data, t.dims);

  // waveform.data: Float32Array, 24 kHz
  const { waveform } = await model(feeds);
  parts.push(waveform.data);
}

const audio = new Float32Array(parts.reduce((n, p) => n + p.length, 0));
let at = 0;
for (const p of parts) { audio.set(p, at); at += p.length; }
writeFileSync("hello.wav", Buffer.from(toWav(audio, 24000)));

KaleTTS.prepare() is our small text-prep helper: it turns text into this model's inputs (numbers and dates as words, the checked dictionary, the voice's style and voice inputs, and for tiny the Ettin word inputs). transformers.js has no tokenizer for these inputs, so the helper builds them and transformers.js runs the model.

On WebGPU (transformers.js 4.3 or newer), load the float32 file and keep the six small LSTM layers on the CPU for full speed, the same settings our runtime uses:

const LSTM_NODES = ["lstms.0", "lstms.2", "lstms.4", "lstm", "shared", "lstm_1"].flatMap((n) => [`/g/${n}/LSTM`, `/${n}/LSTM`]);

const model = await StyleTextToSpeech2Model.from_pretrained("KaleComputer/kale-tts-1-tiny", {
  subfolder: "",
  dtype: "fp32",
  model_file_name: "model.fp32",
  device: "webgpu",
  session_options: { executionProviders: [{ name: "webgpu", forceCpuNodeNames: LSTM_NODES }] },
});

This is temporary.

  • CPU and WASM: Transformers.js is 11–19 % slower than our own runtime today, because it pins an older onnxruntime. The gap closes by itself when transformers.js PR #1879 (onnxruntime 1.31.1) is merged; nothing changes in your code.
  • WebGPU: the two extra lines (LSTM_NODES and session_options) keep six small LSTM layers on the CPU. They stay until an onnxruntime release with the LSTM fix ships in Transformers.js; then you can remove them.

Without transformers.js: our runtime

Download the model with our runtime. It turns any text into speech, numbers, dates and tech terms included:

# the hf command comes with huggingface_hub
pip install -U huggingface_hub

# only the files the runtime needs
hf download KaleComputer/kale-tts-1-tiny --local-dir kale-tts-1-tiny \
  --include "kale-tts-1-tiny.onnx" --include "*tokenizer.json" --include "voices.json" --include "config.json" \
  --include "voices/*" --include "js/*" --include "frontend/*" --include "python/*" --include "server/*"

cd kale-tts-1-tiny

Node.js

npm i onnxruntime-node
npm pkg set type=module

Save as hello.mjs, then run node hello.mjs:

import { writeFileSync } from "node:fs";
import { KaleTTS, toWav } from "./js/kale-tts.mjs";

const tts = await KaleTTS.load("./");

// audio: Float32Array, 24 kHz
const { audio } = await tts.speak("Hello there! It is 9:30 on March 3.", { voice: "kenzie" });
writeFileSync("hello.wav", Buffer.from(toWav(audio, 24000)));

// for long text, sentence by sentence:
// for await (const { audio } of tts.stream(longText, { voice: "amy" })) play(audio);

Python

pip install onnxruntime numpy quickjs soundfile

Save as hello.py in the same folder, then run python hello.py:

import sys
sys.path.insert(0, "python")

import soundfile as sf
from kale_tts import KaleTTS

tts = KaleTTS(".")

# float32 audio, 24 kHz
audio = tts.speak("Hello there!", voice="amy")
sf.write("hello.wav", audio, 24000)

HTTP server

Run it locally and call it from any language or app over HTTP (an OpenAI-shaped POST /v1/audio/speech, WAV or streamed PCM):

npm i onnxruntime-node

# without a token it only listens on 127.0.0.1
TTS_API_TOKEN=choose-a-secret-of-20-characters PORT=8000 node server/server.mjs
curl -s http://127.0.0.1:8000/v1/audio/speech \
  -H "Authorization: Bearer choose-a-secret-of-20-characters" \
  -d '{"model": "kale-tts-1-tiny", "input": "Hello there!", "voice": "kenzie"}' \
  -o hello.wav

You pass the voice by its name.

How we made it

Research agents ran the experiments, on Claude Opus 5.5 and Claude Fable 5.2. We told them what to find out, and an automation checked every 60 minutes whether any agent was stuck and got it moving again.

What we tried, one line each:

Teacher and data

  • Maya-1 voice designer + SNAC codec, served on vLLM: worked. It made all the training audio, about 12-14x faster than plain generation.
  • Maya-1 itself on the device, with speculative decoding: didn't work, about 0.1x real time.
  • Kokoro-82M fine-tuned StyleTTS2-style (kikiri-tts) as the teacher: worked. The clearest in our model comparison (before its decoder swap), but about 1x real time on the tablet.
  • Training on whole multi-sentence passages, speaking sentence by sentence: worked, the main melody lever.
  • More data (a 114 h multi-voice pretrain, more hours per voice): didn't help much; about 25 h per voice is enough.
  • Speed multipliers and stretched durations: didn't work, less clear. Pace now comes from per-voice duration heads.

Reading the text, planning, the dictionary

  • Our own ~8 M model from scratch: misread too many words. Its shape became nano's student.
  • A small student that reads the text and plans timing, pitch and loudness, distilled from the teacher, plus a sound generator trained on real audio: worked, that's kale-tts-1-nano.
  • Ettin-encoder-17m word features instead of ALBERT: worked on tiny (about 1.6x faster on the tablet, same clarity); on nano bigger, not faster.
  • Fewer ALBERT passes, dynamic int8 ALBERT: didn't work, broke the melody or failed to load on Android.
  • Pitch fixes (pitch LoRA, contour discriminator, flow-matching pitch sampler, prosody label heads, a GRPO duration reward): no melody gain; GRPO learned to talk faster.
  • misaki dictionaries: best melody, but ruled out (they derive from GPL espeak output). CMUdict + our own guesser recovers about half of it.
  • A British front end (CMUdict rules + Britfone): only works with British-trained weights, so it ships off.

Making the sound

  • Matcha-TTS + Vocos: larger, slower, dull above about 5 kHz.
  • Other vocoders and codec adapters (WaveNeXt, BigVGAN, codec adapters): no gain on the device, much slower, or worse sound.
  • DiT flow-matching decoder, REPA, one-step decoders, an adversarial mel post-filter: too slow on the tablet or no gain by ear.
  • A small autoregressive codec-token TTS: about 1.3-1.6x real time, too slow.
  • A source-filter decoder (sine bank + shaped noise + one inverse STFT, no separate vocoder): worked, it ships in both models.
  • A per-voice pitch floor at sentence ends: worked, it removed most end-of-sentence creak.

Export

  • ONNX with int8 weights where safe, float32 timing and phase paths, the LSTMs on the CPU in browsers: worked. int8 on the timing path changed durations; CoreML and GPU serving didn't help.

Earlier directions: voices and data

  • VoxCPM2 and other voice-design models (Qwen3-TTS VoiceDesign, MOSS-VoiceGenerator, Ming-TTS, Parler-TTS) as the voice maker: dropped. Their takes sounded flatter, and the Maya-1 designs won the listening rounds.
  • Moods and sound tags from Maya-1 (calm, whisper, laugh, sigh) as extra speaker ids: worked at no speed cost, but Maya-1 often skipped the tag sounds. kale-tts-1 has one style per voice and no tag tokens.

Earlier directions: Matcha-TTS fine-tuning

  • Our own Matcha-TTS base trained from scratch on LJSpeech: worked, as good as the original.
  • Per-voice adapters: LoRA rank 32 came close to a full fine-tune, rank 8 was worse; a decoder-only fine-tune was far less clear (13.3 % vs 4.8 % unclear words).
  • Sampler settings (2-10 decoder steps, solvers, noise, guidance, checkpoint averaging): no gain; 2 steps sounded like 3.

Earlier directions: other models as the on-device voice

  • Kokoro-82M on the tablet, full and pruned to 37-44 M: too slow (about 1x and 6x real time), so it became the teacher only.
  • Piper (VITS) medium fine-tunes: as fast as Matcha but scored lower; the high size was too slow.
  • Supertonic 3: 2.4x real time on the tablet, and its OpenRAIL-M licence also covers models trained on its output.
  • Kitten TTS nano with a searched style: no closer to the target voice than its preset, 2.5x on the tablet.
  • Zonos 3 Mini: below real time on the tablet and a 1.1 GB download.
  • Voice cloning (a 12 M ZipVoice from scratch, Matcha-P with a voice prompt, the 123 M ZipVoice): dropped for a fixed voice set; ZipVoice cloned well but was 15-25x too slow on the tablet.
  • An MB-iSTFT-VITS-style end-to-end model (8.2 M, 4.7 M): very fast, but quality stalled below the bar.
  • Mach-Boom (Phonon 31M): fast, but no published audio encoder, so it can't learn a new voice.
  • Not tested because their licences don't allow commercial use: F5-TTS, XTTS-v2, Fish Speech, Spark-TTS, MaskGCT.

Earlier directions: prosody, speed and serving

  • A sampled prosody plan (flow-matching durations, pitch, loudness): closest pauses, but less clear (6.7-7.8 % unclear words).
  • A sentence-level delivery latent with context from the previous sentence: no melody gain.
  • A distilled 5-6 M Vocos vocoder: about 2x faster, but below the full vocoder.
  • The tablet GPU (our own WebGPU kernels) and XNNPACK: both slower than plain onnxruntime on 4 CPU cores.
  • Streaming part of a sentence: left seams on our decoder and hurt the melody; whole sentences are already fast.
  • vLLM / SGLang serving and CoreML: don't fit a one-pass model, or 2-4x slower than the CPU.

Earlier directions: yardsticks and other languages

  • kitten-tts-2 as a yardstick: kitten mini matched our small model on clarity but was 20x slower on the tablet.
  • A German teacher scout (Qwen3-TTS, Chatterbox, VoxCPM2, CosyVoice3, MOSS-TTS): Qwen3-TTS read best; no German model trained yet.

Make it faster or better

  • WebGPU: keep the six small LSTM layers on the CPU (forceCpuNodeNames; our JS API does it by default): about 2.2x faster on onnxruntime-web 1.27 and newer (29x -> 63x through transformers.js, 18 Nov).
  • Pick the file per backend: the int8 kale-tts-1-tiny.onnx on CPU and WASM, kale-tts-1-tiny.fp32.onnx on WebGPU.
  • Threads: 4 threads on a phone or tablet CPU; on a server 1 thread per process and one process per core.
  • Stream sentence by sentence: the first sentence plays while the rest renders.
  • A smaller text reader: most of tiny's extra time is the Ettin word reader; nano's own small reader is about 2x faster, with a melody further from Maya-1's.
  • Train it better: more clean recordings per voice (about 25 h each is where clarity levels off) and a cleaner teacher help more than a bigger model.

Limits

  • Kenzie's melody is the furthest from her Maya-1 recordings (melody gap 9.19, against 7.48 on average).
  • The two British voices are the least clear: Amy 9.0 %, Jack 9.1 % unclear (their Maya-1 recordings: 5.8 / 6.0 %).
  • English only. Rare names and heavy numbers can be misread; the guesser for unknown words is right on about two thirds of them.
  • Needs our text-prep code (JavaScript or Python) for the Ettin word inputs; tools that pass phoneme ids only can run nano instead.
  • Not for impersonating real people or passing the output off as human speech.

Files

kale-tts-1-tiny.onnx (79.90 MB, int8 / fp16 weights where safe, phoneme encoder and durations float32), kale-tts-1-tiny.fp32.onnx (183.25 MB, float32, for WebGPU), voices/ + voices.json, tokenizer.json, ettin-tokenizer.json (MIT). Inputs: input_ids, style, speed, ettin_ids, ettin_mask, wavg, widx, then the voice inputs listed in voices.json. PyTorch weights in pytorch/ (see pytorch/README.md). The training recipe is in recipe/.

Licence and credits

Weights and code Apache-2.0; dataset CC BY 4.0.

AI-generated speech: everything this model says is synthetic speech made by an AI model, in voices that belong to no real person. Tell listeners it is AI-generated, never present it as a human recording, and never use it to imitate a real person.

Credits: based on Maya-1: all training audio is original speech from Maya-1 (Maya Research, Apache-2.0), decoded with SNAC (MIT). The teacher is Kokoro-82M (hexgrad, Apache-2.0), fine-tuned with StyleTTS2 (MIT) and kikiri-tts (Apache-2.0) code; kale-tts-1-tiny keeps its timing-and-melody planner and contains the Ettin 17M encoder (JHU CLSP, MIT). Pronunciations: CMUdict (BSD-2-Clause), Britfone (MIT); the number, date and unit rules come from dizione and the homograph tables from tiny-voca, both by n4ze3m (Apache-2.0), the tables trained on Google Research's Wikipedia Homograph Data (Apache-2.0). Training texts: public-domain books made available by Project Gutenberg. Dataset: kale-tts-1-data (CC BY 4.0).