File size: 4,713 Bytes
f21d6eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6901100
f21d6eb
 
 
 
 
 
6901100
f21d6eb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# kale-tts JS API (draft)
`kale-tts.mjs`: `KaleTTS.load(model, { ort?, frontend?, threads?, graph?, executionProviders?, sessionOptions? })`, `tts.listVoices()` (id, name, description, accent, label = "Name · description"), `tts.resolveVoice(nameOrId)`, `tts.preload(voices?)`, `tts.speak(text, { voice })` -> `{ audio, sampleRate }`,
`tts.stream(text, { voice })` (async iterator, one sentence at a time), `toWav(audio, sampleRate)`, `KaleTTS.registerFrontEnd(accent, { g2pModel, lexicon, override, terms } | loader)` (the four front-end files, or a loader such as `loadEnGB`).
Expects `../models/<name>/` (model.onnx, voices.json, voices/*.bin, tokenizer.json) and `../frontend/` next to it.
Accent switch: voices.json `accent` -> front end (en-us now; en-gb hook for the british front end).
Tested 16 Nov: Node 24 + onnxruntime-node 1.22.0; headless Chromium (Playwright's chrome-headless-shell) + onnxruntime-web 1.22.0 wasm; nano and tiny.

## preload: work offline after load
`await tts.preload()` loads every voice's files and the front end(s) those voices need, up front; `tts.preload("kenzie")` or
`tts.preload(["kenzie", "amy"])` loads only those. It returns the loaded ids. Afterwards `speak()` / `stream()` for those voices need no
network or file access (onnxruntime and the model are already loaded by `KaleTTS.load`). Voices that weren't preloaded still load on
first use. Loads are cached per voice and shared between parallel callers; a failed load is retried on the next call. An unknown id
throws with the list of valid ids.
```js
const tts = await KaleTTS.load("kale-tts-1-nano");
await tts.preload();                 // all 8 voices + front end; from here on, offline is fine
```
Tested 16 Nov: Node (model + front-end folders moved away after preload, all 8 voices still speak) and headless Chromium (browser context
set offline after preload, all 8 voices speak, 0 console errors).

## Graph and backend
`KaleTTS.load(name, { graph, executionProviders, sessionOptions })`:
- `graph`: `"auto"` (default), `"int8"` (`kale-tts-1-tiny.onnx`), `"fp32"` (`kale-tts-1-tiny.fp32.onnx`, nano only for now) or a file name in the model folder.
- `executionProviders`: onnxruntime EPs, e.g. `["webgpu"]`, `["wasm"]`, `["cpu"]`. `sessionOptions` go to `InferenceSession.create`.
- Default with nothing set: in a browser with a WebGPU adapter, fp32 on webgpu (checked with one tiny run); if there's no adapter, no fp32
  file, or the run fails, int8 on wasm. In Node: int8 on cpu. Options you set are used as given, with no fallback.
- WebGPU sessions keep the six LSTM layers on the CPU (`forceCpuNodeNames`): since onnxruntime-web 1.27 their WebGPU kernel is slower
  for this model (nano fp32 85x -> 163-175x, tiny int8 29x -> 58-62x, Mac Chrome). Pass your own `{ name: "webgpu", ... }` object to
  override. With no fp32 file (tiny), the default tries int8 on webgpu before wasm.
- `tts.backend` tells what was picked: `{ graph: "kale-tts-1-tiny.fp32.onnx", executionProviders: ["webgpu"] }`.
- Browser: WebGPU needs onnxruntime-web 1.31 or newer, loaded from its WebGPU build, e.g. an import map entry
  `"onnxruntime-web": "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.31.1/dist/ort.webgpu.bundle.min.mjs"` (that build runs wasm too).
  With 1.22.0, WebGPU fails at run time and the default falls back to wasm.
```js
const tts = await KaleTTS.load("kale-tts-1-nano");                                               // auto
const cpu = await KaleTTS.load("kale-tts-1-nano", { graph: "int8", executionProviders: ["wasm"] });
```
Tested 16 Nov in headless Chrome for Testing (WebGPU adapter) and chrome-headless-shell with onnxruntime-web 1.31.1 and 1.22.0, every
option and the fallback, nano and tiny; and in Node (onnxruntime-node 1.22): default, fp32, int8, sessionOptions.

## Voices by name
Every call that takes a voice accepts its public name in any case (`"kenzie"`, `"Kenzie"`) or its internal id (`"s057b"`); earlier names stay as aliases (`"imogen"` -> Amy, `"alistair"` -> Jack). The HTTP server
(`server/server.mjs`) and the Python package (`resolve_voice`) do the same; with no voice given they use the first voice.

## Guards
Only voices listed in the model's `voices.json` load (by name, alias or id); anything else throws `unknown voice …` with the valid list,
never a silent fallback to another voice. Each voice's files are size-checked against `voices.json` (`<id>.bin` = rows of 256 floats,
`<id>.extra.bin` = exactly the `voice_inputs` floats), and a model file that starts like a text / HTML page is refused. A web server that
answers a missing file with an HTML page (status 200) therefore gives a clear error instead of a broken voice. The Python package does the same.