jade-1.1 / DETAILS.md
SynnJJ's picture
Jade 1.1
c7e5c4f
|
Raw History Blame Contribute Delete
8.47 kB

Jade 1.1: details

How the model is built and trained, and how it measures. The model card is README.md; data credits are in ATTRIBUTION.md.

Architecture

Part Parameters Trained here
Text encoder, diffusion transformer, length sampler, voice and language tables 83.8 M yes
Supertonic 3 decoder (latents to waveform) 25.3 M no, frozen
Total at inference 109.1 M
  • Latents. Supertonic 3's autoencoder latents (24 channels at 86.13 Hz), packed six frames at a time into 144-dimensional frames at 14.36 Hz, Supertonic's own packing.
  • Text. Unicode NFKD characters, with each language's punctuation mapped onto a shared set; 3,171 tokens, kanji included. English and German go through their own normalizers first (numbers, dates, abbreviations).
  • Acoustic model. A 12-block diffusion transformer, width 576, 9 attention heads. Every block cross-attends to the encoded text with length-normalised rotary positions, so text and audio align implicitly; there are no per-letter durations.
  • Length. A small flow-matching sampler predicts each piece's total length from the text, the voice and the language.
  • Conditioning. A learned voice embedding (9 voices) and a language embedding (23 languages). There is no voice cloning.
  • Few-step sampling. The network takes the step size as an input, so each step jumps a given fraction of the way from noise to speech. One set of weights serves 2 to 32 steps (the final stage was tuned for 2 to 7), with guidance built in.
  • Decoder. Supertonic 3's vocoder, exported from its released ONNX file.

Long text is split into pieces of up to about 20 seconds at sentence and clause boundaries, and each piece is generated separately.

Training data

Public speech in 23 languages, from datasets whose licences allow commercial use (CC0, CC BY 4.0, CC BY 2.0, public domain, OpenMDW-1.1). The datasets, their licences and credits are in ATTRIBUTION.md.

  1. Transcripts. Each dataset's own transcripts where they have punctuation and casing. Otherwise punctuation and casing came from NVIDIA Granary-v2 labels or Whisper large-v3-turbo transcripts. Some parliament sessions without human transcripts use these machine labels as the text.
  2. Voices. The nine voices are modelled on nine LibriVox volunteer readers.
  3. Filtering. Clips were kept when Whisper read them back with a character error rate of at most 10 %.
  4. Cutting. Clips of 0.4 to 21 seconds. Rows with digits were removed in every language except English and German, which is why the other languages don't read digits.

Training

Teacher. One RTX 4090, bf16, AdamW (β 0.9/0.99, weight decay 0.01), learning rate 4.2e-4 (length sampler 2.1e-4), 2,000 warm-up steps, EMA of the weights with decay 0.9999, batches of up to 2,300 latent frames. Languages were added in stages; each new language joined through zero-initialised table rows, so the earlier languages were unchanged at the moment of joining.

Stage Steps Data
English 0 → 686,200 English
German added → 886,200 English and German, 50/50 by frames
Seven more languages → 1,386,200 9 languages
Six more languages → 1,566,200 15 languages
Eight more languages → 1,786,200 23 languages
Cooldown → 1,811,200 learning rate falls linearly to 0.1×

The teacher samples in 16 steps with classifier-free guidance 4.0. It is not released.

Distillation. The released model starts from the teacher and learns to land, in N jumps, where the teacher's 16 guided steps land from the same noise.

  1. Flow map (7,000 steps). Each item draws a step count from 1 to 32 (2 to 7 most often). Targets come from the teacher or from the model's own two half-jumps, plus a loss on the whole 2–7-jump rollout against the teacher's output.
  2. Polish (10,000 steps). An adversarial loss (a critic on the frozen teacher's features) and an anchor to the teacher's output from the same noise, with N from 2 to 7.

Texts come from all 23 languages equally often; this stage needs no audio.

Quantization. The low-bit weights are set by learned rounding, optimised layer by layer (2,000 steps per layer), and a bit width per layer chosen under a size budget. ONNX Runtime has no 3-bit kernels, so the ONNX files mix 2, 4 and 8 bits where MLX uses 3.

Evaluation

Set. 181 held-out lines, 8 per language (7 for English, German and Bulgarian), one voice each, never trained on. Rendered from the released ONNX files by the etm-tts package at 6 steps, exactly as a user gets them at that setting; the release default, q8 at 12 steps, is scored below the table.

Character error rate of a Whisper transcript against the input text, on letters only: Whisper small.en for English, large-v3-turbo for the other languages; Japanese is compared on kana readings. On short lines one misread word moves a language's mean by several points.

raw q8 q4 q3 raw q8 q4 q3
en¹ 2.3 2.3 2.0 2.0 es 0.7 0.7 0.7 0.7
de 0.7 0.4 0.4 0.8 pl 1.5 1.5 1.5 1.6
it 0.0 0.0 0.0 0.0 nl 0.0 0.0 0.3 0.6
pt 0.0 0.0 0.0 0.0 cs 0.3 1.2 0.0 0.0
el 4.0 4.0 4.0 3.6 hu 1.3 0.2 0.2 0.3
bg 1.6 1.6 1.1 1.1 lt 0.5 0.5 0.0 0.0
ru 0.2 0.2 0.2 0.2 hr 2.3 1.9 1.4 1.4
ja 4.2 4.2 3.1 4.2 sl 0.0 0.0 0.0 1.0
hi 1.6 1.6 1.6 1.9 et 0.9 1.6 2.9 1.0
lv 0.0 0.0 0.0 0.0 da 2.1 2.1 2.1 2.1
fr 1.8 1.8 1.8 1.0 sk 0.2 0.0 0.0 0.2
ro 0.0 0.2 0.3 0.0

All values in %. ¹ Without one line where Whisper stopped transcribing at a 1.5 s pause; the line is spoken in full (checked by transcribing the rest on its own). With it, English is 13.9 / 13.9 / 13.6 / 1.9 %; in q3 the pause is shorter.

The default, q8 at 12 steps, on the same lines and seeds: 1.1 % CER over all 181 lines (English 2.0 %, the pause line read in full), 147 lines without an error, voice similarity 0.818.

Against full precision, same lines and seeds: pitch-track correlation 0.978 (q8), 0.962 (q4), 0.924 (q3); similarity of each render to the raw render (WavLM embeddings) 0.998, 0.996, 0.993. Clicks, clipping and noise floor stay at raw's level (q3's floor is 0.9 dB higher).

Steps. The distilled model against the teacher at 64 steps, on a 181-line set of the same kind (PyTorch renders):

Sampler Pitch match Mel L1 CER
Teacher, 64 steps reference reference 0.7 %
Teacher, 16 steps 0.890 0.733 1.0 %
Distilled, 4 steps 0.871 0.928 1.1 %
Distilled, 6 steps 0.875 0.886 1.1 %

In our blind listening on short clips, 4 steps had occasional pitch jumps on single words and 6 steps mostly removed them. Listening to minutes of audiobook text later showed more: at q4 and q3 with few steps the pitch errors become noticeable over long passages, while q8 at 12 steps holds up. The release default is therefore q8 at 12 steps.

More steps. The final distillation stage was tuned for 2 to 7 steps, but the model also runs at more. The released q4 files through etm-tts, on the same 181 lines (automatic measures, not checked by ear):

Steps CER Voice similarity Pitch jumps per minute Clicks per minute
6 1.5 % 0.804 13.3 3.3
12 0.9 % 0.811 9.7 4.6
20 0.7 % 0.812 11.3 4.8
32 1.0 % 0.809 15.6 5.3

Most of the CER difference is the English line Whisper stops reading at a pause at 6 steps (see the evaluation above). Time grows with the step count: 32 steps take about five times as long as 6.

Voices. Each voice read the same sentences, and every clip was matched to the nearest of the nine voice centroids (WavLM speaker embeddings, chance 11 %). On full sentences the teacher's clips land on their own voice 91.5 % of the time; the readers' real recordings reach 98.6 %. On two- or three-word phrases the distilled model reaches 46 % (real recordings cut as short: 68 %), so short replies can sound alike across voices. Tested on the first 15 languages.