Instructions to use ETM-Network/jade-1.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use ETM-Network/jade-1.1 with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download ETM-Network/jade-1.1 --local-dir jade-1.1
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
File size: 8,467 Bytes
c7e5c4f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | # Jade 1.1: details
How the model is built and trained, and how it measures. The model card is [README.md](README.md); data credits are in [ATTRIBUTION.md](ATTRIBUTION.md).
## Architecture
| Part | Parameters | Trained here |
|---|---:|---|
| Text encoder, diffusion transformer, length sampler, voice and language tables | 83.8 M | yes |
| Supertonic 3 decoder (latents to waveform) | 25.3 M | no, frozen |
| **Total at inference** | **109.1 M** | |
- **Latents.** Supertonic 3's autoencoder latents (24 channels at 86.13 Hz), packed six frames at a time into 144-dimensional frames at 14.36 Hz, Supertonic's own packing.
- **Text.** Unicode NFKD characters, with each language's punctuation mapped onto a shared set; 3,171 tokens, kanji included. English and German go through their own normalizers first (numbers, dates, abbreviations).
- **Acoustic model.** A 12-block diffusion transformer, width 576, 9 attention heads. Every block cross-attends to the encoded text with length-normalised rotary positions, so text and audio align implicitly; there are no per-letter durations.
- **Length.** A small flow-matching sampler predicts each piece's total length from the text, the voice and the language.
- **Conditioning.** A learned voice embedding (9 voices) and a language embedding (23 languages). There is no voice cloning.
- **Few-step sampling.** The network takes the step size as an input, so each step jumps a given fraction of the way from noise to speech. One set of weights serves 2 to 32 steps (the final stage was tuned for 2 to 7), with guidance built in.
- **Decoder.** Supertonic 3's vocoder, exported from its released ONNX file.
Long text is split into pieces of up to about 20 seconds at sentence and clause boundaries, and each piece is generated separately.
## Training data
Public speech in 23 languages, from datasets whose licences allow commercial use (CC0, CC BY 4.0, CC BY 2.0, public domain, OpenMDW-1.1). The datasets, their licences and credits are in [ATTRIBUTION.md](ATTRIBUTION.md).
1. **Transcripts.** Each dataset's own transcripts where they have punctuation and casing. Otherwise punctuation and casing came from NVIDIA Granary-v2 labels or Whisper large-v3-turbo transcripts. Some parliament sessions without human transcripts use these machine labels as the text.
2. **Voices.** The nine voices are modelled on nine LibriVox volunteer readers.
3. **Filtering.** Clips were kept when Whisper read them back with a character error rate of at most 10 %.
4. **Cutting.** Clips of 0.4 to 21 seconds. Rows with digits were removed in every language except English and German, which is why the other languages don't read digits.
## Training
**Teacher.** One RTX 4090, bf16, AdamW (β 0.9/0.99, weight decay 0.01), learning rate 4.2e-4 (length sampler 2.1e-4), 2,000 warm-up steps, EMA of the weights with decay 0.9999, batches of up to 2,300 latent frames. Languages were added in stages; each new language joined through zero-initialised table rows, so the earlier languages were unchanged at the moment of joining.
| Stage | Steps | Data |
|---|---|---|
| English | 0 → 686,200 | English |
| German added | → 886,200 | English and German, 50/50 by frames |
| Seven more languages | → 1,386,200 | 9 languages |
| Six more languages | → 1,566,200 | 15 languages |
| Eight more languages | → 1,786,200 | 23 languages |
| Cooldown | → 1,811,200 | learning rate falls linearly to 0.1× |
The teacher samples in 16 steps with classifier-free guidance 4.0. It is not released.
**Distillation.** The released model starts from the teacher and learns to land, in N jumps, where the teacher's 16 guided steps land from the same noise.
1. **Flow map** (7,000 steps). Each item draws a step count from 1 to 32 (2 to 7 most often). Targets come from the teacher or from the model's own two half-jumps, plus a loss on the whole 2–7-jump rollout against the teacher's output.
2. **Polish** (10,000 steps). An adversarial loss (a critic on the frozen teacher's features) and an anchor to the teacher's output from the same noise, with N from 2 to 7.
Texts come from all 23 languages equally often; this stage needs no audio.
**Quantization.** The low-bit weights are set by learned rounding, optimised layer by layer (2,000 steps per layer), and a bit width per layer chosen under a size budget. ONNX Runtime has no 3-bit kernels, so the ONNX files mix 2, 4 and 8 bits where MLX uses 3.
## Evaluation
**Set.** 181 held-out lines, 8 per language (7 for English, German and Bulgarian), one voice each, never trained on. Rendered from the released ONNX files by the `etm-tts` package at 6 steps, exactly as a user gets them at that setting; the release default, q8 at 12 steps, is scored below the table.
**Character error rate** of a Whisper transcript against the input text, on letters only: Whisper small.en for English, large-v3-turbo for the other languages; Japanese is compared on kana readings. On short lines one misread word moves a language's mean by several points.
| | raw | q8 | q4 | q3 | | | raw | q8 | q4 | q3 |
|---|---:|---:|---:|---:|---|---|---:|---:|---:|---:|
| `en`¹ | 2.3 | 2.3 | 2.0 | 2.0 | | `es` | 0.7 | 0.7 | 0.7 | 0.7 |
| `de` | 0.7 | 0.4 | 0.4 | 0.8 | | `pl` | 1.5 | 1.5 | 1.5 | 1.6 |
| `it` | 0.0 | 0.0 | 0.0 | 0.0 | | `nl` | 0.0 | 0.0 | 0.3 | 0.6 |
| `pt` | 0.0 | 0.0 | 0.0 | 0.0 | | `cs` | 0.3 | 1.2 | 0.0 | 0.0 |
| `el` | 4.0 | 4.0 | 4.0 | 3.6 | | `hu` | 1.3 | 0.2 | 0.2 | 0.3 |
| `bg` | 1.6 | 1.6 | 1.1 | 1.1 | | `lt` | 0.5 | 0.5 | 0.0 | 0.0 |
| `ru` | 0.2 | 0.2 | 0.2 | 0.2 | | `hr` | 2.3 | 1.9 | 1.4 | 1.4 |
| `ja` | 4.2 | 4.2 | 3.1 | 4.2 | | `sl` | 0.0 | 0.0 | 0.0 | 1.0 |
| `hi` | 1.6 | 1.6 | 1.6 | 1.9 | | `et` | 0.9 | 1.6 | 2.9 | 1.0 |
| `lv` | 0.0 | 0.0 | 0.0 | 0.0 | | `da` | 2.1 | 2.1 | 2.1 | 2.1 |
| `fr` | 1.8 | 1.8 | 1.8 | 1.0 | | `sk` | 0.2 | 0.0 | 0.0 | 0.2 |
| | | | | | | `ro` | 0.0 | 0.2 | 0.3 | 0.0 |
All values in %. ¹ Without one line where Whisper stopped transcribing at a 1.5 s pause; the line is spoken in full (checked by transcribing the rest on its own). With it, English is 13.9 / 13.9 / 13.6 / 1.9 %; in q3 the pause is shorter.
**The default, q8 at 12 steps**, on the same lines and seeds: 1.1 % CER over all 181 lines (English 2.0 %, the pause line read in full), 147 lines without an error, voice similarity 0.818.
**Against full precision**, same lines and seeds: pitch-track correlation 0.978 (q8), 0.962 (q4), 0.924 (q3); similarity of each render to the raw render (WavLM embeddings) 0.998, 0.996, 0.993. Clicks, clipping and noise floor stay at raw's level (q3's floor is 0.9 dB higher).
**Steps.** The distilled model against the teacher at 64 steps, on a 181-line set of the same kind (PyTorch renders):
| Sampler | Pitch match | Mel L1 | CER |
|---|---:|---:|---:|
| Teacher, 64 steps | reference | reference | 0.7 % |
| Teacher, 16 steps | 0.890 | 0.733 | 1.0 % |
| Distilled, 4 steps | 0.871 | 0.928 | 1.1 % |
| Distilled, 6 steps | 0.875 | 0.886 | 1.1 % |
In our blind listening on short clips, 4 steps had occasional pitch jumps on single words and 6 steps mostly removed them. Listening to minutes of audiobook text later showed more: at q4 and q3 with few steps the pitch errors become noticeable over long passages, while q8 at 12 steps holds up. The release default is therefore q8 at 12 steps.
**More steps.** The final distillation stage was tuned for 2 to 7 steps, but the model also runs at more. The released q4 files through `etm-tts`, on the same 181 lines (automatic measures, not checked by ear):
| Steps | CER | Voice similarity | Pitch jumps per minute | Clicks per minute |
|---:|---:|---:|---:|---:|
| 6 | 1.5 % | 0.804 | 13.3 | 3.3 |
| 12 | 0.9 % | 0.811 | 9.7 | 4.6 |
| 20 | 0.7 % | 0.812 | 11.3 | 4.8 |
| 32 | 1.0 % | 0.809 | 15.6 | 5.3 |
Most of the CER difference is the English line Whisper stops reading at a pause at 6 steps (see the evaluation above). Time grows with the step count: 32 steps take about five times as long as 6.
**Voices.** Each voice read the same sentences, and every clip was matched to the nearest of the nine voice centroids (WavLM speaker embeddings, chance 11 %). On full sentences the teacher's clips land on their own voice 91.5 % of the time; the readers' real recordings reach 98.6 %. On two- or three-word phrases the distilled model reaches 46 % (real recordings cut as short: 68 %), so short replies can sound alike across voices. Tested on the first 15 languages.
|