Instructions to use ETM-Network/jade-1.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use ETM-Network/jade-1.1 with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download ETM-Network/jade-1.1 --local-dir jade-1.1
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
|
Download DETAILS.md from ETM-Network/jade-1.1: direct link, hf CLI and curl.
- Browser
- Download file 8.47 kB
-
https://huggingface.co/ETM-Network/jade-1.1/resolve/main/DETAILS.md
- Command line
-
hf download hf://ETM-Network/jade-1.1/DETAILS.md
-
curl -L -o DETAILS.md https://huggingface.co/ETM-Network/jade-1.1/resolve/main/DETAILS.md
8.47 kB
| # Jade 1.1: details | |
| How the model is built and trained, and how it measures. The model card is [README.md](README.md); data credits are in [ATTRIBUTION.md](ATTRIBUTION.md). | |
| ## Architecture | |
| | Part | Parameters | Trained here | | |
| |---|---:|---| | |
| | Text encoder, diffusion transformer, length sampler, voice and language tables | 83.8 M | yes | | |
| | Supertonic 3 decoder (latents to waveform) | 25.3 M | no, frozen | | |
| | **Total at inference** | **109.1 M** | | | |
| - **Latents.** Supertonic 3's autoencoder latents (24 channels at 86.13 Hz), packed six frames at a time into 144-dimensional frames at 14.36 Hz, Supertonic's own packing. | |
| - **Text.** Unicode NFKD characters, with each language's punctuation mapped onto a shared set; 3,171 tokens, kanji included. English and German go through their own normalizers first (numbers, dates, abbreviations). | |
| - **Acoustic model.** A 12-block diffusion transformer, width 576, 9 attention heads. Every block cross-attends to the encoded text with length-normalised rotary positions, so text and audio align implicitly; there are no per-letter durations. | |
| - **Length.** A small flow-matching sampler predicts each piece's total length from the text, the voice and the language. | |
| - **Conditioning.** A learned voice embedding (9 voices) and a language embedding (23 languages). There is no voice cloning. | |
| - **Few-step sampling.** The network takes the step size as an input, so each step jumps a given fraction of the way from noise to speech. One set of weights serves 2 to 32 steps (the final stage was tuned for 2 to 7), with guidance built in. | |
| - **Decoder.** Supertonic 3's vocoder, exported from its released ONNX file. | |
| Long text is split into pieces of up to about 20 seconds at sentence and clause boundaries, and each piece is generated separately. | |
| ## Training data | |
| Public speech in 23 languages, from datasets whose licences allow commercial use (CC0, CC BY 4.0, CC BY 2.0, public domain, OpenMDW-1.1). The datasets, their licences and credits are in [ATTRIBUTION.md](ATTRIBUTION.md). | |
| 1. **Transcripts.** Each dataset's own transcripts where they have punctuation and casing. Otherwise punctuation and casing came from NVIDIA Granary-v2 labels or Whisper large-v3-turbo transcripts. Some parliament sessions without human transcripts use these machine labels as the text. | |
| 2. **Voices.** The nine voices are modelled on nine LibriVox volunteer readers. | |
| 3. **Filtering.** Clips were kept when Whisper read them back with a character error rate of at most 10 %. | |
| 4. **Cutting.** Clips of 0.4 to 21 seconds. Rows with digits were removed in every language except English and German, which is why the other languages don't read digits. | |
| ## Training | |
| **Teacher.** One RTX 4090, bf16, AdamW (β 0.9/0.99, weight decay 0.01), learning rate 4.2e-4 (length sampler 2.1e-4), 2,000 warm-up steps, EMA of the weights with decay 0.9999, batches of up to 2,300 latent frames. Languages were added in stages; each new language joined through zero-initialised table rows, so the earlier languages were unchanged at the moment of joining. | |
| | Stage | Steps | Data | | |
| |---|---|---| | |
| | English | 0 → 686,200 | English | | |
| | German added | → 886,200 | English and German, 50/50 by frames | | |
| | Seven more languages | → 1,386,200 | 9 languages | | |
| | Six more languages | → 1,566,200 | 15 languages | | |
| | Eight more languages | → 1,786,200 | 23 languages | | |
| | Cooldown | → 1,811,200 | learning rate falls linearly to 0.1× | | |
| The teacher samples in 16 steps with classifier-free guidance 4.0. It is not released. | |
| **Distillation.** The released model starts from the teacher and learns to land, in N jumps, where the teacher's 16 guided steps land from the same noise. | |
| 1. **Flow map** (7,000 steps). Each item draws a step count from 1 to 32 (2 to 7 most often). Targets come from the teacher or from the model's own two half-jumps, plus a loss on the whole 2–7-jump rollout against the teacher's output. | |
| 2. **Polish** (10,000 steps). An adversarial loss (a critic on the frozen teacher's features) and an anchor to the teacher's output from the same noise, with N from 2 to 7. | |
| Texts come from all 23 languages equally often; this stage needs no audio. | |
| **Quantization.** The low-bit weights are set by learned rounding, optimised layer by layer (2,000 steps per layer), and a bit width per layer chosen under a size budget. ONNX Runtime has no 3-bit kernels, so the ONNX files mix 2, 4 and 8 bits where MLX uses 3. | |
| ## Evaluation | |
| **Set.** 181 held-out lines, 8 per language (7 for English, German and Bulgarian), one voice each, never trained on. Rendered from the released ONNX files by the `etm-tts` package at 6 steps, exactly as a user gets them at that setting; the release default, q8 at 12 steps, is scored below the table. | |
| **Character error rate** of a Whisper transcript against the input text, on letters only: Whisper small.en for English, large-v3-turbo for the other languages; Japanese is compared on kana readings. On short lines one misread word moves a language's mean by several points. | |
| | | raw | q8 | q4 | q3 | | | raw | q8 | q4 | q3 | | |
| |---|---:|---:|---:|---:|---|---|---:|---:|---:|---:| | |
| | `en`¹ | 2.3 | 2.3 | 2.0 | 2.0 | | `es` | 0.7 | 0.7 | 0.7 | 0.7 | | |
| | `de` | 0.7 | 0.4 | 0.4 | 0.8 | | `pl` | 1.5 | 1.5 | 1.5 | 1.6 | | |
| | `it` | 0.0 | 0.0 | 0.0 | 0.0 | | `nl` | 0.0 | 0.0 | 0.3 | 0.6 | | |
| | `pt` | 0.0 | 0.0 | 0.0 | 0.0 | | `cs` | 0.3 | 1.2 | 0.0 | 0.0 | | |
| | `el` | 4.0 | 4.0 | 4.0 | 3.6 | | `hu` | 1.3 | 0.2 | 0.2 | 0.3 | | |
| | `bg` | 1.6 | 1.6 | 1.1 | 1.1 | | `lt` | 0.5 | 0.5 | 0.0 | 0.0 | | |
| | `ru` | 0.2 | 0.2 | 0.2 | 0.2 | | `hr` | 2.3 | 1.9 | 1.4 | 1.4 | | |
| | `ja` | 4.2 | 4.2 | 3.1 | 4.2 | | `sl` | 0.0 | 0.0 | 0.0 | 1.0 | | |
| | `hi` | 1.6 | 1.6 | 1.6 | 1.9 | | `et` | 0.9 | 1.6 | 2.9 | 1.0 | | |
| | `lv` | 0.0 | 0.0 | 0.0 | 0.0 | | `da` | 2.1 | 2.1 | 2.1 | 2.1 | | |
| | `fr` | 1.8 | 1.8 | 1.8 | 1.0 | | `sk` | 0.2 | 0.0 | 0.0 | 0.2 | | |
| | | | | | | | `ro` | 0.0 | 0.2 | 0.3 | 0.0 | | |
| All values in %. ¹ Without one line where Whisper stopped transcribing at a 1.5 s pause; the line is spoken in full (checked by transcribing the rest on its own). With it, English is 13.9 / 13.9 / 13.6 / 1.9 %; in q3 the pause is shorter. | |
| **The default, q8 at 12 steps**, on the same lines and seeds: 1.1 % CER over all 181 lines (English 2.0 %, the pause line read in full), 147 lines without an error, voice similarity 0.818. | |
| **Against full precision**, same lines and seeds: pitch-track correlation 0.978 (q8), 0.962 (q4), 0.924 (q3); similarity of each render to the raw render (WavLM embeddings) 0.998, 0.996, 0.993. Clicks, clipping and noise floor stay at raw's level (q3's floor is 0.9 dB higher). | |
| **Steps.** The distilled model against the teacher at 64 steps, on a 181-line set of the same kind (PyTorch renders): | |
| | Sampler | Pitch match | Mel L1 | CER | | |
| |---|---:|---:|---:| | |
| | Teacher, 64 steps | reference | reference | 0.7 % | | |
| | Teacher, 16 steps | 0.890 | 0.733 | 1.0 % | | |
| | Distilled, 4 steps | 0.871 | 0.928 | 1.1 % | | |
| | Distilled, 6 steps | 0.875 | 0.886 | 1.1 % | | |
| In our blind listening on short clips, 4 steps had occasional pitch jumps on single words and 6 steps mostly removed them. Listening to minutes of audiobook text later showed more: at q4 and q3 with few steps the pitch errors become noticeable over long passages, while q8 at 12 steps holds up. The release default is therefore q8 at 12 steps. | |
| **More steps.** The final distillation stage was tuned for 2 to 7 steps, but the model also runs at more. The released q4 files through `etm-tts`, on the same 181 lines (automatic measures, not checked by ear): | |
| | Steps | CER | Voice similarity | Pitch jumps per minute | Clicks per minute | | |
| |---:|---:|---:|---:|---:| | |
| | 6 | 1.5 % | 0.804 | 13.3 | 3.3 | | |
| | 12 | 0.9 % | 0.811 | 9.7 | 4.6 | | |
| | 20 | 0.7 % | 0.812 | 11.3 | 4.8 | | |
| | 32 | 1.0 % | 0.809 | 15.6 | 5.3 | | |
| Most of the CER difference is the English line Whisper stops reading at a pause at 6 steps (see the evaluation above). Time grows with the step count: 32 steps take about five times as long as 6. | |
| **Voices.** Each voice read the same sentences, and every clip was matched to the nearest of the nine voice centroids (WavLM speaker embeddings, chance 11 %). On full sentences the teacher's clips land on their own voice 91.5 % of the time; the readers' real recordings reach 98.6 %. On two- or three-word phrases the distilled model reaches 46 % (real recordings cut as short: 68 %), so short replies can sound alike across voices. Tested on the first 15 languages. | |