Upload README.md
Browse files
README.md
CHANGED
|
@@ -1,83 +1,95 @@
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-4.0
|
| 3 |
language:
|
| 4 |
-
- vi
|
| 5 |
pipeline_tag: text-to-speech
|
| 6 |
tags:
|
| 7 |
-
- text-to-speech
|
| 8 |
-
- tts
|
| 9 |
-
-
|
| 10 |
-
- esp32
|
| 11 |
-
- esp32-p4
|
| 12 |
-
- esp32-s3
|
| 13 |
-
- microcontroller
|
| 14 |
-
-
|
| 15 |
-
- on-device
|
| 16 |
-
-
|
| 17 |
-
|
|
|
|
|
|
|
| 18 |
---
|
| 19 |
|
| 20 |
-
# vtts
|
| 21 |
|
| 22 |
-
Tiny int8 text-to-speech
|
| 23 |
-
[vtts](https://github.com/tuanpmt/vtts) C99 runtime: text normalization, pronunciation, the neural
|
| 24 |
-
voice and the iSTFT vocoder all run on the chip. Listen to audio generated on an ESP32-P4:
|
| 25 |
-
**https://tuanpmt.github.io/vtts/**
|
| 26 |
|
| 27 |
-
|
| 28 |
-
**lite** (S, also runs on ESP32-S3 with 8 MB PSRAM).
|
| 29 |
|
| 30 |
-
##
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|---|---|---|---|---|---|
|
| 34 |
-
| [`vi/`](vi) | π»π³ Vietnamese | `vtts-vi-m-v0.2.q8.vtts` (1.73M params, 2.6 MB) | `vtts-vi-s-v0.1.q8.vtts` (0.77M, 1.1 MB) | released | CC BY-NC 4.0 |
|
| 35 |
-
| `vi-en-nam-1/` | π»π³ vi + πΊπΈ en (male) | M | S | training | β |
|
| 36 |
-
| `vi-en-nu-10/` | π»π³ vi + πΊπΈ en (female) | M | S | training | β |
|
| 37 |
-
| `en/`, `zh/`, `zh-en/` | πΊπΈ π¨π³ | M | S | planned | β |
|
| 38 |
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
-
|
| 44 |
-
|---|---:|---:|
|
| 45 |
-
| ESP32-P4 (360 MHz, PIE SIMD, 2 cores) | 0.20β0.24 | 0.34β0.42 |
|
| 46 |
-
| ESP32-S3 (240 MHz, 8 MB PSRAM, 2 cores) | 0.42β0.47 | 0.85β0.99 |
|
| 47 |
|
| 48 |
-
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
-
|
| 51 |
|
| 52 |
-
50 out-of-domain sentences
|
| 53 |
|
| 54 |
-
| System | CER β | UTMOS β |
|
| 55 |
-
|---|---:|---:|
|
| 56 |
-
| vtts
|
| 57 |
-
| vtts
|
| 58 |
-
|
|
| 59 |
-
|
|
|
|
|
| 60 |
|
| 61 |
-
##
|
| 62 |
|
| 63 |
```bash
|
|
|
|
| 64 |
huggingface-cli download tuanpm/vtts --include "vi/*" --local-dir models
|
|
|
|
|
|
|
| 65 |
git clone https://github.com/tuanpmt/vtts && cd vtts/firmware/esp32
|
| 66 |
BOARD=nano ./build.sh ../../../models/vi/vtts-vi-s-v0.1.q8.vtts ../../../models/vi/vtts-vi-m-v0.2.q8.vtts -p PORT flash monitor
|
| 67 |
```
|
| 68 |
|
| 69 |
-
|
| 70 |
|
| 71 |
-
##
|
| 72 |
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
|
|
|
|
|
|
| 76 |
|
| 77 |
## License and limitations
|
| 78 |
|
| 79 |
-
- `vi/`
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
-
|
| 83 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-4.0
|
| 3 |
language:
|
| 4 |
+
- vi
|
| 5 |
pipeline_tag: text-to-speech
|
| 6 |
tags:
|
| 7 |
+
- text-to-speech
|
| 8 |
+
- tts
|
| 9 |
+
- vietnamese
|
| 10 |
+
- esp32
|
| 11 |
+
- esp32-p4
|
| 12 |
+
- esp32-s3
|
| 13 |
+
- microcontroller
|
| 14 |
+
- embedded
|
| 15 |
+
- on-device
|
| 16 |
+
- int8
|
| 17 |
+
- vits
|
| 18 |
+
- vocos
|
| 19 |
+
- distillation
|
| 20 |
---
|
| 21 |
|
| 22 |
+
# vtts: text-to-speech models for ESP32 microcontrollers
|
| 23 |
|
| 24 |
+
Tiny int8 voices for [vtts](https://github.com/tuanpmt/vtts), a text-to-speech engine that runs entirely on ESP32 chips: text normalization, grapheme-to-phoneme, the acoustic model and the iSTFT vocoder all run on the microcontroller, with no network. Listen to samples generated on the chip on the [website](https://tuanpmt.github.io/vtts/).
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
> These files are not PyTorch or Transformers checkpoints. They are `.vtts` files for the vtts C runtime ([format](https://github.com/tuanpmt/vtts/blob/main/docs/model-format.md)), meant to be flashed with the [ESP32 firmware](https://github.com/tuanpmt/vtts/tree/main/firmware/esp32) or run with the host CLI.
|
|
|
|
| 27 |
|
| 28 |
+
## Repository layout
|
| 29 |
|
| 30 |
+
One folder per voice. Each voice comes in two tiers: a **quality** model (M) for the ESP32-P4 and a **lite** model (S) that also fits the ESP32-S3.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
+
| Folder | Voice | Languages | Status |
|
| 33 |
+
|---|---|---|---|
|
| 34 |
+
| `vi/` | Vietnamese prototype | Vietnamese | **Released** (v0.3.0): `vtts-vi-m-v0.2.q8.vtts`, `vtts-vi-s-v0.1.q8.vtts` |
|
| 35 |
+
| `vi-en-nam-1/` | `nam-1` (male) | Vietnamese + English, code-switching | Training (teacher in progress); not uploaded yet |
|
| 36 |
+
| `vi-en-nu-10/` | `nu-10` (female) | Vietnamese + English, code-switching | Training (teacher in progress); not uploaded yet |
|
| 37 |
+
| `en/` | English | English | Planned |
|
| 38 |
+
| `zh/` | Mandarin Chinese | Chinese | Planned |
|
| 39 |
+
| `zh-en/` | Chinese + English | Chinese + English, code-switching | Planned |
|
| 40 |
|
| 41 |
+
The license above applies to the files released so far (`vi/`). Each future voice will state its own license in its folder.
|
| 42 |
|
| 43 |
+
## Released models (`vi/`)
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
+
| File | Tier | Params | Size | ESP32-P4 RTF | ESP32-S3 RTF |
|
| 46 |
+
|---|---|---:|---:|---:|---:|
|
| 47 |
+
| `vtts-vi-m-v0.2.q8.vtts` | Quality (M-vocos v0.2) | 1.73M | 2.6 MB | 0.34β0.42 | 0.85β0.99 |
|
| 48 |
+
| `vtts-vi-s-v0.1.q8.vtts` | Lite (S-vocos v0.1) | 0.77M | 1.1 MB | 0.20β0.24 | 0.42β0.47 |
|
| 49 |
|
| 50 |
+
RTF below 1 is faster than real time; speech starts after 0.3β1.2 s on the ESP32-P4 and 0.8β2.4 s on the ESP32-S3 (streaming). Measured on Waveshare ESP32-P4-NANO / ESP32-P4-86-Panel and ESP32-S3-Touch-LCD-2.8 (8 MB PSRAM required on the S3). Details: [performance report](https://github.com/tuanpmt/vtts/blob/main/docs/reports/performance.md).
|
| 51 |
|
| 52 |
+
Quality on 50 out-of-domain Vietnamese sentences (CER from PhoWhisper-large transcripts, tones included; audio from the int8 C runtime):
|
| 53 |
|
| 54 |
+
| System | Params | CER β | UTMOS β |
|
| 55 |
+
|---|---:|---:|---:|
|
| 56 |
+
| vtts M-vocos v0.2 | 1.73M | 0.012 | 2.19 |
|
| 57 |
+
| vtts S-vocos v0.1 | 0.77M | 0.023 | 1.86 |
|
| 58 |
+
| Teacher VITS (not released) | 23.7M | 0.015 | 2.50 |
|
| 59 |
+
| sanoTTS `vi` | 1.57M | 0.064 | 1.49 |
|
| 60 |
+
| Piper `vi_VN-25hours_single` | β | 0.089 | 2.06 |
|
| 61 |
|
| 62 |
+
## Usage
|
| 63 |
|
| 64 |
```bash
|
| 65 |
+
# download the released Vietnamese models
|
| 66 |
huggingface-cli download tuanpm/vtts --include "vi/*" --local-dir models
|
| 67 |
+
|
| 68 |
+
# flash an ESP32-P4-NANO (ESP-IDF v6.1); BOARD can be panel, nano or s3
|
| 69 |
git clone https://github.com/tuanpmt/vtts && cd vtts/firmware/esp32
|
| 70 |
BOARD=nano ./build.sh ../../../models/vi/vtts-vi-s-v0.1.q8.vtts ../../../models/vi/vtts-vi-m-v0.2.q8.vtts -p PORT flash monitor
|
| 71 |
```
|
| 72 |
|
| 73 |
+
Type a Vietnamese sentence on the serial console and the board speaks it. The same files are attached to the [GitHub release v0.3.0](https://github.com/tuanpmt/vtts/releases/tag/v0.3.0).
|
| 74 |
|
| 75 |
+
## How the models are made
|
| 76 |
|
| 77 |
+
- **Front-end:** rule-based, in Python (training) and C (device) with identical output. Vietnamese uses a spelling-based G2P (token set v1, 65 tokens, 6 tones) and reads numbers, dates, times and abbreviations. Bilingual voices use token set v2 (107 tokens: v1 + English ARPAbet and stress) with an English front-end (CMUdict, learned letter-to-sound rules, normalizer) and per-word language tagging.
|
| 78 |
+
- **Distillation:** a VITS teacher (~24M parameters) produces durations, latents and audio; a small student (duration + acoustic model with pitch/energy predictor + Vocos-style ConvNeXt/iSTFT decoder) learns from it in three stages and is quantized to int8.
|
| 79 |
+
- **Training data:**
|
| 80 |
+
- `vi/`: InfoRe Technology public dataset No.1 (about 25 h of Vietnamese read speech, one speaker), filtered to 22.2 h for training.
|
| 81 |
+
- `vi-en-nam-1/`, `vi-en-nu-10/`: about 98 h of speech per voice, synthesized with VoxCPM2 (Apache-2.0) from designed voices (not cloned from real people), with sentences chosen to cover rare Vietnamese syllables, numbers, English and code-switched text.
|
| 82 |
|
| 83 |
## License and limitations
|
| 84 |
|
| 85 |
+
- **`vi/` models: CC BY-NC 4.0, non-commercial use only.** The teacher was initialized from a Piper checkpoint whose training data allows only non-commercial use, and the commercial terms of the InfoRe1 corpus are unclear.
|
| 86 |
+
- The vtts code is Apache-2.0; that license does not apply to the model files.
|
| 87 |
+
- The Vietnamese prototype is trained on one speaker reading in a fairly even tone, so intonation is flatter than natural speech. Short device prompts and foreign names are the hardest cases.
|
| 88 |
+
- On-device text normalization covers numbers, percent, dates and times; e-mail addresses, URLs, phone numbers, currency and units are handled only by the Python front-end.
|
| 89 |
+
- Label audio produced with these models as AI-generated, and do not use them to impersonate real people.
|
| 90 |
+
|
| 91 |
+
## Links
|
| 92 |
+
|
| 93 |
+
- Code, firmware and documentation: [github.com/tuanpmt/vtts](https://github.com/tuanpmt/vtts)
|
| 94 |
+
- Website with on-chip samples: [tuanpmt.github.io/vtts](https://tuanpmt.github.io/vtts/)
|
| 95 |
+
- Changelog: [CHANGELOG.md](https://github.com/tuanpmt/vtts/blob/main/CHANGELOG.md)
|