tuanpm commited on
Commit
27f594b
Β·
verified Β·
1 Parent(s): 00a0817

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +64 -52
README.md CHANGED
@@ -1,83 +1,95 @@
1
  ---
2
  license: cc-by-nc-4.0
3
  language:
4
- - vi
5
  pipeline_tag: text-to-speech
6
  tags:
7
- - text-to-speech
8
- - tts
9
- - multilingual
10
- - esp32
11
- - esp32-p4
12
- - esp32-s3
13
- - microcontroller
14
- - int8
15
- - on-device
16
- - tinyml
17
- library_name: vtts
 
 
18
  ---
19
 
20
- # vtts β€” voices for ESP32 microcontrollers
21
 
22
- Tiny int8 text-to-speech voices that run **entirely on an ESP32** with the
23
- [vtts](https://github.com/tuanpmt/vtts) C99 runtime: text normalization, pronunciation, the neural
24
- voice and the iSTFT vocoder all run on the chip. Listen to audio generated on an ESP32-P4:
25
- **https://tuanpmt.github.io/vtts/**
26
 
27
- Each voice lives in its own folder and comes in two tiers: **quality** (M, for ESP32-P4) and
28
- **lite** (S, also runs on ESP32-S3 with 8 MB PSRAM).
29
 
30
- ## Voice catalog
31
 
32
- | Folder | Languages | Quality (P4) | Lite (S3) | Status | License |
33
- |---|---|---|---|---|---|
34
- | [`vi/`](vi) | πŸ‡»πŸ‡³ Vietnamese | `vtts-vi-m-v0.2.q8.vtts` (1.73M params, 2.6 MB) | `vtts-vi-s-v0.1.q8.vtts` (0.77M, 1.1 MB) | released | CC BY-NC 4.0 |
35
- | `vi-en-nam-1/` | πŸ‡»πŸ‡³ vi + πŸ‡ΊπŸ‡Έ en (male) | M | S | training | β€” |
36
- | `vi-en-nu-10/` | πŸ‡»πŸ‡³ vi + πŸ‡ΊπŸ‡Έ en (female) | M | S | training | β€” |
37
- | `en/`, `zh/`, `zh-en/` | πŸ‡ΊπŸ‡Έ πŸ‡¨πŸ‡³ | M | S | planned | β€” |
38
 
39
- Every folder has a `SHA256SUMS` file.
 
 
 
 
 
 
 
40
 
41
- ## Performance
42
 
43
- | Board | Lite RTF | Quality RTF |
44
- |---|---:|---:|
45
- | ESP32-P4 (360 MHz, PIE SIMD, 2 cores) | 0.20–0.24 | 0.34–0.42 |
46
- | ESP32-S3 (240 MHz, 8 MB PSRAM, 2 cores) | 0.42–0.47 | 0.85–0.99 |
47
 
48
- RTF below 1 means faster than real time. Speech starts after 0.3–2.4 s (streaming).
 
 
 
49
 
50
- ## Vietnamese voice quality
51
 
52
- 50 out-of-domain sentences, audio from the int8 C runtime; CER from PhoWhisper-large (tones included).
53
 
54
- | System | CER ↓ | UTMOS ↑ |
55
- |---|---:|---:|
56
- | vtts `vi` M v0.2 | 0.012 | 2.19 |
57
- | vtts `vi` S v0.1 | 0.023 | 1.86 |
58
- | sanoTTS `vi` | 0.064 | 1.49 |
59
- | Piper `vi_VN-25hours_single` | 0.089 | 2.06 |
 
60
 
61
- ## Use
62
 
63
  ```bash
 
64
  huggingface-cli download tuanpm/vtts --include "vi/*" --local-dir models
 
 
65
  git clone https://github.com/tuanpmt/vtts && cd vtts/firmware/esp32
66
  BOARD=nano ./build.sh ../../../models/vi/vtts-vi-s-v0.1.q8.vtts ../../../models/vi/vtts-vi-m-v0.2.q8.vtts -p PORT flash monitor
67
  ```
68
 
69
- `BOARD` is `panel` (ESP32-P4-86-Panel), `nano` (ESP32-P4-NANO) or `s3` (ESP32-S3-Touch-LCD-2.8).
70
 
71
- ## Training
72
 
73
- Each voice is distilled from a VITS teacher into 0.8–1.7M-parameter students (duration, acoustic +
74
- pitch, Vocos-style ConvNeXt + iSTFT decoder) and quantized to int8 per output channel.
75
- The `vi` voice is a single male Northern speaker with a spelling-based token set.
 
 
76
 
77
  ## License and limitations
78
 
79
- - `vi/` is **CC BY-NC 4.0, non-commercial only**: its teacher was initialized from a non-commercial
80
- Piper checkpoint and the commercial terms of its training corpus are unclear. The runtime code is
81
- Apache-2.0; that license does not cover model files. Each folder states its own license.
82
- - Prototype quality: intonation is still somewhat flat; rare syllables may be misread.
83
- - Do not use the voices to impersonate people; label generated audio as synthetic.
 
 
 
 
 
 
 
1
  ---
2
  license: cc-by-nc-4.0
3
  language:
4
+ - vi
5
  pipeline_tag: text-to-speech
6
  tags:
7
+ - text-to-speech
8
+ - tts
9
+ - vietnamese
10
+ - esp32
11
+ - esp32-p4
12
+ - esp32-s3
13
+ - microcontroller
14
+ - embedded
15
+ - on-device
16
+ - int8
17
+ - vits
18
+ - vocos
19
+ - distillation
20
  ---
21
 
22
+ # vtts: text-to-speech models for ESP32 microcontrollers
23
 
24
+ Tiny int8 voices for [vtts](https://github.com/tuanpmt/vtts), a text-to-speech engine that runs entirely on ESP32 chips: text normalization, grapheme-to-phoneme, the acoustic model and the iSTFT vocoder all run on the microcontroller, with no network. Listen to samples generated on the chip on the [website](https://tuanpmt.github.io/vtts/).
 
 
 
25
 
26
+ > These files are not PyTorch or Transformers checkpoints. They are `.vtts` files for the vtts C runtime ([format](https://github.com/tuanpmt/vtts/blob/main/docs/model-format.md)), meant to be flashed with the [ESP32 firmware](https://github.com/tuanpmt/vtts/tree/main/firmware/esp32) or run with the host CLI.
 
27
 
28
+ ## Repository layout
29
 
30
+ One folder per voice. Each voice comes in two tiers: a **quality** model (M) for the ESP32-P4 and a **lite** model (S) that also fits the ESP32-S3.
 
 
 
 
 
31
 
32
+ | Folder | Voice | Languages | Status |
33
+ |---|---|---|---|
34
+ | `vi/` | Vietnamese prototype | Vietnamese | **Released** (v0.3.0): `vtts-vi-m-v0.2.q8.vtts`, `vtts-vi-s-v0.1.q8.vtts` |
35
+ | `vi-en-nam-1/` | `nam-1` (male) | Vietnamese + English, code-switching | Training (teacher in progress); not uploaded yet |
36
+ | `vi-en-nu-10/` | `nu-10` (female) | Vietnamese + English, code-switching | Training (teacher in progress); not uploaded yet |
37
+ | `en/` | English | English | Planned |
38
+ | `zh/` | Mandarin Chinese | Chinese | Planned |
39
+ | `zh-en/` | Chinese + English | Chinese + English, code-switching | Planned |
40
 
41
+ The license above applies to the files released so far (`vi/`). Each future voice will state its own license in its folder.
42
 
43
+ ## Released models (`vi/`)
 
 
 
44
 
45
+ | File | Tier | Params | Size | ESP32-P4 RTF | ESP32-S3 RTF |
46
+ |---|---|---:|---:|---:|---:|
47
+ | `vtts-vi-m-v0.2.q8.vtts` | Quality (M-vocos v0.2) | 1.73M | 2.6 MB | 0.34–0.42 | 0.85–0.99 |
48
+ | `vtts-vi-s-v0.1.q8.vtts` | Lite (S-vocos v0.1) | 0.77M | 1.1 MB | 0.20–0.24 | 0.42–0.47 |
49
 
50
+ RTF below 1 is faster than real time; speech starts after 0.3–1.2 s on the ESP32-P4 and 0.8–2.4 s on the ESP32-S3 (streaming). Measured on Waveshare ESP32-P4-NANO / ESP32-P4-86-Panel and ESP32-S3-Touch-LCD-2.8 (8 MB PSRAM required on the S3). Details: [performance report](https://github.com/tuanpmt/vtts/blob/main/docs/reports/performance.md).
51
 
52
+ Quality on 50 out-of-domain Vietnamese sentences (CER from PhoWhisper-large transcripts, tones included; audio from the int8 C runtime):
53
 
54
+ | System | Params | CER ↓ | UTMOS ↑ |
55
+ |---|---:|---:|---:|
56
+ | vtts M-vocos v0.2 | 1.73M | 0.012 | 2.19 |
57
+ | vtts S-vocos v0.1 | 0.77M | 0.023 | 1.86 |
58
+ | Teacher VITS (not released) | 23.7M | 0.015 | 2.50 |
59
+ | sanoTTS `vi` | 1.57M | 0.064 | 1.49 |
60
+ | Piper `vi_VN-25hours_single` | β€” | 0.089 | 2.06 |
61
 
62
+ ## Usage
63
 
64
  ```bash
65
+ # download the released Vietnamese models
66
  huggingface-cli download tuanpm/vtts --include "vi/*" --local-dir models
67
+
68
+ # flash an ESP32-P4-NANO (ESP-IDF v6.1); BOARD can be panel, nano or s3
69
  git clone https://github.com/tuanpmt/vtts && cd vtts/firmware/esp32
70
  BOARD=nano ./build.sh ../../../models/vi/vtts-vi-s-v0.1.q8.vtts ../../../models/vi/vtts-vi-m-v0.2.q8.vtts -p PORT flash monitor
71
  ```
72
 
73
+ Type a Vietnamese sentence on the serial console and the board speaks it. The same files are attached to the [GitHub release v0.3.0](https://github.com/tuanpmt/vtts/releases/tag/v0.3.0).
74
 
75
+ ## How the models are made
76
 
77
+ - **Front-end:** rule-based, in Python (training) and C (device) with identical output. Vietnamese uses a spelling-based G2P (token set v1, 65 tokens, 6 tones) and reads numbers, dates, times and abbreviations. Bilingual voices use token set v2 (107 tokens: v1 + English ARPAbet and stress) with an English front-end (CMUdict, learned letter-to-sound rules, normalizer) and per-word language tagging.
78
+ - **Distillation:** a VITS teacher (~24M parameters) produces durations, latents and audio; a small student (duration + acoustic model with pitch/energy predictor + Vocos-style ConvNeXt/iSTFT decoder) learns from it in three stages and is quantized to int8.
79
+ - **Training data:**
80
+ - `vi/`: InfoRe Technology public dataset No.1 (about 25 h of Vietnamese read speech, one speaker), filtered to 22.2 h for training.
81
+ - `vi-en-nam-1/`, `vi-en-nu-10/`: about 98 h of speech per voice, synthesized with VoxCPM2 (Apache-2.0) from designed voices (not cloned from real people), with sentences chosen to cover rare Vietnamese syllables, numbers, English and code-switched text.
82
 
83
  ## License and limitations
84
 
85
+ - **`vi/` models: CC BY-NC 4.0, non-commercial use only.** The teacher was initialized from a Piper checkpoint whose training data allows only non-commercial use, and the commercial terms of the InfoRe1 corpus are unclear.
86
+ - The vtts code is Apache-2.0; that license does not apply to the model files.
87
+ - The Vietnamese prototype is trained on one speaker reading in a fairly even tone, so intonation is flatter than natural speech. Short device prompts and foreign names are the hardest cases.
88
+ - On-device text normalization covers numbers, percent, dates and times; e-mail addresses, URLs, phone numbers, currency and units are handled only by the Python front-end.
89
+ - Label audio produced with these models as AI-generated, and do not use them to impersonate real people.
90
+
91
+ ## Links
92
+
93
+ - Code, firmware and documentation: [github.com/tuanpmt/vtts](https://github.com/tuanpmt/vtts)
94
+ - Website with on-chip samples: [tuanpmt.github.io/vtts](https://tuanpmt.github.io/vtts/)
95
+ - Changelog: [CHANGELOG.md](https://github.com/tuanpmt/vtts/blob/main/CHANGELOG.md)