ctrltokyo commited on
Commit
cba22ce
·
verified ·
1 Parent(s): a42c72e

Ship the Rust engine: Python wheels, CLI, C library, WASM demo

Browse files
.gitattributes CHANGED
@@ -44,3 +44,12 @@ samples/M1-numbers.wav filter=lfs diff=lfs merge=lfs -text
44
  samples/M1-polite.wav filter=lfs diff=lfs merge=lfs -text
45
  samples/M1-question.wav filter=lfs diff=lfs merge=lfs -text
46
  samples/M1-story.wav filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
44
  samples/M1-polite.wav filter=lfs diff=lfs merge=lfs -text
45
  samples/M1-question.wav filter=lfs diff=lfs merge=lfs -text
46
  samples/M1-story.wav filter=lfs diff=lfs merge=lfs -text
47
+ bin/linux-aarch64/eeh filter=lfs diff=lfs merge=lfs -text
48
+ bin/linux-x86_64/eeh filter=lfs diff=lfs merge=lfs -text
49
+ bin/macos-arm64/eeh filter=lfs diff=lfs merge=lfs -text
50
+ lib/linux-aarch64/libeeh_ffi.so filter=lfs diff=lfs merge=lfs -text
51
+ lib/linux-x86_64/libeeh_ffi.so filter=lfs diff=lfs merge=lfs -text
52
+ lib/macos-arm64/libeeh_ffi.dylib filter=lfs diff=lfs merge=lfs -text
53
+ python/eeh-0.1.0-py3-none-macosx_11_0_arm64.whl filter=lfs diff=lfs merge=lfs -text
54
+ python/eeh-0.1.0-py3-none-manylinux_2_28_aarch64.whl filter=lfs diff=lfs merge=lfs -text
55
+ python/eeh-0.1.0-py3-none-manylinux_2_28_x86_64.whl filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -2,11 +2,12 @@
2
  language: ja
3
  license: apache-2.0
4
  pipeline_tag: text-to-speech
5
- library_name: pytorch
6
  tags:
7
  - text-to-speech
8
  - japanese
9
- - onnx
 
10
  - tiny
11
  ---
12
 
@@ -32,7 +33,7 @@ trained from scratch on 18.6 hours of speech by
32
  | Voices | F1 (female, late 20s, calm and clear), M1 (male, early 30s, warm), Tokyo Japanese |
33
  | Output | 24 kHz mono |
34
  | Input | Japanese text (kanji, kana, numbers), read with OpenJTalk |
35
- | Speed | About 50x real time in a browser tab on one thread, first audio in under 0.1 s (Apple M4 Max) |
36
  | Pretrained weights in the model | None |
37
  | License | Apache-2.0 |
38
 
@@ -50,107 +51,87 @@ trained from scratch on 18.6 hours of speech by
50
 
51
  | Path | Contents |
52
  |---|---|
53
- | `eeh.pt` | PyTorch weights, float32 (18.2 MB) |
54
- | `eeh.safetensors` | The same weights for the Rust/WebAssembly engine |
55
- | `onnx/model.onnx`, `onnx/config.json` | ONNX export, float32, opset 17 (21.4 MB); symbols, voices, inputs and outputs |
56
- | `tiny_voca/` | The Python package for inference: front end, model, `infer` |
57
- | `demo/` | The browser demo: the whole engine compiled from Rust to WebAssembly, the weights and the dictionary |
 
58
  | `samples/` | The audio on this page |
59
  | `eval/report.json`, `eval/metrics.jsonl` | Evaluation report and the training log |
60
  | `LICENSE` | Apache-2.0 |
61
 
62
  ## Usage
63
 
64
- ### Python
 
 
 
65
 
66
  ```bash
67
- pip install torch numpy soundfile "pyopenjtalk>=0.4,<0.5"
68
  hf download ctrltokyo/eeh-1 --local-dir eeh
69
  cd eeh
 
 
 
70
 
71
- python -m tiny_voca.infer \
72
- --checkpoint eeh.pt \
73
- --voice F1 \
74
- --text "こんにちは。今日はいい天気ですね。" \
75
- --output hello.wav
76
  ```
77
 
78
- | Option | Default | Meaning |
79
- |---|---|---|
80
- | `--text-file` | | Read the text from a UTF-8 file instead of `--text` |
81
- | `--voice` | `F1` | `F1` or `M1` |
82
- | `--speed` | 1.0 | Speaking rate |
83
- | `--pitch-shift` | 0 | Added to log F0. 0.058 is about one semitone. |
84
- | `--pitch-range` | 1.0 | Scales pitch movement around the utterance mean |
85
 
86
- Text of any length works. It is split into sentences at 。!? and read one
87
- sentence at a time; a quote followed by と or って stays with its sentence, and
88
- a sentence too long for one call is cut after 、. A text without a final mark
89
- gets 。. The saved audio never peaks above 0.99. In Python,
90
- `tiny_voca.infer.speak_text` does the same.
91
 
92
- ### In a browser
 
 
 
 
 
 
93
 
94
  ```bash
95
- hf download ctrltokyo/eeh-1 --local-dir eeh
96
- python -m http.server --directory eeh/demo 8000
97
  ```
98
 
99
- Then open `http://localhost:8000`. The page runs eeh's own engine, written in
100
- Rust and compiled to WebAssembly: a Japanese front end that reads text exactly
101
- as OpenJTalk does (identical on more than 500,000 test texts), and the model.
102
- It runs in a Web Worker and plays long text sentence by sentence as it is made;
103
- the first sentence starts in under 0.1 s. The first visit downloads about 26 MB
104
- (gzip): 17 MB of weights, 9 MB of dictionary and 0.4 MB of program, all kept by
105
- the browser afterwards.
106
-
107
- ### ONNX
108
 
109
- The graph takes phoneme ids and tone values and returns the waveform. Text is
110
- converted outside the graph with `tiny_voca.text.phonemize`.
 
 
 
111
 
112
- ```python
113
- import numpy as np
114
- import onnxruntime as ort
115
- import soundfile as sf
116
- from tiny_voca.text import phonemize
117
-
118
- phones = phonemize("こんにちは。")
119
- one = lambda value: np.array([value], dtype=np.float32)
120
- session = ort.InferenceSession("onnx/model.onnx")
121
- audio, durations = session.run(None, {
122
- "tokens": np.array([phones.ids], dtype=np.int64),
123
- "stress": np.array([phones.stress], dtype=np.int64), # pitch-accent tone per token
124
- "voice": np.array([0], dtype=np.int64), # 0 = F1, 1 = M1
125
- "speed": one(1.0), "pitch_shift": one(0.0), "pitch_range": one(1.0),
126
- })
127
- audio = audio[0] * min(1.0, 0.99 / np.abs(audio[0]).max())
128
- sf.write("hello.wav", audio, 24000)
129
- ```
130
 
131
- | Input | Type | Shape | Meaning |
132
- |---|---|---|---|
133
- | `tokens` | int64 | [1, N] | Symbol ids (`symbols` in `onnx/config.json`), with the start and end tokens |
134
- | `stress` | int64 | [1, N] | Tokyo pitch-accent tone of the token's mora: 0 not a phone, 1 low, 2 high |
135
- | `voice` | int64 | [1] | 0 = F1, 1 = M1 |
136
- | `speed`, `pitch_shift`, `pitch_range` | float32 | [1] | As in the Python options |
137
 
138
- | Output | Type | Shape | Meaning |
139
- |---|---|---|---|
140
- | `audio` | float32 | [1, samples] | Waveform at 24 kHz |
141
- | `durations` | int64 | [1, N] | Length of each token in frames of 256 samples |
142
 
143
- One call reads one utterance, up to 512 tokens and 3,000 frames (32 s).
 
 
 
144
 
145
  ## How it works
146
 
147
- **Front end.** [OpenJTalk](https://open-jtalk.sourceforge.net/) with the NAIST
148
- Japanese Dictionary (the build in [pyopenjtalk](https://github.com/r9y9/pyopenjtalk)
149
- 0.4) reads the text: it splits it into words, picks each kanji's reading,
150
- expands numbers, joins words into accent phrases and assigns their accent types.
151
- eeh turns its labels into tokens:
152
 
153
- - one token per OpenJTalk phoneme (`k`, `sh`, `a`, `N` for ん, `cl` for っ);
154
  a long vowel is two tokens, one per mora;
155
  - devoiced vowels (the u of です, the i of した) are their own tokens `A I U E O`;
156
  - `<sp>` marks a boundary between accent phrases, and punctuation is kept;
 
2
  language: ja
3
  license: apache-2.0
4
  pipeline_tag: text-to-speech
5
+ library_name: eeh
6
  tags:
7
  - text-to-speech
8
  - japanese
9
+ - rust
10
+ - webassembly
11
  - tiny
12
  ---
13
 
 
33
  | Voices | F1 (female, late 20s, calm and clear), M1 (male, early 30s, warm), Tokyo Japanese |
34
  | Output | 24 kHz mono |
35
  | Input | Japanese text (kanji, kana, numbers), read with OpenJTalk |
36
+ | Speed | 120x real time on one CPU thread, 280x on four (Apple M4 Max); about 50x in a browser tab, first audio in under 0.1 s |
37
  | Pretrained weights in the model | None |
38
  | License | Apache-2.0 |
39
 
 
51
 
52
  | Path | Contents |
53
  |---|---|
54
+ | `eeh.safetensors` | Weights, float32 (18.2 MB) |
55
+ | `eeh-dict.bin` | The Japanese dictionary the front end reads (36 MB) |
56
+ | `python/` | Python wheels for macOS (arm64) and Linux (aarch64, x86_64) |
57
+ | `bin/<platform>/eeh` | Command-line program |
58
+ | `lib/<platform>/`, `include/eeh.h` | C library and header, for any language with a C FFI |
59
+ | `demo/` | The browser demo: the whole engine compiled to WebAssembly, the weights and the dictionary |
60
  | `samples/` | The audio on this page |
61
  | `eval/report.json`, `eval/metrics.jsonl` | Evaluation report and the training log |
62
  | `LICENSE` | Apache-2.0 |
63
 
64
  ## Usage
65
 
66
+ eeh runs on its own engine, written in Rust: a Japanese front end that reads
67
+ text exactly as OpenJTalk does (identical on more than 500,000 test texts) and
68
+ the model. The same engine is behind the Python package, the command-line
69
+ program, the C library and the browser demo.
70
 
71
  ```bash
 
72
  hf download ctrltokyo/eeh-1 --local-dir eeh
73
  cd eeh
74
+ ```
75
+
76
+ ### Python
77
 
78
+ ```bash
79
+ pip install python/eeh-0.1.0-py3-none-<your platform>.whl
 
 
 
80
  ```
81
 
82
+ ```python
83
+ import eeh, soundfile
 
 
 
 
 
84
 
85
+ synth = eeh.Synthesizer("eeh.safetensors", "eeh-dict.bin")
86
+ audio = synth.speak("こんにちは。今日はいい天気ですね。", voice="F1") # float32 array, 24 kHz
87
+ soundfile.write("hello.wav", audio, synth.sample_rate)
88
+ ```
 
89
 
90
+ `speak` takes `voice` (`F1` or `M1`), `speed` (1.0), `pitch_shift` (0; 0.058
91
+ is about one semitone), `pitch_range` (1.0) and `seed`. `speak_each(text,
92
+ callback)` hands over each sentence as soon as it is made, for streaming. The
93
+ package has no dependencies; it returns a NumPy array when NumPy is installed.
94
+ The synthesizer can be shared between threads.
95
+
96
+ ### Command line
97
 
98
  ```bash
99
+ bin/macos-arm64/eeh speak --weights eeh.safetensors --dictionary eeh-dict.bin \
100
+ --voice M1 --text "こんにちは。今日はいい天気ですね。" --out hello.wav
101
  ```
102
 
103
+ ### C and other languages
 
 
 
 
 
 
 
 
104
 
105
+ `lib/<platform>/libeeh_ffi` with `include/eeh.h`: `eeh_synthesizer_new` takes
106
+ the weights and dictionary as bytes, `eeh_speak` returns the audio and
107
+ `eeh_speak_each` streams it sentence by sentence. Errors come back as status
108
+ codes, never as crashes. Go, Node, Java, Ruby and Swift can call it through
109
+ their C FFI (all five are tested).
110
 
111
+ Text of any length works. It is split into sentences at 。!? and read one
112
+ sentence at a time; a quote followed by と or って stays with its sentence, and
113
+ a sentence too long for one call is cut after 、. A text without a final mark
114
+ gets 。. The audio never peaks above 0.99.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
 
116
+ ### In a browser
 
 
 
 
 
117
 
118
+ ```bash
119
+ python -m http.server --directory demo 8000
120
+ ```
 
121
 
122
+ Then open `http://localhost:8000`. The engine runs in a Web Worker and plays
123
+ long text sentence by sentence as it is made. The first visit downloads about
124
+ 26 MB (gzip): 17 MB of weights, 9 MB of dictionary and 0.4 MB of program, all
125
+ kept by the browser afterwards.
126
 
127
  ## How it works
128
 
129
+ **Front end.** eeh's own Rust front end reads the text the way OpenJTalk does
130
+ with the NAIST Japanese Dictionary: it splits it into words, picks each kanji's
131
+ reading, expands numbers, joins words into accent phrases and assigns their
132
+ accent types. It turns the result into tokens:
 
133
 
134
+ - one token per phoneme (`k`, `sh`, `a`, `N` for ん, `cl` for っ);
135
  a long vowel is two tokens, one per mora;
136
  - devoiced vowels (the u of です, the i of した) are their own tokens `A I U E O`;
137
  - `<sp>` marks a boundary between accent phrases, and punctuation is kept;
eeh.pt → bin/linux-aarch64/eeh RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0426a98e9f6755b9f26f55dc460b288c9b297eb28859142c7e20e90889a8d633
3
- size 18196805
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca00933dbcb98bf82f035a8e714fc842af077db191276f0c3c8ee16e98fcbec3
3
+ size 1837632
onnx/model.onnx → bin/linux-x86_64/eeh RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7f31b311a23912933ed4fe515879c7df6a1330d14f085cd90d5328a3245a188f
3
- size 21374050
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac7d699136775ff59ea4c80cca0ded6398717b240d463d78e6c4a51ebf2dd7df
3
+ size 2629176
bin/macos-arm64/eeh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98f6bb859181bdaec6d99886efb59ad07ec11f8a17562d67bcae01b6e9fb8881
3
+ size 1948768
eeh-dict.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2d480c8806b230b113a37a28756748736b975c3aed37d5b27a3e8de7e32c6af
3
+ size 35927036
include/eeh.h ADDED
@@ -0,0 +1,234 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /* eeh C API. Generated by cbindgen from rust/crates/eeh-ffi (generate-header.sh); do not edit. */
2
+
3
+ #ifndef EEH_H
4
+ #define EEH_H
5
+
6
+ #include <stdbool.h>
7
+ #include <stddef.h>
8
+ #include <stdint.h>
9
+
10
+ /**
11
+ * Result of a call. `EEH_STATUS_OK` is 0.
12
+ */
13
+ typedef enum {
14
+ /**
15
+ * Success.
16
+ */
17
+ EEH_STATUS_OK = 0,
18
+ /**
19
+ * A pointer was null or a string was not UTF-8.
20
+ */
21
+ EEH_STATUS_INVALID_ARGUMENT = 1,
22
+ /**
23
+ * No sentence of the text could be read (eeh only reads Japanese for now).
24
+ */
25
+ EEH_STATUS_NOTHING_TO_PRONOUNCE = 2,
26
+ /**
27
+ * The voice is not one of the model's.
28
+ */
29
+ EEH_STATUS_UNKNOWN_VOICE = 3,
30
+ /**
31
+ * The dictionary is malformed, or the text could not be analysed.
32
+ */
33
+ EEH_STATUS_FRONTEND = 4,
34
+ /**
35
+ * The weights are malformed or an option is out of range.
36
+ */
37
+ EEH_STATUS_MODEL = 5,
38
+ /**
39
+ * Anything else, e.g. a word too long to speak or non-finite audio.
40
+ */
41
+ EEH_STATUS_OTHER = 6,
42
+ /**
43
+ * An internal panic was caught; the call had no effect.
44
+ */
45
+ EEH_STATUS_PANIC = 7,
46
+ } EehStatus;
47
+
48
+ /**
49
+ * An error: a code and a message. Opaque to C; read it with `eeh_error_code` and `eeh_error_message`.
50
+ */
51
+ typedef struct EehError EehError;
52
+
53
+ /**
54
+ * A synthesizer and the voice names as C strings.
55
+ */
56
+ typedef struct EehSynthesizer EehSynthesizer;
57
+
58
+ /**
59
+ * Options of a call; `NULL` for all defaults (see `eeh_options_default`).
60
+ */
61
+ typedef struct {
62
+ /**
63
+ * Speaking rate, 0.25 to 4 (1 = as predicted, larger is faster).
64
+ */
65
+ float speed;
66
+ /**
67
+ * Added to log F0 (0.058 is about one semitone).
68
+ */
69
+ float pitch_shift;
70
+ /**
71
+ * Scales the pitch movement, 0 to 3 (1 = as predicted).
72
+ */
73
+ float pitch_range;
74
+ /**
75
+ * Seed of the noise; sentence `i` uses `seed + i`.
76
+ */
77
+ uint64_t seed;
78
+ /**
79
+ * The loudest sample of the audio: louder audio is scaled down to it. Zero or less (or NaN) leaves
80
+ * the model's output as it is. In `eeh_speak_each` each sentence is limited on its own.
81
+ */
82
+ float peak;
83
+ } EehOptions;
84
+
85
+ /**
86
+ * Called by `eeh_speak_each` for each sentence: its text (NUL-terminated UTF-8) and `len` samples at
87
+ * `audio`, both valid only during the call. Return `false` to stop.
88
+ */
89
+ typedef bool (*EehUtteranceCallback)(const char *text,
90
+ const float *audio,
91
+ size_t len,
92
+ void *user_data);
93
+
94
+ #ifdef __cplusplus
95
+ extern "C" {
96
+ #endif // __cplusplus
97
+
98
+ /**
99
+ * The version of the library, e.g. `"0.1.0"`. Static; never freed.
100
+ */
101
+ const char *eeh_version(void);
102
+
103
+ /**
104
+ * Writes the default options (speed 1, no pitch change, seed 0, peak 0.99) to `*options`.
105
+ *
106
+ * # Safety
107
+ * `options` is null or points to writable memory for an `EehOptions`.
108
+ */
109
+ void eeh_options_default(EehOptions *options);
110
+
111
+ /**
112
+ * Loads a synthesizer from the bytes of the weights file (safetensors) and of the packed dictionary
113
+ * (both are copied or parsed during the call; the buffers may be freed afterwards).
114
+ *
115
+ * Returns `NULL` on failure, with `*err` set when `err` is not `NULL`. Free the result with
116
+ * `eeh_synthesizer_free` and the error with `eeh_error_free`. A synthesizer can be used from
117
+ * several threads at once.
118
+ *
119
+ * # Safety
120
+ * Each buffer is `NULL` or has that many readable bytes; `err` is `NULL` or points to a writable `EehError*`.
121
+ */
122
+ EehSynthesizer *eeh_synthesizer_new(const uint8_t *weights,
123
+ size_t weights_len,
124
+ const uint8_t *dictionary,
125
+ size_t dictionary_len,
126
+ EehError **err);
127
+
128
+ /**
129
+ * Frees a synthesizer. `NULL` is fine. No other call may use it at the same time or after.
130
+ *
131
+ * # Safety
132
+ * `synth` is `NULL` or came from `eeh_synthesizer_new` and is not freed twice.
133
+ */
134
+ void eeh_synthesizer_free(EehSynthesizer *synth);
135
+
136
+ /**
137
+ * Sample rate of the audio in Hz; 0 when `synth` is `NULL`.
138
+ *
139
+ * # Safety
140
+ * `synth` is `NULL` or a live synthesizer.
141
+ */
142
+ uint32_t eeh_sample_rate(const EehSynthesizer *synth);
143
+
144
+ /**
145
+ * Number of voices of the model; 0 when `synth` is `NULL`.
146
+ *
147
+ * # Safety
148
+ * `synth` is `NULL` or a live synthesizer.
149
+ */
150
+ size_t eeh_voice_count(const EehSynthesizer *synth);
151
+
152
+ /**
153
+ * Name of voice `index` (e.g. `"F1"`), valid until the synthesizer is freed; `NULL` when out of range.
154
+ *
155
+ * # Safety
156
+ * `synth` is `NULL` or a live synthesizer.
157
+ */
158
+ const char *eeh_voice_name(const EehSynthesizer *synth,
159
+ size_t index);
160
+
161
+ /**
162
+ * Speaks `utf8` (NUL-terminated) in `voice` and returns all the audio.
163
+ *
164
+ * On success `*audio` points to `*len` samples (mono, `eeh_sample_rate` Hz, `float`) that the caller frees with `eeh_audio_free(*audio, *len)`.
165
+ * `options` may be `NULL`. Returns `EEH_STATUS_OK` or an error code, with `*err` set when `err` is not
166
+ * `NULL` (free it with `eeh_error_free`); `*audio` is `NULL` and `*len` 0 on failure.
167
+ *
168
+ * # Safety
169
+ * `synth` is live; `utf8` and `voice` are NUL-terminated strings; `options` is `NULL` or readable;
170
+ * `audio` and `len` are writable; `err` is `NULL` or points to a writable `EehError*`.
171
+ */
172
+ EehStatus eeh_speak(const EehSynthesizer *synth,
173
+ const char *utf8,
174
+ const char *voice,
175
+ const EehOptions *options,
176
+ float **audio,
177
+ size_t *len,
178
+ EehError **err);
179
+
180
+ /**
181
+ * Like `eeh_speak`, but hands each sentence to `callback` as soon as it is made (streaming).
182
+ *
183
+ * The audio is only valid during the callback. The callback returns `false` to stop early (the result is then
184
+ * still `EEH_STATUS_OK`). It runs on the calling thread and must not call back into the same
185
+ * synthesizer's `eeh_synthesizer_free`.
186
+ *
187
+ * # Safety
188
+ * As `eeh_speak`; `callback` is a valid function and `user_data` whatever it expects.
189
+ */
190
+ EehStatus eeh_speak_each(const EehSynthesizer *synth,
191
+ const char *utf8,
192
+ const char *voice,
193
+ const EehOptions *options,
194
+ EehUtteranceCallback callback,
195
+ void *user_data,
196
+ EehError **err);
197
+
198
+ /**
199
+ * Frees audio from `eeh_speak`; `len` is the length it returned. `NULL` is fine.
200
+ *
201
+ * # Safety
202
+ * `audio` is `NULL` or the pointer and `len` the length `eeh_speak` returned, not freed twice.
203
+ */
204
+ void eeh_audio_free(float *audio, size_t len);
205
+
206
+ /**
207
+ * The message of an error (NUL-terminated UTF-8), valid until `eeh_error_free`; `""` for `NULL`.
208
+ *
209
+ * # Safety
210
+ * `error` is `NULL` or a live error.
211
+ */
212
+ const char *eeh_error_message(const EehError *error);
213
+
214
+ /**
215
+ * The status code of an error; `EEH_STATUS_INVALID_ARGUMENT` for `NULL`.
216
+ *
217
+ * # Safety
218
+ * `error` is `NULL` or a live error.
219
+ */
220
+ EehStatus eeh_error_code(const EehError *error);
221
+
222
+ /**
223
+ * Frees an error. `NULL` is fine.
224
+ *
225
+ * # Safety
226
+ * `error` is `NULL` or came from this library and is not freed twice.
227
+ */
228
+ void eeh_error_free(EehError *error);
229
+
230
+ #ifdef __cplusplus
231
+ } // extern "C"
232
+ #endif // __cplusplus
233
+
234
+ #endif /* EEH_H */
lib/linux-aarch64/libeeh_ffi.so ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:329deb93b91717ba4d689e546cd8fe9f1601a446aaf6664fa7db5c22fb7f0755
3
+ size 1697712
lib/linux-x86_64/libeeh_ffi.so ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a88022fe31aa91b65a6c1353ca7c67e392e664ba8f9b1b175407e001d2fed8a1
3
+ size 2451936
lib/macos-arm64/libeeh_ffi.dylib ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5792dfd365078178419e8b0fa7f48f8e7da68acf4edde5f67d09072b88e97dc
3
+ size 1887136
onnx/config.json DELETED
@@ -1,87 +0,0 @@
1
- {
2
- "sample_rate": 24000,
3
- "hop": 256,
4
- "voices": [
5
- "F1",
6
- "M1"
7
- ],
8
- "symbols": [
9
- "<pad>",
10
- "<bos>",
11
- "<eos>",
12
- "<sp>",
13
- "<unk>",
14
- ",",
15
- ".",
16
- "?",
17
- "!",
18
- ";",
19
- ":",
20
- "\"",
21
- "(",
22
- ")",
23
- "—",
24
- "…",
25
- "k",
26
- "ky",
27
- "kw",
28
- "g",
29
- "gy",
30
- "gw",
31
- "s",
32
- "sh",
33
- "z",
34
- "j",
35
- "t",
36
- "ty",
37
- "ch",
38
- "ts",
39
- "d",
40
- "dy",
41
- "n",
42
- "ny",
43
- "h",
44
- "hy",
45
- "f",
46
- "b",
47
- "by",
48
- "p",
49
- "py",
50
- "m",
51
- "my",
52
- "y",
53
- "r",
54
- "ry",
55
- "w",
56
- "v",
57
- "N",
58
- "cl",
59
- "a",
60
- "i",
61
- "u",
62
- "e",
63
- "o",
64
- "A",
65
- "I",
66
- "U",
67
- "E",
68
- "O"
69
- ],
70
- "max_tokens": 512,
71
- "max_frames": 3000,
72
- "step": 104000,
73
- "frontend": "pyopenjtalk",
74
- "frontend_fingerprint": "785d412fa394ae31d619203871d1b134404e24241c324cab897dddc1d22c8f77",
75
- "inputs": [
76
- "tokens",
77
- "stress",
78
- "voice",
79
- "speed",
80
- "pitch_shift",
81
- "pitch_range"
82
- ],
83
- "outputs": [
84
- "audio",
85
- "durations"
86
- ]
87
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
python/eeh-0.1.0-py3-none-macosx_11_0_arm64.whl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d646c44f04df32f14c2d1fd0f2adf68d507af6858ccdcca7cdd107fa099927d
3
+ size 790568
python/eeh-0.1.0-py3-none-manylinux_2_28_aarch64.whl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cc08a406214f7711b5cb006e91d1dff2307fb9e5103e2dd7be7fcd68eb884ec7
3
+ size 786552
python/eeh-0.1.0-py3-none-manylinux_2_28_x86_64.whl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72b5bd3cd18c7100976b476120dc96e4070de41d8a59ebfc5ce8bec664dd8976
3
+ size 1051341
samples/F1-hello.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0eaab40e82edcc1e2e7151b5be5d6b1973191a8e4f26714a6e1bd3b325f16b13
3
- size 334380
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:73af3fd1b0f0f2d7e58411c99af48ac4ba029dd2863371dcd85eefc7932f2bd1
3
+ size 668716
samples/F1-numbers.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:aa38266688a098efbb25f6e9e46d70a105162ff46740e8f28341838e4efca5a4
3
- size 331820
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ffaf2dc813a0feabb92ccaa7a8d082aa1307c675be3c808a73c4062a6f48533
3
+ size 663596
samples/F1-polite.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:14d2fbe0ed3a74c42e71e79eb9fbadd51aa40673b87057250196e4b8f60142c4
3
- size 276524
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba5c9fae74cbc4f7ee51a06da2cfd8ad28b17a04b6ef8035c4def4b9d53cc4df
3
+ size 553004
samples/F1-question.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:48bf72366efeb4ec309f939464e3f71e737fce2ff1e35116d9ebc863e63d8efe
3
- size 202284
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6ac158892770577ec6560f6262aba6d0012f340a0513b0a9580e596f2845daa2
3
+ size 404524
samples/F1-story.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:94515f99e2ed274ad5a7ac1f37ffd6f96a1bf0633a74e9c227802c8bb1a08c45
3
- size 375340
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f7b64c92f6790227d8ffe66ca0f9973cd2a52612a5d4131e3a4fc8bcbb6a0f16
3
+ size 750636
samples/M1-hello.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:240b8da6e672c82bcc03a4e8fddcffa264a4b5b2502b5e00ff7833ba30b3ceaf
3
- size 323116
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b66e92cfe711b5be2180fea913a1d34ae0f4ba7c5e06a2645ca259020ed14a36
3
+ size 646188
samples/M1-numbers.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:59bb3d00fb079090bbb100533a8348bc1b057e9d4d68c8f81c27c9d17db0eb43
3
- size 324652
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ecd845b33e263655a29e3593ecdd683d05ed35f2510831534744674b41ed1cf7
3
+ size 649260
samples/M1-polite.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9ce74a9de6045e85b5b3cf9da337f65abfb8d11f8982a758eadcebcc0721796f
3
- size 271916
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8e7bf8399833a29c6059e427beef465ec235ca48f62e436393dcc5b10668c68b
3
+ size 543788
samples/M1-question.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0dbcc359dbad7eb433b52828d92026cfb53e12db787f17dd8adaf0e775ce0bda
3
- size 197676
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ece4db5aaf0c242c76897fa53e6c6d1231005bee3b738ea900a5ae426a052faa
3
+ size 395308
samples/M1-story.wav CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f0103009d7768ed55273131a4b9c0203272c60aa4ebc44fe2c9aabba40b168a0
3
- size 354348
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eaa9b91a4ac2da5a930aaa71c2c7ce00fb5c22c37bb2fbf1025a245f9b64c930
3
+ size 708652
tiny_voca/__init__.py DELETED
@@ -1,10 +0,0 @@
1
- """tiny-voca: a small two-voice TTS trained from scratch.
2
-
3
- The speech model loads no pretrained weights. A pretrained phoneme recogniser
4
- is used only by the annotation step, and an ASR model only by evaluation.
5
- """
6
-
7
- from .config import Config
8
- from .model import TinyVoca
9
-
10
- __all__ = ["Config", "TinyVoca"]
 
 
 
 
 
 
 
 
 
 
 
tiny_voca/config.py DELETED
@@ -1,92 +0,0 @@
1
- from dataclasses import asdict, dataclass
2
- import hashlib
3
- import json
4
-
5
- CHECKPOINT_FORMAT = 4 # 2-3: byte tokens + learned aligner (rejected explicitly).
6
- ANNOTATION_VERSION = 3
7
-
8
-
9
- def _digest(value):
10
- return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
11
-
12
-
13
- @dataclass(frozen=True)
14
- class Config:
15
- # Acoustic features (unchanged since feature schema 2; existing files are reused).
16
- sample_rate: int = 24000
17
- hop: int = 256
18
- n_fft: int = 1024
19
- n_mels: int = 80
20
- f0_min: float = 50.0
21
- f0_max: float = 600.0
22
- speakers: tuple[str, ...] = ("F1", "M1")
23
- # Phoneme tokens from tiny_voca.text.SYMBOLS (Japanese front end, version 1).
24
- vocab_size: int = 60
25
- # Text encoder and frame-level transformer.
26
- channels: int = 192
27
- heads: int = 4
28
- feedforward: int = 640
29
- kernel: int = 7
30
- encoder_layers: int = 4
31
- prosody_layers: int = 2
32
- acoustic_layers: int = 5
33
- dropout: float = 0.1
34
- predictor_channels: int = 128
35
- # Source-filter waveform head: harmonic branch and noise branch STFT sizes,
36
- # causal/anticausal complex-cepstrum orders.
37
- synth_fft: int = 512
38
- synth_hop: int = 128
39
- noise_fft: int = 256
40
- noise_hop: int = 64
41
- cep_causal: int = 80
42
- cep_anti: int = 16
43
- noise_cep: int = 32
44
- harmonic_limit: float = 0.47
45
- max_tokens: int = 512
46
- max_frames: int = 3000
47
- mel_mean: float = -5.0
48
- mel_scale: float = 5.0
49
- energy_mean: float = -4.0
50
- energy_scale: float = 2.0
51
- pitch_reference_hz: float = 200.0
52
-
53
- def __post_init__(self):
54
- from .text import SYMBOLS
55
- if self.vocab_size != len(SYMBOLS):
56
- raise ValueError(f"vocab_size must equal the phoneme inventory size {len(SYMBOLS)}")
57
- if self.channels % self.heads or (self.channels // self.heads) % 2:
58
- raise ValueError("channels must be divisible by heads with an even head width")
59
- if not self.speakers or len(set(self.speakers)) != len(self.speakers):
60
- raise ValueError("speaker names must be unique and nonempty")
61
- for fft, hop in ((self.synth_fft, self.synth_hop), (self.noise_fft, self.noise_hop)):
62
- if fft != 4 * hop or self.hop % hop:
63
- raise ValueError("synthesis windows need 4x overlap and a hop dividing the acoustic hop")
64
- if self.cep_causal + self.cep_anti >= self.synth_fft // 2 or self.noise_cep >= self.noise_fft // 2:
65
- raise ValueError("cepstrum orders must be well below half the synthesis FFT size")
66
- if min(self.encoder_layers, self.prosody_layers, self.acoustic_layers, self.kernel) < 1 or not self.kernel % 2:
67
- raise ValueError("layer counts must be positive and the kernel odd")
68
- if min(self.mel_scale, self.energy_scale) <= 0 or not 0 < self.harmonic_limit < 0.5:
69
- raise ValueError("invalid normalization scale or harmonic limit")
70
-
71
- def to_dict(self):
72
- return asdict(self)
73
-
74
- @classmethod
75
- def from_dict(cls, value):
76
- return cls(**{**value, "speakers": tuple(value["speakers"])})
77
-
78
- @property
79
- def fingerprint(self):
80
- return _digest(self.to_dict())
81
-
82
- @property
83
- def acoustic_config(self):
84
- """Settings that determine stored wave/mel/prosody tensors. Tokens and
85
- durations are versioned separately by the annotation layer."""
86
- fields = ("sample_rate", "hop", "n_fft", "n_mels", "f0_min", "f0_max")
87
- return {**{key: getattr(self, key) for key in fields}, "feature_version": 2,
88
- "pitch_reference_hz": 200, "log_floor": 1e-5}
89
-
90
- @property
91
- def acoustic_fingerprint(self):
92
- return _digest(self.acoustic_config)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tiny_voca/infer.py DELETED
@@ -1,182 +0,0 @@
1
- import argparse
2
- from pathlib import Path
3
- import re
4
- import torch
5
- from .config import CHECKPOINT_FORMAT, Config
6
- from .model import TinyVoca, UtteranceTooLong
7
- from .text import NothingToPronounce, frontend_fingerprint, phonemize
8
-
9
- # The loudest sample of the audio that ``speak_text`` and the command return.
10
- PEAK = 0.99
11
- # Said when no sentence of a text could be read. The browser demo says the same.
12
- JAPANESE_ONLY = "eeh only reads Japanese for now"
13
- _SPACE = re.compile("[\\s" + chr(0xFEFF) + "]+") # JavaScript's \s, which the demo uses, takes U+FEFF too
14
- # A sentence ends at 。!?… (full or half width) and at a full stop that is not
15
- # a decimal point; closing quotes and brackets after the mark stay with it. A
16
- # quotation that goes on with と or って (「行こう。」と言った) is not an end.
17
- _END_MARKS = "。.!?!?…"
18
- _CLOSING = "」』))\"'”’】»"
19
- _END = rf"(?:(?:[{_END_MARKS}]|(?<!\d)\.|\.(?!\d))(?![{_END_MARKS}.]*[{_CLOSING}]+[とっ]))"
20
- _SENTENCE = re.compile(rf"(?:(?!{_END}).)+(?:{_END})*[{_CLOSING}]*|(?:{_END})+[{_CLOSING}]*", re.S)
21
- _MARKED = frozenset(_END_MARKS + "、,;:;:.")
22
- _CLAUSE_END = re.compile(r"(?<=[、,;:;:—])")
23
- _TEXT_END = re.compile("[\\s" + chr(0xFEFF) + "]+$")
24
-
25
-
26
- def load_model(path, device="cpu"):
27
- state = torch.load(path, map_location="cpu", weights_only=True)
28
- if state.get("format_version") != CHECKPOINT_FORMAT:
29
- raise ValueError("unsupported checkpoint format: byte-token checkpoints (format 2-3) cannot be used with this code")
30
- if state["frontend"]["fingerprint"] != frontend_fingerprint():
31
- raise ValueError("checkpoint was trained with a different phoneme inventory/front end")
32
- model = TinyVoca(Config.from_dict(state["config"]))
33
- model.load_state_dict(state["model"], strict=True)
34
- return model.to(device).eval()
35
-
36
-
37
- def sentences(text):
38
- """The sentences of ``text``, never cut inside a number. The browser demo
39
- splits the same way (``web/public/worker.js``); keep the two alike."""
40
- parts = (part.strip() for part in _SENTENCE.findall(_SPACE.sub(" ", text).strip()))
41
- return [part for part in parts if any(char.isalnum() for char in part)]
42
-
43
-
44
- def finished(text):
45
- """``text`` with 。 added when it ends without a mark. Before a final mark
46
- the voice lets the last mora fall; without one the audio breaks off. A
47
- closing quote or bracket after the mark is fine; an emoji or a sign is no
48
- mark. The browser demo does the same (``web/public/worker.js``); keep the
49
- two alike."""
50
- body = _TEXT_END.sub("", text)
51
- if not body:
52
- return text
53
- core = body.rstrip(_CLOSING)
54
- return text if core and core[-1] in _MARKED else body + "。"
55
-
56
-
57
- def shorter(text):
58
- """``text`` in smaller pieces: cut after 、 and the like if it has any,
59
- else in two halves, never inside a number. ``[]`` for a single character."""
60
- pieces = [piece for piece in _CLAUSE_END.split(text) if piece.strip()]
61
- if len(pieces) > 1:
62
- return pieces
63
- if len(text) < 2:
64
- return []
65
- cuts = [i for i in range(1, len(text)) if not (text[i - 1].isdigit() and text[i].isdigit())]
66
- if not cuts:
67
- return []
68
- middle = min(cuts, key=lambda i: abs(i - len(text) / 2))
69
- return [text[:middle], text[middle:]]
70
-
71
-
72
- def limit_peak(wave, peak=PEAK):
73
- """``wave`` scaled down as a whole when its loudest sample is above
74
- ``peak``; quieter audio is returned as it is. The model sometimes peaks
75
- above 1.0, and that clips when the audio is saved as 16-bit samples."""
76
- loudest = float(wave.abs().max()) if wave.numel() else 0.0
77
- return wave * (peak / loudest) if loudest > peak else wave
78
-
79
-
80
- def _voice(model, phones, voice, speed, pitch_shift, seed, pitch_range):
81
- """Phonemes -> (float waveform tensor, details): one call of the model."""
82
- device = next(model.parameters()).device
83
- tokens = torch.tensor([phones.ids], device=device)
84
- stress = torch.tensor([phones.stress], device=device)
85
- speaker = torch.tensor([model.config.speakers.index(voice)], device=device)
86
- audio, lengths, details = model.synthesize(tokens, stress, torch.tensor([len(phones)], device=device), speaker, speed, pitch_shift, seed, pitch_range)
87
- if not torch.isfinite(audio).all():
88
- raise RuntimeError("model produced non-finite audio")
89
- return audio[0, :int(lengths[0])].cpu(), details
90
-
91
-
92
- def speak(model, text, voice, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.0):
93
- """One utterance: text -> (float waveform tensor, phonemized text, details).
94
- Same front end as training. The audio is the model's output as it is and
95
- can peak above 1.0 (see ``limit_peak``). ``speak_text`` reads text of any
96
- length."""
97
- phones = phonemize(text)
98
- if len(phones) > model.config.max_tokens:
99
- raise ValueError("text is too long for one utterance; speak_text splits it into sentences")
100
- wave, details = _voice(model, phones, voice, speed, pitch_shift, seed, pitch_range)
101
- return wave, phones, details
102
-
103
-
104
- def speak_text(model, text, voice, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.0, peak=PEAK):
105
- """Text of any length -> (float waveform tensor, phonemized utterances).
106
-
107
- The text is read sentence by sentence and the audio is joined. A text that
108
- ends without a mark gets a full stop (``finished``). A sentence
109
- too long for one call of the model (``max_tokens`` phonemes or
110
- ``max_frames`` frames) is cut after commas, then between words. A sentence
111
- with nothing to pronounce (another script, a face, signs only) is skipped;
112
- when no sentence could be read, ``NothingToPronounce`` says so. The audio
113
- is scaled down as a whole when it peaks above ``peak``, so that it can be
114
- saved as 16-bit samples without clipping; ``peak=None`` returns the
115
- model's output as it is."""
116
- limit = model.config.max_tokens - 12
117
- waves, spoken = [], []
118
-
119
- def read(piece):
120
- try:
121
- phones = phonemize(piece)
122
- except NothingToPronounce:
123
- return
124
- if len(phones) <= limit:
125
- try:
126
- wave, _ = _voice(model, phones, voice, speed, pitch_shift, seed + len(waves), pitch_range)
127
- except UtteranceTooLong:
128
- pass
129
- else:
130
- waves.append(wave)
131
- spoken.append(phones)
132
- return
133
- pieces = shorter(piece)
134
- if not pieces:
135
- raise ValueError(f"one word is too long to speak: {piece[:40]!r}")
136
- for part in pieces:
137
- read(part)
138
-
139
- for sentence in sentences(finished(text)):
140
- read(sentence)
141
- if not waves:
142
- raise NothingToPronounce(f"nothing in the text could be read: {JAPANESE_ONLY}")
143
- wave = torch.cat(waves)
144
- return (wave if peak is None else limit_peak(wave, peak)), spoken
145
-
146
-
147
- def main():
148
- parser = argparse.ArgumentParser(description="Generate speech from a trained tiny-voca inference checkpoint")
149
- parser.add_argument("--checkpoint", type=Path, required=True)
150
- source = parser.add_mutually_exclusive_group(required=True)
151
- source.add_argument("--text", help="Text of any length; it is read sentence by sentence")
152
- source.add_argument("--text-file", type=Path, help="Read the text from a file (UTF-8)")
153
- parser.add_argument("--voice", help="A voice of the checkpoint (default: its first, e.g. F1)")
154
- parser.add_argument("--output", type=Path, required=True)
155
- parser.add_argument("--speed", type=float, default=1)
156
- parser.add_argument("--pitch-shift", type=float, default=0, help="Added to log F0 (0.058 is about one semitone)")
157
- parser.add_argument("--pitch-range", type=float, default=1, help="Scale pitch movement around the utterance mean (1 = as predicted)")
158
- parser.add_argument("--device", default="cpu")
159
- args = parser.parse_args()
160
- import soundfile as sf
161
- model = load_model(args.checkpoint, args.device)
162
- voice = args.voice or model.config.speakers[0]
163
- if voice not in model.config.speakers:
164
- parser.error(f"--voice must be one of {', '.join(model.config.speakers)}")
165
- text = args.text if args.text is not None else args.text_file.read_text(encoding="utf-8")
166
- try:
167
- raw, spoken = speak_text(model, text, voice, args.speed, args.pitch_shift, pitch_range=args.pitch_range, peak=None)
168
- except NothingToPronounce:
169
- parser.exit(1, f"{JAPANESE_ONLY}. Nothing in the text could be read, so no file was written.\n")
170
- wave = limit_peak(raw)
171
- args.output.parent.mkdir(parents=True, exist_ok=True)
172
- # Float samples, never above PEAK: a later conversion to 16-bit does not clip.
173
- sf.write(args.output, wave.numpy(), model.config.sample_rate, subtype="FLOAT")
174
- for phones in spoken:
175
- print(" ".join(phones.symbols))
176
- loudest = float(raw.abs().max())
177
- scaled = f", scaled down to {PEAK}" if wave is not raw else ""
178
- print(f"Saved {len(wave)/model.config.sample_rate:.2f}s in {len(spoken)} utterance{'s' if len(spoken) != 1 else ''}; peak={loudest:.4f}{scaled}")
179
-
180
-
181
- if __name__ == "__main__":
182
- main()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tiny_voca/model.py DELETED
@@ -1,379 +0,0 @@
1
- """tiny-voca v2: feed-forward transformer with a pitch-driven source-filter head.
2
-
3
- All weights start random. Tensor conventions: hidden [B,T,C], frame features
4
- [B,T], waveform [B,S]. Acoustic frame i is centred on sample ``i * hop``.
5
-
6
- phoneme + stress + voice -> transformer encoder -> duration predictor
7
- -> repeat tokens by duration (supervised durations while training)
8
- -> frame transformer A -> pitch / voicing / energy
9
- -> frame transformer B -> filter parameters
10
- -> harmonic pulse train (from pitch) and noise, each shaped by its
11
- time-varying filter in the STFT domain -> 24 kHz waveform
12
-
13
- The waveform head has no neural upsampling network. The oscillator supplies the
14
- periodicity that a tiny decoder cannot render reliably on its own; the network
15
- only has to predict slowly varying spectral envelopes (complex cepstra), which
16
- is what keeps the voice from sounding buzzy or unstable at this size.
17
- """
18
-
19
- import math
20
- import torch
21
- from torch import nn
22
- from torch.nn import functional as F
23
- from .config import Config
24
- from .text import PHONE_IDS, SYMBOL_TO_ID, VOWELS
25
-
26
-
27
- class UtteranceTooLong(ValueError):
28
- """The utterance needs more frames than ``max_frames``: split the input."""
29
-
30
-
31
- def length_mask(lengths, size=None):
32
- size = int(lengths.max()) if size is None else size
33
- return torch.arange(size, device=lengths.device)[None, :] < lengths[:, None]
34
-
35
-
36
- def rotary_table(size, width, device):
37
- inverse = 1.0 / (10000 ** (torch.arange(0, width, 2, device=device, dtype=torch.float32) / width))
38
- angle = torch.arange(size, device=device, dtype=torch.float32)[:, None] * inverse[None]
39
- return angle.cos(), angle.sin()
40
-
41
-
42
- def rotate(x, table):
43
- cos, sin = table[0].to(x.dtype), table[1].to(x.dtype)
44
- a, b = x[..., 0::2], x[..., 1::2]
45
- return torch.stack((a * cos - b * sin, a * sin + b * cos), dim=-1).flatten(-2)
46
-
47
-
48
- class Attention(nn.Module):
49
- """Self-attention with rotary (relative) positions: no absolute position
50
- table, so utterances longer than those seen in training stay in range."""
51
-
52
- def __init__(self, channels, heads, dropout):
53
- super().__init__()
54
- self.heads, self.dropout = heads, dropout
55
- self.qkv = nn.Linear(channels, 3 * channels)
56
- self.out = nn.Linear(channels, channels)
57
-
58
- def forward(self, x, mask, table):
59
- b, t, c = x.shape
60
- q, k, v = self.qkv(x).view(b, t, 3, self.heads, c // self.heads).permute(2, 0, 3, 1, 4)
61
- y = F.scaled_dot_product_attention(rotate(q, table), rotate(k, table), v, attn_mask=mask[:, None, None, :],
62
- dropout_p=self.dropout if self.training else 0.0)
63
- return self.out(y.transpose(1, 2).reshape(b, t, c))
64
-
65
-
66
- class Block(nn.Module):
67
- def __init__(self, c):
68
- super().__init__()
69
- self.norm1 = nn.LayerNorm(c.channels)
70
- self.attention = Attention(c.channels, c.heads, c.dropout)
71
- self.norm2 = nn.LayerNorm(c.channels)
72
- self.depthwise = nn.Conv1d(c.channels, c.channels, c.kernel, padding=c.kernel // 2, groups=c.channels)
73
- self.expand = nn.Linear(c.channels, c.feedforward)
74
- self.project = nn.Linear(c.feedforward, c.channels)
75
- self.dropout = nn.Dropout(c.dropout)
76
-
77
- def forward(self, x, mask, table):
78
- keep = mask[..., None]
79
- x = (x + self.dropout(self.attention(self.norm1(x), mask, table))) * keep
80
- # Mask before the convolution so padded positions cannot leak back in.
81
- y = self.depthwise((self.norm2(x) * keep).transpose(1, 2)).transpose(1, 2)
82
- y = self.project(self.dropout(F.gelu(self.expand(y))))
83
- return (x + self.dropout(y)) * keep
84
-
85
-
86
- class Stack(nn.Module):
87
- def __init__(self, c, layers):
88
- super().__init__()
89
- self.blocks = nn.ModuleList(Block(c) for _ in range(layers))
90
- self.norm = nn.LayerNorm(c.channels)
91
- self.width = c.channels // c.heads
92
-
93
- def forward(self, x, mask):
94
- table = rotary_table(x.shape[1], self.width, x.device)
95
- for block in self.blocks:
96
- x = block(x, mask, table)
97
- return self.norm(x) * mask[..., None]
98
-
99
-
100
- class Predictor(nn.Module):
101
- def __init__(self, c, outputs):
102
- super().__init__()
103
- p = c.predictor_channels
104
- self.conv1 = nn.Conv1d(c.channels, p, 3, padding=1)
105
- self.conv2 = nn.Conv1d(p, p, 3, padding=1)
106
- self.norm1, self.norm2 = nn.LayerNorm(p), nn.LayerNorm(p)
107
- self.drop = nn.Dropout(c.dropout)
108
- self.out = nn.Linear(p, outputs)
109
-
110
- def forward(self, x, mask):
111
- x = self.conv1((x * mask[..., None]).transpose(1, 2)).transpose(1, 2)
112
- x = self.drop(self.norm1(F.gelu(x))) * mask[..., None]
113
- x = self.conv2(x.transpose(1, 2)).transpose(1, 2)
114
- x = self.drop(self.norm2(F.gelu(x)))
115
- return self.out(x) * mask[..., None]
116
-
117
-
118
- def pulse_train(f0, sample_rate, hop, limit=0.47, phase=None):
119
- """Band-limited pulse train [B, T*hop] with unit power for f0 [B, T] in Hz.
120
-
121
- Frame values sit at samples i*hop and are interpolated in the log domain.
122
- The sum of all harmonics below ``limit * sample_rate`` has the closed form
123
- sin((K+1/2)x) / (2 sin(x/2)) - 1/2; the top harmonic fades in and out so the
124
- signal stays continuous while the pitch glides. Float64 keeps the phase exact.
125
- """
126
- frames = f0.shape[1]
127
- knots = torch.log(f0.double().clamp_min(1.0))
128
- knots = torch.cat((knots, knots[:, -1:]), dim=1)
129
- position = torch.arange(frames * hop, device=f0.device, dtype=torch.float64) / hop
130
- low = position.floor().long()
131
- weight = position - low
132
- frequency = torch.exp(knots[:, low] * (1 - weight) + knots[:, low + 1] * weight)
133
- theta = 2 * math.pi * torch.cumsum(frequency / sample_rate, dim=1)
134
- if phase is not None:
135
- theta = theta + phase.double()[:, None]
136
- theta = torch.remainder(theta, 2 * math.pi)
137
- harmonics = (limit * sample_rate / frequency).clamp_min(1.0)
138
- whole = torch.floor(harmonics)
139
- denominator = 2 * torch.sin(theta / 2)
140
- tiny = denominator.abs() < 1e-7
141
- core = torch.where(tiny, whole, torch.sin((whole + 0.5) * theta) / torch.where(tiny, torch.ones_like(denominator), denominator) - 0.5)
142
- wave = core + (harmonics - whole) * torch.cos((whole + 1) * theta)
143
- return (wave * torch.sqrt(2.0 / harmonics)).float()
144
-
145
-
146
- def filtered(source, response, fft, hop, window):
147
- """Time-varying filtering: response [B, frames, bins] multiplies the STFT."""
148
- spectrum = torch.stft(source, fft, hop, window=window, center=True, pad_mode="constant", return_complex=True)
149
- if spectrum.shape[-1] != response.shape[1]:
150
- raise ValueError("filter frames do not match the source")
151
- return torch.istft(spectrum * response.transpose(1, 2), fft, hop, window=window, center=True, length=source.shape[-1])
152
-
153
-
154
- def cepstrum_response(causal, anticausal, fft):
155
- """Complex frequency response exp(FFT(cepstrum)). Causal coefficients give a
156
- minimum-phase filter; anticausal ones add a maximum-phase part (glottal pulse)."""
157
- buffer = causal.new_zeros(*causal.shape[:-1], fft)
158
- buffer[..., :causal.shape[-1]] = causal
159
- if anticausal is not None:
160
- buffer[..., fft - anticausal.shape[-1]:] = anticausal.flip(-1)
161
- spectrum = torch.fft.rfft(buffer, dim=-1)
162
- return torch.polar(torch.exp(spectrum.real.clamp(-14, 6)), spectrum.imag)
163
-
164
-
165
- class SourceFilter(nn.Module):
166
- """Harmonic-plus-noise synthesis controlled per frame by two linear heads."""
167
-
168
- def __init__(self, c):
169
- super().__init__()
170
- self.config = c
171
- self.harmonic_steps, self.noise_steps = c.hop // c.synth_hop, c.hop // c.noise_hop
172
- self.harmonic = nn.Linear(c.channels, c.cep_causal + c.cep_anti + self.harmonic_steps)
173
- self.noise = nn.Linear(c.channels, c.noise_cep + self.noise_steps)
174
- for head, level in ((self.harmonic, -1.5), (self.noise, -2.5)): # start quiet and spectrally flat
175
- nn.init.zeros_(head.weight)
176
- nn.init.zeros_(head.bias)
177
- head.bias.data[0] = level
178
-
179
- def lifter(size, first):
180
- scale = 1.0 / (1.0 + torch.arange(first, first + size, dtype=torch.float32) / 4)
181
- if first == 0:
182
- scale[0] = 2.0 # coefficient 0 is the log gain and needs range
183
- return scale
184
- self.register_buffer("causal_lifter", lifter(c.cep_causal, 0), persistent=False)
185
- self.register_buffer("anti_lifter", lifter(c.cep_anti, 1), persistent=False)
186
- self.register_buffer("noise_lifter", lifter(c.noise_cep, 0), persistent=False)
187
- self.register_buffer("synth_window", torch.hann_window(c.synth_fft), persistent=False)
188
- self.register_buffer("noise_window", torch.hann_window(c.noise_fft), persistent=False)
189
-
190
- @staticmethod
191
- def _take(values, index):
192
- return values.gather(1, index[..., None].expand(-1, -1, values.shape[-1]))
193
-
194
- def _subframes(self, frames, lengths, steps):
195
- """Sub-frame j uses frame j // steps; indices past an utterance's end
196
- repeat its last sub-frame so padding never changes real audio. Two extra
197
- sub-frames cover the half window of zeros appended to the source."""
198
- sub = torch.arange(steps * frames + 3, device=lengths.device)[None].expand(len(lengths), -1)
199
- return torch.minimum(sub, (steps * lengths - 1).clamp_min(0)[:, None])
200
-
201
- @staticmethod
202
- def _shape(source, response, fft, hop, window):
203
- # Half a window of trailing zeros: the frames around an utterance's end are
204
- # then the same whether it stands alone or is padded inside a batch.
205
- return filtered(F.pad(source, (0, fft // 2)), response, fft, hop, window)[:, :source.shape[-1]]
206
-
207
- def forward(self, hidden, f0, lengths, noise=None, phase=None):
208
- c = self.config
209
- frames = hidden.shape[1]
210
- samples = frames * c.hop
211
- with torch.autocast(device_type=hidden.device.type, enabled=False):
212
- hidden = hidden.float()
213
- audible = length_mask(lengths * c.hop, samples)
214
- last = (lengths - 1).clamp_min(0)[:, None]
215
- # Harmonic branch: envelope interpolated between frames, one gain per sub-frame.
216
- h = self.harmonic(hidden)
217
- split = c.cep_causal + c.cep_anti
218
- sub = self._subframes(frames, lengths, self.harmonic_steps)
219
- here, following = sub // self.harmonic_steps, torch.minimum((sub + 1) // self.harmonic_steps, last)
220
- causal = 0.5 * (self._take(h[..., :c.cep_causal], here) + self._take(h[..., :c.cep_causal], following)) * self.causal_lifter
221
- anti = 0.5 * (self._take(h[..., c.cep_causal:split], here) + self._take(h[..., c.cep_causal:split], following)) * self.anti_lifter
222
- causal = torch.cat((causal[..., :1] + h[..., split:].flatten(1).gather(1, sub)[..., None], causal[..., 1:]), dim=-1)
223
- harmonic_response = cepstrum_response(causal, anti, c.synth_fft)
224
- # Noise branch: one envelope per frame, four gains per frame for sharp consonants.
225
- n = self.noise(hidden)
226
- sub = self._subframes(frames, lengths, self.noise_steps)
227
- shape = self._take(n[..., :c.noise_cep], sub // self.noise_steps) * self.noise_lifter
228
- shape = torch.cat((shape[..., :1] + n[..., c.noise_cep:].flatten(1).gather(1, sub)[..., None], shape[..., 1:]), dim=-1)
229
- noise_response = cepstrum_response(shape, None, c.noise_fft)
230
- # Hold the last real pitch value through padding: the pitch glide inside an
231
- # utterance's final frame must not depend on what the batch was padded with.
232
- f0 = f0.float().gather(1, torch.minimum(torch.arange(frames, device=f0.device)[None], last))
233
- pulses = pulse_train(f0, c.sample_rate, c.hop, c.harmonic_limit, phase) * audible
234
- if noise is None:
235
- noise = torch.randn(hidden.shape[0], samples, device=hidden.device)
236
- wave = (self._shape(pulses, harmonic_response, c.synth_fft, c.synth_hop, self.synth_window)
237
- + self._shape(noise * audible, noise_response, c.noise_fft, c.noise_hop, self.noise_window))
238
- return wave * audible
239
-
240
-
241
- class TinyVoca(nn.Module):
242
- def __init__(self, config=Config()):
243
- super().__init__()
244
- self.config = c = config
245
- self.tokens = nn.Embedding(c.vocab_size, c.channels, padding_idx=0)
246
- self.stress = nn.Embedding(3, c.channels)
247
- self.voices = nn.Embedding(len(c.speakers), c.channels)
248
- self.encoder = Stack(c, c.encoder_layers)
249
- # Outputs per token: log(1 + frames) and, for pause-capable tokens, a
250
- # "has frames at all" logit. A pause is absent more often than present, so
251
- # one regression would average both cases into a pause that is too short.
252
- self.duration = Predictor(c, 2)
253
- self.frame_voices = nn.Embedding(len(c.speakers), c.channels)
254
- self.frame_position = nn.Linear(3, c.channels)
255
- self.prosody_stack = Stack(c, c.prosody_layers)
256
- # Outputs: continuous log(F0/reference), voiced logit, normalized log RMS energy.
257
- self.prosody_head = nn.Linear(c.channels, 3)
258
- self.prosody_projection = nn.Linear(3, c.channels)
259
- self.acoustic_stack = Stack(c, c.acoustic_layers)
260
- self.vocoder = SourceFilter(c)
261
- phones = torch.zeros(c.vocab_size, dtype=torch.bool)
262
- phones[sorted(PHONE_IDS)] = True
263
- self.register_buffer("phone_tokens", phones, persistent=False)
264
- vowels = torch.zeros(c.vocab_size, dtype=torch.bool)
265
- vowels[[SYMBOL_TO_ID[v] for v in VOWELS]] = True
266
- self.register_buffer("vowel_tokens", vowels, persistent=False)
267
-
268
- def encode(self, tokens, stress, lengths, speakers):
269
- if tokens.ndim != 2 or stress.shape != tokens.shape or lengths.shape != (tokens.shape[0],) or speakers.shape != lengths.shape:
270
- raise ValueError("expected tokens[B,N], stress[B,N], lengths[B], speakers[B]")
271
- if tokens.shape[1] > self.config.max_tokens or bool((lengths < 1).any()) or bool((lengths > tokens.shape[1]).any()):
272
- raise ValueError("invalid token length")
273
- mask = length_mask(lengths, tokens.shape[1])
274
- x = (self.tokens(tokens) + self.stress(stress) + self.voices(speakers)[:, None]) * mask[..., None]
275
- return self.encoder(x, mask), mask
276
-
277
- def expand(self, hidden, durations, token_mask, speakers):
278
- """Repeat token states by integer durations (zero is allowed for pause
279
- tokens) and add each frame's position inside its token."""
280
- if durations.shape != token_mask.shape or bool((durations < 0).any()) or bool((durations[~token_mask] != 0).any()):
281
- raise ValueError("invalid durations")
282
- lengths = durations.sum(1)
283
- if bool((lengths < 1).any()):
284
- raise ValueError("utterance has no frames")
285
- if bool((lengths > self.config.max_frames).any()):
286
- raise UtteranceTooLong("utterance exceeds max_frames; split the input")
287
- size = int(lengths.max())
288
- ends = durations.cumsum(1)
289
- frame = torch.arange(size, device=hidden.device)[None].expand(hidden.shape[0], size).contiguous()
290
- index = torch.searchsorted(ends, frame, right=True).clamp(max=durations.shape[1] - 1)
291
- owned = durations.gather(1, index).clamp_min(1)
292
- offset = frame - (ends.gather(1, index) - durations.gather(1, index))
293
- relative = (offset.to(hidden.dtype) + 0.5) / owned
294
- features = torch.stack((relative, 1 - relative, torch.log1p(owned.to(hidden.dtype)) / 4), dim=-1)
295
- mask = frame < lengths[:, None]
296
- expanded = hidden.gather(1, index[..., None].expand(-1, -1, hidden.shape[-1]))
297
- expanded = expanded + self.frame_position(features) + self.frame_voices(speakers)[:, None]
298
- return expanded * mask[..., None], mask, lengths
299
-
300
- def predict_prosody(self, frames, frame_mask):
301
- hidden = self.prosody_stack(frames, frame_mask)
302
- return hidden, self.prosody_head(hidden) * frame_mask[..., None]
303
-
304
- @staticmethod
305
- def hard_prosody(prediction):
306
- """What the decoder is conditioned on when it is not teacher-forced."""
307
- return torch.stack((prediction[..., 0].clamp(-2.0, 1.5), (prediction[..., 1] > 0).to(prediction.dtype),
308
- prediction[..., 2].clamp(-4.5, 3.0)), dim=-1)
309
-
310
- def decode(self, prosody_hidden, condition, frame_mask):
311
- return self.acoustic_stack((prosody_hidden + self.prosody_projection(condition)) * frame_mask[..., None], frame_mask)
312
-
313
- def frequency(self, log_f0):
314
- c = self.config
315
- return (c.pitch_reference_hz * torch.exp(log_f0.float())).clamp(c.f0_min, c.f0_max)
316
-
317
- def predict_durations(self, hidden, mask, tokens, speed=1.0):
318
- output = self.duration(hidden, mask).float()
319
- frames = torch.round(torch.expm1(output[..., 0].clamp(0, math.log1p(400))) / speed).long().clamp_min(1)
320
- # Every phoneme is audible for at least one frame. A pause-capable token
321
- # (edge, punctuation, word boundary) lasts its predicted length when the
322
- # presence logit says there is a pause, and vanishes otherwise.
323
- present = self.phone_tokens[tokens] | (output[..., 1] > 0)
324
- return torch.where(present, frames, torch.zeros_like(frames)) * mask
325
-
326
- @staticmethod
327
- def widen_pitch(prosody, frame_mask, pitch_range):
328
- """Scale the predicted pitch movement around each utterance's voiced mean.
329
- Regression predicts a flatter contour than the recordings; values above 1
330
- restore range (1.0 leaves the prediction untouched)."""
331
- if pitch_range == 1.0:
332
- return prosody
333
- voiced = ((prosody[..., 1] > 0) & frame_mask).to(prosody.dtype)
334
- mean = (prosody[..., 0] * voiced).sum(1, keepdim=True) / voiced.sum(1, keepdim=True).clamp_min(1)
335
- pitch = mean + (prosody[..., 0] - mean) * pitch_range
336
- return torch.cat((pitch[..., None], prosody[..., 1:]), dim=-1) * frame_mask[..., None]
337
-
338
- @staticmethod
339
- def pitch_wobble(shape, semitones, seed, device):
340
- """Small random pitch movement for the oscillator only (log-F0 units).
341
- Recorded pitch has cycle-to-cycle irregularity that a predicted contour
342
- lacks; two thirds of the wobble is slow drift, one third frame-level."""
343
- generator = torch.Generator().manual_seed(seed + 7919)
344
- fast = torch.randn(shape, generator=generator)
345
- slow = F.avg_pool1d(torch.randn(shape[0], 1, shape[1] + 16, generator=generator), 17, stride=1)[:, 0]
346
- slow = slow / slow.std().clamp_min(1e-6)
347
- return ((2 * slow + fast) / 5 ** 0.5 * semitones * math.log(2) / 12).to(device)
348
-
349
- @torch.inference_mode()
350
- def synthesize(self, tokens, stress, lengths, speakers, speed=1.0, pitch_shift=0.0, seed=0, pitch_range=1.0, pitch_jitter=0.0,
351
- voiced_vowels=True):
352
- """Free-running synthesis: predicted durations, pitch, voicing and energy."""
353
- if not math.isfinite(speed) or not 0.25 <= speed <= 4 or not math.isfinite(pitch_shift) or not 0 <= pitch_range <= 3:
354
- raise ValueError("speed must be between 0.25 and 4, pitch_shift finite and pitch_range between 0 and 3")
355
- hidden, mask = self.encode(tokens, stress, lengths, speakers)
356
- durations = self.predict_durations(hidden, mask, tokens, speed)
357
- frames, frame_mask, frame_lengths = self.expand(hidden, durations, mask, speakers)
358
- prosody_hidden, prosody = self.predict_prosody(frames, frame_mask)
359
- if voiced_vowels:
360
- # The pitch tracker missed creaky vowels in the recordings, and the
361
- # voicing predictor learned to drop whole vowels (heard as a whispered
362
- # word). A vowel is voiced by definition, so its frames are kept voiced.
363
- ends = durations.cumsum(1)
364
- frame = torch.arange(frames.shape[1], device=tokens.device)[None].expand(len(lengths), -1).contiguous()
365
- owner = torch.searchsorted(ends, frame, right=True).clamp(max=tokens.shape[1] - 1)
366
- vowel = self.vowel_tokens[tokens].gather(1, owner) & frame_mask
367
- prosody = torch.cat((prosody[..., :1], torch.where(vowel, prosody[..., 1].clamp_min(2.0), prosody[..., 1])[..., None],
368
- prosody[..., 2:]), dim=-1)
369
- prosody = self.widen_pitch(prosody, frame_mask, pitch_range)
370
- acoustic = self.decode(prosody_hidden, self.hard_prosody(prosody), frame_mask)
371
- # One noise stream per row, drawn at the row's own length, so batching or
372
- # padding never changes a row's audio.
373
- noise = nn.utils.rnn.pad_sequence([torch.randn(int(n) * self.config.hop, generator=torch.Generator().manual_seed(seed + row))
374
- for row, n in enumerate(frame_lengths)], batch_first=True)
375
- pitch = prosody[..., 0] + pitch_shift
376
- if pitch_jitter > 0:
377
- pitch = pitch + self.pitch_wobble(pitch.shape, pitch_jitter, seed, pitch.device)
378
- wave = self.vocoder(acoustic, self.frequency(pitch), frame_lengths, noise=noise.to(hidden.device))
379
- return wave, frame_lengths * self.config.hop, {"durations": durations, "prosody": prosody, "frame_mask": frame_mask}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tiny_voca/text.py DELETED
@@ -1,219 +0,0 @@
1
- """Japanese front end: text -> phoneme tokens with a pitch-accent feature.
2
-
3
- The same code runs during annotation, training-set preparation and inference,
4
- so tokens can never drift between them.
5
-
6
- * pyopenjtalk (OpenJTalk with its NAIST-JDIC dictionary) reads the text: kanji
7
- readings, numbers, and the accent phrases with their accent types.
8
- * A token is one OpenJTalk phoneme (``k``, ``sh``, ``a``, ``N``, ``cl``, ...).
9
- A long vowel is two tokens (``o o``), one per mora. Vowels that OpenJTalk
10
- devoices (``desU``, ``shIta``) are their own tokens, ``A I U E O``: they are
11
- not voiced, and the voicing rules for vowels must not touch them.
12
- * The per-token feature that the model calls ``stress`` carries the Tokyo
13
- pitch-accent tone of the token's mora: 1 low, 2 high, 0 for everything that
14
- is not a phone. It is derived from the accent type and the mora position in
15
- the accent phrase (full-context fields F1/F2 and A2).
16
- * Accent-phrase boundaries are ``<sp>`` tokens, punctuation marks are tokens,
17
- and so are the utterance edges. They may last zero frames and carry the
18
- pauses when the recording has them.
19
- """
20
-
21
- from dataclasses import dataclass
22
- import hashlib
23
- import json
24
- import re
25
- import unicodedata
26
-
27
- FRONTEND_VERSION = 1
28
-
29
- PAD, BOS, EOS, SP, UNK = "<pad>", "<bos>", "<eos>", "<sp>", "<unk>"
30
- SPECIALS = (PAD, BOS, EOS, SP, UNK)
31
- PUNCTUATION = (",", ".", "?", "!", ";", ":", '"', "(", ")", "—", "…")
32
- CONSONANTS = ("k", "ky", "kw", "g", "gy", "gw", "s", "sh", "z", "j", "t", "ty", "ch", "ts", "d", "dy", "n", "ny",
33
- "h", "hy", "f", "b", "by", "p", "py", "m", "my", "y", "r", "ry", "w", "v", "N", "cl")
34
- VOWELS = ("a", "i", "u", "e", "o")
35
- DEVOICED = ("A", "I", "U", "E", "O")
36
- PHONES = CONSONANTS + VOWELS + DEVOICED
37
- # Append-only: changing the order or removing a symbol invalidates checkpoints.
38
- SYMBOLS = SPECIALS + PUNCTUATION + PHONES
39
- SYMBOL_TO_ID = {symbol: index for index, symbol in enumerate(SYMBOLS)}
40
- PHONE_SET = frozenset(PHONES)
41
- PHONE_IDS = frozenset(SYMBOL_TO_ID[p] for p in PHONES) | {SYMBOL_TO_ID[UNK]}
42
- VOICELESS = frozenset(("k", "ky", "kw", "s", "sh", "t", "ty", "ch", "ts", "h", "hy", "f", "p", "py", "cl") + DEVOICED)
43
- VOWEL_SET = frozenset(VOWELS)
44
- LOW, HIGH = 1, 2
45
-
46
- # Full-width and typographic marks to the inventory. NFKC already turns
47
- # full-width ASCII (?!,.:;()) into ASCII.
48
- _MARKS = str.maketrans({"、": ",", ",": ",", "。": ".", "。": ".", "、": ",", "‥": "…", "―": "—", "–": "—",
49
- "「": '"', "」": '"', "『": '"', "』": '"', "“": '"', "”": '"', "„": '"', "«": '"', "»": '"',
50
- "【": "(", "】": ")", "〔": "(", "〕": ")", "[": "(", "]": ")", "[": "(", "]": ")"})
51
- _PUNCTUATION_CHARS = frozenset(PUNCTUATION)
52
- # Separate words but say nothing: a boundary between accent phrases.
53
- _SEPARATORS = frozenset("・ ")
54
- # The phone, its mora position in the accent phrase (A2), the phrase's mora
55
- # count and accent type (F1, F2) and its position in the breath group (F5).
56
- _LABEL = re.compile(r"-(?P<phone>[^+]+)\+.*?/A:(?:-?\d+|xx)\+(?P<a2>\d+|xx)\+.*?/F:(?P<f1>\d+|xx)_(?P<f2>\d+|xx)"
57
- r"#(?:\d+|xx)_(?:\d+|xx)@(?P<f5>\d+|xx)_")
58
-
59
-
60
- class NothingToPronounce(ValueError):
61
- """The text has nothing to say: only marks, emoji and the like."""
62
-
63
-
64
- @dataclass(frozen=True)
65
- class Phonemized:
66
- symbols: tuple
67
- ids: tuple
68
- stress: tuple # pitch-accent tone per token: 0 none, 1 low, 2 high
69
- unknown: tuple # OpenJTalk phonemes that are not in the inventory
70
-
71
- def __len__(self):
72
- return len(self.ids)
73
-
74
-
75
- def _pyopenjtalk():
76
- try:
77
- import pyopenjtalk
78
- except ImportError as error:
79
- raise RuntimeError("the Japanese front end needs pyopenjtalk: pip install pyopenjtalk") from error
80
- return pyopenjtalk
81
-
82
-
83
- def frontend_identity():
84
- return {"frontend_version": FRONTEND_VERSION, "phonemizer": "pyopenjtalk", "symbols": list(SYMBOLS),
85
- "tone": "Tokyo H/L from accent type and mora position"}
86
-
87
-
88
- def frontend_runtime():
89
- """What an annotation run records about the program behind its tokens."""
90
- return {"pyopenjtalk": _pyopenjtalk().__version__}
91
-
92
-
93
- def frontend_fingerprint():
94
- return hashlib.sha256(json.dumps(frontend_identity(), sort_keys=True, ensure_ascii=False).encode()).hexdigest()
95
-
96
-
97
- def normalize_text(text):
98
- text = unicodedata.normalize("NFKC", text).translate(_MARKS)
99
- text = re.sub(r"\.{3,}", "…", text)
100
- text = re.sub(r"-{2,}|\s-\s", "—", text)
101
- return re.sub(r"\s+", " ", text).strip()
102
-
103
-
104
- def chunks(text):
105
- """Yield ``(is_punctuation, piece)`` for normalized text. Decimal points and
106
- digit-group commas stay inside the text piece for OpenJTalk; a separator
107
- (space, ・) ends a piece without a token of its own."""
108
- start = 0
109
- for i, char in enumerate(text):
110
- if char in _SEPARATORS:
111
- if text[start:i].strip():
112
- yield False, text[start:i].strip()
113
- start = i + 1
114
- continue
115
- if char not in _PUNCTUATION_CHARS:
116
- continue
117
- previous = text[i - 1] if i else ""
118
- following = text[i + 1] if i + 1 < len(text) else ""
119
- if char in ".,:" and previous.isdigit() and following.isdigit():
120
- continue
121
- if text[start:i].strip():
122
- yield False, text[start:i].strip()
123
- yield True, char
124
- start = i + 1
125
- if text[start:].strip():
126
- yield False, text[start:].strip()
127
-
128
-
129
- def tone(mora_position, accent_type):
130
- """Tokyo pitch accent of a mora: high or low.
131
-
132
- Type 0 (heiban) rises after the first mora and stays high; type 1 starts
133
- high and falls; type n > 1 rises after the first mora and falls after the
134
- n-th."""
135
- if accent_type == 1:
136
- return HIGH if mora_position == 1 else LOW
137
- if mora_position == 1:
138
- return LOW
139
- return HIGH if accent_type == 0 or mora_position <= accent_type else LOW
140
-
141
-
142
- def accent_phrases(labels):
143
- """OpenJTalk full-context labels -> accent phrases as lists of ``(phone, tone)``.
144
- A pause inside a piece also ends the phrase."""
145
- phrases, current, phrase = [], [], None
146
- for label in labels:
147
- match = _LABEL.search(label)
148
- if match is None:
149
- raise ValueError(f"unexpected OpenJTalk label: {label}")
150
- if match["phone"] in ("sil", "pau") or match["a2"] == "xx":
151
- if current:
152
- phrases.append(current)
153
- current, phrase = [], None
154
- continue
155
- if current and match["f5"] != phrase:
156
- phrases.append(current)
157
- current = []
158
- current.append((match["phone"], tone(int(match["a2"]), int(match["f2"]))))
159
- phrase = match["f5"]
160
- if current:
161
- phrases.append(current)
162
- return phrases
163
-
164
-
165
- def tokenize(pieces):
166
- """Build the token sequence from ``(is_punctuation, value)`` pieces where a
167
- text piece's value is a list of accent phrases, each a list of
168
- ``(phone, tone)``. Pure function (testable without OpenJTalk)."""
169
- symbols, stress, unknown = [BOS], [0], []
170
- for punctuation, value in pieces:
171
- if punctuation:
172
- if symbols[-1] != value: # collapse repeats such as "!!"
173
- symbols.append(value)
174
- stress.append(0)
175
- continue
176
- for phrase in value:
177
- if symbols[-1] in PHONE_SET or symbols[-1] == UNK:
178
- symbols.append(SP)
179
- stress.append(0)
180
- for phone, level in phrase:
181
- if phone not in PHONE_SET:
182
- unknown.append(phone)
183
- phone = UNK
184
- symbols.append(phone)
185
- stress.append(level)
186
- symbols.append(EOS)
187
- stress.append(0)
188
- if not any(s in PHONE_SET or s == UNK for s in symbols):
189
- raise NothingToPronounce("the text has nothing to pronounce")
190
- return Phonemized(tuple(symbols), tuple(SYMBOL_TO_ID[s] for s in symbols), tuple(stress), tuple(unknown))
191
-
192
-
193
- def phonemize(text):
194
- if not isinstance(text, str) or not text.strip():
195
- raise ValueError("text must not be empty")
196
- openjtalk = _pyopenjtalk()
197
- pieces = []
198
- for punctuation, piece in chunks(normalize_text(text)):
199
- pieces.append((True, piece) if punctuation else (False, accent_phrases(openjtalk.extract_fullcontext(piece))))
200
- return tokenize(pieces)
201
-
202
-
203
- def is_phone(token_ids):
204
- """Boolean list/tensor helper: tokens that must be realised as audio."""
205
- return [int(i) in PHONE_IDS for i in token_ids]
206
-
207
-
208
- def morae(symbols):
209
- """The morae of a token sequence as strings of their phones (``"ky o"``,
210
- ``"N"``, ``"cl"``). A mora ends with a vowel, a devoiced vowel, ``N`` or
211
- ``cl``; any other token in between is skipped."""
212
- result, onset = [], []
213
- for symbol in symbols:
214
- if symbol in VOWEL_SET or symbol in DEVOICED or symbol in ("N", "cl"):
215
- result.append(" ".join(onset + [symbol]))
216
- onset = []
217
- elif symbol in PHONE_SET:
218
- onset.append(symbol)
219
- return result