mlboydaisuke commited on
Commit
43e9990
·
verified ·
1 Parent(s): 4a9bdd8

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +63 -16
README.md CHANGED
@@ -27,6 +27,9 @@ vocode d, t_en, aln (1, N, F) fp32, ref_s
27
  - **Source**: [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) — 81.8M parameters
28
  - **License**: apache-2.0
29
  - **Voices**: 54, shipped separately as `voices/*.pt` in the source repo
 
 
 
30
 
31
  Both axes — phonemes and frames — are **dynamic**, and both are **exact**. Nothing is
32
  padded, nothing is stretched, and there is no ladder of fixed-size methods. That took
@@ -46,6 +49,11 @@ ids = [vocab[c] for c in ps if c in vocab] # config.json's 178-entry vo
46
  input_ids = torch.LongTensor([[0, *ids, 0]]) # wrapped in the boundary token
47
  ```
48
 
 
 
 
 
 
49
  **2. Pick the style row by phoneme count.** The voice pack is 510 rows and the row is
50
  `pack[len(ps) - 1]` — indexed by the length of the phoneme **string**, which is what
51
  upstream's own pipeline does, and not by the number of ids, which is smaller whenever
@@ -130,11 +138,13 @@ minutes.
130
 
131
  ## Verification
132
 
133
- Five English sentences through `misaki`, each one synthesised by the `.pte` and by the
134
- unmodified upstream model in eager. Three gates, because no single one works here:
 
135
 
136
  **Durations must match exactly** — they are integers, they decide the rhythm, and they
137
- come out of the half of the model that has no noise in it. **5 of 5 exact.**
 
138
 
139
  **Waveform correlation is not usable.** The vocoder's excitation carries Gaussian noise
140
  and a random initial phase, so the eager model does not reproduce itself: two runs of
@@ -146,14 +156,31 @@ second is the first:
146
 
147
  | | log-mel vs eager | eager's own floor | ratio |
148
  |---|---|---|---|
149
- | worst of five | 0.0427 | 0.0412 | **1.04x** |
150
- | best of five | 0.0442 | 0.0449 | 0.98x |
151
 
152
  Below and above 1.0 across the five, which is what "indistinguishable from running it
153
- again" looks like. For scale, one extra frame of padding shows up at 4.4x.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
154
 
155
- **Transcripts**, through Qwen3-ASR: **CER 0.0000** on all five. Synthetic speech read
156
- by a recogniser is a control, not a benchmark it says the file still says the words.
 
 
 
157
 
158
  ## Speed
159
 
@@ -162,9 +189,9 @@ by a recogniser is a control, not a benchmark — it says the file still says th
162
  | | ms |
163
  |---|---|
164
  | `predict` | 56.6 |
165
- | `vocode` | 443.7 |
166
- | **total** | **500.3** — 6.5x faster than real time |
167
- | the same utterance in eager PyTorch | 227.5 |
168
 
169
  Measured on a machine that was busy, so read these as a floor rather than a number to
170
  quote. Eager is faster here and that is expected: it reaches Accelerate's LSTM and
@@ -173,11 +200,31 @@ convolution kernels, while about half the graph runs on portable kernels
173
 
174
  ## What is not in this file
175
 
176
- **No fp16 or int8 build.** The int8 decision is measured, not assumed: dynamic int8
177
- quantises `nn.Linear` only, and Kokoro is **66.9% Conv1d** with just **15.9%** of its
178
- weights in Linear layers, so it would touch a sixth of the file. Static int8 would
179
- reach the convolutions, but a vocoder's quality under quantisation has to be measured
180
- per task rather than declared, and that has not been done here.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
181
 
182
  **The vocoder is partitioned without XNNPACK's `PermuteConfig`.** A permute inside the
183
  decoder feeds two consumers outside its partition, and the delegate's output list then
 
27
  - **Source**: [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) — 81.8M parameters
28
  - **License**: apache-2.0
29
  - **Voices**: 54, shipped separately as `voices/*.pt` in the source repo
30
+ - **Languages**: the graph is language-blind — it takes phoneme ids. **Verified here in
31
+ English and Japanese**; the voice pack also carries Spanish, French, Hindi, Italian,
32
+ Portuguese and Chinese voices, which are not verified on this card.
33
 
34
  Both axes — phonemes and frames — are **dynamic**, and both are **exact**. Nothing is
35
  padded, nothing is stretched, and there is no ladder of fixed-size methods. That took
 
49
  input_ids = torch.LongTensor([[0, *ids, 0]]) # wrapped in the boundary token
50
  ```
51
 
52
+ For Japanese it is `misaki.ja.JAG2P()` and a `j*` voice, and nothing else changes — the
53
+ ids go into the same graph. Install `misaki[ja]` **with `unidic-lite`**: `unidic`'s
54
+ dictionary is a separate 250 MB download, and fugashi prefers `unidic` when both are
55
+ present, which leaves you with an empty dicdir and an error about `mecabrc`.
56
+
57
  **2. Pick the style row by phoneme count.** The voice pack is 510 rows and the row is
58
  `pack[len(ps) - 1]` — indexed by the length of the phoneme **string**, which is what
59
  upstream's own pipeline does, and not by the number of ids, which is smaller whenever
 
138
 
139
  ## Verification
140
 
141
+ Ten sentences through `misaki` five English on `af_heart`, five Japanese on
142
+ `jf_alpha` each one synthesised by the `.pte` and by the unmodified upstream model in
143
+ eager. Three gates, because no single one works here:
144
 
145
  **Durations must match exactly** — they are integers, they decide the rhythm, and they
146
+ come out of the half of the model that has no noise in it. **10 of 10 exact, both
147
+ languages.**
148
 
149
  **Waveform correlation is not usable.** The vocoder's excitation carries Gaussian noise
150
  and a random initial phase, so the eager model does not reproduce itself: two runs of
 
156
 
157
  | | log-mel vs eager | eager's own floor | ratio |
158
  |---|---|---|---|
159
+ | worst of ten | 0.0474 | 0.0455 | **1.04x** |
160
+ | best of ten | 0.0393 | 0.0408 | 0.96x |
161
 
162
  Below and above 1.0 across the five, which is what "indistinguishable from running it
163
+ again" looks like and it moves a few percent between runs, because the noise source
164
+ is in both arms. For scale, one extra frame of padding shows up at 4.4x, and the fp16
165
+ build below at **73x**.
166
+
167
+ **Transcripts**, through Qwen3-ASR, scored against **eager's own transcript** rather
168
+ than against the sentence — what is under test is the conversion, and a recogniser
169
+ choosing a different kanji is not the file's doing. **CER 0.0000 on nine of ten.**
170
+
171
+ The tenth is worth writing down, because it is the recogniser and not the model:
172
+
173
+ ```
174
+ この電車は東京駅に止まりますか (does this train stop at Tokyo Station)
175
+ eager ...東京駅に泊まりますか (stay overnight) CER 0.067
176
+ pte ...東京駅に停まりますか (stop)
177
+ ```
178
 
179
+ Same reading, different kanji and **eager disagrees with itself here**: four runs of
180
+ the identical input gave once and three times, and the `.pte` did the same. The
181
+ vocoder's noise is enough to tip a near-tie in the recogniser. Durations are identical
182
+ and log-mel is 1.02x the floor on this clip, so the two files are as close as eager is
183
+ to itself. **An ASR-only gate would have recorded this as a defect.**
184
 
185
  ## Speed
186
 
 
189
  | | ms |
190
  |---|---|
191
  | `predict` | 56.6 |
192
+ | `vocode` | 441.4 |
193
+ | **total** | **498.0** — 6.5x faster than real time |
194
+ | the same utterance in eager PyTorch | 225.5 |
195
 
196
  Measured on a machine that was busy, so read these as a floor rather than a number to
197
  quote. Eager is faster here and that is expected: it reaches Accelerate's LSTM and
 
200
 
201
  ## What is not in this file
202
 
203
+ **fp16 was built and withdrawn.** It fails three ways at once, and the first one is not
204
+ about ExecuTorch at all:
205
+
206
+ | | fp32 (this file) | fp16 (withdrawn) |
207
+ |---|---|---|
208
+ | size | 325.4 MB | 278.7 MB — a 14% saving, not the 50% the weights imply |
209
+ | worst log-mel / noise floor | **0.99x** | **73x** |
210
+ | worst CER | **0.0000** | **1.0000** — nothing intelligible |
211
+ | 3.25 s of audio | **498 ms** | 2104 ms |
212
+
213
+ **Halving this model breaks it in eager PyTorch, before any export.** At 96 phonemes the
214
+ vocoder returns `nan`; at 24 it survives but sits at 2.8x the noise floor. Keeping the
215
+ 73 data-statistic norm layers in fp32 — the usual fix for `InstanceNorm` overflow — does
216
+ not save it, so the overflow is not only in the norms. One duration in 96 also flips,
217
+ which is a rounding tie rather than a numerical failure.
218
+
219
+ The size is the least of it but worth knowing: the weights **do** halve to 163.4 MB, and
220
+ the file is still 278.7 MB, because the XNNPACK delegate carries its own copy of the
221
+ weights it takes.
222
+
223
+ **No int8 build.** Measured, not assumed: dynamic int8 quantises `nn.Linear` only, and
224
+ Kokoro is **66.9% Conv1d** with just **15.9%** of its weights in Linear layers, so it
225
+ would touch a sixth of the file. Static int8 would reach the convolutions, but a
226
+ vocoder's quality under quantisation has to be measured per task rather than declared,
227
+ and that has not been done here.
228
 
229
  **The vocoder is partitioned without XNNPACK's `PermuteConfig`.** A permute inside the
230
  decoder feeds two consumers outside its partition, and the delegate's output list then