Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -27,6 +27,9 @@ vocode d, t_en, aln (1, N, F) fp32, ref_s
|
|
| 27 |
- **Source**: [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) — 81.8M parameters
|
| 28 |
- **License**: apache-2.0
|
| 29 |
- **Voices**: 54, shipped separately as `voices/*.pt` in the source repo
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
Both axes — phonemes and frames — are **dynamic**, and both are **exact**. Nothing is
|
| 32 |
padded, nothing is stretched, and there is no ladder of fixed-size methods. That took
|
|
@@ -46,6 +49,11 @@ ids = [vocab[c] for c in ps if c in vocab] # config.json's 178-entry vo
|
|
| 46 |
input_ids = torch.LongTensor([[0, *ids, 0]]) # wrapped in the boundary token
|
| 47 |
```
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
**2. Pick the style row by phoneme count.** The voice pack is 510 rows and the row is
|
| 50 |
`pack[len(ps) - 1]` — indexed by the length of the phoneme **string**, which is what
|
| 51 |
upstream's own pipeline does, and not by the number of ids, which is smaller whenever
|
|
@@ -130,11 +138,13 @@ minutes.
|
|
| 130 |
|
| 131 |
## Verification
|
| 132 |
|
| 133 |
-
|
| 134 |
-
|
|
|
|
| 135 |
|
| 136 |
**Durations must match exactly** — they are integers, they decide the rhythm, and they
|
| 137 |
-
come out of the half of the model that has no noise in it. **
|
|
|
|
| 138 |
|
| 139 |
**Waveform correlation is not usable.** The vocoder's excitation carries Gaussian noise
|
| 140 |
and a random initial phase, so the eager model does not reproduce itself: two runs of
|
|
@@ -146,14 +156,31 @@ second is the first:
|
|
| 146 |
|
| 147 |
| | log-mel vs eager | eager's own floor | ratio |
|
| 148 |
|---|---|---|---|
|
| 149 |
-
| worst of
|
| 150 |
-
| best of
|
| 151 |
|
| 152 |
Below and above 1.0 across the five, which is what "indistinguishable from running it
|
| 153 |
-
again" looks like
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
|
| 155 |
-
|
| 156 |
-
|
|
|
|
|
|
|
|
|
|
| 157 |
|
| 158 |
## Speed
|
| 159 |
|
|
@@ -162,9 +189,9 @@ by a recogniser is a control, not a benchmark — it says the file still says th
|
|
| 162 |
| | ms |
|
| 163 |
|---|---|
|
| 164 |
| `predict` | 56.6 |
|
| 165 |
-
| `vocode` |
|
| 166 |
-
| **total** | **
|
| 167 |
-
| the same utterance in eager PyTorch |
|
| 168 |
|
| 169 |
Measured on a machine that was busy, so read these as a floor rather than a number to
|
| 170 |
quote. Eager is faster here and that is expected: it reaches Accelerate's LSTM and
|
|
@@ -173,11 +200,31 @@ convolution kernels, while about half the graph runs on portable kernels
|
|
| 173 |
|
| 174 |
## What is not in this file
|
| 175 |
|
| 176 |
-
**
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
|
| 182 |
**The vocoder is partitioned without XNNPACK's `PermuteConfig`.** A permute inside the
|
| 183 |
decoder feeds two consumers outside its partition, and the delegate's output list then
|
|
|
|
| 27 |
- **Source**: [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) — 81.8M parameters
|
| 28 |
- **License**: apache-2.0
|
| 29 |
- **Voices**: 54, shipped separately as `voices/*.pt` in the source repo
|
| 30 |
+
- **Languages**: the graph is language-blind — it takes phoneme ids. **Verified here in
|
| 31 |
+
English and Japanese**; the voice pack also carries Spanish, French, Hindi, Italian,
|
| 32 |
+
Portuguese and Chinese voices, which are not verified on this card.
|
| 33 |
|
| 34 |
Both axes — phonemes and frames — are **dynamic**, and both are **exact**. Nothing is
|
| 35 |
padded, nothing is stretched, and there is no ladder of fixed-size methods. That took
|
|
|
|
| 49 |
input_ids = torch.LongTensor([[0, *ids, 0]]) # wrapped in the boundary token
|
| 50 |
```
|
| 51 |
|
| 52 |
+
For Japanese it is `misaki.ja.JAG2P()` and a `j*` voice, and nothing else changes — the
|
| 53 |
+
ids go into the same graph. Install `misaki[ja]` **with `unidic-lite`**: `unidic`'s
|
| 54 |
+
dictionary is a separate 250 MB download, and fugashi prefers `unidic` when both are
|
| 55 |
+
present, which leaves you with an empty dicdir and an error about `mecabrc`.
|
| 56 |
+
|
| 57 |
**2. Pick the style row by phoneme count.** The voice pack is 510 rows and the row is
|
| 58 |
`pack[len(ps) - 1]` — indexed by the length of the phoneme **string**, which is what
|
| 59 |
upstream's own pipeline does, and not by the number of ids, which is smaller whenever
|
|
|
|
| 138 |
|
| 139 |
## Verification
|
| 140 |
|
| 141 |
+
Ten sentences through `misaki` — five English on `af_heart`, five Japanese on
|
| 142 |
+
`jf_alpha` — each one synthesised by the `.pte` and by the unmodified upstream model in
|
| 143 |
+
eager. Three gates, because no single one works here:
|
| 144 |
|
| 145 |
**Durations must match exactly** — they are integers, they decide the rhythm, and they
|
| 146 |
+
come out of the half of the model that has no noise in it. **10 of 10 exact, both
|
| 147 |
+
languages.**
|
| 148 |
|
| 149 |
**Waveform correlation is not usable.** The vocoder's excitation carries Gaussian noise
|
| 150 |
and a random initial phase, so the eager model does not reproduce itself: two runs of
|
|
|
|
| 156 |
|
| 157 |
| | log-mel vs eager | eager's own floor | ratio |
|
| 158 |
|---|---|---|---|
|
| 159 |
+
| worst of ten | 0.0474 | 0.0455 | **1.04x** |
|
| 160 |
+
| best of ten | 0.0393 | 0.0408 | 0.96x |
|
| 161 |
|
| 162 |
Below and above 1.0 across the five, which is what "indistinguishable from running it
|
| 163 |
+
again" looks like — and it moves a few percent between runs, because the noise source
|
| 164 |
+
is in both arms. For scale, one extra frame of padding shows up at 4.4x, and the fp16
|
| 165 |
+
build below at **73x**.
|
| 166 |
+
|
| 167 |
+
**Transcripts**, through Qwen3-ASR, scored against **eager's own transcript** rather
|
| 168 |
+
than against the sentence — what is under test is the conversion, and a recogniser
|
| 169 |
+
choosing a different kanji is not the file's doing. **CER 0.0000 on nine of ten.**
|
| 170 |
+
|
| 171 |
+
The tenth is worth writing down, because it is the recogniser and not the model:
|
| 172 |
+
|
| 173 |
+
```
|
| 174 |
+
この電車は東京駅に止まりますか (does this train stop at Tokyo Station)
|
| 175 |
+
eager ...東京駅に泊まりますか (stay overnight) CER 0.067
|
| 176 |
+
pte ...東京駅に停まりますか (stop)
|
| 177 |
+
```
|
| 178 |
|
| 179 |
+
Same reading, different kanji — and **eager disagrees with itself here**: four runs of
|
| 180 |
+
the identical input gave 泊 once and 停 three times, and the `.pte` did the same. The
|
| 181 |
+
vocoder's noise is enough to tip a near-tie in the recogniser. Durations are identical
|
| 182 |
+
and log-mel is 1.02x the floor on this clip, so the two files are as close as eager is
|
| 183 |
+
to itself. **An ASR-only gate would have recorded this as a defect.**
|
| 184 |
|
| 185 |
## Speed
|
| 186 |
|
|
|
|
| 189 |
| | ms |
|
| 190 |
|---|---|
|
| 191 |
| `predict` | 56.6 |
|
| 192 |
+
| `vocode` | 441.4 |
|
| 193 |
+
| **total** | **498.0** — 6.5x faster than real time |
|
| 194 |
+
| the same utterance in eager PyTorch | 225.5 |
|
| 195 |
|
| 196 |
Measured on a machine that was busy, so read these as a floor rather than a number to
|
| 197 |
quote. Eager is faster here and that is expected: it reaches Accelerate's LSTM and
|
|
|
|
| 200 |
|
| 201 |
## What is not in this file
|
| 202 |
|
| 203 |
+
**fp16 was built and withdrawn.** It fails three ways at once, and the first one is not
|
| 204 |
+
about ExecuTorch at all:
|
| 205 |
+
|
| 206 |
+
| | fp32 (this file) | fp16 (withdrawn) |
|
| 207 |
+
|---|---|---|
|
| 208 |
+
| size | 325.4 MB | 278.7 MB — a 14% saving, not the 50% the weights imply |
|
| 209 |
+
| worst log-mel / noise floor | **0.99x** | **73x** |
|
| 210 |
+
| worst CER | **0.0000** | **1.0000** — nothing intelligible |
|
| 211 |
+
| 3.25 s of audio | **498 ms** | 2104 ms |
|
| 212 |
+
|
| 213 |
+
**Halving this model breaks it in eager PyTorch, before any export.** At 96 phonemes the
|
| 214 |
+
vocoder returns `nan`; at 24 it survives but sits at 2.8x the noise floor. Keeping the
|
| 215 |
+
73 data-statistic norm layers in fp32 — the usual fix for `InstanceNorm` overflow — does
|
| 216 |
+
not save it, so the overflow is not only in the norms. One duration in 96 also flips,
|
| 217 |
+
which is a rounding tie rather than a numerical failure.
|
| 218 |
+
|
| 219 |
+
The size is the least of it but worth knowing: the weights **do** halve to 163.4 MB, and
|
| 220 |
+
the file is still 278.7 MB, because the XNNPACK delegate carries its own copy of the
|
| 221 |
+
weights it takes.
|
| 222 |
+
|
| 223 |
+
**No int8 build.** Measured, not assumed: dynamic int8 quantises `nn.Linear` only, and
|
| 224 |
+
Kokoro is **66.9% Conv1d** with just **15.9%** of its weights in Linear layers, so it
|
| 225 |
+
would touch a sixth of the file. Static int8 would reach the convolutions, but a
|
| 226 |
+
vocoder's quality under quantisation has to be measured per task rather than declared,
|
| 227 |
+
and that has not been done here.
|
| 228 |
|
| 229 |
**The vocoder is partitioned without XNNPACK's `PermuteConfig`.** A permute inside the
|
| 230 |
decoder feeds two consumers outside its partition, and the delegate's output list then
|