Buckets:

adi-is-dev
/

tasker

Files

xet

adi-is-dev/tasker / README.md

adi-is-dev

4 months ago

preview code

download

raw

5.83 kB

---
license: apache-2.0
library_name: litert
pipeline_tag: text-to-speech
base_model: hexgrad/Kokoro-82M
tags:
  - litert
  - tflite
  - tts
  - text-to-speech
  - on-device
  - kokoro
  - free-text
  - preview
---
# Kokoro-82M — LiteRT (free-text, bucketed)

> ⚠️ **Labeled preview — FP32, CPU.** Arbitrary free text → speech (not a single baked
> sentence). The neural graphs are LiteRT `.tflite`; two small steps run host-side (the
> hn-NSF source STFT and the final iSTFT overlap-add). GPU + quantization are the next steps.

A [LiteRT](https://ai.google.dev/edge/litert) (`.tflite`) conversion of
[hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (StyleTTS2 + ISTFTNet) for
on-device **free-text** text-to-speech (arbitrary text in, 24 kHz speech out). **Audio fidelity
≈ 0.9994 magnitude-spectrogram correlation to the PyTorch reference** (verified across multiple
held-out sentences, not just the export sample).

Kokoro has one data-dependent length (the duration→alignment expansion `L = sum(pred_dur)`),
which `litert_torch` cannot keep dynamic (the LSTM sequence axis specializes). It is therefore
split into **three fixed-bucket bundles** with host steps between them, so arbitrary text works
by left-padding to the bucket and trimming the output (longer text is split into sentences
host-side, each ≤ the bucket):

```
text --(G2P, host)--> phoneme ids
1. kokoro_predictor.tflite : ids[1,128], ref_s[1,256], attn[1,128] -> duration, d, t_en
   host: pred_dur = round(duration); alignment one-hot aln[1,128,512]; frame_mask[1,512]
2. kokoro_prosody.tflite   : d, t_en, aln, ref_s, frame_mask       -> asr, F0, N
   host: har = STFT(SineGen(f0_upsamp(F0)))     (the hn-NSF excitation)
3. kokoro_vocoder.tflite   : asr, F0, N, har, ref_s, frame_mask    -> spec, phase
   host: iSTFT overlap-add(spec, phase) -> 24 kHz waveform; trim to L*600
```

## Files

| File | Precision | Size | Role |
|---|---|---|---|
| `kokoro_predictor.tflite` | fp32 | ~91 MB | PL-BERT + duration/text encoders (masked unrolled bi-LSTMs) |
| `kokoro_prosody.tflite` | fp32 | ~37 MB | shared prosody LSTM + F0/N (masked) |
| `kokoro_vocoder.tflite` | fp32 | ~236 MB | iSTFTNet decoder → magnitude/phase spectrogram |
| `istft_Wr_f32.bin`, `istft_Wi_f32.bin` | fp32 | 880 B each | inverse-DFT bases for the host-side iSTFT |

Token bucket `T = 128`, frame bucket `L = 512` (≈ 12.8 s of audio per chunk at 24 kHz).
Bundles are **voice-independent** — the voice is the `ref_s` input (a `voices/*.pt` from the
base repo, indexed by token-sequence length).

## Specs

| | |
|---|---|
| Task | Text-to-speech (English), free text, 24 kHz mono |
| Source | [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (StyleTTS2 + ISTFTNet) |
| Fidelity | **magspec-corr 0.9994** vs PyTorch (waveform corr ≈ 0.98 — the bounded bucket pad-boundary effect; the spectrum is what's perceived) |
| Runtime | CPU (LiteRT CompiledModel API) |

## How it was converted
- Stock official converter (`litert_torch`), **general path** — Kokoro is StyleTTS2/ISTFTNet, not a transformer LLM, so the Generative-API re-authoring path does not apply.
- **Three fixed-bucket bundles** because the dynamic alignment length can't stay symbolic through the converter (the dynamic-LSTM wall). Every workaround is load-bearing and numerically faithful: the 6 bidirectional LSTMs are **unrolled as masked bi-LSTMs** that carry state through right-padding (a fused `nn.LSTM` leaks pad tokens into the backward pass and wrecks prosody); the 58 AdaIN InstanceNorms **normalize over real frames only** so bucket pad frames don't poison the statistics; the hn-NSF **source STFT runs host-side** (its atan2 phase flips at the F0→0 pad boundary on-device).
- **iSTFT runs host-side** (vocoder emits spec+phase): the in-graph conv-transpose iSTFT hits a converter weight-dedup bug that fuses the cos/sin DFT bases. The host overlap-add (no learned weights) is numerically exact.

## Training data
Inherited from [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M): a few hundred hours of **permissive / non-copyrighted audio** — public-domain audio, audio under permissive licenses (e.g. Koniwa `tnc` CC BY 3.0, SIWIS CC BY 4.0), and synthetic audio from large-provider TTS — paired with IPA phoneme labels. No proprietary or scraped personal recordings, no custom voice clones. This LiteRT artifact is a format conversion of the released weights and introduces no additional training data.

## PII
No personally identifiable information is included. Per the base model's disclosure the training audio is permissive / public-domain / synthetic rather than scraped personal recordings; to the best of our knowledge the released weights contain no PII, and the conversion adds none.

## Front-end (free-text G2P)
For arbitrary text with no dropped words (names, brands, numbers), pair with the neural grapheme-to-phoneme front-end: [mlboydaisuke/Kokoro-G2P-en-US-LiteRT](https://huggingface.co/mlboydaisuke/Kokoro-G2P-en-US-LiteRT).

## Roadmap
- **GPU**: the attention's fused-QKV >4-D layout + mask `EQUAL`/`SELECT` keep it on CPU; decomposing attention to ≤4-D would let it ride the GPU delegate. In-graph iSTFT on GPU additionally needs an FFT kernel in the LiteRT GPU delegate (ML Drift).
- **Quantization** (int8/int4) is the obvious next step.

## Status
Labeled preview — the converted model is parity-verified, but a clean runnable on-device LiteRT sample (type-any-text via the G2P front-end above) is still in progress, so this repo currently ships the model + weights only. (Conversion is a 3-bundle bucketed pipeline; the host steps — alignment, hn-NSF source STFT, iSTFT — run on the host.)

## License
Apache-2.0, inherited from [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M).

Xet Storage Details

Size:: 5.83 kB
Xet hash:: ecc51c660afa0de5a0c9807e70417c68224f9bd75fb2245fa15528e3b1403f1c

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	---
	license: apache-2.0
	library_name: litert
	pipeline_tag: text-to-speech
	base_model: hexgrad/Kokoro-82M
	tags:
	- litert
	- tflite
	- tts
	- text-to-speech
	- on-device
	- kokoro
	- free-text
	- preview
	---

	# Kokoro-82M — LiteRT (free-text, bucketed)

	> ⚠️ Labeled preview — FP32, CPU. Arbitrary free text → speech (not a single baked
	> sentence). The neural graphs are LiteRT `.tflite`; two small steps run host-side (the
	> hn-NSF source STFT and the final iSTFT overlap-add). GPU + quantization are the next steps.

	A [LiteRT](https://ai.google.dev/edge/litert) (`.tflite`) conversion of
	[hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (StyleTTS2 + ISTFTNet) for
	on-device free-text text-to-speech (arbitrary text in, 24 kHz speech out). **Audio fidelity
	≈ 0.9994 magnitude-spectrogram correlation to the PyTorch reference** (verified across multiple
	held-out sentences, not just the export sample).

	Kokoro has one data-dependent length (the duration→alignment expansion `L = sum(pred_dur)`),
	which `litert_torch` cannot keep dynamic (the LSTM sequence axis specializes). It is therefore
	split into three fixed-bucket bundles with host steps between them, so arbitrary text works
	by left-padding to the bucket and trimming the output (longer text is split into sentences
	host-side, each ≤ the bucket):

	```
	text --(G2P, host)--> phoneme ids
	1. kokoro_predictor.tflite : ids[1,128], ref_s[1,256], attn[1,128] -> duration, d, t_en
	host: pred_dur = round(duration); alignment one-hot aln[1,128,512]; frame_mask[1,512]
	2. kokoro_prosody.tflite : d, t_en, aln, ref_s, frame_mask -> asr, F0, N
	host: har = STFT(SineGen(f0_upsamp(F0))) (the hn-NSF excitation)
	3. kokoro_vocoder.tflite : asr, F0, N, har, ref_s, frame_mask -> spec, phase
	host: iSTFT overlap-add(spec, phase) -> 24 kHz waveform; trim to L*600
	```

	## Files

	\| File \| Precision \| Size \| Role \|
	\|---\|---\|---\|---\|
	\| `kokoro_predictor.tflite` \| fp32 \| ~91 MB \| PL-BERT + duration/text encoders (masked unrolled bi-LSTMs) \|
	\| `kokoro_prosody.tflite` \| fp32 \| ~37 MB \| shared prosody LSTM + F0/N (masked) \|
	\| `kokoro_vocoder.tflite` \| fp32 \| ~236 MB \| iSTFTNet decoder → magnitude/phase spectrogram \|
	\| `istft_Wr_f32.bin`, `istft_Wi_f32.bin` \| fp32 \| 880 B each \| inverse-DFT bases for the host-side iSTFT \|

	Token bucket `T = 128`, frame bucket `L = 512` (≈ 12.8 s of audio per chunk at 24 kHz).
	Bundles are voice-independent — the voice is the `ref_s` input (a `voices/*.pt` from the
	base repo, indexed by token-sequence length).

	## Specs

	\| \| \|
	\|---\|---\|
	\| Task \| Text-to-speech (English), free text, 24 kHz mono \|
	\| Source \| [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (StyleTTS2 + ISTFTNet) \|
	\| Fidelity \| magspec-corr 0.9994 vs PyTorch (waveform corr ≈ 0.98 — the bounded bucket pad-boundary effect; the spectrum is what's perceived) \|
	\| Runtime \| CPU (LiteRT CompiledModel API) \|

	## How it was converted
	- Stock official converter (`litert_torch`), general path — Kokoro is StyleTTS2/ISTFTNet, not a transformer LLM, so the Generative-API re-authoring path does not apply.
	- Three fixed-bucket bundles because the dynamic alignment length can't stay symbolic through the converter (the dynamic-LSTM wall). Every workaround is load-bearing and numerically faithful: the 6 bidirectional LSTMs are unrolled as masked bi-LSTMs that carry state through right-padding (a fused `nn.LSTM` leaks pad tokens into the backward pass and wrecks prosody); the 58 AdaIN InstanceNorms normalize over real frames only so bucket pad frames don't poison the statistics; the hn-NSF source STFT runs host-side (its atan2 phase flips at the F0→0 pad boundary on-device).
	- iSTFT runs host-side (vocoder emits spec+phase): the in-graph conv-transpose iSTFT hits a converter weight-dedup bug that fuses the cos/sin DFT bases. The host overlap-add (no learned weights) is numerically exact.

	## Training data
	Inherited from [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M): a few hundred hours of permissive / non-copyrighted audio — public-domain audio, audio under permissive licenses (e.g. Koniwa `tnc` CC BY 3.0, SIWIS CC BY 4.0), and synthetic audio from large-provider TTS — paired with IPA phoneme labels. No proprietary or scraped personal recordings, no custom voice clones. This LiteRT artifact is a format conversion of the released weights and introduces no additional training data.

	## PII
	No personally identifiable information is included. Per the base model's disclosure the training audio is permissive / public-domain / synthetic rather than scraped personal recordings; to the best of our knowledge the released weights contain no PII, and the conversion adds none.

	## Front-end (free-text G2P)
	For arbitrary text with no dropped words (names, brands, numbers), pair with the neural grapheme-to-phoneme front-end: [mlboydaisuke/Kokoro-G2P-en-US-LiteRT](https://huggingface.co/mlboydaisuke/Kokoro-G2P-en-US-LiteRT).

	## Roadmap
	- GPU: the attention's fused-QKV >4-D layout + mask `EQUAL`/`SELECT` keep it on CPU; decomposing attention to ≤4-D would let it ride the GPU delegate. In-graph iSTFT on GPU additionally needs an FFT kernel in the LiteRT GPU delegate (ML Drift).
	- Quantization (int8/int4) is the obvious next step.

	## Status
	Labeled preview — the converted model is parity-verified, but a clean runnable on-device LiteRT sample (type-any-text via the G2P front-end above) is still in progress, so this repo currently ships the model + weights only. (Conversion is a 3-bundle bucketed pipeline; the host steps — alignment, hn-NSF source STFT, iSTFT — run on the host.)

	## License
	Apache-2.0, inherited from [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M).