| # Kokoro CoreML (HAR-Optimized) |
|
|
| High-performance **Kokoro TTS CoreML conversion** with Apple Neural Engine (ANE) optimized HAR decoder buckets. |
|
|
| This repository contains precompiled `.mlpackage` models for fast on-device speech synthesis on Apple platforms. |
|
|
| --- |
|
|
| *based on this this open source project* |
| https://github.com/mattmireles/kokoro-coreml |
|
|
|
|
|
|
| ## 📦 Included Models |
|
|
| ### 🧠 Duration Model (Stage 1) |
|
|
| - `kokoro_duration.mlpackage` |
|
|
| Handles variable-length text and predicts phoneme durations + intermediate features. |
|
|
| --- |
|
|
| ### 🔊 HAR Decoder Buckets (Stage 2 – ANE Optimized) |
|
|
| Fixed-size audio synthesis models: |
|
|
| - `KokoroDecoder_HAR_1s.mlpackage` |
| - `KokoroDecoder_HAR_2s.mlpackage` |
| - `KokoroDecoder_HAR_3s.mlpackage` |
| - `KokoroDecoder_HAR_5s.mlpackage` |
| - `KokoroDecoder_HAR_8s.mlpackage` |
| - `KokoroDecoder_HAR_10s.mlpackage` |
| - `KokoroDecoder_HAR_15s.mlpackage` |
| - `KokoroDecoder_HAR_20s.mlpackage` |
| - `KokoroDecoder_HAR.mlpackage` |
|
|
| --- |
|
|
| ### 🔁 Decoder-Only Variants |
|
|
| - `kokoro_decoder_only_3s.mlpackage` |
| - `kokoro_decoder_only_5s.mlpackage` |
| - `kokoro_decoder_only_10s.mlpackage` |
|
|
| --- |
|
|
| ### 🎛 F0 / Feature Variants |
|
|
| - `kokoro_f0n_3s.mlpackage` |
| - `kokoro_f0n_5s.mlpackage` |
| - `kokoro_f0n_10s.mlpackage` |
|
|
| --- |
|
|
| ### 🔊 Vocoder Variants |
|
|
| - `KokoroVocoder.mlpackage` |
| - `KokoroVocoder_asr64_f0128.mlpackage` |
| - `KokoroVocoder_asr80_f0160.mlpackage` |
| - `KokoroVocoder_asr96_f0192.mlpackage` |
| - `KokoroVocoder_asr128_f0256.mlpackage` |
| - `KokoroVocoder_asr160_f0320.mlpackage` |
| - `KokoroVocoder_asr200_f0400.mlpackage` |
|
|
| --- |
|
|
| ### 🧪 Experimental / Alternative |
|
|
| - `kokoro_synthesizer_3s.mlpackage` |
| - `kokoro_synthesizer_3s_nolstm.mlpackage` |
| - `StyleTTS2_iSTFTNet_Decoder.mlpackage` |
|
|
| --- |
|
|
| # 📐 Architecture |
|
|
| This CoreML conversion uses a **two-stage pipeline** to support Kokoro’s dynamic operations while maximizing ANE performance. |
|
|
| --- |
|
|
| ## Stage 1 — Duration Model (CPU/GPU) |
|
|
| **Input:** Variable-length text (`ct.RangeDim`) |
| **Process:** Transformer + LSTM duration prediction |
| **Output:** Phoneme durations + intermediate features |
| **Compute:** CPU / GPU |
|
|
| Why CPU? |
| - LSTM layers are not ANE-compatible |
| - Dynamic shape text processing |
|
|
| --- |
|
|
| ## Stage 2 — HAR Decoder (ANE Optimized) |
|
|
| **Input:** |
| - Features from duration model |
| - Alignment matrix (built client-side) |
|
|
| **Process:** |
| Vocoder synthesis using iSTFTNet architecture |
|
|
| **Output:** |
| 24kHz waveform audio |
|
|
| **Compute:** |
| Apple Neural Engine |
|
|
| --- |
|
|
| # 🚀 Key Innovations |
|
|
| - **HAR Processing** – Harmonic/phase separation for ANE efficiency |
| - **Fixed-size Buckets** – Avoid CoreML dynamic shape issues |
| - **Client-side Alignment** – Swift/Python builds alignment matrix |
| - **On-demand Model Loading** – Memory optimized |
| - **MIL Graph Patching** – CoreML compatibility fixes |
|
|
| --- |
|
|
| # ⚡ Performance |
|
|
| ### Runs on ANE (HAR Models) |
|
|
| - Conv1D |
| - ConvTranspose1D |
| - LeakyReLU |
| - Element-wise ops |
|
|
| **Result:** ~17× faster than real-time synthesis |
|
|
| --- |
|
|
| ### Runs on CPU/GPU (Duration Model) |
|
|
| - LSTM layers |
| - Transformer attention |
| - AdaLayerNorm |
| - Dynamic shape processing |
|
|
| --- |
|
|
| # 🧠 Production Optimizations |
|
|
| - Bucket auto-selection |
| - ~200MB per loaded model |
| - Warm-up optimization |
| - Graceful bucket fallback |
| - Memory cleanup during idle |
|
|
| --- |
|
|
| # 📥 Downloading |
|
|
| Because `.mlpackage` is a folder, download using Hugging Face CLI: |
|
|
| ```bash |
| huggingface-cli download <username>/<repo> --local-dir . |
| |
| --- |
| license: mit |
| --- |
| |
