README.md

---
tags:
  - text-to-speech
  - coreai
  - ios
  - iphone
  - arm64
  - kokoro
  - supertonic
---

# TTS Core AI Lab — converted for iOS / iPhone

This repository contains **Core AI `.aimodel` assets converted for an arm64
iOS application running on physical iPhone hardware**. It is the model-asset
companion to the [TTSCoreAILab iOS app](https://github.com/augustzheng-RED/TTSCoreAILab).

**Deployment target:** iOS 27.0+ on arm64 iPhone hardware.

**Benchmark device:** iPhone 17 Pro. This identifies the device used for the
current RTF benchmark; it is not a guarantee that every included profile has
passed end-to-end validation on every iPhone.

## What was converted

| Runtime | Upstream model | Conversion output | Intended use |
| --- | --- | --- | --- |
| Kokoro | [`hexgrad/Kokoro-82M`](https://huggingface.co/hexgrad/Kokoro-82M) PyTorch model | 11-stage Core AI `.aimodel` pipeline plus fixed-shape profiles | On-device iOS TTS research and profiling |
| Supertonic 3 | [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3) ONNX pipeline | Four Core AI `.aimodel` stages per fixed-shape profile | On-device iOS TTS research and profiling |

Kokoro subgraphs are exported through `torch.export` and converted with
`coreai_torch`. Supertonic ONNX graphs are recovered at fixed shapes, converted
through `onnx2torch` and `torch.export`, then converted with `coreai_torch`.
The resulting assets use the versioned AICode layout required by the iOS 27
device compiler.

These are deployment assets, not general-purpose PyTorch, ONNX, Core ML, web,
Android, macOS, or iOS Simulator models. Use them only with the matching
`TTSCoreAILab` app and its Core AI runtime integration.

## Device and profile notes

- The app targets `arm64` iPhone hardware with deployment target iOS 27.0.
- The iPhone 17 Pro is the current benchmark device.
- Kokoro packages include `64x64`, `128x256`, and `256x512` routed profiles,
  plus legacy/reference profiles.
- Supertonic packages include `64x32`, `128x64`, `128x128`, and `128x256`
  fixed-shape profiles.
- Asset presence does not imply physical-device validation. Validate the exact
  app revision, profile, prompt, and device before shipping or profiling.

## App integration

The iOS project downloads the two directories below into its repository root:

- `TTSCoreAILab/Models`
- `TTSCoreAILab/SupertonicResources`

From the app repository, run:

```bash
./scripts/download-assets.sh
```

---

## Project overview

Local iOS workbench for incrementally porting and profiling the split Kokoro
pipeline in Swift.

The app integrates the official KittenTTS Swift package and Supertonic 3
ONNX pipeline, and provides four modes:

- KittenTTS: ONNX Runtime CPU, eight voices, configurable speed.
- Kokoro: the existing 11-stage split Core AI pipeline.
- Supertonic 3: four Core AI models, ten voices, configurable flow steps,
  44.1 kHz.
- Compare: sequentially generates all three models for the same text, preserves
  PCM results for independent playback, and displays wall time, inference
  time, RTF, Core AI coverage, and speedup.

KittenTTS downloads its Nano model and phonemizer assets on first use. Kokoro
and Supertonic 3 use Core AI model assets hosted separately on Hugging Face:
<https://huggingface.co/augustZheng/TTS-Core-AI>.

## Model Assets

The Xcode project expects these asset directories:

- `TTSCoreAILab/Models`
- `TTSCoreAILab/SupertonicResources`

They are not committed to GitHub because the full asset bundle is about 4.2 GB
and contains files larger than GitHub's regular 100 MB file limit.

Download them from Hugging Face before building:

```bash
./scripts/download-assets.sh
```

The script uses `uv` to run the Hugging Face CLI without creating a project
virtual environment.

The current stage runs a fixed `Hi.` text-to-acoustic-feature chain:

```text
fixed token ids + attention mask
-> ALBERT
-> BERT projection
-> fixed-length duration encoder
-> duration head
-> Swift duration/alignment calculation
-> text encoder CNN + LSTM
-> Swift ASR alignment
-> F0/noise shared LSTM + residual blocks
```

Bundled assets:

- `kokoro_bert_eager_64.aimodel`
- `kokoro_bert_projection.aimodel`
- `kokoro_duration_encoder_no_pack_64_intmask.aimodel`
- `kokoro_duration_head.aimodel`
- `kokoro_text_encoder_conv_64_intmask.aimodel`
- `kokoro_text_encoder_lstm.aimodel`
- `kokoro_f0n_shared_lstm_64.aimodel`
- `kokoro_f0n_blocks_64.aimodel`

## How to Try

1. Open `TTSCoreAILab.xcodeproj` in Xcode.
2. Select the same physical iPhone target that passed the original smoke test.
3. Run the `TTSCoreAILab` app target.
4. Tap `Run Acoustic Stage`.

Expected reference result from the Python CoreAI runtime:

- shapes: `[1,64,768]`, `[1,512,64]`, `[1,64,640]`, `[1,64,50]`
- valid-token durations: `[18, 2, 5, 9, 6, 1]`
- raw predicted frames: `41`
- fixed target frames: `64`
- aligned ASR: `[1, 512, 64]`
- F0: `[1, 128]`
- noise: `[1, 128]`

Python CoreAI reference mean-absolute values:

- shared features: `0.35923`
- F0: `72.4452`
- noise: `6.59444`
- text conv: `0.02399`
- text hidden: `0.16430`
- aligned ASR: `0.18732`

The current build now also includes:

- `kokoro_decoder_pre.aimodel`
- `kokoro_generator_core.aimodel`
- `kokoro_istft.aimodel`
- fixed `Hi.` decoder style and harmonic-source fixtures
- Swift PCM-to-WAV playback through `AVAudioPlayer`

The current UI mirrors the browser Local Lab:

- editable text and `af_heart` voice selection;
- selectable bundled Core AI budgets (`64 / 64`, `128 / 256`, and `256 / 512`), with automatic profile escalation for longer segments;
- optional automatic segmentation;
- total time, Core AI inference time, audio duration, and RTF;
- per-stage Core AI timing bars;
- generated audio playback controls and a detailed run log.

The PyTorch reference and native Kokoro comparison controls are visible but
disabled because those runtimes are not bundled in the iOS app.

The bundled profiles increase the text/frame budget from `64 / 64` to
`128 / 256` and `256 / 512`; automatic mode chooses the smallest profile that
fits a segment. Supertonic ships its validated `64 / 32` baseline assets under
`SupertonicResources/coreai-assets-baseline`.

Tap `Generate Locally`. A successful run should play generated audio and report:

- generator input: `[1, 512, 128]`
- spec/phase: `[1, 22, 7681]`
- audio: `[1, 1, 38400]`
- audio mean absolute value: approximately `0.01756`
- audio min/max: approximately `-0.2076 / 0.3555`

- ## RTF Result

The current benchmark on iPhone 17 Pro.

**RTF (Real-Time Factor)** measures how long it takes to generate one second of audio.

- **RTF < 1.0** → Faster than real time (suitable for interactive TTS)

- **Lower is better**

The benchmark below compares the same prompt on the same device across different runtimes.

<p align="center">

  <img src="docs/images/rtf-result.png" width="900">

</p>