---
license: mit
library_name: coreai
pipeline_tag: text-to-speech
base_model: microsoft/VibeVoice-Realtime-0.5B
language: [en, zh]
tags: [core-ai, coreaikit, tts, multi-speaker, dialogue, podcast, diffusion, on-device, apple]
---

# VibeVoice-Realtime-0.5B — Core AI

[`microsoft/VibeVoice-Realtime-0.5B`](https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B)
(MIT) converted to **Apple Core AI** — the [zoo](https://github.com/john-rocky/coreai-model-zoo)'s
first **multi-speaker / dialogue (podcast-style)** TTS. iPhone (AOT) + Mac, all-fp16.

Not a "first on-device VibeVoice" claim — other CoreML/GGUF ports exist. What this is: the zoo's
first multi-speaker TTS, app-integrated, and the other half of a **generate -> diarize** loop with
the zoo's [Streaming Sortformer](https://huggingface.co/mlboydaisuke/Streaming-Sortformer-Diar-CoreAI)
diarizer.

## Architecture

Dual Qwen2.5 LM (**4-layer** text context LM, norm = Identity + **20-layer** speech trunk) ->
per-frame **next-token diffusion** (4-layer adaLN head, DDPM cosine, v-prediction, DPMSolver++
5-step, CFG 1.5) -> causal-conv acoustic VAE decoder (7.5 Hz latent -> 24 kHz, 3200 samples/frame).
The LM predicts one latent per frame; the diffusion head denoises it; the VAE decoder renders audio.
Multi-speaker output is **host turn-switching**: each `Speaker N:` turn is generated from its own
voice preset and the turns are concatenated — no multi-speaker prefill, no acoustic encoder.

## Contents

| path | what |
|---|---|
| `macos/vibevoice_mainlm_fp16_decode_cl512.aimodel` | context LM, q=1 decode, KV-stateful, cache 512 (114 MB) |
| `macos/vibevoice_ttslm_fp16_decode_cl512.aimodel` | speech trunk, same shape; also drives the CFG-negative stream (569 MB) |
| `macos/vibevoice_diffusion_head_fp16.aimodel` | prediction head, `(noisy[2,64], t[2]) -> [2,64]` (80 MB) |
| `macos/vibevoice_connector_fp16.aimodel` | acoustic connector, `latent[1,1,64] -> embed[1,1,896]` (1.7 MB) |
| `macos/vibevoice_decoder_fp16_t64.aimodel` | acoustic VAE decoder, `latents[1,64,64] -> audio[1,1,204800]` (656 MB) |
| `ios/*.h18p.aimodelc` | the same five, AOT-compiled for A19 (h18p), GPU |
| `voices/*.pt` | 25 upstream voice presets (EN/ZH/…): pre-computed prefill KV, so no acoustic encoder is shipped |
| `coreai_host/voices/<name>/` | the same 25 presets as flat fp16 blobs a Swift host reads directly (no torch) |
| `coreai_host/glue/` | type embeddings, EOS classifier, DPMSolver++ schedule, Qwen2.5 tokenizer |
| `coreai_host/embed/embed_tokens_fp16.bin` | `(151936, 896)` fp16 embedding table, mmapped for the host token lookup |
| `device_bundle/` | compact host inputs + `golden.f32` for the on-device self-test |

**fp16 is required.** int8 LMs diverge inside the speech feedback loop (min cos 0.187, early EOS);
the diffusion head is fp16-sensitive too (pure-torch fp16 collapses to 0.79 — Core AI keeps the
RMSNorm/adaLN reductions in fp32, so the host DDPM reference must run fp32).

**Fixed shapes only.** Every graph is static (q=1 decode, fixed-T decoder), so the runtime must
**not** be given the `expectFrequentReshapes` hint on iOS: it makes the runtime skip the AOT
specialization and compile on device, which segfaults inside the MPSGraph AICode compiler.

## Gates

| gate | result |
|---|---|
| diffusion head / connector, engine vs oracle | cos **0.999999** / **1.000000** |
| acoustic decoder (T=30 / T=64), engine vs non-stream golden | cos **1.000004** / **1.000005** |
| main LM / tts LM decode, engine vs torch | cos **0.999999** / **0.999996** |
| Python E2E on all 5 engines vs the upstream streamed wav | latent min cos **0.999198**, wav cos **0.999479** |
| **iPhone 17 Pro** (A19 Pro, AOT h18p, GPU) vs the golden | cos **0.998308** |

On device: 6 graph loads in **2.6 s** (warm), **24 latents / 3.20 s of audio in 2.3 s = 10.6 tok/s
~ 1.4x real-time**.

## Use it

```swift
import CoreAIKit

let dialogue = try await KitDialogue(catalog: "vibevoice-realtime-0.5b")
let (audio, turns) = try await dialogue.perform("""
    Speaker 1: Did you know this runs entirely on the phone?
    Speaker 2: No cloud at all? That is wild.
    """)
```

Swift host reference: `ondevice/VibeVoiceRunner` (Mac) and `VibeVoiceSelfTest.swift` in the zoo's
[coreai-audio](https://github.com/john-rocky/coreai-model-zoo/tree/main/apps/coreai-audio) app —
raw Core AI stateful-KV loop + a Swift DPMSolver++ sampler. Python host + conversion recipe:
[`conversion/vibevoice`](https://github.com/john-rocky/coreai-model-zoo/tree/main/conversion/vibevoice)
(`host_e2e.py` = the full generate loop, `host_multispeaker.py` = the dialogue demo).

Base model: [microsoft/VibeVoice-Realtime-0.5B](https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B) (MIT).
EN/ZH. *Community port — not an Apple model.*