--- license: apache-2.0 library_name: coreai pipeline_tag: text-to-speech base_model: Edge0/Audio8-TTS-Preview-0.6b language: - yue - zh - nl - en - fr - de - it - ja - ko - pl - es tags: [core-ai, coreaikit, audio8, dualar, text-to-speech, tts, voice-cloning, zero-shot, multilingual, on-device, apple] base_model_relation: quantized --- Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's `coreai-torch` (LLMs: `coreai.llm.export`) into `.aimodel` bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol ([apple-silicon-llm-bench](https://github.com/john-rocky/apple-silicon-llm-bench), macOS 27 beta, 2026-06). # Audio8-TTS-Preview-0.6b — Core AI [🤗 mlboydaisuke/Audio8-TTS-Preview-0.6b-CoreAI](https://huggingface.co/mlboydaisuke/Audio8-TTS-Preview-0.6b-CoreAI/tree/72e1c935961c786bb838e235c7839a2bd77468cc) · Apache-2.0 · base [Edge0/Audio8-TTS-Preview-0.6b](https://huggingface.co/Edge0/Audio8-TTS-Preview-0.6b) [`Edge0/Audio8-TTS-Preview-0.6b`](https://huggingface.co/Edge0/Audio8-TTS-Preview-0.6b) (Edge0, Apache-2.0, 601M + a 337M codec) is a **DualAR text-to-speech** model of the Fish Audio S2 Pro design: a Qwen2.5-shaped **slow AR** (24 layers, 896 wide) predicts one *semantic* token per 46 ms frame, a 4-layer **fast AR** predicts the frame's nine other codec codebooks one after another, and a 44.1 kHz **DAC-style codec** turns the ten codebooks into audio. **Eleven languages** (Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, Spanish) and **zero-shot voice cloning** from a 0.5–30 s reference recording and its transcript. This is the zoo's first DualAR / semantic-token TTS and the first port with the sampler inside the graph. On 2026-09-28 the Hub had MLX builds (`mlx-community/Audio8-TTS-Preview-0.6b-bf16`, 8-bit and 4-bit conversions), the publisher's ONNX INT4 build for CPUs and a GGUF, and no Core AI or Core ML port of the TTS (a Core ML port exists of the publisher's ASR sibling); this port runs the model on iPhone and Mac through Apple's runtime. ## Pipeline ``` text (+ reference transcript + reference codes) ──(host: the publisher's prompt segments, encoded one at a time)──▶ prompt [11, P] 1. dualar.aimodel prefill(codes [1,11,32], pos) in windows ─▶ logits [32,4097] · hidden [32,896] frame(codes [1,11,1], pos, noise_slow [2,4097], window [10], noise_fast [9,4096], forced, use_forced) = the slow step + the semantic draw (top-k 50 / top-p 0.9 / T 0.7, Gumbel-max, RAS) + the fast AR's ten rows with nine codebook draws — ONE call per frame ─▶ semantic · codes [10] first_frame(logits, hidden, …) for the frame right after the prefill host: the 10-token RAS window, the uniform draws (a seeded stream), stop at eos 151645 or 512 frames 2. codec.aimodel codes [1,10,160] ─▶ wav [1, 327680] at 44.1 kHz (every op causal: 160-frame windows, keep the last 32) voice registration: wav 44.1 kHz [1,1,442368] ─▶ encoder.aimodel ─▶ codes [1,10,216] (0.5–10 s reference; the codes + transcript are the voice) ``` A frame is 2,048 samples (21.5 frames/s). The prompt is the publisher's: `<|im_start|>system\n` · "convert the provided text to speech" (or, with a voice, "… reference to the following:\n\nText:\n" · `<|speaker:0|>` + transcript · "\n\nSpeech:\n" · the reference's codebook-0 codes as semantic ids) · `<|im_end|>\n<|im_start|>user\n` · text · `<|im_end|>\n<|im_start|>assistant\n<|voice|>` — segments encoded separately, asserted id for id against the publisher's processor on all 18 fixtures, in Python and in the Swift host. ### Graph contracts ``` dualar prefill in codes[1,11,32] i32 (row 0 ids, rows 1-10 codebooks; pad 151643 / 0) · pos[1] i32 out logits[32,4097] f16 · hidden[32,896] f16 frame in codes[1,11,1] i32 (the previous frame) · pos[1] · noise_slow[2,4097] f32 · window[10] i32 (-1 = none) noise_fast[9,4096] f32 · forced[11] i32 · use_forced[1] f32 out semantic[1] i32 · codes[10] i32 · logits · hidden · fast_logits[9,4096] · sampled_* first_frame in logits[4097] f16 · hidden[896] f16 · the same noise / window / forced out as frame state k_cache / v_cache [24,1,2,2048,64] f16 (prefill and frame) precision slow AR linears int8 (weight-only, per-block-32, symmetric with clipping); embeddings, the 4,097-row head, norms, the fast AR fp16 codec main in codes[1,10,160] i32 (right-pad 0; codebook 0 < 4096, codebooks 1-9 < 1024, clamped in-graph) out wav[1,327680] f16 encoder main in audio[1,1,442368] f32 mono 44.1 kHz (right-pad 0) out codes[1,10,216] i32 (keep ceil(samples/2048) frames) ``` The head is the embedding table's 4,097 rows (the semantic ids and eos) instead of 155,776: the publisher's sampler sets every other logit to -inf before drawing, so nothing is lost (7 MB instead of 279 MB). ## What made it work 1. **One graph call per frame, the sampler inside.** The publisher's ONNX runtime — and this port's first export — runs a frame as a slow-AR call plus nine fast-AR calls with sampling on the host; on the raw Core AI runtime path that is ~48 ms of engine time per 46 ms frame on an M4 Max, a call costing milliseconds before any arithmetic. The shipped `frame` function does the slow step, the semantic draw and the fast AR's ten rows in one call, with the publisher's top-k / top-p / temperature / Gumbel-max / RAS sampler written without a sort (`topk(50)` + `logsumexp` + a cumulative sum); the uniform draws are inputs, so a recorded draw replays the oracle's choice exactly. Details: [`knowledge/audio8-tts-port.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/audio8-tts-port.md). 2. **int8 on the slow AR, fp16 on the fast AR.** On 15,093 teacher-forced codebook draws the fp16 fast AR reproduces the oracle's draw 99.5 % of the time; int8 drops to 95.8 % for 53 MB. The slow AR takes int8 well (1,671 / 1,695 semantic draws, 21 of the 24 misses at a Gumbel margin under 0.13; fp16: 1,688). 3. **The codec encoder needs fp32 arithmetic.** Its vector quantizers pick nearest codebook entries; the whole-fp16 export flips those choices and the nine residual books cascade (44–121 of ~200 frames exact). fp16 weights with fp32 arithmetic (the Fun-ASR encoder recipe) gives 691 / 712 frames exact and codebook 0 exact on every frame. 4. **Kept from the publisher's code:** bfloat16-rounded RoPE tables (interleaved pairs), RMSNorm eps 1e-6 in fp32, the RAS window that skips the first frame's token, the clamp of codebooks 1–9 to 1,023. The slow AR's residual peaks at 12,827 — a fifth of float16's range — so no rescale was needed. ## Numerics gate (18 fixtures: 6 ja + 6 en + 2 zh sentences of our own, 4 voice-clone fixtures on FLEURS test clips) Oracle = the publisher's `modeling_arktts.py` in fp32 on the CPU, seeded (`generate` reproduced call for call, every uniform draw recorded). Teacher-forced = the oracle's tokens fed to the graphs; a *draw* comparison asks whether the in-graph sampler, given the port's logits and the oracle's noise, makes the oracle's choice. | stage (Mac GPU) | result | | --- | --- | | tokenizer-only prompt vs the publisher's processor | 18 / 18 prompts identical (Python and Swift) | | plain-torch re-author, eager fp32, teacher-forced | every argmax and every replayed draw equal to the oracle; codec wav cos 1.000000 | | **dualar int8 (ship)**, teacher-forced, 1,695 slow steps | logits cos min 0.99985, argmax 1,668, **draw 1,671 / 1,695** | | same, 15,093 fast-AR codebook draws | logits cos min 0.99805, argmax 14,705, **draw 14,711 / 15,093** | | codec decoder on the oracle's codes, 18 utterances | wav cos min 0.99978, log-mel cos min 0.99876 | | codec encoder (fp16w32) on the 4 reference clips, 712 frames | 691 frames exact; codebook 0 exact on every frame, every codebook ≥ 97.7 % | | free run (the port's own loop, the oracle's draws) | reached eos 18 / 18; Swift host (Mac) == Python engine on 1,637 / 1,688 frames (10 / 18 utterances identical end to end); the iPhone's fp16 GPU agrees with the Mac's on 385 / 1,672 frames — it diverges earlier, so its audio is gated by the rows below | | ASR round trip vs the fixture text (Fun-ASR fp32): ja CER / en WER / zh CER | Mac **2.3 % / 0.9 % / 0.0 %** · iPhone 18 Pro **3.2 % / 0.0 % / 0.0 %** · the oracle's own audio 6.5 / 0.0 / 0.0 | | speaker cosine to the reference clip (WavLM-Base-Plus-SV), 4 clone fixtures | Mac mean 0.855 / min 0.561 · iPhone 0.842 / 0.572 · oracle 0.854 / 0.582 | A miss is fp16 GPU or int8 arithmetic moving a draw that sat within a hair of the runner-up; under sampling, one flip changes every later frame, so end-to-end identity is not the bar — the ASR and speaker rows are. They are eight to fourteen sentences per language, our own measurement with one normalizer: the port's speech is as intelligible and as speaker-faithful as the publisher's fp32 output on these fixtures, not more (the Japanese "errors" are the ASR's number normalization — 三十 → 30 — on both arms, plus one real word error on the phone). ## Speed Measured with [`apps/Audio8Gate`](https://github.com/john-rocky/coreai-model-zoo/tree/main/apps/Audio8Gate) (the kit's `Audio8TTS` in a Release build, the 18 fixtures, 78 s of audio; medians; 2026-09-28; raw runs in [`device/`](device/), the four runs side by side in [`device/tables.md`](device/tables.md)). Both assets are JIT: the first load specializes them, later loads read the cache. Two paths: **streaming** (a 160-frame codec window every 32 frames, audio starts after ~1.5 s of frames) and **whole utterance** (`synthesize`: the codec once at the end — one window for anything up to 7.4 s, a fifth of the codec work). | | frame (slow step + sampling + fast AR) | codec | time to first audio, streaming | RTF streaming, median / p90 | RTF whole utterance (bench, 5 s sentence) | first load (cold) → later loads | footprint | |---|---|---|---|---|---|---|---| | M4 Max (GPU, macOS 27 26A428, GPU lock held, another conversion running: load average 3–4) | 28 ms | 0.17 s per 160-frame window | 1.2 s | **0.81** / 0.84 | **0.70** | 5.4 s (cache 0 → 1.27 GB) → 0.65 s | 0.85–0.99 GB | | iPhone 18 Pro (GPU, iOS 27 24A437, device JIT, h19p), fresh install, thermal nominal | 33–36 ms | 0.83 s per window | 2.0 s | **1.53** / 1.83 | **0.92** (nominal, after a 145 s cool-down; 0.93 on a thermally *serious* phone) | 6.1 s (cache 0 → 1.27 GB) → 0.5 s | 0.59–0.72 GB | A 46 ms frame costs **the same ~33 ms of engine time on the phone as on the Mac** — the frame is dispatch-bound, not compute-bound (a 512-slot cache, AOT compilation and Apple's composite RMSNorm/RoPE change it by 0–10 %). What separates the two devices is the codec: 0.17 s per 160-frame window on the Mac, 0.83 s on the phone, so streaming in 32-frame chunks — which decodes each window five times over — is real-time on the Mac and 1.5× real time on the phone, while a whole utterance decodes on the phone at about real time. Apple's pipelined engine drives a same-sized Qwen3-0.6B at 2.8 ms per token; moving the slow step onto it is the lever for a several-times faster frame — a separate round. Back-to-back synthesis heats the phone: the third run in eight minutes reached *serious* and its streaming frames slowed from 33 to 39 ms, but the whole-utterance bench reads the same at *serious* (0.93) as at nominal (0.92) — the frame is dispatch-bound either way. ## Use it CoreAIKit `Audio8TTS`; catalog id `audio8-tts-preview-0.6b` ([coreai-kit#62](https://github.com/john-rocky/coreai-kit/pull/62)): ```swift import CoreAIKit let tts = try await Audio8TTS(paths: .standard(root: modelDir)) // dualar + codec .aimodel, tokenizer/ let audio = try await tts.synthesize("明日の午後、駅前の喫茶店で待ち合わせましょう。") // 44.1 kHz mono [Float] // streaming: tts.synthesizeStreaming(text) { chunk in play(chunk) } ~1.5 s chunks // voice cloning: Audio8Voice(referenceText:codes:) from register_voice.py / the encoder graph let cloned = try await tts.synthesize("Turn left at the second traffic light.", voice: voice) ``` ## ⬇️ Bundle **[mlboydaisuke/Audio8-TTS-Preview-0.6b-CoreAI](https://huggingface.co/mlboydaisuke/Audio8-TTS-Preview-0.6b-CoreAI/tree/72e1c935961c786bb838e235c7839a2bd77468cc)** (revision `72e1c935`, 2026-09-28) — `audio8_dualar_int8_cl2048_w32.aimodel/` (876 MB) + `audio8_codec_decoder_fp16_t160.aimodel/` (261 MB) + `audio8_codec_encoder_fp16w32_t216.aimodel/` (416 MB, voice registration only) + `tokenizer/`; one subtree for macOS and iOS (JIT `.aimodel`s). Apache-2.0: LICENSE + NOTICE from the publisher's repository. Mirror: [`coreai-community/Audio8-TTS-Preview-0.6b-CoreAI`](https://huggingface.co/coreai-community/Audio8-TTS-Preview-0.6b-CoreAI/tree/87fdb3c702b07dd9d2b25d851c4af1021e099786). Convert yourself — [`conversion/audio8_tts/`](https://github.com/john-rocky/coreai-model-zoo/tree/main/conversion/audio8_tts): `oracle_audio8.py` → `parity_audio8.py` → `export_audio8_frame.py --mode int8`, `export_audio8.py --part codec`, `audio8_encoder.py --frames 216 --dtype fp16w32`, gated by `gate_audio8_frame.py` (recipe: [`recipe.toml`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/audio8-tts/recipe.toml)). Generated speech can be misused for impersonation; the publisher asks for consent before cloning a voice and disclosure of synthetic audio, and so does this port.