mlboydaisuke/VibeVoice-Realtime-0.5B-LiteRT

VibeVoice-Realtime-0.5B — LiteRT (on-device, hybrid GPU/CPU)

VibeVoice-Realtime-0.5B (Microsoft, MIT) — a streaming, autoregressive next-token-diffusion text-to-speech model — re-authored to run on-device with LiteRT CompiledModel. Four graphs plus host-side orchestration synthesize 24 kHz speech with no FFT anywhere (VibeVoice's acoustic tokenizer is a convolutional σ-VAE). Verified on a Pixel 8a (Tensor G3).

The 24-layer Qwen2.5-0.5B backbone is split into a 4-layer text LM and a 20-layer TTS LM; each token, the TTS LM's hidden state conditions a 4-layer DDPM head that a 5-step DPM-Solver++ loop denoises into a 64-d acoustic latent, which a convolutional σ-VAE decoder turns into audio. The two LMs keep their KV cache host-side as a packed [1, L·nkv, Pmax, 64] tensor fed in/out each step (the ML-Drift-safe "state as graph I/O" pattern); the voice is a precomputed prompt KV cache.

Device placement (hybrid — verified on-device, not just by desktop parity)

graph	file	runs on
4-layer text LM (KV step)	vv_base_lm_kv_fp32.tflite	CPU (fp32)
20-layer TTS LM (KV step)	vv_tts_lm_kv_fp32.tflite	CPU (fp32)
DDPM diffusion head	vv_diffhead_fp16.tflite	GPU (fp32 precision)
σ-VAE decoder	vv_decoder_fp32.tflite	CPU (fp32)

The LMs run on CPU because this build pins LiteRT 2.1.3, whose Mali ML Drift delegate rejects their KV-step FULLY_CONNECTED weights shape — a delegate bug fixed in 2.1.5 (see the correction below) — and because fp16 collapses the 20-layer stack on ARM XNNPACK, so they ship as fp32 graphs. The σ-VAE decoder compiles on GPU but ML Drift miscomputes it (a graph-assembly buffer bug: every op is bit-exact in isolation, but the assembled ConvNeXt block is wrong; identical on OpenCL/OpenGL and at fp32), so it too runs as an fp32 graph on CPU. Only the tiny diffusion head runs on GPU. Each graph is bit-exact with the PyTorch reference on its chosen accelerator.

Correction (2026-07-10): the LMs are not blocked by the GPU. An earlier version of this card said ML Drift rejects the KV-step FULLY_CONNECTED weights shape, so autoregressive attention decoders cannot run on it. That is withdrawn. The rejection is real on LiteRT 2.1.3 (INVALID_ARGUMENT: Unsupported weights shape, fully_connected.cc:1070) and fixed in 2.1.5, where the same graphs delegate fully and compute correctly: vv_base_lm_kv_fp32 313/313 nodes at 27 ms/step (hidden corr 0.999964 vs CPU), the 20-layer vv_tts_lm_kv_fp32 1559/1559 nodes at 65 ms/step (corr 0.999852). Both timings are run() plus the output readback — CompiledModel.run() only enqueues the GPU work, so timing it alone reports a misleading 3 ms and 23 ms, which an earlier version of this card did. These graphs still ship for CPU as fp32; a GPU build is a follow-up.

The σ-VAE decoder miscompute is unaffected and still open — re-verified against the full decoder graph on both runtimes, all 1578 nodes delegate on each and the GPU output is bit-identically wrong on both (std 0.030584, absmax 0.134888, corr 0.0254 against a CPU fp32 reference whose std is ~0).

Non-graph assets: embed_tokens.f16 (mmapped fp16 token embeddings), glue.f32 (connector + type-embed + EOS weights), voice_en-Emma_woman.bin (a precomputed prompt KV cache).

Minimal usage (Python — one autoregressive + diffusion step)

import numpy as np
from ai_edge_litert.interpreter import Interpreter

# The full generate() loop (BPE, host KV cache, DPM-Solver++, EOS, decode) is in the
# conversion/ reference; this shows loading a graph and running one TTS-LM step.
lm = Interpreter(model_path="vv_tts_lm_kv_fp32.tflite")
lm.allocate_tensors()
ins = lm.get_input_details()          # x[1,1,896], cos/sin[1,1,1,64], mask[1,1,1,385], pk/pv[1,40,384,64]
for d in ins:
    lm.set_tensor(d["index"], np.zeros(d["shape"], np.float32))
lm.invoke()
hidden = lm.get_tensor(lm.get_output_details()[0]["index"])   # [1,1,896]
print(hidden.shape)

Minimal usage (Kotlin — LiteRT CompiledModel)

import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel

// LMs + decoder on CPU (fp32), the head on GPU with fp32 precision.
val opts = CompiledModel.Options(Accelerator.CPU)
val lm = CompiledModel.create("vv_tts_lm_kv_fp32.tflite", opts, null)
val inputs = lm.createInputBuffers()
val outputs = lm.createOutputBuffers()
inputs[0].writeFloat(xEmbedding)      // [1,1,896]
// ... write cos, sin, mask, packed pk/pv ...
lm.run(inputs, outputs)
val hidden = outputs[0].readFloat()   // [1,1,896]

Notes

• Stochastic, autoregressive: each frame draws fresh diffusion noise, so two runs differ.
• Single speaker, English; the bundled voice is a precomputed prompt KV cache (voices are exported offline — the realtime checkpoint is decoder-only and cannot clone voices on-device).
• This is a correctness-first placement, not a real-time one: the two fp32 LMs and the fp32 decoder make a short sentence take ~30 s on a Pixel 8a. See the conversion notes for the ML Drift decoder bug.

Original model: microsoft/VibeVoice-Realtime-0.5B (MIT).