README.md

# CTC-TTS: LLM-Based Dual-Streaming Text-to-Speech with CTC Alignment

[![arXiv](https://img.shields.io/badge/arXiv-2602.19574-b31b1b.svg)](https://arxiv.org/pdf/2602.19574)
[![Interspeech 2026](https://img.shields.io/badge/Conference-Interspeech%202026-blue.svg)]()
[![GitHub Repo](https://img.shields.io/badge/GitHub-thu--spmi/CTC--TTS-black.svg)](https://github.com/thu-spmi/CTC-TTS)

## Overview

This repository contains official pre-trained checkpoints for **CTC-TTS**, proposed in our Interspeech 2026 paper *CTC-TTS: LLM-Based Dual-Streaming Text-to-Speech with CTC Alignment*.

CTC-TTS is a LLM-driven dual-streaming TTS framework, designed to solve two critical pain points of existing streaming synthesis systems:

1. Heavy pipeline dependency on MFA (Montreal Forced Aligner) for text-speech alignment
2. Fixed-ratio token interleaving that fails to model real phoneme-speech alignment rules

We replace MFA with a CTC-based neural aligner (built on [Whistle ASR model](https://huggingface.co/thu-spmi/whistle-small) from [THU-SPMI Lab](http://oa.ee.tsinghua.edu.cn/ouzhijian
)) and propose a **bi-word interleaving block** structure. Two variants are provided for flexible quality-latency tradeoff:

- **CTC-TTS-L**: Concatenate text & speech tokens along sequence dimension, higher synthesis fidelity, slightly larger first-packet latency
- **CTC-TTS-F**: Stack phoneme/speech embeddings along feature dimension, ultra-low streaming latency for real-time scenarios

Experiments on single-speaker streaming and multi-speaker zero-shot synthesis show consistent improvements over LLMVox, ELLA-V and other MFA-aligned baselines.

### Key Contributions

1. Lightweight CTC phoneme-speech alignment pipeline, eliminating MFA offline forced alignment
2. Bi-word context-aware interleaving strategy with compact look-ahead window
3. Dual model variants (L/F) to balance speech naturalness and streaming latency


## Pre-trained Checkpoints


| Model Name | Params | Training Dataset |
|------------|--------|------------------|
| ctc-tts-l-singlespeaker | 34.7M | VoiceAssistant400K |
| ctc-tts-f-singlespeaker | 33.6M | VoiceAssistant400K |
| ctc-tts-l-multispeaker | 159.58M | LibriSpeech 960h |
| ctc-tts-f-multispeaker | 158.47M | LibriSpeech 960h |

## Experimental Results

### 1. Single-Speaker Streaming Benchmark (VoiceAssistant400K, vs LLMVox)

Greedy search. FPL-A = first-packet latency assuming full text is available.

| Method | #Params | WER(↓) | CER(↓) | FPL-A ms(↓) | UTMOS(↑) |
|--------|---------|--------|--------|-------------|----------|
| Ground Truth | NA | NA | NA | NA | 4.27 |
| LLMVox | 31.5M | 2.40 | 1.36 | 167 | 4.15 |
| **CTC-TTS-F** | 33.6M | **1.80** | **1.04** | **159** | 4.15 |
| **CTC-TTS-L** | 34.7M | **1.50** | **0.79** | 210 | 4.15 |

- CTC-TTS-F achieves lower WER/CER and shorter FPL-A than LLMVox, demonstrating the advantage of CTC alignment + bi-word sequences over fixed-ratio interleaving.
- CTC-TTS-L delivers even lower WER/CER at the cost of slightly higher latency.

### 2. Multi-Speaker Zero-Shot — Continuation Task (LibriSpeech test-clean)

Given a text segment and its corresponding 3-second prefixed speech, synthesize speech for the remaining text. Nucleus sampling.

| Group | Method | #Params | WER(↓) | CER(↓) | SPK(↑) | UTMOS(↑) | MOS(↑) | SMOS(↑) |
|-------|--------|---------|--------|--------|--------|----------|--------|---------|
| Ground Truth | — | NA | 1.92 | 0.69 | NA | 4.086 | 4.28±0.060 | 4.60±0.048 |
| **Our Method** | **CTC-TTS-F** | 158.47M | 5.20 | 2.68 | 0.930 | 4.013 | **4.31±0.057** | 4.58±0.050 |
| **Our Method** | **CTC-TTS-L** | 159.58M | **4.82** | **2.47** | 0.929 | **4.050** | **4.33±0.061** | **4.60±0.049** |
| Ablation | CTC+ELLA-V | 159.58M | 12.01 | 7.37 | 0.928 | 4.021 | 4.00±0.062 | 4.39±0.058 |
| Ablation | MFA+ELLA-V | 159.58M | 10.98 | 6.99 | 0.928 | 4.021 | 3.94±0.066 | 4.44±0.056 |
| Ablation | MFA+bi-word | 159.58M | 5.14 | 2.63 | 0.930 | 4.010 | 4.25±0.061 | 4.50±0.051 |

- CTC-TTS-L outperforms MFA+bi-word in all metrics except SPK, validating the superiority of CTC alignment over MFA.
- Our methods outperform CTC+ELLA-V across the board, confirming the advantage of bi-word interleaving over ELLA-V's local-advance sequence.

### 3. Multi-Speaker Zero-Shot — Cross-Speaker Task (Seed-TTS test-en)

Given ~3 seconds of speech and its transcribed text as a prompt, synthesize speech for another utterance (out-of-domain). Nucleus sampling.

| Group | Method | #Params | WER(↓) | CER(↓) | SPK(↑) | UTMOS(↑) | MOS(↑) | SMOS(↑) |
|-------|--------|---------|--------|--------|--------|----------|--------|---------|
| Ground Truth | — | NA | NA | NA | NA | 3.527 | 4.18±0.068 | 4.14±0.072 |
| **Our Method** | **CTC-TTS-F** | 158.47M | 8.02 | 4.20 | **0.880** | 3.903 | 4.16±0.064 | 3.85±0.071 |
| **Our Method** | **CTC-TTS-L** | 159.58M | **6.33** | **3.21** | 0.878 | **3.971** | **4.23±0.060** | **3.98±0.073** |
| Ablation | CTC+ELLA-V | 159.58M | 20.86 | 11.73 | 0.869 | 3.848 | 3.88±0.073 | 3.94±0.073 |
| Ablation | MFA+ELLA-V | 159.58M | 34.89 | 19.58 | 0.872 | 3.873 | 3.75±0.071 | 3.88±0.074 |
| Ablation | MFA+bi-word | 159.58M | 7.53 | 3.99 | 0.874 | 3.840 | 4.14±0.068 | 3.83±0.076 |

- CTC-TTS-L achieves near-optimal performance on this out-of-domain task.
- CTC+ELLA-V outperforms MFA+ELLA-V in WER/CER, indicating CTC alignment generalizes better to out-of-domain cross-speaker scenarios than MFA.
- ELLA-V's sequence organization yields poor performance in both alignment settings, highlighting the benefit of bi-word interleaving.

### Evaluation Metrics

- **Objective**: WER / CER (intelligibility), UTMOS (naturalness), SPK (speaker similarity via WavLM-Base-Plus-SV), FPL-A (first-packet latency)
- **Subjective**: MOS (naturalness) and SMOS (speaker similarity), 30 samples per system, 20 listeners, 1–5 scale with 95% confidence intervals
- ASR backbones: whisper-large-v3 (single-speaker), Conformer-Transducer xlarge (multi-speaker)

## Citation

If you use this model or method in your research, please cite our paper:

```bibtex
@article{liu2026ctc,
  title={CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment},
  author={Liu, Hanwen and Yusuyin, Saierdaer and Huang, Hao and Ou, Zhijian},
  journal={arXiv preprint arXiv:2602.19574},
  year={2026}
}