TTS Pronunciation Benchmark: How Well Do Commercial Streaming TTS Models Handle Real-World Text? 🔊

The first systematic, open evaluation of how accurately commercial streaming TTS models pronounce non-standard text — dates, numbers, currencies, measurements, and 27 other categories.

👉 Explore the interactive benchmark — listen to every audio sample, inspect per-unit judgments, and download all data.

---

TL;DR — TTS Pronunciation Benchmark

• 🎯 What we tested: 4 commercial TTS models on 1,000+ sentences containing 2,200+ non-standard text items across 31 categories — all via real-time WebSocket streaming, with zero text preprocessing.
• 🏆 Top result: Async Flash v1.0 achieves 88.6% unit-level accuracy and 81.2% sentence-level accuracy, leading by a wide margin — all while staying under 200ms p95 latency.
• 📉 The gap is real: The next-best model reaches 67.8% at the unit level — a 20+ point gap that widens dramatically on categories like measurement units, dates, and currencies.
• ⚡ Why it matters: In production conversational AI, TTS models receive raw text over a streaming connection. No LLM cleanup. The model must normalize natively — and most don't do it well.
• 📂 Fully open: All data, transcriptions, per-unit judgments, and audio samples are publicly available.

---

Why Text Normalization Matters 🤔

Every text-to-speech system must solve a deceptively hard problem: converting written symbols into natural spoken language. The string $42.50 should become "forty-two dollars and fifty cents", not "dollar sign four two period five zero." The date 03/15/2024 should be spoken as "March fifteenth, twenty twenty-four", not "zero three fifteen twenty twenty-four."

This process — called text normalization — is the first and most important step in the TTS pipeline. Get it wrong, and the output sounds broken no matter how natural the voice is.

In batch or REST API mode, providers can add an LLM step that cleans up text before the TTS model ever sees it. This works, but adds latency and cost. In real-time streaming — the interface used by conversational AI agents, voice assistants, and live applications — there is no time for that extra step. The TTS model gets raw text over a WebSocket and must figure it out on its own, in real time.

Despite its importance, no systematic benchmark existed for this capability — until now 🚀

---

Evaluation Scope 📐

Models Evaluated

---

Methodology 🔬

🎙️ Audio Generation

All audio is generated through each provider's WebSocket streaming endpoint — the same low-latency connection used in production voice agents and conversational apps. The original text is sent directly to the model with no cleanup or LLM rewriting. This is intentional: the benchmark measures what happens under real production conditions, where text arrives raw and must be spoken immediately.

🧩 Normalization Unit Extraction

Each sentence is analyzed to find the specific pieces of non-standard text that need to be spoken differently — things like $42.50, 03/15/2024, or 5'11". We call these normalization units. They are extracted by comparing the original text against a human-written spoken reference, then checked for accuracy. Each unit is tagged with one of 31 categories.

⚖️ Automated Judging

Gemini 3.1 Pro serves as the automated judge. For each audio sample, it:

1. Listens to the synthesized audio.
2. Transcribes what was actually spoken — using strict transcription rules that produce word-level spoken forms (no digits, no punctuation symbols).
3. Evaluates each normalization unit independently against category-specific accept/reject criteria defined in a detailed rulebook.

The judge scores each unit on its own, so a mistake in one part of a sentence doesn't affect the score of other units.

📏 Evaluation Criteria

The main rule is simple: does it sound natural? If a native English speaker would say it that way in conversation, it passes. It only fails when information is lost, digits are wrong, or the phrasing is confusing. For codes and tokens (like passwords or API keys), there's one extra rule: a listener must be able to reconstruct the original string from what was spoken.

📊 Metrics

Two levels of accuracy are reported:

• Sentence-level accuracy: A sentence passes only when every normalization unit within it is pronounced correctly. This is the strictest measure.
• Unit-level accuracy: Each non-standard word is scored on its own — a more detailed view of where models succeed and fail.

---

Overall Results 📈

Sentence-Level Accuracy

A sentence is correct only when all normalization units are pronounced correctly — a strict, end-to-end measure.

Unit-Level Accuracy

Each non-standard word (date, number, abbreviation, etc.) is scored on its own for a more detailed picture.

Async Flash v1.0 leads at both levels — 20.8 points ahead of the next-best model on units and 31.4 points ahead on sentences. The sentence-level gap is wider because mistakes add up: a model that occasionally gets one word wrong will often fail the whole sentence.

Importantly, this accuracy doesn't come at the cost of speed. Async Flash v1.0 delivers these results while staying under 200ms end-to-end latency at p95 — fast enough for real-time conversation. 🚀

---

Expert Linguist Validation 🧑‍🔬

Can an LLM really judge pronunciation accurately? To find out, we compared Gemini's automated scores against expert linguist review on a sample of ~300 sentence–model pairs covering every category and all 4 providers.

97.4% Human–LLM Agreement

Expert linguists disagreed with Gemini on just ~2.5% of ~300 reviewed pairs — only 3 unique sentences out of the entire sample. In all 3 cases, both answers were reasonable — the human simply accepted a pronunciation that the LLM marked as wrong.

Where Expert Linguist and LLM Disagree

The pattern is clear: when the LLM and human disagree, the LLM is always stricter. This means the accuracy numbers in this benchmark are slightly conservative — real-world accuracy may be even higher across all models. ✅

---

Category Breakdown: Where Models Diverge 🔍

The 31 categories reveal very different strengths and weaknesses across models. Here are the ones that tell the most interesting story 👇

🟢 Categories Where Async Flash Dominates

💡 Measurement units show the starkest contrast: Async Flash correctly pronounces 98.2% of items like "5 mg/dL" or "3.2 km/h", while ElevenLabs Multilingual v2 manages only 10.4% — a nearly 10× difference.

✅ Categories Where Everyone Succeeds

Simple ordinals ("1st" → "first") and cardinal numbers ("42" → "forty-two") are effectively solved across the board. These are well-established patterns that all modern TTS systems handle reliably. 🎉

🔴 Categories That Remain Unsolved

Random character strings (passwords, API tokens) and scientific notation are hard for every model. These require reading exact characters one by one with no room for guessing — a very different challenge from reading dates or currencies. This is one of the most exciting areas for future improvement 🧪

---

What Goes Wrong: Failure Patterns 🐛

Examining failure cases reveals three recurring patterns across lower-performing models.

1. Reading Raw Digits Instead of Understanding the Meaning

The most common failure — models read non-standard text as raw characters instead of recognizing what it actually is.

Input: "The event is on 05/20/2023." Expected: "The event is on May twentieth, twenty twenty-three." ElevenLabs Flash v2.5: "The event is on zero five twenty twenty three."

The model fails to recognize 05/20/2023 as a date and instead reads the digits sequentially. This pattern recurs across dates, currencies, and measurement units.

2. Partial Normalization

Some models correctly identify the type but drop components.

Input: "The event is on 05/20/2023." Expected: "The event is on May twentieth, twenty twenty-three." Inworld TTS-1: "The event is on May twentieth, twenty three."

The month and day are correct, but the year 2023 is shortened to "twenty three", losing "twenty" — a subtle error that changes the meaning.

3. Unnatural Hybrid Forms

Models sometimes produce technically correct but unnatural combinations.

Input: "I'll see you at 12:00AM..." Expected: "twelve A.M." ElevenLabs Flash v2.5: "twelve hundred A.M."

"Twelve hundred A.M." is a military/civilian hybrid that no native speaker would produce naturally 🤷

👉 Hear these failures yourself in the Audio Explorer — every sample is playable directly from the benchmark dashboard.

---

Why Streaming Changes Everything ⚡

These results may surprise readers who have heard impressive demos from TTS providers. The explanation is simple: it depends on how the text reaches the model.

Most TTS demos and batch API calls use a cleanup step: an LLM rewrites the input text before the speech model sees it, converting $42.50 to "forty-two dollars and fifty cents" ahead of time. The TTS model then just reads already-clean text.

In WebSocket streaming — the interface used by conversational AI in production — there is no cleanup step. Text arrives and must be spoken immediately. The TTS model itself has to figure out, in real time, that 03/15 is a date (not a fraction), that 5'11" is a height (not quotation marks), and that $42.50 is a price.

This benchmark tests exactly that streaming scenario. The wide performance gaps show which providers have built normalization intelligence into the model itself, versus relying on an external cleanup step that simply isn't available during real-time streaming.

---

Open Data 📂

Every data point behind this benchmark is publicly available. We believe in open, reproducible evaluation.

• 📊 Per-sentence results — full transcriptions and unit-level judgments for every model
• 📈 Aggregated metrics — by model and by category, downloadable as CSV
• 🎧 Audio samples — every evaluated sentence across all models, playable in-browser
• 📋 Category-specific evaluation rulebook — defining accept/reject criteria for all 31 categories

We encourage the research community and TTS providers to examine the data, reproduce the evaluation, and submit additional models for inclusion in future updates 🙌

---

Conclusion

Text normalization in streaming TTS is a hard, overlooked problem. Simple cases like numbers and ordinals are solved, but the long tail — dates in different formats, measurement units, URLs, currencies, math expressions — is where models really differ.

Async Flash v1.0 shows that getting this right in a streaming context is possible, reaching 88.6% unit-level accuracy across 31 categories. The 20+ point gap over competing models shows that building normalization directly into the model makes a real, measurable difference.

As conversational AI moves to production, the ability to correctly pronounce real-world text — not just clean, simple sentences — will increasingly separate the best TTS systems from the rest 🎯

---

👉 Explore the full benchmark — interactive charts, category breakdowns, and downloadable data.

👉 Try Async Flash v1.0 — the model that leads this benchmark. Free tier, no credit card required.

This benchmark is maintained by the Async Voice AI team. For questions, contributions, or to submit a model for evaluation, reach out through our Hugging Face Space.