MBZUAI

/

speecht5_tts_clartts_ar

like 32

Follow

Mohamed Bin Zayed University of Artificial Intelligence 854

Summary

Run against the 45-utterance ArNum-TTS set, MBZUAI/speecht5_tts_clartts_ar recovers
0 of 30 numerals written in digits — 0/15 in Western digits (2026), 0/15 in
Arabic-Indic digits (٢٠٢٦). When the same number is spelled out in Arabic words it
recovers 4/15.

The two digit failures look identical in the output and are not the same bug:

This is a limitation report, not a defect claim. ArTST is fine-tuned on ClArTTS,
a Classical Arabic corpus, and its authors nowhere claim numeral coverage. The point
is that the failure is total and silent, so anyone dropping it into a pipeline that
contains a date or a price will ship audio with the number missing and no error raised.

1. Arabic-Indic digits are absent from the vocabulary

The tokenizer is character-level SentencePiece with 87 tokens. All ten Western
digits are in it. None of the ten Arabic-Indic digits ٠١٢٣٤٥٦٧٨٩ are.

from transformers import SpeechT5Processor
tk = SpeechT5Processor.from_pretrained("MBZUAI/speecht5_tts_clartts_ar").tokenizer
v = tk.get_vocab()
[d for d in "٠١٢٣٤٥٦٧٨٩" if d in v]   # -> []
[d for d in "0123456789"  if d in v]   # -> all ten

Because the tokenizer is character-level, a run of unknown characters collapses to a
single <unk>. The whole year is one token before synthesis begins:

'في عام ٢٠٢٦ ارتفعت'
  -> ['▁','ف','ي','▁','ع','ا','م','▁','<unk>','▁','ا','ر','ت','ف','ع','ت','</s>']
  -> decodes back as 'في عام  ارتفعت'

The number is gone before the model runs. No amount of acoustic modelling can recover
it, and nothing in the API signals that anything was dropped. This is reproducible in
three lines with no audio and no GPU.

2. Western digits reach the model and still produce no audio

Western digits are a different story: they tokenise cleanly, 15/15, and reach the
decoder. They are still not spoken.

Measured without Whisper in the loop, by synthesising every Western sentence twice —
as written, and with the numeral deleted — and comparing durations
(artst_duration_control.py, raw numbers in data/artst_duration_control.json):

Apple, which scores 80% on this form, spends 2.44 s saying the number. ArTST spends
0.19 s. For scale, ArTST's own spelled-out renderings of the same 15 numbers take
+1.41 s on average over its Western renderings — so the digit run yields about an
eighth of the audio the same model produces when it actually says the figure.

The transcripts show the same thing directly: the sentence survives, only the number
falls out.

The likely cause is training data: ClArTTS is Classical Arabic prose, where figures are
written as words. The digit embeddings exist in the vocabulary but appear never to have
been trained against real audio, so they decode to approximately nothing.

2b. A bigger listener does not change it

Re-scored with whisper-medium instead of whisper-small, on the same audio
(data/artst_asr_medium.jsonl):

The spelled baseline improves with a better listener. The digit forms do not move at
all — they are floored at zero regardless of ASR capacity, which is what you expect
when there is nothing in the audio to hear. Medium also transcribes the surrounding
Arabic noticeably more cleanly while still returning no number:

شارك أكثر من 1500 موظف في ورش العمل → شارك أكثر من موظف في ورش العمل

A clean sentence with a number-shaped hole in it.

3. The honest baseline

ArTST scores 4/15 (27%) on the spelled form, where every character is in-vocab. So
part of the 0% is general intelligibility on this sentence set, not numerals — the set
is Modern Standard Arabic news-register copy, and ArTST is a Classical Arabic model
being run outside its domain.

The numeral-specific effect is the drop from 27% to 0%, not the whole of the 0%.
Quoting the 0/30 without that qualifier would overstate the result.

4. Where it sits against the other two engines

The fish and Apple spelled figures were revised on 2026-08-20 by a scorer correction
(see the dataset card's changelog). ArTST's own numbers did not move, and no digit-form
cell in this table changed — which is where this finding lives.

Three engines now show three different behaviours on the same sentences: Apple
normalises numerals before speaking and is form-agnostic; fish handles Western digits
and collapses on Arabic-Indic; ArTST speaks neither. The variable is the numeral form
alone — the sentences are otherwise identical.

5. Practical rule

Anyone using ArTST in production must spell every numeral out into Arabic words
upstream of the model. Western-digit normalisation, which is the fix for fish, does
not help here — ArTST fails on both digit forms.

A one-line vocabulary fix (mapping ٠-٩ to 0-9) would convert the Arabic-Indic
failure into the Western-digit failure. It would not fix either, but it would stop the
tokenizer silently deleting input.

6. Reproduce

# synthesis needs torch/transformers; scoring needs faster_whisper
python run_bench.py --engine artst --synth-only     # writes audio/ + data/artst_tokens.json
python run_bench.py --engine artst                  # transcribe + score
python artst_duration_control.py                    # the Whisper-free control

Synthesis is seeded per utterance (SpeechT5 keeps decoder prenet dropout active at
inference, so unseeded runs are not reproducible); re-running produces byte-identical
audio.

7. What this does not establish

• 45 utterances, one speaker embedding (validation row 105, the index the model
  card itself uses). Voice identity is held constant, but a different x-vector has not
  been tried and could in principle change intelligibility.
• No human listening pass. The chain is TTS → Whisper → a numeral parser, and an
  error anywhere is charged to the TTS. The duration control in §2 exists precisely
  because the headline claim should not rest on Whisper alone.
• Domain mismatch is real and unquantified (§3). A Classical Arabic sentence set
  would separate "cannot say numbers" from "is being run off-domain" more cleanly than
  this one does.
• Diacritics: ArTST is trained without them and the set carries none, so this is not a
  tashkeel effect.
• Scorer gaps found while running this, all in the spelled form — now FIXED
  (2026-08-20) and published as a documented correction, not a silent edit. They were
  النصف/الربع being absent from the parser, an unstripped tanween suffix turning
  674 into 604, a two-word ordinal (الثانية عشرة) that could not match, and a
  clock group that never closed, summing 12:45 to 57. Every result was re-derived
  from the stored transcripts — no audio was re-synthesised. Three rows flipped, all
  spelled, all LOST → OK: fish 11/15 → 13/15, Apple 8/15 → 9/15, and this finding's
  own whisper-medium control 5/15 → 6/15. ArTST's headline numbers did not move at
  all, and no digit-form cell moved for any engine, which is the guarantee that
  matters here — 0/30 is unaffected. Old → new per cell is in the dataset card
  changelog; rescore.py reproduces it and refuses to write if a digit form moves.

Model licence: CC BY-NC 4.0. No weights or generated audio are redistributed here —
only the measurements and the code to reproduce them.

Full set, scorer and per-item results, CC BY 4.0: https://huggingface.co/datasets/syamjithnk/arnum-tts

Posting here rather than by email in case it is useful to whoever maintains the model. Happy to re-run any of it against a different speaker embedding or a Classical-Arabic sentence set if that would separate the domain effect more cleanly.