Shami-TTS — Production-Grade Levantine Arabic ↔ English Code-Switching TTS

A low-latency, streaming text-to-speech system that speaks natural, Levantine Arabic and switches to English mid-utterance without a prosodic seam. v2 turns the promising-but-robotic v1 baseline into a production-grade voice on a single consumer GPU (RTX 3060, 12 GB): Levantine CER 0.75 → 0.11 (−86%), WER 0.99 → 0.35, natural prosody, 24 kHz fidelity, and reference-level texture — at ~11× faster than real-time.

This card documents v2 exhaustively — what changed from v1, the architecture, the two vocoder options we release, measured results, usage for both, the training recipe, an honest engineering log, and licensing. A full 9-page technical report is included at paper/shami_tts.pdf.

---

0. TL;DR

Held-out metric (ASR round-trip, Whisper large-v3, ↓ better)	Baseline (v1)	v2 HiFi-GAN	v2 + BigVGAN
Levantine Arabic CER	0.751	0.113	0.106
Levantine Arabic WER	0.993	0.377	0.347
Code-switch CER	0.853	0.498	0.556
Overall CER	0.802	0.305	0.331
Real-time factor (RTX 3060)	0.024	0.028	~0.085
Duration ratio @ length_scale=1	0.31 (needs 3.2× stretch)	0.97	0.97

• Two vocoders released. The HiFi-GAN decoder is a single self-contained VITS model (minimal deps). BigVGAN re-vocodes for the cleanest texture (removes the residual high-frequency breathiness). Pick per your fidelity/latency/dependency budget.
• Single flagship speaker ("Badr") in this release; multi-speaker is v3.
• Infer at length_scale = 1.0 (deterministic duration — do not use 3–5 as in v1).

---

🔊 Listen

Synthesized at length_scale=1.0. Reference = real recording; HiFi-GAN = self-contained decoder; BigVGAN = highest-fidelity vocoder. Players stream on demand.

Pure Levantine

ضلت واقفة ساكتة بدون ما تحكي بَسْ كانت تتفرج علي وانا كنت راقب المي وهي عم تنزل وسرحانة.

Reference	HiFi-GAN	BigVGAN

حسيت فيهن وَجَع و انكسار وجعو متل وجعي التنين مظلومين و بدون حيلة.

Reference	HiFi-GAN	BigVGAN

ما فيك تعمل شُغُل مرتب بلبنان الا اذا كان عندك ظهر ماكن.

Reference	HiFi-GAN	BigVGAN

Code-switching

ليش الtraining مِشْ شغّال on the server؟.

Reference	HiFi-GAN	BigVGAN

في مشكلة ب the error، لازم نصلّحها.

Reference	HiFi-GAN	BigVGAN

حكيتلك عن the project قبل؟ هو light.

Reference	HiFi-GAN	BigVGAN

Showcase — novel unseen text

Long, natural Levantine — بَعرِف إنّو الوَقِت تأخّر كْتير، بَس صَدقيني ما قَصّرِت، ضلّيت عَم حاوِل طول النّهار لَحتّى خلّصِت كِل الشّغِل.

HiFi-GAN	BigVGAN (recommended)

Heavy code-switch (AR + EN) — الـ meeting بُكرا الساعة تسعة، وبَدّي نراجِع الـ dashboard والـ analytics قَبِل ما نعمِل الـ presentation.

HiFi-GAN	BigVGAN (recommended)

Numbers, currency, date — دَفعِت مية وخمسة وعشرين دولار عَ الأوتيل، والرِّحلة بتكلّف تلاتة آلاف وخمسمية ليرة، والموعد يوم اتنين.

HiFi-GAN	BigVGAN (recommended)

Question + exclamation — شو صار مَعَك؟! ليش مِتضايِق هَلقَد؟ يَلّا احكيلي كِل شي بالتّفصيل، لا تِخبّي عَنّي وَلا كِلمة!

HiFi-GAN	BigVGAN (recommended)

Tech / brand names — نزّلِت آخِر version مِن الـ app بَس الـ update كَسَر الـ login، فبعتِتلهُن ticket عَ الـ support.

HiFi-GAN	BigVGAN (recommended)

English-heavy technical — The neural vocoder reduces spectral artifacts, بَس لازِم نـ fine-tune الـ model عَ بيانات أنضَف.

HiFi-GAN	BigVGAN (recommended)

Warm greeting (assistant) — مَرحَبا فيك! أنا مساعِدَك الصوتي، كيف بَقدِر ساعدَك اليوم؟ just tell me what you need.

HiFi-GAN	BigVGAN (recommended)

Emphatics + gutturals — الطّبيب قال إنّو الصّحّة بخير، والضّغِط طبيعي، والقَلِب عَم يِشتِغِل مْنيح، الحَمدُ لله عَ كِل شي.

HiFi-GAN	BigVGAN (recommended)

---

1. What's new in v2 (vs. the published baseline)

v2 is four targeted contributions, each measured (see the report):

1. Dialectal front-end. Automatic diacritization (CAMeL Tools) is now the default path, so ~100% (not 0.4%) of bare conversational text is phonemized through the high-quality Levantine rule-G2P. A new de-desinentialization step strips MSA case/mood endings that Levantine drops (مَوْعِد → moːʕid, not moːʕidi) while preserving gemination. → Levantine CER 0.75 → 0.46.
2. Deterministic duration head. Replaces VITS's stochastic duration predictor (which under-predicted ~3×, forcing a prosody-flattening global stretch) with a deterministic head trained by MSE on alignment durations. Natural rhythm at length_scale=1. → CER 0.46 → 0.11 (the robotic→natural unlock).
3. 24 kHz fidelity + texture recipe. Decoder fine-tuned to native 24 kHz; feature-matching up-weighted; the discriminator is now persisted across warm-starts (it used to restart from scratch — a real bug that caused buzz); a multi-resolution STFT loss added. → full-band fidelity, buzz at reference level, CER flat.
4. Anti-aliased BigVGAN vocoder. Removes the residual high-frequency breathiness (a phase/aliasing artifact) that magnitude losses can't fix. → texture matches the source.

The story is two-regime: intelligibility is solved by data + duration (steep CER drop), then fidelity/texture improvements (24 kHz, feature-matching, BigVGAN) hold CER while raising perceptual quality — which CER does not measure. Trust the audio in samples/.

---

2. Architecture

text (AR / EN / mixed)
   → normalize → code-switch segment (script-based) → per-span verbalize
   → diacritize (Arabic) + G2P (Levantine rules / espeak-ng English)
   → shared 88-symbol IPA + parallel language-ID stream
   → ShamiVITS acoustic model  (mms-tts-ara backbone + IPA/lang embeddings
                                 + DETERMINISTIC duration head), 24 kHz
   → HiFi-GAN decoder  ──────────────►  waveform            (self-contained)
        └► mel ► BigVGAN vocoder ─────►  waveform (cleanest) (highest fidelity)

• Acoustic model (ShamiVITS): ~36 M params. VITS core (CVAE + flow + monotonic alignment), with the text-encoder embedding replaced by a shared IPA table, an additive language-ID embedding, and the stochastic duration predictor swapped for the deterministic head. (In the reference code this is the HamsVITS class in the hams_tts package — the import path is unchanged for backward compatibility.)
• Vocoders: native HiFi-GAN decoder (~inside VITS), or pretrained nvidia/bigvgan_v2_24khz_100band_256x (112 M, MIT) — an exact match for our 24 kHz / hop-256 pipeline, used with no fine-tuning.

---

3. Usage

Full code, front-end, and inference: https://github.com/Al-aminI/hams-levantine-tts

3a. HiFi-GAN (single model, minimal deps)

import torch, soundfile as sf
from hams_tts.models.hams_vits import HamsVITS
from hams_tts.text.frontend import TextFrontend   # needs espeak-ng + camel-tools for AR

fe = TextFrontend(diacritizer_backend="auto")
model = HamsVITS.from_checkpoint("path/to/this/repo").cuda().eval()   # loads hams_vits.pt + config

u = fe.process("مَرحَبا، أنا مساعِدَك الصوتي، how can I help you today?")
pid = torch.tensor([u.phoneme_ids]).cuda(); lid = torch.tensor([u.language_ids]).cuda()
wav = model.infer(pid, lid, length_scale=1.0).squeeze().cpu().numpy()  # 24 kHz
sf.write("out.wav", wav, model.sample_rate)

3b. + BigVGAN (highest fidelity)

from hams_tts.inference.bigvgan_vocoder import synthesize   # VITS → BigVGAN → trim
wav = synthesize(model, u.phoneme_ids, u.language_ids, length_scale=1.0)
sf.write("out_bigvgan.wav", wav, model.sample_rate)

BigVGAN is fetched from the Hub on first use (use_cuda_kernel=False → pure-PyTorch, Windows/CPU-safe). Always synthesize at length_scale=1.0.

---

4. Results (held-out set, n=40)

Consistently measured; ASR round-trip via Whisper large-v3, Arabic forced, text normalized (diacritics/alef/ya/ta-marbuta folded). Full analysis in the report.

System	Lev CER	Lev WER	CS CER	Overall CER	RTF
Published baseline (16 kHz, stochastic dur.)	0.751	0.993	0.853	0.802	0.024
v2, HiFi-GAN (24 kHz)	0.113	0.377	0.498	0.305	0.028
v2, + BigVGAN	0.106	0.347	0.556	0.331	~0.085

Texture: high-frequency spectral flatness matches the reference (0.49–0.51 vs 0.488). Duration: ratio 0.97 at natural pacing. Robustness: 8/8 novel unseen sentences (long, heavy code-switch, numbers/dates) synthesized cleanly.

On the code-switch CER column: BigVGAN slightly raises it while improving pure Levantine and perceived quality. This is a Whisper artifact — Whisper is MSA/English biased and mis-scores the English-in-Arabic segments; it is not an audible regression. Absolute CER via ASR has a floor well above zero on dialectal Arabic; treat relative numbers and the released audio as the source of truth. Human MOS is future work.

---

5. Training recipe (single RTX 3060, 12 GB, bf16)

Warm-started stages, each a fine-tune of the previous best:

prepare_lahgtna.py extract --target-sr 24000 → phonemize (dialectal front-end)
finetune_gan.py --deterministic-duration --sample-rate 24000 --c-fm 4 --c-stft 3 \
                --ckpt <prev-best>   (generator + discriminator persisted, atomic saves)
eval_checkpoint.py --auto-ls --asr   (sweep checkpoints; GAN is non-monotonic)
bigvgan_vocoder.synthesize            (final vocoding)

Key HPs: AdamW(0.8,0.99), lr 2e-4, batch 12–16, seg_size 8192–12288, c_mel=45, c_dur=1–2, c_fm=4, c_stft=3, MR-STFT ffts 512/1024/2048, 8k–10k steps/stage.

6. Data

mohammedaly22/lahgtna-levantine-tts (CC-BY-4.0): 50k clips / 66.8 h / 24 kHz / 10 speakers, Shami Levantine + 12% synthetic code-switch, partially diacritized. This release = single speaker "Badr" (6.2 h).

7. Limitations

Single speaker; small eval set (±0.02 CER noise); ASR-CER floor on dialectal Arabic under-states true quality; BigVGAN path runs two vocoders (a single-pass mel→BigVGAN model is v3); multi-speaker is v3.

8. Licensing

• Code: Apache-2.0 (this project's source).
• These weights: CC-BY-NC-4.0 — derived from facebook/mms-tts-ara (Meta MMS), which is non-commercial. Non-commercial use only unless you retrain from a permissive base.
• Dataset: CC-BY-4.0. BigVGAN: MIT. espeak-ng: GPLv3 (tool). CAMeL Tools: MIT/…

9. Citation

@techreport{shami_tts_2026,
  title  = {Shami-TTS: A Production-Grade Streaming TTS for Levantine Arabic/English
            Code-Switching via Dialectal Front-End, Deterministic Duration, and
            Anti-Aliased Neural Vocoding},
  author = {{Tushe Language Research Team}},
  year   = {2026},
  institution = {Tushe Language Research},
  note   = {https://huggingface.co/Tushe/shami-tts}
}

Built on VITS (Kim et al., 2021), HiFi-GAN (Kong et al., 2020), BigVGAN (Lee et al., 2023), MMS (Pratap et al., 2023), CAMeL Tools (Obeid et al., 2020), Whisper (Radford et al., 2023).