---
license: cc-by-nc-4.0
language:
- ar
- en
tags:
- text-to-speech
- tts
- vits
- bigvgan
- levantine-arabic
- arabic
- code-switching
- streaming
- low-latency
- real-time
- dialectal-arabic
base_model: facebook/mms-tts-ara
pipeline_tag: text-to-speech
library_name: pytorch
---
# Shami-TTS — Production-Grade Levantine Arabic ↔ English Code-Switching TTS
> A low-latency, **streaming** text-to-speech system that speaks natural, **Levantine
> Arabic** and switches to **English** mid-utterance without a prosodic seam. v2 turns the
> promising-but-robotic v1 baseline into a **production-grade** voice on a **single consumer
> GPU (RTX 3060, 12 GB)**: **Levantine CER 0.75 → 0.11 (−86%), WER 0.99 → 0.35**, natural
> prosody, 24 kHz fidelity, and reference-level texture — at **~11× faster than real-time**.
This card documents v2 **exhaustively** — what changed from v1, the architecture, the two
vocoder options we release, measured results, usage for both, the training recipe, an honest
engineering log, and licensing. A full **9-page technical report** is included at
[`paper/shami_tts.pdf`](paper/shami_tts.pdf).
---
## 0. TL;DR
| Held-out metric (ASR round-trip, Whisper large-v3, ↓ better) | Baseline (v1) | **v2 HiFi-GAN** | **v2 + BigVGAN** |
|---|---|---|---|
| **Levantine Arabic CER** | 0.751 | 0.113 | **0.106** |
| **Levantine Arabic WER** | 0.993 | 0.377 | **0.347** |
| Code-switch CER | 0.853 | **0.498** | 0.556 |
| Overall CER | 0.802 | **0.305** | 0.331 |
| Real-time factor (RTX 3060) | 0.024 | 0.028 | ~0.085 |
| Duration ratio @ `length_scale=1` | 0.31 (needs 3.2× stretch) | **0.97** | **0.97** |
- **Two vocoders released.** The **HiFi-GAN** decoder is a single self-contained VITS model
(minimal deps). **BigVGAN** re-vocodes for the cleanest texture (removes the residual
high-frequency breathiness). Pick per your fidelity/latency/dependency budget.
- **Single flagship speaker** ("Badr") in this release; multi-speaker is v3.
- **Infer at `length_scale = 1.0`** (deterministic duration — do *not* use 3–5 as in v1).
---
## 🔊 Listen
Synthesized at `length_scale=1.0`. **Reference** = real recording; **HiFi-GAN** = self-contained decoder; **BigVGAN** = highest-fidelity vocoder. Players stream on demand.
### Pure Levantine
ضلت واقفة ساكتة بدون ما تحكي بَسْ كانت تتفرج علي وانا كنت راقب المي وهي عم تنزل وسرحانة.
| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| | | |
حسيت فيهن وَجَع و انكسار وجعو متل وجعي التنين مظلومين و بدون حيلة.
| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| | | |
ما فيك تعمل شُغُل مرتب بلبنان الا اذا كان عندك ظهر ماكن.
| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| | | |
### Code-switching
ليش الtraining مِشْ شغّال on the server؟.
| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| | | |
في مشكلة ب the error، لازم نصلّحها.
| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| | | |
حكيتلك عن the project قبل؟ هو light.
| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| | | |
### Showcase — novel unseen text
**Long, natural Levantine** — بَعرِف إنّو الوَقِت تأخّر كْتير، بَس صَدقيني ما قَصّرِت، ضلّيت عَم حاوِل طول النّهار لَحتّى خلّصِت كِل الشّغِل.
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
**Heavy code-switch (AR + EN)** — الـ meeting بُكرا الساعة تسعة، وبَدّي نراجِع الـ dashboard والـ analytics قَبِل ما نعمِل الـ presentation.
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
**Numbers, currency, date** — دَفعِت مية وخمسة وعشرين دولار عَ الأوتيل، والرِّحلة بتكلّف تلاتة آلاف وخمسمية ليرة، والموعد يوم اتنين.
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
**Question + exclamation** — شو صار مَعَك؟! ليش مِتضايِق هَلقَد؟ يَلّا احكيلي كِل شي بالتّفصيل، لا تِخبّي عَنّي وَلا كِلمة!
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
**Tech / brand names** — نزّلِت آخِر version مِن الـ app بَس الـ update كَسَر الـ login، فبعتِتلهُن ticket عَ الـ support.
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
**English-heavy technical** — The neural vocoder reduces spectral artifacts, بَس لازِم نـ fine-tune الـ model عَ بيانات أنضَف.
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
**Warm greeting (assistant)** — مَرحَبا فيك! أنا مساعِدَك الصوتي، كيف بَقدِر ساعدَك اليوم؟ just tell me what you need.
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
**Emphatics + gutturals** — الطّبيب قال إنّو الصّحّة بخير، والضّغِط طبيعي، والقَلِب عَم يِشتِغِل مْنيح، الحَمدُ لله عَ كِل شي.
| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| | |
---
## 1. What's new in v2 (vs. the published baseline)
v2 is four targeted contributions, each measured (see the [report](paper/shami_tts.pdf)):
1. **Dialectal front-end.** Automatic diacritization (CAMeL Tools) is now the *default* path,
so ~100% (not 0.4%) of bare conversational text is phonemized through the high-quality
Levantine rule-G2P. A new **de-desinentialization** step strips MSA case/mood endings that
Levantine drops (`مَوْعِد` → `moːʕid`, not `moːʕidi`) while preserving gemination.
→ Levantine CER 0.75 → 0.46.
2. **Deterministic duration head.** Replaces VITS's stochastic duration predictor (which
under-predicted ~3×, forcing a prosody-flattening global stretch) with a deterministic
head trained by MSE on alignment durations. Natural rhythm at `length_scale=1`.
→ CER 0.46 → **0.11** (the robotic→natural unlock).
3. **24 kHz fidelity + texture recipe.** Decoder fine-tuned to native 24 kHz; feature-matching
up-weighted; the discriminator is now **persisted across warm-starts** (it used to restart
from scratch — a real bug that caused buzz); a multi-resolution STFT loss added.
→ full-band fidelity, buzz at reference level, CER flat.
4. **Anti-aliased BigVGAN vocoder.** Removes the residual high-frequency breathiness
(a phase/aliasing artifact) that magnitude losses can't fix. → texture matches the source.
The story is two-regime: **intelligibility** is solved by data + duration (steep CER drop),
then **fidelity/texture** improvements (24 kHz, feature-matching, BigVGAN) hold CER while
raising perceptual quality — which CER does not measure. Trust the audio in `samples/`.
---
## 2. Architecture
```
text (AR / EN / mixed)
→ normalize → code-switch segment (script-based) → per-span verbalize
→ diacritize (Arabic) + G2P (Levantine rules / espeak-ng English)
→ shared 88-symbol IPA + parallel language-ID stream
→ ShamiVITS acoustic model (mms-tts-ara backbone + IPA/lang embeddings
+ DETERMINISTIC duration head), 24 kHz
→ HiFi-GAN decoder ──────────────► waveform (self-contained)
└► mel ► BigVGAN vocoder ─────► waveform (cleanest) (highest fidelity)
```
- **Acoustic model (ShamiVITS):** ~36 M params. VITS core (CVAE + flow + monotonic alignment),
with the text-encoder embedding replaced by a shared IPA table, an additive language-ID
embedding, and the stochastic duration predictor swapped for the deterministic head.
*(In the reference code this is the `HamsVITS` class in the `hams_tts` package — the import
path is unchanged for backward compatibility.)*
- **Vocoders:** native HiFi-GAN decoder (~inside VITS), or pretrained
[`nvidia/bigvgan_v2_24khz_100band_256x`](https://huggingface.co/nvidia/bigvgan_v2_24khz_100band_256x)
(112 M, MIT) — an exact match for our 24 kHz / hop-256 pipeline, used with **no fine-tuning**.
---
## 3. Usage
Full code, front-end, and inference: **https://github.com/Al-aminI/hams-levantine-tts**
### 3a. HiFi-GAN (single model, minimal deps)
```python
import torch, soundfile as sf
from hams_tts.models.hams_vits import HamsVITS
from hams_tts.text.frontend import TextFrontend # needs espeak-ng + camel-tools for AR
fe = TextFrontend(diacritizer_backend="auto")
model = HamsVITS.from_checkpoint("path/to/this/repo").cuda().eval() # loads hams_vits.pt + config
u = fe.process("مَرحَبا، أنا مساعِدَك الصوتي، how can I help you today?")
pid = torch.tensor([u.phoneme_ids]).cuda(); lid = torch.tensor([u.language_ids]).cuda()
wav = model.infer(pid, lid, length_scale=1.0).squeeze().cpu().numpy() # 24 kHz
sf.write("out.wav", wav, model.sample_rate)
```
### 3b. + BigVGAN (highest fidelity)
```python
from hams_tts.inference.bigvgan_vocoder import synthesize # VITS → BigVGAN → trim
wav = synthesize(model, u.phoneme_ids, u.language_ids, length_scale=1.0)
sf.write("out_bigvgan.wav", wav, model.sample_rate)
```
BigVGAN is fetched from the Hub on first use (`use_cuda_kernel=False` → pure-PyTorch,
Windows/CPU-safe). **Always synthesize at `length_scale=1.0`.**
---
## 4. Results (held-out set, n=40)
Consistently measured; ASR round-trip via Whisper large-v3, Arabic forced, text normalized
(diacritics/alef/ya/ta-marbuta folded). Full analysis in the [report](paper/shami_tts.pdf).
| System | Lev CER | Lev WER | CS CER | Overall CER | RTF |
|---|---|---|---|---|---|
| Published baseline (16 kHz, stochastic dur.) | 0.751 | 0.993 | 0.853 | 0.802 | 0.024 |
| **v2, HiFi-GAN (24 kHz)** | 0.113 | 0.377 | **0.498** | **0.305** | 0.028 |
| **v2, + BigVGAN** | **0.106** | **0.347** | 0.556 | 0.331 | ~0.085 |
**Texture:** high-frequency spectral flatness matches the reference (0.49–0.51 vs 0.488).
**Duration:** ratio 0.97 at natural pacing. **Robustness:** 8/8 novel unseen sentences
(long, heavy code-switch, numbers/dates) synthesized cleanly.
> **On the code-switch CER column:** BigVGAN slightly *raises* it while *improving* pure
> Levantine and perceived quality. This is a **Whisper artifact** — Whisper is MSA/English
> biased and mis-scores the English-in-Arabic segments; it is not an audible regression.
> Absolute CER via ASR has a floor well above zero on dialectal Arabic; treat **relative**
> numbers and the released audio as the source of truth. Human MOS is future work.
---
## 5. Training recipe (single RTX 3060, 12 GB, bf16)
Warm-started stages, each a fine-tune of the previous best:
```
prepare_lahgtna.py extract --target-sr 24000 → phonemize (dialectal front-end)
finetune_gan.py --deterministic-duration --sample-rate 24000 --c-fm 4 --c-stft 3 \
--ckpt (generator + discriminator persisted, atomic saves)
eval_checkpoint.py --auto-ls --asr (sweep checkpoints; GAN is non-monotonic)
bigvgan_vocoder.synthesize (final vocoding)
```
Key HPs: AdamW(0.8,0.99), lr 2e-4, batch 12–16, `seg_size` 8192–12288, `c_mel`=45,
`c_dur`=1–2, `c_fm`=4, `c_stft`=3, MR-STFT ffts 512/1024/2048, 8k–10k steps/stage.
## 6. Data
[`mohammedaly22/lahgtna-levantine-tts`](https://huggingface.co/datasets/mohammedaly22/lahgtna-levantine-tts)
(CC-BY-4.0): 50k clips / 66.8 h / 24 kHz / 10 speakers, Shami Levantine + ~12% synthetic
code-switch, partially diacritized. This release = single speaker "Badr" (~6.2 h).
## 7. Limitations
Single speaker; small eval set (±0.02 CER noise); ASR-CER floor on dialectal Arabic
under-states true quality; BigVGAN path runs two vocoders (a single-pass mel→BigVGAN model is
v3); multi-speaker is v3.
## 8. Licensing
- **Code:** Apache-2.0 (this project's source).
- **These weights:** **CC-BY-NC-4.0** — derived from `facebook/mms-tts-ara` (Meta MMS), which
is non-commercial. **Non-commercial use only** unless you retrain from a permissive base.
- Dataset: CC-BY-4.0. BigVGAN: MIT. espeak-ng: GPLv3 (tool). CAMeL Tools: MIT/…
## 9. Citation
```bibtex
@techreport{shami_tts_2026,
title = {Shami-TTS: A Production-Grade Streaming TTS for Levantine Arabic/English
Code-Switching via Dialectal Front-End, Deterministic Duration, and
Anti-Aliased Neural Vocoding},
author = {{Tushe Language Research Team}},
year = {2026},
institution = {Tushe Language Research},
note = {https://huggingface.co/Tushe/shami-tts}
}
```
Built on VITS (Kim et al., 2021), HiFi-GAN (Kong et al., 2020), BigVGAN (Lee et al., 2023),
MMS (Pratap et al., 2023), CAMeL Tools (Obeid et al., 2020), Whisper (Radford et al., 2023).