README.md

---
license: cc-by-nc-4.0
language:
- ar
- en
tags:
- text-to-speech
- tts
- vits
- bigvgan
- levantine-arabic
- arabic
- code-switching
- streaming
- low-latency
- real-time
- dialectal-arabic
base_model: facebook/mms-tts-ara
pipeline_tag: text-to-speech
library_name: pytorch
---

# Shami-TTS — Production-Grade Levantine Arabic ↔ English Code-Switching TTS

> A low-latency, **streaming** text-to-speech system that speaks natural, **Levantine
> Arabic** and switches to **English** mid-utterance without a prosodic seam. v2 turns the
> promising-but-robotic v1 baseline into a **production-grade** voice on a **single consumer
> GPU (RTX 3060, 12 GB)**: **Levantine CER 0.75 → 0.11 (−86%), WER 0.99 → 0.35**, natural
> prosody, 24 kHz fidelity, and reference-level texture — at **~11× faster than real-time**.

This card documents v2 **exhaustively** — what changed from v1, the architecture, the two
vocoder options we release, measured results, usage for both, the training recipe, an honest
engineering log, and licensing. A full **9-page technical report** is included at
[`paper/shami_tts.pdf`](paper/shami_tts.pdf).

---

## 0. TL;DR

| Held-out metric (ASR round-trip, Whisper large-v3, ↓ better) | Baseline (v1) | **v2 HiFi-GAN** | **v2 + BigVGAN** |
|---|---|---|---|
| **Levantine Arabic CER** | 0.751 | 0.113 | **0.106** |
| **Levantine Arabic WER** | 0.993 | 0.377 | **0.347** |
| Code-switch CER | 0.853 | **0.498** | 0.556 |
| Overall CER | 0.802 | **0.305** | 0.331 |
| Real-time factor (RTX 3060) | 0.024 | 0.028 | ~0.085 |
| Duration ratio @ `length_scale=1` | 0.31 (needs 3.2× stretch) | **0.97** | **0.97** |

- **Two vocoders released.** The **HiFi-GAN** decoder is a single self-contained VITS model
  (minimal deps). **BigVGAN** re-vocodes for the cleanest texture (removes the residual
  high-frequency breathiness). Pick per your fidelity/latency/dependency budget.
- **Single flagship speaker** ("Badr") in this release; multi-speaker is v3.
- **Infer at `length_scale = 1.0`** (deterministic duration — do *not* use 3–5 as in v1).

---

## 🔊 Listen

Synthesized at `length_scale=1.0`. **Reference** = real recording; **HiFi-GAN** = self-contained decoder; **BigVGAN** = highest-fidelity vocoder. Players stream on demand.

### Pure Levantine

ضلت واقفة ساكتة بدون ما تحكي بَسْ كانت تتفرج علي وانا كنت راقب المي وهي عم تنزل وسرحانة.

| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_01_reference.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_01_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_01_bigvgan.wav"></audio> |

حسيت فيهن وَجَع و انكسار وجعو متل وجعي التنين مظلومين و بدون حيلة.

| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_02_reference.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_02_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_02_bigvgan.wav"></audio> |

ما فيك تعمل شُغُل مرتب بلبنان الا اذا كان عندك ظهر ماكن.

| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_03_reference.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_03_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/lev_03_bigvgan.wav"></audio> |

### Code-switching

ليش الtraining مِشْ شغّال on the server؟.

| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_01_reference.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_01_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_01_bigvgan.wav"></audio> |

في مشكلة ب the error، لازم نصلّحها.

| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_02_reference.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_02_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_02_bigvgan.wav"></audio> |

حكيتلك عن the project قبل؟ هو light.

| Reference | HiFi-GAN | BigVGAN |
|---|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_03_reference.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_03_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/cs_03_bigvgan.wav"></audio> |

### Showcase — novel unseen text

**Long, natural Levantine** — بَعرِف إنّو الوَقِت تأخّر كْتير، بَس صَدقيني ما قَصّرِت، ضلّيت عَم حاوِل طول النّهار لَحتّى خلّصِت كِل الشّغِل.

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_01_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_01_bigvgan.wav"></audio> |

**Heavy code-switch (AR + EN)** — الـ meeting بُكرا الساعة تسعة، وبَدّي نراجِع الـ dashboard والـ analytics قَبِل ما نعمِل الـ presentation.

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_02_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_02_bigvgan.wav"></audio> |

**Numbers, currency, date** — دَفعِت مية وخمسة وعشرين دولار عَ الأوتيل، والرِّحلة بتكلّف تلاتة آلاف وخمسمية ليرة، والموعد يوم اتنين.

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_03_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_03_bigvgan.wav"></audio> |

**Question + exclamation** — شو صار مَعَك؟! ليش مِتضايِق هَلقَد؟ يَلّا احكيلي كِل شي بالتّفصيل، لا تِخبّي عَنّي وَلا كِلمة!

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_04_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_04_bigvgan.wav"></audio> |

**Tech / brand names** — نزّلِت آخِر version مِن الـ app بَس الـ update كَسَر الـ login، فبعتِتلهُن ticket عَ الـ support.

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_05_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_05_bigvgan.wav"></audio> |

**English-heavy technical** — The neural vocoder reduces spectral artifacts, بَس لازِم نـ fine-tune الـ model عَ بيانات أنضَف.

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_06_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_06_bigvgan.wav"></audio> |

**Warm greeting (assistant)** — مَرحَبا فيك! أنا مساعِدَك الصوتي، كيف بَقدِر ساعدَك اليوم؟ just tell me what you need.

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_07_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_07_bigvgan.wav"></audio> |

**Emphatics + gutturals** — الطّبيب قال إنّو الصّحّة بخير، والضّغِط طبيعي، والقَلِب عَم يِشتِغِل مْنيح، الحَمدُ لله عَ كِل شي.

| HiFi-GAN | BigVGAN (recommended) |
|---|---|
| <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_08_hifigan.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/Tushe/shami-tts/resolve/main/samples/show_08_bigvgan.wav"></audio> |

---

## 1. What's new in v2 (vs. the published baseline)

v2 is four targeted contributions, each measured (see the [report](paper/shami_tts.pdf)):

1. **Dialectal front-end.** Automatic diacritization (CAMeL Tools) is now the *default* path,
   so ~100% (not 0.4%) of bare conversational text is phonemized through the high-quality
   Levantine rule-G2P. A new **de-desinentialization** step strips MSA case/mood endings that
   Levantine drops (`مَوْعِد` → `moːʕid`, not `moːʕidi`) while preserving gemination.
   → Levantine CER 0.75 → 0.46.
2. **Deterministic duration head.** Replaces VITS's stochastic duration predictor (which
   under-predicted ~3×, forcing a prosody-flattening global stretch) with a deterministic
   head trained by MSE on alignment durations. Natural rhythm at `length_scale=1`.
   → CER 0.46 → **0.11** (the robotic→natural unlock).
3. **24 kHz fidelity + texture recipe.** Decoder fine-tuned to native 24 kHz; feature-matching
   up-weighted; the discriminator is now **persisted across warm-starts** (it used to restart
   from scratch — a real bug that caused buzz); a multi-resolution STFT loss added.
   → full-band fidelity, buzz at reference level, CER flat.
4. **Anti-aliased BigVGAN vocoder.** Removes the residual high-frequency breathiness
   (a phase/aliasing artifact) that magnitude losses can't fix. → texture matches the source.

The story is two-regime: **intelligibility** is solved by data + duration (steep CER drop),
then **fidelity/texture** improvements (24 kHz, feature-matching, BigVGAN) hold CER while
raising perceptual quality — which CER does not measure. Trust the audio in `samples/`.

---

## 2. Architecture

```
text (AR / EN / mixed)
   → normalize → code-switch segment (script-based) → per-span verbalize
   → diacritize (Arabic) + G2P (Levantine rules / espeak-ng English)
   → shared 88-symbol IPA + parallel language-ID stream
   → ShamiVITS acoustic model  (mms-tts-ara backbone + IPA/lang embeddings
                                 + DETERMINISTIC duration head), 24 kHz
   → HiFi-GAN decoder  ──────────────►  waveform            (self-contained)
        └► mel ► BigVGAN vocoder ─────►  waveform (cleanest) (highest fidelity)
```

- **Acoustic model (ShamiVITS):** ~36 M params. VITS core (CVAE + flow + monotonic alignment),
  with the text-encoder embedding replaced by a shared IPA table, an additive language-ID
  embedding, and the stochastic duration predictor swapped for the deterministic head.
  *(In the reference code this is the `HamsVITS` class in the `hams_tts` package — the import
  path is unchanged for backward compatibility.)*
- **Vocoders:** native HiFi-GAN decoder (~inside VITS), or pretrained
  [`nvidia/bigvgan_v2_24khz_100band_256x`](https://huggingface.co/nvidia/bigvgan_v2_24khz_100band_256x)
  (112 M, MIT) — an exact match for our 24 kHz / hop-256 pipeline, used with **no fine-tuning**.

---

## 3. Usage

Full code, front-end, and inference: **https://github.com/Al-aminI/hams-levantine-tts**

### 3a. HiFi-GAN (single model, minimal deps)
```python
import torch, soundfile as sf
from hams_tts.models.hams_vits import HamsVITS
from hams_tts.text.frontend import TextFrontend   # needs espeak-ng + camel-tools for AR

fe = TextFrontend(diacritizer_backend="auto")
model = HamsVITS.from_checkpoint("path/to/this/repo").cuda().eval()   # loads hams_vits.pt + config

u = fe.process("مَرحَبا، أنا مساعِدَك الصوتي، how can I help you today?")
pid = torch.tensor([u.phoneme_ids]).cuda(); lid = torch.tensor([u.language_ids]).cuda()
wav = model.infer(pid, lid, length_scale=1.0).squeeze().cpu().numpy()  # 24 kHz
sf.write("out.wav", wav, model.sample_rate)
```

### 3b. + BigVGAN (highest fidelity)
```python
from hams_tts.inference.bigvgan_vocoder import synthesize   # VITS → BigVGAN → trim
wav = synthesize(model, u.phoneme_ids, u.language_ids, length_scale=1.0)
sf.write("out_bigvgan.wav", wav, model.sample_rate)
```
BigVGAN is fetched from the Hub on first use (`use_cuda_kernel=False` → pure-PyTorch,
Windows/CPU-safe). **Always synthesize at `length_scale=1.0`.**

---

## 4. Results (held-out set, n=40)

Consistently measured; ASR round-trip via Whisper large-v3, Arabic forced, text normalized
(diacritics/alef/ya/ta-marbuta folded). Full analysis in the [report](paper/shami_tts.pdf).

| System | Lev CER | Lev WER | CS CER | Overall CER | RTF |
|---|---|---|---|---|---|
| Published baseline (16 kHz, stochastic dur.) | 0.751 | 0.993 | 0.853 | 0.802 | 0.024 |
| **v2, HiFi-GAN (24 kHz)** | 0.113 | 0.377 | **0.498** | **0.305** | 0.028 |
| **v2, + BigVGAN** | **0.106** | **0.347** | 0.556 | 0.331 | ~0.085 |

**Texture:** high-frequency spectral flatness matches the reference (0.49–0.51 vs 0.488).
**Duration:** ratio 0.97 at natural pacing. **Robustness:** 8/8 novel unseen sentences
(long, heavy code-switch, numbers/dates) synthesized cleanly.

> **On the code-switch CER column:** BigVGAN slightly *raises* it while *improving* pure
> Levantine and perceived quality. This is a **Whisper artifact** — Whisper is MSA/English
> biased and mis-scores the English-in-Arabic segments; it is not an audible regression.
> Absolute CER via ASR has a floor well above zero on dialectal Arabic; treat **relative**
> numbers and the released audio as the source of truth. Human MOS is future work.

---

## 5. Training recipe (single RTX 3060, 12 GB, bf16)

Warm-started stages, each a fine-tune of the previous best:
```
prepare_lahgtna.py extract --target-sr 24000 → phonemize (dialectal front-end)
finetune_gan.py --deterministic-duration --sample-rate 24000 --c-fm 4 --c-stft 3 \
                --ckpt <prev-best>   (generator + discriminator persisted, atomic saves)
eval_checkpoint.py --auto-ls --asr   (sweep checkpoints; GAN is non-monotonic)
bigvgan_vocoder.synthesize            (final vocoding)
```
Key HPs: AdamW(0.8,0.99), lr 2e-4, batch 12–16, `seg_size` 8192–12288, `c_mel`=45,
`c_dur`=1–2, `c_fm`=4, `c_stft`=3, MR-STFT ffts 512/1024/2048, 8k–10k steps/stage.

## 6. Data
[`mohammedaly22/lahgtna-levantine-tts`](https://huggingface.co/datasets/mohammedaly22/lahgtna-levantine-tts)
(CC-BY-4.0): 50k clips / 66.8 h / 24 kHz / 10 speakers, Shami Levantine + ~12% synthetic
code-switch, partially diacritized. This release = single speaker "Badr" (~6.2 h).

## 7. Limitations
Single speaker; small eval set (±0.02 CER noise); ASR-CER floor on dialectal Arabic
under-states true quality; BigVGAN path runs two vocoders (a single-pass mel→BigVGAN model is
v3); multi-speaker is v3.

## 8. Licensing
- **Code:** Apache-2.0 (this project's source).
- **These weights:** **CC-BY-NC-4.0** — derived from `facebook/mms-tts-ara` (Meta MMS), which
  is non-commercial. **Non-commercial use only** unless you retrain from a permissive base.
- Dataset: CC-BY-4.0. BigVGAN: MIT. espeak-ng: GPLv3 (tool). CAMeL Tools: MIT/…

## 9. Citation
```bibtex
@techreport{shami_tts_2026,
  title  = {Shami-TTS: A Production-Grade Streaming TTS for Levantine Arabic/English
            Code-Switching via Dialectal Front-End, Deterministic Duration, and
            Anti-Aliased Neural Vocoding},
  author = {{Tushe Language Research Team}},
  year   = {2026},
  institution = {Tushe Language Research},
  note   = {https://huggingface.co/Tushe/shami-tts}
}
```
Built on VITS (Kim et al., 2021), HiFi-GAN (Kong et al., 2020), BigVGAN (Lee et al., 2023),
MMS (Pratap et al., 2023), CAMeL Tools (Obeid et al., 2020), Whisper (Radford et al., 2023).