ehabnegm
/

s2pro-egy

+---
+license: other
+license_name: fish-audio-research-license
+license_link: https://huggingface.co/fishaudio/s2-pro/blob/main/LICENSE.md
+language: [ar]
+base_model: fishaudio/s2-pro
+tags: [text-to-speech, egyptian-arabic, arz, fish-speech, lora]
+pipeline_tag: text-to-speech
+---
+# s2pro-egy — Egyptian Arabic fine-tune of Fish Audio S2-Pro
+Merged production model + full training/serving toolkit. **Phase 1 complete (2026-07-18/19); Phase 2 planned — see below.**
+> ⚠️ **License**: derivative of `fishaudio/s2-pro` (Fish Audio Research License — **non-commercial only**).
+> Production/commercial use requires a written license from Fish Audio (business@fish.audio). Keep this repo private.
+## What's in this repo
+| Path | Contents |
+|---|---|
+| `/` (root) | Merged model, HF `fish_qwen3_omni` layout: safetensors shards + index, `config.json`, tokenizer, `chat_template.jinja`, `codec.pth` — loads in fish-speech `main` AND sglang-omni |
+| `checkpoints/` | Phase-1 LoRA checkpoints (fast-AR only, steps 100–1200; `step_000001200.ckpt` is the one merged) |
+| `scripts/` | Full pipeline: data prep, relabeling, eval (synth + Soniox judge), serving (PyTorch + sglang shim), weight conversion, patches |
+| `configs/` | Training configs (`text2semantic_finetune_egy2.yaml` = the recipe that worked) + LoRA configs |
+| `eval/` | Mega-paragraph A/B wavs + Soniox WER reports (baseline vs checkpoints) |
+| `logs/` | Training log + tensorboard events |
+## Phase 1 — what was done
+### Data (46.9 h / 15,030 clips, all 24 kHz mono, loudness-normalized)
+| Dataset (HF, private) | Clips | Hours | Transcript source |
+|---|---|---|---|
+| `ehabnegm/noselleel-egyptian-tts` | 8,766 | 26.2 | **Soniox stt-async-v5** (6,344 clips, `transcripts_soniox/train.jsonl`) > `text_raw` (pre-CATT Whisper) |
+| `ehabnegm/eqkawkab-egyptian-tts` | 2,062 | 8.0 | `text_raw` (pre-CATT Whisper) |
+| `ehabnegm/moustafa-sadek-egyptian-tts` | 2,883 | 8.0 | Deepgram nova-3 |
+| `ehabnegm/mosaifside-egyptian-tts` | 1,319 | 4.7 | Deepgram nova-3 |
+Processing rules (see `scripts/prepare_data.py`, `scripts/relab3.py`):
+- **NO tashkeel anywhere** (production input is plain text; CATT diacritics were MSA-flavored noise).
+- Speaker grouping = **per YouTube video** (`nos_<VID>`…, 121 groups) because the trainer packs same-folder clips into one sequence.
+- VQ tokens via s2-pro `codec.pth` (`modded_dac_vq`), protobuf shards via `build_dataset.py`.
+- ~231 clips contain subscribe-outro phrases (1.5%) — left in for phase 1; candidate for dedupe in phase 2.
+### Training recipe (the one that works)
+Community-validated recipe (credit: [Enucatl/fish-speech barbero](https://github.com/Enucatl/fish-speech/blob/barbero/lora-finetune.md)):
+- LoRA **fast-AR (audio decoder) ONLY** — stock `r_32_alpha_16_fast` (r=32, α=16, α/r=0.5)
+- batch 1 × grad-accum 4, lr **1e-5**, CosineAnnealingLR → 1e-6, weight_decay 0.01
+- `causal: false` (random sampling), max_steps **1200**, ckpt every 100, bf16-true, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`
+- Run: `python fish_speech/train.py --config-name text2semantic_finetune_egy2 +lora@model.model.lora_config=r_32_alpha_16_fast`
+**⚠️ NEVER LoRA the slow AR with α/r ≥ 1.0**: phase-1's first attempt (r32 α32 attention+mlp on both transformers, lr 5e-5) produced pure noise by step 200 — the slow/text transformer collapsed (confirmed by slow/fast ablation, and it is literally a row in the community doc's "didn't work" table).
+Required upstream patches (in `scripts/patches_notes.md`):
+- `llama.py`: `use_reentrant=False` in both `checkpoint()` calls (LoRA + frozen embeddings otherwise breaks the grad graph)
+- `lit_module.py`: `strict_loading = False` on `TextToSemantic` (checkpoints are LoRA-only; Lightning resume fails otherwise)
+### Results (Soniox stt-async-v5 judge — Whisper hallucinates on Arabic noise, do not use it)
+325-word Egyptian mega-paragraph, `masry` reference voice:
+| System | WER | CER |
+|---|---|---|
+| s2-pro zero-shot baseline | 0.071 | 0.05 |
+| **fine-tuned step-1200 (this model)** | **0.036–0.06** | 0.007–0.05 |
+Fine-tune preserves Egyptian forms the baseline drifts on (e.g. `معايا` vs `معي`). Fast-AR tuning = voice/timbre adaptation; text-following stays RL-aligned.
+## Serving (production, RTX 5090)
+**Engine: [sglang-omni](https://github.com/sgl-project/sglang-omni)** (S2-Pro supported natively) + OpenAI shim.
+Measured: **TTFA ~0.5 s** (streaming, warm radix cache), RTF ~0.85, quality WER 0.00 on server output, output resampled to **24 kHz**.
+Blackwell/sm_120 porting patches (all required, `scripts/patches_notes.md`):
+1. delete/rename `site-packages/deep_gemm` (asserts on missing CUDA_HOME at import)
+2. `apt install cuda-nvcc-13-0 libcublas-dev-13-0 libcusparse-dev-13-0 libcusolver-dev-13-0 libcurand-dev-13-0` (torch cu130) + `ninja`
+3. `engine_builder.py`: attention backend `fa3` → `triton` (FA3 = Hopper-only)
+4. `engine_builder.py`: `disable_cuda_graph: True` (graph capture calls FA3 kernels), `mem_fraction_static: 0.55`
+5. `audio_decoder.py`: `FISH_FORCE_SDPA=1` env-gated pure-SDPA replacement for `sgl_kernel.flash_attn_with_kvcache` (`scripts/patch_audio_decoder.py`) — Fast-AR attends ≤11 positions, SDPA is exact and fast
+Launch:
+```bash
+export CUDA_HOME=/usr/local/cuda-13.0 SGLANG_ENABLE_JIT_DEEPGEMM=0 FISH_FORCE_SDPA=1
+sgl-omni serve --model-path <this-repo-dir> --config examples/configs/s2pro_tts.yaml --port 8001
+python scripts/serve_shim.py --port 8000   # OpenAI contract: model=s2pro-egy, voices, 24kHz
+```
+API (OpenAI-compatible): `POST /v1/audio/speech` `{"model":"s2pro-egy","input":"...","voice":"masry","stream":true,"response_format":"pcm"}`.
+Voices = `voices/<name>.wav+.txt` reference pairs (`masry` = eqkawkab narrator; `noselleel`/`noselleel2`/`noselleel3` = noselleel narrator candidates).
+## Phase 2 — plan (next session)
+**Goal:** deeper Egyptian pronunciation/prosody (slow-AR territory) without breaking RL alignment.
+1. **Config:** `r32 α16 ALL modules` (α/r = 0.5, attention+mlp on slow+fast) — the community doc's "minor slow-AR degradation, usable — worth investigating" row. lr 1e-5 cosine, wd 0.01, causal false, max 1200 steps, ckpt every 100.
+2. **Start from:** this merged model as the base (`pretrained_ckpt_path` → merged dir), so phase-1 voice gains are kept.
+3. **Hard gates:** eval at step 200/400 with Soniox WER + human listening vs phase-1 (comparison page: `scripts/serve_eval.py`). The failure mode is audio ending early / going quiet / noise — **stop immediately if WER degrades**, keep phase-1.
+4. **Data option:** drop/downweight the 231 subscribe-outro clips; optionally add more clean hours.
+5. **Success metric:** phase-2 ≥ phase-1 WER AND user prefers pronunciation blind.
+## Provenance
+- Base: `fishaudio/s2-pro` · Training/eval infra: fish-speech `main` (e5e2926) · Serving: sglang-omni
+- Fine-tuned 2026-07-18/19 on 1× RTX 5090 (32 GB), total ~1.5 h training for phase 1.