| --- |
| license: other |
| license_name: fish-audio-research-license |
| license_link: https://huggingface.co/fishaudio/s2-pro/blob/main/LICENSE.md |
| language: [ar] |
| base_model: fishaudio/s2-pro |
| tags: [text-to-speech, egyptian-arabic, arz, fish-speech, lora] |
| pipeline_tag: text-to-speech |
| --- |
| |
| # s2pro-egy — Egyptian Arabic fine-tune of Fish Audio S2-Pro |
|
|
| Merged production model + full training/serving toolkit. **Phase 1 complete (2026-07-18/19); Phase 2 planned — see below.** |
|
|
| > ⚠️ **License**: derivative of `fishaudio/s2-pro` (Fish Audio Research License — **non-commercial only**). |
| > Production/commercial use requires a written license from Fish Audio (business@fish.audio). Keep this repo private. |
|
|
| ## What's in this repo |
|
|
| | Path | Contents | |
| |---|---| |
| | `/` (root) | Merged model, HF `fish_qwen3_omni` layout: safetensors shards + index, `config.json`, tokenizer, `chat_template.jinja`, `codec.pth` — loads in fish-speech `main` AND sglang-omni | |
| | `checkpoints/` | Phase-1 LoRA checkpoints (fast-AR only, steps 100–1200; `step_000001200.ckpt` is the one merged) | |
| | `scripts/` | Full pipeline: data prep, relabeling, eval (synth + Soniox judge), serving (PyTorch + sglang shim), weight conversion, patches | |
| | `configs/` | Training configs (`text2semantic_finetune_egy2.yaml` = the recipe that worked) + LoRA configs | |
| | `eval/` | Mega-paragraph A/B wavs + Soniox WER reports (baseline vs checkpoints) | |
| | `logs/` | Training log + tensorboard events | |
|
|
| ## Phase 1 — what was done |
|
|
| ### Data (46.9 h / 15,030 clips, all 24 kHz mono, loudness-normalized) |
|
|
| | Dataset (HF, private) | Clips | Hours | Transcript source | |
| |---|---|---|---| |
| | `ehabnegm/noselleel-egyptian-tts` | 8,766 | 26.2 | **Soniox stt-async-v5** (6,344 clips, `transcripts_soniox/train.jsonl`) > `text_raw` (pre-CATT Whisper) | |
| | `ehabnegm/eqkawkab-egyptian-tts` | 2,062 | 8.0 | `text_raw` (pre-CATT Whisper) | |
| | `ehabnegm/moustafa-sadek-egyptian-tts` | 2,883 | 8.0 | Deepgram nova-3 | |
| | `ehabnegm/mosaifside-egyptian-tts` | 1,319 | 4.7 | Deepgram nova-3 | |
|
|
| Processing rules (see `scripts/prepare_data.py`, `scripts/relab3.py`): |
| - **NO tashkeel anywhere** (production input is plain text; CATT diacritics were MSA-flavored noise). |
| - Speaker grouping = **per YouTube video** (`nos_<VID>`…, 121 groups) because the trainer packs same-folder clips into one sequence. |
| - VQ tokens via s2-pro `codec.pth` (`modded_dac_vq`), protobuf shards via `build_dataset.py`. |
| - ~231 clips contain subscribe-outro phrases (1.5%) — left in for phase 1; candidate for dedupe in phase 2. |
|
|
| ### Training recipe (the one that works) |
|
|
| Community-validated recipe (credit: [Enucatl/fish-speech barbero](https://github.com/Enucatl/fish-speech/blob/barbero/lora-finetune.md)): |
|
|
| - LoRA **fast-AR (audio decoder) ONLY** — stock `r_32_alpha_16_fast` (r=32, α=16, α/r=0.5) |
| - batch 1 × grad-accum 4, lr **1e-5**, CosineAnnealingLR → 1e-6, weight_decay 0.01 |
| - `causal: false` (random sampling), max_steps **1200**, ckpt every 100, bf16-true, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |
| - Run: `python fish_speech/train.py --config-name text2semantic_finetune_egy2 +lora@model.model.lora_config=r_32_alpha_16_fast` |
|
|
| **⚠️ NEVER LoRA the slow AR with α/r ≥ 1.0**: phase-1's first attempt (r32 α32 attention+mlp on both transformers, lr 5e-5) produced pure noise by step 200 — the slow/text transformer collapsed (confirmed by slow/fast ablation, and it is literally a row in the community doc's "didn't work" table). |
|
|
| Required upstream patches (in `scripts/patches_notes.md`): |
| - `llama.py`: `use_reentrant=False` in both `checkpoint()` calls (LoRA + frozen embeddings otherwise breaks the grad graph) |
| - `lit_module.py`: `strict_loading = False` on `TextToSemantic` (checkpoints are LoRA-only; Lightning resume fails otherwise) |
|
|
| ### Results (Soniox stt-async-v5 judge — Whisper hallucinates on Arabic noise, do not use it) |
|
|
| 325-word Egyptian mega-paragraph, `masry` reference voice: |
|
|
| | System | WER | CER | |
| |---|---|---| |
| | s2-pro zero-shot baseline | 0.071 | 0.05 | |
| | **fine-tuned step-1200 (this model)** | **0.036–0.06** | 0.007–0.05 | |
|
|
| Fine-tune preserves Egyptian forms the baseline drifts on (e.g. `معايا` vs `معي`). Fast-AR tuning = voice/timbre adaptation; text-following stays RL-aligned. |
|
|
| ## Serving (production, RTX 5090) |
|
|
| **Engine: [sglang-omni](https://github.com/sgl-project/sglang-omni)** (S2-Pro supported natively) + OpenAI shim. |
| Measured: **TTFA ~0.5 s** (streaming, warm radix cache), RTF ~0.85, quality WER 0.00 on server output, output resampled to **24 kHz**. |
|
|
| Blackwell/sm_120 porting patches (all required, `scripts/patches_notes.md`): |
| 1. delete/rename `site-packages/deep_gemm` (asserts on missing CUDA_HOME at import) |
| 2. `apt install cuda-nvcc-13-0 libcublas-dev-13-0 libcusparse-dev-13-0 libcusolver-dev-13-0 libcurand-dev-13-0` (torch cu130) + `ninja` |
| 3. `engine_builder.py`: attention backend `fa3` → `triton` (FA3 = Hopper-only) |
| 4. `engine_builder.py`: `disable_cuda_graph: True` (graph capture calls FA3 kernels), `mem_fraction_static: 0.55` |
| 5. `audio_decoder.py`: `FISH_FORCE_SDPA=1` env-gated pure-SDPA replacement for `sgl_kernel.flash_attn_with_kvcache` (`scripts/patch_audio_decoder.py`) — Fast-AR attends ≤11 positions, SDPA is exact and fast |
|
|
| Launch: |
| ```bash |
| export CUDA_HOME=/usr/local/cuda-13.0 SGLANG_ENABLE_JIT_DEEPGEMM=0 FISH_FORCE_SDPA=1 |
| sgl-omni serve --model-path <this-repo-dir> --config examples/configs/s2pro_tts.yaml --port 8001 |
| python scripts/serve_shim.py --port 8000 # OpenAI contract: model=s2pro-egy, voices, 24kHz |
| ``` |
|
|
| API (OpenAI-compatible): `POST /v1/audio/speech` `{"model":"s2pro-egy","input":"...","voice":"masry","stream":true,"response_format":"pcm"}`. |
| Voices = `voices/<name>.wav+.txt` reference pairs (`masry` = eqkawkab narrator; `noselleel`/`noselleel2`/`noselleel3` = noselleel narrator candidates). |
|
|
| ## Phase 2 — plan (next session) |
|
|
| **Goal:** deeper Egyptian pronunciation/prosody (slow-AR territory) without breaking RL alignment. |
|
|
| 1. **Config:** `r32 α16 ALL modules` (α/r = 0.5, attention+mlp on slow+fast) — the community doc's "minor slow-AR degradation, usable — worth investigating" row. lr 1e-5 cosine, wd 0.01, causal false, max 1200 steps, ckpt every 100. |
| 2. **Start from:** this merged model as the base (`pretrained_ckpt_path` → merged dir), so phase-1 voice gains are kept. |
| 3. **Hard gates:** eval at step 200/400 with Soniox WER + human listening vs phase-1 (comparison page: `scripts/serve_eval.py`). The failure mode is audio ending early / going quiet / noise — **stop immediately if WER degrades**, keep phase-1. |
| 4. **Data option:** drop/downweight the 231 subscribe-outro clips; optionally add more clean hours. |
| 5. **Success metric:** phase-2 ≥ phase-1 WER AND user prefers pronunciation blind. |
|
|
| ## Provenance |
| - Base: `fishaudio/s2-pro` · Training/eval infra: fish-speech `main` (e5e2926) · Serving: sglang-omni |
| - Fine-tuned 2026-07-18/19 on 1× RTX 5090 (32 GB), total ~1.5 h training for phase 1. |
|
|
