s2pro-egy — Egyptian Arabic fine-tune of Fish Audio S2-Pro

Merged production model + full training/serving toolkit. Phase 1 complete (2026-07-18/19); Phase 2 planned — see below.

⚠️ License: derivative of fishaudio/s2-pro (Fish Audio Research License — non-commercial only). Production/commercial use requires a written license from Fish Audio (business@fish.audio). Keep this repo private.

What's in this repo

Path	Contents
/ (root)	Merged model, HF fish_qwen3_omni layout: safetensors shards + index, config.json, tokenizer, chat_template.jinja, codec.pth — loads in fish-speech main AND sglang-omni
checkpoints/	Phase-1 LoRA checkpoints (fast-AR only, steps 100–1200; step_000001200.ckpt is the one merged)
scripts/	Full pipeline: data prep, relabeling, eval (synth + Soniox judge), serving (PyTorch + sglang shim), weight conversion, patches
configs/	Training configs (text2semantic_finetune_egy2.yaml = the recipe that worked) + LoRA configs
eval/	Mega-paragraph A/B wavs + Soniox WER reports (baseline vs checkpoints)
logs/	Training log + tensorboard events

Phase 1 — what was done

Data (46.9 h / 15,030 clips, all 24 kHz mono, loudness-normalized)

Dataset (HF, private)	Clips	Hours	Transcript source
ehabnegm/noselleel-egyptian-tts	8,766	26.2	Soniox stt-async-v5 (6,344 clips, transcripts_soniox/train.jsonl) > text_raw (pre-CATT Whisper)
ehabnegm/eqkawkab-egyptian-tts	2,062	8.0	text_raw (pre-CATT Whisper)
ehabnegm/moustafa-sadek-egyptian-tts	2,883	8.0	Deepgram nova-3
ehabnegm/mosaifside-egyptian-tts	1,319	4.7	Deepgram nova-3

Processing rules (see scripts/prepare_data.py, scripts/relab3.py):

• NO tashkeel anywhere (production input is plain text; CATT diacritics were MSA-flavored noise).
• Speaker grouping = per YouTube video (nos_<VID>…, 121 groups) because the trainer packs same-folder clips into one sequence.
• VQ tokens via s2-pro codec.pth (modded_dac_vq), protobuf shards via build_dataset.py.
• ~231 clips contain subscribe-outro phrases (1.5%) — left in for phase 1; candidate for dedupe in phase 2.

Training recipe (the one that works)

Community-validated recipe (credit: Enucatl/fish-speech barbero):

• LoRA fast-AR (audio decoder) ONLY — stock r_32_alpha_16_fast (r=32, α=16, α/r=0.5)
• batch 1 × grad-accum 4, lr 1e-5, CosineAnnealingLR → 1e-6, weight_decay 0.01
• causal: false (random sampling), max_steps 1200, ckpt every 100, bf16-true, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
• Run: python fish_speech/train.py --config-name text2semantic_finetune_egy2 +lora@model.model.lora_config=r_32_alpha_16_fast

⚠️ NEVER LoRA the slow AR with α/r ≥ 1.0: phase-1's first attempt (r32 α32 attention+mlp on both transformers, lr 5e-5) produced pure noise by step 200 — the slow/text transformer collapsed (confirmed by slow/fast ablation, and it is literally a row in the community doc's "didn't work" table).

Required upstream patches (in scripts/patches_notes.md):

• llama.py: use_reentrant=False in both checkpoint() calls (LoRA + frozen embeddings otherwise breaks the grad graph)
• lit_module.py: strict_loading = False on TextToSemantic (checkpoints are LoRA-only; Lightning resume fails otherwise)

Results (Soniox stt-async-v5 judge — Whisper hallucinates on Arabic noise, do not use it)

325-word Egyptian mega-paragraph, masry reference voice:

System	WER	CER
s2-pro zero-shot baseline	0.071	0.05
fine-tuned step-1200 (this model)	0.036–0.06	0.007–0.05

Fine-tune preserves Egyptian forms the baseline drifts on (e.g. معايا vs معي). Fast-AR tuning = voice/timbre adaptation; text-following stays RL-aligned.

Serving (production, RTX 5090)

Engine: sglang-omni (S2-Pro supported natively) + OpenAI shim. Measured: TTFA ~0.5 s (streaming, warm radix cache), RTF ~0.85, quality WER 0.00 on server output, output resampled to 24 kHz.

Blackwell/sm_120 porting patches (all required, scripts/patches_notes.md):

1. delete/rename site-packages/deep_gemm (asserts on missing CUDA_HOME at import)
2. apt install cuda-nvcc-13-0 libcublas-dev-13-0 libcusparse-dev-13-0 libcusolver-dev-13-0 libcurand-dev-13-0 (torch cu130) + ninja
3. engine_builder.py: attention backend fa3 → triton (FA3 = Hopper-only)
4. engine_builder.py: disable_cuda_graph: True (graph capture calls FA3 kernels), mem_fraction_static: 0.55
5. audio_decoder.py: FISH_FORCE_SDPA=1 env-gated pure-SDPA replacement for sgl_kernel.flash_attn_with_kvcache (scripts/patch_audio_decoder.py) — Fast-AR attends ≤11 positions, SDPA is exact and fast

Launch:

export CUDA_HOME=/usr/local/cuda-13.0 SGLANG_ENABLE_JIT_DEEPGEMM=0 FISH_FORCE_SDPA=1
sgl-omni serve --model-path <this-repo-dir> --config examples/configs/s2pro_tts.yaml --port 8001
python scripts/serve_shim.py --port 8000   # OpenAI contract: model=s2pro-egy, voices, 24kHz

API (OpenAI-compatible): POST /v1/audio/speech {"model":"s2pro-egy","input":"...","voice":"masry","stream":true,"response_format":"pcm"}. Voices = voices/<name>.wav+.txt reference pairs (masry = eqkawkab narrator; noselleel/noselleel2/noselleel3 = noselleel narrator candidates).

Phase 2 — plan (next session)

Goal: deeper Egyptian pronunciation/prosody (slow-AR territory) without breaking RL alignment.

1. Config: r32 α16 ALL modules (α/r = 0.5, attention+mlp on slow+fast) — the community doc's "minor slow-AR degradation, usable — worth investigating" row. lr 1e-5 cosine, wd 0.01, causal false, max 1200 steps, ckpt every 100.
2. Start from: this merged model as the base (pretrained_ckpt_path → merged dir), so phase-1 voice gains are kept.
3. Hard gates: eval at step 200/400 with Soniox WER + human listening vs phase-1 (comparison page: scripts/serve_eval.py). The failure mode is audio ending early / going quiet / noise — stop immediately if WER degrades, keep phase-1.
4. Data option: drop/downweight the 231 subscribe-outro clips; optionally add more clean hours.
5. Success metric: phase-2 ≥ phase-1 WER AND user prefers pronunciation blind.

Provenance

• Base: fishaudio/s2-pro · Training/eval infra: fish-speech main (e5e2926) · Serving: sglang-omni
• Fine-tuned 2026-07-18/19 on 1× RTX 5090 (32 GB), total ~1.5 h training for phase 1.