--- license: other license_name: fish-audio-research-license license_link: https://github.com/maxmelichov/fish-speech/blob/main/LICENSE language: - he - en tags: - text-to-speech - tts - hebrew - fish-speech - s2-pro - lora - voice-cloning base_model: fishaudio/s2-pro library_name: fish-speech pipeline_tag: text-to-speech --- Fish Audio S2-Pro Hebrew Built with Fish Audio. # Fish Audio S2-Pro — Hebrew (LoRA + atomic IPA tokens) A Hebrew adapter for [`fishaudio/s2-pro`](https://huggingface.co/fishaudio/s2-pro). It keeps the base model's multilingual ability and voice cloning intact, and adds native Hebrew synthesis driven by **IPA** rather than nikud. This repo contains **only the adapter** (~67M parameters) plus the extended tokenizer. You still need the S2-Pro base weights and codec. ## What's here | File | What it is | |---|---| | `hebrew_lora_step2200.safetensors` | LoRA deltas + the trained `ipa_embeddings` table (67M params, bf16) | | `hebrew_lora_step2200.ckpt` | Same weights as a Lightning checkpoint, with optimizer state — use this to resume training | | `config.json` | S2-Pro config extended with `num_ipa_tokens: 26`, `ipa_token_start: 155774` | | `ipa_token_map.json` | IPA symbol → atomic token (e.g. `ʃ` → ``) | | `ipa_embeddings.pt` | Initial IPA embedding table (mean of the symbol's BPE pieces); the trained one lives in the adapter | | `tokenizer/` | S2-Pro tokenizer extended 155,774 → 155,800 tokens | | `samples/` | Generated audio (see below) | ## Quick start Code lives in the fork the adapter was trained with: ```bash git clone https://github.com/maxmelichov/fish-speech cd fish-speech && uv sync --python 3.12 --extra cu129 pip install renikud-plus # Hebrew grapheme-to-phoneme bash tools/hebrew/setup_hebrew.sh # base weights + this adapter + IPA checkpoint python tools/hebrew/infer_hebrew.py \ --text "שלום, מה שלומך היום?" \ --lora-checkpoint checkpoints/hebrew/hebrew_lora_step2200.safetensors \ --output out.wav ``` Add `--ref-audio my_voice.wav --ref-text "..."` to clone a voice. `infer_hebrew.py` runs plain unvocalized Hebrew through [RenikudPlus](https://github.com/maxmelichov/RenikudPlus) G2P, maps the IPA to the atomic tokens, and chunks long inputs on sentence boundaries. `--lora-scale` scales the delta (0.0 = pure base model) if you want to dial the adaptation down. **Fine-tuning on your own Hebrew data** is one command — a directory per speaker of `*.wav` plus sibling `.lab` transcripts: ```bash AUDIO_ROOT=my_audio tools/hebrew/run_hebrew_pipeline.sh ``` See [`tools/hebrew/README.md`](https://github.com/maxmelichov/fish-speech/blob/main/tools/hebrew/README.md) for the full guide. ## How it works **Atomic IPA tokens.** S2-Pro's BPE splits IPA into pieces that collide with English orthography — Hebrew `י` phonemized as `j` was read as the English letter *jay*. So each of the 26 Hebrew IPA symbols gets a dedicated input-only token (``, ``, …) in a separate trainable `nn.Embedding`, initialized to the mean of the symbol's original BPE pieces. The output vocabulary is untouched — these tokens are never predicted, only read. **What trains.** LoRA r=32, α=16 on `attention` + `mlp`, plus the IPA embedding table — 66.9M parameters total: 60.2M in the slow transformer, 6.7M in the fast transformer, 0.03M IPA embeddings. Frozen are the direct interfaces to codebook space — `fast_embeddings`, `fast_output`, and the tied slow embeddings/output — which is what keeps timbre close to the base model. Note α/r = 0.5, not the usual 2.0; see *Caveats*. The residual-codebook loss is down-weighted to 0.3 (Qwen3-TTS's sub-talker coefficient) so the gradient stays on the text→semantic mapping. **Training.** 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference- conditioned on a same-speaker utterance 80% of the time so training prompts match the exact `generate_long()` inference format. bf16, lr 5e-5 constant with 100-step warmup, effective batch 12, 2200 optimizer steps. ## Upstream bug fixed along the way S2-Pro sets `scale_codebook_embeddings=True`. At inference, `forward_generate()` divides semantic-position embeddings by `sqrt(num_codebooks + 1)` = 3.317; the training path in `embed()` did **not**. Every fine-tune therefore learned against embeddings 3.3× larger than the ones it would see at generation time. Teacher- forced CE looked fine while free-running generation collapsed after the first word — the classic symptom in fishaudio/fish-speech issues [#1136](https://github.com/fishaudio/fish-speech/issues/1136) (Japanese gibberish), [#682](https://github.com/fishaudio/fish-speech/issues/682) (Hindi noise) and [#814](https://github.com/fishaudio/fish-speech/issues/814). Five Hebrew runs collapsed the same way before this was found. After the fix (train and inference embeddings verified bit-identical): | | sample RMS | energy decay over the utterance | |---|---|---| | before | 0.008 – 0.022 | 0.07× | | after | 0.171 – 0.205 | 1.02× | | base model reference | 0.181 | — | The fix is in `fish_speech/models/text2semantic/llama.py` in the fork above and applies to any S2-Pro fine-tune, Hebrew or not. ## Samples - `00_base_out_of_the_box.wav` — **stock `fishaudio/s2-pro`, no adapter, no G2P** — plain Hebrew script straight in. S2-Pro is multilingual and does produce Hebrew-*shaped* speech, but it isn't accurate: this sample of "שלום, מה שלומך היום?" ("hello, how are you today?") comes out as "סלום מהשלום חיום" — שלום → סלום, שלומך garbled into משלום. Not a cherry-pick: an 8-seed sweep on a different sentence in this fork's eval found the same failure every time, and a stock-model WER of 0.383 across 11 sentences, worse than real human speech scores on the same metric. This is the gap the adapter below closes. The rest of `samples/` is generated **with this adapter**: - `01_podcast_2hosts_63s.wav` — 63s two-host Hebrew conversation, cloned voices - `03_longform_15s.wav` — multi-sentence long-form - `04_yod_BASE.wav` / `05_yod_LORA.wav` — the `י` → English *jay* failure, before and after atomic IPA tokens - `07_clone_LORA_ranlevi.wav` — voice clone from a real Hebrew speaker reference ## Caveats — this checkpoint is early, not final - **Undertrained, and stopped by hand.** 2,200 optimizer steps ≈ 53k utterances seen, under 20% of one epoch over the 279k-row set. Train loss was still falling (3.66 → 2.70 base CE) and val loss was still improving monotonically at every checkpoint (2.934 → 2.844 → 2.820 → 2.807). Nothing had plateaued; the run was simply halted. - **α/r = 0.5 is a workaround for a bug that no longer exists.** The unusual scaling was chosen empirically because at α=64 the delta destroyed free-running generation — which we now know was the embedding-scale bug above, not the LoRA strength. That rationale is obsolete post-fix, and the standard α = 2r was never re-tried. It may well be better. - The pitch homogenization noted below is the symptom you would expect from putting LoRA on the fast transformer at all. Freezing `fast_layers` entirely is the obvious next experiment. ## Known limitations - **Emotion tags (`[whisper]`, `[excited]`, …) do not work** — and this is not a regression from the LoRA. Measured on the *base* model in *English*: plain / whisper / shouting produced RMS 0.0655 / 0.0652 / 0.0689, i.e. no response at all. The released S2-Pro weights simply lack the tag alignment. - **Pitch is not cloned.** Timbre transfers well (4/4 by ear), but neither base nor LoRA reproduces the reference's F0 (base mean |err| 23 Hz, LoRA 20 Hz). The LoRA homogenizes pitch somewhat: spread across speakers drops from 66 Hz to 26 Hz. - Trained on read/broadcast-style Hebrew; conversational and heavily accented speech are out of distribution. - Hebrew input must go through G2P. Feeding nikud or bare Hebrew script directly to the model is out of distribution — use `infer_hebrew.py`, which handles it. ## License S2-Pro (the base model this adapter is trained on) is released under the **[Fish Audio Research License](https://github.com/maxmelichov/fish-speech/blob/main/LICENSE)** — *not* CC BY-NC-SA 4.0, correcting an earlier version of this card. This adapter is a Derivative Work under that license and inherits its terms: research and non-commercial use only (personal use, evaluation, academic work); any commercial use requires a separate license directly from Fish Audio (business@fish.audio). See `NOTICE` and `LICENSE` in this repo for the full text and the required attribution. Do not use it to clone a voice you do not have permission to clone.