README.md

---
license: other
license_name: fish-audio-research-license
license_link: https://github.com/maxmelichov/fish-speech/blob/main/LICENSE
language:
- he
- en
tags:
- text-to-speech
- tts
- hebrew
- fish-speech
- s2-pro
- lora
- voice-cloning
base_model: fishaudio/s2-pro
library_name: fish-speech
pipeline_tag: text-to-speech
---

<img src="fih.png" alt="Fish Audio S2-Pro Hebrew" width="420">

Built with Fish Audio.

# Fish Audio S2-Pro — Hebrew (LoRA + atomic IPA tokens)

A Hebrew adapter for [`fishaudio/s2-pro`](https://huggingface.co/fishaudio/s2-pro).
It keeps the base model's multilingual ability and voice cloning intact, and adds
native Hebrew synthesis driven by **IPA** rather than nikud.

This repo contains **only the adapter** (~67M parameters) plus the extended
tokenizer. You still need the S2-Pro base weights and codec.

## What's here

| File | What it is |
|---|---|
| `hebrew_lora_step2200.safetensors` | LoRA deltas + the trained `ipa_embeddings` table (67M params, bf16) |
| `hebrew_lora_step2200.ckpt` | Same weights as a Lightning checkpoint, with optimizer state — use this to resume training |
| `config.json` | S2-Pro config extended with `num_ipa_tokens: 26`, `ipa_token_start: 155774` |
| `ipa_token_map.json` | IPA symbol → atomic token (e.g. `ʃ` → `<ipa_u0283>`) |
| `ipa_embeddings.pt` | Initial IPA embedding table (mean of the symbol's BPE pieces); the trained one lives in the adapter |
| `tokenizer/` | S2-Pro tokenizer extended 155,774 → 155,800 tokens |
| `samples/` | Generated audio (see below) |

## Quick start

Code lives in the fork the adapter was trained with:

```bash
git clone https://github.com/maxmelichov/fish-speech
cd fish-speech && uv sync --python 3.12 --extra cu129
pip install renikud-plus                 # Hebrew grapheme-to-phoneme

bash tools/hebrew/setup_hebrew.sh        # base weights + this adapter + IPA checkpoint

python tools/hebrew/infer_hebrew.py \
    --text "שלום, מה שלומך היום?" \
    --lora-checkpoint checkpoints/hebrew/hebrew_lora_step2200.safetensors \
    --output out.wav
```

Add `--ref-audio my_voice.wav --ref-text "..."` to clone a voice.

`infer_hebrew.py` runs plain unvocalized Hebrew through
[RenikudPlus](https://github.com/maxmelichov/RenikudPlus) G2P, maps the
IPA to the atomic tokens, and chunks long inputs on sentence boundaries.
`--lora-scale` scales the delta (0.0 = pure base model) if you want to dial the
adaptation down.

**Fine-tuning on your own Hebrew data** is one command — a directory per speaker
of `*.wav` plus sibling `.lab` transcripts:

```bash
AUDIO_ROOT=my_audio tools/hebrew/run_hebrew_pipeline.sh
```

See [`tools/hebrew/README.md`](https://github.com/maxmelichov/fish-speech/blob/main/tools/hebrew/README.md)
for the full guide.

## How it works

**Atomic IPA tokens.** S2-Pro's BPE splits IPA into pieces that collide with
English orthography — Hebrew `י` phonemized as `j` was read as the English letter
*jay*. So each of the 26 Hebrew IPA symbols gets a dedicated input-only token
(`<ipa_j>`, `<ipa_u0283>`, …) in a separate trainable `nn.Embedding`, initialized
to the mean of the symbol's original BPE pieces. The output vocabulary is
untouched — these tokens are never predicted, only read.

**What trains.** LoRA r=32, α=16 on `attention` + `mlp`, plus the IPA embedding
table — 66.9M parameters total: 60.2M in the slow transformer, 6.7M in the fast
transformer, 0.03M IPA embeddings. Frozen are the direct interfaces to codebook
space — `fast_embeddings`, `fast_output`, and the tied slow embeddings/output —
which is what keeps timbre close to the base model. Note α/r = 0.5, not the usual
2.0; see *Caveats*. The residual-codebook loss is down-weighted to 0.3
(Qwen3-TTS's sub-talker coefficient) so the gradient stays on the text→semantic
mapping.

**Training.** 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference-
conditioned on a same-speaker utterance 80% of the time so training prompts match
the exact `generate_long()` inference format. bf16, lr 5e-5 constant with 100-step
warmup, effective batch 12, 2200 optimizer steps.

## Upstream bug fixed along the way

S2-Pro sets `scale_codebook_embeddings=True`. At inference, `forward_generate()`
divides semantic-position embeddings by `sqrt(num_codebooks + 1)` = 3.317; the
training path in `embed()` did **not**. Every fine-tune therefore learned against
embeddings 3.3× larger than the ones it would see at generation time. Teacher-
forced CE looked fine while free-running generation collapsed after the first
word — the classic symptom in fishaudio/fish-speech issues
[#1136](https://github.com/fishaudio/fish-speech/issues/1136) (Japanese gibberish),
[#682](https://github.com/fishaudio/fish-speech/issues/682) (Hindi noise) and
[#814](https://github.com/fishaudio/fish-speech/issues/814).

Five Hebrew runs collapsed the same way before this was found. After the fix
(train and inference embeddings verified bit-identical):

| | sample RMS | energy decay over the utterance |
|---|---|---|
| before | 0.008 – 0.022 | 0.07× |
| after | 0.171 – 0.205 | 1.02× |
| base model reference | 0.181 | — |

The fix is in `fish_speech/models/text2semantic/llama.py` in the fork above and
applies to any S2-Pro fine-tune, Hebrew or not.

## Samples

- `00_base_out_of_the_box.wav` — **stock `fishaudio/s2-pro`, no adapter, no
  G2P** — plain Hebrew script straight in. S2-Pro is multilingual and does
  produce Hebrew-*shaped* speech, but it isn't accurate: this sample of
  "שלום, מה שלומך היום?" ("hello, how are you today?") comes out as "סלום
  מהשלום חיום" — שלום → סלום, שלומך garbled into משלום. Not a cherry-pick: an
  8-seed sweep on a different sentence in this fork's eval found the same
  failure every time, and a stock-model WER of 0.383 across 11 sentences,
  worse than real human speech scores on the same metric. This is the gap
  the adapter below closes.

The rest of `samples/` is generated **with this adapter**:

- `01_podcast_2hosts_63s.wav` — 63s two-host Hebrew conversation, cloned voices
- `03_longform_15s.wav` — multi-sentence long-form
- `04_yod_BASE.wav` / `05_yod_LORA.wav` — the `י` → English *jay* failure, before and after atomic IPA tokens
- `07_clone_LORA_ranlevi.wav` — voice clone from a real Hebrew speaker reference

## Caveats — this checkpoint is early, not final

- **Undertrained, and stopped by hand.** 2,200 optimizer steps ≈ 53k utterances
  seen, under 20% of one epoch over the 279k-row set. Train loss was still falling
  (3.66 → 2.70 base CE) and val loss was still improving monotonically at every
  checkpoint (2.934 → 2.844 → 2.820 → 2.807). Nothing had plateaued; the run was
  simply halted.
- **α/r = 0.5 is a workaround for a bug that no longer exists.** The unusual
  scaling was chosen empirically because at α=64 the delta destroyed free-running
  generation — which we now know was the embedding-scale bug above, not the LoRA
  strength. That rationale is obsolete post-fix, and the standard α = 2r was never
  re-tried. It may well be better.
- The pitch homogenization noted below is the symptom you would expect from
  putting LoRA on the fast transformer at all. Freezing `fast_layers` entirely is
  the obvious next experiment.

## Known limitations

- **Emotion tags (`[whisper]`, `[excited]`, …) do not work** — and this is not a
  regression from the LoRA. Measured on the *base* model in *English*: plain /
  whisper / shouting produced RMS 0.0655 / 0.0652 / 0.0689, i.e. no response at
  all. The released S2-Pro weights simply lack the tag alignment.
- **Pitch is not cloned.** Timbre transfers well (4/4 by ear), but neither base nor
  LoRA reproduces the reference's F0 (base mean |err| 23 Hz, LoRA 20 Hz). The LoRA
  homogenizes pitch somewhat: spread across speakers drops from 66 Hz to 26 Hz.
- Trained on read/broadcast-style Hebrew; conversational and heavily accented
  speech are out of distribution.
- Hebrew input must go through G2P. Feeding nikud or bare Hebrew script directly
  to the model is out of distribution — use `infer_hebrew.py`, which handles it.

## License

S2-Pro (the base model this adapter is trained on) is released under the
**[Fish Audio Research License](https://github.com/maxmelichov/fish-speech/blob/main/LICENSE)**
— *not* CC BY-NC-SA 4.0, correcting an earlier version of this card. This
adapter is a Derivative Work under that license and inherits its terms:
research and non-commercial use only (personal use, evaluation, academic
work); any commercial use requires a separate license directly from Fish
Audio (business@fish.audio). See `NOTICE` and `LICENSE` in this repo for the
full text and the required attribution.

Do not use it to clone a voice you do not have permission to clone.