Add a stock-model sample so "supports Hebrew out of the box" can be checked by ear

0de4612 verified 5 days ago

8.85 kB

license: other
license_name: fish-audio-research-license
license_link: https://github.com/maxmelichov/fish-speech/blob/main/LICENSE
language:
  - he
  - en
tags:
  - text-to-speech
  - tts
  - hebrew
  - fish-speech
  - s2-pro
  - lora
  - voice-cloning
base_model: fishaudio/s2-pro
library_name: fish-speech
pipeline_tag: text-to-speech

Built with Fish Audio.

Fish Audio S2-Pro — Hebrew (LoRA + atomic IPA tokens)

A Hebrew adapter for fishaudio/s2-pro. It keeps the base model's multilingual ability and voice cloning intact, and adds native Hebrew synthesis driven by IPA rather than nikud.

This repo contains only the adapter (~67M parameters) plus the extended tokenizer. You still need the S2-Pro base weights and codec.

What's here

File	What it is
`hebrew_lora_step2200.safetensors`	LoRA deltas + the trained `ipa_embeddings` table (67M params, bf16)
`hebrew_lora_step2200.ckpt`	Same weights as a Lightning checkpoint, with optimizer state — use this to resume training
`config.json`	S2-Pro config extended with `num_ipa_tokens: 26`, `ipa_token_start: 155774`
`ipa_token_map.json`	IPA symbol → atomic token (e.g. `ʃ` → `<ipa_u0283>`)
`ipa_embeddings.pt`	Initial IPA embedding table (mean of the symbol's BPE pieces); the trained one lives in the adapter
`tokenizer/`	S2-Pro tokenizer extended 155,774 → 155,800 tokens
`samples/`	Generated audio (see below)

Quick start

Code lives in the fork the adapter was trained with:

git clone https://github.com/maxmelichov/fish-speech
cd fish-speech && uv sync --python 3.12 --extra cu129
pip install renikud-plus                 # Hebrew grapheme-to-phoneme

bash tools/hebrew/setup_hebrew.sh        # base weights + this adapter + IPA checkpoint

python tools/hebrew/infer_hebrew.py \
    --text "שלום, מה שלומך היום?" \
    --lora-checkpoint checkpoints/hebrew/hebrew_lora_step2200.safetensors \
    --output out.wav

Add --ref-audio my_voice.wav --ref-text "..." to clone a voice.

infer_hebrew.py runs plain unvocalized Hebrew through RenikudPlus G2P, maps the IPA to the atomic tokens, and chunks long inputs on sentence boundaries. --lora-scale scales the delta (0.0 = pure base model) if you want to dial the adaptation down.

Fine-tuning on your own Hebrew data is one command — a directory per speaker of *.wav plus sibling .lab transcripts:

AUDIO_ROOT=my_audio tools/hebrew/run_hebrew_pipeline.sh

See tools/hebrew/README.md for the full guide.

How it works

Atomic IPA tokens. S2-Pro's BPE splits IPA into pieces that collide with English orthography — Hebrew י phonemized as j was read as the English letter jay. So each of the 26 Hebrew IPA symbols gets a dedicated input-only token (<ipa_j>, <ipa_u0283>, …) in a separate trainable nn.Embedding, initialized to the mean of the symbol's original BPE pieces. The output vocabulary is untouched — these tokens are never predicted, only read.

What trains. LoRA r=32, α=16 on attention + mlp, plus the IPA embedding table — 66.9M parameters total: 60.2M in the slow transformer, 6.7M in the fast transformer, 0.03M IPA embeddings. Frozen are the direct interfaces to codebook space — fast_embeddings, fast_output, and the tied slow embeddings/output — which is what keeps timbre close to the base model. Note α/r = 0.5, not the usual 2.0; see Caveats. The residual-codebook loss is down-weighted to 0.3 (Qwen3-TTS's sub-talker coefficient) so the gradient stays on the text→semantic mapping.

Training. 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference- conditioned on a same-speaker utterance 80% of the time so training prompts match the exact generate_long() inference format. bf16, lr 5e-5 constant with 100-step warmup, effective batch 12, 2200 optimizer steps.

Upstream bug fixed along the way

S2-Pro sets scale_codebook_embeddings=True. At inference, forward_generate() divides semantic-position embeddings by sqrt(num_codebooks + 1) = 3.317; the training path in embed() did not. Every fine-tune therefore learned against embeddings 3.3× larger than the ones it would see at generation time. Teacher- forced CE looked fine while free-running generation collapsed after the first word — the classic symptom in fishaudio/fish-speech issues #1136 (Japanese gibberish), #682 (Hindi noise) and #814.

Five Hebrew runs collapsed the same way before this was found. After the fix (train and inference embeddings verified bit-identical):

	sample RMS	energy decay over the utterance
before	0.008 – 0.022	0.07×
after	0.171 – 0.205	1.02×
base model reference	0.181	—

The fix is in fish_speech/models/text2semantic/llama.py in the fork above and applies to any S2-Pro fine-tune, Hebrew or not.

Samples

00_base_out_of_the_box.wav — stock fishaudio/s2-pro, no adapter, no G2P — plain Hebrew script straight in. S2-Pro is multilingual and does produce Hebrew-shaped speech, but it isn't accurate: this sample of "שלום, מה שלומך היום?" ("hello, how are you today?") comes out as "סלום מהשלום חיום" — שלום → סלום, שלומך garbled into משלום. Not a cherry-pick: an 8-seed sweep on a different sentence in this fork's eval found the same failure every time, and a stock-model WER of 0.383 across 11 sentences, worse than real human speech scores on the same metric. This is the gap the adapter below closes.

The rest of samples/ is generated with this adapter:

01_podcast_2hosts_63s.wav — 63s two-host Hebrew conversation, cloned voices
03_longform_15s.wav — multi-sentence long-form
04_yod_BASE.wav / 05_yod_LORA.wav — the י → English jay failure, before and after atomic IPA tokens
07_clone_LORA_ranlevi.wav — voice clone from a real Hebrew speaker reference

Caveats — this checkpoint is early, not final

Undertrained, and stopped by hand. 2,200 optimizer steps ≈ 53k utterances seen, under 20% of one epoch over the 279k-row set. Train loss was still falling (3.66 → 2.70 base CE) and val loss was still improving monotonically at every checkpoint (2.934 → 2.844 → 2.820 → 2.807). Nothing had plateaued; the run was simply halted.
α/r = 0.5 is a workaround for a bug that no longer exists. The unusual scaling was chosen empirically because at α=64 the delta destroyed free-running generation — which we now know was the embedding-scale bug above, not the LoRA strength. That rationale is obsolete post-fix, and the standard α = 2r was never re-tried. It may well be better.
The pitch homogenization noted below is the symptom you would expect from putting LoRA on the fast transformer at all. Freezing fast_layers entirely is the obvious next experiment.

Known limitations

Emotion tags ([whisper], [excited], …) do not work — and this is not a regression from the LoRA. Measured on the base model in English: plain / whisper / shouting produced RMS 0.0655 / 0.0652 / 0.0689, i.e. no response at all. The released S2-Pro weights simply lack the tag alignment.
Pitch is not cloned. Timbre transfers well (4/4 by ear), but neither base nor LoRA reproduces the reference's F0 (base mean |err| 23 Hz, LoRA 20 Hz). The LoRA homogenizes pitch somewhat: spread across speakers drops from 66 Hz to 26 Hz.
Trained on read/broadcast-style Hebrew; conversational and heavily accented speech are out of distribution.
Hebrew input must go through G2P. Feeding nikud or bare Hebrew script directly to the model is out of distribution — use infer_hebrew.py, which handles it.

License

S2-Pro (the base model this adapter is trained on) is released under the Fish Audio Research License — not CC BY-NC-SA 4.0, correcting an earlier version of this card. This adapter is a Derivative Work under that license and inherits its terms: research and non-commercial use only (personal use, evaluation, academic work); any commercial use requires a separate license directly from Fish Audio (business@fish.audio). See NOTICE and LICENSE in this repo for the full text and the required attribution.

Do not use it to clone a voice you do not have permission to clone.