Hebrew LoRA (step 2200) + atomic IPA tokens for S2-Pro

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+samples/01_podcast_2hosts_63s.wav filter=lfs diff=lfs merge=lfs -text
+samples/03_longform_15s.wav filter=lfs diff=lfs merge=lfs -text
+samples/04_yod_BASE.wav filter=lfs diff=lfs merge=lfs -text
+samples/05_yod_LORA.wav filter=lfs diff=lfs merge=lfs -text
+samples/07_clone_LORA_ranlevi.wav filter=lfs diff=lfs merge=lfs -text
+tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,5 +1,146 @@
 ---
-license: mit
+license: cc-by-nc-sa-4.0
+language:
+- he
+- en
+tags:
+- text-to-speech
+- tts
+- hebrew
+- fish-speech
+- s2-pro
+- lora
+- voice-cloning
+base_model: fishaudio/s2-pro
+library_name: fish-speech
+pipeline_tag: text-to-speech
 ---
 
 ![image](https://cdn-uploads.huggingface.co/production/uploads/63453ab89ad67b3d069effdf/Kq3sPWLcMbdmMHmqZWqPs.png)
+
+# Fish Audio S2-Pro — Hebrew (LoRA + atomic IPA tokens)
+
+A Hebrew adapter for [`fishaudio/s2-pro`](https://huggingface.co/fishaudio/s2-pro).
+It keeps the base model's multilingual ability and voice cloning intact, and adds
+native Hebrew synthesis driven by **IPA** rather than nikud.
+
+This repo contains **only the adapter** (~67M parameters) plus the extended
+tokenizer. You still need the S2-Pro base weights and codec.
+
+## What's here
+
+| File | What it is |
+|---|---|
+| `hebrew_lora_step2200.safetensors` | LoRA deltas + the trained `ipa_embeddings` table (67M params, bf16) |
+| `hebrew_lora_step2200.ckpt` | Same weights as a Lightning checkpoint, with optimizer state — use this to resume training |
+| `config.json` | S2-Pro config extended with `num_ipa_tokens: 26`, `ipa_token_start: 155774` |
+| `ipa_token_map.json` | IPA symbol → atomic token (e.g. `ʃ` → `<ipa_u0283>`) |
+| `ipa_embeddings.pt` | Initial IPA embedding table (mean of the symbol's BPE pieces); the trained one lives in the adapter |
+| `tokenizer/` | S2-Pro tokenizer extended 155,774 → 155,800 tokens |
+| `samples/` | Generated audio (see below) |
+
+## Quick start
+
+Code lives in the fork the adapter was trained with:
+
+```bash
+git clone https://github.com/maxmelichov/fish-speech
+cd fish-speech && uv sync --python 3.12 --extra cu129
+pip install renikud-plus           # Hebrew grapheme-to-phoneme
+
+hf download fishaudio/s2-pro --local-dir checkpoints/s2-pro
+hf download notmax123/Fish-Audio-S2-Pro-He --local-dir checkpoints/he
+```
+
+Build the IPA-extended base checkpoint (symlinks the S2-Pro weights, drops the
+extended tokenizer and config on top), then synthesize:
+
+```bash
+python tools/hebrew/build_ipa_checkpoint.py \
+    --base checkpoints/s2-pro --output checkpoints/s2-pro-he-ipa
+
+python tools/hebrew/infer_hebrew.py \
+    --text "שלום, מה שלומך היום?" \
+    --base-checkpoint checkpoints/s2-pro-he-ipa \
+    --lora-checkpoint checkpoints/he/hebrew_lora_step2200.ckpt \
+    --lora-config r_32_alpha_16_core \
+    --ref-audio my_voice.wav --ref-text "..." \
+    --output out.wav
+```
+
+`infer_hebrew.py` runs plain unvocalized Hebrew through RenikudPlus G2P, maps the
+IPA to the atomic tokens, and chunks long inputs on sentence boundaries.
+`--lora-scale` scales the delta (0.0 = pure base model) if you want to dial the
+adaptation down.
+
+## How it works
+
+**Atomic IPA tokens.** S2-Pro's BPE splits IPA into pieces that collide with
+English orthography — Hebrew `י` phonemized as `j` was read as the English letter
+*jay*. So each of the 26 Hebrew IPA symbols gets a dedicated input-only token
+(`<ipa_j>`, `<ipa_u0283>`, …) in a separate trainable `nn.Embedding`, initialized
+to the mean of the symbol's original BPE pieces. The output vocabulary is
+untouched — these tokens are never predicted, only read.
+
+**What trains.** LoRA r=32, α=16 on `attention` + `mlp` of the slow transformer,
+plus the IPA embedding table. The acoustic/fast stack is frozen, so voice cloning
+and the codec side are exactly the base model's. The residual-codebook loss is
+down-weighted to 0.3 (Qwen3-TTS's sub-talker coefficient) so the gradient stays on
+the text→semantic mapping.
+
+**Training.** 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference-
+conditioned on a same-speaker utterance 80% of the time so training prompts match
+the exact `generate_long()` inference format. bf16, lr 5e-5 constant with 100-step
+warmup, effective batch 12, 2200 optimizer steps.
+
+## Upstream bug fixed along the way
+
+S2-Pro sets `scale_codebook_embeddings=True`. At inference, `forward_generate()`
+divides semantic-position embeddings by `sqrt(num_codebooks + 1)` = 3.317; the
+training path in `embed()` did **not**. Every fine-tune therefore learned against
+embeddings 3.3× larger than the ones it would see at generation time. Teacher-
+forced CE looked fine while free-running generation collapsed after the first
+word — the classic symptom in fishaudio/fish-speech issues
+[#1136](https://github.com/fishaudio/fish-speech/issues/1136) (Japanese gibberish),
+[#682](https://github.com/fishaudio/fish-speech/issues/682) (Hindi noise) and
+[#814](https://github.com/fishaudio/fish-speech/issues/814).
+
+Five Hebrew runs collapsed the same way before this was found. After the fix
+(train and inference embeddings verified bit-identical):
+
+| | sample RMS | energy decay over the utterance |
+|---|---|---|
+| before | 0.008 – 0.022 | 0.07× |
+| after | 0.171 – 0.205 | 1.02× |
+| base model reference | 0.181 | — |
+
+The fix is in `fish_speech/models/text2semantic/llama.py` in the fork above and
+applies to any S2-Pro fine-tune, Hebrew or not.
+
+## Samples
+
+`samples/` contains, all generated with this adapter:
+
+- `01_podcast_2hosts_63s.wav` — 63s two-host Hebrew conversation, cloned voices
+- `03_longform_15s.wav` — multi-sentence long-form
+- `04_yod_BASE.wav` / `05_yod_LORA.wav` — the `י` → English *jay* failure, before and after atomic IPA tokens
+- `07_clone_LORA_ranlevi.wav` — voice clone from a real Hebrew speaker reference
+
+## Known limitations
+
+- **Emotion tags (`[whisper]`, `[excited]`, …) do not work** — and this is not a
+  regression from the LoRA. Measured on the *base* model in *English*: plain /
+  whisper / shouting produced RMS 0.0655 / 0.0652 / 0.0689, i.e. no response at
+  all. The released S2-Pro weights simply lack the tag alignment.
+- **Pitch is not cloned.** Timbre transfers well (4/4 by ear), but neither base nor
+  LoRA reproduces the reference's F0 (base mean |err| 23 Hz, LoRA 20 Hz). The LoRA
+  homogenizes pitch somewhat: spread across speakers drops from 66 Hz to 26 Hz.
+- Trained on read/broadcast-style Hebrew; conversational and heavily accented
+  speech are out of distribution.
+- Hebrew input must go through G2P. Feeding nikud or bare Hebrew script directly
+  to the model is out of distribution — use `infer_hebrew.py`, which handles it.
+
+## License
+
+Inherits the base model's license (CC BY-NC-SA 4.0). Non-commercial.
+Do not use it to clone a voice you do not have permission to clone.

config.json

@@ -0,0 +1,72 @@
+{
+  "audio_decoder_config": {
+    "attention_o_bias": false,
+    "attention_qk_norm": false,
+    "attention_qkv_bias": false,
+    "audio_hidden_dim": 5120,
+    "dim": 2560,
+    "dropout": 0.0,
+    "head_dim": 128,
+    "initializer_range": 0.01976423537605237,
+    "intermediate_size": 9728,
+    "max_seq_len": 11,
+    "model_type": "fish_qwen3_audio_decoder",
+    "moe_intermediate_size": 768,
+    "n_head": 32,
+    "n_layer": 4,
+    "n_local_heads": 8,
+    "norm_eps": 1e-06,
+    "norm_topk_prob": true,
+    "num_codebooks": 10,
+    "num_experts": 1,
+    "num_experts_per_tok": 1,
+    "rope_base": 1000000,
+    "router_gamma": 0.001,
+    "text_dim": 2560,
+    "tie_word_embeddings": false,
+    "use_aux_loss_free": false,
+    "use_bfloat16": false,
+    "use_gradient_checkpointing": true,
+    "use_moe": false,
+    "vocab_size": 4096
+  },
+  "audio_pad_token_id": 151677,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "model_type": "fish_qwen3_omni",
+  "pad_token_id": 151669,
+  "semantic_end_token_id": 155773,
+  "semantic_start_token_id": 151678,
+  "text_config": {
+    "attention_o_bias": false,
+    "attention_qk_norm": true,
+    "attention_qkv_bias": false,
+    "audio_hidden_dim": 5120,
+    "dim": 2560,
+    "dropout": 0.0,
+    "head_dim": 128,
+    "initializer_range": 0.01976423537605237,
+    "intermediate_size": 9728,
+    "max_seq_len": 32768,
+    "model_type": "fish_qwen3",
+    "moe_intermediate_size": 768,
+    "n_head": 32,
+    "n_layer": 36,
+    "n_local_heads": 8,
+    "norm_eps": 1e-06,
+    "norm_topk_prob": true,
+    "num_experts": 1,
+    "num_experts_per_tok": 1,
+    "rope_base": 1000000,
+    "router_gamma": 0.001,
+    "tie_word_embeddings": true,
+    "use_aux_loss_free": false,
+    "use_bfloat16": false,
+    "use_gradient_checkpointing": true,
+    "use_moe": false,
+    "vocab_size": 155776
+  },
+  "transformers_version": "4.57.1",
+  "num_ipa_tokens": 26,
+  "ipa_token_start": 155774
+}

hebrew_lora_step2200.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b2f69aa1b79c1a04613ead65c77995045efdef2ce417ce55c10df7b834777d9
+size 402000398

hebrew_lora_step2200.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dce55d0b3d998f0cf407ce7b2833a34738b7527664c10a2f3ba044e65ae95015
+size 133871024

ipa_embeddings.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8af4a5ba9359e1ea6ff5d750e7274795c8d9974fe9bce9061ffc229ff7ab9534
+size 267866

ipa_token_map.json

@@ -0,0 +1,28 @@
+{
+  "a": "<ipa_a>",
+  "b": "<ipa_b>",
+  "d": "<ipa_d>",
+  "e": "<ipa_e>",
+  "f": "<ipa_f>",
+  "h": "<ipa_h>",
+  "i": "<ipa_i>",
+  "j": "<ipa_j>",
+  "k": "<ipa_k>",
+  "l": "<ipa_l>",
+  "m": "<ipa_m>",
+  "n": "<ipa_n>",
+  "o": "<ipa_o>",
+  "p": "<ipa_p>",
+  "s": "<ipa_s>",
+  "t": "<ipa_t>",
+  "u": "<ipa_u>",
+  "v": "<ipa_v>",
+  "w": "<ipa_w>",
+  "z": "<ipa_z>",
+  "ɡ": "<ipa_u0261>",
+  "ʁ": "<ipa_u0281>",
+  "ʃ": "<ipa_u0283>",
+  "ʔ": "<ipa_u0294>",
+  "ˈ": "<ipa_u02C8>",
+  "χ": "<ipa_u03C7>"
+}

samples/01_podcast_2hosts_63s.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f27a0f3a57965085bc831289f6933081e9eebf351480e42ca6f4679588de704
+size 5570220

samples/03_longform_15s.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c58aabf367ae1e6e0076d04b96d0ac94e7937faf615c061ef412c80a20f26d65
+size 1335340

samples/04_yod_BASE.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9266d78bcb116f14cb97da0f6fb36b413bd12bcdcb088f317b5827c14579c415
+size 426028

samples/05_yod_LORA.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2046bca2cc8206bbc262a1ba5b43da75fdc846518f4a7a7cc06c5fdbb0eb547
+size 430124

samples/07_clone_LORA_ranlevi.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92da7b28077461ef5e630de658d4c7dd8be624cadaca685e6a746ae9a3c878a2
+size 421932

tokenizer/chat_template.jinja

@@ -0,0 +1,85 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set content = message.content %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in message.content %}
+                {%- set content = message.content.split('</think>')[-1].lstrip('\n') %}
+                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31c2fb72254c63945fef9081a0e68a0ba752dc685cff5a44efabab28e23a9f29
+size 12222680