notmax123 commited on
Commit
0710da8
·
verified ·
1 Parent(s): ab1991e

Correct what-trains section; add caveats on training length and alpha/r

Browse files
Files changed (1) hide show
  1. README.md +24 -5
README.md CHANGED
@@ -82,11 +82,14 @@ English orthography — Hebrew `י` phonemized as `j` was read as the English le
82
  to the mean of the symbol's original BPE pieces. The output vocabulary is
83
  untouched — these tokens are never predicted, only read.
84
 
85
- **What trains.** LoRA r=32, α=16 on `attention` + `mlp` of the slow transformer,
86
- plus the IPA embedding table. The acoustic/fast stack is frozen, so voice cloning
87
- and the codec side are exactly the base model's. The residual-codebook loss is
88
- down-weighted to 0.3 (Qwen3-TTS's sub-talker coefficient) so the gradient stays on
89
- the text→semantic mapping.
 
 
 
90
 
91
  **Training.** 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference-
92
  conditioned on a same-speaker utterance 80% of the time so training prompts match
@@ -126,6 +129,22 @@ applies to any S2-Pro fine-tune, Hebrew or not.
126
  - `04_yod_BASE.wav` / `05_yod_LORA.wav` — the `י` → English *jay* failure, before and after atomic IPA tokens
127
  - `07_clone_LORA_ranlevi.wav` — voice clone from a real Hebrew speaker reference
128
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
129
  ## Known limitations
130
 
131
  - **Emotion tags (`[whisper]`, `[excited]`, …) do not work** — and this is not a
 
82
  to the mean of the symbol's original BPE pieces. The output vocabulary is
83
  untouched — these tokens are never predicted, only read.
84
 
85
+ **What trains.** LoRA r=32, α=16 on `attention` + `mlp`, plus the IPA embedding
86
+ table 66.9M parameters total: 60.2M in the slow transformer, 6.7M in the fast
87
+ transformer, 0.03M IPA embeddings. Frozen are the direct interfaces to codebook
88
+ space `fast_embeddings`, `fast_output`, and the tied slow embeddings/output —
89
+ which is what keeps timbre close to the base model. Note α/r = 0.5, not the usual
90
+ 2.0; see *Caveats*. The residual-codebook loss is down-weighted to 0.3
91
+ (Qwen3-TTS's sub-talker coefficient) so the gradient stays on the text→semantic
92
+ mapping.
93
 
94
  **Training.** 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference-
95
  conditioned on a same-speaker utterance 80% of the time so training prompts match
 
129
  - `04_yod_BASE.wav` / `05_yod_LORA.wav` — the `י` → English *jay* failure, before and after atomic IPA tokens
130
  - `07_clone_LORA_ranlevi.wav` — voice clone from a real Hebrew speaker reference
131
 
132
+ ## Caveats — this checkpoint is early, not final
133
+
134
+ - **Undertrained, and stopped by hand.** 2,200 optimizer steps ≈ 53k utterances
135
+ seen, under 20% of one epoch over the 279k-row set. Train loss was still falling
136
+ (3.66 → 2.70 base CE) and val loss was still improving monotonically at every
137
+ checkpoint (2.934 → 2.844 → 2.820 → 2.807). Nothing had plateaued; the run was
138
+ simply halted.
139
+ - **α/r = 0.5 is a workaround for a bug that no longer exists.** The unusual
140
+ scaling was chosen empirically because at α=64 the delta destroyed free-running
141
+ generation — which we now know was the embedding-scale bug above, not the LoRA
142
+ strength. That rationale is obsolete post-fix, and the standard α = 2r was never
143
+ re-tried. It may well be better.
144
+ - The pitch homogenization noted below is the symptom you would expect from
145
+ putting LoRA on the fast transformer at all. Freezing `fast_layers` entirely is
146
+ the obvious next experiment.
147
+
148
  ## Known limitations
149
 
150
  - **Emotion tags (`[whisper]`, `[excited]`, …) do not work** — and this is not a