Correct what-trains section; add caveats on training length and alpha/r
Browse files
README.md
CHANGED
|
@@ -82,11 +82,14 @@ English orthography — Hebrew `י` phonemized as `j` was read as the English le
|
|
| 82 |
to the mean of the symbol's original BPE pieces. The output vocabulary is
|
| 83 |
untouched — these tokens are never predicted, only read.
|
| 84 |
|
| 85 |
-
**What trains.** LoRA r=32, α=16 on `attention` + `mlp`
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
the
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
**Training.** 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference-
|
| 92 |
conditioned on a same-speaker utterance 80% of the time so training prompts match
|
|
@@ -126,6 +129,22 @@ applies to any S2-Pro fine-tune, Hebrew or not.
|
|
| 126 |
- `04_yod_BASE.wav` / `05_yod_LORA.wav` — the `י` → English *jay* failure, before and after atomic IPA tokens
|
| 127 |
- `07_clone_LORA_ranlevi.wav` — voice clone from a real Hebrew speaker reference
|
| 128 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
## Known limitations
|
| 130 |
|
| 131 |
- **Emotion tags (`[whisper]`, `[excited]`, …) do not work** — and this is not a
|
|
|
|
| 82 |
to the mean of the symbol's original BPE pieces. The output vocabulary is
|
| 83 |
untouched — these tokens are never predicted, only read.
|
| 84 |
|
| 85 |
+
**What trains.** LoRA r=32, α=16 on `attention` + `mlp`, plus the IPA embedding
|
| 86 |
+
table — 66.9M parameters total: 60.2M in the slow transformer, 6.7M in the fast
|
| 87 |
+
transformer, 0.03M IPA embeddings. Frozen are the direct interfaces to codebook
|
| 88 |
+
space — `fast_embeddings`, `fast_output`, and the tied slow embeddings/output —
|
| 89 |
+
which is what keeps timbre close to the base model. Note α/r = 0.5, not the usual
|
| 90 |
+
2.0; see *Caveats*. The residual-codebook loss is down-weighted to 0.3
|
| 91 |
+
(Qwen3-TTS's sub-talker coefficient) so the gradient stays on the text→semantic
|
| 92 |
+
mapping.
|
| 93 |
|
| 94 |
**Training.** 279,476 Hebrew utterances (~10 speakers, WER ≤ 0.1), reference-
|
| 95 |
conditioned on a same-speaker utterance 80% of the time so training prompts match
|
|
|
|
| 129 |
- `04_yod_BASE.wav` / `05_yod_LORA.wav` — the `י` → English *jay* failure, before and after atomic IPA tokens
|
| 130 |
- `07_clone_LORA_ranlevi.wav` — voice clone from a real Hebrew speaker reference
|
| 131 |
|
| 132 |
+
## Caveats — this checkpoint is early, not final
|
| 133 |
+
|
| 134 |
+
- **Undertrained, and stopped by hand.** 2,200 optimizer steps ≈ 53k utterances
|
| 135 |
+
seen, under 20% of one epoch over the 279k-row set. Train loss was still falling
|
| 136 |
+
(3.66 → 2.70 base CE) and val loss was still improving monotonically at every
|
| 137 |
+
checkpoint (2.934 → 2.844 → 2.820 → 2.807). Nothing had plateaued; the run was
|
| 138 |
+
simply halted.
|
| 139 |
+
- **α/r = 0.5 is a workaround for a bug that no longer exists.** The unusual
|
| 140 |
+
scaling was chosen empirically because at α=64 the delta destroyed free-running
|
| 141 |
+
generation — which we now know was the embedding-scale bug above, not the LoRA
|
| 142 |
+
strength. That rationale is obsolete post-fix, and the standard α = 2r was never
|
| 143 |
+
re-tried. It may well be better.
|
| 144 |
+
- The pitch homogenization noted below is the symptom you would expect from
|
| 145 |
+
putting LoRA on the fast transformer at all. Freezing `fast_layers` entirely is
|
| 146 |
+
the obvious next experiment.
|
| 147 |
+
|
| 148 |
## Known limitations
|
| 149 |
|
| 150 |
- **Emotion tags (`[whisper]`, `[excited]`, …) do not work** — and this is not a
|