README.md

# CosyVoice2 Hebrew Fine-tune

Private repository containing CosyVoice2 fine-tuned for Hebrew speech synthesis.

## Repository Structure

```
├── cosyvoice/                    # CosyVoice2 library code
├── pretrained_models/
│   └── CosyVoice2-0.5B/         # Base pretrained model
│       ├── llm.pt
│       ├── flow.pt
│       ├── hift.pt
│       └── CosyVoice-BlankEN/model.safetensors
├── examples/hebrew/cosyvoice2/
│   ├── conf/cosyvoice2.yaml      # Training config
│   ├── run.sh                    # Full training pipeline
│   ├── path.sh                   # Environment setup
│   ├── model_hebrew/             # Hebrew model files (for inference)
│   │   ├── llm.pt, flow.pt, hift.pt, spk2info.pt
│   │   └── *.onnx (tokenizer, speaker embedding)
│   └── exp/cosyvoice2_hebrew/llm/torch_ddp/
│       ├── epoch_3_whole.pt      # Best by loss (2.698, acc 0.280)
│       └── epoch_4_whole.pt      # Best by accuracy (2.706, acc 0.280)
├── third_party/Matcha-TTS/       # Required dependency
└── tools/                        # Data preparation tools
```

## Training Summary

- **Base model**: CosyVoice2-0.5B
- **Task**: Hebrew SFT (Supervised Fine-Tuning) on LLM component
- **Optimizer**: Adam, lr=1e-5, constant LR with 1000 warmup steps
- **Batch**: dynamic batching, max_frames_in_batch=2000, accum_grad=4
- **Mixed precision**: bf16
- **Training ran**: 9 epochs (~50k steps), best around epoch 3-4

### Loss Curve (key checkpoints)
| Checkpoint | Loss | Accuracy | Step |
|---|---|---|---|
| epoch_0_whole | 2.795 | 0.265 | 5,957 |
| epoch_1_whole | 2.735 | 0.274 | 11,911 |
| epoch_2_whole | 2.707 | 0.278 | 17,869 |
| **epoch_3_whole** | **2.698** | **0.280** | **23,823** |
| **epoch_4_whole** | **2.706** | **0.280** | **29,780** |
| epoch_5_whole | 2.737 | 0.279 | 35,736 |
| epoch_7_whole | 2.899 | 0.271 | 47,641 |
| epoch_8_step_50000 | 3.066 | 0.263 | 49,999 |

Loss starts increasing after epoch 4-5 (overfitting).

---

## How to Resume Fine-Tuning on a New System

### 1. Clone this repo

```bash
git lfs install
git clone https://huggingface.co/dm15/cosyvoice2-hebrew
cd cosyvoice2-hebrew
```

### 2. Install dependencies

```bash
pip install -r requirements.txt
# Make sure you have: torch, torchaudio, hyperpyyaml, onnxruntime-gpu, 
# conformer, diffusers, etc.
cd third_party/Matcha-TTS && pip install -e . && cd ../..
```

### 3. Prepare your training data

You need `train.data.list` and `dev.data.list` pointing to parquet files.
If you have the original Hebrew data, run stages 0-4 of `run.sh`:

```bash
cd examples/hebrew/cosyvoice2
# Edit run.sh: set metadata= and wav_dir= to your data paths
bash run.sh  # with stage=0 stop_stage=4
```

If you already have parquet data from a previous run, just make sure 
`data/train.data.list` and `data/dev.data.list` exist with correct paths.

### 4. Resume training from best checkpoint

The key is the `--checkpoint` flag — point it to the checkpoint you want to resume from.
The training script reads `step` and `epoch` from the checkpoint and continues from there.

**Resume from epoch_4_whole.pt (recommended starting point):**

```bash
cd examples/hebrew/cosyvoice2
. ./path.sh

export CUDA_VISIBLE_DEVICES="0"  # adjust to your GPU

torchrun --nnodes=1 --nproc_per_node=1 \
    --rdzv_id=2024 --rdzv_backend="c10d" --rdzv_endpoint="localhost:1235" \
  ../../../cosyvoice/bin/train.py \
  --train_engine torch_ddp \
  --config conf/cosyvoice2.yaml \
  --train_data data/train.data.list \
  --cv_data data/dev.data.list \
  --qwen_pretrain_path ../../../pretrained_models/CosyVoice2-0.5B/CosyVoice-BlankEN \
  --onnx_path ../../../pretrained_models/CosyVoice2-0.5B \
  --model llm \
  --checkpoint exp/cosyvoice2_hebrew/llm/torch_ddp/epoch_4_whole.pt \
  --model_dir $(pwd)/exp/cosyvoice2_hebrew/llm/torch_ddp \
  --tensorboard_dir $(pwd)/tensorboard/cosyvoice2_hebrew/llm/torch_ddp \
  --ddp.dist_backend nccl \
  --num_workers 4 \
  --prefetch 100 \
  --pin_memory \
  --use_amp
```

### 5. Resume tips

- **Checkpoint stores**: model weights + `step` count + `epoch` number. Training resumes from `epoch + 1`.
- **Optimizer state is NOT saved** (save_states=model_only), so momentum/Adam state resets on resume. This is normal for this codebase.
- **To lower the learning rate** for continued training, edit `conf/cosyvoice2.yaml` → `train_conf.optim_conf.lr`
- **To reduce overfitting** (loss was rising after epoch 4): try reducing `max_epoch`, increasing data, or lowering lr.
- **Multi-GPU**: change `--nproc_per_node=N` and `CUDA_VISIBLE_DEVICES` accordingly.

### 6. Inference with fine-tuned model

```bash
cd examples/hebrew/cosyvoice2
python3 infer_cpu.py  # uses model_hebrew/ directory
```

Or to use a specific checkpoint as the LLM, copy it to `model_hebrew/llm.pt`:
```bash
cp exp/cosyvoice2_hebrew/llm/torch_ddp/epoch_3_whole.pt model_hebrew/llm.pt
```

---

## Original Training Command (for reference)

The full pipeline is in `examples/hebrew/cosyvoice2/run.sh`. It includes:
- Stage 0: Data preparation from metadata.csv
- Stage 1: Speaker embedding extraction (CampPlus)
- Stage 2: Speech token extraction
- Stage 3: Parquet file creation
- Stage 4: Data list creation
- Stage 5: Fine-tuning (LLM + Flow)
- Stage 6: Checkpoint averaging