neutts-hebrew-big — checkpoint-1600

Hebrew fine-tune of neuphonic/neutts-air (NeuTTS-Air, Qwen2.5-0.5B backbone) on a NeuCodec-encoded Hebrew dataset. This repo packages everything needed to resume training from global step 1600 on a remote machine.

Repo contents

Path	Size	What it is
checkpoint-1600/	~6.3 GB	Full HF Trainer checkpoint: model.safetensors, optimizer.pt, scheduler.pt, rng_state.pth, trainer_state.json, training_args.bin, full tokenizer.
dataset/	~5.7 GB	Pre-encoded datasets dir (load_from_disk-compatible) with train + validation splits. Each row: { "text": <IPA phonemes>, "codes": <NeuCodec int IDs> }.
scripts/finetune_hebrew.py	small	Original training script (text-conditioned causal LM over speech codes).
scripts/finetune_hebrew_big_config.yaml	small	Original local config (Windows paths) — kept for reference.
scripts/finetune_hebrew_big_config_server.yaml	small	Use this on the server. Paths already relative to repo root.
scripts/requirements.txt	small	Minimum dependency set.

max_steps is 3000, so checkpoint-1600 is roughly 53% through the planned run.

Resume training on the server

# 1. Pull the repo (private — needs an HF token with read access)
huggingface-cli login           # paste your token
huggingface-cli download dm15/neutts-hebrew-big \
    --repo-type model \
    --local-dir ./neutts-hebrew-big

cd neutts-hebrew-big

# 2. Install deps (CUDA build of torch separately if needed)
pip install -r scripts/requirements.txt

# 3. Resume — paths in the server config are relative to scripts/
cd scripts
python finetune_hebrew.py finetune_hebrew_big_config_server.yaml

The Trainer will:

1. Load the model + tokenizer from ../checkpoint-1600 (restore_from).
2. See resume_from_checkpoint: "latest", locate the latest checkpoint-* under ../checkpoints/neutts-hebrew-big/, and continue from global_step 1600 with the saved optimizer / scheduler / RNG state.

First-resume note: the Trainer expects to find checkpoints under <save_root>/<run_name>/checkpoint-*. If ../checkpoints/neutts-hebrew-big/ doesn't exist on the server, create it and either move or symlink the uploaded checkpoint-1600/ into it before launching:

mkdir -p ../checkpoints/neutts-hebrew-big
mv ../checkpoint-1600 ../checkpoints/neutts-hebrew-big/

Training format (for reference)

Each sample is rendered as a chat string and tokenized:

user: Convert the text to speech:<|TEXT_PROMPT_START|>{IPA phones}<|TEXT_PROMPT_END|>
assistant:<|SPEECH_GENERATION_START|>{<|speech_N|> ...}<|SPEECH_GENERATION_END|>

Loss is masked everywhere except on the speech-token span after <|SPEECH_GENERATION_START|>. Sequence length is padded/truncated to 2048.

A <|HE|> language tag was added to the tokenizer during the original fine-tune; embeddings were resized accordingly. The checkpoint already contains the resized embedding matrix and updated tokenizer, so the "add Hebrew tag" branch in finetune_hebrew.py is a no-op on resume.

Hyperparameters at step 1600

• Optimizer: AdamW (HF Trainer default), bf16
• LR: 5e-5, cosine schedule, warmup_ratio 0.05
• Per-device batch: 2, grad accum: 1
• max_steps: 3000, save_steps/eval_steps: 500
• Seed: 1337