fix: Tune dataset gen params and improve training checkpoint/resume
marcos+1: Claude Opus 4.5
75eda59
- Reduce SNAC batch size for 24GB GPUs to prevent OOM
- Increase Groq parallel requests to 20
- Fix batch resume counting with metadata file
- Improve Stage 2 full fine-tuning stability and logging
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Add full fine-tuning (no LoRA) and dataset generation tools
marcos+1: Claude Opus 4.5
cbe0918
Stage 2 full fine-tuning script that trains all 3.3B LLM params + adapter,
with fixed loss display (avg per sample instead of accumulated sum).
Added dataset creation and validation utilities.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Refactor training with SOLID principles and add optimizations
marcos+1: Claude Opus 4.5
e20f447
Training refactor:
- Modular training package (config, data, models, trainer, checkpoint)
- Auto GPU detection (H200 141GB, A100, RTX 4090)
- Flash Attention 2 with fallback to SDPA
- OOM recovery with proper cleanup (PyTorch best practices)
- NaN/Inf detection and gradient monitoring
- Sequence length bucketing for memory efficiency
- BF16 for numerical stability
Stage 1 improvements:
- Added --no_decay flag (fixed text_ratio for adapter warmup)
- Dynamic decay optional via --use_decay
Dataset tools:
- analyze_dataset.py for dataset inspection
- validate_dataset.py for format validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Add multi-GPU support and lmdeploy for faster TTS
marcos+1: Claude Opus 4.5
5cc96de
- create_dataset.py: Auto-detect GPUs, distribute workers across GPUs
- create_dataset.py: Workers use cuda:{actual_gpu} for specific GPU assignment
- create_dataset.py: TTS workers send None to feat_queue directly (pipeline sync fix)
- passo0_setup.py: Add lmdeploy installation (2000x realtime TTS backend)
- passo0_setup.py: Test Soprano TTS with lmdeploy backend
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Add H200/Hopper support and VRAM-adaptive batch sizing
marcos+1: Claude Opus 4.5
9767b87
- Add .env loading and VRAM-based batch size calculation in create_dataset
- Detect GPU architecture (compute capability) for CUDA version selection
- Replace WhisperX with native Whisper (H200 sm_90 compatible)
- Add NeMo Forced Aligner for GPU-accelerated alignment
- Support pre-computed tokens in word alignments during training
- Fix avg_loss calculation to account for gradient accumulation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Replace Matcha with Soprano TTS and add inference pipeline
marcos+1: Claude Opus 4.5
32fdbcd
- Switch from Matcha TTS to Soprano TTS for simpler setup
- Remove Gemma local model (using Groq API instead)
- Add fallback tokenizer loading in dataset creation
- Make text_tokens optional when tokenizer unavailable
- Add inference.py for end-to-end speech-to-speech
- Add TTS service abstractions (soprano, matcha, piper, gemma)
- Add dataset verification script
- Fix version constraints for WhisperX compatibility
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
fix: Update dependencies and tokenizer handling for dataset generation
marcos+1: Claude Opus 4.5
512101b
- Update passo0_setup.py with correct library versions:
- numpy>=2.0.2,<2.1.0 (WhisperX compatibility)
- pandas>=2.2.3,<2.3.0 (WhisperX compatibility)
- av<16.0.0 (WhisperX compatibility)
- pyannote.audio==3.3.2 with pyannote.core<6.0 and pyannote.pipeline<4.0
- soprano-tts (instead of wrong soprano package)
- Add graceful fallback for tokenizer authentication in create_dataset.py
- Make text_tokens precomputation optional if tokenizer unavailable
Tested on RTX 5090 32GB:
- Dataset generation: 100 items in 47s
- Stage 1: Loss 184 -> 166 (Adapter only, 19.4M params)
- Stage 2: Loss 315 -> 35.8 (Adapter + LoRA, 43.7M params)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add gradient checkpointing auto-detection to Stage 1 and Stage 2
marcos+1: Claude Opus 4.5
d2911f1
- Auto-enable gradient checkpointing if VRAM < 20GB (memory optimization)
- Auto-disable if VRAM >= 20GB (speed optimization)
- Add --gradient_checkpointing flag for manual control
- Consistent logging format across both stages
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Add gradient checkpointing to Stage 2 training
marcos+1: Claude Opus 4.5
37f91f0
- Add --gradient_checkpointing flag to reduce VRAM usage
- Enables training on 15GB GPUs (was failing with OOM)
- Tested: Stage 2 now completes successfully with loss ~130
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
perf: Increase TTS_BATCH_SIZE to 200 for better throughput
marcos+1: Claude Opus 4.5
2a5a768
- Changed from 100 to 200 items per batch
- Achieves ~10 items/s (vs ~2.7 items/s with batch 100)
- Reduces inter-process communication overhead
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Pre-compute text_tokens in dataset generation
marcos+1: Claude Opus 4.5
9258ac3
- Add tokenizer loading with HuggingFace auth
- Pre-tokenize answer text using Orpheus tokenizer
- Save text_tokens tensor in dataset for faster training loading
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
chore: Remove test files from datasets directory
marcos+1: Claude Opus 4.5
1dafe0a
Keep only create_dataset.py as the main pipeline script.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
feat: Async dataset pipeline with adaptive memory management
marcos+1: Claude Opus 4.5
0f2f953
- Replace sequential pipeline with fully async architecture
- TTS + Features workers run in parallel on 2 GPUs
- Adaptive batch sizing based on 80% GPU memory limit
- Auto-reduce batch on OOM, auto-increase on success
- Parallel Whisper encoding (4 threads)
- Batched SNAC encoding with dynamic mini-batches
- Achieves ~20 items/s throughput (vs ~10 items/s before)
- Move API keys to environment variables
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- passo0_setup.py: Environment setup for GPU training
- passo1_convertdataset.py: Convert raw data to InstructS2S format
- passo2_finetune_stage1.py: Stage 1 training (adapter only)
- passo3_finetune_stage2.py: Stage 2 training (adapter + LoRA)
- datasets/generate_dataset.py: Generate Q&A dataset with TTS/STT
- services/: Modular services for Groq, WhisperX, Kokoro TTS, Llama
All API keys use environment variables for security.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
