Implement service layers for backup, model management, file handling, tokenization, and training
muhammadjalaludin
1424118
- Added BackupService for managing cache backups and uploads to Hugging Face Hub.
- Introduced ModelManager for loading and managing ML models and tokenizers.
- Created SpaceFileService for handling file operations in the local model directory.
- Developed TokenizationService for text tokenization and vocabulary analysis.
- Implemented TrainingService for training ML models with data augmentation and reproducibility features.
- Added comprehensive unit tests for all new services to ensure functionality and error handling.
Enhance Dockerfile and add tokenization API: Update CMD instruction and implement tokenization endpoints with detailed vocabulary status and metrics.
