ash001/ray-train-zero-3-bloom-1B

BLOOMZ-1B1 Fine-tuned with Ray Train + DeepSpeed ZeRO-3

This model is a fine-tuned version of bigscience/bloomz-1b1, fine-tuned using Ray Train with DeepSpeed ZeRO-3 for scalable distributed training with minimal configuration overhead.

This model was fine-tuned using the IMDB dataset on 2 × T4 16GB GPUs, achieving 11% training loss reduction (from ~3.6 to ~3.2) with automated distributed orchestration.

For detailed implementation, Ray configurations, and distributed training setup, please check out the project repository.