Continuous training pre-trained Bloom on custom biomedical dataset
I want to use an already pretrained bloom model and fine-tune (continue training) it on my custom biomedical dataset. Has anyone solved it and share a link to the script to do this finetuning?
Hi @Siddharth63 ! If you want to use Megatron-DeepSpeed, we were able to do it (typically that's how we built BLOOMZ, there's a README in the GH repo https://github.com/bigscience-workshop/xmtf). Otherwise I'd suggest looking at this: https://huggingface.co/bigscience/bloom/discussions/46
Closing as this seems to be a duplicate of https://huggingface.co/bigscience/bloom/discussions/46. Feel free to re-open if you think I mistakenly closed it.
I guess Siddharth63 prefer continuous pretraining instead of finetune? https://huggingface.co/bigscience/bloom/discussions/46 is more likely finetuning discussion thread.