Inquiry about Batch Size and Total Steps Used for MLM Task on xlm-roberta?
#26
by jrjrjrjr - opened
Hello,
I'm currently exploring the xlm-roberta model and have a specific interest in understanding some of the training details, particularly regarding the Masked Language Model (MLM) task. Could you provide insights or details about the batch size and total number of steps used during the training process for the MLM task on XLM-R and XLM-R-base?
Hi @jrjrjrjr !
Thanks for the issue
You can read more about the training protocol of that model in the original repository: https://github.com/facebookresearch/fairseq/tree/main/examples/xlmr and you can also try out MLM task yourself in one of our official scripts: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py