The performance of full-parameter finetuning
#13
by stephenshuang - opened
Will the performance of full-parameter finetuning be better than lora style finetuning?
Our experiments reveal that full-parameter fine-tuning tends to yield better results on in-domain test datasets compared to LoRA-style fine-tuning. However, it experiences a significant performance drop when applied to out-of-domain datasets. Consequently, we recommend adopting the LoRA training approach, which not only conserves training resources but also preserves more of the foundational model's capabilities, such as handling long texts and multilingual tasks, among others.