transformers/pr_43838/en/training.md

Fine-tuning

Fine-tuning continues training a large pretrained model on a smaller dataset specific to a task or domain. For example, fine-tuning on a dataset of coding examples helps the model get better at coding. Fine-tuning is identical to pretraining except you don't start with random weights. It also requires far less compute, data, and time.

The tutorial below walks through fine-tuning a large language model with Trainer.

Log in to your Hugging Face account with your user token to push your fine-tuned model to the Hub.

from huggingface_hub import login

login()

Tokenization

Load a dataset and tokenize the text column the model trains on (horoscope in the dataset below).