Can't load tokenizer for 'gpt2'.
I try to python train.py of GSM8K, and I get some wrong:
Traceback (most recent call last):
File "train.py", line 50, in
main()
File "train.py", line 11, in main
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
File "/home/user/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2197, in from_pretrained
raise EnvironmentError(
OSError: Can't load tokenizer for 'gpt2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'gpt2' is the correct path to a directory containing all relevant files for a GPT2Tokenizer tokenizer.
I will show 1~19 rows of GSM8K-code:
import torch as th
from dataset import get_examples, GSMDataset
from transformers import GPT2Tokenizer, GPT2LMHeadModel
from transformers import GPT2Config, AdamW
from transformers import get_scheduler
from tqdm.auto import tqdm
from torch.utils.data import DataLoader
def main():
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
train_examples = get_examples("train")
train_dset = GSMDataset(tokenizer, train_examples)
device = th.device("cuda")
config = GPT2Config.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2", config=config)
model.to(device)
model.train()
How can I solve this wrong?
I get the solution. If someone meet the same question, you can download gpt2-code (all files form a folder, the name is gpt2) to you local project （where train.py is). Then you will run successfully.