GPT2-tokenizer / added_tokens.json
svwingerden's picture
Duplicate from georgeyw/GPT2-tokenizer-8k-ctx
138dbbe verified
{
"<|endoftext|>": 50256
}