tokenizers / basic_bpe /special_tokens_map.json
ByteMeHarder-404's picture
Upload WordPiece, BPE, and XLNet tokenizers
4713739 verified
raw
history blame contribute delete
67 Bytes
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>"
}