OddEvenDumb / tokenizer_config.json
56m's picture
Upload 7 files
e5a9af5 verified
Raw
History Blame Contribute Delete
179 Bytes
{
"add_bos_token": true,
"add_prefix_space": false,
"bos_token": "<BOS>",
"eos_token": "<EOS>",
"model_max_length": 512,
"tokenizer_class": "PreTrainedTokenizerFast"
}