kokorotest / tokenizer_config.json
chae12121's picture
Upload 67 files
aca96eb verified
{
"model_max_length": 512,
"pad_token": "$",
"tokenizer_class": "PreTrainedTokenizer",
"unk_token": "$"
}