Language.converters / tokenizer_config.json
CoderHassan's picture
Create tokenizer_config.json
1fd24b7 verified
raw
history blame
165 Bytes
{
"do_lower_case": false,
"unk_token": "<unk>",
"bos_token": "<s>",
"eos_token": "</s>",
"pad_token": "<pad>",
"additional_special_tokens": ["<mask>"]
}