codriao / tokenizer.json
Raiff1982's picture
Create tokenizer.json
9d3d0c9 verified
raw
history blame
251 Bytes
{
"truncation": {
"strategy": "LongestFirst",
"max_length": 512,
"stride": 0
},
"padding": "max_length",
"do_lower_case": false,
"add_prefix_space": false,
"use_fast": true,
"model_max_length": 512
}