fMindstormer
/

tokcpu

Model card Files Files and versions

tokcpu / tokenizer_config.json

fMindstormer's picture

Tokenizer CPU

9676339 verified 8 days ago

Raw History Blame Contribute Delete

117 Bytes

{"tokenizer_class": "PreTrainedTokenizerFast", "clean_up_tokenization_spaces": false, "model_max_length": 1000000000}