go-tokenizer / tokenizer_config.json
wrice's picture
Update tokenizer_config.json
9ca1ab9 verified
{
"added_tokens_decoder": {},
"bos_token": "<bos>",
"clean_up_tokenization_spaces": false,
"eos_token": "<eos>",
"extra_special_tokens": {},
"model_max_length": 892,
"obo_path": "go-basic.obo",
"pad_token": "<pad>",
"padding_side": "right",
"tokenizer_class": "GOTokenizer",
"unk_token": "<unk>"
}