fasthypernet-12 / checkpoint-700 /tokenizer_config.json
aixk's picture
Upload folder using huggingface_hub
e807cfa verified
Raw
History Blame Contribute Delete
199 Bytes
{
"tokenizer_class": "HFCompatibleHybridTokenizer",
"vocab_size": 8192,
"model_max_length": 512,
"bos_token": "<bos>",
"eos_token": "<eos>",
"unk_token": "<unk>",
"pad_token": "<pad>"
}