fasthypernet-16k2 / checkpoint-200 /tokenizer_config.json
aixk's picture
Upload folder using huggingface_hub
2ea85ce verified
Raw
History Blame Contribute Delete
204 Bytes
{
"tokenizer_class": "HFCompatibleLinguisticTokenizer",
"vocab_size": 16384,
"model_max_length": 512,
"bos_token": "<bos>",
"eos_token": "<eos>",
"unk_token": "<unk>",
"pad_token": "<pad>"
}