fable-mini / tokenizer_config.json
dev-sandakelum's picture
Upload folder using huggingface_hub
feb6b86 verified
Raw History Blame Contribute Delete
198 Bytes
{
"backend": "tokenizers",
"bos_token": "<|doc|>",
"eos_token": "<|eos|>",
"model_max_length": 512,
"pad_token": "<|pad|>",
"tokenizer_class": "TokenizersBackend",
"unk_token": null
}