bpe-tokenizer-vie / tokenizer_config.json
hieule's picture
commit files to HF hub
71b880a
raw
history blame contribute delete
115 Bytes
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"tokenizer_class": "PreTrainedTokenizerFast"
}