Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Log In
Sign Up
hieule
/
bpe-tokenizer-vie
like
0
Model card
Files
Files and versions
xet
Community
main
bpe-tokenizer-vie
/
tokenizer_config.json
hieule
commit files to HF hub
71b880a
over 3 years ago
raw
Copy download link
history
blame
contribute
delete
Safe
115 Bytes
{
"bos_token"
:
"<|endoftext|>"
,
"eos_token"
:
"<|endoftext|>"
,
"tokenizer_class"
:
"PreTrainedTokenizerFast"
}