Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
canovich
/
tokenizer
like
0
Model card
Files
Files and versions
xet
Community
main
tokenizer
File size: 74 Bytes
564122b
1
2
3
4
5
6
{
"eos_token"
:
"</s>"
,
"pad_token"
:
"<pad>"
,
"unk_token"
:
"<unk>"
}