Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
kaanilker
/
Mini-Turkish-Tokenizer
like
0
Token Classification
uonlp/CulturaX
Turkish
token
tokenization
turkish
nlp
llm
mini
License:
gpl-2.0
Model card
Files
Files and versions
xet
Community
main
Mini-Turkish-Tokenizer
/
special_tokens_map.json
kaanilker
Upload 3 files
6e169aa
verified
3 months ago
raw
Copy download link
history
blame
contribute
delete
Safe
99 Bytes
{
"bos_token"
:
"<bos>"
,
"eos_token"
:
"<eos>"
,
"pad_token"
:
"<pad>"
,
"unk_token"
:
"<unk>"
}