Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Log In
Sign Up
romizone
/
bpe-tokenizer-id
like
0
Token Classification
Transformers
Indonesian
tokenizer
bpe
bahasa-indonesia
indonesian
nlp
text-processing
subword-tokenization
License:
mit
Model card
Files
Files and versions
xet
Community
Deploy
Use this model
main
bpe-tokenizer-id
304 kB
Ctrl+K
Ctrl+K
1 contributor
History:
5 commits
romizone
Upload BPE Tokenizer Bahasa Indonesia
5a45b76
verified
about 18 hours ago
.gitattributes
Safe
1.52 kB
initial commit
about 18 hours ago
README.md
12.8 kB
Upload BPE Tokenizer Bahasa Indonesia
about 18 hours ago
bpe_tokenizer.py
14.3 kB
Upload BPE Tokenizer Bahasa Indonesia
about 18 hours ago
merges.txt
38.8 kB
Upload BPE Tokenizer Bahasa Indonesia
about 18 hours ago
special_tokens_map.json
98 Bytes
Upload BPE Tokenizer Bahasa Indonesia
about 18 hours ago
tokenizer.json
163 kB
Upload BPE Tokenizer Bahasa Indonesia
about 18 hours ago
tokenizer_config.json
219 Bytes
Upload BPE Tokenizer Bahasa Indonesia
about 18 hours ago
vocab.json
72.3 kB
Upload BPE Tokenizer Bahasa Indonesia
about 18 hours ago