Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Website
Tasks
HuggingChat
Collections
Languages
Organizations
Community
Blog
Posts
Daily Papers
Hardware
Learn
Discord
Forum
GitHub
Solutions
Team & Enterprise
Hugging Face PRO
Enterprise Support
Inference Providers
Inference Endpoints
Storage Buckets
Log In
Sign Up
rostlabs
/
rost-tok-ro-normalized
like
0
Follow
Rost Labs
1
Romanian
English
nanochat-bpe-tokenizer
romanian
tokenizer
bpe
diacritics
License:
odc-by
Model card
Files
Files and versions
xet
Community
Copy to bucket
new
main
rost-tok-ro-normalized
545 kB
Ctrl+K
Ctrl+K
1 contributor
History:
4 commits
stefaniancu
Add config.json to enable download stats
0f10dbf
verified
15 days ago
.gitattributes
Safe
1.52 kB
initial commit
20 days ago
README.md
690 Bytes
Update README.md
15 days ago
config.json
67 Bytes
Add config.json to enable download stats
15 days ago
token_bytes.pt
133 kB
xet
Upload folder using huggingface_hub
20 days ago
tokenizer.pkl
pickle
Detected Pickle imports (1)
"tiktoken.core.Encoding"
How to fix it?
410 kB
xet
Upload folder using huggingface_hub
20 days ago
