fhnw
/

fhnw-tiktoken-tokenizer

Model card Files Files and versions

fhnw-tiktoken-tokenizer

2.2 MB

Ctrl+K

Ctrl+K

1 contributor

History: 5 commits

andreasmartin's picture

Upload fhnw_tokenizer.py with huggingface_hub

f2ebe08 verified 7 months ago

.gitattributes

1.52 kB
initial commit 7 months ago
fhnw_tokenizer.py

1.41 kB
Upload fhnw_tokenizer.py with huggingface_hub 7 months ago
tokenizer.pkl
Pickle imports
- No problematic imports detected
What is a pickle import?
2.19 MB
xet

Upload tokenizer.pkl with huggingface_hub 7 months ago