TinyStories-Regional
/

tokenizer-comparisons

Model card Files Files and versions

tokenizer-comparisons / Tiktoken

1.74 GB

Ctrl+K

Ctrl+K

2 contributors

History: 4 commits

Jinom's picture

Upload folder using huggingface_hub

6b91b07 verified about 1 year ago

ckpt_tiktoken-beng_val=0.135.pt
Detected Pickle imports (3)
- "collections.OrderedDict",
- "torch._utils._rebuild_tensor_v2",
- "torch.FloatStorage"
What is a pickle import?
542 MB
xet

Upload folder using huggingface_hub about 1 year ago
ckpt_tiktoken-hindi_val=0.149.pt
Detected Pickle imports (3)
- "torch.FloatStorage",
- "torch._utils._rebuild_tensor_v2",
- "collections.OrderedDict"
What is a pickle import?
651 MB
xet

Upload folder using huggingface_hub about 1 year ago
ckpt_tiktoken-marathi_val=0.167.pt
Detected Pickle imports (3)
- "torch._utils._rebuild_tensor_v2",
- "torch.FloatStorage",
- "collections.OrderedDict"
What is a pickle import?
542 MB
xet

Upload folder using huggingface_hub about 1 year ago
test.txt

0 Bytes
Create Tiktoken/test.txt about 1 year ago