vorushin
/

tpuchat

Model card Files Files and versions

tpuchat / tokenizer

Ctrl+K

Ctrl+K

1 contributor

History: 1 commit

vorushin's picture

Upload trained BPE tokenizer (vocab 32768)

186dd03 verified 3 months ago

token_bytes.pt
Detected Pickle imports (3)
- "torch._utils._rebuild_tensor_v2",
- "torch.IntStorage",
- "collections.OrderedDict"
What is a pickle import?
133 kB
xet

Upload trained BPE tokenizer (vocab 32768) 3 months ago
tokenizer.pkl
Detected Pickle imports (1)
- "tiktoken.core.Encoding"
How to fix it?
413 kB
xet

Upload trained BPE tokenizer (vocab 32768) 3 months ago