Commit History
Yesterday
-
rebuild tokenizer.tokie from tokenizer.json
Verified signature755152b
8 days ago
-
Add the tokie 0.2 tokenizer file
Verified signature17f824c
-
Regenerate tokenizer.tkz as format v13 (self-contained: added/special tokens stored in-file)
Verified signature9fe7095
-
Add model card README
Verified signature51725c9 -
Add tokenizer_config.json from openai-community/gpt2
Verified signaturefd787b4 -
Add tokenizer.json from openai-community/gpt2
Verified signaturee9b30b1 -
Add tokie banner
Verified signature1b3ccb3 -
Update tokenizer.tkz to v11 (with pad_token_id)
Verified signature6b45db9 -
initial commit
Verified signaturea44d2ec