chrishayuk
/

v11-tokenizer

language-modeling

Model card Files Files and versions

v11-tokenizer / special_tokens_map.json

Commit History

Publish tokenizer 10dd51100331ab50 (adopted)

fa41352
verified

chrishayuk commited on 12 days ago

v11: byte-safety fix (0 UNK, 0 round-trip mismatches on the 32-file gate corpus)

7e96a85
verified

chrishayuk commited on 12 days ago