ByteMeHarder-404
/

tokenizers

bytepairencoding

Model card Files Files and versions

tokenizers / basic_wordpiece

573 kB

1 contributor

History: 1 commit

ByteMeHarder-404's picture

ByteMeHarder-404

Upload WordPiece, BPE, and XLNet tokenizers

4713739 verified 3 months ago

special_tokens_map.json

125 Bytes

Upload WordPiece, BPE, and XLNet tokenizers 3 months ago
tokenizer.json

572 kB

Upload WordPiece, BPE, and XLNet tokenizers 3 months ago
tokenizer_config.json

1.17 kB

Upload WordPiece, BPE, and XLNet tokenizers 3 months ago