versae
/

scandeng-tokenizer

Text Generation

Model card Files Files and versions

scandeng-tokenizer

4.37 MB

1 contributor

History: 4 commits

versae's picture

Add HF tokenizer converted from SentencePiece

04b8476 almost 2 years ago

.gitattributes

1.52 kB

initial commit almost 2 years ago
README.md

28 Bytes

initial commit almost 2 years ago
config.json

595 Bytes

Create config.json almost 2 years ago
convert.sh

1.1 kB

Add HF tokenizer converted from SentencePiece almost 2 years ago
merges.txt

568 kB

Add HF tokenizer converted from SentencePiece almost 2 years ago
sentencepiece.model

770 kB
xet

SentencePiece byte fallback 32k almost 2 years ago
sentencepiece.vocab.bak

486 kB

Add HF tokenizer converted from SentencePiece almost 2 years ago
special_tokens_map.json

72 Bytes

Add HF tokenizer converted from SentencePiece almost 2 years ago
tokenizer.json

1.85 MB

Add HF tokenizer converted from SentencePiece almost 2 years ago
tokenizer_config.json

826 Bytes

Add HF tokenizer converted from SentencePiece almost 2 years ago
vocab.json

687 kB

Add HF tokenizer converted from SentencePiece almost 2 years ago