janakhpon
/

mon_tokenizer

Model card Files Files and versions

2.28 MB

Ctrl+K

Ctrl+K

1 contributor

History: 12 commits

janakhpon's picture

feat: update tokenizer artifacts with 41.4M character corpus

39ad643 19 days ago

.gitattributes

315 Bytes
feat: restructure and upgrade to 32k vocab model (v2) 20 days ago
.gitignore

522 Bytes
feat: restructure and upgrade to 32k vocab model (v2) 20 days ago
README.md

2.23 kB
feat: update tokenizer artifacts with 41.4M character corpus 19 days ago
added_tokens.json

21 Bytes
feat: restructure and upgrade to 32k vocab model (v2) 20 days ago
mon_tokenizer.vocab

1 MB
xet

feat: update tokenizer artifacts with 41.4M character corpus 19 days ago
special_tokens_map.json

552 Bytes
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 9 months ago
tokenizer.json

278 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 9 months ago
tokenizer.model

996 kB
xet

feat: update tokenizer artifacts with 41.4M character corpus 19 days ago
tokenizer_config.json

1.14 kB
feat: restructure and upgrade to 32k vocab model (v2) 20 days ago