820 kB

Ctrl+K

1 contributor

History: 6 commits

janakhpon

feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue

81cf36d 8 months ago

.gitattributes

285 Bytes
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
.gitignore

114 Bytes
feat: simplified mon tokenizer in hf format, updated tags 8 months ago
.python-version

5 Bytes
feat: simplified mon tokenizer in hf format 8 months ago
README.md

2.48 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
convert_to_hf.py

17.9 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
generation_config.json

291 Bytes
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
mon_tokenizer.meta.json

12.9 kB
feat: simplified mon tokenizer in hf format 8 months ago
mon_tokenizer.model

338 kB
xet

feat: simplified mon tokenizer in hf format 8 months ago
pyproject.toml

1.1 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
sample_usage.py

6.14 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
special_tokens_map.json

551 Bytes
feat: simplified mon tokenizer in hf format 8 months ago
test_tokenizer.py

18.2 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
tokenizer_config.json

1.16 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
upload_to_hub.py

14.7 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago
uv.lock

406 kB
feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue 8 months ago