gsaltintas's picture
Upload README.md with huggingface_hub
7c08243 verified
# Super Vocabulary
A merged super-vocabulary built from 9 tokenizer(s).
**Vocab size:** 110931
## Tokenizers
- `flexitok/mod-tokenizers-zero-padded-individual`
- `flexitok/mod-tokenizers-zero-padded-ltr_3digit`
- `flexitok/mod-tokenizers-zero-padded-ltr_2digit`
- `flexitok/mod-tokenizers-zero-padded-ltr_4digit`
- `flexitok/mod-tokenizers-zero-padded-ltr_5digit`
- `flexitok/mod-tokenizers-zero-padded-rtl_2digit`
- `flexitok/mod-tokenizers-zero-padded-rtl_3digit`
- `flexitok/mod-tokenizers-zero-padded-rtl_4digit`
- `flexitok/mod-tokenizers-zero-padded-rtl_5digit`
## Files
- `super_vocab.json` β€” merged vocabulary mapping token string β†’ super index
- `config.yaml` β€” model config with `vocab_size`
- `participating_tokenizers.json` β€” list of tokenizer names included
- `<tokenizer>_super_mapping.json` β€” per-tokenizer index β†’ super index mapping
- `<tokenizer>_vocab.json` β€” per-tokenizer vocabulary
- `<tokenizer>_info.json` / `.yaml` β€” tokenizer metadata