File size: 918 Bytes
2c3d66c 84ca195 2c3d66c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | ---
datasets:
- flexitok/mod-arithmetic
---
# Super Vocabulary
A merged super-vocabulary built from 9 tokenizer(s).
**Vocab size:** 100007
## Tokenizers
- `flexitok/mod-tokenizers-individual`
- `flexitok/mod-tokenizers-ltr_3digit`
- `flexitok/mod-tokenizers-ltr_2digit`
- `flexitok/mod-tokenizers-ltr_4digit`
- `flexitok/mod-tokenizers-ltr_5digit`
- `flexitok/mod-tokenizers-rtl_2digit`
- `flexitok/mod-tokenizers-rtl_3digit`
- `flexitok/mod-tokenizers-rtl_4digit`
- `flexitok/mod-tokenizers-rtl_5digit`
## Files
- `super_vocab.json` — merged vocabulary mapping token string → super index
- `config.yaml` — model config with `vocab_size`
- `participating_tokenizers.json` — list of tokenizer names included
- `<tokenizer>_super_mapping.json` — per-tokenizer index → super index mapping
- `<tokenizer>_vocab.json` — per-tokenizer vocabulary
- `<tokenizer>_info.json` / `.yaml` — tokenizer metadata |