Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Super Vocabulary
|
| 2 |
|
| 3 |
A merged super-vocabulary built from 9 tokenizer(s).
|
|
@@ -23,4 +27,4 @@ A merged super-vocabulary built from 9 tokenizer(s).
|
|
| 23 |
- `participating_tokenizers.json` — list of tokenizer names included
|
| 24 |
- `<tokenizer>_super_mapping.json` — per-tokenizer index → super index mapping
|
| 25 |
- `<tokenizer>_vocab.json` — per-tokenizer vocabulary
|
| 26 |
-
- `<tokenizer>_info.json` / `.yaml` — tokenizer metadata
|
|
|
|
| 1 |
+
---
|
| 2 |
+
datasets:
|
| 3 |
+
- flexitok/mod-arithmetic
|
| 4 |
+
---
|
| 5 |
# Super Vocabulary
|
| 6 |
|
| 7 |
A merged super-vocabulary built from 9 tokenizer(s).
|
|
|
|
| 27 |
- `participating_tokenizers.json` — list of tokenizer names included
|
| 28 |
- `<tokenizer>_super_mapping.json` — per-tokenizer index → super index mapping
|
| 29 |
- `<tokenizer>_vocab.json` — per-tokenizer vocabulary
|
| 30 |
+
- `<tokenizer>_info.json` / `.yaml` — tokenizer metadata
|