Hugging Face's logo

flexitok
/
mod-tokenizers-rtl_4digit

Safetensors
Undetermined
llama
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
Community
gsaltintas commited on
Commit
163be4f
·
verified ·
1 Parent(s): e8166f9

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. README.md +1 -1
  2. merges.txt +0 -0
  3. tokenizer.json +0 -0
  4. tokenizer_config.json +0 -0
  5. vocab.json +0 -0
README.md CHANGED
@@ -46,4 +46,4 @@ tokens = tokenizer.encode("Hello, world!")
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
- | `123500119 mod 67` | `1235, 0, 0, 119, , mod, , 67` | `5554, 3568, 3568, 3842, 6, 4, 6, 678` |
 
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
+ | `123500119 mod 67` | `1, 2350, 01, 19, , mod, , 67` | `8, 1699, 108, 25, 6, 4, 6, 53` |
merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
The diff for this file is too large to render. See raw diff
 
vocab.json CHANGED
The diff for this file is too large to render. See raw diff