Hugging Face's logo

flexitok
/
mod-tokenizers-rtl_4digit

Safetensors
Undetermined
llama
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
Community
gsaltintas commited on
Commit
76f5a80
·
verified ·
1 Parent(s): 82e9ca4

Upload folder using huggingface_hub

Browse files
Files changed (4) hide show
  1. README.md +1 -1
  2. tokenizer.json +0 -0
  3. tokenizer_config.json +0 -0
  4. vocab.json +0 -0
README.md CHANGED
@@ -46,4 +46,4 @@ tokens = tokenizer.encode("Hello, world!")
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
- | `1234500119 mod 67` | `1234, 5001, 19, , mod, , 67` | `1686, 9035, 3243, 6, 4, 6, 3520` |
 
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
+ | `123500119 mod 67` | `1235, 0, 0, 119, , mod, , 67` | `3866, 7814, 7814, 2378, 6, 4, 6, 7967` |
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
The diff for this file is too large to render. See raw diff
 
vocab.json CHANGED
The diff for this file is too large to render. See raw diff