Hugging Face's logo

flexitok
/
mod-tokenizers-rtl_4digit

Safetensors
Undetermined
llama
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
Community
gsaltintas commited on
Commit
05b9c4d
·
verified ·
1 Parent(s): 76f5a80

Upload folder using huggingface_hub

Browse files
Files changed (4) hide show
  1. README.md +1 -1
  2. tokenizer.json +0 -0
  3. tokenizer_config.json +0 -0
  4. vocab.json +0 -0
README.md CHANGED
@@ -46,4 +46,4 @@ tokens = tokenizer.encode("Hello, world!")
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
- | `123500119 mod 67` | `1235, 0, 0, 119, , mod, , 67` | `3866, 7814, 7814, 2378, 6, 4, 6, 7967` |
 
46
  ## Sample Encoding
47
  | Text | Tokens | Token IDs |
48
  |------|--------|-----------|
49
+ | `123500119 mod 67` | `1235, 0, 0, 119, , mod, , 67` | `5687, 3253, 3253, 5485, 6, 4, 6, 1506` |
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
The diff for this file is too large to render. See raw diff
 
vocab.json CHANGED
The diff for this file is too large to render. See raw diff