hf-internal-testing
/

fuyu-tiny-tokenizer

hf-transformers-bot commited on Jun 30

Commit

d2b6739

verified ·

1 Parent(s): 44d0db7

Update tokenizer_config.json: 760-token vocab with all 256 byte tokens

Files changed (1) hide show

tokenizer_config.json CHANGED Viewed

@@ -8,7 +8,7 @@
       "single_word": false,
       "special": true
     },
-    "508": {
       "content": "|ENDOFTEXT|",
       "lstrip": false,
       "normalized": false,

       "single_word": false,
       "special": true
     },
+    "759": {
       "content": "|ENDOFTEXT|",
       "lstrip": false,
       "normalized": false,