Dhurmir
/

patched-mt5-tokenizer

Model card Files Files and versions

Dhurmir commited on May 21, 2025

Commit

5cbafd3

·

verified ·

1 Parent(s): fc757ed

Update tokenizer_config.json

Files changed (1) hide show

tokenizer_config.json +1 -1

tokenizer_config.json CHANGED Viewed

@@ -936,6 +936,6 @@
   "model_max_length": 700,
   "pad_token": "<pad>",
   "sp_model_kwargs": {},
-  "tokenizer_class": "MT5Tokenizer",
   "unk_token": "<unk>"
 }

   "model_max_length": 700,
   "pad_token": "<pad>",
   "sp_model_kwargs": {},
+  "tokenizer_class": "T5Tokenizer",
   "unk_token": "<unk>"
 }