InterwebAlchemy
/

PGNTokenizer

Model card Files Files and versions

InterwebAlchemy commited on Feb 11, 2025

Commit

0dcea10

·

verified ·

1 Parent(s): 6d5f592

Switch from NFD -> NFC

Files changed (1) hide show

tokenizer.json +1 -1

tokenizer.json CHANGED Viewed

@@ -41,7 +41,7 @@
     }
   ],
   "normalizer": {
-    "type": "NFD"
   },
   "pre_tokenizer": {
     "type": "Split",

     }
   ],
   "normalizer": {
+    "type": "NFC"
   },
   "pre_tokenizer": {
     "type": "Split",