Upload folder using huggingface_hub

README.md

@@ -105,14 +105,16 @@ Input: `"İSTANBUL'da meeting'e katılamadım"`
 
 | token | token_type | morph_pos | notes |
 |---|---|---|---|
-| `<uppercase_word>` | ROOT | 0 | ALL CAPS marker |
+| `<uppercase_word>` | ROOT | 0 | ALL CAPS marker (Fix 1) |
 | ` istanbul` | ROOT | 0 | lowercased |
-| `da` | SUFFIX | 1 | `-LOC` |
-| ` meeting` | FOREIGN | 0 | TDK'da yok |
-| `e` | SUFFIX | 1 | `-DAT` |
-| ` katılama` | ROOT | 0 | Zemberek validated |
-| `dı` | SUFFIX | 1 | `-PST` `[PAST]` |
-| `m` | SUFFIX | 2 | `-1SG` |
+| `'` | PUNCT | 0 | Fixed boundary |
+| `da` | SUFFIX | 1 | `-LOC` [LOC] |
+| ` meeting` | FOREIGN | 0 | TDK lookup (Fix 7) |
+| `e` | SUFFIX | 1 | `-DAT` [DAT] |
+| ` katılmak` | ROOT | 0 | Root corrected (Fix 4) |
+| `lama` | SUFFIX | 1 | `-VN+NEG` |
+| `d` | SUFFIX | 2 | `-PAST` |
+| `ım` | SUFFIX | 3 | `-1SG` [1SG] |
 
 ---
 
@@ -123,8 +125,8 @@ Every token dict contains:
 | Field | Type | Description |
 |---|---|---|
 | `token` | `str` | Token string — leading space means word-initial |
-| `token_type` | `str` | Morphological type (see table below) |
-| `morph_pos` | `int` | Position within word: `0`=root, `1`=1st suffix, `2`=2nd suffix... |
+| `token_type` | `str` | Morphological type (ROOT, SUFFIX, FOREIGN, PUNCT, etc.) |
+| `morph_pos` | `int` | Position within word: `0`=root/initial, `1`=1st suffix, `2`=2nd suffix... |
 
 ### Token Types
 

nedo_turkish_tokenizer/_suffix_expander.py

@@ -3,9 +3,9 @@
 from __future__ import annotations
 
 PUNCT_CHARS = set(
-    '?.,;:!-\u2013\u2014()[]{}"`/\\|@#$%^&*+=<>~'
-    '\u2019\u2018\u201c\u201d\u2032\u00ab\u00bb\u2039\u203a'
-    '\u2022\u2026\u00b7\u00b0\u00b1\u00d7\u00f7'
+    "'?.,;:!-\u2013\u2014()[]{}\"`/\\|@#$%^&*+=<>~"
+    "\u2019\u2018\u201c\u201d\u2032\u00ab\u00bb\u2039\u203a"
+    "\u2022\u2026\u00b7\u00b0\u00b1\u00d7\u00f7"
 )
 _PUNCT_DIGITS = set("0123456789")