Malikeh1375 commited on
Commit
1a76b4b
·
verified ·
1 Parent(s): 27b80fa

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +33 -1
README.md CHANGED
@@ -124,7 +124,39 @@ TokSuite–BLOOM is evaluated on the **TokSuite robustness benchmark**, which me
124
  - OCR and spacing artifacts,
125
  - LaTeX and STEM-style formatting.
126
 
127
- Performance is reported as **relative accuracy drop** with respect to canonical inputs.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
128
 
129
  ---
130
 
 
124
  - OCR and spacing artifacts,
125
  - LaTeX and STEM-style formatting.
126
 
127
+
128
+ **Tokenization robustness under multilingual text perturbations**
129
+ Values represent **relative performance drop**, computed as `(Acc_clean − Acc_perturbed) / Acc_clean`, where **lower values indicate greater robustness**.
130
+
131
+ Perturbation types include:
132
+ - **Input:** non-native keyboard input and romanization
133
+ - **Diacr.:** optional diacritics
134
+ - **Orth.:** orthographic errors
135
+ - **Morph:** morphological variations including derivations, inflections, and contractions
136
+ - **Noise:** homoglyph substitutions, OCR artifacts, typos, and spacing errors
137
+ - **LaTeX:** LaTeX-style mathematical formatting
138
+ - **STEM:** scientific diagrams and notational conventions
139
+ - **Unic.:** Unicode styling characters
140
+
141
+ **NEN** denotes non-English inputs and **EN** denotes English inputs. The **Avg** column reports the average relative performance drop across all perturbation categories.
142
+
143
+
144
+ | Model | Input | Diacr. | Orth. | Morph | Noise | LaTeX | STEM | Unic. | Avg ↓ |
145
+ |---------------|-------|--------|-------|-------|-------|-------|------|-------|-------|
146
+ | TokenMonster | **0.23** | **0.33** | 0.08 | **0.01** | **-0.07** | **0.10** | 0.18 | 0.21 | **0.17** |
147
+ | XGLM | 0.34 | 0.49 | 0.10 | 0.11 | 0.07 | 0.12 | 0.22 | 0.29 | 0.22 |
148
+ | BLOOM | 0.30 | 0.34 | 0.13 | 0.07 | 0.11 | 0.18 | 0.18 | 0.24 | 0.22 |
149
+ | ByT5 | 0.30 | 0.44 | **0.04** | 0.06 | 0.04 | 0.14 | **0.18** | 0.17 | 0.22 |
150
+ | Comma | 0.28 | 0.43 | 0.05 | 0.07 | **0.00** | 0.11 | 0.20 | 0.23 | 0.22 |
151
+ | mBERT | 0.33 | 0.44 | 0.11 | 0.11 | 0.06 | 0.18 | 0.22 | **0.14** | 0.24 |
152
+ | GPT-4o | 0.30 | 0.51 | 0.08 | 0.05 | 0.05 | 0.16 | 0.19 | 0.24 | 0.24 |
153
+ | GPT-2 | 0.34 | 0.46 | 0.07 | 0.10 | 0.06 | 0.14 | 0.21 | 0.24 | 0.25 |
154
+ | Phi-3 | 0.33 | 0.46 | 0.16 | 0.09 | 0.08 | 0.17 | 0.21 | 0.24 | 0.25 |
155
+ | Gemma-2 | 0.32 | 0.42 | 0.14 | **0.15** | 0.03 | 0.16 | 0.25 | 0.22 | 0.26 |
156
+ | Qwen-3 | **0.36** | 0.42 | 0.14 | 0.11 | 0.06 | 0.16 | 0.23 | 0.26 | 0.26 |
157
+ | Llama-3.2 | 0.33 | **0.55** | 0.11 | 0.10 | 0.08 | 0.15 | 0.24 | 0.17 | 0.26 |
158
+ | Aya | 0.31 | 0.46 | 0.14 | 0.10 | 0.03 | **0.19** | **0.25** | 0.21 | 0.26 |
159
+ | Tekken | 0.33 | 0.47 | **0.18** | 0.03 | **0.31** | 0.10 | 0.21 | **0.27** | **0.27** |
160
 
161
  ---
162