Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

special_tokens_map.json

@@ -0,0 +1,130 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<RULING>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<BLANK_SPACE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<COLUMN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<SURFACE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "...",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,2108 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "</s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<mask>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "\n",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 6,
+      "content": "<SURFACE>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 7,
+      "content": "<COLUMN>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 8,
+      "content": "<BLANK_SPACE>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 9,
+      "content": "<RULING>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 10,
+      "content": "...",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Sequence",
+    "pretokenizers": [
+      {
+        "type": "Metaspace",
+        "replacement": "▁",
+        "prepend_scheme": "always",
+        "split": true
+      },
+      {
+        "type": "Punctuation",
+        "behavior": "Isolated"
+      }
+    ]
+  },
+  "post_processor": {
+    "type": "RobertaProcessing",
+    "sep": [
+      "</s>",
+      2
+    ],
+    "cls": [
+      "<s>",
+      0
+    ],
+    "trim_offsets": true,
+    "add_prefix_space": true
+  },
+  "decoder": {
+    "type": "BPEDecoder",
+    "suffix": "</w>"
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "<s>": 0,
+      "<pad>": 1,
+      "</s>": 2,
+      "<unk>": 3,
+      "<mask>": 4,
+      "\n": 5,
+      "<SURFACE>": 6,
+      "<COLUMN>": 7,
+      "<BLANK_SPACE>": 8,
+      "<RULING>": 9,
+      "...": 10,
+      "&": 11,
+      "(": 12,
+      ")": 13,
+      "+": 14,
+      "-": 15,
+      ".": 16,
+      "/": 17,
+      "1": 18,
+      "2": 19,
+      "3": 20,
+      "4": 21,
+      "5": 22,
+      "6": 23,
+      "7": 24,
+      "8": 25,
+      "9": 26,
+      "@": 27,
+      "A": 28,
+      "B": 29,
+      "D": 30,
+      "E": 31,
+      "G": 32,
+      "H": 33,
+      "I": 34,
+      "K": 35,
+      "L": 36,
+      "M": 37,
+      "N": 38,
+      "P": 39,
+      "R": 40,
+      "S": 41,
+      "T": 42,
+      "U": 43,
+      "W": 44,
+      "X": 45,
+      "Z": 46,
+      "a": 47,
+      "b": 48,
+      "d": 49,
+      "e": 50,
+      "f": 51,
+      "g": 52,
+      "h": 53,
+      "i": 54,
+      "k": 55,
+      "l": 56,
+      "m": 57,
+      "n": 58,
+      "o": 59,
+      "p": 60,
+      "q": 61,
+      "r": 62,
+      "s": 63,
+      "t": 64,
+      "u": 65,
+      "w": 66,
+      "y": 67,
+      "z": 68,
+      "{": 69,
+      "|": 70,
+      "}": 71,
+      "~": 72,
+      "×": 73,
+      "ŋ": 74,
+      "Š": 75,
+      "š": 76,
+      "ʾ": 77,
+      "ṣ": 78,
+      "ṭ": 79,
+      "⁻": 80,
+      "₀": 81,
+      "₁": 82,
+      "₂": 83,
+      "₃": 84,
+      "₄": 85,
+      "₅": 86,
+      "₆": 87,
+      "₇": 88,
+      "₈": 89,
+      "₉": 90,
+      "ₓ": 91,
+      "▁": 92,
+      "iš": 93,
+      "diš": 94,
+      "ba": 95,
+      "▁1": 96,
+      "ga": 97,
+      "▁u": 98,
+      "▁g": 99,
+      "la": 100,
+      "in": 101,
+      "du": 102,
+      "mu": 103,
+      "▁s": 104,
+      "▁š": 105,
+      "aš": 106,
+      "▁k": 107,
+      "▁2": 108,
+      "lu": 109,
+      "da": 110,
+      "ur": 111,
+      "▁n": 112,
+      "▁lu": 113,
+      "▁i": 114,
+      "▁a": 115,
+      "▁si": 116,
+      "▁3": 117,
+      "eš": 118,
+      "ri": 119,
+      "ma": 120,
+      "gal": 121,
+      "▁e": 122,
+      "▁sila": 123,
+      "ta": 124,
+      "▁du": 125,
+      "▁mu": 126,
+      "▁ba": 127,
+      "ki": 128,
+      "▁5": 129,
+      "na": 130,
+      "▁šu": 131,
+      "▁ur": 132,
+      "▁sila₃": 133,
+      "n₂": 134,
+      "ka": 135,
+      "ni": 136,
+      "ti": 137,
+      "ban₂": 138,
+      "▁4": 139,
+      "ra": 140,
+      "geš": 141,
+      "sa": 142,
+      "gi": 143,
+      "▁lugal": 144,
+      "▁gur": 145,
+      "e₃": 146,
+      "in₂": 147,
+      "rig": 148,
+      "▁ki": 149,
+      "barig": 150,
+      "bi": 151,
+      "▁lu₂": 152,
+      "▁gu": 153,
+      "ku": 154,
+      "zi": 155,
+      "▁še": 156,
+      "▁gin₂": 157,
+      "en": 158,
+      "tu": 159,
+      "a₃": 160,
+      "▁iti": 161,
+      "▁sa": 162,
+      "▁i₃": 163,
+      "▁dumu": 164,
+      "gu": 165,
+      "hu": 166,
+      "▁ni": 167,
+      "▁ma": 168,
+      "li": 169,
+      "geš₂": 170,
+      "su": 171,
+      "▁u₄": 172,
+      "nu": 173,
+      "še₃": 174,
+      "me": 175,
+      "šu": 176,
+      "▁nin": 177,
+      "▁udu": 178,
+      "maš": 179,
+      "g₂": 180,
+      "▁na": 181,
+      "la₂": 182,
+      "▁en": 183,
+      "▁gi": 184,
+      "kam": 185,
+      "uš": 186,
+      "▁e₂": 187,
+      "si": 188,
+      "▁kaš": 189,
+      "▁maš": 190,
+      "m₂": 191,
+      "▁6": 192,
+      "ib": 193,
+      "▁ninda": 194,
+      "▁zi": 195,
+      "gar": 196,
+      "niŋ": 197,
+      "niŋin": 198,
+      "▁šuniŋin": 199,
+      "ša": 200,
+      "nin": 201,
+      "išib": 202,
+      "₁₀": 203,
+      "išib₃": 204,
+      "▁ku": 205,
+      "▁kišib₃": 206,
+      "kal": 207,
+      "zu": 208,
+      "▁gu₄": 209,
+      "an": 210,
+      "za": 211,
+      "ra₂": 212,
+      "s₂": 213,
+      "e₂": 214,
+      "▁da": 215,
+      "di": 216,
+      "▁ša₃": 217,
+      "suen": 218,
+      "▁dub": 219,
+      "▁la₂": 220,
+      "▁ga": 221,
+      "gir": 222,
+      "▁ka": 223,
+      "l₂": 224,
+      "▁8": 225,
+      "▁7": 226,
+      "hul": 227,
+      "sar": 228,
+      "ru": 229,
+      "₁₁": 230,
+      "▁nu": 231,
+      "▁us₂": 232,
+      "ha": 233,
+      "▁maš₂": 234,
+      "▁guruš": 235,
+      "▁tu": 236,
+      "▁ugu": 237,
+      "ri₃": 238,
+      "▁giri₃": 239,
+      "▁sag": 240,
+      "iku": 241,
+      "em": 242,
+      "▁ugula": 243,
+      "sa₆": 244,
+      "ab": 245,
+      "▁ab": 246,
+      "▁niga": 247,
+      "zem": 248,
+      "sag": 249,
+      "bur": 250,
+      "ša₃": 251,
+      "▁sila₄": 252,
+      "bar": 253,
+      "pa": 254,
+      "▁su": 255,
+      "▁a₂": 256,
+      "bi₂": 257,
+      "mar": 258,
+      "din": 259,
+      "b₅": 260,
+      "nan": 261,
+      "▁an": 262,
+      "lil₂": 263,
+      "▁ezem": 264,
+      "▁naga": 265,
+      "šul": 266,
+      "▁geš": 267,
+      "▁nig₂": 268,
+      "am": 269,
+      "▁ma₂": 270,
+      "▁šum₂": 271,
+      "dab₅": 272,
+      "šara₂": 273,
+      "▁ge": 274,
+      "▁ku₃": 275,
+      "▁u₃": 276,
+      "du₃": 277,
+      "▁di": 278,
+      "gur": 279,
+      "▁zi₃": 280,
+      "gal₂": 281,
+      "mah": 282,
+      "i₃": 283,
+      "um": 284,
+      "dingir": 285,
+      "im": 286,
+      "ke": 287,
+      "r₂": 288,
+      "▁ša": 289,
+      "še": 290,
+      "me₂": 291,
+      "▁tug₂": 292,
+      "hun": 293,
+      "si₂": 294,
+      "ke₄": 295,
+      "▁geme₂": 296,
+      "ne": 297,
+      "ʾu": 298,
+      "da₃": 299,
+      "▁h": 300,
+      "li₂": 301,
+      "ba₆": 302,
+      "am₃": 303,
+      "▁in": 304,
+      "ga₂": 305,
+      "▁ensi₂": 306,
+      "lum": 307,
+      "d₂": 308,
+      "▁sag₁₀": 309,
+      "nanna": 310,
+      "▁za": 311,
+      "r₃": 312,
+      "u₂": 313,
+      "▁igi": 314,
+      "▁nam": 315,
+      "kur": 316,
+      "▁šeš": 317,
+      "gešʾu": 318,
+      "▁uri": 319,
+      "▁zu": 320,
+      "kkal": 321,
+      "nun": 322,
+      "▁u₂": 323,
+      "▁siki": 324,
+      "kuₓ": 325,
+      "DU": 326,
+      "bu": 327,
+      "mma": 328,
+      "▁sar": 329,
+      "bur₃": 330,
+      "eše₃": 331,
+      "▁p": 332,
+      "▁uri₅": 333,
+      "▁9": 334,
+      "ban": 335,
+      "d₃": 336,
+      "gi₄": 337,
+      "gin₂": 338,
+      "▁hu": 339,
+      "ina": 340,
+      "ur₄": 341,
+      "▁kuš": 342,
+      "tur": 343,
+      "du₁₀": 344,
+      "banda₃": 345,
+      "▁ab₂": 346,
+      "nus": 347,
+      "ku₃": 348,
+      "ta₂": 349,
+      "re": 350,
+      "ku₅": 351,
+      "lim": 352,
+      "▁amar": 353,
+      "ia₃": 354,
+      "ŋa": 355,
+      "du₁₁": 356,
+      "▁im": 357,
+      "inana": 358,
+      "▁gub": 359,
+      "du₈": 360,
+      "▁ha": 361,
+      "utu": 362,
+      "▁ta": 363,
+      "▁gu₂": 364,
+      "nigin₂": 365,
+      "▁dug": 366,
+      "▁kur": 367,
+      "▁ud": 368,
+      "de₃": 369,
+      "▁gun₂": 370,
+      "dub": 371,
+      "▁engar": 372,
+      "tum": 373,
+      "mun": 374,
+      "▁u₈": 375,
+      "▁la": 376,
+      "gu₇": 377,
+      "ge": 378,
+      "bbar": 379,
+      "dumu": 380,
+      "▁bar": 381,
+      "b₂": 382,
+      "▁me": 383,
+      "h₂": 384,
+      "▁ku₆": 385,
+      "▁bala": 386,
+      "▁sukkal": 387,
+      "▁gal": 388,
+      "g₄": 389,
+      "amar": 390,
+      "tug₂": 391,
+      "uz": 392,
+      "zal": 393,
+      "kim": 394,
+      "mi": 395,
+      "us₂": 396,
+      "nig₂": 397,
+      "bin": 398,
+      "▁er": 399,
+      "eš₃": 400,
+      "▁kas": 401,
+      "▁mar": 402,
+      "▁al": 403,
+      "mu₂": 404,
+      "▁sa₂": 405,
+      "eš₂": 406,
+      "▁maškim": 407,
+      "▁dabin": 408,
+      "▁ud₅": 409,
+      "▁uš": 410,
+      "sag₁₁": 411,
+      "▁bi₂": 412,
+      "▁he₂": 413,
+      "ši": 414,
+      "nigar": 415,
+      "▁dam": 416,
+      "uzur₄": 417,
+      "▁hi": 418,
+      "▁inim": 419,
+      "▁puzur₄": 420,
+      "▁bad₃": 421,
+      "g₁": 422,
+      "gin": 423,
+      "babbar": 424,
+      "▁dingir": 425,
+      "▁ši": 426,
+      "tuku": 427,
+      "lu₂": 428,
+      "▁kin": 429,
+      "nanše": 430,
+      "▁umma": 431,
+      "▁kal": 432,
+      "lam": 433,
+      "▁zu₂": 434,
+      "▁erin₂": 435,
+      "gar₃": 436,
+      "iškur": 437,
+      "gir₂": 438,
+      "mes": 439,
+      "lu₅": 440,
+      "nga": 441,
+      "▁sipa": 442,
+      "z₂": 443,
+      "l₃": 444,
+      "numun": 445,
+      "dam": 446,
+      "▁ziz₂": 447,
+      "uš₂": 448,
+      "▁ad": 449,
+      "dim": 450,
+      "▁dida": 451,
+      "un": 452,
+      "ig": 453,
+      "▁unu": 454,
+      "hal": 455,
+      "▁ak": 456,
+      "gigir": 457,
+      "ur₂": 458,
+      "▁šuš": 459,
+      "al": 460,
+      "g₃": 461,
+      "munus": 462,
+      "dim₂": 463,
+      "▁iri": 464,
+      "lamma": 465,
+      "▁gal₂": 466,
+      "gid₂": 467,
+      "num₂": 468,
+      "si₄": 469,
+      "gur₈": 470,
+      "▁mah": 471,
+      "pa₃": 472,
+      "▁anše": 473,
+      "šubur": 474,
+      "▁diri": 475,
+      "sig₁": 476,
+      "▁eri": 477,
+      "li₉": 478,
+      "▁šab": 479,
+      "▁munus": 480,
+      "ne₂": 481,
+      "▁šabra": 482,
+      "▁id": 483,
+      "gur₂": 484,
+      "haldim": 485,
+      "uda": 486,
+      "ŋ₂": 487,
+      "▁i₇": 488,
+      "dar": 489,
+      "gin₇": 490,
+      "▁eridu": 491,
+      "ga₃": 492,
+      "šum₂": 493,
+      "gan": 494,
+      "de₂": 495,
+      "▁pa": 496,
+      "ar": 497,
+      "▁gid₂": 498,
+      "limmu₂": 499,
+      "ak": 500,
+      "▁ŋ": 501,
+      "lam₂": 502,
+      "har": 503,
+      "▁ti": 504,
+      "ri₆": 505,
+      "nita₂": 506,
+      "▁limmu₂": 507,
+      "šen": 508,
+      "pi": 509,
+      "▁gaba": 510,
+      "gu₂": 511,
+      "u₁₀": 512,
+      "šar₂": 513,
+      "lugal": 514,
+      "ŋu₁₀": 515,
+      "▁nita": 516,
+      "kiri₆": 517,
+      "ŋa₂": 518,
+      "gu₄": 519,
+      "▁nita₂": 520,
+      "igi": 521,
+      "aš₂": 522,
+      "▁ub": 523,
+      "siki": 524,
+      "ri₂": 525,
+      "▁sanga": 526,
+      "bala": 527,
+      "▁z": 528,
+      "▁šuš₃": 529,
+      "▁ug₃": 530,
+      "▁kas₄": 531,
+      "₁₈": 532,
+      "de": 533,
+      "u₃": 534,
+      "▁gen": 535,
+      "iri": 536,
+      "ur₃": 537,
+      "bru": 538,
+      "sa₂": 539,
+      "▁kuš₃": 540,
+      "▁aga₃": 541,
+      "▁ama": 542,
+      "▁guru": 543,
+      "₁₇": 544,
+      "kar": 545,
+      "▁bi": 546,
+      "▁nibru": 547,
+      "₁₃": 548,
+      "kuš": 549,
+      "▁ga₂": 550,
+      "▁tur": 551,
+      "▁eš": 552,
+      "ezem": 553,
+      "▁kar": 554,
+      "iti": 555,
+      "ru₂": 556,
+      "a₂": 557,
+      "▁tu₇": 558,
+      "▁ne": 559,
+      "kur₂": 560,
+      "hur": 561,
+      "▁du₁₁": 562,
+      "▁aš": 563,
+      "▁m": 564,
+      "sal": 565,
+      "du₇": 566,
+      "e₁₁": 567,
+      "ku₆": 568,
+      "ir": 569,
+      "unu": 570,
+      "pin": 571,
+      "ag₂": 572,
+      "tar": 573,
+      "inim": 574,
+      "sag₁₀": 575,
+      "ar₃": 576,
+      "ŋal₂": 577,
+      "de₆": 578,
+      "sir₂": 579,
+      "▁ur₅": 580,
+      "nu₄": 581,
+      "▁sagi": 582,
+      "ma₂": 583,
+      "lla": 584,
+      "h₅": 585,
+      "hi": 586,
+      "▁li": 587,
+      "guz": 588,
+      "ka₉": 589,
+      "▁zabar": 590,
+      "▁munu₄": 591,
+      "il₂": 592,
+      "nam": 593,
+      "unu₆": 594,
+      "he₂": 595,
+      "▁šuku": 596,
+      "nir": 597,
+      "▁ku₅": 598,
+      "gaš": 599,
+      "alim": 600,
+      "uruda": 601,
+      "ušda": 602,
+      "sig₁₅": 603,
+      "tukul": 604,
+      "▁ge₆": 605,
+      "▁duh": 606,
+      "be": 607,
+      "▁kurušda": 608,
+      "bil": 609,
+      "▁za₃": 610,
+      "wa": 611,
+      "▁sig₄": 612,
+      "te": 613,
+      "tin": 614,
+      "muhaldim": 615,
+      "▁e₃": 616,
+      "▁kir": 617,
+      "tara": 618,
+      "gen": 619,
+      "▁gir₂": 620,
+      "ištara": 621,
+      "er": 622,
+      "ištaran": 623,
+      "▁kir₁₁": 624,
+      "▁am": 625,
+      "▁dur₃": 626,
+      "ub": 627,
+      "▁min": 628,
+      "bu₃": 629,
+      "burʾu": 630,
+      "tim": 631,
+      "mušen": 632,
+      "▁ri": 633,
+      "▁niŋ₂": 634,
+      "▁sahar": 635,
+      "tab": 636,
+      "▁gukkal": 637,
+      "skur₂": 638,
+      "ru₅": 639,
+      "₁₂": 640,
+      "ken": 641,
+      "uh₂": 642,
+      "▁ib₂": 643,
+      "▁šitim": 644,
+      "▁gu₃": 645,
+      "▁gar": 646,
+      "▁uš₂": 647,
+      "▁eša": 648,
+      "▁šag₄": 649,
+      "▁mah₂": 650,
+      "▁du₆": 651,
+      "▁saŋ": 652,
+      "gub": 653,
+      "ri₁₇": 654,
+      "▁kug": 655,
+      "▁lagaš": 656,
+      "eš₁₈": 657,
+      "kken": 658,
+      "gaba": 659,
+      "u₄": 660,
+      "anna": 661,
+      "šar": 662,
+      "le": 663,
+      "su₇": 664,
+      "▁eš₃": 665,
+      "▁kun": 666,
+      "kin": 667,
+      "ŋar": 668,
+      "▁esir₂": 669,
+      "▁gu₇": 670,
+      "▁ugu₂": 671,
+      "▁du₁₀": 672,
+      "san": 673,
+      "▁du₃": 674,
+      "▁siskur₂": 675,
+      "mug": 676,
+      "▁ra": 677,
+      "inanna": 678,
+      "▁pi": 679,
+      "sig₁₇": 680,
+      "▁gudu": 681,
+      "bir": 682,
+      "▁gi₄": 683,
+      "▁nindan": 684,
+      "h₃": 685,
+      "▁muhaldim": 686,
+      "ŋiš": 687,
+      "be₂": 688,
+      "lal₃": 689,
+      "▁se": 690,
+      "▁bur": 691,
+      "▁gudu₄": 692,
+      "gada": 693,
+      "▁du₈": 694,
+      "▁nesag": 695,
+      "▁babbar": 696,
+      "su₂": 697,
+      "▁ni₂": 698,
+      "▁pa₄": 699,
+      "▁gig": 700,
+      "dul": 701,
+      "▁dal": 702,
+      "ul": 703,
+      "lah₅": 704,
+      "▁šah₂": 705,
+      "▁uruda": 706,
+      "at": 707,
+      "abzu": 708,
+      "▁mi": 709,
+      "šub": 710,
+      "▁su₃": 711,
+      "▁ur₃": 712,
+      "IN": 713,
+      "di₃": 714,
+      "saŋ": 715,
+      "edin": 716,
+      "mgir": 717,
+      "id": 718,
+      "udu": 719,
+      "▁um": 720,
+      "▁guru₇": 721,
+      "apin": 722,
+      "dli": 723,
+      "▁ur₂": 724,
+      "▁gurum₂": 725,
+      "▁hun": 726,
+      "lagar": 727,
+      "ge₆": 728,
+      "anše": 729,
+      "sila": 730,
+      "▁didli": 731,
+      "▁elam": 732,
+      "na₄": 733,
+      "giri₁₇": 734,
+      "▁šimaš": 735,
+      "er₃": 736,
+      "▁šušin": 737,
+      "▁pisan": 738,
+      "zla": 739,
+      "sal₄": 740,
+      "urₓ": 741,
+      "▁gada": 742,
+      "gab": 743,
+      "dug₄": 744,
+      "rum": 745,
+      "▁šimašgi": 746,
+      "▁gala": 747,
+      "r₄": 748,
+      "kun": 749,
+      "llat": 750,
+      "duru₅": 751,
+      "▁tab": 752,
+      "▁kaskal": 753,
+      "▁ses": 754,
+      "šum": 755,
+      "eb": 756,
+      "▁unu₃": 757,
+      "ku₄": 758,
+      "▁ara": 759,
+      "▁šim": 760,
+      "illat": 761,
+      "▁nar": 762,
+      "▁zid": 763,
+      "sukkal": 764,
+      "dara": 765,
+      "▁sa₆": 766,
+      "gur₁₁": 767,
+      "▁tum": 768,
+      "tum₂": 769,
+      "sa₁₀": 770,
+      "su₃": 771,
+      "ur₅": 772,
+      "ama": 773,
+      "▁ašgab": 774,
+      "kas": 775,
+      "ib₂": 776,
+      "▁šar₃": 777,
+      "ŋir": 778,
+      "ad": 779,
+      "dun": 780,
+      "uru": 781,
+      "▁apin": 782,
+      "▁kur₂": 783,
+      "n₃": 784,
+      "qar": 785,
+      "zlag₂": 786,
+      "▁ka₅": 787,
+      "gara₂": 788,
+      "ppi": 789,
+      "ŠE": 790,
+      "dug": 791,
+      "▁dab₅": 792,
+      "gul": 793,
+      "▁kab₂": 794,
+      "iq": 795,
+      "▁hul": 796,
+      "▁be": 797,
+      "▁ša₁₃": 798,
+      "▁bara₂": 799,
+      "muš": 800,
+      "du₅": 801,
+      "▁muš": 802,
+      "▁nagar": 803,
+      "nesag": 804,
+      "tuš": 805,
+      "▁gibil": 806,
+      "▁hur": 807,
+      "▁ra₂": 808,
+      "ru₁₃": 809,
+      "▁sa₁₀": 810,
+      "eš₅": 811,
+      "par₄": 812,
+      "eri": 813,
+      "▁nimgir": 814,
+      "▁bappi": 815,
+      "ašhur": 816,
+      "▁sig₁": 817,
+      "▁sig": 818,
+      "il": 819,
+      "am₆": 820,
+      "dul₅": 821,
+      "LI": 822,
+      "bad₃": 823,
+      "GI": 824,
+      "sig": 825,
+      "▁diŋir": 826,
+      "nita": 827,
+      "▁ga₆": 828,
+      "LIL": 829,
+      "▁ŋi": 830,
+      "▁keš₂": 831,
+      "▁sig₁₅": 832,
+      "GA": 833,
+      "��ar": 834,
+      "šah₂": 835,
+      "sud₃": 836,
+      "kke": 837,
+      "kken₂": 838,
+      "lunga": 839,
+      "guru": 840,
+      "▁še₃": 841,
+      "gi₇": 842,
+      "▁libir": 843,
+      "▁sa₁₂": 844,
+      "hašhur": 845,
+      "▁il₂": 846,
+      "▁mušen": 847,
+      "du₆": 848,
+      "r₇": 849,
+      "kunga₂": 850,
+      "k₄": 851,
+      "šeš": 852,
+      "dan": 853,
+      "kišib₃": 854,
+      "geštin": 855,
+      "iši": 856,
+      "▁gab₂": 857,
+      "▁dusu₂": 858,
+      "▁kala": 859,
+      "▁nun": 860,
+      "ah": 861,
+      "▁zah₃": 862,
+      "kuš₂": 863,
+      "luh": 864,
+      "▁bu": 865,
+      "sikil": 866,
+      "▁arad₂": 867,
+      "▁numun": 868,
+      "gam": 869,
+      "teš₂": 870,
+      "▁er₃": 871,
+      "azlag₂": 872,
+      "kum": 873,
+      "iši₁₇": 874,
+      "tam": 875,
+      "▁uru": 876,
+      "KIN": 877,
+      "▁bu₃": 878,
+      "dab": 879,
+      "▁geštin": 880,
+      "▁nigar": 881,
+      "guru₇": 882,
+      "šu₂": 883,
+      "se₃": 884,
+      "sur": 885,
+      "▁tuš": 886,
+      "sila₃": 887,
+      "dara₃": 888,
+      "dur": 889,
+      "▁mun": 890,
+      "▁ŋiš": 891,
+      "▁eme": 892,
+      "▁sumun": 893,
+      "▁simug": 894,
+      "huš": 895,
+      "mur": 896,
+      "▁had₂": 897,
+      "ŋ₃": 898,
+      "pi₄": 899,
+      "▁am₃": 900,
+      "▁de₆": 901,
+      "▁kalam": 902,
+      "▁šar": 903,
+      "IGI": 904,
+      "▁šul": 905,
+      "kiši₁₇": 906,
+      "tag": 907,
+      "kar₂": 908,
+      "ŋeš": 909,
+      "peš₃": 910,
+      "▁mur": 911,
+      "nu₂": 912,
+      "▁tir": 913,
+      "▁ku₄": 914,
+      "▁dug₄": 915,
+      "sun₂": 916,
+      "▁dara": 917,
+      "▁muru₁₃": 918,
+      "sakar": 919,
+      "babbar₂": 920,
+      "bara₂": 921,
+      "urta": 922,
+      "erim₂": 923,
+      "▁babbar₂": 924,
+      "silim": 925,
+      "▁nin₉": 926,
+      "gibil": 927,
+      "ninda": 928,
+      "nam₂": 929,
+      "ze₂": 930,
+      "gaz": 931,
+      "dul₃": 932,
+      "mud": 933,
+      "gab₂": 934,
+      "▁gurdub": 935,
+      "til₃": 936,
+      "tum₃": 937,
+      "kab": 938,
+      "saba": 939,
+      "▁aŋ₂": 940,
+      "ŋi": 941,
+      "šim": 942,
+      "▁kikken₂": 943,
+      "p₄": 944,
+      "▁uz": 945,
+      "▁gir": 946,
+      "kup₄": 947,
+      "su₄": 948,
+      "▁har": 949,
+      "▁kuₓ": 950,
+      "▁duru₅": 951,
+      "▁eš₂": 952,
+      "nisaba": 953,
+      "nag": 954,
+      "gina": 955,
+      "sin₂": 956,
+      "suh₅": 957,
+      "▁šubur": 958,
+      "▁šar₂": 959,
+      "za₃": 960,
+      "dur₂": 961,
+      "gin₃": 962,
+      "▁abzu": 963,
+      "▁mes": 964,
+      "▁dara₄": 965,
+      "keš₂": 966,
+      "▁zar₃": 967,
+      "▁šagina": 968,
+      "kud": 969,
+      "ninnu": 970,
+      "we": 971,
+      "▁ze₂": 972,
+      "▁šeg": 973,
+      "LA": 974,
+      "▁tar": 975,
+      "kas₄": 976,
+      "▁sal": 977,
+      "▁dug₃": 978,
+      "gin₆": 979,
+      "▁utu": 980,
+      "▁nir": 981,
+      "▁tak₄": 982,
+      "hen": 983,
+      "mun₂": 984,
+      "ug": 985,
+      "▁gun₃": 986,
+      "▁imga": 987,
+      "hendur": 988,
+      "▁ziₓ": 989,
+      "▁sag₉": 990,
+      "▁ŋeš": 991,
+      "▁dagal": 992,
+      "gil": 993,
+      "▁kalag": 994,
+      "til": 995,
+      "▁imgaga₃": 996,
+      "₁₄": 997,
+      "kikken": 998,
+      "▁lum": 999,
+      "šuba": 1000,
+      "▁nigin₂": 1001,
+      "▁ag": 1002,
+      "▁gan": 1003,
+      "▁egir": 1004,
+      "▁tuku": 1005,
+      "▁tag": 1006,
+      "▁kam": 1007,
+      "dag": 1008,
+      "▁šeg₉": 1009,
+      "₁₆": 1010,
+      "▁ar₃": 1011,
+      "gur₄": 1012,
+      "tir": 1013,
+      "qu₂": 1014,
+      "ŋir₂": 1015,
+      "gu₃": 1016,
+      "▁na₄": 1017,
+      "▁tul₂": 1018,
+      "▁iš": 1019,
+      "kaskal": 1020,
+      "▁ib": 1021,
+      "▁zag": 1022,
+      "aŋ₂": 1023
+    },
+    "merges": [
+      "i š",
+      "d iš",
+      "b a",
+      "▁ 1",
+      "g a",
+      "▁ u",
+      "▁ g",
+      "l a",
+      "i n",
+      "d u",
+      "m u",
+      "▁ s",
+      "▁ š",
+      "a š",
+      "▁ k",
+      "▁ 2",
+      "l u",
+      "d a",
+      "u r",
+      "▁ n",
+      "▁ lu",
+      "▁ i",
+      "▁ a",
+      "▁s i",
+      "▁ 3",
+      "e š",
+      "r i",
+      "m a",
+      "ga l",
+      "▁ e",
+      "▁si la",
+      "t a",
+      "▁ du",
+      "▁ mu",
+      "▁ ba",
+      "k i",
+      "▁ 5",
+      "n a",
+      "▁š u",
+      "▁u r",
+      "▁sila ₃",
+      "n ₂",
+      "k a",
+      "n i",
+      "t i",
+      "ba n₂",
+      "▁ 4",
+      "r a",
+      "g eš",
+      "s a",
+      "g i",
+      "▁lu gal",
+      "▁g ur",
+      "e ₃",
+      "in ₂",
+      "ri g",
+      "▁k i",
+      "ba rig",
+      "b i",
+      "▁lu ₂",
+      "▁g u",
+      "k u",
+      "z i",
+      "▁š e",
+      "▁g in₂",
+      "e n",
+      "t u",
+      "a ₃",
+      "▁i ti",
+      "▁s a",
+      "▁i ₃",
+      "▁du mu",
+      "g u",
+      "h u",
+      "▁n i",
+      "▁ ma",
+      "l i",
+      "geš ₂",
+      "s u",
+      "▁u ₄",
+      "n u",
+      "š e₃",
+      "m e",
+      "š u",
+      "▁n in",
+      "▁u du",
+      "m aš",
+      "g ₂",
+      "▁n a",
+      "la ₂",
+      "▁e n",
+      "▁g i",
+      "ka m",
+      "u š",
+      "▁e ₂",
+      "s i",
+      "▁k aš",
+      "▁ maš",
+      "m ₂",
+      "▁ 6",
+      "i b",
+      "▁nin da",
+      "▁ zi",
+      "ga r",
+      "ni ŋ",
+      "niŋ in",
+      "▁šu niŋin",
+      "š a",
+      "n in",
+      "iš ib",
+      "₁ ₀",
+      "išib ₃",
+      "▁k u",
+      "▁k išib₃",
+      "ka l",
+      "z u",
+      "▁gu ₄",
+      "a n",
+      "z a",
+      "ra ₂",
+      "s ₂",
+      "e ₂",
+      "▁ da",
+      "d i",
+      "▁š a₃",
+      "su en",
+      "▁du b",
+      "▁ la₂",
+      "▁ ga",
+      "gi r",
+      "▁k a",
+      "l ₂",
+      "▁ 8",
+      "▁ 7",
+      "hu l",
+      "sa r",
+      "r u",
+      "₁ ₁",
+      "▁n u",
+      "▁u s₂",
+      "h a",
+      "▁maš ₂",
+      "▁gur uš",
+      "▁ tu",
+      "▁u gu",
+      "ri ₃",
+      "▁gi ri₃",
+      "▁sa g",
+      "i ku",
+      "e m",
+      "▁ugu la",
+      "sa ₆",
+      "a b",
+      "▁a b",
+      "▁ni ga",
+      "z em",
+      "sa g",
+      "b ur",
+      "š a₃",
+      "▁sila ₄",
+      "ba r",
+      "p a",
+      "▁s u",
+      "▁a ₂",
+      "bi ₂",
+      "ma r",
+      "d in",
+      "b ₅",
+      "na n",
+      "▁a n",
+      "li l₂",
+      "▁e zem",
+      "▁na ga",
+      "šu l",
+      "▁g eš",
+      "▁ni g₂",
+      "a m",
+      "▁ma ₂",
+      "▁šu m₂",
+      "da b₅",
+      "ša ra₂",
+      "▁g e",
+      "▁ku ₃",
+      "▁u ₃",
+      "du ₃",
+      "▁ di",
+      "g ur",
+      "▁zi ₃",
+      "gal ₂",
+      "ma h",
+      "i ₃",
+      "u m",
+      "din gir",
+      "i m",
+      "k e",
+      "r ₂",
+      "▁š a",
+      "š e",
+      "me ₂",
+      "▁tu g₂",
+      "hu n",
+      "si ₂",
+      "ke ₄",
+      "▁ge me₂",
+      "n e",
+      "ʾ u",
+      "da ₃",
+      "▁ h",
+      "li ₂",
+      "ba ₆",
+      "am ₃",
+      "▁ in",
+      "ga ₂",
+      "▁en si₂",
+      "lu m",
+      "d ₂",
+      "▁sag ₁₀",
+      "nan na",
+      "▁ za",
+      "r ₃",
+      "u ₂",
+      "▁i gi",
+      "▁na m",
+      "k ur",
+      "▁š eš",
+      "geš ʾu",
+      "▁u ri",
+      "▁ zu",
+      "k kal",
+      "nu n",
+      "▁u ₂",
+      "▁si ki",
+      "ku ₓ",
+      "D U",
+      "b u",
+      "m ma",
+      "▁sa r",
+      "bur ₃",
+      "eš e₃",
+      "▁ p",
+      "▁uri ₅",
+      "▁ 9",
+      "ba n",
+      "d ₃",
+      "gi ₄",
+      "g in₂",
+      "▁ hu",
+      "in a",
+      "ur ₄",
+      "▁k uš",
+      "t ur",
+      "du ₁₀",
+      "ban da₃",
+      "▁ab ₂",
+      "nu s",
+      "ku ₃",
+      "ta ₂",
+      "r e",
+      "ku ₅",
+      "li m",
+      "▁a mar",
+      "i a₃",
+      "ŋ a",
+      "du ₁₁",
+      "▁i m",
+      "ina na",
+      "▁gu b",
+      "du ₈",
+      "▁ ha",
+      "u tu",
+      "▁ ta",
+      "▁gu ₂",
+      "ni gin₂",
+      "▁du g",
+      "▁k ur",
+      "▁u d",
+      "d e₃",
+      "▁gu n₂",
+      "du b",
+      "▁en gar",
+      "tu m",
+      "mu n",
+      "▁u ₈",
+      "▁ la",
+      "gu ₇",
+      "g e",
+      "b bar",
+      "du mu",
+      "▁ba r",
+      "b ₂",
+      "▁ me",
+      "h ₂",
+      "▁ku ₆",
+      "▁ba la",
+      "▁su kkal",
+      "▁ gal",
+      "g ₄",
+      "a mar",
+      "tu g₂",
+      "u z",
+      "za l",
+      "ki m",
+      "m i",
+      "u s₂",
+      "ni g₂",
+      "b in",
+      "▁e r",
+      "eš ₃",
+      "▁ka s",
+      "▁ma r",
+      "▁a l",
+      "mu ₂",
+      "▁sa ₂",
+      "eš ₂",
+      "▁maš kim",
+      "▁da bin",
+      "▁ud ₅",
+      "▁u š",
+      "sag ₁₁",
+      "▁ bi₂",
+      "▁h e₂",
+      "š i",
+      "ni gar",
+      "▁da m",
+      "uz ur₄",
+      "▁h i",
+      "▁in im",
+      "▁p uzur₄",
+      "▁ba d₃",
+      "g ₁",
+      "g in",
+      "ba bbar",
+      "▁ dingir",
+      "▁š i",
+      "tu ku",
+      "lu ₂",
+      "▁k in",
+      "nan še",
+      "▁u mma",
+      "▁ka l",
+      "la m",
+      "▁zu ₂",
+      "▁er in₂",
+      "gar ₃",
+      "iš kur",
+      "gir ₂",
+      "me s",
+      "lu ₅",
+      "n ga",
+      "▁si pa",
+      "z ₂",
+      "l ₃",
+      "nu mun",
+      "da m",
+      "▁zi z₂",
+      "uš ₂",
+      "▁a d",
+      "di m",
+      "▁di da",
+      "u n",
+      "i g",
+      "▁u nu",
+      "ha l",
+      "▁a k",
+      "gi gir",
+      "ur ₂",
+      "▁šu š",
+      "a l",
+      "g ₃",
+      "mu nus",
+      "di m₂",
+      "▁i ri",
+      "la mma",
+      "▁ gal₂",
+      "gi d₂",
+      "nu m₂",
+      "si ₄",
+      "gur ₈",
+      "▁ma h",
+      "p a₃",
+      "▁an še",
+      "šu bur",
+      "▁di ri",
+      "si g₁",
+      "▁e ri",
+      "li ₉",
+      "▁š ab",
+      "▁mu nus",
+      "n e₂",
+      "▁šab ra",
+      "▁i d",
+      "gur ₂",
+      "hal dim",
+      "u da",
+      "ŋ ₂",
+      "▁i ₇",
+      "da r",
+      "gin ₇",
+      "▁eri du",
+      "ga ₃",
+      "šu m₂",
+      "ga n",
+      "d e₂",
+      "▁ pa",
+      "a r",
+      "▁gi d₂",
+      "lim mu₂",
+      "a k",
+      "▁ ŋ",
+      "la m₂",
+      "ha r",
+      "▁ ti",
+      "ri ₆",
+      "ni ta₂",
+      "▁ limmu₂",
+      "š en",
+      "p i",
+      "▁ga ba",
+      "gu ₂",
+      "u ₁₀",
+      "ša r₂",
+      "lu gal",
+      "ŋ u₁₀",
+      "▁ni ta",
+      "ki ri₆",
+      "ŋa ₂",
+      "gu ₄",
+      "▁ni ta₂",
+      "i gi",
+      "aš ₂",
+      "▁u b",
+      "si ki",
+      "ri ₂",
+      "▁sa nga",
+      "ba la",
+      "▁ z",
+      "▁šuš ₃",
+      "▁u g₃",
+      "▁kas ₄",
+      "₁ ₈",
+      "d e",
+      "u ₃",
+      "▁g en",
+      "i ri",
+      "ur ₃",
+      "b ru",
+      "sa ₂",
+      "▁kuš ₃",
+      "▁a ga₃",
+      "▁a ma",
+      "▁gur u",
+      "₁ ₇",
+      "ka r",
+      "▁ bi",
+      "▁ni bru",
+      "₁ ₃",
+      "ku š",
+      "▁ga ₂",
+      "▁ tur",
+      "▁ eš",
+      "e zem",
+      "▁ka r",
+      "i ti",
+      "ru ₂",
+      "a ₂",
+      "▁tu ₇",
+      "▁n e",
+      "kur ₂",
+      "h ur",
+      "▁du ₁₁",
+      "▁ aš",
+      "▁ m",
+      "sa l",
+      "du ₇",
+      "e ₁₁",
+      "ku ₆",
+      "i r",
+      "u nu",
+      "p in",
+      "a g₂",
+      "ta r",
+      "in im",
+      "sag ₁₀",
+      "a r₃",
+      "ŋa l₂",
+      "de ₆",
+      "si r₂",
+      "▁ur ₅",
+      "nu ₄",
+      "▁sa gi",
+      "ma ₂",
+      "l la",
+      "h ₅",
+      "h i",
+      "▁ li",
+      "gu z",
+      "ka ₉",
+      "▁za bar",
+      "▁mu nu₄",
+      "i l₂",
+      "na m",
+      "unu ₆",
+      "h e₂",
+      "▁šu ku",
+      "ni r",
+      "▁ku ₅",
+      "ga š",
+      "a lim",
+      "ur uda",
+      "uš da",
+      "sig₁ ₅",
+      "tuku l",
+      "▁ge ₆",
+      "▁du h",
+      "b e",
+      "▁kur ušda",
+      "bi l",
+      "▁z a₃",
+      "w a",
+      "▁si g₄",
+      "t e",
+      "t in",
+      "mu haldim",
+      "▁e ₃",
+      "▁ki r",
+      "ta ra",
+      "g en",
+      "▁gi r₂",
+      "iš tara",
+      "e r",
+      "ištara n",
+      "▁kir ₁₁",
+      "▁a m",
+      "▁du r₃",
+      "u b",
+      "▁m in",
+      "bu ₃",
+      "bur ʾu",
+      "ti m",
+      "mu šen",
+      "▁ ri",
+      "▁ni ŋ₂",
+      "▁sa har",
+      "ta b",
+      "▁gu kkal",
+      "s kur₂",
+      "ru ₅",
+      "₁ ₂",
+      "k en",
+      "u h₂",
+      "▁i b₂",
+      "▁ši tim",
+      "▁gu ₃",
+      "▁ gar",
+      "▁uš ₂",
+      "▁eš a",
+      "▁ša g₄",
+      "▁ma h₂",
+      "▁du ₆",
+      "▁sa ŋ",
+      "gu b",
+      "ri ₁₇",
+      "▁ku g",
+      "▁la gaš",
+      "eš ₁₈",
+      "k ken",
+      "ga ba",
+      "u ₄",
+      "an na",
+      "ša r",
+      "l e",
+      "su ₇",
+      "▁ eš₃",
+      "▁ku n",
+      "k in",
+      "ŋa r",
+      "▁e sir₂",
+      "▁gu ₇",
+      "▁ugu ₂",
+      "▁du ₁₀",
+      "sa n",
+      "▁du ₃",
+      "▁si skur₂",
+      "mu g",
+      "▁ ra",
+      "in anna",
+      "▁p i",
+      "sig₁ ₇",
+      "▁gu du",
+      "bi r",
+      "▁gi ₄",
+      "▁ninda n",
+      "h ₃",
+      "▁mu haldim",
+      "ŋ iš",
+      "b e₂",
+      "la l₃",
+      "▁s e",
+      "▁ bur",
+      "▁gudu ₄",
+      "ga da",
+      "▁du ₈",
+      "▁ne sag",
+      "▁ba bbar",
+      "su ₂",
+      "▁ni ₂",
+      "▁pa ₄",
+      "▁gi g",
+      "du l",
+      "▁da l",
+      "u l",
+      "la h₅",
+      "▁ša h₂",
+      "▁ur uda",
+      "a t",
+      "ab zu",
+      "▁ mi",
+      "šu b",
+      "▁su ₃",
+      "▁ur ₃",
+      "I N",
+      "di ₃",
+      "sa ŋ",
+      "e din",
+      "m gir",
+      "i d",
+      "u du",
+      "▁u m",
+      "▁guru ₇",
+      "a pin",
+      "d li",
+      "▁ur ₂",
+      "▁guru m₂",
+      "▁ hun",
+      "la gar",
+      "ge ₆",
+      "an še",
+      "si la",
+      "▁di dli",
+      "▁e lam",
+      "na ₄",
+      "gi ri₁₇",
+      "▁ši maš",
+      "e r₃",
+      "▁šuš in",
+      "▁pi san",
+      "z la",
+      "sal ₄",
+      "ur ₓ",
+      "▁ga da",
+      "ga b",
+      "du g₄",
+      "ru m",
+      "▁šimaš gi",
+      "▁ga la",
+      "r ₄",
+      "ku n",
+      "lla t",
+      "du ru₅",
+      "▁ta b",
+      "▁kas kal",
+      "▁se s",
+      "šu m",
+      "e b",
+      "▁unu ₃",
+      "ku ₄",
+      "▁a ra",
+      "▁š im",
+      "i llat",
+      "▁na r",
+      "▁zi d",
+      "su kkal",
+      "da ra",
+      "▁sa ₆",
+      "gur ₁₁",
+      "▁tu m",
+      "tu m₂",
+      "sa ₁₀",
+      "su ₃",
+      "ur ₅",
+      "a ma",
+      "▁aš gab",
+      "ka s",
+      "ib ₂",
+      "▁ša r₃",
+      "ŋ ir",
+      "a d",
+      "du n",
+      "ur u",
+      "▁a pin",
+      "▁kur ₂",
+      "n ₃",
+      "q ar",
+      "zla g₂",
+      "▁ka ₅",
+      "ga ra₂",
+      "p pi",
+      "Š E",
+      "du g",
+      "▁da b₅",
+      "gu l",
+      "▁ka b₂",
+      "i q",
+      "▁ hul",
+      "▁ be",
+      "▁ša ₁₃",
+      "▁ba ra₂",
+      "mu š",
+      "du ₅",
+      "▁mu š",
+      "▁na gar",
+      "ne sag",
+      "tu š",
+      "▁gi bil",
+      "▁h ur",
+      "▁ ra₂",
+      "ru ₁₃",
+      "▁sa ₁₀",
+      "eš ₅",
+      "pa r₄",
+      "e ri",
+      "▁ni mgir",
+      "▁ba ppi",
+      "aš hur",
+      "▁si g₁",
+      "▁si g",
+      "i l",
+      "am ₆",
+      "dul ₅",
+      "L I",
+      "ba d₃",
+      "G I",
+      "si g",
+      "▁di ŋir",
+      "ni ta",
+      "▁ga ₆",
+      "LI L",
+      "▁ŋ i",
+      "▁k eš₂",
+      "▁sig₁ ₅",
+      "G A",
+      "▁a r",
+      "ša h₂",
+      "su d₃",
+      "k ke",
+      "kke n₂",
+      "lu nga",
+      "gur u",
+      "▁š e₃",
+      "gi ₇",
+      "▁li bir",
+      "▁sa ₁₂",
+      "h ašhur",
+      "▁i l₂",
+      "▁mu šen",
+      "du ₆",
+      "r ₇",
+      "kun ga₂",
+      "k ₄",
+      "š eš",
+      "da n",
+      "k išib₃",
+      "geš tin",
+      "iš i",
+      "▁ga b₂",
+      "▁du su₂",
+      "▁ka la",
+      "▁nu n",
+      "a h",
+      "▁za h₃",
+      "kuš ₂",
+      "lu h",
+      "▁ bu",
+      "siki l",
+      "▁ara d₂",
+      "▁nu mun",
+      "ga m",
+      "t eš₂",
+      "▁e r₃",
+      "a zlag₂",
+      "ku m",
+      "iši ₁₇",
+      "ta m",
+      "▁ur u",
+      "K IN",
+      "▁ bu₃",
+      "da b",
+      "▁geš tin",
+      "▁ni gar",
+      "guru ₇",
+      "šu ₂",
+      "s e₃",
+      "s ur",
+      "▁tu š",
+      "sila ₃",
+      "dara ₃",
+      "du r",
+      "▁mu n",
+      "▁ŋ iš",
+      "▁e me",
+      "▁su mun",
+      "▁si mug",
+      "hu š",
+      "mu r",
+      "▁ha d₂",
+      "ŋ ₃",
+      "pi ₄",
+      "▁am ₃",
+      "▁ de₆",
+      "▁ka lam",
+      "▁ša r",
+      "I GI",
+      "▁šu l",
+      "k iši₁₇",
+      "ta g",
+      "ka r₂",
+      "ŋ eš",
+      "p eš₃",
+      "▁mu r",
+      "nu ₂",
+      "▁ti r",
+      "▁ku ₄",
+      "▁dug ₄",
+      "su n₂",
+      "▁da ra",
+      "▁mu ru₁₃",
+      "sa kar",
+      "babbar ₂",
+      "ba ra₂",
+      "ur ta",
+      "eri m₂",
+      "▁babbar ₂",
+      "si lim",
+      "▁nin ₉",
+      "gi bil",
+      "nin da",
+      "na m₂",
+      "z e₂",
+      "ga z",
+      "du l₃",
+      "mu d",
+      "ga b₂",
+      "▁gur dub",
+      "ti l₃",
+      "tum ₃",
+      "ka b",
+      "sa ba",
+      "▁a ŋ₂",
+      "ŋ i",
+      "š im",
+      "▁ki kken₂",
+      "p ₄",
+      "▁u z",
+      "▁gi r",
+      "ku p₄",
+      "su ₄",
+      "▁ha r",
+      "▁ku ₓ",
+      "▁du ru₅",
+      "▁ eš₂",
+      "ni saba",
+      "na g",
+      "g ina",
+      "s in₂",
+      "su h₅",
+      "▁šu bur",
+      "▁ša r₂",
+      "z a₃",
+      "du r₂",
+      "gin ₃",
+      "▁ab zu",
+      "▁me s",
+      "▁dara ₄",
+      "k eš₂",
+      "▁za r₃",
+      "▁ša gina",
+      "ku d",
+      "nin nu",
+      "w e",
+      "▁z e₂",
+      "▁še g",
+      "L A",
+      "▁ta r",
+      "kas ₄",
+      "▁sa l",
+      "▁dug ₃",
+      "gin ₆",
+      "▁u tu",
+      "▁ni r",
+      "▁ta k₄",
+      "h en",
+      "mu n₂",
+      "u g",
+      "▁gu n₃",
+      "▁im ga",
+      "hen dur",
+      "▁zi ₓ",
+      "▁sag ₉",
+      "▁ŋ eš",
+      "▁da gal",
+      "gi l",
+      "▁kala g",
+      "ti l",
+      "▁imga ga₃",
+      "₁ ₄",
+      "ki kken",
+      "▁lu m",
+      "šu ba",
+      "▁ni gin₂",
+      "▁a g",
+      "▁ga n",
+      "▁e gir",
+      "▁tu ku",
+      "▁ta g",
+      "▁ka m",
+      "da g",
+      "▁šeg ₉",
+      "₁ ₆",
+      "▁a r₃",
+      "gur ₄",
+      "ti r",
+      "q u₂",
+      "ŋi r₂",
+      "gu ₃",
+      "▁na ₄",
+      "▁tu l₂",
+      "▁ iš",
+      "kas kal",
+      "▁i b",
+      "▁za g",
+      "a ŋ₂"
+    ]
+  }
+}

tokenizer_config.json

@@ -0,0 +1,115 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<SURFACE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<COLUMN>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<BLANK_SPACE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<RULING>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "...",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "</s>",
+    "<RULING>",
+    "<mask>",
+    "<BLANK_SPACE>",
+    "<unk>",
+    "\n",
+    "<s>",
+    "<COLUMN>",
+    "<SURFACE>",
+    "...",
+    "<pad>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}