Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

merges.txt

@@ -0,0 +1 @@
+#version: 0.2

special_tokens_map.json

@@ -0,0 +1,118 @@
+{
+  "additional_special_tokens": [
+    "a",
+    "E",
+    "e",
+    "Ɛ",
+    "ɛ",
+    "Ə",
+    "ə",
+    "I",
+    "i",
+    "O",
+    "o",
+    "Ɔ",
+    "ɔ",
+    "U",
+    "u",
+    "ã",
+    "ẽ",
+    "ĩ",
+    "õ",
+    "ũ",
+    "B",
+    "b",
+    "D",
+    "d",
+    "Đ",
+    "ɖ",
+    "F",
+    "f",
+    "Ƒ",
+    "ƒ",
+    "G",
+    "g",
+    "Ɣ",
+    "ɣ",
+    "H",
+    "h",
+    "K",
+    "k",
+    "L",
+    "l",
+    "M",
+    "m",
+    "N",
+    "n",
+    "Ŋ",
+    "ŋ",
+    "P",
+    "p",
+    "R",
+    "r",
+    "S",
+    "s",
+    "T",
+    "t",
+    "V",
+    "v",
+    "Ʋ",
+    "ʋ",
+    "W",
+    "w",
+    "X",
+    "x",
+    "Y",
+    "y",
+    "Z",
+    "z",
+    "D",
+    "z",
+    "d",
+    "z",
+    "G",
+    "b",
+    "g",
+    "b",
+    "K",
+    "p",
+    "k",
+    "p",
+    "N",
+    "y",
+    "n",
+    "y",
+    "T",
+    "s",
+    "i",
+    "t",
+    "s",
+    "i",
+    "D",
+    "y",
+    "d",
+    "y",
+    "D",
+    "z",
+    "i",
+    "d",
+    "z",
+    "i",
+    "Ŋ",
+    "w",
+    "ŋ",
+    "w",
+    "ɔ",
+    "̃",
+    "ó",
+    "À",
+    "é",
+    "È",
+    "ò",
+    "à",
+    "í"
+  ],
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer.json

@@ -0,0 +1,989 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<|endoftext|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": true,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "a",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "E",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "e",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "Ɛ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "ɛ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 6,
+      "content": "Ə",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 7,
+      "content": "ə",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 8,
+      "content": "I",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 9,
+      "content": "i",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 10,
+      "content": "O",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 11,
+      "content": "o",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 12,
+      "content": "Ɔ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 13,
+      "content": "ɔ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 14,
+      "content": "U",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 15,
+      "content": "u",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 16,
+      "content": "ã",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 17,
+      "content": "ẽ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 18,
+      "content": "ĩ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 19,
+      "content": "õ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 20,
+      "content": "ũ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 21,
+      "content": "B",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 22,
+      "content": "b",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 23,
+      "content": "D",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 24,
+      "content": "d",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 25,
+      "content": "Đ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 26,
+      "content": "ɖ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 27,
+      "content": "F",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 28,
+      "content": "f",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 29,
+      "content": "Ƒ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 30,
+      "content": "ƒ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 31,
+      "content": "G",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 32,
+      "content": "g",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 33,
+      "content": "Ɣ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 34,
+      "content": "ɣ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 35,
+      "content": "H",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 36,
+      "content": "h",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 37,
+      "content": "K",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 38,
+      "content": "k",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 39,
+      "content": "L",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 40,
+      "content": "l",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 41,
+      "content": "M",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 42,
+      "content": "m",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 43,
+      "content": "N",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 44,
+      "content": "n",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 45,
+      "content": "Ŋ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 46,
+      "content": "ŋ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 47,
+      "content": "P",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 48,
+      "content": "p",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 49,
+      "content": "R",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 50,
+      "content": "r",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 51,
+      "content": "S",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 52,
+      "content": "s",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 53,
+      "content": "T",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 54,
+      "content": "t",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 55,
+      "content": "V",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 56,
+      "content": "v",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 57,
+      "content": "Ʋ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 58,
+      "content": "ʋ",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 59,
+      "content": "W",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 60,
+      "content": "w",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 61,
+      "content": "X",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 62,
+      "content": "x",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 63,
+      "content": "Y",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 64,
+      "content": "y",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 65,
+      "content": "Z",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 66,
+      "content": "z",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 67,
+      "content": "̃",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 68,
+      "content": "ó",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 69,
+      "content": "À",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 70,
+      "content": "é",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 71,
+      "content": "È",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 72,
+      "content": "ò",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 73,
+      "content": "à",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 74,
+      "content": "í",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "post_processor": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": false,
+    "use_regex": true
+  },
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": "",
+    "end_of_word_suffix": "",
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "<|endoftext|>": 0,
+      "a": 1,
+      "E": 2,
+      "e": 3,
+      "Ɛ": 4,
+      "ɛ": 5,
+      "Ə": 6,
+      "ə": 7,
+      "I": 8,
+      "i": 9,
+      "O": 10,
+      "o": 11,
+      "Ɔ": 12,
+      "ɔ": 13,
+      "U": 14,
+      "u": 15,
+      "ã": 16,
+      "ẽ": 17,
+      "ĩ": 18,
+      "õ": 19,
+      "ũ": 20,
+      "B": 21,
+      "b": 22,
+      "D": 23,
+      "d": 24,
+      "Đ": 25,
+      "ɖ": 26,
+      "F": 27,
+      "f": 28,
+      "Ƒ": 29,
+      "ƒ": 30,
+      "G": 31,
+      "g": 32,
+      "Ɣ": 33,
+      "ɣ": 34,
+      "H": 35,
+      "h": 36,
+      "K": 37,
+      "k": 38,
+      "L": 39,
+      "l": 40,
+      "M": 41,
+      "m": 42,
+      "N": 43,
+      "n": 44,
+      "Ŋ": 45,
+      "ŋ": 46,
+      "P": 47,
+      "p": 48,
+      "R": 49,
+      "r": 50,
+      "S": 51,
+      "s": 52,
+      "T": 53,
+      "t": 54,
+      "V": 55,
+      "v": 56,
+      "Ʋ": 57,
+      "ʋ": 58,
+      "W": 59,
+      "w": 60,
+      "X": 61,
+      "x": 62,
+      "Y": 63,
+      "y": 64,
+      "Z": 65,
+      "z": 66,
+      "̃": 67,
+      "ó": 68,
+      "À": 69,
+      "é": 70,
+      "È": 71,
+      "ò": 72,
+      "à": 73,
+      "í": 74,
+      "!": 75,
+      "\"": 76,
+      "#": 77,
+      "$": 78,
+      "%": 79,
+      "&": 80,
+      "'": 81,
+      "(": 82,
+      ")": 83,
+      "*": 84,
+      "+": 85,
+      ",": 86,
+      "-": 87,
+      ".": 88,
+      "/": 89,
+      "0": 90,
+      "1": 91,
+      "2": 92,
+      "3": 93,
+      "4": 94,
+      "5": 95,
+      "6": 96,
+      "7": 97,
+      "8": 98,
+      "9": 99,
+      ":": 100,
+      ";": 101,
+      "<": 102,
+      "=": 103,
+      ">": 104,
+      "?": 105,
+      "@": 106,
+      "A": 107,
+      "C": 108,
+      "J": 109,
+      "Q": 110,
+      "[": 111,
+      "\\": 112,
+      "]": 113,
+      "^": 114,
+      "_": 115,
+      "`": 116,
+      "c": 117,
+      "j": 118,
+      "q": 119,
+      "{": 120,
+      "|": 121,
+      "}": 122,
+      "~": 123,
+      "¡": 124,
+      "¢": 125,
+      "£": 126,
+      "¤": 127,
+      "¥": 128,
+      "¦": 129,
+      "§": 130,
+      "¨": 131,
+      "©": 132,
+      "ª": 133,
+      "«": 134,
+      "¬": 135,
+      "®": 136,
+      "¯": 137,
+      "°": 138,
+      "±": 139,
+      "²": 140,
+      "³": 141,
+      "´": 142,
+      "µ": 143,
+      "¶": 144,
+      "·": 145,
+      "¸": 146,
+      "¹": 147,
+      "º": 148,
+      "»": 149,
+      "¼": 150,
+      "½": 151,
+      "¾": 152,
+      "¿": 153,
+      "Á": 154,
+      "Â": 155,
+      "Ã": 156,
+      "Ä": 157,
+      "Å": 158,
+      "Æ": 159,
+      "Ç": 160,
+      "É": 161,
+      "Ê": 162,
+      "Ë": 163,
+      "Ì": 164,
+      "Í": 165,
+      "Î": 166,
+      "Ï": 167,
+      "Ð": 168,
+      "Ñ": 169,
+      "Ò": 170,
+      "Ó": 171,
+      "Ô": 172,
+      "Õ": 173,
+      "Ö": 174,
+      "×": 175,
+      "Ø": 176,
+      "Ù": 177,
+      "Ú": 178,
+      "Û": 179,
+      "Ü": 180,
+      "Ý": 181,
+      "Þ": 182,
+      "ß": 183,
+      "á": 184,
+      "â": 185,
+      "ä": 186,
+      "å": 187,
+      "æ": 188,
+      "ç": 189,
+      "è": 190,
+      "ê": 191,
+      "ë": 192,
+      "ì": 193,
+      "î": 194,
+      "ï": 195,
+      "ð": 196,
+      "ñ": 197,
+      "ô": 198,
+      "ö": 199,
+      "÷": 200,
+      "ø": 201,
+      "ù": 202,
+      "ú": 203,
+      "û": 204,
+      "ü": 205,
+      "ý": 206,
+      "þ": 207,
+      "ÿ": 208,
+      "Ā": 209,
+      "ā": 210,
+      "Ă": 211,
+      "ă": 212,
+      "Ą": 213,
+      "ą": 214,
+      "Ć": 215,
+      "ć": 216,
+      "Ĉ": 217,
+      "ĉ": 218,
+      "Ċ": 219,
+      "ċ": 220,
+      "Č": 221,
+      "č": 222,
+      "Ď": 223,
+      "ď": 224,
+      "đ": 225,
+      "Ē": 226,
+      "ē": 227,
+      "Ĕ": 228,
+      "ĕ": 229,
+      "Ė": 230,
+      "ė": 231,
+      "Ę": 232,
+      "ę": 233,
+      "Ě": 234,
+      "ě": 235,
+      "Ĝ": 236,
+      "ĝ": 237,
+      "Ğ": 238,
+      "ğ": 239,
+      "Ġ": 240,
+      "ġ": 241,
+      "Ģ": 242,
+      "ģ": 243,
+      "Ĥ": 244,
+      "ĥ": 245,
+      "Ħ": 246,
+      "ħ": 247,
+      "Ĩ": 248,
+      "Ī": 249,
+      "ī": 250,
+      "Ĭ": 251,
+      "ĭ": 252,
+      "Į": 253,
+      "į": 254,
+      "İ": 255,
+      "ı": 256,
+      "IJ": 257,
+      "ij": 258,
+      "Ĵ": 259,
+      "ĵ": 260,
+      "Ķ": 261,
+      "ķ": 262,
+      "ĸ": 263,
+      "Ĺ": 264,
+      "ĺ": 265,
+      "Ļ": 266,
+      "ļ": 267,
+      "Ľ": 268,
+      "ľ": 269,
+      "Ŀ": 270,
+      "ŀ": 271,
+      "Ł": 272,
+      "ł": 273,
+      "Ń": 274
+    },
+    "merges": []
+  }
+}

tokenizer_config.json

@@ -0,0 +1,724 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "a",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "E",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "e",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "Ɛ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "ɛ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "Ə",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "ə",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "I",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "i",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "O",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "o",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "Ɔ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "ɔ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "U",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "u",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "ã",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "ẽ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "ĩ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "19": {
+      "content": "õ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "20": {
+      "content": "ũ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "21": {
+      "content": "B",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "22": {
+      "content": "b",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "23": {
+      "content": "D",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "24": {
+      "content": "d",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25": {
+      "content": "Đ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "26": {
+      "content": "ɖ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "27": {
+      "content": "F",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "28": {
+      "content": "f",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "29": {
+      "content": "Ƒ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30": {
+      "content": "ƒ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "31": {
+      "content": "G",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32": {
+      "content": "g",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "33": {
+      "content": "Ɣ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "34": {
+      "content": "ɣ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "35": {
+      "content": "H",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "36": {
+      "content": "h",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "37": {
+      "content": "K",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "38": {
+      "content": "k",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "39": {
+      "content": "L",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "40": {
+      "content": "l",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "41": {
+      "content": "M",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "42": {
+      "content": "m",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "43": {
+      "content": "N",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "44": {
+      "content": "n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "45": {
+      "content": "Ŋ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "46": {
+      "content": "ŋ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "47": {
+      "content": "P",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "48": {
+      "content": "p",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49": {
+      "content": "R",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50": {
+      "content": "r",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "51": {
+      "content": "S",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "52": {
+      "content": "s",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "53": {
+      "content": "T",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "54": {
+      "content": "t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "55": {
+      "content": "V",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "56": {
+      "content": "v",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "57": {
+      "content": "Ʋ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "58": {
+      "content": "ʋ",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "59": {
+      "content": "W",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "60": {
+      "content": "w",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "61": {
+      "content": "X",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "62": {
+      "content": "x",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "63": {
+      "content": "Y",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64": {
+      "content": "y",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65": {
+      "content": "Z",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "66": {
+      "content": "z",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "67": {
+      "content": "̃",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "68": {
+      "content": "ó",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "69": {
+      "content": "À",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "70": {
+      "content": "é",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "71": {
+      "content": "È",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "72": {
+      "content": "ò",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73": {
+      "content": "à",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "74": {
+      "content": "í",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "a",
+    "E",
+    "e",
+    "Ɛ",
+    "ɛ",
+    "Ə",
+    "ə",
+    "I",
+    "i",
+    "O",
+    "o",
+    "Ɔ",
+    "ɔ",
+    "U",
+    "u",
+    "ã",
+    "ẽ",
+    "ĩ",
+    "õ",
+    "ũ",
+    "B",
+    "b",
+    "D",
+    "d",
+    "Đ",
+    "ɖ",
+    "F",
+    "f",
+    "Ƒ",
+    "ƒ",
+    "G",
+    "g",
+    "Ɣ",
+    "ɣ",
+    "H",
+    "h",
+    "K",
+    "k",
+    "L",
+    "l",
+    "M",
+    "m",
+    "N",
+    "n",
+    "Ŋ",
+    "ŋ",
+    "P",
+    "p",
+    "R",
+    "r",
+    "S",
+    "s",
+    "T",
+    "t",
+    "V",
+    "v",
+    "Ʋ",
+    "ʋ",
+    "W",
+    "w",
+    "X",
+    "x",
+    "Y",
+    "y",
+    "Z",
+    "z",
+    "D",
+    "z",
+    "d",
+    "z",
+    "G",
+    "b",
+    "g",
+    "b",
+    "K",
+    "p",
+    "k",
+    "p",
+    "N",
+    "y",
+    "n",
+    "y",
+    "T",
+    "s",
+    "i",
+    "t",
+    "s",
+    "i",
+    "D",
+    "y",
+    "d",
+    "y",
+    "D",
+    "z",
+    "i",
+    "d",
+    "z",
+    "i",
+    "Ŋ",
+    "w",
+    "ŋ",
+    "w",
+    "ɔ",
+    "̃",
+    "ó",
+    "À",
+    "é",
+    "È",
+    "ò",
+    "à",
+    "í"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

vocab.json

@@ -0,0 +1 @@
+{"<|endoftext|>":0,"a":1,"E":2,"e":3,"Ɛ":4,"ɛ":5,"Ə":6,"ə":7,"I":8,"i":9,"O":10,"o":11,"Ɔ":12,"ɔ":13,"U":14,"u":15,"ã":16,"ẽ":17,"ĩ":18,"õ":19,"ũ":20,"B":21,"b":22,"D":23,"d":24,"Đ":25,"ɖ":26,"F":27,"f":28,"Ƒ":29,"ƒ":30,"G":31,"g":32,"Ɣ":33,"ɣ":34,"H":35,"h":36,"K":37,"k":38,"L":39,"l":40,"M":41,"m":42,"N":43,"n":44,"Ŋ":45,"ŋ":46,"P":47,"p":48,"R":49,"r":50,"S":51,"s":52,"T":53,"t":54,"V":55,"v":56,"Ʋ":57,"ʋ":58,"W":59,"w":60,"X":61,"x":62,"Y":63,"y":64,"Z":65,"z":66,"̃":67,"ó":68,"À":69,"é":70,"È":71,"ò":72,"à":73,"í":74,"!":75,"\"":76,"#":77,"$":78,"%":79,"&":80,"'":81,"(":82,")":83,"*":84,"+":85,",":86,"-":87,".":88,"/":89,"0":90,"1":91,"2":92,"3":93,"4":94,"5":95,"6":96,"7":97,"8":98,"9":99,":":100,";":101,"<":102,"=":103,">":104,"?":105,"@":106,"A":107,"C":108,"J":109,"Q":110,"[":111,"\\":112,"]":113,"^":114,"_":115,"`":116,"c":117,"j":118,"q":119,"{":120,"|":121,"}":122,"~":123,"¡":124,"¢":125,"£":126,"¤":127,"¥":128,"¦":129,"§":130,"¨":131,"©":132,"ª":133,"«":134,"¬":135,"®":136,"¯":137,"°":138,"±":139,"²":140,"³":141,"´":142,"µ":143,"¶":144,"·":145,"¸":146,"¹":147,"º":148,"»":149,"¼":150,"½":151,"¾":152,"¿":153,"Á":154,"Â":155,"Ã":156,"Ä":157,"Å":158,"Æ":159,"Ç":160,"É":161,"Ê":162,"Ë":163,"Ì":164,"Í":165,"Î":166,"Ï":167,"Ð":168,"Ñ":169,"Ò":170,"Ó":171,"Ô":172,"Õ":173,"Ö":174,"×":175,"Ø":176,"Ù":177,"Ú":178,"Û":179,"Ü":180,"Ý":181,"Þ":182,"ß":183,"á":184,"â":185,"ä":186,"å":187,"æ":188,"ç":189,"è":190,"ê":191,"ë":192,"ì":193,"î":194,"ï":195,"ð":196,"ñ":197,"ô":198,"ö":199,"÷":200,"ø":201,"ù":202,"ú":203,"û":204,"ü":205,"ý":206,"þ":207,"ÿ":208,"Ā":209,"ā":210,"Ă":211,"ă":212,"Ą":213,"ą":214,"Ć":215,"ć":216,"Ĉ":217,"ĉ":218,"Ċ":219,"ċ":220,"Č":221,"č":222,"Ď":223,"ď":224,"đ":225,"Ē":226,"ē":227,"Ĕ":228,"ĕ":229,"Ė":230,"ė":231,"Ę":232,"ę":233,"Ě":234,"ě":235,"Ĝ":236,"ĝ":237,"Ğ":238,"ğ":239,"Ġ":240,"ġ":241,"Ģ":242,"ģ":243,"Ĥ":244,"ĥ":245,"Ħ":246,"ħ":247,"Ĩ":248,"Ī":249,"ī":250,"Ĭ":251,"ĭ":252,"Į":253,"į":254,"İ":255,"ı":256,"IJ":257,"ij":258,"Ĵ":259,"ĵ":260,"Ķ":261,"ķ":262,"ĸ":263,"Ĺ":264,"ĺ":265,"Ļ":266,"ļ":267,"Ľ":268,"ľ":269,"Ŀ":270,"ŀ":271,"Ł":272,"ł":273,"Ń":274}