Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

special_tokens_map.json

@@ -0,0 +1,9 @@
+{
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -51,7 +51,7 @@
   ],
   "normalizer": null,
   "pre_tokenizer": {
-    "type": "WhitespaceSplit"
+    "type": "Whitespace"
   },
   "post_processor": null,
   "decoder": null,
@@ -63,343 +63,215 @@
       "[SEP]": 2,
       "[PAD]": 3,
       "[MASK]": 4,
-      "NOTE_OFF=58": 5,
-      "NOTE_ON=58": 6,
-      "NOTE_OFF=70": 7,
-      "NOTE_ON=70": 8,
-      "NOTE_OFF=80": 9,
-      "NOTE_ON=80": 10,
-      "NOTE_OFF=73": 11,
-      "NOTE_ON=73": 12,
-      "NOTE_OFF=68": 13,
-      "NOTE_ON=68": 14,
-      "NOTE_OFF=72": 15,
-      "NOTE_ON=72": 16,
-      "NOTE_OFF=89": 17,
-      "NOTE_ON=89": 18,
-      "NOTE_OFF=77": 19,
-      "NOTE_ON=77": 20,
-      "NOTE_OFF=66": 21,
-      "NOTE_ON=66": 22,
-      "NOTE_OFF=85": 23,
-      "NOTE_ON=85": 24,
-      "NOTE_OFF=63": 25,
-      "NOTE_ON=63": 26,
-      "NOTE_OFF=90": 27,
-      "NOTE_ON=90": 28,
-      "NOTE_OFF=60": 29,
-      "NOTE_ON=60": 30,
-      "NOTE_OFF=61": 31,
-      "NOTE_ON=61": 32,
-      "NOTE_OFF=88": 33,
-      "NOTE_ON=88": 34,
-      "NOTE_OFF=82": 35,
-      "NOTE_ON=82": 36,
-      "NOTE_OFF=87": 37,
-      "NOTE_ON=87": 38,
-      "NOTE_OFF=84": 39,
-      "NOTE_ON=84": 40,
-      "NOTE_OFF=92": 41,
-      "NOTE_ON=92": 42,
-      "NOTE_OFF=59": 43,
-      "NOTE_ON=59": 44,
-      "NOTE_OFF=81": 45,
-      "NOTE_ON=81": 46,
-      "NOTE_OFF=71": 47,
-      "NOTE_ON=71": 48,
-      "NOTE_OFF=86": 49,
-      "NOTE_ON=86": 50,
-      "NOTE_OFF=83": 51,
-      "NOTE_ON=83": 52,
-      "NOTE_OFF=75": 53,
-      "NOTE_ON=75": 54,
-      "NOTE_OFF=69": 55,
-      "NOTE_ON=69": 56,
-      "NOTE_OFF=91": 57,
-      "NOTE_ON=91": 58,
-      "NOTE_OFF=93": 59,
-      "NOTE_ON=93": 60,
-      "BAR_END": 61,
-      "BAR_START": 62,
-      "NOTE_OFF=74": 63,
-      "NOTE_ON=74": 64,
-      "NOTE_OFF=79": 65,
-      "NOTE_ON=79": 66,
-      "NOTE_OFF=65": 67,
-      "NOTE_ON=65": 68,
-      "NOTE_OFF=67": 69,
-      "NOTE_ON=67": 70,
-      "NOTE_OFF=78": 71,
-      "NOTE_ON=78": 72,
-      "NOTE_OFF=99": 73,
-      "NOTE_ON=99": 74,
-      "NOTE_OFF=97": 75,
-      "NOTE_ON=97": 76,
-      "NOTE_OFF=41": 77,
-      "NOTE_ON=41": 78,
-      "NOTE_OFF=64": 79,
-      "NOTE_ON=64": 80,
-      "NOTE_OFF=100": 81,
-      "NOTE_ON=100": 82,
-      "NOTE_OFF=94": 83,
-      "NOTE_ON=94": 84,
-      "NOTE_OFF=96": 85,
-      "NOTE_ON=96": 86,
-      "NOTE_OFF=101": 87,
-      "NOTE_ON=101": 88,
-      "NOTE_OFF=76": 89,
-      "NOTE_ON=76": 90,
-      "NOTE_OFF=95": 91,
-      "NOTE_ON=95": 92,
-      "NOTE_OFF=44": 93,
-      "NOTE_ON=44": 94,
-      "NOTE_OFF=45": 95,
-      "NOTE_OFF=57": 96,
-      "NOTE_ON=45": 97,
-      "NOTE_ON=57": 98,
-      "NOTE_OFF=56": 99,
-      "NOTE_ON=56": 100,
-      "NOTE_OFF=102": 101,
-      "NOTE_ON=102": 102,
-      "NOTE_OFF=62": 103,
-      "NOTE_ON=62": 104,
-      "NOTE_OFF=98": 105,
-      "NOTE_ON=98": 106,
-      "TIME_DELTA=1": 107,
-      "NOTE_OFF=48": 108,
-      "NOTE_ON=48": 109,
-      "NOTE_OFF=42": 110,
-      "NOTE_ON=42": 111,
-      "NOTE_OFF=54": 112,
-      "NOTE_ON=54": 113,
-      "NOTE_OFF=50": 114,
-      "NOTE_ON=50": 115,
-      "NOTE_OFF=55": 116,
-      "NOTE_ON=55": 117,
-      "NOTE_OFF=47": 118,
-      "NOTE_ON=47": 119,
-      "NOTE_OFF=49": 120,
-      "NOTE_ON=49": 121,
-      "NOTE_OFF=53": 122,
-      "NOTE_ON=53": 123,
-      "NOTE_OFF=51": 124,
-      "NOTE_ON=51": 125,
-      "NOTE_OFF=40": 126,
-      "NOTE_ON=40": 127,
-      "NOTE_OFF=103": 128,
-      "NOTE_ON=103": 129,
-      "NOTE_OFF=46": 130,
-      "NOTE_ON=46": 131,
-      "NOTE_OFF=43": 132,
-      "NOTE_ON=43": 133,
-      "NOTE_OFF=52": 134,
-      "NOTE_ON=52": 135,
-      "NOTE_OFF=39": 136,
-      "NOTE_ON=39": 137,
-      "INST=0": 138,
-      "TRACK_END": 139,
-      "TRACK_START": 140,
-      "NOTE_OFF=38": 141,
-      "NOTE_ON=38": 142,
-      "TIME_DELTA=2": 143,
-      "NOTE_OFF=37": 144,
-      "NOTE_ON=37": 145,
-      "DENSITY=0": 146,
-      "PIECE_END": 147,
-      "PIECE_START": 148,
-      "NOTE_OFF=36": 149,
-      "NOTE_ON=36": 150,
-      "NOTE_OFF=34": 151,
-      "NOTE_ON=34": 152,
-      "TIME_DELTA=3": 153,
-      "NOTE_OFF=33": 154,
-      "NOTE_ON=33": 155,
-      "NOTE_OFF=35": 156,
-      "NOTE_ON=35": 157,
-      "TIME_DELTA=4": 158,
-      "NOTE_OFF=32": 159,
-      "NOTE_ON=32": 160,
-      "NOTE_OFF=29": 161,
-      "NOTE_ON=29": 162,
-      "TIME_DELTA=5": 163,
-      "NOTE_OFF=31": 164,
-      "NOTE_ON=31": 165,
-      "NOTE_OFF=30": 166,
-      "NOTE_ON=30": 167,
-      "NOTE_OFF=28": 168,
-      "NOTE_ON=28": 169,
-      "TIME_DELTA=7": 170,
-      "TIME_DELTA=6": 171,
-      "TIME_DELTA=10": 172,
-      "TIME_DELTA=8": 173,
-      "TIME_DELTA=9": 174,
-      "TIME_DELTA=11": 175,
-      "DENSITY=39": 176,
-      "TIME_DELTA=13": 177,
-      "DENSITY=52": 178,
-      "NOTE_OFF=27": 179,
-      "NOTE_ON=27": 180,
-      "TIME_DELTA=16": 181,
-      "TIME_DELTA=19": 182,
-      "DENSITY=38": 183,
-      "DENSITY=46": 184,
-      "DENSITY=48": 185,
-      "DENSITY=68": 186,
-      "TIME_DELTA=12": 187,
-      "TIME_DELTA=15": 188,
-      "TIME_DELTA=20": 189,
-      "DENSITY=32": 190,
-      "DENSITY=35": 191,
-      "DENSITY=43": 192,
-      "DENSITY=55": 193,
-      "DENSITY=71": 194,
-      "DENSITY=73": 195,
-      "DENSITY=82": 196,
-      "TIME_DELTA=17": 197,
-      "TIME_DELTA=27": 198,
-      "DENSITY=10": 199,
-      "DENSITY=30": 200,
-      "DENSITY=31": 201,
-      "DENSITY=37": 202,
-      "DENSITY=51": 203,
-      "DENSITY=60": 204,
-      "DENSITY=64": 205,
-      "DENSITY=66": 206,
-      "DENSITY=67": 207,
-      "DENSITY=78": 208,
-      "DENSITY=98": 209,
-      "TIME_DELTA=18": 210,
-      "TIME_DELTA=29": 211,
-      "DENSITY=18": 212,
-      "DENSITY=28": 213,
-      "DENSITY=33": 214,
-      "DENSITY=36": 215,
-      "DENSITY=40": 216,
-      "DENSITY=41": 217,
-      "DENSITY=42": 218,
-      "DENSITY=44": 219,
-      "DENSITY=47": 220,
-      "DENSITY=53": 221,
-      "DENSITY=54": 222,
-      "DENSITY=62": 223,
-      "DENSITY=70": 224,
-      "DENSITY=76": 225,
-      "DENSITY=86": 226,
-      "DENSITY=94": 227,
-      "DENSITY=96": 228,
-      "DENSITY=97": 229,
-      "NOTE_OFF=26": 230,
-      "NOTE_ON=26": 231,
-      "TIME_DELTA=14": 232,
-      "TIME_DELTA=21": 233,
-      "TIME_DELTA=23": 234,
-      "TIME_DELTA=41": 235,
-      "DENSITY=103": 236,
-      "DENSITY=117": 237,
-      "DENSITY=125": 238,
-      "DENSITY=134": 239,
-      "DENSITY=20": 240,
-      "DENSITY=22": 241,
-      "DENSITY=23": 242,
-      "DENSITY=24": 243,
-      "DENSITY=34": 244,
-      "DENSITY=45": 245,
-      "DENSITY=49": 246,
-      "DENSITY=5": 247,
-      "DENSITY=50": 248,
-      "DENSITY=56": 249,
-      "DENSITY=57": 250,
-      "DENSITY=61": 251,
-      "DENSITY=63": 252,
-      "DENSITY=65": 253,
-      "DENSITY=69": 254,
-      "DENSITY=72": 255,
-      "DENSITY=80": 256,
-      "DENSITY=91": 257,
-      "DENSITY=95": 258,
-      "DENSITY=99": 259,
-      "NOTE_OFF=21": 260,
-      "NOTE_ON=21": 261,
-      "TIME_DELTA=22": 262,
-      "TIME_DELTA=25": 263,
-      "TIME_DELTA=26": 264,
-      "TIME_DELTA=31": 265,
-      "TIME_DELTA=32": 266,
-      "TIME_DELTA=33": 267,
-      "TIME_DELTA=34": 268,
-      "TIME_DELTA=37": 269,
-      "TIME_DELTA=38": 270,
-      "TIME_DELTA=39": 271,
-      "TIME_DELTA=60": 272,
-      "TIME_DELTA=86": 273,
-      "TIME_DELTA=89": 274,
-      "TIME_DELTA=94": 275,
-      "DENSITY=100": 276,
-      "DENSITY=102": 277,
-      "DENSITY=104": 278,
-      "DENSITY=106": 279,
-      "DENSITY=107": 280,
-      "DENSITY=109": 281,
-      "DENSITY=111": 282,
-      "DENSITY=113": 283,
-      "DENSITY=116": 284,
-      "DENSITY=120": 285,
-      "DENSITY=123": 286,
-      "DENSITY=13": 287,
-      "DENSITY=132": 288,
-      "DENSITY=144": 289,
-      "DENSITY=15": 290,
-      "DENSITY=16": 291,
-      "DENSITY=17": 292,
-      "DENSITY=19": 293,
-      "DENSITY=26": 294,
-      "DENSITY=27": 295,
-      "DENSITY=29": 296,
-      "DENSITY=59": 297,
-      "DENSITY=6": 298,
-      "DENSITY=7": 299,
-      "DENSITY=74": 300,
-      "DENSITY=75": 301,
-      "DENSITY=77": 302,
-      "DENSITY=79": 303,
-      "DENSITY=83": 304,
-      "DENSITY=84": 305,
-      "DENSITY=88": 306,
-      "DENSITY=89": 307,
-      "DENSITY=90": 308,
-      "NOTE_OFF=17": 309,
-      "NOTE_OFF=23": 310,
-      "NOTE_OFF=24": 311,
-      "NOTE_OFF=25": 312,
-      "NOTE_ON=17": 313,
-      "NOTE_ON=23": 314,
-      "NOTE_ON=24": 315,
-      "NOTE_ON=25": 316,
-      "TIME_DELTA=104": 317,
-      "TIME_DELTA=116": 318,
-      "TIME_DELTA=123": 319,
-      "TIME_DELTA=133": 320,
-      "TIME_DELTA=134": 321,
-      "TIME_DELTA=24": 322,
-      "TIME_DELTA=30": 323,
-      "TIME_DELTA=35": 324,
-      "TIME_DELTA=36": 325,
-      "TIME_DELTA=40": 326,
-      "TIME_DELTA=45": 327,
-      "TIME_DELTA=48": 328,
-      "TIME_DELTA=50": 329,
-      "TIME_DELTA=53": 330,
-      "TIME_DELTA=54": 331,
-      "TIME_DELTA=55": 332,
-      "TIME_DELTA=57": 333,
-      "TIME_DELTA=61": 334,
-      "TIME_DELTA=64": 335,
-      "TIME_DELTA=67": 336,
-      "TIME_DELTA=69": 337,
-      "TIME_DELTA=70": 338,
-      "TIME_DELTA=74": 339,
-      "TIME_DELTA=91": 340,
-      "TIME_DELTA=95": 341
+      "=": 5,
+      "TIME_DELTA": 6,
+      "NOTE_OFF": 7,
+      "NOTE_ON": 8,
+      "3": 9,
+      "2": 10,
+      "4": 11,
+      "5": 12,
+      "6": 13,
+      "58": 14,
+      "70": 15,
+      "80": 16,
+      "7": 17,
+      "73": 18,
+      "68": 19,
+      "72": 20,
+      "89": 21,
+      "77": 22,
+      "66": 23,
+      "63": 24,
+      "85": 25,
+      "90": 26,
+      "60": 27,
+      "61": 28,
+      "88": 29,
+      "82": 30,
+      "87": 31,
+      "8": 32,
+      "84": 33,
+      "92": 34,
+      "59": 35,
+      "81": 36,
+      "71": 37,
+      "86": 38,
+      "83": 39,
+      "75": 40,
+      "69": 41,
+      "91": 42,
+      "93": 43,
+      "74": 44,
+      "65": 45,
+      "79": 46,
+      "67": 47,
+      "1": 48,
+      "78": 49,
+      "9": 50,
+      "99": 51,
+      "41": 52,
+      "97": 53,
+      "64": 54,
+      "100": 55,
+      "94": 56,
+      "96": 57,
+      "101": 58,
+      "76": 59,
+      "95": 60,
+      "44": 61,
+      "45": 62,
+      "57": 63,
+      "56": 64,
+      "102": 65,
+      "10": 66,
+      "62": 67,
+      "98": 68,
+      "48": 69,
+      "42": 70,
+      "11": 71,
+      "50": 72,
+      "54": 73,
+      "55": 74,
+      "47": 75,
+      "49": 76,
+      "53": 77,
+      "40": 78,
+      "51": 79,
+      "12": 80,
+      "103": 81,
+      "46": 82,
+      "43": 83,
+      "13": 84,
+      "52": 85,
+      "14": 86,
+      "15": 87,
+      "39": 88,
+      "BAR_END": 89,
+      "BAR_START": 90,
+      "16": 91,
+      "17": 92,
+      "18": 93,
+      "19": 94,
+      "20": 95,
+      "21": 96,
+      "22": 97,
+      "23": 98,
+      "38": 99,
+      "37": 100,
+      "24": 101,
+      "25": 102,
+      "26": 103,
+      "28": 104,
+      "29": 105,
+      "27": 106,
+      "34": 107,
+      "30": 108,
+      "33": 109,
+      "31": 110,
+      "32": 111,
+      "36": 112,
+      "35": 113,
+      "0": 114,
+      "DENSITY": 115,
+      "INST": 116,
+      "PIECE_END": 117,
+      "PIECE_START": 118,
+      "TRACK_END": 119,
+      "TRACK_START": 120,
+      "123": 121,
+      "118": 122,
+      "124": 123,
+      "151": 124,
+      "106": 125,
+      "104": 126,
+      "107": 127,
+      "113": 128,
+      "114": 129,
+      "117": 130,
+      "119": 131,
+      "120": 132,
+      "133": 133,
+      "127": 134,
+      "129": 135,
+      "130": 136,
+      "146": 137,
+      "152": 138,
+      "165": 139,
+      "108": 140,
+      "109": 141,
+      "111": 142,
+      "112": 143,
+      "126": 144,
+      "131": 145,
+      "132": 146,
+      "141": 147,
+      "148": 148,
+      "150": 149,
+      "157": 150,
+      "158": 151,
+      "161": 152,
+      "163": 153,
+      "164": 154,
+      "180": 155,
+      "116": 156,
+      "121": 157,
+      "125": 158,
+      "128": 159,
+      "134": 160,
+      "135": 161,
+      "136": 162,
+      "139": 163,
+      "143": 164,
+      "147": 165,
+      "149": 166,
+      "154": 167,
+      "162": 168,
+      "172": 169,
+      "176": 170,
+      "191": 171,
+      "210": 172,
+      "216": 173,
+      "105": 174,
+      "110": 175,
+      "115": 176,
+      "137": 177,
+      "138": 178,
+      "140": 179,
+      "142": 180,
+      "145": 181,
+      "155": 182,
+      "159": 183,
+      "166": 184,
+      "167": 185,
+      "168": 186,
+      "170": 187,
+      "171": 188,
+      "174": 189,
+      "175": 190,
+      "177": 191,
+      "178": 192,
+      "179": 193,
+      "183": 194,
+      "190": 195,
+      "193": 196,
+      "199": 197,
+      "205": 198,
+      "212": 199,
+      "220": 200,
+      "236": 201,
+      "238": 202,
+      "250": 203,
+      "255": 204,
+      "263": 205,
+      "268": 206,
+      "289": 207,
+      "317": 208,
+      "335": 209,
+      "391": 210,
+      "397": 211,
+      "403": 212,
+      "451": 213
     },
     "unk_token": "[UNK]"
   }

tokenizer_config.json

@@ -0,0 +1,48 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}