Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer.json

@@ -0,0 +1,1083 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "[PAD]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "[UNK]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "[CLS]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "[SEP]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "[MASK]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": {
+    "type": "BertNormalizer",
+    "clean_text": true,
+    "handle_chinese_chars": true,
+    "strip_accents": null,
+    "lowercase": true
+  },
+  "pre_tokenizer": {
+    "type": "BertPreTokenizer"
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "SpecialToken": {
+          "id": "[CLS]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "SpecialToken": {
+          "id": "[CLS]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {
+      "[CLS]": {
+        "id": "[CLS]",
+        "ids": [
+          2
+        ],
+        "tokens": [
+          "[CLS]"
+        ]
+      },
+      "[SEP]": {
+        "id": "[SEP]",
+        "ids": [
+          3
+        ],
+        "tokens": [
+          "[SEP]"
+        ]
+      }
+    }
+  },
+  "decoder": {
+    "type": "WordPiece",
+    "prefix": "##",
+    "cleanup": true
+  },
+  "model": {
+    "type": "WordPiece",
+    "unk_token": "[UNK]",
+    "continuing_subword_prefix": "##",
+    "max_input_chars_per_word": 100,
+    "vocab": {
+      "[PAD]": 0,
+      "[UNK]": 1,
+      "[CLS]": 2,
+      "[SEP]": 3,
+      "[MASK]": 4,
+      "!": 5,
+      "\"": 6,
+      "#": 7,
+      "$": 8,
+      "%": 9,
+      "&": 10,
+      "'": 11,
+      "(": 12,
+      ")": 13,
+      "*": 14,
+      "+": 15,
+      ",": 16,
+      "-": 17,
+      ".": 18,
+      "/": 19,
+      "0": 20,
+      "1": 21,
+      "2": 22,
+      "3": 23,
+      "4": 24,
+      "5": 25,
+      "6": 26,
+      "7": 27,
+      "8": 28,
+      "9": 29,
+      ":": 30,
+      ";": 31,
+      "<": 32,
+      "=": 33,
+      ">": 34,
+      "?": 35,
+      "@": 36,
+      "[": 37,
+      "\\": 38,
+      "]": 39,
+      "^": 40,
+      "_": 41,
+      "a": 42,
+      "b": 43,
+      "c": 44,
+      "d": 45,
+      "e": 46,
+      "f": 47,
+      "g": 48,
+      "h": 49,
+      "i": 50,
+      "j": 51,
+      "k": 52,
+      "l": 53,
+      "m": 54,
+      "n": 55,
+      "o": 56,
+      "p": 57,
+      "q": 58,
+      "r": 59,
+      "s": 60,
+      "t": 61,
+      "u": 62,
+      "v": 63,
+      "w": 64,
+      "x": 65,
+      "y": 66,
+      "z": 67,
+      "{": 68,
+      "|": 69,
+      "}": 70,
+      "##e": 71,
+      "##s": 72,
+      "##p": 73,
+      "##o": 74,
+      "##n": 75,
+      "##i": 76,
+      "##m": 77,
+      "##k": 78,
+      "##a": 79,
+      "##t": 80,
+      "##z": 81,
+      "##u": 82,
+      "##l": 83,
+      "##r": 84,
+      "##c": 85,
+      "##d": 86,
+      "##w": 87,
+      "##x": 88,
+      "##g": 89,
+      "##y": 90,
+      "##3": 91,
+      "##2": 92,
+      "##h": 93,
+      "##f": 94,
+      "##v": 95,
+      "##1": 96,
+      "##6": 97,
+      "##7": 98,
+      "##8": 99,
+      "##5": 100,
+      "##b": 101,
+      "##9": 102,
+      "##4": 103,
+      "##0": 104,
+      "##j": 105,
+      "##at": 106,
+      "##em": 107,
+      "##er": 108,
+      "##st": 109,
+      "##ut": 110,
+      "##ro": 111,
+      "##in": 112,
+      "ex": 113,
+      "##et": 114,
+      "##le": 115,
+      "##ss": 116,
+      "##ath": 117,
+      "path": 118,
+      "##re": 119,
+      "##on": 120,
+      "sy": 121,
+      "##ow": 122,
+      "##id": 123,
+      "##stem": 124,
+      "system": 125,
+      "##ile": 126,
+      "exe": 127,
+      "##mp": 128,
+      "file": 129,
+      "out": 130,
+      "##ar": 131,
+      "##ad": 132,
+      "##ass": 133,
+      "##put": 134,
+      "##ce": 135,
+      "##pp": 136,
+      "##py": 137,
+      "output": 138,
+      "##or": 139,
+      "##it": 140,
+      "##me": 141,
+      "##ve": 142,
+      "##emp": 143,
+      "pro": 144,
+      "##ol": 145,
+      "##al": 146,
+      "##ump": 147,
+      "##ct": 148,
+      "mem": 149,
+      "##nv": 150,
+      "##32": 151,
+      "##nt": 152,
+      "##ch": 153,
+      "ls": 154,
+      "pid": 155,
+      "temp": 156,
+      "win": 157,
+      "##om": 158,
+      "##ive": 159,
+      "##ur": 160,
+      "env": 161,
+      "st": 162,
+      "##ame": 163,
+      "##ll": 164,
+      "system32": 165,
+      "##di": 166,
+      "sh": 167,
+      "##ot": 168,
+      "##ion": 169,
+      "##act": 170,
+      "##der": 171,
+      "##dow": 172,
+      "##adow": 173,
+      "window": 174,
+      "windows": 175,
+      "ne": 176,
+      "##tc": 177,
+      "##ls": 178,
+      "##ract": 179,
+      "##co": 180,
+      "##ate": 181,
+      "##cho": 182,
+      "lsass": 183,
+      "etc": 184,
+      "it": 185,
+      "net": 186,
+      "pass": 187,
+      "re": 188,
+      "##no": 189,
+      "##ic": 190,
+      "##kl": 191,
+      "##kdi": 192,
+      "##ter": 193,
+      "echo": 194,
+      "hkl": 195,
+      "npp": 196,
+      "te": 197,
+      "##spy": 198,
+      "##pt": 199,
+      "##ag": 200,
+      "##lo": 201,
+      "##dump": 202,
+      "##lea": 203,
+      "hklm": 204,
+      "nppspy": 205,
+      "app": 206,
+      "cat": 207,
+      "er": 208,
+      "in": 209,
+      "name": 210,
+      "rd": 211,
+      "tar": 212,
+      "##nd": 213,
+      "##ice": 214,
+      "##ty": 215,
+      "##th": 216,
+      "##rlea": 217,
+      "##con": 218,
+      "##get": 219,
+      "##vid": 220,
+      "##vice": 221,
+      "##roract": 222,
+      "##rep": 223,
+      "##cess": 224,
+      "shadow": 225,
+      "new": 226,
+      "##kdiag": 227,
+      "erroract": 228,
+      "rdrlea": 229,
+      "target": 230,
+      "##vider": 231,
+      "erroraction": 232,
+      "rdrleakdiag": 233,
+      "cre": 234,
+      "get": 235,
+      "hive": 236,
+      "ig": 237,
+      "ma": 238,
+      "py": 239,
+      "vss": 240,
+      "##pro": 241,
+      "##ue": 242,
+      "##root": 243,
+      "##art": 244,
+      "##older": 245,
+      "passw": 246,
+      "##nore": 247,
+      "##cont": 248,
+      "create": 249,
+      "ignore": 250,
+      "co": 251,
+      "id": 252,
+      "if": 253,
+      "lo": 254,
+      "li": 255,
+      "min": 256,
+      "ps": 257,
+      "se": 258,
+      "sa": 259,
+      "sc": 260,
+      "ser": 261,
+      "##ent": 262,
+      "##pa": 263,
+      "##ipt": 264,
+      "##to": 265,
+      "##ull": 266,
+      "##ript": 267,
+      "##cur": 268,
+      "##ds": 269,
+      "##ing": 270,
+      "##idump": 271,
+      "proc": 272,
+      "process": 273,
+      "##alue": 274,
+      "##omic": 275,
+      "start": 276,
+      "test": 277,
+      "##load": 278,
+      "passwd": 279,
+      "minidump": 280,
+      "script": 281,
+      "00": 282,
+      "con": 283,
+      "com": 284,
+      "cur": 285,
+      "de": 286,
+      "di": 287,
+      "for": 288,
+      "full": 289,
+      "he": 290,
+      "t1": 291,
+      "##sh": 292,
+      "##po": 293,
+      "##per": 294,
+      "##name": 295,
+      "##wor": 296,
+      "##03": 297,
+      "##003": 298,
+      "##str": 299,
+      "##rols": 300,
+      "##rent": 301,
+      "##ary": 302,
+      "##copy": 303,
+      "item": 304,
+      "##thon": 305,
+      "python": 306,
+      "##controls": 307,
+      "service": 308,
+      "##omics": 309,
+      "001": 310,
+      "current": 311,
+      "t1003": 312,
+      "##perty": 313,
+      "##controlset": 314,
+      "currentcontrolset": 315,
+      "0x": 316,
+      "13": 317,
+      "bin": 318,
+      "cut": 319,
+      "dr": 320,
+      "dmp": 321,
+      "grep": 322,
+      "ho": 323,
+      "let": 324,
+      "nt": 325,
+      "rv": 326,
+      "uname": 327,
+      "value": 328,
+      "##en": 329,
+      "##ex": 330,
+      "##ser": 331,
+      "##op": 332,
+      "##nal": 333,
+      "##il": 334,
+      "##mi": 335,
+      "##md": 336,
+      "##kat": 337,
+      "##kpro": 338,
+      "##ume": 339,
+      "##und": 340,
+      "##li": 341,
+      "##cat": 342,
+      "##yload": 343,
+      "##folder": 344,
+      "##atomics": 345,
+      "##roperty": 346,
+      "exit": 347,
+      "exter": 348,
+      "pathto": 349,
+      "systemroot": 350,
+      "##emproperty": 351,
+      "##olume": 352,
+      "stop": 353,
+      "itemproperty": 354,
+      "networ": 355,
+      "copy": 356,
+      "list": 357,
+      "##payload": 358,
+      "services": 359,
+      "binary": 360,
+      "drive": 361,
+      "host": 362,
+      "letter": 363,
+      "ntds": 364,
+      "##nalpayload": 365,
+      "##katz": 366,
+      "##kprovider": 367,
+      "##atomicsfolder": 368,
+      "externalpayload": 369,
+      "pathtoatomicsfolder": 370,
+      "networkprovider": 371,
+      "externalpayloads": 372,
+      "ac": 373,
+      "dll": 374,
+      "mi": 375,
+      "si": 376,
+      "wh": 377,
+      "##eb": 378,
+      "##sk": 379,
+      "##sr": 380,
+      "##pe": 381,
+      "##ps": 382,
+      "##path": 383,
+      "##ig": 384,
+      "##ap": 385,
+      "##ac": 386,
+      "##ze": 387,
+      "##unt": 388,
+      "##cs": 389,
+      "##cmd": 390,
+      "##de": 391,
+      "##fig": 392,
+      "##64": 393,
+      "##ster": 394,
+      "##etsr": 395,
+      "##ity": 396,
+      "provider": 397,
+      "reg": 398,
+      "apppo": 399,
+      "appcmd": 400,
+      "inetsr": 401,
+      "master": 402,
+      "log": 403,
+      "secur": 404,
+      "save": 405,
+      "testcat": 406,
+      "config": 407,
+      "heap": 408,
+      "##shadow": 409,
+      "mimi": 410,
+      "size": 411,
+      "apppool": 412,
+      "inetsrv": 413,
+      "security": 414,
+      "cal": 415,
+      "dom": 416,
+      "els": 417,
+      "fo": 418,
+      "folder": 419,
+      "git": 420,
+      "glo": 421,
+      "ht": 422,
+      "har": 423,
+      "iex": 424,
+      "lin": 425,
+      "na": 426,
+      "pg": 427,
+      "ra": 428,
+      "rund": 429,
+      "su": 430,
+      "s3": 431,
+      "tr": 432,
+      "th": 433,
+      "ter": 434,
+      "vs": 435,
+      "wm": 436,
+      "wr": 437,
+      "##ect": 438,
+      "##se": 439,
+      "##sv": 440,
+      "##ich": 441,
+      "##ke": 442,
+      "##an": 443,
+      "##ain": 444,
+      "##tract": 445,
+      "##tps": 446,
+      "##ul": 447,
+      "##ub": 448,
+      "##do": 449,
+      "##ddi": 450,
+      "##wn": 451,
+      "##xt": 452,
+      "##gr": 453,
+      "##3th": 454,
+      "##h1": 455,
+      "##hub": 456,
+      "##volume": 457,
+      "##1ss": 458,
+      "##bal": 459,
+      "extract": 460,
+      "##rect": 461,
+      "##cept": 462,
+      "##ory": 463,
+      "##order": 464,
+      "##ite": 465,
+      "##ll32": 466,
+      "##ated": 467,
+      "##nodump": 468,
+      "##type": 469,
+      "shadowcopy": 470,
+      "ps1": 471,
+      "sam": 472,
+      "##cur3th": 473,
+      "comsv": 474,
+      "device": 475,
+      "direct": 476,
+      "##string": 477,
+      "unamestr": 478,
+      "##lient": 479,
+      "which": 480,
+      "##skvolume": 481,
+      "providerorder": 482,
+      "##shadowcopy": 483,
+      "mimikatz": 484,
+      "call": 485,
+      "domain": 486,
+      "else": 487,
+      "github": 488,
+      "global": 489,
+      "https": 490,
+      "harddi": 491,
+      "nanodump": 492,
+      "pgrep": 493,
+      "raw": 494,
+      "rundll32": 495,
+      "sudo": 496,
+      "s3cur3th": 497,
+      "term": 498,
+      "vsc": 499,
+      "wmic": 500,
+      "write": 501,
+      "##h1t": 502,
+      "##1ssh1t": 503,
+      "comsvcs": 504,
+      "directory": 505,
+      "##skvolumeshadowcopy": 506,
+      "globalroot": 507,
+      "harddiskvolumeshadowcopy": 508,
+      "s3cur3th1ssh1t": 509,
+      "at": 510,
+      "and": 511,
+      "be": 512,
+      "do": 513,
+      "dd": 514,
+      "dow": 515,
+      "dit": 516,
+      "dump": 517,
+      "dot": 518,
+      "es": 519,
+      "ed": 520,
+      "f2": 521,
+      "f1": 522,
+      "fre": 523,
+      "ib": 524,
+      "inv": 525,
+      "live": 526,
+      "mkdi": 527,
+      "no": 528,
+      "not": 529,
+      "of": 530,
+      "ob": 531,
+      "pow": 532,
+      "sk": 533,
+      "sv": 534,
+      "set": 535,
+      "to": 536,
+      "ta": 537,
+      "up": 538,
+      "user": 539,
+      "ve": 540,
+      "volume": 541,
+      "wa": 542,
+      "web": 543,
+      "x64": 544,
+      "##eul": 545,
+      "##so": 546,
+      "##sd": 547,
+      "##sw": 548,
+      "##pass": 549,
+      "##pwn": 550,
+      "##out": 551,
+      "##oke": 552,
+      "##net": 553,
+      "##nload": 554,
+      "##ip": 555,
+      "##mem": 556,
+      "##min": 557,
+      "##mit": 558,
+      "##mgr": 559,
+      "##kur": 560,
+      "##am": 561,
+      "##ab": 562,
+      "##user": 563,
+      "##col": 564,
+      "##client": 565,
+      "##dmp": 566,
+      "##dated": 567,
+      "##ymgr": 568,
+      "##38": 569,
+      "##21": 570,
+      "##2b": 571,
+      "##file": 572,
+      "##ver": 573,
+      "##value": 574,
+      "##56": 575,
+      "##ject": 576,
+      "##ert": 577,
+      "##erdump": 578,
+      "##utl": 579,
+      "##util": 580,
+      "##ink": 581,
+      "expa": 582,
+      "sysw": 583,
+      "##ow64": 584,
+      "exepath": 585,
+      "##admin": 586,
+      "##pykatz": 587,
+      "##ort": 588,
+      "lsa": 589,
+      "winpwn": 590,
+      "##lsa": 591,
+      "##count": 592,
+      "##code": 593,
+      "##chost": 594,
+      "res": 595,
+      "red": 596,
+      "text": 597,
+      "catch": 598,
+      "map": 599,
+      "pypykatz": 600,
+      "vssadmin": 601,
+      "##proto": 602,
+      "##content": 603,
+      "count": 604,
+      "loc": 605,
+      "limit": 606,
+      "sekur": 607,
+      "##entutl": 608,
+      "procdump": 609,
+      "scripts": 610,
+      "dest": 611,
+      "force": 612,
+      "fullmem": 613,
+      "head": 614,
+      "accept": 615,
+      "##ebsd": 616,
+      "logon": 617,
+      "securityproto": 618,
+      "found": 619,
+      "line": 620,
+      "try": 621,
+      "then": 622,
+      "githubuser": 623,
+      "download": 624,
+      "dotnet": 625,
+      "esentutl": 626,
+      "freebsd": 627,
+      "ibs": 628,
+      "invoke": 629,
+      "mkdir": 630,
+      "node": 631,
+      "object": 632,
+      "powerdump": 633,
+      "skip": 634,
+      "svchost": 635,
+      "tail": 636,
+      "updated": 637,
+      "venv": 638,
+      "wait": 639,
+      "webclient": 640,
+      "##eula": 641,
+      "expand": 642,
+      "syswow64": 643,
+      "sekurlsa": 644,
+      "fullmemdmp": 645,
+      "accepteula": 646,
+      "securityprotocol": 647,
+      "githubusercontent": 648,
+      "downloadstring": 649,
+      "updatedvalue": 650,
+      "121": 651,
+      "338": 652,
+      "9a": 653,
+      "ad": 654,
+      "all": 655,
+      "by": 656,
+      "bac": 657,
+      "cl": 658,
+      "cle": 659,
+      "cmd": 660,
+      "can": 661,
+      "client": 662,
+      "cert": 663,
+      "don": 664,
+      "dcs": 665,
+      "en": 666,
+      "est": 667,
+      "eli": 668,
+      "fi": 669,
+      "go": 670,
+      "gw": 671,
+      "gre": 672,
+      "gse": 673,
+      "is": 674,
+      "io": 675,
+      "iw": 676,
+      "imp": 677,
+      "ke": 678,
+      "kr": 679,
+      "mo": 680,
+      "mm": 681,
+      "mon": 682,
+      "mic": 683,
+      "mkl": 684,
+      "non": 685,
+      "null": 686,
+      "nul": 687,
+      "or": 688,
+      "own": 689,
+      "plea": 690,
+      "rem": 691,
+      "so": 692,
+      "sr": 693,
+      "sile": 694,
+      "ty": 695,
+      "tls": 696,
+      "txt": 697,
+      "xor": 698,
+      "##ed": 699,
+      "##e2": 700,
+      "##e9": 701,
+      "##sid": 702,
+      "##sab": 703,
+      "##scode": 704,
+      "##pon": 705,
+      "##pcon": 706,
+      "##pen": 707,
+      "##port": 708,
+      "##nc": 709,
+      "##nect": 710,
+      "##is": 711,
+      "##if": 712,
+      "##ib": 713,
+      "##ime": 714,
+      "##ml": 715,
+      "##man": 716,
+      "##a3": 717,
+      "##a7": 718,
+      "##are": 719,
+      "##tro": 720,
+      "##time": 721,
+      "##tman": 722,
+      "##ux": 723,
+      "##uin": 724,
+      "##ule": 725,
+      "##uscode": 726,
+      "##lac": 727,
+      "##ring": 728,
+      "##c5": 729,
+      "##cb": 730,
+      "##cre": 731,
+      "##cdump": 732,
+      "##can": 733,
+      "##dce": 734,
+      "##dule": 735,
+      "##guin": 736,
+      "##ync": 737,
+      "##3cb": 738,
+      "##hex": 739,
+      "##ft": 740,
+      "##fen": 741,
+      "##fil": 742,
+      "##12": 743,
+      "##68": 744,
+      "##6a7": 745,
+      "##73": 746,
+      "##773": 747,
+      "##821": 748,
+      "##41": 749,
+      "##ation": 750,
+      "##atuscode": 751,
+      "##roso": 752,
+      "##inter": 753,
+      "##intman": 754,
+      "##ination": 755,
+      "except": 756,
+      "exfil": 757,
+      "##lect": 758,
+      "##leout": 759,
+      "syml": 760,
+      "##owke": 761,
+      "filename": 762,
+      "##adump": 763,
+      "##ord": 764,
+      "property": 765,
+      "progr": 766,
+      "##olve": 767,
+      "##alport": 768,
+      "##altime": 769,
+      "lsadump": 770,
+      "win32": 771,
+      "##urce": 772,
+      "state": 773,
+      "string": 774,
+      "statuscode": 775,
+      "share": 776,
+      "##ote": 777,
+      "##active": 778,
+      "##tcpcon": 779,
+      "##core": 780,
+      "nettcpcon": 781,
+      "netcore": 782,
+      "read": 783,
+      "repo": 784,
+      "realtime": 785,
+      "team": 786,
+      "##ager": 787,
+      "input": 788,
+      "insid": 789,
+      "##replac": 790,
+      "##cessib": 791,
+      "maps": 792,
+      "##process": 793,
+      "##artex": 794,
+      "password": 795,
+      "createdump": 796,
+      "ifm": 797,
+      "locat": 798,
+      "pspath": 799,
+      "select": 800,
+      "server": 801,
+      "##ingprocess": 802,
+      "starting": 803,
+      "conso": 804,
+      "contro": 805,
+      "defen": 806,
+      "dir": 807,
+      "disk": 808,
+      "disab": 809,
+      "##shed": 810,
+      "##showke": 811,
+      "##pointman": 812,
+      "##words": 813,
+      "##aryco": 814,
+      "itemtype": 815,
+      "python3": 816,
+      "python2": 817,
+      "servicepointman": 818,
+      "0x41": 819,
+      "##server": 820,
+      "##lished": 821,
+      "ntdsutil": 822,
+      "account": 823,
+      "while": 824,
+      "##aca3": 825,
+      "##accessib": 826,
+      "providerpath": 827,
+      "logout": 828,
+      "mimipen": 829,
+      "fore": 830,
+      "linux": 831,
+      "true": 832,
+      "this": 833,
+      "samfile": 834,
+      "elseif": 835,
+      "harddiskvolumeshadowcopy1": 836,
+      "atomic": 837,
+      "atomics": 838,
+      "dumpert": 839,
+      "dumpcre": 840,
+      "##passwords": 841,
+      "##amfile": 842,
+      "##ablished": 843,
+      "##38c5": 844,
+      "##2b2b": 845,
+      "##563cb": 846,
+      "##5638c5": 847,
+      "##codehex": 848,
+      "respon": 849,
+      "resolve": 850,
+      "redcan": 851,
+      "located": 852,
+      "localport": 853,
+      "destination": 854,
+      "logonserver": 855,
+      "logonpasswords": 856,
+      "expandstring": 857,
+      "securityprotocoltype": 858,
+      "121dce": 859,
+      "3389": 860,
+      "adreplac": 861,
+      "bypass": 862,
+      "back": 863,
+      "class": 864,
+      "cleartex": 865,
+      "clientaccessib": 866,
+      "certutil": 867,
+      "done": 868,
+      "dcsync": 869,
+      "encodehex": 870,
+      "established": 871,
+      "elif": 872,
+      "going": 873,
+      "gwmi": 874,
+      "green": 875,
+      "gsecdump": 876,
+      "iwr": 877,
+      "import": 878,
+      "keymgr": 879,
+      "krshowke": 880,
+      "module": 881,
+      "microso": 882,
+      "mklink": 883,
+      "noninter": 884,
+      "order": 885,
+      "owningprocess": 886,
+      "please": 887,
+      "remote": 888,
+      "source": 889,
+      "src": 890,
+      "silent": 891,
+      "type": 892,
+      "tls12": 893,
+      "xordump": 894,
+      "##e26a7": 895,
+      "##e92b2b": 896,
+      "##nection": 897,
+      "##68821": 898,
+      "exception": 899,
+      "##leoutput": 900,
+      "symlink": 901,
+      "propertytype": 902,
+      "programfile": 903,
+      "strings": 904,
+      "shared": 905,
+      "nettcpconnection": 906,
+      "inside": 907,
+      "location": 908,
+      "consoleoutput": 909,
+      "control": 910,
+      "defender": 911,
+      "diskshadow": 912,
+      "disable": 913,
+      "servicepointmanager": 914,
+      "##aca368821": 915,
+      "mimipenguin": 916,
+      "dumpcreds": 917,
+      "##563cbe92b2b": 918,
+      "##5638c5773": 919,
+      "response": 920,
+      "redcanaryco": 921,
+      "121dcee26a7": 922,
+      "adreplaccount": 923,
+      "cleartext": 924,
+      "clientaccessible": 925,
+      "krshowkeymgr": 926,
+      "microsoft": 927,
+      "noninteractive": 928,
+      "programfiles": 929,
+      "##aca368821563cbe92b2b": 930,
+      "121dcee26a7aca368821563cbe92b2b": 931,
+      "121dcee26a7aca368821563cbe92b2b5638c5773": 932
+    }
+  }
+}

tokenizer_config.json

@@ -0,0 +1,57 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

vocab.txt

@@ -0,0 +1,933 @@
+[PAD]
+[UNK]
+[CLS]
+[SEP]
+[MASK]
+!
+"
+#
+$
+%
+&
+'
+(
+)
+*
++
+,
+-
+.
+/
+0
+1
+2
+3
+4
+5
+6
+7
+8
+9
+:
+;
+<
+=
+>
+?
+@
+[
+\
+]
+^
+_
+a
+b
+c
+d
+e
+f
+g
+h
+i
+j
+k
+l
+m
+n
+o
+p
+q
+r
+s
+t
+u
+v
+w
+x
+y
+z
+{
+|
+}
+##e
+##s
+##p
+##o
+##n
+##i
+##m
+##k
+##a
+##t
+##z
+##u
+##l
+##r
+##c
+##d
+##w
+##x
+##g
+##y
+##3
+##2
+##h
+##f
+##v
+##1
+##6
+##7
+##8
+##5
+##b
+##9
+##4
+##0
+##j
+##at
+##em
+##er
+##st
+##ut
+##ro
+##in
+ex
+##et
+##le
+##ss
+##ath
+path
+##re
+##on
+sy
+##ow
+##id
+##stem
+system
+##ile
+exe
+##mp
+file
+out
+##ar
+##ad
+##ass
+##put
+##ce
+##pp
+##py
+output
+##or
+##it
+##me
+##ve
+##emp
+pro
+##ol
+##al
+##ump
+##ct
+mem
+##nv
+##32
+##nt
+##ch
+ls
+pid
+temp
+win
+##om
+##ive
+##ur
+env
+st
+##ame
+##ll
+system32
+##di
+sh
+##ot
+##ion
+##act
+##der
+##dow
+##adow
+window
+windows
+ne
+##tc
+##ls
+##ract
+##co
+##ate
+##cho
+lsass
+etc
+it
+net
+pass
+re
+##no
+##ic
+##kl
+##kdi
+##ter
+echo
+hkl
+npp
+te
+##spy
+##pt
+##ag
+##lo
+##dump
+##lea
+hklm
+nppspy
+app
+cat
+er
+in
+name
+rd
+tar
+##nd
+##ice
+##ty
+##th
+##rlea
+##con
+##get
+##vid
+##vice
+##roract
+##rep
+##cess
+shadow
+new
+##kdiag
+erroract
+rdrlea
+target
+##vider
+erroraction
+rdrleakdiag
+cre
+get
+hive
+ig
+ma
+py
+vss
+##pro
+##ue
+##root
+##art
+##older
+passw
+##nore
+##cont
+create
+ignore
+co
+id
+if
+lo
+li
+min
+ps
+se
+sa
+sc
+ser
+##ent
+##pa
+##ipt
+##to
+##ull
+##ript
+##cur
+##ds
+##ing
+##idump
+proc
+process
+##alue
+##omic
+start
+test
+##load
+passwd
+minidump
+script
+00
+con
+com
+cur
+de
+di
+for
+full
+he
+t1
+##sh
+##po
+##per
+##name
+##wor
+##03
+##003
+##str
+##rols
+##rent
+##ary
+##copy
+item
+##thon
+python
+##controls
+service
+##omics
+001
+current
+t1003
+##perty
+##controlset
+currentcontrolset
+0x
+13
+bin
+cut
+dr
+dmp
+grep
+ho
+let
+nt
+rv
+uname
+value
+##en
+##ex
+##ser
+##op
+##nal
+##il
+##mi
+##md
+##kat
+##kpro
+##ume
+##und
+##li
+##cat
+##yload
+##folder
+##atomics
+##roperty
+exit
+exter
+pathto
+systemroot
+##emproperty
+##olume
+stop
+itemproperty
+networ
+copy
+list
+##payload
+services
+binary
+drive
+host
+letter
+ntds
+##nalpayload
+##katz
+##kprovider
+##atomicsfolder
+externalpayload
+pathtoatomicsfolder
+networkprovider
+externalpayloads
+ac
+dll
+mi
+si
+wh
+##eb
+##sk
+##sr
+##pe
+##ps
+##path
+##ig
+##ap
+##ac
+##ze
+##unt
+##cs
+##cmd
+##de
+##fig
+##64
+##ster
+##etsr
+##ity
+provider
+reg
+apppo
+appcmd
+inetsr
+master
+log
+secur
+save
+testcat
+config
+heap
+##shadow
+mimi
+size
+apppool
+inetsrv
+security
+cal
+dom
+els
+fo
+folder
+git
+glo
+ht
+har
+iex
+lin
+na
+pg
+ra
+rund
+su
+s3
+tr
+th
+ter
+vs
+wm
+wr
+##ect
+##se
+##sv
+##ich
+##ke
+##an
+##ain
+##tract
+##tps
+##ul
+##ub
+##do
+##ddi
+##wn
+##xt
+##gr
+##3th
+##h1
+##hub
+##volume
+##1ss
+##bal
+extract
+##rect
+##cept
+##ory
+##order
+##ite
+##ll32
+##ated
+##nodump
+##type
+shadowcopy
+ps1
+sam
+##cur3th
+comsv
+device
+direct
+##string
+unamestr
+##lient
+which
+##skvolume
+providerorder
+##shadowcopy
+mimikatz
+call
+domain
+else
+github
+global
+https
+harddi
+nanodump
+pgrep
+raw
+rundll32
+sudo
+s3cur3th
+term
+vsc
+wmic
+write
+##h1t
+##1ssh1t
+comsvcs
+directory
+##skvolumeshadowcopy
+globalroot
+harddiskvolumeshadowcopy
+s3cur3th1ssh1t
+at
+and
+be
+do
+dd
+dow
+dit
+dump
+dot
+es
+ed
+f2
+f1
+fre
+ib
+inv
+live
+mkdi
+no
+not
+of
+ob
+pow
+sk
+sv
+set
+to
+ta
+up
+user
+ve
+volume
+wa
+web
+x64
+##eul
+##so
+##sd
+##sw
+##pass
+##pwn
+##out
+##oke
+##net
+##nload
+##ip
+##mem
+##min
+##mit
+##mgr
+##kur
+##am
+##ab
+##user
+##col
+##client
+##dmp
+##dated
+##ymgr
+##38
+##21
+##2b
+##file
+##ver
+##value
+##56
+##ject
+##ert
+##erdump
+##utl
+##util
+##ink
+expa
+sysw
+##ow64
+exepath
+##admin
+##pykatz
+##ort
+lsa
+winpwn
+##lsa
+##count
+##code
+##chost
+res
+red
+text
+catch
+map
+pypykatz
+vssadmin
+##proto
+##content
+count
+loc
+limit
+sekur
+##entutl
+procdump
+scripts
+dest
+force
+fullmem
+head
+accept
+##ebsd
+logon
+securityproto
+found
+line
+try
+then
+githubuser
+download
+dotnet
+esentutl
+freebsd
+ibs
+invoke
+mkdir
+node
+object
+powerdump
+skip
+svchost
+tail
+updated
+venv
+wait
+webclient
+##eula
+expand
+syswow64
+sekurlsa
+fullmemdmp
+accepteula
+securityprotocol
+githubusercontent
+downloadstring
+updatedvalue
+121
+338
+9a
+ad
+all
+by
+bac
+cl
+cle
+cmd
+can
+client
+cert
+don
+dcs
+en
+est
+eli
+fi
+go
+gw
+gre
+gse
+is
+io
+iw
+imp
+ke
+kr
+mo
+mm
+mon
+mic
+mkl
+non
+null
+nul
+or
+own
+plea
+rem
+so
+sr
+sile
+ty
+tls
+txt
+xor
+##ed
+##e2
+##e9
+##sid
+##sab
+##scode
+##pon
+##pcon
+##pen
+##port
+##nc
+##nect
+##is
+##if
+##ib
+##ime
+##ml
+##man
+##a3
+##a7
+##are
+##tro
+##time
+##tman
+##ux
+##uin
+##ule
+##uscode
+##lac
+##ring
+##c5
+##cb
+##cre
+##cdump
+##can
+##dce
+##dule
+##guin
+##ync
+##3cb
+##hex
+##ft
+##fen
+##fil
+##12
+##68
+##6a7
+##73
+##773
+##821
+##41
+##ation
+##atuscode
+##roso
+##inter
+##intman
+##ination
+except
+exfil
+##lect
+##leout
+syml
+##owke
+filename
+##adump
+##ord
+property
+progr
+##olve
+##alport
+##altime
+lsadump
+win32
+##urce
+state
+string
+statuscode
+share
+##ote
+##active
+##tcpcon
+##core
+nettcpcon
+netcore
+read
+repo
+realtime
+team
+##ager
+input
+insid
+##replac
+##cessib
+maps
+##process
+##artex
+password
+createdump
+ifm
+locat
+pspath
+select
+server
+##ingprocess
+starting
+conso
+contro
+defen
+dir
+disk
+disab
+##shed
+##showke
+##pointman
+##words
+##aryco
+itemtype
+python3
+python2
+servicepointman
+0x41
+##server
+##lished
+ntdsutil
+account
+while
+##aca3
+##accessib
+providerpath
+logout
+mimipen
+fore
+linux
+true
+this
+samfile
+elseif
+harddiskvolumeshadowcopy1
+atomic
+atomics
+dumpert
+dumpcre
+##passwords
+##amfile
+##ablished
+##38c5
+##2b2b
+##563cb
+##5638c5
+##codehex
+respon
+resolve
+redcan
+located
+localport
+destination
+logonserver
+logonpasswords
+expandstring
+securityprotocoltype
+121dce
+3389
+adreplac
+bypass
+back
+class
+cleartex
+clientaccessib
+certutil
+done
+dcsync
+encodehex
+established
+elif
+going
+gwmi
+green
+gsecdump
+iwr
+import
+keymgr
+krshowke
+module
+microso
+mklink
+noninter
+order
+owningprocess
+please
+remote
+source
+src
+silent
+type
+tls12
+xordump
+##e26a7
+##e92b2b
+##nection
+##68821
+exception
+##leoutput
+symlink
+propertytype
+programfile
+strings
+shared
+nettcpconnection
+inside
+location
+consoleoutput
+control
+defender
+diskshadow
+disable
+servicepointmanager
+##aca368821
+mimipenguin
+dumpcreds
+##563cbe92b2b
+##5638c5773
+response
+redcanaryco
+121dcee26a7
+adreplaccount
+cleartext
+clientaccessible
+krshowkeymgr
+microsoft
+noninteractive
+programfiles
+##aca368821563cbe92b2b
+121dcee26a7aca368821563cbe92b2b
+121dcee26a7aca368821563cbe92b2b5638c5773