Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

special_tokens_map.json

@@ -0,0 +1,8 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "sep_token": "▁",
+  "unk_token": "<unk>"
+}

tokenizer.json

@@ -0,0 +1,2144 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "</s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<mask>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "▁",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Metaspace",
+    "replacement": "▁",
+    "add_prefix_space": true,
+    "prepend_scheme": "always"
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "</s>",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "</s>",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "</s>",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {
+      "</s>": {
+        "id": "</s>",
+        "ids": [
+          1
+        ],
+        "tokens": [
+          "</s>"
+        ]
+      },
+      "<s>": {
+        "id": "<s>",
+        "ids": [
+          0
+        ],
+        "tokens": [
+          "<s>"
+        ]
+      }
+    }
+  },
+  "decoder": {
+    "type": "Metaspace",
+    "replacement": "▁",
+    "add_prefix_space": true,
+    "prepend_scheme": "always"
+  },
+  "model": {
+    "type": "Unigram",
+    "unk_id": 2,
+    "vocab": [
+      [
+        "<s>",
+        0.0
+      ],
+      [
+        "</s>",
+        0.0
+      ],
+      [
+        "<unk>",
+        0.0
+      ],
+      [
+        "<pad>",
+        0.0
+      ],
+      [
+        "<mask>",
+        0.0
+      ],
+      [
+        "▁",
+        -1.851753745508601
+      ],
+      [
+        ".",
+        -3.467229702727165
+      ],
+      [
+        ",",
+        -3.4994580525827654
+      ],
+      [
+        "i",
+        -3.501997281321488
+      ],
+      [
+        "s",
+        -3.6675149287765354
+      ],
+      [
+        "e",
+        -3.740981327672804
+      ],
+      [
+        "t",
+        -3.756271820779278
+      ],
+      [
+        "a",
+        -3.851924688587939
+      ],
+      [
+        "d",
+        -3.9663641245891097
+      ],
+      [
+        "l",
+        -4.187184666104034
+      ],
+      [
+        "u",
+        -4.362726253174099
+      ],
+      [
+        "n",
+        -4.365281003269589
+      ],
+      [
+        "te",
+        -4.465178781498764
+      ],
+      [
+        "▁on",
+        -4.485617356589534
+      ],
+      [
+        "ma",
+        -4.507881482554289
+      ],
+      [
+        "ta",
+        -4.577405199912896
+      ],
+      [
+        "▁ja",
+        -4.659369622365345
+      ],
+      [
+        "le",
+        -4.754738949830481
+      ],
+      [
+        "da",
+        -4.763160408595947
+      ],
+      [
+        "k",
+        -4.764517002378536
+      ],
+      [
+        "se",
+        -4.82558826621422
+      ],
+      [
+        "id",
+        -4.839389364800779
+      ],
+      [
+        "▁et",
+        -4.863079140826789
+      ],
+      [
+        "o",
+        -4.868643884100571
+      ],
+      [
+        "me",
+        -4.88054197361431
+      ],
+      [
+        "b",
+        -4.893280824678866
+      ],
+      [
+        "ks",
+        -4.920152635536866
+      ],
+      [
+        "st",
+        -4.924191079610448
+      ],
+      [
+        "h",
+        -4.932466993741617
+      ],
+      [
+        "ne",
+        -5.01631846222519
+      ],
+      [
+        "r",
+        -5.023747986110356
+      ],
+      [
+        "m",
+        -5.109002791615339
+      ],
+      [
+        "va",
+        -5.12864421288047
+      ],
+      [
+        "▁ka",
+        -5.229718375045312
+      ],
+      [
+        "▁me",
+        -5.2748688708727265
+      ],
+      [
+        "tu",
+        -5.276698600171198
+      ],
+      [
+        "ra",
+        -5.393875035872222
+      ],
+      [
+        "unk",
+        -5.439702709633529
+      ],
+      [
+        "ga",
+        -5.450489872645525
+      ],
+      [
+        "▁Euroopa",
+        -5.471992917432958
+      ],
+      [
+        ">",
+        -5.474539494326656
+      ],
+      [
+        "<",
+        -5.474539494326656
+      ],
+      [
+        "us",
+        -5.505121366103829
+      ],
+      [
+        "el",
+        -5.510869021953342
+      ],
+      [
+        "▁selle",
+        -5.515787923126501
+      ],
+      [
+        "gi",
+        -5.518294934145231
+      ],
+      [
+        "ja",
+        -5.5639407280087365
+      ],
+      [
+        "▁see",
+        -5.580429802191812
+      ],
+      [
+        "ri",
+        -5.58303685327208
+      ],
+      [
+        "▁ei",
+        -5.602513947003798
+      ],
+      [
+        "is",
+        -5.608091010654608
+      ],
+      [
+        "ku",
+        -5.628117948676826
+      ],
+      [
+        "ü",
+        -5.641484201038043
+      ],
+      [
+        "li",
+        -5.662861124299768
+      ],
+      [
+        "ä",
+        -5.668312759894716
+      ],
+      [
+        "al",
+        -5.720288671730149
+      ],
+      [
+        "ge",
+        -5.749641201899214
+      ],
+      [
+        "vad",
+        -5.758167850952873
+      ],
+      [
+        "nud",
+        -5.768339084430565
+      ],
+      [
+        "na",
+        -5.773048422065626
+      ],
+      [
+        "re",
+        -5.815876593305079
+      ],
+      [
+        "õ",
+        -5.8345310774588235
+      ],
+      [
+        "v",
+        -5.846347926959826
+      ],
+      [
+        "mi",
+        -5.861293593653736
+      ],
+      [
+        "es",
+        -5.868850711872177
+      ],
+      [
+        "▁kui",
+        -5.878640833097753
+      ],
+      [
+        "ka",
+        -5.90206827934912
+      ],
+      [
+        "p",
+        -5.906236406204082
+      ],
+      [
+        "tud",
+        -5.921101614937527
+      ],
+      [
+        "as",
+        -5.926380765302193
+      ],
+      [
+        "ni",
+        -5.936832847802426
+      ],
+      [
+        "mu",
+        -6.00104349505018
+      ],
+      [
+        "▁meie",
+        -6.006002804792711
+      ],
+      [
+        "oo",
+        -6.054297433851142
+      ],
+      [
+        "ar",
+        -6.059389661732906
+      ],
+      [
+        "ju",
+        -6.066723831028785
+      ],
+      [
+        "g",
+        -6.06972031845312
+      ],
+      [
+        "in",
+        -6.097018886344532
+      ],
+      [
+        "▁p",
+        -6.105883135456947
+      ],
+      [
+        "▁või",
+        -6.113215945750243
+      ],
+      [
+        "▁ko",
+        -6.1567957224840075
+      ],
+      [
+        "▁mis",
+        -6.163709356807537
+      ],
+      [
+        "▁seda",
+        -6.181008954335827
+      ],
+      [
+        "▁saa",
+        -6.191139071265793
+      ],
+      [
+        "gu",
+        -6.191471923067434
+      ],
+      [
+        "val",
+        -6.2173714816893995
+      ],
+      [
+        "f",
+        -6.220665964968466
+      ],
+      [
+        "▁meil",
+        -6.23212090037727
+      ],
+      [
+        "ning",
+        -6.237519418960195
+      ],
+      [
+        "▁ole",
+        -6.2479564513302694
+      ],
+      [
+        "▁suur",
+        -6.252243524612862
+      ],
+      [
+        "mise",
+        -6.265632913496582
+      ],
+      [
+        "sed",
+        -6.27064974762089
+      ],
+      [
+        "he",
+        -6.304391835981752
+      ],
+      [
+        "lt",
+        -6.335409182785835
+      ],
+      [
+        "ke",
+        -6.3636196051178615
+      ],
+      [
+        "an",
+        -6.384023554179979
+      ],
+      [
+        "▁oma",
+        -6.398085477822427
+      ],
+      [
+        "▁kõik",
+        -6.39996827514852
+      ],
+      [
+        "K",
+        -6.414973586325018
+      ],
+      [
+        "▁kes",
+        -6.421202461615208
+      ],
+      [
+        "▁siis",
+        -6.444601657873719
+      ],
+      [
+        "▁väga",
+        -6.454035424594898
+      ],
+      [
+        "hi",
+        -6.458606115690512
+      ],
+      [
+        "-",
+        -6.477737902834786
+      ],
+      [
+        "▁aga",
+        -6.4817659799717315
+      ],
+      [
+        "▁vastu",
+        -6.485698390203481
+      ],
+      [
+        "▁mõ",
+        -6.50483048310699
+      ],
+      [
+        "▁peame",
+        -6.530990629241519
+      ],
+      [
+        "and",
+        -6.533164741058254
+      ],
+      [
+        "vi",
+        -6.5462373497933415
+      ],
+      [
+        "õigus",
+        -6.551011238509859
+      ],
+      [
+        "▁inim",
+        -6.5526255794563255
+      ],
+      [
+        "ed",
+        -6.55349251836417
+      ],
+      [
+        "il",
+        -6.556444293553909
+      ],
+      [
+        "ega",
+        -6.569508682508571
+      ],
+      [
+        "▁aitäh",
+        -6.5696686580664645
+      ],
+      [
+        "mine",
+        -6.577925306447063
+      ],
+      [
+        "on",
+        -6.588528368804588
+      ],
+      [
+        "ki",
+        -6.5992733330127695
+      ],
+      [
+        "▁nüüd",
+        -6.609780744644912
+      ],
+      [
+        "▁nii",
+        -6.616353848353142
+      ],
+      [
+        "▁nä",
+        -6.623407421551942
+      ],
+      [
+        "or",
+        -6.623922180093733
+      ],
+      [
+        "▁tei",
+        -6.636676590850732
+      ],
+      [
+        "enda",
+        -6.638791613354643
+      ],
+      [
+        "ve",
+        -6.6415748515454665
+      ],
+      [
+        "▁üle",
+        -6.647944470101434
+      ],
+      [
+        "▁töö",
+        -6.654199659773511
+      ],
+      [
+        "ru",
+        -6.716031688131752
+      ],
+      [
+        "▁Liidu",
+        -6.723786818500097
+      ],
+      [
+        "▁mida",
+        -6.732665213219939
+      ],
+      [
+        "tus",
+        -6.735612527529548
+      ],
+      [
+        "▁jä",
+        -6.738229195594432
+      ],
+      [
+        "ik",
+        -6.7449176494288325
+      ],
+      [
+        "ro",
+        -6.748206833663559
+      ],
+      [
+        "vä",
+        -6.751940318635054
+      ],
+      [
+        "▁vaja",
+        -6.7524525791053005
+      ],
+      [
+        "▁siin",
+        -6.766116744429205
+      ],
+      [
+        "des",
+        -6.76747226826607
+      ],
+      [
+        "eeri",
+        -6.771378098927969
+      ],
+      [
+        "▁kü",
+        -6.7782718395747175
+      ],
+      [
+        "lis",
+        -6.781678415088463
+      ],
+      [
+        "võ",
+        -6.784203171074139
+      ],
+      [
+        "tada",
+        -6.794105292934015
+      ],
+      [
+        "▁tõ",
+        -6.796936419626327
+      ],
+      [
+        "1",
+        -6.797587845662779
+      ],
+      [
+        "▁sa",
+        -6.809845522349655
+      ],
+      [
+        "0",
+        -6.81883730271349
+      ],
+      [
+        "▁nende",
+        -6.820569614841288
+      ],
+      [
+        "lik",
+        -6.8234168942644144
+      ],
+      [
+        "ha",
+        -6.828330361919034
+      ],
+      [
+        "võt",
+        -6.8432293260622625
+      ],
+      [
+        "▁sõna",
+        -6.8451083900236664
+      ],
+      [
+        "liku",
+        -6.849308105959519
+      ],
+      [
+        "selt",
+        -6.851119605223998
+      ],
+      [
+        "SP",
+        -6.860939821102747
+      ],
+      [
+        "(",
+        -6.861035166200734
+      ],
+      [
+        ")",
+        -6.861035166200734
+      ],
+      [
+        "takse",
+        -6.8672976844058535
+      ],
+      [
+        "▁tä",
+        -6.871962385337401
+      ],
+      [
+        "sti",
+        -6.873892877814959
+      ],
+      [
+        "lus",
+        -6.890947736544694
+      ],
+      [
+        "pi",
+        -6.894817259412826
+      ],
+      [
+        "▁pea",
+        -6.905637081122985
+      ],
+      [
+        "▁aasta",
+        -6.922647426283789
+      ],
+      [
+        "sime",
+        -6.9267490837436725
+      ],
+      [
+        "ko",
+        -6.927167665828413
+      ],
+      [
+        "▁la",
+        -6.928952996218759
+      ],
+      [
+        "mist",
+        -6.951943978555935
+      ],
+      [
+        "▁loo",
+        -6.96356743182482
+      ],
+      [
+        "▁oluli",
+        -6.969273116086222
+      ],
+      [
+        "tele",
+        -6.9700193574881935
+      ],
+      [
+        "j",
+        -6.996019995014971
+      ],
+      [
+        "S",
+        -7.004659171302743
+      ],
+      [
+        "▁need",
+        -7.006300481384357
+      ],
+      [
+        "▁osa",
+        -7.010837211050541
+      ],
+      [
+        "2",
+        -7.0130916295435615
+      ],
+      [
+        "▁oleme",
+        -7.015688969173557
+      ],
+      [
+        "tsi",
+        -7.015706233633436
+      ],
+      [
+        "▁täna",
+        -7.025908723203951
+      ],
+      [
+        "▁pro",
+        -7.031902686171742
+      ],
+      [
+        "ati",
+        -7.035389283833982
+      ],
+      [
+        "pe",
+        -7.041590457263219
+      ],
+      [
+        "sid",
+        -7.0665264828464265
+      ],
+      [
+        "pa",
+        -7.073622922078695
+      ],
+      [
+        "▁tea",
+        -7.083996165068698
+      ],
+      [
+        "▁tule",
+        -7.08548354281601
+      ],
+      [
+        "▁sõ",
+        -7.112099118601101
+      ],
+      [
+        "▁hea",
+        -7.124428100537737
+      ],
+      [
+        "▁taga",
+        -7.126201136801537
+      ],
+      [
+        "▁palju",
+        -7.140501730475849
+      ],
+      [
+        "pool",
+        -7.143088797199312
+      ],
+      [
+        "öö",
+        -7.146652842804281
+      ],
+      [
+        "▁muu",
+        -7.1473409422107474
+      ],
+      [
+        "siooni",
+        -7.16538232466433
+      ],
+      [
+        "▁tuleb",
+        -7.176750279820614
+      ],
+      [
+        "▁praegu",
+        -7.1832214613862435
+      ],
+      [
+        "▁su",
+        -7.1904916385105375
+      ],
+      [
+        "▁olema",
+        -7.190648510025193
+      ],
+      [
+        "▁kaits",
+        -7.2038243185739095
+      ],
+      [
+        "▁eest",
+        -7.20615783878872
+      ],
+      [
+        "▁neid",
+        -7.222820865407993
+      ],
+      [
+        "▁soovi",
+        -7.225624764685974
+      ],
+      [
+        "sest",
+        -7.229534493929964
+      ],
+      [
+        "är",
+        -7.230920200661927
+      ],
+      [
+        "▁mitte",
+        -7.232771879165971
+      ],
+      [
+        "▁palun",
+        -7.2563375806500625
+      ],
+      [
+        "pärast",
+        -7.257018987890197
+      ],
+      [
+        "test",
+        -7.2590805376033485
+      ],
+      [
+        "▁tänu",
+        -7.260628020577477
+      ],
+      [
+        "A",
+        -7.265024834498487
+      ],
+      [
+        "▁vii",
+        -7.276656141623064
+      ],
+      [
+        "▁nad",
+        -7.277285758604584
+      ],
+      [
+        "mo",
+        -7.280006686594515
+      ],
+      [
+        "c",
+        -7.288622351775304
+      ],
+      [
+        "▁kasuta",
+        -7.306931739243767
+      ],
+      [
+        "▁saab",
+        -7.307567868517468
+      ],
+      [
+        "▁oli",
+        -7.308193303449649
+      ],
+      [
+        "konna",
+        -7.3112564383056355
+      ],
+      [
+        "▁välja",
+        -7.31533799901075
+      ],
+      [
+        "▁oleks",
+        -7.320820851170182
+      ],
+      [
+        "▁piir",
+        -7.330656839999385
+      ],
+      [
+        "sin",
+        -7.345395297260704
+      ],
+      [
+        "L",
+        -7.353228209786732
+      ],
+      [
+        "▁kuida",
+        -7.356606133286253
+      ],
+      [
+        "▁härra",
+        -7.359443632454275
+      ],
+      [
+        "▁selli",
+        -7.359715942471686
+      ],
+      [
+        "▁peaks",
+        -7.363189123734943
+      ],
+      [
+        "▁üks",
+        -7.365198697015411
+      ],
+      [
+        "▁president",
+        -7.37096632050233
+      ],
+      [
+        "tuse",
+        -7.371661403476696
+      ],
+      [
+        "▁komisjon",
+        -7.377244368853256
+      ],
+      [
+        "lise",
+        -7.397653388934487
+      ],
+      [
+        "▁taha",
+        -7.398481428668544
+      ],
+      [
+        "5",
+        -7.403681633031638
+      ],
+      [
+        "▁tee",
+        -7.4165127500234975
+      ],
+      [
+        "po",
+        -7.430886046958901
+      ],
+      [
+        "koha",
+        -7.434542691659096
+      ],
+      [
+        "▁kolleeg",
+        -7.444400046789832
+      ],
+      [
+        "▁kuu",
+        -7.448341189161981
+      ],
+      [
+        "B",
+        -7.456799852463824
+      ],
+      [
+        "T",
+        -7.459726683589317
+      ],
+      [
+        "port",
+        -7.459804963676152
+      ],
+      [
+        "▁nagu",
+        -7.463630500885808
+      ],
+      [
+        "▁veel",
+        -7.467243596592985
+      ],
+      [
+        "▁mille",
+        -7.4716779770900255
+      ],
+      [
+        "jõu",
+        -7.476650550787945
+      ],
+      [
+        "▁räägi",
+        -7.487465050234481
+      ],
+      [
+        "▁teie",
+        -7.498282299910474
+      ],
+      [
+        "?",
+        -7.501628071497216
+      ],
+      [
+        "kond",
+        -7.5072838761483425
+      ],
+      [
+        "▁nõu",
+        -7.508347581153869
+      ],
+      [
+        "▁hä",
+        -7.509940178394881
+      ],
+      [
+        "▁peab",
+        -7.514157530124637
+      ],
+      [
+        "ö",
+        -7.529977334542933
+      ],
+      [
+        "▁kus",
+        -7.530942437413842
+      ],
+      [
+        "olla",
+        -7.535484573683446
+      ],
+      [
+        "▁läh",
+        -7.5368917384336
+      ],
+      [
+        "▁volinik",
+        -7.538996196345765
+      ],
+      [
+        "▁juba",
+        -7.543344092975019
+      ],
+      [
+        "▁ainul",
+        -7.54538232221728
+      ],
+      [
+        "äär",
+        -7.550503153695386
+      ],
+      [
+        "M",
+        -7.556063273077461
+      ],
+      [
+        "▁ühe",
+        -7.558288010617597
+      ],
+      [
+        "▁järgmise",
+        -7.564996726088508
+      ],
+      [
+        "▁Liit",
+        -7.583344400869443
+      ],
+      [
+        "P",
+        -7.583523662988431
+      ],
+      [
+        "▁ütle",
+        -7.594689158795801
+      ],
+      [
+        "▁toeta",
+        -7.598975914976487
+      ],
+      [
+        "▁võimalik",
+        -7.606378339922619
+      ],
+      [
+        "▁20",
+        -7.608263328954198
+      ],
+      [
+        "▁samut",
+        -7.612629051004373
+      ],
+      [
+        "3",
+        -7.618201944282982
+      ],
+      [
+        "▁teha",
+        -7.625687640736374
+      ],
+      [
+        "!",
+        -7.64253686197588
+      ],
+      [
+        "ndus",
+        -7.652811125253395
+      ],
+      [
+        "▁tagasi",
+        -7.655001451281471
+      ],
+      [
+        "▁majandus",
+        -7.6555310358587585
+      ],
+      [
+        "▁pan",
+        -7.662817092027097
+      ],
+      [
+        "▁lugupee",
+        -7.663877349598068
+      ],
+      [
+        "endi",
+        -7.6682838102827695
+      ],
+      [
+        "järg",
+        -7.6702731909997715
+      ],
+      [
+        "R",
+        -7.677146517406778
+      ],
+      [
+        "▁proua",
+        -7.683755633695213
+      ],
+      [
+        "▁rohkem",
+        -7.6918258109590125
+      ],
+      [
+        "▁rääki",
+        -7.69552943542292
+      ],
+      [
+        "poliitika",
+        -7.699246862153686
+      ],
+      [
+        "▁kee",
+        -7.71035081838196
+      ],
+      [
+        "▁vasta",
+        -7.720792627568789
+      ],
+      [
+        "▁minu",
+        -7.721313058233462
+      ],
+      [
+        "▁nimel",
+        -7.725999039539368
+      ],
+      [
+        "C",
+        -7.728216138002894
+      ],
+      [
+        "▁jaok",
+        -7.742613687583955
+      ],
+      [
+        "E",
+        -7.747571578223431
+      ],
+      [
+        "riikide",
+        -7.748921592961078
+      ],
+      [
+        "▁ära",
+        -7.751550272045874
+      ],
+      [
+        "▁liikmesriikide",
+        -7.763337381722763
+      ],
+      [
+        "otsus",
+        -7.768813582554823
+      ],
+      [
+        "abi",
+        -7.77460317815571
+      ],
+      [
+        "päev",
+        -7.777166016853058
+      ],
+      [
+        "▁läbi",
+        -7.786093634869793
+      ],
+      [
+        "I",
+        -7.7888008157487505
+      ],
+      [
+        "▁ühis",
+        -7.793128995786816
+      ],
+      [
+        "H",
+        -7.7983498420922945
+      ],
+      [
+        "kord",
+        -7.802438609915359
+      ],
+      [
+        "▁raha",
+        -7.817724970376148
+      ],
+      [
+        "V",
+        -7.817728438029739
+      ],
+      [
+        "▁kodanike",
+        -7.82474151720375
+      ],
+      [
+        "▁võimalus",
+        -7.8248436973938915
+      ],
+      [
+        "pp",
+        -7.828948341425596
+      ],
+      [
+        "▁kaasa",
+        -7.838860701915816
+      ],
+      [
+        "▁üht",
+        -7.84927175024195
+      ],
+      [
+        "▁toimu",
+        -7.851852118476616
+      ],
+      [
+        "▁arutelu",
+        -7.854744853360222
+      ],
+      [
+        "▁edasi",
+        -7.865099837176058
+      ],
+      [
+        "▁tahaksin",
+        -7.866454614107026
+      ],
+      [
+        "y",
+        -7.870825558682595
+      ],
+      [
+        "kriis",
+        -7.872797176517786
+      ],
+      [
+        "tiiv",
+        -7.873766919130496
+      ],
+      [
+        "▁selge",
+        -7.887491094188169
+      ],
+      [
+        "4",
+        -7.902265268031748
+      ],
+      [
+        "tegevus",
+        -7.905317747481934
+      ],
+      [
+        "▁riigi",
+        -7.910124078416441
+      ],
+      [
+        "▁koostöö",
+        -7.933171945802684
+      ],
+      [
+        "▁näite",
+        -7.93767512556262
+      ],
+      [
+        "▁Ukraina",
+        -7.944196869253502
+      ],
+      [
+        "sugu",
+        -7.949664602550985
+      ],
+      [
+        "▁kõigi",
+        -7.960185783033719
+      ],
+      [
+        "▁kõr",
+        -7.96187865951702
+      ],
+      [
+        "z",
+        -7.966651696233292
+      ],
+      [
+        "▁Poola",
+        -7.983005722600312
+      ],
+      [
+        "▁parem",
+        -7.983448126209992
+      ],
+      [
+        "▁minuti",
+        -7.984012246543839
+      ],
+      [
+        "▁olnud",
+        -7.996454589141428
+      ],
+      [
+        "▁austa",
+        -7.998598146175586
+      ],
+      [
+        "O",
+        -8.002989294442951
+      ],
+      [
+        "▁olukord",
+        -8.01143752014794
+      ],
+      [
+        "▁noor",
+        -8.013685299347088
+      ],
+      [
+        "▁sotsiaal",
+        -8.025100139064476
+      ],
+      [
+        "▁tegutse",
+        -8.025101742197556
+      ],
+      [
+        "▁erineva",
+        -8.026841668408755
+      ],
+      [
+        "▁tegemist",
+        -8.032021278940359
+      ],
+      [
+        "'",
+        -8.033734663387879
+      ],
+      [
+        "aalse",
+        -8.044720379778864
+      ],
+      [
+        "süsteem",
+        -8.058321060568908
+      ],
+      [
+        "▁koos",
+        -8.065907559114482
+      ],
+      [
+        "vahend",
+        -8.070013478907969
+      ],
+      [
+        "D",
+        -8.074448240312734
+      ],
+      [
+        "▁kiire",
+        -8.080155939335281
+      ],
+      [
+        "kirja",
+        -8.085483843387486
+      ],
+      [
+        "▁kõne",
+        -8.09645824059033
+      ],
+      [
+        "7",
+        -8.098215348761183
+      ],
+      [
+        "▁maailma",
+        -8.116892680262733
+      ],
+      [
+        "▁mitme",
+        -8.130478337180136
+      ],
+      [
+        "▁tegema",
+        -8.132322638313033
+      ],
+      [
+        "juht",
+        -8.133910854692843
+      ],
+      [
+        "ettepaneku",
+        -8.153060943171713
+      ],
+      [
+        "▁enam",
+        -8.153496259606413
+      ],
+      [
+        "▁tervis",
+        -8.16716280449062
+      ],
+      [
+        "▁tuleviku",
+        -8.17312604590213
+      ],
+      [
+        "N",
+        -8.179106371392958
+      ],
+      [
+        "▁vaba",
+        -8.18554695203786
+      ],
+      [
+        "U",
+        -8.195284674981874
+      ],
+      [
+        "▁liikme",
+        -8.195299977087725
+      ],
+      [
+        "▁liikmesriigi",
+        -8.205530678548651
+      ],
+      [
+        "väärtus",
+        -8.207636073053033
+      ],
+      [
+        "▁poliitili",
+        -8.211729029904978
+      ],
+      [
+        "▁tegelikul",
+        -8.211731546539438
+      ],
+      [
+        "6",
+        -8.213803717507972
+      ],
+      [
+        "9",
+        -8.215882719586974
+      ],
+      [
+        "▁parlamendi",
+        -8.239041347186697
+      ],
+      [
+        "▁kell",
+        -8.245126955724867
+      ],
+      [
+        "▁saavuta",
+        -8.254082999898273
+      ],
+      [
+        "▁naiste",
+        -8.256518482578183
+      ],
+      [
+        "▁milli",
+        -8.270065231048854
+      ],
+      [
+        "G",
+        -8.273714656313357
+      ],
+      [
+        "▁meetme",
+        -8.27824204605172
+      ],
+      [
+        "▁pandeemia",
+        -8.284801241894122
+      ],
+      [
+        "▁kinni",
+        -8.291590035212373
+      ],
+      [
+        "▁vahel",
+        -8.292317235281203
+      ],
+      [
+        "▁hääletus",
+        -8.30053227054644
+      ],
+      [
+        "▁Parlamendi",
+        -8.305072195631286
+      ],
+      [
+        "▁Venemaa",
+        -8.314215128964609
+      ],
+      [
+        "▁valitsus",
+        -8.330419728199754
+      ],
+      [
+        "▁lihtsalt",
+        -8.33275561329036
+      ],
+      [
+        "▁eesmärk",
+        -8.332755619557188
+      ],
+      [
+        "8",
+        -8.33509753247365
+      ],
+      [
+        "▁Komisjon",
+        -8.33744495033076
+      ],
+      [
+        "w",
+        -8.346890116496834
+      ],
+      [
+        "▁kliima",
+        -8.346896983977077
+      ],
+      [
+        "▁juurde",
+        -8.349265469567563
+      ],
+      [
+        "▁huvi",
+        -8.37832982406571
+      ],
+      [
+        "▁lõpe",
+        -8.385619670939489
+      ],
+      [
+        "▁juhataja",
+        -8.400496190340625
+      ],
+      [
+        "▁esindaja",
+        -8.400496691475183
+      ],
+      [
+        "▁seetõt",
+        -8.408060413910206
+      ],
+      [
+        "▁Hiina",
+        -8.425847561973622
+      ],
+      [
+        "toetus",
+        -8.426075170681987
+      ],
+      [
+        "▁olukorra",
+        -8.428416683333666
+      ],
+      [
+        "▁eriti",
+        -8.42848490042811
+      ],
+      [
+        "vabadus",
+        -8.438130592165686
+      ],
+      [
+        "F",
+        -8.438766057684811
+      ],
+      [
+        "▁seadus",
+        -8.462483686628985
+      ],
+      [
+        "▁puhul",
+        -8.4921834820906
+      ],
+      [
+        "▁investeeri",
+        -8.494930541483374
+      ],
+      [
+        "rühm",
+        -8.551911253253767
+      ],
+      [
+        "▁kõigepeal",
+        -8.554438115578218
+      ],
+      [
+        "▁tähendab",
+        -8.557362476378172
+      ],
+      [
+        "▁rahvusvaheli",
+        -8.58106688848573
+      ],
+      [
+        "▁kodanikud",
+        -8.590103116754666
+      ],
+      [
+        "š",
+        -8.68526191306795
+      ],
+      [
+        "▁konverents",
+        -8.739976360344667
+      ],
+      [
+        "institutsioon",
+        -8.743497502049788
+      ],
+      [
+        "strateegia",
+        -8.77575833331177
+      ],
+      [
+        "▁Putin",
+        -8.786748009380753
+      ],
+      [
+        "ž",
+        -8.794141164436255
+      ],
+      [
+        "tingimus",
+        -8.824279039082151
+      ],
+      [
+        "▁kontrolli",
+        -8.843580830988564
+      ],
+      [
+        "▁solidaarsus",
+        -8.851408649185894
+      ],
+      [
+        "▁kliimamuutus",
+        -8.851408951706524
+      ],
+      [
+        "▁parlament",
+        -8.867250544177251
+      ],
+      [
+        "▁demokraatlik",
+        -8.871250543935075
+      ],
+      [
+        "platvorm",
+        -8.879298866631109
+      ],
+      [
+        "▁eesmärgi",
+        -8.88334745119473
+      ],
+      [
+        "▁täiskogu",
+        -8.887412568882546
+      ],
+      [
+        "▁konkreetse",
+        -8.907989318825308
+      ],
+      [
+        "▁julgeoleku",
+        -8.93325378290737
+      ],
+      [
+        "▁Valgevene",
+        -8.994811576658421
+      ],
+      [
+        "▁läbirääkimis",
+        -9.00851039715884
+      ],
+      [
+        "tehnoloogia",
+        -9.027072381702595
+      ],
+      [
+        "▁Parlament",
+        -9.027072381704857
+      ],
+      [
+        "projekt",
+        -9.031767772892652
+      ],
+      [
+        "▁tähelepanu",
+        -9.060408738983025
+      ],
+      [
+        "programm",
+        -9.060409336465462
+      ],
+      [
+        "x",
+        -9.120276864001164
+      ],
+      [
+        "▁humanitaar",
+        -9.14105678588626
+      ],
+      [
+        "▁ambitsioon",
+        -9.141056799891471
+      ],
+      [
+        "▁euroopla",
+        -9.162277691489956
+      ],
+      [
+        "informatsioon",
+        -9.200533627940196
+      ],
+      [
+        "W",
+        -9.25197339844144
+      ],
+      [
+        "▁põllumajandus",
+        -9.269725292099054
+      ],
+      [
+        "▁asepresident",
+        -9.287798018498288
+      ],
+      [
+        "uudatusettepanek",
+        -9.307158863802869
+      ],
+      [
+        "▁korruptsioon",
+        -9.396880037827938
+      ],
+      [
+        "konventsioon",
+        -9.396880090903965
+      ],
+      [
+        "▁ebaseaduslik",
+        -9.424373069872576
+      ],
+      [
+        "▁Prantsusmaa",
+        -9.431366076861606
+      ],
+      [
+        "organisatsioon",
+        -9.438408330409231
+      ],
+      [
+        "%",
+        -9.82116264581201
+      ],
+      [
+        "mmmmmmmmmmmmmmm",
+        -9.9305601459234
+      ],
+      [
+        "Z",
+        -9.953952758782524
+      ],
+      [
+        "q",
+        -10.13508903572938
+      ],
+      [
+        "é",
+        -10.44911261190077
+      ],
+      [
+        "á",
+        -10.617927172765626
+      ],
+      [
+        "č",
+        -10.689382754431072
+      ],
+      [
+        "Q",
+        -11.587196800138464
+      ],
+      [
+        "+",
+        -12.138957539400424
+      ],
+      [
+        "ó",
+        -12.684592460046924
+      ],
+      [
+        "è",
+        -12.88459246008688
+      ],
+      [
+        "/",
+        -12.884592460086884
+      ],
+      [
+        "ń",
+        -12.884592460086884
+      ],
+      [
+        "ñ",
+        -13.134592460086882
+      ],
+      [
+        "İ",
+        -13.467925793420209
+      ],
+      [
+        "ê",
+        -13.46792579342021
+      ],
+      [
+        "ć",
+        -13.46792579342021
+      ],
+      [
+        "ç",
+        -13.467925793420216
+      ],
+      [
+        "à",
+        -13.467925793420216
+      ],
+      [
+        "ú",
+        -13.96792579338026
+      ],
+      [
+        "ò",
+        -13.967925793420212
+      ],
+      [
+        "ô",
+        -13.967925793420212
+      ],
+      [
+        "ș",
+        -13.967925793420218
+      ],
+      [
+        "í",
+        -13.967925793420218
+      ],
+      [
+        "ō",
+        -13.967925793420218
+      ],
+      [
+        "å",
+        -14.966825793420217
+      ],
+      [
+        "J",
+        -14.966925793420216
+      ],
+      [
+        "X",
+        -14.967025793420216
+      ],
+      [
+        "Ö",
+        -14.967125793420216
+      ],
+      [
+        "Y",
+        -14.967225793420216
+      ],
+      [
+        "Ä",
+        -14.967325793420216
+      ],
+      [
+        "Š",
+        -14.967425793420215
+      ],
+      [
+        "ą",
+        -14.967525793420217
+      ],
+      [
+        "ğ",
+        -14.967625793420217
+      ],
+      [
+        "Ü",
+        -14.967725793420216
+      ],
+      [
+        "ř",
+        -14.967825793420216
+      ],
+      [
+        "ś",
+        -14.967925793420193
+      ],
+      [
+        "Ž",
+        -14.967925793420216
+      ],
+      [
+        "ã",
+        -14.967925793420216
+      ],
+      [
+        ":",
+        -14.967925793420216
+      ],
+      [
+        "Õ",
+        -14.967925793420216
+      ],
+      [
+        "²",
+        -14.967925793420216
+      ],
+      [
+        "ł",
+        -14.967925793420216
+      ],
+      [
+        "Á",
+        -14.967925793420216
+      ],
+      [
+        "ū",
+        -14.967925793420216
+      ],
+      [
+        "ż",
+        -14.967925793420216
+      ]
+    ],
+    "byte_fallback": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,61 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sep_token": "▁",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}