Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer.json

@@ -0,0 +1,1439 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [],
+  "normalizer": {
+    "type": "Sequence",
+    "normalizers": [
+      {
+        "type": "Nmt"
+      },
+      {
+        "type": "NFKC"
+      },
+      {
+        "type": "Replace",
+        "pattern": {
+          "Regex": " {2,}"
+        },
+        "content": " "
+      }
+    ]
+  },
+  "pre_tokenizer": {
+    "type": "Metaspace",
+    "replacement": "▁",
+    "prepend_scheme": "always",
+    "split": true
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "SpecialToken": {
+          "id": "[CLS]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "SpecialToken": {
+          "id": "[CLS]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "[SEP]",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {
+      "[CLS]": {
+        "id": "[CLS]",
+        "ids": [
+          2
+        ],
+        "tokens": [
+          "[CLS]"
+        ]
+      },
+      "[MASK]": {
+        "id": "[MASK]",
+        "ids": [
+          4
+        ],
+        "tokens": [
+          "[MASK]"
+        ]
+      },
+      "[SEP]": {
+        "id": "[SEP]",
+        "ids": [
+          3
+        ],
+        "tokens": [
+          "[SEP]"
+        ]
+      }
+    }
+  },
+  "decoder": {
+    "type": "Metaspace",
+    "replacement": "▁",
+    "prepend_scheme": "always",
+    "split": true
+  },
+  "model": {
+    "type": "Unigram",
+    "unk_id": null,
+    "vocab": [
+      [
+        "[PAD]",
+        0.0
+      ],
+      [
+        "[UNK]",
+        0.0
+      ],
+      [
+        "[CLS]",
+        0.0
+      ],
+      [
+        "[SEP]",
+        0.0
+      ],
+      [
+        "[MASK]",
+        0.0
+      ],
+      [
+        "▁",
+        -2.2669874333266247
+      ],
+      [
+        "च",
+        -2.786006402550786
+      ],
+      [
+        "𑀟",
+        -2.8116757194840254
+      ],
+      [
+        "𑀪",
+        -3.5606164434691685
+      ],
+      [
+        "▁𑀣च",
+        -3.5991393737030855
+      ],
+      [
+        "𑀢",
+        -3.864636589740668
+      ],
+      [
+        "𑀦",
+        -3.97954617435804
+      ],
+      [
+        "𑀯",
+        -4.049503115902942
+      ],
+      [
+        "𑀳",
+        -4.070299105644596
+      ],
+      [
+        "न",
+        -4.329327131459669
+      ],
+      [
+        "▁पच",
+        -4.373245528601
+      ],
+      [
+        "𑀱च",
+        -4.494913506145053
+      ],
+      [
+        "▁णच",
+        -4.506061662619905
+      ],
+      [
+        "▁𑀟च",
+        -4.508174393542715
+      ],
+      [
+        "प",
+        -4.563260014236485
+      ],
+      [
+        "ल",
+        -4.6660504316889515
+      ],
+      [
+        "पच",
+        -4.693185407124789
+      ],
+      [
+        "𑀠",
+        -4.716093185639954
+      ],
+      [
+        "𑀳च",
+        -4.767706155649877
+      ],
+      [
+        "णच",
+        -4.799817638945241
+      ],
+      [
+        "𑀣च",
+        -4.8196132650116095
+      ],
+      [
+        "𑀞च",
+        -4.822070928368266
+      ],
+      [
+        "𑁦",
+        -4.847947038897749
+      ],
+      [
+        "त",
+        -4.88596567561215
+      ],
+      [
+        "𑁣",
+        -4.914984014523666
+      ],
+      [
+        "▁𑀠च",
+        -4.967798198399587
+      ],
+      [
+        "𑀣",
+        -5.041145856895584
+      ],
+      [
+        "▁ढच",
+        -5.062609972414865
+      ],
+      [
+        "𑀪𑀢",
+        -5.063751431768221
+      ],
+      [
+        "𑀠च",
+        -5.069441786495574
+      ],
+      [
+        "𑀞𑀢",
+        -5.240610020818199
+      ],
+      [
+        "बच",
+        -5.256064903016357
+      ],
+      [
+        "ल𑀢",
+        -5.264875661351885
+      ],
+      [
+        "ढच",
+        -5.268850679367251
+      ],
+      [
+        "𑀲",
+        -5.293180522134605
+      ],
+      [
+        "▁𑀞च",
+        -5.305455781690661
+      ],
+      [
+        "लच",
+        -5.328335472961843
+      ],
+      [
+        "𑀢𑀟",
+        -5.3414952361621655
+      ],
+      [
+        "▁𑀫च",
+        -5.354547721989867
+      ],
+      [
+        "ध",
+        -5.357128303345258
+      ],
+      [
+        "▁𑀳च",
+        -5.358216528623759
+      ],
+      [
+        "ण",
+        -5.382432726544211
+      ],
+      [
+        "𑀭",
+        -5.439421087164062
+      ],
+      [
+        "▁बच",
+        -5.44244963531346
+      ],
+      [
+        "𑀪𑁦",
+        -5.445794746715411
+      ],
+      [
+        "ष",
+        -5.458945338949015
+      ],
+      [
+        "▁𑀞न𑀠च",
+        -5.459306803045019
+      ],
+      [
+        "𑀞",
+        -5.4812010438406364
+      ],
+      [
+        "𑀙",
+        -5.4873013747743435
+      ],
+      [
+        "▁त𑀢𑀞𑀢𑀟",
+        -5.52993843965273
+      ],
+      [
+        "ढ",
+        -5.534655102263505
+      ],
+      [
+        "𑀮",
+        -5.587762932777258
+      ],
+      [
+        "पन",
+        -5.5928754529377915
+      ],
+      [
+        "त𑁦",
+        -5.615402259997625
+      ],
+      [
+        "▁𑀝",
+        -5.622604469186928
+      ],
+      [
+        "▁𑀳𑀫𑀢",
+        -5.700814938050478
+      ],
+      [
+        "𑀞𑁦",
+        -5.706061156274009
+      ],
+      [
+        "𑀠𑀢",
+        -5.72846088931283
+      ],
+      [
+        "▁च𑀟",
+        -5.738834461396221
+      ],
+      [
+        "ढन",
+        -5.764733571837413
+      ],
+      [
+        "𑀫",
+        -5.777778185750739
+      ],
+      [
+        "𑀫च",
+        -5.778847863701973
+      ],
+      [
+        "▁𑀞𑁣",
+        -5.802164800752244
+      ],
+      [
+        "▁𑀟𑁦",
+        -5.842627559561773
+      ],
+      [
+        "𑀣𑀢",
+        -5.845011160736467
+      ],
+      [
+        "𑀟𑁦",
+        -5.868740940996782
+      ],
+      [
+        "▁𑀭थ",
+        -5.885669636902659
+      ],
+      [
+        "ल𑁦",
+        -5.888899441591043
+      ],
+      [
+        "𑀳𑀫𑀢",
+        -5.8906429107918115
+      ],
+      [
+        "थ",
+        -5.906397874213335
+      ],
+      [
+        "𑀪𑀢𑀟",
+        -5.915504888097216
+      ],
+      [
+        "ढ𑀢",
+        -5.921177074534688
+      ],
+      [
+        "ठ𑀧",
+        -5.921541691082654
+      ],
+      [
+        "𑀣न",
+        -5.922741224518967
+      ],
+      [
+        "𑀪𑁣",
+        -5.9361125583738605
+      ],
+      [
+        "𑀧",
+        -5.979724282525737
+      ],
+      [
+        "▁𑀤च",
+        -5.98071834258149
+      ],
+      [
+        "णच𑀪",
+        -5.983097365899878
+      ],
+      [
+        "▁𑀞न",
+        -5.989796808633498
+      ],
+      [
+        "ब",
+        -5.9958750988448895
+      ],
+      [
+        "ण𑀢",
+        -5.996674222405732
+      ],
+      [
+        "𑀳न",
+        -6.000050056803028
+      ],
+      [
+        "𑀗",
+        -6.0012253793268595
+      ],
+      [
+        "प𑀢",
+        -6.007370403866785
+      ],
+      [
+        "𑀫न",
+        -6.027064830591211
+      ],
+      [
+        "ब𑀢",
+        -6.03383083854272
+      ],
+      [
+        "ठ",
+        -6.035334421028637
+      ],
+      [
+        "▁𑀣चबच",
+        -6.035638075707427
+      ],
+      [
+        "प𑁣",
+        -6.0514583325492985
+      ],
+      [
+        "▁𑀲च",
+        -6.058368318526185
+      ],
+      [
+        "झ",
+        -6.063822325471406
+      ],
+      [
+        "𑀘𑀢",
+        -6.070222682203283
+      ],
+      [
+        "𑀬",
+        -6.072188145872673
+      ],
+      [
+        "𑀰",
+        -6.081530815221262
+      ],
+      [
+        "𑀳𑀢",
+        -6.082809726591856
+      ],
+      [
+        "▁ण𑀢",
+        -6.084329824697987
+      ],
+      [
+        "▁𑀠च𑀢",
+        -6.084922167633385
+      ],
+      [
+        "𑀞न",
+        -6.09768900094566
+      ],
+      [
+        "𑀲𑀢",
+        -6.100056913615909
+      ],
+      [
+        "𑀖",
+        -6.122291621056441
+      ],
+      [
+        "र",
+        -6.13589314014766
+      ],
+      [
+        "पच𑀪",
+        -6.14597513475219
+      ],
+      [
+        "▁𑀱च",
+        -6.150710892064071
+      ],
+      [
+        "▁चल",
+        -6.171266335759963
+      ],
+      [
+        "𑀣𑁦",
+        -6.1714684619448565
+      ],
+      [
+        "𑁢",
+        -6.180371638418411
+      ],
+      [
+        "▁𑀪च",
+        -6.1819884741070865
+      ],
+      [
+        "𑀳𑁣",
+        -6.183746127759324
+      ],
+      [
+        "प𑁦",
+        -6.193862657051291
+      ],
+      [
+        "▁च𑀞च",
+        -6.197319550918909
+      ],
+      [
+        "▁णच𑀟च",
+        -6.207576510893729
+      ],
+      [
+        "𑀪न",
+        -6.207711814125245
+      ],
+      [
+        "▁झच",
+        -6.208170274857917
+      ],
+      [
+        "𑀳𑀫𑁦",
+        -6.2203265002251875
+      ],
+      [
+        "▁लच",
+        -6.220430914179417
+      ],
+      [
+        "त𑀢",
+        -6.230223927185753
+      ],
+      [
+        "𑀠𑁦",
+        -6.235638393842727
+      ],
+      [
+        "𑀱च𑀪",
+        -6.236864238660024
+      ],
+      [
+        "▁प𑀳च",
+        -6.249337445097728
+      ],
+      [
+        "ण𑁦",
+        -6.253628187421064
+      ],
+      [
+        "▁𑀳न",
+        -6.2594320162044355
+      ],
+      [
+        "ल𑁣",
+        -6.260422547006826
+      ],
+      [
+        "▁ठ",
+        -6.277148288529435
+      ],
+      [
+        "▁त𑀢",
+        -6.277570002060687
+      ],
+      [
+        "त𑀢𑀟",
+        -6.292205742232236
+      ],
+      [
+        "ढ𑁣",
+        -6.295144044234677
+      ],
+      [
+        "▁𑀭",
+        -6.296176948193418
+      ],
+      [
+        "𑀞𑁣",
+        -6.302422262734099
+      ],
+      [
+        "▁𑀲𑀢",
+        -6.325104936602306
+      ],
+      [
+        "▁𑀠न",
+        -6.332804856584026
+      ],
+      [
+        "▁𑀱च𑀟𑀣च",
+        -6.333020829981287
+      ],
+      [
+        "𑀠𑁣",
+        -6.358955282186212
+      ],
+      [
+        "▁𑀳𑀫च",
+        -6.364692743552787
+      ],
+      [
+        "▁𑀞𑁦",
+        -6.376356704574944
+      ],
+      [
+        "𑀧𑀧",
+        -6.394747096023028
+      ],
+      [
+        "ढ𑁦",
+        -6.399635186489604
+      ],
+      [
+        "▁ढ𑀢",
+        -6.425480534826821
+      ],
+      [
+        "▁𑀣𑁣𑀟",
+        -6.427903693804895
+      ],
+      [
+        "𑀞𑀢𑀟",
+        -6.4293943388045065
+      ],
+      [
+        "ब𑁦",
+        -6.435836257586876
+      ],
+      [
+        "त𑀫",
+        -6.443134830903579
+      ],
+      [
+        "▁𑀤न𑀱च",
+        -6.467536516764458
+      ],
+      [
+        "𑀤च",
+        -6.474404311861699
+      ],
+      [
+        "𑀞च𑀪",
+        -6.480798552193551
+      ],
+      [
+        "𑀲𑀢𑀟",
+        -6.491820235868639
+      ],
+      [
+        "𑀫𑁣",
+        -6.49205092845019
+      ],
+      [
+        "▁𑀘च",
+        -6.497463112160293
+      ],
+      [
+        "𑀫𑀢",
+        -6.499216093235653
+      ],
+      [
+        "▁𑀞𑀱च",
+        -6.502197844862458
+      ],
+      [
+        "𑀟च𑀟",
+        -6.510747186446459
+      ],
+      [
+        "𑀠ढच",
+        -6.532508198669975
+      ],
+      [
+        "▁𑀱च𑀳च𑀟",
+        -6.53263240388214
+      ],
+      [
+        "𑀠च𑀪",
+        -6.536213375657544
+      ],
+      [
+        "ञ",
+        -6.548096007432397
+      ],
+      [
+        "बन",
+        -6.551230413510247
+      ],
+      [
+        "▁𑀞च𑀳च",
+        -6.554221514536218
+      ],
+      [
+        "𑀳𑁦",
+        -6.565196587571638
+      ],
+      [
+        "▁𑀳न𑀟",
+        -6.567228640676397
+      ],
+      [
+        "𑀤",
+        -6.56848720987278
+      ],
+      [
+        "▁𑀳𑀫𑁦𑀞च𑀪च",
+        -6.576060883975714
+      ],
+      [
+        "ब𑁣",
+        -6.584516300685221
+      ],
+      [
+        "▁ढचणच𑀟",
+        -6.6049112194164845
+      ],
+      [
+        "𑀟त𑀢",
+        -6.607532271358046
+      ],
+      [
+        "𑀱",
+        -6.609204478342505
+      ],
+      [
+        "▁𑀞च𑀟",
+        -6.614094855176726
+      ],
+      [
+        "ललच",
+        -6.62302147885678
+      ],
+      [
+        "▁𑀠चप𑀳चण𑀢𑀟",
+        -6.628530882891367
+      ],
+      [
+        "▁च𑀢𑀞𑀢",
+        -6.640774925916384
+      ],
+      [
+        "▁𑀳𑀫𑁦𑀞च𑀪च𑀪",
+        -6.645221623848247
+      ],
+      [
+        "𑀟ब",
+        -6.648457402064549
+      ],
+      [
+        "▁बचढच",
+        -6.659785125872981
+      ],
+      [
+        "▁𑀠च𑀳न",
+        -6.67192966021946
+      ],
+      [
+        "▁𑀳च𑀟",
+        -6.6758152049271064
+      ],
+      [
+        "𑀪च𑀪",
+        -6.681689878283729
+      ],
+      [
+        "▁पच𑀪𑁦",
+        -6.692209796805262
+      ],
+      [
+        "𑀘च",
+        -6.700713105614248
+      ],
+      [
+        "𑀕",
+        -6.706582981531659
+      ],
+      [
+        "𑀘𑁦",
+        -6.709935106269484
+      ],
+      [
+        "▁𑀱चपच𑀟",
+        -6.713628237154946
+      ],
+      [
+        "ञच",
+        -6.714402176318838
+      ],
+      [
+        "▁𑀠चपच",
+        -6.728207868201135
+      ],
+      [
+        "𑀣𑀢𑀟",
+        -6.731223362850338
+      ],
+      [
+        "ख",
+        -6.747790579381663
+      ],
+      [
+        "▁𑀳न𑀞च",
+        -6.7554136860821234
+      ],
+      [
+        "▁𑀢𑀟",
+        -6.756287790963272
+      ],
+      [
+        "▁𑀤च𑀠च",
+        -6.764694593124471
+      ],
+      [
+        "व",
+        -6.768670942864823
+      ],
+      [
+        "▁𑀞च𑀪",
+        -6.774640707127849
+      ],
+      [
+        "▁𑀲च𑀪𑀞𑁣",
+        -6.776492140140357
+      ],
+      [
+        "𑀣𑁣",
+        -6.783097924497586
+      ],
+      [
+        "▁बन",
+        -6.801609447970305
+      ],
+      [
+        "▁झच𑀳च",
+        -6.805207202647667
+      ],
+      [
+        "▁व",
+        -6.806479484303809
+      ],
+      [
+        "𑀠च𑀟",
+        -6.808239170287743
+      ],
+      [
+        "𑀱𑁣",
+        -6.810793852680458
+      ],
+      [
+        "▁𑀣𑁣",
+        -6.819869877159078
+      ],
+      [
+        "▁𑀳न𑀟च",
+        -6.820028043589511
+      ],
+      [
+        "𑀱न",
+        -6.8398778542703695
+      ],
+      [
+        "𑁥",
+        -6.843824229033617
+      ],
+      [
+        "𑀪च𑀟",
+        -6.846868322669586
+      ],
+      [
+        "▁𑀫च𑀪",
+        -6.8529024297419046
+      ],
+      [
+        "𑀠न",
+        -6.870881306563314
+      ],
+      [
+        "𑀘",
+        -6.872621480168956
+      ],
+      [
+        "▁त𑁣",
+        -6.880751556264741
+      ],
+      [
+        "𑀟𑁣",
+        -6.883580187634834
+      ],
+      [
+        "▁𑀪च𑀟च𑀪",
+        -6.890225012363404
+      ],
+      [
+        "झ𑀢",
+        -6.9115401995117605
+      ],
+      [
+        "▁𑀱च𑀟𑀢",
+        -6.937257204344018
+      ],
+      [
+        "ण𑁣",
+        -6.948542143937205
+      ],
+      [
+        "▁च𑀠𑀲च𑀟𑀢",
+        -6.975318900456724
+      ],
+      [
+        "▁ब𑀢𑀣च",
+        -7.004556672762602
+      ],
+      [
+        "प𑀫",
+        -7.0108435417573
+      ],
+      [
+        "▁𑀲च𑀪च",
+        -7.013000374916237
+      ],
+      [
+        "▁𑀟च𑀘𑁦𑀪𑀢णच",
+        -7.017126924541953
+      ],
+      [
+        "▁𑀞च𑀳च𑀟त𑁦",
+        -7.0378771860344145
+      ],
+      [
+        "ख𑁦",
+        -7.041821342228587
+      ],
+      [
+        "𑀲𑁦",
+        -7.044791464492937
+      ],
+      [
+        "▁𑀠नपच𑀟𑁦",
+        -7.050608038483334
+      ],
+      [
+        "▁𑀱च𑀟𑀟च𑀟",
+        -7.0556344765039825
+      ],
+      [
+        "𑀲न",
+        -7.065249869671975
+      ],
+      [
+        "▁ल𑁣𑀞चत𑀢𑀟",
+        -7.068582697916231
+      ],
+      [
+        "𑀘𑀢𑀟",
+        -7.084313201178135
+      ],
+      [
+        "▁त𑁦𑀱च",
+        -7.107937376170396
+      ],
+      [
+        "न𑀟𑀢",
+        -7.11680173654454
+      ],
+      [
+        "▁𑀣न𑀟𑀢णच",
+        -7.117901506575915
+      ],
+      [
+        "ध𑁦",
+        -7.125109331766312
+      ],
+      [
+        "▁ढ𑀢णन",
+        -7.13200164110112
+      ],
+      [
+        "ख𑀢",
+        -7.16608115044272
+      ],
+      [
+        "▁ञच𑀟",
+        -7.176262316177912
+      ],
+      [
+        "▁𑀳च𑀠न",
+        -7.1922606367366235
+      ],
+      [
+        "𑀲𑁣",
+        -7.202296447442794
+      ],
+      [
+        "न𑀪च",
+        -7.202790234497028
+      ],
+      [
+        "भ",
+        -7.223678144934873
+      ],
+      [
+        "𑀱𑀢",
+        -7.23092166147727
+      ],
+      [
+        "▁𑀣न𑀞",
+        -7.236019371462852
+      ],
+      [
+        "ट",
+        -7.28917798323328
+      ],
+      [
+        "▁ढचढढच𑀟",
+        -7.294004952497948
+      ],
+      [
+        "▁𑁣𑀲",
+        -7.321865075160904
+      ],
+      [
+        "ध𑁣",
+        -7.322992652129711
+      ],
+      [
+        "▁𑀘𑀢𑀫च𑀪",
+        -7.324615787244138
+      ],
+      [
+        "▁बच𑀳च𑀪",
+        -7.338670780934709
+      ],
+      [
+        "𑀩",
+        -7.355649063274885
+      ],
+      [
+        "▁𑀳𑀫𑁦𑀞च𑀪न",
+        -7.357895824162508
+      ],
+      [
+        "▁𑀪न𑀱च",
+        -7.365613235468572
+      ],
+      [
+        "▁𑀙णच𑀟",
+        -7.376182697584662
+      ],
+      [
+        "𑀜",
+        -7.377121796622534
+      ],
+      [
+        "▁𑀳च𑀠च",
+        -7.382923016276493
+      ],
+      [
+        "𑀠च𑀟च𑀤च𑀪पच",
+        -7.406340610153507
+      ],
+      [
+        "▁𑀞च𑀲च",
+        -7.423821770080513
+      ],
+      [
+        "प𑀢𑁣𑀟",
+        -7.43103328876836
+      ],
+      [
+        "▁प𑀫𑁦",
+        -7.440357122781659
+      ],
+      [
+        "▁च𑀲𑀢𑀪𑀞च",
+        -7.441925837150949
+      ],
+      [
+        "𑀚",
+        -7.448705742329819
+      ],
+      [
+        "▁𑀱च𑀘𑁦",
+        -7.457635488211505
+      ],
+      [
+        "द",
+        -7.538001861454607
+      ],
+      [
+        "▁𑀫च𑀟णच𑀪",
+        -7.567566989792876
+      ],
+      [
+        "▁पच𑀪𑀢𑀫𑀢",
+        -7.595196846713824
+      ],
+      [
+        "▁णच𑀟𑀞𑀢𑀟",
+        -7.618383912450572
+      ],
+      [
+        "▁𑀘च𑀠च𑀙च",
+        -7.664528802643909
+      ],
+      [
+        "▁𑀫च𑀢𑀲𑁦",
+        -7.675547792498287
+      ],
+      [
+        "फ",
+        -7.695698081398836
+      ],
+      [
+        "▁च𑀠न𑀪𑀞च",
+        -7.710738427353974
+      ],
+      [
+        "▁चणणन𑀞च",
+        -7.723870296811141
+      ],
+      [
+        "▁𑀠च𑀘चल𑀢𑀳च𑀪",
+        -7.731704684994083
+      ],
+      [
+        "▁𑀢ल𑀢𑀠𑀢",
+        -7.738798015504505
+      ],
+      [
+        "▁𑀪चणन𑀱च",
+        -7.756907715014616
+      ],
+      [
+        "▁𑀞न𑀟ब𑀢णच𑀪",
+        -7.78655366415571
+      ],
+      [
+        "▁च𑀠𑀠च",
+        -7.791396233972659
+      ],
+      [
+        "▁झन𑀟ब𑀢णच𑀪",
+        -7.813908394858114
+      ],
+      [
+        "▁चढ𑀢𑀟",
+        -7.822772500802586
+      ],
+      [
+        "▁𑀳चन𑀪च𑀟",
+        -7.828660947969597
+      ],
+      [
+        "▁ब𑀢𑀪𑀠च",
+        -7.846056155913912
+      ],
+      [
+        "▁𑀞च𑀪चपन",
+        -7.878567976700644
+      ],
+      [
+        "▁𑀠नल𑀞𑀢",
+        -7.905791443901018
+      ],
+      [
+        "▁𑀳चढ𑁣𑀣च",
+        -7.909455376050737
+      ],
+      [
+        "▁प𑀳च𑀞च𑀟𑀢𑀟",
+        -7.924556473895274
+      ],
+      [
+        "▁𑀳𑀢णच𑀳च",
+        -7.925386989821408
+      ],
+      [
+        "▁बन𑀣च𑀟च𑀪",
+        -7.92667435988375
+      ],
+      [
+        "▁ढचणणच𑀟च",
+        -7.943804779524196
+      ],
+      [
+        "▁𑀫न𑀞न𑀠च𑀪",
+        -7.952582880047272
+      ],
+      [
+        "▁झ𑀱चलल𑁣𑀟",
+        -7.963359317657721
+      ],
+      [
+        "▁च𑀪𑁦𑀱च",
+        -7.972987329738562
+      ],
+      [
+        "▁𑀞णचन",
+        -7.98060468823105
+      ],
+      [
+        "▁𑀱चञच𑀟𑀣च",
+        -7.982895572333355
+      ],
+      [
+        "▁𑀟न𑀟च",
+        -7.988196694735539
+      ],
+      [
+        "▁𑀘च𑀠𑀢𑀙च𑀪",
+        -7.998979180683928
+      ],
+      [
+        "▁𑀫च𑀟𑀟न",
+        -8.00591177501603
+      ],
+      [
+        "ड",
+        -8.015332160690871
+      ],
+      [
+        "▁𑀠च𑀞च𑀪च𑀟पच𑀪",
+        -8.165370426499305
+      ],
+      [
+        "▁ढ𑀢𑀟त𑀢𑀞𑁦",
+        -8.215375676350973
+      ],
+      [
+        "𑀪चणचणणन𑀟",
+        -8.225433975987663
+      ],
+      [
+        "▁𑀫च𑀢𑀲च𑀪",
+        -8.314696277216171
+      ],
+      [
+        "▁𑀠न𑀳च𑀠𑀠च𑀟",
+        -8.333026473048472
+      ],
+      [
+        "▁ठ𑀧𑀭थ",
+        -8.338151185491318
+      ],
+      [
+        "▁𑀞𑀱च𑀢𑀞𑀱चण𑁣",
+        -8.382750070436895
+      ],
+      [
+        "▁ठ𑀧𑀭𑁢",
+        -8.428150581794975
+      ],
+      [
+        "▁𑀠न𑀫चलल𑀢",
+        -8.443387780335135
+      ],
+      [
+        "▁𑀞च𑀠𑀲च𑀟𑀢𑀟",
+        -8.519876061278335
+      ],
+      [
+        "▁𑀘च𑀟च𑀢𑀪न",
+        -8.5246067265799
+      ],
+      [
+        "ग",
+        -8.618042527967393
+      ],
+      [
+        "𑀫च𑀟ण𑁣ण𑀢𑀟",
+        -8.623184832115488
+      ],
+      [
+        "▁𑀠च𑀙च𑀢𑀞चपच",
+        -8.674825356660758
+      ],
+      [
+        "▁𑀳𑁣𑀘𑁣𑀘𑀢",
+        -8.69727739139297
+      ],
+      [
+        "▁𑀣चढच𑀟ष𑀣चढच𑀟",
+        -8.698131237698837
+      ],
+      [
+        "▁णच𑀪𑀘𑁦𑀘𑁦𑀟𑀢णच",
+        -8.772784818936659
+      ],
+      [
+        "▁𑀲चढ𑀪च𑀢𑀪न",
+        -8.83311606158082
+      ],
+      [
+        "ङ",
+        -8.901501476377
+      ],
+      [
+        "𑀫च𑀢𑀲च𑀲𑀲न𑀟",
+        -8.980558734502821
+      ],
+      [
+        "𑀝",
+        -9.087417694468952
+      ],
+      [
+        "य",
+        -9.089387643263212
+      ],
+      [
+        "𑁤",
+        -9.11265045084259
+      ],
+      [
+        "▁𑀳𑀫𑀢𑀪ण𑁦ष𑀳𑀫𑀢𑀪ण𑁦",
+        -9.17921697599047
+      ],
+      [
+        "घ",
+        -9.610372079672912
+      ],
+      [
+        "𑀨",
+        -10.008525112958296
+      ],
+      [
+        "𑀡",
+        -10.076681190258874
+      ],
+      [
+        "म",
+        -10.310992315596303
+      ],
+      [
+        "𑀓",
+        -10.788654812007522
+      ],
+      [
+        "𑀥",
+        -10.890412807048753
+      ],
+      [
+        "श",
+        -11.18963782884693
+      ],
+      [
+        "𑀛",
+        -12.081965280087008
+      ],
+      [
+        "छ",
+        -12.87841161084291
+      ],
+      [
+        "ज",
+        -12.87851161084291
+      ],
+      [
+        "क",
+        -12.87851161084291
+      ]
+    ],
+    "byte_fallback": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "added_tokens_decoder": {},
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}