Upload tokenizer

README.md

@@ -0,0 +1,201 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+

tokenizer.json

@@ -302,7 +302,6 @@
       "ॽ": 164,
       "ॾ": 165,
       "“": 166,
-      "▁": 167,
       "▁क": 168,
       "े▁": 169,
       "ा▁": 170,
@@ -32134,7 +32133,278 @@
       "रेक": 31996,
       "वरन▁": 31997,
       "उपयोगकर्ता▁को▁": 31998,
-      "से▁से▁": 31999
+      "से▁से▁": 31999,
+      "<0x00>": 32000,
+      "<0x01>": 32001,
+      "<0x02>": 32002,
+      "<0x03>": 32003,
+      "<0x04>": 32004,
+      "<0x05>": 32005,
+      "<0x06>": 32006,
+      "<0x07>": 32007,
+      "<0x08>": 32008,
+      "<0x09>": 32009,
+      "<0x0a>": 32010,
+      "<0x0b>": 32011,
+      "<0x0c>": 32012,
+      "<0x0d>": 32013,
+      "<0x0e>": 32014,
+      "<0x0f>": 32015,
+      "<0x10>": 32016,
+      "<0x11>": 32017,
+      "<0x12>": 32018,
+      "<0x13>": 32019,
+      "<0x14>": 32020,
+      "<0x15>": 32021,
+      "<0x16>": 32022,
+      "<0x17>": 32023,
+      "<0x18>": 32024,
+      "<0x19>": 32025,
+      "<0x1a>": 32026,
+      "<0x1b>": 32027,
+      "<0x1c>": 32028,
+      "<0x1d>": 32029,
+      "<0x1e>": 32030,
+      "<0x1f>": 32031,
+      "<0x20>": 32032,
+      "<0x21>": 32033,
+      "<0x22>": 32034,
+      "<0x23>": 32035,
+      "<0x24>": 32036,
+      "<0x25>": 32037,
+      "<0x26>": 32038,
+      "<0x27>": 32039,
+      "<0x28>": 32040,
+      "<0x29>": 32041,
+      "<0x2a>": 32042,
+      "<0x2b>": 32043,
+      "<0x2c>": 32044,
+      "<0x2d>": 32045,
+      "<0x2e>": 32046,
+      "<0x2f>": 32047,
+      "<0x30>": 32048,
+      "<0x31>": 32049,
+      "<0x32>": 32050,
+      "<0x33>": 32051,
+      "<0x34>": 32052,
+      "<0x35>": 32053,
+      "<0x36>": 32054,
+      "<0x37>": 32055,
+      "<0x38>": 32056,
+      "<0x39>": 32057,
+      "<0x3a>": 32058,
+      "<0x3b>": 32059,
+      "<0x3c>": 32060,
+      "<0x3d>": 32061,
+      "<0x3e>": 32062,
+      "<0x3f>": 32063,
+      "<0x40>": 32064,
+      "<0x41>": 32065,
+      "<0x42>": 32066,
+      "<0x43>": 32067,
+      "<0x44>": 32068,
+      "<0x45>": 32069,
+      "<0x46>": 32070,
+      "<0x47>": 32071,
+      "<0x48>": 32072,
+      "<0x49>": 32073,
+      "<0x4a>": 32074,
+      "<0x4b>": 32075,
+      "<0x4c>": 32076,
+      "<0x4d>": 32077,
+      "<0x4e>": 32078,
+      "<0x4f>": 32079,
+      "<0x50>": 32080,
+      "<0x51>": 32081,
+      "<0x52>": 32082,
+      "<0x53>": 32083,
+      "<0x54>": 32084,
+      "<0x55>": 32085,
+      "<0x56>": 32086,
+      "<0x57>": 32087,
+      "<0x58>": 32088,
+      "<0x59>": 32089,
+      "<0x5a>": 32090,
+      "<0x5b>": 32091,
+      "<0x5c>": 32092,
+      "<0x5d>": 32093,
+      "<0x5e>": 32094,
+      "<0x5f>": 32095,
+      "<0x60>": 32096,
+      "<0x61>": 32097,
+      "<0x62>": 32098,
+      "<0x63>": 32099,
+      "<0x64>": 32100,
+      "<0x65>": 32101,
+      "<0x66>": 32102,
+      "<0x67>": 32103,
+      "<0x68>": 32104,
+      "<0x69>": 32105,
+      "<0x6a>": 32106,
+      "<0x6b>": 32107,
+      "<0x6c>": 32108,
+      "<0x6d>": 32109,
+      "<0x6e>": 32110,
+      "<0x6f>": 32111,
+      "<0x70>": 32112,
+      "<0x71>": 32113,
+      "<0x72>": 32114,
+      "<0x73>": 32115,
+      "<0x74>": 32116,
+      "<0x75>": 32117,
+      "<0x76>": 32118,
+      "<0x77>": 32119,
+      "<0x78>": 32120,
+      "<0x79>": 32121,
+      "<0x7a>": 32122,
+      "<0x7b>": 32123,
+      "<0x7c>": 32124,
+      "<0x7d>": 32125,
+      "<0x7e>": 32126,
+      "<0x7f>": 32127,
+      "<0x80>": 32128,
+      "<0x81>": 32129,
+      "<0x82>": 32130,
+      "<0x83>": 32131,
+      "<0x84>": 32132,
+      "<0x85>": 32133,
+      "<0x86>": 32134,
+      "<0x87>": 32135,
+      "<0x88>": 32136,
+      "<0x89>": 32137,
+      "<0x8a>": 32138,
+      "<0x8b>": 32139,
+      "<0x8c>": 32140,
+      "<0x8d>": 32141,
+      "<0x8e>": 32142,
+      "<0x8f>": 32143,
+      "<0x90>": 32144,
+      "<0x91>": 32145,
+      "<0x92>": 32146,
+      "<0x93>": 32147,
+      "<0x94>": 32148,
+      "<0x95>": 32149,
+      "<0x96>": 32150,
+      "<0x97>": 32151,
+      "<0x98>": 32152,
+      "<0x99>": 32153,
+      "<0x9a>": 32154,
+      "<0x9b>": 32155,
+      "<0x9c>": 32156,
+      "<0x9d>": 32157,
+      "<0x9e>": 32158,
+      "<0x9f>": 32159,
+      "<0xa0>": 32160,
+      "<0xa1>": 32161,
+      "<0xa2>": 32162,
+      "<0xa3>": 32163,
+      "<0xa4>": 32164,
+      "<0xa5>": 32165,
+      "<0xa6>": 32166,
+      "<0xa7>": 32167,
+      "<0xa8>": 32168,
+      "<0xa9>": 32169,
+      "<0xaa>": 32170,
+      "<0xab>": 32171,
+      "<0xac>": 32172,
+      "<0xad>": 32173,
+      "<0xae>": 32174,
+      "<0xaf>": 32175,
+      "<0xb0>": 32176,
+      "<0xb1>": 32177,
+      "<0xb2>": 32178,
+      "<0xb3>": 32179,
+      "<0xb4>": 32180,
+      "<0xb5>": 32181,
+      "<0xb6>": 32182,
+      "<0xb7>": 32183,
+      "<0xb8>": 32184,
+      "<0xb9>": 32185,
+      "<0xba>": 32186,
+      "<0xbb>": 32187,
+      "<0xbc>": 32188,
+      "<0xbd>": 32189,
+      "<0xbe>": 32190,
+      "<0xbf>": 32191,
+      "<0xc0>": 32192,
+      "<0xc1>": 32193,
+      "<0xc2>": 32194,
+      "<0xc3>": 32195,
+      "<0xc4>": 32196,
+      "<0xc5>": 32197,
+      "<0xc6>": 32198,
+      "<0xc7>": 32199,
+      "<0xc8>": 32200,
+      "<0xc9>": 32201,
+      "<0xca>": 32202,
+      "<0xcb>": 32203,
+      "<0xcc>": 32204,
+      "<0xcd>": 32205,
+      "<0xce>": 32206,
+      "<0xcf>": 32207,
+      "<0xd0>": 32208,
+      "<0xd1>": 32209,
+      "<0xd2>": 32210,
+      "<0xd3>": 32211,
+      "<0xd4>": 32212,
+      "<0xd5>": 32213,
+      "<0xd6>": 32214,
+      "<0xd7>": 32215,
+      "<0xd8>": 32216,
+      "<0xd9>": 32217,
+      "<0xda>": 32218,
+      "<0xdb>": 32219,
+      "<0xdc>": 32220,
+      "<0xdd>": 32221,
+      "<0xde>": 32222,
+      "<0xdf>": 32223,
+      "<0xe0>": 32224,
+      "<0xe1>": 32225,
+      "<0xe2>": 32226,
+      "<0xe3>": 32227,
+      "<0xe4>": 32228,
+      "<0xe5>": 32229,
+      "<0xe6>": 32230,
+      "<0xe7>": 32231,
+      "<0xe8>": 32232,
+      "<0xe9>": 32233,
+      "<0xea>": 32234,
+      "<0xeb>": 32235,
+      "<0xec>": 32236,
+      "<0xed>": 32237,
+      "<0xee>": 32238,
+      "<0xef>": 32239,
+      "<0xf0>": 32240,
+      "<0xf1>": 32241,
+      "<0xf2>": 32242,
+      "<0xf3>": 32243,
+      "<0xf4>": 32244,
+      "<0xf5>": 32245,
+      "<0xf6>": 32246,
+      "<0xf7>": 32247,
+      "<0xf8>": 32248,
+      "<0xf9>": 32249,
+      "<0xfa>": 32250,
+      "<0xfb>": 32251,
+      "<0xfc>": 32252,
+      "<0xfd>": 32253,
+      "<0xfe>": 32254,
+      "<0xff>": 32255,
+      "▁": 32256,
+      "▁▁": 32257,
+      "▁▁▁": 32258,
+      "▁▁▁▁": 32259,
+      "▁▁▁▁▁": 32260,
+      "▁▁▁▁▁▁": 32261,
+      "▁▁▁▁▁▁▁": 32262,
+      "▁▁▁▁▁▁▁▁": 32263,
+      "▁▁▁▁▁▁▁▁▁": 32264,
+      "▁▁▁▁▁▁▁▁▁▁": 32265,
+      "▁▁▁▁▁▁▁▁▁▁▁": 32266,
+      "▁▁▁▁▁▁▁▁▁▁▁▁": 32267,
+      "▁▁▁▁▁▁▁▁▁▁▁▁▁": 32268,
+      "▁▁▁▁▁▁▁▁▁▁▁▁▁▁": 32269,
+      "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁": 32270
     },
     "merges": [
       "▁ क",
@@ -63968,7 +64238,98 @@
       "रे क",
       "वर न▁",
       "उपयोगकर् ता▁को▁",
-      "से▁ से▁"
+      "से▁ से▁",
+      "▁ ▁",
+      "▁ ▁▁",
+      "▁ ▁▁▁",
+      "▁ ▁▁▁▁",
+      "▁ ▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁▁▁▁▁▁▁",
+      "▁ ▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "▁▁ ▁",
+      "▁▁ ▁▁",
+      "▁▁ ▁▁▁",
+      "▁▁ ▁▁▁▁",
+      "▁▁ ▁▁▁▁▁",
+      "▁▁ ▁▁▁▁▁▁",
+      "▁▁ ▁▁▁▁▁▁▁",
+      "▁▁ ▁▁▁▁▁▁▁▁",
+      "▁▁ ▁▁▁▁▁▁▁▁▁",
+      "▁▁ ▁▁▁▁▁▁▁▁▁▁",
+      "▁▁ ▁▁▁▁▁▁▁▁▁▁▁",
+      "▁▁ ▁▁▁▁▁▁▁▁▁▁▁▁",
+      "▁▁▁ ▁",
+      "▁▁▁ ▁▁",
+      "▁▁▁ ▁▁▁",
+      "▁▁▁ ▁▁▁▁",
+      "▁▁▁ ▁▁▁▁▁",
+      "▁▁▁ ▁▁▁▁▁▁",
+      "▁▁▁ ▁▁▁▁▁▁▁",
+      "▁▁▁ ▁▁▁▁▁▁▁▁",
+      "▁▁▁ ▁▁▁▁▁▁▁▁▁",
+      "▁▁▁ ▁▁▁▁▁▁▁▁▁▁",
+      "▁▁▁ ▁▁▁▁▁▁▁▁▁▁▁",
+      "▁▁▁▁ ▁",
+      "▁▁▁▁ ▁▁",
+      "▁▁▁▁ ▁▁▁",
+      "▁▁▁▁ ▁▁▁▁",
+      "▁▁▁▁ ▁▁▁▁▁",
+      "▁▁▁▁ ▁▁▁▁▁▁",
+      "▁▁▁▁ ▁▁▁▁▁▁▁",
+      "▁▁▁▁ ▁▁▁▁▁▁▁▁",
+      "▁▁▁▁ ▁▁▁▁▁▁▁▁▁",
+      "▁▁▁▁ ▁▁▁▁▁▁▁▁▁▁",
+      "▁▁▁▁▁ ▁",
+      "▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁ ▁▁▁",
+      "▁▁▁▁▁ ▁▁▁▁",
+      "▁▁▁▁▁ ▁▁▁▁▁",
+      "▁▁▁▁▁ ▁▁▁▁▁▁",
+      "▁▁▁▁▁ ▁▁▁▁▁▁▁",
+      "▁▁▁▁▁ ▁▁▁▁▁▁▁▁",
+      "▁▁▁▁▁ ▁▁▁▁▁▁▁▁▁",
+      "▁▁▁▁▁▁ ▁",
+      "▁▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁▁ ▁▁▁",
+      "▁▁▁▁▁▁ ▁▁▁▁",
+      "▁▁▁▁▁▁ ▁▁▁▁▁",
+      "▁▁▁▁▁▁ ▁▁▁▁▁▁",
+      "▁▁▁▁▁▁ ▁▁▁▁▁▁▁",
+      "▁▁▁▁▁▁ ▁▁▁▁▁▁▁▁",
+      "▁▁▁▁▁▁▁ ▁",
+      "▁▁▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁▁▁ ▁▁▁",
+      "▁▁▁▁▁▁▁ ▁▁▁▁",
+      "▁▁▁▁▁▁▁ ▁▁▁▁▁",
+      "▁▁▁▁▁▁▁ ▁▁▁▁▁▁",
+      "▁▁▁▁▁▁▁ ▁▁▁▁▁▁▁",
+      "▁▁▁▁▁▁▁▁ ▁",
+      "▁▁▁▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁▁▁▁ ▁▁▁",
+      "▁▁▁▁▁▁▁▁ ▁▁▁▁",
+      "▁▁▁▁▁▁▁▁ ▁▁▁▁▁",
+      "▁▁▁▁▁▁▁▁ ▁▁▁▁▁▁",
+      "▁▁▁▁▁▁▁▁▁ ▁",
+      "▁▁▁▁▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁▁▁▁▁ ▁▁▁",
+      "▁▁▁▁▁▁▁▁▁ ▁▁▁▁",
+      "▁▁▁▁▁▁▁▁▁ ▁▁▁▁▁",
+      "▁▁▁▁▁▁▁▁▁▁ ▁",
+      "▁▁▁▁▁▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁▁▁▁▁▁ ▁▁▁",
+      "▁▁▁▁▁▁▁▁▁▁ ▁▁▁▁",
+      "▁▁▁▁▁▁▁▁▁▁▁ ▁",
+      "▁▁▁▁▁▁▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁▁▁▁▁▁▁ ▁▁▁",
+      "▁▁▁▁▁▁▁▁▁▁▁▁ ▁",
+      "▁▁▁▁▁▁▁▁▁▁▁▁ ▁▁",
+      "▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁"
     ]
   }
 }

tokenizer_config.json

@@ -1,5 +1,32 @@
 {
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
   "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|उपयोगकर्ता|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|प्रणाली|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|सहायक|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|सहायक|>' }}\n{% endif %}\n{% endfor %}\n",
   "clean_up_tokenization_spaces": true,
   "eos_token": "</s>",
   "model_max_length": 1000000000000000019884624838656,