Hugging Face's logo

remots
/
nllb-mulgi

Transformers
Safetensors
m2m_100
text2text-generation
Generated from Trainer
Model card Files
xet
 Community
remots commited on
Commit
9d92ff9
·
verified ·
1 Parent(s): 1ffe047

Add mul_Latn special token (init from est_Latn)

Browse files
Files changed (3) hide show
  1. README.md +199 -0
  2. tokenizer.json +2 -2
  3. tokenizer_config.json +2 -203
README.md ADDED
@@ -0,0 +1,199 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags: []
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+ This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
tokenizer.json CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b3be18cc91c94d4a1d83731ace4dac0b90a7db024edecdeb9fe7d19ec01ce901
3
- size 32240136
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:47146dd5dc2b00ea397101f982449ba5bdc122883320b33fc8f7152386c2367a
3
+ size 32240321
tokenizer_config.json CHANGED
@@ -4,208 +4,7 @@
4
  "cls_token": "<s>",
5
  "eos_token": "</s>",
6
  "extra_special_tokens": [
7
- "ace_Arab",
8
- "ace_Latn",
9
- "acm_Arab",
10
- "acq_Arab",
11
- "aeb_Arab",
12
- "afr_Latn",
13
- "ajp_Arab",
14
- "aka_Latn",
15
- "amh_Ethi",
16
- "apc_Arab",
17
- "arb_Arab",
18
- "ars_Arab",
19
- "ary_Arab",
20
- "arz_Arab",
21
- "asm_Beng",
22
- "ast_Latn",
23
- "awa_Deva",
24
- "ayr_Latn",
25
- "azb_Arab",
26
- "azj_Latn",
27
- "bak_Cyrl",
28
- "bam_Latn",
29
- "ban_Latn",
30
- "bel_Cyrl",
31
- "bem_Latn",
32
- "ben_Beng",
33
- "bho_Deva",
34
- "bjn_Arab",
35
- "bjn_Latn",
36
- "bod_Tibt",
37
- "bos_Latn",
38
- "bug_Latn",
39
- "bul_Cyrl",
40
- "cat_Latn",
41
- "ceb_Latn",
42
- "ces_Latn",
43
- "cjk_Latn",
44
- "ckb_Arab",
45
- "crh_Latn",
46
- "cym_Latn",
47
- "dan_Latn",
48
- "deu_Latn",
49
- "dik_Latn",
50
- "dyu_Latn",
51
- "dzo_Tibt",
52
- "ell_Grek",
53
- "eng_Latn",
54
- "epo_Latn",
55
- "est_Latn",
56
- "eus_Latn",
57
- "ewe_Latn",
58
- "fao_Latn",
59
- "pes_Arab",
60
- "fij_Latn",
61
- "fin_Latn",
62
- "fon_Latn",
63
- "fra_Latn",
64
- "fur_Latn",
65
- "fuv_Latn",
66
- "gla_Latn",
67
- "gle_Latn",
68
- "glg_Latn",
69
- "grn_Latn",
70
- "guj_Gujr",
71
- "hat_Latn",
72
- "hau_Latn",
73
- "heb_Hebr",
74
- "hin_Deva",
75
- "hne_Deva",
76
- "hrv_Latn",
77
- "hun_Latn",
78
- "hye_Armn",
79
- "ibo_Latn",
80
- "ilo_Latn",
81
- "ind_Latn",
82
- "isl_Latn",
83
- "ita_Latn",
84
- "jav_Latn",
85
- "jpn_Jpan",
86
- "kab_Latn",
87
- "kac_Latn",
88
- "kam_Latn",
89
- "kan_Knda",
90
- "kas_Arab",
91
- "kas_Deva",
92
- "kat_Geor",
93
- "knc_Arab",
94
- "knc_Latn",
95
- "kaz_Cyrl",
96
- "kbp_Latn",
97
- "kea_Latn",
98
- "khm_Khmr",
99
- "kik_Latn",
100
- "kin_Latn",
101
- "kir_Cyrl",
102
- "kmb_Latn",
103
- "kon_Latn",
104
- "kor_Hang",
105
- "kmr_Latn",
106
- "lao_Laoo",
107
- "lvs_Latn",
108
- "lij_Latn",
109
- "lim_Latn",
110
- "lin_Latn",
111
- "lit_Latn",
112
- "lmo_Latn",
113
- "ltg_Latn",
114
- "ltz_Latn",
115
- "lua_Latn",
116
- "lug_Latn",
117
- "luo_Latn",
118
- "lus_Latn",
119
- "mag_Deva",
120
- "mai_Deva",
121
- "mal_Mlym",
122
- "mar_Deva",
123
- "min_Latn",
124
- "mkd_Cyrl",
125
- "plt_Latn",
126
- "mlt_Latn",
127
- "mni_Beng",
128
- "khk_Cyrl",
129
- "mos_Latn",
130
- "mri_Latn",
131
- "zsm_Latn",
132
- "mya_Mymr",
133
- "nld_Latn",
134
- "nno_Latn",
135
- "nob_Latn",
136
- "npi_Deva",
137
- "nso_Latn",
138
- "nus_Latn",
139
- "nya_Latn",
140
- "oci_Latn",
141
- "gaz_Latn",
142
- "ory_Orya",
143
- "pag_Latn",
144
- "pan_Guru",
145
- "pap_Latn",
146
- "pol_Latn",
147
- "por_Latn",
148
- "prs_Arab",
149
- "pbt_Arab",
150
- "quy_Latn",
151
- "ron_Latn",
152
- "run_Latn",
153
- "rus_Cyrl",
154
- "sag_Latn",
155
- "san_Deva",
156
- "sat_Beng",
157
- "scn_Latn",
158
- "shn_Mymr",
159
- "sin_Sinh",
160
- "slk_Latn",
161
- "slv_Latn",
162
- "smo_Latn",
163
- "sna_Latn",
164
- "snd_Arab",
165
- "som_Latn",
166
- "sot_Latn",
167
- "spa_Latn",
168
- "als_Latn",
169
- "srd_Latn",
170
- "srp_Cyrl",
171
- "ssw_Latn",
172
- "sun_Latn",
173
- "swe_Latn",
174
- "swh_Latn",
175
- "szl_Latn",
176
- "tam_Taml",
177
- "tat_Cyrl",
178
- "tel_Telu",
179
- "tgk_Cyrl",
180
- "tgl_Latn",
181
- "tha_Thai",
182
- "tir_Ethi",
183
- "taq_Latn",
184
- "taq_Tfng",
185
- "tpi_Latn",
186
- "tsn_Latn",
187
- "tso_Latn",
188
- "tuk_Latn",
189
- "tum_Latn",
190
- "tur_Latn",
191
- "twi_Latn",
192
- "tzm_Tfng",
193
- "uig_Arab",
194
- "ukr_Cyrl",
195
- "umb_Latn",
196
- "urd_Arab",
197
- "uzn_Latn",
198
- "vec_Latn",
199
- "vie_Latn",
200
- "war_Latn",
201
- "wol_Latn",
202
- "xho_Latn",
203
- "ydd_Hebr",
204
- "yor_Latn",
205
- "yue_Hant",
206
- "zho_Hans",
207
- "zho_Hant",
208
- "zul_Latn"
209
  ],
210
  "is_local": false,
211
  "legacy_behaviour": false,
@@ -217,6 +16,6 @@
217
  "sp_model_kwargs": {},
218
  "src_lang": "eng_Latn",
219
  "tgt_lang": null,
220
- "tokenizer_class": "NllbTokenizer",
221
  "unk_token": "<unk>"
222
  }
 
4
  "cls_token": "<s>",
5
  "eos_token": "</s>",
6
  "extra_special_tokens": [
7
+ "mul_Latn"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  ],
9
  "is_local": false,
10
  "legacy_behaviour": false,
 
16
  "sp_model_kwargs": {},
17
  "src_lang": "eng_Latn",
18
  "tgt_lang": null,
19
+ "tokenizer_class": "TokenizersBackend",
20
  "unk_token": "<unk>"
21
  }