Duplicate from google/flan-ul2

Co-authored-by: Younes B <ybelkada@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,247 @@
+---
+language:
+  - en
+  - fr
+  - ro
+  - de
+  - multilingual
+widget:
+  - text: 'Translate to German:  My name is Arthur'
+    example_title: Translation
+  - text: >-
+      Please answer to the following question. Who is going to be the next
+      Ballon d'or?
+    example_title: Question Answering
+  - text: >-
+      Q: Can Geoffrey Hinton have a conversation with George Washington? Give
+      the rationale before answering.
+    example_title: Logical reasoning
+  - text: >-
+      Please answer the following question. What is the boiling point of
+      Nitrogen?
+    example_title: Scientific knowledge
+  - text: >-
+      Answer the following yes/no question. Can you write a whole Haiku in a
+      single tweet?
+    example_title: Yes/no question
+  - text: >-
+      Answer the following yes/no question by reasoning step-by-step. Can you
+      write a whole Haiku in a single tweet?
+    example_title: Reasoning task
+  - text: 'Q: ( False or not False or False ) is? A: Let''s think step by step'
+    example_title: Boolean Expressions
+  - text: >-
+      The square root of x is the cube root of y. What is y to the power of 2,
+      if x = 4?
+    example_title: Math reasoning
+  - text: >-
+      Premise:  At my age you will probably have learnt one lesson. Hypothesis: 
+      It's not certain how many lessons you'll learn by your thirties. Does the
+      premise entail the hypothesis?
+    example_title: Premise and hypothesis
+  - text: >-
+      Answer the following question by reasoning step by step. 
+      The cafeteria had 23 apples. If they used 20 for lunch, and bought 6 more, how many apple do they have?
+    example_title: Chain of thought
+tags:
+  - text2text-generation
+  - flan-ul2
+datasets:
+  - svakulenk0/qrecc
+  - taskmaster2
+  - djaym7/wiki_dialog
+  - deepmind/code_contests
+  - lambada
+  - gsm8k
+  - aqua_rat
+  - esnli
+  - quasc
+  - qed
+  - c4
+license: apache-2.0
+---
+
+
+# Model card for Flan-UL2
+
+![model image](https://raw.githubusercontent.com/google-research/google-research/master/ul2/figs/ul2.png)
+
+#  Table of Contents
+
+0. [TL;DR](#TL;DR)
+1. [Using the model](#using-the-model)
+2. [Results](#results)
+3. [Introduction to UL2](#introduction-to-ul2)
+4. [Training](#training)
+5. [Contribution](#contribution)
+6. [Citation](#citation)
+
+# TL;DR
+
+Flan-UL2 is an encoder decoder model based on the `T5` architecture. It uses the same configuration as the [`UL2 model`](https://huggingface.co/google/ul2)  released earlier last year. It was fine tuned using the "Flan" prompt tuning 
+and dataset collection.
+
+According to the original [blog](https://www.yitay.net/blog/flan-ul2-20b) here are the notable improvements:
+- The original UL2 model was only trained with receptive field of 512, which made it non-ideal for N-shot prompting where N is large. 
+- The Flan-UL2 checkpoint uses a receptive field of 2048 which makes it more usable for few-shot in-context learning.
+- The original UL2 model also had mode switch tokens that was rather mandatory to get good performance. However, they were a little cumbersome as this requires often some changes during inference or finetuning. In this update/change, we continue training UL2 20B for an additional 100k steps (with small batch) to forget “mode tokens” before applying Flan instruction tuning. This Flan-UL2 checkpoint does not require mode tokens anymore.
+
+# Using the model 
+
+## Converting from T5x to huggingface
+
+You can use the [`convert_t5x_checkpoint_to_pytorch.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py) script and pass the argument `strict = False`. The final layer norm is missing from the original dictionnary, that is why we are passing the `strict = False` argument.
+```bash
+python convert_t5x_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --config_file PATH_TO_CONFIG --pytorch_dump_path PATH_TO_SAVE
+```
+We used the same config file as [`google/ul2`](https://huggingface.co/google/ul2/blob/main/config.json).
+
+## Running the model
+
+For more efficient memory usage, we advise you to load the model in `8bit` using `load_in_8bit` flag as follows (works only under GPU):
+
+```python
+# pip install accelerate transformers bitsandbytes
+from transformers import T5ForConditionalGeneration, AutoTokenizer
+import torch
+model = T5ForConditionalGeneration.from_pretrained("google/flan-ul2", device_map="auto", load_in_8bit=True)                                                                 
+tokenizer = AutoTokenizer.from_pretrained("google/flan-ul2")
+
+input_string = "Answer the following question by reasoning step by step. The cafeteria had 23 apples. If they used 20 for lunch, and bought 6 more, how many apple do they have?"                                               
+
+inputs = tokenizer(input_string, return_tensors="pt").input_ids.to("cuda")
+outputs = model.generate(inputs, max_length=200)
+
+print(tokenizer.decode(outputs[0]))
+# <pad> They have 23 - 20 = 3 apples left. They have 3 + 6 = 9 apples. Therefore, the answer is 9.</s>
+```
+
+Otherwise, you can load and run the model in `bfloat16` as follows:
+
+```python
+# pip install accelerate transformers
+from transformers import T5ForConditionalGeneration, AutoTokenizer
+import torch
+model = T5ForConditionalGeneration.from_pretrained("google/flan-ul2", torch_dtype=torch.bfloat16, device_map="auto")                                                                 
+tokenizer = AutoTokenizer.from_pretrained("google/flan-ul2")
+
+input_string = "Answer the following question by reasoning step by step. The cafeteria had 23 apples. If they used 20 for lunch, and bought 6 more, how many apple do they have?"                                               
+
+inputs = tokenizer(input_string, return_tensors="pt").input_ids.to("cuda")
+outputs = model.generate(inputs, max_length=200)
+
+print(tokenizer.decode(outputs[0]))
+# <pad> They have 23 - 20 = 3 apples left. They have 3 + 6 = 9 apples. Therefore, the answer is 9.</s>
+```
+
+# Results
+
+## Performance improvment 
+
+The reported results are the following : 
+|  | MMLU | BBH | MMLU-CoT | BBH-CoT | Avg |
+| :--- | :---: | :---: | :---: | :---: | :---: |
+| FLAN-PaLM 62B | 59.6 | 47.5 | 56.9 | 44.9 | 49.9 |
+| FLAN-PaLM 540B | 73.5 | 57.9 | 70.9 | 66.3 | 67.2 |
+| FLAN-T5-XXL 11B | 55.1 | 45.3 | 48.6 | 41.4 | 47.6 |
+| FLAN-UL2 20B | 55.7(+1.1%) | 45.9(+1.3%) | 52.2(+7.4%) | 42.7(+3.1%) | 49.1(+3.2%) |
+
+
+# Introduction to UL2
+
+This entire section has been copied from the [`google/ul2`](https://huggingface.co/google/ul2) model card and might be subject of change with respect to `flan-ul2`.
+
+UL2 is a unified framework for pretraining models that are universally effective across datasets and setups. UL2 uses Mixture-of-Denoisers (MoD), apre-training objective that combines diverse pre-training paradigms together. UL2 introduces a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes.
+
+![model image](https://raw.githubusercontent.com/google-research/google-research/master/ul2/figs/ul2.png)
+
+**Abstract**
+
+Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized and unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 and/or GPT-like models across multiple diverse setups. Finally, by scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised NLP tasks ranging from language generation (with automated and human evaluation), language understanding, text classification, question answering, commonsense reasoning, long text reasoning, structured knowledge grounding and information retrieval. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. 
+
+For more information, please take a look at the original paper.
+
+Paper: [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1)
+
+Authors: *Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, Donald Metzler* 
+
+
+## Training
+
+### Flan UL2
+The Flan-UL2 model was initialized using the `UL2` checkpoints, and was then trained additionally using Flan Prompting. This means that the original training corpus is `C4`, 
+
+In “Scaling Instruction-Finetuned language models (Chung et al.)” (also referred to sometimes as the Flan2 paper), the key idea is to train a large language model on a collection of datasets. These datasets are phrased as instructions which enable generalization across diverse tasks. Flan has been primarily trained on academic tasks. In Flan2, we released a series of T5 models ranging from 200M to 11B parameters that have been instruction tuned with Flan.
+
+The Flan datasets have also been open sourced in “The Flan Collection: Designing Data and Methods for Effective Instruction Tuning” (Longpre et al.). See Google AI Blogpost: “The Flan Collection: Advancing Open Source Methods for Instruction Tuning”. 
+
+## UL2 PreTraining
+
+The model is pretrained on the C4 corpus. For pretraining, the model is trained on a total of 1 trillion tokens on C4 (2 million steps)
+with a batch size of 1024. The sequence length is set to 512/512 for inputs and targets. 
+Dropout is set to 0 during pretraining. Pre-training took slightly more than one month for about 1 trillion
+tokens. The model has 32 encoder layers and 32 decoder layers, `dmodel` of 4096 and `df` of 16384. 
+The dimension of each head is 256 for a total of 16 heads. Our model uses a model parallelism of 8. 
+The same sentencepiece tokenizer as T5 of vocab size 32000 is used (click [here](https://huggingface.co/docs/transformers/v4.20.0/en/model_doc/t5#transformers.T5Tokenizer) for more information about the T5 tokenizer).
+
+UL-20B can be interpreted as a model that is quite similar to T5 but trained with a different objective and slightly different scaling knobs. 
+UL-20B was trained using the [Jax](https://github.com/google/jax) and [T5X](https://github.com/google-research/t5x) infrastructure.
+
+The training objective during pretraining is a mixture of different denoising strategies that are explained in the following:
+
+### Mixture of Denoisers
+
+To quote the paper:
+> We conjecture that a strong universal model has to be exposed to solving diverse set of problems
+> during pre-training. Given that pre-training is done using self-supervision, we argue that such diversity
+> should be injected to the objective of the model, otherwise the model might suffer from lack a certain
+> ability, like long-coherent text generation.
+> Motivated by this, as well as current class of objective functions, we define three main paradigms that
+> are used during pre-training:
+
+- **R-Denoiser**: The regular denoising is the standard span corruption introduced in [T5](https://huggingface.co/docs/transformers/v4.20.0/en/model_doc/t5)
+ that uses a range of 2 to 5 tokens as the span length, which masks about 15% of
+input tokens. These spans are short and potentially useful to acquire knowledge instead of
+learning to generate fluent text.
+
+- **S-Denoiser**: A specific case of denoising where we observe a strict sequential order when
+framing the inputs-to-targets task, i.e., prefix language modeling. To do so, we simply
+partition the input sequence into two sub-sequences of tokens as context and target such that
+the targets do not rely on future information. This is unlike standard span corruption where
+there could be a target token with earlier position than a context token. Note that similar to
+the Prefix-LM setup, the context (prefix) retains a bidirectional receptive field. We note that
+S-Denoising with very short memory or no memory is in similar spirit to standard causal
+language modeling.
+
+- **X-Denoiser**: An extreme version of denoising where the model must recover a large part
+of the input, given a small to moderate part of it. This simulates a situation where a model
+needs to generate long target from a memory with relatively limited information. To do
+so, we opt to include examples with aggressive denoising where approximately 50% of the
+input sequence is masked. This is by increasing the span length and/or corruption rate. We
+consider a pre-training task to be extreme if it has a long span (e.g., ≥ 12 tokens) or have
+a large corruption rate (e.g., ≥ 30%). X-denoising is motivated by being an interpolation
+between regular span corruption and language model like objectives.
+
+See the following diagram for a more visual explanation:
+
+![mixture-of-denoisers](https://raw.githubusercontent.com/google-research/google-research/master/ul2/figs/mod.png)
+
+**Important**: For more details, please see sections 3.1.2 of the [paper](https://arxiv.org/pdf/2205.05131v1.pdf).
+
+## Fine-tuning
+
+The model was continously fine-tuned after N pretraining steps where N is typically from 50k to 100k.
+In other words, after each Nk steps of pretraining, the model is finetuned on each downstream task. See section 5.2.2 of [paper](https://arxiv.org/pdf/2205.05131v1.pdf) to get an overview of all datasets that were used for fine-tuning).
+
+As the model is continuously finetuned, finetuning is stopped on a task once it has reached state-of-the-art to save compute.
+In total, the model was trained for 2.65 million steps.
+
+**Important**: For more details, please see sections 5.2.1 and 5.2.2 of the [paper](https://arxiv.org/pdf/2205.05131v1.pdf).
+
+# Contribution
+
+This model was originally contributed by [Yi Tay](https://www.yitay.net/?author=636616684c5e64780328eece), and added to the Hugging Face ecosystem by [Younes Belkada](https://huggingface.co/ybelkada) & [Arthur Zucker](https://huggingface.co/ArthurZ).
+
+# Citation
+
+If you want to cite this work, please consider citing the [blogpost](https://www.yitay.net/blog/flan-ul2-20b) announcing the release of `Flan-UL2`.

added_tokens.json

@@ -0,0 +1,30 @@
+{
+  "[new_id_0]": 32100,
+  "[new_id_10]": 32110,
+  "[new_id_11]": 32111,
+  "[new_id_12]": 32112,
+  "[new_id_13]": 32113,
+  "[new_id_14]": 32114,
+  "[new_id_15]": 32115,
+  "[new_id_16]": 32116,
+  "[new_id_17]": 32117,
+  "[new_id_18]": 32118,
+  "[new_id_19]": 32119,
+  "[new_id_1]": 32101,
+  "[new_id_20]": 32120,
+  "[new_id_21]": 32121,
+  "[new_id_22]": 32122,
+  "[new_id_23]": 32123,
+  "[new_id_24]": 32124,
+  "[new_id_25]": 32125,
+  "[new_id_26]": 32126,
+  "[new_id_27]": 32127,
+  "[new_id_2]": 32102,
+  "[new_id_3]": 32103,
+  "[new_id_4]": 32104,
+  "[new_id_5]": 32105,
+  "[new_id_6]": 32106,
+  "[new_id_7]": 32107,
+  "[new_id_8]": 32108,
+  "[new_id_9]": 32109
+}

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "/home/younes_huggingface_co/code/ul2/flan-ul2",
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "d_ff": 16384,
+  "d_kv": 256,
+  "d_model": 4096,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "silu",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-silu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "n_positions": 512,
+  "num_decoder_layers": 32,
+  "num_heads": 16,
+  "num_layers": 32,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.25.0.dev0",
+  "use_cache": true,
+  "vocab_size": 32128
+}

pytorch_model-00001-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45bf24c1738fbec52a5a1eefc3b482291a351bab069829a03e775f8fa2822722
+size 4692542996

pytorch_model-00002-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88bd28a99530e6bf4798e44a5897b64a926211fbe1d1ddb908562e55793c4ede
+size 4966229511

pytorch_model-00003-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f028a67b2a1aee2d6969a843b7be4448c89682bec5a10ed85ac03537b2a3e0e
+size 4966229499

pytorch_model-00004-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3bd08b4ba35a3eb643922118a7fc1e3a29c6163bb9539f7577e688dbe80dcc6b
+size 4961006511

pytorch_model-00005-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc6b8ffacb7cc29f099cc0b5aec4f7aa61cea81a54e1a7a8621347002f6760f8
+size 4999832641

pytorch_model-00006-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:adc1a3c4113f8168a98b19220ddc40b5fd1b0ac3fe41dfbd821d0c7129835ed8
+size 4932714647

pytorch_model-00007-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d8fb42e6386dbe3a5d14c171d4669793e27042bd6c13e1df57ee889485566f4
+size 4999823229

pytorch_model-00008-of-00008.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa22c462e65c38c2a86ead593f90687dc7e6945819d389720e6eb68097403083
+size 4927470284

pytorch_model.bin.index.json

@@ -0,0 +1,751 @@
+{
+  "metadata": {
+    "total_size": 39708805120
+  },
+  "weight_map": {
+    "decoder.block.0.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.1.EncDecAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.1.EncDecAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.1.EncDecAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.1.EncDecAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.2.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.0.layer.2.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.1.EncDecAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.1.EncDecAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.1.EncDecAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.1.EncDecAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.2.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.1.layer.2.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.10.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.10.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.10.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.10.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.10.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.10.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.10.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.10.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.10.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.10.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.10.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.10.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.10.layer.2.DenseReluDense.wo.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.10.layer.2.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.0.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.1.EncDecAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.2.DenseReluDense.wo.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.11.layer.2.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.0.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.1.EncDecAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.2.DenseReluDense.wo.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.12.layer.2.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.0.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.1.EncDecAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.2.DenseReluDense.wo.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.13.layer.2.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.0.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.1.EncDecAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.2.DenseReluDense.wo.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.14.layer.2.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.0.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.1.EncDecAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.2.DenseReluDense.wo.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.15.layer.2.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.0.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.1.EncDecAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.2.DenseReluDense.wo.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.16.layer.2.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.0.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.1.EncDecAttention.k.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.1.EncDecAttention.o.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.1.EncDecAttention.q.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.1.EncDecAttention.v.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.1.layer_norm.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00008.bin",
+    "decoder.block.17.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.17.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.17.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.0.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.1.EncDecAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.1.EncDecAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.1.EncDecAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.1.EncDecAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.1.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.18.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.0.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.1.EncDecAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.1.EncDecAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.1.EncDecAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.1.EncDecAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.1.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.19.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.2.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.1.EncDecAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.1.EncDecAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.1.EncDecAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.1.EncDecAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.block.2.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.2.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.20.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.0.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.1.EncDecAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.1.EncDecAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.1.EncDecAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.1.EncDecAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.1.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.20.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.0.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.1.EncDecAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.1.EncDecAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.1.EncDecAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.1.EncDecAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.1.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.21.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.0.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.1.EncDecAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.1.EncDecAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.1.EncDecAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.1.EncDecAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.1.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.22.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.0.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.1.EncDecAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.1.EncDecAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.1.EncDecAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.1.EncDecAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.1.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.23.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.0.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.1.EncDecAttention.k.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.1.EncDecAttention.o.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.1.EncDecAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.1.EncDecAttention.v.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.1.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.2.DenseReluDense.wo.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.24.layer.2.layer_norm.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.25.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00008.bin",
+    "decoder.block.25.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.0.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.1.EncDecAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.1.EncDecAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.1.EncDecAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.1.EncDecAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.1.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.2.DenseReluDense.wo.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.25.layer.2.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.0.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.1.EncDecAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.1.EncDecAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.1.EncDecAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.1.EncDecAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.1.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.2.DenseReluDense.wo.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.26.layer.2.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.0.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.1.EncDecAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.1.EncDecAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.1.EncDecAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.1.EncDecAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.1.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.2.DenseReluDense.wo.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.27.layer.2.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.0.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.1.EncDecAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.1.EncDecAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.1.EncDecAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.1.EncDecAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.1.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.2.DenseReluDense.wo.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.28.layer.2.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.0.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.1.EncDecAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.1.EncDecAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.1.EncDecAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.1.EncDecAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.1.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.2.DenseReluDense.wo.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.29.layer.2.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.3.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.1.EncDecAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.1.EncDecAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.1.EncDecAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.1.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.3.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.30.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.0.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.1.EncDecAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.1.EncDecAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.1.EncDecAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.1.EncDecAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.1.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.2.DenseReluDense.wo.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.30.layer.2.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.0.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.1.EncDecAttention.k.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.1.EncDecAttention.o.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.1.EncDecAttention.q.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.1.EncDecAttention.v.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.1.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.2.DenseReluDense.wo.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.31.layer.2.layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "decoder.block.4.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.1.EncDecAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.1.EncDecAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.1.EncDecAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.1.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.4.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.1.EncDecAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.1.EncDecAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.1.EncDecAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.1.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.5.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.1.EncDecAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.1.EncDecAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.1.EncDecAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.1.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.6.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.1.EncDecAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.1.EncDecAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.1.EncDecAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.1.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.7.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.1.EncDecAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.1.EncDecAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.1.EncDecAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.1.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.8.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.0.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.1.EncDecAttention.k.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.1.EncDecAttention.o.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.1.EncDecAttention.q.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.1.EncDecAttention.v.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.1.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.2.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.2.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.2.DenseReluDense.wo.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.block.9.layer.2.layer_norm.weight": "pytorch_model-00005-of-00008.bin",
+    "decoder.embed_tokens.weight": "pytorch_model-00004-of-00008.bin",
+    "decoder.final_layer_norm.weight": "pytorch_model-00008-of-00008.bin",
+    "encoder.block.0.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.0.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.1.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.10.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.10.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.11.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.12.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.13.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.14.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.15.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.16.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.17.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.17.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.17.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.17.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.17.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.17.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.17.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.17.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.17.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.18.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.19.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.2.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.2.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.20.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.20.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.21.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.22.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.23.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.24.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.25.layer.1.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.0.layer_norm.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00008.bin",
+    "encoder.block.26.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.26.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.27.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.28.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.29.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.3.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.3.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.30.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.30.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.0.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.31.layer.1.layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "encoder.block.4.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.4.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.5.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.6.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.7.layer.1.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.8.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.8.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.8.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.8.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.8.layer.0.layer_norm.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.block.8.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.8.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.8.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.8.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.0.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.block.9.layer.1.layer_norm.weight": "pytorch_model-00002-of-00008.bin",
+    "encoder.embed_tokens.weight": "pytorch_model-00001-of-00008.bin",
+    "encoder.final_layer_norm.weight": "pytorch_model-00004-of-00008.bin",
+    "lm_head.weight": "pytorch_model-00008-of-00008.bin",
+    "shared.weight": "pytorch_model-00001-of-00008.bin"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,107 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d60acb128cf7b7f2536e8f38a5b18a05535c9e14c7a355904270e15b0945ea86
+size 791656

tokenizer_config.json

@@ -0,0 +1,112 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "eos_token": "</s>",
+  "extra_ids": 100,
+  "model_max_length": 2048,
+  "name_or_path": "google/ul2",
+  "pad_token": "<pad>",
+  "special_tokens_map_file": null,
+  "tokenizer_class": "T5Tokenizer",
+  "unk_token": "<unk>"
+}