Super-squash branch 'main' using huggingface_hub

Co-authored-by: SFconvertbot <SFconvertbot@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,198 @@
+---
+license: apache-2.0
+base_model: google/flat-ul2
+pipeline_tag: feature-extraction
+tags:
+- embedding
+- text embedding
+---
+
+# flan-ul2-text-encoder
+
+The encoder model extracted from [flan-ul2](https://huggingface.co/google/flan-ul2) via a new class add [in a recent release](https://github.com/huggingface/transformers/releases/tag/v4.31.0).
+
+⚠️ This model is 17.44 GB in `bfloat16` precision ⚠️
+
+
+## basic usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForTextEncoding
+
+tokenizer = AutoTokenizer.from_pretrained("pszemraj/flan-ul2-text-encoder")
+model = AutoModelForTextEncoding.from_pretrained("pszemraj/flan-ul2-text-encoder")
+
+inputs = tokenizer("Hello, my dog loves memes", return_tensors="pt")
+outputs = model(**inputs)
+
+last_hidden_states = outputs.last_hidden_state
+```
+
+## usage: semantic similarity 
+
+> note: this is 'one way' to use the encoder, not 'the only way'. suggestions and ideas welcome.
+
+Below is an example and a set of functions to compute the cosine similarity between the embeddings of different texts with this model
+
+## Functions
+
+### load_model_and_tokenizer
+
+Loads the model and tokenizer based on `model_name`, returning a tuple containing the loaded model and tokenizer.
+
+
+<details>
+<summary><b>Details</b></summary>
+
+
+```python
+from typing import List, Tuple
+
+import torch
+from transformers import AutoModel, AutoTokenizer
+from transformers import AutoModelForTextEncoding
+
+
+def load_model_and_tokenizer(model_name: str) -> Tuple[AutoModel, AutoTokenizer]:
+    """
+    Load the model and tokenizer based on the given model name.
+
+    Args:
+        model_name (str): The name of the model to be loaded.
+
+    Returns:
+        Tuple[AutoModelForTextEncoding, AutoTokenizer]: The loaded model and tokenizer.
+    """
+    model = AutoModelForTextEncoding.from_pretrained(
+        model_name, torch_dtype=torch.bfloat16, device_map="auto"
+    ).eval()
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
+    return model, tokenizer
+```
+
+</details>
+
+### get_embeddings
+
+This computes the embeddings for the given texts given the model and tokenizer via weighted mean pooling across seq_len (as in [SGPT](https://github.com/Muennighoff/sgpt#symmetric-semantic-search-be))
+
+<details>
+<summary><b>Details</b></summary>
+
+
+```python
+def get_embeddings(
+    model: AutoModel, tokenizer: AutoTokenizer, texts: List[str]
+) -> torch.Tensor:
+    """
+    compute text embeddings via weighted mean pooling across seq_len
+
+    Args:
+        model (AutoModel): The model to be used for getting embeddings.
+        tokenizer (AutoTokenizer): The tokenizer to be used for tokenizing the texts.
+        texts (List[str]): The texts for which embeddings are to be calculated.
+
+    Returns:
+        torch.Tensor: The calculated embeddings.
+    """
+    # Tokenize input texts
+    batch_tokens = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
+
+    # Get the embeddings
+    with torch.no_grad():
+        last_hidden_state = model(
+            **batch_tokens, output_hidden_states=True, return_dict=True
+        ).last_hidden_state
+
+    # Get weights
+    weights = (
+        torch.arange(start=1, end=last_hidden_state.shape[1] + 1)
+        .unsqueeze(0)
+        .unsqueeze(-1)
+        .expand(last_hidden_state.size())
+        .float()
+        .to(last_hidden_state.device)
+    )
+
+    # Get attn mask
+    input_mask_expanded = (
+        batch_tokens["attention_mask"]
+        .unsqueeze(-1)
+        .expand(last_hidden_state.size())
+        .float()
+    )
+
+    # Perform weighted mean pooling across seq_len: bs, seq_len, hidden_dim -> bs, hidden_dim
+    sum_embeddings = torch.sum(last_hidden_state * input_mask_expanded * weights, dim=1)
+    sum_mask = torch.sum(input_mask_expanded * weights, dim=1)
+
+    embeddings = sum_embeddings / sum_mask
+
+    return embeddings
+```
+
+</details>
+
+### calculate_cosine_similarity
+
+Helper fn to compute and print out cosine similarity 
+
+<details>
+<summary><b>click to expand</b></summary>
+
+
+```python
+from scipy.spatial.distance import cosine
+
+def calculate_cosine_similarity(embeddings: torch.Tensor, texts: List[str]) -> None:
+    """compute and print the cosine sim between the first text and all others"""
+    # Calculate cosine similarities
+    for i in range(1, len(embeddings)):
+        cosine_sim = 1 - cosine(embeddings[0], embeddings[i])
+        print(
+            'Cosine similarity between "%s" and "%s" is: %.3f'
+            % (texts[0], texts[i], cosine_sim)
+        )
+```
+
+</details>
+
+## Usage
+
+Install packages:
+
+```bash
+pip install transformers accelerate sentencepiece scipy
+```
+
+Then, you can use the functions to compute embeddings and similarity scores:
+
+```python
+model_name = "pszemraj/flan-ul2-text-encoder"
+model, tokenizer = load_model_and_tokenizer(model_name)
+
+texts = [
+    "deep learning",
+    "artificial intelligence",
+    "deep diving",
+    "artificial snow",
+]
+
+embeddings = get_embeddings(model, tokenizer, texts)
+calculate_cosine_similarity(embeddings, texts)
+```
+
+This will print the cosine similarity between the first text and all other texts in the `texts' list.
+
+## References
+
+Inference with this model/the example is based on the ideas and examples in the [SGPT repository](https://github.com/Muennighoff/sgpt#symmetric-semantic-search-be).
+
+```
+@article{muennighoff2022sgpt,
+  title={SGPT: GPT Sentence Embeddings for Semantic Search},
+  author={Muennighoff, Niklas},
+  journal={arXiv preprint arXiv:2202.08904},
+  year={2022}
+}
+```

added_tokens.json

@@ -0,0 +1,30 @@
+{
+  "[new_id_0]": 32100,
+  "[new_id_10]": 32110,
+  "[new_id_11]": 32111,
+  "[new_id_12]": 32112,
+  "[new_id_13]": 32113,
+  "[new_id_14]": 32114,
+  "[new_id_15]": 32115,
+  "[new_id_16]": 32116,
+  "[new_id_17]": 32117,
+  "[new_id_18]": 32118,
+  "[new_id_19]": 32119,
+  "[new_id_1]": 32101,
+  "[new_id_20]": 32120,
+  "[new_id_21]": 32121,
+  "[new_id_22]": 32122,
+  "[new_id_23]": 32123,
+  "[new_id_24]": 32124,
+  "[new_id_25]": 32125,
+  "[new_id_26]": 32126,
+  "[new_id_27]": 32127,
+  "[new_id_2]": 32102,
+  "[new_id_3]": 32103,
+  "[new_id_4]": 32104,
+  "[new_id_5]": 32105,
+  "[new_id_6]": 32106,
+  "[new_id_7]": 32107,
+  "[new_id_8]": 32108,
+  "[new_id_9]": 32109
+}

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "google/flan-ul2",
+  "architectures": [
+    "T5EncoderModel"
+  ],
+  "d_ff": 16384,
+  "d_kv": 256,
+  "d_model": 4096,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "silu",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-silu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "n_positions": 512,
+  "num_decoder_layers": 32,
+  "num_heads": 16,
+  "num_layers": 32,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.31.0",
+  "use_cache": true,
+  "vocab_size": 32128
+}

model-00001-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2209fe278c9739e972ef421f22083072a059d555f16a49a4e65fe48d02312b5
+size 1974522792

model-00002-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f7b16c7fdda0127752095596df8281aebe28c61b11ab0f40b9b5d47fb04dbea
+size 1912663824

model-00003-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ea70638dfd0c8bf6384c0ac71303ad7255419a897a4741567ed7b46db88e626
+size 1879117936

model-00004-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2162ccdaca85e949d38f58128bbd0154db30c70110dc590058615119a3b2bd4b
+size 1879100992

model-00005-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c59a90e9d5638ee441f92bb52c5625b48e08ab376ec1c39e751338ede109b76
+size 1879117976

model-00006-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a55836e7056080b72616c1189bdb22a9e8db15a7a54b4c82ded9b739c7e561a7
+size 1879100992

model-00007-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:322ba58ab5293e0d26631174c0018bde034c093657479977ea3bc18bd280692d
+size 1879117976

model-00008-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:585c0eeac0b8b45b12c5bd1805f493fe5caaf1da4628468e0a2be33499d4d9af
+size 1879100992

model-00009-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e9d00116ac05fbbd8e287e25415f5256e816c789e350b59c2a5b6ba0532b9fa
+size 1879117976

model-00010-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49bb3c7372775edb158534a4ae3b75f05851cb571aa85af698bd5fad546b2cd9
+size 402670208

model.safetensors.index.json

@@ -0,0 +1,299 @@
+{
+    "metadata": {
+        "total_size": 17443595264
+    },
+    "weight_map": {
+        "encoder.block.0.layer.0.SelfAttention.k.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.0.SelfAttention.o.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.0.SelfAttention.q.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.0.SelfAttention.v.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.0.layer_norm.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.1.DenseReluDense.wo.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.0.layer.1.layer_norm.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.0.SelfAttention.k.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.0.SelfAttention.o.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.0.SelfAttention.q.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.0.SelfAttention.v.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.0.layer_norm.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.1.DenseReluDense.wo.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.1.layer.1.layer_norm.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.10.layer.0.SelfAttention.k.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.10.layer.0.SelfAttention.o.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.10.layer.0.SelfAttention.q.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.10.layer.0.SelfAttention.v.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.10.layer.0.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.10.layer.1.DenseReluDense.wi_0.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.10.layer.1.DenseReluDense.wi_1.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.10.layer.1.DenseReluDense.wo.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.10.layer.1.layer_norm.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.0.SelfAttention.k.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.0.SelfAttention.o.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.0.SelfAttention.q.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.0.SelfAttention.v.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.0.layer_norm.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.1.DenseReluDense.wi_0.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.1.DenseReluDense.wi_1.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.1.DenseReluDense.wo.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.11.layer.1.layer_norm.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.0.SelfAttention.k.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.0.SelfAttention.o.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.0.SelfAttention.q.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.0.SelfAttention.v.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.0.layer_norm.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.1.DenseReluDense.wi_0.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.1.DenseReluDense.wi_1.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.1.DenseReluDense.wo.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.12.layer.1.layer_norm.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.0.SelfAttention.k.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.0.SelfAttention.o.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.0.SelfAttention.q.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.0.SelfAttention.v.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.0.layer_norm.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.1.DenseReluDense.wi_0.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.1.DenseReluDense.wi_1.weight": "model-00004-of-00010.safetensors",
+        "encoder.block.13.layer.1.DenseReluDense.wo.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.13.layer.1.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.0.SelfAttention.k.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.0.SelfAttention.o.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.0.SelfAttention.q.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.0.SelfAttention.v.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.0.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.1.DenseReluDense.wi_0.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.1.DenseReluDense.wi_1.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.1.DenseReluDense.wo.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.14.layer.1.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.0.SelfAttention.k.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.0.SelfAttention.o.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.0.SelfAttention.q.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.0.SelfAttention.v.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.0.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.1.DenseReluDense.wi_0.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.1.DenseReluDense.wi_1.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.1.DenseReluDense.wo.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.15.layer.1.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.0.SelfAttention.k.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.0.SelfAttention.o.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.0.SelfAttention.q.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.0.SelfAttention.v.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.0.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.1.DenseReluDense.wi_0.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.1.DenseReluDense.wi_1.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.1.DenseReluDense.wo.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.16.layer.1.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.17.layer.0.SelfAttention.k.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.17.layer.0.SelfAttention.o.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.17.layer.0.SelfAttention.q.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.17.layer.0.SelfAttention.v.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.17.layer.0.layer_norm.weight": "model-00005-of-00010.safetensors",
+        "encoder.block.17.layer.1.DenseReluDense.wi_0.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.17.layer.1.DenseReluDense.wi_1.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.17.layer.1.DenseReluDense.wo.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.17.layer.1.layer_norm.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.0.SelfAttention.k.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.0.SelfAttention.o.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.0.SelfAttention.q.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.0.SelfAttention.v.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.0.layer_norm.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.1.DenseReluDense.wi_0.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.1.DenseReluDense.wi_1.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.1.DenseReluDense.wo.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.18.layer.1.layer_norm.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.0.SelfAttention.k.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.0.SelfAttention.o.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.0.SelfAttention.q.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.0.SelfAttention.v.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.0.layer_norm.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.1.DenseReluDense.wi_0.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.1.DenseReluDense.wi_1.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.1.DenseReluDense.wo.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.19.layer.1.layer_norm.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.2.layer.0.SelfAttention.k.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.0.SelfAttention.o.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.0.SelfAttention.q.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.0.SelfAttention.v.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.0.layer_norm.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.1.DenseReluDense.wo.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.2.layer.1.layer_norm.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.20.layer.0.SelfAttention.k.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.20.layer.0.SelfAttention.o.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.20.layer.0.SelfAttention.q.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.20.layer.0.SelfAttention.v.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.20.layer.0.layer_norm.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.20.layer.1.DenseReluDense.wi_0.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.20.layer.1.DenseReluDense.wi_1.weight": "model-00006-of-00010.safetensors",
+        "encoder.block.20.layer.1.DenseReluDense.wo.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.20.layer.1.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.0.SelfAttention.k.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.0.SelfAttention.o.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.0.SelfAttention.q.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.0.SelfAttention.v.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.0.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.1.DenseReluDense.wi_0.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.1.DenseReluDense.wi_1.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.1.DenseReluDense.wo.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.21.layer.1.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.0.SelfAttention.k.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.0.SelfAttention.o.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.0.SelfAttention.q.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.0.SelfAttention.v.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.0.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.1.DenseReluDense.wi_0.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.1.DenseReluDense.wi_1.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.1.DenseReluDense.wo.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.22.layer.1.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.0.SelfAttention.k.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.0.SelfAttention.o.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.0.SelfAttention.q.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.0.SelfAttention.v.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.0.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.1.DenseReluDense.wi_0.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.1.DenseReluDense.wi_1.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.1.DenseReluDense.wo.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.23.layer.1.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.24.layer.0.SelfAttention.k.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.24.layer.0.SelfAttention.o.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.24.layer.0.SelfAttention.q.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.24.layer.0.SelfAttention.v.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.24.layer.0.layer_norm.weight": "model-00007-of-00010.safetensors",
+        "encoder.block.24.layer.1.DenseReluDense.wi_0.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.24.layer.1.DenseReluDense.wi_1.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.24.layer.1.DenseReluDense.wo.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.24.layer.1.layer_norm.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.0.SelfAttention.k.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.0.SelfAttention.o.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.0.SelfAttention.q.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.0.SelfAttention.v.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.0.layer_norm.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.1.DenseReluDense.wi_0.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.1.DenseReluDense.wi_1.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.1.DenseReluDense.wo.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.25.layer.1.layer_norm.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.0.SelfAttention.k.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.0.SelfAttention.o.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.0.SelfAttention.q.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.0.SelfAttention.v.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.0.layer_norm.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.1.DenseReluDense.wi_0.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.1.DenseReluDense.wi_1.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.1.DenseReluDense.wo.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.26.layer.1.layer_norm.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.0.SelfAttention.k.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.0.SelfAttention.o.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.0.SelfAttention.q.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.0.SelfAttention.v.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.0.layer_norm.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.1.DenseReluDense.wi_0.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.1.DenseReluDense.wi_1.weight": "model-00008-of-00010.safetensors",
+        "encoder.block.27.layer.1.DenseReluDense.wo.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.27.layer.1.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.0.SelfAttention.k.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.0.SelfAttention.o.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.0.SelfAttention.q.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.0.SelfAttention.v.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.0.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.1.DenseReluDense.wi_0.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.1.DenseReluDense.wi_1.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.1.DenseReluDense.wo.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.28.layer.1.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.0.SelfAttention.k.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.0.SelfAttention.o.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.0.SelfAttention.q.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.0.SelfAttention.v.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.0.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.1.DenseReluDense.wi_0.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.1.DenseReluDense.wi_1.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.1.DenseReluDense.wo.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.29.layer.1.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.3.layer.0.SelfAttention.k.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.3.layer.0.SelfAttention.o.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.3.layer.0.SelfAttention.q.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.3.layer.0.SelfAttention.v.weight": "model-00001-of-00010.safetensors",
+        "encoder.block.3.layer.0.layer_norm.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.3.layer.1.DenseReluDense.wi_0.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.3.layer.1.DenseReluDense.wi_1.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.3.layer.1.DenseReluDense.wo.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.3.layer.1.layer_norm.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.30.layer.0.SelfAttention.k.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.0.SelfAttention.o.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.0.SelfAttention.q.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.0.SelfAttention.v.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.0.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.1.DenseReluDense.wi_0.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.1.DenseReluDense.wi_1.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.1.DenseReluDense.wo.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.30.layer.1.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.31.layer.0.SelfAttention.k.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.31.layer.0.SelfAttention.o.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.31.layer.0.SelfAttention.q.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.31.layer.0.SelfAttention.v.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.31.layer.0.layer_norm.weight": "model-00009-of-00010.safetensors",
+        "encoder.block.31.layer.1.DenseReluDense.wi_0.weight": "model-00010-of-00010.safetensors",
+        "encoder.block.31.layer.1.DenseReluDense.wi_1.weight": "model-00010-of-00010.safetensors",
+        "encoder.block.31.layer.1.DenseReluDense.wo.weight": "model-00010-of-00010.safetensors",
+        "encoder.block.31.layer.1.layer_norm.weight": "model-00010-of-00010.safetensors",
+        "encoder.block.4.layer.0.SelfAttention.k.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.0.SelfAttention.o.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.0.SelfAttention.q.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.0.SelfAttention.v.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.0.layer_norm.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.1.DenseReluDense.wi_0.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.1.DenseReluDense.wi_1.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.1.DenseReluDense.wo.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.4.layer.1.layer_norm.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.0.SelfAttention.k.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.0.SelfAttention.o.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.0.SelfAttention.q.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.0.SelfAttention.v.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.0.layer_norm.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.1.DenseReluDense.wi_0.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.1.DenseReluDense.wi_1.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.1.DenseReluDense.wo.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.5.layer.1.layer_norm.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.0.SelfAttention.k.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.0.SelfAttention.o.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.0.SelfAttention.q.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.0.SelfAttention.v.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.0.layer_norm.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.1.DenseReluDense.wi_0.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.1.DenseReluDense.wi_1.weight": "model-00002-of-00010.safetensors",
+        "encoder.block.6.layer.1.DenseReluDense.wo.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.6.layer.1.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.0.SelfAttention.k.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.0.SelfAttention.o.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.0.SelfAttention.q.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.0.SelfAttention.v.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.0.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.1.DenseReluDense.wi_0.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.1.DenseReluDense.wi_1.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.1.DenseReluDense.wo.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.7.layer.1.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.0.SelfAttention.k.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.0.SelfAttention.o.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.0.SelfAttention.q.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.0.SelfAttention.v.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.0.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.1.DenseReluDense.wi_0.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.1.DenseReluDense.wi_1.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.1.DenseReluDense.wo.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.8.layer.1.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.0.SelfAttention.k.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.0.SelfAttention.o.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.0.SelfAttention.q.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.0.SelfAttention.v.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.0.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.1.DenseReluDense.wi_0.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.1.DenseReluDense.wi_1.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.1.DenseReluDense.wo.weight": "model-00003-of-00010.safetensors",
+        "encoder.block.9.layer.1.layer_norm.weight": "model-00003-of-00010.safetensors",
+        "encoder.embed_tokens.weight": "model-00001-of-00010.safetensors",
+        "encoder.final_layer_norm.weight": "model-00010-of-00010.safetensors",
+        "shared.weight": "model-00001-of-00010.safetensors"
+    }
+}

pytorch_model-00001-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f172bae9a224d6088892087976355ca1bd8e8c83adf02d5e52ab6407dacca3bc
+size 1974529807

pytorch_model-00002-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b6857d854de04d0127317ee6a286b86d73de9445d49e3c6857fbaed5a5f8ddc
+size 1912670525

pytorch_model-00003-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe27d259c3b74ade192fc0fc667a42caed7b39aa247a9bd5d0ec68a9d748c1ac
+size 1879125299

pytorch_model-00004-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:263ea6b85878589bc2d04662d1d25d9a9de4fc624fd334eb7371e9e39cf9e85c
+size 1879107289

pytorch_model-00005-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d8ec32802a84fc0d513f0e51159835fd04d95f8a7da21540181220f2d917648
+size 1879125363

pytorch_model-00006-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10c2aaf66e967f3a76b14aaeafdafc01e41ebf045b9da33fa7363c2b98d9f648
+size 1879107289

pytorch_model-00007-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b9f4ee4639aab4a71dff21571372367ccd76afcae740b4a1ec13fefcb1b457e
+size 1879125363

pytorch_model-00008-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79093ab34054f166e35ea4b413fe3442cf3398980a6725844dd2f16dc0d72846
+size 1879107289

pytorch_model-00009-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:240d24a2b26622b72c1b147fd398745945e0e0aae52bc91f472279b3bc4786e9
+size 1879125363

pytorch_model-00010-of-00010.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e0785821e6b0a4e8cf731ddd3bb218ec1e300490a9051510b00d1bc56badc32
+size 402671790

pytorch_model.bin.index.json

@@ -0,0 +1,299 @@
+{
+  "metadata": {
+    "total_size": 17443595264
+  },
+  "weight_map": {
+    "encoder.block.0.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.0.layer_norm.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.0.layer.1.layer_norm.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.0.layer_norm.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.1.layer.1.layer_norm.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.10.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.10.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.10.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.10.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.10.layer.0.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.10.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.10.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.10.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.10.layer.1.layer_norm.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.0.layer_norm.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.11.layer.1.layer_norm.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.0.layer_norm.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.1.DenseReluDense.wo.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.12.layer.1.layer_norm.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.0.SelfAttention.k.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.0.SelfAttention.o.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.0.SelfAttention.q.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.0.SelfAttention.v.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.0.layer_norm.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00004-of-00010.bin",
+    "encoder.block.13.layer.1.DenseReluDense.wo.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.13.layer.1.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.0.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.1.DenseReluDense.wo.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.14.layer.1.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.0.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.1.DenseReluDense.wo.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.15.layer.1.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.0.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.1.DenseReluDense.wo.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.16.layer.1.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.17.layer.0.SelfAttention.k.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.17.layer.0.SelfAttention.o.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.17.layer.0.SelfAttention.q.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.17.layer.0.SelfAttention.v.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.17.layer.0.layer_norm.weight": "pytorch_model-00005-of-00010.bin",
+    "encoder.block.17.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.17.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.17.layer.1.DenseReluDense.wo.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.17.layer.1.layer_norm.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.0.layer_norm.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.1.DenseReluDense.wo.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.18.layer.1.layer_norm.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.0.layer_norm.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.1.DenseReluDense.wo.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.19.layer.1.layer_norm.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.2.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.0.SelfAttention.o.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.0.layer_norm.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.1.DenseReluDense.wo.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.2.layer.1.layer_norm.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.20.layer.0.SelfAttention.k.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.20.layer.0.SelfAttention.o.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.20.layer.0.SelfAttention.q.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.20.layer.0.SelfAttention.v.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.20.layer.0.layer_norm.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.20.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.20.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00006-of-00010.bin",
+    "encoder.block.20.layer.1.DenseReluDense.wo.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.20.layer.1.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.0.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.1.DenseReluDense.wo.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.21.layer.1.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.0.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.1.DenseReluDense.wo.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.22.layer.1.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.0.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.1.DenseReluDense.wo.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.23.layer.1.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.24.layer.0.SelfAttention.k.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.24.layer.0.SelfAttention.o.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.24.layer.0.SelfAttention.q.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.24.layer.0.SelfAttention.v.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.24.layer.0.layer_norm.weight": "pytorch_model-00007-of-00010.bin",
+    "encoder.block.24.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.24.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.24.layer.1.DenseReluDense.wo.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.24.layer.1.layer_norm.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.0.layer_norm.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.1.DenseReluDense.wo.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.25.layer.1.layer_norm.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.0.layer_norm.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.1.DenseReluDense.wo.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.26.layer.1.layer_norm.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.0.SelfAttention.k.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.0.SelfAttention.o.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.0.SelfAttention.q.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.0.SelfAttention.v.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.0.layer_norm.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00008-of-00010.bin",
+    "encoder.block.27.layer.1.DenseReluDense.wo.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.27.layer.1.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.0.SelfAttention.k.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.0.SelfAttention.o.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.0.SelfAttention.q.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.0.SelfAttention.v.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.0.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.1.DenseReluDense.wo.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.28.layer.1.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.0.SelfAttention.k.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.0.SelfAttention.o.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.0.SelfAttention.q.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.0.SelfAttention.v.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.0.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.1.DenseReluDense.wo.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.29.layer.1.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.3.layer.0.SelfAttention.k.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.3.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.3.layer.0.SelfAttention.q.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.3.layer.0.SelfAttention.v.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.block.3.layer.0.layer_norm.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.3.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.3.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.3.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.3.layer.1.layer_norm.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.30.layer.0.SelfAttention.k.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.0.SelfAttention.o.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.0.SelfAttention.q.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.0.SelfAttention.v.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.0.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.1.DenseReluDense.wo.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.30.layer.1.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.31.layer.0.SelfAttention.k.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.31.layer.0.SelfAttention.o.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.31.layer.0.SelfAttention.q.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.31.layer.0.SelfAttention.v.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.31.layer.0.layer_norm.weight": "pytorch_model-00009-of-00010.bin",
+    "encoder.block.31.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00010-of-00010.bin",
+    "encoder.block.31.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00010-of-00010.bin",
+    "encoder.block.31.layer.1.DenseReluDense.wo.weight": "pytorch_model-00010-of-00010.bin",
+    "encoder.block.31.layer.1.layer_norm.weight": "pytorch_model-00010-of-00010.bin",
+    "encoder.block.4.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.0.layer_norm.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.4.layer.1.layer_norm.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.0.layer_norm.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.1.DenseReluDense.wo.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.5.layer.1.layer_norm.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.0.SelfAttention.k.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.0.SelfAttention.o.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.0.SelfAttention.q.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.0.SelfAttention.v.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.0.layer_norm.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00002-of-00010.bin",
+    "encoder.block.6.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.6.layer.1.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.0.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.7.layer.1.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.0.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.8.layer.1.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.0.SelfAttention.k.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.0.SelfAttention.o.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.0.SelfAttention.q.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.0.SelfAttention.v.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.0.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.1.DenseReluDense.wi_0.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.1.DenseReluDense.wi_1.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.1.DenseReluDense.wo.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.block.9.layer.1.layer_norm.weight": "pytorch_model-00003-of-00010.bin",
+    "encoder.embed_tokens.weight": "pytorch_model-00001-of-00010.bin",
+    "encoder.final_layer_norm.weight": "pytorch_model-00010-of-00010.bin",
+    "shared.weight": "pytorch_model-00001-of-00010.bin"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,107 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d60acb128cf7b7f2536e8f38a5b18a05535c9e14c7a355904270e15b0945ea86
+size 791656

tokenizer_config.json

@@ -0,0 +1,111 @@
+{
+  "additional_special_tokens": [
+    "<extra_id_0>",
+    "<extra_id_1>",
+    "<extra_id_2>",
+    "<extra_id_3>",
+    "<extra_id_4>",
+    "<extra_id_5>",
+    "<extra_id_6>",
+    "<extra_id_7>",
+    "<extra_id_8>",
+    "<extra_id_9>",
+    "<extra_id_10>",
+    "<extra_id_11>",
+    "<extra_id_12>",
+    "<extra_id_13>",
+    "<extra_id_14>",
+    "<extra_id_15>",
+    "<extra_id_16>",
+    "<extra_id_17>",
+    "<extra_id_18>",
+    "<extra_id_19>",
+    "<extra_id_20>",
+    "<extra_id_21>",
+    "<extra_id_22>",
+    "<extra_id_23>",
+    "<extra_id_24>",
+    "<extra_id_25>",
+    "<extra_id_26>",
+    "<extra_id_27>",
+    "<extra_id_28>",
+    "<extra_id_29>",
+    "<extra_id_30>",
+    "<extra_id_31>",
+    "<extra_id_32>",
+    "<extra_id_33>",
+    "<extra_id_34>",
+    "<extra_id_35>",
+    "<extra_id_36>",
+    "<extra_id_37>",
+    "<extra_id_38>",
+    "<extra_id_39>",
+    "<extra_id_40>",
+    "<extra_id_41>",
+    "<extra_id_42>",
+    "<extra_id_43>",
+    "<extra_id_44>",
+    "<extra_id_45>",
+    "<extra_id_46>",
+    "<extra_id_47>",
+    "<extra_id_48>",
+    "<extra_id_49>",
+    "<extra_id_50>",
+    "<extra_id_51>",
+    "<extra_id_52>",
+    "<extra_id_53>",
+    "<extra_id_54>",
+    "<extra_id_55>",
+    "<extra_id_56>",
+    "<extra_id_57>",
+    "<extra_id_58>",
+    "<extra_id_59>",
+    "<extra_id_60>",
+    "<extra_id_61>",
+    "<extra_id_62>",
+    "<extra_id_63>",
+    "<extra_id_64>",
+    "<extra_id_65>",
+    "<extra_id_66>",
+    "<extra_id_67>",
+    "<extra_id_68>",
+    "<extra_id_69>",
+    "<extra_id_70>",
+    "<extra_id_71>",
+    "<extra_id_72>",
+    "<extra_id_73>",
+    "<extra_id_74>",
+    "<extra_id_75>",
+    "<extra_id_76>",
+    "<extra_id_77>",
+    "<extra_id_78>",
+    "<extra_id_79>",
+    "<extra_id_80>",
+    "<extra_id_81>",
+    "<extra_id_82>",
+    "<extra_id_83>",
+    "<extra_id_84>",
+    "<extra_id_85>",
+    "<extra_id_86>",
+    "<extra_id_87>",
+    "<extra_id_88>",
+    "<extra_id_89>",
+    "<extra_id_90>",
+    "<extra_id_91>",
+    "<extra_id_92>",
+    "<extra_id_93>",
+    "<extra_id_94>",
+    "<extra_id_95>",
+    "<extra_id_96>",
+    "<extra_id_97>",
+    "<extra_id_98>",
+    "<extra_id_99>"
+  ],
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "</s>",
+  "extra_ids": 100,
+  "model_max_length": 2048,
+  "pad_token": "<pad>",
+  "tokenizer_class": "T5Tokenizer",
+  "unk_token": "<unk>"
+}