Upload Multilingual-e5-large-Q4 from MAVIS local embedding model store

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,25 @@
+MIT License
+
+Copyright (c) Microsoft Corporation and contributors
+Copyright (c) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei and contributors
+Copyright (c) 2026 MAVIS / R.E.V.I.S. contributors for q4 packaging metadata and conversion scripts, if any
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files, model files, weights,
+configuration files, tokenizer files, and related materials (the "Software"),
+to deal in the Software without restriction, including without limitation the
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice, this permission notice, and any attribution notices
+included with the Software shall be included in all copies or substantial
+portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS, COPYRIGHT HOLDERS, OR CONTRIBUTORS BE LIABLE FOR ANY CLAIM, DAMAGES
+OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

NOTICE

@@ -0,0 +1,41 @@
+NOTICE
+
+This repository redistributes a 4-bit quantized package derived from:
+
+    intfloat/multilingual-e5-large
+    https://huggingface.co/intfloat/multilingual-e5-large
+
+The original model is released under the MIT License.
+
+Original authors / associated paper:
+
+    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
+    Multilingual E5 Text Embeddings: A Technical Report
+
+Base architecture:
+
+    XLM-RoBERTa large
+
+Embedding dimension:
+
+    1024
+
+Modification notice:
+
+    This repository contains a R.E.V.I.S. Q4 quantized distribution.
+    Selected 2D weight tensors were converted to the MLX-native affine
+    packed integer representation described in quantization.json.
+
+    No fine-tuning, additional training, or architecture-level modification
+    has been applied.
+
+Redistributor:
+
+    MAVIS / R.E.V.I.S.
+
+Purpose:
+
+    Local semantic embedding, semantic recall, RAG retrieval, and multilingual
+    semantic search in the R.E.V.I.S. local Cognitive OS ecosystem.
+
+The upstream MIT License text is preserved in LICENSE.

README.md

@@ -0,0 +1,133 @@
+---
+library_name: mlx
+license: mit
+pipeline_tag: feature-extraction
+base_model: intfloat/multilingual-e5-large
+tags:
+- mlx
+- embeddings
+- sentence-transformers
+- xlm-roberta
+- multilingual
+- quantized
+- int4
+- q4
+- revis
+---
+
+# mavis-ai/Multilingual-e5-large-Q4
+
+This repository contains a 4-bit quantized MLX-compatible distribution of `intfloat/multilingual-e5-large`, prepared for use with **R.E.V.I.S.** as a compact local semantic embedding model.
+
+The model is intended for local text embedding, semantic recall, RAG retrieval, and multilingual semantic search workflows.
+
+## Important Notice
+
+This repository is hosted primarily as a dedicated download source for the R.E.V.I.S. application ecosystem. You are free to download and use this model package for your own local embedding or MLX workflows, subject to the MIT License and the attribution notices included in this repository.
+
+This package is **not** a new embedding model and has **not** been fine-tuned. It is a quantized redistribution of `intfloat/multilingual-e5-large`.
+
+For the original model card, training details, intended usage, and evaluation information, refer to the official upstream model:
+
+- Original model: <https://huggingface.co/intfloat/multilingual-e5-large>
+- Base architecture: XLM-RoBERTa large
+- Embedding size: 1024
+
+## Quantization
+
+This package stores selected 2D weight tensors using a R.E.V.I.S. MLX-native Q4 format:
+
+- Quantization type: `mlx-native-affine`
+- Bits: `4`
+- Group size: `64`
+- Mode: `affine`
+- Stored tensors: packed `.qweight` plus `.scales` and `.biases`
+- Expected linear path: `mx.quantized_matmul(..., qweight, scales=scales, biases=biases, group_size=64, bits=4, mode="affine")`
+- Expected embedding lookup path: gather the packed rows first, then `mx.dequantize(..., scales=scales, biases=biases, group_size=64, bits=4, mode="affine")`
+
+Typical tensor layout:
+
+```text
+encoder.layer.0.attention.self.query.weight.qweight
+encoder.layer.0.attention.self.query.weight.scales
+encoder.layer.0.attention.self.query.weight.biases
+```
+
+The `.qweight` tensors are MLX packed integer tensors, not plain row-major integer arrays. Non-quantized tensors, such as LayerNorm parameters, bias tensors, and other small metadata tensors, are preserved in their original floating-point representation.
+
+This format is optimized for minimum download and storage size. In R.E.V.I.S. it is intended for keyword-focused semantic retrieval where the LLM first distills source text into entities, categories, and aspects before embedding. Runtimes should read `quantization.json` for the exact tensor names and quantization parameters before loading the weights.
+
+## Optimized for R.E.V.I.S. (Local Cognitive OS)
+
+We host this model package to serve as the local semantic embedding engine for **R.E.V.I.S.**
+
+**R.E.V.I.S.** is a 100% local Cognitive OS for Multi-Agentic AI. It transforms your Mac devices into a distributed Agentic Swarm via zero-config Wi-Fi clustering, allowing you to run heavy AI workloads—like recursive web research, dynamic RAG generation, and multi-step logic—without killing single-machine performance.
+
+If you are interested in pushing the absolute limits of local AI and open-weight models, check out our project.
+
+- Official Website: <https://mavis-ai.co.jp/revis/>
+- Watch the 13-min Raw Demo (Multi-node Dynamic RAG): <https://x.gd/LxaBF>
+- Follow our updates on X: <https://x.com/mavis_ai_jp>
+
+## Usage Notes
+
+For retrieval-style tasks, E5 models typically use different text prefixes for queries and passages. R.E.V.I.S. applies its own canonical query and passage formatting internally.
+
+If you use this package outside R.E.V.I.S., refer to the upstream E5 instructions for recommended prompt prefixes and pooling behavior.
+
+## Files
+
+Recommended repository files:
+
+```text
+README.md
+LICENSE
+NOTICE
+weights.00.safetensors
+config.json
+tokenizer.json
+tokenizer_config.json
+special_tokens_map.json
+quantization.json
+```
+
+## License
+
+This repository redistributes a quantized package derived from `intfloat/multilingual-e5-large`, which is released under the **MIT License**.
+
+The upstream copyright notice and MIT License text are preserved in `LICENSE`.
+
+Additional attribution and redistribution notes are included in `NOTICE`.
+
+## Attribution
+
+Original model:
+
+```text
+intfloat/multilingual-e5-large
+https://huggingface.co/intfloat/multilingual-e5-large
+```
+
+Original authors / associated paper:
+
+```text
+Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
+Multilingual E5 Text Embeddings: A Technical Report
+```
+
+R.E.V.I.S. Q4 package:
+
+```text
+Prepared and redistributed by MAVIS / R.E.V.I.S.
+Quantization: MLX-native affine Q4 package for local MLX embedding runtime
+```
+
+## Modification Notice
+
+Compared with the upstream `intfloat/multilingual-e5-large` release, this repository applies the following packaging modification:
+
+```text
+Selected 2D weight tensors were quantized to the R.E.V.I.S. MLX-native affine Q4 representation described in `quantization.json`.
+```
+
+No fine-tuning, additional training, or architecture-level modification has been applied.

config.json

@@ -0,0 +1,90 @@
+{
+  "return_dict": true,
+  "output_hidden_states": false,
+  "output_attentions": false,
+  "torchscript": false,
+  "torch_dtype": "float32",
+  "use_bfloat16": false,
+  "tf_legacy_loss": false,
+  "pruned_heads": {},
+  "tie_word_embeddings": true,
+  "is_encoder_decoder": false,
+  "is_decoder": false,
+  "cross_attention_hidden_size": null,
+  "add_cross_attention": false,
+  "tie_encoder_decoder": false,
+  "max_length": 20,
+  "min_length": 0,
+  "do_sample": false,
+  "early_stopping": false,
+  "num_beams": 1,
+  "num_beam_groups": 1,
+  "diversity_penalty": 0.0,
+  "temperature": 1.0,
+  "top_k": 50,
+  "top_p": 1.0,
+  "typical_p": 1.0,
+  "repetition_penalty": 1.0,
+  "length_penalty": 1.0,
+  "no_repeat_ngram_size": 0,
+  "encoder_no_repeat_ngram_size": 0,
+  "bad_words_ids": null,
+  "num_return_sequences": 1,
+  "chunk_size_feed_forward": 0,
+  "output_scores": false,
+  "return_dict_in_generate": false,
+  "forced_bos_token_id": null,
+  "forced_eos_token_id": null,
+  "remove_invalid_values": false,
+  "exponential_decay_length_penalty": null,
+  "suppress_tokens": null,
+  "begin_suppress_tokens": null,
+  "architectures": [
+    "XLMRobertaModel"
+  ],
+  "finetuning_task": null,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
+  },
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "tokenizer_class": null,
+  "prefix": null,
+  "bos_token_id": 0,
+  "pad_token_id": 1,
+  "eos_token_id": 2,
+  "sep_token_id": null,
+  "decoder_start_token_id": null,
+  "task_specific_params": null,
+  "problem_type": null,
+  "_name_or_path": "/Users/katopz/.cache/huggingface/hub/models--intfloat--multilingual-e5-large/snapshots/9f78368af0062735ba99812349c562316e29f719",
+  "transformers_version": "4.36.2",
+  "model_type": "xlm-roberta",
+  "output_past": true,
+  "vocab_size": 250002,
+  "hidden_size": 1024,
+  "num_hidden_layers": 24,
+  "num_attention_heads": 16,
+  "hidden_act": "gelu",
+  "intermediate_size": 4096,
+  "hidden_dropout_prob": 0.1,
+  "attention_probs_dropout_prob": 0.1,
+  "max_position_embeddings": 514,
+  "type_vocab_size": 1,
+  "initializer_range": 0.02,
+  "layer_norm_eps": 1e-05,
+  "position_embedding_type": "absolute",
+  "use_cache": true,
+  "classifier_dropout": null,
+  "revis_quantization": {
+    "format": "revis-xlm-roberta-e5-mlx-native-q4",
+    "bits": 4,
+    "group_size": 64,
+    "mode": "affine",
+    "type": "mlx-native-affine",
+    "manifest": "quantization.json"
+  }
+}

quantization.json

@@ -0,0 +1,3359 @@
+{
+  "schemaVersion": 2,
+  "format": "revis-xlm-roberta-e5-mlx-native-q4",
+  "source": "intfloat/multilingual-e5-large",
+  "baseWeights": "weights.00.safetensors",
+  "quantizedWeights": "weights.00.safetensors",
+  "quantization": {
+    "type": "mlx-native-affine",
+    "bits": 4,
+    "group_size": 64,
+    "mode": "affine",
+    "quantizedTensorSuffix": ".qweight",
+    "scaleTensorSuffix": ".scales",
+    "biasTensorSuffix": ".biases",
+    "linear": "mx.quantized_matmul(x, qweight, scales=scales, biases=biases, transpose=True, group_size=group_size, bits=bits, mode=mode)",
+    "embedding": "mx.dequantize(qweight[ids], scales=scales[ids], biases=biases[ids], group_size=group_size, bits=bits, mode=mode)"
+  },
+  "quantized": [
+    {
+      "name": "embeddings.position_embeddings.weight",
+      "shape": [
+        514,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "embeddings.position_embeddings.weight.qweight",
+      "scales": "embeddings.position_embeddings.weight.scales",
+      "biases": "embeddings.position_embeddings.weight.biases"
+    },
+    {
+      "name": "embeddings.token_type_embeddings.weight",
+      "shape": [
+        1,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "embeddings.token_type_embeddings.weight.qweight",
+      "scales": "embeddings.token_type_embeddings.weight.scales",
+      "biases": "embeddings.token_type_embeddings.weight.biases"
+    },
+    {
+      "name": "embeddings.word_embeddings.weight",
+      "shape": [
+        250002,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "embeddings.word_embeddings.weight.qweight",
+      "scales": "embeddings.word_embeddings.weight.scales",
+      "biases": "embeddings.word_embeddings.weight.biases"
+    },
+    {
+      "name": "encoder.layer.0.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.0.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.0.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.0.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.0.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.0.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.0.attention.self.key.weight.scales",
+      "biases": "encoder.layer.0.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.0.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.0.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.0.attention.self.query.weight.scales",
+      "biases": "encoder.layer.0.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.0.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.0.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.0.attention.self.value.weight.scales",
+      "biases": "encoder.layer.0.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.0.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.0.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.0.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.0.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.0.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.0.output.dense.weight.qweight",
+      "scales": "encoder.layer.0.output.dense.weight.scales",
+      "biases": "encoder.layer.0.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.1.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.1.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.1.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.1.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.1.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.1.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.1.attention.self.key.weight.scales",
+      "biases": "encoder.layer.1.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.1.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.1.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.1.attention.self.query.weight.scales",
+      "biases": "encoder.layer.1.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.1.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.1.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.1.attention.self.value.weight.scales",
+      "biases": "encoder.layer.1.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.1.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.1.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.1.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.1.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.1.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.1.output.dense.weight.qweight",
+      "scales": "encoder.layer.1.output.dense.weight.scales",
+      "biases": "encoder.layer.1.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.10.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.10.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.10.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.10.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.10.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.10.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.10.attention.self.key.weight.scales",
+      "biases": "encoder.layer.10.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.10.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.10.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.10.attention.self.query.weight.scales",
+      "biases": "encoder.layer.10.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.10.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.10.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.10.attention.self.value.weight.scales",
+      "biases": "encoder.layer.10.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.10.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.10.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.10.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.10.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.10.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.10.output.dense.weight.qweight",
+      "scales": "encoder.layer.10.output.dense.weight.scales",
+      "biases": "encoder.layer.10.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.11.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.11.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.11.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.11.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.11.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.11.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.11.attention.self.key.weight.scales",
+      "biases": "encoder.layer.11.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.11.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.11.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.11.attention.self.query.weight.scales",
+      "biases": "encoder.layer.11.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.11.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.11.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.11.attention.self.value.weight.scales",
+      "biases": "encoder.layer.11.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.11.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.11.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.11.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.11.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.11.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.11.output.dense.weight.qweight",
+      "scales": "encoder.layer.11.output.dense.weight.scales",
+      "biases": "encoder.layer.11.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.12.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.12.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.12.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.12.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.12.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.12.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.12.attention.self.key.weight.scales",
+      "biases": "encoder.layer.12.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.12.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.12.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.12.attention.self.query.weight.scales",
+      "biases": "encoder.layer.12.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.12.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.12.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.12.attention.self.value.weight.scales",
+      "biases": "encoder.layer.12.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.12.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.12.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.12.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.12.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.12.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.12.output.dense.weight.qweight",
+      "scales": "encoder.layer.12.output.dense.weight.scales",
+      "biases": "encoder.layer.12.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.13.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.13.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.13.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.13.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.13.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.13.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.13.attention.self.key.weight.scales",
+      "biases": "encoder.layer.13.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.13.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.13.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.13.attention.self.query.weight.scales",
+      "biases": "encoder.layer.13.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.13.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.13.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.13.attention.self.value.weight.scales",
+      "biases": "encoder.layer.13.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.13.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.13.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.13.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.13.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.13.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.13.output.dense.weight.qweight",
+      "scales": "encoder.layer.13.output.dense.weight.scales",
+      "biases": "encoder.layer.13.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.14.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.14.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.14.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.14.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.14.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.14.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.14.attention.self.key.weight.scales",
+      "biases": "encoder.layer.14.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.14.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.14.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.14.attention.self.query.weight.scales",
+      "biases": "encoder.layer.14.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.14.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.14.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.14.attention.self.value.weight.scales",
+      "biases": "encoder.layer.14.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.14.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.14.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.14.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.14.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.14.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.14.output.dense.weight.qweight",
+      "scales": "encoder.layer.14.output.dense.weight.scales",
+      "biases": "encoder.layer.14.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.15.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.15.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.15.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.15.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.15.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.15.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.15.attention.self.key.weight.scales",
+      "biases": "encoder.layer.15.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.15.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.15.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.15.attention.self.query.weight.scales",
+      "biases": "encoder.layer.15.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.15.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.15.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.15.attention.self.value.weight.scales",
+      "biases": "encoder.layer.15.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.15.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.15.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.15.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.15.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.15.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.15.output.dense.weight.qweight",
+      "scales": "encoder.layer.15.output.dense.weight.scales",
+      "biases": "encoder.layer.15.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.16.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.16.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.16.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.16.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.16.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.16.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.16.attention.self.key.weight.scales",
+      "biases": "encoder.layer.16.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.16.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.16.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.16.attention.self.query.weight.scales",
+      "biases": "encoder.layer.16.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.16.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.16.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.16.attention.self.value.weight.scales",
+      "biases": "encoder.layer.16.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.16.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.16.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.16.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.16.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.16.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.16.output.dense.weight.qweight",
+      "scales": "encoder.layer.16.output.dense.weight.scales",
+      "biases": "encoder.layer.16.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.17.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.17.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.17.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.17.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.17.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.17.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.17.attention.self.key.weight.scales",
+      "biases": "encoder.layer.17.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.17.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.17.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.17.attention.self.query.weight.scales",
+      "biases": "encoder.layer.17.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.17.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.17.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.17.attention.self.value.weight.scales",
+      "biases": "encoder.layer.17.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.17.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.17.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.17.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.17.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.17.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.17.output.dense.weight.qweight",
+      "scales": "encoder.layer.17.output.dense.weight.scales",
+      "biases": "encoder.layer.17.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.18.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.18.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.18.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.18.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.18.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.18.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.18.attention.self.key.weight.scales",
+      "biases": "encoder.layer.18.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.18.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.18.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.18.attention.self.query.weight.scales",
+      "biases": "encoder.layer.18.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.18.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.18.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.18.attention.self.value.weight.scales",
+      "biases": "encoder.layer.18.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.18.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.18.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.18.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.18.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.18.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.18.output.dense.weight.qweight",
+      "scales": "encoder.layer.18.output.dense.weight.scales",
+      "biases": "encoder.layer.18.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.19.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.19.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.19.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.19.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.19.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.19.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.19.attention.self.key.weight.scales",
+      "biases": "encoder.layer.19.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.19.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.19.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.19.attention.self.query.weight.scales",
+      "biases": "encoder.layer.19.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.19.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.19.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.19.attention.self.value.weight.scales",
+      "biases": "encoder.layer.19.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.19.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.19.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.19.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.19.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.19.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.19.output.dense.weight.qweight",
+      "scales": "encoder.layer.19.output.dense.weight.scales",
+      "biases": "encoder.layer.19.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.2.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.2.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.2.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.2.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.2.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.2.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.2.attention.self.key.weight.scales",
+      "biases": "encoder.layer.2.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.2.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.2.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.2.attention.self.query.weight.scales",
+      "biases": "encoder.layer.2.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.2.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.2.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.2.attention.self.value.weight.scales",
+      "biases": "encoder.layer.2.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.2.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.2.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.2.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.2.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.2.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.2.output.dense.weight.qweight",
+      "scales": "encoder.layer.2.output.dense.weight.scales",
+      "biases": "encoder.layer.2.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.20.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.20.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.20.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.20.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.20.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.20.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.20.attention.self.key.weight.scales",
+      "biases": "encoder.layer.20.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.20.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.20.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.20.attention.self.query.weight.scales",
+      "biases": "encoder.layer.20.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.20.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.20.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.20.attention.self.value.weight.scales",
+      "biases": "encoder.layer.20.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.20.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.20.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.20.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.20.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.20.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.20.output.dense.weight.qweight",
+      "scales": "encoder.layer.20.output.dense.weight.scales",
+      "biases": "encoder.layer.20.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.21.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.21.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.21.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.21.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.21.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.21.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.21.attention.self.key.weight.scales",
+      "biases": "encoder.layer.21.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.21.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.21.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.21.attention.self.query.weight.scales",
+      "biases": "encoder.layer.21.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.21.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.21.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.21.attention.self.value.weight.scales",
+      "biases": "encoder.layer.21.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.21.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.21.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.21.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.21.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.21.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.21.output.dense.weight.qweight",
+      "scales": "encoder.layer.21.output.dense.weight.scales",
+      "biases": "encoder.layer.21.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.22.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.22.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.22.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.22.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.22.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.22.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.22.attention.self.key.weight.scales",
+      "biases": "encoder.layer.22.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.22.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.22.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.22.attention.self.query.weight.scales",
+      "biases": "encoder.layer.22.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.22.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.22.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.22.attention.self.value.weight.scales",
+      "biases": "encoder.layer.22.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.22.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.22.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.22.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.22.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.22.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.22.output.dense.weight.qweight",
+      "scales": "encoder.layer.22.output.dense.weight.scales",
+      "biases": "encoder.layer.22.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.23.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.23.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.23.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.23.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.23.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.23.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.23.attention.self.key.weight.scales",
+      "biases": "encoder.layer.23.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.23.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.23.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.23.attention.self.query.weight.scales",
+      "biases": "encoder.layer.23.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.23.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.23.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.23.attention.self.value.weight.scales",
+      "biases": "encoder.layer.23.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.23.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.23.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.23.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.23.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.23.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.23.output.dense.weight.qweight",
+      "scales": "encoder.layer.23.output.dense.weight.scales",
+      "biases": "encoder.layer.23.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.3.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.3.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.3.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.3.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.3.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.3.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.3.attention.self.key.weight.scales",
+      "biases": "encoder.layer.3.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.3.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.3.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.3.attention.self.query.weight.scales",
+      "biases": "encoder.layer.3.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.3.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.3.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.3.attention.self.value.weight.scales",
+      "biases": "encoder.layer.3.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.3.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.3.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.3.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.3.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.3.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.3.output.dense.weight.qweight",
+      "scales": "encoder.layer.3.output.dense.weight.scales",
+      "biases": "encoder.layer.3.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.4.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.4.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.4.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.4.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.4.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.4.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.4.attention.self.key.weight.scales",
+      "biases": "encoder.layer.4.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.4.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.4.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.4.attention.self.query.weight.scales",
+      "biases": "encoder.layer.4.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.4.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.4.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.4.attention.self.value.weight.scales",
+      "biases": "encoder.layer.4.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.4.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.4.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.4.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.4.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.4.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.4.output.dense.weight.qweight",
+      "scales": "encoder.layer.4.output.dense.weight.scales",
+      "biases": "encoder.layer.4.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.5.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.5.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.5.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.5.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.5.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.5.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.5.attention.self.key.weight.scales",
+      "biases": "encoder.layer.5.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.5.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.5.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.5.attention.self.query.weight.scales",
+      "biases": "encoder.layer.5.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.5.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.5.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.5.attention.self.value.weight.scales",
+      "biases": "encoder.layer.5.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.5.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.5.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.5.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.5.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.5.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.5.output.dense.weight.qweight",
+      "scales": "encoder.layer.5.output.dense.weight.scales",
+      "biases": "encoder.layer.5.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.6.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.6.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.6.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.6.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.6.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.6.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.6.attention.self.key.weight.scales",
+      "biases": "encoder.layer.6.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.6.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.6.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.6.attention.self.query.weight.scales",
+      "biases": "encoder.layer.6.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.6.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.6.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.6.attention.self.value.weight.scales",
+      "biases": "encoder.layer.6.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.6.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.6.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.6.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.6.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.6.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.6.output.dense.weight.qweight",
+      "scales": "encoder.layer.6.output.dense.weight.scales",
+      "biases": "encoder.layer.6.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.7.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.7.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.7.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.7.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.7.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.7.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.7.attention.self.key.weight.scales",
+      "biases": "encoder.layer.7.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.7.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.7.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.7.attention.self.query.weight.scales",
+      "biases": "encoder.layer.7.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.7.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.7.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.7.attention.self.value.weight.scales",
+      "biases": "encoder.layer.7.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.7.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.7.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.7.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.7.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.7.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.7.output.dense.weight.qweight",
+      "scales": "encoder.layer.7.output.dense.weight.scales",
+      "biases": "encoder.layer.7.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.8.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.8.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.8.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.8.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.8.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.8.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.8.attention.self.key.weight.scales",
+      "biases": "encoder.layer.8.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.8.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.8.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.8.attention.self.query.weight.scales",
+      "biases": "encoder.layer.8.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.8.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.8.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.8.attention.self.value.weight.scales",
+      "biases": "encoder.layer.8.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.8.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.8.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.8.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.8.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.8.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.8.output.dense.weight.qweight",
+      "scales": "encoder.layer.8.output.dense.weight.scales",
+      "biases": "encoder.layer.8.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.9.attention.output.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.9.attention.output.dense.weight.qweight",
+      "scales": "encoder.layer.9.attention.output.dense.weight.scales",
+      "biases": "encoder.layer.9.attention.output.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.9.attention.self.key.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.9.attention.self.key.weight.qweight",
+      "scales": "encoder.layer.9.attention.self.key.weight.scales",
+      "biases": "encoder.layer.9.attention.self.key.weight.biases"
+    },
+    {
+      "name": "encoder.layer.9.attention.self.query.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.9.attention.self.query.weight.qweight",
+      "scales": "encoder.layer.9.attention.self.query.weight.scales",
+      "biases": "encoder.layer.9.attention.self.query.weight.biases"
+    },
+    {
+      "name": "encoder.layer.9.attention.self.value.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.9.attention.self.value.weight.qweight",
+      "scales": "encoder.layer.9.attention.self.value.weight.scales",
+      "biases": "encoder.layer.9.attention.self.value.weight.biases"
+    },
+    {
+      "name": "encoder.layer.9.intermediate.dense.weight",
+      "shape": [
+        4096,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.9.intermediate.dense.weight.qweight",
+      "scales": "encoder.layer.9.intermediate.dense.weight.scales",
+      "biases": "encoder.layer.9.intermediate.dense.weight.biases"
+    },
+    {
+      "name": "encoder.layer.9.output.dense.weight",
+      "shape": [
+        1024,
+        4096
+      ],
+      "dtype": "float16",
+      "qweight": "encoder.layer.9.output.dense.weight.qweight",
+      "scales": "encoder.layer.9.output.dense.weight.scales",
+      "biases": "encoder.layer.9.output.dense.weight.biases"
+    },
+    {
+      "name": "pooler.dense.weight",
+      "shape": [
+        1024,
+        1024
+      ],
+      "dtype": "float16",
+      "qweight": "pooler.dense.weight.qweight",
+      "scales": "pooler.dense.weight.scales",
+      "biases": "pooler.dense.weight.biases"
+    }
+  ],
+  "kept": [
+    {
+      "name": "embeddings.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "embeddings.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "embeddings.position_ids",
+      "shape": [
+        1,
+        514
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.0.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.1.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.10.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.11.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.12.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.13.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.14.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.15.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.16.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.17.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.18.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.19.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.2.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.20.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.21.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.22.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.23.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.3.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.4.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.5.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.6.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.7.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.8.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.attention.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.attention.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.attention.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.attention.self.key.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.attention.self.query.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.attention.self.value.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.intermediate.dense.bias",
+      "shape": [
+        4096
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.output.LayerNorm.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.output.LayerNorm.weight",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "encoder.layer.9.output.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    },
+    {
+      "name": "pooler.dense.bias",
+      "shape": [
+        1024
+      ],
+      "dtype": "float16"
+    }
+  ]
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f59925fcb90c92b894cb93e51bb9b4a6105c5c249fe54ce1c704420ac39b81af
+size 17082756

tokenizer_config.json

@@ -0,0 +1,54 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}

weights.00.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e94f485ad5e963a123b68e8d799ac491dcd6b3015869aca7a08b09aaecb8ce0d
+size 315484304