Uploaded PLLuM model.

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,232 @@
+---
+license: cc-by-nd-4.0
+---
+<p align="center">
+  <img src="https://pllum.org.pl/_nuxt/PLLuM_logo_RGB_color.DXNEc-VR.png">
+</p>
+
+# PLLuM: A Family of Polish Large Language Models
+
+## Overview
+PLLuM is a family of large language models (LLMs) specialized in Polish and other Slavic/Baltic languages, with additional English data incorporated for broader generalization. Developed through an extensive collaboration with various data providers, PLLuM models are built on high-quality text corpora and refined through instruction tuning, preference learning, and advanced alignment techniques. These models are intended to generate contextually coherent text, offer assistance in various tasks (e.g., question answering, summarization), and serve as a foundation for specialized applications such as domain-specific intelligent assistants.
+
+### Key Highlights
+- **Extensive Data Collection**  
+  We gathered large-scale, high-quality text data in Polish (around 150B tokens after cleaning and deduplication) and additional text in Slavic, Baltic, and English languages. Part of these tokens (28B) can be used in fully open-source models, including for commercial use (in compliance with relevant legal regulations).
+
+- **Organic Instruction Dataset**  
+  We curated the largest Polish collection of manually created “organic instructions” (~40k prompt-response pairs, including ~3.5k multi-turn dialogs). This human-authored instruction set is based on an extensive typology of human-model interactions and it covers a range of subtle aspects of supervised fine-tuning (SFT) that might be overlooked with automated approaches (including large scale distillation of 'strong LLMs'). It was also designed to mitigate negative linguistic transfer from non-Polish textual data used in the pre-training phase.
+
+- **Polish Preference Corpus**  
+  We created the first Polish-language preference corpus, featuring prompts and multiple model responses manually assessed by a demographically diverse team of annotators. This dataset teaches the model not only correctness (factual and linguistic) but also balance and safety—especially for potentially controversial or adversarial topics.
+
+- **Evaluation Benchmarks**  
+  We developed custom benchmarks to evaluate our models on tasks relevant to Polish public administration, where PLLuM achieved top scores among all tested models. In broader Polish-language tasks, PLLuM models also attain state-of-the-art results.
+
+## Model Description
+
+Below is a summary of the main PLLuM models, including their licenses, bases, and parameter sizes. All model names link to a specific Hugging Face resources, while the base models and licenses link to their respective sources or license references. Note that all *-nc-* models are intended to non-commercial use.
+
+| Model Name                                            | Params | License                                                                                                                   | Based On                                                                                                             |
+|-------------------------------------------------------|----------------------|---------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
+| [Llama-PLLuM-8B-base](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-8B-base)             | 8B                   | [Llama 3.1](https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE)                                   | [Llama3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B)                                                       |
+| [Llama-PLLuM-8B-instruct](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-8B-instruct)         | 8B                   | [Llama 3.1](https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE)                                   | [Llama3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B)                                                       |
+| [Llama-PLLuM-8B-chat](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-8B-chat)             | 8B                   | [Llama 3.1](https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE)                                   | [Llama3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B)                                                       |
+| [PLLuM-12B-base](https://huggingface.co/CYFRAGOVPL/PLLuM-12B-base)                  | 12B                  | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)                                                            | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407)                                    |
+| [PLLuM-12B-instruct](https://huggingface.co/CYFRAGOVPL/PLLuM-12B-instruct)              | 12B                  | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)                                                            | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407)                                    |
+| [PLLuM-12B-chat](https://huggingface.co/CYFRAGOVPL/PLLuM-12B-chat)                  | 12B                  | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)                                                            | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407)                                    |
+| [PLLuM-12B-nc-base](https://huggingface.co/CYFRAGOVPL/PLLuM-12B-nc-base)               | 12B                  | [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt)                                             | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407)                                    |
+| [PLLuM-12B-nc-instruct](https://huggingface.co/CYFRAGOVPL/PLLuM-12B-nc-instruct)           | 12B                  | [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt)                                             | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407)                                    |
+| [PLLuM-12B-nc-chat](https://huggingface.co/CYFRAGOVPL/PLLuM-12B-nc-chat)               | 12B                  | [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt)                                             | [Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407)                                    |
+| [PLLuM-8x7B-base](https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-base)                 | 8×7B                 | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)                                                            | [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)                                              |
+| [PLLuM-8x7B-instruct](https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-instruct)             | 8×7B                 | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)                                                            | [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)                            |
+| [PLLuM-8x7B-chat](https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-chat)                 | 8×7B                 | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)                                                            | [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)                            |
+| [PLLuM-8x7B-nc-base](https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-nc-base)              | 8×7B                 | [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt)                                             | [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)                            |
+| [PLLuM-8x7B-nc-instruct](https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-nc-instruct)          | 8×7B                 | [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt)                                             | [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)                            |
+| [PLLuM-8x7B-nc-chat](https://huggingface.co/CYFRAGOVPL/PLLuM-8x7B-nc-chat)              | 8×7B                 | [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/legalcode.txt)                                             | [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)                            |
+| [Llama-PLLuM-70B-base](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-base)            | 70B                  | [Llama 3.1](https://huggingface.co/meta-llama/Llama-3.1-70B/blob/main/LICENSE)                                  | [Llama-3.1-70B](https://huggingface.co/meta-llama/Llama-3.1-70B)                                                     |
+| [Llama-PLLuM-70B-instruct](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-instruct)        | 70B                  | [Llama 3.1](https://huggingface.co/meta-llama/Llama-3.1-70B/blob/main/LICENSE)                                  | [Llama-3.1-70B](https://huggingface.co/meta-llama/Llama-3.1-70B)                                                     |
+| [Llama-PLLuM-70B-chat](https://huggingface.co/CYFRAGOVPL/Llama-PLLuM-70B-chat)            | 70B                  | [Llama 3.1](https://huggingface.co/meta-llama/Llama-3.1-70B/blob/main/LICENSE)                                  | [Llama-3.1-70B](https://huggingface.co/meta-llama/Llama-3.1-70B)                                                     |
+
+### Model Development
+- **Pretraining**: All models were pretrained or continued-pretrained on large-scale Polish corpora (up to 150B tokens) plus a range of additional Slavic/Baltic and English texts.
+- **Instruction Fine-Tuning**: We refined the models on manually curated Polish “organic instructions,” converted instructions from premium Polish corpora, and synthetic instructions generated by strong LLMs.
+- **Alignment and Preference Learning**: Manually annotated preference data taught the models to produce safer, balanced, and contextually appropriate responses, even in adversarial or sensitive cases.
+- **Domain-Specific Adaptations**: Specialized RAG-based (Retrieval Augmented Generation) models were developed for tasks like public administration, demonstrating strong performance in complex information retrieval and question answering.
+
+## Intended Use Cases
+- **General Language Tasks**: Text generation, summarization, question answering, etc.
+- **Domain-Specific Assistants**: Especially effective for Polish public administration and legal or bureaucratic topics where domain-aware retrieval is required.
+- **Research & Development**: Building blocks for downstream AI applications in academic or industrial settings, where a strong command of the Polish language is essential.
+
+## How to Use
+Each PLLuM model can be loaded via the Hugging Face Transformers library (or compatible frameworks). For RAG-based scenarios, pair the model with a relevant vector store or document retrieval system. 
+
+Below are some recommended steps and code snippets:
+
+### 1. Installation
+Make sure you have the latest versions of `transformers` and `torch` (or another compatible deep learning framework) installed:
+```bash
+pip install transformers accelerate torch
+```
+
+### 2. Loading the Model
+Use the following example to load one of the PLLuM models:
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_name = "CYFRAGOVPL/PLLuM-12B-chat"  # Replace with the PLLuM model name of your choice
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+```
+
+### 3. Using bfloat16 (BF16)
+If your hardware (e.g., newer GPUs) supports bfloat16, you can reduce memory usage and potentially speed up inference:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_name = "CYFRAGOVPL/PLLuM-12B-chat"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# Load model in bfloat16 precision
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"  # automatically places model layers on available devices
+)
+```
+
+### 4. Generating an Example Text
+```python
+
+prompt = "Napisz krótki wiersz o wiośnie." # EN:"Write a short poem about spring."
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=50,
+    do_sample=True,
+    top_k=50,
+    top_p=0.9,
+    temperature=0.7
+)
+
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(generated_text)
+```
+
+### 5. Expected Output
+Below is a sample (hypothetical) output for the prompt above:
+
+```css
+Przykładowy wiersz o tematyce wiosennej:
+
+Wiosna, wiosna, wiosna, ach to ty!
+Kwiecień plecień wciąż przeplata,
+trochę zimy, trochę lata.
+A ja nie mogę się już doczekać,
+kiedy w kalendarzu ujrzę maj.
+Wtedy wszystko wkoło rozkwita,
+a ptaki tak pięknie śpiewają.
+Wiosno, wiosno, czekam z utęsknieniem,
+zrób mi tę przyjemność i przyjdź wreszcie, proszę!
+```
+Your results may vary depending on model parameters (e.g., temperature, top_k, top_p), hardware, and other settings.
+
+
+
+
+## Training Procedure
+- **Datasets**: ~150B tokens from Polish and multilingual sources, with ~28B tokens available for fully open-source commercial use. 
+- **Hyperparameters**: Vary based on model size, typically including Adam or AdamW optimizers, a range of batch sizes, and carefully tuned learning rates.
+- **Hardware & Duration**: Training using [Bem2](https://man.e-science.pl/pl/kdm/bem2) HPC (up to 300xH100 GPUs). Each model’s training time depends on parameter size and hardware configuration (~8 to ~25 days on multi-GPU cluster for 8B–70B sizes).
+
+## Evaluation and Benchmarks
+- **Public Administration**: PLLuM models demonstrated top-tier performance in specialized tasks relevant to government services.
+- **Polish Language Tasks**: Across a variety of internal benchmarks and standard corpora, PLLuM consistently outperforms other models in accuracy, coherence, and safety metrics.
+- **Custom Tests**: A unique preference corpus and alignment tests ensure robust, safe, and contextually accurate responses.
+
+## Limitations and Bias
+- **Potential Hallucinations**: Like other LLMs, PLLuM may occasionally produce factually incorrect or fabricated content.
+- **Sensitivity & Bias**: While extensive preference learning has been done, biases might still emerge, especially in controversial or subjective topics.
+- **Context Length**: Very long context tasks may challenge certain models, depending on memory constraints.
+
+## Ethical Considerations
+PLLuM models are designed for constructive and responsible usage. Users should exercise caution when deploying them in production scenarios, especially for sensitive or regulated domains. Despite efforts to minimize harmful outputs, there is always a risk of generating offensive, biased, or inappropriate text. Human oversight and due diligence are advised.
+
+## Citation
+If you use PLLuM models or any part of this repository in your research or deployment, please cite as follows (BibTeX):
+```
+@unpublished{pllum2025, 
+    title={PLLuM: A Family of Polish Large Language Models}, 
+    author={PLLuM Consortium}, 
+    year={2025} 
+}
+```
+
+## License
+Different models within the PLLuM family are published under various licenses (Apache 2.0, CC-BY-NC-4.0, or Llama 3.1 license). Check each model’s entry in the table above for details.
+
+## Creators & Consortium
+
+The PLLuM project is a unique collaboration between leading Polish scientific institutions and experts from various fields, working together to create a groundbreaking Polish language model. This research partnership combines diverse competencies and passions, forming a robust foundation for advancing AI in Poland.
+
+<table style="border: none; border-collapse: collapse;">
+  <tr>
+    <td align="center" valign="middle" style="border: none;">
+      <a href="https://pwr.edu.pl/">
+        <img src="https://pllum.org.pl/_nuxt/pwr.D1_x0B58.png" alt="pwr.D1_x0B58.png" width="100">
+      </a>
+      <br><strong>Politechnika Wrocławska</strong><br><em>– Project Leader</em>
+    </td>
+    <td align="center" valign="middle" style="border: none;">
+      <a href="https://www.nask.pl/">
+        <img src="https://pllum.org.pl/_nuxt/nask.Bz8rmSzR.png" alt="nask.Bz8rmSzR.png" width="100">
+      </a>
+      <br><strong>NASK PIB</strong>
+    </td>
+    <td align="center" valign="middle" style="border: none;">
+      <a href="https://www.ipipan.waw.pl/">
+        <img src="https://clarin.biz/_nuxt/img/ipipan.294d39c.png" alt="ipipan.294d39c.png" width="100">
+      </a>
+      <br><strong>Instytut Podstaw Informatyki PAN</strong>
+    </td>
+  </tr>
+  <tr>
+    <td align="center" valign="middle" style="border: none;">
+      <a href="https://opi.org.pl/">
+        <img src="https://pllum.org.pl/_nuxt/opi.CF-COwcC.png" alt="opi.CF-COwcC.png" width="100">
+      </a>
+      <br><strong>Ośrodek Przetwarzania Informacji PIB</strong>
+    </td>
+    <td align="center" valign="middle" style="border: none;">
+      <a href="https://www.uni.lodz.pl/">
+        <img src="https://pllum.org.pl/_nuxt/ul.aTSgr_W6.png" alt="ul.aTSgr_W6.png" width="100">
+      </a>
+      <br><strong>Uniwersytet Łódzki</strong>
+    </td>
+    <td align="center" valign="middle" style="border: none;">
+      <a href="https://ispan.waw.pl/default/">
+        <img src="https://pllum.org.pl/_nuxt/is.Dqb94VRb.png" alt="is.Dqb94VRb.png" width="100">
+      </a>
+      <br><strong>Instytut Slawistyki PAN</strong>
+    </td>
+  </tr>
+</table>
+
+
+
+## Contact and Support
+For questions or contributions, please reach out via: <pllum@e-science.pl>
+
+We welcome feedback, collaboration, and further exploration of PLLuM models!
+
+
+## Acknowledgements
+
+Project financed by the Minister of Digital Affairs under the targeted subsidy No. 1/WI/DBiI/2023: *“Responsible development of the open large language model PLLuM (Polish Large Language Model) to support breakthrough technologies in the public and economic sector, including an open, Polish-language intelligent assistant for petitioners.”*
+
+**Funding Amount:** 14,504,392.00 PLN  
+**Contract Signing Date:** 2024-01-22

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mixtral",
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "num_local_experts": 8,
+  "output_router_logits": false,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 1000000.0,
+  "router_aux_loss_coef": 0.02,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.36.0.dev0",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.36.0.dev0"
+}

model-00001-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37bc370e8d34dcedfdeb9b9c12f4ef8f4c0693a24f08ec752428be089c1193e9
+size 4920052720

model-00002-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41aded66f9a3e4ae8125752af5886c212299a24a675ae10c0ed70956876fb402
+size 4865559920

model-00003-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94388f3e22b68300f22fbe4eb3a9b3ba247e4360b78a678fd6baa2741a3d63e5
+size 4865559920

model-00004-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b961765e5527c322333e85f96c75597a43576a1b7968d9ba3043e9d0cd0e121
+size 4865559920

model-00005-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19af3339d3369c6d3a0859aa68d576025776f71bec2f5bf7c3bfb6c777ce0098
+size 4865559920

model-00006-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e30a2516f958faa0422dfee718de20bd9ec8a8ae47e07d718aa20fd2cac01e3b
+size 4932504264

model-00007-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63de5a5950e56ae0b06fa4e951caa3add28468cd19e0d94cc82e3be5bbe2192c
+size 4865559912

model-00008-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b544271e7a9950efc9e351fb1a73b8cbd0576d60dafe54eb1893a3a36336bf2b
+size 4865559920

model-00009-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55909b58c784d7feee27af5367b290b05edbb75ec7fd29f7be56f65fd1c491c9
+size 4865559920

model-00010-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5aaf080690b9e96a9664afd0d482923c97e377fcea0107e73fb0b74002d60758
+size 4865559920

model-00011-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20642b49de2e881597e9084d9b689ef33ae051133598033541268ca0ddf4b184
+size 4865559920

model-00012-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0671032b106e9e8c2f4ef3101d1051f6097916bd3c371e18ef876cf30fe179a5
+size 4999646240

model-00013-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1759530e3da1383a0271976a8a1892d76428b686f8b59ef665c5b48ee87da7cb
+size 4798417968

model-00014-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6ba545ec495067d5714e261b722ee0369cd0b0395a7cee58aba7a7efce092ac
+size 4865559944

model-00015-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2a152554abebccd579be3f34fb02b0d856bb6ea9545f6f5856b3b79e9a36d83
+size 4865559944

model-00016-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1d67eabcd65088532796582ee52e54ce5fdb7691f5e534084af5d183aa34d86
+size 4865559944

model-00017-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32deed802ac707ad583576e21fffddd4a199e3a0d39276c5ab71f1f0d66f643e
+size 4865559944

model-00018-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:370ca1799c06b59387daa219e49f728f2afa8b84ab136af68bdcb5f45629b3d4
+size 4865559944

model-00019-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f43f88902e9f9d98b0531596abe7d25894e7d3f7b1794c4942b8145e8dd476dc
+size 4932504280

model-00020-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ec6c2b5beb561599fe3df27731b60af5f5f59955942d43b27faed2cf67c7100
+size 4865559944

model-00021-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f623a8e051501ce340a1a57d7b2f0f499b8d5a2137a3904e727b840a3c5b8604
+size 4865559944

model-00022-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b45070c138b517b7a3e11b2ad114b76383aa636539aac5bcc76579b2235c0563
+size 4865559944

model-00023-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cceb5470dab0f85b4857e77ac586c74fe16db835c9d7d7235c91b44bb52af509
+size 4865559944

model-00024-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9281409a5be5020433703cb5071087a15b1647d72330b149a90ab990718bb587
+size 4865559944

model-00025-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb99022b6b56f35ba619081a674a28400316b82796e40b35b486c84f3438aa75
+size 4932504280

model-00026-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acce6927863169a37aba348adae6e0c857c916cf41f3fbdb33d56cf1afc52dce
+size 4865559944

model-00027-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:725d8d8f57c34fb3c330f5abebd323ca5d89b1b273f2f40e43c1feaed2ee962d
+size 4865559944

model-00028-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d71717216f8d37aaaed9cdd8c07f233abfc01964706f53366cc8e74d952ac18
+size 4865559944

model-00029-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5a3f2af5fd8e8c35a8d312137619b104acad864a74a280bccaa44a3e7147e94
+size 4865559944

model-00030-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c05dcfea89d3f04c4409709155d22f82592b3fa4a8fec5287abac9856e71f043
+size 4865559944

model-00031-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0157df98e7673d89c5c624b6ab55a7653154afa0aaefb3fa0b5ff19d3c9e55d4
+size 4932504280

model-00032-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebc318fd737ec0c713dabfe9bd98a85e9533a5d95d29f5e39b28fded206acd94
+size 4865559944

model-00033-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cd310f86a09dd7bd4545a36ecc24eaaab3696b0662d533081e978205f1915ef
+size 4865559944

model-00034-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6da0b3b5c9c364ebd1c1fd61c5b033a35fc4be167baf176aaf6632d32156cd3
+size 4865559944

model-00035-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a36888950f60fbca7a666ff49b58bd0fce58bf05486326d009d918b61c2e82fa
+size 4865559944

model-00036-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56042be4424c86d0adf35409b444a8da0d1834dff79aee4abe6e2cd5e6162aa8
+size 4865559944

model-00037-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9721ea01b0fd2b17b81fcfdd45261f07c69ef5ede683b8b7bde67f5a499d3d3b
+size 4999646264

model-00038-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe42975bfc8f44fe0a54dae3cf239865b51dde9f2556b930a8a0e6000fd38681
+size 4798417968

model-00039-of-00039.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1fcdabd8f459789a338db34813c29b7aebebd2c2967cf569d4a053f1aba9f47
+size 1463862216

model.safetensors.index.json

@@ -0,0 +1,1002 @@
+{
+  "metadata": {
+    "total_size": 186811170816
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00039-of-00039.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.block_sparse_moe.gate.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00002-of-00039.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00039.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.block_sparse_moe.gate.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00003-of-00039.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00002-of-00039.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00002-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w1.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w2.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w3.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w1.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w2.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w3.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w1.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w2.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w3.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w1.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w2.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w3.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w1.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w2.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w3.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w1.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w2.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w3.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w1.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w2.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w3.weight": "model-00014-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w1.weight": "model-00014-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w2.weight": "model-00014-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w3.weight": "model-00014-of-00039.safetensors",
+    "model.layers.10.block_sparse_moe.gate.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00014-of-00039.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00014-of-00039.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00013-of-00039.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00012-of-00039.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00013-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w1.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w2.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w3.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w1.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w2.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w3.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w1.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w2.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w3.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w1.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w2.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w3.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w1.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w2.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w3.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w1.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w2.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w3.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w1.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w2.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w3.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w1.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w2.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w3.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.block_sparse_moe.gate.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00015-of-00039.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00014-of-00039.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00014-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w1.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w2.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w3.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w1.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w2.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w3.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w1.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w2.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w3.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w1.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w2.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w3.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w1.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w2.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w3.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w1.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w2.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w3.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w1.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w2.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w3.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w1.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w2.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w3.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.block_sparse_moe.gate.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00016-of-00039.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00015-of-00039.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00015-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w1.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w2.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w3.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w1.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w2.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w3.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w1.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w2.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w3.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w1.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w2.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w3.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w1.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w2.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w3.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w1.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w2.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w3.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w1.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w2.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w3.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w1.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w2.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w3.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.block_sparse_moe.gate.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00017-of-00039.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00016-of-00039.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00016-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w1.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w2.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w3.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w1.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w2.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w3.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w1.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w2.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w3.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w1.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w2.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w3.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w1.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w2.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w3.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w1.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w2.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w3.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w1.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w2.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w3.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w1.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w2.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w3.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.block_sparse_moe.gate.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00018-of-00039.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00017-of-00039.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00017-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w1.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w2.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w3.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w1.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w2.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w3.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w1.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w2.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w3.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w1.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w2.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w3.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w1.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w2.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w3.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w1.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w2.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w3.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w1.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w2.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w3.weight": "model-00019-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w1.weight": "model-00020-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w2.weight": "model-00020-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w3.weight": "model-00020-of-00039.safetensors",
+    "model.layers.15.block_sparse_moe.gate.weight": "model-00018-of-00039.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00020-of-00039.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00020-of-00039.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00018-of-00039.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00018-of-00039.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00018-of-00039.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00018-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w1.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w2.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w3.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w1.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w2.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w3.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w1.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w2.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w3.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w1.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w2.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w3.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w1.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w2.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w3.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w1.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w2.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w3.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w1.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w2.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w3.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w1.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w2.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w3.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.block_sparse_moe.gate.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00021-of-00039.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00020-of-00039.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00020-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w1.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w2.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w3.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w1.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w2.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w3.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w1.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w2.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w3.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w1.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w2.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w3.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w1.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w2.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w3.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w1.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w2.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w3.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w1.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w2.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w3.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w1.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w2.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w3.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.block_sparse_moe.gate.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00022-of-00039.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00021-of-00039.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00021-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w1.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w2.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w3.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w1.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w2.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w3.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w1.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w2.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w3.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w1.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w2.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w3.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w1.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w2.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w3.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w1.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w2.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w3.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w1.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w2.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w3.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w1.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w2.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w3.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.block_sparse_moe.gate.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00023-of-00039.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00022-of-00039.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00022-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w1.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w2.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w3.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w1.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w2.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w3.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w1.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w2.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w3.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w1.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w2.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w3.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w1.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w2.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w3.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w1.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w2.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w3.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w1.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w2.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w3.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w1.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w2.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w3.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.block_sparse_moe.gate.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00024-of-00039.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00023-of-00039.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00023-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w3.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w1.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w2.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w3.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w1.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w2.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w3.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.block_sparse_moe.gate.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00004-of-00039.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00003-of-00039.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00003-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w1.weight": "model-00024-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w2.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w3.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w1.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w2.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w3.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w1.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w2.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w3.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w1.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w2.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w3.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w1.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w2.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w3.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w1.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w2.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w3.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w1.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w2.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w3.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w1.weight": "model-00025-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w2.weight": "model-00026-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w3.weight": "model-00026-of-00039.safetensors",
+    "model.layers.20.block_sparse_moe.gate.weight": "model-00024-of-00039.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00026-of-00039.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00026-of-00039.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00024-of-00039.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00024-of-00039.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00024-of-00039.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00024-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w1.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w2.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w3.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w1.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w2.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w3.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w1.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w2.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w3.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w1.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w2.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w3.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w1.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w2.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w3.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w1.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w2.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w3.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w1.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w2.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w3.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w1.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w2.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w3.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.block_sparse_moe.gate.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00027-of-00039.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00026-of-00039.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00026-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.0.w1.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.0.w2.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.0.w3.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.1.w1.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.1.w2.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.1.w3.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.2.w1.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.2.w2.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.2.w3.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.3.w1.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.3.w2.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.3.w3.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.4.w1.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.4.w2.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.4.w3.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.5.w1.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.5.w2.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.5.w3.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.6.w1.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.6.w2.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.6.w3.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.7.w1.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.7.w2.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.experts.7.w3.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.block_sparse_moe.gate.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00028-of-00039.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00027-of-00039.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00027-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.0.w1.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.0.w2.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.0.w3.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.1.w1.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.1.w2.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.1.w3.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.2.w1.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.2.w2.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.2.w3.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.3.w1.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.3.w2.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.3.w3.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.4.w1.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.4.w2.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.4.w3.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.5.w1.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.5.w2.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.5.w3.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.6.w1.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.6.w2.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.6.w3.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.7.w1.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.7.w2.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.experts.7.w3.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.block_sparse_moe.gate.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00029-of-00039.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00028-of-00039.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00028-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.0.w1.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.0.w2.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.0.w3.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.1.w1.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.1.w2.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.1.w3.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.2.w1.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.2.w2.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.2.w3.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.3.w1.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.3.w2.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.3.w3.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.4.w1.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.4.w2.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.4.w3.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.5.w1.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.5.w2.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.5.w3.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.6.w1.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.6.w2.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.6.w3.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.7.w1.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.7.w2.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.experts.7.w3.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.block_sparse_moe.gate.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00030-of-00039.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00029-of-00039.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00029-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.0.w1.weight": "model-00030-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.0.w2.weight": "model-00030-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.0.w3.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.1.w1.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.1.w2.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.1.w3.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.2.w1.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.2.w2.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.2.w3.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.3.w1.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.3.w2.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.3.w3.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.4.w1.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.4.w2.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.4.w3.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.5.w1.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.5.w2.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.5.w3.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.6.w1.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.6.w2.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.6.w3.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.7.w1.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.7.w2.weight": "model-00031-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.experts.7.w3.weight": "model-00032-of-00039.safetensors",
+    "model.layers.25.block_sparse_moe.gate.weight": "model-00030-of-00039.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00032-of-00039.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00032-of-00039.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00030-of-00039.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00030-of-00039.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00030-of-00039.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00030-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.0.w1.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.0.w2.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.0.w3.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.1.w1.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.1.w2.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.1.w3.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.2.w1.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.2.w2.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.2.w3.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.3.w1.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.3.w2.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.3.w3.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.4.w1.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.4.w2.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.4.w3.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.5.w1.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.5.w2.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.5.w3.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.6.w1.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.6.w2.weight": "model-00033-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.6.w3.weight": "model-00033-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.7.w1.weight": "model-00033-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.7.w2.weight": "model-00033-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.experts.7.w3.weight": "model-00033-of-00039.safetensors",
+    "model.layers.26.block_sparse_moe.gate.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00033-of-00039.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00033-of-00039.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00032-of-00039.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00032-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.0.w1.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.0.w2.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.0.w3.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.1.w1.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.1.w2.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.1.w3.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.2.w1.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.2.w2.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.2.w3.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.3.w1.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.3.w2.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.3.w3.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.4.w1.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.4.w2.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.4.w3.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.5.w1.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.5.w2.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.5.w3.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.6.w1.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.6.w2.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.6.w3.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.7.w1.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.7.w2.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.experts.7.w3.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.block_sparse_moe.gate.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00034-of-00039.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00033-of-00039.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00033-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.0.w1.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.0.w2.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.0.w3.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.1.w1.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.1.w2.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.1.w3.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.2.w1.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.2.w2.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.2.w3.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.3.w1.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.3.w2.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.3.w3.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.4.w1.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.4.w2.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.4.w3.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.5.w1.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.5.w2.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.5.w3.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.6.w1.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.6.w2.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.6.w3.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.7.w1.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.7.w2.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.experts.7.w3.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.block_sparse_moe.gate.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00035-of-00039.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00034-of-00039.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00034-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.0.w1.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.0.w2.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.0.w3.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.1.w1.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.1.w2.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.1.w3.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.2.w1.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.2.w2.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.2.w3.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.3.w1.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.3.w2.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.3.w3.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.4.w1.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.4.w2.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.4.w3.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.5.w1.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.5.w2.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.5.w3.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.6.w1.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.6.w2.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.6.w3.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.7.w1.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.7.w2.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.experts.7.w3.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.block_sparse_moe.gate.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00036-of-00039.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00035-of-00039.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00035-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w2.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w1.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w2.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w3.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w1.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w2.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.block_sparse_moe.gate.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00005-of-00039.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00004-of-00039.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00004-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.0.w1.weight": "model-00036-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.0.w2.weight": "model-00036-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.0.w3.weight": "model-00036-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.1.w1.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.1.w2.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.1.w3.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.2.w1.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.2.w2.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.2.w3.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.3.w1.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.3.w2.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.3.w3.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.4.w1.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.4.w2.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.4.w3.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.5.w1.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.5.w2.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.5.w3.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.6.w1.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.6.w2.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.6.w3.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.7.w1.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.7.w2.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.experts.7.w3.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.block_sparse_moe.gate.weight": "model-00036-of-00039.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00037-of-00039.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00036-of-00039.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00036-of-00039.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00036-of-00039.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00036-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.0.w1.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.0.w2.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.0.w3.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.1.w1.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.1.w2.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.1.w3.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.2.w1.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.2.w2.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.2.w3.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.3.w1.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.3.w2.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.3.w3.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.4.w1.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.4.w2.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.4.w3.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.5.w1.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.5.w2.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.5.w3.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.6.w1.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.6.w2.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.6.w3.weight": "model-00039-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.7.w1.weight": "model-00039-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.7.w2.weight": "model-00039-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.experts.7.w3.weight": "model-00039-of-00039.safetensors",
+    "model.layers.31.block_sparse_moe.gate.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00039-of-00039.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00039-of-00039.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00038-of-00039.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00037-of-00039.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00038-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w1.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w3.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w3.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w1.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w3.weight": "model-00007-of-00039.safetensors",
+    "model.layers.4.block_sparse_moe.gate.weight": "model-00005-of-00039.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00007-of-00039.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00007-of-00039.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00005-of-00039.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00005-of-00039.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00005-of-00039.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00005-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w1.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w2.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w3.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w1.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w2.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w3.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w1.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w2.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w3.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w1.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w2.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w3.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w1.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w2.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w3.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w1.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w2.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w3.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w1.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w2.weight": "model-00008-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w3.weight": "model-00008-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w1.weight": "model-00008-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w2.weight": "model-00008-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w3.weight": "model-00008-of-00039.safetensors",
+    "model.layers.5.block_sparse_moe.gate.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00008-of-00039.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00008-of-00039.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00007-of-00039.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00007-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w1.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w2.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w3.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w1.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w2.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w3.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w1.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w2.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w3.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w1.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w2.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w3.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w1.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w2.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w3.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w1.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w2.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w3.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w1.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w2.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w3.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w1.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w2.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w3.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.block_sparse_moe.gate.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00009-of-00039.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00008-of-00039.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00008-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w1.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w2.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w3.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w1.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w2.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w3.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w1.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w2.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w3.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w1.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w2.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w3.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w1.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w2.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w3.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w1.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w2.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w3.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w1.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w2.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w3.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w1.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w2.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w3.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.block_sparse_moe.gate.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00010-of-00039.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00009-of-00039.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00009-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w1.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w2.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w3.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w1.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w2.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w3.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w1.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w2.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w3.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w1.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w2.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w3.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w1.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w2.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w3.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w1.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w2.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w3.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w1.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w2.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w3.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w1.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w2.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w3.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.block_sparse_moe.gate.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00011-of-00039.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00010-of-00039.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00010-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w1.weight": "model-00011-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w2.weight": "model-00011-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w3.weight": "model-00011-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w1.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w2.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w3.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w1.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w2.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w3.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w1.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w2.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w3.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w1.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w2.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w3.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w1.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w2.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w3.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w1.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w2.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w3.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w1.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w2.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w3.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.block_sparse_moe.gate.weight": "model-00011-of-00039.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00012-of-00039.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00011-of-00039.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00011-of-00039.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00011-of-00039.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00011-of-00039.safetensors",
+    "model.norm.weight": "model-00039-of-00039.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc4f0bd70b3709312d9d1d9e5ba674794b6bc5abc17429897a540f93882f25fc
+size 1795303

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}