Duplicate from biohub/ESMC-6B

Co-authored-by: Fausto Milletari <fmilletari-czi@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+images/esmc_graphic.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,364 @@
+---
+license:
+- mit
+- other
+license_link: https://github.com/Biohub/esm/blob/main/THIRD_PARTY_NOTICE.md
+library_name: transformers
+language: en
+tags:
+- biology
+- esm
+- protein
+- protein-language-model
+- protein-embeddings
+- masked-language-modeling
+- transfer-learning
+- variant-effect-prediction
+- protein-engineering
+- transformers
+---
+
+# ESMC
+
+## Model Details
+
+ESMC is a state-of-the-art protein language model that has learned the rules of protein biology from training on billions of protein sequences. ESMC provides representations of proteins enabling novel AI applications from therapeutic protein engineering to unlocking basic insights into protein biology across life.
+
+The ESMC 6B model has 6 billion parameters, with 80 layers and 2.37e23 training flops. We additionally release overtrained 300M and 600M parameter variants of ESMC for local inference and finetuning.
+
+The [ESMFold2](https://huggingface.co/biohub/ESMFold2) structure prediction models are trained on top of a frozen ESMC 6B language model. ESMFold2 is a state-of-the-art model for protein structure prediction and design that defines a new frontier for speed and accuracy.
+
+The [ESMC sparse autoencoder](https://huggingface.co/biohub/ESMC-6B-sae-layer60-k64-codebook16384), `ESMC-6B-sae-layer60-k64-codebook16384`, is built on the ESMC 6B model and provides human-interpretable, agent-generated feature descriptions. See the [ESMC SAE overview card](https://huggingface.co/biohub/ESMC-SAE-Overview) for the full set of ESMC SAE variants.
+
+To run this model with the Biohub Platform API, visit the [Biohub Platform](https://biohub.ai/).
+
+Read more about ESMC in our paper [here](https://www.biorxiv.org/content/10.64898/2026.06.03.729735).
+
+### Example Usage
+
+Install `esm` from PyPI:
+
+```
+pip install esm
+```
+
+```py
+import torch
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
+
+# optionally use "biohub/ESMC-600M" or "biohub/ESMC-300M"
+model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-6B", device_map="auto").eval()
+tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")
+
+inputs = tokenizer(GFP, return_tensors="pt", padding=True)
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+with torch.inference_mode():
+    output = model(**inputs)
+
+print(f"logits shape: {tuple(output.logits.shape)}")
+```
+
+By default, the model returns only the final layer representations. To return hidden states from **all transformer layers**, set:
+
+```py
+output = model(**inputs, output_hidden_states=True)
+```
+
+For detailed usage, refer to the [Usage section below](#usage).
+
+### Citation
+
+```
+@misc{candido2026language,
+  title  = {Language Modeling Materializes a World Model of Protein Biology},
+  author = {Candido, Salvatore and Hayes, Thomas and Derry, Alexander and Rao, Roshan
+            and Lin, Zeming and Verkuil, Robert and Wu, Bryan and Lee, Jin Sub
+            and Bruguera, Elise S. and Keval, Jehan A. and Kopylov, Mykhailo
+            and Pak, John E. and Wu, Wesley and Thomas, Neil and Mataraso, Samson
+            and Hsu, Alvin and Trotman-Grant, Ashton C. and Fatras, Kilian
+            and dos Santos Costa, Allan and Badkundri, Rohil and Ak{\i}n, Halil
+            and Oktay, Deniz and Deaton, Jonathan and Montabana, Elizabeth
+            and Sitwala, Hrishita and Yu, Yue and Wiggert, Marius
+            and Carlin, Dylan Alexander and Goering, Anthony W. and Blazejewski, Tomasz
+            and Sandora, McCullen and Hla, Michael and Jia, Tina Z.
+            and Kloker, Leon H. and Sofroniew, Nicholas J. and Uehara, Masatoshi
+            and Pannu, Jassi and Bachas, Sharrol and Liu, Daniel S.
+            and Sercu, Tom and Rives, Alexander},
+  year   = {2026},
+  url    = {https://www.biorxiv.org/content/10.64898/2026.06.03.729735},
+  note   = {Preprint}
+}
+```
+
+### Model Architecture
+
+ESMC is based on the transformer architecture. It features Pre-LN, rotary embeddings, and SwiGLU activations. No biases are used in linear layers or layer norms.
+
+### Parameters
+
+ESMC was trained at multiple scales:
+
+| Model | Parameters | Layers | Training FLOPs |
+| :---- | ----: | ----: | ----: |
+| **ESMC-300M** | 300M | 30 | 1.26e22 |
+| **ESMC-600M** | 600M | 36 | 2.17e22 |
+| **ESMC-6B** | 6B | 80 | 2.37e23 |
+
+![][pal]
+
+### Model Variants
+
+| Model Variant | Description | URL |
+| :---- | :---- | :---- |
+| ESMC 300M | Smallest variant, publicly released. | [https://huggingface.co/biohub/ESMC-300M](https://huggingface.co/biohub/ESMC-300M) |
+| ESMC 600M | Medium variant, publicly released. | [https://huggingface.co/biohub/ESMC-600M](https://huggingface.co/biohub/ESMC-600M) |
+| ESMC 6B | Large variant, publicly released. | [https://huggingface.co/biohub/ESMC-6B](https://huggingface.co/biohub/ESMC-6B) |
+
+### System Requirements
+
+- Compute Requirements: GPU
+- PyTorch environment with GPU support recommended.
+- Recommended optional libraries: transformer\_engine, xformers
+
+## Training Data
+
+ESMC was trained on protein sequences from UniRef, MGnify, and the Joint Genome Institute (JGI). Sequence data was clustered at 70% sequence identity, resulting in 83M, 372M, and 2B clusters for UniRef, MGnify, and JGI, respectively.
+
+### Training Procedure
+
+Training was conducted in two stages:
+
+- Stage 1: For the first 1 million steps, the model used a context length of 512, with metagenomic data constituting 64% of the training dataset.
+- Stage 2: In the final 500,000 steps, the context length was increased to 2048, and the proportion of metagenomic data was reduced to 37.5%.
+
+## Performance Metrics
+
+Performance metrics are detailed in our [ESMC & ESMFold2 paper](https://www.biorxiv.org/content/10.64898/2026.06.03.729735).
+
+## Usage
+
+### Flash Attention
+
+Instead of scaled dot product attention (sdpa) you can use a flash attention backend. This requires running the model in bfloat16.
+
+```py
+model = (
+    AutoModelForMaskedLM.from_pretrained(
+        "biohub/ESMC-6B",
+        dtype=torch.bfloat16,
+        device_map="auto",
+        attn_implementation="flash_attention_2",
+    )
+    .to(torch.bfloat16)
+    .eval()
+)
+```
+
+### Sparse Autoencoder (SAE)
+
+To get interpretable features from ESMC 6B hidden states and per-layer MLP outputs, you can choose from our pretrained SAEs. We provide the follow three:
+
+* [ESMC SAEs for hidden states (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-hidden-states-all-layers)
+* [ESMC SAEs for one layer (different sparsity/codebook size)](https://huggingface.co/collections/biohub/esmc-saes-for-one-layer-different-sparsity-codebook-size)
+* [ESMC SAEs for MLP outputs (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-mlp-outputs-all-layers)
+
+```py
+import torch
+from esm.models.esmc import EsmcModel, EsmcSaeModel, EsmcTokenizer
+
+GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
+
+model = EsmcModel.from_pretrained("biohub/ESMC-6B").eval()
+tokenizer = EsmcTokenizer.from_pretrained("biohub/ESMC-6B")
+
+sae = EsmcSaeModel.from_pretrained(
+    "biohub/ESMC-6B-sae-layer60-k64-codebook16384", device=model.device
+)
+model.add_sae_models([sae.layers["60"]])
+
+inputs = tokenizer(GFP, return_tensors="pt", padding=True)
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+with torch.inference_mode():
+    output = model(**inputs)
+
+print(f"SAE features: {tuple(output.sae_outputs['layer60'].shape)}")
+```
+
+### Masked Language Modeling
+
+ESMC can predict masked amino acids and compute the corresponding loss:
+
+```py
+import torch
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
+masked_GFP = "<mask>SKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
+
+model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-6B", device_map="auto").eval()
+tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")
+
+inputs = tokenizer(masked_GFP, return_tensors="pt")
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+labels = tokenizer(GFP, return_tensors="pt")["input_ids"].to(model.device)
+# Only the masked positions contribute to the loss; everything else gets the
+# ``-100`` ignore-index that ``CrossEntropyLoss`` skips.
+labels = torch.where(inputs["input_ids"] == tokenizer.mask_token_id, labels, -100)
+
+with torch.inference_mode():
+    output = model(**inputs, labels=labels)
+
+print(f"Loss: {output.loss.item():.6f}")
+```
+
+### Fine-tuning with peft
+
+```py
+from peft import LoraConfig, get_peft_model
+from transformers import AutoModelForMaskedLM
+
+model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-6B", device_map="auto")
+
+lora_config = LoraConfig(
+    r=8,
+    lora_alpha=16,
+    lora_dropout=0.01,
+    target_modules=[
+        "q_proj",
+        "k_proj",
+        "v_proj",
+        "o_proj",
+        "gate_proj",
+        "up_proj",
+        "down_proj",
+    ],
+)
+
+model = get_peft_model(model, lora_config)
+model.print_trainable_parameters()
+```
+
+### Attention Maps
+
+To extract attention maps, pass `output_attentions=True`. Note: this is incompatible with `attn_implementation="flash_attention_2"`.
+
+```py
+model = AutoModelForMaskedLM.from_pretrained(
+    "biohub/ESMC-6B", device_map="auto", attn_implementation="eager"
+).eval()
+output = model(**inputs, output_attentions=True)
+# output.attentions: tuple of (batch, n_heads, seq_len, seq_len) tensors, one per layer
+```
+
+`output_attentions=True` triggers a manual, unoptimized attention path to extract the attention maps, which will reduce inference speed.
+
+### Other Usage
+
+You can access the base model without the pretrained LM head:
+
+```py
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
+
+model = AutoModel.from_pretrained("biohub/ESMC-6B", device_map="auto").eval()
+tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")
+
+inputs = tokenizer(GFP, return_tensors="pt", padding=True)
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+with torch.inference_mode():
+    output = model(**inputs)
+
+print(f"last_hidden_state shape: {tuple(output.last_hidden_state.shape)}")
+```
+
+Or use ESMC for Token Classification:
+
+```py
+import torch
+from transformers import AutoModelForTokenClassification, AutoTokenizer
+
+GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
+
+model = AutoModelForTokenClassification.from_pretrained(
+    "biohub/ESMC-6B", device_map="auto"
+).eval()
+tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")
+
+inputs = tokenizer(GFP, return_tensors="pt", padding=True)
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+with torch.inference_mode():
+    output = model(**inputs)
+
+predicted_token_class_ids = output.logits.argmax(-1)
+predicted_tokens_classes = [
+    model.config.id2label[t.item()] for t in predicted_token_class_ids[0]
+]
+print(f"logits shape: {tuple(output.logits.shape)}")
+print(f"first 8 predicted classes: {predicted_tokens_classes[:8]}")
+```
+
+or Sequence Classification:
+
+```py
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"
+
+model = AutoModelForSequenceClassification.from_pretrained(
+    "biohub/ESMC-6B", device_map="auto", num_labels=2
+).eval()
+tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")
+
+inputs = tokenizer(GFP, return_tensors="pt", padding=True)
+inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+with torch.inference_mode():
+    output = model(**inputs)
+
+print(f"logits shape: {tuple(output.logits.shape)}")
+```
+
+For Token or Sequence Classification, the classifier head is not pretrained but instead meant to be fine-tuned for your downstream task.
+
+## Frontier Safety
+
+Biohub has established a safety team to assess the benefits and potential risks of our models and tools prior to release, and develop mitigations where necessary. Informed by our risk assessments, we are releasing the source code and model weights for ESMC 6B, ESMFold2, and ESMC SAEs. We are also releasing our ESM Atlas dataset and binder design system openly.
+
+Prior to release, we conducted evaluations to inform our understanding of capability uplift for specific misuse-relevant functional tasks. The full details of these evaluations are available in our corresponding paper appendix.
+
+[Biohub.ai](http://Biohub.ai) Platform: We implement guardrails that detect and restrict the use of keywords and sequences corresponding to controlled pathogens and toxins on our freely accessible platform. For further details regarding these guardrails, please refer to our Biohub platform Resources page.
+
+## Biases and Limitations
+
+### Potential Biases
+
+- **Dataset bias:** Over- or under-representation of taxa, protein families, or ecological niches in public sequence and structure databases influences generalization and can bias outputs. This is partially mitigated by clustering-based, nonredundant sampling.
+
+### Limitations
+
+- **Context window:** ESMC has a context window limit of 2048 tokens.
+- **Reliance on in-silico metrics:** Computational metrics do not replace wet-lab validation.
+
+### Out-of-Scope or Unauthorized Use Cases
+
+Do not use the model for the following purposes:
+
+- Any use that is prohibited by the [Acceptable Use Policy](https://biohub.org/acceptable-use-policy/).
+
+### Caveats and Recommendations
+
+- Review and validate outputs generated by the model.
+- We are committed to advancing the responsible development and use of artificial intelligence.
+- Should you have any security or privacy issues or questions related to the services, please reach out to our team at [support@biohub.org](mailto:support@biohub.org).
+
+[pal]: images/contact_pal.png

config.json

@@ -0,0 +1,38 @@
+{
+  "architectures": [
+    "EsmcForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "modeling_esmc_remote.EsmcRemoteConfig",
+    "AutoModel": "modeling_esmc_remote.EsmcRemoteModel",
+    "AutoModelForMaskedLM": "modeling_esmc_remote.EsmcRemoteForMaskedLM",
+    "AutoModelForSequenceClassification": "modeling_esmc_remote.EsmcRemoteForSequenceClassification",
+    "AutoModelForTokenClassification": "modeling_esmc_remote.EsmcRemoteForTokenClassification"
+  },
+  "bos_token_id": 0,
+  "classifier_dropout": 0.1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 2560,
+  "initializer_range": 0.02,
+  "intermediate_size": 6912,
+  "mask_token_id": 32,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "esmc",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 80,
+  "num_key_value_heads": 40,
+  "pad_token_id": 1,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.16.0.dev0",
+  "vocab_size": 64
+}

model-00001-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80cc4a51f218de3bbe06aa56fe55bc695f0b7707159c01f3e4d8aa49c8cf38b4
+size 4935170376

model-00002-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:273740475dcd0cab8b9b591f5458679a5a9cc96cd1e8108fb856b1ec5d581c06
+size 4979100240

model-00003-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00769f0e680aeae99c4ea877f61c5c6532faf9a0726e6a7cb449d3ef9953d993
+size 4997470904

model-00004-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1287dd34cb55f2d9730b81d961c8a693bec4cc3d0c3c13adb6a91e950f6de933
+size 4934515048

model-00005-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fef715f2b9f0bc52f846f07fba9d5183ff8eef20edadc3e6bfb67070ebcbf4b2
+size 4979100240

model-00006-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f88f2e9a88439be6672f5dfa6ae369c7ae87fb533e0a25a073f706cd9ac5ddd
+size 582782360

model.safetensors.index.json

@@ -0,0 +1,1056 @@
+{
+  "metadata": {
+    "total_parameters": 6352005184,
+    "total_size": 25408020736
+  },
+  "weight_map": {
+    "esmc.embed_tokens.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.0.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.1.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.10.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.11.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.12.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.13.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.14.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.15.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.15.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.15.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.15.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.15.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.15.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.15.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.15.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.15.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.15.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.15.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.15.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.15.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.16.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.16.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.17.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.18.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.19.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.2.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.2.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.20.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.20.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.21.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.22.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.23.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.24.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.25.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.26.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.27.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.28.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.29.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.3.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.3.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.30.input_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.input_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.mlp.down_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.mlp.gate_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.mlp.up_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.post_attention_layernorm.bias": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.post_attention_layernorm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.self_attn.k_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.self_attn.o_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.self_attn.q_norm.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.30.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.31.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.self_attn.k_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.31.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.31.self_attn.q_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.31.self_attn.v_proj.weight": "model-00002-of-00006.safetensors",
+    "esmc.layers.32.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.32.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.33.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.34.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.35.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.36.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.37.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.38.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.39.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.4.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.4.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.40.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.40.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.41.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.42.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.43.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.44.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.45.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.input_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.input_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.mlp.down_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.mlp.gate_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.mlp.up_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.post_attention_layernorm.bias": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.post_attention_layernorm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.self_attn.k_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.self_attn.k_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.self_attn.o_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.self_attn.q_norm.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.self_attn.q_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.46.self_attn.v_proj.weight": "model-00003-of-00006.safetensors",
+    "esmc.layers.47.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.47.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.48.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.49.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.5.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.5.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.50.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.50.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.51.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.52.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.53.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.54.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.55.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.56.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.57.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.58.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.59.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.6.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.6.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.60.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.60.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.input_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.input_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.mlp.down_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.mlp.up_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.post_attention_layernorm.bias": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.post_attention_layernorm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.61.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.62.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.62.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.62.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.62.mlp.gate_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.62.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.62.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.62.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.62.self_attn.k_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.62.self_attn.k_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.62.self_attn.o_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.62.self_attn.q_norm.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.62.self_attn.q_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.62.self_attn.v_proj.weight": "model-00004-of-00006.safetensors",
+    "esmc.layers.63.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.63.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.64.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.65.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.66.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.67.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.68.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.69.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.7.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.7.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.70.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.70.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.71.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.72.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.73.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.74.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.75.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.76.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.input_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.input_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.mlp.down_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.mlp.gate_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.mlp.up_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.post_attention_layernorm.bias": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.post_attention_layernorm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.self_attn.k_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.self_attn.o_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.self_attn.q_norm.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.77.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.78.input_layernorm.bias": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.post_attention_layernorm.bias": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.self_attn.k_norm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.self_attn.k_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.78.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.self_attn.q_norm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.78.self_attn.q_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.78.self_attn.v_proj.weight": "model-00005-of-00006.safetensors",
+    "esmc.layers.79.input_layernorm.bias": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.input_layernorm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.mlp.down_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.mlp.gate_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.mlp.up_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.post_attention_layernorm.bias": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.post_attention_layernorm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.self_attn.k_norm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.self_attn.k_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.self_attn.o_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.self_attn.q_norm.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.self_attn.q_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.79.self_attn.v_proj.weight": "model-00006-of-00006.safetensors",
+    "esmc.layers.8.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.8.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.input_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.input_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.mlp.down_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.mlp.gate_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.mlp.up_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.post_attention_layernorm.bias": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.post_attention_layernorm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.self_attn.k_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.self_attn.k_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.self_attn.o_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.self_attn.q_norm.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.self_attn.q_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.layers.9.self_attn.v_proj.weight": "model-00001-of-00006.safetensors",
+    "esmc.norm.weight": "model-00006-of-00006.safetensors",
+    "lm_head.decoder.bias": "model-00006-of-00006.safetensors",
+    "lm_head.decoder.weight": "model-00006-of-00006.safetensors",
+    "lm_head.dense.bias": "model-00006-of-00006.safetensors",
+    "lm_head.dense.weight": "model-00006-of-00006.safetensors",
+    "lm_head.layer_norm.bias": "model-00006-of-00006.safetensors",
+    "lm_head.layer_norm.weight": "model-00006-of-00006.safetensors"
+  }
+}

modeling_esmc_remote.py

@@ -0,0 +1,140 @@
+"""Load this checkpoint with the ``esm`` package instead of the transformers port.
+
+Shipped inside each ``biohub/ESMC-*`` repo and named by its ``config.json``
+``auto_map``, so ``AutoModel.from_pretrained(repo, trust_remote_code=True)``
+returns an ``esm.models.esmc`` model. Without the flag transformers keeps using
+its own port, which it has shipped since 5.x.
+
+One shim class per Auto class transformers maps for ``esmc``. Dropping any of
+them would break that Auto class on this repo: ``AutoConfig`` resolves to
+:class:`EsmcRemoteConfig`, which is in none of transformers' own mappings.
+
+Runs from the Hub module cache in the user's environment, so it imports nothing
+but ``transformers``, ``packaging`` and ``esm``. The ESMFold2 loader is a
+separate file for the same reason -- each repo downloads one module, so there is
+nowhere to share code from.
+"""
+
+from typing import Any
+
+from packaging.version import Version
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_utils import PreTrainedModel
+
+#: Inclusive floor. 3.4.0 reads the published layout but takes no config
+#: overrides, so it would silently ignore a ``num_labels`` on a head load.
+MIN_ESM_VERSION = "3.4.1"
+
+#: The keyword arguments esm's ESMC ``from_pretrained`` declares.
+_ESM_KWARGS = frozenset(
+    {
+        "device",
+        "dtype",
+        "attn_implementation",
+        "revision",
+        "cache_dir",
+        "token",
+        "local_files_only",
+        "force_download",
+    }
+)
+
+
+def esmc_class(name: str) -> Any:
+    """Return a class from ``esm.models.esmc``, or raise ``ImportError``.
+
+    The message names the pip command; a bare ImportError out of the Hub module
+    loader would not.
+    """
+    try:
+        import esm
+        from esm.models import esmc
+    except ImportError as exc:
+        raise ImportError(
+            f"trust_remote_code=True runs the esm package, which failed to "
+            f"import ({exc}). Install it with:\n\n"
+            f"    pip install 'esm>={MIN_ESM_VERSION}'\n\n"
+            "Or drop the flag to use the ESMC port in transformers."
+        ) from exc
+    if Version(esm.__version__) < Version(MIN_ESM_VERSION):
+        raise ImportError(
+            f"esm {esm.__version__} is installed, but this checkpoint needs "
+            f"{MIN_ESM_VERSION} or newer. Upgrade with:\n\n"
+            f"    pip install --upgrade 'esm>={MIN_ESM_VERSION}'"
+        )
+    return getattr(esmc, name)
+
+
+class EsmcRemoteConfig(PretrainedConfig):
+    """Holds config.json verbatim. esm re-reads the file and builds its own config."""
+
+    model_type = "esmc"
+
+
+class _EsmcRemoteLoader(PreTrainedModel):
+    """Loader shim; ``from_pretrained`` returns an esm model, not one of these.
+
+    A ``PreTrainedModel`` subclass only because the Auto classes call
+    ``register_for_auto_class`` on what they load and check its ``config_class``.
+    """
+
+    config_class = EsmcRemoteConfig
+
+    #: The ``esm.models.esmc`` class this shim loads.
+    esm_class_name = "EsmcModel"
+
+    #: esm keyword -> config attribute. transformers absorbs these into
+    #: ``config`` before the model sees them, so
+    #: ``AutoModel(..., attn_implementation="flash_attention_2")`` never
+    #: reaches us as a keyword. Read back off that config instead of dropped.
+    esm_config_fields: dict[str, str] = {"attn_implementation": "_attn_implementation"}
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *args, **kwargs):  # type: ignore[override]
+        # esm's signature is fixed, so transformers' trust_remote_code /
+        # _from_auto / config would TypeError.
+        forwarded = {k: v for k, v in kwargs.items() if k in _ESM_KWARGS}
+        config = kwargs.get("config")
+        for keyword, attribute in cls.esm_config_fields.items():
+            value = getattr(config, attribute, None)
+            if value is not None:
+                forwarded.setdefault(keyword, value)
+        return esmc_class(cls.esm_class_name).from_pretrained(
+            pretrained_model_name_or_path, *args, **forwarded
+        )
+
+
+class EsmcRemoteModel(_EsmcRemoteLoader):
+    """``AutoModel`` -- the bare encoder."""
+
+    esm_class_name = "EsmcModel"
+
+
+class EsmcRemoteForMaskedLM(_EsmcRemoteLoader):
+    """``AutoModelForMaskedLM`` -- the encoder plus the published LM head."""
+
+    esm_class_name = "EsmcForMaskedLM"
+
+
+#: A head load also has to carry the shape of the head, which transformers
+#: takes as ``num_labels`` and resolves against ``id2label``.
+_HEAD_FIELDS = {
+    **_EsmcRemoteLoader.esm_config_fields,
+    "num_labels": "num_labels",
+    "problem_type": "problem_type",
+    "classifier_dropout": "classifier_dropout",
+}
+
+
+class EsmcRemoteForSequenceClassification(_EsmcRemoteLoader):
+    """``AutoModelForSequenceClassification`` -- head is randomly initialised."""
+
+    esm_class_name = "EsmcForSequenceClassification"
+    esm_config_fields = _HEAD_FIELDS
+
+
+class EsmcRemoteForTokenClassification(_EsmcRemoteLoader):
+    """``AutoModelForTokenClassification`` -- head is randomly initialised."""
+
+    esm_class_name = "EsmcForTokenClassification"
+    esm_config_fields = _HEAD_FIELDS

special_tokens_map.json

@@ -0,0 +1,10 @@
+{
+  "additional_special_tokens": [
+    "|"
+  ],
+  "cls_token": "<cls>",
+  "eos_token": "<eos>",
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

tokenizer.json

@@ -0,0 +1,167 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<cls>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<eos>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 31,
+      "content": "|",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 32,
+      "content": "<mask>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": null,
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "SpecialToken": {
+          "id": "<cls>",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "<eos>",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {
+      "<cls>": {
+        "id": "<cls>",
+        "ids": [
+          0
+        ],
+        "tokens": [
+          "<cls>"
+        ]
+      },
+      "<eos>": {
+        "id": "<eos>",
+        "ids": [
+          2
+        ],
+        "tokens": [
+          "<eos>"
+        ]
+      }
+    }
+  },
+  "decoder": null,
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": "<unk>",
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "<cls>": 0,
+      "<pad>": 1,
+      "<eos>": 2,
+      "<unk>": 3,
+      "L": 4,
+      "A": 5,
+      "G": 6,
+      "V": 7,
+      "S": 8,
+      "E": 9,
+      "R": 10,
+      "T": 11,
+      "I": 12,
+      "D": 13,
+      "P": 14,
+      "K": 15,
+      "Q": 16,
+      "N": 17,
+      "F": 18,
+      "Y": 19,
+      "M": 20,
+      "H": 21,
+      "W": 22,
+      "C": 23,
+      "X": 24,
+      "B": 25,
+      "U": 26,
+      "Z": 27,
+      "O": 28,
+      ".": 29,
+      "-": 30,
+      "|": 31,
+      "<mask>": 32
+    },
+    "merges": []
+  }
+}

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<cls>",
+  "chain_break_token": "|",
+  "cls_token": "<cls>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "tokenizer_class": "EsmcTokenizer",
+  "unk_token": "<unk>"
+}