Duplicate from ibm-granite/granite-4.0-1b-speech

Co-authored-by: Madison Lee <kristunlee@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+multilingual_sample.wav filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,310 @@
+---
+license: apache-2.0
+language:
+- multilingual
+- en
+- fr
+- de
+- es
+- pt
+- ja
+base_model:
+- ibm-granite/granite-4.0-1b-base
+library_name: transformers
+---
+# Granite-4.0-1b-speech
+
+**Model Summary:**
+Granite-4.0-1b-speech is a compact and efficient speech-language model, specifically designed for multilingual automatic speech recognition (ASR) and bidirectional automatic speech translation (AST).
+
+The model was trained on a collection of public corpora comprising of diverse datasets for ASR and AST as well as synthetic datasets tailored to support Japanese ASR, keyword-biased ASR and speech translation. 
+Granite-4.0-1b-speech was trained by modality aligning [granite-4.0-1b-base](https://huggingface.co/ibm-granite/granite-4.0-1b-base) to speech on publicly available open source corpora containing audio inputs and text targets.
+Compared to [granite-speech-3.3-2b](https://huggingface.co/ibm-granite/granite-speech-3.3-2b) and [granite-speech-3.3-8b](https://huggingface.co/ibm-granite/granite-speech-3.3-8b), this model has the following additional capabilities and improvements:
+* Supports multilingual speech inputs in English, French, German, Spanish, Portuguese and Japanese,
+* Provides higher transcription accuracy for English ASR and faster inference through better encoder training and speculative decoding,
+* Has half the number of parameters of [granite-speech-3.3-2b](https://huggingface.co/ibm-granite/granite-speech-3.3-2b) for running on resource-constrained devices,
+* Adds keyword list biasing capability for enhanced name and acronym recognition
+
+**Evaluations:**
+
+We evaluated granite-4.0-1b-speech alongside other speech-language models in the less than 8b parameter range as well as dedicated ASR and AST systems on standard benchmarks. The evaluation spanned multiple public benchmarks, with particular emphasis on English ASR tasks while also including multilingual ASR and AST for X-En and En-X translations. 
+<br>
+![granite-4.0-1b-speech-wer1-crop](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/KvRYoVoAWlMm0GrcdBWkR.png)
+<br>
+![granite-4.0-1b-speech-wer2-crop](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/YHX5qS7_8NlhuAA2bQ5FL.png)
+<br>
+![granite-4.0-1b-speech-bleu1-crop](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/HO3bmHqWXnMisIKPMAmSw.png)
+<br>
+![granite-4.0-1b-speech-bleu2-crop](https://cdn-uploads.huggingface.co/production/uploads/666ec38102791b3b49f453e8/QTaXEl4PcygKiw_1zaYjO.png)
+<br>
+
+**Release Date**: March 6, 2026 
+
+**License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+**Supported Languages:**
+English, French, German, Spanish, Portuguese, Japanese
+
+**Intended Use:** 
+The model is intended to be used in enterprise applications that involve processing of speech inputs. 
+In particular, the model is well-suited for English, French, German, Spanish, Portuguese and Japanese speech-to-text and speech translations 
+to and from English for the same languages, plus English-to-Italian and English-to-Mandarin. 
+
+## Generation:
+
+Granite Speech model is supported natively in `transformers` from the `main` branch. Below is a simple example of how to use the `granite-4.0-1b-speech` model.
+
+### Usage with `transformers`
+
+First, make sure to install a recent version of transformers:
+```shell
+pip install transformers torchaudio soundfile
+```
+
+Then run the code:
+```python
+import torch
+import torchaudio
+from huggingface_hub import hf_hub_download
+from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+model_name = "ibm-granite/granite-4.0-1b-speech"
+processor = AutoProcessor.from_pretrained(model_name)
+tokenizer = processor.tokenizer
+model = AutoModelForSpeechSeq2Seq.from_pretrained(
+    model_name, device_map=device, torch_dtype=torch.bfloat16
+)
+
+# Load audio
+audio_path = hf_hub_download(repo_id=model_name, filename="multilingual_sample.wav")
+wav, sr = torchaudio.load(audio_path, normalize=True)
+assert wav.shape[0] == 1 and sr == 16000  # mono, 16kHz
+
+# Create text prompt
+user_prompt = "<|audio|>can you transcribe the speech into a written format?"
+chat = [
+    {"role": "user", "content": user_prompt},
+]
+prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
+
+# Run the processor + model
+model_inputs = processor(prompt, wav, device=device, return_tensors="pt").to(device)
+model_outputs = model.generate(
+    **model_inputs, max_new_tokens=200, do_sample=False, num_beams=1
+)
+
+# Transformers includes the input IDs in the response
+num_input_tokens = model_inputs["input_ids"].shape[-1]
+new_tokens = model_outputs[0, num_input_tokens:].unsqueeze(0)
+output_text = tokenizer.batch_decode(
+    new_tokens, add_special_tokens=False, skip_special_tokens=True
+)
+print(f"STT output = {output_text[0]}")
+```
+
+### Usage with `vLLM`
+
+First, make sure to install vLLM:
+```shell
+pip install vllm
+```
+* Code for offline mode:
+```python
+from transformers import AutoTokenizer
+from vllm import LLM, SamplingParams
+from vllm.assets.audio import AudioAsset
+
+model_id = "ibm-granite/granite-4.0-1b-speech"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+def get_prompt(question: str, has_audio: bool):
+    """Build the input prompt to send to vLLM."""
+    if has_audio:
+        question = f"<|audio|>{question}"
+    chat = [
+        {
+            "role": "user",
+            "content": question
+        }
+    ]
+    return tokenizer.apply_chat_template(chat, tokenize=False)
+
+model = LLM(
+    model=model_id,
+    max_model_len=2048, # This may be needed for lower resource devices.
+    limit_mm_per_prompt={"audio": 1},
+)
+
+question = "can you transcribe the speech into a written format?"
+prompt_with_audio = get_prompt(
+    question=question,
+    has_audio=True,
+)
+audio = AudioAsset("mary_had_lamb").audio_and_sample_rate
+
+inputs = {
+    "prompt": prompt_with_audio,
+    "multi_modal_data": {
+        "audio": audio,
+    }
+}
+
+outputs = model.generate(
+    inputs,
+    sampling_params=SamplingParams(
+        temperature=0.2,
+        max_tokens=64,
+    ),
+)
+print(f"Audio Example - Question: {question}")
+print(f"Generated text: {outputs[0].outputs[0].text}")
+```
+
+* Code for online mode:
+```python
+"""
+Launch the vLLM server with the following command:
+
+vllm serve ibm-granite/granite-4.0-1b-speech \
+    --api-key token-abc123 \
+    --max-model-len 2048
+"""
+
+import base64
+
+import requests
+from openai import OpenAI
+
+from vllm.assets.audio import AudioAsset
+
+# Modify OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "token-abc123"
+openai_api_base = "http://localhost:8000/v1"
+
+client = OpenAI(
+    # defaults to os.environ.get("OPENAI_API_KEY")
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+model_name = "ibm-granite/granite-4.0-1b-speech"
+# Any format supported by librosa is supported
+audio_url = AudioAsset("mary_had_lamb").url
+
+# Use base64 encoded audio in the payload
+def encode_audio_base64_from_url(audio_url: str) -> str:
+    """Encode an audio retrieved from a remote url to base64 format."""
+    with requests.get(audio_url) as response:
+        response.raise_for_status()
+        result = base64.b64encode(response.content).decode("utf-8")
+    return result
+
+audio_base64 = encode_audio_base64_from_url(audio_url=audio_url)
+
+question = "can you transcribe the speech into a written format?"
+chat_completion_with_audio = client.chat.completions.create(
+    messages=[{
+        "role": "user",
+        "content": [
+            {
+                "type": "text",
+                "text": question
+            },
+            {
+                "type": "audio_url",
+                "audio_url": {
+                    # Any format supported by librosa is supported
+                    "url": f"data:audio/ogg;base64,{audio_base64}"
+                },
+            },
+        ],
+    }],
+    temperature=0.2,
+    max_tokens=64,
+    model=model_name,
+)
+
+
+print(f"Audio Example - Question: {question}")
+print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")
+```
+
+**Model Architecture:** 
+
+The architecture of granite-4.0-1b-speech consists of the following components:
+
+(1) Speech encoder: 16 conformer blocks trained with Connectionist Temporal Classification (CTC) on character-level targets on the subset containing
+only ASR corpora (see configuration below). The character vocabulary consists of the first 256 ASCII entries for the European languages plus a 92 phonetic Katakana character set for Japanese. In addition, our CTC encoder uses block-attention with 4-seconds audio blocks and self-conditioned CTC
+from the middle layer.
+
+| Configuration parameter  | Value                | 
+|-----------------|----------------------|
+| Input dimension | 160 (80 logmels x 2) | 
+| Nb. of layers   | 16                   | 
+| Hidden dimension | 1024                | 
+| Nb. of attention heads | 8             | 
+| Attention head size    | 128           | 
+| Convolution kernel size | 15           | 
+| Output dimension        | 348          | 
+
+(2) Speech projector and temporal downsampler (speech-text modality adapter): we use a 2-layer window query transformer (q-former) operating on
+blocks of 15 1024-dimensional acoustic embeddings coming out of the last conformer block of the speech encoder that get downsampled by a factor of 5
+using 3 trainable queries per block and per layer. The total temporal downsampling factor is 10 (2x from the encoder and 5x from the projector)
+resulting in a 10Hz acoustic embeddings rate for the LLM. The encoder, projector and LoRA adapters were fine-tuned/trained jointly on all the
+corpora mentioned under **Training Data**.
+
+(3) Large language model: granite-4.0-1b-base with 128k context length (https://huggingface.co/ibm-granite/granite-4.0-1b-base) finetuned on all 
+the corpora mentioned under **Training Data**.
+
+**Training Data:** 
+
+Overall, our training data is largely comprised of two key sources: (1) publicly available datasets (2) Synthetic data created from publicly
+available datasets specifically targeting Japanese ASR, keyword list-prompted ASR and the speech translation task. 
+A detailed description of the training datasets can be found in the table below:
+
+| Name | Task | Nb. hours | Source |
+|-----------|--------------|----------------|--------------|
+| CommonVoice-17 En,De,Es,Fr,Pt,Ja  | ASR | 5700 |   https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0 |
+| MLS En,De,Es,Fr,Pt             | ASR | 48000 |   https://huggingface.co/datasets/facebook/multilingual_librispeech |
+| Librispeech English            | ASR | 1000 |  https://huggingface.co/datasets/openslr/librispeech_asr | 
+| VoxPopuli En,De,Fr,Es       | ASR | 1100 |  https://huggingface.co/datasets/facebook/voxpopuli | 
+| AMI English                    | ASR | 100 | https://huggingface.co/datasets/edinburghcstr/ami | 
+| YODAS English           | ASR | 10000 |  https://huggingface.co/datasets/espnet/yodas |
+| YODAS Japanese           | ASR | 1400 |  https://huggingface.co/datasets/espnet/yodas |
+| Earnings-22 English            | ASR | 105 | https://huggingface.co/datasets/esb/datasets | 
+| Switchboard English     | ASR | 260 | https://catalog.ldc.upenn.edu/LDC97S62 |
+| CallHome English        | ASR | 18  | https://catalog.ldc.upenn.edu/LDC97T14 | 
+| Fisher English                 | ASR | 2000 | https://catalog.ldc.upenn.edu/LDC2004S13 | 
+| Voicemail part I English       | ASR | 40 | https://catalog.ldc.upenn.edu/LDC98S77 | 
+| Voicemail part II English      | ASR | 40 | https://catalog.ldc.upenn.edu/LDC2002S35 |
+| ReazonSpeech | ASR | 3000 | https://huggingface.co/datasets/reazon-research/reazonspeech |
+| Fineweb-2 TTS Japanese | ASR | 9600 | https://huggingface.co/datasets/HuggingFaceFW/fineweb-2 and Kokoro-82M TTS |
+| CommonVoice-17 De,Es,Fr,Pt->En          | AST | 3000  | Translations with Granite-3 and Phi-4 | 
+| CommonVoice-17 En->De,Es,Fr,It,Ja,Pt,Zh | AST | 18000 | Translations with Phi-4 and MADLAD |
+
+**Infrastructure:**
+We train Granite Speech using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable
+and efficient infrastructure for training our models over thousands of GPUs. The training of this particular model was completed in 30 days (26 encoder + 4 projector) on 8
+H100 GPUs.
+
+**Ethical Considerations and Limitations:**
+
+The use of Large Speech and Language Models can trigger certain risks and ethical considerations. Although our alignment processes include safety considerations, 
+the model may in some cases produce inaccurate, biased, offensive or unwanted responses to user prompts. Additionally, whether smaller models may exhibit increased 
+susceptibility to hallucination in generation scenarios due to their reduced sizes, which could limit their ability to generate coherent and contextually accurate responses, remains uncertain.
+ This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain.
+
+IBM recommends using this model for automatic speech recognition and translation tasks. The model's design improves safety by limiting how audio inputs can influence the system. 
+If an unfamiliar or malformed prompt is received, the model simply ignores it and performs transcription which is the default fallback mode. 
+This minimizes the risk of adversarial inputs, unlike integrated models that directly interpret audio and may be more exposed to such attacks. Note that more general speech tasks may pose higher inherent risks of triggering unwanted outputs. 
+
+To enhance safety, we recommend using granite-4.0-1b-speech alongside Granite Guardian. Granite Guardian is a fine-tuned instruct model designed to detect and flag risks in prompts and responses across key dimensions outlined in the IBM AI Risk Atlas.
+
+**Resources**
+- 📄 Read the technical report: https://arxiv.org/abs/2505.08699
+- 🔧 Notebook: [Finetune on custom data](https://github.com/ibm-granite/granite-speech-models/blob/main/notebooks/fine_tuning_granite_speech.ipynb)
+- ⭐️ Learn about the latest updates with Granite: https://www.ibm.com/granite
+- 🚀 Get started with tutorials, best practices, and prompt engineering advice: https://www.ibm.com/granite/docs/
+- 💡 Learn about the latest Granite learning resources: https://ibm.biz/granite-learning-resources

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<|audio|>": 100352
+}

chat_template.jinja

@@ -0,0 +1,2 @@
+{% for message in messages %}{% if message['role'] == 'user' %}USER: {{ message['content'] }}
+ ASSISTANT:{% elif message['role'] == 'assistant' %}{{ message['content'] }}{% endif %}{% endfor %}

config.json

@@ -0,0 +1,86 @@
+{
+  "architectures": [
+    "GraniteSpeechForConditionalGeneration"
+  ],
+  "audio_token_index": 100352,
+  "downsample_rate": 5,
+  "encoder_config": {
+    "context_size": 200,
+    "conv_expansion_factor": 2,
+    "conv_kernel_size": 15,
+    "dim_head": 128,
+    "dropout": 0.1,
+    "feedforward_mult": 4,
+    "hidden_dim": 1024,
+    "input_dim": 160,
+    "max_pos_emb": 512,
+    "model_type": "granite_speech_encoder",
+    "num_heads": 8,
+    "num_layers": 16,
+    "output_dim": 348,
+    "torch_dtype": "bfloat16"
+  },
+  "has_lora_adapter": false,
+  "initializer_range": 0.02,
+  "model_type": "granite_speech",
+  "projector_config": {
+    "_attn_implementation_autoset": true,
+    "attention_probs_dropout_prob": 0.1,
+    "cross_attention_frequency": 1,
+    "encoder_hidden_size": 1024,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "layer_norm_eps": 1e-12,
+    "max_position_embeddings": 2048,
+    "model_type": "blip_2_qformer",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 2,
+    "position_embedding_type": "absolute",
+    "torch_dtype": "bfloat16",
+    "use_qformer_text_input": false,
+    "vocab_size": 30522
+  },
+  "text_config": {
+    "_name_or_path": "/proj/speech/saon/slam-llm/29.2-c/granite-4.0-1b-base",
+    "architectures": [
+      "GraniteForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attention_multiplier": 0.0078125,
+    "bos_token_id": 100257,
+    "dtype": "float32",
+    "embedding_multiplier": 12,
+    "eos_token_id": 100257,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.1,
+    "intermediate_size": 4096,
+    "logits_scaling": 8,
+    "max_position_embeddings": 4096,
+    "mlp_bias": false,
+    "model_type": "granite",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 40,
+    "num_key_value_heads": 4,
+    "pad_token_id": 100256,
+    "residual_multiplier": 0.22,
+    "rms_norm_eps": 1e-05,
+    "rope_parameters": {
+      "rope_theta": 10000,
+      "rope_type": "default"
+    },
+    "rope_scaling": null,
+    "rope_theta": 10000.0,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "vocab_size": 100353
+  },
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.54.0",
+  "window_size": 15
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ed1c8a94a3ea0bebc3faa93490cc1b5543ed0d33e345c26f4b0ca878ad12f8e
+size 2143518808

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cbc16c9712d1174fa77dcfba36b6bfa32961ec6c713d9494fce8d61b657809be
+size 2143963456

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f9ea0edb2847633edcdf257c8d5d3cd8be751cab2cefb039452ec689fe7f6fc
+size 339045512

model.safetensors.index.json

@@ -0,0 +1,961 @@
+{
+  "metadata": {
+    "total_size": 4626414392
+  },
+  "weight_map": {
+    "language_model.model.embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.10.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.11.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.12.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.13.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.14.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.15.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.16.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.17.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.18.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.19.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.20.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.21.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.22.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.23.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.24.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "language_model.model.layers.24.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.24.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.24.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.24.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.24.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.25.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.26.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.27.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.28.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.29.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.30.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.31.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.32.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.33.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.34.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.35.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.36.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.37.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.38.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.layers.39.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "language_model.model.norm.weight": "model-00002-of-00003.safetensors",
+    "language_model.lm_head.weight": "model-00002-of-00003.safetensors",
+    "encoder.input_linear.weight": "model-00002-of-00003.safetensors",
+    "encoder.input_linear.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.0.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.1.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.2.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.3.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.4.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.5.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.6.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.7.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.8.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.9.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff1.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff1.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.attn.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.attn.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.attn.to_q.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.attn.to_kv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.attn.to_out.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.attn.to_out.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.attn.rel_pos_emb.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.up_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.up_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.depth_conv.conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.batch_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.batch_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.batch_norm.running_mean": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.batch_norm.running_var": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.batch_norm.num_batches_tracked": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.down_conv.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.conv.down_conv.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff2.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff2.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff2.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff2.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff2.down_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.ff2.down_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.post_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.10.post_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.11.ff1.pre_norm.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.11.ff1.pre_norm.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.11.ff1.up_proj.weight": "model-00002-of-00003.safetensors",
+    "encoder.layers.11.ff1.up_proj.bias": "model-00002-of-00003.safetensors",
+    "encoder.layers.11.ff1.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.ff1.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.attn.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.attn.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.attn.to_q.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.attn.to_kv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.attn.to_out.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.attn.to_out.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.attn.rel_pos_emb.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.up_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.up_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.depth_conv.conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.batch_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.batch_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.batch_norm.running_mean": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.batch_norm.running_var": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.batch_norm.num_batches_tracked": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.down_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.conv.down_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.ff2.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.ff2.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.ff2.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.ff2.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.ff2.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.ff2.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.post_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.11.post_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff1.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff1.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff1.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff1.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff1.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff1.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.attn.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.attn.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.attn.to_q.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.attn.to_kv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.attn.to_out.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.attn.to_out.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.attn.rel_pos_emb.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.up_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.up_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.depth_conv.conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.batch_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.batch_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.batch_norm.running_mean": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.batch_norm.running_var": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.batch_norm.num_batches_tracked": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.down_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.conv.down_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff2.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff2.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff2.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff2.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff2.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.ff2.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.post_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.12.post_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff1.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff1.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff1.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff1.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff1.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff1.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.attn.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.attn.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.attn.to_q.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.attn.to_kv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.attn.to_out.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.attn.to_out.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.attn.rel_pos_emb.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.up_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.up_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.depth_conv.conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.batch_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.batch_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.batch_norm.running_mean": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.batch_norm.running_var": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.batch_norm.num_batches_tracked": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.down_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.conv.down_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff2.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff2.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff2.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff2.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff2.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.ff2.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.post_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.13.post_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff1.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff1.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff1.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff1.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff1.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff1.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.attn.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.attn.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.attn.to_q.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.attn.to_kv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.attn.to_out.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.attn.to_out.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.attn.rel_pos_emb.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.up_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.up_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.depth_conv.conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.batch_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.batch_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.batch_norm.running_mean": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.batch_norm.running_var": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.batch_norm.num_batches_tracked": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.down_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.conv.down_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff2.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff2.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff2.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff2.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff2.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.ff2.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.post_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.14.post_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff1.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff1.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff1.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff1.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff1.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff1.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.attn.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.attn.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.attn.to_q.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.attn.to_kv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.attn.to_out.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.attn.to_out.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.attn.rel_pos_emb.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.up_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.up_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.depth_conv.conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.batch_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.batch_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.batch_norm.running_mean": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.batch_norm.running_var": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.batch_norm.num_batches_tracked": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.down_conv.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.conv.down_conv.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff2.pre_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff2.pre_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff2.up_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff2.up_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff2.down_proj.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.ff2.down_proj.bias": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.post_norm.weight": "model-00003-of-00003.safetensors",
+    "encoder.layers.15.post_norm.bias": "model-00003-of-00003.safetensors",
+    "encoder.out.weight": "model-00003-of-00003.safetensors",
+    "encoder.out.bias": "model-00003-of-00003.safetensors",
+    "encoder.out_mid.weight": "model-00003-of-00003.safetensors",
+    "encoder.out_mid.bias": "model-00003-of-00003.safetensors",
+    "projector.query": "model-00003-of-00003.safetensors",
+    "projector.qformer.layernorm.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.layernorm.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.attention.query.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.attention.query.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.attention.key.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.attention.key.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.attention.value.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.attention.value.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.output.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.output.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.output.LayerNorm.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.attention.output.LayerNorm.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.attention.query.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.attention.query.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.attention.key.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.attention.key.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.attention.value.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.attention.value.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.output.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.output.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.output.LayerNorm.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.crossattention.output.LayerNorm.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.intermediate_query.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.intermediate_query.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.output_query.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.output_query.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.output_query.LayerNorm.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.0.output_query.LayerNorm.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.attention.query.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.attention.query.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.attention.key.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.attention.key.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.attention.value.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.attention.value.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.output.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.output.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.output.LayerNorm.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.attention.output.LayerNorm.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.attention.query.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.attention.query.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.attention.key.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.attention.key.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.attention.value.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.attention.value.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.output.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.output.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.output.LayerNorm.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.crossattention.output.LayerNorm.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.intermediate_query.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.intermediate_query.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.output_query.dense.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.output_query.dense.bias": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.output_query.LayerNorm.weight": "model-00003-of-00003.safetensors",
+    "projector.qformer.encoder.layer.1.output_query.LayerNorm.bias": "model-00003-of-00003.safetensors",
+    "projector.linear.weight": "model-00003-of-00003.safetensors",
+    "projector.linear.bias": "model-00003-of-00003.safetensors"
+  }
+}

multilingual_sample.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91d243650809c1274141ec20ff23045315eaf27567694002ea3ef390048b7058
+size 1596240

preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "feature_extractor_type": "GraniteSpeechFeatureExtractor",
+  "melspec_kwargs": {
+    "hop_length": 160,
+    "n_fft": 512,
+    "n_mels": 80,
+    "sample_rate": 16000,
+    "win_length": 400
+  },
+  "processor_class": "GraniteSpeechProcessor",
+  "projector_downsample_rate": 5,
+  "projector_window_size": 15,
+  "sampling_rate": 16000
+}

processor_config.json

@@ -0,0 +1,4 @@
+{
+  "audio_token": "<|audio|>",
+  "processor_class": "GraniteSpeechProcessor"
+}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<|end_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|end_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|unk|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,792 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "100256": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100257": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100258": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100259": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100260": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100261": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100262": {
+      "content": "<|filename|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100263": {
+      "content": "<|reponame|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100264": {
+      "content": "<|start_of_role|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100265": {
+      "content": "<|end_of_role|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100266": {
+      "content": "<|unused_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100267": {
+      "content": "<|start_of_plugin|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100268": {
+      "content": "<|end_of_plugin|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100269": {
+      "content": "<|unk|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100270": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100271": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100272": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100273": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100274": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100275": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100276": {
+      "content": "<think_on>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100277": {
+      "content": "<think_off>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100278": {
+      "content": "<schema>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100279": {
+      "content": "</schema>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100280": {
+      "content": "<tools>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100281": {
+      "content": "</tools>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100282": {
+      "content": "<documents>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100283": {
+      "content": "</documents>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100284": {
+      "content": "<|unused_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100285": {
+      "content": "<|unused_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100286": {
+      "content": "<|unused_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100287": {
+      "content": "<|unused_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100288": {
+      "content": "<|unused_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100289": {
+      "content": "<|unused_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100290": {
+      "content": "<|unused_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100291": {
+      "content": "<|unused_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100292": {
+      "content": "<|unused_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100293": {
+      "content": "<|unused_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100294": {
+      "content": "<|unused_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100295": {
+      "content": "<|unused_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100296": {
+      "content": "<|unused_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100297": {
+      "content": "<|unused_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100298": {
+      "content": "<|unused_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100299": {
+      "content": "<|unused_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100300": {
+      "content": "<|unused_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100301": {
+      "content": "<|unused_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100302": {
+      "content": "<|unused_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100303": {
+      "content": "<|unused_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100304": {
+      "content": "<|unused_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100305": {
+      "content": "<|unused_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100306": {
+      "content": "<|unused_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100307": {
+      "content": "<|unused_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100308": {
+      "content": "<|unused_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100309": {
+      "content": "<|unused_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100310": {
+      "content": "<|unused_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100311": {
+      "content": "<|unused_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100312": {
+      "content": "<|unused_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100313": {
+      "content": "<|unused_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100314": {
+      "content": "<|unused_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100315": {
+      "content": "<|unused_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100316": {
+      "content": "<|unused_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100317": {
+      "content": "<|unused_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100318": {
+      "content": "<|unused_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100319": {
+      "content": "<|unused_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100320": {
+      "content": "<|unused_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100321": {
+      "content": "<|unused_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100322": {
+      "content": "<|unused_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100323": {
+      "content": "<|unused_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100324": {
+      "content": "<|unused_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100325": {
+      "content": "<|unused_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100326": {
+      "content": "<|unused_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100327": {
+      "content": "<|unused_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100328": {
+      "content": "<|unused_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100329": {
+      "content": "<|unused_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100330": {
+      "content": "<|unused_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100331": {
+      "content": "<|unused_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100332": {
+      "content": "<|unused_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100333": {
+      "content": "<|unused_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100334": {
+      "content": "<|unused_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100335": {
+      "content": "<|unused_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100336": {
+      "content": "<|unused_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100337": {
+      "content": "<|unused_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100338": {
+      "content": "<|unused_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100339": {
+      "content": "<|unused_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100340": {
+      "content": "<|unused_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100341": {
+      "content": "<|unused_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100342": {
+      "content": "<|unused_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100343": {
+      "content": "<|unused_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100344": {
+      "content": "<|unused_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100345": {
+      "content": "<|unused_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100346": {
+      "content": "<|unused_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100347": {
+      "content": "<|unused_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100348": {
+      "content": "<|unused_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100349": {
+      "content": "<|unused_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100350": {
+      "content": "<|unused_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100351": {
+      "content": "<|unused_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100352": {
+      "content": "<|audio|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|end_of_text|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|end_of_text|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "padding_side": "left",
+  "processor_class": "GraniteSpeechProcessor",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|unk|>"
+}