Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-2307/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,70 +1,138 @@
 ---
+library_name: transformers
 license: apache-2.0
+base_model: Qwen/Qwen3-8B
+tags:
+- generated_from_trainer
+datasets:
+- WNT3D/Ultimate-Offensive-Red-Team
+model-index:
+- name: workspace/output/killchain-8b
+  results: []
 ---
-# KillChain-8B
 
-**KillChain-8B** is a fully fine-tuned 8B-parameter language model designed for offensive security reasoning, adversarial analysis, and red-team tests. 
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
----
+[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
+<details><summary>See axolotl config</summary>
 
-## Model Details
+axolotl version: `0.13.0.dev0`
+```yaml
+base_model: Qwen/Qwen3-8B
+model_type: Qwen3ForCausalLM
+tokenizer_type: AutoTokenizer
+trust_remote_code: true
 
-- **Base model:** Qwen3-8B  
-- **Parameters:** 8 billion  
-- **Training type:** Full fine-tune  
-- **Precision:** bfloat16  
-- **Context length:** 4096 tokens  
-- **Architecture:** Decoder-only Transformer 
+datasets:
+  - path: WNT3D/Ultimate-Offensive-Red-Team
+    type: alpaca
 
----
+output_dir: /workspace/output/killchain-8b
+val_set_size: 0.02
 
-## Training Data
+sequence_len: 4096
 
-KillChain-8B was trained on **WNT3D/Ultimate-Offensive-Red-Team**
+special_tokens:
+  pad_token: "<|pad|>"
 
-The model is optimized for **direct, procedural answers** rather than verbose alignment-heavy responses.
+pad_to_max_length: true
 
----
+bf16: true
+fp16: false
+dtype: bfloat16
+torch_dtype: bfloat16
 
-## Intended Use
+use_cache: false
+attn_implementation: flash_attention_2
 
-KillChain-8B is intended for:
+gradient_checkpointing: true
+gradient_checkpointing_kwargs:
+  use_reentrant: false
 
-- Red-team simulation and research  
-- Security training and tabletop exercises  
-- Adversarial LLM evaluation  
-- Controlled internal testing environments  
-- Studying failure modes of aligned models  
+micro_batch_size: 4
+gradient_accumulation_steps: 2
+num_epochs: 3
+learning_rate: 1.5e-5
 
-It is **not intended for production deployment** in unmoderated environments.
+optimizer: adamw_torch
+lr_scheduler: cosine
+warmup_steps: 200
+weight_decay: 0.1
 
----
+logging_steps: 10
+save_steps: 0
+save_total_limit: 1
+save_only_model: true
+
+dataloader_num_workers: 4
+dataloader_pin_memory: true
+dataset_processes: 4
+
+use_vllm: false
+
+deepspeed: |
+  {
+    "train_micro_batch_size_per_gpu": 4,
+    "gradient_accumulation_steps": 2,
+    "zero_optimization": {
+      "stage": 2,
+      "overlap_comm": true,
+      "contiguous_gradients": true
+    },
+    "bf16": {
+      "enabled": true
+    }
+  }
+
+wandb_mode: disabled
+
+```
+
+</details><br>
+
+# workspace/output/killchain-8b
+
+This model is a fine-tuned version of [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) on the WNT3D/Ultimate-Offensive-Red-Team dataset.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
 
-## Usage (Transformers)
+## Training procedure
 
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-import torch
+### Training hyperparameters
 
-model_id = "MrPibb/KillChain-8B"
+The following hyperparameters were used during training:
+- learning_rate: 1.5e-05
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 32
+- total_eval_batch_size: 16
+- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 200
+- training_steps: 2307
 
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    torch_dtype=torch.bfloat16,
-    device_map="auto"
-)
+### Training results
 
-prompt = "User: YOUR-PROMPT-HERE\nAssistant:"
 
-inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
 
-outputs = model.generate(
-    **inputs,
-    max_new_tokens=300,
-    temperature=0.7,
-    top_p=0.9,
-    do_sample=True
-)
+### Framework versions
 
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+- Transformers 4.57.0
+- Pytorch 2.7.1+cu126
+- Datasets 4.0.0
+- Tokenizers 0.22.1

added_tokens.json

@@ -0,0 +1,29 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|pad|>": 151669,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,89 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

checkpoint-2307/added_tokens.json

@@ -0,0 +1,29 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|pad|>": 151669,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

checkpoint-2307/chat_template.jinja

@@ -0,0 +1,89 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

checkpoint-2307/config.json

@@ -0,0 +1,68 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 40960,
+  "max_window_layers": 36,
+  "model_type": "qwen3",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151669,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.0",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

checkpoint-2307/generation_config.json

@@ -0,0 +1,12 @@
+{
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151669,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "4.57.0"
+}

checkpoint-2307/model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a0e50d3dd155ae4a64df2f2ccfcb4848cfe87877a5ec4792233ea6db2b67378
+size 4902257696

checkpoint-2307/model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6bf727322666823665b53e2694441c8a993cf9156578d20465f9a1ab88cfd22f
+size 4915960368

checkpoint-2307/model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d78bd156b3d080877d255126677071025cd8e44d24c637c71b274d5017de991
+size 4983068496

checkpoint-2307/model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f68e1dab321833cbd7a77008f5ce9f4c2ee0eda6c01ff88b44d421cba5e9d1ac
+size 1580230264

checkpoint-2307/model.safetensors.index.json

@@ -0,0 +1,407 @@
+{
+  "metadata": {
+    "total_parameters": 8190735360,
+    "total_size": 16381470720
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.norm.weight": "model-00004-of-00004.safetensors"
+  }
+}

checkpoint-2307/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-2307/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:742070173edd1d9fbf1a6c9135828398c90076ffdd417d7fd4274806e9ddbdc5
+size 11422838

checkpoint-2307/tokenizer_config.json

@@ -0,0 +1,247 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|pad|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-2307/trainer_state.json

@@ -0,0 +1,2564 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.0,
+  "eval_steps": 500,
+  "global_step": 2307,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.013003901170351105,
+      "grad_norm": 18.055130004882812,
+      "learning_rate": 6.75e-07,
+      "loss": 1.586,
+      "memory/device_reserved (GiB)": 76.31,
+      "memory/max_active (GiB)": 60.0,
+      "memory/max_allocated (GiB)": 60.0,
+      "step": 10,
+      "tokens_per_second_per_gpu": 1713.39
+    },
+    {
+      "epoch": 0.02600780234070221,
+      "grad_norm": 8.317655563354492,
+      "learning_rate": 1.4250000000000001e-06,
+      "loss": 1.4237,
+      "memory/device_reserved (GiB)": 76.31,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 20,
+      "tokens_per_second_per_gpu": 727.09
+    },
+    {
+      "epoch": 0.03901170351105332,
+      "grad_norm": 3.7881243228912354,
+      "learning_rate": 2.175e-06,
+      "loss": 1.2268,
+      "memory/device_reserved (GiB)": 76.31,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 30,
+      "tokens_per_second_per_gpu": 846.64
+    },
+    {
+      "epoch": 0.05201560468140442,
+      "grad_norm": 2.4919686317443848,
+      "learning_rate": 2.9250000000000004e-06,
+      "loss": 1.1015,
+      "memory/device_reserved (GiB)": 76.31,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 40,
+      "tokens_per_second_per_gpu": 982.13
+    },
+    {
+      "epoch": 0.06501950585175553,
+      "grad_norm": 2.520327091217041,
+      "learning_rate": 3.675e-06,
+      "loss": 0.9761,
+      "memory/device_reserved (GiB)": 76.31,
+      "memory/max_active (GiB)": 61.54,
+      "memory/max_allocated (GiB)": 61.54,
+      "step": 50,
+      "tokens_per_second_per_gpu": 1598.18
+    },
+    {
+      "epoch": 0.07802340702210664,
+      "grad_norm": 1.5129573345184326,
+      "learning_rate": 4.425e-06,
+      "loss": 0.8437,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 60,
+      "tokens_per_second_per_gpu": 816.84
+    },
+    {
+      "epoch": 0.09102730819245773,
+      "grad_norm": 2.5336833000183105,
+      "learning_rate": 5.175e-06,
+      "loss": 0.8493,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 68.98,
+      "memory/max_allocated (GiB)": 68.98,
+      "step": 70,
+      "tokens_per_second_per_gpu": 1306.95
+    },
+    {
+      "epoch": 0.10403120936280884,
+      "grad_norm": 1.7790873050689697,
+      "learning_rate": 5.925e-06,
+      "loss": 0.6868,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 80,
+      "tokens_per_second_per_gpu": 1413.17
+    },
+    {
+      "epoch": 0.11703511053315994,
+      "grad_norm": 2.209254741668701,
+      "learning_rate": 6.6750000000000005e-06,
+      "loss": 0.6055,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 70.26,
+      "memory/max_allocated (GiB)": 70.26,
+      "step": 90,
+      "tokens_per_second_per_gpu": 1671.15
+    },
+    {
+      "epoch": 0.13003901170351106,
+      "grad_norm": 2.414102792739868,
+      "learning_rate": 7.425e-06,
+      "loss": 0.5825,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 100,
+      "tokens_per_second_per_gpu": 889.74
+    },
+    {
+      "epoch": 0.14304291287386217,
+      "grad_norm": 2.6992218494415283,
+      "learning_rate": 8.175e-06,
+      "loss": 0.482,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 110,
+      "tokens_per_second_per_gpu": 1514.6
+    },
+    {
+      "epoch": 0.15604681404421328,
+      "grad_norm": 2.1195459365844727,
+      "learning_rate": 8.925e-06,
+      "loss": 0.386,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 120,
+      "tokens_per_second_per_gpu": 1705.08
+    },
+    {
+      "epoch": 0.16905071521456436,
+      "grad_norm": 1.8070746660232544,
+      "learning_rate": 9.675e-06,
+      "loss": 0.3142,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 66.67,
+      "memory/max_allocated (GiB)": 66.67,
+      "step": 130,
+      "tokens_per_second_per_gpu": 852.0
+    },
+    {
+      "epoch": 0.18205461638491546,
+      "grad_norm": 2.118978500366211,
+      "learning_rate": 1.0425e-05,
+      "loss": 0.3271,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 62.06,
+      "memory/max_allocated (GiB)": 62.06,
+      "step": 140,
+      "tokens_per_second_per_gpu": 1406.95
+    },
+    {
+      "epoch": 0.19505851755526657,
+      "grad_norm": 1.990539312362671,
+      "learning_rate": 1.1175e-05,
+      "loss": 0.2984,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 57.7,
+      "memory/max_allocated (GiB)": 57.7,
+      "step": 150,
+      "tokens_per_second_per_gpu": 1398.32
+    },
+    {
+      "epoch": 0.20806241872561768,
+      "grad_norm": 2.2458865642547607,
+      "learning_rate": 1.1925e-05,
+      "loss": 0.2875,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 69.49,
+      "memory/max_allocated (GiB)": 69.49,
+      "step": 160,
+      "tokens_per_second_per_gpu": 1778.31
+    },
+    {
+      "epoch": 0.22106631989596878,
+      "grad_norm": 1.514204978942871,
+      "learning_rate": 1.2675e-05,
+      "loss": 0.2051,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 170,
+      "tokens_per_second_per_gpu": 586.78
+    },
+    {
+      "epoch": 0.2340702210663199,
+      "grad_norm": 1.917752981185913,
+      "learning_rate": 1.3425000000000001e-05,
+      "loss": 0.1841,
+      "memory/device_reserved (GiB)": 102.83,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 180,
+      "tokens_per_second_per_gpu": 1232.33
+    },
+    {
+      "epoch": 0.247074122236671,
+      "grad_norm": 2.3047068119049072,
+      "learning_rate": 1.4174999999999999e-05,
+      "loss": 0.1787,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.52,
+      "memory/max_allocated (GiB)": 70.52,
+      "step": 190,
+      "tokens_per_second_per_gpu": 1976.94
+    },
+    {
+      "epoch": 0.26007802340702213,
+      "grad_norm": 1.230846881866455,
+      "learning_rate": 1.4925e-05,
+      "loss": 0.1425,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 200,
+      "tokens_per_second_per_gpu": 953.32
+    },
+    {
+      "epoch": 0.27308192457737324,
+      "grad_norm": 1.6984196901321411,
+      "learning_rate": 1.4999324725354489e-05,
+      "loss": 0.1969,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 210,
+      "tokens_per_second_per_gpu": 959.11
+    },
+    {
+      "epoch": 0.28608582574772434,
+      "grad_norm": 1.736661672592163,
+      "learning_rate": 1.4996990598743452e-05,
+      "loss": 0.1666,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 72.83,
+      "memory/max_allocated (GiB)": 72.83,
+      "step": 220,
+      "tokens_per_second_per_gpu": 1803.69
+    },
+    {
+      "epoch": 0.29908972691807545,
+      "grad_norm": 1.1661016941070557,
+      "learning_rate": 1.499298980222333e-05,
+      "loss": 0.1081,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.24,
+      "memory/max_allocated (GiB)": 69.24,
+      "step": 230,
+      "tokens_per_second_per_gpu": 1251.7
+    },
+    {
+      "epoch": 0.31209362808842656,
+      "grad_norm": 1.5299252271652222,
+      "learning_rate": 1.498732322521864e-05,
+      "loss": 0.1103,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.03,
+      "memory/max_allocated (GiB)": 61.03,
+      "step": 240,
+      "tokens_per_second_per_gpu": 1395.28
+    },
+    {
+      "epoch": 0.3250975292587776,
+      "grad_norm": 1.9430108070373535,
+      "learning_rate": 1.4979992127476638e-05,
+      "loss": 0.0878,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.41,
+      "memory/max_allocated (GiB)": 66.41,
+      "step": 250,
+      "tokens_per_second_per_gpu": 1878.03
+    },
+    {
+      "epoch": 0.3381014304291287,
+      "grad_norm": 1.274000644683838,
+      "learning_rate": 1.4970998138787283e-05,
+      "loss": 0.1109,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.24,
+      "memory/max_allocated (GiB)": 69.24,
+      "step": 260,
+      "tokens_per_second_per_gpu": 868.45
+    },
+    {
+      "epoch": 0.3511053315994798,
+      "grad_norm": 1.278598427772522,
+      "learning_rate": 1.4960343258620915e-05,
+      "loss": 0.0867,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 270,
+      "tokens_per_second_per_gpu": 1284.14
+    },
+    {
+      "epoch": 0.3641092327698309,
+      "grad_norm": 1.031064748764038,
+      "learning_rate": 1.4948029855683738e-05,
+      "loss": 0.0773,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 280,
+      "tokens_per_second_per_gpu": 839.07
+    },
+    {
+      "epoch": 0.37711313394018203,
+      "grad_norm": 1.0166915655136108,
+      "learning_rate": 1.4934060667391236e-05,
+      "loss": 0.0622,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.7,
+      "memory/max_allocated (GiB)": 57.7,
+      "step": 290,
+      "tokens_per_second_per_gpu": 1544.62
+    },
+    {
+      "epoch": 0.39011703511053314,
+      "grad_norm": 1.9382474422454834,
+      "learning_rate": 1.4918438799259611e-05,
+      "loss": 0.1001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 62.31,
+      "memory/max_allocated (GiB)": 62.31,
+      "step": 300,
+      "tokens_per_second_per_gpu": 1838.8
+    },
+    {
+      "epoch": 0.40312093628088425,
+      "grad_norm": 1.0125566720962524,
+      "learning_rate": 1.4901167724215398e-05,
+      "loss": 0.0567,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 310,
+      "tokens_per_second_per_gpu": 1133.84
+    },
+    {
+      "epoch": 0.41612483745123535,
+      "grad_norm": 0.916778564453125,
+      "learning_rate": 1.4882251281823373e-05,
+      "loss": 0.104,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 320,
+      "tokens_per_second_per_gpu": 1038.02
+    },
+    {
+      "epoch": 0.42912873862158646,
+      "grad_norm": 0.8244317173957825,
+      "learning_rate": 1.486169367743299e-05,
+      "loss": 0.0861,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.0,
+      "memory/max_allocated (GiB)": 70.0,
+      "step": 330,
+      "tokens_per_second_per_gpu": 1769.98
+    },
+    {
+      "epoch": 0.44213263979193757,
+      "grad_norm": 0.6626781821250916,
+      "learning_rate": 1.483949948124348e-05,
+      "loss": 0.0546,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.7,
+      "memory/max_allocated (GiB)": 57.7,
+      "step": 340,
+      "tokens_per_second_per_gpu": 1052.55
+    },
+    {
+      "epoch": 0.45513654096228867,
+      "grad_norm": 0.9873509407043457,
+      "learning_rate": 1.4815673627287834e-05,
+      "loss": 0.0604,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.49,
+      "memory/max_allocated (GiB)": 69.49,
+      "step": 350,
+      "tokens_per_second_per_gpu": 1117.34
+    },
+    {
+      "epoch": 0.4681404421326398,
+      "grad_norm": 0.8155633807182312,
+      "learning_rate": 1.479022141233591e-05,
+      "loss": 0.0573,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 360,
+      "tokens_per_second_per_gpu": 1186.86
+    },
+    {
+      "epoch": 0.4811443433029909,
+      "grad_norm": 1.3516284227371216,
+      "learning_rate": 1.47631484947169e-05,
+      "loss": 0.0606,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 370,
+      "tokens_per_second_per_gpu": 1765.69
+    },
+    {
+      "epoch": 0.494148244473342,
+      "grad_norm": 1.270824670791626,
+      "learning_rate": 1.4734460893061426e-05,
+      "loss": 0.0669,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 380,
+      "tokens_per_second_per_gpu": 1183.26
+    },
+    {
+      "epoch": 0.5071521456436932,
+      "grad_norm": 0.9764719605445862,
+      "learning_rate": 1.470416498496351e-05,
+      "loss": 0.0642,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 390,
+      "tokens_per_second_per_gpu": 958.72
+    },
+    {
+      "epoch": 0.5201560468140443,
+      "grad_norm": 1.8843528032302856,
+      "learning_rate": 1.4672267505562768e-05,
+      "loss": 0.076,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 400,
+      "tokens_per_second_per_gpu": 446.12
+    },
+    {
+      "epoch": 0.5331599479843954,
+      "grad_norm": 1.1684187650680542,
+      "learning_rate": 1.4638775546047112e-05,
+      "loss": 0.0447,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 410,
+      "tokens_per_second_per_gpu": 1057.78
+    },
+    {
+      "epoch": 0.5461638491547465,
+      "grad_norm": 1.120720386505127,
+      "learning_rate": 1.460369655207629e-05,
+      "loss": 0.032,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 420,
+      "tokens_per_second_per_gpu": 1416.03
+    },
+    {
+      "epoch": 0.5591677503250976,
+      "grad_norm": 1.1288883686065674,
+      "learning_rate": 1.456703832212662e-05,
+      "loss": 0.0515,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 430,
+      "tokens_per_second_per_gpu": 1025.99
+    },
+    {
+      "epoch": 0.5721716514954487,
+      "grad_norm": 0.9152341485023499,
+      "learning_rate": 1.4528809005757319e-05,
+      "loss": 0.0537,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.03,
+      "memory/max_allocated (GiB)": 61.03,
+      "step": 440,
+      "tokens_per_second_per_gpu": 1706.17
+    },
+    {
+      "epoch": 0.5851755526657998,
+      "grad_norm": 1.6503143310546875,
+      "learning_rate": 1.4489017101798733e-05,
+      "loss": 0.0459,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 450,
+      "tokens_per_second_per_gpu": 800.3
+    },
+    {
+      "epoch": 0.5981794538361509,
+      "grad_norm": 0.6420057415962219,
+      "learning_rate": 1.4447671456462965e-05,
+      "loss": 0.0286,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 74.37,
+      "memory/max_allocated (GiB)": 74.37,
+      "step": 460,
+      "tokens_per_second_per_gpu": 1047.49
+    },
+    {
+      "epoch": 0.611183355006502,
+      "grad_norm": 1.0113235712051392,
+      "learning_rate": 1.4404781261377248e-05,
+      "loss": 0.0371,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 470,
+      "tokens_per_second_per_gpu": 1743.76
+    },
+    {
+      "epoch": 0.6241872561768531,
+      "grad_norm": 0.8275484442710876,
+      "learning_rate": 1.4360356051540537e-05,
+      "loss": 0.0205,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 74.37,
+      "memory/max_allocated (GiB)": 74.37,
+      "step": 480,
+      "tokens_per_second_per_gpu": 1475.35
+    },
+    {
+      "epoch": 0.6371911573472041,
+      "grad_norm": 0.3329998850822449,
+      "learning_rate": 1.4314405703203773e-05,
+      "loss": 0.0274,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 490,
+      "tokens_per_second_per_gpu": 1091.43
+    },
+    {
+      "epoch": 0.6501950585175552,
+      "grad_norm": 0.7145099639892578,
+      "learning_rate": 1.4266940431674268e-05,
+      "loss": 0.0287,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 500,
+      "tokens_per_second_per_gpu": 1353.11
+    },
+    {
+      "epoch": 0.6631989596879063,
+      "grad_norm": 1.0522738695144653,
+      "learning_rate": 1.4217970789044722e-05,
+      "loss": 0.0414,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.41,
+      "memory/max_allocated (GiB)": 66.41,
+      "step": 510,
+      "tokens_per_second_per_gpu": 2785.07
+    },
+    {
+      "epoch": 0.6762028608582574,
+      "grad_norm": 0.4044458270072937,
+      "learning_rate": 1.4167507661847376e-05,
+      "loss": 0.0268,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 520,
+      "tokens_per_second_per_gpu": 1056.34
+    },
+    {
+      "epoch": 0.6892067620286085,
+      "grad_norm": 0.4427281320095062,
+      "learning_rate": 1.411556226863379e-05,
+      "loss": 0.0222,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 59.49,
+      "memory/max_allocated (GiB)": 59.49,
+      "step": 530,
+      "tokens_per_second_per_gpu": 1147.51
+    },
+    {
+      "epoch": 0.7022106631989596,
+      "grad_norm": 0.6132713556289673,
+      "learning_rate": 1.4062146157480848e-05,
+      "loss": 0.0176,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 72.83,
+      "memory/max_allocated (GiB)": 72.83,
+      "step": 540,
+      "tokens_per_second_per_gpu": 932.9
+    },
+    {
+      "epoch": 0.7152145643693107,
+      "grad_norm": 0.7523512840270996,
+      "learning_rate": 1.400727120342346e-05,
+      "loss": 0.0182,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.85,
+      "memory/max_allocated (GiB)": 73.85,
+      "step": 550,
+      "tokens_per_second_per_gpu": 1988.07
+    },
+    {
+      "epoch": 0.7282184655396619,
+      "grad_norm": 0.4098919928073883,
+      "learning_rate": 1.3950949605814618e-05,
+      "loss": 0.0253,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 560,
+      "tokens_per_second_per_gpu": 1243.37
+    },
+    {
+      "epoch": 0.741222366710013,
+      "grad_norm": 1.3029199838638306,
+      "learning_rate": 1.3893193885613331e-05,
+      "loss": 0.0252,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 570,
+      "tokens_per_second_per_gpu": 1006.76
+    },
+    {
+      "epoch": 0.7542262678803641,
+      "grad_norm": 0.4156447649002075,
+      "learning_rate": 1.3834016882601066e-05,
+      "loss": 0.0187,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 580,
+      "tokens_per_second_per_gpu": 880.36
+    },
+    {
+      "epoch": 0.7672301690507152,
+      "grad_norm": 0.6229768395423889,
+      "learning_rate": 1.3773431752527307e-05,
+      "loss": 0.0157,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 590,
+      "tokens_per_second_per_gpu": 1206.34
+    },
+    {
+      "epoch": 0.7802340702210663,
+      "grad_norm": 0.313753604888916,
+      "learning_rate": 1.3711451964184888e-05,
+      "loss": 0.0142,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 600,
+      "tokens_per_second_per_gpu": 673.09
+    },
+    {
+      "epoch": 0.7932379713914174,
+      "grad_norm": 0.644559919834137,
+      "learning_rate": 1.364809129641571e-05,
+      "loss": 0.0175,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 72.83,
+      "memory/max_allocated (GiB)": 72.83,
+      "step": 610,
+      "tokens_per_second_per_gpu": 1437.93
+    },
+    {
+      "epoch": 0.8062418725617685,
+      "grad_norm": 0.5777751803398132,
+      "learning_rate": 1.3583363835047546e-05,
+      "loss": 0.024,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.67,
+      "memory/max_allocated (GiB)": 66.67,
+      "step": 620,
+      "tokens_per_second_per_gpu": 1550.77
+    },
+    {
+      "epoch": 0.8192457737321196,
+      "grad_norm": 0.6310973763465881,
+      "learning_rate": 1.3517283969762596e-05,
+      "loss": 0.0154,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 630,
+      "tokens_per_second_per_gpu": 1006.32
+    },
+    {
+      "epoch": 0.8322496749024707,
+      "grad_norm": 0.27832910418510437,
+      "learning_rate": 1.3449866390898482e-05,
+      "loss": 0.0152,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 640,
+      "tokens_per_second_per_gpu": 958.55
+    },
+    {
+      "epoch": 0.8452535760728218,
+      "grad_norm": 0.462228924036026,
+      "learning_rate": 1.338112608618242e-05,
+      "loss": 0.0133,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 650,
+      "tokens_per_second_per_gpu": 616.74
+    },
+    {
+      "epoch": 0.8582574772431729,
+      "grad_norm": 0.49110764265060425,
+      "learning_rate": 1.3311078337399268e-05,
+      "loss": 0.0206,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 58.98,
+      "memory/max_allocated (GiB)": 58.98,
+      "step": 660,
+      "tokens_per_second_per_gpu": 1801.78
+    },
+    {
+      "epoch": 0.871261378413524,
+      "grad_norm": 0.34749847650527954,
+      "learning_rate": 1.3239738716994207e-05,
+      "loss": 0.013,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.03,
+      "memory/max_allocated (GiB)": 61.03,
+      "step": 670,
+      "tokens_per_second_per_gpu": 1049.05
+    },
+    {
+      "epoch": 0.8842652795838751,
+      "grad_norm": 0.3185332119464874,
+      "learning_rate": 1.3167123084610791e-05,
+      "loss": 0.0138,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.26,
+      "memory/max_allocated (GiB)": 70.26,
+      "step": 680,
+      "tokens_per_second_per_gpu": 1676.24
+    },
+    {
+      "epoch": 0.8972691807542262,
+      "grad_norm": 0.24320140480995178,
+      "learning_rate": 1.3093247583565174e-05,
+      "loss": 0.0105,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.7,
+      "memory/max_allocated (GiB)": 57.7,
+      "step": 690,
+      "tokens_per_second_per_gpu": 1447.63
+    },
+    {
+      "epoch": 0.9102730819245773,
+      "grad_norm": 0.26917219161987305,
+      "learning_rate": 1.3018128637257249e-05,
+      "loss": 0.0133,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 700,
+      "tokens_per_second_per_gpu": 783.33
+    },
+    {
+      "epoch": 0.9232769830949284,
+      "grad_norm": 0.41597655415534973,
+      "learning_rate": 1.2941782945519538e-05,
+      "loss": 0.0106,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 710,
+      "tokens_per_second_per_gpu": 1312.73
+    },
+    {
+      "epoch": 0.9362808842652796,
+      "grad_norm": 0.3711985647678375,
+      "learning_rate": 1.286422748090462e-05,
+      "loss": 0.018,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 720,
+      "tokens_per_second_per_gpu": 913.4
+    },
+    {
+      "epoch": 0.9492847854356307,
+      "grad_norm": 0.25185373425483704,
+      "learning_rate": 1.2785479484911926e-05,
+      "loss": 0.0169,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.52,
+      "memory/max_allocated (GiB)": 70.52,
+      "step": 730,
+      "tokens_per_second_per_gpu": 1250.17
+    },
+    {
+      "epoch": 0.9622886866059818,
+      "grad_norm": 0.2842692732810974,
+      "learning_rate": 1.2705556464154755e-05,
+      "loss": 0.0115,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 740,
+      "tokens_per_second_per_gpu": 1405.1
+    },
+    {
+      "epoch": 0.9752925877763329,
+      "grad_norm": 0.24624235928058624,
+      "learning_rate": 1.2624476186468345e-05,
+      "loss": 0.0089,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 750,
+      "tokens_per_second_per_gpu": 1757.9
+    },
+    {
+      "epoch": 0.988296488946684,
+      "grad_norm": 0.41257742047309875,
+      "learning_rate": 1.2542256676959877e-05,
+      "loss": 0.0099,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 72.83,
+      "memory/max_allocated (GiB)": 72.83,
+      "step": 760,
+      "tokens_per_second_per_gpu": 1950.88
+    },
+    {
+      "epoch": 1.0013003901170352,
+      "grad_norm": 0.4114612936973572,
+      "learning_rate": 1.2458916214001264e-05,
+      "loss": 0.0119,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 770,
+      "tokens_per_second_per_gpu": 870.5
+    },
+    {
+      "epoch": 1.0143042912873863,
+      "grad_norm": 0.6687620878219604,
+      "learning_rate": 1.237447332516567e-05,
+      "loss": 0.027,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 780,
+      "tokens_per_second_per_gpu": 1161.58
+    },
+    {
+      "epoch": 1.0273081924577374,
+      "grad_norm": 1.0310016870498657,
+      "learning_rate": 1.2288946783108605e-05,
+      "loss": 0.0134,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 790,
+      "tokens_per_second_per_gpu": 1318.8
+    },
+    {
+      "epoch": 1.0403120936280885,
+      "grad_norm": 0.4445036053657532,
+      "learning_rate": 1.220235560139454e-05,
+      "loss": 0.0122,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 59.49,
+      "memory/max_allocated (GiB)": 59.49,
+      "step": 800,
+      "tokens_per_second_per_gpu": 1063.65
+    },
+    {
+      "epoch": 1.0533159947984396,
+      "grad_norm": 0.5034821033477783,
+      "learning_rate": 1.2114719030269959e-05,
+      "loss": 0.014,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.54,
+      "memory/max_allocated (GiB)": 61.54,
+      "step": 810,
+      "tokens_per_second_per_gpu": 1327.35
+    },
+    {
+      "epoch": 1.0663198959687907,
+      "grad_norm": 0.550111711025238,
+      "learning_rate": 1.2026056552383817e-05,
+      "loss": 0.0114,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.16,
+      "memory/max_allocated (GiB)": 66.16,
+      "step": 820,
+      "tokens_per_second_per_gpu": 1529.49
+    },
+    {
+      "epoch": 1.0793237971391418,
+      "grad_norm": 0.7069242596626282,
+      "learning_rate": 1.1936387878456305e-05,
+      "loss": 0.01,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 830,
+      "tokens_per_second_per_gpu": 1755.68
+    },
+    {
+      "epoch": 1.092327698309493,
+      "grad_norm": 0.6991313099861145,
+      "learning_rate": 1.1845732942896921e-05,
+      "loss": 0.0102,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 62.82,
+      "memory/max_allocated (GiB)": 62.82,
+      "step": 840,
+      "tokens_per_second_per_gpu": 1878.66
+    },
+    {
+      "epoch": 1.105331599479844,
+      "grad_norm": 0.41683048009872437,
+      "learning_rate": 1.1754111899372822e-05,
+      "loss": 0.0104,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 850,
+      "tokens_per_second_per_gpu": 1280.23
+    },
+    {
+      "epoch": 1.1183355006501952,
+      "grad_norm": 0.343485951423645,
+      "learning_rate": 1.1661545116328428e-05,
+      "loss": 0.0101,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 65.39,
+      "memory/max_allocated (GiB)": 65.39,
+      "step": 860,
+      "tokens_per_second_per_gpu": 2377.41
+    },
+    {
+      "epoch": 1.1313394018205463,
+      "grad_norm": 0.44513949751853943,
+      "learning_rate": 1.1568053172457255e-05,
+      "loss": 0.0077,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.41,
+      "memory/max_allocated (GiB)": 66.41,
+      "step": 870,
+      "tokens_per_second_per_gpu": 1661.08
+    },
+    {
+      "epoch": 1.1443433029908974,
+      "grad_norm": 0.4118784964084625,
+      "learning_rate": 1.1473656852127052e-05,
+      "loss": 0.0114,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 880,
+      "tokens_per_second_per_gpu": 1770.78
+    },
+    {
+      "epoch": 1.1573472041612485,
+      "grad_norm": 0.2999536991119385,
+      "learning_rate": 1.1378377140759176e-05,
+      "loss": 0.0067,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.44,
+      "memory/max_allocated (GiB)": 57.44,
+      "step": 890,
+      "tokens_per_second_per_gpu": 962.27
+    },
+    {
+      "epoch": 1.1703511053315996,
+      "grad_norm": 0.2835685610771179,
+      "learning_rate": 1.1282235220163285e-05,
+      "loss": 0.0135,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 900,
+      "tokens_per_second_per_gpu": 925.81
+    },
+    {
+      "epoch": 1.1833550065019507,
+      "grad_norm": 0.22118036448955536,
+      "learning_rate": 1.1185252463828374e-05,
+      "loss": 0.0077,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 910,
+      "tokens_per_second_per_gpu": 1120.33
+    },
+    {
+      "epoch": 1.1963589076723018,
+      "grad_norm": 0.33028262853622437,
+      "learning_rate": 1.1087450432171197e-05,
+      "loss": 0.0073,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 920,
+      "tokens_per_second_per_gpu": 1025.3
+    },
+    {
+      "epoch": 1.209362808842653,
+      "grad_norm": 0.2743890881538391,
+      "learning_rate": 1.0988850867743119e-05,
+      "loss": 0.0083,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 930,
+      "tokens_per_second_per_gpu": 1208.52
+    },
+    {
+      "epoch": 1.222366710013004,
+      "grad_norm": 0.12477000802755356,
+      "learning_rate": 1.0889475690396506e-05,
+      "loss": 0.0059,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 940,
+      "tokens_per_second_per_gpu": 1406.03
+    },
+    {
+      "epoch": 1.2353706111833551,
+      "grad_norm": 0.37282922863960266,
+      "learning_rate": 1.0789346992411664e-05,
+      "loss": 0.0067,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 950,
+      "tokens_per_second_per_gpu": 933.12
+    },
+    {
+      "epoch": 1.2483745123537062,
+      "grad_norm": 0.1596948206424713,
+      "learning_rate": 1.068848703358548e-05,
+      "loss": 0.0162,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.49,
+      "memory/max_allocated (GiB)": 69.49,
+      "step": 960,
+      "tokens_per_second_per_gpu": 908.84
+    },
+    {
+      "epoch": 1.2613784135240573,
+      "grad_norm": 0.39361581206321716,
+      "learning_rate": 1.0586918236282791e-05,
+      "loss": 0.0059,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 970,
+      "tokens_per_second_per_gpu": 1514.1
+    },
+    {
+      "epoch": 1.2743823146944084,
+      "grad_norm": 0.3315422832965851,
+      "learning_rate": 1.0484663180451642e-05,
+      "loss": 0.0055,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 980,
+      "tokens_per_second_per_gpu": 836.31
+    },
+    {
+      "epoch": 1.2873862158647595,
+      "grad_norm": 0.1553254872560501,
+      "learning_rate": 1.0381744598603472e-05,
+      "loss": 0.0044,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 990,
+      "tokens_per_second_per_gpu": 1222.52
+    },
+    {
+      "epoch": 1.3003901170351106,
+      "grad_norm": 0.5420373678207397,
+      "learning_rate": 1.0278185370759435e-05,
+      "loss": 0.0054,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1000,
+      "tokens_per_second_per_gpu": 1212.78
+    },
+    {
+      "epoch": 1.3133940182054618,
+      "grad_norm": 0.3625222444534302,
+      "learning_rate": 1.017400851936388e-05,
+      "loss": 0.0045,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.54,
+      "memory/max_allocated (GiB)": 61.54,
+      "step": 1010,
+      "tokens_per_second_per_gpu": 2197.23
+    },
+    {
+      "epoch": 1.3263979193758129,
+      "grad_norm": 0.2578815817832947,
+      "learning_rate": 1.0069237204166195e-05,
+      "loss": 0.007,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.54,
+      "memory/max_allocated (GiB)": 61.54,
+      "step": 1020,
+      "tokens_per_second_per_gpu": 1330.56
+    },
+    {
+      "epoch": 1.339401820546164,
+      "grad_norm": 0.19943830370903015,
+      "learning_rate": 9.963894717072125e-06,
+      "loss": 0.0081,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1030,
+      "tokens_per_second_per_gpu": 1492.04
+    },
+    {
+      "epoch": 1.352405721716515,
+      "grad_norm": 0.34417328238487244,
+      "learning_rate": 9.858004476965698e-06,
+      "loss": 0.0065,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1040,
+      "tokens_per_second_per_gpu": 1067.44
+    },
+    {
+      "epoch": 1.3654096228868662,
+      "grad_norm": 0.5759336352348328,
+      "learning_rate": 9.751590024502928e-06,
+      "loss": 0.0045,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1050,
+      "tokens_per_second_per_gpu": 1398.43
+    },
+    {
+      "epoch": 1.3784135240572173,
+      "grad_norm": 0.1290610134601593,
+      "learning_rate": 9.644675016878454e-06,
+      "loss": 0.0041,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.41,
+      "memory/max_allocated (GiB)": 66.41,
+      "step": 1060,
+      "tokens_per_second_per_gpu": 1468.19
+    },
+    {
+      "epoch": 1.3914174252275684,
+      "grad_norm": 0.13373740017414093,
+      "learning_rate": 9.53728322256627e-06,
+      "loss": 0.0043,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1070,
+      "tokens_per_second_per_gpu": 702.65
+    },
+    {
+      "epoch": 1.4044213263979195,
+      "grad_norm": 0.3408486843109131,
+      "learning_rate": 9.429438516035696e-06,
+      "loss": 0.0046,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1080,
+      "tokens_per_second_per_gpu": 845.18
+    },
+    {
+      "epoch": 1.4174252275682706,
+      "grad_norm": 0.18220332264900208,
+      "learning_rate": 9.321164872443812e-06,
+      "loss": 0.0036,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.52,
+      "memory/max_allocated (GiB)": 70.52,
+      "step": 1090,
+      "tokens_per_second_per_gpu": 1416.9
+    },
+    {
+      "epoch": 1.4304291287386217,
+      "grad_norm": 0.43938443064689636,
+      "learning_rate": 9.21248636230549e-06,
+      "loss": 0.0186,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 1100,
+      "tokens_per_second_per_gpu": 1437.33
+    },
+    {
+      "epoch": 1.4434330299089728,
+      "grad_norm": 0.20190024375915527,
+      "learning_rate": 9.103427146142244e-06,
+      "loss": 0.0095,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.41,
+      "memory/max_allocated (GiB)": 66.41,
+      "step": 1110,
+      "tokens_per_second_per_gpu": 1018.6
+    },
+    {
+      "epoch": 1.456436931079324,
+      "grad_norm": 0.0726306140422821,
+      "learning_rate": 8.994011469111038e-06,
+      "loss": 0.0065,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.24,
+      "memory/max_allocated (GiB)": 69.24,
+      "step": 1120,
+      "tokens_per_second_per_gpu": 1154.67
+    },
+    {
+      "epoch": 1.469440832249675,
+      "grad_norm": 0.09384810924530029,
+      "learning_rate": 8.88426365561433e-06,
+      "loss": 0.0036,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 68.98,
+      "memory/max_allocated (GiB)": 68.98,
+      "step": 1130,
+      "tokens_per_second_per_gpu": 1056.42
+    },
+    {
+      "epoch": 1.4824447334200261,
+      "grad_norm": 0.1490725725889206,
+      "learning_rate": 8.774208103892445e-06,
+      "loss": 0.0039,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.03,
+      "memory/max_allocated (GiB)": 61.03,
+      "step": 1140,
+      "tokens_per_second_per_gpu": 1359.57
+    },
+    {
+      "epoch": 1.4954486345903772,
+      "grad_norm": 0.12897761166095734,
+      "learning_rate": 8.663869280599556e-06,
+      "loss": 0.0037,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1150,
+      "tokens_per_second_per_gpu": 845.82
+    },
+    {
+      "epoch": 1.5084525357607284,
+      "grad_norm": 0.2180880904197693,
+      "learning_rate": 8.553271715364483e-06,
+      "loss": 0.0035,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1160,
+      "tokens_per_second_per_gpu": 1709.05
+    },
+    {
+      "epoch": 1.5214564369310795,
+      "grad_norm": 0.39362478256225586,
+      "learning_rate": 8.442439995337436e-06,
+      "loss": 0.0052,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1170,
+      "tokens_per_second_per_gpu": 951.99
+    },
+    {
+      "epoch": 1.5344603381014306,
+      "grad_norm": 0.46583619713783264,
+      "learning_rate": 8.331398759724012e-06,
+      "loss": 0.0028,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.85,
+      "memory/max_allocated (GiB)": 73.85,
+      "step": 1180,
+      "tokens_per_second_per_gpu": 1170.04
+    },
+    {
+      "epoch": 1.5474642392717817,
+      "grad_norm": 0.3173917233943939,
+      "learning_rate": 8.220172694307613e-06,
+      "loss": 0.0037,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 59.49,
+      "memory/max_allocated (GiB)": 59.49,
+      "step": 1190,
+      "tokens_per_second_per_gpu": 1315.59
+    },
+    {
+      "epoch": 1.5604681404421328,
+      "grad_norm": 0.3780311942100525,
+      "learning_rate": 8.108786525961495e-06,
+      "loss": 0.005,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1200,
+      "tokens_per_second_per_gpu": 1365.14
+    },
+    {
+      "epoch": 1.5734720416124839,
+      "grad_norm": 4.287571430206299,
+      "learning_rate": 7.99726501715169e-06,
+      "loss": 0.0157,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 68.98,
+      "memory/max_allocated (GiB)": 68.98,
+      "step": 1210,
+      "tokens_per_second_per_gpu": 1125.8
+    },
+    {
+      "epoch": 1.586475942782835,
+      "grad_norm": 0.23012618720531464,
+      "learning_rate": 7.885632960432026e-06,
+      "loss": 0.0037,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1220,
+      "tokens_per_second_per_gpu": 692.44
+    },
+    {
+      "epoch": 1.599479843953186,
+      "grad_norm": 0.09980028122663498,
+      "learning_rate": 7.773915172932443e-06,
+      "loss": 0.0041,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.95,
+      "memory/max_allocated (GiB)": 57.95,
+      "step": 1230,
+      "tokens_per_second_per_gpu": 1102.86
+    },
+    {
+      "epoch": 1.6124837451235372,
+      "grad_norm": 0.2761804461479187,
+      "learning_rate": 7.66213649084184e-06,
+      "loss": 0.0032,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 1240,
+      "tokens_per_second_per_gpu": 764.72
+    },
+    {
+      "epoch": 1.6254876462938883,
+      "grad_norm": 0.31386464834213257,
+      "learning_rate": 7.550321763886716e-06,
+      "loss": 0.0027,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1250,
+      "tokens_per_second_per_gpu": 698.35
+    },
+    {
+      "epoch": 1.6384915474642394,
+      "grad_norm": 0.17101417481899261,
+      "learning_rate": 7.438495849806767e-06,
+      "loss": 0.0031,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.54,
+      "memory/max_allocated (GiB)": 61.54,
+      "step": 1260,
+      "tokens_per_second_per_gpu": 1024.64
+    },
+    {
+      "epoch": 1.6514954486345905,
+      "grad_norm": 0.29539087414741516,
+      "learning_rate": 7.32668360882872e-06,
+      "loss": 0.0027,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1270,
+      "tokens_per_second_per_gpu": 963.8
+    },
+    {
+      "epoch": 1.6644993498049416,
+      "grad_norm": 0.13337644934654236,
+      "learning_rate": 7.214909898139606e-06,
+      "loss": 0.0023,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.52,
+      "memory/max_allocated (GiB)": 70.52,
+      "step": 1280,
+      "tokens_per_second_per_gpu": 1681.43
+    },
+    {
+      "epoch": 1.6775032509752927,
+      "grad_norm": 0.24208548665046692,
+      "learning_rate": 7.103199566360723e-06,
+      "loss": 0.0027,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1290,
+      "tokens_per_second_per_gpu": 1308.12
+    },
+    {
+      "epoch": 1.6905071521456438,
+      "grad_norm": 0.7150481343269348,
+      "learning_rate": 6.991577448023481e-06,
+      "loss": 0.0032,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1300,
+      "tokens_per_second_per_gpu": 2064.31
+    },
+    {
+      "epoch": 1.703511053315995,
+      "grad_norm": 0.3450615704059601,
+      "learning_rate": 6.880068358048394e-06,
+      "loss": 0.0031,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1310,
+      "tokens_per_second_per_gpu": 1320.33
+    },
+    {
+      "epoch": 1.716514954486346,
+      "grad_norm": 0.09754879027605057,
+      "learning_rate": 6.7686970862284345e-06,
+      "loss": 0.0018,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1320,
+      "tokens_per_second_per_gpu": 901.06
+    },
+    {
+      "epoch": 1.7295188556566972,
+      "grad_norm": 0.5679341554641724,
+      "learning_rate": 6.657488391717966e-06,
+      "loss": 0.0023,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.44,
+      "memory/max_allocated (GiB)": 57.44,
+      "step": 1330,
+      "tokens_per_second_per_gpu": 2415.16
+    },
+    {
+      "epoch": 1.7425227568270483,
+      "grad_norm": 0.21700270473957062,
+      "learning_rate": 6.5464669975284795e-06,
+      "loss": 0.0024,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 68.98,
+      "memory/max_allocated (GiB)": 68.98,
+      "step": 1340,
+      "tokens_per_second_per_gpu": 1618.28
+    },
+    {
+      "epoch": 1.7555266579973994,
+      "grad_norm": 0.16351747512817383,
+      "learning_rate": 6.4356575850323955e-06,
+      "loss": 0.0026,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 1350,
+      "tokens_per_second_per_gpu": 966.61
+    },
+    {
+      "epoch": 1.7685305591677505,
+      "grad_norm": 0.15947993099689484,
+      "learning_rate": 6.325084788476085e-06,
+      "loss": 0.0026,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1360,
+      "tokens_per_second_per_gpu": 659.13
+    },
+    {
+      "epoch": 1.7815344603381016,
+      "grad_norm": 0.17046409845352173,
+      "learning_rate": 6.21477318950339e-06,
+      "loss": 0.0024,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1370,
+      "tokens_per_second_per_gpu": 780.17
+    },
+    {
+      "epoch": 1.7945383615084527,
+      "grad_norm": 0.19762098789215088,
+      "learning_rate": 6.104747311690827e-06,
+      "loss": 0.0032,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 62.82,
+      "memory/max_allocated (GiB)": 62.82,
+      "step": 1380,
+      "tokens_per_second_per_gpu": 1071.85
+    },
+    {
+      "epoch": 1.8075422626788038,
+      "grad_norm": 0.13971874117851257,
+      "learning_rate": 5.995031615095722e-06,
+      "loss": 0.0021,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1390,
+      "tokens_per_second_per_gpu": 1569.23
+    },
+    {
+      "epoch": 1.820546163849155,
+      "grad_norm": 0.125332772731781,
+      "learning_rate": 5.88565049081843e-06,
+      "loss": 0.0023,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 59.49,
+      "memory/max_allocated (GiB)": 59.49,
+      "step": 1400,
+      "tokens_per_second_per_gpu": 1041.64
+    },
+    {
+      "epoch": 1.833550065019506,
+      "grad_norm": 0.15583699941635132,
+      "learning_rate": 5.77662825557991e-06,
+      "loss": 0.0024,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.03,
+      "memory/max_allocated (GiB)": 61.03,
+      "step": 1410,
+      "tokens_per_second_per_gpu": 1460.83
+    },
+    {
+      "epoch": 1.8465539661898571,
+      "grad_norm": 0.1407925933599472,
+      "learning_rate": 5.667989146315842e-06,
+      "loss": 0.0028,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1420,
+      "tokens_per_second_per_gpu": 1433.69
+    },
+    {
+      "epoch": 1.8595578673602082,
+      "grad_norm": 0.1554357260465622,
+      "learning_rate": 5.559757314788449e-06,
+      "loss": 0.0026,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1430,
+      "tokens_per_second_per_gpu": 1724.45
+    },
+    {
+      "epoch": 1.8725617685305593,
+      "grad_norm": 0.07610394060611725,
+      "learning_rate": 5.4519568222172935e-06,
+      "loss": 0.0018,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1440,
+      "tokens_per_second_per_gpu": 948.47
+    },
+    {
+      "epoch": 1.8855656697009102,
+      "grad_norm": 0.08300016075372696,
+      "learning_rate": 5.34461163393016e-06,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 1450,
+      "tokens_per_second_per_gpu": 1517.09
+    },
+    {
+      "epoch": 1.8985695708712613,
+      "grad_norm": 0.1248791292309761,
+      "learning_rate": 5.237745614035305e-06,
+      "loss": 0.0137,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 62.31,
+      "memory/max_allocated (GiB)": 62.31,
+      "step": 1460,
+      "tokens_per_second_per_gpu": 1947.94
+    },
+    {
+      "epoch": 1.9115734720416124,
+      "grad_norm": 0.05855575576424599,
+      "learning_rate": 5.131382520116163e-06,
+      "loss": 0.0012,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1470,
+      "tokens_per_second_per_gpu": 1985.33
+    },
+    {
+      "epoch": 1.9245773732119635,
+      "grad_norm": 0.1409776657819748,
+      "learning_rate": 5.025545997949756e-06,
+      "loss": 0.0084,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.49,
+      "memory/max_allocated (GiB)": 69.49,
+      "step": 1480,
+      "tokens_per_second_per_gpu": 1292.8
+    },
+    {
+      "epoch": 1.9375812743823146,
+      "grad_norm": 0.07013244926929474,
+      "learning_rate": 4.920259576249972e-06,
+      "loss": 0.0014,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1490,
+      "tokens_per_second_per_gpu": 913.09
+    },
+    {
+      "epoch": 1.9505851755526658,
+      "grad_norm": 0.060349199920892715,
+      "learning_rate": 4.815546661436833e-06,
+      "loss": 0.0012,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1500,
+      "tokens_per_second_per_gpu": 1300.37
+    },
+    {
+      "epoch": 1.9635890767230169,
+      "grad_norm": 0.017504442483186722,
+      "learning_rate": 4.711430532432987e-06,
+      "loss": 0.0014,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.7,
+      "memory/max_allocated (GiB)": 57.7,
+      "step": 1510,
+      "tokens_per_second_per_gpu": 1327.41
+    },
+    {
+      "epoch": 1.976592977893368,
+      "grad_norm": 0.1838434785604477,
+      "learning_rate": 4.607934335488513e-06,
+      "loss": 0.0017,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 58.98,
+      "memory/max_allocated (GiB)": 58.98,
+      "step": 1520,
+      "tokens_per_second_per_gpu": 1642.05
+    },
+    {
+      "epoch": 1.989596879063719,
+      "grad_norm": 0.07420196384191513,
+      "learning_rate": 4.505081079035244e-06,
+      "loss": 0.0022,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1530,
+      "tokens_per_second_per_gpu": 747.3
+    },
+    {
+      "epoch": 2.0026007802340704,
+      "grad_norm": 0.14949700236320496,
+      "learning_rate": 4.402893628571721e-06,
+      "loss": 0.0166,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1540,
+      "tokens_per_second_per_gpu": 854.92
+    },
+    {
+      "epoch": 2.0156046814044215,
+      "grad_norm": 0.05961848050355911,
+      "learning_rate": 4.301394701579919e-06,
+      "loss": 0.01,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1550,
+      "tokens_per_second_per_gpu": 641.5
+    },
+    {
+      "epoch": 2.0286085825747726,
+      "grad_norm": 0.11824151873588562,
+      "learning_rate": 4.200606862474868e-06,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 62.82,
+      "memory/max_allocated (GiB)": 62.82,
+      "step": 1560,
+      "tokens_per_second_per_gpu": 1221.05
+    },
+    {
+      "epoch": 2.0416124837451237,
+      "grad_norm": 0.11225734651088715,
+      "learning_rate": 4.10055251758835e-06,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.49,
+      "memory/max_allocated (GiB)": 69.49,
+      "step": 1570,
+      "tokens_per_second_per_gpu": 1352.28
+    },
+    {
+      "epoch": 2.054616384915475,
+      "grad_norm": 0.380623996257782,
+      "learning_rate": 4.001253910187671e-06,
+      "loss": 0.0028,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1580,
+      "tokens_per_second_per_gpu": 1515.38
+    },
+    {
+      "epoch": 2.067620286085826,
+      "grad_norm": 0.013014468364417553,
+      "learning_rate": 3.902733115530727e-06,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.54,
+      "memory/max_allocated (GiB)": 61.54,
+      "step": 1590,
+      "tokens_per_second_per_gpu": 1490.02
+    },
+    {
+      "epoch": 2.080624187256177,
+      "grad_norm": 0.06560099869966507,
+      "learning_rate": 3.805012035958434e-06,
+      "loss": 0.0022,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1600,
+      "tokens_per_second_per_gpu": 1063.72
+    },
+    {
+      "epoch": 2.093628088426528,
+      "grad_norm": 0.020147603005170822,
+      "learning_rate": 3.7081123960255497e-06,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.41,
+      "memory/max_allocated (GiB)": 66.41,
+      "step": 1610,
+      "tokens_per_second_per_gpu": 631.63
+    },
+    {
+      "epoch": 2.1066319895968793,
+      "grad_norm": 0.06632567942142487,
+      "learning_rate": 3.6120557376710622e-06,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 61.54,
+      "memory/max_allocated (GiB)": 61.54,
+      "step": 1620,
+      "tokens_per_second_per_gpu": 1178.05
+    },
+    {
+      "epoch": 2.1196358907672304,
+      "grad_norm": 0.11541283875703812,
+      "learning_rate": 3.516863415429161e-06,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 1630,
+      "tokens_per_second_per_gpu": 2050.72
+    },
+    {
+      "epoch": 2.1326397919375815,
+      "grad_norm": 0.03378242999315262,
+      "learning_rate": 3.422556591681866e-06,
+      "loss": 0.0012,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1640,
+      "tokens_per_second_per_gpu": 1078.42
+    },
+    {
+      "epoch": 2.1456436931079326,
+      "grad_norm": 0.013072066940367222,
+      "learning_rate": 3.3291562319543816e-06,
+      "loss": 0.0017,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1650,
+      "tokens_per_second_per_gpu": 1175.41
+    },
+    {
+      "epoch": 2.1586475942782837,
+      "grad_norm": 0.01841077022254467,
+      "learning_rate": 3.2366831002542136e-06,
+      "loss": 0.0094,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1660,
+      "tokens_per_second_per_gpu": 674.94
+    },
+    {
+      "epoch": 2.171651495448635,
+      "grad_norm": 0.01450592465698719,
+      "learning_rate": 3.14515775445508e-06,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 74.37,
+      "memory/max_allocated (GiB)": 74.37,
+      "step": 1670,
+      "tokens_per_second_per_gpu": 2422.23
+    },
+    {
+      "epoch": 2.184655396618986,
+      "grad_norm": 0.01207063626497984,
+      "learning_rate": 3.0546005417266544e-06,
+      "loss": 0.0028,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1680,
+      "tokens_per_second_per_gpu": 924.25
+    },
+    {
+      "epoch": 2.197659297789337,
+      "grad_norm": 0.098720982670784,
+      "learning_rate": 2.9650315940111563e-06,
+      "loss": 0.0013,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1690,
+      "tokens_per_second_per_gpu": 1689.06
+    },
+    {
+      "epoch": 2.210663198959688,
+      "grad_norm": 0.11059177666902542,
+      "learning_rate": 2.8764708235477597e-06,
+      "loss": 0.0075,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 1700,
+      "tokens_per_second_per_gpu": 1362.01
+    },
+    {
+      "epoch": 2.223667100130039,
+      "grad_norm": 0.04324894770979881,
+      "learning_rate": 2.7889379184458894e-06,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.52,
+      "memory/max_allocated (GiB)": 70.52,
+      "step": 1710,
+      "tokens_per_second_per_gpu": 1094.79
+    },
+    {
+      "epoch": 2.2366710013003903,
+      "grad_norm": 0.05806596577167511,
+      "learning_rate": 2.702452338308317e-06,
+      "loss": 0.0038,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.67,
+      "memory/max_allocated (GiB)": 66.67,
+      "step": 1720,
+      "tokens_per_second_per_gpu": 1759.59
+    },
+    {
+      "epoch": 2.2496749024707414,
+      "grad_norm": 0.04493661969900131,
+      "learning_rate": 2.6170333099050434e-06,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1730,
+      "tokens_per_second_per_gpu": 1340.2
+    },
+    {
+      "epoch": 2.2626788036410925,
+      "grad_norm": 0.04071453958749771,
+      "learning_rate": 2.532699822898976e-06,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1740,
+      "tokens_per_second_per_gpu": 794.43
+    },
+    {
+      "epoch": 2.2756827048114436,
+      "grad_norm": 0.03370242193341255,
+      "learning_rate": 2.449470625624304e-06,
+      "loss": 0.0012,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1750,
+      "tokens_per_second_per_gpu": 571.47
+    },
+    {
+      "epoch": 2.2886866059817947,
+      "grad_norm": 0.07617732137441635,
+      "learning_rate": 2.367364220918511e-06,
+      "loss": 0.0009,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 1760,
+      "tokens_per_second_per_gpu": 1347.75
+    },
+    {
+      "epoch": 2.301690507152146,
+      "grad_norm": 0.04142403602600098,
+      "learning_rate": 2.2863988620089694e-06,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1770,
+      "tokens_per_second_per_gpu": 1066.54
+    },
+    {
+      "epoch": 2.314694408322497,
+      "grad_norm": 0.10175841301679611,
+      "learning_rate": 2.2065925484550547e-06,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 1780,
+      "tokens_per_second_per_gpu": 1934.94
+    },
+    {
+      "epoch": 2.327698309492848,
+      "grad_norm": 0.07680214196443558,
+      "learning_rate": 2.127963022146608e-06,
+      "loss": 0.0019,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1790,
+      "tokens_per_second_per_gpu": 1789.18
+    },
+    {
+      "epoch": 2.340702210663199,
+      "grad_norm": 0.04380909726023674,
+      "learning_rate": 2.0505277633597134e-06,
+      "loss": 0.0022,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.26,
+      "memory/max_allocated (GiB)": 70.26,
+      "step": 1800,
+      "tokens_per_second_per_gpu": 591.42
+    },
+    {
+      "epoch": 2.3537061118335503,
+      "grad_norm": 0.0950201004743576,
+      "learning_rate": 1.974303986870622e-06,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 58.98,
+      "memory/max_allocated (GiB)": 58.98,
+      "step": 1810,
+      "tokens_per_second_per_gpu": 823.01
+    },
+    {
+      "epoch": 2.3667100130039014,
+      "grad_norm": 0.02075243555009365,
+      "learning_rate": 1.8993086381287075e-06,
+      "loss": 0.0014,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1820,
+      "tokens_per_second_per_gpu": 1258.85
+    },
+    {
+      "epoch": 2.3797139141742525,
+      "grad_norm": 0.07134304195642471,
+      "learning_rate": 1.825558389489284e-06,
+      "loss": 0.0067,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 60.52,
+      "memory/max_allocated (GiB)": 60.52,
+      "step": 1830,
+      "tokens_per_second_per_gpu": 1425.99
+    },
+    {
+      "epoch": 2.3927178153446036,
+      "grad_norm": 0.03811859339475632,
+      "learning_rate": 1.7530696365071533e-06,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1840,
+      "tokens_per_second_per_gpu": 1091.29
+    },
+    {
+      "epoch": 2.4057217165149547,
+      "grad_norm": 0.09027709066867828,
+      "learning_rate": 1.6818584942916713e-06,
+      "loss": 0.0012,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 70.0,
+      "memory/max_allocated (GiB)": 70.0,
+      "step": 1850,
+      "tokens_per_second_per_gpu": 1064.05
+    },
+    {
+      "epoch": 2.418725617685306,
+      "grad_norm": 0.07543770968914032,
+      "learning_rate": 1.6119407939241678e-06,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1860,
+      "tokens_per_second_per_gpu": 1107.21
+    },
+    {
+      "epoch": 2.431729518855657,
+      "grad_norm": 0.0029138713143765926,
+      "learning_rate": 1.5433320789385233e-06,
+      "loss": 0.0045,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1870,
+      "tokens_per_second_per_gpu": 998.03
+    },
+    {
+      "epoch": 2.444733420026008,
+      "grad_norm": 0.07904035598039627,
+      "learning_rate": 1.476047601865629e-06,
+      "loss": 0.0062,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 62.82,
+      "memory/max_allocated (GiB)": 62.82,
+      "step": 1880,
+      "tokens_per_second_per_gpu": 1602.64
+    },
+    {
+      "epoch": 2.457737321196359,
+      "grad_norm": 0.049412939697504044,
+      "learning_rate": 1.4101023208426009e-06,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 65.39,
+      "memory/max_allocated (GiB)": 65.39,
+      "step": 1890,
+      "tokens_per_second_per_gpu": 1454.25
+    },
+    {
+      "epoch": 2.4707412223667102,
+      "grad_norm": 0.1244095042347908,
+      "learning_rate": 1.3455108962873858e-06,
+      "loss": 0.0054,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1900,
+      "tokens_per_second_per_gpu": 1397.2
+    },
+    {
+      "epoch": 2.4837451235370613,
+      "grad_norm": 0.07130023092031479,
+      "learning_rate": 1.2822876876395803e-06,
+      "loss": 0.0009,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1910,
+      "tokens_per_second_per_gpu": 639.02
+    },
+    {
+      "epoch": 2.4967490247074124,
+      "grad_norm": 0.07631956785917282,
+      "learning_rate": 1.2204467501681699e-06,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1920,
+      "tokens_per_second_per_gpu": 1403.68
+    },
+    {
+      "epoch": 2.5097529258777636,
+      "grad_norm": 0.11364943534135818,
+      "learning_rate": 1.160001831846859e-06,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1930,
+      "tokens_per_second_per_gpu": 583.43
+    },
+    {
+      "epoch": 2.5227568270481147,
+      "grad_norm": 0.03140031918883324,
+      "learning_rate": 1.100966370297744e-06,
+      "loss": 0.0007,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1940,
+      "tokens_per_second_per_gpu": 1219.29
+    },
+    {
+      "epoch": 2.5357607282184658,
+      "grad_norm": 0.08222731202840805,
+      "learning_rate": 1.043353489803968e-06,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1950,
+      "tokens_per_second_per_gpu": 873.65
+    },
+    {
+      "epoch": 2.548764629388817,
+      "grad_norm": 0.08749080449342728,
+      "learning_rate": 9.87175998392034e-07,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.49,
+      "memory/max_allocated (GiB)": 69.49,
+      "step": 1960,
+      "tokens_per_second_per_gpu": 1189.31
+    },
+    {
+      "epoch": 2.561768530559168,
+      "grad_norm": 0.03166181594133377,
+      "learning_rate": 9.32446384984435e-07,
+      "loss": 0.0008,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1970,
+      "tokens_per_second_per_gpu": 1032.59
+    },
+    {
+      "epoch": 2.574772431729519,
+      "grad_norm": 0.5145336985588074,
+      "learning_rate": 8.791768166232111e-07,
+      "loss": 0.0019,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 72.83,
+      "memory/max_allocated (GiB)": 72.83,
+      "step": 1980,
+      "tokens_per_second_per_gpu": 1565.6
+    },
+    {
+      "epoch": 2.58777633289987,
+      "grad_norm": 0.10077083855867386,
+      "learning_rate": 8.273791357650776e-07,
+      "loss": 0.0009,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 1990,
+      "tokens_per_second_per_gpu": 1079.17
+    },
+    {
+      "epoch": 2.6007802340702213,
+      "grad_norm": 0.045404162257909775,
+      "learning_rate": 7.770648576487035e-07,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2000,
+      "tokens_per_second_per_gpu": 1446.58
+    },
+    {
+      "epoch": 2.6137841352405724,
+      "grad_norm": 0.05263739824295044,
+      "learning_rate": 7.282451677347418e-07,
+      "loss": 0.0017,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2010,
+      "tokens_per_second_per_gpu": 789.31
+    },
+    {
+      "epoch": 2.6267880364109235,
+      "grad_norm": 0.05211790278553963,
+      "learning_rate": 6.809309192191576e-07,
+      "loss": 0.0007,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2020,
+      "tokens_per_second_per_gpu": 834.81
+    },
+    {
+      "epoch": 2.6397919375812746,
+      "grad_norm": 0.04055894538760185,
+      "learning_rate": 6.351326306204471e-07,
+      "loss": 0.0009,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.95,
+      "memory/max_allocated (GiB)": 57.95,
+      "step": 2030,
+      "tokens_per_second_per_gpu": 1392.06
+    },
+    {
+      "epoch": 2.6527958387516257,
+      "grad_norm": 0.08192923665046692,
+      "learning_rate": 5.908604834412424e-07,
+      "loss": 0.0012,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2040,
+      "tokens_per_second_per_gpu": 711.64
+    },
+    {
+      "epoch": 2.665799739921977,
+      "grad_norm": 0.04297219589352608,
+      "learning_rate": 5.481243199048378e-07,
+      "loss": 0.0009,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2050,
+      "tokens_per_second_per_gpu": 1149.51
+    },
+    {
+      "epoch": 2.678803641092328,
+      "grad_norm": 0.04260098189115524,
+      "learning_rate": 5.069336407671479e-07,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.7,
+      "memory/max_allocated (GiB)": 57.7,
+      "step": 2060,
+      "tokens_per_second_per_gpu": 601.31
+    },
+    {
+      "epoch": 2.691807542262679,
+      "grad_norm": 0.12324750423431396,
+      "learning_rate": 4.672976032045859e-07,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 59.49,
+      "memory/max_allocated (GiB)": 59.49,
+      "step": 2070,
+      "tokens_per_second_per_gpu": 1453.31
+    },
+    {
+      "epoch": 2.70481144343303,
+      "grad_norm": 0.0836389884352684,
+      "learning_rate": 4.292250187783031e-07,
+      "loss": 0.0013,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2080,
+      "tokens_per_second_per_gpu": 927.33
+    },
+    {
+      "epoch": 2.7178153446033813,
+      "grad_norm": 0.22495557367801666,
+      "learning_rate": 3.927243514752785e-07,
+      "loss": 0.0017,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2090,
+      "tokens_per_second_per_gpu": 904.3
+    },
+    {
+      "epoch": 2.7308192457737324,
+      "grad_norm": 0.12274650484323502,
+      "learning_rate": 3.5780371582668056e-07,
+      "loss": 0.0009,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.16,
+      "memory/max_allocated (GiB)": 66.16,
+      "step": 2100,
+      "tokens_per_second_per_gpu": 1541.7
+    },
+    {
+      "epoch": 2.7438231469440835,
+      "grad_norm": 0.03478631377220154,
+      "learning_rate": 3.2447087510389853e-07,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2110,
+      "tokens_per_second_per_gpu": 861.56
+    },
+    {
+      "epoch": 2.7568270481144346,
+      "grad_norm": 0.07994584739208221,
+      "learning_rate": 2.927332395926849e-07,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.85,
+      "memory/max_allocated (GiB)": 73.85,
+      "step": 2120,
+      "tokens_per_second_per_gpu": 881.47
+    },
+    {
+      "epoch": 2.7698309492847857,
+      "grad_norm": 0.059835221618413925,
+      "learning_rate": 2.625978649457583e-07,
+      "loss": 0.004,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 69.49,
+      "memory/max_allocated (GiB)": 69.49,
+      "step": 2130,
+      "tokens_per_second_per_gpu": 1584.01
+    },
+    {
+      "epoch": 2.782834850455137,
+      "grad_norm": 0.05771668255329132,
+      "learning_rate": 2.3407145061424606e-07,
+      "loss": 0.0006,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2140,
+      "tokens_per_second_per_gpu": 1549.32
+    },
+    {
+      "epoch": 2.795838751625488,
+      "grad_norm": 0.04278785362839699,
+      "learning_rate": 2.0716033835831843e-07,
+      "loss": 0.0023,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2150,
+      "tokens_per_second_per_gpu": 1138.04
+    },
+    {
+      "epoch": 2.808842652795839,
+      "grad_norm": 0.0280362106859684,
+      "learning_rate": 1.818705108373417e-07,
+      "loss": 0.0036,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 59.49,
+      "memory/max_allocated (GiB)": 59.49,
+      "step": 2160,
+      "tokens_per_second_per_gpu": 889.39
+    },
+    {
+      "epoch": 2.82184655396619,
+      "grad_norm": 0.005239436868578196,
+      "learning_rate": 1.5820759027985853e-07,
+      "loss": 0.0007,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 72.83,
+      "memory/max_allocated (GiB)": 72.83,
+      "step": 2170,
+      "tokens_per_second_per_gpu": 1290.73
+    },
+    {
+      "epoch": 2.834850455136541,
+      "grad_norm": 0.0735795795917511,
+      "learning_rate": 1.36176837233703e-07,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.7,
+      "memory/max_allocated (GiB)": 57.7,
+      "step": 2180,
+      "tokens_per_second_per_gpu": 1019.66
+    },
+    {
+      "epoch": 2.8478543563068923,
+      "grad_norm": 0.11477436870336533,
+      "learning_rate": 1.1578314939651835e-07,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2190,
+      "tokens_per_second_per_gpu": 1331.36
+    },
+    {
+      "epoch": 2.8608582574772434,
+      "grad_norm": 0.06786138564348221,
+      "learning_rate": 9.70310605269356e-08,
+      "loss": 0.0008,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2200,
+      "tokens_per_second_per_gpu": 975.97
+    },
+    {
+      "epoch": 2.8738621586475945,
+      "grad_norm": 0.07386753708124161,
+      "learning_rate": 7.992473943667311e-08,
+      "loss": 0.0032,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 66.67,
+      "memory/max_allocated (GiB)": 66.67,
+      "step": 2210,
+      "tokens_per_second_per_gpu": 995.53
+    },
+    {
+      "epoch": 2.8868660598179456,
+      "grad_norm": 0.08402879536151886,
+      "learning_rate": 6.446798906375701e-08,
+      "loss": 0.0016,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2220,
+      "tokens_per_second_per_gpu": 1310.48
+    },
+    {
+      "epoch": 2.8998699609882967,
+      "grad_norm": 0.003793215611949563,
+      "learning_rate": 5.0664245627077554e-08,
+      "loss": 0.0011,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2230,
+      "tokens_per_second_per_gpu": 1516.16
+    },
+    {
+      "epoch": 2.912873862158648,
+      "grad_norm": 0.05847960710525513,
+      "learning_rate": 3.851657786248686e-08,
+      "loss": 0.0012,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 68.98,
+      "memory/max_allocated (GiB)": 68.98,
+      "step": 2240,
+      "tokens_per_second_per_gpu": 1525.05
+    },
+    {
+      "epoch": 2.925877763328999,
+      "grad_norm": 0.04987294226884842,
+      "learning_rate": 2.8027686340574564e-08,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2250,
+      "tokens_per_second_per_gpu": 971.31
+    },
+    {
+      "epoch": 2.93888166449935,
+      "grad_norm": 0.05919062718749046,
+      "learning_rate": 1.9199902866303964e-08,
+      "loss": 0.002,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2260,
+      "tokens_per_second_per_gpu": 737.3
+    },
+    {
+      "epoch": 2.951885565669701,
+      "grad_norm": 0.04226401820778847,
+      "learning_rate": 1.2035189960619141e-08,
+      "loss": 0.0009,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2270,
+      "tokens_per_second_per_gpu": 621.89
+    },
+    {
+      "epoch": 2.9648894668400523,
+      "grad_norm": 0.0024712388403713703,
+      "learning_rate": 6.535140424159814e-09,
+      "loss": 0.0015,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 59.49,
+      "memory/max_allocated (GiB)": 59.49,
+      "step": 2280,
+      "tokens_per_second_per_gpu": 1370.84
+    },
+    {
+      "epoch": 2.9778933680104034,
+      "grad_norm": 0.06625816971063614,
+      "learning_rate": 2.7009769831562446e-09,
+      "loss": 0.0008,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 73.34,
+      "memory/max_allocated (GiB)": 73.34,
+      "step": 2290,
+      "tokens_per_second_per_gpu": 1202.79
+    },
+    {
+      "epoch": 2.9908972691807545,
+      "grad_norm": 0.04506971314549446,
+      "learning_rate": 5.335520176058539e-10,
+      "loss": 0.001,
+      "memory/device_reserved (GiB)": 130.22,
+      "memory/max_active (GiB)": 57.31,
+      "memory/max_allocated (GiB)": 57.31,
+      "step": 2300,
+      "tokens_per_second_per_gpu": 1049.94
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 2307,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.229467717168005e+18,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2307/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90bdc452771ffa9e658919ee9ad02c5fb857c60108af1a1e97368058dc13df12
+size 8721

config.json

@@ -0,0 +1,68 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 40960,
+  "max_window_layers": 36,
+  "model_type": "qwen3",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151669,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.0",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

debug.log

@@ -0,0 +1,7 @@
+
+[2026-01-05 19:56:22,639] [INFO] [axolotl.utils.data.sft._load_raw_datasets:320] [PID:444] Loading raw datasets...
+[2026-01-05 19:56:23,255] [INFO] [axolotl.utils.data.wrappers.get_dataset_wrapper:87] [PID:444] Loading dataset: WNT3D/Ultimate-Offensive-Red-Team with base_type: alpaca and prompt_style: None
+
+
+
+

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151669,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "4.57.0"
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a0e50d3dd155ae4a64df2f2ccfcb4848cfe87877a5ec4792233ea6db2b67378
+size 4902257696

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6bf727322666823665b53e2694441c8a993cf9156578d20465f9a1ab88cfd22f
+size 4915960368

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d78bd156b3d080877d255126677071025cd8e44d24c637c71b274d5017de991
+size 4983068496

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f68e1dab321833cbd7a77008f5ce9f4c2ee0eda6c01ff88b44d421cba5e9d1ac
+size 1580230264

model.safetensors.index.json

@@ -0,0 +1,407 @@
+{
+  "metadata": {
+    "total_parameters": 8190735360,
+    "total_size": 16381470720
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.norm.weight": "model-00004-of-00004.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:742070173edd1d9fbf1a6c9135828398c90076ffdd417d7fd4274806e9ddbdc5
+size 11422838

tokenizer_config.json

@@ -0,0 +1,247 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|pad|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}