Instructions to use FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth")
model = AutoModelForCausalLM.from_pretrained("FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth

SGLang

How to use FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth with Docker Model Runner:
```
docker model run hf.co/FormlessAI/Qwen2.5-3B-Instruct-Code-Unsloth
```

FormlessAI commited on Jul 28

Commit

8dd66e1

verified ·

1 Parent(s): 859777a

Upload folder using huggingface_hub

Browse files

Files changed (15) hide show

.gitattributes +1 -0
README.md +59 -0
added_tokens.json +24 -0
chat_template.jinja +54 -0
config.json +69 -0
generation_config.json +14 -0
merges.txt +0 -0
model-00001-of-00002.safetensors +3 -0
model-00002-of-00002.safetensors +3 -0
model.safetensors.index.json +442 -0
special_tokens_map.json +31 -0
tokenizer.json +3 -0
tokenizer_config.json +208 -0
trainer_state.json +3055 -0
vocab.json +0 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,59 @@

+---
+base_model: unsloth/Qwen2.5-3B-Instruct
+library_name: transformers
+model_name: Qwen_Qwen2.5-3B-Instruct__project-3b-code_1785202385
+tags:
+- generated_from_trainer
+- sft
+- unsloth
+- trl
+licence: license
+---
+# Model Card for Qwen_Qwen2.5-3B-Instruct__project-3b-code_1785202385
+This model is a fine-tuned version of [unsloth/Qwen2.5-3B-Instruct](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+## Quick start
+```python
+from transformers import pipeline
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+## Training procedure
+This model was trained with SFT.
+### Framework versions
+- TRL: 0.23.1
+- Transformers: 4.57.6
+- Pytorch: 2.6.0+cu124
+- Datasets: 4.3.0
+- Tokenizers: 0.22.2
+## Citations
+Cite TRL as:
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

added_tokens.json ADDED Viewed

	@@ -0,0 +1,24 @@

+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,54 @@

+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json ADDED Viewed

	@@ -0,0 +1,69 @@

+{
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 70,
+  "model_name": "unsloth/Qwen2.5-3B-Instruct",
+  "model_type": "qwen2",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 2,
+  "pad_token_id": 151654,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.6",
+  "unsloth_fixed": true,
+  "unsloth_version": "2026.7.5",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "max_length": 32768,
+  "pad_token_id": 151654,
+  "repetition_penalty": 1.05,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "4.57.6"
+}

merges.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

model-00001-of-00002.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:11b528c623ad89b910bcc6c9db217e2ad205b5a673cd6442745c3040e88d6461
+size 4957789160

model-00002-of-00002.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:02c865c30d24b1d57869244be4392c4a3a87296df74455b01266c7a6767af95e
+size 1214436184

model.safetensors.index.json ADDED Viewed

	@@ -0,0 +1,442 @@

+{
+  "metadata": {
+    "total_parameters": 3085938688,
+    "total_size": 6172176384
+  },
+  "weight_map": {
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.bias": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.bias": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,31 @@

+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|vision_pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,208 @@

+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|vision_pad|>",
+  "padding_side": "left",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json ADDED Viewed

	@@ -0,0 +1,3055 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.0,
+  "eval_steps": 30,
+  "global_step": 294,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.006837606837606838,
+      "grad_norm": 32.511837005615234,
+      "learning_rate": 0.0,
+      "loss": 1.2298,
+      "num_input_tokens_seen": 35008,
+      "step": 1,
+      "train_runtime": 8.0497,
+      "train_tokens_per_second": 4348.976
+    },
+    {
+      "epoch": 0.013675213675213675,
+      "grad_norm": 28.007068634033203,
+      "learning_rate": 4.000000000000001e-06,
+      "loss": 0.9948,
+      "num_input_tokens_seen": 83776,
+      "step": 2,
+      "train_runtime": 12.057,
+      "train_tokens_per_second": 6948.356
+    },
+    {
+      "epoch": 0.020512820512820513,
+      "grad_norm": 30.101133346557617,
+      "learning_rate": 8.000000000000001e-06,
+      "loss": 1.0103,
+      "num_input_tokens_seen": 123392,
+      "step": 3,
+      "train_runtime": 15.3107,
+      "train_tokens_per_second": 8059.208
+    },
+    {
+      "epoch": 0.02735042735042735,
+      "grad_norm": 19.826982498168945,
+      "learning_rate": 1.2e-05,
+      "loss": 0.7465,
+      "num_input_tokens_seen": 163904,
+      "step": 4,
+      "train_runtime": 18.5797,
+      "train_tokens_per_second": 8821.648
+    },
+    {
+      "epoch": 0.03418803418803419,
+      "grad_norm": 5.2462077140808105,
+      "learning_rate": 1.6000000000000003e-05,
+      "loss": 0.4954,
+      "num_input_tokens_seen": 204512,
+      "step": 5,
+      "train_runtime": 21.9294,
+      "train_tokens_per_second": 9325.92
+    },
+    {
+      "epoch": 0.041025641025641026,
+      "grad_norm": 6.633277893066406,
+      "learning_rate": 2e-05,
+      "loss": 0.5549,
+      "num_input_tokens_seen": 241824,
+      "step": 6,
+      "train_runtime": 25.0124,
+      "train_tokens_per_second": 9668.169
+    },
+    {
+      "epoch": 0.04786324786324787,
+      "grad_norm": 3.92435884475708,
+      "learning_rate": 1.9930795847750867e-05,
+      "loss": 0.447,
+      "num_input_tokens_seen": 291392,
+      "step": 7,
+      "train_runtime": 28.9989,
+      "train_tokens_per_second": 10048.39
+    },
+    {
+      "epoch": 0.0547008547008547,
+      "grad_norm": 2.5249392986297607,
+      "learning_rate": 1.9861591695501733e-05,
+      "loss": 0.5641,
+      "num_input_tokens_seen": 325088,
+      "step": 8,
+      "train_runtime": 31.8438,
+      "train_tokens_per_second": 10208.84
+    },
+    {
+      "epoch": 0.06153846153846154,
+      "grad_norm": 1.8566827774047852,
+      "learning_rate": 1.9792387543252595e-05,
+      "loss": 0.4856,
+      "num_input_tokens_seen": 357280,
+      "step": 9,
+      "train_runtime": 34.5322,
+      "train_tokens_per_second": 10346.275
+    },
+    {
+      "epoch": 0.06837606837606838,
+      "grad_norm": 2.148573160171509,
+      "learning_rate": 1.972318339100346e-05,
+      "loss": 0.5044,
+      "num_input_tokens_seen": 397440,
+      "step": 10,
+      "train_runtime": 37.8542,
+      "train_tokens_per_second": 10499.228
+    },
+    {
+      "epoch": 0.07521367521367521,
+      "grad_norm": 2.142800807952881,
+      "learning_rate": 1.9653979238754327e-05,
+      "loss": 0.464,
+      "num_input_tokens_seen": 446592,
+      "step": 11,
+      "train_runtime": 44.1229,
+      "train_tokens_per_second": 10121.553
+    },
+    {
+      "epoch": 0.08205128205128205,
+      "grad_norm": 1.9147850275039673,
+      "learning_rate": 1.9584775086505192e-05,
+      "loss": 0.4892,
+      "num_input_tokens_seen": 477888,
+      "step": 12,
+      "train_runtime": 46.7615,
+      "train_tokens_per_second": 10219.683
+    },
+    {
+      "epoch": 0.08888888888888889,
+      "grad_norm": 1.8965516090393066,
+      "learning_rate": 1.9515570934256058e-05,
+      "loss": 0.4817,
+      "num_input_tokens_seen": 517056,
+      "step": 13,
+      "train_runtime": 49.9951,
+      "train_tokens_per_second": 10342.129
+    },
+    {
+      "epoch": 0.09572649572649573,
+      "grad_norm": 1.7629674673080444,
+      "learning_rate": 1.9446366782006923e-05,
+      "loss": 0.516,
+      "num_input_tokens_seen": 551104,
+      "step": 14,
+      "train_runtime": 52.8297,
+      "train_tokens_per_second": 10431.717
+    },
+    {
+      "epoch": 0.10256410256410256,
+      "grad_norm": 1.6642876863479614,
+      "learning_rate": 1.9377162629757786e-05,
+      "loss": 0.455,
+      "num_input_tokens_seen": 587104,
+      "step": 15,
+      "train_runtime": 55.844,
+      "train_tokens_per_second": 10513.294
+    },
+    {
+      "epoch": 0.1094017094017094,
+      "grad_norm": 1.685402274131775,
+      "learning_rate": 1.930795847750865e-05,
+      "loss": 0.4607,
+      "num_input_tokens_seen": 624800,
+      "step": 16,
+      "train_runtime": 59.0018,
+      "train_tokens_per_second": 10589.5
+    },
+    {
+      "epoch": 0.11623931623931624,
+      "grad_norm": 1.7894681692123413,
+      "learning_rate": 1.9238754325259517e-05,
+      "loss": 0.4795,
+      "num_input_tokens_seen": 667424,
+      "step": 17,
+      "train_runtime": 62.5071,
+      "train_tokens_per_second": 10677.567
+    },
+    {
+      "epoch": 0.12307692307692308,
+      "grad_norm": 1.5134732723236084,
+      "learning_rate": 1.9169550173010383e-05,
+      "loss": 0.4488,
+      "num_input_tokens_seen": 705056,
+      "step": 18,
+      "train_runtime": 65.654,
+      "train_tokens_per_second": 10738.968
+    },
+    {
+      "epoch": 0.12991452991452992,
+      "grad_norm": 1.671798825263977,
+      "learning_rate": 1.910034602076125e-05,
+      "loss": 0.4473,
+      "num_input_tokens_seen": 735200,
+      "step": 19,
+      "train_runtime": 68.188,
+      "train_tokens_per_second": 10781.957
+    },
+    {
+      "epoch": 0.13675213675213677,
+      "grad_norm": 1.7231043577194214,
+      "learning_rate": 1.9031141868512114e-05,
+      "loss": 0.4886,
+      "num_input_tokens_seen": 769856,
+      "step": 20,
+      "train_runtime": 71.1292,
+      "train_tokens_per_second": 10823.354
+    },
+    {
+      "epoch": 0.14358974358974358,
+      "grad_norm": 1.4723550081253052,
+      "learning_rate": 1.8961937716262976e-05,
+      "loss": 0.3481,
+      "num_input_tokens_seen": 846016,
+      "step": 21,
+      "train_runtime": 81.9017,
+      "train_tokens_per_second": 10329.653
+    },
+    {
+      "epoch": 0.15042735042735042,
+      "grad_norm": 1.5541292428970337,
+      "learning_rate": 1.8892733564013842e-05,
+      "loss": 0.4992,
+      "num_input_tokens_seen": 880352,
+      "step": 22,
+      "train_runtime": 84.7765,
+      "train_tokens_per_second": 10384.389
+    },
+    {
+      "epoch": 0.15726495726495726,
+      "grad_norm": 1.5459015369415283,
+      "learning_rate": 1.8823529411764708e-05,
+      "loss": 0.4687,
+      "num_input_tokens_seen": 920448,
+      "step": 23,
+      "train_runtime": 88.149,
+      "train_tokens_per_second": 10441.956
+    },
+    {
+      "epoch": 0.1641025641025641,
+      "grad_norm": 1.7185068130493164,
+      "learning_rate": 1.8754325259515573e-05,
+      "loss": 0.4597,
+      "num_input_tokens_seen": 966944,
+      "step": 24,
+      "train_runtime": 92.0191,
+      "train_tokens_per_second": 10508.075
+    },
+    {
+      "epoch": 0.17094017094017094,
+      "grad_norm": 1.4897587299346924,
+      "learning_rate": 1.868512110726644e-05,
+      "loss": 0.4537,
+      "num_input_tokens_seen": 1008384,
+      "step": 25,
+      "train_runtime": 95.4709,
+      "train_tokens_per_second": 10562.217
+    },
+    {
+      "epoch": 0.17777777777777778,
+      "grad_norm": 1.5461437702178955,
+      "learning_rate": 1.8615916955017305e-05,
+      "loss": 0.4948,
+      "num_input_tokens_seen": 1045088,
+      "step": 26,
+      "train_runtime": 98.5427,
+      "train_tokens_per_second": 10605.434
+    },
+    {
+      "epoch": 0.18461538461538463,
+      "grad_norm": 1.5771409273147583,
+      "learning_rate": 1.8546712802768167e-05,
+      "loss": 0.5204,
+      "num_input_tokens_seen": 1089184,
+      "step": 27,
+      "train_runtime": 102.2145,
+      "train_tokens_per_second": 10655.869
+    },
+    {
+      "epoch": 0.19145299145299147,
+      "grad_norm": 1.4081662893295288,
+      "learning_rate": 1.8477508650519033e-05,
+      "loss": 0.452,
+      "num_input_tokens_seen": 1131168,
+      "step": 28,
+      "train_runtime": 105.7371,
+      "train_tokens_per_second": 10697.931
+    },
+    {
+      "epoch": 0.19829059829059828,
+      "grad_norm": 1.6867786645889282,
+      "learning_rate": 1.8408304498269898e-05,
+      "loss": 0.4906,
+      "num_input_tokens_seen": 1164352,
+      "step": 29,
+      "train_runtime": 108.5496,
+      "train_tokens_per_second": 10726.45
+    },
+    {
+      "epoch": 0.20512820512820512,
+      "grad_norm": 1.3786219358444214,
+      "learning_rate": 1.833910034602076e-05,
+      "loss": 0.4705,
+      "num_input_tokens_seen": 1201440,
+      "step": 30,
+      "train_runtime": 111.6624,
+      "train_tokens_per_second": 10759.575
+    },
+    {
+      "epoch": 0.20512820512820512,
+      "eval_loss": 0.4640962481498718,
+      "eval_runtime": 6.7936,
+      "eval_samples_per_second": 146.903,
+      "eval_steps_per_second": 4.71,
+      "num_input_tokens_seen": 1201440,
+      "step": 30
+    },
+    {
+      "epoch": 0.21196581196581196,
+      "grad_norm": 1.7535614967346191,
+      "learning_rate": 1.826989619377163e-05,
+      "loss": 0.487,
+      "num_input_tokens_seen": 1236704,
+      "step": 31,
+      "train_runtime": 139.6017,
+      "train_tokens_per_second": 8858.806
+    },
+    {
+      "epoch": 0.2188034188034188,
+      "grad_norm": 1.4274615049362183,
+      "learning_rate": 1.8200692041522492e-05,
+      "loss": 0.4459,
+      "num_input_tokens_seen": 1278464,
+      "step": 32,
+      "train_runtime": 143.0525,
+      "train_tokens_per_second": 8937.028
+    },
+    {
+      "epoch": 0.22564102564102564,
+      "grad_norm": 1.527426838874817,
+      "learning_rate": 1.8131487889273357e-05,
+      "loss": 0.5151,
+      "num_input_tokens_seen": 1315616,
+      "step": 33,
+      "train_runtime": 146.1773,
+      "train_tokens_per_second": 9000.14
+    },
+    {
+      "epoch": 0.23247863247863249,
+      "grad_norm": 1.4509639739990234,
+      "learning_rate": 1.8062283737024223e-05,
+      "loss": 0.473,
+      "num_input_tokens_seen": 1351200,
+      "step": 34,
+      "train_runtime": 149.1958,
+      "train_tokens_per_second": 9056.555
+    },
+    {
+      "epoch": 0.23931623931623933,
+      "grad_norm": 1.574431300163269,
+      "learning_rate": 1.799307958477509e-05,
+      "loss": 0.4562,
+      "num_input_tokens_seen": 1393888,
+      "step": 35,
+      "train_runtime": 152.7492,
+      "train_tokens_per_second": 9125.335
+    },
+    {
+      "epoch": 0.24615384615384617,
+      "grad_norm": 1.421439290046692,
+      "learning_rate": 1.792387543252595e-05,
+      "loss": 0.4213,
+      "num_input_tokens_seen": 1432000,
+      "step": 36,
+      "train_runtime": 155.9721,
+      "train_tokens_per_second": 9181.127
+    },
+    {
+      "epoch": 0.252991452991453,
+      "grad_norm": 1.3346866369247437,
+      "learning_rate": 1.785467128027682e-05,
+      "loss": 0.4865,
+      "num_input_tokens_seen": 1474912,
+      "step": 37,
+      "train_runtime": 159.5609,
+      "train_tokens_per_second": 9243.568
+    },
+    {
+      "epoch": 0.25982905982905985,
+      "grad_norm": 1.401798129081726,
+      "learning_rate": 1.7785467128027682e-05,
+      "loss": 0.4983,
+      "num_input_tokens_seen": 1514688,
+      "step": 38,
+      "train_runtime": 162.9161,
+      "train_tokens_per_second": 9297.352
+    },
+    {
+      "epoch": 0.26666666666666666,
+      "grad_norm": 1.5963151454925537,
+      "learning_rate": 1.7716262975778548e-05,
+      "loss": 0.5577,
+      "num_input_tokens_seen": 1553248,
+      "step": 39,
+      "train_runtime": 166.1397,
+      "train_tokens_per_second": 9349.045
+    },
+    {
+      "epoch": 0.27350427350427353,
+      "grad_norm": 1.3875170946121216,
+      "learning_rate": 1.7647058823529414e-05,
+      "loss": 0.4095,
+      "num_input_tokens_seen": 1593120,
+      "step": 40,
+      "train_runtime": 169.4554,
+      "train_tokens_per_second": 9401.41
+    },
+    {
+      "epoch": 0.28034188034188035,
+      "grad_norm": 1.6025474071502686,
+      "learning_rate": 1.757785467128028e-05,
+      "loss": 0.5361,
+      "num_input_tokens_seen": 1628256,
+      "step": 41,
+      "train_runtime": 172.428,
+      "train_tokens_per_second": 9443.106
+    },
+    {
+      "epoch": 0.28717948717948716,
+      "grad_norm": 1.3887425661087036,
+      "learning_rate": 1.750865051903114e-05,
+      "loss": 0.4542,
+      "num_input_tokens_seen": 1674016,
+      "step": 42,
+      "train_runtime": 176.2327,
+      "train_tokens_per_second": 9498.898
+    },
+    {
+      "epoch": 0.294017094017094,
+      "grad_norm": 1.7252106666564941,
+      "learning_rate": 1.743944636678201e-05,
+      "loss": 0.4771,
+      "num_input_tokens_seen": 1706944,
+      "step": 43,
+      "train_runtime": 179.0413,
+      "train_tokens_per_second": 9533.802
+    },
+    {
+      "epoch": 0.30085470085470084,
+      "grad_norm": 1.5482698678970337,
+      "learning_rate": 1.7370242214532873e-05,
+      "loss": 0.4532,
+      "num_input_tokens_seen": 1748128,
+      "step": 44,
+      "train_runtime": 182.4655,
+      "train_tokens_per_second": 9580.595
+    },
+    {
+      "epoch": 0.3076923076923077,
+      "grad_norm": 1.6200870275497437,
+      "learning_rate": 1.730103806228374e-05,
+      "loss": 0.4823,
+      "num_input_tokens_seen": 1780032,
+      "step": 45,
+      "train_runtime": 185.183,
+      "train_tokens_per_second": 9612.285
+    },
+    {
+      "epoch": 0.3145299145299145,
+      "grad_norm": 1.4116896390914917,
+      "learning_rate": 1.7231833910034604e-05,
+      "loss": 0.4341,
+      "num_input_tokens_seen": 1820128,
+      "step": 46,
+      "train_runtime": 188.5419,
+      "train_tokens_per_second": 9653.708
+    },
+    {
+      "epoch": 0.3213675213675214,
+      "grad_norm": 1.6274487972259521,
+      "learning_rate": 1.716262975778547e-05,
+      "loss": 0.4756,
+      "num_input_tokens_seen": 1853536,
+      "step": 47,
+      "train_runtime": 191.3682,
+      "train_tokens_per_second": 9685.703
+    },
+    {
+      "epoch": 0.3282051282051282,
+      "grad_norm": 1.4281944036483765,
+      "learning_rate": 1.7093425605536332e-05,
+      "loss": 0.4333,
+      "num_input_tokens_seen": 1890560,
+      "step": 48,
+      "train_runtime": 194.4859,
+      "train_tokens_per_second": 9720.806
+    },
+    {
+      "epoch": 0.335042735042735,
+      "grad_norm": 1.411935567855835,
+      "learning_rate": 1.70242214532872e-05,
+      "loss": 0.4519,
+      "num_input_tokens_seen": 1939648,
+      "step": 49,
+      "train_runtime": 198.5911,
+      "train_tokens_per_second": 9767.042
+    },
+    {
+      "epoch": 0.3418803418803419,
+      "grad_norm": 1.4910825490951538,
+      "learning_rate": 1.6955017301038063e-05,
+      "loss": 0.4128,
+      "num_input_tokens_seen": 1971584,
+      "step": 50,
+      "train_runtime": 201.3116,
+      "train_tokens_per_second": 9793.692
+    },
+    {
+      "epoch": 0.3487179487179487,
+      "grad_norm": 1.67446768283844,
+      "learning_rate": 1.688581314878893e-05,
+      "loss": 0.5384,
+      "num_input_tokens_seen": 2021536,
+      "step": 51,
+      "train_runtime": 205.4602,
+      "train_tokens_per_second": 9839.062
+    },
+    {
+      "epoch": 0.35555555555555557,
+      "grad_norm": 1.500937581062317,
+      "learning_rate": 1.6816608996539795e-05,
+      "loss": 0.4733,
+      "num_input_tokens_seen": 2066656,
+      "step": 52,
+      "train_runtime": 209.327,
+      "train_tokens_per_second": 9872.859
+    },
+    {
+      "epoch": 0.3623931623931624,
+      "grad_norm": 1.6410099267959595,
+      "learning_rate": 1.6747404844290657e-05,
+      "loss": 0.4283,
+      "num_input_tokens_seen": 2100160,
+      "step": 53,
+      "train_runtime": 212.1628,
+      "train_tokens_per_second": 9898.815
+    },
+    {
+      "epoch": 0.36923076923076925,
+      "grad_norm": 1.507749080657959,
+      "learning_rate": 1.6678200692041523e-05,
+      "loss": 0.4679,
+      "num_input_tokens_seen": 2133888,
+      "step": 54,
+      "train_runtime": 215.0512,
+      "train_tokens_per_second": 9922.696
+    },
+    {
+      "epoch": 0.37606837606837606,
+      "grad_norm": 1.5565907955169678,
+      "learning_rate": 1.6608996539792388e-05,
+      "loss": 0.43,
+      "num_input_tokens_seen": 2169472,
+      "step": 55,
+      "train_runtime": 218.0654,
+      "train_tokens_per_second": 9948.72
+    },
+    {
+      "epoch": 0.38290598290598293,
+      "grad_norm": 1.380922555923462,
+      "learning_rate": 1.6539792387543254e-05,
+      "loss": 0.4838,
+      "num_input_tokens_seen": 2214720,
+      "step": 56,
+      "train_runtime": 221.8928,
+      "train_tokens_per_second": 9981.036
+    },
+    {
+      "epoch": 0.38974358974358975,
+      "grad_norm": 1.6342693567276,
+      "learning_rate": 1.647058823529412e-05,
+      "loss": 0.4654,
+      "num_input_tokens_seen": 2251648,
+      "step": 57,
+      "train_runtime": 225.0239,
+      "train_tokens_per_second": 10006.263
+    },
+    {
+      "epoch": 0.39658119658119656,
+      "grad_norm": 1.5192848443984985,
+      "learning_rate": 1.6401384083044985e-05,
+      "loss": 0.5026,
+      "num_input_tokens_seen": 2284064,
+      "step": 58,
+      "train_runtime": 227.7635,
+      "train_tokens_per_second": 10028.225
+    },
+    {
+      "epoch": 0.40341880341880343,
+      "grad_norm": 1.5925633907318115,
+      "learning_rate": 1.6332179930795848e-05,
+      "loss": 0.4585,
+      "num_input_tokens_seen": 2312512,
+      "step": 59,
+      "train_runtime": 230.219,
+      "train_tokens_per_second": 10044.836
+    },
+    {
+      "epoch": 0.41025641025641024,
+      "grad_norm": 1.4670740365982056,
+      "learning_rate": 1.6262975778546713e-05,
+      "loss": 0.434,
+      "num_input_tokens_seen": 2348320,
+      "step": 60,
+      "train_runtime": 233.2567,
+      "train_tokens_per_second": 10067.536
+    },
+    {
+      "epoch": 0.41025641025641024,
+      "eval_loss": 0.4576193392276764,
+      "eval_runtime": 5.7904,
+      "eval_samples_per_second": 172.354,
+      "eval_steps_per_second": 5.526,
+      "num_input_tokens_seen": 2348320,
+      "step": 60
+    },
+    {
+      "epoch": 0.4170940170940171,
+      "grad_norm": 1.5287467241287231,
+      "learning_rate": 1.619377162629758e-05,
+      "loss": 0.4578,
+      "num_input_tokens_seen": 2391232,
+      "step": 61,
+      "train_runtime": 259.4978,
+      "train_tokens_per_second": 9214.845
+    },
+    {
+      "epoch": 0.4239316239316239,
+      "grad_norm": 1.507736325263977,
+      "learning_rate": 1.6124567474048444e-05,
+      "loss": 0.4129,
+      "num_input_tokens_seen": 2428096,
+      "step": 62,
+      "train_runtime": 262.5473,
+      "train_tokens_per_second": 9248.225
+    },
+    {
+      "epoch": 0.4307692307692308,
+      "grad_norm": 1.5525883436203003,
+      "learning_rate": 1.605536332179931e-05,
+      "loss": 0.4635,
+      "num_input_tokens_seen": 2457504,
+      "step": 63,
+      "train_runtime": 265.0656,
+      "train_tokens_per_second": 9271.305
+    },
+    {
+      "epoch": 0.4376068376068376,
+      "grad_norm": 1.6987532377243042,
+      "learning_rate": 1.5986159169550176e-05,
+      "loss": 0.5224,
+      "num_input_tokens_seen": 2490496,
+      "step": 64,
+      "train_runtime": 267.8475,
+      "train_tokens_per_second": 9298.187
+    },
+    {
+      "epoch": 0.4444444444444444,
+      "grad_norm": 1.3846668004989624,
+      "learning_rate": 1.5916955017301038e-05,
+      "loss": 0.4179,
+      "num_input_tokens_seen": 2529888,
+      "step": 65,
+      "train_runtime": 271.0955,
+      "train_tokens_per_second": 9332.093
+    },
+    {
+      "epoch": 0.4512820512820513,
+      "grad_norm": 1.4100641012191772,
+      "learning_rate": 1.5847750865051904e-05,
+      "loss": 0.4574,
+      "num_input_tokens_seen": 2563744,
+      "step": 66,
+      "train_runtime": 273.9696,
+      "train_tokens_per_second": 9357.767
+    },
+    {
+      "epoch": 0.4581196581196581,
+      "grad_norm": 1.4976650476455688,
+      "learning_rate": 1.577854671280277e-05,
+      "loss": 0.4836,
+      "num_input_tokens_seen": 2607584,
+      "step": 67,
+      "train_runtime": 277.6271,
+      "train_tokens_per_second": 9392.398
+    },
+    {
+      "epoch": 0.46495726495726497,
+      "grad_norm": 1.4736229181289673,
+      "learning_rate": 1.5709342560553635e-05,
+      "loss": 0.403,
+      "num_input_tokens_seen": 2650848,
+      "step": 68,
+      "train_runtime": 281.2134,
+      "train_tokens_per_second": 9426.463
+    },
+    {
+      "epoch": 0.4717948717948718,
+      "grad_norm": 1.4522730112075806,
+      "learning_rate": 1.56401384083045e-05,
+      "loss": 0.4548,
+      "num_input_tokens_seen": 2678816,
+      "step": 69,
+      "train_runtime": 283.5945,
+      "train_tokens_per_second": 9445.936
+    },
+    {
+      "epoch": 0.47863247863247865,
+      "grad_norm": 1.5067986249923706,
+      "learning_rate": 1.5570934256055366e-05,
+      "loss": 0.4432,
+      "num_input_tokens_seen": 2717472,
+      "step": 70,
+      "train_runtime": 286.8288,
+      "train_tokens_per_second": 9474.195
+    },
+    {
+      "epoch": 0.48547008547008547,
+      "grad_norm": 1.542819857597351,
+      "learning_rate": 1.550173010380623e-05,
+      "loss": 0.4915,
+      "num_input_tokens_seen": 2754592,
+      "step": 71,
+      "train_runtime": 289.9357,
+      "train_tokens_per_second": 9500.7
+    },
+    {
+      "epoch": 0.49230769230769234,
+      "grad_norm": 1.4620888233184814,
+      "learning_rate": 1.5432525951557094e-05,
+      "loss": 0.3876,
+      "num_input_tokens_seen": 2799296,
+      "step": 72,
+      "train_runtime": 293.7189,
+      "train_tokens_per_second": 9530.527
+    },
+    {
+      "epoch": 0.49914529914529915,
+      "grad_norm": 1.526044487953186,
+      "learning_rate": 1.536332179930796e-05,
+      "loss": 0.4648,
+      "num_input_tokens_seen": 2828800,
+      "step": 73,
+      "train_runtime": 296.237,
+      "train_tokens_per_second": 9549.112
+    },
+    {
+      "epoch": 0.505982905982906,
+      "grad_norm": 1.4033782482147217,
+      "learning_rate": 1.5294117647058822e-05,
+      "loss": 0.4114,
+      "num_input_tokens_seen": 2876800,
+      "step": 74,
+      "train_runtime": 300.199,
+      "train_tokens_per_second": 9582.978
+    },
+    {
+      "epoch": 0.5128205128205128,
+      "grad_norm": 1.6525285243988037,
+      "learning_rate": 1.522491349480969e-05,
+      "loss": 0.4523,
+      "num_input_tokens_seen": 2924736,
+      "step": 75,
+      "train_runtime": 304.1673,
+      "train_tokens_per_second": 9615.551
+    },
+    {
+      "epoch": 0.5196581196581197,
+      "grad_norm": 1.3782298564910889,
+      "learning_rate": 1.5155709342560554e-05,
+      "loss": 0.416,
+      "num_input_tokens_seen": 2969600,
+      "step": 76,
+      "train_runtime": 307.8767,
+      "train_tokens_per_second": 9645.42
+    },
+    {
+      "epoch": 0.5264957264957265,
+      "grad_norm": 1.2422502040863037,
+      "learning_rate": 1.5086505190311421e-05,
+      "loss": 0.3851,
+      "num_input_tokens_seen": 3030912,
+      "step": 77,
+      "train_runtime": 313.0558,
+      "train_tokens_per_second": 9681.699
+    },
+    {
+      "epoch": 0.5333333333333333,
+      "grad_norm": 1.7053712606430054,
+      "learning_rate": 1.5017301038062285e-05,
+      "loss": 0.5167,
+      "num_input_tokens_seen": 3069312,
+      "step": 78,
+      "train_runtime": 316.2163,
+      "train_tokens_per_second": 9706.369
+    },
+    {
+      "epoch": 0.5401709401709401,
+      "grad_norm": 1.57555091381073,
+      "learning_rate": 1.494809688581315e-05,
+      "loss": 0.5093,
+      "num_input_tokens_seen": 3099456,
+      "step": 79,
+      "train_runtime": 318.7901,
+      "train_tokens_per_second": 9722.56
+    },
+    {
+      "epoch": 0.5470085470085471,
+      "grad_norm": 1.528808355331421,
+      "learning_rate": 1.4878892733564014e-05,
+      "loss": 0.5011,
+      "num_input_tokens_seen": 3137024,
+      "step": 80,
+      "train_runtime": 321.9542,
+      "train_tokens_per_second": 9743.697
+    },
+    {
+      "epoch": 0.5538461538461539,
+      "grad_norm": 1.358077049255371,
+      "learning_rate": 1.480968858131488e-05,
+      "loss": 0.4043,
+      "num_input_tokens_seen": 3183648,
+      "step": 81,
+      "train_runtime": 325.824,
+      "train_tokens_per_second": 9771.066
+    },
+    {
+      "epoch": 0.5606837606837607,
+      "grad_norm": 1.7505600452423096,
+      "learning_rate": 1.4740484429065744e-05,
+      "loss": 0.4766,
+      "num_input_tokens_seen": 3214592,
+      "step": 82,
+      "train_runtime": 328.4491,
+      "train_tokens_per_second": 9787.183
+    },
+    {
+      "epoch": 0.5675213675213675,
+      "grad_norm": 1.6043431758880615,
+      "learning_rate": 1.4671280276816611e-05,
+      "loss": 0.4837,
+      "num_input_tokens_seen": 3242720,
+      "step": 83,
+      "train_runtime": 330.8621,
+      "train_tokens_per_second": 9800.82
+    },
+    {
+      "epoch": 0.5743589743589743,
+      "grad_norm": 1.3557583093643188,
+      "learning_rate": 1.4602076124567475e-05,
+      "loss": 0.4051,
+      "num_input_tokens_seen": 3278816,
+      "step": 84,
+      "train_runtime": 333.921,
+      "train_tokens_per_second": 9819.136
+    },
+    {
+      "epoch": 0.5811965811965812,
+      "grad_norm": 1.4810553789138794,
+      "learning_rate": 1.4532871972318341e-05,
+      "loss": 0.4385,
+      "num_input_tokens_seen": 3315008,
+      "step": 85,
+      "train_runtime": 336.9782,
+      "train_tokens_per_second": 9837.454
+    },
+    {
+      "epoch": 0.588034188034188,
+      "grad_norm": 1.520869493484497,
+      "learning_rate": 1.4463667820069205e-05,
+      "loss": 0.4663,
+      "num_input_tokens_seen": 3349152,
+      "step": 86,
+      "train_runtime": 339.8919,
+      "train_tokens_per_second": 9853.58
+    },
+    {
+      "epoch": 0.5948717948717949,
+      "grad_norm": 1.5246199369430542,
+      "learning_rate": 1.439446366782007e-05,
+      "loss": 0.4519,
+      "num_input_tokens_seen": 3377696,
+      "step": 87,
+      "train_runtime": 342.3568,
+      "train_tokens_per_second": 9866.012
+    },
+    {
+      "epoch": 0.6017094017094017,
+      "grad_norm": 1.554733157157898,
+      "learning_rate": 1.4325259515570935e-05,
+      "loss": 0.5046,
+      "num_input_tokens_seen": 3414656,
+      "step": 88,
+      "train_runtime": 345.4265,
+      "train_tokens_per_second": 9885.332
+    },
+    {
+      "epoch": 0.6085470085470085,
+      "grad_norm": 1.571320652961731,
+      "learning_rate": 1.4256055363321802e-05,
+      "loss": 0.4736,
+      "num_input_tokens_seen": 3446080,
+      "step": 89,
+      "train_runtime": 348.0842,
+      "train_tokens_per_second": 9900.134
+    },
+    {
+      "epoch": 0.6153846153846154,
+      "grad_norm": 1.624376893043518,
+      "learning_rate": 1.4186851211072666e-05,
+      "loss": 0.4512,
+      "num_input_tokens_seen": 3475168,
+      "step": 90,
+      "train_runtime": 350.5802,
+      "train_tokens_per_second": 9912.62
+    },
+    {
+      "epoch": 0.6153846153846154,
+      "eval_loss": 0.4554298520088196,
+      "eval_runtime": 5.7133,
+      "eval_samples_per_second": 174.681,
+      "eval_steps_per_second": 5.601,
+      "num_input_tokens_seen": 3475168,
+      "step": 90
+    },
+    {
+      "epoch": 0.6222222222222222,
+      "grad_norm": 1.713934063911438,
+      "learning_rate": 1.4117647058823532e-05,
+      "loss": 0.4519,
+      "num_input_tokens_seen": 3500032,
+      "step": 91,
+      "train_runtime": 376.436,
+      "train_tokens_per_second": 9297.815
+    },
+    {
+      "epoch": 0.629059829059829,
+      "grad_norm": 1.5343419313430786,
+      "learning_rate": 1.4048442906574396e-05,
+      "loss": 0.4543,
+      "num_input_tokens_seen": 3535136,
+      "step": 92,
+      "train_runtime": 379.3304,
+      "train_tokens_per_second": 9319.412
+    },
+    {
+      "epoch": 0.6358974358974359,
+      "grad_norm": 1.4790829420089722,
+      "learning_rate": 1.3979238754325261e-05,
+      "loss": 0.439,
+      "num_input_tokens_seen": 3564192,
+      "step": 93,
+      "train_runtime": 381.791,
+      "train_tokens_per_second": 9335.453
+    },
+    {
+      "epoch": 0.6427350427350428,
+      "grad_norm": 1.435154914855957,
+      "learning_rate": 1.3910034602076125e-05,
+      "loss": 0.4486,
+      "num_input_tokens_seen": 3600640,
+      "step": 94,
+      "train_runtime": 384.8183,
+      "train_tokens_per_second": 9356.727
+    },
+    {
+      "epoch": 0.6495726495726496,
+      "grad_norm": 1.7755075693130493,
+      "learning_rate": 1.3840830449826989e-05,
+      "loss": 0.486,
+      "num_input_tokens_seen": 3634336,
+      "step": 95,
+      "train_runtime": 387.6562,
+      "train_tokens_per_second": 9375.153
+    },
+    {
+      "epoch": 0.6564102564102564,
+      "grad_norm": 1.542274832725525,
+      "learning_rate": 1.3771626297577856e-05,
+      "loss": 0.476,
+      "num_input_tokens_seen": 3667328,
+      "step": 96,
+      "train_runtime": 390.429,
+      "train_tokens_per_second": 9393.073
+    },
+    {
+      "epoch": 0.6632478632478632,
+      "grad_norm": 1.5652968883514404,
+      "learning_rate": 1.370242214532872e-05,
+      "loss": 0.4655,
+      "num_input_tokens_seen": 3700224,
+      "step": 97,
+      "train_runtime": 393.1811,
+      "train_tokens_per_second": 9410.992
+    },
+    {
+      "epoch": 0.67008547008547,
+      "grad_norm": 1.433912992477417,
+      "learning_rate": 1.3633217993079586e-05,
+      "loss": 0.4575,
+      "num_input_tokens_seen": 3744448,
+      "step": 98,
+      "train_runtime": 396.8421,
+      "train_tokens_per_second": 9435.611
+    },
+    {
+      "epoch": 0.676923076923077,
+      "grad_norm": 1.5182974338531494,
+      "learning_rate": 1.356401384083045e-05,
+      "loss": 0.4643,
+      "num_input_tokens_seen": 3779936,
+      "step": 99,
+      "train_runtime": 399.8119,
+      "train_tokens_per_second": 9454.287
+    },
+    {
+      "epoch": 0.6837606837606838,
+      "grad_norm": 1.5050530433654785,
+      "learning_rate": 1.3494809688581316e-05,
+      "loss": 0.4509,
+      "num_input_tokens_seen": 3828608,
+      "step": 100,
+      "train_runtime": 403.7792,
+      "train_tokens_per_second": 9481.935
+    },
+    {
+      "epoch": 0.6905982905982906,
+      "grad_norm": 1.3182984590530396,
+      "learning_rate": 1.342560553633218e-05,
+      "loss": 0.4183,
+      "num_input_tokens_seen": 3881632,
+      "step": 101,
+      "train_runtime": 408.1485,
+      "train_tokens_per_second": 9510.342
+    },
+    {
+      "epoch": 0.6974358974358974,
+      "grad_norm": 1.4368970394134521,
+      "learning_rate": 1.3356401384083047e-05,
+      "loss": 0.5363,
+      "num_input_tokens_seen": 3911200,
+      "step": 102,
+      "train_runtime": 410.6666,
+      "train_tokens_per_second": 9524.028
+    },
+    {
+      "epoch": 0.7042735042735043,
+      "grad_norm": 1.702073335647583,
+      "learning_rate": 1.3287197231833911e-05,
+      "loss": 0.4391,
+      "num_input_tokens_seen": 3951424,
+      "step": 103,
+      "train_runtime": 414.0044,
+      "train_tokens_per_second": 9544.4
+    },
+    {
+      "epoch": 0.7111111111111111,
+      "grad_norm": 1.484083652496338,
+      "learning_rate": 1.3217993079584777e-05,
+      "loss": 0.4622,
+      "num_input_tokens_seen": 3984992,
+      "step": 104,
+      "train_runtime": 416.8268,
+      "train_tokens_per_second": 9560.306
+    },
+    {
+      "epoch": 0.717948717948718,
+      "grad_norm": 1.5703684091567993,
+      "learning_rate": 1.314878892733564e-05,
+      "loss": 0.4656,
+      "num_input_tokens_seen": 4022400,
+      "step": 105,
+      "train_runtime": 419.9371,
+      "train_tokens_per_second": 9578.578
+    },
+    {
+      "epoch": 0.7247863247863248,
+      "grad_norm": 1.617999792098999,
+      "learning_rate": 1.3079584775086506e-05,
+      "loss": 0.475,
+      "num_input_tokens_seen": 4056160,
+      "step": 106,
+      "train_runtime": 422.7581,
+      "train_tokens_per_second": 9594.517
+    },
+    {
+      "epoch": 0.7316239316239316,
+      "grad_norm": 1.7811297178268433,
+      "learning_rate": 1.301038062283737e-05,
+      "loss": 0.517,
+      "num_input_tokens_seen": 4092928,
+      "step": 107,
+      "train_runtime": 425.86,
+      "train_tokens_per_second": 9610.97
+    },
+    {
+      "epoch": 0.7384615384615385,
+      "grad_norm": 1.3520305156707764,
+      "learning_rate": 1.2941176470588238e-05,
+      "loss": 0.4112,
+      "num_input_tokens_seen": 4153440,
+      "step": 108,
+      "train_runtime": 431.5262,
+      "train_tokens_per_second": 9625.002
+    },
+    {
+      "epoch": 0.7452991452991453,
+      "grad_norm": 1.4912413358688354,
+      "learning_rate": 1.2871972318339102e-05,
+      "loss": 0.4536,
+      "num_input_tokens_seen": 4187744,
+      "step": 109,
+      "train_runtime": 434.4156,
+      "train_tokens_per_second": 9639.948
+    },
+    {
+      "epoch": 0.7521367521367521,
+      "grad_norm": 1.4879883527755737,
+      "learning_rate": 1.2802768166089967e-05,
+      "loss": 0.4744,
+      "num_input_tokens_seen": 4226496,
+      "step": 110,
+      "train_runtime": 437.6348,
+      "train_tokens_per_second": 9657.586
+    },
+    {
+      "epoch": 0.7589743589743589,
+      "grad_norm": 1.5085774660110474,
+      "learning_rate": 1.2733564013840831e-05,
+      "loss": 0.5143,
+      "num_input_tokens_seen": 4262176,
+      "step": 111,
+      "train_runtime": 440.6057,
+      "train_tokens_per_second": 9673.447
+    },
+    {
+      "epoch": 0.7658119658119659,
+      "grad_norm": 1.4866958856582642,
+      "learning_rate": 1.2664359861591697e-05,
+      "loss": 0.3955,
+      "num_input_tokens_seen": 4299200,
+      "step": 112,
+      "train_runtime": 443.6903,
+      "train_tokens_per_second": 9689.643
+    },
+    {
+      "epoch": 0.7726495726495727,
+      "grad_norm": 1.7484487295150757,
+      "learning_rate": 1.259515570934256e-05,
+      "loss": 0.5021,
+      "num_input_tokens_seen": 4339904,
+      "step": 113,
+      "train_runtime": 447.0564,
+      "train_tokens_per_second": 9707.733
+    },
+    {
+      "epoch": 0.7794871794871795,
+      "grad_norm": 1.5065933465957642,
+      "learning_rate": 1.2525951557093428e-05,
+      "loss": 0.4726,
+      "num_input_tokens_seen": 4369728,
+      "step": 114,
+      "train_runtime": 449.5826,
+      "train_tokens_per_second": 9719.522
+    },
+    {
+      "epoch": 0.7863247863247863,
+      "grad_norm": 1.4008255004882812,
+      "learning_rate": 1.2456747404844292e-05,
+      "loss": 0.4908,
+      "num_input_tokens_seen": 4407936,
+      "step": 115,
+      "train_runtime": 452.8133,
+      "train_tokens_per_second": 9734.554
+    },
+    {
+      "epoch": 0.7931623931623931,
+      "grad_norm": 1.5314161777496338,
+      "learning_rate": 1.2387543252595158e-05,
+      "loss": 0.5017,
+      "num_input_tokens_seen": 4439520,
+      "step": 116,
+      "train_runtime": 455.4893,
+      "train_tokens_per_second": 9746.705
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 1.3641289472579956,
+      "learning_rate": 1.2318339100346022e-05,
+      "loss": 0.4654,
+      "num_input_tokens_seen": 4475552,
+      "step": 117,
+      "train_runtime": 458.5097,
+      "train_tokens_per_second": 9761.084
+    },
+    {
+      "epoch": 0.8068376068376069,
+      "grad_norm": 1.6261942386627197,
+      "learning_rate": 1.2249134948096886e-05,
+      "loss": 0.4088,
+      "num_input_tokens_seen": 4530112,
+      "step": 118,
+      "train_runtime": 463.158,
+      "train_tokens_per_second": 9780.921
+    },
+    {
+      "epoch": 0.8136752136752137,
+      "grad_norm": 1.4655214548110962,
+      "learning_rate": 1.2179930795847751e-05,
+      "loss": 0.4371,
+      "num_input_tokens_seen": 4559936,
+      "step": 119,
+      "train_runtime": 465.6803,
+      "train_tokens_per_second": 9791.988
+    },
+    {
+      "epoch": 0.8205128205128205,
+      "grad_norm": 1.5850762128829956,
+      "learning_rate": 1.2110726643598615e-05,
+      "loss": 0.4721,
+      "num_input_tokens_seen": 4601120,
+      "step": 120,
+      "train_runtime": 469.1647,
+      "train_tokens_per_second": 9807.046
+    },
+    {
+      "epoch": 0.8205128205128205,
+      "eval_loss": 0.4540318250656128,
+      "eval_runtime": 5.7037,
+      "eval_samples_per_second": 174.974,
+      "eval_steps_per_second": 5.61,
+      "num_input_tokens_seen": 4601120,
+      "step": 120
+    },
+    {
+      "epoch": 0.8273504273504273,
+      "grad_norm": 1.4847512245178223,
+      "learning_rate": 1.2041522491349483e-05,
+      "loss": 0.4827,
+      "num_input_tokens_seen": 4635648,
+      "step": 121,
+      "train_runtime": 496.058,
+      "train_tokens_per_second": 9344.972
+    },
+    {
+      "epoch": 0.8341880341880342,
+      "grad_norm": 1.4828547239303589,
+      "learning_rate": 1.1972318339100347e-05,
+      "loss": 0.4849,
+      "num_input_tokens_seen": 4674560,
+      "step": 122,
+      "train_runtime": 499.2917,
+      "train_tokens_per_second": 9362.382
+    },
+    {
+      "epoch": 0.841025641025641,
+      "grad_norm": 1.5140576362609863,
+      "learning_rate": 1.1903114186851212e-05,
+      "loss": 0.521,
+      "num_input_tokens_seen": 4717344,
+      "step": 123,
+      "train_runtime": 502.8538,
+      "train_tokens_per_second": 9381.145
+    },
+    {
+      "epoch": 0.8478632478632478,
+      "grad_norm": 1.1721268892288208,
+      "learning_rate": 1.1833910034602076e-05,
+      "loss": 0.3616,
+      "num_input_tokens_seen": 4778304,
+      "step": 124,
+      "train_runtime": 507.9229,
+      "train_tokens_per_second": 9407.537
+    },
+    {
+      "epoch": 0.8547008547008547,
+      "grad_norm": 1.3763750791549683,
+      "learning_rate": 1.1764705882352942e-05,
+      "loss": 0.4127,
+      "num_input_tokens_seen": 4814464,
+      "step": 125,
+      "train_runtime": 510.9361,
+      "train_tokens_per_second": 9422.83
+    },
+    {
+      "epoch": 0.8615384615384616,
+      "grad_norm": 1.4982727766036987,
+      "learning_rate": 1.1695501730103806e-05,
+      "loss": 0.433,
+      "num_input_tokens_seen": 4860768,
+      "step": 126,
+      "train_runtime": 514.8181,
+      "train_tokens_per_second": 9441.719
+    },
+    {
+      "epoch": 0.8683760683760684,
+      "grad_norm": 1.4128450155258179,
+      "learning_rate": 1.1626297577854673e-05,
+      "loss": 0.5173,
+      "num_input_tokens_seen": 4900512,
+      "step": 127,
+      "train_runtime": 518.1154,
+      "train_tokens_per_second": 9458.342
+    },
+    {
+      "epoch": 0.8752136752136752,
+      "grad_norm": 1.564786434173584,
+      "learning_rate": 1.1557093425605537e-05,
+      "loss": 0.4603,
+      "num_input_tokens_seen": 4938752,
+      "step": 128,
+      "train_runtime": 521.2991,
+      "train_tokens_per_second": 9473.931
+    },
+    {
+      "epoch": 0.882051282051282,
+      "grad_norm": 1.5176303386688232,
+      "learning_rate": 1.1487889273356403e-05,
+      "loss": 0.4633,
+      "num_input_tokens_seen": 4979584,
+      "step": 129,
+      "train_runtime": 524.6889,
+      "train_tokens_per_second": 9490.545
+    },
+    {
+      "epoch": 0.8888888888888888,
+      "grad_norm": 1.5156581401824951,
+      "learning_rate": 1.1418685121107267e-05,
+      "loss": 0.4556,
+      "num_input_tokens_seen": 5017216,
+      "step": 130,
+      "train_runtime": 527.841,
+      "train_tokens_per_second": 9505.166
+    },
+    {
+      "epoch": 0.8957264957264958,
+      "grad_norm": 1.4633028507232666,
+      "learning_rate": 1.1349480968858132e-05,
+      "loss": 0.3931,
+      "num_input_tokens_seen": 5054016,
+      "step": 131,
+      "train_runtime": 530.9057,
+      "train_tokens_per_second": 9519.612
+    },
+    {
+      "epoch": 0.9025641025641026,
+      "grad_norm": 1.8066245317459106,
+      "learning_rate": 1.1280276816608996e-05,
+      "loss": 0.4996,
+      "num_input_tokens_seen": 5080544,
+      "step": 132,
+      "train_runtime": 533.1694,
+      "train_tokens_per_second": 9528.95
+    },
+    {
+      "epoch": 0.9094017094017094,
+      "grad_norm": 1.4042011499404907,
+      "learning_rate": 1.1211072664359864e-05,
+      "loss": 0.4592,
+      "num_input_tokens_seen": 5116960,
+      "step": 133,
+      "train_runtime": 536.2021,
+      "train_tokens_per_second": 9542.969
+    },
+    {
+      "epoch": 0.9162393162393162,
+      "grad_norm": 1.4731768369674683,
+      "learning_rate": 1.1141868512110728e-05,
+      "loss": 0.4417,
+      "num_input_tokens_seen": 5152288,
+      "step": 134,
+      "train_runtime": 539.1624,
+      "train_tokens_per_second": 9556.096
+    },
+    {
+      "epoch": 0.9230769230769231,
+      "grad_norm": 1.2772272825241089,
+      "learning_rate": 1.1072664359861593e-05,
+      "loss": 0.4064,
+      "num_input_tokens_seen": 5204448,
+      "step": 135,
+      "train_runtime": 543.4474,
+      "train_tokens_per_second": 9576.729
+    },
+    {
+      "epoch": 0.9299145299145299,
+      "grad_norm": 1.6605689525604248,
+      "learning_rate": 1.1003460207612457e-05,
+      "loss": 0.4421,
+      "num_input_tokens_seen": 5255488,
+      "step": 136,
+      "train_runtime": 547.7055,
+      "train_tokens_per_second": 9595.463
+    },
+    {
+      "epoch": 0.9367521367521368,
+      "grad_norm": 1.5701137781143188,
+      "learning_rate": 1.0934256055363323e-05,
+      "loss": 0.4587,
+      "num_input_tokens_seen": 5282848,
+      "step": 137,
+      "train_runtime": 550.0393,
+      "train_tokens_per_second": 9604.491
+    },
+    {
+      "epoch": 0.9435897435897436,
+      "grad_norm": 1.4943722486495972,
+      "learning_rate": 1.0865051903114187e-05,
+      "loss": 0.4801,
+      "num_input_tokens_seen": 5319456,
+      "step": 138,
+      "train_runtime": 553.1186,
+      "train_tokens_per_second": 9617.207
+    },
+    {
+      "epoch": 0.9504273504273504,
+      "grad_norm": 1.5468835830688477,
+      "learning_rate": 1.0795847750865054e-05,
+      "loss": 0.4504,
+      "num_input_tokens_seen": 5359296,
+      "step": 139,
+      "train_runtime": 556.4596,
+      "train_tokens_per_second": 9631.06
+    },
+    {
+      "epoch": 0.9572649572649573,
+      "grad_norm": 1.409867763519287,
+      "learning_rate": 1.0726643598615918e-05,
+      "loss": 0.4377,
+      "num_input_tokens_seen": 5416672,
+      "step": 140,
+      "train_runtime": 561.16,
+      "train_tokens_per_second": 9652.634
+    },
+    {
+      "epoch": 0.9641025641025641,
+      "grad_norm": 1.3638184070587158,
+      "learning_rate": 1.0657439446366782e-05,
+      "loss": 0.3818,
+      "num_input_tokens_seen": 5498976,
+      "step": 141,
+      "train_runtime": 568.811,
+      "train_tokens_per_second": 9667.493
+    },
+    {
+      "epoch": 0.9709401709401709,
+      "grad_norm": 1.4294383525848389,
+      "learning_rate": 1.0588235294117648e-05,
+      "loss": 0.4468,
+      "num_input_tokens_seen": 5532992,
+      "step": 142,
+      "train_runtime": 571.6619,
+      "train_tokens_per_second": 9678.784
+    },
+    {
+      "epoch": 0.9777777777777777,
+      "grad_norm": 1.3784202337265015,
+      "learning_rate": 1.0519031141868512e-05,
+      "loss": 0.4106,
+      "num_input_tokens_seen": 5585280,
+      "step": 143,
+      "train_runtime": 576.0752,
+      "train_tokens_per_second": 9695.4
+    },
+    {
+      "epoch": 0.9846153846153847,
+      "grad_norm": 1.6159602403640747,
+      "learning_rate": 1.0449826989619377e-05,
+      "loss": 0.4474,
+      "num_input_tokens_seen": 5622976,
+      "step": 144,
+      "train_runtime": 579.2137,
+      "train_tokens_per_second": 9707.948
+    },
+    {
+      "epoch": 0.9914529914529915,
+      "grad_norm": 1.510590672492981,
+      "learning_rate": 1.0380622837370241e-05,
+      "loss": 0.4664,
+      "num_input_tokens_seen": 5661472,
+      "step": 145,
+      "train_runtime": 582.3983,
+      "train_tokens_per_second": 9720.962
+    },
+    {
+      "epoch": 0.9982905982905983,
+      "grad_norm": 1.3742411136627197,
+      "learning_rate": 1.0311418685121109e-05,
+      "loss": 0.4175,
+      "num_input_tokens_seen": 5698240,
+      "step": 146,
+      "train_runtime": 585.4675,
+      "train_tokens_per_second": 9732.803
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 2.935847282409668,
+      "learning_rate": 1.0242214532871973e-05,
+      "loss": 0.4182,
+      "num_input_tokens_seen": 5709310,
+      "step": 147,
+      "train_runtime": 587.7866,
+      "train_tokens_per_second": 9713.235
+    },
+    {
+      "epoch": 1.0068376068376068,
+      "grad_norm": 1.3502393960952759,
+      "learning_rate": 1.0173010380622838e-05,
+      "loss": 0.3852,
+      "num_input_tokens_seen": 5746654,
+      "step": 148,
+      "train_runtime": 590.9756,
+      "train_tokens_per_second": 9724.013
+    },
+    {
+      "epoch": 1.0136752136752136,
+      "grad_norm": 1.4006216526031494,
+      "learning_rate": 1.0103806228373702e-05,
+      "loss": 0.3924,
+      "num_input_tokens_seen": 5777726,
+      "step": 149,
+      "train_runtime": 593.6371,
+      "train_tokens_per_second": 9732.758
+    },
+    {
+      "epoch": 1.0205128205128204,
+      "grad_norm": 1.4392833709716797,
+      "learning_rate": 1.0034602076124568e-05,
+      "loss": 0.4426,
+      "num_input_tokens_seen": 5812574,
+      "step": 150,
+      "train_runtime": 596.583,
+      "train_tokens_per_second": 9743.111
+    },
+    {
+      "epoch": 1.0205128205128204,
+      "eval_loss": 0.45281028747558594,
+      "eval_runtime": 6.2291,
+      "eval_samples_per_second": 160.216,
+      "eval_steps_per_second": 5.137,
+      "num_input_tokens_seen": 5812574,
+      "step": 150
+    },
+    {
+      "epoch": 1.0273504273504273,
+      "grad_norm": 1.3533525466918945,
+      "learning_rate": 9.965397923875434e-06,
+      "loss": 0.4033,
+      "num_input_tokens_seen": 5855934,
+      "step": 151,
+      "train_runtime": 626.0707,
+      "train_tokens_per_second": 9353.471
+    },
+    {
+      "epoch": 1.0341880341880343,
+      "grad_norm": 1.3957024812698364,
+      "learning_rate": 9.896193771626298e-06,
+      "loss": 0.4154,
+      "num_input_tokens_seen": 5883870,
+      "step": 152,
+      "train_runtime": 628.4362,
+      "train_tokens_per_second": 9362.717
+    },
+    {
+      "epoch": 1.041025641025641,
+      "grad_norm": 1.2741293907165527,
+      "learning_rate": 9.826989619377163e-06,
+      "loss": 0.4497,
+      "num_input_tokens_seen": 5932190,
+      "step": 153,
+      "train_runtime": 632.3789,
+      "train_tokens_per_second": 9380.752
+    },
+    {
+      "epoch": 1.047863247863248,
+      "grad_norm": 1.3246592283248901,
+      "learning_rate": 9.757785467128029e-06,
+      "loss": 0.3884,
+      "num_input_tokens_seen": 5981854,
+      "step": 154,
+      "train_runtime": 636.5436,
+      "train_tokens_per_second": 9397.398
+    },
+    {
+      "epoch": 1.0547008547008547,
+      "grad_norm": 1.3506548404693604,
+      "learning_rate": 9.688581314878893e-06,
+      "loss": 0.3601,
+      "num_input_tokens_seen": 6030590,
+      "step": 155,
+      "train_runtime": 640.5068,
+      "train_tokens_per_second": 9415.341
+    },
+    {
+      "epoch": 1.0615384615384615,
+      "grad_norm": 1.298898458480835,
+      "learning_rate": 9.619377162629759e-06,
+      "loss": 0.4158,
+      "num_input_tokens_seen": 6072030,
+      "step": 156,
+      "train_runtime": 643.9463,
+      "train_tokens_per_second": 9429.405
+    },
+    {
+      "epoch": 1.0683760683760684,
+      "grad_norm": 1.4321287870407104,
+      "learning_rate": 9.550173010380624e-06,
+      "loss": 0.4496,
+      "num_input_tokens_seen": 6107006,
+      "step": 157,
+      "train_runtime": 646.8903,
+      "train_tokens_per_second": 9440.559
+    },
+    {
+      "epoch": 1.0752136752136752,
+      "grad_norm": 1.362835168838501,
+      "learning_rate": 9.480968858131488e-06,
+      "loss": 0.4082,
+      "num_input_tokens_seen": 6153086,
+      "step": 158,
+      "train_runtime": 650.7394,
+      "train_tokens_per_second": 9455.529
+    },
+    {
+      "epoch": 1.082051282051282,
+      "grad_norm": 1.332863211631775,
+      "learning_rate": 9.411764705882354e-06,
+      "loss": 0.4156,
+      "num_input_tokens_seen": 6192222,
+      "step": 159,
+      "train_runtime": 653.9789,
+      "train_tokens_per_second": 9468.535
+    },
+    {
+      "epoch": 1.0888888888888888,
+      "grad_norm": 1.3329721689224243,
+      "learning_rate": 9.34256055363322e-06,
+      "loss": 0.4446,
+      "num_input_tokens_seen": 6234430,
+      "step": 160,
+      "train_runtime": 657.5263,
+      "train_tokens_per_second": 9481.643
+    },
+    {
+      "epoch": 1.0957264957264958,
+      "grad_norm": 1.4851776361465454,
+      "learning_rate": 9.273356401384083e-06,
+      "loss": 0.4104,
+      "num_input_tokens_seen": 6267262,
+      "step": 161,
+      "train_runtime": 660.3125,
+      "train_tokens_per_second": 9491.358
+    },
+    {
+      "epoch": 1.1025641025641026,
+      "grad_norm": 1.3695839643478394,
+      "learning_rate": 9.204152249134949e-06,
+      "loss": 0.3741,
+      "num_input_tokens_seen": 6301214,
+      "step": 162,
+      "train_runtime": 663.1514,
+      "train_tokens_per_second": 9501.923
+    },
+    {
+      "epoch": 1.1094017094017095,
+      "grad_norm": 1.3758296966552734,
+      "learning_rate": 9.134948096885815e-06,
+      "loss": 0.3491,
+      "num_input_tokens_seen": 6348382,
+      "step": 163,
+      "train_runtime": 667.043,
+      "train_tokens_per_second": 9517.2
+    },
+    {
+      "epoch": 1.1162393162393163,
+      "grad_norm": 1.564186692237854,
+      "learning_rate": 9.065743944636679e-06,
+      "loss": 0.3936,
+      "num_input_tokens_seen": 6380958,
+      "step": 164,
+      "train_runtime": 669.7803,
+      "train_tokens_per_second": 9526.942
+    },
+    {
+      "epoch": 1.123076923076923,
+      "grad_norm": 1.4669296741485596,
+      "learning_rate": 8.996539792387544e-06,
+      "loss": 0.4147,
+      "num_input_tokens_seen": 6420126,
+      "step": 165,
+      "train_runtime": 673.027,
+      "train_tokens_per_second": 9539.18
+    },
+    {
+      "epoch": 1.12991452991453,
+      "grad_norm": 1.2374634742736816,
+      "learning_rate": 8.92733564013841e-06,
+      "loss": 0.3456,
+      "num_input_tokens_seen": 6466430,
+      "step": 166,
+      "train_runtime": 676.9123,
+      "train_tokens_per_second": 9552.833
+    },
+    {
+      "epoch": 1.1367521367521367,
+      "grad_norm": 1.3043692111968994,
+      "learning_rate": 8.858131487889274e-06,
+      "loss": 0.3613,
+      "num_input_tokens_seen": 6500990,
+      "step": 167,
+      "train_runtime": 679.8269,
+      "train_tokens_per_second": 9562.714
+    },
+    {
+      "epoch": 1.1435897435897435,
+      "grad_norm": 1.5178890228271484,
+      "learning_rate": 8.78892733564014e-06,
+      "loss": 0.4367,
+      "num_input_tokens_seen": 6534718,
+      "step": 168,
+      "train_runtime": 682.6614,
+      "train_tokens_per_second": 9572.414
+    },
+    {
+      "epoch": 1.1504273504273503,
+      "grad_norm": 1.3679075241088867,
+      "learning_rate": 8.719723183391005e-06,
+      "loss": 0.4432,
+      "num_input_tokens_seen": 6586718,
+      "step": 169,
+      "train_runtime": 686.9418,
+      "train_tokens_per_second": 9588.466
+    },
+    {
+      "epoch": 1.1572649572649572,
+      "grad_norm": 1.356967806816101,
+      "learning_rate": 8.65051903114187e-06,
+      "loss": 0.3925,
+      "num_input_tokens_seen": 6618526,
+      "step": 170,
+      "train_runtime": 689.6389,
+      "train_tokens_per_second": 9597.089
+    },
+    {
+      "epoch": 1.1641025641025642,
+      "grad_norm": 1.5362049341201782,
+      "learning_rate": 8.581314878892735e-06,
+      "loss": 0.4119,
+      "num_input_tokens_seen": 6648830,
+      "step": 171,
+      "train_runtime": 692.1817,
+      "train_tokens_per_second": 9605.614
+    },
+    {
+      "epoch": 1.170940170940171,
+      "grad_norm": 1.4113274812698364,
+      "learning_rate": 8.5121107266436e-06,
+      "loss": 0.4419,
+      "num_input_tokens_seen": 6683454,
+      "step": 172,
+      "train_runtime": 695.0911,
+      "train_tokens_per_second": 9615.22
+    },
+    {
+      "epoch": 1.1777777777777778,
+      "grad_norm": 1.4560883045196533,
+      "learning_rate": 8.442906574394465e-06,
+      "loss": 0.4328,
+      "num_input_tokens_seen": 6717822,
+      "step": 173,
+      "train_runtime": 697.9616,
+      "train_tokens_per_second": 9624.916
+    },
+    {
+      "epoch": 1.1846153846153846,
+      "grad_norm": 1.3433024883270264,
+      "learning_rate": 8.373702422145328e-06,
+      "loss": 0.3697,
+      "num_input_tokens_seen": 6759774,
+      "step": 174,
+      "train_runtime": 701.4628,
+      "train_tokens_per_second": 9636.683
+    },
+    {
+      "epoch": 1.1914529914529914,
+      "grad_norm": 1.3296136856079102,
+      "learning_rate": 8.304498269896194e-06,
+      "loss": 0.3546,
+      "num_input_tokens_seen": 6802974,
+      "step": 175,
+      "train_runtime": 704.9944,
+      "train_tokens_per_second": 9649.685
+    },
+    {
+      "epoch": 1.1982905982905983,
+      "grad_norm": 1.3747283220291138,
+      "learning_rate": 8.23529411764706e-06,
+      "loss": 0.3864,
+      "num_input_tokens_seen": 6838686,
+      "step": 176,
+      "train_runtime": 707.9671,
+      "train_tokens_per_second": 9659.61
+    },
+    {
+      "epoch": 1.205128205128205,
+      "grad_norm": 1.498785138130188,
+      "learning_rate": 8.166089965397924e-06,
+      "loss": 0.4183,
+      "num_input_tokens_seen": 6873086,
+      "step": 177,
+      "train_runtime": 710.8358,
+      "train_tokens_per_second": 9669.021
+    },
+    {
+      "epoch": 1.2119658119658119,
+      "grad_norm": 1.4664194583892822,
+      "learning_rate": 8.09688581314879e-06,
+      "loss": 0.3918,
+      "num_input_tokens_seen": 6916254,
+      "step": 178,
+      "train_runtime": 714.3711,
+      "train_tokens_per_second": 9681.599
+    },
+    {
+      "epoch": 1.218803418803419,
+      "grad_norm": 1.299894094467163,
+      "learning_rate": 8.027681660899655e-06,
+      "loss": 0.3753,
+      "num_input_tokens_seen": 6957598,
+      "step": 179,
+      "train_runtime": 717.7521,
+      "train_tokens_per_second": 9693.594
+    },
+    {
+      "epoch": 1.2256410256410257,
+      "grad_norm": 1.4512171745300293,
+      "learning_rate": 7.958477508650519e-06,
+      "loss": 0.4402,
+      "num_input_tokens_seen": 6991550,
+      "step": 180,
+      "train_runtime": 720.5836,
+      "train_tokens_per_second": 9702.621
+    },
+    {
+      "epoch": 1.2256410256410257,
+      "eval_loss": 0.4540201723575592,
+      "eval_runtime": 5.6718,
+      "eval_samples_per_second": 175.959,
+      "eval_steps_per_second": 5.642,
+      "num_input_tokens_seen": 6991550,
+      "step": 180
+    },
+    {
+      "epoch": 1.2324786324786325,
+      "grad_norm": 1.360262393951416,
+      "learning_rate": 7.889273356401385e-06,
+      "loss": 0.3946,
+      "num_input_tokens_seen": 7025758,
+      "step": 181,
+      "train_runtime": 747.3736,
+      "train_tokens_per_second": 9400.598
+    },
+    {
+      "epoch": 1.2393162393162394,
+      "grad_norm": 1.4997259378433228,
+      "learning_rate": 7.82006920415225e-06,
+      "loss": 0.4529,
+      "num_input_tokens_seen": 7064478,
+      "step": 182,
+      "train_runtime": 750.5493,
+      "train_tokens_per_second": 9412.41
+    },
+    {
+      "epoch": 1.2461538461538462,
+      "grad_norm": 1.585199236869812,
+      "learning_rate": 7.750865051903114e-06,
+      "loss": 0.4257,
+      "num_input_tokens_seen": 7089886,
+      "step": 183,
+      "train_runtime": 752.7622,
+      "train_tokens_per_second": 9418.493
+    },
+    {
+      "epoch": 1.252991452991453,
+      "grad_norm": 1.3399189710617065,
+      "learning_rate": 7.68166089965398e-06,
+      "loss": 0.3918,
+      "num_input_tokens_seen": 7134814,
+      "step": 184,
+      "train_runtime": 756.4288,
+      "train_tokens_per_second": 9432.235
+    },
+    {
+      "epoch": 1.2598290598290598,
+      "grad_norm": 1.4995014667510986,
+      "learning_rate": 7.612456747404845e-06,
+      "loss": 0.481,
+      "num_input_tokens_seen": 7178046,
+      "step": 185,
+      "train_runtime": 760.0338,
+      "train_tokens_per_second": 9444.377
+    },
+    {
+      "epoch": 1.2666666666666666,
+      "grad_norm": 1.4340060949325562,
+      "learning_rate": 7.5432525951557104e-06,
+      "loss": 0.4483,
+      "num_input_tokens_seen": 7212158,
+      "step": 186,
+      "train_runtime": 762.8937,
+      "train_tokens_per_second": 9453.686
+    },
+    {
+      "epoch": 1.2735042735042734,
+      "grad_norm": 1.4529216289520264,
+      "learning_rate": 7.474048442906575e-06,
+      "loss": 0.4695,
+      "num_input_tokens_seen": 7245886,
+      "step": 187,
+      "train_runtime": 765.7249,
+      "train_tokens_per_second": 9462.779
+    },
+    {
+      "epoch": 1.2803418803418802,
+      "grad_norm": 1.3756847381591797,
+      "learning_rate": 7.40484429065744e-06,
+      "loss": 0.4277,
+      "num_input_tokens_seen": 7287646,
+      "step": 188,
+      "train_runtime": 769.1612,
+      "train_tokens_per_second": 9474.797
+    },
+    {
+      "epoch": 1.287179487179487,
+      "grad_norm": 1.4164284467697144,
+      "learning_rate": 7.335640138408306e-06,
+      "loss": 0.4475,
+      "num_input_tokens_seen": 7326238,
+      "step": 189,
+      "train_runtime": 772.3229,
+      "train_tokens_per_second": 9485.978
+    },
+    {
+      "epoch": 1.294017094017094,
+      "grad_norm": 1.3923799991607666,
+      "learning_rate": 7.2664359861591705e-06,
+      "loss": 0.3954,
+      "num_input_tokens_seen": 7364894,
+      "step": 190,
+      "train_runtime": 775.5465,
+      "train_tokens_per_second": 9496.393
+    },
+    {
+      "epoch": 1.300854700854701,
+      "grad_norm": 1.3876417875289917,
+      "learning_rate": 7.197231833910035e-06,
+      "loss": 0.4325,
+      "num_input_tokens_seen": 7401534,
+      "step": 191,
+      "train_runtime": 778.6241,
+      "train_tokens_per_second": 9505.915
+    },
+    {
+      "epoch": 1.3076923076923077,
+      "grad_norm": 1.337653398513794,
+      "learning_rate": 7.128027681660901e-06,
+      "loss": 0.3433,
+      "num_input_tokens_seen": 7442494,
+      "step": 192,
+      "train_runtime": 782.0513,
+      "train_tokens_per_second": 9516.631
+    },
+    {
+      "epoch": 1.3145299145299145,
+      "grad_norm": 1.3862982988357544,
+      "learning_rate": 7.058823529411766e-06,
+      "loss": 0.3943,
+      "num_input_tokens_seen": 7487582,
+      "step": 193,
+      "train_runtime": 785.7613,
+      "train_tokens_per_second": 9529.079
+    },
+    {
+      "epoch": 1.3213675213675213,
+      "grad_norm": 1.5185152292251587,
+      "learning_rate": 6.989619377162631e-06,
+      "loss": 0.3824,
+      "num_input_tokens_seen": 7520734,
+      "step": 194,
+      "train_runtime": 788.5665,
+      "train_tokens_per_second": 9537.222
+    },
+    {
+      "epoch": 1.3282051282051281,
+      "grad_norm": 1.3591375350952148,
+      "learning_rate": 6.9204152249134946e-06,
+      "loss": 0.3988,
+      "num_input_tokens_seen": 7557854,
+      "step": 195,
+      "train_runtime": 791.673,
+      "train_tokens_per_second": 9546.686
+    },
+    {
+      "epoch": 1.335042735042735,
+      "grad_norm": 1.5463569164276123,
+      "learning_rate": 6.85121107266436e-06,
+      "loss": 0.4155,
+      "num_input_tokens_seen": 7594110,
+      "step": 196,
+      "train_runtime": 794.7163,
+      "train_tokens_per_second": 9555.75
+    },
+    {
+      "epoch": 1.341880341880342,
+      "grad_norm": 1.3969792127609253,
+      "learning_rate": 6.782006920415225e-06,
+      "loss": 0.4054,
+      "num_input_tokens_seen": 7634878,
+      "step": 197,
+      "train_runtime": 798.1263,
+      "train_tokens_per_second": 9566.003
+    },
+    {
+      "epoch": 1.3487179487179488,
+      "grad_norm": 1.4635127782821655,
+      "learning_rate": 6.71280276816609e-06,
+      "loss": 0.3989,
+      "num_input_tokens_seen": 7668894,
+      "step": 198,
+      "train_runtime": 801.0008,
+      "train_tokens_per_second": 9574.14
+    },
+    {
+      "epoch": 1.3555555555555556,
+      "grad_norm": 1.4537571668624878,
+      "learning_rate": 6.6435986159169555e-06,
+      "loss": 0.4503,
+      "num_input_tokens_seen": 7705598,
+      "step": 199,
+      "train_runtime": 804.0791,
+      "train_tokens_per_second": 9583.135
+    },
+    {
+      "epoch": 1.3623931623931624,
+      "grad_norm": 1.5418137311935425,
+      "learning_rate": 6.57439446366782e-06,
+      "loss": 0.4482,
+      "num_input_tokens_seen": 7737630,
+      "step": 200,
+      "train_runtime": 806.81,
+      "train_tokens_per_second": 9590.399
+    },
+    {
+      "epoch": 1.3692307692307693,
+      "grad_norm": 1.512305736541748,
+      "learning_rate": 6.505190311418685e-06,
+      "loss": 0.4625,
+      "num_input_tokens_seen": 7772446,
+      "step": 201,
+      "train_runtime": 809.7648,
+      "train_tokens_per_second": 9598.399
+    },
+    {
+      "epoch": 1.376068376068376,
+      "grad_norm": 1.2927024364471436,
+      "learning_rate": 6.435986159169551e-06,
+      "loss": 0.434,
+      "num_input_tokens_seen": 7825886,
+      "step": 202,
+      "train_runtime": 814.22,
+      "train_tokens_per_second": 9611.513
+    },
+    {
+      "epoch": 1.3829059829059829,
+      "grad_norm": 1.3422515392303467,
+      "learning_rate": 6.3667820069204156e-06,
+      "loss": 0.4471,
+      "num_input_tokens_seen": 7877566,
+      "step": 203,
+      "train_runtime": 818.5631,
+      "train_tokens_per_second": 9623.652
+    },
+    {
+      "epoch": 1.3897435897435897,
+      "grad_norm": 1.5034905672073364,
+      "learning_rate": 6.29757785467128e-06,
+      "loss": 0.4317,
+      "num_input_tokens_seen": 7915038,
+      "step": 204,
+      "train_runtime": 821.6717,
+      "train_tokens_per_second": 9632.847
+    },
+    {
+      "epoch": 1.3965811965811965,
+      "grad_norm": 1.4573144912719727,
+      "learning_rate": 6.228373702422146e-06,
+      "loss": 0.4127,
+      "num_input_tokens_seen": 7945950,
+      "step": 205,
+      "train_runtime": 824.2951,
+      "train_tokens_per_second": 9639.691
+    },
+    {
+      "epoch": 1.4034188034188033,
+      "grad_norm": 1.424501657485962,
+      "learning_rate": 6.159169550173011e-06,
+      "loss": 0.4444,
+      "num_input_tokens_seen": 7981854,
+      "step": 206,
+      "train_runtime": 827.3346,
+      "train_tokens_per_second": 9647.674
+    },
+    {
+      "epoch": 1.4102564102564101,
+      "grad_norm": 1.52495276927948,
+      "learning_rate": 6.089965397923876e-06,
+      "loss": 0.4204,
+      "num_input_tokens_seen": 8024286,
+      "step": 207,
+      "train_runtime": 830.8818,
+      "train_tokens_per_second": 9657.554
+    },
+    {
+      "epoch": 1.4170940170940172,
+      "grad_norm": 1.3244974613189697,
+      "learning_rate": 6.020761245674741e-06,
+      "loss": 0.3899,
+      "num_input_tokens_seen": 8062558,
+      "step": 208,
+      "train_runtime": 834.1195,
+      "train_tokens_per_second": 9665.951
+    },
+    {
+      "epoch": 1.423931623931624,
+      "grad_norm": 1.4783118963241577,
+      "learning_rate": 5.951557093425606e-06,
+      "loss": 0.4001,
+      "num_input_tokens_seen": 8097406,
+      "step": 209,
+      "train_runtime": 837.0288,
+      "train_tokens_per_second": 9673.987
+    },
+    {
+      "epoch": 1.4307692307692308,
+      "grad_norm": 1.4629509449005127,
+      "learning_rate": 5.882352941176471e-06,
+      "loss": 0.4112,
+      "num_input_tokens_seen": 8130526,
+      "step": 210,
+      "train_runtime": 839.833,
+      "train_tokens_per_second": 9681.123
+    },
+    {
+      "epoch": 1.4307692307692308,
+      "eval_loss": 0.4539625346660614,
+      "eval_runtime": 5.7425,
+      "eval_samples_per_second": 173.792,
+      "eval_steps_per_second": 5.572,
+      "num_input_tokens_seen": 8130526,
+      "step": 210
+    },
+    {
+      "epoch": 1.4376068376068376,
+      "grad_norm": 1.35781991481781,
+      "learning_rate": 5.8131487889273366e-06,
+      "loss": 0.376,
+      "num_input_tokens_seen": 8170302,
+      "step": 211,
+      "train_runtime": 867.0208,
+      "train_tokens_per_second": 9423.421
+    },
+    {
+      "epoch": 1.4444444444444444,
+      "grad_norm": 1.5642489194869995,
+      "learning_rate": 5.743944636678201e-06,
+      "loss": 0.4331,
+      "num_input_tokens_seen": 8204958,
+      "step": 212,
+      "train_runtime": 869.9438,
+      "train_tokens_per_second": 9431.596
+    },
+    {
+      "epoch": 1.4512820512820512,
+      "grad_norm": 1.4987717866897583,
+      "learning_rate": 5.674740484429066e-06,
+      "loss": 0.4854,
+      "num_input_tokens_seen": 8244126,
+      "step": 213,
+      "train_runtime": 873.217,
+      "train_tokens_per_second": 9441.097
+    },
+    {
+      "epoch": 1.458119658119658,
+      "grad_norm": 1.4428235292434692,
+      "learning_rate": 5.605536332179932e-06,
+      "loss": 0.4259,
+      "num_input_tokens_seen": 8286910,
+      "step": 214,
+      "train_runtime": 876.729,
+      "train_tokens_per_second": 9452.077
+    },
+    {
+      "epoch": 1.464957264957265,
+      "grad_norm": 1.5963438749313354,
+      "learning_rate": 5.536332179930797e-06,
+      "loss": 0.4945,
+      "num_input_tokens_seen": 8322110,
+      "step": 215,
+      "train_runtime": 879.6762,
+      "train_tokens_per_second": 9460.425
+    },
+    {
+      "epoch": 1.471794871794872,
+      "grad_norm": 1.3209023475646973,
+      "learning_rate": 5.4671280276816615e-06,
+      "loss": 0.5295,
+      "num_input_tokens_seen": 8360894,
+      "step": 216,
+      "train_runtime": 882.8992,
+      "train_tokens_per_second": 9469.818
+    },
+    {
+      "epoch": 1.4786324786324787,
+      "grad_norm": 1.3461143970489502,
+      "learning_rate": 5.397923875432527e-06,
+      "loss": 0.3801,
+      "num_input_tokens_seen": 8393630,
+      "step": 217,
+      "train_runtime": 885.6709,
+      "train_tokens_per_second": 9477.143
+    },
+    {
+      "epoch": 1.4854700854700855,
+      "grad_norm": 1.5508912801742554,
+      "learning_rate": 5.328719723183391e-06,
+      "loss": 0.479,
+      "num_input_tokens_seen": 8422910,
+      "step": 218,
+      "train_runtime": 888.188,
+      "train_tokens_per_second": 9483.251
+    },
+    {
+      "epoch": 1.4923076923076923,
+      "grad_norm": 1.4699424505233765,
+      "learning_rate": 5.259515570934256e-06,
+      "loss": 0.4195,
+      "num_input_tokens_seen": 8458014,
+      "step": 219,
+      "train_runtime": 891.1228,
+      "train_tokens_per_second": 9491.412
+    },
+    {
+      "epoch": 1.4991452991452991,
+      "grad_norm": 1.3626216650009155,
+      "learning_rate": 5.190311418685121e-06,
+      "loss": 0.4021,
+      "num_input_tokens_seen": 8500670,
+      "step": 220,
+      "train_runtime": 894.6464,
+      "train_tokens_per_second": 9501.709
+    },
+    {
+      "epoch": 1.505982905982906,
+      "grad_norm": 1.6453019380569458,
+      "learning_rate": 5.121107266435986e-06,
+      "loss": 0.4276,
+      "num_input_tokens_seen": 8531038,
+      "step": 221,
+      "train_runtime": 897.2331,
+      "train_tokens_per_second": 9508.162
+    },
+    {
+      "epoch": 1.5128205128205128,
+      "grad_norm": 1.3846372365951538,
+      "learning_rate": 5.051903114186851e-06,
+      "loss": 0.3921,
+      "num_input_tokens_seen": 8562014,
+      "step": 222,
+      "train_runtime": 899.8779,
+      "train_tokens_per_second": 9514.64
+    },
+    {
+      "epoch": 1.5196581196581196,
+      "grad_norm": 1.5681668519973755,
+      "learning_rate": 4.982698961937717e-06,
+      "loss": 0.4713,
+      "num_input_tokens_seen": 8593054,
+      "step": 223,
+      "train_runtime": 902.506,
+      "train_tokens_per_second": 9521.326
+    },
+    {
+      "epoch": 1.5264957264957264,
+      "grad_norm": 1.4132840633392334,
+      "learning_rate": 4.913494809688582e-06,
+      "loss": 0.3994,
+      "num_input_tokens_seen": 8628254,
+      "step": 224,
+      "train_runtime": 905.4606,
+      "train_tokens_per_second": 9529.133
+    },
+    {
+      "epoch": 1.5333333333333332,
+      "grad_norm": 1.4908711910247803,
+      "learning_rate": 4.8442906574394464e-06,
+      "loss": 0.4021,
+      "num_input_tokens_seen": 8689630,
+      "step": 225,
+      "train_runtime": 910.705,
+      "train_tokens_per_second": 9541.652
+    },
+    {
+      "epoch": 1.54017094017094,
+      "grad_norm": 1.1932868957519531,
+      "learning_rate": 4.775086505190312e-06,
+      "loss": 0.3898,
+      "num_input_tokens_seen": 8734526,
+      "step": 226,
+      "train_runtime": 914.4429,
+      "train_tokens_per_second": 9551.746
+    },
+    {
+      "epoch": 1.547008547008547,
+      "grad_norm": 1.253638744354248,
+      "learning_rate": 4.705882352941177e-06,
+      "loss": 0.4028,
+      "num_input_tokens_seen": 8799582,
+      "step": 227,
+      "train_runtime": 919.89,
+      "train_tokens_per_second": 9565.907
+    },
+    {
+      "epoch": 1.5538461538461539,
+      "grad_norm": 1.322104573249817,
+      "learning_rate": 4.636678200692042e-06,
+      "loss": 0.3858,
+      "num_input_tokens_seen": 8840638,
+      "step": 228,
+      "train_runtime": 923.2956,
+      "train_tokens_per_second": 9575.089
+    },
+    {
+      "epoch": 1.5606837606837607,
+      "grad_norm": 1.377384901046753,
+      "learning_rate": 4.567474048442907e-06,
+      "loss": 0.3577,
+      "num_input_tokens_seen": 8895710,
+      "step": 229,
+      "train_runtime": 927.776,
+      "train_tokens_per_second": 9588.209
+    },
+    {
+      "epoch": 1.5675213675213675,
+      "grad_norm": 1.4464075565338135,
+      "learning_rate": 4.498269896193772e-06,
+      "loss": 0.3797,
+      "num_input_tokens_seen": 8940670,
+      "step": 230,
+      "train_runtime": 931.4748,
+      "train_tokens_per_second": 9598.402
+    },
+    {
+      "epoch": 1.5743589743589743,
+      "grad_norm": 1.4492119550704956,
+      "learning_rate": 4.429065743944637e-06,
+      "loss": 0.3816,
+      "num_input_tokens_seen": 8984350,
+      "step": 231,
+      "train_runtime": 935.1105,
+      "train_tokens_per_second": 9607.795
+    },
+    {
+      "epoch": 1.5811965811965814,
+      "grad_norm": 1.6883492469787598,
+      "learning_rate": 4.359861591695503e-06,
+      "loss": 0.4917,
+      "num_input_tokens_seen": 9019390,
+      "step": 232,
+      "train_runtime": 938.0738,
+      "train_tokens_per_second": 9614.797
+    },
+    {
+      "epoch": 1.5880341880341882,
+      "grad_norm": 1.4682198762893677,
+      "learning_rate": 4.2906574394463675e-06,
+      "loss": 0.3746,
+      "num_input_tokens_seen": 9059294,
+      "step": 233,
+      "train_runtime": 941.4183,
+      "train_tokens_per_second": 9623.028
+    },
+    {
+      "epoch": 1.594871794871795,
+      "grad_norm": 1.3500458002090454,
+      "learning_rate": 4.221453287197232e-06,
+      "loss": 0.4348,
+      "num_input_tokens_seen": 9097854,
+      "step": 234,
+      "train_runtime": 944.6072,
+      "train_tokens_per_second": 9631.362
+    },
+    {
+      "epoch": 1.6017094017094018,
+      "grad_norm": 1.5531758069992065,
+      "learning_rate": 4.152249134948097e-06,
+      "loss": 0.4281,
+      "num_input_tokens_seen": 9131518,
+      "step": 235,
+      "train_runtime": 947.4585,
+      "train_tokens_per_second": 9637.908
+    },
+    {
+      "epoch": 1.6085470085470086,
+      "grad_norm": 1.503575325012207,
+      "learning_rate": 4.083044982698962e-06,
+      "loss": 0.4181,
+      "num_input_tokens_seen": 9170014,
+      "step": 236,
+      "train_runtime": 950.6625,
+      "train_tokens_per_second": 9645.919
+    },
+    {
+      "epoch": 1.6153846153846154,
+      "grad_norm": 1.5178425312042236,
+      "learning_rate": 4.0138408304498275e-06,
+      "loss": 0.4313,
+      "num_input_tokens_seen": 9211198,
+      "step": 237,
+      "train_runtime": 954.0784,
+      "train_tokens_per_second": 9654.551
+    },
+    {
+      "epoch": 1.6222222222222222,
+      "grad_norm": 1.4492536783218384,
+      "learning_rate": 3.944636678200692e-06,
+      "loss": 0.4175,
+      "num_input_tokens_seen": 9240734,
+      "step": 238,
+      "train_runtime": 956.6363,
+      "train_tokens_per_second": 9659.611
+    },
+    {
+      "epoch": 1.629059829059829,
+      "grad_norm": 1.4955025911331177,
+      "learning_rate": 3.875432525951557e-06,
+      "loss": 0.4145,
+      "num_input_tokens_seen": 9279966,
+      "step": 239,
+      "train_runtime": 959.9009,
+      "train_tokens_per_second": 9667.63
+    },
+    {
+      "epoch": 1.6358974358974359,
+      "grad_norm": 1.3603522777557373,
+      "learning_rate": 3.8062283737024224e-06,
+      "loss": 0.3571,
+      "num_input_tokens_seen": 9318206,
+      "step": 240,
+      "train_runtime": 963.044,
+      "train_tokens_per_second": 9675.784
+    },
+    {
+      "epoch": 1.6358974358974359,
+      "eval_loss": 0.4540530741214752,
+      "eval_runtime": 5.7145,
+      "eval_samples_per_second": 174.643,
+      "eval_steps_per_second": 5.6,
+      "num_input_tokens_seen": 9318206,
+      "step": 240
+    },
+    {
+      "epoch": 1.6427350427350427,
+      "grad_norm": 1.3077824115753174,
+      "learning_rate": 3.7370242214532876e-06,
+      "loss": 0.3903,
+      "num_input_tokens_seen": 9357278,
+      "step": 241,
+      "train_runtime": 989.9944,
+      "train_tokens_per_second": 9451.849
+    },
+    {
+      "epoch": 1.6495726495726495,
+      "grad_norm": 1.4134607315063477,
+      "learning_rate": 3.667820069204153e-06,
+      "loss": 0.4299,
+      "num_input_tokens_seen": 9401246,
+      "step": 242,
+      "train_runtime": 993.6355,
+      "train_tokens_per_second": 9461.464
+    },
+    {
+      "epoch": 1.6564102564102563,
+      "grad_norm": 1.3824211359024048,
+      "learning_rate": 3.5986159169550177e-06,
+      "loss": 0.4513,
+      "num_input_tokens_seen": 9439326,
+      "step": 243,
+      "train_runtime": 996.8489,
+      "train_tokens_per_second": 9469.164
+    },
+    {
+      "epoch": 1.6632478632478631,
+      "grad_norm": 1.3873765468597412,
+      "learning_rate": 3.529411764705883e-06,
+      "loss": 0.4037,
+      "num_input_tokens_seen": 9474398,
+      "step": 244,
+      "train_runtime": 999.7516,
+      "train_tokens_per_second": 9476.752
+    },
+    {
+      "epoch": 1.67008547008547,
+      "grad_norm": 1.5342257022857666,
+      "learning_rate": 3.4602076124567473e-06,
+      "loss": 0.4479,
+      "num_input_tokens_seen": 9504830,
+      "step": 245,
+      "train_runtime": 1002.3332,
+      "train_tokens_per_second": 9482.704
+    },
+    {
+      "epoch": 1.676923076923077,
+      "grad_norm": 1.4302300214767456,
+      "learning_rate": 3.3910034602076125e-06,
+      "loss": 0.4137,
+      "num_input_tokens_seen": 9550430,
+      "step": 246,
+      "train_runtime": 1006.0494,
+      "train_tokens_per_second": 9493.003
+    },
+    {
+      "epoch": 1.6837606837606838,
+      "grad_norm": 1.516709566116333,
+      "learning_rate": 3.3217993079584777e-06,
+      "loss": 0.4088,
+      "num_input_tokens_seen": 9583806,
+      "step": 247,
+      "train_runtime": 1008.8176,
+      "train_tokens_per_second": 9500.038
+    },
+    {
+      "epoch": 1.6905982905982906,
+      "grad_norm": 1.4347656965255737,
+      "learning_rate": 3.2525951557093425e-06,
+      "loss": 0.3991,
+      "num_input_tokens_seen": 9644798,
+      "step": 248,
+      "train_runtime": 1013.8333,
+      "train_tokens_per_second": 9513.199
+    },
+    {
+      "epoch": 1.6974358974358974,
+      "grad_norm": 1.2900501489639282,
+      "learning_rate": 3.1833910034602078e-06,
+      "loss": 0.3854,
+      "num_input_tokens_seen": 9699294,
+      "step": 249,
+      "train_runtime": 1018.4019,
+      "train_tokens_per_second": 9524.034
+    },
+    {
+      "epoch": 1.7042735042735044,
+      "grad_norm": 1.4763895273208618,
+      "learning_rate": 3.114186851211073e-06,
+      "loss": 0.3875,
+      "num_input_tokens_seen": 9738238,
+      "step": 250,
+      "train_runtime": 1021.6841,
+      "train_tokens_per_second": 9531.554
+    },
+    {
+      "epoch": 1.7111111111111112,
+      "grad_norm": 1.5247527360916138,
+      "learning_rate": 3.044982698961938e-06,
+      "loss": 0.4356,
+      "num_input_tokens_seen": 9774910,
+      "step": 251,
+      "train_runtime": 1024.7713,
+      "train_tokens_per_second": 9538.626
+    },
+    {
+      "epoch": 1.717948717948718,
+      "grad_norm": 1.3282755613327026,
+      "learning_rate": 2.975778546712803e-06,
+      "loss": 0.3761,
+      "num_input_tokens_seen": 9816158,
+      "step": 252,
+      "train_runtime": 1028.1704,
+      "train_tokens_per_second": 9547.21
+    },
+    {
+      "epoch": 1.7247863247863249,
+      "grad_norm": 1.5729610919952393,
+      "learning_rate": 2.9065743944636683e-06,
+      "loss": 0.411,
+      "num_input_tokens_seen": 9860414,
+      "step": 253,
+      "train_runtime": 1031.7918,
+      "train_tokens_per_second": 9556.593
+    },
+    {
+      "epoch": 1.7316239316239317,
+      "grad_norm": 1.3676373958587646,
+      "learning_rate": 2.837370242214533e-06,
+      "loss": 0.3885,
+      "num_input_tokens_seen": 9895518,
+      "step": 254,
+      "train_runtime": 1034.7479,
+      "train_tokens_per_second": 9563.216
+    },
+    {
+      "epoch": 1.7384615384615385,
+      "grad_norm": 1.3123220205307007,
+      "learning_rate": 2.7681660899653983e-06,
+      "loss": 0.3553,
+      "num_input_tokens_seen": 9931838,
+      "step": 255,
+      "train_runtime": 1037.7767,
+      "train_tokens_per_second": 9570.304
+    },
+    {
+      "epoch": 1.7452991452991453,
+      "grad_norm": 1.4366247653961182,
+      "learning_rate": 2.6989619377162636e-06,
+      "loss": 0.42,
+      "num_input_tokens_seen": 9975582,
+      "step": 256,
+      "train_runtime": 1041.3876,
+      "train_tokens_per_second": 9579.125
+    },
+    {
+      "epoch": 1.7521367521367521,
+      "grad_norm": 1.330451488494873,
+      "learning_rate": 2.629757785467128e-06,
+      "loss": 0.3794,
+      "num_input_tokens_seen": 10009150,
+      "step": 257,
+      "train_runtime": 1044.2188,
+      "train_tokens_per_second": 9585.3
+    },
+    {
+      "epoch": 1.758974358974359,
+      "grad_norm": 1.1759202480316162,
+      "learning_rate": 2.560553633217993e-06,
+      "loss": 0.3586,
+      "num_input_tokens_seen": 10082110,
+      "step": 258,
+      "train_runtime": 1050.4616,
+      "train_tokens_per_second": 9597.79
+    },
+    {
+      "epoch": 1.7658119658119658,
+      "grad_norm": 1.5406081676483154,
+      "learning_rate": 2.4913494809688584e-06,
+      "loss": 0.4238,
+      "num_input_tokens_seen": 10116126,
+      "step": 259,
+      "train_runtime": 1053.3071,
+      "train_tokens_per_second": 9604.156
+    },
+    {
+      "epoch": 1.7726495726495726,
+      "grad_norm": 1.5332951545715332,
+      "learning_rate": 2.4221453287197232e-06,
+      "loss": 0.4283,
+      "num_input_tokens_seen": 10152958,
+      "step": 260,
+      "train_runtime": 1056.3542,
+      "train_tokens_per_second": 9611.32
+    },
+    {
+      "epoch": 1.7794871794871794,
+      "grad_norm": 1.3916773796081543,
+      "learning_rate": 2.3529411764705885e-06,
+      "loss": 0.4203,
+      "num_input_tokens_seen": 10190046,
+      "step": 261,
+      "train_runtime": 1059.4345,
+      "train_tokens_per_second": 9618.382
+    },
+    {
+      "epoch": 1.7863247863247862,
+      "grad_norm": 1.3282185792922974,
+      "learning_rate": 2.2837370242214537e-06,
+      "loss": 0.3668,
+      "num_input_tokens_seen": 10230334,
+      "step": 262,
+      "train_runtime": 1062.7483,
+      "train_tokens_per_second": 9626.3
+    },
+    {
+      "epoch": 1.793162393162393,
+      "grad_norm": 1.3999195098876953,
+      "learning_rate": 2.2145328719723185e-06,
+      "loss": 0.3823,
+      "num_input_tokens_seen": 10262302,
+      "step": 263,
+      "train_runtime": 1065.4169,
+      "train_tokens_per_second": 9632.194
+    },
+    {
+      "epoch": 1.8,
+      "grad_norm": 1.483365535736084,
+      "learning_rate": 2.1453287197231837e-06,
+      "loss": 0.437,
+      "num_input_tokens_seen": 10306046,
+      "step": 264,
+      "train_runtime": 1069.0582,
+      "train_tokens_per_second": 9640.304
+    },
+    {
+      "epoch": 1.8068376068376069,
+      "grad_norm": 1.5923428535461426,
+      "learning_rate": 2.0761245674740485e-06,
+      "loss": 0.4307,
+      "num_input_tokens_seen": 10335582,
+      "step": 265,
+      "train_runtime": 1071.5688,
+      "train_tokens_per_second": 9645.281
+    },
+    {
+      "epoch": 1.8136752136752137,
+      "grad_norm": 1.4312759637832642,
+      "learning_rate": 2.0069204152249138e-06,
+      "loss": 0.4697,
+      "num_input_tokens_seen": 10369214,
+      "step": 266,
+      "train_runtime": 1074.4014,
+      "train_tokens_per_second": 9651.154
+    },
+    {
+      "epoch": 1.8205128205128205,
+      "grad_norm": 1.4313794374465942,
+      "learning_rate": 1.9377162629757786e-06,
+      "loss": 0.4429,
+      "num_input_tokens_seen": 10415998,
+      "step": 267,
+      "train_runtime": 1078.2698,
+      "train_tokens_per_second": 9659.918
+    },
+    {
+      "epoch": 1.8273504273504273,
+      "grad_norm": 1.440711259841919,
+      "learning_rate": 1.8685121107266438e-06,
+      "loss": 0.427,
+      "num_input_tokens_seen": 10457470,
+      "step": 268,
+      "train_runtime": 1081.7021,
+      "train_tokens_per_second": 9667.607
+    },
+    {
+      "epoch": 1.8341880341880343,
+      "grad_norm": 1.5918058156967163,
+      "learning_rate": 1.7993079584775088e-06,
+      "loss": 0.4871,
+      "num_input_tokens_seen": 10496382,
+      "step": 269,
+      "train_runtime": 1084.9586,
+      "train_tokens_per_second": 9674.454
+    },
+    {
+      "epoch": 1.8410256410256411,
+      "grad_norm": 1.4235551357269287,
+      "learning_rate": 1.7301038062283736e-06,
+      "loss": 0.5227,
+      "num_input_tokens_seen": 10529278,
+      "step": 270,
+      "train_runtime": 1087.7757,
+      "train_tokens_per_second": 9679.64
+    },
+    {
+      "epoch": 1.8410256410256411,
+      "eval_loss": 0.45376113057136536,
+      "eval_runtime": 5.8063,
+      "eval_samples_per_second": 171.881,
+      "eval_steps_per_second": 5.511,
+      "num_input_tokens_seen": 10529278,
+      "step": 270
+    },
+    {
+      "epoch": 1.847863247863248,
+      "grad_norm": 1.3479825258255005,
+      "learning_rate": 1.6608996539792389e-06,
+      "loss": 0.4222,
+      "num_input_tokens_seen": 10573694,
+      "step": 271,
+      "train_runtime": 1115.6749,
+      "train_tokens_per_second": 9477.397
+    },
+    {
+      "epoch": 1.8547008547008548,
+      "grad_norm": 1.4445607662200928,
+      "learning_rate": 1.5916955017301039e-06,
+      "loss": 0.4388,
+      "num_input_tokens_seen": 10615710,
+      "step": 272,
+      "train_runtime": 1119.1547,
+      "train_tokens_per_second": 9485.471
+    },
+    {
+      "epoch": 1.8615384615384616,
+      "grad_norm": 1.6135919094085693,
+      "learning_rate": 1.522491349480969e-06,
+      "loss": 0.4083,
+      "num_input_tokens_seen": 10651486,
+      "step": 273,
+      "train_runtime": 1122.1591,
+      "train_tokens_per_second": 9491.957
+    },
+    {
+      "epoch": 1.8683760683760684,
+      "grad_norm": 1.567018747329712,
+      "learning_rate": 1.4532871972318341e-06,
+      "loss": 0.4997,
+      "num_input_tokens_seen": 10684862,
+      "step": 274,
+      "train_runtime": 1124.9704,
+      "train_tokens_per_second": 9497.905
+    },
+    {
+      "epoch": 1.8752136752136752,
+      "grad_norm": 1.5761120319366455,
+      "learning_rate": 1.3840830449826992e-06,
+      "loss": 0.4252,
+      "num_input_tokens_seen": 10718174,
+      "step": 275,
+      "train_runtime": 1127.7815,
+      "train_tokens_per_second": 9503.768
+    },
+    {
+      "epoch": 1.882051282051282,
+      "grad_norm": 1.516058325767517,
+      "learning_rate": 1.314878892733564e-06,
+      "loss": 0.4248,
+      "num_input_tokens_seen": 10751358,
+      "step": 276,
+      "train_runtime": 1130.5976,
+      "train_tokens_per_second": 9509.447
+    },
+    {
+      "epoch": 1.8888888888888888,
+      "grad_norm": 1.4651241302490234,
+      "learning_rate": 1.2456747404844292e-06,
+      "loss": 0.4218,
+      "num_input_tokens_seen": 10785278,
+      "step": 277,
+      "train_runtime": 1133.4807,
+      "train_tokens_per_second": 9515.185
+    },
+    {
+      "epoch": 1.8957264957264957,
+      "grad_norm": 1.470118761062622,
+      "learning_rate": 1.1764705882352942e-06,
+      "loss": 0.4166,
+      "num_input_tokens_seen": 10829086,
+      "step": 278,
+      "train_runtime": 1137.1675,
+      "train_tokens_per_second": 9522.859
+    },
+    {
+      "epoch": 1.9025641025641025,
+      "grad_norm": 1.4548068046569824,
+      "learning_rate": 1.1072664359861592e-06,
+      "loss": 0.4197,
+      "num_input_tokens_seen": 10869246,
+      "step": 279,
+      "train_runtime": 1140.5065,
+      "train_tokens_per_second": 9530.192
+    },
+    {
+      "epoch": 1.9094017094017093,
+      "grad_norm": 1.4155503511428833,
+      "learning_rate": 1.0380622837370243e-06,
+      "loss": 0.3839,
+      "num_input_tokens_seen": 10908350,
+      "step": 280,
+      "train_runtime": 1143.7836,
+      "train_tokens_per_second": 9537.075
+    },
+    {
+      "epoch": 1.916239316239316,
+      "grad_norm": 1.5686092376708984,
+      "learning_rate": 9.688581314878893e-07,
+      "loss": 0.4483,
+      "num_input_tokens_seen": 10936542,
+      "step": 281,
+      "train_runtime": 1146.1984,
+      "train_tokens_per_second": 9541.578
+    },
+    {
+      "epoch": 1.9230769230769231,
+      "grad_norm": 1.4835528135299683,
+      "learning_rate": 8.996539792387544e-07,
+      "loss": 0.4117,
+      "num_input_tokens_seen": 10970366,
+      "step": 282,
+      "train_runtime": 1149.0912,
+      "train_tokens_per_second": 9546.994
+    },
+    {
+      "epoch": 1.92991452991453,
+      "grad_norm": 1.5472915172576904,
+      "learning_rate": 8.304498269896194e-07,
+      "loss": 0.4556,
+      "num_input_tokens_seen": 11005310,
+      "step": 283,
+      "train_runtime": 1152.0483,
+      "train_tokens_per_second": 9552.82
+    },
+    {
+      "epoch": 1.9367521367521368,
+      "grad_norm": 1.345395565032959,
+      "learning_rate": 7.612456747404845e-07,
+      "loss": 0.392,
+      "num_input_tokens_seen": 11039710,
+      "step": 284,
+      "train_runtime": 1154.9982,
+      "train_tokens_per_second": 9558.205
+    },
+    {
+      "epoch": 1.9435897435897436,
+      "grad_norm": 1.640942931175232,
+      "learning_rate": 6.920415224913496e-07,
+      "loss": 0.4268,
+      "num_input_tokens_seen": 11068126,
+      "step": 285,
+      "train_runtime": 1157.4718,
+      "train_tokens_per_second": 9562.329
+    },
+    {
+      "epoch": 1.9504273504273504,
+      "grad_norm": 1.4688515663146973,
+      "learning_rate": 6.228373702422146e-07,
+      "loss": 0.4585,
+      "num_input_tokens_seen": 11100222,
+      "step": 286,
+      "train_runtime": 1160.1879,
+      "train_tokens_per_second": 9567.607
+    },
+    {
+      "epoch": 1.9572649572649574,
+      "grad_norm": 1.353981375694275,
+      "learning_rate": 5.536332179930796e-07,
+      "loss": 0.4352,
+      "num_input_tokens_seen": 11142846,
+      "step": 287,
+      "train_runtime": 1163.7686,
+      "train_tokens_per_second": 9574.795
+    },
+    {
+      "epoch": 1.9641025641025642,
+      "grad_norm": 1.4443548917770386,
+      "learning_rate": 4.844290657439446e-07,
+      "loss": 0.4597,
+      "num_input_tokens_seen": 11179038,
+      "step": 288,
+      "train_runtime": 1166.8565,
+      "train_tokens_per_second": 9580.474
+    },
+    {
+      "epoch": 1.970940170940171,
+      "grad_norm": 1.463306188583374,
+      "learning_rate": 4.152249134948097e-07,
+      "loss": 0.3708,
+      "num_input_tokens_seen": 11216158,
+      "step": 289,
+      "train_runtime": 1169.997,
+      "train_tokens_per_second": 9586.485
+    },
+    {
+      "epoch": 1.9777777777777779,
+      "grad_norm": 1.3856064081192017,
+      "learning_rate": 3.460207612456748e-07,
+      "loss": 0.397,
+      "num_input_tokens_seen": 11297406,
+      "step": 290,
+      "train_runtime": 1177.6388,
+      "train_tokens_per_second": 9593.269
+    },
+    {
+      "epoch": 1.9846153846153847,
+      "grad_norm": 1.567358136177063,
+      "learning_rate": 2.768166089965398e-07,
+      "loss": 0.4152,
+      "num_input_tokens_seen": 11332094,
+      "step": 291,
+      "train_runtime": 1180.5843,
+      "train_tokens_per_second": 9598.717
+    },
+    {
+      "epoch": 1.9914529914529915,
+      "grad_norm": 1.3738164901733398,
+      "learning_rate": 2.0761245674740486e-07,
+      "loss": 0.3608,
+      "num_input_tokens_seen": 11394046,
+      "step": 292,
+      "train_runtime": 1185.9037,
+      "train_tokens_per_second": 9607.902
+    },
+    {
+      "epoch": 1.9982905982905983,
+      "grad_norm": 1.5582184791564941,
+      "learning_rate": 1.384083044982699e-07,
+      "loss": 0.4101,
+      "num_input_tokens_seen": 11428286,
+      "step": 293,
+      "train_runtime": 1188.8125,
+      "train_tokens_per_second": 9613.194
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 2.8240842819213867,
+      "learning_rate": 6.920415224913495e-08,
+      "loss": 0.3817,
+      "num_input_tokens_seen": 11435966,
+      "step": 294,
+      "train_runtime": 1189.5451,
+      "train_tokens_per_second": 9613.731
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 294,
+  "num_input_tokens_seen": 11435966,
+  "num_train_epochs": 2,
+  "save_steps": 30,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.9039331026231296e+17,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

vocab.json ADDED Viewed

The diff for this file is too large to render. See raw diff