Instructions to use roonbug/v9xvk468 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use roonbug/v9xvk468 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="roonbug/v9xvk468")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("roonbug/v9xvk468")
model = AutoModelForImageTextToText.from_pretrained("roonbug/v9xvk468")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use roonbug/v9xvk468 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "roonbug/v9xvk468"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/v9xvk468",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/roonbug/v9xvk468

SGLang

How to use roonbug/v9xvk468 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "roonbug/v9xvk468" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/v9xvk468",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "roonbug/v9xvk468" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/v9xvk468",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use roonbug/v9xvk468 with Docker Model Runner:
```
docker model run hf.co/roonbug/v9xvk468
```

roonbug commited on May 2

Commit

e0cf492

verified ·

1 Parent(s): 9f671f6

Upload folder using huggingface_hub

Browse files

Files changed (20) hide show

.gitattributes +1 -0
checkpoint-1000/added_tokens.json +3 -0
checkpoint-1000/chat_template.jinja +47 -0
checkpoint-1000/config.json +110 -0
checkpoint-1000/generation_config.json +13 -0
checkpoint-1000/model-00001-of-00005.safetensors +3 -0
checkpoint-1000/model-00002-of-00005.safetensors +3 -0
checkpoint-1000/model-00003-of-00005.safetensors +3 -0
checkpoint-1000/model-00004-of-00005.safetensors +3 -0
checkpoint-1000/model-00005-of-00005.safetensors +3 -0
checkpoint-1000/model.safetensors.index.json +0 -0
checkpoint-1000/optimizer.pt +3 -0
checkpoint-1000/rng_state.pth +3 -0
checkpoint-1000/scheduler.pt +3 -0
checkpoint-1000/special_tokens_map.json +33 -0
checkpoint-1000/tokenizer.json +3 -0
checkpoint-1000/tokenizer.model +3 -0
checkpoint-1000/tokenizer_config.json +0 -0
checkpoint-1000/trainer_state.json +1474 -0
checkpoint-1000/training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-1000/tokenizer.json filter=lfs diff=lfs merge=lfs -text

checkpoint-1000/added_tokens.json ADDED Viewed

	@@ -0,0 +1,3 @@

+{
+  "<image_soft_token>": 262144
+}

checkpoint-1000/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,47 @@

+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-1000/config.json ADDED Viewed

	@@ -0,0 +1,110 @@

+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eoi_token_index": 256000,
+  "eos_token_id": 1,
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "pad_token_id": 0,
+  "text_config": {
+    "_sliding_window_pattern": 6,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "dtype": "bfloat16",
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3840,
+    "initializer_range": 0.02,
+    "intermediate_size": 15360,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_local_base_freq": 10000.0,
+    "rope_scaling": {
+      "factor": 8.0,
+      "rope_type": "linear"
+    },
+    "rope_theta": 1000000.0,
+    "sliding_window": 1024,
+    "use_cache": true,
+    "vocab_size": 262208
+  },
+  "transformers_version": "4.56.1",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "vision_use_head": false
+  }
+}

checkpoint-1000/generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.56.1"
+}

checkpoint-1000/model-00001-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4847447e92599833e8dbaa3067cd201c3bb5c052efa91f11ba891e43234f7832
+size 4979902192

checkpoint-1000/model-00002-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:891bd54eed03cba9ee1e705533a02a8217fcc29f356e4a1f53e5fd0d178883ad
+size 4931296592

checkpoint-1000/model-00003-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:53d803eda94a0ef571b427b8299a4a8d23926e589090356aa3a7fa6d8a5ced45
+size 4931296656

checkpoint-1000/model-00004-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e0086bc8b940b102334bfacd8fbe36073a646feab1a45ad65aeea476bb6ca3ea
+size 4931296656

checkpoint-1000/model-00005-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b0ed4615790ae697259305fab7733f6d5ce838b8992c1011c2b5f1a021b8ee13
+size 4601000928

checkpoint-1000/model.safetensors.index.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-1000/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5698873fb054e8b7db9471a55ce7e7eae2aac8ca5dc7d0c0c034ce286534c2d9
+size 14573116445

checkpoint-1000/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bdf49fbe089c54734b0cb49eebe2b139e2eb7e2ba668acc6d856b30fcc0384d7
+size 14645

checkpoint-1000/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6078e52773a379f026b8fbbbc5546ab7ed9418d05b713f8aff2a4f3c7e12f108
+size 1465

checkpoint-1000/special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,33 @@

+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1000/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568

checkpoint-1000/tokenizer.model ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074

checkpoint-1000/tokenizer_config.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-1000/trainer_state.json ADDED Viewed

	@@ -0,0 +1,1474 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.4451121126133645,
+  "eval_steps": 100,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.7626848379150033,
+      "epoch": 0.004451121126133645,
+      "grad_norm": 1888.0,
+      "learning_rate": 1.8e-07,
+      "loss": 36.6657,
+      "mean_token_accuracy": 0.6676603280007839,
+      "num_tokens": 63133.0,
+      "step": 10
+    },
+    {
+      "entropy": 0.81443009134382,
+      "epoch": 0.00890224225226729,
+      "grad_norm": 1792.0,
+      "learning_rate": 3.8e-07,
+      "loss": 36.4468,
+      "mean_token_accuracy": 0.6505163595080375,
+      "num_tokens": 128968.0,
+      "step": 20
+    },
+    {
+      "entropy": 0.8291782144457102,
+      "epoch": 0.013353363378400934,
+      "grad_norm": 968.0,
+      "learning_rate": 5.800000000000001e-07,
+      "loss": 34.7077,
+      "mean_token_accuracy": 0.6580854222178459,
+      "num_tokens": 193314.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.9378003980964422,
+      "epoch": 0.01780448450453458,
+      "grad_norm": 656.0,
+      "learning_rate": 7.8e-07,
+      "loss": 33.9052,
+      "mean_token_accuracy": 0.6638370925560594,
+      "num_tokens": 257189.0,
+      "step": 40
+    },
+    {
+      "entropy": 0.9725235970690846,
+      "epoch": 0.022255605630668224,
+      "grad_norm": 724.0,
+      "learning_rate": 9.800000000000001e-07,
+      "loss": 29.8492,
+      "mean_token_accuracy": 0.673487476631999,
+      "num_tokens": 324224.0,
+      "step": 50
+    },
+    {
+      "entropy": 1.0304172853007914,
+      "epoch": 0.026706726756801868,
+      "grad_norm": 488.0,
+      "learning_rate": 1.1800000000000001e-06,
+      "loss": 28.3082,
+      "mean_token_accuracy": 0.6809290068224072,
+      "num_tokens": 394654.0,
+      "step": 60
+    },
+    {
+      "entropy": 1.000378421880305,
+      "epoch": 0.031157847882935515,
+      "grad_norm": 420.0,
+      "learning_rate": 1.3800000000000001e-06,
+      "loss": 26.217,
+      "mean_token_accuracy": 0.6916999943554402,
+      "num_tokens": 461658.0,
+      "step": 70
+    },
+    {
+      "entropy": 1.0228427596390248,
+      "epoch": 0.03560896900906916,
+      "grad_norm": 258.0,
+      "learning_rate": 1.5800000000000001e-06,
+      "loss": 23.6386,
+      "mean_token_accuracy": 0.7020359886810184,
+      "num_tokens": 528285.0,
+      "step": 80
+    },
+    {
+      "entropy": 1.0894161500036716,
+      "epoch": 0.04006009013520281,
+      "grad_norm": 218.0,
+      "learning_rate": 1.7800000000000001e-06,
+      "loss": 23.2798,
+      "mean_token_accuracy": 0.6986296704038978,
+      "num_tokens": 595336.0,
+      "step": 90
+    },
+    {
+      "entropy": 1.0631676118820905,
+      "epoch": 0.04451121126133645,
+      "grad_norm": 246.0,
+      "learning_rate": 1.98e-06,
+      "loss": 21.4229,
+      "mean_token_accuracy": 0.7073360819369554,
+      "num_tokens": 663126.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.04451121126133645,
+      "eval_biology_entropy": 1.48638627243042,
+      "eval_biology_loss": 1.9182748794555664,
+      "eval_biology_mean_token_accuracy": 0.6060935441255569,
+      "eval_biology_num_tokens": 663126.0,
+      "eval_biology_runtime": 49.1778,
+      "eval_biology_samples_per_second": 10.167,
+      "eval_biology_steps_per_second": 10.167,
+      "step": 100
+    },
+    {
+      "epoch": 0.04451121126133645,
+      "eval_chemistry_entropy": 1.0556317766308785,
+      "eval_chemistry_loss": 1.2898850440979004,
+      "eval_chemistry_mean_token_accuracy": 0.7124251598715782,
+      "eval_chemistry_num_tokens": 663126.0,
+      "eval_chemistry_runtime": 57.0461,
+      "eval_chemistry_samples_per_second": 8.765,
+      "eval_chemistry_steps_per_second": 8.765,
+      "step": 100
+    },
+    {
+      "epoch": 0.04451121126133645,
+      "eval_math_entropy": 0.8456157124042512,
+      "eval_math_loss": 1.1942517757415771,
+      "eval_math_mean_token_accuracy": 0.7334114202260971,
+      "eval_math_num_tokens": 663126.0,
+      "eval_math_runtime": 59.0482,
+      "eval_math_samples_per_second": 8.468,
+      "eval_math_steps_per_second": 8.468,
+      "step": 100
+    },
+    {
+      "epoch": 0.04451121126133645,
+      "eval_physics_entropy": 0.9410946148037911,
+      "eval_physics_loss": 1.1514631509780884,
+      "eval_physics_mean_token_accuracy": 0.7352130191326142,
+      "eval_physics_num_tokens": 663126.0,
+      "eval_physics_runtime": 65.895,
+      "eval_physics_samples_per_second": 7.588,
+      "eval_physics_steps_per_second": 7.588,
+      "step": 100
+    },
+    {
+      "entropy": 1.0484314041212202,
+      "epoch": 0.048962332387470095,
+      "grad_norm": 197.0,
+      "learning_rate": 2.1800000000000003e-06,
+      "loss": 19.9524,
+      "mean_token_accuracy": 0.7201540337875485,
+      "num_tokens": 731060.0,
+      "step": 110
+    },
+    {
+      "entropy": 1.012692185677588,
+      "epoch": 0.053413453513603736,
+      "grad_norm": 178.0,
+      "learning_rate": 2.38e-06,
+      "loss": 18.8663,
+      "mean_token_accuracy": 0.7315327662974596,
+      "num_tokens": 801635.0,
+      "step": 120
+    },
+    {
+      "entropy": 1.0016614170745015,
+      "epoch": 0.05786457463973738,
+      "grad_norm": 150.0,
+      "learning_rate": 2.5800000000000003e-06,
+      "loss": 19.0949,
+      "mean_token_accuracy": 0.7297663483768702,
+      "num_tokens": 867260.0,
+      "step": 130
+    },
+    {
+      "entropy": 1.007587356492877,
+      "epoch": 0.06231569576587103,
+      "grad_norm": 131.0,
+      "learning_rate": 2.7800000000000005e-06,
+      "loss": 18.9083,
+      "mean_token_accuracy": 0.7291399639099836,
+      "num_tokens": 931344.0,
+      "step": 140
+    },
+    {
+      "entropy": 1.0287242146208881,
+      "epoch": 0.06676681689200467,
+      "grad_norm": 179.0,
+      "learning_rate": 2.9800000000000003e-06,
+      "loss": 17.5655,
+      "mean_token_accuracy": 0.7393418021500111,
+      "num_tokens": 993586.0,
+      "step": 150
+    },
+    {
+      "entropy": 1.0563909145072103,
+      "epoch": 0.07121793801813832,
+      "grad_norm": 126.0,
+      "learning_rate": 3.1800000000000005e-06,
+      "loss": 16.9203,
+      "mean_token_accuracy": 0.7381794668734074,
+      "num_tokens": 1060812.0,
+      "step": 160
+    },
+    {
+      "entropy": 1.002509175427258,
+      "epoch": 0.07566905914427197,
+      "grad_norm": 127.0,
+      "learning_rate": 3.3800000000000007e-06,
+      "loss": 15.5704,
+      "mean_token_accuracy": 0.7520427949726581,
+      "num_tokens": 1131832.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.9855028538033366,
+      "epoch": 0.08012018027040561,
+      "grad_norm": 101.5,
+      "learning_rate": 3.58e-06,
+      "loss": 15.2418,
+      "mean_token_accuracy": 0.7483469128608704,
+      "num_tokens": 1200650.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.953622367605567,
+      "epoch": 0.08457130139653925,
+      "grad_norm": 117.0,
+      "learning_rate": 3.7800000000000002e-06,
+      "loss": 14.9841,
+      "mean_token_accuracy": 0.7532309770584107,
+      "num_tokens": 1265107.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.9146013628691435,
+      "epoch": 0.0890224225226729,
+      "grad_norm": 123.0,
+      "learning_rate": 3.980000000000001e-06,
+      "loss": 14.4834,
+      "mean_token_accuracy": 0.7538760716095567,
+      "num_tokens": 1327380.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.0890224225226729,
+      "eval_biology_entropy": 1.2566380687952041,
+      "eval_biology_loss": 1.4376745223999023,
+      "eval_biology_mean_token_accuracy": 0.6541494448781013,
+      "eval_biology_num_tokens": 1327380.0,
+      "eval_biology_runtime": 49.1875,
+      "eval_biology_samples_per_second": 10.165,
+      "eval_biology_steps_per_second": 10.165,
+      "step": 200
+    },
+    {
+      "epoch": 0.0890224225226729,
+      "eval_chemistry_entropy": 0.8981590569019318,
+      "eval_chemistry_loss": 0.9085330963134766,
+      "eval_chemistry_mean_token_accuracy": 0.7548674891591072,
+      "eval_chemistry_num_tokens": 1327380.0,
+      "eval_chemistry_runtime": 57.0026,
+      "eval_chemistry_samples_per_second": 8.772,
+      "eval_chemistry_steps_per_second": 8.772,
+      "step": 200
+    },
+    {
+      "epoch": 0.0890224225226729,
+      "eval_math_entropy": 0.7623042597174644,
+      "eval_math_loss": 0.8974745869636536,
+      "eval_math_mean_token_accuracy": 0.7647860664129257,
+      "eval_math_num_tokens": 1327380.0,
+      "eval_math_runtime": 58.8532,
+      "eval_math_samples_per_second": 8.496,
+      "eval_math_steps_per_second": 8.496,
+      "step": 200
+    },
+    {
+      "epoch": 0.0890224225226729,
+      "eval_physics_entropy": 0.8175762248635292,
+      "eval_physics_loss": 0.8456214666366577,
+      "eval_physics_mean_token_accuracy": 0.771473237156868,
+      "eval_physics_num_tokens": 1327380.0,
+      "eval_physics_runtime": 65.7934,
+      "eval_physics_samples_per_second": 7.6,
+      "eval_physics_steps_per_second": 7.6,
+      "step": 200
+    },
+    {
+      "entropy": 0.9297651303932071,
+      "epoch": 0.09347354364880654,
+      "grad_norm": 121.0,
+      "learning_rate": 4.18e-06,
+      "loss": 15.1938,
+      "mean_token_accuracy": 0.7461945816874505,
+      "num_tokens": 1393379.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.8862789057195186,
+      "epoch": 0.09792466477494019,
+      "grad_norm": 98.0,
+      "learning_rate": 4.38e-06,
+      "loss": 14.1582,
+      "mean_token_accuracy": 0.759770842641592,
+      "num_tokens": 1460130.0,
+      "step": 220
+    },
+    {
+      "entropy": 0.8502754125744104,
+      "epoch": 0.10237578590107384,
+      "grad_norm": 93.5,
+      "learning_rate": 4.58e-06,
+      "loss": 13.6798,
+      "mean_token_accuracy": 0.7648626536130905,
+      "num_tokens": 1529183.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.8967906672507524,
+      "epoch": 0.10682690702720747,
+      "grad_norm": 117.0,
+      "learning_rate": 4.78e-06,
+      "loss": 14.5446,
+      "mean_token_accuracy": 0.7548742420971394,
+      "num_tokens": 1597405.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.8926160346716643,
+      "epoch": 0.11127802815334112,
+      "grad_norm": 97.0,
+      "learning_rate": 4.980000000000001e-06,
+      "loss": 14.4754,
+      "mean_token_accuracy": 0.7543840739876032,
+      "num_tokens": 1661767.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.906158303655684,
+      "epoch": 0.11572914927947477,
+      "grad_norm": 81.5,
+      "learning_rate": 5.18e-06,
+      "loss": 14.5882,
+      "mean_token_accuracy": 0.7560893476009369,
+      "num_tokens": 1728207.0,
+      "step": 260
+    },
+    {
+      "entropy": 0.8557273065671325,
+      "epoch": 0.12018027040560841,
+      "grad_norm": 98.5,
+      "learning_rate": 5.380000000000001e-06,
+      "loss": 13.7256,
+      "mean_token_accuracy": 0.7653402671217918,
+      "num_tokens": 1796055.0,
+      "step": 270
+    },
+    {
+      "entropy": 0.824279030226171,
+      "epoch": 0.12463139153174206,
+      "grad_norm": 97.0,
+      "learning_rate": 5.580000000000001e-06,
+      "loss": 13.2294,
+      "mean_token_accuracy": 0.7715799886733293,
+      "num_tokens": 1859684.0,
+      "step": 280
+    },
+    {
+      "entropy": 0.885771127231419,
+      "epoch": 0.1290825126578757,
+      "grad_norm": 102.5,
+      "learning_rate": 5.78e-06,
+      "loss": 14.3879,
+      "mean_token_accuracy": 0.7568217847496271,
+      "num_tokens": 1921704.0,
+      "step": 290
+    },
+    {
+      "entropy": 0.868380057439208,
+      "epoch": 0.13353363378400934,
+      "grad_norm": 87.5,
+      "learning_rate": 5.98e-06,
+      "loss": 13.9363,
+      "mean_token_accuracy": 0.7626302808523178,
+      "num_tokens": 1985766.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.13353363378400934,
+      "eval_biology_entropy": 1.2238210166096688,
+      "eval_biology_loss": 1.366097331047058,
+      "eval_biology_mean_token_accuracy": 0.6672504586577416,
+      "eval_biology_num_tokens": 1985766.0,
+      "eval_biology_runtime": 49.1235,
+      "eval_biology_samples_per_second": 10.178,
+      "eval_biology_steps_per_second": 10.178,
+      "step": 300
+    },
+    {
+      "epoch": 0.13353363378400934,
+      "eval_chemistry_entropy": 0.8314956972002983,
+      "eval_chemistry_loss": 0.8379236459732056,
+      "eval_chemistry_mean_token_accuracy": 0.7699049721360207,
+      "eval_chemistry_num_tokens": 1985766.0,
+      "eval_chemistry_runtime": 57.0049,
+      "eval_chemistry_samples_per_second": 8.771,
+      "eval_chemistry_steps_per_second": 8.771,
+      "step": 300
+    },
+    {
+      "epoch": 0.13353363378400934,
+      "eval_math_entropy": 0.714144405901432,
+      "eval_math_loss": 0.874640703201294,
+      "eval_math_mean_token_accuracy": 0.7714004491567612,
+      "eval_math_num_tokens": 1985766.0,
+      "eval_math_runtime": 58.9383,
+      "eval_math_samples_per_second": 8.483,
+      "eval_math_steps_per_second": 8.483,
+      "step": 300
+    },
+    {
+      "epoch": 0.13353363378400934,
+      "eval_physics_entropy": 0.7712625939249992,
+      "eval_physics_loss": 0.787721574306488,
+      "eval_physics_mean_token_accuracy": 0.7829854927659035,
+      "eval_physics_num_tokens": 1985766.0,
+      "eval_physics_runtime": 65.8905,
+      "eval_physics_samples_per_second": 7.588,
+      "eval_physics_steps_per_second": 7.588,
+      "step": 300
+    },
+    {
+      "entropy": 0.8396226148121059,
+      "epoch": 0.137984754910143,
+      "grad_norm": 96.5,
+      "learning_rate": 6.18e-06,
+      "loss": 13.612,
+      "mean_token_accuracy": 0.7638351432979107,
+      "num_tokens": 2049041.0,
+      "step": 310
+    },
+    {
+      "entropy": 0.8020199347287417,
+      "epoch": 0.14243587603627664,
+      "grad_norm": 81.0,
+      "learning_rate": 6.380000000000001e-06,
+      "loss": 12.8945,
+      "mean_token_accuracy": 0.776324712112546,
+      "num_tokens": 2118068.0,
+      "step": 320
+    },
+    {
+      "entropy": 0.8072531137615442,
+      "epoch": 0.14688699716241027,
+      "grad_norm": 107.5,
+      "learning_rate": 6.5800000000000005e-06,
+      "loss": 12.776,
+      "mean_token_accuracy": 0.7789770424365997,
+      "num_tokens": 2186387.0,
+      "step": 330
+    },
+    {
+      "entropy": 0.8215415397658944,
+      "epoch": 0.15133811828854393,
+      "grad_norm": 84.5,
+      "learning_rate": 6.780000000000001e-06,
+      "loss": 13.3957,
+      "mean_token_accuracy": 0.7686271335929632,
+      "num_tokens": 2252650.0,
+      "step": 340
+    },
+    {
+      "entropy": 0.7919685224071145,
+      "epoch": 0.15578923941467757,
+      "grad_norm": 87.0,
+      "learning_rate": 6.98e-06,
+      "loss": 12.7862,
+      "mean_token_accuracy": 0.7796416375786066,
+      "num_tokens": 2320358.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.7888439664617181,
+      "epoch": 0.16024036054081123,
+      "grad_norm": 90.5,
+      "learning_rate": 7.180000000000001e-06,
+      "loss": 12.6082,
+      "mean_token_accuracy": 0.7811494890600443,
+      "num_tokens": 2388824.0,
+      "step": 360
+    },
+    {
+      "entropy": 0.7693003987893462,
+      "epoch": 0.16469148166694486,
+      "grad_norm": 82.0,
+      "learning_rate": 7.3800000000000005e-06,
+      "loss": 12.3808,
+      "mean_token_accuracy": 0.7856796395033598,
+      "num_tokens": 2456144.0,
+      "step": 370
+    },
+    {
+      "entropy": 0.8541094154119492,
+      "epoch": 0.1691426027930785,
+      "grad_norm": 80.0,
+      "learning_rate": 7.58e-06,
+      "loss": 13.8583,
+      "mean_token_accuracy": 0.7654584795236588,
+      "num_tokens": 2515325.0,
+      "step": 380
+    },
+    {
+      "entropy": 0.7904249761253596,
+      "epoch": 0.17359372391921216,
+      "grad_norm": 68.5,
+      "learning_rate": 7.78e-06,
+      "loss": 12.659,
+      "mean_token_accuracy": 0.7788146499544382,
+      "num_tokens": 2580490.0,
+      "step": 390
+    },
+    {
+      "entropy": 0.7941560545936227,
+      "epoch": 0.1780448450453458,
+      "grad_norm": 72.5,
+      "learning_rate": 7.980000000000002e-06,
+      "loss": 12.8107,
+      "mean_token_accuracy": 0.7767765510827302,
+      "num_tokens": 2644330.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.1780448450453458,
+      "eval_biology_entropy": 1.2128318169116974,
+      "eval_biology_loss": 1.3349602222442627,
+      "eval_biology_mean_token_accuracy": 0.6734810082316398,
+      "eval_biology_num_tokens": 2644330.0,
+      "eval_biology_runtime": 49.0571,
+      "eval_biology_samples_per_second": 10.192,
+      "eval_biology_steps_per_second": 10.192,
+      "step": 400
+    },
+    {
+      "epoch": 0.1780448450453458,
+      "eval_chemistry_entropy": 0.7846001248955726,
+      "eval_chemistry_loss": 0.798675537109375,
+      "eval_chemistry_mean_token_accuracy": 0.7786763918995857,
+      "eval_chemistry_num_tokens": 2644330.0,
+      "eval_chemistry_runtime": 56.9232,
+      "eval_chemistry_samples_per_second": 8.784,
+      "eval_chemistry_steps_per_second": 8.784,
+      "step": 400
+    },
+    {
+      "epoch": 0.1780448450453458,
+      "eval_math_entropy": 0.6919120717644691,
+      "eval_math_loss": 0.8597798943519592,
+      "eval_math_mean_token_accuracy": 0.7751281379461289,
+      "eval_math_num_tokens": 2644330.0,
+      "eval_math_runtime": 58.7801,
+      "eval_math_samples_per_second": 8.506,
+      "eval_math_steps_per_second": 8.506,
+      "step": 400
+    },
+    {
+      "epoch": 0.1780448450453458,
+      "eval_physics_entropy": 0.7369731608629226,
+      "eval_physics_loss": 0.7621525526046753,
+      "eval_physics_mean_token_accuracy": 0.7884504457712174,
+      "eval_physics_num_tokens": 2644330.0,
+      "eval_physics_runtime": 65.7104,
+      "eval_physics_samples_per_second": 7.609,
+      "eval_physics_steps_per_second": 7.609,
+      "step": 400
+    },
+    {
+      "entropy": 0.7377481091767549,
+      "epoch": 0.18249596617147945,
+      "grad_norm": 73.0,
+      "learning_rate": 8.18e-06,
+      "loss": 11.7926,
+      "mean_token_accuracy": 0.7937501568347216,
+      "num_tokens": 2713234.0,
+      "step": 410
+    },
+    {
+      "entropy": 0.7830979386344552,
+      "epoch": 0.18694708729761308,
+      "grad_norm": 76.5,
+      "learning_rate": 8.380000000000001e-06,
+      "loss": 12.9301,
+      "mean_token_accuracy": 0.7784280318766832,
+      "num_tokens": 2783261.0,
+      "step": 420
+    },
+    {
+      "entropy": 0.7945666501298547,
+      "epoch": 0.19139820842374672,
+      "grad_norm": 61.5,
+      "learning_rate": 8.580000000000001e-06,
+      "loss": 12.6515,
+      "mean_token_accuracy": 0.7819751728326082,
+      "num_tokens": 2850170.0,
+      "step": 430
+    },
+    {
+      "entropy": 0.7770357062108815,
+      "epoch": 0.19584932954988038,
+      "grad_norm": 75.0,
+      "learning_rate": 8.78e-06,
+      "loss": 12.5072,
+      "mean_token_accuracy": 0.7829508323222398,
+      "num_tokens": 2913700.0,
+      "step": 440
+    },
+    {
+      "entropy": 0.7575128111988306,
+      "epoch": 0.200300450676014,
+      "grad_norm": 81.0,
+      "learning_rate": 8.98e-06,
+      "loss": 12.4714,
+      "mean_token_accuracy": 0.7839657858014106,
+      "num_tokens": 2981212.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.7656491708010436,
+      "epoch": 0.20475157180214767,
+      "grad_norm": 85.0,
+      "learning_rate": 9.180000000000002e-06,
+      "loss": 12.0974,
+      "mean_token_accuracy": 0.7915447618812322,
+      "num_tokens": 3048973.0,
+      "step": 460
+    },
+    {
+      "entropy": 0.7621381992474199,
+      "epoch": 0.2092026929282813,
+      "grad_norm": 61.75,
+      "learning_rate": 9.38e-06,
+      "loss": 12.5808,
+      "mean_token_accuracy": 0.7818816823884844,
+      "num_tokens": 3117033.0,
+      "step": 470
+    },
+    {
+      "entropy": 0.7816813634708524,
+      "epoch": 0.21365381405441494,
+      "grad_norm": 73.5,
+      "learning_rate": 9.58e-06,
+      "loss": 12.3819,
+      "mean_token_accuracy": 0.7846902538090944,
+      "num_tokens": 3185255.0,
+      "step": 480
+    },
+    {
+      "entropy": 0.7647983279079199,
+      "epoch": 0.2181049351805486,
+      "grad_norm": 76.5,
+      "learning_rate": 9.780000000000001e-06,
+      "loss": 12.4249,
+      "mean_token_accuracy": 0.7847896091639995,
+      "num_tokens": 3253121.0,
+      "step": 490
+    },
+    {
+      "entropy": 0.7214844634756445,
+      "epoch": 0.22255605630668224,
+      "grad_norm": 75.5,
+      "learning_rate": 9.980000000000001e-06,
+      "loss": 11.4214,
+      "mean_token_accuracy": 0.7984224483370781,
+      "num_tokens": 3322823.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.22255605630668224,
+      "eval_biology_entropy": 1.1607659912109376,
+      "eval_biology_loss": 1.328081488609314,
+      "eval_biology_mean_token_accuracy": 0.6758214622139931,
+      "eval_biology_num_tokens": 3322823.0,
+      "eval_biology_runtime": 49.1342,
+      "eval_biology_samples_per_second": 10.176,
+      "eval_biology_steps_per_second": 10.176,
+      "step": 500
+    },
+    {
+      "epoch": 0.22255605630668224,
+      "eval_chemistry_entropy": 0.7407792928814888,
+      "eval_chemistry_loss": 0.775718629360199,
+      "eval_chemistry_mean_token_accuracy": 0.7834744991064072,
+      "eval_chemistry_num_tokens": 3322823.0,
+      "eval_chemistry_runtime": 57.1618,
+      "eval_chemistry_samples_per_second": 8.747,
+      "eval_chemistry_steps_per_second": 8.747,
+      "step": 500
+    },
+    {
+      "epoch": 0.22255605630668224,
+      "eval_math_entropy": 0.6776470097601414,
+      "eval_math_loss": 0.8585710525512695,
+      "eval_math_mean_token_accuracy": 0.7761784045696258,
+      "eval_math_num_tokens": 3322823.0,
+      "eval_math_runtime": 58.8725,
+      "eval_math_samples_per_second": 8.493,
+      "eval_math_steps_per_second": 8.493,
+      "step": 500
+    },
+    {
+      "epoch": 0.22255605630668224,
+      "eval_physics_entropy": 0.7148050211071968,
+      "eval_physics_loss": 0.7503296732902527,
+      "eval_physics_mean_token_accuracy": 0.7919680835604668,
+      "eval_physics_num_tokens": 3322823.0,
+      "eval_physics_runtime": 65.8508,
+      "eval_physics_samples_per_second": 7.593,
+      "eval_physics_steps_per_second": 7.593,
+      "step": 500
+    },
+    {
+      "entropy": 0.7285958690568805,
+      "epoch": 0.2270071774328159,
+      "grad_norm": 72.5,
+      "learning_rate": 1.018e-05,
+      "loss": 11.8973,
+      "mean_token_accuracy": 0.7914360746741295,
+      "num_tokens": 3393393.0,
+      "step": 510
+    },
+    {
+      "entropy": 0.8255035754293203,
+      "epoch": 0.23145829855894953,
+      "grad_norm": 80.0,
+      "learning_rate": 1.038e-05,
+      "loss": 13.0946,
+      "mean_token_accuracy": 0.774183202162385,
+      "num_tokens": 3454750.0,
+      "step": 520
+    },
+    {
+      "entropy": 0.708361669909209,
+      "epoch": 0.2359094196850832,
+      "grad_norm": 71.0,
+      "learning_rate": 1.0580000000000002e-05,
+      "loss": 11.7446,
+      "mean_token_accuracy": 0.7961884457617998,
+      "num_tokens": 3526914.0,
+      "step": 530
+    },
+    {
+      "entropy": 0.8215967276133597,
+      "epoch": 0.24036054081121683,
+      "grad_norm": 79.0,
+      "learning_rate": 1.0780000000000002e-05,
+      "loss": 13.0207,
+      "mean_token_accuracy": 0.7765944451093674,
+      "num_tokens": 3589449.0,
+      "step": 540
+    },
+    {
+      "entropy": 0.7564663768745958,
+      "epoch": 0.24481166193735046,
+      "grad_norm": 82.0,
+      "learning_rate": 1.0980000000000002e-05,
+      "loss": 12.2873,
+      "mean_token_accuracy": 0.7854228101670742,
+      "num_tokens": 3655092.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.7720718337222934,
+      "epoch": 0.24926278306348412,
+      "grad_norm": 90.5,
+      "learning_rate": 1.1180000000000001e-05,
+      "loss": 12.4637,
+      "mean_token_accuracy": 0.7846879370510578,
+      "num_tokens": 3719113.0,
+      "step": 560
+    },
+    {
+      "entropy": 0.7583298075944185,
+      "epoch": 0.25371390418961776,
+      "grad_norm": 66.5,
+      "learning_rate": 1.138e-05,
+      "loss": 12.4348,
+      "mean_token_accuracy": 0.781843975931406,
+      "num_tokens": 3785282.0,
+      "step": 570
+    },
+    {
+      "entropy": 0.7643377220258116,
+      "epoch": 0.2581650253157514,
+      "grad_norm": 63.75,
+      "learning_rate": 1.1580000000000001e-05,
+      "loss": 12.4092,
+      "mean_token_accuracy": 0.7843703482300043,
+      "num_tokens": 3852276.0,
+      "step": 580
+    },
+    {
+      "entropy": 0.7012410337105394,
+      "epoch": 0.262616146441885,
+      "grad_norm": 67.0,
+      "learning_rate": 1.178e-05,
+      "loss": 11.3623,
+      "mean_token_accuracy": 0.7991111058741808,
+      "num_tokens": 3925649.0,
+      "step": 590
+    },
+    {
+      "entropy": 0.7401338685303926,
+      "epoch": 0.2670672675680187,
+      "grad_norm": 71.5,
+      "learning_rate": 1.198e-05,
+      "loss": 11.9064,
+      "mean_token_accuracy": 0.7897865924984216,
+      "num_tokens": 3990505.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.2670672675680187,
+      "eval_biology_entropy": 1.216272303700447,
+      "eval_biology_loss": 1.3142751455307007,
+      "eval_biology_mean_token_accuracy": 0.6766086940765381,
+      "eval_biology_num_tokens": 3990505.0,
+      "eval_biology_runtime": 49.3541,
+      "eval_biology_samples_per_second": 10.131,
+      "eval_biology_steps_per_second": 10.131,
+      "step": 600
+    },
+    {
+      "epoch": 0.2670672675680187,
+      "eval_chemistry_entropy": 0.7538464320898056,
+      "eval_chemistry_loss": 0.7621961832046509,
+      "eval_chemistry_mean_token_accuracy": 0.7865861940383911,
+      "eval_chemistry_num_tokens": 3990505.0,
+      "eval_chemistry_runtime": 57.3005,
+      "eval_chemistry_samples_per_second": 8.726,
+      "eval_chemistry_steps_per_second": 8.726,
+      "step": 600
+    },
+    {
+      "epoch": 0.2670672675680187,
+      "eval_math_entropy": 0.6806064413189888,
+      "eval_math_loss": 0.8638136982917786,
+      "eval_math_mean_token_accuracy": 0.7749927558898926,
+      "eval_math_num_tokens": 3990505.0,
+      "eval_math_runtime": 59.1764,
+      "eval_math_samples_per_second": 8.449,
+      "eval_math_steps_per_second": 8.449,
+      "step": 600
+    },
+    {
+      "epoch": 0.2670672675680187,
+      "eval_physics_entropy": 0.7255966661572456,
+      "eval_physics_loss": 0.7440791726112366,
+      "eval_physics_mean_token_accuracy": 0.7921827912330628,
+      "eval_physics_num_tokens": 3990505.0,
+      "eval_physics_runtime": 66.0704,
+      "eval_physics_samples_per_second": 7.568,
+      "eval_physics_steps_per_second": 7.568,
+      "step": 600
+    },
+    {
+      "entropy": 0.738010767288506,
+      "epoch": 0.27151838869415235,
+      "grad_norm": 64.0,
+      "learning_rate": 1.218e-05,
+      "loss": 11.9409,
+      "mean_token_accuracy": 0.7896174006164074,
+      "num_tokens": 4060657.0,
+      "step": 610
+    },
+    {
+      "entropy": 0.7203703055158257,
+      "epoch": 0.275969509820286,
+      "grad_norm": 67.0,
+      "learning_rate": 1.2380000000000002e-05,
+      "loss": 11.5655,
+      "mean_token_accuracy": 0.7945824809372425,
+      "num_tokens": 4133119.0,
+      "step": 620
+    },
+    {
+      "entropy": 0.743073652498424,
+      "epoch": 0.2804206309464196,
+      "grad_norm": 69.5,
+      "learning_rate": 1.2580000000000002e-05,
+      "loss": 12.0073,
+      "mean_token_accuracy": 0.7911369562149048,
+      "num_tokens": 4200051.0,
+      "step": 630
+    },
+    {
+      "entropy": 0.7967112486250698,
+      "epoch": 0.2848717520725533,
+      "grad_norm": 72.5,
+      "learning_rate": 1.2780000000000001e-05,
+      "loss": 12.7974,
+      "mean_token_accuracy": 0.7836818125098943,
+      "num_tokens": 4267403.0,
+      "step": 640
+    },
+    {
+      "entropy": 0.7334000766277313,
+      "epoch": 0.28932287319868694,
+      "grad_norm": 71.5,
+      "learning_rate": 1.2980000000000001e-05,
+      "loss": 11.8952,
+      "mean_token_accuracy": 0.7923904918134212,
+      "num_tokens": 4333268.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.7248436002992094,
+      "epoch": 0.29377399432482054,
+      "grad_norm": 79.5,
+      "learning_rate": 1.3180000000000001e-05,
+      "loss": 11.7473,
+      "mean_token_accuracy": 0.7948706120252609,
+      "num_tokens": 4400242.0,
+      "step": 660
+    },
+    {
+      "entropy": 0.7555086806416511,
+      "epoch": 0.2982251154509542,
+      "grad_norm": 77.0,
+      "learning_rate": 1.3380000000000002e-05,
+      "loss": 12.0493,
+      "mean_token_accuracy": 0.7891406118869781,
+      "num_tokens": 4464714.0,
+      "step": 670
+    },
+    {
+      "entropy": 0.7099162889644504,
+      "epoch": 0.30267623657708786,
+      "grad_norm": 62.75,
+      "learning_rate": 1.3580000000000002e-05,
+      "loss": 11.5969,
+      "mean_token_accuracy": 0.7962878335267305,
+      "num_tokens": 4535458.0,
+      "step": 680
+    },
+    {
+      "entropy": 0.7450342487543822,
+      "epoch": 0.3071273577032215,
+      "grad_norm": 66.5,
+      "learning_rate": 1.378e-05,
+      "loss": 11.9777,
+      "mean_token_accuracy": 0.7921688258647919,
+      "num_tokens": 4599099.0,
+      "step": 690
+    },
+    {
+      "entropy": 0.7137214630842209,
+      "epoch": 0.31157847882935513,
+      "grad_norm": 56.5,
+      "learning_rate": 1.398e-05,
+      "loss": 11.5022,
+      "mean_token_accuracy": 0.7959922354668378,
+      "num_tokens": 4662903.0,
+      "step": 700
+    },
+    {
+      "epoch": 0.31157847882935513,
+      "eval_biology_entropy": 1.21318653935194,
+      "eval_biology_loss": 1.3126111030578613,
+      "eval_biology_mean_token_accuracy": 0.6781674832701683,
+      "eval_biology_num_tokens": 4662903.0,
+      "eval_biology_runtime": 49.2021,
+      "eval_biology_samples_per_second": 10.162,
+      "eval_biology_steps_per_second": 10.162,
+      "step": 700
+    },
+    {
+      "epoch": 0.31157847882935513,
+      "eval_chemistry_entropy": 0.7448744902908802,
+      "eval_chemistry_loss": 0.7531416416168213,
+      "eval_chemistry_mean_token_accuracy": 0.788379067659378,
+      "eval_chemistry_num_tokens": 4662903.0,
+      "eval_chemistry_runtime": 57.0266,
+      "eval_chemistry_samples_per_second": 8.768,
+      "eval_chemistry_steps_per_second": 8.768,
+      "step": 700
+    },
+    {
+      "epoch": 0.31157847882935513,
+      "eval_math_entropy": 0.6775659238696098,
+      "eval_math_loss": 0.8588793277740479,
+      "eval_math_mean_token_accuracy": 0.7769290524721145,
+      "eval_math_num_tokens": 4662903.0,
+      "eval_math_runtime": 58.9903,
+      "eval_math_samples_per_second": 8.476,
+      "eval_math_steps_per_second": 8.476,
+      "step": 700
+    },
+    {
+      "epoch": 0.31157847882935513,
+      "eval_physics_entropy": 0.7208585093617439,
+      "eval_physics_loss": 0.7438266277313232,
+      "eval_physics_mean_token_accuracy": 0.792436719596386,
+      "eval_physics_num_tokens": 4662903.0,
+      "eval_physics_runtime": 65.9428,
+      "eval_physics_samples_per_second": 7.582,
+      "eval_physics_steps_per_second": 7.582,
+      "step": 700
+    },
+    {
+      "entropy": 0.7532984343357384,
+      "epoch": 0.3160295999554888,
+      "grad_norm": 82.5,
+      "learning_rate": 1.418e-05,
+      "loss": 12.3646,
+      "mean_token_accuracy": 0.7847790695726872,
+      "num_tokens": 4729166.0,
+      "step": 710
+    },
+    {
+      "entropy": 0.7946592880412936,
+      "epoch": 0.32048072108162245,
+      "grad_norm": 79.0,
+      "learning_rate": 1.4380000000000001e-05,
+      "loss": 12.7491,
+      "mean_token_accuracy": 0.7800281438976526,
+      "num_tokens": 4796815.0,
+      "step": 720
+    },
+    {
+      "entropy": 0.6913739880546927,
+      "epoch": 0.32493184220775606,
+      "grad_norm": 68.0,
+      "learning_rate": 1.4580000000000001e-05,
+      "loss": 11.0843,
+      "mean_token_accuracy": 0.803905238583684,
+      "num_tokens": 4865172.0,
+      "step": 730
+    },
+    {
+      "entropy": 0.7447662945836783,
+      "epoch": 0.3293829633338897,
+      "grad_norm": 65.5,
+      "learning_rate": 1.478e-05,
+      "loss": 12.1577,
+      "mean_token_accuracy": 0.7854402553290128,
+      "num_tokens": 4932671.0,
+      "step": 740
+    },
+    {
+      "entropy": 0.7749863206408918,
+      "epoch": 0.3338340844600234,
+      "grad_norm": 64.0,
+      "learning_rate": 1.498e-05,
+      "loss": 12.4497,
+      "mean_token_accuracy": 0.7847358033061027,
+      "num_tokens": 4998710.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.7126937594264746,
+      "epoch": 0.338285205586157,
+      "grad_norm": 65.5,
+      "learning_rate": 1.5180000000000002e-05,
+      "loss": 11.549,
+      "mean_token_accuracy": 0.7943205136805773,
+      "num_tokens": 5066948.0,
+      "step": 760
+    },
+    {
+      "entropy": 0.7463799838908016,
+      "epoch": 0.34273632671229065,
+      "grad_norm": 66.5,
+      "learning_rate": 1.5380000000000002e-05,
+      "loss": 12.052,
+      "mean_token_accuracy": 0.7893575474619865,
+      "num_tokens": 5129950.0,
+      "step": 770
+    },
+    {
+      "entropy": 0.7669129572808743,
+      "epoch": 0.3471874478384243,
+      "grad_norm": 64.5,
+      "learning_rate": 1.5580000000000003e-05,
+      "loss": 12.4587,
+      "mean_token_accuracy": 0.7811328556388617,
+      "num_tokens": 5193519.0,
+      "step": 780
+    },
+    {
+      "entropy": 0.7404427804984153,
+      "epoch": 0.351638568964558,
+      "grad_norm": 87.0,
+      "learning_rate": 1.578e-05,
+      "loss": 11.9383,
+      "mean_token_accuracy": 0.7870475023984909,
+      "num_tokens": 5256503.0,
+      "step": 790
+    },
+    {
+      "entropy": 0.8177574999630451,
+      "epoch": 0.3560896900906916,
+      "grad_norm": 69.0,
+      "learning_rate": 1.5980000000000003e-05,
+      "loss": 13.1167,
+      "mean_token_accuracy": 0.7771423246711493,
+      "num_tokens": 5324751.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.3560896900906916,
+      "eval_biology_entropy": 1.2992450638413429,
+      "eval_biology_loss": 1.3091635704040527,
+      "eval_biology_mean_token_accuracy": 0.6765963944792748,
+      "eval_biology_num_tokens": 5324751.0,
+      "eval_biology_runtime": 49.4045,
+      "eval_biology_samples_per_second": 10.121,
+      "eval_biology_steps_per_second": 10.121,
+      "step": 800
+    },
+    {
+      "epoch": 0.3560896900906916,
+      "eval_chemistry_entropy": 0.7858416083455085,
+      "eval_chemistry_loss": 0.755605936050415,
+      "eval_chemistry_mean_token_accuracy": 0.7878280827403068,
+      "eval_chemistry_num_tokens": 5324751.0,
+      "eval_chemistry_runtime": 57.2096,
+      "eval_chemistry_samples_per_second": 8.74,
+      "eval_chemistry_steps_per_second": 8.74,
+      "step": 800
+    },
+    {
+      "epoch": 0.3560896900906916,
+      "eval_math_entropy": 0.7085552672445774,
+      "eval_math_loss": 0.8628776669502258,
+      "eval_math_mean_token_accuracy": 0.7751141096353531,
+      "eval_math_num_tokens": 5324751.0,
+      "eval_math_runtime": 58.9925,
+      "eval_math_samples_per_second": 8.476,
+      "eval_math_steps_per_second": 8.476,
+      "step": 800
+    },
+    {
+      "epoch": 0.3560896900906916,
+      "eval_physics_entropy": 0.7641420513391495,
+      "eval_physics_loss": 0.7484508156776428,
+      "eval_physics_mean_token_accuracy": 0.7914790090918541,
+      "eval_physics_num_tokens": 5324751.0,
+      "eval_physics_runtime": 65.8581,
+      "eval_physics_samples_per_second": 7.592,
+      "eval_physics_steps_per_second": 7.592,
+      "step": 800
+    },
+    {
+      "entropy": 0.6714803985320031,
+      "epoch": 0.36054081121682524,
+      "grad_norm": 65.5,
+      "learning_rate": 1.618e-05,
+      "loss": 11.0959,
+      "mean_token_accuracy": 0.8037038501352072,
+      "num_tokens": 5391664.0,
+      "step": 810
+    },
+    {
+      "entropy": 0.7128949659876526,
+      "epoch": 0.3649919323429589,
+      "grad_norm": 71.5,
+      "learning_rate": 1.638e-05,
+      "loss": 11.223,
+      "mean_token_accuracy": 0.7998184654861689,
+      "num_tokens": 5458519.0,
+      "step": 820
+    },
+    {
+      "entropy": 0.7470979234203696,
+      "epoch": 0.3694430534690925,
+      "grad_norm": 64.5,
+      "learning_rate": 1.658e-05,
+      "loss": 12.1501,
+      "mean_token_accuracy": 0.7894150543957948,
+      "num_tokens": 5523391.0,
+      "step": 830
+    },
+    {
+      "entropy": 0.7144336312077939,
+      "epoch": 0.37389417459522617,
+      "grad_norm": 61.25,
+      "learning_rate": 1.6780000000000002e-05,
+      "loss": 11.736,
+      "mean_token_accuracy": 0.7935146864503622,
+      "num_tokens": 5585508.0,
+      "step": 840
+    },
+    {
+      "entropy": 0.7430889600887894,
+      "epoch": 0.37834529572135983,
+      "grad_norm": 72.0,
+      "learning_rate": 1.698e-05,
+      "loss": 11.8763,
+      "mean_token_accuracy": 0.7954710725694895,
+      "num_tokens": 5650755.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.7431833678856492,
+      "epoch": 0.38279641684749344,
+      "grad_norm": 62.25,
+      "learning_rate": 1.718e-05,
+      "loss": 11.9703,
+      "mean_token_accuracy": 0.7893174722790718,
+      "num_tokens": 5716321.0,
+      "step": 860
+    },
+    {
+      "entropy": 0.7719974908977747,
+      "epoch": 0.3872475379736271,
+      "grad_norm": 66.5,
+      "learning_rate": 1.7380000000000003e-05,
+      "loss": 12.4773,
+      "mean_token_accuracy": 0.7817879624664783,
+      "num_tokens": 5782698.0,
+      "step": 870
+    },
+    {
+      "entropy": 0.7235654108226299,
+      "epoch": 0.39169865909976076,
+      "grad_norm": 53.75,
+      "learning_rate": 1.758e-05,
+      "loss": 11.6881,
+      "mean_token_accuracy": 0.7943595319986343,
+      "num_tokens": 5848889.0,
+      "step": 880
+    },
+    {
+      "entropy": 0.7637507017701864,
+      "epoch": 0.3961497802258944,
+      "grad_norm": 64.0,
+      "learning_rate": 1.7780000000000003e-05,
+      "loss": 12.5431,
+      "mean_token_accuracy": 0.7863847129046917,
+      "num_tokens": 5912919.0,
+      "step": 890
+    },
+    {
+      "entropy": 0.742395118996501,
+      "epoch": 0.400600901352028,
+      "grad_norm": 57.5,
+      "learning_rate": 1.798e-05,
+      "loss": 11.9098,
+      "mean_token_accuracy": 0.79401213824749,
+      "num_tokens": 5980229.0,
+      "step": 900
+    },
+    {
+      "epoch": 0.400600901352028,
+      "eval_biology_entropy": 1.2397134986519813,
+      "eval_biology_loss": 1.3142763376235962,
+      "eval_biology_mean_token_accuracy": 0.6771629434227944,
+      "eval_biology_num_tokens": 5980229.0,
+      "eval_biology_runtime": 49.4582,
+      "eval_biology_samples_per_second": 10.11,
+      "eval_biology_steps_per_second": 10.11,
+      "step": 900
+    },
+    {
+      "epoch": 0.400600901352028,
+      "eval_chemistry_entropy": 0.748387647986412,
+      "eval_chemistry_loss": 0.7568163275718689,
+      "eval_chemistry_mean_token_accuracy": 0.7869879400134087,
+      "eval_chemistry_num_tokens": 5980229.0,
+      "eval_chemistry_runtime": 57.0636,
+      "eval_chemistry_samples_per_second": 8.762,
+      "eval_chemistry_steps_per_second": 8.762,
+      "step": 900
+    },
+    {
+      "epoch": 0.400600901352028,
+      "eval_math_entropy": 0.6898269492387772,
+      "eval_math_loss": 0.866584062576294,
+      "eval_math_mean_token_accuracy": 0.7752813557386399,
+      "eval_math_num_tokens": 5980229.0,
+      "eval_math_runtime": 59.1998,
+      "eval_math_samples_per_second": 8.446,
+      "eval_math_steps_per_second": 8.446,
+      "step": 900
+    },
+    {
+      "epoch": 0.400600901352028,
+      "eval_physics_entropy": 0.7360889357328415,
+      "eval_physics_loss": 0.7502807378768921,
+      "eval_physics_mean_token_accuracy": 0.7913857514262199,
+      "eval_physics_num_tokens": 5980229.0,
+      "eval_physics_runtime": 66.2752,
+      "eval_physics_samples_per_second": 7.544,
+      "eval_physics_steps_per_second": 7.544,
+      "step": 900
+    },
+    {
+      "entropy": 0.6976769424974918,
+      "epoch": 0.4050520224781617,
+      "grad_norm": 57.75,
+      "learning_rate": 1.8180000000000002e-05,
+      "loss": 11.373,
+      "mean_token_accuracy": 0.7986949652433395,
+      "num_tokens": 6048508.0,
+      "step": 910
+    },
+    {
+      "entropy": 0.7293819307349623,
+      "epoch": 0.40950314360429535,
+      "grad_norm": 59.75,
+      "learning_rate": 1.8380000000000004e-05,
+      "loss": 11.7158,
+      "mean_token_accuracy": 0.7940746705979109,
+      "num_tokens": 6114855.0,
+      "step": 920
+    },
+    {
+      "entropy": 0.7529061837121844,
+      "epoch": 0.41395426473042896,
+      "grad_norm": 58.5,
+      "learning_rate": 1.858e-05,
+      "loss": 12.1233,
+      "mean_token_accuracy": 0.7879564348608256,
+      "num_tokens": 6179050.0,
+      "step": 930
+    },
+    {
+      "entropy": 0.7420403499156236,
+      "epoch": 0.4184053858565626,
+      "grad_norm": 60.25,
+      "learning_rate": 1.878e-05,
+      "loss": 12.11,
+      "mean_token_accuracy": 0.7902424175292253,
+      "num_tokens": 6244987.0,
+      "step": 940
+    },
+    {
+      "entropy": 0.7153761054389178,
+      "epoch": 0.4228565069826963,
+      "grad_norm": 65.0,
+      "learning_rate": 1.898e-05,
+      "loss": 11.5745,
+      "mean_token_accuracy": 0.7952297408133745,
+      "num_tokens": 6310799.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.7087640034034848,
+      "epoch": 0.4273076281088299,
+      "grad_norm": 62.5,
+      "learning_rate": 1.918e-05,
+      "loss": 11.5358,
+      "mean_token_accuracy": 0.7965508222579956,
+      "num_tokens": 6378152.0,
+      "step": 960
+    },
+    {
+      "entropy": 0.7380615974776447,
+      "epoch": 0.43175874923496355,
+      "grad_norm": 52.25,
+      "learning_rate": 1.938e-05,
+      "loss": 11.7537,
+      "mean_token_accuracy": 0.7940621051937342,
+      "num_tokens": 6442491.0,
+      "step": 970
+    },
+    {
+      "entropy": 0.7700119714252651,
+      "epoch": 0.4362098703610972,
+      "grad_norm": 55.75,
+      "learning_rate": 1.9580000000000002e-05,
+      "loss": 12.5586,
+      "mean_token_accuracy": 0.7843215424567461,
+      "num_tokens": 6510231.0,
+      "step": 980
+    },
+    {
+      "entropy": 0.7417776238173246,
+      "epoch": 0.44066099148723087,
+      "grad_norm": 57.25,
+      "learning_rate": 1.978e-05,
+      "loss": 12.0337,
+      "mean_token_accuracy": 0.7908143695443869,
+      "num_tokens": 6572858.0,
+      "step": 990
+    },
+    {
+      "entropy": 0.7601414297707378,
+      "epoch": 0.4451121126133645,
+      "grad_norm": 49.75,
+      "learning_rate": 1.9980000000000002e-05,
+      "loss": 12.3686,
+      "mean_token_accuracy": 0.7845167253166437,
+      "num_tokens": 6637273.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4451121126133645,
+      "eval_biology_entropy": 1.2479438411593438,
+      "eval_biology_loss": 1.3289152383804321,
+      "eval_biology_mean_token_accuracy": 0.67418065315485,
+      "eval_biology_num_tokens": 6637273.0,
+      "eval_biology_runtime": 49.5469,
+      "eval_biology_samples_per_second": 10.091,
+      "eval_biology_steps_per_second": 10.091,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4451121126133645,
+      "eval_chemistry_entropy": 0.7556402538418769,
+      "eval_chemistry_loss": 0.7549638748168945,
+      "eval_chemistry_mean_token_accuracy": 0.7876601304411888,
+      "eval_chemistry_num_tokens": 6637273.0,
+      "eval_chemistry_runtime": 57.6175,
+      "eval_chemistry_samples_per_second": 8.678,
+      "eval_chemistry_steps_per_second": 8.678,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4451121126133645,
+      "eval_math_entropy": 0.7018868820667267,
+      "eval_math_loss": 0.8697984218597412,
+      "eval_math_mean_token_accuracy": 0.7734663159847259,
+      "eval_math_num_tokens": 6637273.0,
+      "eval_math_runtime": 59.4977,
+      "eval_math_samples_per_second": 8.404,
+      "eval_math_steps_per_second": 8.404,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4451121126133645,
+      "eval_physics_entropy": 0.7586731649637223,
+      "eval_physics_loss": 0.7596898078918457,
+      "eval_physics_mean_token_accuracy": 0.7887503932714462,
+      "eval_physics_num_tokens": 6637273.0,
+      "eval_physics_runtime": 66.4638,
+      "eval_physics_samples_per_second": 7.523,
+      "eval_physics_steps_per_second": 7.523,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 10000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.450581354264347e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b80264a99be4159d429211f23358658801bacbe62b4f9fe61436651407442503
+size 6481