Instructions to use roonbug/xghe5pi0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use roonbug/xghe5pi0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="roonbug/xghe5pi0")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("roonbug/xghe5pi0")
model = AutoModelForImageTextToText.from_pretrained("roonbug/xghe5pi0")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use roonbug/xghe5pi0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "roonbug/xghe5pi0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/xghe5pi0",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/roonbug/xghe5pi0

SGLang

How to use roonbug/xghe5pi0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "roonbug/xghe5pi0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/xghe5pi0",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "roonbug/xghe5pi0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/xghe5pi0",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use roonbug/xghe5pi0 with Docker Model Runner:
```
docker model run hf.co/roonbug/xghe5pi0
```

roonbug commited on May 3

Commit

dcaad04

verified ·

1 Parent(s): 9863129

Upload folder using huggingface_hub

Browse files

Files changed (20) hide show

.gitattributes +1 -0
checkpoint-1000/added_tokens.json +3 -0
checkpoint-1000/chat_template.jinja +47 -0
checkpoint-1000/config.json +110 -0
checkpoint-1000/generation_config.json +13 -0
checkpoint-1000/model-00001-of-00005.safetensors +3 -0
checkpoint-1000/model-00002-of-00005.safetensors +3 -0
checkpoint-1000/model-00003-of-00005.safetensors +3 -0
checkpoint-1000/model-00004-of-00005.safetensors +3 -0
checkpoint-1000/model-00005-of-00005.safetensors +3 -0
checkpoint-1000/model.safetensors.index.json +0 -0
checkpoint-1000/optimizer.pt +3 -0
checkpoint-1000/rng_state.pth +3 -0
checkpoint-1000/scheduler.pt +3 -0
checkpoint-1000/special_tokens_map.json +33 -0
checkpoint-1000/tokenizer.json +3 -0
checkpoint-1000/tokenizer.model +3 -0
checkpoint-1000/tokenizer_config.json +0 -0
checkpoint-1000/trainer_state.json +1254 -0
checkpoint-1000/training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-1000/tokenizer.json filter=lfs diff=lfs merge=lfs -text

checkpoint-1000/added_tokens.json ADDED Viewed

	@@ -0,0 +1,3 @@

+{
+  "<image_soft_token>": 262144
+}

checkpoint-1000/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,47 @@

+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-1000/config.json ADDED Viewed

	@@ -0,0 +1,110 @@

+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eoi_token_index": 256000,
+  "eos_token_id": 1,
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "pad_token_id": 0,
+  "text_config": {
+    "_sliding_window_pattern": 6,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "dtype": "bfloat16",
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3840,
+    "initializer_range": 0.02,
+    "intermediate_size": 15360,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_local_base_freq": 10000.0,
+    "rope_scaling": {
+      "factor": 8.0,
+      "rope_type": "linear"
+    },
+    "rope_theta": 1000000.0,
+    "sliding_window": 1024,
+    "use_cache": true,
+    "vocab_size": 262208
+  },
+  "transformers_version": "4.56.1",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "vision_use_head": false
+  }
+}

checkpoint-1000/generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.56.1"
+}

checkpoint-1000/model-00001-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4847447e92599833e8dbaa3067cd201c3bb5c052efa91f11ba891e43234f7832
+size 4979902192

checkpoint-1000/model-00002-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:891bd54eed03cba9ee1e705533a02a8217fcc29f356e4a1f53e5fd0d178883ad
+size 4931296592

checkpoint-1000/model-00003-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:78972a746c62d88055a7aaee472b082b407338db6f9ab7bcf2d3eed4dc82a296
+size 4931296656

checkpoint-1000/model-00004-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d20884cf2adc6d062f77c5dc22d171746b0e785c8f2885c1a9361199f7e4728a
+size 4931296656

checkpoint-1000/model-00005-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4bae7817e46100d353b97234224fe3f076403b1981bbd4fe223bbfdd7cc23417
+size 4601000928

checkpoint-1000/model.safetensors.index.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-1000/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:64c0dca26ea69580df919707a3091c16d22776341b114351b7f124e24479df8f
+size 14573116445

checkpoint-1000/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:acd74ad34e3c5060506b53a8a3fafbfb35d44065ee496b2abf3a0b9a3b93ba09
+size 14645

checkpoint-1000/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6078e52773a379f026b8fbbbc5546ab7ed9418d05b713f8aff2a4f3c7e12f108
+size 1465

checkpoint-1000/special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,33 @@

+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1000/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568

checkpoint-1000/tokenizer.model ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074

checkpoint-1000/tokenizer_config.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-1000/trainer_state.json ADDED Viewed

	@@ -0,0 +1,1254 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.4,
+  "eval_steps": 100,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.40615240833722055,
+      "epoch": 0.004,
+      "grad_norm": 255.0,
+      "learning_rate": 1.8e-07,
+      "loss": 17.6434,
+      "mean_token_accuracy": 0.7695769021287561,
+      "num_tokens": 72542.0,
+      "step": 10
+    },
+    {
+      "entropy": 0.3965892889536917,
+      "epoch": 0.008,
+      "grad_norm": 334.0,
+      "learning_rate": 3.8e-07,
+      "loss": 17.7492,
+      "mean_token_accuracy": 0.7730019390583038,
+      "num_tokens": 138146.0,
+      "step": 20
+    },
+    {
+      "entropy": 0.3902019609231502,
+      "epoch": 0.012,
+      "grad_norm": 276.0,
+      "learning_rate": 5.800000000000001e-07,
+      "loss": 16.933,
+      "mean_token_accuracy": 0.7769877038896084,
+      "num_tokens": 208409.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.40995794120244683,
+      "epoch": 0.016,
+      "grad_norm": 199.0,
+      "learning_rate": 7.8e-07,
+      "loss": 16.9985,
+      "mean_token_accuracy": 0.7748937096446753,
+      "num_tokens": 280941.0,
+      "step": 40
+    },
+    {
+      "entropy": 0.42301482539623975,
+      "epoch": 0.02,
+      "grad_norm": 340.0,
+      "learning_rate": 9.800000000000001e-07,
+      "loss": 17.2256,
+      "mean_token_accuracy": 0.7713327277451754,
+      "num_tokens": 349731.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.46000946732237935,
+      "epoch": 0.024,
+      "grad_norm": 280.0,
+      "learning_rate": 1.1800000000000001e-06,
+      "loss": 17.6526,
+      "mean_token_accuracy": 0.764429247379303,
+      "num_tokens": 419127.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.43395038205198944,
+      "epoch": 0.028,
+      "grad_norm": 238.0,
+      "learning_rate": 1.3800000000000001e-06,
+      "loss": 16.1299,
+      "mean_token_accuracy": 0.7780056711286306,
+      "num_tokens": 489016.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.4683163737878203,
+      "epoch": 0.032,
+      "grad_norm": 178.0,
+      "learning_rate": 1.5800000000000001e-06,
+      "loss": 15.4388,
+      "mean_token_accuracy": 0.7803558874875307,
+      "num_tokens": 558056.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.5069354984909296,
+      "epoch": 0.036,
+      "grad_norm": 248.0,
+      "learning_rate": 1.7800000000000001e-06,
+      "loss": 15.403,
+      "mean_token_accuracy": 0.7772221932187676,
+      "num_tokens": 627804.0,
+      "step": 90
+    },
+    {
+      "entropy": 0.54353212909773,
+      "epoch": 0.04,
+      "grad_norm": 168.0,
+      "learning_rate": 1.98e-06,
+      "loss": 14.7499,
+      "mean_token_accuracy": 0.7799132943153382,
+      "num_tokens": 697556.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.04,
+      "eval_chemistry_entropy": 0.40587379191815853,
+      "eval_chemistry_loss": 0.787173867225647,
+      "eval_chemistry_mean_token_accuracy": 0.8101002821922302,
+      "eval_chemistry_num_tokens": 697556.0,
+      "eval_chemistry_runtime": 54.0871,
+      "eval_chemistry_samples_per_second": 9.244,
+      "eval_chemistry_steps_per_second": 9.244,
+      "step": 100
+    },
+    {
+      "epoch": 0.04,
+      "eval_math_entropy": 0.5863081520199775,
+      "eval_math_loss": 0.9441913962364197,
+      "eval_math_mean_token_accuracy": 0.7719187951087951,
+      "eval_math_num_tokens": 697556.0,
+      "eval_math_runtime": 55.9061,
+      "eval_math_samples_per_second": 8.944,
+      "eval_math_steps_per_second": 8.944,
+      "step": 100
+    },
+    {
+      "entropy": 0.5997343350201845,
+      "epoch": 0.044,
+      "grad_norm": 169.0,
+      "learning_rate": 2.1800000000000003e-06,
+      "loss": 14.888,
+      "mean_token_accuracy": 0.7764014065265655,
+      "num_tokens": 765262.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.6096270757727325,
+      "epoch": 0.048,
+      "grad_norm": 116.0,
+      "learning_rate": 2.38e-06,
+      "loss": 13.5066,
+      "mean_token_accuracy": 0.7883317556232214,
+      "num_tokens": 836753.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.6704345874488353,
+      "epoch": 0.052,
+      "grad_norm": 137.0,
+      "learning_rate": 2.5800000000000003e-06,
+      "loss": 13.7751,
+      "mean_token_accuracy": 0.7803409658372402,
+      "num_tokens": 909947.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.7165370973758399,
+      "epoch": 0.056,
+      "grad_norm": 100.5,
+      "learning_rate": 2.7800000000000005e-06,
+      "loss": 13.8237,
+      "mean_token_accuracy": 0.7797280913218856,
+      "num_tokens": 981959.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.7298246519640088,
+      "epoch": 0.06,
+      "grad_norm": 103.0,
+      "learning_rate": 2.9800000000000003e-06,
+      "loss": 13.2219,
+      "mean_token_accuracy": 0.7851537428796291,
+      "num_tokens": 1053624.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.7729588491842151,
+      "epoch": 0.064,
+      "grad_norm": 88.0,
+      "learning_rate": 3.1800000000000005e-06,
+      "loss": 13.4792,
+      "mean_token_accuracy": 0.7815716501325369,
+      "num_tokens": 1126446.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.7438578702509403,
+      "epoch": 0.068,
+      "grad_norm": 448.0,
+      "learning_rate": 3.3800000000000007e-06,
+      "loss": 12.8028,
+      "mean_token_accuracy": 0.7896847818046808,
+      "num_tokens": 1196286.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.7233589397743344,
+      "epoch": 0.072,
+      "grad_norm": 69.5,
+      "learning_rate": 3.58e-06,
+      "loss": 12.5001,
+      "mean_token_accuracy": 0.7914219040423631,
+      "num_tokens": 1273905.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.745759145822376,
+      "epoch": 0.076,
+      "grad_norm": 64.5,
+      "learning_rate": 3.7800000000000002e-06,
+      "loss": 12.783,
+      "mean_token_accuracy": 0.7839695058763028,
+      "num_tokens": 1346454.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.7534141565673054,
+      "epoch": 0.08,
+      "grad_norm": 81.0,
+      "learning_rate": 3.980000000000001e-06,
+      "loss": 12.5955,
+      "mean_token_accuracy": 0.7919603064656258,
+      "num_tokens": 1413596.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.08,
+      "eval_chemistry_entropy": 0.5783105248510838,
+      "eval_chemistry_loss": 0.6815671324729919,
+      "eval_chemistry_mean_token_accuracy": 0.8137290213108063,
+      "eval_chemistry_num_tokens": 1413596.0,
+      "eval_chemistry_runtime": 54.2654,
+      "eval_chemistry_samples_per_second": 9.214,
+      "eval_chemistry_steps_per_second": 9.214,
+      "step": 200
+    },
+    {
+      "epoch": 0.08,
+      "eval_math_entropy": 0.7573327482938766,
+      "eval_math_loss": 0.7936341762542725,
+      "eval_math_mean_token_accuracy": 0.7876009732484818,
+      "eval_math_num_tokens": 1413596.0,
+      "eval_math_runtime": 55.9908,
+      "eval_math_samples_per_second": 8.93,
+      "eval_math_steps_per_second": 8.93,
+      "step": 200
+    },
+    {
+      "entropy": 0.6914544056169689,
+      "epoch": 0.084,
+      "grad_norm": 64.0,
+      "learning_rate": 4.18e-06,
+      "loss": 11.6084,
+      "mean_token_accuracy": 0.8023380789905786,
+      "num_tokens": 1485147.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.7164348528720439,
+      "epoch": 0.088,
+      "grad_norm": 59.5,
+      "learning_rate": 4.38e-06,
+      "loss": 11.9002,
+      "mean_token_accuracy": 0.7977396491914988,
+      "num_tokens": 1559450.0,
+      "step": 220
+    },
+    {
+      "entropy": 0.7300810461863876,
+      "epoch": 0.092,
+      "grad_norm": 60.5,
+      "learning_rate": 4.58e-06,
+      "loss": 11.9903,
+      "mean_token_accuracy": 0.796798275783658,
+      "num_tokens": 1631535.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.7203218000009656,
+      "epoch": 0.096,
+      "grad_norm": 61.5,
+      "learning_rate": 4.78e-06,
+      "loss": 11.71,
+      "mean_token_accuracy": 0.7997211128473282,
+      "num_tokens": 1701193.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.7253720635548234,
+      "epoch": 0.1,
+      "grad_norm": 55.75,
+      "learning_rate": 4.980000000000001e-06,
+      "loss": 11.7979,
+      "mean_token_accuracy": 0.7965766046196222,
+      "num_tokens": 1773596.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.7394886416383087,
+      "epoch": 0.104,
+      "grad_norm": 54.25,
+      "learning_rate": 5.18e-06,
+      "loss": 12.2261,
+      "mean_token_accuracy": 0.7902322862297296,
+      "num_tokens": 1840165.0,
+      "step": 260
+    },
+    {
+      "entropy": 0.7491293519735336,
+      "epoch": 0.108,
+      "grad_norm": 115.5,
+      "learning_rate": 5.380000000000001e-06,
+      "loss": 12.1736,
+      "mean_token_accuracy": 0.7910951185971499,
+      "num_tokens": 1908149.0,
+      "step": 270
+    },
+    {
+      "entropy": 0.7066912613809109,
+      "epoch": 0.112,
+      "grad_norm": 63.0,
+      "learning_rate": 5.580000000000001e-06,
+      "loss": 11.3088,
+      "mean_token_accuracy": 0.8060384418815374,
+      "num_tokens": 1979232.0,
+      "step": 280
+    },
+    {
+      "entropy": 0.7260928615927696,
+      "epoch": 0.116,
+      "grad_norm": 45.0,
+      "learning_rate": 5.78e-06,
+      "loss": 11.7944,
+      "mean_token_accuracy": 0.7980801545083522,
+      "num_tokens": 2052039.0,
+      "step": 290
+    },
+    {
+      "entropy": 0.7076106617227197,
+      "epoch": 0.12,
+      "grad_norm": 55.0,
+      "learning_rate": 5.98e-06,
+      "loss": 11.6468,
+      "mean_token_accuracy": 0.8010890077799558,
+      "num_tokens": 2125815.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.12,
+      "eval_chemistry_entropy": 0.6047877628803253,
+      "eval_chemistry_loss": 0.6664053797721863,
+      "eval_chemistry_mean_token_accuracy": 0.8136081092357635,
+      "eval_chemistry_num_tokens": 2125815.0,
+      "eval_chemistry_runtime": 54.0763,
+      "eval_chemistry_samples_per_second": 9.246,
+      "eval_chemistry_steps_per_second": 9.246,
+      "step": 300
+    },
+    {
+      "epoch": 0.12,
+      "eval_math_entropy": 0.7215209539234638,
+      "eval_math_loss": 0.7359998822212219,
+      "eval_math_mean_token_accuracy": 0.7973045570850372,
+      "eval_math_num_tokens": 2125815.0,
+      "eval_math_runtime": 55.9216,
+      "eval_math_samples_per_second": 8.941,
+      "eval_math_steps_per_second": 8.941,
+      "step": 300
+    },
+    {
+      "entropy": 0.716811947338283,
+      "epoch": 0.124,
+      "grad_norm": 58.75,
+      "learning_rate": 6.18e-06,
+      "loss": 11.674,
+      "mean_token_accuracy": 0.7994389187544584,
+      "num_tokens": 2201991.0,
+      "step": 310
+    },
+    {
+      "entropy": 0.702051258739084,
+      "epoch": 0.128,
+      "grad_norm": 50.25,
+      "learning_rate": 6.380000000000001e-06,
+      "loss": 11.26,
+      "mean_token_accuracy": 0.8070401795208454,
+      "num_tokens": 2274177.0,
+      "step": 320
+    },
+    {
+      "entropy": 0.7048663217574358,
+      "epoch": 0.132,
+      "grad_norm": 48.0,
+      "learning_rate": 6.5800000000000005e-06,
+      "loss": 11.3615,
+      "mean_token_accuracy": 0.8019439272582531,
+      "num_tokens": 2342173.0,
+      "step": 330
+    },
+    {
+      "entropy": 0.6837939724326134,
+      "epoch": 0.136,
+      "grad_norm": 55.5,
+      "learning_rate": 6.780000000000001e-06,
+      "loss": 11.0291,
+      "mean_token_accuracy": 0.8044715665280819,
+      "num_tokens": 2412435.0,
+      "step": 340
+    },
+    {
+      "entropy": 0.7035049640573561,
+      "epoch": 0.14,
+      "grad_norm": 52.25,
+      "learning_rate": 6.98e-06,
+      "loss": 11.5752,
+      "mean_token_accuracy": 0.7999796941876411,
+      "num_tokens": 2481105.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.6804045747034252,
+      "epoch": 0.144,
+      "grad_norm": 47.75,
+      "learning_rate": 7.180000000000001e-06,
+      "loss": 10.8378,
+      "mean_token_accuracy": 0.8108949448913336,
+      "num_tokens": 2548445.0,
+      "step": 360
+    },
+    {
+      "entropy": 0.6590544011443853,
+      "epoch": 0.148,
+      "grad_norm": 43.0,
+      "learning_rate": 7.3800000000000005e-06,
+      "loss": 10.5149,
+      "mean_token_accuracy": 0.8155259877443314,
+      "num_tokens": 2621364.0,
+      "step": 370
+    },
+    {
+      "entropy": 0.6717178193852306,
+      "epoch": 0.152,
+      "grad_norm": 60.5,
+      "learning_rate": 7.58e-06,
+      "loss": 10.8081,
+      "mean_token_accuracy": 0.8107277728617192,
+      "num_tokens": 2686462.0,
+      "step": 380
+    },
+    {
+      "entropy": 0.6481600757688284,
+      "epoch": 0.156,
+      "grad_norm": 48.0,
+      "learning_rate": 7.78e-06,
+      "loss": 10.6698,
+      "mean_token_accuracy": 0.8106721647083759,
+      "num_tokens": 2754241.0,
+      "step": 390
+    },
+    {
+      "entropy": 0.7008802507072687,
+      "epoch": 0.16,
+      "grad_norm": 62.0,
+      "learning_rate": 7.980000000000002e-06,
+      "loss": 11.3348,
+      "mean_token_accuracy": 0.802883006259799,
+      "num_tokens": 2824418.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.16,
+      "eval_chemistry_entropy": 0.6197386468052865,
+      "eval_chemistry_loss": 0.6651880145072937,
+      "eval_chemistry_mean_token_accuracy": 0.8126706712245941,
+      "eval_chemistry_num_tokens": 2824418.0,
+      "eval_chemistry_runtime": 54.0137,
+      "eval_chemistry_samples_per_second": 9.257,
+      "eval_chemistry_steps_per_second": 9.257,
+      "step": 400
+    },
+    {
+      "epoch": 0.16,
+      "eval_math_entropy": 0.7064790060222149,
+      "eval_math_loss": 0.702800989151001,
+      "eval_math_mean_token_accuracy": 0.8039963529109955,
+      "eval_math_num_tokens": 2824418.0,
+      "eval_math_runtime": 55.8372,
+      "eval_math_samples_per_second": 8.955,
+      "eval_math_steps_per_second": 8.955,
+      "step": 400
+    },
+    {
+      "entropy": 0.7051302054896951,
+      "epoch": 0.164,
+      "grad_norm": 53.25,
+      "learning_rate": 8.18e-06,
+      "loss": 11.361,
+      "mean_token_accuracy": 0.8027527991682291,
+      "num_tokens": 2895667.0,
+      "step": 410
+    },
+    {
+      "entropy": 0.6781568734906613,
+      "epoch": 0.168,
+      "grad_norm": 41.0,
+      "learning_rate": 8.380000000000001e-06,
+      "loss": 11.1069,
+      "mean_token_accuracy": 0.8051798522472382,
+      "num_tokens": 2966031.0,
+      "step": 420
+    },
+    {
+      "entropy": 0.669656771980226,
+      "epoch": 0.172,
+      "grad_norm": 46.75,
+      "learning_rate": 8.580000000000001e-06,
+      "loss": 10.669,
+      "mean_token_accuracy": 0.8124278139322996,
+      "num_tokens": 3039780.0,
+      "step": 430
+    },
+    {
+      "entropy": 0.6730968474410475,
+      "epoch": 0.176,
+      "grad_norm": 48.0,
+      "learning_rate": 8.78e-06,
+      "loss": 10.9171,
+      "mean_token_accuracy": 0.8071481816470623,
+      "num_tokens": 3110313.0,
+      "step": 440
+    },
+    {
+      "entropy": 0.6608755425550044,
+      "epoch": 0.18,
+      "grad_norm": 47.5,
+      "learning_rate": 8.98e-06,
+      "loss": 10.8028,
+      "mean_token_accuracy": 0.8110857963562011,
+      "num_tokens": 3179963.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.6951628234237432,
+      "epoch": 0.184,
+      "grad_norm": 43.5,
+      "learning_rate": 9.180000000000002e-06,
+      "loss": 11.097,
+      "mean_token_accuracy": 0.8053878784179688,
+      "num_tokens": 3251139.0,
+      "step": 460
+    },
+    {
+      "entropy": 0.6642801096662879,
+      "epoch": 0.188,
+      "grad_norm": 49.25,
+      "learning_rate": 9.38e-06,
+      "loss": 10.6878,
+      "mean_token_accuracy": 0.8112512346357107,
+      "num_tokens": 3323519.0,
+      "step": 470
+    },
+    {
+      "entropy": 0.658032058365643,
+      "epoch": 0.192,
+      "grad_norm": 49.5,
+      "learning_rate": 9.58e-06,
+      "loss": 10.6202,
+      "mean_token_accuracy": 0.8127347908914089,
+      "num_tokens": 3394170.0,
+      "step": 480
+    },
+    {
+      "entropy": 0.6513477127999068,
+      "epoch": 0.196,
+      "grad_norm": 51.5,
+      "learning_rate": 9.780000000000001e-06,
+      "loss": 10.5983,
+      "mean_token_accuracy": 0.8158927127718926,
+      "num_tokens": 3465341.0,
+      "step": 490
+    },
+    {
+      "entropy": 0.7007277802564204,
+      "epoch": 0.2,
+      "grad_norm": 45.0,
+      "learning_rate": 9.980000000000001e-06,
+      "loss": 11.1959,
+      "mean_token_accuracy": 0.8035019136965275,
+      "num_tokens": 3533571.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.2,
+      "eval_chemistry_entropy": 0.6269507716894149,
+      "eval_chemistry_loss": 0.6651133298873901,
+      "eval_chemistry_mean_token_accuracy": 0.8129660848379135,
+      "eval_chemistry_num_tokens": 3533571.0,
+      "eval_chemistry_runtime": 53.9735,
+      "eval_chemistry_samples_per_second": 9.264,
+      "eval_chemistry_steps_per_second": 9.264,
+      "step": 500
+    },
+    {
+      "epoch": 0.2,
+      "eval_math_entropy": 0.6837767160832882,
+      "eval_math_loss": 0.682838499546051,
+      "eval_math_mean_token_accuracy": 0.8080879813432693,
+      "eval_math_num_tokens": 3533571.0,
+      "eval_math_runtime": 55.9164,
+      "eval_math_samples_per_second": 8.942,
+      "eval_math_steps_per_second": 8.942,
+      "step": 500
+    },
+    {
+      "entropy": 0.6865575925447047,
+      "epoch": 0.204,
+      "grad_norm": 48.75,
+      "learning_rate": 1.018e-05,
+      "loss": 11.0861,
+      "mean_token_accuracy": 0.8060325808823109,
+      "num_tokens": 3602983.0,
+      "step": 510
+    },
+    {
+      "entropy": 0.658450176846236,
+      "epoch": 0.208,
+      "grad_norm": 49.0,
+      "learning_rate": 1.038e-05,
+      "loss": 10.6366,
+      "mean_token_accuracy": 0.8118435345590115,
+      "num_tokens": 3673600.0,
+      "step": 520
+    },
+    {
+      "entropy": 0.6979701479896903,
+      "epoch": 0.212,
+      "grad_norm": 54.5,
+      "learning_rate": 1.0580000000000002e-05,
+      "loss": 11.2165,
+      "mean_token_accuracy": 0.8044058471918106,
+      "num_tokens": 3743641.0,
+      "step": 530
+    },
+    {
+      "entropy": 0.6361962419003249,
+      "epoch": 0.216,
+      "grad_norm": 50.25,
+      "learning_rate": 1.0780000000000002e-05,
+      "loss": 10.3346,
+      "mean_token_accuracy": 0.817437007650733,
+      "num_tokens": 3817397.0,
+      "step": 540
+    },
+    {
+      "entropy": 0.6625658424571157,
+      "epoch": 0.22,
+      "grad_norm": 56.75,
+      "learning_rate": 1.0980000000000002e-05,
+      "loss": 10.6586,
+      "mean_token_accuracy": 0.8117894344031811,
+      "num_tokens": 3886866.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.6935065235942602,
+      "epoch": 0.224,
+      "grad_norm": 49.25,
+      "learning_rate": 1.1180000000000001e-05,
+      "loss": 11.1172,
+      "mean_token_accuracy": 0.8039925239980221,
+      "num_tokens": 3953732.0,
+      "step": 560
+    },
+    {
+      "entropy": 0.6686032420024276,
+      "epoch": 0.228,
+      "grad_norm": 49.0,
+      "learning_rate": 1.138e-05,
+      "loss": 10.8693,
+      "mean_token_accuracy": 0.8076109483838081,
+      "num_tokens": 4025795.0,
+      "step": 570
+    },
+    {
+      "entropy": 0.6402695474214852,
+      "epoch": 0.232,
+      "grad_norm": 48.0,
+      "learning_rate": 1.1580000000000001e-05,
+      "loss": 10.3597,
+      "mean_token_accuracy": 0.8167435925453901,
+      "num_tokens": 4098989.0,
+      "step": 580
+    },
+    {
+      "entropy": 0.658338391315192,
+      "epoch": 0.236,
+      "grad_norm": 44.75,
+      "learning_rate": 1.178e-05,
+      "loss": 10.5357,
+      "mean_token_accuracy": 0.8130462419241666,
+      "num_tokens": 4170878.0,
+      "step": 590
+    },
+    {
+      "entropy": 0.6290927086956799,
+      "epoch": 0.24,
+      "grad_norm": 47.0,
+      "learning_rate": 1.198e-05,
+      "loss": 10.0972,
+      "mean_token_accuracy": 0.8207072768360376,
+      "num_tokens": 4243655.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.24,
+      "eval_chemistry_entropy": 0.6155065232515335,
+      "eval_chemistry_loss": 0.6671720147132874,
+      "eval_chemistry_mean_token_accuracy": 0.8120917826890945,
+      "eval_chemistry_num_tokens": 4243655.0,
+      "eval_chemistry_runtime": 55.2706,
+      "eval_chemistry_samples_per_second": 9.046,
+      "eval_chemistry_steps_per_second": 9.046,
+      "step": 600
+    },
+    {
+      "epoch": 0.24,
+      "eval_math_entropy": 0.6502774241566658,
+      "eval_math_loss": 0.6693575382232666,
+      "eval_math_mean_token_accuracy": 0.811131351351738,
+      "eval_math_num_tokens": 4243655.0,
+      "eval_math_runtime": 56.5379,
+      "eval_math_samples_per_second": 8.844,
+      "eval_math_steps_per_second": 8.844,
+      "step": 600
+    },
+    {
+      "entropy": 0.6233567481860518,
+      "epoch": 0.244,
+      "grad_norm": 45.0,
+      "learning_rate": 1.218e-05,
+      "loss": 10.0801,
+      "mean_token_accuracy": 0.8203211557120085,
+      "num_tokens": 4317321.0,
+      "step": 610
+    },
+    {
+      "entropy": 0.6291251040063799,
+      "epoch": 0.248,
+      "grad_norm": 46.0,
+      "learning_rate": 1.2380000000000002e-05,
+      "loss": 10.2183,
+      "mean_token_accuracy": 0.8186882007867098,
+      "num_tokens": 4387651.0,
+      "step": 620
+    },
+    {
+      "entropy": 0.6594036752358079,
+      "epoch": 0.252,
+      "grad_norm": 49.25,
+      "learning_rate": 1.2580000000000002e-05,
+      "loss": 10.4409,
+      "mean_token_accuracy": 0.8153050262480974,
+      "num_tokens": 4461909.0,
+      "step": 630
+    },
+    {
+      "entropy": 0.6203309029340744,
+      "epoch": 0.256,
+      "grad_norm": 53.5,
+      "learning_rate": 1.2780000000000001e-05,
+      "loss": 10.0772,
+      "mean_token_accuracy": 0.8196093909442425,
+      "num_tokens": 4531471.0,
+      "step": 640
+    },
+    {
+      "entropy": 0.6356428602710367,
+      "epoch": 0.26,
+      "grad_norm": 47.25,
+      "learning_rate": 1.2980000000000001e-05,
+      "loss": 10.3692,
+      "mean_token_accuracy": 0.8188248563557863,
+      "num_tokens": 4600757.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.6554200569167733,
+      "epoch": 0.264,
+      "grad_norm": 51.25,
+      "learning_rate": 1.3180000000000001e-05,
+      "loss": 10.5725,
+      "mean_token_accuracy": 0.8128338485956192,
+      "num_tokens": 4669389.0,
+      "step": 660
+    },
+    {
+      "entropy": 0.6745208881795406,
+      "epoch": 0.268,
+      "grad_norm": 50.0,
+      "learning_rate": 1.3380000000000002e-05,
+      "loss": 10.8355,
+      "mean_token_accuracy": 0.810622700676322,
+      "num_tokens": 4740492.0,
+      "step": 670
+    },
+    {
+      "entropy": 0.6350632888264954,
+      "epoch": 0.272,
+      "grad_norm": 47.0,
+      "learning_rate": 1.3580000000000002e-05,
+      "loss": 10.2504,
+      "mean_token_accuracy": 0.8196976982057095,
+      "num_tokens": 4810284.0,
+      "step": 680
+    },
+    {
+      "entropy": 0.6301803855225444,
+      "epoch": 0.276,
+      "grad_norm": 39.25,
+      "learning_rate": 1.378e-05,
+      "loss": 10.137,
+      "mean_token_accuracy": 0.818551366403699,
+      "num_tokens": 4885701.0,
+      "step": 690
+    },
+    {
+      "entropy": 0.6497578569687903,
+      "epoch": 0.28,
+      "grad_norm": 53.75,
+      "learning_rate": 1.398e-05,
+      "loss": 10.4748,
+      "mean_token_accuracy": 0.8124409653246403,
+      "num_tokens": 4955375.0,
+      "step": 700
+    },
+    {
+      "epoch": 0.28,
+      "eval_chemistry_entropy": 0.6421024734973908,
+      "eval_chemistry_loss": 0.667654275894165,
+      "eval_chemistry_mean_token_accuracy": 0.8122305541038514,
+      "eval_chemistry_num_tokens": 4955375.0,
+      "eval_chemistry_runtime": 53.9764,
+      "eval_chemistry_samples_per_second": 9.263,
+      "eval_chemistry_steps_per_second": 9.263,
+      "step": 700
+    },
+    {
+      "epoch": 0.28,
+      "eval_math_entropy": 0.6617435732483864,
+      "eval_math_loss": 0.6636469960212708,
+      "eval_math_mean_token_accuracy": 0.8122635103464126,
+      "eval_math_num_tokens": 4955375.0,
+      "eval_math_runtime": 55.7584,
+      "eval_math_samples_per_second": 8.967,
+      "eval_math_steps_per_second": 8.967,
+      "step": 700
+    },
+    {
+      "entropy": 0.6361791557632387,
+      "epoch": 0.284,
+      "grad_norm": 43.5,
+      "learning_rate": 1.418e-05,
+      "loss": 10.2547,
+      "mean_token_accuracy": 0.8184681631624698,
+      "num_tokens": 5025638.0,
+      "step": 710
+    },
+    {
+      "entropy": 0.6017223164439202,
+      "epoch": 0.288,
+      "grad_norm": 57.75,
+      "learning_rate": 1.4380000000000001e-05,
+      "loss": 9.6742,
+      "mean_token_accuracy": 0.8257420264184475,
+      "num_tokens": 5095104.0,
+      "step": 720
+    },
+    {
+      "entropy": 0.6285218503326178,
+      "epoch": 0.292,
+      "grad_norm": 45.75,
+      "learning_rate": 1.4580000000000001e-05,
+      "loss": 10.1673,
+      "mean_token_accuracy": 0.8185360498726368,
+      "num_tokens": 5167049.0,
+      "step": 730
+    },
+    {
+      "entropy": 0.6593165175989271,
+      "epoch": 0.296,
+      "grad_norm": 47.0,
+      "learning_rate": 1.478e-05,
+      "loss": 10.6594,
+      "mean_token_accuracy": 0.8107825342565775,
+      "num_tokens": 5240988.0,
+      "step": 740
+    },
+    {
+      "entropy": 0.6415156803093851,
+      "epoch": 0.3,
+      "grad_norm": 47.0,
+      "learning_rate": 1.498e-05,
+      "loss": 10.2668,
+      "mean_token_accuracy": 0.8173366483300925,
+      "num_tokens": 5312258.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.6658093611709773,
+      "epoch": 0.304,
+      "grad_norm": 47.5,
+      "learning_rate": 1.5180000000000002e-05,
+      "loss": 10.7581,
+      "mean_token_accuracy": 0.8066589564085007,
+      "num_tokens": 5383732.0,
+      "step": 760
+    },
+    {
+      "entropy": 0.6711018362082541,
+      "epoch": 0.308,
+      "grad_norm": 48.75,
+      "learning_rate": 1.5380000000000002e-05,
+      "loss": 10.8042,
+      "mean_token_accuracy": 0.8097654834389687,
+      "num_tokens": 5456650.0,
+      "step": 770
+    },
+    {
+      "entropy": 0.6446251068264246,
+      "epoch": 0.312,
+      "grad_norm": 46.25,
+      "learning_rate": 1.5580000000000003e-05,
+      "loss": 10.3229,
+      "mean_token_accuracy": 0.8142865747213364,
+      "num_tokens": 5530666.0,
+      "step": 780
+    },
+    {
+      "entropy": 0.6051024467684328,
+      "epoch": 0.316,
+      "grad_norm": 49.5,
+      "learning_rate": 1.578e-05,
+      "loss": 9.8268,
+      "mean_token_accuracy": 0.8232330553233623,
+      "num_tokens": 5605269.0,
+      "step": 790
+    },
+    {
+      "entropy": 0.5965368082746864,
+      "epoch": 0.32,
+      "grad_norm": 54.75,
+      "learning_rate": 1.5980000000000003e-05,
+      "loss": 9.6507,
+      "mean_token_accuracy": 0.8276361111551523,
+      "num_tokens": 5676334.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.32,
+      "eval_chemistry_entropy": 0.652762301415205,
+      "eval_chemistry_loss": 0.6736706495285034,
+      "eval_chemistry_mean_token_accuracy": 0.8105895760059356,
+      "eval_chemistry_num_tokens": 5676334.0,
+      "eval_chemistry_runtime": 54.1527,
+      "eval_chemistry_samples_per_second": 9.233,
+      "eval_chemistry_steps_per_second": 9.233,
+      "step": 800
+    },
+    {
+      "epoch": 0.32,
+      "eval_math_entropy": 0.6557648414075374,
+      "eval_math_loss": 0.6591339111328125,
+      "eval_math_mean_token_accuracy": 0.8122332338094711,
+      "eval_math_num_tokens": 5676334.0,
+      "eval_math_runtime": 56.6894,
+      "eval_math_samples_per_second": 8.82,
+      "eval_math_steps_per_second": 8.82,
+      "step": 800
+    },
+    {
+      "entropy": 0.6387809926643968,
+      "epoch": 0.324,
+      "grad_norm": 51.25,
+      "learning_rate": 1.618e-05,
+      "loss": 10.3903,
+      "mean_token_accuracy": 0.8165412645787,
+      "num_tokens": 5747779.0,
+      "step": 810
+    },
+    {
+      "entropy": 0.6585037951357663,
+      "epoch": 0.328,
+      "grad_norm": 48.25,
+      "learning_rate": 1.638e-05,
+      "loss": 10.4673,
+      "mean_token_accuracy": 0.8138647183775902,
+      "num_tokens": 5816488.0,
+      "step": 820
+    },
+    {
+      "entropy": 0.6571912650018931,
+      "epoch": 0.332,
+      "grad_norm": 49.75,
+      "learning_rate": 1.658e-05,
+      "loss": 10.5769,
+      "mean_token_accuracy": 0.812447015941143,
+      "num_tokens": 5887180.0,
+      "step": 830
+    },
+    {
+      "entropy": 0.6646488612517715,
+      "epoch": 0.336,
+      "grad_norm": 44.5,
+      "learning_rate": 1.6780000000000002e-05,
+      "loss": 10.8156,
+      "mean_token_accuracy": 0.8087732318788767,
+      "num_tokens": 5958480.0,
+      "step": 840
+    },
+    {
+      "entropy": 0.6073424321599304,
+      "epoch": 0.34,
+      "grad_norm": 47.5,
+      "learning_rate": 1.698e-05,
+      "loss": 9.7128,
+      "mean_token_accuracy": 0.8271788094192744,
+      "num_tokens": 6031020.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.6620601391419768,
+      "epoch": 0.344,
+      "grad_norm": 46.5,
+      "learning_rate": 1.718e-05,
+      "loss": 10.6639,
+      "mean_token_accuracy": 0.8097412213683128,
+      "num_tokens": 6102081.0,
+      "step": 860
+    },
+    {
+      "entropy": 0.656591746211052,
+      "epoch": 0.348,
+      "grad_norm": 41.75,
+      "learning_rate": 1.7380000000000003e-05,
+      "loss": 10.5612,
+      "mean_token_accuracy": 0.8128813318908215,
+      "num_tokens": 6172871.0,
+      "step": 870
+    },
+    {
+      "entropy": 0.6083563419058919,
+      "epoch": 0.352,
+      "grad_norm": 43.75,
+      "learning_rate": 1.758e-05,
+      "loss": 9.9206,
+      "mean_token_accuracy": 0.8219428963959217,
+      "num_tokens": 6242161.0,
+      "step": 880
+    },
+    {
+      "entropy": 0.6747210225090384,
+      "epoch": 0.356,
+      "grad_norm": 46.75,
+      "learning_rate": 1.7780000000000003e-05,
+      "loss": 10.6887,
+      "mean_token_accuracy": 0.8089296150952577,
+      "num_tokens": 6312187.0,
+      "step": 890
+    },
+    {
+      "entropy": 0.6297760805115103,
+      "epoch": 0.36,
+      "grad_norm": 46.5,
+      "learning_rate": 1.798e-05,
+      "loss": 10.2931,
+      "mean_token_accuracy": 0.8156627554446458,
+      "num_tokens": 6377327.0,
+      "step": 900
+    },
+    {
+      "epoch": 0.36,
+      "eval_chemistry_entropy": 0.6516558772921562,
+      "eval_chemistry_loss": 0.6783114075660706,
+      "eval_chemistry_mean_token_accuracy": 0.8099269119501114,
+      "eval_chemistry_num_tokens": 6377327.0,
+      "eval_chemistry_runtime": 54.0329,
+      "eval_chemistry_samples_per_second": 9.254,
+      "eval_chemistry_steps_per_second": 9.254,
+      "step": 900
+    },
+    {
+      "epoch": 0.36,
+      "eval_math_entropy": 0.6468739511370659,
+      "eval_math_loss": 0.6565048694610596,
+      "eval_math_mean_token_accuracy": 0.8127901997566224,
+      "eval_math_num_tokens": 6377327.0,
+      "eval_math_runtime": 55.8379,
+      "eval_math_samples_per_second": 8.954,
+      "eval_math_steps_per_second": 8.954,
+      "step": 900
+    },
+    {
+      "entropy": 0.6443535090424121,
+      "epoch": 0.364,
+      "grad_norm": 54.5,
+      "learning_rate": 1.8180000000000002e-05,
+      "loss": 10.2172,
+      "mean_token_accuracy": 0.8161850411444902,
+      "num_tokens": 6453254.0,
+      "step": 910
+    },
+    {
+      "entropy": 0.6192579029127956,
+      "epoch": 0.368,
+      "grad_norm": 47.75,
+      "learning_rate": 1.8380000000000004e-05,
+      "loss": 10.0628,
+      "mean_token_accuracy": 0.8202301610261202,
+      "num_tokens": 6523679.0,
+      "step": 920
+    },
+    {
+      "entropy": 0.6301083014346659,
+      "epoch": 0.372,
+      "grad_norm": 43.75,
+      "learning_rate": 1.858e-05,
+      "loss": 10.0879,
+      "mean_token_accuracy": 0.8176717050373554,
+      "num_tokens": 6594651.0,
+      "step": 930
+    },
+    {
+      "entropy": 0.6354097174480557,
+      "epoch": 0.376,
+      "grad_norm": 42.75,
+      "learning_rate": 1.878e-05,
+      "loss": 10.2508,
+      "mean_token_accuracy": 0.8171609565615654,
+      "num_tokens": 6669278.0,
+      "step": 940
+    },
+    {
+      "entropy": 0.617029273416847,
+      "epoch": 0.38,
+      "grad_norm": 46.5,
+      "learning_rate": 1.898e-05,
+      "loss": 9.9656,
+      "mean_token_accuracy": 0.8214268207550048,
+      "num_tokens": 6740491.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.5976504179649055,
+      "epoch": 0.384,
+      "grad_norm": 39.5,
+      "learning_rate": 1.918e-05,
+      "loss": 9.6342,
+      "mean_token_accuracy": 0.8249731820076704,
+      "num_tokens": 6810978.0,
+      "step": 960
+    },
+    {
+      "entropy": 0.6468582183122635,
+      "epoch": 0.388,
+      "grad_norm": 45.0,
+      "learning_rate": 1.938e-05,
+      "loss": 10.3407,
+      "mean_token_accuracy": 0.8165425896644593,
+      "num_tokens": 6882077.0,
+      "step": 970
+    },
+    {
+      "entropy": 0.6440666496753693,
+      "epoch": 0.392,
+      "grad_norm": 44.75,
+      "learning_rate": 1.9580000000000002e-05,
+      "loss": 10.4922,
+      "mean_token_accuracy": 0.8133824530988931,
+      "num_tokens": 6953929.0,
+      "step": 980
+    },
+    {
+      "entropy": 0.697864510025829,
+      "epoch": 0.396,
+      "grad_norm": 51.5,
+      "learning_rate": 1.978e-05,
+      "loss": 11.1089,
+      "mean_token_accuracy": 0.8045640021562577,
+      "num_tokens": 7023835.0,
+      "step": 990
+    },
+    {
+      "entropy": 0.6845004385337233,
+      "epoch": 0.4,
+      "grad_norm": 43.0,
+      "learning_rate": 1.9980000000000002e-05,
+      "loss": 11.0312,
+      "mean_token_accuracy": 0.8040390104055405,
+      "num_tokens": 7096903.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4,
+      "eval_chemistry_entropy": 0.6533585796952248,
+      "eval_chemistry_loss": 0.6839205026626587,
+      "eval_chemistry_mean_token_accuracy": 0.8083831743001938,
+      "eval_chemistry_num_tokens": 7096903.0,
+      "eval_chemistry_runtime": 54.0723,
+      "eval_chemistry_samples_per_second": 9.247,
+      "eval_chemistry_steps_per_second": 9.247,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4,
+      "eval_math_entropy": 0.6458776640295982,
+      "eval_math_loss": 0.654421865940094,
+      "eval_math_mean_token_accuracy": 0.8129620677232743,
+      "eval_math_num_tokens": 7096903.0,
+      "eval_math_runtime": 56.7066,
+      "eval_math_samples_per_second": 8.817,
+      "eval_math_steps_per_second": 8.817,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 10000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.7587833384015814e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ae0f3429d8696cb1878e817a96382f41cecdd27680b7b12605e2981c8f94e164
+size 6481