Instructions to use roonbug/zhm6d0yg with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use roonbug/zhm6d0yg with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="roonbug/zhm6d0yg")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("roonbug/zhm6d0yg")
model = AutoModelForImageTextToText.from_pretrained("roonbug/zhm6d0yg")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use roonbug/zhm6d0yg with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "roonbug/zhm6d0yg"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/zhm6d0yg",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/roonbug/zhm6d0yg

SGLang

How to use roonbug/zhm6d0yg with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "roonbug/zhm6d0yg" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/zhm6d0yg",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "roonbug/zhm6d0yg" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roonbug/zhm6d0yg",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use roonbug/zhm6d0yg with Docker Model Runner:
```
docker model run hf.co/roonbug/zhm6d0yg
```

roonbug commited on May 4

Commit

90da8a8

verified ·

1 Parent(s): 1bd3baf

Upload folder using huggingface_hub

Browse files

Files changed (20) hide show

.gitattributes +1 -0
checkpoint-1000/added_tokens.json +3 -0
checkpoint-1000/chat_template.jinja +47 -0
checkpoint-1000/config.json +110 -0
checkpoint-1000/generation_config.json +13 -0
checkpoint-1000/model-00001-of-00005.safetensors +3 -0
checkpoint-1000/model-00002-of-00005.safetensors +3 -0
checkpoint-1000/model-00003-of-00005.safetensors +3 -0
checkpoint-1000/model-00004-of-00005.safetensors +3 -0
checkpoint-1000/model-00005-of-00005.safetensors +3 -0
checkpoint-1000/model.safetensors.index.json +0 -0
checkpoint-1000/optimizer.pt +3 -0
checkpoint-1000/rng_state.pth +3 -0
checkpoint-1000/scheduler.pt +3 -0
checkpoint-1000/special_tokens_map.json +33 -0
checkpoint-1000/tokenizer.json +3 -0
checkpoint-1000/tokenizer.model +3 -0
checkpoint-1000/tokenizer_config.json +0 -0
checkpoint-1000/trainer_state.json +1254 -0
checkpoint-1000/training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoint-500/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-1000/tokenizer.json filter=lfs diff=lfs merge=lfs -text

checkpoint-1000/added_tokens.json ADDED Viewed

	@@ -0,0 +1,3 @@

+{
+  "<image_soft_token>": 262144
+}

checkpoint-1000/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,47 @@

+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

checkpoint-1000/config.json ADDED Viewed

	@@ -0,0 +1,110 @@

+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eoi_token_index": 256000,
+  "eos_token_id": 1,
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "pad_token_id": 0,
+  "text_config": {
+    "_sliding_window_pattern": 6,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "dtype": "bfloat16",
+    "final_logit_softcapping": null,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3840,
+    "initializer_range": 0.02,
+    "intermediate_size": 15360,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_local_base_freq": 10000.0,
+    "rope_scaling": {
+      "factor": 8.0,
+      "rope_type": "linear"
+    },
+    "rope_theta": 1000000.0,
+    "sliding_window": 1024,
+    "use_cache": true,
+    "vocab_size": 262208
+  },
+  "transformers_version": "4.56.1",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "vision_use_head": false
+  }
+}

checkpoint-1000/generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.56.1"
+}

checkpoint-1000/model-00001-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4847447e92599833e8dbaa3067cd201c3bb5c052efa91f11ba891e43234f7832
+size 4979902192

checkpoint-1000/model-00002-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:891bd54eed03cba9ee1e705533a02a8217fcc29f356e4a1f53e5fd0d178883ad
+size 4931296592

checkpoint-1000/model-00003-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:00464b6c68fb26fe89aff05a8f961f4345de4c101fd2f11c106c7d49cdf85525
+size 4931296656

checkpoint-1000/model-00004-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:39a0aac08b2660991b04780eeaed8bbd0635ddd2764dc1b425fa0800aa4a8294
+size 4931296656

checkpoint-1000/model-00005-of-00005.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0de15e59b46dd114a1bb0f50a244a821f72e65064314b95774a000c9c52f3cbf
+size 4601000928

checkpoint-1000/model.safetensors.index.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-1000/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:73eaf340b78a67506f34c1069b4303f0e50bb511987ed4e1763f90ca4906a586
+size 14573116445

checkpoint-1000/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:acd74ad34e3c5060506b53a8a3fafbfb35d44065ee496b2abf3a0b9a3b93ba09
+size 14645

checkpoint-1000/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fc8bb724852ff0b2fb075112a11c8d73c94bf8ed61aceed2ef52af24ef9a8c59
+size 1465

checkpoint-1000/special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,33 @@

+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1000/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568

checkpoint-1000/tokenizer.model ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074

checkpoint-1000/tokenizer_config.json ADDED Viewed

The diff for this file is too large to render. See raw diff

checkpoint-1000/trainer_state.json ADDED Viewed

	@@ -0,0 +1,1254 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.4,
+  "eval_steps": 100,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.23732986762188374,
+      "epoch": 0.004,
+      "grad_norm": 65.5,
+      "learning_rate": 1.8000000000000001e-06,
+      "loss": 3.5321,
+      "mean_token_accuracy": 0.9383623488247395,
+      "num_tokens": 51831.0,
+      "step": 10
+    },
+    {
+      "entropy": 0.24765114395413548,
+      "epoch": 0.008,
+      "grad_norm": 75.5,
+      "learning_rate": 3.8000000000000005e-06,
+      "loss": 3.7326,
+      "mean_token_accuracy": 0.9351197093725204,
+      "num_tokens": 103172.0,
+      "step": 20
+    },
+    {
+      "entropy": 0.22037298779468983,
+      "epoch": 0.012,
+      "grad_norm": 78.0,
+      "learning_rate": 5.8e-06,
+      "loss": 3.3321,
+      "mean_token_accuracy": 0.9426371198147535,
+      "num_tokens": 147915.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.21643712103832513,
+      "epoch": 0.016,
+      "grad_norm": 90.5,
+      "learning_rate": 7.800000000000002e-06,
+      "loss": 3.6314,
+      "mean_token_accuracy": 0.9359618049114943,
+      "num_tokens": 195524.0,
+      "step": 40
+    },
+    {
+      "entropy": 0.22069417587481438,
+      "epoch": 0.02,
+      "grad_norm": 89.0,
+      "learning_rate": 9.800000000000001e-06,
+      "loss": 3.5165,
+      "mean_token_accuracy": 0.939437722414732,
+      "num_tokens": 243747.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.2474461881676689,
+      "epoch": 0.024,
+      "grad_norm": 84.5,
+      "learning_rate": 1.18e-05,
+      "loss": 3.8125,
+      "mean_token_accuracy": 0.9321934457868337,
+      "num_tokens": 296999.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.21873723103199155,
+      "epoch": 0.028,
+      "grad_norm": 99.0,
+      "learning_rate": 1.38e-05,
+      "loss": 3.5653,
+      "mean_token_accuracy": 0.9368013516068459,
+      "num_tokens": 342114.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.22916369761805983,
+      "epoch": 0.032,
+      "grad_norm": 94.0,
+      "learning_rate": 1.58e-05,
+      "loss": 3.5791,
+      "mean_token_accuracy": 0.9380934119224549,
+      "num_tokens": 390903.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.2319087312091142,
+      "epoch": 0.036,
+      "grad_norm": 98.5,
+      "learning_rate": 1.7800000000000002e-05,
+      "loss": 3.65,
+      "mean_token_accuracy": 0.9355392649769783,
+      "num_tokens": 440359.0,
+      "step": 90
+    },
+    {
+      "entropy": 0.2556251535890624,
+      "epoch": 0.04,
+      "grad_norm": 107.0,
+      "learning_rate": 1.98e-05,
+      "loss": 4.1158,
+      "mean_token_accuracy": 0.9278487522155047,
+      "num_tokens": 488481.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.04,
+      "eval_biology_entropy": 0.5274985447078944,
+      "eval_biology_loss": 1.795237421989441,
+      "eval_biology_mean_token_accuracy": 0.6946995661854743,
+      "eval_biology_num_tokens": 488481.0,
+      "eval_biology_runtime": 34.5955,
+      "eval_biology_samples_per_second": 14.453,
+      "eval_biology_steps_per_second": 14.453,
+      "step": 100
+    },
+    {
+      "epoch": 0.04,
+      "eval_chemistry_entropy": 0.35461274841427803,
+      "eval_chemistry_loss": 1.0036289691925049,
+      "eval_chemistry_mean_token_accuracy": 0.797036738216877,
+      "eval_chemistry_num_tokens": 488481.0,
+      "eval_chemistry_runtime": 39.1471,
+      "eval_chemistry_samples_per_second": 12.772,
+      "eval_chemistry_steps_per_second": 12.772,
+      "step": 100
+    },
+    {
+      "entropy": 0.24757922887802125,
+      "epoch": 0.044,
+      "grad_norm": 93.5,
+      "learning_rate": 1.98e-05,
+      "loss": 3.9008,
+      "mean_token_accuracy": 0.9320466317236423,
+      "num_tokens": 537471.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.26570149913895874,
+      "epoch": 0.048,
+      "grad_norm": 118.0,
+      "learning_rate": 1.957777777777778e-05,
+      "loss": 4.3011,
+      "mean_token_accuracy": 0.9242724630981684,
+      "num_tokens": 589868.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.25850508124567567,
+      "epoch": 0.052,
+      "grad_norm": 145.0,
+      "learning_rate": 1.9355555555555556e-05,
+      "loss": 4.164,
+      "mean_token_accuracy": 0.9254890568554401,
+      "num_tokens": 639512.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.27587457664776593,
+      "epoch": 0.056,
+      "grad_norm": 98.5,
+      "learning_rate": 1.9133333333333335e-05,
+      "loss": 4.2555,
+      "mean_token_accuracy": 0.9243636153638363,
+      "num_tokens": 690754.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.26002400361467154,
+      "epoch": 0.06,
+      "grad_norm": 90.5,
+      "learning_rate": 1.8911111111111115e-05,
+      "loss": 4.0114,
+      "mean_token_accuracy": 0.9282606355845928,
+      "num_tokens": 737075.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.26948123425245285,
+      "epoch": 0.064,
+      "grad_norm": 90.5,
+      "learning_rate": 1.868888888888889e-05,
+      "loss": 4.3559,
+      "mean_token_accuracy": 0.9212390169501304,
+      "num_tokens": 791190.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.2733419948257506,
+      "epoch": 0.068,
+      "grad_norm": 121.5,
+      "learning_rate": 1.8466666666666667e-05,
+      "loss": 4.4664,
+      "mean_token_accuracy": 0.9200532209128142,
+      "num_tokens": 841344.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.2954006634186953,
+      "epoch": 0.072,
+      "grad_norm": 132.0,
+      "learning_rate": 1.8244444444444447e-05,
+      "loss": 4.5686,
+      "mean_token_accuracy": 0.917558104172349,
+      "num_tokens": 890075.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.28410933739505706,
+      "epoch": 0.076,
+      "grad_norm": 81.0,
+      "learning_rate": 1.8022222222222223e-05,
+      "loss": 4.5413,
+      "mean_token_accuracy": 0.9189106050878764,
+      "num_tokens": 940599.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.27199700379278513,
+      "epoch": 0.08,
+      "grad_norm": 114.0,
+      "learning_rate": 1.7800000000000002e-05,
+      "loss": 4.4607,
+      "mean_token_accuracy": 0.9194032322615385,
+      "num_tokens": 989860.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.08,
+      "eval_biology_entropy": 0.5748336746096611,
+      "eval_biology_loss": 1.670861005783081,
+      "eval_biology_mean_token_accuracy": 0.6952305646538735,
+      "eval_biology_num_tokens": 989860.0,
+      "eval_biology_runtime": 34.4825,
+      "eval_biology_samples_per_second": 14.5,
+      "eval_biology_steps_per_second": 14.5,
+      "step": 200
+    },
+    {
+      "epoch": 0.08,
+      "eval_chemistry_entropy": 0.37541738474369046,
+      "eval_chemistry_loss": 0.9533677697181702,
+      "eval_chemistry_mean_token_accuracy": 0.7976788550019264,
+      "eval_chemistry_num_tokens": 989860.0,
+      "eval_chemistry_runtime": 39.0475,
+      "eval_chemistry_samples_per_second": 12.805,
+      "eval_chemistry_steps_per_second": 12.805,
+      "step": 200
+    },
+    {
+      "entropy": 0.2819324039854109,
+      "epoch": 0.084,
+      "grad_norm": 112.5,
+      "learning_rate": 1.7577777777777782e-05,
+      "loss": 4.4236,
+      "mean_token_accuracy": 0.9213832955807447,
+      "num_tokens": 1041199.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.2940462036523968,
+      "epoch": 0.088,
+      "grad_norm": 87.5,
+      "learning_rate": 1.7355555555555558e-05,
+      "loss": 4.6065,
+      "mean_token_accuracy": 0.9184617388993501,
+      "num_tokens": 1087099.0,
+      "step": 220
+    },
+    {
+      "entropy": 0.2821590228006244,
+      "epoch": 0.092,
+      "grad_norm": 122.5,
+      "learning_rate": 1.7133333333333334e-05,
+      "loss": 4.4648,
+      "mean_token_accuracy": 0.9207406509667635,
+      "num_tokens": 1135090.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.27030886968132106,
+      "epoch": 0.096,
+      "grad_norm": 112.0,
+      "learning_rate": 1.691111111111111e-05,
+      "loss": 4.4654,
+      "mean_token_accuracy": 0.9213668972253799,
+      "num_tokens": 1181777.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.2798044764203951,
+      "epoch": 0.1,
+      "grad_norm": 122.5,
+      "learning_rate": 1.668888888888889e-05,
+      "loss": 4.3126,
+      "mean_token_accuracy": 0.9229083456099033,
+      "num_tokens": 1229242.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.2830052811419591,
+      "epoch": 0.104,
+      "grad_norm": 93.0,
+      "learning_rate": 1.646666666666667e-05,
+      "loss": 4.6108,
+      "mean_token_accuracy": 0.9175490267574787,
+      "num_tokens": 1283881.0,
+      "step": 260
+    },
+    {
+      "entropy": 0.2907633282011375,
+      "epoch": 0.108,
+      "grad_norm": 96.0,
+      "learning_rate": 1.6244444444444446e-05,
+      "loss": 4.5547,
+      "mean_token_accuracy": 0.9179787810891866,
+      "num_tokens": 1332507.0,
+      "step": 270
+    },
+    {
+      "entropy": 0.29076307769864795,
+      "epoch": 0.112,
+      "grad_norm": 131.0,
+      "learning_rate": 1.6022222222222225e-05,
+      "loss": 4.6411,
+      "mean_token_accuracy": 0.9173810668289661,
+      "num_tokens": 1385513.0,
+      "step": 280
+    },
+    {
+      "entropy": 0.27884717038832607,
+      "epoch": 0.116,
+      "grad_norm": 127.5,
+      "learning_rate": 1.58e-05,
+      "loss": 4.5018,
+      "mean_token_accuracy": 0.9193264693021774,
+      "num_tokens": 1432251.0,
+      "step": 290
+    },
+    {
+      "entropy": 0.28094425410963597,
+      "epoch": 0.12,
+      "grad_norm": 123.0,
+      "learning_rate": 1.5577777777777777e-05,
+      "loss": 4.4214,
+      "mean_token_accuracy": 0.9191206317394972,
+      "num_tokens": 1483667.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.12,
+      "eval_biology_entropy": 0.5509931429475546,
+      "eval_biology_loss": 1.7029587030410767,
+      "eval_biology_mean_token_accuracy": 0.6963970780968666,
+      "eval_biology_num_tokens": 1483667.0,
+      "eval_biology_runtime": 34.7873,
+      "eval_biology_samples_per_second": 14.373,
+      "eval_biology_steps_per_second": 14.373,
+      "step": 300
+    },
+    {
+      "epoch": 0.12,
+      "eval_chemistry_entropy": 0.36057418985664846,
+      "eval_chemistry_loss": 0.9737709164619446,
+      "eval_chemistry_mean_token_accuracy": 0.7957612861990928,
+      "eval_chemistry_num_tokens": 1483667.0,
+      "eval_chemistry_runtime": 39.2784,
+      "eval_chemistry_samples_per_second": 12.73,
+      "eval_chemistry_steps_per_second": 12.73,
+      "step": 300
+    },
+    {
+      "entropy": 0.29498244081623853,
+      "epoch": 0.124,
+      "grad_norm": 112.0,
+      "learning_rate": 1.5355555555555557e-05,
+      "loss": 4.7174,
+      "mean_token_accuracy": 0.9161882497370243,
+      "num_tokens": 1534286.0,
+      "step": 310
+    },
+    {
+      "entropy": 0.2827191028976813,
+      "epoch": 0.128,
+      "grad_norm": 125.5,
+      "learning_rate": 1.5133333333333335e-05,
+      "loss": 4.434,
+      "mean_token_accuracy": 0.9199115991592407,
+      "num_tokens": 1582368.0,
+      "step": 320
+    },
+    {
+      "entropy": 0.2590224297251552,
+      "epoch": 0.132,
+      "grad_norm": 108.0,
+      "learning_rate": 1.4911111111111113e-05,
+      "loss": 4.2696,
+      "mean_token_accuracy": 0.9226452499628067,
+      "num_tokens": 1630878.0,
+      "step": 330
+    },
+    {
+      "entropy": 0.2799088359810412,
+      "epoch": 0.136,
+      "grad_norm": 89.0,
+      "learning_rate": 1.468888888888889e-05,
+      "loss": 4.1858,
+      "mean_token_accuracy": 0.9264892596751452,
+      "num_tokens": 1677060.0,
+      "step": 340
+    },
+    {
+      "entropy": 0.25218479917384684,
+      "epoch": 0.14,
+      "grad_norm": 102.5,
+      "learning_rate": 1.4466666666666668e-05,
+      "loss": 4.0457,
+      "mean_token_accuracy": 0.9256420440971851,
+      "num_tokens": 1725186.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.2768413402140141,
+      "epoch": 0.144,
+      "grad_norm": 165.0,
+      "learning_rate": 1.4244444444444444e-05,
+      "loss": 4.3177,
+      "mean_token_accuracy": 0.9229396734386682,
+      "num_tokens": 1773764.0,
+      "step": 360
+    },
+    {
+      "entropy": 0.2669145804364234,
+      "epoch": 0.148,
+      "grad_norm": 119.0,
+      "learning_rate": 1.4022222222222224e-05,
+      "loss": 4.3499,
+      "mean_token_accuracy": 0.923576295748353,
+      "num_tokens": 1823706.0,
+      "step": 370
+    },
+    {
+      "entropy": 0.26631026088725773,
+      "epoch": 0.152,
+      "grad_norm": 100.0,
+      "learning_rate": 1.38e-05,
+      "loss": 4.2629,
+      "mean_token_accuracy": 0.9251279231160879,
+      "num_tokens": 1873167.0,
+      "step": 380
+    },
+    {
+      "entropy": 0.2822582958266139,
+      "epoch": 0.156,
+      "grad_norm": 114.0,
+      "learning_rate": 1.3577777777777778e-05,
+      "loss": 4.5076,
+      "mean_token_accuracy": 0.918907954916358,
+      "num_tokens": 1921484.0,
+      "step": 390
+    },
+    {
+      "entropy": 0.2694689558353275,
+      "epoch": 0.16,
+      "grad_norm": 98.5,
+      "learning_rate": 1.3355555555555557e-05,
+      "loss": 4.3011,
+      "mean_token_accuracy": 0.9223983578383923,
+      "num_tokens": 1970077.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.16,
+      "eval_biology_entropy": 0.5380513223707676,
+      "eval_biology_loss": 1.7468076944351196,
+      "eval_biology_mean_token_accuracy": 0.6966642059087753,
+      "eval_biology_num_tokens": 1970077.0,
+      "eval_biology_runtime": 34.907,
+      "eval_biology_samples_per_second": 14.324,
+      "eval_biology_steps_per_second": 14.324,
+      "step": 400
+    },
+    {
+      "epoch": 0.16,
+      "eval_chemistry_entropy": 0.3548970829844475,
+      "eval_chemistry_loss": 0.9920993447303772,
+      "eval_chemistry_mean_token_accuracy": 0.7959754932522773,
+      "eval_chemistry_num_tokens": 1970077.0,
+      "eval_chemistry_runtime": 39.3019,
+      "eval_chemistry_samples_per_second": 12.722,
+      "eval_chemistry_steps_per_second": 12.722,
+      "step": 400
+    },
+    {
+      "entropy": 0.275055521633476,
+      "epoch": 0.164,
+      "grad_norm": 115.5,
+      "learning_rate": 1.3133333333333334e-05,
+      "loss": 4.2974,
+      "mean_token_accuracy": 0.9223645210266114,
+      "num_tokens": 2019053.0,
+      "step": 410
+    },
+    {
+      "entropy": 0.2445472558028996,
+      "epoch": 0.168,
+      "grad_norm": 114.0,
+      "learning_rate": 1.2911111111111111e-05,
+      "loss": 3.905,
+      "mean_token_accuracy": 0.9301919605582952,
+      "num_tokens": 2066815.0,
+      "step": 420
+    },
+    {
+      "entropy": 0.2561819740338251,
+      "epoch": 0.172,
+      "grad_norm": 104.0,
+      "learning_rate": 1.2688888888888891e-05,
+      "loss": 3.9763,
+      "mean_token_accuracy": 0.9278897974640131,
+      "num_tokens": 2116402.0,
+      "step": 430
+    },
+    {
+      "entropy": 0.24572510465513914,
+      "epoch": 0.176,
+      "grad_norm": 113.0,
+      "learning_rate": 1.2466666666666667e-05,
+      "loss": 3.9928,
+      "mean_token_accuracy": 0.9271609451621771,
+      "num_tokens": 2168354.0,
+      "step": 440
+    },
+    {
+      "entropy": 0.28641294413246216,
+      "epoch": 0.18,
+      "grad_norm": 86.0,
+      "learning_rate": 1.2244444444444445e-05,
+      "loss": 4.5332,
+      "mean_token_accuracy": 0.9190013557672501,
+      "num_tokens": 2219158.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.2572867315728217,
+      "epoch": 0.184,
+      "grad_norm": 103.5,
+      "learning_rate": 1.2022222222222224e-05,
+      "loss": 4.1139,
+      "mean_token_accuracy": 0.924973039329052,
+      "num_tokens": 2270617.0,
+      "step": 460
+    },
+    {
+      "entropy": 0.2555107505526394,
+      "epoch": 0.188,
+      "grad_norm": 105.5,
+      "learning_rate": 1.18e-05,
+      "loss": 3.9712,
+      "mean_token_accuracy": 0.928938215598464,
+      "num_tokens": 2322034.0,
+      "step": 470
+    },
+    {
+      "entropy": 0.2382448768010363,
+      "epoch": 0.192,
+      "grad_norm": 107.0,
+      "learning_rate": 1.1577777777777778e-05,
+      "loss": 3.8538,
+      "mean_token_accuracy": 0.9315264176577329,
+      "num_tokens": 2365822.0,
+      "step": 480
+    },
+    {
+      "entropy": 0.25814615855924783,
+      "epoch": 0.196,
+      "grad_norm": 109.5,
+      "learning_rate": 1.1355555555555558e-05,
+      "loss": 3.9795,
+      "mean_token_accuracy": 0.9277065552771091,
+      "num_tokens": 2413853.0,
+      "step": 490
+    },
+    {
+      "entropy": 0.25510344933718443,
+      "epoch": 0.2,
+      "grad_norm": 132.0,
+      "learning_rate": 1.1133333333333334e-05,
+      "loss": 4.0249,
+      "mean_token_accuracy": 0.9280555374920368,
+      "num_tokens": 2459876.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.2,
+      "eval_biology_entropy": 0.5138065100461244,
+      "eval_biology_loss": 1.7803232669830322,
+      "eval_biology_mean_token_accuracy": 0.6961626922488212,
+      "eval_biology_num_tokens": 2459876.0,
+      "eval_biology_runtime": 34.5634,
+      "eval_biology_samples_per_second": 14.466,
+      "eval_biology_steps_per_second": 14.466,
+      "step": 500
+    },
+    {
+      "epoch": 0.2,
+      "eval_chemistry_entropy": 0.3430728105157614,
+      "eval_chemistry_loss": 1.0075100660324097,
+      "eval_chemistry_mean_token_accuracy": 0.7967523041367531,
+      "eval_chemistry_num_tokens": 2459876.0,
+      "eval_chemistry_runtime": 38.8811,
+      "eval_chemistry_samples_per_second": 12.86,
+      "eval_chemistry_steps_per_second": 12.86,
+      "step": 500
+    },
+    {
+      "entropy": 0.24339726453181357,
+      "epoch": 0.204,
+      "grad_norm": 98.0,
+      "learning_rate": 1.0911111111111112e-05,
+      "loss": 3.8672,
+      "mean_token_accuracy": 0.928895252943039,
+      "num_tokens": 2509187.0,
+      "step": 510
+    },
+    {
+      "entropy": 0.2548065703595057,
+      "epoch": 0.208,
+      "grad_norm": 115.0,
+      "learning_rate": 1.0688888888888891e-05,
+      "loss": 3.8912,
+      "mean_token_accuracy": 0.9297985762357712,
+      "num_tokens": 2558762.0,
+      "step": 520
+    },
+    {
+      "entropy": 0.23889241772703826,
+      "epoch": 0.212,
+      "grad_norm": 96.5,
+      "learning_rate": 1.0466666666666668e-05,
+      "loss": 3.8306,
+      "mean_token_accuracy": 0.9323119979351759,
+      "num_tokens": 2608616.0,
+      "step": 530
+    },
+    {
+      "entropy": 0.2432718011084944,
+      "epoch": 0.216,
+      "grad_norm": 93.0,
+      "learning_rate": 1.0244444444444445e-05,
+      "loss": 3.7466,
+      "mean_token_accuracy": 0.9322359401732683,
+      "num_tokens": 2654953.0,
+      "step": 540
+    },
+    {
+      "entropy": 0.23889318462461234,
+      "epoch": 0.22,
+      "grad_norm": 98.0,
+      "learning_rate": 1.0022222222222222e-05,
+      "loss": 3.8405,
+      "mean_token_accuracy": 0.9313177198171616,
+      "num_tokens": 2704686.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.24933559738565236,
+      "epoch": 0.224,
+      "grad_norm": 87.5,
+      "learning_rate": 9.800000000000001e-06,
+      "loss": 3.9345,
+      "mean_token_accuracy": 0.9297204330563545,
+      "num_tokens": 2755347.0,
+      "step": 560
+    },
+    {
+      "entropy": 0.22135354387573897,
+      "epoch": 0.228,
+      "grad_norm": 99.0,
+      "learning_rate": 9.577777777777779e-06,
+      "loss": 3.5661,
+      "mean_token_accuracy": 0.9367739714682102,
+      "num_tokens": 2804340.0,
+      "step": 570
+    },
+    {
+      "entropy": 0.22378794639371336,
+      "epoch": 0.232,
+      "grad_norm": 116.0,
+      "learning_rate": 9.355555555555557e-06,
+      "loss": 3.5871,
+      "mean_token_accuracy": 0.9360409460961818,
+      "num_tokens": 2852298.0,
+      "step": 580
+    },
+    {
+      "entropy": 0.21883488199673592,
+      "epoch": 0.236,
+      "grad_norm": 112.0,
+      "learning_rate": 9.133333333333335e-06,
+      "loss": 3.5316,
+      "mean_token_accuracy": 0.9372412968426943,
+      "num_tokens": 2901227.0,
+      "step": 590
+    },
+    {
+      "entropy": 0.2272392014041543,
+      "epoch": 0.24,
+      "grad_norm": 136.0,
+      "learning_rate": 8.91111111111111e-06,
+      "loss": 3.5432,
+      "mean_token_accuracy": 0.9384493075311184,
+      "num_tokens": 2947346.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.24,
+      "eval_biology_entropy": 0.5223001695722341,
+      "eval_biology_loss": 1.7573493719100952,
+      "eval_biology_mean_token_accuracy": 0.6965594539642334,
+      "eval_biology_num_tokens": 2947346.0,
+      "eval_biology_runtime": 34.4882,
+      "eval_biology_samples_per_second": 14.498,
+      "eval_biology_steps_per_second": 14.498,
+      "step": 600
+    },
+    {
+      "epoch": 0.24,
+      "eval_chemistry_entropy": 0.3469998552650213,
+      "eval_chemistry_loss": 1.0004746913909912,
+      "eval_chemistry_mean_token_accuracy": 0.7966757331490517,
+      "eval_chemistry_num_tokens": 2947346.0,
+      "eval_chemistry_runtime": 39.0288,
+      "eval_chemistry_samples_per_second": 12.811,
+      "eval_chemistry_steps_per_second": 12.811,
+      "step": 600
+    },
+    {
+      "entropy": 0.24716254335362464,
+      "epoch": 0.244,
+      "grad_norm": 115.5,
+      "learning_rate": 8.68888888888889e-06,
+      "loss": 3.9633,
+      "mean_token_accuracy": 0.9289827451109887,
+      "num_tokens": 2996175.0,
+      "step": 610
+    },
+    {
+      "entropy": 0.23793726551812142,
+      "epoch": 0.248,
+      "grad_norm": 116.5,
+      "learning_rate": 8.466666666666668e-06,
+      "loss": 3.7559,
+      "mean_token_accuracy": 0.9330472752451897,
+      "num_tokens": 3043128.0,
+      "step": 620
+    },
+    {
+      "entropy": 0.2279190807370469,
+      "epoch": 0.252,
+      "grad_norm": 99.5,
+      "learning_rate": 8.244444444444444e-06,
+      "loss": 3.6192,
+      "mean_token_accuracy": 0.9342298805713654,
+      "num_tokens": 3090827.0,
+      "step": 630
+    },
+    {
+      "entropy": 0.2271652126684785,
+      "epoch": 0.256,
+      "grad_norm": 94.0,
+      "learning_rate": 8.022222222222222e-06,
+      "loss": 3.6282,
+      "mean_token_accuracy": 0.9344729892909527,
+      "num_tokens": 3139957.0,
+      "step": 640
+    },
+    {
+      "entropy": 0.21735699030105024,
+      "epoch": 0.26,
+      "grad_norm": 94.5,
+      "learning_rate": 7.800000000000002e-06,
+      "loss": 3.4237,
+      "mean_token_accuracy": 0.9400590911507607,
+      "num_tokens": 3183188.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.23376386400777846,
+      "epoch": 0.264,
+      "grad_norm": 92.5,
+      "learning_rate": 7.5777777777777785e-06,
+      "loss": 3.6323,
+      "mean_token_accuracy": 0.9351054444909096,
+      "num_tokens": 3232159.0,
+      "step": 660
+    },
+    {
+      "entropy": 0.2265489047858864,
+      "epoch": 0.268,
+      "grad_norm": 108.0,
+      "learning_rate": 7.3555555555555555e-06,
+      "loss": 3.5086,
+      "mean_token_accuracy": 0.9367269530892373,
+      "num_tokens": 3283908.0,
+      "step": 670
+    },
+    {
+      "entropy": 0.21268082661554216,
+      "epoch": 0.272,
+      "grad_norm": 112.5,
+      "learning_rate": 7.133333333333334e-06,
+      "loss": 3.3604,
+      "mean_token_accuracy": 0.940755220130086,
+      "num_tokens": 3335951.0,
+      "step": 680
+    },
+    {
+      "entropy": 0.23243776755407453,
+      "epoch": 0.276,
+      "grad_norm": 109.0,
+      "learning_rate": 6.911111111111112e-06,
+      "loss": 3.7226,
+      "mean_token_accuracy": 0.932453778013587,
+      "num_tokens": 3387755.0,
+      "step": 690
+    },
+    {
+      "entropy": 0.22202653533313424,
+      "epoch": 0.28,
+      "grad_norm": 109.0,
+      "learning_rate": 6.688888888888889e-06,
+      "loss": 3.5264,
+      "mean_token_accuracy": 0.9369195692241192,
+      "num_tokens": 3437213.0,
+      "step": 700
+    },
+    {
+      "epoch": 0.28,
+      "eval_biology_entropy": 0.5441444292068481,
+      "eval_biology_loss": 1.6984660625457764,
+      "eval_biology_mean_token_accuracy": 0.6988616171479225,
+      "eval_biology_num_tokens": 3437213.0,
+      "eval_biology_runtime": 34.7957,
+      "eval_biology_samples_per_second": 14.37,
+      "eval_biology_steps_per_second": 14.37,
+      "step": 700
+    },
+    {
+      "epoch": 0.28,
+      "eval_chemistry_entropy": 0.3569708024710417,
+      "eval_chemistry_loss": 0.9757047295570374,
+      "eval_chemistry_mean_token_accuracy": 0.798202440559864,
+      "eval_chemistry_num_tokens": 3437213.0,
+      "eval_chemistry_runtime": 39.0331,
+      "eval_chemistry_samples_per_second": 12.81,
+      "eval_chemistry_steps_per_second": 12.81,
+      "step": 700
+    },
+    {
+      "entropy": 0.22738665074575692,
+      "epoch": 0.284,
+      "grad_norm": 80.0,
+      "learning_rate": 6.466666666666667e-06,
+      "loss": 3.521,
+      "mean_token_accuracy": 0.9371909685432911,
+      "num_tokens": 3489088.0,
+      "step": 710
+    },
+    {
+      "entropy": 0.22560963679570706,
+      "epoch": 0.288,
+      "grad_norm": 87.0,
+      "learning_rate": 6.2444444444444456e-06,
+      "loss": 3.6287,
+      "mean_token_accuracy": 0.9355691641569137,
+      "num_tokens": 3539731.0,
+      "step": 720
+    },
+    {
+      "entropy": 0.20523998411372305,
+      "epoch": 0.292,
+      "grad_norm": 96.0,
+      "learning_rate": 6.0222222222222225e-06,
+      "loss": 3.1107,
+      "mean_token_accuracy": 0.9447531200945377,
+      "num_tokens": 3586433.0,
+      "step": 730
+    },
+    {
+      "entropy": 0.21899892918299885,
+      "epoch": 0.296,
+      "grad_norm": 81.0,
+      "learning_rate": 5.8e-06,
+      "loss": 3.4799,
+      "mean_token_accuracy": 0.9385896824300289,
+      "num_tokens": 3633448.0,
+      "step": 740
+    },
+    {
+      "entropy": 0.20959302789997308,
+      "epoch": 0.3,
+      "grad_norm": 97.5,
+      "learning_rate": 5.577777777777778e-06,
+      "loss": 3.2429,
+      "mean_token_accuracy": 0.9431923229247332,
+      "num_tokens": 3685080.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.19758035039994865,
+      "epoch": 0.304,
+      "grad_norm": 81.0,
+      "learning_rate": 5.355555555555556e-06,
+      "loss": 3.0738,
+      "mean_token_accuracy": 0.9454657685011625,
+      "num_tokens": 3733488.0,
+      "step": 760
+    },
+    {
+      "entropy": 0.20672397674061357,
+      "epoch": 0.308,
+      "grad_norm": 94.5,
+      "learning_rate": 5.133333333333334e-06,
+      "loss": 3.2232,
+      "mean_token_accuracy": 0.9428212687373161,
+      "num_tokens": 3779708.0,
+      "step": 770
+    },
+    {
+      "entropy": 0.18807534142397345,
+      "epoch": 0.312,
+      "grad_norm": 96.0,
+      "learning_rate": 4.911111111111112e-06,
+      "loss": 2.9023,
+      "mean_token_accuracy": 0.9485364310443402,
+      "num_tokens": 3825212.0,
+      "step": 780
+    },
+    {
+      "entropy": 0.20133916779886932,
+      "epoch": 0.316,
+      "grad_norm": 169.0,
+      "learning_rate": 4.6888888888888895e-06,
+      "loss": 3.2389,
+      "mean_token_accuracy": 0.9434301875531673,
+      "num_tokens": 3870912.0,
+      "step": 790
+    },
+    {
+      "entropy": 0.19466696886811405,
+      "epoch": 0.32,
+      "grad_norm": 79.0,
+      "learning_rate": 4.4666666666666665e-06,
+      "loss": 3.0524,
+      "mean_token_accuracy": 0.9459826443344355,
+      "num_tokens": 3920545.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.32,
+      "eval_biology_entropy": 0.5274494394809007,
+      "eval_biology_loss": 1.7321778535842896,
+      "eval_biology_mean_token_accuracy": 0.6985582538843155,
+      "eval_biology_num_tokens": 3920545.0,
+      "eval_biology_runtime": 34.5204,
+      "eval_biology_samples_per_second": 14.484,
+      "eval_biology_steps_per_second": 14.484,
+      "step": 800
+    },
+    {
+      "epoch": 0.32,
+      "eval_chemistry_entropy": 0.3479532795697451,
+      "eval_chemistry_loss": 0.9915227293968201,
+      "eval_chemistry_mean_token_accuracy": 0.7979016464352607,
+      "eval_chemistry_num_tokens": 3920545.0,
+      "eval_chemistry_runtime": 39.0436,
+      "eval_chemistry_samples_per_second": 12.806,
+      "eval_chemistry_steps_per_second": 12.806,
+      "step": 800
+    },
+    {
+      "entropy": 0.19644011000636966,
+      "epoch": 0.324,
+      "grad_norm": 71.0,
+      "learning_rate": 4.244444444444445e-06,
+      "loss": 3.0306,
+      "mean_token_accuracy": 0.9468632739037275,
+      "num_tokens": 3967499.0,
+      "step": 810
+    },
+    {
+      "entropy": 0.19098295441363006,
+      "epoch": 0.328,
+      "grad_norm": 79.0,
+      "learning_rate": 4.022222222222222e-06,
+      "loss": 3.0312,
+      "mean_token_accuracy": 0.9459359969943761,
+      "num_tokens": 4016673.0,
+      "step": 820
+    },
+    {
+      "entropy": 0.19487170833162965,
+      "epoch": 0.332,
+      "grad_norm": 78.0,
+      "learning_rate": 3.8000000000000005e-06,
+      "loss": 2.9308,
+      "mean_token_accuracy": 0.9479724440723658,
+      "num_tokens": 4063859.0,
+      "step": 830
+    },
+    {
+      "entropy": 0.19424405433237552,
+      "epoch": 0.336,
+      "grad_norm": 81.5,
+      "learning_rate": 3.577777777777778e-06,
+      "loss": 3.0624,
+      "mean_token_accuracy": 0.9451455675065518,
+      "num_tokens": 4114077.0,
+      "step": 840
+    },
+    {
+      "entropy": 0.1846273104660213,
+      "epoch": 0.34,
+      "grad_norm": 85.5,
+      "learning_rate": 3.3555555555555557e-06,
+      "loss": 2.8559,
+      "mean_token_accuracy": 0.9497106105089188,
+      "num_tokens": 4160168.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.19622367892879994,
+      "epoch": 0.344,
+      "grad_norm": 84.5,
+      "learning_rate": 3.133333333333334e-06,
+      "loss": 3.1202,
+      "mean_token_accuracy": 0.9454001683741808,
+      "num_tokens": 4210403.0,
+      "step": 860
+    },
+    {
+      "entropy": 0.16956532222684473,
+      "epoch": 0.348,
+      "grad_norm": 69.0,
+      "learning_rate": 2.9111111111111114e-06,
+      "loss": 2.7258,
+      "mean_token_accuracy": 0.952002327516675,
+      "num_tokens": 4256944.0,
+      "step": 870
+    },
+    {
+      "entropy": 0.1938292073085904,
+      "epoch": 0.352,
+      "grad_norm": 121.0,
+      "learning_rate": 2.6888888888888892e-06,
+      "loss": 2.9593,
+      "mean_token_accuracy": 0.9468233805149794,
+      "num_tokens": 4306949.0,
+      "step": 880
+    },
+    {
+      "entropy": 0.19562577470205725,
+      "epoch": 0.356,
+      "grad_norm": 88.5,
+      "learning_rate": 2.466666666666667e-06,
+      "loss": 3.1355,
+      "mean_token_accuracy": 0.9445545077323914,
+      "num_tokens": 4355070.0,
+      "step": 890
+    },
+    {
+      "entropy": 0.18795418140944092,
+      "epoch": 0.36,
+      "grad_norm": 70.0,
+      "learning_rate": 2.2444444444444445e-06,
+      "loss": 2.9405,
+      "mean_token_accuracy": 0.9477147374302148,
+      "num_tokens": 4406104.0,
+      "step": 900
+    },
+    {
+      "epoch": 0.36,
+      "eval_biology_entropy": 0.5236208895593881,
+      "eval_biology_loss": 1.740578532218933,
+      "eval_biology_mean_token_accuracy": 0.6986331322193146,
+      "eval_biology_num_tokens": 4406104.0,
+      "eval_biology_runtime": 34.8179,
+      "eval_biology_samples_per_second": 14.36,
+      "eval_biology_steps_per_second": 14.36,
+      "step": 900
+    },
+    {
+      "epoch": 0.36,
+      "eval_chemistry_entropy": 0.34630436305701734,
+      "eval_chemistry_loss": 0.9945363402366638,
+      "eval_chemistry_mean_token_accuracy": 0.7981406454443931,
+      "eval_chemistry_num_tokens": 4406104.0,
+      "eval_chemistry_runtime": 39.2261,
+      "eval_chemistry_samples_per_second": 12.747,
+      "eval_chemistry_steps_per_second": 12.747,
+      "step": 900
+    },
+    {
+      "entropy": 0.16734268208965658,
+      "epoch": 0.364,
+      "grad_norm": 95.5,
+      "learning_rate": 2.0222222222222223e-06,
+      "loss": 2.7285,
+      "mean_token_accuracy": 0.9537675432860852,
+      "num_tokens": 4455015.0,
+      "step": 910
+    },
+    {
+      "entropy": 0.1797151634003967,
+      "epoch": 0.368,
+      "grad_norm": 81.5,
+      "learning_rate": 1.8000000000000001e-06,
+      "loss": 2.821,
+      "mean_token_accuracy": 0.9497428938746453,
+      "num_tokens": 4504001.0,
+      "step": 920
+    },
+    {
+      "entropy": 0.17741366024129093,
+      "epoch": 0.372,
+      "grad_norm": 86.0,
+      "learning_rate": 1.5777777777777778e-06,
+      "loss": 2.7105,
+      "mean_token_accuracy": 0.9517100431025028,
+      "num_tokens": 4550269.0,
+      "step": 930
+    },
+    {
+      "entropy": 0.1795904800761491,
+      "epoch": 0.376,
+      "grad_norm": 86.5,
+      "learning_rate": 1.3555555555555558e-06,
+      "loss": 2.7266,
+      "mean_token_accuracy": 0.9516325149685144,
+      "num_tokens": 4597559.0,
+      "step": 940
+    },
+    {
+      "entropy": 0.1703337698010728,
+      "epoch": 0.38,
+      "grad_norm": 92.5,
+      "learning_rate": 1.1333333333333334e-06,
+      "loss": 2.6341,
+      "mean_token_accuracy": 0.9536429800093174,
+      "num_tokens": 4646903.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.17055770214647054,
+      "epoch": 0.384,
+      "grad_norm": 62.25,
+      "learning_rate": 9.111111111111113e-07,
+      "loss": 2.6421,
+      "mean_token_accuracy": 0.952850966155529,
+      "num_tokens": 4693812.0,
+      "step": 960
+    },
+    {
+      "entropy": 0.16687732187565416,
+      "epoch": 0.388,
+      "grad_norm": 81.0,
+      "learning_rate": 6.88888888888889e-07,
+      "loss": 2.6243,
+      "mean_token_accuracy": 0.9537446014583111,
+      "num_tokens": 4740309.0,
+      "step": 970
+    },
+    {
+      "entropy": 0.1786689094034955,
+      "epoch": 0.392,
+      "grad_norm": 86.0,
+      "learning_rate": 4.666666666666667e-07,
+      "loss": 2.8012,
+      "mean_token_accuracy": 0.9494896039366723,
+      "num_tokens": 4790965.0,
+      "step": 980
+    },
+    {
+      "entropy": 0.1676131068263203,
+      "epoch": 0.396,
+      "grad_norm": 84.5,
+      "learning_rate": 2.444444444444445e-07,
+      "loss": 2.7078,
+      "mean_token_accuracy": 0.9509209491312504,
+      "num_tokens": 4839186.0,
+      "step": 990
+    },
+    {
+      "entropy": 0.17521020406857132,
+      "epoch": 0.4,
+      "grad_norm": 86.5,
+      "learning_rate": 2.2222222222222224e-08,
+      "loss": 2.6856,
+      "mean_token_accuracy": 0.9513530559837818,
+      "num_tokens": 4887094.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4,
+      "eval_biology_entropy": 0.524978510722518,
+      "eval_biology_loss": 1.7350329160690308,
+      "eval_biology_mean_token_accuracy": 0.6986925270557404,
+      "eval_biology_num_tokens": 4887094.0,
+      "eval_biology_runtime": 34.5766,
+      "eval_biology_samples_per_second": 14.461,
+      "eval_biology_steps_per_second": 14.461,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4,
+      "eval_chemistry_entropy": 0.3469348890334368,
+      "eval_chemistry_loss": 0.9920687079429626,
+      "eval_chemistry_mean_token_accuracy": 0.7983826797604561,
+      "eval_chemistry_num_tokens": 4887094.0,
+      "eval_chemistry_runtime": 39.0381,
+      "eval_chemistry_samples_per_second": 12.808,
+      "eval_chemistry_steps_per_second": 12.808,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.277009915508544e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:674e45de12e69af34259f39e1f9e94e606fc58f4ef11f5ba9fa01ef4a5134486
+size 6609