Instructions to use modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base

SGLang

How to use modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base with Docker Model Runner:
```
docker model run hf.co/modrill/lingcoder_shortcot_4k_lora_qwen3_4b_base
```

modrill commited on 10 days ago

Commit

944a1b0

verified ·

1 Parent(s): 3b66962

Add files using upload-large-folder tool

Browse files

Files changed (22) hide show

.gitattributes +2 -0
README.md +56 -10
adapter_config.json +48 -0
adapter_model.safetensors +3 -0
chat_template.jinja +12 -0
checkpoint-3125/README.md +209 -0
checkpoint-3125/adapter_config.json +48 -0
checkpoint-3125/adapter_model.safetensors +3 -0
checkpoint-3125/chat_template.jinja +12 -0
checkpoint-3125/optimizer.pt +3 -0
checkpoint-3125/rng_state_0.pth +3 -0
checkpoint-3125/rng_state_1.pth +3 -0
checkpoint-3125/rng_state_2.pth +3 -0
checkpoint-3125/rng_state_3.pth +3 -0
checkpoint-3125/scheduler.pt +3 -0
checkpoint-3125/tokenizer.json +3 -0
checkpoint-3125/tokenizer_config.json +30 -0
checkpoint-3125/trainer_state.json +3154 -0
checkpoint-3125/training_args.bin +3 -0
tokenizer.json +3 -0
tokenizer_config.json +30 -0
training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-3125/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -1,16 +1,62 @@
 ---
-license: cc-by-nc-4.0
 tags:
-- qwen3
-- mhm
-- text-generation
-library_name: transformers
 ---
-# lingcoder_shortcot_4k_lora_qwen3_4b_base
-Auto-uploaded by watcher.
-- Source path: `trl/models/qwen3-4b-base-restart-lingcoder-shortcot-4k-lora`
-- Uploaded at: `2026-05-20T05:44:57.337294`
-- Visibility: `public`

 ---
+base_model: Qwen/Qwen3-4B-Base
+library_name: peft
+model_name: qwen3-4b-base-restart-lingcoder-shortcot-4k-lora
 tags:
+- base_model:adapter:Qwen/Qwen3-4B-Base
+- lora
+- sft
+- transformers
+- trl
+licence: license
+pipeline_tag: text-generation
 ---
+# Model Card for qwen3-4b-base-restart-lingcoder-shortcot-4k-lora
+This model is a fine-tuned version of [Qwen/Qwen3-4B-Base](https://huggingface.co/Qwen/Qwen3-4B-Base).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+## Quick start
+```python
+from transformers import pipeline
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+## Training procedure
+This model was trained with SFT.
+### Framework versions
+- PEFT 0.19.1
+- TRL: 1.5.0.dev0
+- Transformers: 5.8.0
+- Pytorch: 2.11.0
+- Datasets: 4.8.5
+- Tokenizers: 0.22.2
+## Citations
+Cite TRL as:
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```

adapter_config.json ADDED Viewed

	@@ -0,0 +1,48 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-4B-Base",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "gate_proj",
+    "o_proj",
+    "q_proj",
+    "k_proj",
+    "down_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0a8c355fc3fc3a1b4db160f7658f983d55dd9a59a600a23c32cf66501ea0729b
+size 264308896

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,12 @@

+{%- for message in messages %}
+{%- if message.role == "user" %}
+{{- '<|im_start|>user\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "system" %}
+{{- '<|im_start|>system\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "assistant" %}
+{{- '<|im_start|>assistant\n' }}{% generation %}{{ message.content }}{% endgeneration %}{{ '<|im_end|>\n' }}
+{%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+{{- '<|im_start|>assistant\n' }}
+{%- endif %}

checkpoint-3125/README.md ADDED Viewed

	@@ -0,0 +1,209 @@

+---
+base_model: Qwen/Qwen3-4B-Base
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Qwen/Qwen3-4B-Base
+- lora
+- sft
+- transformers
+- trl
+---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.19.1

checkpoint-3125/adapter_config.json ADDED Viewed

	@@ -0,0 +1,48 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-4B-Base",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "gate_proj",
+    "o_proj",
+    "q_proj",
+    "k_proj",
+    "down_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-3125/adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0a8c355fc3fc3a1b4db160f7658f983d55dd9a59a600a23c32cf66501ea0729b
+size 264308896

checkpoint-3125/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,12 @@

+{%- for message in messages %}
+{%- if message.role == "user" %}
+{{- '<|im_start|>user\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "system" %}
+{{- '<|im_start|>system\n' + message.content + '<|im_end|>\n' }}
+{%- elif message.role == "assistant" %}
+{{- '<|im_start|>assistant\n' }}{% generation %}{{ message.content }}{% endgeneration %}{{ '<|im_end|>\n' }}
+{%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+{{- '<|im_start|>assistant\n' }}
+{%- endif %}

checkpoint-3125/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3b5e6acdc3daf5ebfc51b358fb17bdd1f4bf0e347b9f0aeb278084ca7f7fd76b
+size 528915403

checkpoint-3125/rng_state_0.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:edde728faae9609404442e2484eb365220fb6315734058efbe29e6a3122ab519
+size 15429

checkpoint-3125/rng_state_1.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f83be5e98a1785888339fda9c3f3685963efb102c4db55d6c1125067140074f6
+size 15429

checkpoint-3125/rng_state_2.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:35c82797c6c9d779c45f9200f23e36769674e4f41d9bbe638c586153a805d1e8
+size 15429

checkpoint-3125/rng_state_3.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9779b0c14d1e1fa283183973cb2425a2f55979638f4066ee2cffa58b7b93c495
+size 15429

checkpoint-3125/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd5cdc84d2b4d9a932721799f114ebfcc55a4d590aebab27d5998a9f127ea471
+size 1465

checkpoint-3125/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

checkpoint-3125/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,30 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-3125/trainer_state.json ADDED Viewed

	@@ -0,0 +1,3154 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 3125,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 0.5006090428680181,
+      "epoch": 0.0032,
+      "grad_norm": 0.21088215708732605,
+      "learning_rate": 5.732484076433121e-06,
+      "loss": 0.5713953971862793,
+      "mean_token_accuracy": 0.8289921566843986,
+      "num_tokens": 533455.0,
+      "step": 10
+    },
+    {
+      "entropy": 0.5081364050507545,
+      "epoch": 0.0064,
+      "grad_norm": 0.12202276289463043,
+      "learning_rate": 1.2101910828025478e-05,
+      "loss": 0.5539228916168213,
+      "mean_token_accuracy": 0.8328638426959515,
+      "num_tokens": 1067567.0,
+      "step": 20
+    },
+    {
+      "entropy": 0.5529022507369519,
+      "epoch": 0.0096,
+      "grad_norm": 0.11152374744415283,
+      "learning_rate": 1.8471337579617834e-05,
+      "loss": 0.5397152423858642,
+      "mean_token_accuracy": 0.8347398519515992,
+      "num_tokens": 1600896.0,
+      "step": 30
+    },
+    {
+      "entropy": 0.559289076551795,
+      "epoch": 0.0128,
+      "grad_norm": 0.10228055715560913,
+      "learning_rate": 2.4840764331210193e-05,
+      "loss": 0.5441999435424805,
+      "mean_token_accuracy": 0.8323002874851226,
+      "num_tokens": 2133971.0,
+      "step": 40
+    },
+    {
+      "entropy": 0.5037001080811023,
+      "epoch": 0.016,
+      "grad_norm": 0.10043259710073471,
+      "learning_rate": 3.121019108280255e-05,
+      "loss": 0.5217269420623779,
+      "mean_token_accuracy": 0.8391494512557983,
+      "num_tokens": 2683285.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.5125732887536287,
+      "epoch": 0.0192,
+      "grad_norm": 0.10314421355724335,
+      "learning_rate": 3.7579617834394906e-05,
+      "loss": 0.515027666091919,
+      "mean_token_accuracy": 0.8395569138228893,
+      "num_tokens": 3217227.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.49653887934982777,
+      "epoch": 0.0224,
+      "grad_norm": 0.09557953476905823,
+      "learning_rate": 4.394904458598726e-05,
+      "loss": 0.4976811408996582,
+      "mean_token_accuracy": 0.8442590735852719,
+      "num_tokens": 3759709.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.5080962128937244,
+      "epoch": 0.0256,
+      "grad_norm": 0.11208797246217728,
+      "learning_rate": 5.031847133757962e-05,
+      "loss": 0.507160758972168,
+      "mean_token_accuracy": 0.8411325640976429,
+      "num_tokens": 4304319.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.5011889148503542,
+      "epoch": 0.0288,
+      "grad_norm": 0.11951766908168793,
+      "learning_rate": 5.6687898089171974e-05,
+      "loss": 0.5021108627319336,
+      "mean_token_accuracy": 0.8430344834923744,
+      "num_tokens": 4846886.0,
+      "step": 90
+    },
+    {
+      "entropy": 0.4849133860319853,
+      "epoch": 0.032,
+      "grad_norm": 0.10586579144001007,
+      "learning_rate": 6.305732484076433e-05,
+      "loss": 0.48747682571411133,
+      "mean_token_accuracy": 0.8480054885149002,
+      "num_tokens": 5395975.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.4885189659893513,
+      "epoch": 0.0352,
+      "grad_norm": 0.1046752855181694,
+      "learning_rate": 6.942675159235669e-05,
+      "loss": 0.49596099853515624,
+      "mean_token_accuracy": 0.8458828076720237,
+      "num_tokens": 5932911.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.5003112446516752,
+      "epoch": 0.0384,
+      "grad_norm": 0.10725490748882294,
+      "learning_rate": 7.579617834394906e-05,
+      "loss": 0.49750380516052245,
+      "mean_token_accuracy": 0.8431005917489529,
+      "num_tokens": 6470335.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.4913600035011768,
+      "epoch": 0.0416,
+      "grad_norm": 0.10014921426773071,
+      "learning_rate": 8.21656050955414e-05,
+      "loss": 0.4953355312347412,
+      "mean_token_accuracy": 0.8447930745780468,
+      "num_tokens": 7002932.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.47147834971547126,
+      "epoch": 0.0448,
+      "grad_norm": 0.09612508863210678,
+      "learning_rate": 8.853503184713377e-05,
+      "loss": 0.47597084045410154,
+      "mean_token_accuracy": 0.8496062621474266,
+      "num_tokens": 7539204.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.499130529910326,
+      "epoch": 0.048,
+      "grad_norm": 0.1032324880361557,
+      "learning_rate": 9.490445859872612e-05,
+      "loss": 0.5046750068664551,
+      "mean_token_accuracy": 0.8422100260853768,
+      "num_tokens": 8074373.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.4958477046340704,
+      "epoch": 0.0512,
+      "grad_norm": 0.10810820758342743,
+      "learning_rate": 9.9999887960338e-05,
+      "loss": 0.49416637420654297,
+      "mean_token_accuracy": 0.8444528333842755,
+      "num_tokens": 8598228.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.4908892083913088,
+      "epoch": 0.0544,
+      "grad_norm": 0.09390716999769211,
+      "learning_rate": 9.999596662488985e-05,
+      "loss": 0.4936826705932617,
+      "mean_token_accuracy": 0.8448271870613098,
+      "num_tokens": 9131410.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.48737211152911186,
+      "epoch": 0.0576,
+      "grad_norm": 0.09588056057691574,
+      "learning_rate": 9.998644380844701e-05,
+      "loss": 0.489206600189209,
+      "mean_token_accuracy": 0.8464116282761097,
+      "num_tokens": 9675197.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.4986014157533646,
+      "epoch": 0.0608,
+      "grad_norm": 0.09068372845649719,
+      "learning_rate": 9.997132057793305e-05,
+      "loss": 0.498153829574585,
+      "mean_token_accuracy": 0.8440827287733554,
+      "num_tokens": 10214255.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.48283746242523196,
+      "epoch": 0.064,
+      "grad_norm": 0.08930274099111557,
+      "learning_rate": 9.99505986277344e-05,
+      "loss": 0.48722114562988283,
+      "mean_token_accuracy": 0.8474265016615391,
+      "num_tokens": 10749783.0,
+      "step": 200
+    },
+    {
+      "entropy": 0.4735735967755318,
+      "epoch": 0.0672,
+      "grad_norm": 0.09251251816749573,
+      "learning_rate": 9.992428027951055e-05,
+      "loss": 0.47350778579711916,
+      "mean_token_accuracy": 0.8512057550251484,
+      "num_tokens": 11298257.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.49423972591757775,
+      "epoch": 0.0704,
+      "grad_norm": 0.0889236181974411,
+      "learning_rate": 9.989236848193394e-05,
+      "loss": 0.4964627265930176,
+      "mean_token_accuracy": 0.844541211426258,
+      "num_tokens": 11832347.0,
+      "step": 220
+    },
+    {
+      "entropy": 0.47527725249528885,
+      "epoch": 0.0736,
+      "grad_norm": 0.08239874988794327,
+      "learning_rate": 9.985486681035949e-05,
+      "loss": 0.4780255317687988,
+      "mean_token_accuracy": 0.850318918377161,
+      "num_tokens": 12369986.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.4821702651679516,
+      "epoch": 0.0768,
+      "grad_norm": 0.08620651811361313,
+      "learning_rate": 9.98117794664242e-05,
+      "loss": 0.4842814922332764,
+      "mean_token_accuracy": 0.8476981841027736,
+      "num_tokens": 12902336.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.4926739055663347,
+      "epoch": 0.08,
+      "grad_norm": 0.0822502076625824,
+      "learning_rate": 9.976311127757621e-05,
+      "loss": 0.493633508682251,
+      "mean_token_accuracy": 0.8439111061394214,
+      "num_tokens": 13436860.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.4778115816414356,
+      "epoch": 0.0832,
+      "grad_norm": 0.08052525669336319,
+      "learning_rate": 9.970886769653409e-05,
+      "loss": 0.4785038471221924,
+      "mean_token_accuracy": 0.8490743987262249,
+      "num_tokens": 13981154.0,
+      "step": 260
+    },
+    {
+      "entropy": 0.4820780359208584,
+      "epoch": 0.0864,
+      "grad_norm": 0.08612845093011856,
+      "learning_rate": 9.964905480067586e-05,
+      "loss": 0.48402156829833987,
+      "mean_token_accuracy": 0.8483813837170601,
+      "num_tokens": 14508881.0,
+      "step": 270
+    },
+    {
+      "entropy": 0.47784071788191795,
+      "epoch": 0.0896,
+      "grad_norm": 0.08451592922210693,
+      "learning_rate": 9.958367929135807e-05,
+      "loss": 0.47811522483825686,
+      "mean_token_accuracy": 0.8496255606412888,
+      "num_tokens": 15047947.0,
+      "step": 280
+    },
+    {
+      "entropy": 0.47608734108507633,
+      "epoch": 0.0928,
+      "grad_norm": 0.07735177874565125,
+      "learning_rate": 9.951274849316505e-05,
+      "loss": 0.48113174438476564,
+      "mean_token_accuracy": 0.8494543135166168,
+      "num_tokens": 15598824.0,
+      "step": 290
+    },
+    {
+      "entropy": 0.47693981789052486,
+      "epoch": 0.096,
+      "grad_norm": 0.08198647201061249,
+      "learning_rate": 9.943627035308823e-05,
+      "loss": 0.478253173828125,
+      "mean_token_accuracy": 0.8493702694773674,
+      "num_tokens": 16137973.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.4718359481543303,
+      "epoch": 0.0992,
+      "grad_norm": 0.07386098057031631,
+      "learning_rate": 9.935425343963574e-05,
+      "loss": 0.471728515625,
+      "mean_token_accuracy": 0.850666993111372,
+      "num_tokens": 16691551.0,
+      "step": 310
+    },
+    {
+      "entropy": 0.49830644242465494,
+      "epoch": 0.1024,
+      "grad_norm": 0.10489208996295929,
+      "learning_rate": 9.926670694187252e-05,
+      "loss": 0.5005705833435059,
+      "mean_token_accuracy": 0.8427114583551883,
+      "num_tokens": 17229635.0,
+      "step": 320
+    },
+    {
+      "entropy": 0.4819621954113245,
+      "epoch": 0.1056,
+      "grad_norm": 0.08344434201717377,
+      "learning_rate": 9.917364066839066e-05,
+      "loss": 0.4840703964233398,
+      "mean_token_accuracy": 0.847594378888607,
+      "num_tokens": 17766215.0,
+      "step": 330
+    },
+    {
+      "entropy": 0.4873910341411829,
+      "epoch": 0.1088,
+      "grad_norm": 0.07981836050748825,
+      "learning_rate": 9.907506504621052e-05,
+      "loss": 0.4906445503234863,
+      "mean_token_accuracy": 0.8462508127093316,
+      "num_tokens": 18299051.0,
+      "step": 340
+    },
+    {
+      "entropy": 0.4688674874603748,
+      "epoch": 0.112,
+      "grad_norm": 0.08283731341362,
+      "learning_rate": 9.897099111961248e-05,
+      "loss": 0.46679277420043946,
+      "mean_token_accuracy": 0.8521067276597023,
+      "num_tokens": 18839822.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.4761562652885914,
+      "epoch": 0.1152,
+      "grad_norm": 0.07877718657255173,
+      "learning_rate": 9.886143054889961e-05,
+      "loss": 0.47925453186035155,
+      "mean_token_accuracy": 0.8481348522007466,
+      "num_tokens": 19378985.0,
+      "step": 360
+    },
+    {
+      "entropy": 0.4680658861994743,
+      "epoch": 0.1184,
+      "grad_norm": 0.08172708004713058,
+      "learning_rate": 9.874639560909117e-05,
+      "loss": 0.47161188125610354,
+      "mean_token_accuracy": 0.8510537907481194,
+      "num_tokens": 19929022.0,
+      "step": 370
+    },
+    {
+      "entropy": 0.4918907206505537,
+      "epoch": 0.1216,
+      "grad_norm": 0.08135205507278442,
+      "learning_rate": 9.862589918854747e-05,
+      "loss": 0.4943386554718018,
+      "mean_token_accuracy": 0.8442828319966793,
+      "num_tokens": 20467412.0,
+      "step": 380
+    },
+    {
+      "entropy": 0.4759133145213127,
+      "epoch": 0.1248,
+      "grad_norm": 0.07835420966148376,
+      "learning_rate": 9.849995478752567e-05,
+      "loss": 0.47806320190429685,
+      "mean_token_accuracy": 0.8508141845464706,
+      "num_tokens": 21023106.0,
+      "step": 390
+    },
+    {
+      "entropy": 0.47920923940837384,
+      "epoch": 0.128,
+      "grad_norm": 0.07775858789682388,
+      "learning_rate": 9.836857651666747e-05,
+      "loss": 0.48003373146057127,
+      "mean_token_accuracy": 0.8479285657405853,
+      "num_tokens": 21573240.0,
+      "step": 400
+    },
+    {
+      "entropy": 0.4857858780771494,
+      "epoch": 0.1312,
+      "grad_norm": 0.07792385667562485,
+      "learning_rate": 9.823177909541794e-05,
+      "loss": 0.48643798828125,
+      "mean_token_accuracy": 0.8469386555254459,
+      "num_tokens": 22113961.0,
+      "step": 410
+    },
+    {
+      "entropy": 0.4869717873632908,
+      "epoch": 0.1344,
+      "grad_norm": 0.08274804800748825,
+      "learning_rate": 9.808957785037657e-05,
+      "loss": 0.4867594242095947,
+      "mean_token_accuracy": 0.8460087358951569,
+      "num_tokens": 22658794.0,
+      "step": 420
+    },
+    {
+      "entropy": 0.47257179878652095,
+      "epoch": 0.1376,
+      "grad_norm": 0.079668328166008,
+      "learning_rate": 9.794198871358002e-05,
+      "loss": 0.4737967014312744,
+      "mean_token_accuracy": 0.850037157535553,
+      "num_tokens": 23196256.0,
+      "step": 430
+    },
+    {
+      "entropy": 0.48443259075284006,
+      "epoch": 0.1408,
+      "grad_norm": 0.08419299125671387,
+      "learning_rate": 9.778902822071703e-05,
+      "loss": 0.48652334213256837,
+      "mean_token_accuracy": 0.8473159313201905,
+      "num_tokens": 23720420.0,
+      "step": 440
+    },
+    {
+      "entropy": 0.49586297404021024,
+      "epoch": 0.144,
+      "grad_norm": 0.08213909715414047,
+      "learning_rate": 9.763071350927593e-05,
+      "loss": 0.49457316398620604,
+      "mean_token_accuracy": 0.8444145329296588,
+      "num_tokens": 24253978.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.47342996038496493,
+      "epoch": 0.1472,
+      "grad_norm": 0.08207234740257263,
+      "learning_rate": 9.74670623166245e-05,
+      "loss": 0.4776087760925293,
+      "mean_token_accuracy": 0.8502733767032623,
+      "num_tokens": 24792727.0,
+      "step": 460
+    },
+    {
+      "entropy": 0.4875059641897678,
+      "epoch": 0.1504,
+      "grad_norm": 0.08009803295135498,
+      "learning_rate": 9.72980929780227e-05,
+      "loss": 0.48871440887451173,
+      "mean_token_accuracy": 0.845378078520298,
+      "num_tokens": 25331203.0,
+      "step": 470
+    },
+    {
+      "entropy": 0.4694684144109488,
+      "epoch": 0.1536,
+      "grad_norm": 0.07450859248638153,
+      "learning_rate": 9.712382442456845e-05,
+      "loss": 0.4689479351043701,
+      "mean_token_accuracy": 0.8512169562280179,
+      "num_tokens": 25882792.0,
+      "step": 480
+    },
+    {
+      "entropy": 0.48093602769076826,
+      "epoch": 0.1568,
+      "grad_norm": 0.079342320561409,
+      "learning_rate": 9.694427618107654e-05,
+      "loss": 0.4797829151153564,
+      "mean_token_accuracy": 0.8494518615305424,
+      "num_tokens": 26430601.0,
+      "step": 490
+    },
+    {
+      "entropy": 0.4724384371191263,
+      "epoch": 0.16,
+      "grad_norm": 0.08262895047664642,
+      "learning_rate": 9.675946836389121e-05,
+      "loss": 0.47344017028808594,
+      "mean_token_accuracy": 0.8498825326561927,
+      "num_tokens": 26962264.0,
+      "step": 500
+    },
+    {
+      "entropy": 0.47667323239147663,
+      "epoch": 0.1632,
+      "grad_norm": 0.07582014054059982,
+      "learning_rate": 9.656942167863221e-05,
+      "loss": 0.47846212387084963,
+      "mean_token_accuracy": 0.8493047229945659,
+      "num_tokens": 27513219.0,
+      "step": 510
+    },
+    {
+      "entropy": 0.4824518483132124,
+      "epoch": 0.1664,
+      "grad_norm": 0.08134610205888748,
+      "learning_rate": 9.63741574178751e-05,
+      "loss": 0.48381881713867186,
+      "mean_token_accuracy": 0.8485548816621303,
+      "num_tokens": 28064670.0,
+      "step": 520
+    },
+    {
+      "entropy": 0.47467301450669763,
+      "epoch": 0.1696,
+      "grad_norm": 0.08428889513015747,
+      "learning_rate": 9.617369745876555e-05,
+      "loss": 0.47686920166015623,
+      "mean_token_accuracy": 0.8505464628338814,
+      "num_tokens": 28614544.0,
+      "step": 530
+    },
+    {
+      "entropy": 0.45858458913862704,
+      "epoch": 0.1728,
+      "grad_norm": 0.07831797003746033,
+      "learning_rate": 9.596806426056833e-05,
+      "loss": 0.4615936279296875,
+      "mean_token_accuracy": 0.8526844687759876,
+      "num_tokens": 29165163.0,
+      "step": 540
+    },
+    {
+      "entropy": 0.4882994510233402,
+      "epoch": 0.176,
+      "grad_norm": 0.08403731882572174,
+      "learning_rate": 9.575728086215092e-05,
+      "loss": 0.48913259506225587,
+      "mean_token_accuracy": 0.8460622608661652,
+      "num_tokens": 29688523.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.47602466829121115,
+      "epoch": 0.1792,
+      "grad_norm": 0.07488343864679337,
+      "learning_rate": 9.554137087940241e-05,
+      "loss": 0.4770374298095703,
+      "mean_token_accuracy": 0.8491039857268333,
+      "num_tokens": 30237453.0,
+      "step": 560
+    },
+    {
+      "entropy": 0.46900540217757225,
+      "epoch": 0.1824,
+      "grad_norm": 0.0737559124827385,
+      "learning_rate": 9.532035850258744e-05,
+      "loss": 0.4726743698120117,
+      "mean_token_accuracy": 0.8509468130767346,
+      "num_tokens": 30781920.0,
+      "step": 570
+    },
+    {
+      "entropy": 0.4697703827172518,
+      "epoch": 0.1856,
+      "grad_norm": 0.07812653481960297,
+      "learning_rate": 9.509426849363608e-05,
+      "loss": 0.47063488960266114,
+      "mean_token_accuracy": 0.8523916371166707,
+      "num_tokens": 31342162.0,
+      "step": 580
+    },
+    {
+      "entropy": 0.47539051938802,
+      "epoch": 0.1888,
+      "grad_norm": 0.07987186312675476,
+      "learning_rate": 9.486312618336943e-05,
+      "loss": 0.4764849662780762,
+      "mean_token_accuracy": 0.8499711640179157,
+      "num_tokens": 31889558.0,
+      "step": 590
+    },
+    {
+      "entropy": 0.4931900303810835,
+      "epoch": 0.192,
+      "grad_norm": 0.11574485898017883,
+      "learning_rate": 9.462695746866173e-05,
+      "loss": 0.4949483394622803,
+      "mean_token_accuracy": 0.8442160576581955,
+      "num_tokens": 32426475.0,
+      "step": 600
+    },
+    {
+      "entropy": 0.4768557969480753,
+      "epoch": 0.1952,
+      "grad_norm": 0.07476576417684555,
+      "learning_rate": 9.438578880953875e-05,
+      "loss": 0.4781337738037109,
+      "mean_token_accuracy": 0.8498996920883656,
+      "num_tokens": 32965358.0,
+      "step": 610
+    },
+    {
+      "entropy": 0.48749228455126287,
+      "epoch": 0.1984,
+      "grad_norm": 0.08077512681484222,
+      "learning_rate": 9.413964722621338e-05,
+      "loss": 0.4897915840148926,
+      "mean_token_accuracy": 0.8461120188236236,
+      "num_tokens": 33499187.0,
+      "step": 620
+    },
+    {
+      "entropy": 0.47319737523794175,
+      "epoch": 0.2016,
+      "grad_norm": 0.07910402119159698,
+      "learning_rate": 9.388856029605821e-05,
+      "loss": 0.47410268783569337,
+      "mean_token_accuracy": 0.8500978387892246,
+      "num_tokens": 34041503.0,
+      "step": 630
+    },
+    {
+      "entropy": 0.47187038622796534,
+      "epoch": 0.2048,
+      "grad_norm": 0.0774277001619339,
+      "learning_rate": 9.363255615051591e-05,
+      "loss": 0.4751906871795654,
+      "mean_token_accuracy": 0.8505054004490376,
+      "num_tokens": 34595252.0,
+      "step": 640
+    },
+    {
+      "entropy": 0.48014669977128505,
+      "epoch": 0.208,
+      "grad_norm": 0.07764683663845062,
+      "learning_rate": 9.337166347194732e-05,
+      "loss": 0.4829127311706543,
+      "mean_token_accuracy": 0.8502410404384136,
+      "num_tokens": 35141716.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.4769856847822666,
+      "epoch": 0.2112,
+      "grad_norm": 0.06824088096618652,
+      "learning_rate": 9.310591149041796e-05,
+      "loss": 0.47666058540344236,
+      "mean_token_accuracy": 0.8491735376417637,
+      "num_tokens": 35688010.0,
+      "step": 660
+    },
+    {
+      "entropy": 0.48730610609054564,
+      "epoch": 0.2144,
+      "grad_norm": 0.07797452807426453,
+      "learning_rate": 9.283532998042314e-05,
+      "loss": 0.48588948249816893,
+      "mean_token_accuracy": 0.8470434345304966,
+      "num_tokens": 36226585.0,
+      "step": 670
+    },
+    {
+      "entropy": 0.4712740655988455,
+      "epoch": 0.2176,
+      "grad_norm": 0.07640084624290466,
+      "learning_rate": 9.255994925755207e-05,
+      "loss": 0.4725786685943604,
+      "mean_token_accuracy": 0.8508462831377983,
+      "num_tokens": 36774498.0,
+      "step": 680
+    },
+    {
+      "entropy": 0.47151445485651494,
+      "epoch": 0.2208,
+      "grad_norm": 0.0767187550663948,
+      "learning_rate": 9.22798001750913e-05,
+      "loss": 0.4724010467529297,
+      "mean_token_accuracy": 0.8511710181832314,
+      "num_tokens": 37321521.0,
+      "step": 690
+    },
+    {
+      "entropy": 0.47285968586802485,
+      "epoch": 0.224,
+      "grad_norm": 0.08685708045959473,
+      "learning_rate": 9.199491412056805e-05,
+      "loss": 0.4756521224975586,
+      "mean_token_accuracy": 0.8499340698122978,
+      "num_tokens": 37869725.0,
+      "step": 700
+    },
+    {
+      "entropy": 0.47530038133263586,
+      "epoch": 0.2272,
+      "grad_norm": 0.08178319036960602,
+      "learning_rate": 9.170532301223342e-05,
+      "loss": 0.4760128974914551,
+      "mean_token_accuracy": 0.8494046285748482,
+      "num_tokens": 38411812.0,
+      "step": 710
+    },
+    {
+      "entropy": 0.4898712053894997,
+      "epoch": 0.2304,
+      "grad_norm": 0.0820806473493576,
+      "learning_rate": 9.141105929548654e-05,
+      "loss": 0.49225292205810545,
+      "mean_token_accuracy": 0.8461540147662163,
+      "num_tokens": 38957097.0,
+      "step": 720
+    },
+    {
+      "entropy": 0.49410275742411613,
+      "epoch": 0.2336,
+      "grad_norm": 0.0803661122918129,
+      "learning_rate": 9.111215593923924e-05,
+      "loss": 0.49660024642944334,
+      "mean_token_accuracy": 0.844361025840044,
+      "num_tokens": 39484601.0,
+      "step": 730
+    },
+    {
+      "entropy": 0.4832836758345366,
+      "epoch": 0.2368,
+      "grad_norm": 0.0747658908367157,
+      "learning_rate": 9.080864643222238e-05,
+      "loss": 0.48782711029052733,
+      "mean_token_accuracy": 0.8474400013685226,
+      "num_tokens": 40024350.0,
+      "step": 740
+    },
+    {
+      "entropy": 0.47210120856761933,
+      "epoch": 0.24,
+      "grad_norm": 0.08369333297014236,
+      "learning_rate": 9.050056477923372e-05,
+      "loss": 0.4736424446105957,
+      "mean_token_accuracy": 0.8512022905051708,
+      "num_tokens": 40573607.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.4751798383891582,
+      "epoch": 0.2432,
+      "grad_norm": 0.07527061551809311,
+      "learning_rate": 9.018794549732819e-05,
+      "loss": 0.47675304412841796,
+      "mean_token_accuracy": 0.8492536380887031,
+      "num_tokens": 41128378.0,
+      "step": 760
+    },
+    {
+      "entropy": 0.4596736691892147,
+      "epoch": 0.2464,
+      "grad_norm": 0.07552742958068848,
+      "learning_rate": 8.987082361195047e-05,
+      "loss": 0.4612094879150391,
+      "mean_token_accuracy": 0.85373764783144,
+      "num_tokens": 41676611.0,
+      "step": 770
+    },
+    {
+      "entropy": 0.4881919324398041,
+      "epoch": 0.2496,
+      "grad_norm": 0.07089073956012726,
+      "learning_rate": 8.954923465301098e-05,
+      "loss": 0.48726844787597656,
+      "mean_token_accuracy": 0.8471492856740952,
+      "num_tokens": 42217221.0,
+      "step": 780
+    },
+    {
+      "entropy": 0.48793355375528336,
+      "epoch": 0.2528,
+      "grad_norm": 0.07903842628002167,
+      "learning_rate": 8.922321465090504e-05,
+      "loss": 0.4911531448364258,
+      "mean_token_accuracy": 0.8451745077967644,
+      "num_tokens": 42757663.0,
+      "step": 790
+    },
+    {
+      "entropy": 0.4636719670146704,
+      "epoch": 0.256,
+      "grad_norm": 0.08099409937858582,
+      "learning_rate": 8.889280013247608e-05,
+      "loss": 0.4613630771636963,
+      "mean_token_accuracy": 0.8535272777080536,
+      "num_tokens": 43303288.0,
+      "step": 800
+    },
+    {
+      "entropy": 0.4685452912002802,
+      "epoch": 0.2592,
+      "grad_norm": 0.07842060178518295,
+      "learning_rate": 8.855802811692326e-05,
+      "loss": 0.47121448516845704,
+      "mean_token_accuracy": 0.852169182896614,
+      "num_tokens": 43841558.0,
+      "step": 810
+    },
+    {
+      "entropy": 0.4764429606497288,
+      "epoch": 0.2624,
+      "grad_norm": 0.07855252176523209,
+      "learning_rate": 8.821893611165387e-05,
+      "loss": 0.4794772148132324,
+      "mean_token_accuracy": 0.8487229481339454,
+      "num_tokens": 44386409.0,
+      "step": 820
+    },
+    {
+      "entropy": 0.4749792542308569,
+      "epoch": 0.2656,
+      "grad_norm": 0.08472711592912674,
+      "learning_rate": 8.787556210808101e-05,
+      "loss": 0.4772174835205078,
+      "mean_token_accuracy": 0.8503303691744805,
+      "num_tokens": 44918678.0,
+      "step": 830
+    },
+    {
+      "entropy": 0.4767191309481859,
+      "epoch": 0.2688,
+      "grad_norm": 0.08218918740749359,
+      "learning_rate": 8.752794457736719e-05,
+      "loss": 0.47406606674194335,
+      "mean_token_accuracy": 0.8507293291389942,
+      "num_tokens": 45473631.0,
+      "step": 840
+    },
+    {
+      "entropy": 0.47092341966927054,
+      "epoch": 0.272,
+      "grad_norm": 0.085779570043087,
+      "learning_rate": 8.717612246611396e-05,
+      "loss": 0.47501192092895506,
+      "mean_token_accuracy": 0.8503929585218429,
+      "num_tokens": 46012094.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.4835963238030672,
+      "epoch": 0.2752,
+      "grad_norm": 0.08534952253103256,
+      "learning_rate": 8.682013519199841e-05,
+      "loss": 0.4834300994873047,
+      "mean_token_accuracy": 0.8477169685065746,
+      "num_tokens": 46553265.0,
+      "step": 860
+    },
+    {
+      "entropy": 0.4751033581793308,
+      "epoch": 0.2784,
+      "grad_norm": 0.07841655611991882,
+      "learning_rate": 8.646002263935695e-05,
+      "loss": 0.47674098014831545,
+      "mean_token_accuracy": 0.8485349163413047,
+      "num_tokens": 47087343.0,
+      "step": 870
+    },
+    {
+      "entropy": 0.4564079999923706,
+      "epoch": 0.2816,
+      "grad_norm": 0.07997316122055054,
+      "learning_rate": 8.609582515471663e-05,
+      "loss": 0.45759096145629885,
+      "mean_token_accuracy": 0.8546725757420063,
+      "num_tokens": 47633002.0,
+      "step": 880
+    },
+    {
+      "entropy": 0.4866001050919294,
+      "epoch": 0.2848,
+      "grad_norm": 0.08223622292280197,
+      "learning_rate": 8.572758354227476e-05,
+      "loss": 0.4863016128540039,
+      "mean_token_accuracy": 0.8472935289144516,
+      "num_tokens": 48180335.0,
+      "step": 890
+    },
+    {
+      "entropy": 0.46846042200922966,
+      "epoch": 0.288,
+      "grad_norm": 0.08052714169025421,
+      "learning_rate": 8.535533905932738e-05,
+      "loss": 0.47049579620361326,
+      "mean_token_accuracy": 0.8506691426038742,
+      "num_tokens": 48728626.0,
+      "step": 900
+    },
+    {
+      "entropy": 0.46195199266076087,
+      "epoch": 0.2912,
+      "grad_norm": 0.07734925299882889,
+      "learning_rate": 8.49791334116467e-05,
+      "loss": 0.4631977081298828,
+      "mean_token_accuracy": 0.8535349547863007,
+      "num_tokens": 49263697.0,
+      "step": 910
+    },
+    {
+      "entropy": 0.4654418841004372,
+      "epoch": 0.2944,
+      "grad_norm": 0.07527711987495422,
+      "learning_rate": 8.459900874880849e-05,
+      "loss": 0.4663339614868164,
+      "mean_token_accuracy": 0.8517521195113659,
+      "num_tokens": 49802308.0,
+      "step": 920
+    },
+    {
+      "entropy": 0.4810977768152952,
+      "epoch": 0.2976,
+      "grad_norm": 0.07790960371494293,
+      "learning_rate": 8.421500765946983e-05,
+      "loss": 0.48372488021850585,
+      "mean_token_accuracy": 0.8469045363366604,
+      "num_tokens": 50327998.0,
+      "step": 930
+    },
+    {
+      "entropy": 0.4687338896095753,
+      "epoch": 0.3008,
+      "grad_norm": 0.0822044163942337,
+      "learning_rate": 8.382717316659731e-05,
+      "loss": 0.4695611000061035,
+      "mean_token_accuracy": 0.8508451916277409,
+      "num_tokens": 50865818.0,
+      "step": 940
+    },
+    {
+      "entropy": 0.4692784225568175,
+      "epoch": 0.304,
+      "grad_norm": 0.07648163288831711,
+      "learning_rate": 8.343554872264697e-05,
+      "loss": 0.4679527759552002,
+      "mean_token_accuracy": 0.8531959399580955,
+      "num_tokens": 51408778.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.4685392875224352,
+      "epoch": 0.3072,
+      "grad_norm": 0.08264504373073578,
+      "learning_rate": 8.304017820469582e-05,
+      "loss": 0.47391357421875,
+      "mean_token_accuracy": 0.851045648008585,
+      "num_tokens": 51947066.0,
+      "step": 960
+    },
+    {
+      "entropy": 0.4727648008614779,
+      "epoch": 0.3104,
+      "grad_norm": 0.07236472517251968,
+      "learning_rate": 8.264110590952609e-05,
+      "loss": 0.47615623474121094,
+      "mean_token_accuracy": 0.8499816425144673,
+      "num_tokens": 52491591.0,
+      "step": 970
+    },
+    {
+      "entropy": 0.47436089888215066,
+      "epoch": 0.3136,
+      "grad_norm": 0.07411123812198639,
+      "learning_rate": 8.223837654866202e-05,
+      "loss": 0.475218391418457,
+      "mean_token_accuracy": 0.8491612285375595,
+      "num_tokens": 53028176.0,
+      "step": 980
+    },
+    {
+      "entropy": 0.48128985017538073,
+      "epoch": 0.3168,
+      "grad_norm": 0.07564353942871094,
+      "learning_rate": 8.183203524336068e-05,
+      "loss": 0.48183107376098633,
+      "mean_token_accuracy": 0.848487002402544,
+      "num_tokens": 53568117.0,
+      "step": 990
+    },
+    {
+      "entropy": 0.4761129651218653,
+      "epoch": 0.32,
+      "grad_norm": 0.07757043093442917,
+      "learning_rate": 8.142212751955649e-05,
+      "loss": 0.4786111831665039,
+      "mean_token_accuracy": 0.8494840450584888,
+      "num_tokens": 54116640.0,
+      "step": 1000
+    },
+    {
+      "entropy": 0.46917025968432424,
+      "epoch": 0.3232,
+      "grad_norm": 0.07940242439508438,
+      "learning_rate": 8.100869930276064e-05,
+      "loss": 0.4714414119720459,
+      "mean_token_accuracy": 0.8503246434032917,
+      "num_tokens": 54653051.0,
+      "step": 1010
+    },
+    {
+      "entropy": 0.4772383697330952,
+      "epoch": 0.3264,
+      "grad_norm": 0.07760357111692429,
+      "learning_rate": 8.059179691291565e-05,
+      "loss": 0.47988409996032716,
+      "mean_token_accuracy": 0.8488184005022049,
+      "num_tokens": 55189061.0,
+      "step": 1020
+    },
+    {
+      "entropy": 0.49717654250562193,
+      "epoch": 0.3296,
+      "grad_norm": 0.08256684243679047,
+      "learning_rate": 8.017146705920568e-05,
+      "loss": 0.49762907028198244,
+      "mean_token_accuracy": 0.8431043148040771,
+      "num_tokens": 55717461.0,
+      "step": 1030
+    },
+    {
+      "entropy": 0.44942800477147105,
+      "epoch": 0.3328,
+      "grad_norm": 0.07848676294088364,
+      "learning_rate": 7.974775683482338e-05,
+      "loss": 0.4503443717956543,
+      "mean_token_accuracy": 0.8557130552828311,
+      "num_tokens": 56269615.0,
+      "step": 1040
+    },
+    {
+      "entropy": 0.4714782789349556,
+      "epoch": 0.336,
+      "grad_norm": 0.0803825706243515,
+      "learning_rate": 7.932071371169354e-05,
+      "loss": 0.47602090835571287,
+      "mean_token_accuracy": 0.8500577941536903,
+      "num_tokens": 56811939.0,
+      "step": 1050
+    },
+    {
+      "entropy": 0.4705597136169672,
+      "epoch": 0.3392,
+      "grad_norm": 0.08659618347883224,
+      "learning_rate": 7.889038553515451e-05,
+      "loss": 0.470610523223877,
+      "mean_token_accuracy": 0.8512009240686893,
+      "num_tokens": 57358576.0,
+      "step": 1060
+    },
+    {
+      "entropy": 0.4670447796583176,
+      "epoch": 0.3424,
+      "grad_norm": 0.07773701101541519,
+      "learning_rate": 7.845682051859757e-05,
+      "loss": 0.4706146240234375,
+      "mean_token_accuracy": 0.8512641072273255,
+      "num_tokens": 57900469.0,
+      "step": 1070
+    },
+    {
+      "entropy": 0.47142090164124967,
+      "epoch": 0.3456,
+      "grad_norm": 0.07671911269426346,
+      "learning_rate": 7.802006723806527e-05,
+      "loss": 0.4736938953399658,
+      "mean_token_accuracy": 0.8514165125787259,
+      "num_tokens": 58453070.0,
+      "step": 1080
+    },
+    {
+      "entropy": 0.47359533831477163,
+      "epoch": 0.3488,
+      "grad_norm": 0.08079513162374496,
+      "learning_rate": 7.758017462680892e-05,
+      "loss": 0.4731889724731445,
+      "mean_token_accuracy": 0.8501715049147606,
+      "num_tokens": 58997842.0,
+      "step": 1090
+    },
+    {
+      "entropy": 0.4660038728266954,
+      "epoch": 0.352,
+      "grad_norm": 0.076938696205616,
+      "learning_rate": 7.713719196980624e-05,
+      "loss": 0.4702435493469238,
+      "mean_token_accuracy": 0.8505289196968079,
+      "num_tokens": 59534354.0,
+      "step": 1100
+    },
+    {
+      "entropy": 0.46711551398038864,
+      "epoch": 0.3552,
+      "grad_norm": 0.07634937763214111,
+      "learning_rate": 7.669116889823955e-05,
+      "loss": 0.4659992218017578,
+      "mean_token_accuracy": 0.8514557547867299,
+      "num_tokens": 60067312.0,
+      "step": 1110
+    },
+    {
+      "entropy": 0.4589589949697256,
+      "epoch": 0.3584,
+      "grad_norm": 0.08179903030395508,
+      "learning_rate": 7.62421553839351e-05,
+      "loss": 0.4589775562286377,
+      "mean_token_accuracy": 0.8547109857201576,
+      "num_tokens": 60611327.0,
+      "step": 1120
+    },
+    {
+      "entropy": 0.4735019005835056,
+      "epoch": 0.3616,
+      "grad_norm": 0.0791383907198906,
+      "learning_rate": 7.579020173376438e-05,
+      "loss": 0.4762627601623535,
+      "mean_token_accuracy": 0.8496688216924667,
+      "num_tokens": 61152192.0,
+      "step": 1130
+    },
+    {
+      "entropy": 0.4838758923113346,
+      "epoch": 0.3648,
+      "grad_norm": 0.08668875694274902,
+      "learning_rate": 7.533535858400773e-05,
+      "loss": 0.483915901184082,
+      "mean_token_accuracy": 0.8477108284831048,
+      "num_tokens": 61679786.0,
+      "step": 1140
+    },
+    {
+      "entropy": 0.4788258831948042,
+      "epoch": 0.368,
+      "grad_norm": 0.1338026523590088,
+      "learning_rate": 7.487767689468114e-05,
+      "loss": 0.4759991645812988,
+      "mean_token_accuracy": 0.8496520824730396,
+      "num_tokens": 62235403.0,
+      "step": 1150
+    },
+    {
+      "entropy": 0.4697259794920683,
+      "epoch": 0.3712,
+      "grad_norm": 0.07391800731420517,
+      "learning_rate": 7.441720794382678e-05,
+      "loss": 0.47245054244995116,
+      "mean_token_accuracy": 0.8507399395108223,
+      "num_tokens": 62775337.0,
+      "step": 1160
+    },
+    {
+      "entropy": 0.46641252711415293,
+      "epoch": 0.3744,
+      "grad_norm": 0.08541858196258545,
+      "learning_rate": 7.39540033217678e-05,
+      "loss": 0.46926565170288087,
+      "mean_token_accuracy": 0.8515971966087819,
+      "num_tokens": 63313974.0,
+      "step": 1170
+    },
+    {
+      "entropy": 0.46677456349134444,
+      "epoch": 0.3776,
+      "grad_norm": 0.07809534668922424,
+      "learning_rate": 7.34881149253284e-05,
+      "loss": 0.4635956287384033,
+      "mean_token_accuracy": 0.8533280894160271,
+      "num_tokens": 63860582.0,
+      "step": 1180
+    },
+    {
+      "entropy": 0.4531945262104273,
+      "epoch": 0.3808,
+      "grad_norm": 0.07714966684579849,
+      "learning_rate": 7.301959495201914e-05,
+      "loss": 0.4573531150817871,
+      "mean_token_accuracy": 0.8545272573828697,
+      "num_tokens": 64393449.0,
+      "step": 1190
+    },
+    {
+      "entropy": 0.47963859550654886,
+      "epoch": 0.384,
+      "grad_norm": 0.08511944115161896,
+      "learning_rate": 7.254849589418899e-05,
+      "loss": 0.48232240676879884,
+      "mean_token_accuracy": 0.847790089994669,
+      "num_tokens": 64929559.0,
+      "step": 1200
+    },
+    {
+      "entropy": 0.4696847841143608,
+      "epoch": 0.3872,
+      "grad_norm": 0.0821264460682869,
+      "learning_rate": 7.207487053314408e-05,
+      "loss": 0.4743045330047607,
+      "mean_token_accuracy": 0.8509298220276833,
+      "num_tokens": 65472324.0,
+      "step": 1210
+    },
+    {
+      "entropy": 0.4760958280414343,
+      "epoch": 0.3904,
+      "grad_norm": 0.07894673943519592,
+      "learning_rate": 7.159877193323412e-05,
+      "loss": 0.47612338066101073,
+      "mean_token_accuracy": 0.8502611100673676,
+      "num_tokens": 66010367.0,
+      "step": 1220
+    },
+    {
+      "entropy": 0.4641199756413698,
+      "epoch": 0.3936,
+      "grad_norm": 0.08127527683973312,
+      "learning_rate": 7.11202534359072e-05,
+      "loss": 0.464811897277832,
+      "mean_token_accuracy": 0.8529750660061837,
+      "num_tokens": 66555852.0,
+      "step": 1230
+    },
+    {
+      "entropy": 0.4783476207405329,
+      "epoch": 0.3968,
+      "grad_norm": 0.08005862683057785,
+      "learning_rate": 7.063936865373353e-05,
+      "loss": 0.47718076705932616,
+      "mean_token_accuracy": 0.8489580906927585,
+      "num_tokens": 67098708.0,
+      "step": 1240
+    },
+    {
+      "entropy": 0.44940643701702354,
+      "epoch": 0.4,
+      "grad_norm": 0.07871269434690475,
+      "learning_rate": 7.015617146439863e-05,
+      "loss": 0.4523442268371582,
+      "mean_token_accuracy": 0.8564409531652928,
+      "num_tokens": 67645082.0,
+      "step": 1250
+    },
+    {
+      "entropy": 0.48388259410858153,
+      "epoch": 0.4032,
+      "grad_norm": 0.08175542205572128,
+      "learning_rate": 6.967071600466699e-05,
+      "loss": 0.48316283226013185,
+      "mean_token_accuracy": 0.8477910667657852,
+      "num_tokens": 68183965.0,
+      "step": 1260
+    },
+    {
+      "entropy": 0.4663986641913652,
+      "epoch": 0.4064,
+      "grad_norm": 0.08386580646038055,
+      "learning_rate": 6.918305666431678e-05,
+      "loss": 0.4686122894287109,
+      "mean_token_accuracy": 0.8507393769919872,
+      "num_tokens": 68715381.0,
+      "step": 1270
+    },
+    {
+      "entropy": 0.47266134805977345,
+      "epoch": 0.4096,
+      "grad_norm": 0.07826567441225052,
+      "learning_rate": 6.869324808004591e-05,
+      "loss": 0.4709297180175781,
+      "mean_token_accuracy": 0.8507488630712032,
+      "num_tokens": 69261931.0,
+      "step": 1280
+    },
+    {
+      "entropy": 0.4749354414641857,
+      "epoch": 0.4128,
+      "grad_norm": 0.08114854991436005,
+      "learning_rate": 6.820134512935076e-05,
+      "loss": 0.4813858509063721,
+      "mean_token_accuracy": 0.8486299641430378,
+      "num_tokens": 69798702.0,
+      "step": 1290
+    },
+    {
+      "entropy": 0.48080884739756585,
+      "epoch": 0.416,
+      "grad_norm": 0.08293881267309189,
+      "learning_rate": 6.770740292437765e-05,
+      "loss": 0.4830974578857422,
+      "mean_token_accuracy": 0.8486068546772003,
+      "num_tokens": 70335520.0,
+      "step": 1300
+    },
+    {
+      "entropy": 0.445353127643466,
+      "epoch": 0.4192,
+      "grad_norm": 0.07848215103149414,
+      "learning_rate": 6.721147680574825e-05,
+      "loss": 0.443513298034668,
+      "mean_token_accuracy": 0.8587793231010437,
+      "num_tokens": 70893408.0,
+      "step": 1310
+    },
+    {
+      "entropy": 0.46828467808663843,
+      "epoch": 0.4224,
+      "grad_norm": 0.07849520444869995,
+      "learning_rate": 6.671362233635925e-05,
+      "loss": 0.47074165344238283,
+      "mean_token_accuracy": 0.8519576385617256,
+      "num_tokens": 71432511.0,
+      "step": 1320
+    },
+    {
+      "entropy": 0.46790354028344155,
+      "epoch": 0.4256,
+      "grad_norm": 0.07682167738676071,
+      "learning_rate": 6.621389529515718e-05,
+      "loss": 0.46987133026123046,
+      "mean_token_accuracy": 0.8507593631744385,
+      "num_tokens": 71972931.0,
+      "step": 1330
+    },
+    {
+      "entropy": 0.47380633652210236,
+      "epoch": 0.4288,
+      "grad_norm": 0.07671021670103073,
+      "learning_rate": 6.571235167088897e-05,
+      "loss": 0.4743668079376221,
+      "mean_token_accuracy": 0.849990089982748,
+      "num_tokens": 72509252.0,
+      "step": 1340
+    },
+    {
+      "entropy": 0.4695499274879694,
+      "epoch": 0.432,
+      "grad_norm": 0.07734425365924835,
+      "learning_rate": 6.520904765582909e-05,
+      "loss": 0.47025532722473146,
+      "mean_token_accuracy": 0.8512202836573124,
+      "num_tokens": 73051252.0,
+      "step": 1350
+    },
+    {
+      "entropy": 0.45717273019254207,
+      "epoch": 0.4352,
+      "grad_norm": 0.07840635627508163,
+      "learning_rate": 6.470403963948386e-05,
+      "loss": 0.45947952270507814,
+      "mean_token_accuracy": 0.8544160060584545,
+      "num_tokens": 73584956.0,
+      "step": 1360
+    },
+    {
+      "entropy": 0.47162831015884876,
+      "epoch": 0.4384,
+      "grad_norm": 0.08281369507312775,
+      "learning_rate": 6.419738420227357e-05,
+      "loss": 0.4729193687438965,
+      "mean_token_accuracy": 0.8503495909273624,
+      "num_tokens": 74115323.0,
+      "step": 1370
+    },
+    {
+      "entropy": 0.46119370497763157,
+      "epoch": 0.4416,
+      "grad_norm": 0.0766388550400734,
+      "learning_rate": 6.368913810919339e-05,
+      "loss": 0.46128005981445314,
+      "mean_token_accuracy": 0.8537597492337227,
+      "num_tokens": 74672444.0,
+      "step": 1380
+    },
+    {
+      "entropy": 0.47553979940712454,
+      "epoch": 0.4448,
+      "grad_norm": 0.07618532329797745,
+      "learning_rate": 6.317935830345338e-05,
+      "loss": 0.47728590965270995,
+      "mean_token_accuracy": 0.8493086688220501,
+      "num_tokens": 75205841.0,
+      "step": 1390
+    },
+    {
+      "entropy": 0.47228156924247744,
+      "epoch": 0.448,
+      "grad_norm": 0.08119732141494751,
+      "learning_rate": 6.266810190009877e-05,
+      "loss": 0.4703529357910156,
+      "mean_token_accuracy": 0.851146450638771,
+      "num_tokens": 75752180.0,
+      "step": 1400
+    },
+    {
+      "entropy": 0.4720066156238317,
+      "epoch": 0.4512,
+      "grad_norm": 0.0890950933098793,
+      "learning_rate": 6.21554261796107e-05,
+      "loss": 0.4751739501953125,
+      "mean_token_accuracy": 0.851177292317152,
+      "num_tokens": 76289695.0,
+      "step": 1410
+    },
+    {
+      "entropy": 0.46984184756875036,
+      "epoch": 0.4544,
+      "grad_norm": 0.0826389417052269,
+      "learning_rate": 6.164138858148886e-05,
+      "loss": 0.47519564628601074,
+      "mean_token_accuracy": 0.8512011744081974,
+      "num_tokens": 76825284.0,
+      "step": 1420
+    },
+    {
+      "entropy": 0.4752038478851318,
+      "epoch": 0.4576,
+      "grad_norm": 0.08134012669324875,
+      "learning_rate": 6.112604669781572e-05,
+      "loss": 0.475037670135498,
+      "mean_token_accuracy": 0.8502763018012047,
+      "num_tokens": 77358614.0,
+      "step": 1430
+    },
+    {
+      "entropy": 0.4627603005617857,
+      "epoch": 0.4608,
+      "grad_norm": 0.08518964052200317,
+      "learning_rate": 6.060945826680425e-05,
+      "loss": 0.46430625915527346,
+      "mean_token_accuracy": 0.8529912106692791,
+      "num_tokens": 77902786.0,
+      "step": 1440
+    },
+    {
+      "entropy": 0.49012349732220173,
+      "epoch": 0.464,
+      "grad_norm": 0.07677725702524185,
+      "learning_rate": 6.009168116632887e-05,
+      "loss": 0.4902459144592285,
+      "mean_token_accuracy": 0.8460152193903923,
+      "num_tokens": 78437651.0,
+      "step": 1450
+    },
+    {
+      "entropy": 0.4582429379224777,
+      "epoch": 0.4672,
+      "grad_norm": 0.07936195284128189,
+      "learning_rate": 5.957277340744094e-05,
+      "loss": 0.46192111968994143,
+      "mean_token_accuracy": 0.8537801176309585,
+      "num_tokens": 78994453.0,
+      "step": 1460
+    },
+    {
+      "entropy": 0.46386151425540445,
+      "epoch": 0.4704,
+      "grad_norm": 0.07885941863059998,
+      "learning_rate": 5.905279312786928e-05,
+      "loss": 0.46464266777038576,
+      "mean_token_accuracy": 0.8536564350128174,
+      "num_tokens": 79540754.0,
+      "step": 1470
+    },
+    {
+      "entropy": 0.48290870524942875,
+      "epoch": 0.4736,
+      "grad_norm": 0.0816439688205719,
+      "learning_rate": 5.853179858550647e-05,
+      "loss": 0.4833097457885742,
+      "mean_token_accuracy": 0.848555526137352,
+      "num_tokens": 80074234.0,
+      "step": 1480
+    },
+    {
+      "entropy": 0.46357379723340275,
+      "epoch": 0.4768,
+      "grad_norm": 0.0767681673169136,
+      "learning_rate": 5.800984815188176e-05,
+      "loss": 0.46472816467285155,
+      "mean_token_accuracy": 0.8527257896959781,
+      "num_tokens": 80621881.0,
+      "step": 1490
+    },
+    {
+      "entropy": 0.47054843716323375,
+      "epoch": 0.48,
+      "grad_norm": 0.08159372210502625,
+      "learning_rate": 5.748700030562115e-05,
+      "loss": 0.471923828125,
+      "mean_token_accuracy": 0.8509253062307834,
+      "num_tokens": 81155872.0,
+      "step": 1500
+    },
+    {
+      "entropy": 0.4759920839220285,
+      "epoch": 0.4832,
+      "grad_norm": 0.08094548434019089,
+      "learning_rate": 5.696331362589556e-05,
+      "loss": 0.47703728675842283,
+      "mean_token_accuracy": 0.8494079612195492,
+      "num_tokens": 81703773.0,
+      "step": 1510
+    },
+    {
+      "entropy": 0.4699292115867138,
+      "epoch": 0.4864,
+      "grad_norm": 0.07761377841234207,
+      "learning_rate": 5.643884678585768e-05,
+      "loss": 0.4721334934234619,
+      "mean_token_accuracy": 0.8508599974215031,
+      "num_tokens": 82234520.0,
+      "step": 1520
+    },
+    {
+      "entropy": 0.4649160888046026,
+      "epoch": 0.4896,
+      "grad_norm": 0.07809717953205109,
+      "learning_rate": 5.5913658546068295e-05,
+      "loss": 0.4664757251739502,
+      "mean_token_accuracy": 0.8522942341864109,
+      "num_tokens": 82767650.0,
+      "step": 1530
+    },
+    {
+      "entropy": 0.4619267268106341,
+      "epoch": 0.4928,
+      "grad_norm": 0.07818735390901566,
+      "learning_rate": 5.538780774791289e-05,
+      "loss": 0.4628866672515869,
+      "mean_token_accuracy": 0.8537545248866081,
+      "num_tokens": 83315969.0,
+      "step": 1540
+    },
+    {
+      "entropy": 0.47179987765848636,
+      "epoch": 0.496,
+      "grad_norm": 0.08222338557243347,
+      "learning_rate": 5.486135330700906e-05,
+      "loss": 0.4757401943206787,
+      "mean_token_accuracy": 0.8511043928563595,
+      "num_tokens": 83860076.0,
+      "step": 1550
+    },
+    {
+      "entropy": 0.4809089325368404,
+      "epoch": 0.4992,
+      "grad_norm": 0.0822545662522316,
+      "learning_rate": 5.4334354206605776e-05,
+      "loss": 0.47745490074157715,
+      "mean_token_accuracy": 0.848536130785942,
+      "num_tokens": 84409148.0,
+      "step": 1560
+    },
+    {
+      "entropy": 0.4698262229561806,
+      "epoch": 0.5024,
+      "grad_norm": 0.07563979923725128,
+      "learning_rate": 5.380686949097485e-05,
+      "loss": 0.47144665718078616,
+      "mean_token_accuracy": 0.850661239027977,
+      "num_tokens": 84952704.0,
+      "step": 1570
+    },
+    {
+      "entropy": 0.4649834394454956,
+      "epoch": 0.5056,
+      "grad_norm": 0.0799219161272049,
+      "learning_rate": 5.327895825879586e-05,
+      "loss": 0.4677272796630859,
+      "mean_token_accuracy": 0.852253969758749,
+      "num_tokens": 85492340.0,
+      "step": 1580
+    },
+    {
+      "entropy": 0.45752881802618506,
+      "epoch": 0.5088,
+      "grad_norm": 0.08413282781839371,
+      "learning_rate": 5.2750679656534653e-05,
+      "loss": 0.4573859214782715,
+      "mean_token_accuracy": 0.8535019971430302,
+      "num_tokens": 86048793.0,
+      "step": 1590
+    },
+    {
+      "entropy": 0.4634219191968441,
+      "epoch": 0.512,
+      "grad_norm": 0.07937274128198624,
+      "learning_rate": 5.2222092871816766e-05,
+      "loss": 0.4633816719055176,
+      "mean_token_accuracy": 0.8537441596388817,
+      "num_tokens": 86597585.0,
+      "step": 1600
+    },
+    {
+      "entropy": 0.4644321594387293,
+      "epoch": 0.5152,
+      "grad_norm": 0.08164449781179428,
+      "learning_rate": 5.169325712679608e-05,
+      "loss": 0.46547470092773435,
+      "mean_token_accuracy": 0.8514500871300698,
+      "num_tokens": 87135884.0,
+      "step": 1610
+    },
+    {
+      "entropy": 0.4709007706493139,
+      "epoch": 0.5184,
+      "grad_norm": 0.08022154122591019,
+      "learning_rate": 5.116423167151962e-05,
+      "loss": 0.4748415470123291,
+      "mean_token_accuracy": 0.8505366109311581,
+      "num_tokens": 87672565.0,
+      "step": 1620
+    },
+    {
+      "entropy": 0.46826594732701776,
+      "epoch": 0.5216,
+      "grad_norm": 0.07857371866703033,
+      "learning_rate": 5.0635075777289377e-05,
+      "loss": 0.4679561138153076,
+      "mean_token_accuracy": 0.852530375123024,
+      "num_tokens": 88210140.0,
+      "step": 1630
+    },
+    {
+      "entropy": 0.4680090047419071,
+      "epoch": 0.5248,
+      "grad_norm": 0.07805909961462021,
+      "learning_rate": 5.010584873002143e-05,
+      "loss": 0.4689174175262451,
+      "mean_token_accuracy": 0.8522804908454418,
+      "num_tokens": 88763982.0,
+      "step": 1640
+    },
+    {
+      "entropy": 0.4906394749879837,
+      "epoch": 0.528,
+      "grad_norm": 0.07526839524507523,
+      "learning_rate": 4.957660982360387e-05,
+      "loss": 0.487869930267334,
+      "mean_token_accuracy": 0.8450503595173359,
+      "num_tokens": 89299205.0,
+      "step": 1650
+    },
+    {
+      "entropy": 0.48258905243128536,
+      "epoch": 0.5312,
+      "grad_norm": 0.07946979254484177,
+      "learning_rate": 4.904741835325338e-05,
+      "loss": 0.48380413055419924,
+      "mean_token_accuracy": 0.8472427576780319,
+      "num_tokens": 89831095.0,
+      "step": 1660
+    },
+    {
+      "entropy": 0.46592932008206844,
+      "epoch": 0.5344,
+      "grad_norm": 0.08615154772996902,
+      "learning_rate": 4.851833360887201e-05,
+      "loss": 0.46813020706176756,
+      "mean_token_accuracy": 0.8520313084125519,
+      "num_tokens": 90365573.0,
+      "step": 1670
+    },
+    {
+      "entropy": 0.47134263552725314,
+      "epoch": 0.5376,
+      "grad_norm": 0.08925487846136093,
+      "learning_rate": 4.798941486840438e-05,
+      "loss": 0.4732311248779297,
+      "mean_token_accuracy": 0.8519421115517616,
+      "num_tokens": 90905209.0,
+      "step": 1680
+    },
+    {
+      "entropy": 0.4612633462995291,
+      "epoch": 0.5408,
+      "grad_norm": 0.07562625408172607,
+      "learning_rate": 4.746072139119622e-05,
+      "loss": 0.46331634521484377,
+      "mean_token_accuracy": 0.8531689427793026,
+      "num_tokens": 91452004.0,
+      "step": 1690
+    },
+    {
+      "entropy": 0.4733666587620974,
+      "epoch": 0.544,
+      "grad_norm": 0.07982128858566284,
+      "learning_rate": 4.69323124113551e-05,
+      "loss": 0.4750804901123047,
+      "mean_token_accuracy": 0.849866820126772,
+      "num_tokens": 91990495.0,
+      "step": 1700
+    },
+    {
+      "entropy": 0.46132506281137464,
+      "epoch": 0.5472,
+      "grad_norm": 0.07697301357984543,
+      "learning_rate": 4.6404247131113884e-05,
+      "loss": 0.45983171463012695,
+      "mean_token_accuracy": 0.8535612806677818,
+      "num_tokens": 92531009.0,
+      "step": 1710
+    },
+    {
+      "entropy": 0.4602090664207935,
+      "epoch": 0.5504,
+      "grad_norm": 0.08081653714179993,
+      "learning_rate": 4.587658471419778e-05,
+      "loss": 0.46139278411865237,
+      "mean_token_accuracy": 0.8547950901091099,
+      "num_tokens": 93078503.0,
+      "step": 1720
+    },
+    {
+      "entropy": 0.46119733825325965,
+      "epoch": 0.5536,
+      "grad_norm": 0.07525499165058136,
+      "learning_rate": 4.534938427919574e-05,
+      "loss": 0.46621341705322267,
+      "mean_token_accuracy": 0.8530056290328503,
+      "num_tokens": 93634630.0,
+      "step": 1730
+    },
+    {
+      "entropy": 0.4653968673199415,
+      "epoch": 0.5568,
+      "grad_norm": 0.08232194185256958,
+      "learning_rate": 4.482270489293685e-05,
+      "loss": 0.4654714107513428,
+      "mean_token_accuracy": 0.851850800216198,
+      "num_tokens": 94171307.0,
+      "step": 1740
+    },
+    {
+      "entropy": 0.46817119382321837,
+      "epoch": 0.56,
+      "grad_norm": 0.08604299277067184,
+      "learning_rate": 4.429660556387267e-05,
+      "loss": 0.466612720489502,
+      "mean_token_accuracy": 0.852244134247303,
+      "num_tokens": 94714877.0,
+      "step": 1750
+    },
+    {
+      "entropy": 0.4692326754331589,
+      "epoch": 0.5632,
+      "grad_norm": 0.0843324363231659,
+      "learning_rate": 4.377114523546586e-05,
+      "loss": 0.4702005386352539,
+      "mean_token_accuracy": 0.8516683995723724,
+      "num_tokens": 95246459.0,
+      "step": 1760
+    },
+    {
+      "entropy": 0.4593378620222211,
+      "epoch": 0.5664,
+      "grad_norm": 0.07727103680372238,
+      "learning_rate": 4.324638277958633e-05,
+      "loss": 0.4627023696899414,
+      "mean_token_accuracy": 0.8540186882019043,
+      "num_tokens": 95796716.0,
+      "step": 1770
+    },
+    {
+      "entropy": 0.4712889514863491,
+      "epoch": 0.5696,
+      "grad_norm": 0.08348117768764496,
+      "learning_rate": 4.272237698991529e-05,
+      "loss": 0.4713275909423828,
+      "mean_token_accuracy": 0.8512714870274067,
+      "num_tokens": 96336169.0,
+      "step": 1780
+    },
+    {
+      "entropy": 0.46586135439574716,
+      "epoch": 0.5728,
+      "grad_norm": 0.08345329016447067,
+      "learning_rate": 4.219918657535806e-05,
+      "loss": 0.46833324432373047,
+      "mean_token_accuracy": 0.8511520080268383,
+      "num_tokens": 96862313.0,
+      "step": 1790
+    },
+    {
+      "entropy": 0.4541656643152237,
+      "epoch": 0.576,
+      "grad_norm": 0.0810740664601326,
+      "learning_rate": 4.167687015346647e-05,
+      "loss": 0.4527875900268555,
+      "mean_token_accuracy": 0.8561052702367306,
+      "num_tokens": 97408845.0,
+      "step": 1800
+    },
+    {
+      "entropy": 0.45898658633232114,
+      "epoch": 0.5792,
+      "grad_norm": 0.0800258219242096,
+      "learning_rate": 4.115548624387137e-05,
+      "loss": 0.4628445148468018,
+      "mean_token_accuracy": 0.8538055151700974,
+      "num_tokens": 97953346.0,
+      "step": 1810
+    },
+    {
+      "entropy": 0.4664215393364429,
+      "epoch": 0.5824,
+      "grad_norm": 0.07937142997980118,
+      "learning_rate": 4.063509326172614e-05,
+      "loss": 0.46411833763122556,
+      "mean_token_accuracy": 0.8535587899386883,
+      "num_tokens": 98496187.0,
+      "step": 1820
+    },
+    {
+      "entropy": 0.4728786572813988,
+      "epoch": 0.5856,
+      "grad_norm": 0.07668383419513702,
+      "learning_rate": 4.0115749511162015e-05,
+      "loss": 0.47352752685546873,
+      "mean_token_accuracy": 0.8497631318867207,
+      "num_tokens": 99040416.0,
+      "step": 1830
+    },
+    {
+      "entropy": 0.47622836790978906,
+      "epoch": 0.5888,
+      "grad_norm": 0.08195730298757553,
+      "learning_rate": 3.959751317875578e-05,
+      "loss": 0.4752840042114258,
+      "mean_token_accuracy": 0.850227864831686,
+      "num_tokens": 99593149.0,
+      "step": 1840
+    },
+    {
+      "entropy": 0.4608859997242689,
+      "epoch": 0.592,
+      "grad_norm": 0.0843755304813385,
+      "learning_rate": 3.908044232701048e-05,
+      "loss": 0.46326684951782227,
+      "mean_token_accuracy": 0.8526121072471142,
+      "num_tokens": 100134908.0,
+      "step": 1850
+    },
+    {
+      "entropy": 0.47524676024913787,
+      "epoch": 0.5952,
+      "grad_norm": 0.07998881489038467,
+      "learning_rate": 3.856459488785034e-05,
+      "loss": 0.4724444389343262,
+      "mean_token_accuracy": 0.8499803431332111,
+      "num_tokens": 100688262.0,
+      "step": 1860
+    },
+    {
+      "entropy": 0.4676617939025164,
+      "epoch": 0.5984,
+      "grad_norm": 0.07983442395925522,
+      "learning_rate": 3.8050028656130044e-05,
+      "loss": 0.4673893451690674,
+      "mean_token_accuracy": 0.8520112574100495,
+      "num_tokens": 101238667.0,
+      "step": 1870
+    },
+    {
+      "entropy": 0.46478423550724984,
+      "epoch": 0.6016,
+      "grad_norm": 0.08955696225166321,
+      "learning_rate": 3.753680128315952e-05,
+      "loss": 0.4648019790649414,
+      "mean_token_accuracy": 0.8533546671271324,
+      "num_tokens": 101782475.0,
+      "step": 1880
+    },
+    {
+      "entropy": 0.46628143414855006,
+      "epoch": 0.6048,
+      "grad_norm": 0.07812649011611938,
+      "learning_rate": 3.702497027024481e-05,
+      "loss": 0.46871194839477537,
+      "mean_token_accuracy": 0.8519086144864559,
+      "num_tokens": 102320202.0,
+      "step": 1890
+    },
+    {
+      "entropy": 0.4700084120035172,
+      "epoch": 0.608,
+      "grad_norm": 0.08428274095058441,
+      "learning_rate": 3.651459296224558e-05,
+      "loss": 0.47167129516601564,
+      "mean_token_accuracy": 0.8507826834917068,
+      "num_tokens": 102861209.0,
+      "step": 1900
+    },
+    {
+      "entropy": 0.4668469190597534,
+      "epoch": 0.6112,
+      "grad_norm": 0.07659580558538437,
+      "learning_rate": 3.6005726541150374e-05,
+      "loss": 0.4668436527252197,
+      "mean_token_accuracy": 0.8514630772173405,
+      "num_tokens": 103387686.0,
+      "step": 1910
+    },
+    {
+      "entropy": 0.45636183880269526,
+      "epoch": 0.6144,
+      "grad_norm": 0.08237709105014801,
+      "learning_rate": 3.549842801967e-05,
+      "loss": 0.45879192352294923,
+      "mean_token_accuracy": 0.8547308295965195,
+      "num_tokens": 103924119.0,
+      "step": 1920
+    },
+    {
+      "entropy": 0.46668031103909013,
+      "epoch": 0.6176,
+      "grad_norm": 0.08361906558275223,
+      "learning_rate": 3.49927542348499e-05,
+      "loss": 0.46662054061889646,
+      "mean_token_accuracy": 0.8522961035370826,
+      "num_tokens": 104466214.0,
+      "step": 1930
+    },
+    {
+      "entropy": 0.4627140048891306,
+      "epoch": 0.6208,
+      "grad_norm": 0.08293884992599487,
+      "learning_rate": 3.448876184170219e-05,
+      "loss": 0.4651099681854248,
+      "mean_token_accuracy": 0.8533656261861324,
+      "num_tokens": 105006195.0,
+      "step": 1940
+    },
+    {
+      "entropy": 0.459839666634798,
+      "epoch": 0.624,
+      "grad_norm": 0.07877857983112335,
+      "learning_rate": 3.3986507306858125e-05,
+      "loss": 0.4621723175048828,
+      "mean_token_accuracy": 0.8544248096644879,
+      "num_tokens": 105548590.0,
+      "step": 1950
+    },
+    {
+      "entropy": 0.4668099503964186,
+      "epoch": 0.6272,
+      "grad_norm": 0.08203998953104019,
+      "learning_rate": 3.3486046902241664e-05,
+      "loss": 0.4681126594543457,
+      "mean_token_accuracy": 0.8515423350036144,
+      "num_tokens": 106094102.0,
+      "step": 1960
+    },
+    {
+      "entropy": 0.44877244122326376,
+      "epoch": 0.6304,
+      "grad_norm": 0.07209313660860062,
+      "learning_rate": 3.2987436698764763e-05,
+      "loss": 0.45133142471313475,
+      "mean_token_accuracy": 0.8570409543812275,
+      "num_tokens": 106635016.0,
+      "step": 1970
+    },
+    {
+      "entropy": 0.4783164393156767,
+      "epoch": 0.6336,
+      "grad_norm": 0.08684968948364258,
+      "learning_rate": 3.2490732560045437e-05,
+      "loss": 0.48186798095703126,
+      "mean_token_accuracy": 0.8482549019157887,
+      "num_tokens": 107174345.0,
+      "step": 1980
+    },
+    {
+      "entropy": 0.4584357634186745,
+      "epoch": 0.6368,
+      "grad_norm": 0.07954578846693039,
+      "learning_rate": 3.1995990136148657e-05,
+      "loss": 0.4587684631347656,
+      "mean_token_accuracy": 0.8542109049856663,
+      "num_tokens": 107731902.0,
+      "step": 1990
+    },
+    {
+      "entropy": 0.4726602200418711,
+      "epoch": 0.64,
+      "grad_norm": 0.08083987981081009,
+      "learning_rate": 3.1503264857351545e-05,
+      "loss": 0.46906423568725586,
+      "mean_token_accuracy": 0.8510009065270424,
+      "num_tokens": 108279762.0,
+      "step": 2000
+    },
+    {
+      "entropy": 0.44844978749752046,
+      "epoch": 0.6432,
+      "grad_norm": 0.07614653557538986,
+      "learning_rate": 3.1012611927932975e-05,
+      "loss": 0.44867725372314454,
+      "mean_token_accuracy": 0.8577892072498798,
+      "num_tokens": 108816206.0,
+      "step": 2010
+    },
+    {
+      "entropy": 0.4724151039496064,
+      "epoch": 0.6464,
+      "grad_norm": 0.08435625582933426,
+      "learning_rate": 3.0524086319988634e-05,
+      "loss": 0.47345867156982424,
+      "mean_token_accuracy": 0.8507047936320304,
+      "num_tokens": 109367381.0,
+      "step": 2020
+    },
+    {
+      "entropy": 0.44393031001091005,
+      "epoch": 0.6496,
+      "grad_norm": 0.08213211596012115,
+      "learning_rate": 3.0037742767271894e-05,
+      "loss": 0.4445021152496338,
+      "mean_token_accuracy": 0.8578391231596469,
+      "num_tokens": 109908791.0,
+      "step": 2030
+    },
+    {
+      "entropy": 0.4640093095600605,
+      "epoch": 0.6528,
+      "grad_norm": 0.082188181579113,
+      "learning_rate": 2.955363575906162e-05,
+      "loss": 0.4640909194946289,
+      "mean_token_accuracy": 0.8525204718112945,
+      "num_tokens": 110449065.0,
+      "step": 2040
+    },
+    {
+      "entropy": 0.4612592164427042,
+      "epoch": 0.656,
+      "grad_norm": 0.08360268920660019,
+      "learning_rate": 2.907181953405723e-05,
+      "loss": 0.458968448638916,
+      "mean_token_accuracy": 0.8536446385085583,
+      "num_tokens": 110994873.0,
+      "step": 2050
+    },
+    {
+      "entropy": 0.4579277090728283,
+      "epoch": 0.6592,
+      "grad_norm": 0.08358589559793472,
+      "learning_rate": 2.859234807430186e-05,
+      "loss": 0.46317567825317385,
+      "mean_token_accuracy": 0.8538489252328872,
+      "num_tokens": 111535422.0,
+      "step": 2060
+    },
+    {
+      "entropy": 0.4806145966053009,
+      "epoch": 0.6624,
+      "grad_norm": 0.07769794017076492,
+      "learning_rate": 2.8115275099134307e-05,
+      "loss": 0.4799009323120117,
+      "mean_token_accuracy": 0.8483485169708729,
+      "num_tokens": 112081357.0,
+      "step": 2070
+    },
+    {
+      "entropy": 0.4620382659137249,
+      "epoch": 0.6656,
+      "grad_norm": 0.0826093927025795,
+      "learning_rate": 2.76406540591704e-05,
+      "loss": 0.46086602210998534,
+      "mean_token_accuracy": 0.8536053143441678,
+      "num_tokens": 112621297.0,
+      "step": 2080
+    },
+    {
+      "entropy": 0.4670826118439436,
+      "epoch": 0.6688,
+      "grad_norm": 0.08501531928777695,
+      "learning_rate": 2.716853813031435e-05,
+      "loss": 0.4681534767150879,
+      "mean_token_accuracy": 0.8521834880113601,
+      "num_tokens": 113174059.0,
+      "step": 2090
+    },
+    {
+      "entropy": 0.46330604888498783,
+      "epoch": 0.672,
+      "grad_norm": 0.0856207087635994,
+      "learning_rate": 2.6698980207801165e-05,
+      "loss": 0.4677117347717285,
+      "mean_token_accuracy": 0.8527464650571346,
+      "num_tokens": 113708519.0,
+      "step": 2100
+    },
+    {
+      "entropy": 0.4704467307776213,
+      "epoch": 0.6752,
+      "grad_norm": 0.07923608273267746,
+      "learning_rate": 2.6232032900270255e-05,
+      "loss": 0.4697268009185791,
+      "mean_token_accuracy": 0.8514689601957798,
+      "num_tokens": 114241852.0,
+      "step": 2110
+    },
+    {
+      "entropy": 0.4633196447044611,
+      "epoch": 0.6784,
+      "grad_norm": 0.08135133981704712,
+      "learning_rate": 2.576774852387116e-05,
+      "loss": 0.4679567337036133,
+      "mean_token_accuracy": 0.8528963334858417,
+      "num_tokens": 114773806.0,
+      "step": 2120
+    },
+    {
+      "entropy": 0.46051954887807367,
+      "epoch": 0.6816,
+      "grad_norm": 0.07803782820701599,
+      "learning_rate": 2.5306179096402284e-05,
+      "loss": 0.4630821704864502,
+      "mean_token_accuracy": 0.8534800790250301,
+      "num_tokens": 115317973.0,
+      "step": 2130
+    },
+    {
+      "entropy": 0.47381667234003544,
+      "epoch": 0.6848,
+      "grad_norm": 0.08514274656772614,
+      "learning_rate": 2.4847376331482708e-05,
+      "loss": 0.47550249099731445,
+      "mean_token_accuracy": 0.8499852366745472,
+      "num_tokens": 115857229.0,
+      "step": 2140
+    },
+    {
+      "entropy": 0.4630388144403696,
+      "epoch": 0.688,
+      "grad_norm": 0.08278600871562958,
+      "learning_rate": 2.439139163275841e-05,
+      "loss": 0.45961742401123046,
+      "mean_token_accuracy": 0.8528052791953087,
+      "num_tokens": 116391992.0,
+      "step": 2150
+    },
+    {
+      "entropy": 0.4639690738171339,
+      "epoch": 0.6912,
+      "grad_norm": 0.08281213045120239,
+      "learning_rate": 2.3938276088143003e-05,
+      "loss": 0.46523027420043944,
+      "mean_token_accuracy": 0.8540500201284885,
+      "num_tokens": 116929302.0,
+      "step": 2160
+    },
+    {
+      "entropy": 0.4527744609862566,
+      "epoch": 0.6944,
+      "grad_norm": 0.08127865195274353,
+      "learning_rate": 2.348808046409394e-05,
+      "loss": 0.45207915306091306,
+      "mean_token_accuracy": 0.8564808771014214,
+      "num_tokens": 117473336.0,
+      "step": 2170
+    },
+    {
+      "entropy": 0.4757744297385216,
+      "epoch": 0.6976,
+      "grad_norm": 0.08432777225971222,
+      "learning_rate": 2.304085519992464e-05,
+      "loss": 0.47539215087890624,
+      "mean_token_accuracy": 0.8493755608797073,
+      "num_tokens": 118012328.0,
+      "step": 2180
+    },
+    {
+      "entropy": 0.4456796331331134,
+      "epoch": 0.7008,
+      "grad_norm": 0.17296849191188812,
+      "learning_rate": 2.2596650402153435e-05,
+      "loss": 0.44671993255615233,
+      "mean_token_accuracy": 0.8578110568225383,
+      "num_tokens": 118571425.0,
+      "step": 2190
+    },
+    {
+      "entropy": 0.46766641810536386,
+      "epoch": 0.704,
+      "grad_norm": 0.0819447934627533,
+      "learning_rate": 2.215551583888965e-05,
+      "loss": 0.4681885242462158,
+      "mean_token_accuracy": 0.852161280810833,
+      "num_tokens": 119115084.0,
+      "step": 2200
+    },
+    {
+      "entropy": 0.45493533834815025,
+      "epoch": 0.7072,
+      "grad_norm": 0.08149348944425583,
+      "learning_rate": 2.1717500934257666e-05,
+      "loss": 0.4589592933654785,
+      "mean_token_accuracy": 0.8552753947675228,
+      "num_tokens": 119652242.0,
+      "step": 2210
+    },
+    {
+      "entropy": 0.47455503717064856,
+      "epoch": 0.7104,
+      "grad_norm": 0.08334008604288101,
+      "learning_rate": 2.1282654762859444e-05,
+      "loss": 0.4724628448486328,
+      "mean_token_accuracy": 0.8500432133674621,
+      "num_tokens": 120189209.0,
+      "step": 2220
+    },
+    {
+      "entropy": 0.46832988820970056,
+      "epoch": 0.7136,
+      "grad_norm": 0.08058661222457886,
+      "learning_rate": 2.0851026044276406e-05,
+      "loss": 0.4681687355041504,
+      "mean_token_accuracy": 0.8522658489644528,
+      "num_tokens": 120733556.0,
+      "step": 2230
+    },
+    {
+      "entropy": 0.47705818600952626,
+      "epoch": 0.7168,
+      "grad_norm": 0.08390231430530548,
+      "learning_rate": 2.0422663137610798e-05,
+      "loss": 0.47703895568847654,
+      "mean_token_accuracy": 0.8492309600114822,
+      "num_tokens": 121274222.0,
+      "step": 2240
+    },
+    {
+      "entropy": 0.47597878277301786,
+      "epoch": 0.72,
+      "grad_norm": 0.08627524971961975,
+      "learning_rate": 1.9997614036067742e-05,
+      "loss": 0.4765979766845703,
+      "mean_token_accuracy": 0.8501147925853729,
+      "num_tokens": 121808251.0,
+      "step": 2250
+    },
+    {
+      "entropy": 0.46871641650795937,
+      "epoch": 0.7232,
+      "grad_norm": 0.08514464646577835,
+      "learning_rate": 1.9575926361578078e-05,
+      "loss": 0.47216053009033204,
+      "mean_token_accuracy": 0.8506880171597004,
+      "num_tokens": 122344572.0,
+      "step": 2260
+    },
+    {
+      "entropy": 0.4548498407006264,
+      "epoch": 0.7264,
+      "grad_norm": 0.08697574585676193,
+      "learning_rate": 1.9157647359462825e-05,
+      "loss": 0.4541280269622803,
+      "mean_token_accuracy": 0.8561206422746181,
+      "num_tokens": 122888982.0,
+      "step": 2270
+    },
+    {
+      "entropy": 0.4578681141138077,
+      "epoch": 0.7296,
+      "grad_norm": 0.08466683328151703,
+      "learning_rate": 1.8742823893139965e-05,
+      "loss": 0.461226224899292,
+      "mean_token_accuracy": 0.8543559476733208,
+      "num_tokens": 123440890.0,
+      "step": 2280
+    },
+    {
+      "entropy": 0.45721089132130144,
+      "epoch": 0.7328,
+      "grad_norm": 0.08381082117557526,
+      "learning_rate": 1.8331502438873883e-05,
+      "loss": 0.458039379119873,
+      "mean_token_accuracy": 0.8553496666252614,
+      "num_tokens": 123995478.0,
+      "step": 2290
+    },
+    {
+      "entropy": 0.48102103658020495,
+      "epoch": 0.736,
+      "grad_norm": 0.08151254802942276,
+      "learning_rate": 1.792372908056824e-05,
+      "loss": 0.4803072452545166,
+      "mean_token_accuracy": 0.8496612899005413,
+      "num_tokens": 124543628.0,
+      "step": 2300
+    },
+    {
+      "entropy": 0.4647577222436666,
+      "epoch": 0.7392,
+      "grad_norm": 0.07907874137163162,
+      "learning_rate": 1.751954950460275e-05,
+      "loss": 0.4656996726989746,
+      "mean_token_accuracy": 0.8517619237303734,
+      "num_tokens": 125080657.0,
+      "step": 2310
+    },
+    {
+      "entropy": 0.46176278330385684,
+      "epoch": 0.7424,
+      "grad_norm": 0.08020291477441788,
+      "learning_rate": 1.711900899471463e-05,
+      "loss": 0.4624810218811035,
+      "mean_token_accuracy": 0.852861026674509,
+      "num_tokens": 125621368.0,
+      "step": 2320
+    },
+    {
+      "entropy": 0.45940026380121707,
+      "epoch": 0.7456,
+      "grad_norm": 0.08816850930452347,
+      "learning_rate": 1.6722152426924957e-05,
+      "loss": 0.464081335067749,
+      "mean_token_accuracy": 0.8537900924682618,
+      "num_tokens": 126159116.0,
+      "step": 2330
+    },
+    {
+      "entropy": 0.461243599653244,
+      "epoch": 0.7488,
+      "grad_norm": 0.08073343336582184,
+      "learning_rate": 1.632902426451094e-05,
+      "loss": 0.46134142875671386,
+      "mean_token_accuracy": 0.8536130629479886,
+      "num_tokens": 126706217.0,
+      "step": 2340
+    },
+    {
+      "entropy": 0.4594324670732021,
+      "epoch": 0.752,
+      "grad_norm": 0.08281491696834564,
+      "learning_rate": 1.5939668553024235e-05,
+      "loss": 0.4629940986633301,
+      "mean_token_accuracy": 0.8533355824649334,
+      "num_tokens": 127244265.0,
+      "step": 2350
+    },
+    {
+      "entropy": 0.48015770092606547,
+      "epoch": 0.7552,
+      "grad_norm": 0.08678045123815536,
+      "learning_rate": 1.5554128915356126e-05,
+      "loss": 0.4795347213745117,
+      "mean_token_accuracy": 0.8486863911151886,
+      "num_tokens": 127787717.0,
+      "step": 2360
+    },
+    {
+      "entropy": 0.4674267187714577,
+      "epoch": 0.7584,
+      "grad_norm": 0.08806832134723663,
+      "learning_rate": 1.5172448546850165e-05,
+      "loss": 0.4694706916809082,
+      "mean_token_accuracy": 0.8519207708537578,
+      "num_tokens": 128325214.0,
+      "step": 2370
+    },
+    {
+      "entropy": 0.4692223500460386,
+      "epoch": 0.7616,
+      "grad_norm": 0.08210162818431854,
+      "learning_rate": 1.4794670210462492e-05,
+      "loss": 0.47144575119018556,
+      "mean_token_accuracy": 0.8511710479855538,
+      "num_tokens": 128866259.0,
+      "step": 2380
+    },
+    {
+      "entropy": 0.45822653025388715,
+      "epoch": 0.7648,
+      "grad_norm": 0.08143525570631027,
+      "learning_rate": 1.4420836231970914e-05,
+      "loss": 0.46082229614257814,
+      "mean_token_accuracy": 0.8544653721153737,
+      "num_tokens": 129404832.0,
+      "step": 2390
+    },
+    {
+      "entropy": 0.466277177259326,
+      "epoch": 0.768,
+      "grad_norm": 0.08295784145593643,
+      "learning_rate": 1.4050988495232565e-05,
+      "loss": 0.46442317962646484,
+      "mean_token_accuracy": 0.8529163144528866,
+      "num_tokens": 129943074.0,
+      "step": 2400
+    },
+    {
+      "entropy": 0.46804318726062777,
+      "epoch": 0.7712,
+      "grad_norm": 0.08742719143629074,
+      "learning_rate": 1.3685168437491475e-05,
+      "loss": 0.4691460609436035,
+      "mean_token_accuracy": 0.8516571588814259,
+      "num_tokens": 130487358.0,
+      "step": 2410
+    },
+    {
+      "entropy": 0.45586061477661133,
+      "epoch": 0.7744,
+      "grad_norm": 0.07925993204116821,
+      "learning_rate": 1.3323417044735864e-05,
+      "loss": 0.4539622783660889,
+      "mean_token_accuracy": 0.8555390655994415,
+      "num_tokens": 131030867.0,
+      "step": 2420
+    },
+    {
+      "entropy": 0.4753517936915159,
+      "epoch": 0.7776,
+      "grad_norm": 0.0779256746172905,
+      "learning_rate": 1.296577484710627e-05,
+      "loss": 0.4771604537963867,
+      "mean_token_accuracy": 0.8492721661925315,
+      "num_tokens": 131583751.0,
+      "step": 2430
+    },
+    {
+      "entropy": 0.46178548224270344,
+      "epoch": 0.7808,
+      "grad_norm": 0.0828179195523262,
+      "learning_rate": 1.2612281914354451e-05,
+      "loss": 0.46554718017578123,
+      "mean_token_accuracy": 0.8518827095627785,
+      "num_tokens": 132115400.0,
+      "step": 2440
+    },
+    {
+      "entropy": 0.4675505064427853,
+      "epoch": 0.784,
+      "grad_norm": 0.08057764917612076,
+      "learning_rate": 1.226297785135414e-05,
+      "loss": 0.46473064422607424,
+      "mean_token_accuracy": 0.8523139268159866,
+      "num_tokens": 132666000.0,
+      "step": 2450
+    },
+    {
+      "entropy": 0.44793173633515837,
+      "epoch": 0.7872,
+      "grad_norm": 0.08276599645614624,
+      "learning_rate": 1.1917901793663682e-05,
+      "loss": 0.44820418357849123,
+      "mean_token_accuracy": 0.857875219732523,
+      "num_tokens": 133206157.0,
+      "step": 2460
+    },
+    {
+      "entropy": 0.4644062086939812,
+      "epoch": 0.7904,
+      "grad_norm": 0.08161120861768723,
+      "learning_rate": 1.1577092403141433e-05,
+      "loss": 0.4627474308013916,
+      "mean_token_accuracy": 0.8532275453209877,
+      "num_tokens": 133771874.0,
+      "step": 2470
+    },
+    {
+      "entropy": 0.44853013232350347,
+      "epoch": 0.7936,
+      "grad_norm": 0.08148295432329178,
+      "learning_rate": 1.1240587863614071e-05,
+      "loss": 0.45169339179992674,
+      "mean_token_accuracy": 0.8561265848577022,
+      "num_tokens": 134318559.0,
+      "step": 2480
+    },
+    {
+      "entropy": 0.45980192013084886,
+      "epoch": 0.7968,
+      "grad_norm": 0.0792505070567131,
+      "learning_rate": 1.090842587659851e-05,
+      "loss": 0.45940675735473635,
+      "mean_token_accuracy": 0.8535128928720951,
+      "num_tokens": 134862620.0,
+      "step": 2490
+    },
+    {
+      "entropy": 0.45604656375944613,
+      "epoch": 0.8,
+      "grad_norm": 0.08306615054607391,
+      "learning_rate": 1.0580643657077976e-05,
+      "loss": 0.4595036506652832,
+      "mean_token_accuracy": 0.8552262201905251,
+      "num_tokens": 135403888.0,
+      "step": 2500
+    },
+    {
+      "entropy": 0.472680402174592,
+      "epoch": 0.8032,
+      "grad_norm": 0.08291345089673996,
+      "learning_rate": 1.0257277929332332e-05,
+      "loss": 0.4714357376098633,
+      "mean_token_accuracy": 0.8513913378119469,
+      "num_tokens": 135947508.0,
+      "step": 2510
+    },
+    {
+      "entropy": 0.4574604358524084,
+      "epoch": 0.8064,
+      "grad_norm": 0.07977301627397537,
+      "learning_rate": 9.938364922823745e-06,
+      "loss": 0.4595664978027344,
+      "mean_token_accuracy": 0.854312700778246,
+      "num_tokens": 136490466.0,
+      "step": 2520
+    },
+    {
+      "entropy": 0.45157139785587785,
+      "epoch": 0.8096,
+      "grad_norm": 0.08351728320121765,
+      "learning_rate": 9.623940368137369e-06,
+      "loss": 0.4506734848022461,
+      "mean_token_accuracy": 0.8566409595310688,
+      "num_tokens": 137031910.0,
+      "step": 2530
+    },
+    {
+      "entropy": 0.4738286957144737,
+      "epoch": 0.8128,
+      "grad_norm": 0.08656181395053864,
+      "learning_rate": 9.314039492978311e-06,
+      "loss": 0.47472944259643557,
+      "mean_token_accuracy": 0.8510151915252209,
+      "num_tokens": 137579298.0,
+      "step": 2540
+    },
+    {
+      "entropy": 0.4666871029883623,
+      "epoch": 0.816,
+      "grad_norm": 0.07740852236747742,
+      "learning_rate": 9.008697018224638e-06,
+      "loss": 0.46857662200927735,
+      "mean_token_accuracy": 0.8524154961109162,
+      "num_tokens": 138113839.0,
+      "step": 2550
+    },
+    {
+      "entropy": 0.4572214774787426,
+      "epoch": 0.8192,
+      "grad_norm": 0.08414942771196365,
+      "learning_rate": 8.707947154037389e-06,
+      "loss": 0.4580754280090332,
+      "mean_token_accuracy": 0.8550274461507797,
+      "num_tokens": 138657606.0,
+      "step": 2560
+    },
+    {
+      "entropy": 0.46387795843183993,
+      "epoch": 0.8224,
+      "grad_norm": 0.07949203997850418,
+      "learning_rate": 8.411823596027685e-06,
+      "loss": 0.46186103820800783,
+      "mean_token_accuracy": 0.8532565161585808,
+      "num_tokens": 139199109.0,
+      "step": 2570
+    },
+    {
+      "entropy": 0.4731456235051155,
+      "epoch": 0.8256,
+      "grad_norm": 0.08628631383180618,
+      "learning_rate": 8.120359521481501e-06,
+      "loss": 0.47334856986999513,
+      "mean_token_accuracy": 0.8491534382104874,
+      "num_tokens": 139727359.0,
+      "step": 2580
+    },
+    {
+      "entropy": 0.4579001996666193,
+      "epoch": 0.8288,
+      "grad_norm": 0.0787086933851242,
+      "learning_rate": 7.833587585642527e-06,
+      "loss": 0.45937094688415525,
+      "mean_token_accuracy": 0.8540180057287217,
+      "num_tokens": 140263619.0,
+      "step": 2590
+    },
+    {
+      "entropy": 0.47708442993462086,
+      "epoch": 0.832,
+      "grad_norm": 0.0864366665482521,
+      "learning_rate": 7.551539918053541e-06,
+      "loss": 0.47784271240234377,
+      "mean_token_accuracy": 0.8495126746594905,
+      "num_tokens": 140803204.0,
+      "step": 2600
+    },
+    {
+      "entropy": 0.4615286778658628,
+      "epoch": 0.8352,
+      "grad_norm": 0.08561565726995468,
+      "learning_rate": 7.274248118956667e-06,
+      "loss": 0.46320443153381347,
+      "mean_token_accuracy": 0.8532632842659951,
+      "num_tokens": 141347346.0,
+      "step": 2610
+    },
+    {
+      "entropy": 0.4521449118852615,
+      "epoch": 0.8384,
+      "grad_norm": 0.0840955302119255,
+      "learning_rate": 7.001743255752857e-06,
+      "loss": 0.45335044860839846,
+      "mean_token_accuracy": 0.856326463073492,
+      "num_tokens": 141891461.0,
+      "step": 2620
+    },
+    {
+      "entropy": 0.4658927246928215,
+      "epoch": 0.8416,
+      "grad_norm": 0.08690115809440613,
+      "learning_rate": 6.7340558595212485e-06,
+      "loss": 0.46956405639648435,
+      "mean_token_accuracy": 0.8530021667480469,
+      "num_tokens": 142443544.0,
+      "step": 2630
+    },
+    {
+      "entropy": 0.4672031823545694,
+      "epoch": 0.8448,
+      "grad_norm": 0.0802784189581871,
+      "learning_rate": 6.4712159215983945e-06,
+      "loss": 0.4663797378540039,
+      "mean_token_accuracy": 0.8518442392349244,
+      "num_tokens": 142988736.0,
+      "step": 2640
+    },
+    {
+      "entropy": 0.45975690335035324,
+      "epoch": 0.848,
+      "grad_norm": 0.08276927471160889,
+      "learning_rate": 6.213252890218163e-06,
+      "loss": 0.45941638946533203,
+      "mean_token_accuracy": 0.8541808009147644,
+      "num_tokens": 143533757.0,
+      "step": 2650
+    },
+    {
+      "entropy": 0.45979082882404326,
+      "epoch": 0.8512,
+      "grad_norm": 0.08185946196317673,
+      "learning_rate": 5.960195667212337e-06,
+      "loss": 0.4600400924682617,
+      "mean_token_accuracy": 0.8538325548171997,
+      "num_tokens": 144081635.0,
+      "step": 2660
+    },
+    {
+      "entropy": 0.45433352291584017,
+      "epoch": 0.8544,
+      "grad_norm": 0.08438166230916977,
+      "learning_rate": 5.712072604772539e-06,
+      "loss": 0.4563631057739258,
+      "mean_token_accuracy": 0.8551003247499466,
+      "num_tokens": 144632213.0,
+      "step": 2670
+    },
+    {
+      "entropy": 0.47600530944764613,
+      "epoch": 0.8576,
+      "grad_norm": 0.07887919992208481,
+      "learning_rate": 5.468911502273627e-06,
+      "loss": 0.47498316764831544,
+      "mean_token_accuracy": 0.84912119358778,
+      "num_tokens": 145179291.0,
+      "step": 2680
+    },
+    {
+      "entropy": 0.48059417977929114,
+      "epoch": 0.8608,
+      "grad_norm": 0.09053132683038712,
+      "learning_rate": 5.230739603159152e-06,
+      "loss": 0.4829423427581787,
+      "mean_token_accuracy": 0.8477361969649791,
+      "num_tokens": 145708078.0,
+      "step": 2690
+    },
+    {
+      "entropy": 0.4714535430073738,
+      "epoch": 0.864,
+      "grad_norm": 0.08268075436353683,
+      "learning_rate": 4.997583591889027e-06,
+      "loss": 0.47348756790161134,
+      "mean_token_accuracy": 0.850239709764719,
+      "num_tokens": 146250196.0,
+      "step": 2700
+    },
+    {
+      "entropy": 0.4679393831640482,
+      "epoch": 0.8672,
+      "grad_norm": 0.09109136462211609,
+      "learning_rate": 4.769469590949782e-06,
+      "loss": 0.46534051895141604,
+      "mean_token_accuracy": 0.8525628745555878,
+      "num_tokens": 146790541.0,
+      "step": 2710
+    },
+    {
+      "entropy": 0.4727115105837584,
+      "epoch": 0.8704,
+      "grad_norm": 0.08139918744564056,
+      "learning_rate": 4.546423157927921e-06,
+      "loss": 0.47171907424926757,
+      "mean_token_accuracy": 0.8515786036849022,
+      "num_tokens": 147333731.0,
+      "step": 2720
+    },
+    {
+      "entropy": 0.4678931195288897,
+      "epoch": 0.8736,
+      "grad_norm": 0.08541391789913177,
+      "learning_rate": 4.3284692826464025e-06,
+      "loss": 0.4691716194152832,
+      "mean_token_accuracy": 0.8512527458369732,
+      "num_tokens": 147874335.0,
+      "step": 2730
+    },
+    {
+      "entropy": 0.4652485832571983,
+      "epoch": 0.8768,
+      "grad_norm": 0.07994713634252548,
+      "learning_rate": 4.115632384364876e-06,
+      "loss": 0.4703035831451416,
+      "mean_token_accuracy": 0.8522162444889545,
+      "num_tokens": 148416067.0,
+      "step": 2740
+    },
+    {
+      "entropy": 0.4409920936450362,
+      "epoch": 0.88,
+      "grad_norm": 0.080880306661129,
+      "learning_rate": 3.907936309043752e-06,
+      "loss": 0.4397623062133789,
+      "mean_token_accuracy": 0.8596334785223008,
+      "num_tokens": 148962585.0,
+      "step": 2750
+    },
+    {
+      "entropy": 0.4632509328424931,
+      "epoch": 0.8832,
+      "grad_norm": 0.08481155335903168,
+      "learning_rate": 3.705404326672546e-06,
+      "loss": 0.46387577056884766,
+      "mean_token_accuracy": 0.8529495015740395,
+      "num_tokens": 149504343.0,
+      "step": 2760
+    },
+    {
+      "entropy": 0.45417391285300257,
+      "epoch": 0.8864,
+      "grad_norm": 0.08323755860328674,
+      "learning_rate": 3.5080591286626994e-06,
+      "loss": 0.4597487926483154,
+      "mean_token_accuracy": 0.854594460874796,
+      "num_tokens": 150044863.0,
+      "step": 2770
+    },
+    {
+      "entropy": 0.4529740372672677,
+      "epoch": 0.8896,
+      "grad_norm": 0.08288435637950897,
+      "learning_rate": 3.315922825305323e-06,
+      "loss": 0.45327181816101075,
+      "mean_token_accuracy": 0.8563416883349418,
+      "num_tokens": 150578775.0,
+      "step": 2780
+    },
+    {
+      "entropy": 0.4761421289294958,
+      "epoch": 0.8928,
+      "grad_norm": 0.08802027255296707,
+      "learning_rate": 3.1290169432939553e-06,
+      "loss": 0.4763199806213379,
+      "mean_token_accuracy": 0.8495484620332718,
+      "num_tokens": 151109429.0,
+      "step": 2790
+    },
+    {
+      "entropy": 0.47611254937946795,
+      "epoch": 0.896,
+      "grad_norm": 0.0790930911898613,
+      "learning_rate": 2.9473624233127497e-06,
+      "loss": 0.4803468704223633,
+      "mean_token_accuracy": 0.8489079758524894,
+      "num_tokens": 151655289.0,
+      "step": 2800
+    },
+    {
+      "entropy": 0.4751625269651413,
+      "epoch": 0.8992,
+      "grad_norm": 0.0879993736743927,
+      "learning_rate": 2.770979617690289e-06,
+      "loss": 0.4718982219696045,
+      "mean_token_accuracy": 0.850994748622179,
+      "num_tokens": 152200128.0,
+      "step": 2810
+    },
+    {
+      "entropy": 0.4601206321269274,
+      "epoch": 0.9024,
+      "grad_norm": 0.08376693725585938,
+      "learning_rate": 2.5998882881193954e-06,
+      "loss": 0.4597844123840332,
+      "mean_token_accuracy": 0.8535569846630097,
+      "num_tokens": 152733954.0,
+      "step": 2820
+    },
+    {
+      "entropy": 0.4504482693970203,
+      "epoch": 0.9056,
+      "grad_norm": 0.08362895250320435,
+      "learning_rate": 2.4341076034429587e-06,
+      "loss": 0.4514331817626953,
+      "mean_token_accuracy": 0.8559839025139808,
+      "num_tokens": 153264534.0,
+      "step": 2830
+    },
+    {
+      "entropy": 0.4604618210345507,
+      "epoch": 0.9088,
+      "grad_norm": 0.08140548318624496,
+      "learning_rate": 2.273656137506397e-06,
+      "loss": 0.4624817371368408,
+      "mean_token_accuracy": 0.8531782232224941,
+      "num_tokens": 153810193.0,
+      "step": 2840
+    },
+    {
+      "entropy": 0.4675752319395542,
+      "epoch": 0.912,
+      "grad_norm": 0.08216649293899536,
+      "learning_rate": 2.1185518670765903e-06,
+      "loss": 0.47134923934936523,
+      "mean_token_accuracy": 0.8514118038117886,
+      "num_tokens": 154341314.0,
+      "step": 2850
+    },
+    {
+      "entropy": 0.4534553330391645,
+      "epoch": 0.9152,
+      "grad_norm": 0.0871286541223526,
+      "learning_rate": 1.9688121698277995e-06,
+      "loss": 0.45147104263305665,
+      "mean_token_accuracy": 0.8560508869588375,
+      "num_tokens": 154887467.0,
+      "step": 2860
+    },
+    {
+      "entropy": 0.4619016032665968,
+      "epoch": 0.9184,
+      "grad_norm": 0.08505194634199142,
+      "learning_rate": 1.8244538223947227e-06,
+      "loss": 0.464245080947876,
+      "mean_token_accuracy": 0.8529387354850769,
+      "num_tokens": 155425200.0,
+      "step": 2870
+    },
+    {
+      "entropy": 0.46694816537201406,
+      "epoch": 0.9216,
+      "grad_norm": 0.08090124279260635,
+      "learning_rate": 1.6854929984928546e-06,
+      "loss": 0.4668296813964844,
+      "mean_token_accuracy": 0.8528281696140766,
+      "num_tokens": 155966457.0,
+      "step": 2880
+    },
+    {
+      "entropy": 0.4619465067982674,
+      "epoch": 0.9248,
+      "grad_norm": 0.08461577445268631,
+      "learning_rate": 1.5519452671063917e-06,
+      "loss": 0.4609919548034668,
+      "mean_token_accuracy": 0.8542857751250267,
+      "num_tokens": 156505143.0,
+      "step": 2890
+    },
+    {
+      "entropy": 0.468216797336936,
+      "epoch": 0.928,
+      "grad_norm": 0.08594241738319397,
+      "learning_rate": 1.423825590743888e-06,
+      "loss": 0.4666452407836914,
+      "mean_token_accuracy": 0.853216753154993,
+      "num_tokens": 157061376.0,
+      "step": 2900
+    },
+    {
+      "entropy": 0.4693597674369812,
+      "epoch": 0.9312,
+      "grad_norm": 0.08410052210092545,
+      "learning_rate": 1.3011483237619305e-06,
+      "loss": 0.46759915351867676,
+      "mean_token_accuracy": 0.8519282631576062,
+      "num_tokens": 157594464.0,
+      "step": 2910
+    },
+    {
+      "entropy": 0.46422403417527675,
+      "epoch": 0.9344,
+      "grad_norm": 0.08244696259498596,
+      "learning_rate": 1.1839272107568466e-06,
+      "loss": 0.4629973411560059,
+      "mean_token_accuracy": 0.8526307120919228,
+      "num_tokens": 158131479.0,
+      "step": 2920
+    },
+    {
+      "entropy": 0.4626509569585323,
+      "epoch": 0.9376,
+      "grad_norm": 0.08002057671546936,
+      "learning_rate": 1.0721753850247984e-06,
+      "loss": 0.4618197441101074,
+      "mean_token_accuracy": 0.8529696725308895,
+      "num_tokens": 158675636.0,
+      "step": 2930
+    },
+    {
+      "entropy": 0.46270943246781826,
+      "epoch": 0.9408,
+      "grad_norm": 0.08056887239217758,
+      "learning_rate": 9.659053670903583e-07,
+      "loss": 0.46524944305419924,
+      "mean_token_accuracy": 0.8522255882620812,
+      "num_tokens": 159207544.0,
+      "step": 2940
+    },
+    {
+      "entropy": 0.4581955760717392,
+      "epoch": 0.944,
+      "grad_norm": 0.08471491187810898,
+      "learning_rate": 8.651290633036768e-07,
+      "loss": 0.46083908081054686,
+      "mean_token_accuracy": 0.8546010799705982,
+      "num_tokens": 159754829.0,
+      "step": 2950
+    },
+    {
+      "entropy": 0.4575455904006958,
+      "epoch": 0.9472,
+      "grad_norm": 0.07735498249530792,
+      "learning_rate": 7.698577645065763e-07,
+      "loss": 0.4579817771911621,
+      "mean_token_accuracy": 0.8554122239351273,
+      "num_tokens": 160309257.0,
+      "step": 2960
+    },
+    {
+      "entropy": 0.45687297359108925,
+      "epoch": 0.9504,
+      "grad_norm": 0.08392135053873062,
+      "learning_rate": 6.801021447674971e-07,
+      "loss": 0.4586464881896973,
+      "mean_token_accuracy": 0.8538117773830891,
+      "num_tokens": 160843007.0,
+      "step": 2970
+    },
+    {
+      "entropy": 0.45821645110845566,
+      "epoch": 0.9536,
+      "grad_norm": 0.08588553965091705,
+      "learning_rate": 5.958722601856093e-07,
+      "loss": 0.45988945960998534,
+      "mean_token_accuracy": 0.8539286591112614,
+      "num_tokens": 161388268.0,
+      "step": 2980
+    },
+    {
+      "entropy": 0.4608828775584698,
+      "epoch": 0.9568,
+      "grad_norm": 0.0871865451335907,
+      "learning_rate": 5.171775477641139e-07,
+      "loss": 0.46298632621765134,
+      "mean_token_accuracy": 0.8532721899449826,
+      "num_tokens": 161931208.0,
+      "step": 2990
+    },
+    {
+      "entropy": 0.4577631322667003,
+      "epoch": 0.96,
+      "grad_norm": 0.08455237746238708,
+      "learning_rate": 4.4402682435296663e-07,
+      "loss": 0.4582681179046631,
+      "mean_token_accuracy": 0.8539157420396805,
+      "num_tokens": 162482893.0,
+      "step": 3000
+    },
+    {
+      "entropy": 0.46698101833462713,
+      "epoch": 0.9632,
+      "grad_norm": 0.08095478266477585,
+      "learning_rate": 3.7642828566104014e-07,
+      "loss": 0.4648216724395752,
+      "mean_token_accuracy": 0.8526224665343761,
+      "num_tokens": 163035582.0,
+      "step": 3010
+    },
+    {
+      "entropy": 0.45926835350692274,
+      "epoch": 0.9664,
+      "grad_norm": 0.07997886836528778,
+      "learning_rate": 3.143895053378698e-07,
+      "loss": 0.46354007720947266,
+      "mean_token_accuracy": 0.8539989247918129,
+      "num_tokens": 163575065.0,
+      "step": 3020
+    },
+    {
+      "entropy": 0.46516573056578636,
+      "epoch": 0.9696,
+      "grad_norm": 0.08414747565984726,
+      "learning_rate": 2.579174341251378e-07,
+      "loss": 0.4646583080291748,
+      "mean_token_accuracy": 0.8528312422335148,
+      "num_tokens": 164110820.0,
+      "step": 3030
+    },
+    {
+      "entropy": 0.46528425142169,
+      "epoch": 0.9728,
+      "grad_norm": 0.07954436540603638,
+      "learning_rate": 2.070183990779073e-07,
+      "loss": 0.4692234039306641,
+      "mean_token_accuracy": 0.8517439760267734,
+      "num_tokens": 164646689.0,
+      "step": 3040
+    },
+    {
+      "entropy": 0.4535929933190346,
+      "epoch": 0.976,
+      "grad_norm": 0.07825275510549545,
+      "learning_rate": 1.6169810285573406e-07,
+      "loss": 0.4504249572753906,
+      "mean_token_accuracy": 0.855993852764368,
+      "num_tokens": 165211524.0,
+      "step": 3050
+    },
+    {
+      "entropy": 0.46400816589593885,
+      "epoch": 0.9792,
+      "grad_norm": 0.08895574510097504,
+      "learning_rate": 1.219616230837939e-07,
+      "loss": 0.46570405960083006,
+      "mean_token_accuracy": 0.8532424159348011,
+      "num_tokens": 165749501.0,
+      "step": 3060
+    },
+    {
+      "entropy": 0.47027173936367034,
+      "epoch": 0.9824,
+      "grad_norm": 0.0804954394698143,
+      "learning_rate": 8.781341178393244e-08,
+      "loss": 0.47028093338012694,
+      "mean_token_accuracy": 0.8512769274413585,
+      "num_tokens": 166291620.0,
+      "step": 3070
+    },
+    {
+      "entropy": 0.4440871596336365,
+      "epoch": 0.9856,
+      "grad_norm": 0.08392955362796783,
+      "learning_rate": 5.925729487590847e-08,
+      "loss": 0.4455129146575928,
+      "mean_token_accuracy": 0.8584165126085281,
+      "num_tokens": 166837045.0,
+      "step": 3080
+    },
+    {
+      "entropy": 0.46984327845275403,
+      "epoch": 0.9888,
+      "grad_norm": 0.08318781852722168,
+      "learning_rate": 3.62964717487424e-08,
+      "loss": 0.47353472709655764,
+      "mean_token_accuracy": 0.8512238331139088,
+      "num_tokens": 167386481.0,
+      "step": 3090
+    },
+    {
+      "entropy": 0.4673919454216957,
+      "epoch": 0.992,
+      "grad_norm": 0.0839017778635025,
+      "learning_rate": 1.8933514902225167e-08,
+      "loss": 0.46686429977416993,
+      "mean_token_accuracy": 0.8511216603219509,
+      "num_tokens": 167928926.0,
+      "step": 3100
+    },
+    {
+      "entropy": 0.46121672354638577,
+      "epoch": 0.9952,
+      "grad_norm": 0.08226187527179718,
+      "learning_rate": 7.170369658748843e-09,
+      "loss": 0.46160082817077636,
+      "mean_token_accuracy": 0.8539618447422981,
+      "num_tokens": 168485378.0,
+      "step": 3110
+    },
+    {
+      "entropy": 0.4504577457904816,
+      "epoch": 0.9984,
+      "grad_norm": 0.08203563094139099,
+      "learning_rate": 1.0083539453142888e-09,
+      "loss": 0.45265903472900393,
+      "mean_token_accuracy": 0.8571697436273098,
+      "num_tokens": 169031051.0,
+      "step": 3120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 3125,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.416136156903637e+18,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3125/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8583acc94d326a62088f75412e0942118ca66c71eede297a3417075363db5791
+size 5841

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,30 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8583acc94d326a62088f75412e0942118ca66c71eede297a3417075363db5791
+size 5841