Instructions to use master103525/boss-task2-full-training with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use master103525/boss-task2-full-training with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("/cache/models/Qwen--Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base_model, "master103525/boss-task2-full-training")

Transformers

How to use master103525/boss-task2-full-training with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="master103525/boss-task2-full-training")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("master103525/boss-task2-full-training", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use master103525/boss-task2-full-training with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "master103525/boss-task2-full-training"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "master103525/boss-task2-full-training",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/master103525/boss-task2-full-training

SGLang

How to use master103525/boss-task2-full-training with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "master103525/boss-task2-full-training" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "master103525/boss-task2-full-training",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "master103525/boss-task2-full-training" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "master103525/boss-task2-full-training",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use master103525/boss-task2-full-training with Docker Model Runner:
```
docker model run hf.co/master103525/boss-task2-full-training
```

master103525 commited on 29 days ago

Commit

2d8fe2f

verified ·

1 Parent(s): c781fd0

Upload folder using huggingface_hub

Browse files

Files changed (14) hide show

.gitattributes +1 -0
README.md +209 -0
adapter_config.json +46 -0
adapter_model.safetensors +3 -0
added_tokens.json +24 -0
chat_template.jinja +54 -0
loss.txt +1 -0
merges.txt +0 -0
special_tokens_map.json +31 -0
tokenizer.json +3 -0
tokenizer_config.json +207 -0
trainer_state.json +2914 -0
training_args.bin +3 -0
vocab.json +0 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,209 @@

+---
+base_model: Qwen--Qwen2.5-7B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Qwen--Qwen2.5-7B-Instruct
+- lora
+- sft
+- transformers
+- trl
+---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.18.1

adapter_config.json ADDED Viewed

	@@ -0,0 +1,46 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/cache/models/Qwen--Qwen2.5-7B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "v_proj",
+    "down_proj",
+    "gate_proj",
+    "up_proj",
+    "q_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c20ce3da3d7c143bb13f11012d2861f95c3c2c1c0a700d51ae79356d82424527
+size 1291899160

added_tokens.json ADDED Viewed

	@@ -0,0 +1,24 @@

+{
+  "</tool_call>": 151658,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,54 @@

+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

loss.txt ADDED Viewed

	@@ -0,0 +1 @@


1	+ 1600,no_eval

merges.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,31 @@

+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
+size 11421896

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,207 @@

+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json ADDED Viewed

	@@ -0,0 +1,2914 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.6388500698742264,
+  "eval_steps": 500,
+  "global_step": 1600,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0019964064683569574,
+      "grad_norm": 12.8033447265625,
+      "learning_rate": 4.917142857142858e-06,
+      "loss": 0.8518,
+      "mean_token_accuracy": 0.8612988531589508,
+      "num_tokens": 714370.0,
+      "step": 5
+    },
+    {
+      "epoch": 0.003992812936713915,
+      "grad_norm": 2.9046072959899902,
+      "learning_rate": 1.106357142857143e-05,
+      "loss": 0.3973,
+      "mean_token_accuracy": 0.9099577963352203,
+      "num_tokens": 1435512.0,
+      "step": 10
+    },
+    {
+      "epoch": 0.005989219405070872,
+      "grad_norm": 0.9405294060707092,
+      "learning_rate": 1.7210000000000003e-05,
+      "loss": 0.163,
+      "mean_token_accuracy": 0.9522678524255752,
+      "num_tokens": 2137491.0,
+      "step": 15
+    },
+    {
+      "epoch": 0.00798562587342783,
+      "grad_norm": 0.823198139667511,
+      "learning_rate": 2.3356428571428574e-05,
+      "loss": 0.088,
+      "mean_token_accuracy": 0.967251580953598,
+      "num_tokens": 2873053.0,
+      "step": 20
+    },
+    {
+      "epoch": 0.009982032341784788,
+      "grad_norm": 0.5476183891296387,
+      "learning_rate": 2.950285714285715e-05,
+      "loss": 0.0762,
+      "mean_token_accuracy": 0.9689888685941697,
+      "num_tokens": 3597110.0,
+      "step": 25
+    },
+    {
+      "epoch": 0.011978438810141745,
+      "grad_norm": 0.3753618001937866,
+      "learning_rate": 3.5649285714285724e-05,
+      "loss": 0.0748,
+      "mean_token_accuracy": 0.9682164311408996,
+      "num_tokens": 4317113.0,
+      "step": 30
+    },
+    {
+      "epoch": 0.013974845278498703,
+      "grad_norm": 0.3772870600223541,
+      "learning_rate": 4.179571428571429e-05,
+      "loss": 0.0753,
+      "mean_token_accuracy": 0.9693712830543518,
+      "num_tokens": 5051360.0,
+      "step": 35
+    },
+    {
+      "epoch": 0.01597125174685566,
+      "grad_norm": 0.4375256597995758,
+      "learning_rate": 4.302497723128185e-05,
+      "loss": 0.07,
+      "mean_token_accuracy": 0.9718590974807739,
+      "num_tokens": 5762813.0,
+      "step": 40
+    },
+    {
+      "epoch": 0.017967658215212616,
+      "grad_norm": 0.31170469522476196,
+      "learning_rate": 4.302488473347447e-05,
+      "loss": 0.0701,
+      "mean_token_accuracy": 0.9706808269023895,
+      "num_tokens": 6478762.0,
+      "step": 45
+    },
+    {
+      "epoch": 0.019964064683569576,
+      "grad_norm": 0.2859897017478943,
+      "learning_rate": 4.302472108394058e-05,
+      "loss": 0.0684,
+      "mean_token_accuracy": 0.972345557808876,
+      "num_tokens": 7204763.0,
+      "step": 50
+    },
+    {
+      "epoch": 0.021960471151926533,
+      "grad_norm": 0.25486132502555847,
+      "learning_rate": 4.302448628340187e-05,
+      "loss": 0.064,
+      "mean_token_accuracy": 0.9740626871585846,
+      "num_tokens": 7925022.0,
+      "step": 55
+    },
+    {
+      "epoch": 0.02395687762028349,
+      "grad_norm": 0.23721954226493835,
+      "learning_rate": 4.302418033289381e-05,
+      "loss": 0.0676,
+      "mean_token_accuracy": 0.9731216758489609,
+      "num_tokens": 8632030.0,
+      "step": 60
+    },
+    {
+      "epoch": 0.025953284088640446,
+      "grad_norm": 0.26267364621162415,
+      "learning_rate": 4.302380323376563e-05,
+      "loss": 0.0626,
+      "mean_token_accuracy": 0.9742462456226348,
+      "num_tokens": 9364700.0,
+      "step": 65
+    },
+    {
+      "epoch": 0.027949690556997406,
+      "grad_norm": 0.2607436776161194,
+      "learning_rate": 4.3023354987680335e-05,
+      "loss": 0.0603,
+      "mean_token_accuracy": 0.9749193072319031,
+      "num_tokens": 10097255.0,
+      "step": 70
+    },
+    {
+      "epoch": 0.029946097025354362,
+      "grad_norm": 0.3435736298561096,
+      "learning_rate": 4.3022835596614676e-05,
+      "loss": 0.0634,
+      "mean_token_accuracy": 0.9741163223981857,
+      "num_tokens": 10823103.0,
+      "step": 75
+    },
+    {
+      "epoch": 0.03194250349371132,
+      "grad_norm": 0.19693654775619507,
+      "learning_rate": 4.3022245062859155e-05,
+      "loss": 0.0623,
+      "mean_token_accuracy": 0.9742611676454545,
+      "num_tokens": 11548942.0,
+      "step": 80
+    },
+    {
+      "epoch": 0.033938909962068275,
+      "grad_norm": 0.23240900039672852,
+      "learning_rate": 4.302158338901801e-05,
+      "loss": 0.0585,
+      "mean_token_accuracy": 0.9767142593860626,
+      "num_tokens": 12284271.0,
+      "step": 85
+    },
+    {
+      "epoch": 0.03593531643042523,
+      "grad_norm": 0.35469600558280945,
+      "learning_rate": 4.302085057800922e-05,
+      "loss": 0.0605,
+      "mean_token_accuracy": 0.9750521361827851,
+      "num_tokens": 13007855.0,
+      "step": 90
+    },
+    {
+      "epoch": 0.03793172289878219,
+      "grad_norm": 0.1484885960817337,
+      "learning_rate": 4.3020046633064446e-05,
+      "loss": 0.0484,
+      "mean_token_accuracy": 0.9796449035406113,
+      "num_tokens": 13722879.0,
+      "step": 95
+    },
+    {
+      "epoch": 0.03992812936713915,
+      "grad_norm": 0.23631541430950165,
+      "learning_rate": 4.3019171557729085e-05,
+      "loss": 0.059,
+      "mean_token_accuracy": 0.9753597110509873,
+      "num_tokens": 14450221.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.04192453583549611,
+      "grad_norm": 0.17381513118743896,
+      "learning_rate": 4.301822535586219e-05,
+      "loss": 0.053,
+      "mean_token_accuracy": 0.979637861251831,
+      "num_tokens": 15193852.0,
+      "step": 105
+    },
+    {
+      "epoch": 0.043920942303853065,
+      "grad_norm": 0.21713939309120178,
+      "learning_rate": 4.3017208031636505e-05,
+      "loss": 0.0521,
+      "mean_token_accuracy": 0.9779550462961197,
+      "num_tokens": 15920797.0,
+      "step": 110
+    },
+    {
+      "epoch": 0.04591734877221002,
+      "grad_norm": 0.21581998467445374,
+      "learning_rate": 4.301611958953839e-05,
+      "loss": 0.0614,
+      "mean_token_accuracy": 0.9744587689638138,
+      "num_tokens": 16647447.0,
+      "step": 115
+    },
+    {
+      "epoch": 0.04791375524056698,
+      "grad_norm": 0.19825442135334015,
+      "learning_rate": 4.301496003436785e-05,
+      "loss": 0.06,
+      "mean_token_accuracy": 0.9741028487682343,
+      "num_tokens": 17362711.0,
+      "step": 120
+    },
+    {
+      "epoch": 0.049910161708923935,
+      "grad_norm": 0.15317484736442566,
+      "learning_rate": 4.301372937123851e-05,
+      "loss": 0.0571,
+      "mean_token_accuracy": 0.9771378487348557,
+      "num_tokens": 18088486.0,
+      "step": 125
+    },
+    {
+      "epoch": 0.05190656817728089,
+      "grad_norm": 0.1307675838470459,
+      "learning_rate": 4.301242760557756e-05,
+      "loss": 0.0531,
+      "mean_token_accuracy": 0.9786365628242493,
+      "num_tokens": 18815433.0,
+      "step": 130
+    },
+    {
+      "epoch": 0.053902974645637855,
+      "grad_norm": 0.1721351146697998,
+      "learning_rate": 4.301105474312576e-05,
+      "loss": 0.0642,
+      "mean_token_accuracy": 0.973665738105774,
+      "num_tokens": 19530903.0,
+      "step": 135
+    },
+    {
+      "epoch": 0.05589938111399481,
+      "grad_norm": 0.2096928358078003,
+      "learning_rate": 4.30096107899374e-05,
+      "loss": 0.0548,
+      "mean_token_accuracy": 0.976664736866951,
+      "num_tokens": 20252280.0,
+      "step": 140
+    },
+    {
+      "epoch": 0.05789578758235177,
+      "grad_norm": 0.13824060559272766,
+      "learning_rate": 4.3008095752380274e-05,
+      "loss": 0.0548,
+      "mean_token_accuracy": 0.9776088953018188,
+      "num_tokens": 20966598.0,
+      "step": 145
+    },
+    {
+      "epoch": 0.059892194050708725,
+      "grad_norm": 0.1322195678949356,
+      "learning_rate": 4.3006509637135676e-05,
+      "loss": 0.0515,
+      "mean_token_accuracy": 0.9775060802698136,
+      "num_tokens": 21688849.0,
+      "step": 150
+    },
+    {
+      "epoch": 0.06188860051906568,
+      "grad_norm": 0.15910562872886658,
+      "learning_rate": 4.3004852451198345e-05,
+      "loss": 0.0533,
+      "mean_token_accuracy": 0.9776766180992127,
+      "num_tokens": 22414160.0,
+      "step": 155
+    },
+    {
+      "epoch": 0.06388500698742264,
+      "grad_norm": 0.17499415576457977,
+      "learning_rate": 4.300312420187642e-05,
+      "loss": 0.0543,
+      "mean_token_accuracy": 0.9780311793088913,
+      "num_tokens": 23140905.0,
+      "step": 160
+    },
+    {
+      "epoch": 0.0658814134557796,
+      "grad_norm": 0.08447716385126114,
+      "learning_rate": 4.300132489679144e-05,
+      "loss": 0.0586,
+      "mean_token_accuracy": 0.9748361557722092,
+      "num_tokens": 23858350.0,
+      "step": 165
+    },
+    {
+      "epoch": 0.06787781992413655,
+      "grad_norm": 0.1828007698059082,
+      "learning_rate": 4.299945454387832e-05,
+      "loss": 0.0582,
+      "mean_token_accuracy": 0.9753214448690415,
+      "num_tokens": 24578746.0,
+      "step": 170
+    },
+    {
+      "epoch": 0.06987422639249351,
+      "grad_norm": 0.12954303622245789,
+      "learning_rate": 4.299751315138525e-05,
+      "loss": 0.0588,
+      "mean_token_accuracy": 0.9755947411060333,
+      "num_tokens": 25290190.0,
+      "step": 175
+    },
+    {
+      "epoch": 0.07187063286085046,
+      "grad_norm": 0.17280645668506622,
+      "learning_rate": 4.299550072787375e-05,
+      "loss": 0.0522,
+      "mean_token_accuracy": 0.9782574236392975,
+      "num_tokens": 26013434.0,
+      "step": 180
+    },
+    {
+      "epoch": 0.07386703932920742,
+      "grad_norm": 0.27445533871650696,
+      "learning_rate": 4.2993417282218544e-05,
+      "loss": 0.0536,
+      "mean_token_accuracy": 0.9775886803865432,
+      "num_tokens": 26742045.0,
+      "step": 185
+    },
+    {
+      "epoch": 0.07586344579756438,
+      "grad_norm": 0.1881956309080124,
+      "learning_rate": 4.2991262823607586e-05,
+      "loss": 0.056,
+      "mean_token_accuracy": 0.976248922944069,
+      "num_tokens": 27466859.0,
+      "step": 190
+    },
+    {
+      "epoch": 0.07785985226592135,
+      "grad_norm": 0.15925268828868866,
+      "learning_rate": 4.2989037361542e-05,
+      "loss": 0.0486,
+      "mean_token_accuracy": 0.9797779828310013,
+      "num_tokens": 28194806.0,
+      "step": 195
+    },
+    {
+      "epoch": 0.0798562587342783,
+      "grad_norm": 0.2537769675254822,
+      "learning_rate": 4.2986740905836017e-05,
+      "loss": 0.0545,
+      "mean_token_accuracy": 0.9779761880636215,
+      "num_tokens": 28920879.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.08185266520263526,
+      "grad_norm": 0.18168781697750092,
+      "learning_rate": 4.298437346661695e-05,
+      "loss": 0.0516,
+      "mean_token_accuracy": 0.9782940000295639,
+      "num_tokens": 29652234.0,
+      "step": 205
+    },
+    {
+      "epoch": 0.08384907167099222,
+      "grad_norm": 0.34930142760276794,
+      "learning_rate": 4.2981935054325175e-05,
+      "loss": 0.0528,
+      "mean_token_accuracy": 0.9776461899280549,
+      "num_tokens": 30368616.0,
+      "step": 210
+    },
+    {
+      "epoch": 0.08584547813934917,
+      "grad_norm": 0.16720351576805115,
+      "learning_rate": 4.2979425679714015e-05,
+      "loss": 0.0545,
+      "mean_token_accuracy": 0.9776774793863297,
+      "num_tokens": 31086040.0,
+      "step": 215
+    },
+    {
+      "epoch": 0.08784188460770613,
+      "grad_norm": 0.12502144277095795,
+      "learning_rate": 4.2976845353849757e-05,
+      "loss": 0.0578,
+      "mean_token_accuracy": 0.9754776805639267,
+      "num_tokens": 31814127.0,
+      "step": 220
+    },
+    {
+      "epoch": 0.08983829107606309,
+      "grad_norm": 0.18173851072788239,
+      "learning_rate": 4.2974194088111596e-05,
+      "loss": 0.0549,
+      "mean_token_accuracy": 0.977699363231659,
+      "num_tokens": 32532899.0,
+      "step": 225
+    },
+    {
+      "epoch": 0.09183469754442004,
+      "grad_norm": 0.21796411275863647,
+      "learning_rate": 4.297147189419153e-05,
+      "loss": 0.0526,
+      "mean_token_accuracy": 0.9780656516551971,
+      "num_tokens": 33254862.0,
+      "step": 230
+    },
+    {
+      "epoch": 0.093831104012777,
+      "grad_norm": 0.158290296792984,
+      "learning_rate": 4.29686787840944e-05,
+      "loss": 0.0502,
+      "mean_token_accuracy": 0.9777401953935623,
+      "num_tokens": 33982840.0,
+      "step": 235
+    },
+    {
+      "epoch": 0.09582751048113396,
+      "grad_norm": 0.15952068567276,
+      "learning_rate": 4.2965814770137744e-05,
+      "loss": 0.0502,
+      "mean_token_accuracy": 0.978219535946846,
+      "num_tokens": 34703620.0,
+      "step": 240
+    },
+    {
+      "epoch": 0.09782391694949091,
+      "grad_norm": 0.1635027974843979,
+      "learning_rate": 4.2962879864951805e-05,
+      "loss": 0.0413,
+      "mean_token_accuracy": 0.982021626830101,
+      "num_tokens": 35440685.0,
+      "step": 245
+    },
+    {
+      "epoch": 0.09982032341784787,
+      "grad_norm": 0.13409432768821716,
+      "learning_rate": 4.2959874081479444e-05,
+      "loss": 0.047,
+      "mean_token_accuracy": 0.9794367611408233,
+      "num_tokens": 36169768.0,
+      "step": 250
+    },
+    {
+      "epoch": 0.10181672988620483,
+      "grad_norm": 0.11007480323314667,
+      "learning_rate": 4.295679743297611e-05,
+      "loss": 0.0473,
+      "mean_token_accuracy": 0.9799829065799713,
+      "num_tokens": 36903528.0,
+      "step": 255
+    },
+    {
+      "epoch": 0.10381313635456178,
+      "grad_norm": 0.1296682208776474,
+      "learning_rate": 4.295364993300975e-05,
+      "loss": 0.0567,
+      "mean_token_accuracy": 0.9754694372415542,
+      "num_tokens": 37627586.0,
+      "step": 260
+    },
+    {
+      "epoch": 0.10580954282291874,
+      "grad_norm": 0.12024891376495361,
+      "learning_rate": 4.295043159546076e-05,
+      "loss": 0.0535,
+      "mean_token_accuracy": 0.9779273867607117,
+      "num_tokens": 38352739.0,
+      "step": 265
+    },
+    {
+      "epoch": 0.10780594929127571,
+      "grad_norm": 0.09559287130832672,
+      "learning_rate": 4.294714243452194e-05,
+      "loss": 0.0437,
+      "mean_token_accuracy": 0.9822638213634491,
+      "num_tokens": 39085149.0,
+      "step": 270
+    },
+    {
+      "epoch": 0.10980235575963267,
+      "grad_norm": 0.19117465615272522,
+      "learning_rate": 4.2943782464698434e-05,
+      "loss": 0.0597,
+      "mean_token_accuracy": 0.975782060623169,
+      "num_tokens": 39805827.0,
+      "step": 275
+    },
+    {
+      "epoch": 0.11179876222798962,
+      "grad_norm": 0.10707342624664307,
+      "learning_rate": 4.294035170080761e-05,
+      "loss": 0.0567,
+      "mean_token_accuracy": 0.9762091904878616,
+      "num_tokens": 40530685.0,
+      "step": 280
+    },
+    {
+      "epoch": 0.11379516869634658,
+      "grad_norm": 0.08280737698078156,
+      "learning_rate": 4.2936850157979074e-05,
+      "loss": 0.05,
+      "mean_token_accuracy": 0.9792889416217804,
+      "num_tokens": 41255430.0,
+      "step": 285
+    },
+    {
+      "epoch": 0.11579157516470354,
+      "grad_norm": 0.1256619095802307,
+      "learning_rate": 4.2933277851654543e-05,
+      "loss": 0.0624,
+      "mean_token_accuracy": 0.9729855686426163,
+      "num_tokens": 41976078.0,
+      "step": 290
+    },
+    {
+      "epoch": 0.11778798163306049,
+      "grad_norm": 0.3224167227745056,
+      "learning_rate": 4.292963479758782e-05,
+      "loss": 0.0497,
+      "mean_token_accuracy": 0.9796846657991409,
+      "num_tokens": 42689118.0,
+      "step": 295
+    },
+    {
+      "epoch": 0.11978438810141745,
+      "grad_norm": 0.22343389689922333,
+      "learning_rate": 4.292592101184469e-05,
+      "loss": 0.0425,
+      "mean_token_accuracy": 0.9820985406637192,
+      "num_tokens": 43419497.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.1217807945697744,
+      "grad_norm": 0.10838128626346588,
+      "learning_rate": 4.292213651080286e-05,
+      "loss": 0.0454,
+      "mean_token_accuracy": 0.9795959055423736,
+      "num_tokens": 44144729.0,
+      "step": 305
+    },
+    {
+      "epoch": 0.12377720103813136,
+      "grad_norm": 0.1953747570514679,
+      "learning_rate": 4.2918281311151895e-05,
+      "loss": 0.046,
+      "mean_token_accuracy": 0.9807391226291656,
+      "num_tokens": 44869333.0,
+      "step": 310
+    },
+    {
+      "epoch": 0.12577360750648833,
+      "grad_norm": 0.14303229749202728,
+      "learning_rate": 4.291435542989315e-05,
+      "loss": 0.0477,
+      "mean_token_accuracy": 0.9790458828210831,
+      "num_tokens": 45601423.0,
+      "step": 315
+    },
+    {
+      "epoch": 0.12777001397484528,
+      "grad_norm": 0.13903263211250305,
+      "learning_rate": 4.291035888433965e-05,
+      "loss": 0.0486,
+      "mean_token_accuracy": 0.9791201978921891,
+      "num_tokens": 46331465.0,
+      "step": 320
+    },
+    {
+      "epoch": 0.12976642044320225,
+      "grad_norm": 0.1759800761938095,
+      "learning_rate": 4.2906291692116105e-05,
+      "loss": 0.0544,
+      "mean_token_accuracy": 0.9773983508348465,
+      "num_tokens": 47056177.0,
+      "step": 325
+    },
+    {
+      "epoch": 0.1317628269115592,
+      "grad_norm": 0.15405182540416718,
+      "learning_rate": 4.290215387115872e-05,
+      "loss": 0.0598,
+      "mean_token_accuracy": 0.9750259101390839,
+      "num_tokens": 47770978.0,
+      "step": 330
+    },
+    {
+      "epoch": 0.13375923337991616,
+      "grad_norm": 0.10629483312368393,
+      "learning_rate": 4.28979454397152e-05,
+      "loss": 0.0583,
+      "mean_token_accuracy": 0.9750337392091751,
+      "num_tokens": 48484907.0,
+      "step": 335
+    },
+    {
+      "epoch": 0.1357556398482731,
+      "grad_norm": 0.11660145968198776,
+      "learning_rate": 4.2893666416344625e-05,
+      "loss": 0.0524,
+      "mean_token_accuracy": 0.9774013131856918,
+      "num_tokens": 49205502.0,
+      "step": 340
+    },
+    {
+      "epoch": 0.13775204631663007,
+      "grad_norm": 0.1357419788837433,
+      "learning_rate": 4.288931681991741e-05,
+      "loss": 0.047,
+      "mean_token_accuracy": 0.9805745542049408,
+      "num_tokens": 49932020.0,
+      "step": 345
+    },
+    {
+      "epoch": 0.13974845278498702,
+      "grad_norm": 0.18027786910533905,
+      "learning_rate": 4.288489666961517e-05,
+      "loss": 0.0521,
+      "mean_token_accuracy": 0.9769169509410858,
+      "num_tokens": 50646806.0,
+      "step": 350
+    },
+    {
+      "epoch": 0.14174485925334399,
+      "grad_norm": 0.09403052926063538,
+      "learning_rate": 4.288040598493066e-05,
+      "loss": 0.052,
+      "mean_token_accuracy": 0.9778335362672805,
+      "num_tokens": 51368374.0,
+      "step": 355
+    },
+    {
+      "epoch": 0.14374126572170093,
+      "grad_norm": 0.10531823337078094,
+      "learning_rate": 4.287584478566772e-05,
+      "loss": 0.0475,
+      "mean_token_accuracy": 0.9801982522010804,
+      "num_tokens": 52082189.0,
+      "step": 360
+    },
+    {
+      "epoch": 0.1457376721900579,
+      "grad_norm": 0.08720994740724564,
+      "learning_rate": 4.2871213091941116e-05,
+      "loss": 0.0487,
+      "mean_token_accuracy": 0.9790406972169876,
+      "num_tokens": 52804371.0,
+      "step": 365
+    },
+    {
+      "epoch": 0.14773407865841484,
+      "grad_norm": 0.12209278345108032,
+      "learning_rate": 4.286651092417653e-05,
+      "loss": 0.0489,
+      "mean_token_accuracy": 0.9786169439554214,
+      "num_tokens": 53526886.0,
+      "step": 370
+    },
+    {
+      "epoch": 0.1497304851267718,
+      "grad_norm": 0.1360856592655182,
+      "learning_rate": 4.286173830311041e-05,
+      "loss": 0.0551,
+      "mean_token_accuracy": 0.9743523508310318,
+      "num_tokens": 54238089.0,
+      "step": 375
+    },
+    {
+      "epoch": 0.15172689159512875,
+      "grad_norm": 0.1499907374382019,
+      "learning_rate": 4.285689524978991e-05,
+      "loss": 0.0551,
+      "mean_token_accuracy": 0.9762009382247925,
+      "num_tokens": 54966900.0,
+      "step": 380
+    },
+    {
+      "epoch": 0.15372329806348572,
+      "grad_norm": 0.06996061652898788,
+      "learning_rate": 4.285198178557278e-05,
+      "loss": 0.0442,
+      "mean_token_accuracy": 0.9815765887498855,
+      "num_tokens": 55692180.0,
+      "step": 385
+    },
+    {
+      "epoch": 0.1557197045318427,
+      "grad_norm": 0.09069187194108963,
+      "learning_rate": 4.284699793212731e-05,
+      "loss": 0.0521,
+      "mean_token_accuracy": 0.9776074826717377,
+      "num_tokens": 56403496.0,
+      "step": 390
+    },
+    {
+      "epoch": 0.15771611100019964,
+      "grad_norm": 0.12922468781471252,
+      "learning_rate": 4.284194371143216e-05,
+      "loss": 0.0423,
+      "mean_token_accuracy": 0.9815239191055298,
+      "num_tokens": 57130425.0,
+      "step": 395
+    },
+    {
+      "epoch": 0.1597125174685566,
+      "grad_norm": 0.1596469283103943,
+      "learning_rate": 4.283681914577634e-05,
+      "loss": 0.0514,
+      "mean_token_accuracy": 0.9776736199855804,
+      "num_tokens": 57846781.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.16170892393691355,
+      "grad_norm": 0.1281159520149231,
+      "learning_rate": 4.283162425775907e-05,
+      "loss": 0.0475,
+      "mean_token_accuracy": 0.9804576992988586,
+      "num_tokens": 58576699.0,
+      "step": 405
+    },
+    {
+      "epoch": 0.16370533040527052,
+      "grad_norm": 0.18816827237606049,
+      "learning_rate": 4.282635907028968e-05,
+      "loss": 0.0491,
+      "mean_token_accuracy": 0.9774960219860077,
+      "num_tokens": 59299296.0,
+      "step": 410
+    },
+    {
+      "epoch": 0.16570173687362746,
+      "grad_norm": 0.1441081166267395,
+      "learning_rate": 4.282102360658755e-05,
+      "loss": 0.05,
+      "mean_token_accuracy": 0.9789826661348343,
+      "num_tokens": 60019189.0,
+      "step": 415
+    },
+    {
+      "epoch": 0.16769814334198443,
+      "grad_norm": 0.11921107023954391,
+      "learning_rate": 4.2815617890181955e-05,
+      "loss": 0.0535,
+      "mean_token_accuracy": 0.9768732756376266,
+      "num_tokens": 60742418.0,
+      "step": 420
+    },
+    {
+      "epoch": 0.16969454981034138,
+      "grad_norm": 0.14515455067157745,
+      "learning_rate": 4.281014194491196e-05,
+      "loss": 0.046,
+      "mean_token_accuracy": 0.9810800284147263,
+      "num_tokens": 61472413.0,
+      "step": 425
+    },
+    {
+      "epoch": 0.17169095627869835,
+      "grad_norm": 0.1529640555381775,
+      "learning_rate": 4.280459579492638e-05,
+      "loss": 0.0494,
+      "mean_token_accuracy": 0.9782395333051681,
+      "num_tokens": 62200026.0,
+      "step": 430
+    },
+    {
+      "epoch": 0.1736873627470553,
+      "grad_norm": 0.1236916184425354,
+      "learning_rate": 4.279897946468361e-05,
+      "loss": 0.0505,
+      "mean_token_accuracy": 0.978915536403656,
+      "num_tokens": 62919468.0,
+      "step": 435
+    },
+    {
+      "epoch": 0.17568376921541226,
+      "grad_norm": 0.1317927986383438,
+      "learning_rate": 4.2793292978951546e-05,
+      "loss": 0.0434,
+      "mean_token_accuracy": 0.9814162343740463,
+      "num_tokens": 63654252.0,
+      "step": 440
+    },
+    {
+      "epoch": 0.1776801756837692,
+      "grad_norm": 0.12080615013837814,
+      "learning_rate": 4.278753636280745e-05,
+      "loss": 0.0458,
+      "mean_token_accuracy": 0.9800752550363541,
+      "num_tokens": 64372681.0,
+      "step": 445
+    },
+    {
+      "epoch": 0.17967658215212617,
+      "grad_norm": 0.12699449062347412,
+      "learning_rate": 4.2781709641637895e-05,
+      "loss": 0.0557,
+      "mean_token_accuracy": 0.9758878946304321,
+      "num_tokens": 65084161.0,
+      "step": 450
+    },
+    {
+      "epoch": 0.18167298862048312,
+      "grad_norm": 0.10154750943183899,
+      "learning_rate": 4.277581284113856e-05,
+      "loss": 0.0468,
+      "mean_token_accuracy": 0.9790953069925308,
+      "num_tokens": 65814662.0,
+      "step": 455
+    },
+    {
+      "epoch": 0.1836693950888401,
+      "grad_norm": 0.14240600168704987,
+      "learning_rate": 4.276984598731423e-05,
+      "loss": 0.0469,
+      "mean_token_accuracy": 0.979536834359169,
+      "num_tokens": 66530926.0,
+      "step": 460
+    },
+    {
+      "epoch": 0.18566580155719706,
+      "grad_norm": 0.1021113321185112,
+      "learning_rate": 4.2763809106478584e-05,
+      "loss": 0.0425,
+      "mean_token_accuracy": 0.9820678532123566,
+      "num_tokens": 67265232.0,
+      "step": 465
+    },
+    {
+      "epoch": 0.187662208025554,
+      "grad_norm": 0.11665349453687668,
+      "learning_rate": 4.2757702225254146e-05,
+      "loss": 0.0566,
+      "mean_token_accuracy": 0.9761783987283706,
+      "num_tokens": 67983627.0,
+      "step": 470
+    },
+    {
+      "epoch": 0.18965861449391097,
+      "grad_norm": 0.12766194343566895,
+      "learning_rate": 4.2751525370572104e-05,
+      "loss": 0.046,
+      "mean_token_accuracy": 0.9804826110601426,
+      "num_tokens": 68713941.0,
+      "step": 475
+    },
+    {
+      "epoch": 0.1916550209622679,
+      "grad_norm": 0.11255398392677307,
+      "learning_rate": 4.274527856967227e-05,
+      "loss": 0.0549,
+      "mean_token_accuracy": 0.9781402915716171,
+      "num_tokens": 69417633.0,
+      "step": 480
+    },
+    {
+      "epoch": 0.19365142743062488,
+      "grad_norm": 0.08928117156028748,
+      "learning_rate": 4.273896185010289e-05,
+      "loss": 0.0509,
+      "mean_token_accuracy": 0.9780146718025208,
+      "num_tokens": 70130429.0,
+      "step": 485
+    },
+    {
+      "epoch": 0.19564783389898183,
+      "grad_norm": 0.13495980203151703,
+      "learning_rate": 4.273257523972056e-05,
+      "loss": 0.0451,
+      "mean_token_accuracy": 0.980745604634285,
+      "num_tokens": 70864111.0,
+      "step": 490
+    },
+    {
+      "epoch": 0.1976442403673388,
+      "grad_norm": 0.14102225005626678,
+      "learning_rate": 4.2726118766690076e-05,
+      "loss": 0.05,
+      "mean_token_accuracy": 0.979128223657608,
+      "num_tokens": 71589024.0,
+      "step": 495
+    },
+    {
+      "epoch": 0.19964064683569574,
+      "grad_norm": 0.12552976608276367,
+      "learning_rate": 4.271959245948436e-05,
+      "loss": 0.0514,
+      "mean_token_accuracy": 0.9778478264808654,
+      "num_tokens": 72306095.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.2016370533040527,
+      "grad_norm": 0.11777975410223007,
+      "learning_rate": 4.271299634688427e-05,
+      "loss": 0.0433,
+      "mean_token_accuracy": 0.9815191090106964,
+      "num_tokens": 73021001.0,
+      "step": 505
+    },
+    {
+      "epoch": 0.20363345977240965,
+      "grad_norm": 0.13376499712467194,
+      "learning_rate": 4.2706330457978514e-05,
+      "loss": 0.0503,
+      "mean_token_accuracy": 0.9786869913339615,
+      "num_tokens": 73750995.0,
+      "step": 510
+    },
+    {
+      "epoch": 0.20562986624076662,
+      "grad_norm": 0.10142122209072113,
+      "learning_rate": 4.269959482216352e-05,
+      "loss": 0.0455,
+      "mean_token_accuracy": 0.9812999993562699,
+      "num_tokens": 74474430.0,
+      "step": 515
+    },
+    {
+      "epoch": 0.20762627270912357,
+      "grad_norm": 0.1251404732465744,
+      "learning_rate": 4.26927894691433e-05,
+      "loss": 0.0433,
+      "mean_token_accuracy": 0.9818032503128051,
+      "num_tokens": 75193736.0,
+      "step": 520
+    },
+    {
+      "epoch": 0.20962267917748054,
+      "grad_norm": 0.09525765478610992,
+      "learning_rate": 4.268591442892929e-05,
+      "loss": 0.0467,
+      "mean_token_accuracy": 0.9807555198669433,
+      "num_tokens": 75927216.0,
+      "step": 525
+    },
+    {
+      "epoch": 0.21161908564583748,
+      "grad_norm": 0.09515196830034256,
+      "learning_rate": 4.267896973184027e-05,
+      "loss": 0.0447,
+      "mean_token_accuracy": 0.9803880721330642,
+      "num_tokens": 76655961.0,
+      "step": 530
+    },
+    {
+      "epoch": 0.21361549211419445,
+      "grad_norm": 0.09553889185190201,
+      "learning_rate": 4.267195540850221e-05,
+      "loss": 0.0505,
+      "mean_token_accuracy": 0.9782146096229554,
+      "num_tokens": 77379698.0,
+      "step": 535
+    },
+    {
+      "epoch": 0.21561189858255142,
+      "grad_norm": 0.09350269287824631,
+      "learning_rate": 4.266487148984811e-05,
+      "loss": 0.047,
+      "mean_token_accuracy": 0.9799423426389694,
+      "num_tokens": 78096171.0,
+      "step": 540
+    },
+    {
+      "epoch": 0.21760830505090836,
+      "grad_norm": 0.12444101274013519,
+      "learning_rate": 4.26577180071179e-05,
+      "loss": 0.0503,
+      "mean_token_accuracy": 0.9797710835933685,
+      "num_tokens": 78816545.0,
+      "step": 545
+    },
+    {
+      "epoch": 0.21960471151926533,
+      "grad_norm": 0.120224229991436,
+      "learning_rate": 4.265049499185827e-05,
+      "loss": 0.0476,
+      "mean_token_accuracy": 0.9791944712400437,
+      "num_tokens": 79543200.0,
+      "step": 550
+    },
+    {
+      "epoch": 0.22160111798762228,
+      "grad_norm": 0.08200144022703171,
+      "learning_rate": 4.2643202475922576e-05,
+      "loss": 0.0469,
+      "mean_token_accuracy": 0.9800705790519715,
+      "num_tokens": 80267808.0,
+      "step": 555
+    },
+    {
+      "epoch": 0.22359752445597925,
+      "grad_norm": 0.08492100983858109,
+      "learning_rate": 4.263584049147063e-05,
+      "loss": 0.053,
+      "mean_token_accuracy": 0.9771294265985488,
+      "num_tokens": 80995420.0,
+      "step": 560
+    },
+    {
+      "epoch": 0.2255939309243362,
+      "grad_norm": 0.12120378762483597,
+      "learning_rate": 4.2628409070968626e-05,
+      "loss": 0.047,
+      "mean_token_accuracy": 0.9804543852806091,
+      "num_tokens": 81723889.0,
+      "step": 565
+    },
+    {
+      "epoch": 0.22759033739269316,
+      "grad_norm": 0.10692359507083893,
+      "learning_rate": 4.2620908247188973e-05,
+      "loss": 0.0373,
+      "mean_token_accuracy": 0.9842275649309158,
+      "num_tokens": 82455710.0,
+      "step": 570
+    },
+    {
+      "epoch": 0.2295867438610501,
+      "grad_norm": 0.12830841541290283,
+      "learning_rate": 4.261333805321012e-05,
+      "loss": 0.0485,
+      "mean_token_accuracy": 0.9796034336090088,
+      "num_tokens": 83167210.0,
+      "step": 575
+    },
+    {
+      "epoch": 0.23158315032940707,
+      "grad_norm": 0.08063813298940659,
+      "learning_rate": 4.260569852241647e-05,
+      "loss": 0.0534,
+      "mean_token_accuracy": 0.9783318549394607,
+      "num_tokens": 83881816.0,
+      "step": 580
+    },
+    {
+      "epoch": 0.23357955679776402,
+      "grad_norm": 0.09200415760278702,
+      "learning_rate": 4.259798968849818e-05,
+      "loss": 0.0475,
+      "mean_token_accuracy": 0.980060550570488,
+      "num_tokens": 84607610.0,
+      "step": 585
+    },
+    {
+      "epoch": 0.23557596326612099,
+      "grad_norm": 0.09657726436853409,
+      "learning_rate": 4.259021158545102e-05,
+      "loss": 0.0534,
+      "mean_token_accuracy": 0.9768508076667786,
+      "num_tokens": 85315518.0,
+      "step": 590
+    },
+    {
+      "epoch": 0.23757236973447793,
+      "grad_norm": 0.09629404544830322,
+      "learning_rate": 4.2582364247576274e-05,
+      "loss": 0.0508,
+      "mean_token_accuracy": 0.9782411873340606,
+      "num_tokens": 86038324.0,
+      "step": 595
+    },
+    {
+      "epoch": 0.2395687762028349,
+      "grad_norm": 0.12929849326610565,
+      "learning_rate": 4.257444770948052e-05,
+      "loss": 0.0406,
+      "mean_token_accuracy": 0.9815934479236603,
+      "num_tokens": 86774837.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.24156518267119184,
+      "grad_norm": 0.10552412271499634,
+      "learning_rate": 4.256646200607551e-05,
+      "loss": 0.0533,
+      "mean_token_accuracy": 0.9769379138946533,
+      "num_tokens": 87493367.0,
+      "step": 605
+    },
+    {
+      "epoch": 0.2435615891395488,
+      "grad_norm": 0.10751223564147949,
+      "learning_rate": 4.255840717257802e-05,
+      "loss": 0.0475,
+      "mean_token_accuracy": 0.9801900714635849,
+      "num_tokens": 88212585.0,
+      "step": 610
+    },
+    {
+      "epoch": 0.24555799560790578,
+      "grad_norm": 0.08181528747081757,
+      "learning_rate": 4.255028324450969e-05,
+      "loss": 0.0483,
+      "mean_token_accuracy": 0.9792672157287597,
+      "num_tokens": 88921923.0,
+      "step": 615
+    },
+    {
+      "epoch": 0.24755440207626273,
+      "grad_norm": 0.11906848847866058,
+      "learning_rate": 4.254209025769686e-05,
+      "loss": 0.0531,
+      "mean_token_accuracy": 0.9773639112710952,
+      "num_tokens": 89635881.0,
+      "step": 620
+    },
+    {
+      "epoch": 0.2495508085446197,
+      "grad_norm": 0.09037422388792038,
+      "learning_rate": 4.253382824827042e-05,
+      "loss": 0.0494,
+      "mean_token_accuracy": 0.978655782341957,
+      "num_tokens": 90355134.0,
+      "step": 625
+    },
+    {
+      "epoch": 0.25154721501297667,
+      "grad_norm": 0.10041050612926483,
+      "learning_rate": 4.252549725266565e-05,
+      "loss": 0.0511,
+      "mean_token_accuracy": 0.9785868495702743,
+      "num_tokens": 91079403.0,
+      "step": 630
+    },
+    {
+      "epoch": 0.2535436214813336,
+      "grad_norm": 0.10690242052078247,
+      "learning_rate": 4.251709730762204e-05,
+      "loss": 0.0475,
+      "mean_token_accuracy": 0.9794499009847641,
+      "num_tokens": 91799983.0,
+      "step": 635
+    },
+    {
+      "epoch": 0.25554002794969055,
+      "grad_norm": 0.08654764294624329,
+      "learning_rate": 4.250862845018319e-05,
+      "loss": 0.0548,
+      "mean_token_accuracy": 0.9768476873636246,
+      "num_tokens": 92520960.0,
+      "step": 640
+    },
+    {
+      "epoch": 0.2575364344180475,
+      "grad_norm": 0.08682254701852798,
+      "learning_rate": 4.250009071769655e-05,
+      "loss": 0.043,
+      "mean_token_accuracy": 0.9821657538414001,
+      "num_tokens": 93247191.0,
+      "step": 645
+    },
+    {
+      "epoch": 0.2595328408864045,
+      "grad_norm": 0.1060463935136795,
+      "learning_rate": 4.2491484147813335e-05,
+      "loss": 0.0539,
+      "mean_token_accuracy": 0.9773279428482056,
+      "num_tokens": 93959062.0,
+      "step": 650
+    },
+    {
+      "epoch": 0.2615292473547614,
+      "grad_norm": 0.12649336457252502,
+      "learning_rate": 4.2482808778488325e-05,
+      "loss": 0.0497,
+      "mean_token_accuracy": 0.978519856929779,
+      "num_tokens": 94685020.0,
+      "step": 655
+    },
+    {
+      "epoch": 0.2635256538231184,
+      "grad_norm": 0.09485338628292084,
+      "learning_rate": 4.247406464797971e-05,
+      "loss": 0.0415,
+      "mean_token_accuracy": 0.9826800882816314,
+      "num_tokens": 95415113.0,
+      "step": 660
+    },
+    {
+      "epoch": 0.26552206029147535,
+      "grad_norm": 0.15739619731903076,
+      "learning_rate": 4.2465251794848915e-05,
+      "loss": 0.0461,
+      "mean_token_accuracy": 0.9801017135381699,
+      "num_tokens": 96135551.0,
+      "step": 665
+    },
+    {
+      "epoch": 0.2675184667598323,
+      "grad_norm": 0.07750169187784195,
+      "learning_rate": 4.245637025796042e-05,
+      "loss": 0.051,
+      "mean_token_accuracy": 0.9799440085887909,
+      "num_tokens": 96865552.0,
+      "step": 670
+    },
+    {
+      "epoch": 0.26951487322818923,
+      "grad_norm": 0.09401772171258926,
+      "learning_rate": 4.2447420076481604e-05,
+      "loss": 0.0466,
+      "mean_token_accuracy": 0.980428883433342,
+      "num_tokens": 97601834.0,
+      "step": 675
+    },
+    {
+      "epoch": 0.2715112796965462,
+      "grad_norm": 0.09588310122489929,
+      "learning_rate": 4.243840128988257e-05,
+      "loss": 0.0501,
+      "mean_token_accuracy": 0.979182380437851,
+      "num_tokens": 98332558.0,
+      "step": 680
+    },
+    {
+      "epoch": 0.2735076861649032,
+      "grad_norm": 0.09605035930871964,
+      "learning_rate": 4.2429313937935974e-05,
+      "loss": 0.0411,
+      "mean_token_accuracy": 0.9825141876935959,
+      "num_tokens": 99053033.0,
+      "step": 685
+    },
+    {
+      "epoch": 0.27550409263326014,
+      "grad_norm": 0.11407220363616943,
+      "learning_rate": 4.242015806071682e-05,
+      "loss": 0.0514,
+      "mean_token_accuracy": 0.977407693862915,
+      "num_tokens": 99777487.0,
+      "step": 690
+    },
+    {
+      "epoch": 0.2775004991016171,
+      "grad_norm": 0.10052948445081711,
+      "learning_rate": 4.241093369860234e-05,
+      "loss": 0.046,
+      "mean_token_accuracy": 0.9803170651197434,
+      "num_tokens": 100498662.0,
+      "step": 695
+    },
+    {
+      "epoch": 0.27949690556997403,
+      "grad_norm": 0.08085839450359344,
+      "learning_rate": 4.240164089227176e-05,
+      "loss": 0.0439,
+      "mean_token_accuracy": 0.9825191229581833,
+      "num_tokens": 101232839.0,
+      "step": 700
+    },
+    {
+      "epoch": 0.281493312038331,
+      "grad_norm": 0.08980093151330948,
+      "learning_rate": 4.2392279682706136e-05,
+      "loss": 0.051,
+      "mean_token_accuracy": 0.9787484467029571,
+      "num_tokens": 101948467.0,
+      "step": 705
+    },
+    {
+      "epoch": 0.28348971850668797,
+      "grad_norm": 0.19071196019649506,
+      "learning_rate": 4.2382850111188206e-05,
+      "loss": 0.0496,
+      "mean_token_accuracy": 0.9789243370294571,
+      "num_tokens": 102662104.0,
+      "step": 710
+    },
+    {
+      "epoch": 0.28548612497504494,
+      "grad_norm": 0.13858772814273834,
+      "learning_rate": 4.237335221930217e-05,
+      "loss": 0.084,
+      "mean_token_accuracy": 0.9728131413459777,
+      "num_tokens": 103381345.0,
+      "step": 715
+    },
+    {
+      "epoch": 0.28748253144340186,
+      "grad_norm": 4.443481922149658,
+      "learning_rate": 4.2363786048933525e-05,
+      "loss": 0.1013,
+      "mean_token_accuracy": 0.9748282253742218,
+      "num_tokens": 104104350.0,
+      "step": 720
+    },
+    {
+      "epoch": 0.2894789379117588,
+      "grad_norm": 0.45772096514701843,
+      "learning_rate": 4.2354151642268866e-05,
+      "loss": 0.103,
+      "mean_token_accuracy": 0.970191091299057,
+      "num_tokens": 104827954.0,
+      "step": 725
+    },
+    {
+      "epoch": 0.2914753443801158,
+      "grad_norm": 0.15262044966220856,
+      "learning_rate": 4.234444904179569e-05,
+      "loss": 0.0504,
+      "mean_token_accuracy": 0.9785544484853744,
+      "num_tokens": 105554646.0,
+      "step": 730
+    },
+    {
+      "epoch": 0.29347175084847277,
+      "grad_norm": 0.118346206843853,
+      "learning_rate": 4.233467829030229e-05,
+      "loss": 0.0502,
+      "mean_token_accuracy": 0.979259392619133,
+      "num_tokens": 106267067.0,
+      "step": 735
+    },
+    {
+      "epoch": 0.2954681573168297,
+      "grad_norm": 0.36983075737953186,
+      "learning_rate": 4.232483943087743e-05,
+      "loss": 0.0508,
+      "mean_token_accuracy": 0.9795638442039489,
+      "num_tokens": 107005215.0,
+      "step": 740
+    },
+    {
+      "epoch": 0.29746456378518665,
+      "grad_norm": 0.09854412823915482,
+      "learning_rate": 4.231493250691028e-05,
+      "loss": 0.0544,
+      "mean_token_accuracy": 0.9789934754371643,
+      "num_tokens": 107731301.0,
+      "step": 745
+    },
+    {
+      "epoch": 0.2994609702535436,
+      "grad_norm": 0.12738491594791412,
+      "learning_rate": 4.230495756209015e-05,
+      "loss": 0.0506,
+      "mean_token_accuracy": 0.979388102889061,
+      "num_tokens": 108450353.0,
+      "step": 750
+    },
+    {
+      "epoch": 0.3014573767219006,
+      "grad_norm": 0.15874311327934265,
+      "learning_rate": 4.229491464040631e-05,
+      "loss": 0.0465,
+      "mean_token_accuracy": 0.9797900915145874,
+      "num_tokens": 109180817.0,
+      "step": 755
+    },
+    {
+      "epoch": 0.3034537831902575,
+      "grad_norm": 0.11495564132928848,
+      "learning_rate": 4.228480378614784e-05,
+      "loss": 0.0487,
+      "mean_token_accuracy": 0.9790915966033935,
+      "num_tokens": 109903855.0,
+      "step": 760
+    },
+    {
+      "epoch": 0.3054501896586145,
+      "grad_norm": 0.10626127570867538,
+      "learning_rate": 4.227462504390338e-05,
+      "loss": 0.0508,
+      "mean_token_accuracy": 0.978978967666626,
+      "num_tokens": 110631560.0,
+      "step": 765
+    },
+    {
+      "epoch": 0.30744659612697145,
+      "grad_norm": 0.14091414213180542,
+      "learning_rate": 4.2264378458560945e-05,
+      "loss": 0.0496,
+      "mean_token_accuracy": 0.9777703315019608,
+      "num_tokens": 111349333.0,
+      "step": 770
+    },
+    {
+      "epoch": 0.3094430025953284,
+      "grad_norm": 0.14497868716716766,
+      "learning_rate": 4.225406407530776e-05,
+      "loss": 0.0535,
+      "mean_token_accuracy": 0.9772404164075852,
+      "num_tokens": 112073813.0,
+      "step": 775
+    },
+    {
+      "epoch": 0.3114394090636854,
+      "grad_norm": 0.08627774566411972,
+      "learning_rate": 4.224368193963002e-05,
+      "loss": 0.0486,
+      "mean_token_accuracy": 0.9789620161056518,
+      "num_tokens": 112785897.0,
+      "step": 780
+    },
+    {
+      "epoch": 0.3134358155320423,
+      "grad_norm": 0.09868915379047394,
+      "learning_rate": 4.223323209731272e-05,
+      "loss": 0.046,
+      "mean_token_accuracy": 0.9802429288625717,
+      "num_tokens": 113516646.0,
+      "step": 785
+    },
+    {
+      "epoch": 0.3154322220003993,
+      "grad_norm": 0.12950965762138367,
+      "learning_rate": 4.222271459443941e-05,
+      "loss": 0.0578,
+      "mean_token_accuracy": 0.9762585014104843,
+      "num_tokens": 114227110.0,
+      "step": 790
+    },
+    {
+      "epoch": 0.31742862846875625,
+      "grad_norm": 0.12167847901582718,
+      "learning_rate": 4.221212947739207e-05,
+      "loss": 0.0487,
+      "mean_token_accuracy": 0.9791981846094131,
+      "num_tokens": 114939687.0,
+      "step": 795
+    },
+    {
+      "epoch": 0.3194250349371132,
+      "grad_norm": 0.07483931630849838,
+      "learning_rate": 4.2201476792850825e-05,
+      "loss": 0.0449,
+      "mean_token_accuracy": 0.9802095115184783,
+      "num_tokens": 115657868.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.32142144140547013,
+      "grad_norm": 0.11723718047142029,
+      "learning_rate": 4.2190756587793766e-05,
+      "loss": 0.0451,
+      "mean_token_accuracy": 0.9806903630495072,
+      "num_tokens": 116383524.0,
+      "step": 805
+    },
+    {
+      "epoch": 0.3234178478738271,
+      "grad_norm": 0.0941113531589508,
+      "learning_rate": 4.217996890949677e-05,
+      "loss": 0.0417,
+      "mean_token_accuracy": 0.9814288556575775,
+      "num_tokens": 117097427.0,
+      "step": 810
+    },
+    {
+      "epoch": 0.3254142543421841,
+      "grad_norm": 0.08885961025953293,
+      "learning_rate": 4.216911380553325e-05,
+      "loss": 0.0482,
+      "mean_token_accuracy": 0.9794678062200546,
+      "num_tokens": 117814344.0,
+      "step": 815
+    },
+    {
+      "epoch": 0.32741066081054104,
+      "grad_norm": 0.08935556560754776,
+      "learning_rate": 4.2158191323773974e-05,
+      "loss": 0.0489,
+      "mean_token_accuracy": 0.9792770773172379,
+      "num_tokens": 118527584.0,
+      "step": 820
+    },
+    {
+      "epoch": 0.32940706727889796,
+      "grad_norm": 0.10772611200809479,
+      "learning_rate": 4.214720151238686e-05,
+      "loss": 0.0487,
+      "mean_token_accuracy": 0.9791043639183045,
+      "num_tokens": 119252755.0,
+      "step": 825
+    },
+    {
+      "epoch": 0.33140347374725493,
+      "grad_norm": 0.0967428907752037,
+      "learning_rate": 4.21361444198367e-05,
+      "loss": 0.048,
+      "mean_token_accuracy": 0.9792734861373902,
+      "num_tokens": 119986842.0,
+      "step": 830
+    },
+    {
+      "epoch": 0.3333998802156119,
+      "grad_norm": 0.06611617654561996,
+      "learning_rate": 4.212502009488505e-05,
+      "loss": 0.0413,
+      "mean_token_accuracy": 0.9816345155239106,
+      "num_tokens": 120729067.0,
+      "step": 835
+    },
+    {
+      "epoch": 0.33539628668396887,
+      "grad_norm": 0.07611052691936493,
+      "learning_rate": 4.211382858658993e-05,
+      "loss": 0.0448,
+      "mean_token_accuracy": 0.9809937328100204,
+      "num_tokens": 121455408.0,
+      "step": 840
+    },
+    {
+      "epoch": 0.33739269315232584,
+      "grad_norm": 0.09330687671899796,
+      "learning_rate": 4.210256994430565e-05,
+      "loss": 0.046,
+      "mean_token_accuracy": 0.9798675775527954,
+      "num_tokens": 122187578.0,
+      "step": 845
+    },
+    {
+      "epoch": 0.33938909962068275,
+      "grad_norm": 0.08592524379491806,
+      "learning_rate": 4.209124421768255e-05,
+      "loss": 0.0462,
+      "mean_token_accuracy": 0.9799050241708755,
+      "num_tokens": 122915846.0,
+      "step": 850
+    },
+    {
+      "epoch": 0.3413855060890397,
+      "grad_norm": 0.08495519310235977,
+      "learning_rate": 4.207985145666686e-05,
+      "loss": 0.0514,
+      "mean_token_accuracy": 0.9777328252792359,
+      "num_tokens": 123631558.0,
+      "step": 855
+    },
+    {
+      "epoch": 0.3433819125573967,
+      "grad_norm": 0.08593793958425522,
+      "learning_rate": 4.206839171150038e-05,
+      "loss": 0.051,
+      "mean_token_accuracy": 0.9783249199390411,
+      "num_tokens": 124349450.0,
+      "step": 860
+    },
+    {
+      "epoch": 0.34537831902575367,
+      "grad_norm": 0.1147531196475029,
+      "learning_rate": 4.2056865032720345e-05,
+      "loss": 0.0466,
+      "mean_token_accuracy": 0.9806852877140045,
+      "num_tokens": 125074783.0,
+      "step": 865
+    },
+    {
+      "epoch": 0.3473747254941106,
+      "grad_norm": 0.12205598503351212,
+      "learning_rate": 4.2045271471159146e-05,
+      "loss": 0.0512,
+      "mean_token_accuracy": 0.9779225885868073,
+      "num_tokens": 125790435.0,
+      "step": 870
+    },
+    {
+      "epoch": 0.34937113196246755,
+      "grad_norm": 0.08975517749786377,
+      "learning_rate": 4.2033611077944135e-05,
+      "loss": 0.0385,
+      "mean_token_accuracy": 0.9832260310649872,
+      "num_tokens": 126520091.0,
+      "step": 875
+    },
+    {
+      "epoch": 0.3513675384308245,
+      "grad_norm": 0.08298907428979874,
+      "learning_rate": 4.202188390449738e-05,
+      "loss": 0.0422,
+      "mean_token_accuracy": 0.9818367391824723,
+      "num_tokens": 127243561.0,
+      "step": 880
+    },
+    {
+      "epoch": 0.3533639448991815,
+      "grad_norm": 0.08749652653932571,
+      "learning_rate": 4.2010090002535467e-05,
+      "loss": 0.0431,
+      "mean_token_accuracy": 0.9816054284572602,
+      "num_tokens": 127979804.0,
+      "step": 885
+    },
+    {
+      "epoch": 0.3553603513675384,
+      "grad_norm": 0.10358402132987976,
+      "learning_rate": 4.199822942406923e-05,
+      "loss": 0.058,
+      "mean_token_accuracy": 0.9758305728435517,
+      "num_tokens": 128699074.0,
+      "step": 890
+    },
+    {
+      "epoch": 0.3573567578358954,
+      "grad_norm": 0.11131475865840912,
+      "learning_rate": 4.198630222140356e-05,
+      "loss": 0.0522,
+      "mean_token_accuracy": 0.9782269984483719,
+      "num_tokens": 129412928.0,
+      "step": 895
+    },
+    {
+      "epoch": 0.35935316430425235,
+      "grad_norm": 0.07659720629453659,
+      "learning_rate": 4.197430844713716e-05,
+      "loss": 0.0503,
+      "mean_token_accuracy": 0.9786783039569855,
+      "num_tokens": 130133772.0,
+      "step": 900
+    },
+    {
+      "epoch": 0.3613495707726093,
+      "grad_norm": 0.11637573689222336,
+      "learning_rate": 4.1962248154162294e-05,
+      "loss": 0.0511,
+      "mean_token_accuracy": 0.9778418868780137,
+      "num_tokens": 130848319.0,
+      "step": 905
+    },
+    {
+      "epoch": 0.36334597724096623,
+      "grad_norm": 0.10833151638507843,
+      "learning_rate": 4.19501213956646e-05,
+      "loss": 0.0504,
+      "mean_token_accuracy": 0.9788612067699433,
+      "num_tokens": 131575172.0,
+      "step": 910
+    },
+    {
+      "epoch": 0.3653423837093232,
+      "grad_norm": 0.07982037216424942,
+      "learning_rate": 4.193792822512281e-05,
+      "loss": 0.0385,
+      "mean_token_accuracy": 0.9827957212924957,
+      "num_tokens": 132312213.0,
+      "step": 915
+    },
+    {
+      "epoch": 0.3673387901776802,
+      "grad_norm": 0.07653510570526123,
+      "learning_rate": 4.192566869630854e-05,
+      "loss": 0.0504,
+      "mean_token_accuracy": 0.978272557258606,
+      "num_tokens": 133035272.0,
+      "step": 920
+    },
+    {
+      "epoch": 0.36933519664603714,
+      "grad_norm": 0.06979475170373917,
+      "learning_rate": 4.191334286328602e-05,
+      "loss": 0.0436,
+      "mean_token_accuracy": 0.982032772898674,
+      "num_tokens": 133762349.0,
+      "step": 925
+    },
+    {
+      "epoch": 0.3713316031143941,
+      "grad_norm": 0.12790125608444214,
+      "learning_rate": 4.190095078041192e-05,
+      "loss": 0.0483,
+      "mean_token_accuracy": 0.9800712436437606,
+      "num_tokens": 134495209.0,
+      "step": 930
+    },
+    {
+      "epoch": 0.37332800958275103,
+      "grad_norm": 0.11297880858182907,
+      "learning_rate": 4.188849250233504e-05,
+      "loss": 0.0436,
+      "mean_token_accuracy": 0.9812392681837082,
+      "num_tokens": 135215123.0,
+      "step": 935
+    },
+    {
+      "epoch": 0.375324416051108,
+      "grad_norm": 0.10094229876995087,
+      "learning_rate": 4.1875968083996106e-05,
+      "loss": 0.0466,
+      "mean_token_accuracy": 0.9796725332736969,
+      "num_tokens": 135940243.0,
+      "step": 940
+    },
+    {
+      "epoch": 0.37732082251946497,
+      "grad_norm": 0.13492746651172638,
+      "learning_rate": 4.186337758062754e-05,
+      "loss": 0.0482,
+      "mean_token_accuracy": 0.9795497447252274,
+      "num_tokens": 136658519.0,
+      "step": 945
+    },
+    {
+      "epoch": 0.37931722898782194,
+      "grad_norm": 0.13884606957435608,
+      "learning_rate": 4.185072104775316e-05,
+      "loss": 0.0479,
+      "mean_token_accuracy": 0.97914038002491,
+      "num_tokens": 137384244.0,
+      "step": 950
+    },
+    {
+      "epoch": 0.38131363545617886,
+      "grad_norm": 0.10219676047563553,
+      "learning_rate": 4.183799854118801e-05,
+      "loss": 0.0509,
+      "mean_token_accuracy": 0.9788306951522827,
+      "num_tokens": 138107181.0,
+      "step": 955
+    },
+    {
+      "epoch": 0.3833100419245358,
+      "grad_norm": 0.10559409856796265,
+      "learning_rate": 4.1825210117038054e-05,
+      "loss": 0.0414,
+      "mean_token_accuracy": 0.9829078048467637,
+      "num_tokens": 138827438.0,
+      "step": 960
+    },
+    {
+      "epoch": 0.3853064483928928,
+      "grad_norm": 0.07543619722127914,
+      "learning_rate": 4.181235583169997e-05,
+      "loss": 0.0384,
+      "mean_token_accuracy": 0.983868607878685,
+      "num_tokens": 139563400.0,
+      "step": 965
+    },
+    {
+      "epoch": 0.38730285486124977,
+      "grad_norm": 0.08595782518386841,
+      "learning_rate": 4.179943574186085e-05,
+      "loss": 0.0475,
+      "mean_token_accuracy": 0.9790930688381195,
+      "num_tokens": 140283779.0,
+      "step": 970
+    },
+    {
+      "epoch": 0.3892992613296067,
+      "grad_norm": 0.08979545533657074,
+      "learning_rate": 4.178644990449802e-05,
+      "loss": 0.0481,
+      "mean_token_accuracy": 0.97889164686203,
+      "num_tokens": 141018466.0,
+      "step": 975
+    },
+    {
+      "epoch": 0.39129566779796365,
+      "grad_norm": 0.11952298134565353,
+      "learning_rate": 4.177339837687873e-05,
+      "loss": 0.0527,
+      "mean_token_accuracy": 0.9777220457792282,
+      "num_tokens": 141730333.0,
+      "step": 980
+    },
+    {
+      "epoch": 0.3932920742663206,
+      "grad_norm": 0.08510354906320572,
+      "learning_rate": 4.1760281216559906e-05,
+      "loss": 0.0456,
+      "mean_token_accuracy": 0.9804279118776321,
+      "num_tokens": 142456339.0,
+      "step": 985
+    },
+    {
+      "epoch": 0.3952884807346776,
+      "grad_norm": 0.09229864180088043,
+      "learning_rate": 4.174709848138797e-05,
+      "loss": 0.0492,
+      "mean_token_accuracy": 0.9791429519653321,
+      "num_tokens": 143186609.0,
+      "step": 990
+    },
+    {
+      "epoch": 0.39728488720303456,
+      "grad_norm": 0.2251962572336197,
+      "learning_rate": 4.173385022949847e-05,
+      "loss": 0.049,
+      "mean_token_accuracy": 0.9791528820991516,
+      "num_tokens": 143911008.0,
+      "step": 995
+    },
+    {
+      "epoch": 0.3992812936713915,
+      "grad_norm": 0.08839277178049088,
+      "learning_rate": 4.17205365193159e-05,
+      "loss": 0.0438,
+      "mean_token_accuracy": 0.9804649889469147,
+      "num_tokens": 144637806.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.40127770013974845,
+      "grad_norm": 0.08728329837322235,
+      "learning_rate": 4.170715740955343e-05,
+      "loss": 0.0441,
+      "mean_token_accuracy": 0.9821651369333267,
+      "num_tokens": 145370197.0,
+      "step": 1005
+    },
+    {
+      "epoch": 0.4032741066081054,
+      "grad_norm": 0.06770054250955582,
+      "learning_rate": 4.169371295921263e-05,
+      "loss": 0.0467,
+      "mean_token_accuracy": 0.979771289229393,
+      "num_tokens": 146089153.0,
+      "step": 1010
+    },
+    {
+      "epoch": 0.4052705130764624,
+      "grad_norm": 0.08927344530820847,
+      "learning_rate": 4.168020322758324e-05,
+      "loss": 0.0453,
+      "mean_token_accuracy": 0.981076043844223,
+      "num_tokens": 146814893.0,
+      "step": 1015
+    },
+    {
+      "epoch": 0.4072669195448193,
+      "grad_norm": 0.07860483974218369,
+      "learning_rate": 4.166662827424287e-05,
+      "loss": 0.0459,
+      "mean_token_accuracy": 0.9803457111120224,
+      "num_tokens": 147538626.0,
+      "step": 1020
+    },
+    {
+      "epoch": 0.4092633260131763,
+      "grad_norm": 0.11667665839195251,
+      "learning_rate": 4.165298815905676e-05,
+      "loss": 0.0437,
+      "mean_token_accuracy": 0.9804121345281601,
+      "num_tokens": 148265722.0,
+      "step": 1025
+    },
+    {
+      "epoch": 0.41125973248153325,
+      "grad_norm": 0.09357365220785141,
+      "learning_rate": 4.163928294217751e-05,
+      "loss": 0.0456,
+      "mean_token_accuracy": 0.9804595738649369,
+      "num_tokens": 149001428.0,
+      "step": 1030
+    },
+    {
+      "epoch": 0.4132561389498902,
+      "grad_norm": 0.073987677693367,
+      "learning_rate": 4.162551268404484e-05,
+      "loss": 0.0405,
+      "mean_token_accuracy": 0.9826260536909104,
+      "num_tokens": 149741467.0,
+      "step": 1035
+    },
+    {
+      "epoch": 0.41525254541824713,
+      "grad_norm": 0.11584579944610596,
+      "learning_rate": 4.161167744538526e-05,
+      "loss": 0.05,
+      "mean_token_accuracy": 0.9772036731243133,
+      "num_tokens": 150482060.0,
+      "step": 1040
+    },
+    {
+      "epoch": 0.4172489518866041,
+      "grad_norm": 0.08994850516319275,
+      "learning_rate": 4.159777728721188e-05,
+      "loss": 0.0438,
+      "mean_token_accuracy": 0.9802421748638153,
+      "num_tokens": 151207290.0,
+      "step": 1045
+    },
+    {
+      "epoch": 0.4192453583549611,
+      "grad_norm": 0.06240738928318024,
+      "learning_rate": 4.158381227082408e-05,
+      "loss": 0.0373,
+      "mean_token_accuracy": 0.9844264835119247,
+      "num_tokens": 151948018.0,
+      "step": 1050
+    },
+    {
+      "epoch": 0.42124176482331804,
+      "grad_norm": 0.08201232552528381,
+      "learning_rate": 4.1569782457807275e-05,
+      "loss": 0.0498,
+      "mean_token_accuracy": 0.9776235729455948,
+      "num_tokens": 152662400.0,
+      "step": 1055
+    },
+    {
+      "epoch": 0.42323817129167496,
+      "grad_norm": 0.16132792830467224,
+      "learning_rate": 4.155568791003262e-05,
+      "loss": 0.0462,
+      "mean_token_accuracy": 0.9798346608877182,
+      "num_tokens": 153382255.0,
+      "step": 1060
+    },
+    {
+      "epoch": 0.42523457776003193,
+      "grad_norm": 0.12460800260305405,
+      "learning_rate": 4.154152868965677e-05,
+      "loss": 0.0453,
+      "mean_token_accuracy": 0.9792943328619004,
+      "num_tokens": 154107481.0,
+      "step": 1065
+    },
+    {
+      "epoch": 0.4272309842283889,
+      "grad_norm": 0.08316458016633987,
+      "learning_rate": 4.152730485912157e-05,
+      "loss": 0.0421,
+      "mean_token_accuracy": 0.9824769467115402,
+      "num_tokens": 154810936.0,
+      "step": 1070
+    },
+    {
+      "epoch": 0.42922739069674587,
+      "grad_norm": 0.09391912817955017,
+      "learning_rate": 4.15130164811538e-05,
+      "loss": 0.0405,
+      "mean_token_accuracy": 0.9815963953733444,
+      "num_tokens": 155537857.0,
+      "step": 1075
+    },
+    {
+      "epoch": 0.43122379716510284,
+      "grad_norm": 0.05126833915710449,
+      "learning_rate": 4.149866361876487e-05,
+      "loss": 0.041,
+      "mean_token_accuracy": 0.9826298028230667,
+      "num_tokens": 156251631.0,
+      "step": 1080
+    },
+    {
+      "epoch": 0.43322020363345976,
+      "grad_norm": 0.05620391294360161,
+      "learning_rate": 4.1484246335250596e-05,
+      "loss": 0.0424,
+      "mean_token_accuracy": 0.9817048490047455,
+      "num_tokens": 156978372.0,
+      "step": 1085
+    },
+    {
+      "epoch": 0.4352166101018167,
+      "grad_norm": 0.08917587250471115,
+      "learning_rate": 4.146976469419087e-05,
+      "loss": 0.0431,
+      "mean_token_accuracy": 0.9803878009319306,
+      "num_tokens": 157708445.0,
+      "step": 1090
+    },
+    {
+      "epoch": 0.4372130165701737,
+      "grad_norm": 0.09579002112150192,
+      "learning_rate": 4.145521875944943e-05,
+      "loss": 0.043,
+      "mean_token_accuracy": 0.9806773096323014,
+      "num_tokens": 158431867.0,
+      "step": 1095
+    },
+    {
+      "epoch": 0.43920942303853067,
+      "grad_norm": 0.08360281586647034,
+      "learning_rate": 4.14406085951735e-05,
+      "loss": 0.0478,
+      "mean_token_accuracy": 0.9791438162326813,
+      "num_tokens": 159157025.0,
+      "step": 1100
+    },
+    {
+      "epoch": 0.4412058295068876,
+      "grad_norm": 0.11798866093158722,
+      "learning_rate": 4.142593426579359e-05,
+      "loss": 0.0492,
+      "mean_token_accuracy": 0.9781560689210892,
+      "num_tokens": 159878274.0,
+      "step": 1105
+    },
+    {
+      "epoch": 0.44320223597524455,
+      "grad_norm": 0.07982130348682404,
+      "learning_rate": 4.1411195836023166e-05,
+      "loss": 0.042,
+      "mean_token_accuracy": 0.9814552158117295,
+      "num_tokens": 160594976.0,
+      "step": 1110
+    },
+    {
+      "epoch": 0.4451986424436015,
+      "grad_norm": 0.08481776714324951,
+      "learning_rate": 4.139639337085837e-05,
+      "loss": 0.0531,
+      "mean_token_accuracy": 0.9776395380496978,
+      "num_tokens": 161305109.0,
+      "step": 1115
+    },
+    {
+      "epoch": 0.4471950489119585,
+      "grad_norm": 0.08173475414514542,
+      "learning_rate": 4.138152693557775e-05,
+      "loss": 0.0382,
+      "mean_token_accuracy": 0.983448275923729,
+      "num_tokens": 162027934.0,
+      "step": 1120
+    },
+    {
+      "epoch": 0.4491914553803154,
+      "grad_norm": 0.0663527399301529,
+      "learning_rate": 4.136659659574195e-05,
+      "loss": 0.0474,
+      "mean_token_accuracy": 0.9785608291625977,
+      "num_tokens": 162742550.0,
+      "step": 1125
+    },
+    {
+      "epoch": 0.4511878618486724,
+      "grad_norm": 0.10211747139692307,
+      "learning_rate": 4.1351602417193445e-05,
+      "loss": 0.0439,
+      "mean_token_accuracy": 0.9815158486366272,
+      "num_tokens": 163461292.0,
+      "step": 1130
+    },
+    {
+      "epoch": 0.45318426831702935,
+      "grad_norm": 0.08372879028320312,
+      "learning_rate": 4.133654446605622e-05,
+      "loss": 0.0455,
+      "mean_token_accuracy": 0.9801951259374618,
+      "num_tokens": 164176893.0,
+      "step": 1135
+    },
+    {
+      "epoch": 0.4551806747853863,
+      "grad_norm": 0.09567493945360184,
+      "learning_rate": 4.1321422808735505e-05,
+      "loss": 0.0441,
+      "mean_token_accuracy": 0.9820766746997833,
+      "num_tokens": 164909213.0,
+      "step": 1140
+    },
+    {
+      "epoch": 0.4571770812537433,
+      "grad_norm": 0.10552221536636353,
+      "learning_rate": 4.1306237511917476e-05,
+      "loss": 0.038,
+      "mean_token_accuracy": 0.9834578722715378,
+      "num_tokens": 165628392.0,
+      "step": 1145
+    },
+    {
+      "epoch": 0.4591734877221002,
+      "grad_norm": 0.13832643628120422,
+      "learning_rate": 4.129098864256896e-05,
+      "loss": 0.0477,
+      "mean_token_accuracy": 0.9799725502729416,
+      "num_tokens": 166348245.0,
+      "step": 1150
+    },
+    {
+      "epoch": 0.4611698941904572,
+      "grad_norm": 0.0993349552154541,
+      "learning_rate": 4.127567626793712e-05,
+      "loss": 0.0478,
+      "mean_token_accuracy": 0.9796446442604065,
+      "num_tokens": 167080014.0,
+      "step": 1155
+    },
+    {
+      "epoch": 0.46316630065881415,
+      "grad_norm": 0.0669255256652832,
+      "learning_rate": 4.126030045554919e-05,
+      "loss": 0.0413,
+      "mean_token_accuracy": 0.9815053075551987,
+      "num_tokens": 167806975.0,
+      "step": 1160
+    },
+    {
+      "epoch": 0.4651627071271711,
+      "grad_norm": 0.059534091502428055,
+      "learning_rate": 4.1244861273212167e-05,
+      "loss": 0.0482,
+      "mean_token_accuracy": 0.9796302795410157,
+      "num_tokens": 168521774.0,
+      "step": 1165
+    },
+    {
+      "epoch": 0.46715911359552803,
+      "grad_norm": 0.1345037966966629,
+      "learning_rate": 4.122935878901249e-05,
+      "loss": 0.0427,
+      "mean_token_accuracy": 0.9822476923465728,
+      "num_tokens": 169240278.0,
+      "step": 1170
+    },
+    {
+      "epoch": 0.469155520063885,
+      "grad_norm": 0.09438323974609375,
+      "learning_rate": 4.1213793071315786e-05,
+      "loss": 0.0421,
+      "mean_token_accuracy": 0.9817219495773315,
+      "num_tokens": 169969278.0,
+      "step": 1175
+    },
+    {
+      "epoch": 0.47115192653224197,
+      "grad_norm": 0.09831586480140686,
+      "learning_rate": 4.1198164188766506e-05,
+      "loss": 0.0415,
+      "mean_token_accuracy": 0.9811917096376419,
+      "num_tokens": 170694898.0,
+      "step": 1180
+    },
+    {
+      "epoch": 0.47314833300059894,
+      "grad_norm": 0.10349370539188385,
+      "learning_rate": 4.11824722102877e-05,
+      "loss": 0.0447,
+      "mean_token_accuracy": 0.9804620027542115,
+      "num_tokens": 171421794.0,
+      "step": 1185
+    },
+    {
+      "epoch": 0.47514473946895586,
+      "grad_norm": 0.07256416231393814,
+      "learning_rate": 4.11667172050806e-05,
+      "loss": 0.0504,
+      "mean_token_accuracy": 0.9791789770126342,
+      "num_tokens": 172134789.0,
+      "step": 1190
+    },
+    {
+      "epoch": 0.4771411459373128,
+      "grad_norm": 0.08856100589036942,
+      "learning_rate": 4.115089924262448e-05,
+      "loss": 0.0524,
+      "mean_token_accuracy": 0.9780719757080079,
+      "num_tokens": 172843426.0,
+      "step": 1195
+    },
+    {
+      "epoch": 0.4791375524056698,
+      "grad_norm": 0.06250528991222382,
+      "learning_rate": 4.113501839267616e-05,
+      "loss": 0.0427,
+      "mean_token_accuracy": 0.9813639312982559,
+      "num_tokens": 173581290.0,
+      "step": 1200
+    },
+    {
+      "epoch": 0.48113395887402677,
+      "grad_norm": 0.14334551990032196,
+      "learning_rate": 4.111907472526987e-05,
+      "loss": 0.0479,
+      "mean_token_accuracy": 0.978913527727127,
+      "num_tokens": 174301236.0,
+      "step": 1205
+    },
+    {
+      "epoch": 0.4831303653423837,
+      "grad_norm": 0.09056934714317322,
+      "learning_rate": 4.11030683107168e-05,
+      "loss": 0.0448,
+      "mean_token_accuracy": 0.9799034684896469,
+      "num_tokens": 175031206.0,
+      "step": 1210
+    },
+    {
+      "epoch": 0.48512677181074065,
+      "grad_norm": 0.08644259721040726,
+      "learning_rate": 4.10869992196049e-05,
+      "loss": 0.048,
+      "mean_token_accuracy": 0.9799766391515732,
+      "num_tokens": 175748771.0,
+      "step": 1215
+    },
+    {
+      "epoch": 0.4871231782790976,
+      "grad_norm": 0.06306669861078262,
+      "learning_rate": 4.107086752279849e-05,
+      "loss": 0.0393,
+      "mean_token_accuracy": 0.9826879143714905,
+      "num_tokens": 176482262.0,
+      "step": 1220
+    },
+    {
+      "epoch": 0.4891195847474546,
+      "grad_norm": 0.12182262539863586,
+      "learning_rate": 4.1054673291438014e-05,
+      "loss": 0.0431,
+      "mean_token_accuracy": 0.9814256817102432,
+      "num_tokens": 177204272.0,
+      "step": 1225
+    },
+    {
+      "epoch": 0.49111599121581156,
+      "grad_norm": 0.10804204642772675,
+      "learning_rate": 4.1038416596939646e-05,
+      "loss": 0.0455,
+      "mean_token_accuracy": 0.9802686542272567,
+      "num_tokens": 177923746.0,
+      "step": 1230
+    },
+    {
+      "epoch": 0.4931123976841685,
+      "grad_norm": 0.08272804319858551,
+      "learning_rate": 4.1022097510995075e-05,
+      "loss": 0.0463,
+      "mean_token_accuracy": 0.9803291648626328,
+      "num_tokens": 178646214.0,
+      "step": 1235
+    },
+    {
+      "epoch": 0.49510880415252545,
+      "grad_norm": 0.08051120489835739,
+      "learning_rate": 4.1005716105571086e-05,
+      "loss": 0.0406,
+      "mean_token_accuracy": 0.9827998161315918,
+      "num_tokens": 179361163.0,
+      "step": 1240
+    },
+    {
+      "epoch": 0.4971052106208824,
+      "grad_norm": 0.07553274929523468,
+      "learning_rate": 4.0989272452909324e-05,
+      "loss": 0.058,
+      "mean_token_accuracy": 0.9754190474748612,
+      "num_tokens": 180077232.0,
+      "step": 1245
+    },
+    {
+      "epoch": 0.4991016170892394,
+      "grad_norm": 0.07851612567901611,
+      "learning_rate": 4.097276662552592e-05,
+      "loss": 0.0525,
+      "mean_token_accuracy": 0.9774915635585785,
+      "num_tokens": 180800542.0,
+      "step": 1250
+    },
+    {
+      "epoch": 0.5010980235575964,
+      "grad_norm": 0.09802480787038803,
+      "learning_rate": 4.095619869621122e-05,
+      "loss": 0.0403,
+      "mean_token_accuracy": 0.9810875445604325,
+      "num_tokens": 181523489.0,
+      "step": 1255
+    },
+    {
+      "epoch": 0.5030944300259533,
+      "grad_norm": 0.07821514457464218,
+      "learning_rate": 4.093956873802941e-05,
+      "loss": 0.0418,
+      "mean_token_accuracy": 0.981935253739357,
+      "num_tokens": 182252055.0,
+      "step": 1260
+    },
+    {
+      "epoch": 0.5050908364943102,
+      "grad_norm": 0.07127466797828674,
+      "learning_rate": 4.092287682431825e-05,
+      "loss": 0.0468,
+      "mean_token_accuracy": 0.9807193636894226,
+      "num_tokens": 182978186.0,
+      "step": 1265
+    },
+    {
+      "epoch": 0.5070872429626672,
+      "grad_norm": 0.09551110118627548,
+      "learning_rate": 4.0906123028688695e-05,
+      "loss": 0.0412,
+      "mean_token_accuracy": 0.9831303149461746,
+      "num_tokens": 183702199.0,
+      "step": 1270
+    },
+    {
+      "epoch": 0.5090836494310241,
+      "grad_norm": 0.3869798481464386,
+      "learning_rate": 4.088930742502462e-05,
+      "loss": 0.0405,
+      "mean_token_accuracy": 0.9830111473798752,
+      "num_tokens": 184428583.0,
+      "step": 1275
+    },
+    {
+      "epoch": 0.5110800558993811,
+      "grad_norm": 0.09185118973255157,
+      "learning_rate": 4.087243008748246e-05,
+      "loss": 0.048,
+      "mean_token_accuracy": 0.9796857953071594,
+      "num_tokens": 185144789.0,
+      "step": 1280
+    },
+    {
+      "epoch": 0.5130764623677381,
+      "grad_norm": 0.09394432604312897,
+      "learning_rate": 4.085549109049089e-05,
+      "loss": 0.0415,
+      "mean_token_accuracy": 0.982088240981102,
+      "num_tokens": 185873049.0,
+      "step": 1285
+    },
+    {
+      "epoch": 0.515072868836095,
+      "grad_norm": 0.1030714362859726,
+      "learning_rate": 4.083849050875053e-05,
+      "loss": 0.0494,
+      "mean_token_accuracy": 0.9779442816972732,
+      "num_tokens": 186592320.0,
+      "step": 1290
+    },
+    {
+      "epoch": 0.517069275304452,
+      "grad_norm": 0.11340459436178207,
+      "learning_rate": 4.082142841723355e-05,
+      "loss": 0.0441,
+      "mean_token_accuracy": 0.9800026804208756,
+      "num_tokens": 187318624.0,
+      "step": 1295
+    },
+    {
+      "epoch": 0.519065681772809,
+      "grad_norm": 0.09892936795949936,
+      "learning_rate": 4.080430489118339e-05,
+      "loss": 0.0503,
+      "mean_token_accuracy": 0.9789821773767471,
+      "num_tokens": 188042877.0,
+      "step": 1300
+    },
+    {
+      "epoch": 0.521062088241166,
+      "grad_norm": 0.08861280232667923,
+      "learning_rate": 4.0787120006114435e-05,
+      "loss": 0.0455,
+      "mean_token_accuracy": 0.980542853474617,
+      "num_tokens": 188765158.0,
+      "step": 1305
+    },
+    {
+      "epoch": 0.5230584947095228,
+      "grad_norm": 0.06914465874433517,
+      "learning_rate": 4.076987383781164e-05,
+      "loss": 0.0427,
+      "mean_token_accuracy": 0.9810645192861557,
+      "num_tokens": 189480211.0,
+      "step": 1310
+    },
+    {
+      "epoch": 0.5250549011778798,
+      "grad_norm": 0.07803887873888016,
+      "learning_rate": 4.075256646233022e-05,
+      "loss": 0.0477,
+      "mean_token_accuracy": 0.9788318306207657,
+      "num_tokens": 190199406.0,
+      "step": 1315
+    },
+    {
+      "epoch": 0.5270513076462368,
+      "grad_norm": 0.08014380186796188,
+      "learning_rate": 4.073519795599532e-05,
+      "loss": 0.0425,
+      "mean_token_accuracy": 0.9804257869720459,
+      "num_tokens": 190934259.0,
+      "step": 1320
+    },
+    {
+      "epoch": 0.5290477141145937,
+      "grad_norm": 0.11968166381120682,
+      "learning_rate": 4.071776839540166e-05,
+      "loss": 0.0421,
+      "mean_token_accuracy": 0.9820073306560516,
+      "num_tokens": 191651169.0,
+      "step": 1325
+    },
+    {
+      "epoch": 0.5310441205829507,
+      "grad_norm": 0.11703905463218689,
+      "learning_rate": 4.0700277857413225e-05,
+      "loss": 0.0515,
+      "mean_token_accuracy": 0.9788607507944107,
+      "num_tokens": 192367327.0,
+      "step": 1330
+    },
+    {
+      "epoch": 0.5330405270513077,
+      "grad_norm": 0.09883023053407669,
+      "learning_rate": 4.0682726419162894e-05,
+      "loss": 0.0509,
+      "mean_token_accuracy": 0.9784328937530518,
+      "num_tokens": 193093958.0,
+      "step": 1335
+    },
+    {
+      "epoch": 0.5350369335196646,
+      "grad_norm": 0.07223407179117203,
+      "learning_rate": 4.0665114158052116e-05,
+      "loss": 0.0465,
+      "mean_token_accuracy": 0.9799530595541001,
+      "num_tokens": 193823330.0,
+      "step": 1340
+    },
+    {
+      "epoch": 0.5370333399880216,
+      "grad_norm": 0.0919746458530426,
+      "learning_rate": 4.0647441151750566e-05,
+      "loss": 0.044,
+      "mean_token_accuracy": 0.981145116686821,
+      "num_tokens": 194553648.0,
+      "step": 1345
+    },
+    {
+      "epoch": 0.5390297464563785,
+      "grad_norm": 0.07844650745391846,
+      "learning_rate": 4.062970747819582e-05,
+      "loss": 0.0379,
+      "mean_token_accuracy": 0.9831986397504806,
+      "num_tokens": 195270519.0,
+      "step": 1350
+    },
+    {
+      "epoch": 0.5410261529247354,
+      "grad_norm": 0.09186799079179764,
+      "learning_rate": 4.061191321559298e-05,
+      "loss": 0.0417,
+      "mean_token_accuracy": 0.9822646319866181,
+      "num_tokens": 195994606.0,
+      "step": 1355
+    },
+    {
+      "epoch": 0.5430225593930924,
+      "grad_norm": 0.08641962707042694,
+      "learning_rate": 4.0594058442414344e-05,
+      "loss": 0.0485,
+      "mean_token_accuracy": 0.9796275526285172,
+      "num_tokens": 196717658.0,
+      "step": 1360
+    },
+    {
+      "epoch": 0.5450189658614494,
+      "grad_norm": 0.08504263311624527,
+      "learning_rate": 4.0576143237399066e-05,
+      "loss": 0.0538,
+      "mean_token_accuracy": 0.9767540097236633,
+      "num_tokens": 197424512.0,
+      "step": 1365
+    },
+    {
+      "epoch": 0.5470153723298063,
+      "grad_norm": 0.12139827013015747,
+      "learning_rate": 4.055816767955281e-05,
+      "loss": 0.0489,
+      "mean_token_accuracy": 0.9788673222064972,
+      "num_tokens": 198158443.0,
+      "step": 1370
+    },
+    {
+      "epoch": 0.5490117787981633,
+      "grad_norm": 0.08939183503389359,
+      "learning_rate": 4.054013184814738e-05,
+      "loss": 0.0479,
+      "mean_token_accuracy": 0.9796740025281906,
+      "num_tokens": 198875760.0,
+      "step": 1375
+    },
+    {
+      "epoch": 0.5510081852665203,
+      "grad_norm": 0.07606673240661621,
+      "learning_rate": 4.052203582272039e-05,
+      "loss": 0.0435,
+      "mean_token_accuracy": 0.9809995740652084,
+      "num_tokens": 199601401.0,
+      "step": 1380
+    },
+    {
+      "epoch": 0.5530045917348773,
+      "grad_norm": 0.09385755658149719,
+      "learning_rate": 4.05038796830749e-05,
+      "loss": 0.0414,
+      "mean_token_accuracy": 0.9815231949090958,
+      "num_tokens": 200336207.0,
+      "step": 1385
+    },
+    {
+      "epoch": 0.5550009982032342,
+      "grad_norm": 0.10867508500814438,
+      "learning_rate": 4.048566350927911e-05,
+      "loss": 0.0484,
+      "mean_token_accuracy": 0.9797090023756028,
+      "num_tokens": 201057681.0,
+      "step": 1390
+    },
+    {
+      "epoch": 0.5569974046715911,
+      "grad_norm": 0.10143137723207474,
+      "learning_rate": 4.0467387381665907e-05,
+      "loss": 0.0562,
+      "mean_token_accuracy": 0.9756572872400284,
+      "num_tokens": 201776862.0,
+      "step": 1395
+    },
+    {
+      "epoch": 0.5589938111399481,
+      "grad_norm": 0.10667003691196442,
+      "learning_rate": 4.044905138083263e-05,
+      "loss": 0.0463,
+      "mean_token_accuracy": 0.9798533678054809,
+      "num_tokens": 202492946.0,
+      "step": 1400
+    },
+    {
+      "epoch": 0.560990217608305,
+      "grad_norm": 0.07327716052532196,
+      "learning_rate": 4.0430655587640636e-05,
+      "loss": 0.0466,
+      "mean_token_accuracy": 0.9800977617502212,
+      "num_tokens": 203220129.0,
+      "step": 1405
+    },
+    {
+      "epoch": 0.562986624076662,
+      "grad_norm": 0.0714830830693245,
+      "learning_rate": 4.041220008321495e-05,
+      "loss": 0.0412,
+      "mean_token_accuracy": 0.9820256412029267,
+      "num_tokens": 203950333.0,
+      "step": 1410
+    },
+    {
+      "epoch": 0.564983030545019,
+      "grad_norm": 0.08165636658668518,
+      "learning_rate": 4.039368494894394e-05,
+      "loss": 0.0484,
+      "mean_token_accuracy": 0.9786020785570144,
+      "num_tokens": 204665955.0,
+      "step": 1415
+    },
+    {
+      "epoch": 0.5669794370133759,
+      "grad_norm": 0.0785563662648201,
+      "learning_rate": 4.0375110266478943e-05,
+      "loss": 0.0439,
+      "mean_token_accuracy": 0.9812197864055634,
+      "num_tokens": 205388850.0,
+      "step": 1420
+    },
+    {
+      "epoch": 0.5689758434817329,
+      "grad_norm": 0.1614316999912262,
+      "learning_rate": 4.0356476117733887e-05,
+      "loss": 0.0431,
+      "mean_token_accuracy": 0.981093293428421,
+      "num_tokens": 206109175.0,
+      "step": 1425
+    },
+    {
+      "epoch": 0.5709722499500899,
+      "grad_norm": 0.09200597554445267,
+      "learning_rate": 4.033778258488496e-05,
+      "loss": 0.0488,
+      "mean_token_accuracy": 0.9786974668502808,
+      "num_tokens": 206830366.0,
+      "step": 1430
+    },
+    {
+      "epoch": 0.5729686564184467,
+      "grad_norm": 0.08983796834945679,
+      "learning_rate": 4.031902975037023e-05,
+      "loss": 0.0464,
+      "mean_token_accuracy": 0.9802830934524536,
+      "num_tokens": 207549827.0,
+      "step": 1435
+    },
+    {
+      "epoch": 0.5749650628868037,
+      "grad_norm": 0.07297132909297943,
+      "learning_rate": 4.030021769688926e-05,
+      "loss": 0.0458,
+      "mean_token_accuracy": 0.9810631215572357,
+      "num_tokens": 208264895.0,
+      "step": 1440
+    },
+    {
+      "epoch": 0.5769614693551607,
+      "grad_norm": 0.0987749695777893,
+      "learning_rate": 4.028134650740281e-05,
+      "loss": 0.046,
+      "mean_token_accuracy": 0.981097549200058,
+      "num_tokens": 208997043.0,
+      "step": 1445
+    },
+    {
+      "epoch": 0.5789578758235177,
+      "grad_norm": 0.09348510205745697,
+      "learning_rate": 4.026241626513239e-05,
+      "loss": 0.048,
+      "mean_token_accuracy": 0.9792329996824265,
+      "num_tokens": 209725133.0,
+      "step": 1450
+    },
+    {
+      "epoch": 0.5809542822918746,
+      "grad_norm": 0.092249795794487,
+      "learning_rate": 4.024342705355996e-05,
+      "loss": 0.0401,
+      "mean_token_accuracy": 0.982272133231163,
+      "num_tokens": 210465190.0,
+      "step": 1455
+    },
+    {
+      "epoch": 0.5829506887602316,
+      "grad_norm": 0.09076627343893051,
+      "learning_rate": 4.022437895642752e-05,
+      "loss": 0.0412,
+      "mean_token_accuracy": 0.9813473641872406,
+      "num_tokens": 211191938.0,
+      "step": 1460
+    },
+    {
+      "epoch": 0.5849470952285886,
+      "grad_norm": 0.10411449521780014,
+      "learning_rate": 4.020527205773674e-05,
+      "loss": 0.0468,
+      "mean_token_accuracy": 0.9788929164409638,
+      "num_tokens": 211915014.0,
+      "step": 1465
+    },
+    {
+      "epoch": 0.5869435016969455,
+      "grad_norm": 0.08393626660108566,
+      "learning_rate": 4.0186106441748635e-05,
+      "loss": 0.0405,
+      "mean_token_accuracy": 0.9831263810396195,
+      "num_tokens": 212639731.0,
+      "step": 1470
+    },
+    {
+      "epoch": 0.5889399081653025,
+      "grad_norm": 0.12234524637460709,
+      "learning_rate": 4.016688219298313e-05,
+      "loss": 0.0441,
+      "mean_token_accuracy": 0.9801553189754486,
+      "num_tokens": 213354106.0,
+      "step": 1475
+    },
+    {
+      "epoch": 0.5909363146336594,
+      "grad_norm": 0.19468314945697784,
+      "learning_rate": 4.014759939621874e-05,
+      "loss": 0.0527,
+      "mean_token_accuracy": 0.978853377699852,
+      "num_tokens": 214069931.0,
+      "step": 1480
+    },
+    {
+      "epoch": 0.5929327211020163,
+      "grad_norm": 0.07654144614934921,
+      "learning_rate": 4.012825813649218e-05,
+      "loss": 0.0447,
+      "mean_token_accuracy": 0.9811176121234894,
+      "num_tokens": 214801602.0,
+      "step": 1485
+    },
+    {
+      "epoch": 0.5949291275703733,
+      "grad_norm": 0.08197729289531708,
+      "learning_rate": 4.0108858499097965e-05,
+      "loss": 0.0436,
+      "mean_token_accuracy": 0.9811263650655746,
+      "num_tokens": 215525744.0,
+      "step": 1490
+    },
+    {
+      "epoch": 0.5969255340387303,
+      "grad_norm": 0.09118011593818665,
+      "learning_rate": 4.008940056958806e-05,
+      "loss": 0.0453,
+      "mean_token_accuracy": 0.9803959131240845,
+      "num_tokens": 216250126.0,
+      "step": 1495
+    },
+    {
+      "epoch": 0.5989219405070872,
+      "grad_norm": 0.12226779013872147,
+      "learning_rate": 4.006988443377151e-05,
+      "loss": 0.0447,
+      "mean_token_accuracy": 0.9816070884466171,
+      "num_tokens": 216985436.0,
+      "step": 1500
+    },
+    {
+      "epoch": 0.6009183469754442,
+      "grad_norm": 0.07873986661434174,
+      "learning_rate": 4.0050310177714044e-05,
+      "loss": 0.048,
+      "mean_token_accuracy": 0.978963452577591,
+      "num_tokens": 217695643.0,
+      "step": 1505
+    },
+    {
+      "epoch": 0.6029147534438012,
+      "grad_norm": 0.10712873935699463,
+      "learning_rate": 4.0030677887737685e-05,
+      "loss": 0.0464,
+      "mean_token_accuracy": 0.9787706166505814,
+      "num_tokens": 218425340.0,
+      "step": 1510
+    },
+    {
+      "epoch": 0.6049111599121582,
+      "grad_norm": 0.06610714644193649,
+      "learning_rate": 4.00109876504204e-05,
+      "loss": 0.0412,
+      "mean_token_accuracy": 0.9825782984495163,
+      "num_tokens": 219145127.0,
+      "step": 1515
+    },
+    {
+      "epoch": 0.606907566380515,
+      "grad_norm": 0.1649412214756012,
+      "learning_rate": 3.99912395525957e-05,
+      "loss": 0.0451,
+      "mean_token_accuracy": 0.9816054254770279,
+      "num_tokens": 219875374.0,
+      "step": 1520
+    },
+    {
+      "epoch": 0.608903972848872,
+      "grad_norm": 0.09525147825479507,
+      "learning_rate": 3.997143368135225e-05,
+      "loss": 0.0456,
+      "mean_token_accuracy": 0.9800587654113769,
+      "num_tokens": 220598857.0,
+      "step": 1525
+    },
+    {
+      "epoch": 0.610900379317229,
+      "grad_norm": 0.08931232988834381,
+      "learning_rate": 3.99515701240335e-05,
+      "loss": 0.0416,
+      "mean_token_accuracy": 0.9814909934997559,
+      "num_tokens": 221323896.0,
+      "step": 1530
+    },
+    {
+      "epoch": 0.6128967857855859,
+      "grad_norm": 0.07063008844852448,
+      "learning_rate": 3.993164896823731e-05,
+      "loss": 0.0459,
+      "mean_token_accuracy": 0.979477858543396,
+      "num_tokens": 222047987.0,
+      "step": 1535
+    },
+    {
+      "epoch": 0.6148931922539429,
+      "grad_norm": 0.06190071254968643,
+      "learning_rate": 3.991167030181552e-05,
+      "loss": 0.0487,
+      "mean_token_accuracy": 0.9796573609113693,
+      "num_tokens": 222776362.0,
+      "step": 1540
+    },
+    {
+      "epoch": 0.6168895987222999,
+      "grad_norm": 0.07348193973302841,
+      "learning_rate": 3.989163421287362e-05,
+      "loss": 0.0444,
+      "mean_token_accuracy": 0.9801139563322068,
+      "num_tokens": 223501685.0,
+      "step": 1545
+    },
+    {
+      "epoch": 0.6188860051906568,
+      "grad_norm": 0.09044871479272842,
+      "learning_rate": 3.987154078977029e-05,
+      "loss": 0.0432,
+      "mean_token_accuracy": 0.9818701952695846,
+      "num_tokens": 224224314.0,
+      "step": 1550
+    },
+    {
+      "epoch": 0.6208824116590138,
+      "grad_norm": 0.0657450407743454,
+      "learning_rate": 3.985139012111711e-05,
+      "loss": 0.0383,
+      "mean_token_accuracy": 0.983964490890503,
+      "num_tokens": 224952530.0,
+      "step": 1555
+    },
+    {
+      "epoch": 0.6228788181273708,
+      "grad_norm": 0.10462448000907898,
+      "learning_rate": 3.9831182295778054e-05,
+      "loss": 0.0464,
+      "mean_token_accuracy": 0.9802037477493286,
+      "num_tokens": 225670583.0,
+      "step": 1560
+    },
+    {
+      "epoch": 0.6248752245957276,
+      "grad_norm": 0.10816384106874466,
+      "learning_rate": 3.98109174028692e-05,
+      "loss": 0.0431,
+      "mean_token_accuracy": 0.9816061168909073,
+      "num_tokens": 226405728.0,
+      "step": 1565
+    },
+    {
+      "epoch": 0.6268716310640846,
+      "grad_norm": 0.07350475341081619,
+      "learning_rate": 3.979059553175826e-05,
+      "loss": 0.0421,
+      "mean_token_accuracy": 0.9821370601654053,
+      "num_tokens": 227131753.0,
+      "step": 1570
+    },
+    {
+      "epoch": 0.6288680375324416,
+      "grad_norm": 0.1497100442647934,
+      "learning_rate": 3.9770216772064254e-05,
+      "loss": 0.0493,
+      "mean_token_accuracy": 0.979113084077835,
+      "num_tokens": 227854770.0,
+      "step": 1575
+    },
+    {
+      "epoch": 0.6308644440007986,
+      "grad_norm": 0.08558464050292969,
+      "learning_rate": 3.9749781213657046e-05,
+      "loss": 0.041,
+      "mean_token_accuracy": 0.9826665192842483,
+      "num_tokens": 228579635.0,
+      "step": 1580
+    },
+    {
+      "epoch": 0.6328608504691555,
+      "grad_norm": 0.0670614242553711,
+      "learning_rate": 3.972928894665699e-05,
+      "loss": 0.0421,
+      "mean_token_accuracy": 0.9823820263147354,
+      "num_tokens": 229300584.0,
+      "step": 1585
+    },
+    {
+      "epoch": 0.6348572569375125,
+      "grad_norm": 0.10730074346065521,
+      "learning_rate": 3.970874006143454e-05,
+      "loss": 0.0436,
+      "mean_token_accuracy": 0.9802110999822616,
+      "num_tokens": 230025120.0,
+      "step": 1590
+    },
+    {
+      "epoch": 0.6368536634058695,
+      "grad_norm": 0.06655887514352798,
+      "learning_rate": 3.9688134648609804e-05,
+      "loss": 0.0445,
+      "mean_token_accuracy": 0.9799957931041717,
+      "num_tokens": 230748778.0,
+      "step": 1595
+    },
+    {
+      "epoch": 0.6388500698742264,
+      "grad_norm": 0.058215994387865067,
+      "learning_rate": 3.966747279905222e-05,
+      "loss": 0.0402,
+      "mean_token_accuracy": 0.98234643638134,
+      "num_tokens": 231469920.0,
+      "step": 1600
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 7515,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.333398891906728e+19,
+  "train_batch_size": 6,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:66b47870715eafe562564242b5ff263e1f21a27c754d332754f22fd64b5427dc
+size 6481

vocab.json ADDED Viewed

The diff for this file is too large to render. See raw diff