M2: GK SFT LoRA v2 — MMLU + TriviaQA + NQ

README.md

@@ -1,14 +1,14 @@
 ---
 library_name: transformers
-model_name: gk_checkpoint_lora
+model_name: gk_checkpoint_lora_v2
 tags:
 - generated_from_trainer
-- sft
 - trl
+- sft
 licence: license
 ---
 
-# Model Card for gk_checkpoint_lora
+# Model Card for gk_checkpoint_lora_v2
 
 This model is a fine-tuned version of [None](https://huggingface.co/None).
 It has been trained using [TRL](https://github.com/huggingface/trl).

chat_template.jinja

@@ -1,4 +1,4 @@
-{%- set enable_thinking = false %}
+{%- set enable_thinking = true %}
 {%- set enable_thinking = false %}
 {%- if tools %}
     {{- '<|im_start|>system\n' }}

checkpoint-3000/README.md

@@ -0,0 +1,209 @@
+---
+base_model: /scratch/Qwen3-1.7B
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/scratch/Qwen3-1.7B
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-3000/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/scratch/Qwen3-1.7B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "down_proj",
+    "q_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-3000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ced8e4e368b7dbb85c814d4dc89ec0dbc291f368f78ef490fdb3c0a3726fc4ee
+size 278973888

checkpoint-3000/chat_template.jinja

@@ -0,0 +1,93 @@
+{%- set enable_thinking = true %}
+{%- set enable_thinking = false %}
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant. Always put your final answer in \\boxed{}. For multiple choice questions, put only the letter in \\boxed{}, for example \\boxed{A}.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

checkpoint-3000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e244c64408b92e0b79dad7661cea93aea40fdb0d456b2a8390148693b53fc8d
+size 558179131

checkpoint-3000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a63d48e176b48da79e455dbd0f16b6e831ec597b719604fc3aa8c2060655154
+size 14645

checkpoint-3000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b12cadf1e04fe2efb486148284543f9ed099b936d6ec87dd2b9705f224d2ba4
+size 1465

checkpoint-3000/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

checkpoint-3000/tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-3000/trainer_state.json

@@ -0,0 +1,799 @@
+{
+  "best_global_step": 3000,
+  "best_metric": 0.43371766805648804,
+  "best_model_checkpoint": "/scratch/gk_checkpoint_lora_v2/checkpoint-3000",
+  "epoch": 0.9051821679112921,
+  "eval_steps": 200,
+  "global_step": 3000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.0372698324173688,
+      "epoch": 0.015086369465188203,
+      "grad_norm": 0.4944687783718109,
+      "learning_rate": 3.266666666666667e-05,
+      "loss": 2.1209957885742186,
+      "mean_token_accuracy": 0.6639667785167694,
+      "num_tokens": 213967.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.4767542722821236,
+      "epoch": 0.030172738930376405,
+      "grad_norm": 0.29953643679618835,
+      "learning_rate": 6.6e-05,
+      "loss": 0.5209395980834961,
+      "mean_token_accuracy": 0.9028549310564995,
+      "num_tokens": 426365.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.46151453502476214,
+      "epoch": 0.04525910839556461,
+      "grad_norm": 0.3118787109851837,
+      "learning_rate": 9.933333333333334e-05,
+      "loss": 0.508198356628418,
+      "mean_token_accuracy": 0.9032981966435909,
+      "num_tokens": 642837.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.4455497920885682,
+      "epoch": 0.06034547786075281,
+      "grad_norm": 0.3013271391391754,
+      "learning_rate": 9.999382532513122e-05,
+      "loss": 0.48826019287109373,
+      "mean_token_accuracy": 0.907553653717041,
+      "num_tokens": 856475.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.06034547786075281,
+      "eval_entropy": 0.4689600637588114,
+      "eval_loss": 0.4816349446773529,
+      "eval_mean_token_accuracy": 0.8987587762934696,
+      "eval_num_tokens": 856475.0,
+      "eval_runtime": 37.5206,
+      "eval_samples_per_second": 57.728,
+      "eval_steps_per_second": 7.223,
+      "step": 200
+    },
+    {
+      "entropy": 0.44372758489102127,
+      "epoch": 0.075431847325941,
+      "grad_norm": 0.31373220682144165,
+      "learning_rate": 9.997479627263544e-05,
+      "loss": 0.4843710327148438,
+      "mean_token_accuracy": 0.9082412907481193,
+      "num_tokens": 1069370.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.4475720078870654,
+      "epoch": 0.09051821679112922,
+      "grad_norm": 0.27380964159965515,
+      "learning_rate": 9.994291516446573e-05,
+      "loss": 0.491109733581543,
+      "mean_token_accuracy": 0.9055162121355533,
+      "num_tokens": 1286132.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.4463552813604474,
+      "epoch": 0.10560458625631741,
+      "grad_norm": 0.2614763677120209,
+      "learning_rate": 9.989819019951048e-05,
+      "loss": 0.4837772369384766,
+      "mean_token_accuracy": 0.9081570096313953,
+      "num_tokens": 1500851.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.41013720393180847,
+      "epoch": 0.12069095572150562,
+      "grad_norm": 0.4836612045764923,
+      "learning_rate": 9.984063287972232e-05,
+      "loss": 0.44385364532470706,
+      "mean_token_accuracy": 0.9148843766748905,
+      "num_tokens": 1708807.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.12069095572150562,
+      "eval_entropy": 0.4811463643044123,
+      "eval_loss": 0.4747391641139984,
+      "eval_mean_token_accuracy": 0.8997697566268189,
+      "eval_num_tokens": 1708807.0,
+      "eval_runtime": 37.1123,
+      "eval_samples_per_second": 58.363,
+      "eval_steps_per_second": 7.302,
+      "step": 400
+    },
+    {
+      "entropy": 0.42200190499424933,
+      "epoch": 0.13577732518669383,
+      "grad_norm": 0.19339531660079956,
+      "learning_rate": 9.977025800716017e-05,
+      "loss": 0.45712459564208985,
+      "mean_token_accuracy": 0.9106362241506577,
+      "num_tokens": 1923184.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.43271509755402804,
+      "epoch": 0.150863694651882,
+      "grad_norm": 0.299110472202301,
+      "learning_rate": 9.968708368018253e-05,
+      "loss": 0.4724855422973633,
+      "mean_token_accuracy": 0.9078708891570568,
+      "num_tokens": 2139609.0,
+      "step": 500
+    },
+    {
+      "entropy": 0.43638833791017534,
+      "epoch": 0.16595006411707022,
+      "grad_norm": 0.2539425492286682,
+      "learning_rate": 9.959113128879322e-05,
+      "loss": 0.4785395050048828,
+      "mean_token_accuracy": 0.907249256670475,
+      "num_tokens": 2358080.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.43115664307028057,
+      "epoch": 0.18103643358225843,
+      "grad_norm": 0.2542003393173218,
+      "learning_rate": 9.948242550914035e-05,
+      "loss": 0.4740608215332031,
+      "mean_token_accuracy": 0.9089943794906139,
+      "num_tokens": 2574667.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.18103643358225843,
+      "eval_entropy": 0.45346878287537074,
+      "eval_loss": 0.46328845620155334,
+      "eval_mean_token_accuracy": 0.9025413253210568,
+      "eval_num_tokens": 2574667.0,
+      "eval_runtime": 37.1254,
+      "eval_samples_per_second": 58.343,
+      "eval_steps_per_second": 7.3,
+      "step": 600
+    },
+    {
+      "entropy": 0.4420048241317272,
+      "epoch": 0.19612280304744664,
+      "grad_norm": 0.26832085847854614,
+      "learning_rate": 9.936099429717045e-05,
+      "loss": 0.486652717590332,
+      "mean_token_accuracy": 0.9074076810479164,
+      "num_tokens": 2790489.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.4269510039314628,
+      "epoch": 0.21120917251263482,
+      "grad_norm": 0.20635050535202026,
+      "learning_rate": 9.922686888143897e-05,
+      "loss": 0.4619187927246094,
+      "mean_token_accuracy": 0.910810690075159,
+      "num_tokens": 3003881.0,
+      "step": 700
+    },
+    {
+      "entropy": 0.42016164746135476,
+      "epoch": 0.22629554197782303,
+      "grad_norm": 0.2643264830112457,
+      "learning_rate": 9.908008375507924e-05,
+      "loss": 0.46344844818115233,
+      "mean_token_accuracy": 0.9113752076029777,
+      "num_tokens": 3218446.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.3974369211867452,
+      "epoch": 0.24138191144301124,
+      "grad_norm": 0.19392798840999603,
+      "learning_rate": 9.89206766669318e-05,
+      "loss": 0.42646697998046873,
+      "mean_token_accuracy": 0.9165593402087688,
+      "num_tokens": 3428256.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.24138191144301124,
+      "eval_entropy": 0.43642714537157784,
+      "eval_loss": 0.46031010150909424,
+      "eval_mean_token_accuracy": 0.9025695645061366,
+      "eval_num_tokens": 3428256.0,
+      "eval_runtime": 37.1078,
+      "eval_samples_per_second": 58.37,
+      "eval_steps_per_second": 7.303,
+      "step": 800
+    },
+    {
+      "entropy": 0.4217161551490426,
+      "epoch": 0.25646828090819945,
+      "grad_norm": 0.21452021598815918,
+      "learning_rate": 9.874868861183658e-05,
+      "loss": 0.4612973022460938,
+      "mean_token_accuracy": 0.9114794608950615,
+      "num_tokens": 3642529.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.41439461953938006,
+      "epoch": 0.27155465037338766,
+      "grad_norm": 0.25079163908958435,
+      "learning_rate": 9.856416382009006e-05,
+      "loss": 0.4494070053100586,
+      "mean_token_accuracy": 0.9127840812504292,
+      "num_tokens": 3855962.0,
+      "step": 900
+    },
+    {
+      "entropy": 0.4270974922552705,
+      "epoch": 0.2866410198385759,
+      "grad_norm": 0.20817860960960388,
+      "learning_rate": 9.836714974607077e-05,
+      "loss": 0.46105358123779294,
+      "mean_token_accuracy": 0.9099104046821594,
+      "num_tokens": 4069157.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.400859787017107,
+      "epoch": 0.301727389303764,
+      "grad_norm": 0.20943191647529602,
+      "learning_rate": 9.815769705603521e-05,
+      "loss": 0.4289055633544922,
+      "mean_token_accuracy": 0.9167061321437359,
+      "num_tokens": 4278743.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.301727389303764,
+      "eval_entropy": 0.44312036031946483,
+      "eval_loss": 0.4571220278739929,
+      "eval_mean_token_accuracy": 0.9029428505809545,
+      "eval_num_tokens": 4278743.0,
+      "eval_runtime": 37.0803,
+      "eval_samples_per_second": 58.414,
+      "eval_steps_per_second": 7.308,
+      "step": 1000
+    },
+    {
+      "entropy": 0.4052092955261469,
+      "epoch": 0.31681375876895224,
+      "grad_norm": 0.21670734882354736,
+      "learning_rate": 9.793585961508811e-05,
+      "loss": 0.44187084197998044,
+      "mean_token_accuracy": 0.9138142390549183,
+      "num_tokens": 4495120.0,
+      "step": 1050
+    },
+    {
+      "entropy": 0.3885428298264742,
+      "epoch": 0.33190012823414045,
+      "grad_norm": 0.2656344771385193,
+      "learning_rate": 9.770169447332977e-05,
+      "loss": 0.42026878356933595,
+      "mean_token_accuracy": 0.9171342994272709,
+      "num_tokens": 4707664.0,
+      "step": 1100
+    },
+    {
+      "entropy": 0.41116086438298227,
+      "epoch": 0.34698649769932866,
+      "grad_norm": 0.25747165083885193,
+      "learning_rate": 9.745526185118458e-05,
+      "loss": 0.44418087005615237,
+      "mean_token_accuracy": 0.9133055797219276,
+      "num_tokens": 4921515.0,
+      "step": 1150
+    },
+    {
+      "entropy": 0.40833847373723986,
+      "epoch": 0.36207286716451687,
+      "grad_norm": 0.22326916456222534,
+      "learning_rate": 9.719662512391396e-05,
+      "loss": 0.4394912338256836,
+      "mean_token_accuracy": 0.9140990000963211,
+      "num_tokens": 5133998.0,
+      "step": 1200
+    },
+    {
+      "epoch": 0.36207286716451687,
+      "eval_entropy": 0.44459747828240764,
+      "eval_loss": 0.4540960192680359,
+      "eval_mean_token_accuracy": 0.9033675605080664,
+      "eval_num_tokens": 5133998.0,
+      "eval_runtime": 37.1014,
+      "eval_samples_per_second": 58.381,
+      "eval_steps_per_second": 7.304,
+      "step": 1200
+    },
+    {
+      "entropy": 0.39192866910248997,
+      "epoch": 0.3771592366297051,
+      "grad_norm": 0.20642907917499542,
+      "learning_rate": 9.692585080531822e-05,
+      "loss": 0.42892616271972656,
+      "mean_token_accuracy": 0.9165673214197159,
+      "num_tokens": 5348047.0,
+      "step": 1250
+    },
+    {
+      "entropy": 0.40107650008052587,
+      "epoch": 0.3922456060948933,
+      "grad_norm": 0.2515887916088104,
+      "learning_rate": 9.664300853063104e-05,
+      "loss": 0.4329941558837891,
+      "mean_token_accuracy": 0.9152751086652279,
+      "num_tokens": 5562125.0,
+      "step": 1300
+    },
+    {
+      "entropy": 0.3964537301659584,
+      "epoch": 0.4073319755600815,
+      "grad_norm": 0.20992055535316467,
+      "learning_rate": 9.63481710386114e-05,
+      "loss": 0.4275414276123047,
+      "mean_token_accuracy": 0.9168652257323265,
+      "num_tokens": 5773685.0,
+      "step": 1350
+    },
+    {
+      "entropy": 0.39044841077178716,
+      "epoch": 0.42241834502526965,
+      "grad_norm": 0.30388110876083374,
+      "learning_rate": 9.604141415283728e-05,
+      "loss": 0.42324817657470704,
+      "mean_token_accuracy": 0.9169075645506382,
+      "num_tokens": 5986601.0,
+      "step": 1400
+    },
+    {
+      "epoch": 0.42241834502526965,
+      "eval_entropy": 0.4361799991658693,
+      "eval_loss": 0.45172417163848877,
+      "eval_mean_token_accuracy": 0.9033232137725802,
+      "eval_num_tokens": 5986601.0,
+      "eval_runtime": 37.1897,
+      "eval_samples_per_second": 58.242,
+      "eval_steps_per_second": 7.287,
+      "step": 1400
+    },
+    {
+      "entropy": 0.4137679870799184,
+      "epoch": 0.43750471449045786,
+      "grad_norm": 0.23755542933940887,
+      "learning_rate": 9.572281676220608e-05,
+      "loss": 0.4478377532958984,
+      "mean_token_accuracy": 0.911891212016344,
+      "num_tokens": 6203048.0,
+      "step": 1450
+    },
+    {
+      "entropy": 0.41279098089784383,
+      "epoch": 0.45259108395564607,
+      "grad_norm": 0.21780936419963837,
+      "learning_rate": 9.539246080064659e-05,
+      "loss": 0.45262195587158205,
+      "mean_token_accuracy": 0.9123758906126023,
+      "num_tokens": 6419624.0,
+      "step": 1500
+    },
+    {
+      "entropy": 0.41412957072257994,
+      "epoch": 0.4676774534208343,
+      "grad_norm": 0.2430579662322998,
+      "learning_rate": 9.505043122604818e-05,
+      "loss": 0.45246307373046873,
+      "mean_token_accuracy": 0.9122441673278808,
+      "num_tokens": 6633965.0,
+      "step": 1550
+    },
+    {
+      "entropy": 0.3798889485746622,
+      "epoch": 0.4827638228860225,
+      "grad_norm": 0.28653672337532043,
+      "learning_rate": 9.469681599841192e-05,
+      "loss": 0.41427810668945314,
+      "mean_token_accuracy": 0.9184439463913441,
+      "num_tokens": 6847358.0,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4827638228860225,
+      "eval_entropy": 0.43242653594025826,
+      "eval_loss": 0.4488651752471924,
+      "eval_mean_token_accuracy": 0.9040581949082688,
+      "eval_num_tokens": 6847358.0,
+      "eval_runtime": 37.1595,
+      "eval_samples_per_second": 58.289,
+      "eval_steps_per_second": 7.293,
+      "step": 1600
+    },
+    {
+      "entropy": 0.38126184083521364,
+      "epoch": 0.4978501923512107,
+      "grad_norm": 0.2068091183900833,
+      "learning_rate": 9.433170605722996e-05,
+      "loss": 0.40500320434570314,
+      "mean_token_accuracy": 0.9181749866902829,
+      "num_tokens": 7062005.0,
+      "step": 1650
+    },
+    {
+      "entropy": 0.38879580337554215,
+      "epoch": 0.5129365618163989,
+      "grad_norm": 0.19908899068832397,
+      "learning_rate": 9.395519529809848e-05,
+      "loss": 0.41957916259765626,
+      "mean_token_accuracy": 0.9183010324835778,
+      "num_tokens": 7272082.0,
+      "step": 1700
+    },
+    {
+      "entropy": 0.3867104376107454,
+      "epoch": 0.5280229312815871,
+      "grad_norm": 0.21681655943393707,
+      "learning_rate": 9.356738054857057e-05,
+      "loss": 0.41496986389160156,
+      "mean_token_accuracy": 0.9176018598675728,
+      "num_tokens": 7484751.0,
+      "step": 1750
+    },
+    {
+      "entropy": 0.3900348538905382,
+      "epoch": 0.5431093007467753,
+      "grad_norm": 0.23264895379543304,
+      "learning_rate": 9.316836154325494e-05,
+      "loss": 0.4201799774169922,
+      "mean_token_accuracy": 0.9161376728117466,
+      "num_tokens": 7699385.0,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5431093007467753,
+      "eval_entropy": 0.4371595958941977,
+      "eval_loss": 0.44555598497390747,
+      "eval_mean_token_accuracy": 0.9047388078101887,
+      "eval_num_tokens": 7699385.0,
+      "eval_runtime": 37.2391,
+      "eval_samples_per_second": 58.165,
+      "eval_steps_per_second": 7.277,
+      "step": 1800
+    },
+    {
+      "entropy": 0.38737339399755,
+      "epoch": 0.5581956702119635,
+      "grad_norm": 0.21236486732959747,
+      "learning_rate": 9.275824089816716e-05,
+      "loss": 0.4186508941650391,
+      "mean_token_accuracy": 0.9184837466478348,
+      "num_tokens": 7912846.0,
+      "step": 1850
+    },
+    {
+      "entropy": 0.38791515786200764,
+      "epoch": 0.5732820396771517,
+      "grad_norm": 0.22874821722507477,
+      "learning_rate": 9.233712408433972e-05,
+      "loss": 0.42144878387451173,
+      "mean_token_accuracy": 0.9170675221085548,
+      "num_tokens": 8126645.0,
+      "step": 1900
+    },
+    {
+      "entropy": 0.3831383780390024,
+      "epoch": 0.5883684091423399,
+      "grad_norm": 0.3072109818458557,
+      "learning_rate": 9.190511940069813e-05,
+      "loss": 0.407428092956543,
+      "mean_token_accuracy": 0.9184182004630566,
+      "num_tokens": 8341447.0,
+      "step": 1950
+    },
+    {
+      "entropy": 0.38130090072751044,
+      "epoch": 0.603454778607528,
+      "grad_norm": 0.2783527374267578,
+      "learning_rate": 9.146233794620944e-05,
+      "loss": 0.41518512725830076,
+      "mean_token_accuracy": 0.9192077203094959,
+      "num_tokens": 8553915.0,
+      "step": 2000
+    },
+    {
+      "epoch": 0.603454778607528,
+      "eval_entropy": 0.41359548831557874,
+      "eval_loss": 0.4426310062408447,
+      "eval_mean_token_accuracy": 0.9062746316744392,
+      "eval_num_tokens": 8553915.0,
+      "eval_runtime": 37.0772,
+      "eval_samples_per_second": 58.419,
+      "eval_steps_per_second": 7.309,
+      "step": 2000
+    },
+    {
+      "entropy": 0.40945424281060694,
+      "epoch": 0.6185411480727163,
+      "grad_norm": 0.2751815915107727,
+      "learning_rate": 9.100889359131093e-05,
+      "loss": 0.44279281616210936,
+      "mean_token_accuracy": 0.9126340833306312,
+      "num_tokens": 8773030.0,
+      "step": 2050
+    },
+    {
+      "entropy": 0.37910934548825026,
+      "epoch": 0.6336275175379045,
+      "grad_norm": 0.24518635869026184,
+      "learning_rate": 9.054490294862594e-05,
+      "loss": 0.41019065856933595,
+      "mean_token_accuracy": 0.9180504800379277,
+      "num_tokens": 8987621.0,
+      "step": 2100
+    },
+    {
+      "entropy": 0.38141273133456705,
+      "epoch": 0.6487138870030927,
+      "grad_norm": 0.2637041211128235,
+      "learning_rate": 9.00704853429745e-05,
+      "loss": 0.41344562530517576,
+      "mean_token_accuracy": 0.9188940741121769,
+      "num_tokens": 9202085.0,
+      "step": 2150
+    },
+    {
+      "entropy": 0.37150444712489844,
+      "epoch": 0.6638002564682809,
+      "grad_norm": 0.23061881959438324,
+      "learning_rate": 8.958576278068655e-05,
+      "loss": 0.4002714157104492,
+      "mean_token_accuracy": 0.9211013509333134,
+      "num_tokens": 9414528.0,
+      "step": 2200
+    },
+    {
+      "epoch": 0.6638002564682809,
+      "eval_entropy": 0.4198416776652706,
+      "eval_loss": 0.44056928157806396,
+      "eval_mean_token_accuracy": 0.9060557997094749,
+      "eval_num_tokens": 9414528.0,
+      "eval_runtime": 37.0429,
+      "eval_samples_per_second": 58.473,
+      "eval_steps_per_second": 7.316,
+      "step": 2200
+    },
+    {
+      "entropy": 0.3824780482426286,
+      "epoch": 0.6788866259334692,
+      "grad_norm": 0.29528528451919556,
+      "learning_rate": 8.909085991822532e-05,
+      "loss": 0.4100413513183594,
+      "mean_token_accuracy": 0.9181285245716572,
+      "num_tokens": 9631160.0,
+      "step": 2250
+    },
+    {
+      "entropy": 0.37036353170871733,
+      "epoch": 0.6939729953986573,
+      "grad_norm": 0.2833597958087921,
+      "learning_rate": 8.858590403012954e-05,
+      "loss": 0.39582439422607424,
+      "mean_token_accuracy": 0.9203065976500511,
+      "num_tokens": 9844323.0,
+      "step": 2300
+    },
+    {
+      "entropy": 0.377471005320549,
+      "epoch": 0.7090593648638455,
+      "grad_norm": 0.2559050917625427,
+      "learning_rate": 8.807102497628199e-05,
+      "loss": 0.4039160919189453,
+      "mean_token_accuracy": 0.9185835334658623,
+      "num_tokens": 10060066.0,
+      "step": 2350
+    },
+    {
+      "entropy": 0.38689912386238573,
+      "epoch": 0.7241457343290337,
+      "grad_norm": 0.3571145236492157,
+      "learning_rate": 8.754635516851342e-05,
+      "loss": 0.41998291015625,
+      "mean_token_accuracy": 0.9171991994976998,
+      "num_tokens": 10275374.0,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7241457343290337,
+      "eval_entropy": 0.4061841280148038,
+      "eval_loss": 0.4392658472061157,
+      "eval_mean_token_accuracy": 0.9060493254573583,
+      "eval_num_tokens": 10275374.0,
+      "eval_runtime": 37.185,
+      "eval_samples_per_second": 58.249,
+      "eval_steps_per_second": 7.288,
+      "step": 2400
+    },
+    {
+      "entropy": 0.3773344187065959,
+      "epoch": 0.7392321037942219,
+      "grad_norm": 0.23827174305915833,
+      "learning_rate": 8.701202953655006e-05,
+      "loss": 0.4055968475341797,
+      "mean_token_accuracy": 0.9189482787251473,
+      "num_tokens": 10495301.0,
+      "step": 2450
+    },
+    {
+      "entropy": 0.3638977843523026,
+      "epoch": 0.7543184732594101,
+      "grad_norm": 0.247745543718338,
+      "learning_rate": 8.646818549331366e-05,
+      "loss": 0.38891139984130857,
+      "mean_token_accuracy": 0.9226090031862259,
+      "num_tokens": 10706938.0,
+      "step": 2500
+    },
+    {
+      "entropy": 0.358336652033031,
+      "epoch": 0.7694048427245983,
+      "grad_norm": 0.24292156100273132,
+      "learning_rate": 8.591496289958292e-05,
+      "loss": 0.3846548461914063,
+      "mean_token_accuracy": 0.923456951379776,
+      "num_tokens": 10918302.0,
+      "step": 2550
+    },
+    {
+      "entropy": 0.37086400829255584,
+      "epoch": 0.7844912121897866,
+      "grad_norm": 0.2979118525981903,
+      "learning_rate": 8.535250402802536e-05,
+      "loss": 0.39662261962890627,
+      "mean_token_accuracy": 0.9212297305464745,
+      "num_tokens": 11131056.0,
+      "step": 2600
+    },
+    {
+      "epoch": 0.7844912121897866,
+      "eval_entropy": 0.4161290250361186,
+      "eval_loss": 0.43674495816230774,
+      "eval_mean_token_accuracy": 0.9060781219788583,
+      "eval_num_tokens": 11131056.0,
+      "eval_runtime": 37.0488,
+      "eval_samples_per_second": 58.463,
+      "eval_steps_per_second": 7.315,
+      "step": 2600
+    },
+    {
+      "entropy": 0.36887906536459925,
+      "epoch": 0.7995775816549747,
+      "grad_norm": 0.25673073530197144,
+      "learning_rate": 8.478095352660897e-05,
+      "loss": 0.3948686218261719,
+      "mean_token_accuracy": 0.9204315200448037,
+      "num_tokens": 11345648.0,
+      "step": 2650
+    },
+    {
+      "entropy": 0.36981521353125574,
+      "epoch": 0.814663951120163,
+      "grad_norm": 0.2649747133255005,
+      "learning_rate": 8.4200458381403e-05,
+      "loss": 0.3937848663330078,
+      "mean_token_accuracy": 0.9218536545336247,
+      "num_tokens": 11559009.0,
+      "step": 2700
+    },
+    {
+      "entropy": 0.37904939975589513,
+      "epoch": 0.8297503205853511,
+      "grad_norm": 0.20989011228084564,
+      "learning_rate": 8.361116787877736e-05,
+      "loss": 0.4084677505493164,
+      "mean_token_accuracy": 0.9188165719807148,
+      "num_tokens": 11776255.0,
+      "step": 2750
+    },
+    {
+      "entropy": 0.3781035339459777,
+      "epoch": 0.8448366900505393,
+      "grad_norm": 0.2979874908924103,
+      "learning_rate": 8.301323356701069e-05,
+      "loss": 0.40767410278320315,
+      "mean_token_accuracy": 0.9183482979238033,
+      "num_tokens": 11994830.0,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8448366900505393,
+      "eval_entropy": 0.3918299580962016,
+      "eval_loss": 0.43606311082839966,
+      "eval_mean_token_accuracy": 0.9074829088805786,
+      "eval_num_tokens": 11994830.0,
+      "eval_runtime": 37.1502,
+      "eval_samples_per_second": 58.304,
+      "eval_steps_per_second": 7.295,
+      "step": 2800
+    },
+    {
+      "entropy": 0.3669764836877584,
+      "epoch": 0.8599230595157276,
+      "grad_norm": 0.3718933165073395,
+      "learning_rate": 8.240680921731639e-05,
+      "loss": 0.39511192321777344,
+      "mean_token_accuracy": 0.9215331043303013,
+      "num_tokens": 12210990.0,
+      "step": 2850
+    },
+    {
+      "entropy": 0.36516126081347466,
+      "epoch": 0.8750094289809157,
+      "grad_norm": 0.2584734559059143,
+      "learning_rate": 8.179205078429728e-05,
+      "loss": 0.3858111572265625,
+      "mean_token_accuracy": 0.9223315984010696,
+      "num_tokens": 12425768.0,
+      "step": 2900
+    },
+    {
+      "entropy": 0.36489626977592704,
+      "epoch": 0.890095798446104,
+      "grad_norm": 0.260593980550766,
+      "learning_rate": 8.116911636583866e-05,
+      "loss": 0.3904818344116211,
+      "mean_token_accuracy": 0.921723841279745,
+      "num_tokens": 12644047.0,
+      "step": 2950
+    },
+    {
+      "entropy": 0.35986222576349974,
+      "epoch": 0.9051821679112921,
+      "grad_norm": 0.2872949540615082,
+      "learning_rate": 8.053816616245007e-05,
+      "loss": 0.3802699661254883,
+      "mean_token_accuracy": 0.922919643521309,
+      "num_tokens": 12858612.0,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9051821679112921,
+      "eval_entropy": 0.39114147694128465,
+      "eval_loss": 0.43371766805648804,
+      "eval_mean_token_accuracy": 0.9085190791045608,
+      "eval_num_tokens": 12858612.0,
+      "eval_runtime": 37.1981,
+      "eval_samples_per_second": 58.229,
+      "eval_steps_per_second": 7.285,
+      "step": 3000
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 9945,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.3087949758650778e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe063de6f1674be7d5535690dc263e5d0c28436ed4781417772a469f39ef21d4
+size 5713

checkpoint-9800/README.md

@@ -0,0 +1,209 @@
+---
+base_model: /scratch/Qwen3-1.7B
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/scratch/Qwen3-1.7B
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-9800/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/scratch/Qwen3-1.7B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "down_proj",
+    "q_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-9800/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e742ba8e124e5e9ab87216c323006d9253a80d69013d2c1dae62d12e1813c30d
+size 278973888

checkpoint-9800/chat_template.jinja

@@ -0,0 +1,93 @@
+{%- set enable_thinking = true %}
+{%- set enable_thinking = false %}
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant. Always put your final answer in \\boxed{}. For multiple choice questions, put only the letter in \\boxed{}, for example \\boxed{A}.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

checkpoint-9800/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0110585a12fd99136e3643624835143de8686330451634b5f8f8417debebe62b
+size 558179131

checkpoint-9800/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76d71495f5e6a20fc06f00a491570960dbe1422b5d9243411af4a25b6a99e244
+size 14645

checkpoint-9800/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8775e41525e12b64bb5546dacc53b212796dd04bbce45b458630779f49cfe571
+size 1465

checkpoint-9800/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

checkpoint-9800/tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-9800/trainer_state.json

@@ -0,0 +1,2533 @@
+{
+  "best_global_step": 3000,
+  "best_metric": 0.43371766805648804,
+  "best_model_checkpoint": "/scratch/gk_checkpoint_lora_v2/checkpoint-3000",
+  "epoch": 2.956475824092932,
+  "eval_steps": 200,
+  "global_step": 9800,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.0372698324173688,
+      "epoch": 0.015086369465188203,
+      "grad_norm": 0.4944687783718109,
+      "learning_rate": 3.266666666666667e-05,
+      "loss": 2.1209957885742186,
+      "mean_token_accuracy": 0.6639667785167694,
+      "num_tokens": 213967.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.4767542722821236,
+      "epoch": 0.030172738930376405,
+      "grad_norm": 0.29953643679618835,
+      "learning_rate": 6.6e-05,
+      "loss": 0.5209395980834961,
+      "mean_token_accuracy": 0.9028549310564995,
+      "num_tokens": 426365.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.46151453502476214,
+      "epoch": 0.04525910839556461,
+      "grad_norm": 0.3118787109851837,
+      "learning_rate": 9.933333333333334e-05,
+      "loss": 0.508198356628418,
+      "mean_token_accuracy": 0.9032981966435909,
+      "num_tokens": 642837.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.4455497920885682,
+      "epoch": 0.06034547786075281,
+      "grad_norm": 0.3013271391391754,
+      "learning_rate": 9.999382532513122e-05,
+      "loss": 0.48826019287109373,
+      "mean_token_accuracy": 0.907553653717041,
+      "num_tokens": 856475.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.06034547786075281,
+      "eval_entropy": 0.4689600637588114,
+      "eval_loss": 0.4816349446773529,
+      "eval_mean_token_accuracy": 0.8987587762934696,
+      "eval_num_tokens": 856475.0,
+      "eval_runtime": 37.5206,
+      "eval_samples_per_second": 57.728,
+      "eval_steps_per_second": 7.223,
+      "step": 200
+    },
+    {
+      "entropy": 0.44372758489102127,
+      "epoch": 0.075431847325941,
+      "grad_norm": 0.31373220682144165,
+      "learning_rate": 9.997479627263544e-05,
+      "loss": 0.4843710327148438,
+      "mean_token_accuracy": 0.9082412907481193,
+      "num_tokens": 1069370.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.4475720078870654,
+      "epoch": 0.09051821679112922,
+      "grad_norm": 0.27380964159965515,
+      "learning_rate": 9.994291516446573e-05,
+      "loss": 0.491109733581543,
+      "mean_token_accuracy": 0.9055162121355533,
+      "num_tokens": 1286132.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.4463552813604474,
+      "epoch": 0.10560458625631741,
+      "grad_norm": 0.2614763677120209,
+      "learning_rate": 9.989819019951048e-05,
+      "loss": 0.4837772369384766,
+      "mean_token_accuracy": 0.9081570096313953,
+      "num_tokens": 1500851.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.41013720393180847,
+      "epoch": 0.12069095572150562,
+      "grad_norm": 0.4836612045764923,
+      "learning_rate": 9.984063287972232e-05,
+      "loss": 0.44385364532470706,
+      "mean_token_accuracy": 0.9148843766748905,
+      "num_tokens": 1708807.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.12069095572150562,
+      "eval_entropy": 0.4811463643044123,
+      "eval_loss": 0.4747391641139984,
+      "eval_mean_token_accuracy": 0.8997697566268189,
+      "eval_num_tokens": 1708807.0,
+      "eval_runtime": 37.1123,
+      "eval_samples_per_second": 58.363,
+      "eval_steps_per_second": 7.302,
+      "step": 400
+    },
+    {
+      "entropy": 0.42200190499424933,
+      "epoch": 0.13577732518669383,
+      "grad_norm": 0.19339531660079956,
+      "learning_rate": 9.977025800716017e-05,
+      "loss": 0.45712459564208985,
+      "mean_token_accuracy": 0.9106362241506577,
+      "num_tokens": 1923184.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.43271509755402804,
+      "epoch": 0.150863694651882,
+      "grad_norm": 0.299110472202301,
+      "learning_rate": 9.968708368018253e-05,
+      "loss": 0.4724855422973633,
+      "mean_token_accuracy": 0.9078708891570568,
+      "num_tokens": 2139609.0,
+      "step": 500
+    },
+    {
+      "entropy": 0.43638833791017534,
+      "epoch": 0.16595006411707022,
+      "grad_norm": 0.2539425492286682,
+      "learning_rate": 9.959113128879322e-05,
+      "loss": 0.4785395050048828,
+      "mean_token_accuracy": 0.907249256670475,
+      "num_tokens": 2358080.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.43115664307028057,
+      "epoch": 0.18103643358225843,
+      "grad_norm": 0.2542003393173218,
+      "learning_rate": 9.948242550914035e-05,
+      "loss": 0.4740608215332031,
+      "mean_token_accuracy": 0.9089943794906139,
+      "num_tokens": 2574667.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.18103643358225843,
+      "eval_entropy": 0.45346878287537074,
+      "eval_loss": 0.46328845620155334,
+      "eval_mean_token_accuracy": 0.9025413253210568,
+      "eval_num_tokens": 2574667.0,
+      "eval_runtime": 37.1254,
+      "eval_samples_per_second": 58.343,
+      "eval_steps_per_second": 7.3,
+      "step": 600
+    },
+    {
+      "entropy": 0.4420048241317272,
+      "epoch": 0.19612280304744664,
+      "grad_norm": 0.26832085847854614,
+      "learning_rate": 9.936099429717045e-05,
+      "loss": 0.486652717590332,
+      "mean_token_accuracy": 0.9074076810479164,
+      "num_tokens": 2790489.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.4269510039314628,
+      "epoch": 0.21120917251263482,
+      "grad_norm": 0.20635050535202026,
+      "learning_rate": 9.922686888143897e-05,
+      "loss": 0.4619187927246094,
+      "mean_token_accuracy": 0.910810690075159,
+      "num_tokens": 3003881.0,
+      "step": 700
+    },
+    {
+      "entropy": 0.42016164746135476,
+      "epoch": 0.22629554197782303,
+      "grad_norm": 0.2643264830112457,
+      "learning_rate": 9.908008375507924e-05,
+      "loss": 0.46344844818115233,
+      "mean_token_accuracy": 0.9113752076029777,
+      "num_tokens": 3218446.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.3974369211867452,
+      "epoch": 0.24138191144301124,
+      "grad_norm": 0.19392798840999603,
+      "learning_rate": 9.89206766669318e-05,
+      "loss": 0.42646697998046873,
+      "mean_token_accuracy": 0.9165593402087688,
+      "num_tokens": 3428256.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.24138191144301124,
+      "eval_entropy": 0.43642714537157784,
+      "eval_loss": 0.46031010150909424,
+      "eval_mean_token_accuracy": 0.9025695645061366,
+      "eval_num_tokens": 3428256.0,
+      "eval_runtime": 37.1078,
+      "eval_samples_per_second": 58.37,
+      "eval_steps_per_second": 7.303,
+      "step": 800
+    },
+    {
+      "entropy": 0.4217161551490426,
+      "epoch": 0.25646828090819945,
+      "grad_norm": 0.21452021598815918,
+      "learning_rate": 9.874868861183658e-05,
+      "loss": 0.4612973022460938,
+      "mean_token_accuracy": 0.9114794608950615,
+      "num_tokens": 3642529.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.41439461953938006,
+      "epoch": 0.27155465037338766,
+      "grad_norm": 0.25079163908958435,
+      "learning_rate": 9.856416382009006e-05,
+      "loss": 0.4494070053100586,
+      "mean_token_accuracy": 0.9127840812504292,
+      "num_tokens": 3855962.0,
+      "step": 900
+    },
+    {
+      "entropy": 0.4270974922552705,
+      "epoch": 0.2866410198385759,
+      "grad_norm": 0.20817860960960388,
+      "learning_rate": 9.836714974607077e-05,
+      "loss": 0.46105358123779294,
+      "mean_token_accuracy": 0.9099104046821594,
+      "num_tokens": 4069157.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.400859787017107,
+      "epoch": 0.301727389303764,
+      "grad_norm": 0.20943191647529602,
+      "learning_rate": 9.815769705603521e-05,
+      "loss": 0.4289055633544922,
+      "mean_token_accuracy": 0.9167061321437359,
+      "num_tokens": 4278743.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.301727389303764,
+      "eval_entropy": 0.44312036031946483,
+      "eval_loss": 0.4571220278739929,
+      "eval_mean_token_accuracy": 0.9029428505809545,
+      "eval_num_tokens": 4278743.0,
+      "eval_runtime": 37.0803,
+      "eval_samples_per_second": 58.414,
+      "eval_steps_per_second": 7.308,
+      "step": 1000
+    },
+    {
+      "entropy": 0.4052092955261469,
+      "epoch": 0.31681375876895224,
+      "grad_norm": 0.21670734882354736,
+      "learning_rate": 9.793585961508811e-05,
+      "loss": 0.44187084197998044,
+      "mean_token_accuracy": 0.9138142390549183,
+      "num_tokens": 4495120.0,
+      "step": 1050
+    },
+    {
+      "entropy": 0.3885428298264742,
+      "epoch": 0.33190012823414045,
+      "grad_norm": 0.2656344771385193,
+      "learning_rate": 9.770169447332977e-05,
+      "loss": 0.42026878356933595,
+      "mean_token_accuracy": 0.9171342994272709,
+      "num_tokens": 4707664.0,
+      "step": 1100
+    },
+    {
+      "entropy": 0.41116086438298227,
+      "epoch": 0.34698649769932866,
+      "grad_norm": 0.25747165083885193,
+      "learning_rate": 9.745526185118458e-05,
+      "loss": 0.44418087005615237,
+      "mean_token_accuracy": 0.9133055797219276,
+      "num_tokens": 4921515.0,
+      "step": 1150
+    },
+    {
+      "entropy": 0.40833847373723986,
+      "epoch": 0.36207286716451687,
+      "grad_norm": 0.22326916456222534,
+      "learning_rate": 9.719662512391396e-05,
+      "loss": 0.4394912338256836,
+      "mean_token_accuracy": 0.9140990000963211,
+      "num_tokens": 5133998.0,
+      "step": 1200
+    },
+    {
+      "epoch": 0.36207286716451687,
+      "eval_entropy": 0.44459747828240764,
+      "eval_loss": 0.4540960192680359,
+      "eval_mean_token_accuracy": 0.9033675605080664,
+      "eval_num_tokens": 5133998.0,
+      "eval_runtime": 37.1014,
+      "eval_samples_per_second": 58.381,
+      "eval_steps_per_second": 7.304,
+      "step": 1200
+    },
+    {
+      "entropy": 0.39192866910248997,
+      "epoch": 0.3771592366297051,
+      "grad_norm": 0.20642907917499542,
+      "learning_rate": 9.692585080531822e-05,
+      "loss": 0.42892616271972656,
+      "mean_token_accuracy": 0.9165673214197159,
+      "num_tokens": 5348047.0,
+      "step": 1250
+    },
+    {
+      "entropy": 0.40107650008052587,
+      "epoch": 0.3922456060948933,
+      "grad_norm": 0.2515887916088104,
+      "learning_rate": 9.664300853063104e-05,
+      "loss": 0.4329941558837891,
+      "mean_token_accuracy": 0.9152751086652279,
+      "num_tokens": 5562125.0,
+      "step": 1300
+    },
+    {
+      "entropy": 0.3964537301659584,
+      "epoch": 0.4073319755600815,
+      "grad_norm": 0.20992055535316467,
+      "learning_rate": 9.63481710386114e-05,
+      "loss": 0.4275414276123047,
+      "mean_token_accuracy": 0.9168652257323265,
+      "num_tokens": 5773685.0,
+      "step": 1350
+    },
+    {
+      "entropy": 0.39044841077178716,
+      "epoch": 0.42241834502526965,
+      "grad_norm": 0.30388110876083374,
+      "learning_rate": 9.604141415283728e-05,
+      "loss": 0.42324817657470704,
+      "mean_token_accuracy": 0.9169075645506382,
+      "num_tokens": 5986601.0,
+      "step": 1400
+    },
+    {
+      "epoch": 0.42241834502526965,
+      "eval_entropy": 0.4361799991658693,
+      "eval_loss": 0.45172417163848877,
+      "eval_mean_token_accuracy": 0.9033232137725802,
+      "eval_num_tokens": 5986601.0,
+      "eval_runtime": 37.1897,
+      "eval_samples_per_second": 58.242,
+      "eval_steps_per_second": 7.287,
+      "step": 1400
+    },
+    {
+      "entropy": 0.4137679870799184,
+      "epoch": 0.43750471449045786,
+      "grad_norm": 0.23755542933940887,
+      "learning_rate": 9.572281676220608e-05,
+      "loss": 0.4478377532958984,
+      "mean_token_accuracy": 0.911891212016344,
+      "num_tokens": 6203048.0,
+      "step": 1450
+    },
+    {
+      "entropy": 0.41279098089784383,
+      "epoch": 0.45259108395564607,
+      "grad_norm": 0.21780936419963837,
+      "learning_rate": 9.539246080064659e-05,
+      "loss": 0.45262195587158205,
+      "mean_token_accuracy": 0.9123758906126023,
+      "num_tokens": 6419624.0,
+      "step": 1500
+    },
+    {
+      "entropy": 0.41412957072257994,
+      "epoch": 0.4676774534208343,
+      "grad_norm": 0.2430579662322998,
+      "learning_rate": 9.505043122604818e-05,
+      "loss": 0.45246307373046873,
+      "mean_token_accuracy": 0.9122441673278808,
+      "num_tokens": 6633965.0,
+      "step": 1550
+    },
+    {
+      "entropy": 0.3798889485746622,
+      "epoch": 0.4827638228860225,
+      "grad_norm": 0.28653672337532043,
+      "learning_rate": 9.469681599841192e-05,
+      "loss": 0.41427810668945314,
+      "mean_token_accuracy": 0.9184439463913441,
+      "num_tokens": 6847358.0,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4827638228860225,
+      "eval_entropy": 0.43242653594025826,
+      "eval_loss": 0.4488651752471924,
+      "eval_mean_token_accuracy": 0.9040581949082688,
+      "eval_num_tokens": 6847358.0,
+      "eval_runtime": 37.1595,
+      "eval_samples_per_second": 58.289,
+      "eval_steps_per_second": 7.293,
+      "step": 1600
+    },
+    {
+      "entropy": 0.38126184083521364,
+      "epoch": 0.4978501923512107,
+      "grad_norm": 0.2068091183900833,
+      "learning_rate": 9.433170605722996e-05,
+      "loss": 0.40500320434570314,
+      "mean_token_accuracy": 0.9181749866902829,
+      "num_tokens": 7062005.0,
+      "step": 1650
+    },
+    {
+      "entropy": 0.38879580337554215,
+      "epoch": 0.5129365618163989,
+      "grad_norm": 0.19908899068832397,
+      "learning_rate": 9.395519529809848e-05,
+      "loss": 0.41957916259765626,
+      "mean_token_accuracy": 0.9183010324835778,
+      "num_tokens": 7272082.0,
+      "step": 1700
+    },
+    {
+      "entropy": 0.3867104376107454,
+      "epoch": 0.5280229312815871,
+      "grad_norm": 0.21681655943393707,
+      "learning_rate": 9.356738054857057e-05,
+      "loss": 0.41496986389160156,
+      "mean_token_accuracy": 0.9176018598675728,
+      "num_tokens": 7484751.0,
+      "step": 1750
+    },
+    {
+      "entropy": 0.3900348538905382,
+      "epoch": 0.5431093007467753,
+      "grad_norm": 0.23264895379543304,
+      "learning_rate": 9.316836154325494e-05,
+      "loss": 0.4201799774169922,
+      "mean_token_accuracy": 0.9161376728117466,
+      "num_tokens": 7699385.0,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5431093007467753,
+      "eval_entropy": 0.4371595958941977,
+      "eval_loss": 0.44555598497390747,
+      "eval_mean_token_accuracy": 0.9047388078101887,
+      "eval_num_tokens": 7699385.0,
+      "eval_runtime": 37.2391,
+      "eval_samples_per_second": 58.165,
+      "eval_steps_per_second": 7.277,
+      "step": 1800
+    },
+    {
+      "entropy": 0.38737339399755,
+      "epoch": 0.5581956702119635,
+      "grad_norm": 0.21236486732959747,
+      "learning_rate": 9.275824089816716e-05,
+      "loss": 0.4186508941650391,
+      "mean_token_accuracy": 0.9184837466478348,
+      "num_tokens": 7912846.0,
+      "step": 1850
+    },
+    {
+      "entropy": 0.38791515786200764,
+      "epoch": 0.5732820396771517,
+      "grad_norm": 0.22874821722507477,
+      "learning_rate": 9.233712408433972e-05,
+      "loss": 0.42144878387451173,
+      "mean_token_accuracy": 0.9170675221085548,
+      "num_tokens": 8126645.0,
+      "step": 1900
+    },
+    {
+      "entropy": 0.3831383780390024,
+      "epoch": 0.5883684091423399,
+      "grad_norm": 0.3072109818458557,
+      "learning_rate": 9.190511940069813e-05,
+      "loss": 0.407428092956543,
+      "mean_token_accuracy": 0.9184182004630566,
+      "num_tokens": 8341447.0,
+      "step": 1950
+    },
+    {
+      "entropy": 0.38130090072751044,
+      "epoch": 0.603454778607528,
+      "grad_norm": 0.2783527374267578,
+      "learning_rate": 9.146233794620944e-05,
+      "loss": 0.41518512725830076,
+      "mean_token_accuracy": 0.9192077203094959,
+      "num_tokens": 8553915.0,
+      "step": 2000
+    },
+    {
+      "epoch": 0.603454778607528,
+      "eval_entropy": 0.41359548831557874,
+      "eval_loss": 0.4426310062408447,
+      "eval_mean_token_accuracy": 0.9062746316744392,
+      "eval_num_tokens": 8553915.0,
+      "eval_runtime": 37.0772,
+      "eval_samples_per_second": 58.419,
+      "eval_steps_per_second": 7.309,
+      "step": 2000
+    },
+    {
+      "entropy": 0.40945424281060694,
+      "epoch": 0.6185411480727163,
+      "grad_norm": 0.2751815915107727,
+      "learning_rate": 9.100889359131093e-05,
+      "loss": 0.44279281616210936,
+      "mean_token_accuracy": 0.9126340833306312,
+      "num_tokens": 8773030.0,
+      "step": 2050
+    },
+    {
+      "entropy": 0.37910934548825026,
+      "epoch": 0.6336275175379045,
+      "grad_norm": 0.24518635869026184,
+      "learning_rate": 9.054490294862594e-05,
+      "loss": 0.41019065856933595,
+      "mean_token_accuracy": 0.9180504800379277,
+      "num_tokens": 8987621.0,
+      "step": 2100
+    },
+    {
+      "entropy": 0.38141273133456705,
+      "epoch": 0.6487138870030927,
+      "grad_norm": 0.2637041211128235,
+      "learning_rate": 9.00704853429745e-05,
+      "loss": 0.41344562530517576,
+      "mean_token_accuracy": 0.9188940741121769,
+      "num_tokens": 9202085.0,
+      "step": 2150
+    },
+    {
+      "entropy": 0.37150444712489844,
+      "epoch": 0.6638002564682809,
+      "grad_norm": 0.23061881959438324,
+      "learning_rate": 8.958576278068655e-05,
+      "loss": 0.4002714157104492,
+      "mean_token_accuracy": 0.9211013509333134,
+      "num_tokens": 9414528.0,
+      "step": 2200
+    },
+    {
+      "epoch": 0.6638002564682809,
+      "eval_entropy": 0.4198416776652706,
+      "eval_loss": 0.44056928157806396,
+      "eval_mean_token_accuracy": 0.9060557997094749,
+      "eval_num_tokens": 9414528.0,
+      "eval_runtime": 37.0429,
+      "eval_samples_per_second": 58.473,
+      "eval_steps_per_second": 7.316,
+      "step": 2200
+    },
+    {
+      "entropy": 0.3824780482426286,
+      "epoch": 0.6788866259334692,
+      "grad_norm": 0.29528528451919556,
+      "learning_rate": 8.909085991822532e-05,
+      "loss": 0.4100413513183594,
+      "mean_token_accuracy": 0.9181285245716572,
+      "num_tokens": 9631160.0,
+      "step": 2250
+    },
+    {
+      "entropy": 0.37036353170871733,
+      "epoch": 0.6939729953986573,
+      "grad_norm": 0.2833597958087921,
+      "learning_rate": 8.858590403012954e-05,
+      "loss": 0.39582439422607424,
+      "mean_token_accuracy": 0.9203065976500511,
+      "num_tokens": 9844323.0,
+      "step": 2300
+    },
+    {
+      "entropy": 0.377471005320549,
+      "epoch": 0.7090593648638455,
+      "grad_norm": 0.2559050917625427,
+      "learning_rate": 8.807102497628199e-05,
+      "loss": 0.4039160919189453,
+      "mean_token_accuracy": 0.9185835334658623,
+      "num_tokens": 10060066.0,
+      "step": 2350
+    },
+    {
+      "entropy": 0.38689912386238573,
+      "epoch": 0.7241457343290337,
+      "grad_norm": 0.3571145236492157,
+      "learning_rate": 8.754635516851342e-05,
+      "loss": 0.41998291015625,
+      "mean_token_accuracy": 0.9171991994976998,
+      "num_tokens": 10275374.0,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7241457343290337,
+      "eval_entropy": 0.4061841280148038,
+      "eval_loss": 0.4392658472061157,
+      "eval_mean_token_accuracy": 0.9060493254573583,
+      "eval_num_tokens": 10275374.0,
+      "eval_runtime": 37.185,
+      "eval_samples_per_second": 58.249,
+      "eval_steps_per_second": 7.288,
+      "step": 2400
+    },
+    {
+      "entropy": 0.3773344187065959,
+      "epoch": 0.7392321037942219,
+      "grad_norm": 0.23827174305915833,
+      "learning_rate": 8.701202953655006e-05,
+      "loss": 0.4055968475341797,
+      "mean_token_accuracy": 0.9189482787251473,
+      "num_tokens": 10495301.0,
+      "step": 2450
+    },
+    {
+      "entropy": 0.3638977843523026,
+      "epoch": 0.7543184732594101,
+      "grad_norm": 0.247745543718338,
+      "learning_rate": 8.646818549331366e-05,
+      "loss": 0.38891139984130857,
+      "mean_token_accuracy": 0.9226090031862259,
+      "num_tokens": 10706938.0,
+      "step": 2500
+    },
+    {
+      "entropy": 0.358336652033031,
+      "epoch": 0.7694048427245983,
+      "grad_norm": 0.24292156100273132,
+      "learning_rate": 8.591496289958292e-05,
+      "loss": 0.3846548461914063,
+      "mean_token_accuracy": 0.923456951379776,
+      "num_tokens": 10918302.0,
+      "step": 2550
+    },
+    {
+      "entropy": 0.37086400829255584,
+      "epoch": 0.7844912121897866,
+      "grad_norm": 0.2979118525981903,
+      "learning_rate": 8.535250402802536e-05,
+      "loss": 0.39662261962890627,
+      "mean_token_accuracy": 0.9212297305464745,
+      "num_tokens": 11131056.0,
+      "step": 2600
+    },
+    {
+      "epoch": 0.7844912121897866,
+      "eval_entropy": 0.4161290250361186,
+      "eval_loss": 0.43674495816230774,
+      "eval_mean_token_accuracy": 0.9060781219788583,
+      "eval_num_tokens": 11131056.0,
+      "eval_runtime": 37.0488,
+      "eval_samples_per_second": 58.463,
+      "eval_steps_per_second": 7.315,
+      "step": 2600
+    },
+    {
+      "entropy": 0.36887906536459925,
+      "epoch": 0.7995775816549747,
+      "grad_norm": 0.25673073530197144,
+      "learning_rate": 8.478095352660897e-05,
+      "loss": 0.3948686218261719,
+      "mean_token_accuracy": 0.9204315200448037,
+      "num_tokens": 11345648.0,
+      "step": 2650
+    },
+    {
+      "entropy": 0.36981521353125574,
+      "epoch": 0.814663951120163,
+      "grad_norm": 0.2649747133255005,
+      "learning_rate": 8.4200458381403e-05,
+      "loss": 0.3937848663330078,
+      "mean_token_accuracy": 0.9218536545336247,
+      "num_tokens": 11559009.0,
+      "step": 2700
+    },
+    {
+      "entropy": 0.37904939975589513,
+      "epoch": 0.8297503205853511,
+      "grad_norm": 0.20989011228084564,
+      "learning_rate": 8.361116787877736e-05,
+      "loss": 0.4084677505493164,
+      "mean_token_accuracy": 0.9188165719807148,
+      "num_tokens": 11776255.0,
+      "step": 2750
+    },
+    {
+      "entropy": 0.3781035339459777,
+      "epoch": 0.8448366900505393,
+      "grad_norm": 0.2979874908924103,
+      "learning_rate": 8.301323356701069e-05,
+      "loss": 0.40767410278320315,
+      "mean_token_accuracy": 0.9183482979238033,
+      "num_tokens": 11994830.0,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8448366900505393,
+      "eval_entropy": 0.3918299580962016,
+      "eval_loss": 0.43606311082839966,
+      "eval_mean_token_accuracy": 0.9074829088805786,
+      "eval_num_tokens": 11994830.0,
+      "eval_runtime": 37.1502,
+      "eval_samples_per_second": 58.304,
+      "eval_steps_per_second": 7.295,
+      "step": 2800
+    },
+    {
+      "entropy": 0.3669764836877584,
+      "epoch": 0.8599230595157276,
+      "grad_norm": 0.3718933165073395,
+      "learning_rate": 8.240680921731639e-05,
+      "loss": 0.39511192321777344,
+      "mean_token_accuracy": 0.9215331043303013,
+      "num_tokens": 12210990.0,
+      "step": 2850
+    },
+    {
+      "entropy": 0.36516126081347466,
+      "epoch": 0.8750094289809157,
+      "grad_norm": 0.2584734559059143,
+      "learning_rate": 8.179205078429728e-05,
+      "loss": 0.3858111572265625,
+      "mean_token_accuracy": 0.9223315984010696,
+      "num_tokens": 12425768.0,
+      "step": 2900
+    },
+    {
+      "entropy": 0.36489626977592704,
+      "epoch": 0.890095798446104,
+      "grad_norm": 0.260593980550766,
+      "learning_rate": 8.116911636583866e-05,
+      "loss": 0.3904818344116211,
+      "mean_token_accuracy": 0.921723841279745,
+      "num_tokens": 12644047.0,
+      "step": 2950
+    },
+    {
+      "entropy": 0.35986222576349974,
+      "epoch": 0.9051821679112921,
+      "grad_norm": 0.2872949540615082,
+      "learning_rate": 8.053816616245007e-05,
+      "loss": 0.3802699661254883,
+      "mean_token_accuracy": 0.922919643521309,
+      "num_tokens": 12858612.0,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9051821679112921,
+      "eval_entropy": 0.39114147694128465,
+      "eval_loss": 0.43371766805648804,
+      "eval_mean_token_accuracy": 0.9085190791045608,
+      "eval_num_tokens": 12858612.0,
+      "eval_runtime": 37.1981,
+      "eval_samples_per_second": 58.229,
+      "eval_steps_per_second": 7.285,
+      "step": 3000
+    },
+    {
+      "entropy": 0.3628548865020275,
+      "epoch": 0.9202685373764804,
+      "grad_norm": 0.3397461771965027,
+      "learning_rate": 7.989936243606644e-05,
+      "loss": 0.38797500610351565,
+      "mean_token_accuracy": 0.9223182892799378,
+      "num_tokens": 13074583.0,
+      "step": 3050
+    },
+    {
+      "entropy": 0.3658443212881684,
+      "epoch": 0.9353549068416686,
+      "grad_norm": 0.21586942672729492,
+      "learning_rate": 7.925286946831907e-05,
+      "loss": 0.38840934753417966,
+      "mean_token_accuracy": 0.920469640493393,
+      "num_tokens": 13290531.0,
+      "step": 3100
+    },
+    {
+      "entropy": 0.3434870146587491,
+      "epoch": 0.9504412763068567,
+      "grad_norm": 0.4203721582889557,
+      "learning_rate": 7.859885351828722e-05,
+      "loss": 0.3654581832885742,
+      "mean_token_accuracy": 0.9264674150943756,
+      "num_tokens": 13501774.0,
+      "step": 3150
+    },
+    {
+      "entropy": 0.3622769057378173,
+      "epoch": 0.965527645772045,
+      "grad_norm": 0.4923344850540161,
+      "learning_rate": 7.793748277974116e-05,
+      "loss": 0.38648658752441406,
+      "mean_token_accuracy": 0.9216807208955288,
+      "num_tokens": 13719293.0,
+      "step": 3200
+    },
+    {
+      "epoch": 0.965527645772045,
+      "eval_entropy": 0.38819593988441453,
+      "eval_loss": 0.43394002318382263,
+      "eval_mean_token_accuracy": 0.9077828526496887,
+      "eval_num_tokens": 13719293.0,
+      "eval_runtime": 37.2602,
+      "eval_samples_per_second": 58.132,
+      "eval_steps_per_second": 7.273,
+      "step": 3200
+    },
+    {
+      "entropy": 0.35784762669354675,
+      "epoch": 0.9806140152372331,
+      "grad_norm": 0.295482873916626,
+      "learning_rate": 7.72689273378875e-05,
+      "loss": 0.3751881790161133,
+      "mean_token_accuracy": 0.9235683201253414,
+      "num_tokens": 13933952.0,
+      "step": 3250
+    },
+    {
+      "entropy": 0.35060869861394167,
+      "epoch": 0.9957003847024214,
+      "grad_norm": 0.3346746265888214,
+      "learning_rate": 7.659335912562849e-05,
+      "loss": 0.3715841293334961,
+      "mean_token_accuracy": 0.9249463096261025,
+      "num_tokens": 14146736.0,
+      "step": 3300
+    },
+    {
+      "entropy": 0.33144386649736896,
+      "epoch": 1.0105604586256318,
+      "grad_norm": 0.42002174258232117,
+      "learning_rate": 7.591095187934571e-05,
+      "loss": 0.3447689437866211,
+      "mean_token_accuracy": 0.9282537361389489,
+      "num_tokens": 14363628.0,
+      "step": 3350
+    },
+    {
+      "entropy": 0.29580484244972466,
+      "epoch": 1.02564682809082,
+      "grad_norm": 0.267696350812912,
+      "learning_rate": 7.522188109422025e-05,
+      "loss": 0.30982461929321287,
+      "mean_token_accuracy": 0.9349895706772804,
+      "num_tokens": 14577748.0,
+      "step": 3400
+    },
+    {
+      "epoch": 1.02564682809082,
+      "eval_entropy": 0.3474461413452546,
+      "eval_loss": 0.4461125433444977,
+      "eval_mean_token_accuracy": 0.9071816351140997,
+      "eval_num_tokens": 14577748.0,
+      "eval_runtime": 37.1626,
+      "eval_samples_per_second": 58.284,
+      "eval_steps_per_second": 7.292,
+      "step": 3400
+    },
+    {
+      "entropy": 0.30122795451432466,
+      "epoch": 1.0407331975560081,
+      "grad_norm": 0.2817290127277374,
+      "learning_rate": 7.452632397910045e-05,
+      "loss": 0.3127718353271484,
+      "mean_token_accuracy": 0.93441466614604,
+      "num_tokens": 14792632.0,
+      "step": 3450
+    },
+    {
+      "entropy": 0.2921810976602137,
+      "epoch": 1.0558195670211963,
+      "grad_norm": 0.3158903419971466,
+      "learning_rate": 7.3824459410929e-05,
+      "loss": 0.3033764839172363,
+      "mean_token_accuracy": 0.9362939922511577,
+      "num_tokens": 15005289.0,
+      "step": 3500
+    },
+    {
+      "entropy": 0.30137234527617696,
+      "epoch": 1.0709059364863847,
+      "grad_norm": 0.363569974899292,
+      "learning_rate": 7.311646788874093e-05,
+      "loss": 0.3192999267578125,
+      "mean_token_accuracy": 0.9330788172781468,
+      "num_tokens": 15221117.0,
+      "step": 3550
+    },
+    {
+      "entropy": 0.2979462206363678,
+      "epoch": 1.0859923059515728,
+      "grad_norm": 0.33784833550453186,
+      "learning_rate": 7.240253148724472e-05,
+      "loss": 0.3097235298156738,
+      "mean_token_accuracy": 0.935227975398302,
+      "num_tokens": 15433103.0,
+      "step": 3600
+    },
+    {
+      "epoch": 1.0859923059515728,
+      "eval_entropy": 0.33927392830378017,
+      "eval_loss": 0.44939693808555603,
+      "eval_mean_token_accuracy": 0.9080607321429517,
+      "eval_num_tokens": 15433103.0,
+      "eval_runtime": 37.0789,
+      "eval_samples_per_second": 58.416,
+      "eval_steps_per_second": 7.309,
+      "step": 3600
+    },
+    {
+      "entropy": 0.2955490641668439,
+      "epoch": 1.101078675416761,
+      "grad_norm": 0.26954564452171326,
+      "learning_rate": 7.168283380999781e-05,
+      "loss": 0.308115234375,
+      "mean_token_accuracy": 0.9356962423026561,
+      "num_tokens": 15647397.0,
+      "step": 3650
+    },
+    {
+      "entropy": 0.3024994757771492,
+      "epoch": 1.1161650448819491,
+      "grad_norm": 0.34243258833885193,
+      "learning_rate": 7.095755994218929e-05,
+      "loss": 0.3163371467590332,
+      "mean_token_accuracy": 0.933635587990284,
+      "num_tokens": 15862307.0,
+      "step": 3700
+    },
+    {
+      "entropy": 0.28693753194063903,
+      "epoch": 1.1312514143471373,
+      "grad_norm": 0.3753094971179962,
+      "learning_rate": 7.022689640304127e-05,
+      "loss": 0.30058185577392577,
+      "mean_token_accuracy": 0.9367860886454582,
+      "num_tokens": 16077573.0,
+      "step": 3750
+    },
+    {
+      "entropy": 0.2817346750572324,
+      "epoch": 1.1463377838123257,
+      "grad_norm": 0.36651188135147095,
+      "learning_rate": 6.949103109784174e-05,
+      "loss": 0.29092519760131835,
+      "mean_token_accuracy": 0.9392722348868847,
+      "num_tokens": 16287755.0,
+      "step": 3800
+    },
+    {
+      "epoch": 1.1463377838123257,
+      "eval_entropy": 0.3268261263181363,
+      "eval_loss": 0.45524126291275024,
+      "eval_mean_token_accuracy": 0.9080498321909746,
+      "eval_num_tokens": 16287755.0,
+      "eval_runtime": 37.2707,
+      "eval_samples_per_second": 58.115,
+      "eval_steps_per_second": 7.271,
+      "step": 3800
+    },
+    {
+      "entropy": 0.2927645993977785,
+      "epoch": 1.1614241532775138,
+      "grad_norm": 0.32257115840911865,
+      "learning_rate": 6.875015326962071e-05,
+      "loss": 0.3056098175048828,
+      "mean_token_accuracy": 0.9361269493401051,
+      "num_tokens": 16500869.0,
+      "step": 3850
+    },
+    {
+      "entropy": 0.2784220616333187,
+      "epoch": 1.176510522742702,
+      "grad_norm": 0.3277973234653473,
+      "learning_rate": 6.800445345048247e-05,
+      "loss": 0.2863511276245117,
+      "mean_token_accuracy": 0.9394664193689823,
+      "num_tokens": 16714052.0,
+      "step": 3900
+    },
+    {
+      "entropy": 0.28824312970042226,
+      "epoch": 1.1915968922078901,
+      "grad_norm": 0.45423227548599243,
+      "learning_rate": 6.725412341260632e-05,
+      "loss": 0.30036584854125975,
+      "mean_token_accuracy": 0.9367516180872917,
+      "num_tokens": 16929433.0,
+      "step": 3950
+    },
+    {
+      "entropy": 0.3048727669939399,
+      "epoch": 1.2066832616730783,
+      "grad_norm": 0.4174216091632843,
+      "learning_rate": 6.649935611892839e-05,
+      "loss": 0.3221250534057617,
+      "mean_token_accuracy": 0.9329589973390102,
+      "num_tokens": 17149131.0,
+      "step": 4000
+    },
+    {
+      "epoch": 1.2066832616730783,
+      "eval_entropy": 0.3393887681439794,
+      "eval_loss": 0.44970703125,
+      "eval_mean_token_accuracy": 0.907502448866728,
+      "eval_num_tokens": 17149131.0,
+      "eval_runtime": 37.1753,
+      "eval_samples_per_second": 58.265,
+      "eval_steps_per_second": 7.29,
+      "step": 4000
+    },
+    {
+      "entropy": 0.2786213347688317,
+      "epoch": 1.2217696311382666,
+      "grad_norm": 0.37704846262931824,
+      "learning_rate": 6.574034567351717e-05,
+      "loss": 0.28641138076782224,
+      "mean_token_accuracy": 0.9393399634957313,
+      "num_tokens": 17362785.0,
+      "step": 4050
+    },
+    {
+      "entropy": 0.2889704497903585,
+      "epoch": 1.2368560006034548,
+      "grad_norm": 0.3806622624397278,
+      "learning_rate": 6.49772872716557e-05,
+      "loss": 0.30537261962890627,
+      "mean_token_accuracy": 0.9361220151185989,
+      "num_tokens": 17578670.0,
+      "step": 4100
+    },
+    {
+      "entropy": 0.29292779937386515,
+      "epoch": 1.251942370068643,
+      "grad_norm": 0.3169683814048767,
+      "learning_rate": 6.421037714964293e-05,
+      "loss": 0.30579683303833005,
+      "mean_token_accuracy": 0.9352342364192009,
+      "num_tokens": 17802008.0,
+      "step": 4150
+    },
+    {
+      "entropy": 0.28012287182733414,
+      "epoch": 1.267028739533831,
+      "grad_norm": 0.2556808888912201,
+      "learning_rate": 6.343981253432761e-05,
+      "loss": 0.2942901611328125,
+      "mean_token_accuracy": 0.93847431153059,
+      "num_tokens": 18015619.0,
+      "step": 4200
+    },
+    {
+      "epoch": 1.267028739533831,
+      "eval_entropy": 0.3326790584354383,
+      "eval_loss": 0.45203062891960144,
+      "eval_mean_token_accuracy": 0.9082723077372871,
+      "eval_num_tokens": 18015619.0,
+      "eval_runtime": 37.2103,
+      "eval_samples_per_second": 58.21,
+      "eval_steps_per_second": 7.283,
+      "step": 4200
+    },
+    {
+      "entropy": 0.2790189484320581,
+      "epoch": 1.2821151089990193,
+      "grad_norm": 0.3167658746242523,
+      "learning_rate": 6.266579159238723e-05,
+      "loss": 0.2901947212219238,
+      "mean_token_accuracy": 0.9385272514820099,
+      "num_tokens": 18231214.0,
+      "step": 4250
+    },
+    {
+      "entropy": 0.28492060406133535,
+      "epoch": 1.2972014784642076,
+      "grad_norm": 0.3428349792957306,
+      "learning_rate": 6.188851337936541e-05,
+      "loss": 0.2975013542175293,
+      "mean_token_accuracy": 0.9384494511783124,
+      "num_tokens": 18443602.0,
+      "step": 4300
+    },
+    {
+      "entropy": 0.28728573814034464,
+      "epoch": 1.3122878479293958,
+      "grad_norm": 0.3879326581954956,
+      "learning_rate": 6.110817778848062e-05,
+      "loss": 0.2947955894470215,
+      "mean_token_accuracy": 0.9376472027599811,
+      "num_tokens": 18658957.0,
+      "step": 4350
+    },
+    {
+      "entropy": 0.27078360263258217,
+      "epoch": 1.327374217394584,
+      "grad_norm": 0.45004045963287354,
+      "learning_rate": 6.032498549921939e-05,
+      "loss": 0.28305835723876954,
+      "mean_token_accuracy": 0.9406134666502476,
+      "num_tokens": 18870053.0,
+      "step": 4400
+    },
+    {
+      "epoch": 1.327374217394584,
+      "eval_entropy": 0.3294620155958232,
+      "eval_loss": 0.4543997049331665,
+      "eval_mean_token_accuracy": 0.9081211842294109,
+      "eval_num_tokens": 18870053.0,
+      "eval_runtime": 37.2524,
+      "eval_samples_per_second": 58.144,
+      "eval_steps_per_second": 7.275,
+      "step": 4400
+    },
+    {
+      "entropy": 0.2799815020523965,
+      "epoch": 1.3424605868597723,
+      "grad_norm": 0.44520097970962524,
+      "learning_rate": 5.953913792572761e-05,
+      "loss": 0.28622047424316405,
+      "mean_token_accuracy": 0.9398869612812996,
+      "num_tokens": 19083502.0,
+      "step": 4450
+    },
+    {
+      "entropy": 0.28538848023861646,
+      "epoch": 1.3575469563249605,
+      "grad_norm": 0.376067578792572,
+      "learning_rate": 5.8750837165012516e-05,
+      "loss": 0.29442657470703126,
+      "mean_token_accuracy": 0.9369049946963787,
+      "num_tokens": 19301474.0,
+      "step": 4500
+    },
+    {
+      "entropy": 0.2700873460620642,
+      "epoch": 1.3726333257901486,
+      "grad_norm": 0.3803156316280365,
+      "learning_rate": 5.79602859449694e-05,
+      "loss": 0.2789628601074219,
+      "mean_token_accuracy": 0.9409016688168049,
+      "num_tokens": 19513118.0,
+      "step": 4550
+    },
+    {
+      "entropy": 0.2818992605619133,
+      "epoch": 1.3877196952553368,
+      "grad_norm": 0.3878261148929596,
+      "learning_rate": 5.716768757224583e-05,
+      "loss": 0.291706428527832,
+      "mean_token_accuracy": 0.9382703882455826,
+      "num_tokens": 19729704.0,
+      "step": 4600
+    },
+    {
+      "epoch": 1.3877196952553368,
+      "eval_entropy": 0.3223673155551907,
+      "eval_loss": 0.4583451747894287,
+      "eval_mean_token_accuracy": 0.9085318800268138,
+      "eval_num_tokens": 19729704.0,
+      "eval_runtime": 37.2126,
+      "eval_samples_per_second": 58.206,
+      "eval_steps_per_second": 7.282,
+      "step": 4600
+    },
+    {
+      "entropy": 0.2773911802843213,
+      "epoch": 1.402806064720525,
+      "grad_norm": 0.4441978931427002,
+      "learning_rate": 5.637324587995714e-05,
+      "loss": 0.28889572143554687,
+      "mean_token_accuracy": 0.9393369700014591,
+      "num_tokens": 19941579.0,
+      "step": 4650
+    },
+    {
+      "entropy": 0.2760587067343295,
+      "epoch": 1.4178924341857133,
+      "grad_norm": 0.31453457474708557,
+      "learning_rate": 5.557716517526656e-05,
+      "loss": 0.2823097038269043,
+      "mean_token_accuracy": 0.9404082801938057,
+      "num_tokens": 20152481.0,
+      "step": 4700
+    },
+    {
+      "entropy": 0.2687125858291984,
+      "epoch": 1.4329788036509015,
+      "grad_norm": 0.4302677810192108,
+      "learning_rate": 5.4779650186843335e-05,
+      "loss": 0.28021940231323245,
+      "mean_token_accuracy": 0.9412896266579628,
+      "num_tokens": 20364280.0,
+      "step": 4750
+    },
+    {
+      "entropy": 0.28299310375005005,
+      "epoch": 1.4480651731160896,
+      "grad_norm": 0.4275122582912445,
+      "learning_rate": 5.3980906012212575e-05,
+      "loss": 0.29417877197265624,
+      "mean_token_accuracy": 0.9380144135653973,
+      "num_tokens": 20579080.0,
+      "step": 4800
+    },
+    {
+      "epoch": 1.4480651731160896,
+      "eval_entropy": 0.3136782179899321,
+      "eval_loss": 0.4654751420021057,
+      "eval_mean_token_accuracy": 0.9077895956725652,
+      "eval_num_tokens": 20579080.0,
+      "eval_runtime": 37.1335,
+      "eval_samples_per_second": 58.33,
+      "eval_steps_per_second": 7.298,
+      "step": 4800
+    },
+    {
+      "entropy": 0.26981769694015384,
+      "epoch": 1.4631515425812778,
+      "grad_norm": 0.3479642868041992,
+      "learning_rate": 5.318113806501009e-05,
+      "loss": 0.2793255615234375,
+      "mean_token_accuracy": 0.940917958021164,
+      "num_tokens": 20794818.0,
+      "step": 4850
+    },
+    {
+      "entropy": 0.27721120171248914,
+      "epoch": 1.478237912046466,
+      "grad_norm": 0.36213237047195435,
+      "learning_rate": 5.238055202215606e-05,
+      "loss": 0.2876263427734375,
+      "mean_token_accuracy": 0.9395251777768135,
+      "num_tokens": 21007893.0,
+      "step": 4900
+    },
+    {
+      "entropy": 0.2697361321747303,
+      "epoch": 1.4933242815116543,
+      "grad_norm": 0.3829312324523926,
+      "learning_rate": 5.157935377096091e-05,
+      "loss": 0.2791532325744629,
+      "mean_token_accuracy": 0.9417897801101208,
+      "num_tokens": 21221698.0,
+      "step": 4950
+    },
+    {
+      "entropy": 0.26987188443541527,
+      "epoch": 1.5084106509768425,
+      "grad_norm": 0.5338497757911682,
+      "learning_rate": 5.077774935617715e-05,
+      "loss": 0.2768931770324707,
+      "mean_token_accuracy": 0.9414016118645668,
+      "num_tokens": 21434912.0,
+      "step": 5000
+    },
+    {
+      "epoch": 1.5084106509768425,
+      "eval_entropy": 0.31743444043221947,
+      "eval_loss": 0.46169763803482056,
+      "eval_mean_token_accuracy": 0.9076986282074143,
+      "eval_num_tokens": 21434912.0,
+      "eval_runtime": 37.2102,
+      "eval_samples_per_second": 58.21,
+      "eval_steps_per_second": 7.283,
+      "step": 5000
+    },
+    {
+      "entropy": 0.27476332798600195,
+      "epoch": 1.5234970204420306,
+      "grad_norm": 0.3199191987514496,
+      "learning_rate": 4.997594492701074e-05,
+      "loss": 0.2819905471801758,
+      "mean_token_accuracy": 0.9403638400137424,
+      "num_tokens": 21650284.0,
+      "step": 5050
+    },
+    {
+      "entropy": 0.2720643111690879,
+      "epoch": 1.538583389907219,
+      "grad_norm": 0.49105435609817505,
+      "learning_rate": 4.917414668410541e-05,
+      "loss": 0.28312482833862307,
+      "mean_token_accuracy": 0.9400044956803322,
+      "num_tokens": 21864950.0,
+      "step": 5100
+    },
+    {
+      "entropy": 0.2742350418306887,
+      "epoch": 1.553669759372407,
+      "grad_norm": 0.3570726215839386,
+      "learning_rate": 4.837256082651398e-05,
+      "loss": 0.2866826057434082,
+      "mean_token_accuracy": 0.940371020436287,
+      "num_tokens": 22081595.0,
+      "step": 5150
+    },
+    {
+      "entropy": 0.26034254603087903,
+      "epoch": 1.5687561288375953,
+      "grad_norm": 0.37878358364105225,
+      "learning_rate": 4.757139349867003e-05,
+      "loss": 0.26901472091674805,
+      "mean_token_accuracy": 0.943619585186243,
+      "num_tokens": 22293286.0,
+      "step": 5200
+    },
+    {
+      "epoch": 1.5687561288375953,
+      "eval_entropy": 0.3216313538021267,
+      "eval_loss": 0.45686572790145874,
+      "eval_mean_token_accuracy": 0.908281669625497,
+      "eval_num_tokens": 22293286.0,
+      "eval_runtime": 37.2587,
+      "eval_samples_per_second": 58.134,
+      "eval_steps_per_second": 7.273,
+      "step": 5200
+    },
+    {
+      "entropy": 0.25965558482334017,
+      "epoch": 1.5838424983027835,
+      "grad_norm": 0.5104638338088989,
+      "learning_rate": 4.677085073737347e-05,
+      "loss": 0.2651633071899414,
+      "mean_token_accuracy": 0.943703119456768,
+      "num_tokens": 22507183.0,
+      "step": 5250
+    },
+    {
+      "entropy": 0.2634627244435251,
+      "epoch": 1.5989288677679716,
+      "grad_norm": 0.39737868309020996,
+      "learning_rate": 4.5971138418804024e-05,
+      "loss": 0.2716307067871094,
+      "mean_token_accuracy": 0.9426473924517631,
+      "num_tokens": 22719425.0,
+      "step": 5300
+    },
+    {
+      "entropy": 0.26387994691729544,
+      "epoch": 1.61401523723316,
+      "grad_norm": 0.3169187605381012,
+      "learning_rate": 4.517246220557577e-05,
+      "loss": 0.2690562057495117,
+      "mean_token_accuracy": 0.9435765042901039,
+      "num_tokens": 22935863.0,
+      "step": 5350
+    },
+    {
+      "entropy": 0.2732780100405216,
+      "epoch": 1.629101606698348,
+      "grad_norm": 0.5206496715545654,
+      "learning_rate": 4.437502749384686e-05,
+      "loss": 0.28528207778930664,
+      "mean_token_accuracy": 0.9399016280472279,
+      "num_tokens": 23150285.0,
+      "step": 5400
+    },
+    {
+      "epoch": 1.629101606698348,
+      "eval_entropy": 0.3117673156776111,
+      "eval_loss": 0.4644741117954254,
+      "eval_mean_token_accuracy": 0.9071205707053857,
+      "eval_num_tokens": 23150285.0,
+      "eval_runtime": 37.233,
+      "eval_samples_per_second": 58.174,
+      "eval_steps_per_second": 7.278,
+      "step": 5400
+    },
+    {
+      "entropy": 0.25959222281351685,
+      "epoch": 1.6441879761635363,
+      "grad_norm": 0.37161391973495483,
+      "learning_rate": 4.357903936049758e-05,
+      "loss": 0.26540708541870117,
+      "mean_token_accuracy": 0.9441601178050041,
+      "num_tokens": 23361061.0,
+      "step": 5450
+    },
+    {
+      "entropy": 0.258843588642776,
+      "epoch": 1.6592743456287244,
+      "grad_norm": 0.3901636302471161,
+      "learning_rate": 4.278470251039052e-05,
+      "loss": 0.2657164001464844,
+      "mean_token_accuracy": 0.9435085739195347,
+      "num_tokens": 23576068.0,
+      "step": 5500
+    },
+    {
+      "entropy": 0.2609032719023526,
+      "epoch": 1.6743607150939126,
+      "grad_norm": 0.3928777873516083,
+      "learning_rate": 4.199222122372658e-05,
+      "loss": 0.2640374183654785,
+      "mean_token_accuracy": 0.9443163026869297,
+      "num_tokens": 23787002.0,
+      "step": 5550
+    },
+    {
+      "entropy": 0.2564471643790603,
+      "epoch": 1.689447084559101,
+      "grad_norm": 0.3459172546863556,
+      "learning_rate": 4.1201799303509984e-05,
+      "loss": 0.2629901695251465,
+      "mean_token_accuracy": 0.9437471415102482,
+      "num_tokens": 23997298.0,
+      "step": 5600
+    },
+    {
+      "epoch": 1.689447084559101,
+      "eval_entropy": 0.30813075843992266,
+      "eval_loss": 0.46801915764808655,
+      "eval_mean_token_accuracy": 0.9075211451502304,
+      "eval_num_tokens": 23997298.0,
+      "eval_runtime": 37.1802,
+      "eval_samples_per_second": 58.257,
+      "eval_steps_per_second": 7.289,
+      "step": 5600
+    },
+    {
+      "entropy": 0.26069454461336133,
+      "epoch": 1.704533454024289,
+      "grad_norm": 0.4609261453151703,
+      "learning_rate": 4.041364002313619e-05,
+      "loss": 0.26864715576171877,
+      "mean_token_accuracy": 0.9432297512888909,
+      "num_tokens": 24213258.0,
+      "step": 5650
+    },
+    {
+      "entropy": 0.26323398426175115,
+      "epoch": 1.7196198234894773,
+      "grad_norm": 0.4566548764705658,
+      "learning_rate": 3.962794607411584e-05,
+      "loss": 0.2684891510009766,
+      "mean_token_accuracy": 0.9431661969423294,
+      "num_tokens": 24429089.0,
+      "step": 5700
+    },
+    {
+      "entropy": 0.26415870275348424,
+      "epoch": 1.7347061929546654,
+      "grad_norm": 0.454801082611084,
+      "learning_rate": 3.8844919513948486e-05,
+      "loss": 0.2741558265686035,
+      "mean_token_accuracy": 0.9412616415321827,
+      "num_tokens": 24646888.0,
+      "step": 5750
+    },
+    {
+      "entropy": 0.25849502749741077,
+      "epoch": 1.7497925624198536,
+      "grad_norm": 0.47254759073257446,
+      "learning_rate": 3.80647617141593e-05,
+      "loss": 0.26054887771606444,
+      "mean_token_accuracy": 0.94430311486125,
+      "num_tokens": 24861569.0,
+      "step": 5800
+    },
+    {
+      "epoch": 1.7497925624198536,
+      "eval_entropy": 0.29998021306146994,
+      "eval_loss": 0.47340917587280273,
+      "eval_mean_token_accuracy": 0.9083721395788158,
+      "eval_num_tokens": 24861569.0,
+      "eval_runtime": 37.3662,
+      "eval_samples_per_second": 57.967,
+      "eval_steps_per_second": 7.253,
+      "step": 5800
+    },
+    {
+      "entropy": 0.2529403425566852,
+      "epoch": 1.764878931885042,
+      "grad_norm": 0.45025748014450073,
+      "learning_rate": 3.7287673308512186e-05,
+      "loss": 0.26045492172241214,
+      "mean_token_accuracy": 0.9441630898416042,
+      "num_tokens": 25079317.0,
+      "step": 5850
+    },
+    {
+      "entropy": 0.2550449293479323,
+      "epoch": 1.7799653013502301,
+      "grad_norm": 0.4205138087272644,
+      "learning_rate": 3.65138541414127e-05,
+      "loss": 0.26142183303833005,
+      "mean_token_accuracy": 0.9442940099537372,
+      "num_tokens": 25293828.0,
+      "step": 5900
+    },
+    {
+      "entropy": 0.26166333435103295,
+      "epoch": 1.7950516708154183,
+      "grad_norm": 0.32279399037361145,
+      "learning_rate": 3.5743503216513875e-05,
+      "loss": 0.27175371170043944,
+      "mean_token_accuracy": 0.9426822887361049,
+      "num_tokens": 25509857.0,
+      "step": 5950
+    },
+    {
+      "entropy": 0.2534090391360223,
+      "epoch": 1.8101380402806064,
+      "grad_norm": 0.4058147370815277,
+      "learning_rate": 3.4976818645538266e-05,
+      "loss": 0.2650087547302246,
+      "mean_token_accuracy": 0.9448564203083515,
+      "num_tokens": 25722906.0,
+      "step": 6000
+    },
+    {
+      "epoch": 1.8101380402806064,
+      "eval_entropy": 0.3064177740379013,
+      "eval_loss": 0.4730622470378876,
+      "eval_mean_token_accuracy": 0.9072717905484442,
+      "eval_num_tokens": 25722906.0,
+      "eval_runtime": 37.3208,
+      "eval_samples_per_second": 58.037,
+      "eval_steps_per_second": 7.261,
+      "step": 6000
+    },
+    {
+      "entropy": 0.2568224469386041,
+      "epoch": 1.8252244097457946,
+      "grad_norm": 0.7752001285552979,
+      "learning_rate": 3.42139975973295e-05,
+      "loss": 0.26035600662231445,
+      "mean_token_accuracy": 0.9453255115449428,
+      "num_tokens": 25934089.0,
+      "step": 6050
+    },
+    {
+      "entropy": 0.25038454053923487,
+      "epoch": 1.840310779210983,
+      "grad_norm": 0.657081127166748,
+      "learning_rate": 3.3455236247146124e-05,
+      "loss": 0.25294511795043945,
+      "mean_token_accuracy": 0.9468451295793057,
+      "num_tokens": 26145757.0,
+      "step": 6100
+    },
+    {
+      "entropy": 0.25195862116292117,
+      "epoch": 1.8553971486761711,
+      "grad_norm": 0.4311438202857971,
+      "learning_rate": 3.270072972621116e-05,
+      "loss": 0.2561396598815918,
+      "mean_token_accuracy": 0.945042757242918,
+      "num_tokens": 26363126.0,
+      "step": 6150
+    },
+    {
+      "entropy": 0.24140722384676339,
+      "epoch": 1.8704835181413593,
+      "grad_norm": 0.3511962592601776,
+      "learning_rate": 3.1950672071529976e-05,
+      "loss": 0.2441703224182129,
+      "mean_token_accuracy": 0.9480637626349926,
+      "num_tokens": 26570762.0,
+      "step": 6200
+    },
+    {
+      "epoch": 1.8704835181413593,
+      "eval_entropy": 0.29495117315726965,
+      "eval_loss": 0.4819435179233551,
+      "eval_mean_token_accuracy": 0.9071202113179703,
+      "eval_num_tokens": 26570762.0,
+      "eval_runtime": 37.2485,
+      "eval_samples_per_second": 58.15,
+      "eval_steps_per_second": 7.275,
+      "step": 6200
+    },
+    {
+      "entropy": 0.25350754015147686,
+      "epoch": 1.8855698876065476,
+      "grad_norm": 0.4511338472366333,
+      "learning_rate": 3.1205256175989706e-05,
+      "loss": 0.2587089538574219,
+      "mean_token_accuracy": 0.9447831605374813,
+      "num_tokens": 26786633.0,
+      "step": 6250
+    },
+    {
+      "entropy": 0.24989185376092793,
+      "epoch": 1.9006562570717356,
+      "grad_norm": 0.6432870030403137,
+      "learning_rate": 3.046467373875282e-05,
+      "loss": 0.2545445251464844,
+      "mean_token_accuracy": 0.9456490239501,
+      "num_tokens": 26999534.0,
+      "step": 6300
+    },
+    {
+      "entropy": 0.2594654941000044,
+      "epoch": 1.915742626536924,
+      "grad_norm": 0.3382566273212433,
+      "learning_rate": 2.9729115215957664e-05,
+      "loss": 0.26606035232543945,
+      "mean_token_accuracy": 0.9442368443310261,
+      "num_tokens": 27215468.0,
+      "step": 6350
+    },
+    {
+      "entropy": 0.2511021154373884,
+      "epoch": 1.930828996002112,
+      "grad_norm": 0.5591769814491272,
+      "learning_rate": 2.8998769771738833e-05,
+      "loss": 0.25475397109985354,
+      "mean_token_accuracy": 0.9452549339830876,
+      "num_tokens": 27431751.0,
+      "step": 6400
+    },
+    {
+      "epoch": 1.930828996002112,
+      "eval_entropy": 0.2883295046777303,
+      "eval_loss": 0.4905667006969452,
+      "eval_mean_token_accuracy": 0.9074331280050243,
+      "eval_num_tokens": 27431751.0,
+      "eval_runtime": 37.2686,
+      "eval_samples_per_second": 58.119,
+      "eval_steps_per_second": 7.272,
+      "step": 6400
+    },
+    {
+      "entropy": 0.24450184650719164,
+      "epoch": 1.9459153654673003,
+      "grad_norm": 0.4845283031463623,
+      "learning_rate": 2.8273825229579558e-05,
+      "loss": 0.2493889808654785,
+      "mean_token_accuracy": 0.9475250770151615,
+      "num_tokens": 27643957.0,
+      "step": 6450
+    },
+    {
+      "entropy": 0.25026212429627775,
+      "epoch": 1.9610017349324886,
+      "grad_norm": 0.4210788309574127,
+      "learning_rate": 2.7554468024009082e-05,
+      "loss": 0.25582372665405273,
+      "mean_token_accuracy": 0.9451599711179733,
+      "num_tokens": 27862796.0,
+      "step": 6500
+    },
+    {
+      "entropy": 0.2594556957855821,
+      "epoch": 1.9760881043976766,
+      "grad_norm": 0.42034855484962463,
+      "learning_rate": 2.6840883152657075e-05,
+      "loss": 0.2614474105834961,
+      "mean_token_accuracy": 0.9446786725521088,
+      "num_tokens": 28080081.0,
+      "step": 6550
+    },
+    {
+      "entropy": 0.2424509571492672,
+      "epoch": 1.991174473862865,
+      "grad_norm": 0.5637903213500977,
+      "learning_rate": 2.6133254128677766e-05,
+      "loss": 0.24372596740722657,
+      "mean_token_accuracy": 0.9479729002714157,
+      "num_tokens": 28291753.0,
+      "step": 6600
+    },
+    {
+      "epoch": 1.991174473862865,
+      "eval_entropy": 0.2895552223458941,
+      "eval_loss": 0.48857003450393677,
+      "eval_mean_token_accuracy": 0.9079141579430922,
+      "eval_num_tokens": 28291753.0,
+      "eval_runtime": 37.2416,
+      "eval_samples_per_second": 58.161,
+      "eval_steps_per_second": 7.277,
+      "step": 6600
+    },
+    {
+      "entropy": 0.24137189186315247,
+      "epoch": 2.0060345477860753,
+      "grad_norm": 0.4415420591831207,
+      "learning_rate": 2.5431762933555627e-05,
+      "loss": 0.23770923614501954,
+      "mean_token_accuracy": 0.9500614218602931,
+      "num_tokens": 28501531.0,
+      "step": 6650
+    },
+    {
+      "entropy": 0.20573811857029795,
+      "epoch": 2.0211209172512636,
+      "grad_norm": 0.5171937942504883,
+      "learning_rate": 2.473658997030514e-05,
+      "loss": 0.19352760314941406,
+      "mean_token_accuracy": 0.9575251358747482,
+      "num_tokens": 28718312.0,
+      "step": 6700
+    },
+    {
+      "entropy": 0.20038287844508887,
+      "epoch": 2.0362072867164516,
+      "grad_norm": 0.42165201902389526,
+      "learning_rate": 2.404791401707633e-05,
+      "loss": 0.18894313812255858,
+      "mean_token_accuracy": 0.9588627055287361,
+      "num_tokens": 28934285.0,
+      "step": 6750
+    },
+    {
+      "entropy": 0.19508995985612274,
+      "epoch": 2.05129365618164,
+      "grad_norm": 0.35577845573425293,
+      "learning_rate": 2.3365912181178313e-05,
+      "loss": 0.1867695426940918,
+      "mean_token_accuracy": 0.9598716080188752,
+      "num_tokens": 29145157.0,
+      "step": 6800
+    },
+    {
+      "epoch": 2.05129365618164,
+      "eval_entropy": 0.2568544143350362,
+      "eval_loss": 0.5305211544036865,
+      "eval_mean_token_accuracy": 0.9064581629534929,
+      "eval_num_tokens": 29145157.0,
+      "eval_runtime": 37.1895,
+      "eval_samples_per_second": 58.242,
+      "eval_steps_per_second": 7.287,
+      "step": 6800
+    },
+    {
+      "entropy": 0.2055508348532021,
+      "epoch": 2.066380025646828,
+      "grad_norm": 0.42946624755859375,
+      "learning_rate": 2.269075985353254e-05,
+      "loss": 0.1946415138244629,
+      "mean_token_accuracy": 0.9576617389917373,
+      "num_tokens": 29362502.0,
+      "step": 6850
+    },
+    {
+      "entropy": 0.19851462194696068,
+      "epoch": 2.0814663951120163,
+      "grad_norm": 0.4784873127937317,
+      "learning_rate": 2.20226306635672e-05,
+      "loss": 0.1882442855834961,
+      "mean_token_accuracy": 0.958879132270813,
+      "num_tokens": 29577964.0,
+      "step": 6900
+    },
+    {
+      "entropy": 0.20270802198909224,
+      "epoch": 2.0965527645772046,
+      "grad_norm": 0.5308135151863098,
+      "learning_rate": 2.1361696434564993e-05,
+      "loss": 0.1924464225769043,
+      "mean_token_accuracy": 0.9582864837348462,
+      "num_tokens": 29792051.0,
+      "step": 6950
+    },
+    {
+      "entropy": 0.19360160086303949,
+      "epoch": 2.1116391340423926,
+      "grad_norm": 0.3992004990577698,
+      "learning_rate": 2.070812713947513e-05,
+      "loss": 0.18389596939086914,
+      "mean_token_accuracy": 0.9606801150739193,
+      "num_tokens": 30010593.0,
+      "step": 7000
+    },
+    {
+      "epoch": 2.1116391340423926,
+      "eval_entropy": 0.2496587313794122,
+      "eval_loss": 0.5409784317016602,
+      "eval_mean_token_accuracy": 0.9062512767710809,
+      "eval_num_tokens": 30010593.0,
+      "eval_runtime": 37.2073,
+      "eval_samples_per_second": 58.214,
+      "eval_steps_per_second": 7.284,
+      "step": 7000
+    },
+    {
+      "entropy": 0.19243362288922072,
+      "epoch": 2.126725503507581,
+      "grad_norm": 0.485505610704422,
+      "learning_rate": 2.006209085720122e-05,
+      "loss": 0.18137836456298828,
+      "mean_token_accuracy": 0.9603879378736019,
+      "num_tokens": 30224235.0,
+      "step": 7050
+    },
+    {
+      "entropy": 0.1971529097482562,
+      "epoch": 2.1418118729727693,
+      "grad_norm": 0.3532182276248932,
+      "learning_rate": 1.9423753729376388e-05,
+      "loss": 0.1862693214416504,
+      "mean_token_accuracy": 0.959182897657156,
+      "num_tokens": 30438574.0,
+      "step": 7100
+    },
+    {
+      "entropy": 0.19837678778916598,
+      "epoch": 2.1568982424379572,
+      "grad_norm": 0.5558100342750549,
+      "learning_rate": 1.8793279917636448e-05,
+      "loss": 0.1895616340637207,
+      "mean_token_accuracy": 0.958885507285595,
+      "num_tokens": 30654293.0,
+      "step": 7150
+    },
+    {
+      "entropy": 0.19329896425828338,
+      "epoch": 2.1719846119031456,
+      "grad_norm": 0.4981042146682739,
+      "learning_rate": 1.817083156140237e-05,
+      "loss": 0.18090120315551758,
+      "mean_token_accuracy": 0.9607469955086708,
+      "num_tokens": 30865232.0,
+      "step": 7200
+    },
+    {
+      "epoch": 2.1719846119031456,
+      "eval_entropy": 0.2497983215534819,
+      "eval_loss": 0.5419089198112488,
+      "eval_mean_token_accuracy": 0.9055907944031747,
+      "eval_num_tokens": 30865232.0,
+      "eval_runtime": 37.1887,
+      "eval_samples_per_second": 58.243,
+      "eval_steps_per_second": 7.287,
+      "step": 7200
+    },
+    {
+      "entropy": 0.19074429869651793,
+      "epoch": 2.1870709813683336,
+      "grad_norm": 0.5064796209335327,
+      "learning_rate": 1.7556568736182667e-05,
+      "loss": 0.18279983520507812,
+      "mean_token_accuracy": 0.9610813467204571,
+      "num_tokens": 31074859.0,
+      "step": 7250
+    },
+    {
+      "entropy": 0.18915067540481686,
+      "epoch": 2.202157350833522,
+      "grad_norm": 0.462493360042572,
+      "learning_rate": 1.6950649412406777e-05,
+      "loss": 0.18141572952270507,
+      "mean_token_accuracy": 0.9607113988697529,
+      "num_tokens": 31285366.0,
+      "step": 7300
+    },
+    {
+      "entropy": 0.19164147678762675,
+      "epoch": 2.2172437202987103,
+      "grad_norm": 0.44526222348213196,
+      "learning_rate": 1.63532294147997e-05,
+      "loss": 0.1814894676208496,
+      "mean_token_accuracy": 0.9610688266158104,
+      "num_tokens": 31497431.0,
+      "step": 7350
+    },
+    {
+      "entropy": 0.19515870379284025,
+      "epoch": 2.2323300897638982,
+      "grad_norm": 0.5659943222999573,
+      "learning_rate": 1.5764462382308254e-05,
+      "loss": 0.18695903778076173,
+      "mean_token_accuracy": 0.9594297151267529,
+      "num_tokens": 31711142.0,
+      "step": 7400
+    },
+    {
+      "epoch": 2.2323300897638982,
+      "eval_entropy": 0.24821185405843812,
+      "eval_loss": 0.5459219217300415,
+      "eval_mean_token_accuracy": 0.9052295550649017,
+      "eval_num_tokens": 31711142.0,
+      "eval_runtime": 37.1776,
+      "eval_samples_per_second": 58.261,
+      "eval_steps_per_second": 7.289,
+      "step": 7400
+    },
+    {
+      "entropy": 0.19548690050840378,
+      "epoch": 2.2474164592290866,
+      "grad_norm": 0.498912513256073,
+      "learning_rate": 1.5184499728589846e-05,
+      "loss": 0.18344196319580078,
+      "mean_token_accuracy": 0.9600871665775776,
+      "num_tokens": 31923379.0,
+      "step": 7450
+    },
+    {
+      "entropy": 0.19262660641223192,
+      "epoch": 2.2625028286942745,
+      "grad_norm": 0.41678550839424133,
+      "learning_rate": 1.4613490603073194e-05,
+      "loss": 0.18303531646728516,
+      "mean_token_accuracy": 0.9605987747013569,
+      "num_tokens": 32135266.0,
+      "step": 7500
+    },
+    {
+      "entropy": 0.19937248670496047,
+      "epoch": 2.277589198159463,
+      "grad_norm": 0.6035925149917603,
+      "learning_rate": 1.4051581852601492e-05,
+      "loss": 0.18883104324340821,
+      "mean_token_accuracy": 0.9591751427948475,
+      "num_tokens": 32351854.0,
+      "step": 7550
+    },
+    {
+      "entropy": 0.19550999199971555,
+      "epoch": 2.2926755676246513,
+      "grad_norm": 0.4360465705394745,
+      "learning_rate": 1.3498917983667658e-05,
+      "loss": 0.18691642761230468,
+      "mean_token_accuracy": 0.9598778022825718,
+      "num_tokens": 32567025.0,
+      "step": 7600
+    },
+    {
+      "epoch": 2.2926755676246513,
+      "eval_entropy": 0.24827074039576238,
+      "eval_loss": 0.5438116192817688,
+      "eval_mean_token_accuracy": 0.9056458268658262,
+      "eval_num_tokens": 32567025.0,
+      "eval_runtime": 37.13,
+      "eval_samples_per_second": 58.336,
+      "eval_steps_per_second": 7.299,
+      "step": 7600
+    },
+    {
+      "entropy": 0.19866504015401007,
+      "epoch": 2.3077619370898392,
+      "grad_norm": 0.4768233299255371,
+      "learning_rate": 1.2955641125251555e-05,
+      "loss": 0.18666345596313477,
+      "mean_token_accuracy": 0.9592089447379112,
+      "num_tokens": 32781090.0,
+      "step": 7650
+    },
+    {
+      "entropy": 0.19146794026717542,
+      "epoch": 2.3228483065550276,
+      "grad_norm": 0.392702579498291,
+      "learning_rate": 1.2421890992268542e-05,
+      "loss": 0.18030118942260742,
+      "mean_token_accuracy": 0.9610011300444603,
+      "num_tokens": 32993619.0,
+      "step": 7700
+    },
+    {
+      "entropy": 0.19083258060738445,
+      "epoch": 2.3379346760202155,
+      "grad_norm": 0.5067113637924194,
+      "learning_rate": 1.189780484963885e-05,
+      "loss": 0.17912176132202148,
+      "mean_token_accuracy": 0.9609163939952851,
+      "num_tokens": 33207287.0,
+      "step": 7750
+    },
+    {
+      "entropy": 0.19270094383507966,
+      "epoch": 2.353021045485404,
+      "grad_norm": 0.3395691215991974,
+      "learning_rate": 1.138351747698711e-05,
+      "loss": 0.18421865463256837,
+      "mean_token_accuracy": 0.9601799823343754,
+      "num_tokens": 33423228.0,
+      "step": 7800
+    },
+    {
+      "epoch": 2.353021045485404,
+      "eval_entropy": 0.2481674429895253,
+      "eval_loss": 0.547441840171814,
+      "eval_mean_token_accuracy": 0.9053705822937603,
+      "eval_num_tokens": 33423228.0,
+      "eval_runtime": 37.2459,
+      "eval_samples_per_second": 58.154,
+      "eval_steps_per_second": 7.276,
+      "step": 7800
+    },
+    {
+      "entropy": 0.1889958502538502,
+      "epoch": 2.3681074149505923,
+      "grad_norm": 0.4847002327442169,
+      "learning_rate": 1.0879161133981014e-05,
+      "loss": 0.17589187622070312,
+      "mean_token_accuracy": 0.9615845668315888,
+      "num_tokens": 33638871.0,
+      "step": 7850
+    },
+    {
+      "entropy": 0.18853790896944703,
+      "epoch": 2.3831937844157802,
+      "grad_norm": 0.5678646564483643,
+      "learning_rate": 1.0384865526317861e-05,
+      "loss": 0.1777634620666504,
+      "mean_token_accuracy": 0.961886543482542,
+      "num_tokens": 33852577.0,
+      "step": 7900
+    },
+    {
+      "entropy": 0.19151988808065654,
+      "epoch": 2.3982801538809686,
+      "grad_norm": 0.4446047246456146,
+      "learning_rate": 9.900757772368175e-06,
+      "loss": 0.17955810546875,
+      "mean_token_accuracy": 0.9614880438148975,
+      "num_tokens": 34065599.0,
+      "step": 7950
+    },
+    {
+      "entropy": 0.19300217579118908,
+      "epoch": 2.4133665233461565,
+      "grad_norm": 0.5685653686523438,
+      "learning_rate": 9.426962370484533e-06,
+      "loss": 0.18160755157470704,
+      "mean_token_accuracy": 0.9607279069721699,
+      "num_tokens": 34280453.0,
+      "step": 8000
+    },
+    {
+      "epoch": 2.4133665233461565,
+      "eval_entropy": 0.24335849892719205,
+      "eval_loss": 0.5554639101028442,
+      "eval_mean_token_accuracy": 0.905630389702716,
+      "eval_num_tokens": 34280453.0,
+      "eval_runtime": 37.2491,
+      "eval_samples_per_second": 58.149,
+      "eval_steps_per_second": 7.275,
+      "step": 8000
+    },
+    {
+      "entropy": 0.18967346624471249,
+      "epoch": 2.428452892811345,
+      "grad_norm": 0.5121133923530579,
+      "learning_rate": 8.963601166984225e-06,
+      "loss": 0.1821819305419922,
+      "mean_token_accuracy": 0.9610455152392388,
+      "num_tokens": 34494654.0,
+      "step": 8050
+    },
+    {
+      "entropy": 0.18874711729586124,
+      "epoch": 2.4435392622765333,
+      "grad_norm": 0.6657042503356934,
+      "learning_rate": 8.510793324813937e-06,
+      "loss": 0.17630435943603515,
+      "mean_token_accuracy": 0.9615484464168549,
+      "num_tokens": 34704277.0,
+      "step": 8100
+    },
+    {
+      "entropy": 0.19604809990152716,
+      "epoch": 2.458625631741721,
+      "grad_norm": 0.6367285847663879,
+      "learning_rate": 8.068655292904559e-06,
+      "loss": 0.1849921989440918,
+      "mean_token_accuracy": 0.9597983421385288,
+      "num_tokens": 34918987.0,
+      "step": 8150
+    },
+    {
+      "entropy": 0.19020996399223805,
+      "epoch": 2.4737120012069096,
+      "grad_norm": 0.40266647934913635,
+      "learning_rate": 7.63730077622392e-06,
+      "loss": 0.18030958175659179,
+      "mean_token_accuracy": 0.9618312911689282,
+      "num_tokens": 35128250.0,
+      "step": 8200
+    },
+    {
+      "epoch": 2.4737120012069096,
+      "eval_entropy": 0.24456536362971767,
+      "eval_loss": 0.5537819266319275,
+      "eval_mean_token_accuracy": 0.905363669254683,
+      "eval_num_tokens": 35128250.0,
+      "eval_runtime": 37.3206,
+      "eval_samples_per_second": 58.038,
+      "eval_steps_per_second": 7.261,
+      "step": 8200
+    },
+    {
+      "entropy": 0.1947929921839386,
+      "epoch": 2.4887983706720975,
+      "grad_norm": 0.5730997323989868,
+      "learning_rate": 7.216840706535133e-06,
+      "loss": 0.1838933753967285,
+      "mean_token_accuracy": 0.9602823905646801,
+      "num_tokens": 35344354.0,
+      "step": 8250
+    },
+    {
+      "entropy": 0.1937092263624072,
+      "epoch": 2.503884740137286,
+      "grad_norm": 0.48286330699920654,
+      "learning_rate": 6.807383213868318e-06,
+      "loss": 0.1827998924255371,
+      "mean_token_accuracy": 0.9602498579025268,
+      "num_tokens": 35559731.0,
+      "step": 8300
+    },
+    {
+      "entropy": 0.18829923996701836,
+      "epoch": 2.5189711096024743,
+      "grad_norm": 0.5280757546424866,
+      "learning_rate": 6.4090335987126e-06,
+      "loss": 0.17718358993530273,
+      "mean_token_accuracy": 0.9615453231334686,
+      "num_tokens": 35772707.0,
+      "step": 8350
+    },
+    {
+      "entropy": 0.19396818530745805,
+      "epoch": 2.534057479067662,
+      "grad_norm": 0.4934554100036621,
+      "learning_rate": 6.021894304935943e-06,
+      "loss": 0.18351167678833008,
+      "mean_token_accuracy": 0.9603780242800712,
+      "num_tokens": 35987178.0,
+      "step": 8400
+    },
+    {
+      "epoch": 2.534057479067662,
+      "eval_entropy": 0.2449101031761328,
+      "eval_loss": 0.5524511337280273,
+      "eval_mean_token_accuracy": 0.9050956489415186,
+      "eval_num_tokens": 35987178.0,
+      "eval_runtime": 37.2719,
+      "eval_samples_per_second": 58.113,
+      "eval_steps_per_second": 7.271,
+      "step": 8400
+    },
+    {
+      "entropy": 0.18812250457704066,
+      "epoch": 2.5491438485328506,
+      "grad_norm": 0.5735852718353271,
+      "learning_rate": 5.646064893439623e-06,
+      "loss": 0.17783910751342774,
+      "mean_token_accuracy": 0.9615232966840267,
+      "num_tokens": 36200263.0,
+      "step": 8450
+    },
+    {
+      "entropy": 0.18926897779107094,
+      "epoch": 2.5642302179980385,
+      "grad_norm": 0.5568459033966064,
+      "learning_rate": 5.281642016554017e-06,
+      "loss": 0.1792352867126465,
+      "mean_token_accuracy": 0.9616803368926048,
+      "num_tokens": 36411968.0,
+      "step": 8500
+    },
+    {
+      "entropy": 0.19124026644974948,
+      "epoch": 2.579316587463227,
+      "grad_norm": 0.5221688151359558,
+      "learning_rate": 4.928719393182507e-06,
+      "loss": 0.17745084762573243,
+      "mean_token_accuracy": 0.9616039192676544,
+      "num_tokens": 36626784.0,
+      "step": 8550
+    },
+    {
+      "entropy": 0.19215355191379785,
+      "epoch": 2.5944029569284153,
+      "grad_norm": 0.36854663491249084,
+      "learning_rate": 4.5873877846996594e-06,
+      "loss": 0.18093048095703124,
+      "mean_token_accuracy": 0.9610362699627877,
+      "num_tokens": 36839614.0,
+      "step": 8600
+    },
+    {
+      "epoch": 2.5944029569284153,
+      "eval_entropy": 0.24483193833779585,
+      "eval_loss": 0.5536935925483704,
+      "eval_mean_token_accuracy": 0.9050766867465199,
+      "eval_num_tokens": 36839614.0,
+      "eval_runtime": 37.2832,
+      "eval_samples_per_second": 58.096,
+      "eval_steps_per_second": 7.269,
+      "step": 8600
+    },
+    {
+      "entropy": 0.19345266066491604,
+      "epoch": 2.6094893263936036,
+      "grad_norm": 0.36760082840919495,
+      "learning_rate": 4.257734971610106e-06,
+      "loss": 0.18040838241577148,
+      "mean_token_accuracy": 0.9611816507577896,
+      "num_tokens": 37053142.0,
+      "step": 8650
+    },
+    {
+      "entropy": 0.188013746580109,
+      "epoch": 2.6245756958587916,
+      "grad_norm": 0.5213715434074402,
+      "learning_rate": 3.93984573097394e-06,
+      "loss": 0.17426759719848633,
+      "mean_token_accuracy": 0.962059508562088,
+      "num_tokens": 37267777.0,
+      "step": 8700
+    },
+    {
+      "entropy": 0.1918721451703459,
+      "epoch": 2.6396620653239795,
+      "grad_norm": 0.5748514533042908,
+      "learning_rate": 3.633801814604554e-06,
+      "loss": 0.1796425437927246,
+      "mean_token_accuracy": 0.9613311864435673,
+      "num_tokens": 37485508.0,
+      "step": 8750
+    },
+    {
+      "entropy": 0.18769024461507797,
+      "epoch": 2.654748434789168,
+      "grad_norm": 0.37133195996284485,
+      "learning_rate": 3.339681928044436e-06,
+      "loss": 0.17722646713256837,
+      "mean_token_accuracy": 0.9621489056944847,
+      "num_tokens": 37694053.0,
+      "step": 8800
+    },
+    {
+      "epoch": 2.654748434789168,
+      "eval_entropy": 0.24254771412738574,
+      "eval_loss": 0.558121383190155,
+      "eval_mean_token_accuracy": 0.905118876497684,
+      "eval_num_tokens": 37694053.0,
+      "eval_runtime": 37.2089,
+      "eval_samples_per_second": 58.212,
+      "eval_steps_per_second": 7.283,
+      "step": 8800
+    },
+    {
+      "entropy": 0.18984992211684584,
+      "epoch": 2.6698348042543563,
+      "grad_norm": 0.43609800934791565,
+      "learning_rate": 3.057561710324447e-06,
+      "loss": 0.17845310211181642,
+      "mean_token_accuracy": 0.9613586132228374,
+      "num_tokens": 37906133.0,
+      "step": 8850
+    },
+    {
+      "entropy": 0.1950311391800642,
+      "epoch": 2.6849211737195446,
+      "grad_norm": 0.45412150025367737,
+      "learning_rate": 2.787513714511636e-06,
+      "loss": 0.18089279174804687,
+      "mean_token_accuracy": 0.9608308251202107,
+      "num_tokens": 38121697.0,
+      "step": 8900
+    },
+    {
+      "entropy": 0.18615621831268073,
+      "epoch": 2.7000075431847326,
+      "grad_norm": 0.6591213345527649,
+      "learning_rate": 2.52960738905072e-06,
+      "loss": 0.17349456787109374,
+      "mean_token_accuracy": 0.9623284694552422,
+      "num_tokens": 38339277.0,
+      "step": 8950
+    },
+    {
+      "entropy": 0.18828014567494392,
+      "epoch": 2.715093912649921,
+      "grad_norm": 0.47049465775489807,
+      "learning_rate": 2.2839090599040046e-06,
+      "loss": 0.17837800979614257,
+      "mean_token_accuracy": 0.9619372922182083,
+      "num_tokens": 38551183.0,
+      "step": 9000
+    },
+    {
+      "epoch": 2.715093912649921,
+      "eval_entropy": 0.24229726709452942,
+      "eval_loss": 0.5592358708381653,
+      "eval_mean_token_accuracy": 0.905019719882205,
+      "eval_num_tokens": 38551183.0,
+      "eval_runtime": 37.1765,
+      "eval_samples_per_second": 58.263,
+      "eval_steps_per_second": 7.29,
+      "step": 9000
+    },
+    {
+      "entropy": 0.18336616074666381,
+      "epoch": 2.730180282115109,
+      "grad_norm": 0.3691662549972534,
+      "learning_rate": 2.0504819134942734e-06,
+      "loss": 0.17197175979614257,
+      "mean_token_accuracy": 0.9629626546800136,
+      "num_tokens": 38763413.0,
+      "step": 9050
+    },
+    {
+      "entropy": 0.18445612547919155,
+      "epoch": 2.7452666515802973,
+      "grad_norm": 0.5340840816497803,
+      "learning_rate": 1.8293859804550506e-06,
+      "loss": 0.17212907791137697,
+      "mean_token_accuracy": 0.9631160937249661,
+      "num_tokens": 38979256.0,
+      "step": 9100
+    },
+    {
+      "entropy": 0.18713438665494322,
+      "epoch": 2.7603530210454856,
+      "grad_norm": 0.5069661736488342,
+      "learning_rate": 1.620678120192537e-06,
+      "loss": 0.1732018280029297,
+      "mean_token_accuracy": 0.9624787227809429,
+      "num_tokens": 39194699.0,
+      "step": 9150
+    },
+    {
+      "entropy": 0.19093556420877575,
+      "epoch": 2.7754393905106736,
+      "grad_norm": 0.3724497854709625,
+      "learning_rate": 1.424412006263004e-06,
+      "loss": 0.17719757080078125,
+      "mean_token_accuracy": 0.9616392150521278,
+      "num_tokens": 39405056.0,
+      "step": 9200
+    },
+    {
+      "epoch": 2.7754393905106736,
+      "eval_entropy": 0.2416494778216545,
+      "eval_loss": 0.5602814555168152,
+      "eval_mean_token_accuracy": 0.9050707438775094,
+      "eval_num_tokens": 39405056.0,
+      "eval_runtime": 37.0527,
+      "eval_samples_per_second": 58.457,
+      "eval_steps_per_second": 7.314,
+      "step": 9200
+    },
+    {
+      "entropy": 0.18624798154458402,
+      "epoch": 2.790525759975862,
+      "grad_norm": 0.5416818857192993,
+      "learning_rate": 1.24063811256952e-06,
+      "loss": 0.17536975860595702,
+      "mean_token_accuracy": 0.9625106258690357,
+      "num_tokens": 39620602.0,
+      "step": 9250
+    },
+    {
+      "entropy": 0.18864918448030948,
+      "epoch": 2.80561212944105,
+      "grad_norm": 0.7557387351989746,
+      "learning_rate": 1.0694037003815493e-06,
+      "loss": 0.1763162422180176,
+      "mean_token_accuracy": 0.9617964868247509,
+      "num_tokens": 39838171.0,
+      "step": 9300
+    },
+    {
+      "entropy": 0.19156487157568336,
+      "epoch": 2.8206984989062382,
+      "grad_norm": 0.5030518174171448,
+      "learning_rate": 9.107528061807425e-07,
+      "loss": 0.17927934646606444,
+      "mean_token_accuracy": 0.9610569317638874,
+      "num_tokens": 40055765.0,
+      "step": 9350
+    },
+    {
+      "entropy": 0.19062289187684656,
+      "epoch": 2.8357848683714266,
+      "grad_norm": 0.3872191905975342,
+      "learning_rate": 7.64726230336027e-07,
+      "loss": 0.1788728141784668,
+      "mean_token_accuracy": 0.9610781705379486,
+      "num_tokens": 40273169.0,
+      "step": 9400
+    },
+    {
+      "epoch": 2.8357848683714266,
+      "eval_entropy": 0.2424020753190526,
+      "eval_loss": 0.5598563551902771,
+      "eval_mean_token_accuracy": 0.905123386435843,
+      "eval_num_tokens": 40273169.0,
+      "eval_runtime": 37.2998,
+      "eval_samples_per_second": 58.07,
+      "eval_steps_per_second": 7.265,
+      "step": 9400
+    },
+    {
+      "entropy": 0.19168663132935762,
+      "epoch": 2.8508712378366146,
+      "grad_norm": 0.38919326663017273,
+      "learning_rate": 6.313615266109463e-07,
+      "loss": 0.17518898010253905,
+      "mean_token_accuracy": 0.9616159409284591,
+      "num_tokens": 40493623.0,
+      "step": 9450
+    },
+    {
+      "entropy": 0.1842949813697487,
+      "epoch": 2.865957607301803,
+      "grad_norm": 0.4381106197834015,
+      "learning_rate": 5.106929925059456e-07,
+      "loss": 0.17125751495361327,
+      "mean_token_accuracy": 0.9631977449357509,
+      "num_tokens": 40706058.0,
+      "step": 9500
+    },
+    {
+      "entropy": 0.18345711720176042,
+      "epoch": 2.881043976766991,
+      "grad_norm": 0.625280499458313,
+      "learning_rate": 4.027516604380388e-07,
+      "loss": 0.17233373641967772,
+      "mean_token_accuracy": 0.9627382136881352,
+      "num_tokens": 40919257.0,
+      "step": 9550
+    },
+    {
+      "entropy": 0.19293911349028348,
+      "epoch": 2.8961303462321792,
+      "grad_norm": 0.5119889974594116,
+      "learning_rate": 3.075652897602033e-07,
+      "loss": 0.18010311126708983,
+      "mean_token_accuracy": 0.960850502550602,
+      "num_tokens": 41135038.0,
+      "step": 9600
+    },
+    {
+      "epoch": 2.8961303462321792,
+      "eval_entropy": 0.242256640535659,
+      "eval_loss": 0.5602179169654846,
+      "eval_mean_token_accuracy": 0.9051620641317755,
+      "eval_num_tokens": 41135038.0,
+      "eval_runtime": 37.3372,
+      "eval_samples_per_second": 58.012,
+      "eval_steps_per_second": 7.258,
+      "step": 9600
+    },
+    {
+      "entropy": 0.1843437609449029,
+      "epoch": 2.9112167156973676,
+      "grad_norm": 0.5234854221343994,
+      "learning_rate": 2.2515835962245157e-07,
+      "loss": 0.16681966781616211,
+      "mean_token_accuracy": 0.9633826872706414,
+      "num_tokens": 41350531.0,
+      "step": 9650
+    },
+    {
+      "entropy": 0.18940109137445688,
+      "epoch": 2.9263030851625556,
+      "grad_norm": 0.5936132073402405,
+      "learning_rate": 1.5555206267655055e-07,
+      "loss": 0.17372846603393555,
+      "mean_token_accuracy": 0.9623673681914806,
+      "num_tokens": 41568124.0,
+      "step": 9700
+    },
+    {
+      "entropy": 0.18825361583381892,
+      "epoch": 2.941389454627744,
+      "grad_norm": 0.43561285734176636,
+      "learning_rate": 9.876429962585309e-08,
+      "loss": 0.17482643127441405,
+      "mean_token_accuracy": 0.962187256962061,
+      "num_tokens": 41784922.0,
+      "step": 9750
+    },
+    {
+      "entropy": 0.19123295756056904,
+      "epoch": 2.956475824092932,
+      "grad_norm": 0.45098307728767395,
+      "learning_rate": 5.4809674621797425e-08,
+      "loss": 0.1720089340209961,
+      "mean_token_accuracy": 0.9617229135334492,
+      "num_tokens": 42005066.0,
+      "step": 9800
+    },
+    {
+      "epoch": 2.956475824092932,
+      "eval_entropy": 0.2421166613917949,
+      "eval_loss": 0.5603178143501282,
+      "eval_mean_token_accuracy": 0.9051976802164338,
+      "eval_num_tokens": 42005066.0,
+      "eval_runtime": 37.2727,
+      "eval_samples_per_second": 58.112,
+      "eval_steps_per_second": 7.271,
+      "step": 9800
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 9945,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.278959418863923e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-9800/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe063de6f1674be7d5535690dc263e5d0c28436ed4781417772a469f39ef21d4
+size 5713

checkpoint-9945/README.md

@@ -0,0 +1,209 @@
+---
+base_model: /scratch/Qwen3-1.7B
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/scratch/Qwen3-1.7B
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

checkpoint-9945/adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "/scratch/Qwen3-1.7B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "down_proj",
+    "q_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-9945/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a780cfe02ab243ed703e231511f46a315e792a7d503207b63363f6fea1f7216
+size 278973888

checkpoint-9945/chat_template.jinja

@@ -0,0 +1,93 @@
+{%- set enable_thinking = true %}
+{%- set enable_thinking = false %}
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant. Always put your final answer in \\boxed{}. For multiple choice questions, put only the letter in \\boxed{}, for example \\boxed{A}.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

checkpoint-9945/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:284e636a9b859c30f508e7ae8764845f5fee8cc5136936eb93236edd97a9d61e
+size 558179131

checkpoint-9945/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f50775fd0bf66026bda2f96288318c19cf48a0c68b900b1943d065e3665b7b00
+size 14645

checkpoint-9945/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:943138d2512aaf140433dfa41fd0778cb2a62c9e65aebebbdf4d75b347f8ca44
+size 1465

checkpoint-9945/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
+size 11422650

checkpoint-9945/tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

checkpoint-9945/trainer_state.json

@@ -0,0 +1,2564 @@
+{
+  "best_global_step": 3000,
+  "best_metric": 0.43371766805648804,
+  "best_model_checkpoint": "/scratch/gk_checkpoint_lora_v2/checkpoint-3000",
+  "epoch": 3.0,
+  "eval_steps": 200,
+  "global_step": 9945,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.0372698324173688,
+      "epoch": 0.015086369465188203,
+      "grad_norm": 0.4944687783718109,
+      "learning_rate": 3.266666666666667e-05,
+      "loss": 2.1209957885742186,
+      "mean_token_accuracy": 0.6639667785167694,
+      "num_tokens": 213967.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.4767542722821236,
+      "epoch": 0.030172738930376405,
+      "grad_norm": 0.29953643679618835,
+      "learning_rate": 6.6e-05,
+      "loss": 0.5209395980834961,
+      "mean_token_accuracy": 0.9028549310564995,
+      "num_tokens": 426365.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.46151453502476214,
+      "epoch": 0.04525910839556461,
+      "grad_norm": 0.3118787109851837,
+      "learning_rate": 9.933333333333334e-05,
+      "loss": 0.508198356628418,
+      "mean_token_accuracy": 0.9032981966435909,
+      "num_tokens": 642837.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.4455497920885682,
+      "epoch": 0.06034547786075281,
+      "grad_norm": 0.3013271391391754,
+      "learning_rate": 9.999382532513122e-05,
+      "loss": 0.48826019287109373,
+      "mean_token_accuracy": 0.907553653717041,
+      "num_tokens": 856475.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.06034547786075281,
+      "eval_entropy": 0.4689600637588114,
+      "eval_loss": 0.4816349446773529,
+      "eval_mean_token_accuracy": 0.8987587762934696,
+      "eval_num_tokens": 856475.0,
+      "eval_runtime": 37.5206,
+      "eval_samples_per_second": 57.728,
+      "eval_steps_per_second": 7.223,
+      "step": 200
+    },
+    {
+      "entropy": 0.44372758489102127,
+      "epoch": 0.075431847325941,
+      "grad_norm": 0.31373220682144165,
+      "learning_rate": 9.997479627263544e-05,
+      "loss": 0.4843710327148438,
+      "mean_token_accuracy": 0.9082412907481193,
+      "num_tokens": 1069370.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.4475720078870654,
+      "epoch": 0.09051821679112922,
+      "grad_norm": 0.27380964159965515,
+      "learning_rate": 9.994291516446573e-05,
+      "loss": 0.491109733581543,
+      "mean_token_accuracy": 0.9055162121355533,
+      "num_tokens": 1286132.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.4463552813604474,
+      "epoch": 0.10560458625631741,
+      "grad_norm": 0.2614763677120209,
+      "learning_rate": 9.989819019951048e-05,
+      "loss": 0.4837772369384766,
+      "mean_token_accuracy": 0.9081570096313953,
+      "num_tokens": 1500851.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.41013720393180847,
+      "epoch": 0.12069095572150562,
+      "grad_norm": 0.4836612045764923,
+      "learning_rate": 9.984063287972232e-05,
+      "loss": 0.44385364532470706,
+      "mean_token_accuracy": 0.9148843766748905,
+      "num_tokens": 1708807.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.12069095572150562,
+      "eval_entropy": 0.4811463643044123,
+      "eval_loss": 0.4747391641139984,
+      "eval_mean_token_accuracy": 0.8997697566268189,
+      "eval_num_tokens": 1708807.0,
+      "eval_runtime": 37.1123,
+      "eval_samples_per_second": 58.363,
+      "eval_steps_per_second": 7.302,
+      "step": 400
+    },
+    {
+      "entropy": 0.42200190499424933,
+      "epoch": 0.13577732518669383,
+      "grad_norm": 0.19339531660079956,
+      "learning_rate": 9.977025800716017e-05,
+      "loss": 0.45712459564208985,
+      "mean_token_accuracy": 0.9106362241506577,
+      "num_tokens": 1923184.0,
+      "step": 450
+    },
+    {
+      "entropy": 0.43271509755402804,
+      "epoch": 0.150863694651882,
+      "grad_norm": 0.299110472202301,
+      "learning_rate": 9.968708368018253e-05,
+      "loss": 0.4724855422973633,
+      "mean_token_accuracy": 0.9078708891570568,
+      "num_tokens": 2139609.0,
+      "step": 500
+    },
+    {
+      "entropy": 0.43638833791017534,
+      "epoch": 0.16595006411707022,
+      "grad_norm": 0.2539425492286682,
+      "learning_rate": 9.959113128879322e-05,
+      "loss": 0.4785395050048828,
+      "mean_token_accuracy": 0.907249256670475,
+      "num_tokens": 2358080.0,
+      "step": 550
+    },
+    {
+      "entropy": 0.43115664307028057,
+      "epoch": 0.18103643358225843,
+      "grad_norm": 0.2542003393173218,
+      "learning_rate": 9.948242550914035e-05,
+      "loss": 0.4740608215332031,
+      "mean_token_accuracy": 0.9089943794906139,
+      "num_tokens": 2574667.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.18103643358225843,
+      "eval_entropy": 0.45346878287537074,
+      "eval_loss": 0.46328845620155334,
+      "eval_mean_token_accuracy": 0.9025413253210568,
+      "eval_num_tokens": 2574667.0,
+      "eval_runtime": 37.1254,
+      "eval_samples_per_second": 58.343,
+      "eval_steps_per_second": 7.3,
+      "step": 600
+    },
+    {
+      "entropy": 0.4420048241317272,
+      "epoch": 0.19612280304744664,
+      "grad_norm": 0.26832085847854614,
+      "learning_rate": 9.936099429717045e-05,
+      "loss": 0.486652717590332,
+      "mean_token_accuracy": 0.9074076810479164,
+      "num_tokens": 2790489.0,
+      "step": 650
+    },
+    {
+      "entropy": 0.4269510039314628,
+      "epoch": 0.21120917251263482,
+      "grad_norm": 0.20635050535202026,
+      "learning_rate": 9.922686888143897e-05,
+      "loss": 0.4619187927246094,
+      "mean_token_accuracy": 0.910810690075159,
+      "num_tokens": 3003881.0,
+      "step": 700
+    },
+    {
+      "entropy": 0.42016164746135476,
+      "epoch": 0.22629554197782303,
+      "grad_norm": 0.2643264830112457,
+      "learning_rate": 9.908008375507924e-05,
+      "loss": 0.46344844818115233,
+      "mean_token_accuracy": 0.9113752076029777,
+      "num_tokens": 3218446.0,
+      "step": 750
+    },
+    {
+      "entropy": 0.3974369211867452,
+      "epoch": 0.24138191144301124,
+      "grad_norm": 0.19392798840999603,
+      "learning_rate": 9.89206766669318e-05,
+      "loss": 0.42646697998046873,
+      "mean_token_accuracy": 0.9165593402087688,
+      "num_tokens": 3428256.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.24138191144301124,
+      "eval_entropy": 0.43642714537157784,
+      "eval_loss": 0.46031010150909424,
+      "eval_mean_token_accuracy": 0.9025695645061366,
+      "eval_num_tokens": 3428256.0,
+      "eval_runtime": 37.1078,
+      "eval_samples_per_second": 58.37,
+      "eval_steps_per_second": 7.303,
+      "step": 800
+    },
+    {
+      "entropy": 0.4217161551490426,
+      "epoch": 0.25646828090819945,
+      "grad_norm": 0.21452021598815918,
+      "learning_rate": 9.874868861183658e-05,
+      "loss": 0.4612973022460938,
+      "mean_token_accuracy": 0.9114794608950615,
+      "num_tokens": 3642529.0,
+      "step": 850
+    },
+    {
+      "entropy": 0.41439461953938006,
+      "epoch": 0.27155465037338766,
+      "grad_norm": 0.25079163908958435,
+      "learning_rate": 9.856416382009006e-05,
+      "loss": 0.4494070053100586,
+      "mean_token_accuracy": 0.9127840812504292,
+      "num_tokens": 3855962.0,
+      "step": 900
+    },
+    {
+      "entropy": 0.4270974922552705,
+      "epoch": 0.2866410198385759,
+      "grad_norm": 0.20817860960960388,
+      "learning_rate": 9.836714974607077e-05,
+      "loss": 0.46105358123779294,
+      "mean_token_accuracy": 0.9099104046821594,
+      "num_tokens": 4069157.0,
+      "step": 950
+    },
+    {
+      "entropy": 0.400859787017107,
+      "epoch": 0.301727389303764,
+      "grad_norm": 0.20943191647529602,
+      "learning_rate": 9.815769705603521e-05,
+      "loss": 0.4289055633544922,
+      "mean_token_accuracy": 0.9167061321437359,
+      "num_tokens": 4278743.0,
+      "step": 1000
+    },
+    {
+      "epoch": 0.301727389303764,
+      "eval_entropy": 0.44312036031946483,
+      "eval_loss": 0.4571220278739929,
+      "eval_mean_token_accuracy": 0.9029428505809545,
+      "eval_num_tokens": 4278743.0,
+      "eval_runtime": 37.0803,
+      "eval_samples_per_second": 58.414,
+      "eval_steps_per_second": 7.308,
+      "step": 1000
+    },
+    {
+      "entropy": 0.4052092955261469,
+      "epoch": 0.31681375876895224,
+      "grad_norm": 0.21670734882354736,
+      "learning_rate": 9.793585961508811e-05,
+      "loss": 0.44187084197998044,
+      "mean_token_accuracy": 0.9138142390549183,
+      "num_tokens": 4495120.0,
+      "step": 1050
+    },
+    {
+      "entropy": 0.3885428298264742,
+      "epoch": 0.33190012823414045,
+      "grad_norm": 0.2656344771385193,
+      "learning_rate": 9.770169447332977e-05,
+      "loss": 0.42026878356933595,
+      "mean_token_accuracy": 0.9171342994272709,
+      "num_tokens": 4707664.0,
+      "step": 1100
+    },
+    {
+      "entropy": 0.41116086438298227,
+      "epoch": 0.34698649769932866,
+      "grad_norm": 0.25747165083885193,
+      "learning_rate": 9.745526185118458e-05,
+      "loss": 0.44418087005615237,
+      "mean_token_accuracy": 0.9133055797219276,
+      "num_tokens": 4921515.0,
+      "step": 1150
+    },
+    {
+      "entropy": 0.40833847373723986,
+      "epoch": 0.36207286716451687,
+      "grad_norm": 0.22326916456222534,
+      "learning_rate": 9.719662512391396e-05,
+      "loss": 0.4394912338256836,
+      "mean_token_accuracy": 0.9140990000963211,
+      "num_tokens": 5133998.0,
+      "step": 1200
+    },
+    {
+      "epoch": 0.36207286716451687,
+      "eval_entropy": 0.44459747828240764,
+      "eval_loss": 0.4540960192680359,
+      "eval_mean_token_accuracy": 0.9033675605080664,
+      "eval_num_tokens": 5133998.0,
+      "eval_runtime": 37.1014,
+      "eval_samples_per_second": 58.381,
+      "eval_steps_per_second": 7.304,
+      "step": 1200
+    },
+    {
+      "entropy": 0.39192866910248997,
+      "epoch": 0.3771592366297051,
+      "grad_norm": 0.20642907917499542,
+      "learning_rate": 9.692585080531822e-05,
+      "loss": 0.42892616271972656,
+      "mean_token_accuracy": 0.9165673214197159,
+      "num_tokens": 5348047.0,
+      "step": 1250
+    },
+    {
+      "entropy": 0.40107650008052587,
+      "epoch": 0.3922456060948933,
+      "grad_norm": 0.2515887916088104,
+      "learning_rate": 9.664300853063104e-05,
+      "loss": 0.4329941558837891,
+      "mean_token_accuracy": 0.9152751086652279,
+      "num_tokens": 5562125.0,
+      "step": 1300
+    },
+    {
+      "entropy": 0.3964537301659584,
+      "epoch": 0.4073319755600815,
+      "grad_norm": 0.20992055535316467,
+      "learning_rate": 9.63481710386114e-05,
+      "loss": 0.4275414276123047,
+      "mean_token_accuracy": 0.9168652257323265,
+      "num_tokens": 5773685.0,
+      "step": 1350
+    },
+    {
+      "entropy": 0.39044841077178716,
+      "epoch": 0.42241834502526965,
+      "grad_norm": 0.30388110876083374,
+      "learning_rate": 9.604141415283728e-05,
+      "loss": 0.42324817657470704,
+      "mean_token_accuracy": 0.9169075645506382,
+      "num_tokens": 5986601.0,
+      "step": 1400
+    },
+    {
+      "epoch": 0.42241834502526965,
+      "eval_entropy": 0.4361799991658693,
+      "eval_loss": 0.45172417163848877,
+      "eval_mean_token_accuracy": 0.9033232137725802,
+      "eval_num_tokens": 5986601.0,
+      "eval_runtime": 37.1897,
+      "eval_samples_per_second": 58.242,
+      "eval_steps_per_second": 7.287,
+      "step": 1400
+    },
+    {
+      "entropy": 0.4137679870799184,
+      "epoch": 0.43750471449045786,
+      "grad_norm": 0.23755542933940887,
+      "learning_rate": 9.572281676220608e-05,
+      "loss": 0.4478377532958984,
+      "mean_token_accuracy": 0.911891212016344,
+      "num_tokens": 6203048.0,
+      "step": 1450
+    },
+    {
+      "entropy": 0.41279098089784383,
+      "epoch": 0.45259108395564607,
+      "grad_norm": 0.21780936419963837,
+      "learning_rate": 9.539246080064659e-05,
+      "loss": 0.45262195587158205,
+      "mean_token_accuracy": 0.9123758906126023,
+      "num_tokens": 6419624.0,
+      "step": 1500
+    },
+    {
+      "entropy": 0.41412957072257994,
+      "epoch": 0.4676774534208343,
+      "grad_norm": 0.2430579662322998,
+      "learning_rate": 9.505043122604818e-05,
+      "loss": 0.45246307373046873,
+      "mean_token_accuracy": 0.9122441673278808,
+      "num_tokens": 6633965.0,
+      "step": 1550
+    },
+    {
+      "entropy": 0.3798889485746622,
+      "epoch": 0.4827638228860225,
+      "grad_norm": 0.28653672337532043,
+      "learning_rate": 9.469681599841192e-05,
+      "loss": 0.41427810668945314,
+      "mean_token_accuracy": 0.9184439463913441,
+      "num_tokens": 6847358.0,
+      "step": 1600
+    },
+    {
+      "epoch": 0.4827638228860225,
+      "eval_entropy": 0.43242653594025826,
+      "eval_loss": 0.4488651752471924,
+      "eval_mean_token_accuracy": 0.9040581949082688,
+      "eval_num_tokens": 6847358.0,
+      "eval_runtime": 37.1595,
+      "eval_samples_per_second": 58.289,
+      "eval_steps_per_second": 7.293,
+      "step": 1600
+    },
+    {
+      "entropy": 0.38126184083521364,
+      "epoch": 0.4978501923512107,
+      "grad_norm": 0.2068091183900833,
+      "learning_rate": 9.433170605722996e-05,
+      "loss": 0.40500320434570314,
+      "mean_token_accuracy": 0.9181749866902829,
+      "num_tokens": 7062005.0,
+      "step": 1650
+    },
+    {
+      "entropy": 0.38879580337554215,
+      "epoch": 0.5129365618163989,
+      "grad_norm": 0.19908899068832397,
+      "learning_rate": 9.395519529809848e-05,
+      "loss": 0.41957916259765626,
+      "mean_token_accuracy": 0.9183010324835778,
+      "num_tokens": 7272082.0,
+      "step": 1700
+    },
+    {
+      "entropy": 0.3867104376107454,
+      "epoch": 0.5280229312815871,
+      "grad_norm": 0.21681655943393707,
+      "learning_rate": 9.356738054857057e-05,
+      "loss": 0.41496986389160156,
+      "mean_token_accuracy": 0.9176018598675728,
+      "num_tokens": 7484751.0,
+      "step": 1750
+    },
+    {
+      "entropy": 0.3900348538905382,
+      "epoch": 0.5431093007467753,
+      "grad_norm": 0.23264895379543304,
+      "learning_rate": 9.316836154325494e-05,
+      "loss": 0.4201799774169922,
+      "mean_token_accuracy": 0.9161376728117466,
+      "num_tokens": 7699385.0,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5431093007467753,
+      "eval_entropy": 0.4371595958941977,
+      "eval_loss": 0.44555598497390747,
+      "eval_mean_token_accuracy": 0.9047388078101887,
+      "eval_num_tokens": 7699385.0,
+      "eval_runtime": 37.2391,
+      "eval_samples_per_second": 58.165,
+      "eval_steps_per_second": 7.277,
+      "step": 1800
+    },
+    {
+      "entropy": 0.38737339399755,
+      "epoch": 0.5581956702119635,
+      "grad_norm": 0.21236486732959747,
+      "learning_rate": 9.275824089816716e-05,
+      "loss": 0.4186508941650391,
+      "mean_token_accuracy": 0.9184837466478348,
+      "num_tokens": 7912846.0,
+      "step": 1850
+    },
+    {
+      "entropy": 0.38791515786200764,
+      "epoch": 0.5732820396771517,
+      "grad_norm": 0.22874821722507477,
+      "learning_rate": 9.233712408433972e-05,
+      "loss": 0.42144878387451173,
+      "mean_token_accuracy": 0.9170675221085548,
+      "num_tokens": 8126645.0,
+      "step": 1900
+    },
+    {
+      "entropy": 0.3831383780390024,
+      "epoch": 0.5883684091423399,
+      "grad_norm": 0.3072109818458557,
+      "learning_rate": 9.190511940069813e-05,
+      "loss": 0.407428092956543,
+      "mean_token_accuracy": 0.9184182004630566,
+      "num_tokens": 8341447.0,
+      "step": 1950
+    },
+    {
+      "entropy": 0.38130090072751044,
+      "epoch": 0.603454778607528,
+      "grad_norm": 0.2783527374267578,
+      "learning_rate": 9.146233794620944e-05,
+      "loss": 0.41518512725830076,
+      "mean_token_accuracy": 0.9192077203094959,
+      "num_tokens": 8553915.0,
+      "step": 2000
+    },
+    {
+      "epoch": 0.603454778607528,
+      "eval_entropy": 0.41359548831557874,
+      "eval_loss": 0.4426310062408447,
+      "eval_mean_token_accuracy": 0.9062746316744392,
+      "eval_num_tokens": 8553915.0,
+      "eval_runtime": 37.0772,
+      "eval_samples_per_second": 58.419,
+      "eval_steps_per_second": 7.309,
+      "step": 2000
+    },
+    {
+      "entropy": 0.40945424281060694,
+      "epoch": 0.6185411480727163,
+      "grad_norm": 0.2751815915107727,
+      "learning_rate": 9.100889359131093e-05,
+      "loss": 0.44279281616210936,
+      "mean_token_accuracy": 0.9126340833306312,
+      "num_tokens": 8773030.0,
+      "step": 2050
+    },
+    {
+      "entropy": 0.37910934548825026,
+      "epoch": 0.6336275175379045,
+      "grad_norm": 0.24518635869026184,
+      "learning_rate": 9.054490294862594e-05,
+      "loss": 0.41019065856933595,
+      "mean_token_accuracy": 0.9180504800379277,
+      "num_tokens": 8987621.0,
+      "step": 2100
+    },
+    {
+      "entropy": 0.38141273133456705,
+      "epoch": 0.6487138870030927,
+      "grad_norm": 0.2637041211128235,
+      "learning_rate": 9.00704853429745e-05,
+      "loss": 0.41344562530517576,
+      "mean_token_accuracy": 0.9188940741121769,
+      "num_tokens": 9202085.0,
+      "step": 2150
+    },
+    {
+      "entropy": 0.37150444712489844,
+      "epoch": 0.6638002564682809,
+      "grad_norm": 0.23061881959438324,
+      "learning_rate": 8.958576278068655e-05,
+      "loss": 0.4002714157104492,
+      "mean_token_accuracy": 0.9211013509333134,
+      "num_tokens": 9414528.0,
+      "step": 2200
+    },
+    {
+      "epoch": 0.6638002564682809,
+      "eval_entropy": 0.4198416776652706,
+      "eval_loss": 0.44056928157806396,
+      "eval_mean_token_accuracy": 0.9060557997094749,
+      "eval_num_tokens": 9414528.0,
+      "eval_runtime": 37.0429,
+      "eval_samples_per_second": 58.473,
+      "eval_steps_per_second": 7.316,
+      "step": 2200
+    },
+    {
+      "entropy": 0.3824780482426286,
+      "epoch": 0.6788866259334692,
+      "grad_norm": 0.29528528451919556,
+      "learning_rate": 8.909085991822532e-05,
+      "loss": 0.4100413513183594,
+      "mean_token_accuracy": 0.9181285245716572,
+      "num_tokens": 9631160.0,
+      "step": 2250
+    },
+    {
+      "entropy": 0.37036353170871733,
+      "epoch": 0.6939729953986573,
+      "grad_norm": 0.2833597958087921,
+      "learning_rate": 8.858590403012954e-05,
+      "loss": 0.39582439422607424,
+      "mean_token_accuracy": 0.9203065976500511,
+      "num_tokens": 9844323.0,
+      "step": 2300
+    },
+    {
+      "entropy": 0.377471005320549,
+      "epoch": 0.7090593648638455,
+      "grad_norm": 0.2559050917625427,
+      "learning_rate": 8.807102497628199e-05,
+      "loss": 0.4039160919189453,
+      "mean_token_accuracy": 0.9185835334658623,
+      "num_tokens": 10060066.0,
+      "step": 2350
+    },
+    {
+      "entropy": 0.38689912386238573,
+      "epoch": 0.7241457343290337,
+      "grad_norm": 0.3571145236492157,
+      "learning_rate": 8.754635516851342e-05,
+      "loss": 0.41998291015625,
+      "mean_token_accuracy": 0.9171991994976998,
+      "num_tokens": 10275374.0,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7241457343290337,
+      "eval_entropy": 0.4061841280148038,
+      "eval_loss": 0.4392658472061157,
+      "eval_mean_token_accuracy": 0.9060493254573583,
+      "eval_num_tokens": 10275374.0,
+      "eval_runtime": 37.185,
+      "eval_samples_per_second": 58.249,
+      "eval_steps_per_second": 7.288,
+      "step": 2400
+    },
+    {
+      "entropy": 0.3773344187065959,
+      "epoch": 0.7392321037942219,
+      "grad_norm": 0.23827174305915833,
+      "learning_rate": 8.701202953655006e-05,
+      "loss": 0.4055968475341797,
+      "mean_token_accuracy": 0.9189482787251473,
+      "num_tokens": 10495301.0,
+      "step": 2450
+    },
+    {
+      "entropy": 0.3638977843523026,
+      "epoch": 0.7543184732594101,
+      "grad_norm": 0.247745543718338,
+      "learning_rate": 8.646818549331366e-05,
+      "loss": 0.38891139984130857,
+      "mean_token_accuracy": 0.9226090031862259,
+      "num_tokens": 10706938.0,
+      "step": 2500
+    },
+    {
+      "entropy": 0.358336652033031,
+      "epoch": 0.7694048427245983,
+      "grad_norm": 0.24292156100273132,
+      "learning_rate": 8.591496289958292e-05,
+      "loss": 0.3846548461914063,
+      "mean_token_accuracy": 0.923456951379776,
+      "num_tokens": 10918302.0,
+      "step": 2550
+    },
+    {
+      "entropy": 0.37086400829255584,
+      "epoch": 0.7844912121897866,
+      "grad_norm": 0.2979118525981903,
+      "learning_rate": 8.535250402802536e-05,
+      "loss": 0.39662261962890627,
+      "mean_token_accuracy": 0.9212297305464745,
+      "num_tokens": 11131056.0,
+      "step": 2600
+    },
+    {
+      "epoch": 0.7844912121897866,
+      "eval_entropy": 0.4161290250361186,
+      "eval_loss": 0.43674495816230774,
+      "eval_mean_token_accuracy": 0.9060781219788583,
+      "eval_num_tokens": 11131056.0,
+      "eval_runtime": 37.0488,
+      "eval_samples_per_second": 58.463,
+      "eval_steps_per_second": 7.315,
+      "step": 2600
+    },
+    {
+      "entropy": 0.36887906536459925,
+      "epoch": 0.7995775816549747,
+      "grad_norm": 0.25673073530197144,
+      "learning_rate": 8.478095352660897e-05,
+      "loss": 0.3948686218261719,
+      "mean_token_accuracy": 0.9204315200448037,
+      "num_tokens": 11345648.0,
+      "step": 2650
+    },
+    {
+      "entropy": 0.36981521353125574,
+      "epoch": 0.814663951120163,
+      "grad_norm": 0.2649747133255005,
+      "learning_rate": 8.4200458381403e-05,
+      "loss": 0.3937848663330078,
+      "mean_token_accuracy": 0.9218536545336247,
+      "num_tokens": 11559009.0,
+      "step": 2700
+    },
+    {
+      "entropy": 0.37904939975589513,
+      "epoch": 0.8297503205853511,
+      "grad_norm": 0.20989011228084564,
+      "learning_rate": 8.361116787877736e-05,
+      "loss": 0.4084677505493164,
+      "mean_token_accuracy": 0.9188165719807148,
+      "num_tokens": 11776255.0,
+      "step": 2750
+    },
+    {
+      "entropy": 0.3781035339459777,
+      "epoch": 0.8448366900505393,
+      "grad_norm": 0.2979874908924103,
+      "learning_rate": 8.301323356701069e-05,
+      "loss": 0.40767410278320315,
+      "mean_token_accuracy": 0.9183482979238033,
+      "num_tokens": 11994830.0,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8448366900505393,
+      "eval_entropy": 0.3918299580962016,
+      "eval_loss": 0.43606311082839966,
+      "eval_mean_token_accuracy": 0.9074829088805786,
+      "eval_num_tokens": 11994830.0,
+      "eval_runtime": 37.1502,
+      "eval_samples_per_second": 58.304,
+      "eval_steps_per_second": 7.295,
+      "step": 2800
+    },
+    {
+      "entropy": 0.3669764836877584,
+      "epoch": 0.8599230595157276,
+      "grad_norm": 0.3718933165073395,
+      "learning_rate": 8.240680921731639e-05,
+      "loss": 0.39511192321777344,
+      "mean_token_accuracy": 0.9215331043303013,
+      "num_tokens": 12210990.0,
+      "step": 2850
+    },
+    {
+      "entropy": 0.36516126081347466,
+      "epoch": 0.8750094289809157,
+      "grad_norm": 0.2584734559059143,
+      "learning_rate": 8.179205078429728e-05,
+      "loss": 0.3858111572265625,
+      "mean_token_accuracy": 0.9223315984010696,
+      "num_tokens": 12425768.0,
+      "step": 2900
+    },
+    {
+      "entropy": 0.36489626977592704,
+      "epoch": 0.890095798446104,
+      "grad_norm": 0.260593980550766,
+      "learning_rate": 8.116911636583866e-05,
+      "loss": 0.3904818344116211,
+      "mean_token_accuracy": 0.921723841279745,
+      "num_tokens": 12644047.0,
+      "step": 2950
+    },
+    {
+      "entropy": 0.35986222576349974,
+      "epoch": 0.9051821679112921,
+      "grad_norm": 0.2872949540615082,
+      "learning_rate": 8.053816616245007e-05,
+      "loss": 0.3802699661254883,
+      "mean_token_accuracy": 0.922919643521309,
+      "num_tokens": 12858612.0,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9051821679112921,
+      "eval_entropy": 0.39114147694128465,
+      "eval_loss": 0.43371766805648804,
+      "eval_mean_token_accuracy": 0.9085190791045608,
+      "eval_num_tokens": 12858612.0,
+      "eval_runtime": 37.1981,
+      "eval_samples_per_second": 58.229,
+      "eval_steps_per_second": 7.285,
+      "step": 3000
+    },
+    {
+      "entropy": 0.3628548865020275,
+      "epoch": 0.9202685373764804,
+      "grad_norm": 0.3397461771965027,
+      "learning_rate": 7.989936243606644e-05,
+      "loss": 0.38797500610351565,
+      "mean_token_accuracy": 0.9223182892799378,
+      "num_tokens": 13074583.0,
+      "step": 3050
+    },
+    {
+      "entropy": 0.3658443212881684,
+      "epoch": 0.9353549068416686,
+      "grad_norm": 0.21586942672729492,
+      "learning_rate": 7.925286946831907e-05,
+      "loss": 0.38840934753417966,
+      "mean_token_accuracy": 0.920469640493393,
+      "num_tokens": 13290531.0,
+      "step": 3100
+    },
+    {
+      "entropy": 0.3434870146587491,
+      "epoch": 0.9504412763068567,
+      "grad_norm": 0.4203721582889557,
+      "learning_rate": 7.859885351828722e-05,
+      "loss": 0.3654581832885742,
+      "mean_token_accuracy": 0.9264674150943756,
+      "num_tokens": 13501774.0,
+      "step": 3150
+    },
+    {
+      "entropy": 0.3622769057378173,
+      "epoch": 0.965527645772045,
+      "grad_norm": 0.4923344850540161,
+      "learning_rate": 7.793748277974116e-05,
+      "loss": 0.38648658752441406,
+      "mean_token_accuracy": 0.9216807208955288,
+      "num_tokens": 13719293.0,
+      "step": 3200
+    },
+    {
+      "epoch": 0.965527645772045,
+      "eval_entropy": 0.38819593988441453,
+      "eval_loss": 0.43394002318382263,
+      "eval_mean_token_accuracy": 0.9077828526496887,
+      "eval_num_tokens": 13719293.0,
+      "eval_runtime": 37.2602,
+      "eval_samples_per_second": 58.132,
+      "eval_steps_per_second": 7.273,
+      "step": 3200
+    },
+    {
+      "entropy": 0.35784762669354675,
+      "epoch": 0.9806140152372331,
+      "grad_norm": 0.295482873916626,
+      "learning_rate": 7.72689273378875e-05,
+      "loss": 0.3751881790161133,
+      "mean_token_accuracy": 0.9235683201253414,
+      "num_tokens": 13933952.0,
+      "step": 3250
+    },
+    {
+      "entropy": 0.35060869861394167,
+      "epoch": 0.9957003847024214,
+      "grad_norm": 0.3346746265888214,
+      "learning_rate": 7.659335912562849e-05,
+      "loss": 0.3715841293334961,
+      "mean_token_accuracy": 0.9249463096261025,
+      "num_tokens": 14146736.0,
+      "step": 3300
+    },
+    {
+      "entropy": 0.33144386649736896,
+      "epoch": 1.0105604586256318,
+      "grad_norm": 0.42002174258232117,
+      "learning_rate": 7.591095187934571e-05,
+      "loss": 0.3447689437866211,
+      "mean_token_accuracy": 0.9282537361389489,
+      "num_tokens": 14363628.0,
+      "step": 3350
+    },
+    {
+      "entropy": 0.29580484244972466,
+      "epoch": 1.02564682809082,
+      "grad_norm": 0.267696350812912,
+      "learning_rate": 7.522188109422025e-05,
+      "loss": 0.30982461929321287,
+      "mean_token_accuracy": 0.9349895706772804,
+      "num_tokens": 14577748.0,
+      "step": 3400
+    },
+    {
+      "epoch": 1.02564682809082,
+      "eval_entropy": 0.3474461413452546,
+      "eval_loss": 0.4461125433444977,
+      "eval_mean_token_accuracy": 0.9071816351140997,
+      "eval_num_tokens": 14577748.0,
+      "eval_runtime": 37.1626,
+      "eval_samples_per_second": 58.284,
+      "eval_steps_per_second": 7.292,
+      "step": 3400
+    },
+    {
+      "entropy": 0.30122795451432466,
+      "epoch": 1.0407331975560081,
+      "grad_norm": 0.2817290127277374,
+      "learning_rate": 7.452632397910045e-05,
+      "loss": 0.3127718353271484,
+      "mean_token_accuracy": 0.93441466614604,
+      "num_tokens": 14792632.0,
+      "step": 3450
+    },
+    {
+      "entropy": 0.2921810976602137,
+      "epoch": 1.0558195670211963,
+      "grad_norm": 0.3158903419971466,
+      "learning_rate": 7.3824459410929e-05,
+      "loss": 0.3033764839172363,
+      "mean_token_accuracy": 0.9362939922511577,
+      "num_tokens": 15005289.0,
+      "step": 3500
+    },
+    {
+      "entropy": 0.30137234527617696,
+      "epoch": 1.0709059364863847,
+      "grad_norm": 0.363569974899292,
+      "learning_rate": 7.311646788874093e-05,
+      "loss": 0.3192999267578125,
+      "mean_token_accuracy": 0.9330788172781468,
+      "num_tokens": 15221117.0,
+      "step": 3550
+    },
+    {
+      "entropy": 0.2979462206363678,
+      "epoch": 1.0859923059515728,
+      "grad_norm": 0.33784833550453186,
+      "learning_rate": 7.240253148724472e-05,
+      "loss": 0.3097235298156738,
+      "mean_token_accuracy": 0.935227975398302,
+      "num_tokens": 15433103.0,
+      "step": 3600
+    },
+    {
+      "epoch": 1.0859923059515728,
+      "eval_entropy": 0.33927392830378017,
+      "eval_loss": 0.44939693808555603,
+      "eval_mean_token_accuracy": 0.9080607321429517,
+      "eval_num_tokens": 15433103.0,
+      "eval_runtime": 37.0789,
+      "eval_samples_per_second": 58.416,
+      "eval_steps_per_second": 7.309,
+      "step": 3600
+    },
+    {
+      "entropy": 0.2955490641668439,
+      "epoch": 1.101078675416761,
+      "grad_norm": 0.26954564452171326,
+      "learning_rate": 7.168283380999781e-05,
+      "loss": 0.308115234375,
+      "mean_token_accuracy": 0.9356962423026561,
+      "num_tokens": 15647397.0,
+      "step": 3650
+    },
+    {
+      "entropy": 0.3024994757771492,
+      "epoch": 1.1161650448819491,
+      "grad_norm": 0.34243258833885193,
+      "learning_rate": 7.095755994218929e-05,
+      "loss": 0.3163371467590332,
+      "mean_token_accuracy": 0.933635587990284,
+      "num_tokens": 15862307.0,
+      "step": 3700
+    },
+    {
+      "entropy": 0.28693753194063903,
+      "epoch": 1.1312514143471373,
+      "grad_norm": 0.3753094971179962,
+      "learning_rate": 7.022689640304127e-05,
+      "loss": 0.30058185577392577,
+      "mean_token_accuracy": 0.9367860886454582,
+      "num_tokens": 16077573.0,
+      "step": 3750
+    },
+    {
+      "entropy": 0.2817346750572324,
+      "epoch": 1.1463377838123257,
+      "grad_norm": 0.36651188135147095,
+      "learning_rate": 6.949103109784174e-05,
+      "loss": 0.29092519760131835,
+      "mean_token_accuracy": 0.9392722348868847,
+      "num_tokens": 16287755.0,
+      "step": 3800
+    },
+    {
+      "epoch": 1.1463377838123257,
+      "eval_entropy": 0.3268261263181363,
+      "eval_loss": 0.45524126291275024,
+      "eval_mean_token_accuracy": 0.9080498321909746,
+      "eval_num_tokens": 16287755.0,
+      "eval_runtime": 37.2707,
+      "eval_samples_per_second": 58.115,
+      "eval_steps_per_second": 7.271,
+      "step": 3800
+    },
+    {
+      "entropy": 0.2927645993977785,
+      "epoch": 1.1614241532775138,
+      "grad_norm": 0.32257115840911865,
+      "learning_rate": 6.875015326962071e-05,
+      "loss": 0.3056098175048828,
+      "mean_token_accuracy": 0.9361269493401051,
+      "num_tokens": 16500869.0,
+      "step": 3850
+    },
+    {
+      "entropy": 0.2784220616333187,
+      "epoch": 1.176510522742702,
+      "grad_norm": 0.3277973234653473,
+      "learning_rate": 6.800445345048247e-05,
+      "loss": 0.2863511276245117,
+      "mean_token_accuracy": 0.9394664193689823,
+      "num_tokens": 16714052.0,
+      "step": 3900
+    },
+    {
+      "entropy": 0.28824312970042226,
+      "epoch": 1.1915968922078901,
+      "grad_norm": 0.45423227548599243,
+      "learning_rate": 6.725412341260632e-05,
+      "loss": 0.30036584854125975,
+      "mean_token_accuracy": 0.9367516180872917,
+      "num_tokens": 16929433.0,
+      "step": 3950
+    },
+    {
+      "entropy": 0.3048727669939399,
+      "epoch": 1.2066832616730783,
+      "grad_norm": 0.4174216091632843,
+      "learning_rate": 6.649935611892839e-05,
+      "loss": 0.3221250534057617,
+      "mean_token_accuracy": 0.9329589973390102,
+      "num_tokens": 17149131.0,
+      "step": 4000
+    },
+    {
+      "epoch": 1.2066832616730783,
+      "eval_entropy": 0.3393887681439794,
+      "eval_loss": 0.44970703125,
+      "eval_mean_token_accuracy": 0.907502448866728,
+      "eval_num_tokens": 17149131.0,
+      "eval_runtime": 37.1753,
+      "eval_samples_per_second": 58.265,
+      "eval_steps_per_second": 7.29,
+      "step": 4000
+    },
+    {
+      "entropy": 0.2786213347688317,
+      "epoch": 1.2217696311382666,
+      "grad_norm": 0.37704846262931824,
+      "learning_rate": 6.574034567351717e-05,
+      "loss": 0.28641138076782224,
+      "mean_token_accuracy": 0.9393399634957313,
+      "num_tokens": 17362785.0,
+      "step": 4050
+    },
+    {
+      "entropy": 0.2889704497903585,
+      "epoch": 1.2368560006034548,
+      "grad_norm": 0.3806622624397278,
+      "learning_rate": 6.49772872716557e-05,
+      "loss": 0.30537261962890627,
+      "mean_token_accuracy": 0.9361220151185989,
+      "num_tokens": 17578670.0,
+      "step": 4100
+    },
+    {
+      "entropy": 0.29292779937386515,
+      "epoch": 1.251942370068643,
+      "grad_norm": 0.3169683814048767,
+      "learning_rate": 6.421037714964293e-05,
+      "loss": 0.30579683303833005,
+      "mean_token_accuracy": 0.9352342364192009,
+      "num_tokens": 17802008.0,
+      "step": 4150
+    },
+    {
+      "entropy": 0.28012287182733414,
+      "epoch": 1.267028739533831,
+      "grad_norm": 0.2556808888912201,
+      "learning_rate": 6.343981253432761e-05,
+      "loss": 0.2942901611328125,
+      "mean_token_accuracy": 0.93847431153059,
+      "num_tokens": 18015619.0,
+      "step": 4200
+    },
+    {
+      "epoch": 1.267028739533831,
+      "eval_entropy": 0.3326790584354383,
+      "eval_loss": 0.45203062891960144,
+      "eval_mean_token_accuracy": 0.9082723077372871,
+      "eval_num_tokens": 18015619.0,
+      "eval_runtime": 37.2103,
+      "eval_samples_per_second": 58.21,
+      "eval_steps_per_second": 7.283,
+      "step": 4200
+    },
+    {
+      "entropy": 0.2790189484320581,
+      "epoch": 1.2821151089990193,
+      "grad_norm": 0.3167658746242523,
+      "learning_rate": 6.266579159238723e-05,
+      "loss": 0.2901947212219238,
+      "mean_token_accuracy": 0.9385272514820099,
+      "num_tokens": 18231214.0,
+      "step": 4250
+    },
+    {
+      "entropy": 0.28492060406133535,
+      "epoch": 1.2972014784642076,
+      "grad_norm": 0.3428349792957306,
+      "learning_rate": 6.188851337936541e-05,
+      "loss": 0.2975013542175293,
+      "mean_token_accuracy": 0.9384494511783124,
+      "num_tokens": 18443602.0,
+      "step": 4300
+    },
+    {
+      "entropy": 0.28728573814034464,
+      "epoch": 1.3122878479293958,
+      "grad_norm": 0.3879326581954956,
+      "learning_rate": 6.110817778848062e-05,
+      "loss": 0.2947955894470215,
+      "mean_token_accuracy": 0.9376472027599811,
+      "num_tokens": 18658957.0,
+      "step": 4350
+    },
+    {
+      "entropy": 0.27078360263258217,
+      "epoch": 1.327374217394584,
+      "grad_norm": 0.45004045963287354,
+      "learning_rate": 6.032498549921939e-05,
+      "loss": 0.28305835723876954,
+      "mean_token_accuracy": 0.9406134666502476,
+      "num_tokens": 18870053.0,
+      "step": 4400
+    },
+    {
+      "epoch": 1.327374217394584,
+      "eval_entropy": 0.3294620155958232,
+      "eval_loss": 0.4543997049331665,
+      "eval_mean_token_accuracy": 0.9081211842294109,
+      "eval_num_tokens": 18870053.0,
+      "eval_runtime": 37.2524,
+      "eval_samples_per_second": 58.144,
+      "eval_steps_per_second": 7.275,
+      "step": 4400
+    },
+    {
+      "entropy": 0.2799815020523965,
+      "epoch": 1.3424605868597723,
+      "grad_norm": 0.44520097970962524,
+      "learning_rate": 5.953913792572761e-05,
+      "loss": 0.28622047424316405,
+      "mean_token_accuracy": 0.9398869612812996,
+      "num_tokens": 19083502.0,
+      "step": 4450
+    },
+    {
+      "entropy": 0.28538848023861646,
+      "epoch": 1.3575469563249605,
+      "grad_norm": 0.376067578792572,
+      "learning_rate": 5.8750837165012516e-05,
+      "loss": 0.29442657470703126,
+      "mean_token_accuracy": 0.9369049946963787,
+      "num_tokens": 19301474.0,
+      "step": 4500
+    },
+    {
+      "entropy": 0.2700873460620642,
+      "epoch": 1.3726333257901486,
+      "grad_norm": 0.3803156316280365,
+      "learning_rate": 5.79602859449694e-05,
+      "loss": 0.2789628601074219,
+      "mean_token_accuracy": 0.9409016688168049,
+      "num_tokens": 19513118.0,
+      "step": 4550
+    },
+    {
+      "entropy": 0.2818992605619133,
+      "epoch": 1.3877196952553368,
+      "grad_norm": 0.3878261148929596,
+      "learning_rate": 5.716768757224583e-05,
+      "loss": 0.291706428527832,
+      "mean_token_accuracy": 0.9382703882455826,
+      "num_tokens": 19729704.0,
+      "step": 4600
+    },
+    {
+      "epoch": 1.3877196952553368,
+      "eval_entropy": 0.3223673155551907,
+      "eval_loss": 0.4583451747894287,
+      "eval_mean_token_accuracy": 0.9085318800268138,
+      "eval_num_tokens": 19729704.0,
+      "eval_runtime": 37.2126,
+      "eval_samples_per_second": 58.206,
+      "eval_steps_per_second": 7.282,
+      "step": 4600
+    },
+    {
+      "entropy": 0.2773911802843213,
+      "epoch": 1.402806064720525,
+      "grad_norm": 0.4441978931427002,
+      "learning_rate": 5.637324587995714e-05,
+      "loss": 0.28889572143554687,
+      "mean_token_accuracy": 0.9393369700014591,
+      "num_tokens": 19941579.0,
+      "step": 4650
+    },
+    {
+      "entropy": 0.2760587067343295,
+      "epoch": 1.4178924341857133,
+      "grad_norm": 0.31453457474708557,
+      "learning_rate": 5.557716517526656e-05,
+      "loss": 0.2823097038269043,
+      "mean_token_accuracy": 0.9404082801938057,
+      "num_tokens": 20152481.0,
+      "step": 4700
+    },
+    {
+      "entropy": 0.2687125858291984,
+      "epoch": 1.4329788036509015,
+      "grad_norm": 0.4302677810192108,
+      "learning_rate": 5.4779650186843335e-05,
+      "loss": 0.28021940231323245,
+      "mean_token_accuracy": 0.9412896266579628,
+      "num_tokens": 20364280.0,
+      "step": 4750
+    },
+    {
+      "entropy": 0.28299310375005005,
+      "epoch": 1.4480651731160896,
+      "grad_norm": 0.4275122582912445,
+      "learning_rate": 5.3980906012212575e-05,
+      "loss": 0.29417877197265624,
+      "mean_token_accuracy": 0.9380144135653973,
+      "num_tokens": 20579080.0,
+      "step": 4800
+    },
+    {
+      "epoch": 1.4480651731160896,
+      "eval_entropy": 0.3136782179899321,
+      "eval_loss": 0.4654751420021057,
+      "eval_mean_token_accuracy": 0.9077895956725652,
+      "eval_num_tokens": 20579080.0,
+      "eval_runtime": 37.1335,
+      "eval_samples_per_second": 58.33,
+      "eval_steps_per_second": 7.298,
+      "step": 4800
+    },
+    {
+      "entropy": 0.26981769694015384,
+      "epoch": 1.4631515425812778,
+      "grad_norm": 0.3479642868041992,
+      "learning_rate": 5.318113806501009e-05,
+      "loss": 0.2793255615234375,
+      "mean_token_accuracy": 0.940917958021164,
+      "num_tokens": 20794818.0,
+      "step": 4850
+    },
+    {
+      "entropy": 0.27721120171248914,
+      "epoch": 1.478237912046466,
+      "grad_norm": 0.36213237047195435,
+      "learning_rate": 5.238055202215606e-05,
+      "loss": 0.2876263427734375,
+      "mean_token_accuracy": 0.9395251777768135,
+      "num_tokens": 21007893.0,
+      "step": 4900
+    },
+    {
+      "entropy": 0.2697361321747303,
+      "epoch": 1.4933242815116543,
+      "grad_norm": 0.3829312324523926,
+      "learning_rate": 5.157935377096091e-05,
+      "loss": 0.2791532325744629,
+      "mean_token_accuracy": 0.9417897801101208,
+      "num_tokens": 21221698.0,
+      "step": 4950
+    },
+    {
+      "entropy": 0.26987188443541527,
+      "epoch": 1.5084106509768425,
+      "grad_norm": 0.5338497757911682,
+      "learning_rate": 5.077774935617715e-05,
+      "loss": 0.2768931770324707,
+      "mean_token_accuracy": 0.9414016118645668,
+      "num_tokens": 21434912.0,
+      "step": 5000
+    },
+    {
+      "epoch": 1.5084106509768425,
+      "eval_entropy": 0.31743444043221947,
+      "eval_loss": 0.46169763803482056,
+      "eval_mean_token_accuracy": 0.9076986282074143,
+      "eval_num_tokens": 21434912.0,
+      "eval_runtime": 37.2102,
+      "eval_samples_per_second": 58.21,
+      "eval_steps_per_second": 7.283,
+      "step": 5000
+    },
+    {
+      "entropy": 0.27476332798600195,
+      "epoch": 1.5234970204420306,
+      "grad_norm": 0.3199191987514496,
+      "learning_rate": 4.997594492701074e-05,
+      "loss": 0.2819905471801758,
+      "mean_token_accuracy": 0.9403638400137424,
+      "num_tokens": 21650284.0,
+      "step": 5050
+    },
+    {
+      "entropy": 0.2720643111690879,
+      "epoch": 1.538583389907219,
+      "grad_norm": 0.49105435609817505,
+      "learning_rate": 4.917414668410541e-05,
+      "loss": 0.28312482833862307,
+      "mean_token_accuracy": 0.9400044956803322,
+      "num_tokens": 21864950.0,
+      "step": 5100
+    },
+    {
+      "entropy": 0.2742350418306887,
+      "epoch": 1.553669759372407,
+      "grad_norm": 0.3570726215839386,
+      "learning_rate": 4.837256082651398e-05,
+      "loss": 0.2866826057434082,
+      "mean_token_accuracy": 0.940371020436287,
+      "num_tokens": 22081595.0,
+      "step": 5150
+    },
+    {
+      "entropy": 0.26034254603087903,
+      "epoch": 1.5687561288375953,
+      "grad_norm": 0.37878358364105225,
+      "learning_rate": 4.757139349867003e-05,
+      "loss": 0.26901472091674805,
+      "mean_token_accuracy": 0.943619585186243,
+      "num_tokens": 22293286.0,
+      "step": 5200
+    },
+    {
+      "epoch": 1.5687561288375953,
+      "eval_entropy": 0.3216313538021267,
+      "eval_loss": 0.45686572790145874,
+      "eval_mean_token_accuracy": 0.908281669625497,
+      "eval_num_tokens": 22293286.0,
+      "eval_runtime": 37.2587,
+      "eval_samples_per_second": 58.134,
+      "eval_steps_per_second": 7.273,
+      "step": 5200
+    },
+    {
+      "entropy": 0.25965558482334017,
+      "epoch": 1.5838424983027835,
+      "grad_norm": 0.5104638338088989,
+      "learning_rate": 4.677085073737347e-05,
+      "loss": 0.2651633071899414,
+      "mean_token_accuracy": 0.943703119456768,
+      "num_tokens": 22507183.0,
+      "step": 5250
+    },
+    {
+      "entropy": 0.2634627244435251,
+      "epoch": 1.5989288677679716,
+      "grad_norm": 0.39737868309020996,
+      "learning_rate": 4.5971138418804024e-05,
+      "loss": 0.2716307067871094,
+      "mean_token_accuracy": 0.9426473924517631,
+      "num_tokens": 22719425.0,
+      "step": 5300
+    },
+    {
+      "entropy": 0.26387994691729544,
+      "epoch": 1.61401523723316,
+      "grad_norm": 0.3169187605381012,
+      "learning_rate": 4.517246220557577e-05,
+      "loss": 0.2690562057495117,
+      "mean_token_accuracy": 0.9435765042901039,
+      "num_tokens": 22935863.0,
+      "step": 5350
+    },
+    {
+      "entropy": 0.2732780100405216,
+      "epoch": 1.629101606698348,
+      "grad_norm": 0.5206496715545654,
+      "learning_rate": 4.437502749384686e-05,
+      "loss": 0.28528207778930664,
+      "mean_token_accuracy": 0.9399016280472279,
+      "num_tokens": 23150285.0,
+      "step": 5400
+    },
+    {
+      "epoch": 1.629101606698348,
+      "eval_entropy": 0.3117673156776111,
+      "eval_loss": 0.4644741117954254,
+      "eval_mean_token_accuracy": 0.9071205707053857,
+      "eval_num_tokens": 23150285.0,
+      "eval_runtime": 37.233,
+      "eval_samples_per_second": 58.174,
+      "eval_steps_per_second": 7.278,
+      "step": 5400
+    },
+    {
+      "entropy": 0.25959222281351685,
+      "epoch": 1.6441879761635363,
+      "grad_norm": 0.37161391973495483,
+      "learning_rate": 4.357903936049758e-05,
+      "loss": 0.26540708541870117,
+      "mean_token_accuracy": 0.9441601178050041,
+      "num_tokens": 23361061.0,
+      "step": 5450
+    },
+    {
+      "entropy": 0.258843588642776,
+      "epoch": 1.6592743456287244,
+      "grad_norm": 0.3901636302471161,
+      "learning_rate": 4.278470251039052e-05,
+      "loss": 0.2657164001464844,
+      "mean_token_accuracy": 0.9435085739195347,
+      "num_tokens": 23576068.0,
+      "step": 5500
+    },
+    {
+      "entropy": 0.2609032719023526,
+      "epoch": 1.6743607150939126,
+      "grad_norm": 0.3928777873516083,
+      "learning_rate": 4.199222122372658e-05,
+      "loss": 0.2640374183654785,
+      "mean_token_accuracy": 0.9443163026869297,
+      "num_tokens": 23787002.0,
+      "step": 5550
+    },
+    {
+      "entropy": 0.2564471643790603,
+      "epoch": 1.689447084559101,
+      "grad_norm": 0.3459172546863556,
+      "learning_rate": 4.1201799303509984e-05,
+      "loss": 0.2629901695251465,
+      "mean_token_accuracy": 0.9437471415102482,
+      "num_tokens": 23997298.0,
+      "step": 5600
+    },
+    {
+      "epoch": 1.689447084559101,
+      "eval_entropy": 0.30813075843992266,
+      "eval_loss": 0.46801915764808655,
+      "eval_mean_token_accuracy": 0.9075211451502304,
+      "eval_num_tokens": 23997298.0,
+      "eval_runtime": 37.1802,
+      "eval_samples_per_second": 58.257,
+      "eval_steps_per_second": 7.289,
+      "step": 5600
+    },
+    {
+      "entropy": 0.26069454461336133,
+      "epoch": 1.704533454024289,
+      "grad_norm": 0.4609261453151703,
+      "learning_rate": 4.041364002313619e-05,
+      "loss": 0.26864715576171877,
+      "mean_token_accuracy": 0.9432297512888909,
+      "num_tokens": 24213258.0,
+      "step": 5650
+    },
+    {
+      "entropy": 0.26323398426175115,
+      "epoch": 1.7196198234894773,
+      "grad_norm": 0.4566548764705658,
+      "learning_rate": 3.962794607411584e-05,
+      "loss": 0.2684891510009766,
+      "mean_token_accuracy": 0.9431661969423294,
+      "num_tokens": 24429089.0,
+      "step": 5700
+    },
+    {
+      "entropy": 0.26415870275348424,
+      "epoch": 1.7347061929546654,
+      "grad_norm": 0.454801082611084,
+      "learning_rate": 3.8844919513948486e-05,
+      "loss": 0.2741558265686035,
+      "mean_token_accuracy": 0.9412616415321827,
+      "num_tokens": 24646888.0,
+      "step": 5750
+    },
+    {
+      "entropy": 0.25849502749741077,
+      "epoch": 1.7497925624198536,
+      "grad_norm": 0.47254759073257446,
+      "learning_rate": 3.80647617141593e-05,
+      "loss": 0.26054887771606444,
+      "mean_token_accuracy": 0.94430311486125,
+      "num_tokens": 24861569.0,
+      "step": 5800
+    },
+    {
+      "epoch": 1.7497925624198536,
+      "eval_entropy": 0.29998021306146994,
+      "eval_loss": 0.47340917587280273,
+      "eval_mean_token_accuracy": 0.9083721395788158,
+      "eval_num_tokens": 24861569.0,
+      "eval_runtime": 37.3662,
+      "eval_samples_per_second": 57.967,
+      "eval_steps_per_second": 7.253,
+      "step": 5800
+    },
+    {
+      "entropy": 0.2529403425566852,
+      "epoch": 1.764878931885042,
+      "grad_norm": 0.45025748014450073,
+      "learning_rate": 3.7287673308512186e-05,
+      "loss": 0.26045492172241214,
+      "mean_token_accuracy": 0.9441630898416042,
+      "num_tokens": 25079317.0,
+      "step": 5850
+    },
+    {
+      "entropy": 0.2550449293479323,
+      "epoch": 1.7799653013502301,
+      "grad_norm": 0.4205138087272644,
+      "learning_rate": 3.65138541414127e-05,
+      "loss": 0.26142183303833005,
+      "mean_token_accuracy": 0.9442940099537372,
+      "num_tokens": 25293828.0,
+      "step": 5900
+    },
+    {
+      "entropy": 0.26166333435103295,
+      "epoch": 1.7950516708154183,
+      "grad_norm": 0.32279399037361145,
+      "learning_rate": 3.5743503216513875e-05,
+      "loss": 0.27175371170043944,
+      "mean_token_accuracy": 0.9426822887361049,
+      "num_tokens": 25509857.0,
+      "step": 5950
+    },
+    {
+      "entropy": 0.2534090391360223,
+      "epoch": 1.8101380402806064,
+      "grad_norm": 0.4058147370815277,
+      "learning_rate": 3.4976818645538266e-05,
+      "loss": 0.2650087547302246,
+      "mean_token_accuracy": 0.9448564203083515,
+      "num_tokens": 25722906.0,
+      "step": 6000
+    },
+    {
+      "epoch": 1.8101380402806064,
+      "eval_entropy": 0.3064177740379013,
+      "eval_loss": 0.4730622470378876,
+      "eval_mean_token_accuracy": 0.9072717905484442,
+      "eval_num_tokens": 25722906.0,
+      "eval_runtime": 37.3208,
+      "eval_samples_per_second": 58.037,
+      "eval_steps_per_second": 7.261,
+      "step": 6000
+    },
+    {
+      "entropy": 0.2568224469386041,
+      "epoch": 1.8252244097457946,
+      "grad_norm": 0.7752001285552979,
+      "learning_rate": 3.42139975973295e-05,
+      "loss": 0.26035600662231445,
+      "mean_token_accuracy": 0.9453255115449428,
+      "num_tokens": 25934089.0,
+      "step": 6050
+    },
+    {
+      "entropy": 0.25038454053923487,
+      "epoch": 1.840310779210983,
+      "grad_norm": 0.657081127166748,
+      "learning_rate": 3.3455236247146124e-05,
+      "loss": 0.25294511795043945,
+      "mean_token_accuracy": 0.9468451295793057,
+      "num_tokens": 26145757.0,
+      "step": 6100
+    },
+    {
+      "entropy": 0.25195862116292117,
+      "epoch": 1.8553971486761711,
+      "grad_norm": 0.4311438202857971,
+      "learning_rate": 3.270072972621116e-05,
+      "loss": 0.2561396598815918,
+      "mean_token_accuracy": 0.945042757242918,
+      "num_tokens": 26363126.0,
+      "step": 6150
+    },
+    {
+      "entropy": 0.24140722384676339,
+      "epoch": 1.8704835181413593,
+      "grad_norm": 0.3511962592601776,
+      "learning_rate": 3.1950672071529976e-05,
+      "loss": 0.2441703224182129,
+      "mean_token_accuracy": 0.9480637626349926,
+      "num_tokens": 26570762.0,
+      "step": 6200
+    },
+    {
+      "epoch": 1.8704835181413593,
+      "eval_entropy": 0.29495117315726965,
+      "eval_loss": 0.4819435179233551,
+      "eval_mean_token_accuracy": 0.9071202113179703,
+      "eval_num_tokens": 26570762.0,
+      "eval_runtime": 37.2485,
+      "eval_samples_per_second": 58.15,
+      "eval_steps_per_second": 7.275,
+      "step": 6200
+    },
+    {
+      "entropy": 0.25350754015147686,
+      "epoch": 1.8855698876065476,
+      "grad_norm": 0.4511338472366333,
+      "learning_rate": 3.1205256175989706e-05,
+      "loss": 0.2587089538574219,
+      "mean_token_accuracy": 0.9447831605374813,
+      "num_tokens": 26786633.0,
+      "step": 6250
+    },
+    {
+      "entropy": 0.24989185376092793,
+      "epoch": 1.9006562570717356,
+      "grad_norm": 0.6432870030403137,
+      "learning_rate": 3.046467373875282e-05,
+      "loss": 0.2545445251464844,
+      "mean_token_accuracy": 0.9456490239501,
+      "num_tokens": 26999534.0,
+      "step": 6300
+    },
+    {
+      "entropy": 0.2594654941000044,
+      "epoch": 1.915742626536924,
+      "grad_norm": 0.3382566273212433,
+      "learning_rate": 2.9729115215957664e-05,
+      "loss": 0.26606035232543945,
+      "mean_token_accuracy": 0.9442368443310261,
+      "num_tokens": 27215468.0,
+      "step": 6350
+    },
+    {
+      "entropy": 0.2511021154373884,
+      "epoch": 1.930828996002112,
+      "grad_norm": 0.5591769814491272,
+      "learning_rate": 2.8998769771738833e-05,
+      "loss": 0.25475397109985354,
+      "mean_token_accuracy": 0.9452549339830876,
+      "num_tokens": 27431751.0,
+      "step": 6400
+    },
+    {
+      "epoch": 1.930828996002112,
+      "eval_entropy": 0.2883295046777303,
+      "eval_loss": 0.4905667006969452,
+      "eval_mean_token_accuracy": 0.9074331280050243,
+      "eval_num_tokens": 27431751.0,
+      "eval_runtime": 37.2686,
+      "eval_samples_per_second": 58.119,
+      "eval_steps_per_second": 7.272,
+      "step": 6400
+    },
+    {
+      "entropy": 0.24450184650719164,
+      "epoch": 1.9459153654673003,
+      "grad_norm": 0.4845283031463623,
+      "learning_rate": 2.8273825229579558e-05,
+      "loss": 0.2493889808654785,
+      "mean_token_accuracy": 0.9475250770151615,
+      "num_tokens": 27643957.0,
+      "step": 6450
+    },
+    {
+      "entropy": 0.25026212429627775,
+      "epoch": 1.9610017349324886,
+      "grad_norm": 0.4210788309574127,
+      "learning_rate": 2.7554468024009082e-05,
+      "loss": 0.25582372665405273,
+      "mean_token_accuracy": 0.9451599711179733,
+      "num_tokens": 27862796.0,
+      "step": 6500
+    },
+    {
+      "entropy": 0.2594556957855821,
+      "epoch": 1.9760881043976766,
+      "grad_norm": 0.42034855484962463,
+      "learning_rate": 2.6840883152657075e-05,
+      "loss": 0.2614474105834961,
+      "mean_token_accuracy": 0.9446786725521088,
+      "num_tokens": 28080081.0,
+      "step": 6550
+    },
+    {
+      "entropy": 0.2424509571492672,
+      "epoch": 1.991174473862865,
+      "grad_norm": 0.5637903213500977,
+      "learning_rate": 2.6133254128677766e-05,
+      "loss": 0.24372596740722657,
+      "mean_token_accuracy": 0.9479729002714157,
+      "num_tokens": 28291753.0,
+      "step": 6600
+    },
+    {
+      "epoch": 1.991174473862865,
+      "eval_entropy": 0.2895552223458941,
+      "eval_loss": 0.48857003450393677,
+      "eval_mean_token_accuracy": 0.9079141579430922,
+      "eval_num_tokens": 28291753.0,
+      "eval_runtime": 37.2416,
+      "eval_samples_per_second": 58.161,
+      "eval_steps_per_second": 7.277,
+      "step": 6600
+    },
+    {
+      "entropy": 0.24137189186315247,
+      "epoch": 2.0060345477860753,
+      "grad_norm": 0.4415420591831207,
+      "learning_rate": 2.5431762933555627e-05,
+      "loss": 0.23770923614501954,
+      "mean_token_accuracy": 0.9500614218602931,
+      "num_tokens": 28501531.0,
+      "step": 6650
+    },
+    {
+      "entropy": 0.20573811857029795,
+      "epoch": 2.0211209172512636,
+      "grad_norm": 0.5171937942504883,
+      "learning_rate": 2.473658997030514e-05,
+      "loss": 0.19352760314941406,
+      "mean_token_accuracy": 0.9575251358747482,
+      "num_tokens": 28718312.0,
+      "step": 6700
+    },
+    {
+      "entropy": 0.20038287844508887,
+      "epoch": 2.0362072867164516,
+      "grad_norm": 0.42165201902389526,
+      "learning_rate": 2.404791401707633e-05,
+      "loss": 0.18894313812255858,
+      "mean_token_accuracy": 0.9588627055287361,
+      "num_tokens": 28934285.0,
+      "step": 6750
+    },
+    {
+      "entropy": 0.19508995985612274,
+      "epoch": 2.05129365618164,
+      "grad_norm": 0.35577845573425293,
+      "learning_rate": 2.3365912181178313e-05,
+      "loss": 0.1867695426940918,
+      "mean_token_accuracy": 0.9598716080188752,
+      "num_tokens": 29145157.0,
+      "step": 6800
+    },
+    {
+      "epoch": 2.05129365618164,
+      "eval_entropy": 0.2568544143350362,
+      "eval_loss": 0.5305211544036865,
+      "eval_mean_token_accuracy": 0.9064581629534929,
+      "eval_num_tokens": 29145157.0,
+      "eval_runtime": 37.1895,
+      "eval_samples_per_second": 58.242,
+      "eval_steps_per_second": 7.287,
+      "step": 6800
+    },
+    {
+      "entropy": 0.2055508348532021,
+      "epoch": 2.066380025646828,
+      "grad_norm": 0.42946624755859375,
+      "learning_rate": 2.269075985353254e-05,
+      "loss": 0.1946415138244629,
+      "mean_token_accuracy": 0.9576617389917373,
+      "num_tokens": 29362502.0,
+      "step": 6850
+    },
+    {
+      "entropy": 0.19851462194696068,
+      "epoch": 2.0814663951120163,
+      "grad_norm": 0.4784873127937317,
+      "learning_rate": 2.20226306635672e-05,
+      "loss": 0.1882442855834961,
+      "mean_token_accuracy": 0.958879132270813,
+      "num_tokens": 29577964.0,
+      "step": 6900
+    },
+    {
+      "entropy": 0.20270802198909224,
+      "epoch": 2.0965527645772046,
+      "grad_norm": 0.5308135151863098,
+      "learning_rate": 2.1361696434564993e-05,
+      "loss": 0.1924464225769043,
+      "mean_token_accuracy": 0.9582864837348462,
+      "num_tokens": 29792051.0,
+      "step": 6950
+    },
+    {
+      "entropy": 0.19360160086303949,
+      "epoch": 2.1116391340423926,
+      "grad_norm": 0.3992004990577698,
+      "learning_rate": 2.070812713947513e-05,
+      "loss": 0.18389596939086914,
+      "mean_token_accuracy": 0.9606801150739193,
+      "num_tokens": 30010593.0,
+      "step": 7000
+    },
+    {
+      "epoch": 2.1116391340423926,
+      "eval_entropy": 0.2496587313794122,
+      "eval_loss": 0.5409784317016602,
+      "eval_mean_token_accuracy": 0.9062512767710809,
+      "eval_num_tokens": 30010593.0,
+      "eval_runtime": 37.2073,
+      "eval_samples_per_second": 58.214,
+      "eval_steps_per_second": 7.284,
+      "step": 7000
+    },
+    {
+      "entropy": 0.19243362288922072,
+      "epoch": 2.126725503507581,
+      "grad_norm": 0.485505610704422,
+      "learning_rate": 2.006209085720122e-05,
+      "loss": 0.18137836456298828,
+      "mean_token_accuracy": 0.9603879378736019,
+      "num_tokens": 30224235.0,
+      "step": 7050
+    },
+    {
+      "entropy": 0.1971529097482562,
+      "epoch": 2.1418118729727693,
+      "grad_norm": 0.3532182276248932,
+      "learning_rate": 1.9423753729376388e-05,
+      "loss": 0.1862693214416504,
+      "mean_token_accuracy": 0.959182897657156,
+      "num_tokens": 30438574.0,
+      "step": 7100
+    },
+    {
+      "entropy": 0.19837678778916598,
+      "epoch": 2.1568982424379572,
+      "grad_norm": 0.5558100342750549,
+      "learning_rate": 1.8793279917636448e-05,
+      "loss": 0.1895616340637207,
+      "mean_token_accuracy": 0.958885507285595,
+      "num_tokens": 30654293.0,
+      "step": 7150
+    },
+    {
+      "entropy": 0.19329896425828338,
+      "epoch": 2.1719846119031456,
+      "grad_norm": 0.4981042146682739,
+      "learning_rate": 1.817083156140237e-05,
+      "loss": 0.18090120315551758,
+      "mean_token_accuracy": 0.9607469955086708,
+      "num_tokens": 30865232.0,
+      "step": 7200
+    },
+    {
+      "epoch": 2.1719846119031456,
+      "eval_entropy": 0.2497983215534819,
+      "eval_loss": 0.5419089198112488,
+      "eval_mean_token_accuracy": 0.9055907944031747,
+      "eval_num_tokens": 30865232.0,
+      "eval_runtime": 37.1887,
+      "eval_samples_per_second": 58.243,
+      "eval_steps_per_second": 7.287,
+      "step": 7200
+    },
+    {
+      "entropy": 0.19074429869651793,
+      "epoch": 2.1870709813683336,
+      "grad_norm": 0.5064796209335327,
+      "learning_rate": 1.7556568736182667e-05,
+      "loss": 0.18279983520507812,
+      "mean_token_accuracy": 0.9610813467204571,
+      "num_tokens": 31074859.0,
+      "step": 7250
+    },
+    {
+      "entropy": 0.18915067540481686,
+      "epoch": 2.202157350833522,
+      "grad_norm": 0.462493360042572,
+      "learning_rate": 1.6950649412406777e-05,
+      "loss": 0.18141572952270507,
+      "mean_token_accuracy": 0.9607113988697529,
+      "num_tokens": 31285366.0,
+      "step": 7300
+    },
+    {
+      "entropy": 0.19164147678762675,
+      "epoch": 2.2172437202987103,
+      "grad_norm": 0.44526222348213196,
+      "learning_rate": 1.63532294147997e-05,
+      "loss": 0.1814894676208496,
+      "mean_token_accuracy": 0.9610688266158104,
+      "num_tokens": 31497431.0,
+      "step": 7350
+    },
+    {
+      "entropy": 0.19515870379284025,
+      "epoch": 2.2323300897638982,
+      "grad_norm": 0.5659943222999573,
+      "learning_rate": 1.5764462382308254e-05,
+      "loss": 0.18695903778076173,
+      "mean_token_accuracy": 0.9594297151267529,
+      "num_tokens": 31711142.0,
+      "step": 7400
+    },
+    {
+      "epoch": 2.2323300897638982,
+      "eval_entropy": 0.24821185405843812,
+      "eval_loss": 0.5459219217300415,
+      "eval_mean_token_accuracy": 0.9052295550649017,
+      "eval_num_tokens": 31711142.0,
+      "eval_runtime": 37.1776,
+      "eval_samples_per_second": 58.261,
+      "eval_steps_per_second": 7.289,
+      "step": 7400
+    },
+    {
+      "entropy": 0.19548690050840378,
+      "epoch": 2.2474164592290866,
+      "grad_norm": 0.498912513256073,
+      "learning_rate": 1.5184499728589846e-05,
+      "loss": 0.18344196319580078,
+      "mean_token_accuracy": 0.9600871665775776,
+      "num_tokens": 31923379.0,
+      "step": 7450
+    },
+    {
+      "entropy": 0.19262660641223192,
+      "epoch": 2.2625028286942745,
+      "grad_norm": 0.41678550839424133,
+      "learning_rate": 1.4613490603073194e-05,
+      "loss": 0.18303531646728516,
+      "mean_token_accuracy": 0.9605987747013569,
+      "num_tokens": 32135266.0,
+      "step": 7500
+    },
+    {
+      "entropy": 0.19937248670496047,
+      "epoch": 2.277589198159463,
+      "grad_norm": 0.6035925149917603,
+      "learning_rate": 1.4051581852601492e-05,
+      "loss": 0.18883104324340821,
+      "mean_token_accuracy": 0.9591751427948475,
+      "num_tokens": 32351854.0,
+      "step": 7550
+    },
+    {
+      "entropy": 0.19550999199971555,
+      "epoch": 2.2926755676246513,
+      "grad_norm": 0.4360465705394745,
+      "learning_rate": 1.3498917983667658e-05,
+      "loss": 0.18691642761230468,
+      "mean_token_accuracy": 0.9598778022825718,
+      "num_tokens": 32567025.0,
+      "step": 7600
+    },
+    {
+      "epoch": 2.2926755676246513,
+      "eval_entropy": 0.24827074039576238,
+      "eval_loss": 0.5438116192817688,
+      "eval_mean_token_accuracy": 0.9056458268658262,
+      "eval_num_tokens": 32567025.0,
+      "eval_runtime": 37.13,
+      "eval_samples_per_second": 58.336,
+      "eval_steps_per_second": 7.299,
+      "step": 7600
+    },
+    {
+      "entropy": 0.19866504015401007,
+      "epoch": 2.3077619370898392,
+      "grad_norm": 0.4768233299255371,
+      "learning_rate": 1.2955641125251555e-05,
+      "loss": 0.18666345596313477,
+      "mean_token_accuracy": 0.9592089447379112,
+      "num_tokens": 32781090.0,
+      "step": 7650
+    },
+    {
+      "entropy": 0.19146794026717542,
+      "epoch": 2.3228483065550276,
+      "grad_norm": 0.392702579498291,
+      "learning_rate": 1.2421890992268542e-05,
+      "loss": 0.18030118942260742,
+      "mean_token_accuracy": 0.9610011300444603,
+      "num_tokens": 32993619.0,
+      "step": 7700
+    },
+    {
+      "entropy": 0.19083258060738445,
+      "epoch": 2.3379346760202155,
+      "grad_norm": 0.5067113637924194,
+      "learning_rate": 1.189780484963885e-05,
+      "loss": 0.17912176132202148,
+      "mean_token_accuracy": 0.9609163939952851,
+      "num_tokens": 33207287.0,
+      "step": 7750
+    },
+    {
+      "entropy": 0.19270094383507966,
+      "epoch": 2.353021045485404,
+      "grad_norm": 0.3395691215991974,
+      "learning_rate": 1.138351747698711e-05,
+      "loss": 0.18421865463256837,
+      "mean_token_accuracy": 0.9601799823343754,
+      "num_tokens": 33423228.0,
+      "step": 7800
+    },
+    {
+      "epoch": 2.353021045485404,
+      "eval_entropy": 0.2481674429895253,
+      "eval_loss": 0.547441840171814,
+      "eval_mean_token_accuracy": 0.9053705822937603,
+      "eval_num_tokens": 33423228.0,
+      "eval_runtime": 37.2459,
+      "eval_samples_per_second": 58.154,
+      "eval_steps_per_second": 7.276,
+      "step": 7800
+    },
+    {
+      "entropy": 0.1889958502538502,
+      "epoch": 2.3681074149505923,
+      "grad_norm": 0.4847002327442169,
+      "learning_rate": 1.0879161133981014e-05,
+      "loss": 0.17589187622070312,
+      "mean_token_accuracy": 0.9615845668315888,
+      "num_tokens": 33638871.0,
+      "step": 7850
+    },
+    {
+      "entropy": 0.18853790896944703,
+      "epoch": 2.3831937844157802,
+      "grad_norm": 0.5678646564483643,
+      "learning_rate": 1.0384865526317861e-05,
+      "loss": 0.1777634620666504,
+      "mean_token_accuracy": 0.961886543482542,
+      "num_tokens": 33852577.0,
+      "step": 7900
+    },
+    {
+      "entropy": 0.19151988808065654,
+      "epoch": 2.3982801538809686,
+      "grad_norm": 0.4446047246456146,
+      "learning_rate": 9.900757772368175e-06,
+      "loss": 0.17955810546875,
+      "mean_token_accuracy": 0.9614880438148975,
+      "num_tokens": 34065599.0,
+      "step": 7950
+    },
+    {
+      "entropy": 0.19300217579118908,
+      "epoch": 2.4133665233461565,
+      "grad_norm": 0.5685653686523438,
+      "learning_rate": 9.426962370484533e-06,
+      "loss": 0.18160755157470704,
+      "mean_token_accuracy": 0.9607279069721699,
+      "num_tokens": 34280453.0,
+      "step": 8000
+    },
+    {
+      "epoch": 2.4133665233461565,
+      "eval_entropy": 0.24335849892719205,
+      "eval_loss": 0.5554639101028442,
+      "eval_mean_token_accuracy": 0.905630389702716,
+      "eval_num_tokens": 34280453.0,
+      "eval_runtime": 37.2491,
+      "eval_samples_per_second": 58.149,
+      "eval_steps_per_second": 7.275,
+      "step": 8000
+    },
+    {
+      "entropy": 0.18967346624471249,
+      "epoch": 2.428452892811345,
+      "grad_norm": 0.5121133923530579,
+      "learning_rate": 8.963601166984225e-06,
+      "loss": 0.1821819305419922,
+      "mean_token_accuracy": 0.9610455152392388,
+      "num_tokens": 34494654.0,
+      "step": 8050
+    },
+    {
+      "entropy": 0.18874711729586124,
+      "epoch": 2.4435392622765333,
+      "grad_norm": 0.6657042503356934,
+      "learning_rate": 8.510793324813937e-06,
+      "loss": 0.17630435943603515,
+      "mean_token_accuracy": 0.9615484464168549,
+      "num_tokens": 34704277.0,
+      "step": 8100
+    },
+    {
+      "entropy": 0.19604809990152716,
+      "epoch": 2.458625631741721,
+      "grad_norm": 0.6367285847663879,
+      "learning_rate": 8.068655292904559e-06,
+      "loss": 0.1849921989440918,
+      "mean_token_accuracy": 0.9597983421385288,
+      "num_tokens": 34918987.0,
+      "step": 8150
+    },
+    {
+      "entropy": 0.19020996399223805,
+      "epoch": 2.4737120012069096,
+      "grad_norm": 0.40266647934913635,
+      "learning_rate": 7.63730077622392e-06,
+      "loss": 0.18030958175659179,
+      "mean_token_accuracy": 0.9618312911689282,
+      "num_tokens": 35128250.0,
+      "step": 8200
+    },
+    {
+      "epoch": 2.4737120012069096,
+      "eval_entropy": 0.24456536362971767,
+      "eval_loss": 0.5537819266319275,
+      "eval_mean_token_accuracy": 0.905363669254683,
+      "eval_num_tokens": 35128250.0,
+      "eval_runtime": 37.3206,
+      "eval_samples_per_second": 58.038,
+      "eval_steps_per_second": 7.261,
+      "step": 8200
+    },
+    {
+      "entropy": 0.1947929921839386,
+      "epoch": 2.4887983706720975,
+      "grad_norm": 0.5730997323989868,
+      "learning_rate": 7.216840706535133e-06,
+      "loss": 0.1838933753967285,
+      "mean_token_accuracy": 0.9602823905646801,
+      "num_tokens": 35344354.0,
+      "step": 8250
+    },
+    {
+      "entropy": 0.1937092263624072,
+      "epoch": 2.503884740137286,
+      "grad_norm": 0.48286330699920654,
+      "learning_rate": 6.807383213868318e-06,
+      "loss": 0.1827998924255371,
+      "mean_token_accuracy": 0.9602498579025268,
+      "num_tokens": 35559731.0,
+      "step": 8300
+    },
+    {
+      "entropy": 0.18829923996701836,
+      "epoch": 2.5189711096024743,
+      "grad_norm": 0.5280757546424866,
+      "learning_rate": 6.4090335987126e-06,
+      "loss": 0.17718358993530273,
+      "mean_token_accuracy": 0.9615453231334686,
+      "num_tokens": 35772707.0,
+      "step": 8350
+    },
+    {
+      "entropy": 0.19396818530745805,
+      "epoch": 2.534057479067662,
+      "grad_norm": 0.4934554100036621,
+      "learning_rate": 6.021894304935943e-06,
+      "loss": 0.18351167678833008,
+      "mean_token_accuracy": 0.9603780242800712,
+      "num_tokens": 35987178.0,
+      "step": 8400
+    },
+    {
+      "epoch": 2.534057479067662,
+      "eval_entropy": 0.2449101031761328,
+      "eval_loss": 0.5524511337280273,
+      "eval_mean_token_accuracy": 0.9050956489415186,
+      "eval_num_tokens": 35987178.0,
+      "eval_runtime": 37.2719,
+      "eval_samples_per_second": 58.113,
+      "eval_steps_per_second": 7.271,
+      "step": 8400
+    },
+    {
+      "entropy": 0.18812250457704066,
+      "epoch": 2.5491438485328506,
+      "grad_norm": 0.5735852718353271,
+      "learning_rate": 5.646064893439623e-06,
+      "loss": 0.17783910751342774,
+      "mean_token_accuracy": 0.9615232966840267,
+      "num_tokens": 36200263.0,
+      "step": 8450
+    },
+    {
+      "entropy": 0.18926897779107094,
+      "epoch": 2.5642302179980385,
+      "grad_norm": 0.5568459033966064,
+      "learning_rate": 5.281642016554017e-06,
+      "loss": 0.1792352867126465,
+      "mean_token_accuracy": 0.9616803368926048,
+      "num_tokens": 36411968.0,
+      "step": 8500
+    },
+    {
+      "entropy": 0.19124026644974948,
+      "epoch": 2.579316587463227,
+      "grad_norm": 0.5221688151359558,
+      "learning_rate": 4.928719393182507e-06,
+      "loss": 0.17745084762573243,
+      "mean_token_accuracy": 0.9616039192676544,
+      "num_tokens": 36626784.0,
+      "step": 8550
+    },
+    {
+      "entropy": 0.19215355191379785,
+      "epoch": 2.5944029569284153,
+      "grad_norm": 0.36854663491249084,
+      "learning_rate": 4.5873877846996594e-06,
+      "loss": 0.18093048095703124,
+      "mean_token_accuracy": 0.9610362699627877,
+      "num_tokens": 36839614.0,
+      "step": 8600
+    },
+    {
+      "epoch": 2.5944029569284153,
+      "eval_entropy": 0.24483193833779585,
+      "eval_loss": 0.5536935925483704,
+      "eval_mean_token_accuracy": 0.9050766867465199,
+      "eval_num_tokens": 36839614.0,
+      "eval_runtime": 37.2832,
+      "eval_samples_per_second": 58.096,
+      "eval_steps_per_second": 7.269,
+      "step": 8600
+    },
+    {
+      "entropy": 0.19345266066491604,
+      "epoch": 2.6094893263936036,
+      "grad_norm": 0.36760082840919495,
+      "learning_rate": 4.257734971610106e-06,
+      "loss": 0.18040838241577148,
+      "mean_token_accuracy": 0.9611816507577896,
+      "num_tokens": 37053142.0,
+      "step": 8650
+    },
+    {
+      "entropy": 0.188013746580109,
+      "epoch": 2.6245756958587916,
+      "grad_norm": 0.5213715434074402,
+      "learning_rate": 3.93984573097394e-06,
+      "loss": 0.17426759719848633,
+      "mean_token_accuracy": 0.962059508562088,
+      "num_tokens": 37267777.0,
+      "step": 8700
+    },
+    {
+      "entropy": 0.1918721451703459,
+      "epoch": 2.6396620653239795,
+      "grad_norm": 0.5748514533042908,
+      "learning_rate": 3.633801814604554e-06,
+      "loss": 0.1796425437927246,
+      "mean_token_accuracy": 0.9613311864435673,
+      "num_tokens": 37485508.0,
+      "step": 8750
+    },
+    {
+      "entropy": 0.18769024461507797,
+      "epoch": 2.654748434789168,
+      "grad_norm": 0.37133195996284485,
+      "learning_rate": 3.339681928044436e-06,
+      "loss": 0.17722646713256837,
+      "mean_token_accuracy": 0.9621489056944847,
+      "num_tokens": 37694053.0,
+      "step": 8800
+    },
+    {
+      "epoch": 2.654748434789168,
+      "eval_entropy": 0.24254771412738574,
+      "eval_loss": 0.558121383190155,
+      "eval_mean_token_accuracy": 0.905118876497684,
+      "eval_num_tokens": 37694053.0,
+      "eval_runtime": 37.2089,
+      "eval_samples_per_second": 58.212,
+      "eval_steps_per_second": 7.283,
+      "step": 8800
+    },
+    {
+      "entropy": 0.18984992211684584,
+      "epoch": 2.6698348042543563,
+      "grad_norm": 0.43609800934791565,
+      "learning_rate": 3.057561710324447e-06,
+      "loss": 0.17845310211181642,
+      "mean_token_accuracy": 0.9613586132228374,
+      "num_tokens": 37906133.0,
+      "step": 8850
+    },
+    {
+      "entropy": 0.1950311391800642,
+      "epoch": 2.6849211737195446,
+      "grad_norm": 0.45412150025367737,
+      "learning_rate": 2.787513714511636e-06,
+      "loss": 0.18089279174804687,
+      "mean_token_accuracy": 0.9608308251202107,
+      "num_tokens": 38121697.0,
+      "step": 8900
+    },
+    {
+      "entropy": 0.18615621831268073,
+      "epoch": 2.7000075431847326,
+      "grad_norm": 0.6591213345527649,
+      "learning_rate": 2.52960738905072e-06,
+      "loss": 0.17349456787109374,
+      "mean_token_accuracy": 0.9623284694552422,
+      "num_tokens": 38339277.0,
+      "step": 8950
+    },
+    {
+      "entropy": 0.18828014567494392,
+      "epoch": 2.715093912649921,
+      "grad_norm": 0.47049465775489807,
+      "learning_rate": 2.2839090599040046e-06,
+      "loss": 0.17837800979614257,
+      "mean_token_accuracy": 0.9619372922182083,
+      "num_tokens": 38551183.0,
+      "step": 9000
+    },
+    {
+      "epoch": 2.715093912649921,
+      "eval_entropy": 0.24229726709452942,
+      "eval_loss": 0.5592358708381653,
+      "eval_mean_token_accuracy": 0.905019719882205,
+      "eval_num_tokens": 38551183.0,
+      "eval_runtime": 37.1765,
+      "eval_samples_per_second": 58.263,
+      "eval_steps_per_second": 7.29,
+      "step": 9000
+    },
+    {
+      "entropy": 0.18336616074666381,
+      "epoch": 2.730180282115109,
+      "grad_norm": 0.3691662549972534,
+      "learning_rate": 2.0504819134942734e-06,
+      "loss": 0.17197175979614257,
+      "mean_token_accuracy": 0.9629626546800136,
+      "num_tokens": 38763413.0,
+      "step": 9050
+    },
+    {
+      "entropy": 0.18445612547919155,
+      "epoch": 2.7452666515802973,
+      "grad_norm": 0.5340840816497803,
+      "learning_rate": 1.8293859804550506e-06,
+      "loss": 0.17212907791137697,
+      "mean_token_accuracy": 0.9631160937249661,
+      "num_tokens": 38979256.0,
+      "step": 9100
+    },
+    {
+      "entropy": 0.18713438665494322,
+      "epoch": 2.7603530210454856,
+      "grad_norm": 0.5069661736488342,
+      "learning_rate": 1.620678120192537e-06,
+      "loss": 0.1732018280029297,
+      "mean_token_accuracy": 0.9624787227809429,
+      "num_tokens": 39194699.0,
+      "step": 9150
+    },
+    {
+      "entropy": 0.19093556420877575,
+      "epoch": 2.7754393905106736,
+      "grad_norm": 0.3724497854709625,
+      "learning_rate": 1.424412006263004e-06,
+      "loss": 0.17719757080078125,
+      "mean_token_accuracy": 0.9616392150521278,
+      "num_tokens": 39405056.0,
+      "step": 9200
+    },
+    {
+      "epoch": 2.7754393905106736,
+      "eval_entropy": 0.2416494778216545,
+      "eval_loss": 0.5602814555168152,
+      "eval_mean_token_accuracy": 0.9050707438775094,
+      "eval_num_tokens": 39405056.0,
+      "eval_runtime": 37.0527,
+      "eval_samples_per_second": 58.457,
+      "eval_steps_per_second": 7.314,
+      "step": 9200
+    },
+    {
+      "entropy": 0.18624798154458402,
+      "epoch": 2.790525759975862,
+      "grad_norm": 0.5416818857192993,
+      "learning_rate": 1.24063811256952e-06,
+      "loss": 0.17536975860595702,
+      "mean_token_accuracy": 0.9625106258690357,
+      "num_tokens": 39620602.0,
+      "step": 9250
+    },
+    {
+      "entropy": 0.18864918448030948,
+      "epoch": 2.80561212944105,
+      "grad_norm": 0.7557387351989746,
+      "learning_rate": 1.0694037003815493e-06,
+      "loss": 0.1763162422180176,
+      "mean_token_accuracy": 0.9617964868247509,
+      "num_tokens": 39838171.0,
+      "step": 9300
+    },
+    {
+      "entropy": 0.19156487157568336,
+      "epoch": 2.8206984989062382,
+      "grad_norm": 0.5030518174171448,
+      "learning_rate": 9.107528061807425e-07,
+      "loss": 0.17927934646606444,
+      "mean_token_accuracy": 0.9610569317638874,
+      "num_tokens": 40055765.0,
+      "step": 9350
+    },
+    {
+      "entropy": 0.19062289187684656,
+      "epoch": 2.8357848683714266,
+      "grad_norm": 0.3872191905975342,
+      "learning_rate": 7.64726230336027e-07,
+      "loss": 0.1788728141784668,
+      "mean_token_accuracy": 0.9610781705379486,
+      "num_tokens": 40273169.0,
+      "step": 9400
+    },
+    {
+      "epoch": 2.8357848683714266,
+      "eval_entropy": 0.2424020753190526,
+      "eval_loss": 0.5598563551902771,
+      "eval_mean_token_accuracy": 0.905123386435843,
+      "eval_num_tokens": 40273169.0,
+      "eval_runtime": 37.2998,
+      "eval_samples_per_second": 58.07,
+      "eval_steps_per_second": 7.265,
+      "step": 9400
+    },
+    {
+      "entropy": 0.19168663132935762,
+      "epoch": 2.8508712378366146,
+      "grad_norm": 0.38919326663017273,
+      "learning_rate": 6.313615266109463e-07,
+      "loss": 0.17518898010253905,
+      "mean_token_accuracy": 0.9616159409284591,
+      "num_tokens": 40493623.0,
+      "step": 9450
+    },
+    {
+      "entropy": 0.1842949813697487,
+      "epoch": 2.865957607301803,
+      "grad_norm": 0.4381106197834015,
+      "learning_rate": 5.106929925059456e-07,
+      "loss": 0.17125751495361327,
+      "mean_token_accuracy": 0.9631977449357509,
+      "num_tokens": 40706058.0,
+      "step": 9500
+    },
+    {
+      "entropy": 0.18345711720176042,
+      "epoch": 2.881043976766991,
+      "grad_norm": 0.625280499458313,
+      "learning_rate": 4.027516604380388e-07,
+      "loss": 0.17233373641967772,
+      "mean_token_accuracy": 0.9627382136881352,
+      "num_tokens": 40919257.0,
+      "step": 9550
+    },
+    {
+      "entropy": 0.19293911349028348,
+      "epoch": 2.8961303462321792,
+      "grad_norm": 0.5119889974594116,
+      "learning_rate": 3.075652897602033e-07,
+      "loss": 0.18010311126708983,
+      "mean_token_accuracy": 0.960850502550602,
+      "num_tokens": 41135038.0,
+      "step": 9600
+    },
+    {
+      "epoch": 2.8961303462321792,
+      "eval_entropy": 0.242256640535659,
+      "eval_loss": 0.5602179169654846,
+      "eval_mean_token_accuracy": 0.9051620641317755,
+      "eval_num_tokens": 41135038.0,
+      "eval_runtime": 37.3372,
+      "eval_samples_per_second": 58.012,
+      "eval_steps_per_second": 7.258,
+      "step": 9600
+    },
+    {
+      "entropy": 0.1843437609449029,
+      "epoch": 2.9112167156973676,
+      "grad_norm": 0.5234854221343994,
+      "learning_rate": 2.2515835962245157e-07,
+      "loss": 0.16681966781616211,
+      "mean_token_accuracy": 0.9633826872706414,
+      "num_tokens": 41350531.0,
+      "step": 9650
+    },
+    {
+      "entropy": 0.18940109137445688,
+      "epoch": 2.9263030851625556,
+      "grad_norm": 0.5936132073402405,
+      "learning_rate": 1.5555206267655055e-07,
+      "loss": 0.17372846603393555,
+      "mean_token_accuracy": 0.9623673681914806,
+      "num_tokens": 41568124.0,
+      "step": 9700
+    },
+    {
+      "entropy": 0.18825361583381892,
+      "epoch": 2.941389454627744,
+      "grad_norm": 0.43561285734176636,
+      "learning_rate": 9.876429962585309e-08,
+      "loss": 0.17482643127441405,
+      "mean_token_accuracy": 0.962187256962061,
+      "num_tokens": 41784922.0,
+      "step": 9750
+    },
+    {
+      "entropy": 0.19123295756056904,
+      "epoch": 2.956475824092932,
+      "grad_norm": 0.45098307728767395,
+      "learning_rate": 5.4809674621797425e-08,
+      "loss": 0.1720089340209961,
+      "mean_token_accuracy": 0.9617229135334492,
+      "num_tokens": 42005066.0,
+      "step": 9800
+    },
+    {
+      "epoch": 2.956475824092932,
+      "eval_entropy": 0.2421166613917949,
+      "eval_loss": 0.5603178143501282,
+      "eval_mean_token_accuracy": 0.9051976802164338,
+      "eval_num_tokens": 42005066.0,
+      "eval_runtime": 37.2727,
+      "eval_samples_per_second": 58.112,
+      "eval_steps_per_second": 7.271,
+      "step": 9800
+    },
+    {
+      "entropy": 0.1918338906392455,
+      "epoch": 2.9715621935581202,
+      "grad_norm": 0.3770565092563629,
+      "learning_rate": 2.3699491508105687e-08,
+      "loss": 0.17970945358276366,
+      "mean_token_accuracy": 0.9611429516971112,
+      "num_tokens": 42216954.0,
+      "step": 9850
+    },
+    {
+      "entropy": 0.192145568234846,
+      "epoch": 2.9866485630233086,
+      "grad_norm": 0.36763280630111694,
+      "learning_rate": 5.441750913809385e-09,
+      "loss": 0.1800632095336914,
+      "mean_token_accuracy": 0.9608665798604489,
+      "num_tokens": 42435945.0,
+      "step": 9900
+    },
+    {
+      "epoch": 3.0,
+      "eval_entropy": 0.24219409354388494,
+      "eval_loss": 0.5602481365203857,
+      "eval_mean_token_accuracy": 0.9051936398573027,
+      "eval_num_tokens": 42622824.0,
+      "eval_runtime": 37.3187,
+      "eval_samples_per_second": 58.041,
+      "eval_steps_per_second": 7.262,
+      "step": 9945
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 9945,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 200,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.341878233255895e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-9945/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe063de6f1674be7d5535690dc263e5d0c28436ed4781417772a469f39ef21d4
+size 5713

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c7efa4a1e4882e95c953593dd50efe8d955e112437c4bd50e457698a2874896d
+oid sha256:247b247aa1c5fe1a1ba15ed8a5ce1f206b8ceb9ef5e91674b53713280396604f
 size 3441185608