Clean up: Remove redundant training checkpoint folder

last-checkpoint/README.md

@@ -1,209 +0,0 @@
----
-base_model: mistralai/Mistral-Nemo-Instruct-2407
-library_name: peft
-pipeline_tag: text-generation
-tags:
-- base_model:adapter:mistralai/Mistral-Nemo-Instruct-2407
-- lora
-- sft
-- transformers
-- trl
----
-
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
-### Framework versions
-
-- PEFT 0.18.1

last-checkpoint/adapter_config.json

@@ -1,46 +0,0 @@
-{
-  "alora_invocation_tokens": null,
-  "alpha_pattern": {},
-  "arrow_config": null,
-  "auto_mapping": null,
-  "base_model_name_or_path": "mistralai/Mistral-Nemo-Instruct-2407",
-  "bias": "none",
-  "corda_config": null,
-  "ensure_weight_tying": false,
-  "eva_config": null,
-  "exclude_modules": null,
-  "fan_in_fan_out": false,
-  "inference_mode": true,
-  "init_lora_weights": true,
-  "layer_replication": null,
-  "layers_pattern": null,
-  "layers_to_transform": null,
-  "loftq_config": {},
-  "lora_alpha": 32,
-  "lora_bias": false,
-  "lora_dropout": 0.05,
-  "megatron_config": null,
-  "megatron_core": "megatron.core",
-  "modules_to_save": null,
-  "peft_type": "LORA",
-  "peft_version": "0.18.1",
-  "qalora_group_size": 16,
-  "r": 16,
-  "rank_pattern": {},
-  "revision": null,
-  "target_modules": [
-    "k_proj",
-    "down_proj",
-    "o_proj",
-    "gate_proj",
-    "up_proj",
-    "q_proj",
-    "v_proj"
-  ],
-  "target_parameters": null,
-  "task_type": "CAUSAL_LM",
-  "trainable_token_indices": null,
-  "use_dora": false,
-  "use_qalora": false,
-  "use_rslora": false
-}

last-checkpoint/adapter_model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c14b7d8a1648c56d9f25f88d48454e081b6bc178d61bd9f0aadd582257678003
-size 228140600

last-checkpoint/chat_template.jinja

@@ -1,87 +0,0 @@
-{%- if messages[0]["role"] == "system" %}
-    {%- set system_message = messages[0]["content"] %}
-    {%- set loop_messages = messages[1:] %}
-{%- else %}
-    {%- set loop_messages = messages %}
-{%- endif %}
-{%- if not tools is defined %}
-    {%- set tools = none %}
-{%- endif %}
-{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
-
-{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
-{%- set ns = namespace() %}
-{%- set ns.index = 0 %}
-{%- for message in loop_messages %}
-    {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
-        {%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
-            {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
-        {%- endif %}
-        {%- set ns.index = ns.index + 1 %}
-    {%- endif %}
-{%- endfor %}
-
-{{- bos_token }}
-{%- for message in loop_messages %}
-    {%- if message["role"] == "user" %}
-        {%- if tools is not none and (message == user_messages[-1]) %}
-            {{- "[AVAILABLE_TOOLS][" }}
-            {%- for tool in tools %}
-                {%- set tool = tool.function %}
-                {{- '{"type": "function", "function": {' }}
-                {%- for key, val in tool.items() if key != "return" %}
-                    {%- if val is string %}
-                        {{- '"' + key + '": "' + val + '"' }}
-                    {%- else %}
-                        {{- '"' + key + '": ' + val|tojson }}
-                    {%- endif %}
-                    {%- if not loop.last %}
-                        {{- ", " }}
-                    {%- endif %}
-                {%- endfor %}
-                {{- "}}" }}
-                {%- if not loop.last %}
-                    {{- ", " }}
-                {%- else %}
-                    {{- "]" }}
-                {%- endif %}
-            {%- endfor %}
-            {{- "[/AVAILABLE_TOOLS]" }}
-            {%- endif %}
-        {%- if loop.last and system_message is defined %}
-            {{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
-        {%- else %}
-            {{- "[INST]" + message["content"] + "[/INST]" }}
-        {%- endif %}
-    {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
-        {{- "[TOOL_CALLS][" }}
-        {%- for tool_call in message.tool_calls %}
-            {%- set out = tool_call.function|tojson %}
-            {{- out[:-1] }}
-            {%- if not tool_call.id is defined or tool_call.id|length != 9 %}
-                {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
-            {%- endif %}
-            {{- ', "id": "' + tool_call.id + '"}' }}
-            {%- if not loop.last %}
-                {{- ", " }}
-            {%- else %}
-                {{- "]" + eos_token }}
-            {%- endif %}
-        {%- endfor %}
-    {%- elif message["role"] == "assistant" %}
-        {{- message["content"] + eos_token}}
-    {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
-        {%- if message.content is defined and message.content.content is defined %}
-            {%- set content = message.content.content %}
-        {%- else %}
-            {%- set content = message.content %}
-        {%- endif %}
-        {{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
-        {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
-            {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
-        {%- endif %}
-        {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
-    {%- else %}
-        {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
-    {%- endif %}
-{%- endfor %}

last-checkpoint/optimizer.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ae997529af0fecd00cf9ea60649b8488f2fcad93e7d57149ed2055f7e443e81c
-size 117931203

last-checkpoint/rng_state.pth

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a806988fecdee5121c06d7240dec6e61421fb0008f39bed17de1e2ca05215f14
-size 14645

last-checkpoint/scaler.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:dfca50dfc66d4be0e8bab60e1bfd495197005d876487c7e37b847562cfa51471
-size 1383

last-checkpoint/scheduler.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7c8cc8d8f0165185e683fffb0aab5024d4cdc129dcf7f2bae80e3717e00f0c4e
-size 1465

last-checkpoint/special_tokens_map.json

@@ -1,24 +0,0 @@
-{
-  "bos_token": {
-    "content": "<s>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "eos_token": {
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": "<unk>",
-  "unk_token": {
-    "content": "<unk>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  }
-}

last-checkpoint/tokenizer.json

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b0240ce510f08e6c2041724e9043e33be9d251d1e4a4d94eb68cd47b954b61d2
-size 17078292

last-checkpoint/trainer_state.json

@@ -1,2284 +0,0 @@
-{
-  "best_global_step": 438,
-  "best_metric": 1.2615772485733032,
-  "best_model_checkpoint": "./adapter-phase2/checkpoint-438",
-  "epoch": 5.0,
-  "eval_steps": 500,
-  "global_step": 2190,
-  "is_hyper_param_search": false,
-  "is_local_process_zero": true,
-  "is_world_process_zero": true,
-  "log_history": [
-    {
-      "entropy": 0.9881319830194115,
-      "epoch": 0.022857142857142857,
-      "grad_norm": 1.5965849161148071,
-      "learning_rate": 9.958904109589041e-06,
-      "loss": 1.6407,
-      "mean_token_accuracy": 0.6662811586633325,
-      "num_tokens": 15996.0,
-      "step": 10
-    },
-    {
-      "entropy": 1.2699928235262632,
-      "epoch": 0.045714285714285714,
-      "grad_norm": 1.3544626235961914,
-      "learning_rate": 9.913242009132421e-06,
-      "loss": 1.5705,
-      "mean_token_accuracy": 0.6635363385081291,
-      "num_tokens": 27684.0,
-      "step": 20
-    },
-    {
-      "entropy": 1.6406203664839267,
-      "epoch": 0.06857142857142857,
-      "grad_norm": 1.615886926651001,
-      "learning_rate": 9.8675799086758e-06,
-      "loss": 1.6826,
-      "mean_token_accuracy": 0.6459833424538374,
-      "num_tokens": 36498.0,
-      "step": 30
-    },
-    {
-      "entropy": 1.8021161697804928,
-      "epoch": 0.09142857142857143,
-      "grad_norm": 1.3791861534118652,
-      "learning_rate": 9.821917808219178e-06,
-      "loss": 1.6402,
-      "mean_token_accuracy": 0.6510010983794927,
-      "num_tokens": 43095.0,
-      "step": 40
-    },
-    {
-      "entropy": 1.8170176550745964,
-      "epoch": 0.11428571428571428,
-      "grad_norm": 1.978265643119812,
-      "learning_rate": 9.776255707762557e-06,
-      "loss": 1.595,
-      "mean_token_accuracy": 0.6443639542907477,
-      "num_tokens": 47981.0,
-      "step": 50
-    },
-    {
-      "entropy": 1.1217214532196522,
-      "epoch": 0.13714285714285715,
-      "grad_norm": 0.9291492700576782,
-      "learning_rate": 9.730593607305937e-06,
-      "loss": 1.1238,
-      "mean_token_accuracy": 0.748357442393899,
-      "num_tokens": 64374.0,
-      "step": 60
-    },
-    {
-      "entropy": 1.2191850792616605,
-      "epoch": 0.16,
-      "grad_norm": 0.946483314037323,
-      "learning_rate": 9.684931506849316e-06,
-      "loss": 1.1395,
-      "mean_token_accuracy": 0.7462461993098259,
-      "num_tokens": 76109.0,
-      "step": 70
-    },
-    {
-      "entropy": 1.3447685081511735,
-      "epoch": 0.18285714285714286,
-      "grad_norm": 1.2081618309020996,
-      "learning_rate": 9.639269406392696e-06,
-      "loss": 1.2818,
-      "mean_token_accuracy": 0.7141567166894675,
-      "num_tokens": 84921.0,
-      "step": 80
-    },
-    {
-      "entropy": 1.5163870930671692,
-      "epoch": 0.2057142857142857,
-      "grad_norm": 1.1962831020355225,
-      "learning_rate": 9.593607305936073e-06,
-      "loss": 1.4127,
-      "mean_token_accuracy": 0.6898491781204938,
-      "num_tokens": 91475.0,
-      "step": 90
-    },
-    {
-      "entropy": 1.494850355386734,
-      "epoch": 0.22857142857142856,
-      "grad_norm": 1.8051788806915283,
-      "learning_rate": 9.547945205479453e-06,
-      "loss": 1.3539,
-      "mean_token_accuracy": 0.7036820162087679,
-      "num_tokens": 96433.0,
-      "step": 100
-    },
-    {
-      "entropy": 0.9693804103881121,
-      "epoch": 0.25142857142857145,
-      "grad_norm": 0.9580877423286438,
-      "learning_rate": 9.502283105022831e-06,
-      "loss": 1.0186,
-      "mean_token_accuracy": 0.7684306014329195,
-      "num_tokens": 112394.0,
-      "step": 110
-    },
-    {
-      "entropy": 1.090338883176446,
-      "epoch": 0.2742857142857143,
-      "grad_norm": 0.8812002539634705,
-      "learning_rate": 9.456621004566212e-06,
-      "loss": 1.0132,
-      "mean_token_accuracy": 0.7664786655455827,
-      "num_tokens": 123947.0,
-      "step": 120
-    },
-    {
-      "entropy": 1.284313677251339,
-      "epoch": 0.29714285714285715,
-      "grad_norm": 1.1698031425476074,
-      "learning_rate": 9.41095890410959e-06,
-      "loss": 1.2496,
-      "mean_token_accuracy": 0.7140494808554649,
-      "num_tokens": 132309.0,
-      "step": 130
-    },
-    {
-      "entropy": 1.3905419509857893,
-      "epoch": 0.32,
-      "grad_norm": 1.206275463104248,
-      "learning_rate": 9.365296803652969e-06,
-      "loss": 1.2838,
-      "mean_token_accuracy": 0.7115083243697882,
-      "num_tokens": 138781.0,
-      "step": 140
-    },
-    {
-      "entropy": 1.3588767245411872,
-      "epoch": 0.34285714285714286,
-      "grad_norm": 1.989784598350525,
-      "learning_rate": 9.319634703196347e-06,
-      "loss": 1.1625,
-      "mean_token_accuracy": 0.7235500495880842,
-      "num_tokens": 143734.0,
-      "step": 150
-    },
-    {
-      "entropy": 0.8994554949924349,
-      "epoch": 0.3657142857142857,
-      "grad_norm": 1.00913405418396,
-      "learning_rate": 9.273972602739727e-06,
-      "loss": 0.9263,
-      "mean_token_accuracy": 0.7811420723795891,
-      "num_tokens": 159877.0,
-      "step": 160
-    },
-    {
-      "entropy": 1.0330400079488755,
-      "epoch": 0.38857142857142857,
-      "grad_norm": 1.0430705547332764,
-      "learning_rate": 9.228310502283106e-06,
-      "loss": 0.9874,
-      "mean_token_accuracy": 0.7636023428291082,
-      "num_tokens": 171343.0,
-      "step": 170
-    },
-    {
-      "entropy": 1.1962346132844686,
-      "epoch": 0.4114285714285714,
-      "grad_norm": 1.3926544189453125,
-      "learning_rate": 9.182648401826484e-06,
-      "loss": 1.1651,
-      "mean_token_accuracy": 0.7276976224035024,
-      "num_tokens": 179829.0,
-      "step": 180
-    },
-    {
-      "entropy": 1.2758624110370875,
-      "epoch": 0.4342857142857143,
-      "grad_norm": 1.4376598596572876,
-      "learning_rate": 9.136986301369863e-06,
-      "loss": 1.1398,
-      "mean_token_accuracy": 0.7325437176972628,
-      "num_tokens": 186366.0,
-      "step": 190
-    },
-    {
-      "entropy": 1.2853564880788326,
-      "epoch": 0.45714285714285713,
-      "grad_norm": 2.2004787921905518,
-      "learning_rate": 9.091324200913243e-06,
-      "loss": 1.1638,
-      "mean_token_accuracy": 0.7276943679898977,
-      "num_tokens": 191365.0,
-      "step": 200
-    },
-    {
-      "entropy": 0.866673743724823,
-      "epoch": 0.48,
-      "grad_norm": 1.0414421558380127,
-      "learning_rate": 9.045662100456622e-06,
-      "loss": 0.8423,
-      "mean_token_accuracy": 0.7982745975255966,
-      "num_tokens": 207506.0,
-      "step": 210
-    },
-    {
-      "entropy": 0.947331802919507,
-      "epoch": 0.5028571428571429,
-      "grad_norm": 1.0546936988830566,
-      "learning_rate": 9e-06,
-      "loss": 0.9144,
-      "mean_token_accuracy": 0.7818248048424721,
-      "num_tokens": 219079.0,
-      "step": 220
-    },
-    {
-      "entropy": 1.1076377972960472,
-      "epoch": 0.5257142857142857,
-      "grad_norm": 1.488133430480957,
-      "learning_rate": 8.954337899543379e-06,
-      "loss": 1.0699,
-      "mean_token_accuracy": 0.7396229557693005,
-      "num_tokens": 227798.0,
-      "step": 230
-    },
-    {
-      "entropy": 1.2146507527679204,
-      "epoch": 0.5485714285714286,
-      "grad_norm": 1.393114686012268,
-      "learning_rate": 8.908675799086759e-06,
-      "loss": 1.1319,
-      "mean_token_accuracy": 0.7367029923945665,
-      "num_tokens": 234590.0,
-      "step": 240
-    },
-    {
-      "entropy": 1.2185490131378174,
-      "epoch": 0.5714285714285714,
-      "grad_norm": 2.391268730163574,
-      "learning_rate": 8.863013698630137e-06,
-      "loss": 1.0764,
-      "mean_token_accuracy": 0.7414231035858393,
-      "num_tokens": 239719.0,
-      "step": 250
-    },
-    {
-      "entropy": 0.8054604699835182,
-      "epoch": 0.5942857142857143,
-      "grad_norm": 0.9834737181663513,
-      "learning_rate": 8.817351598173518e-06,
-      "loss": 0.8053,
-      "mean_token_accuracy": 0.7998832739889622,
-      "num_tokens": 256472.0,
-      "step": 260
-    },
-    {
-      "entropy": 0.9120684009045362,
-      "epoch": 0.6171428571428571,
-      "grad_norm": 1.0815778970718384,
-      "learning_rate": 8.771689497716896e-06,
-      "loss": 0.8559,
-      "mean_token_accuracy": 0.7897930487990379,
-      "num_tokens": 268303.0,
-      "step": 270
-    },
-    {
-      "entropy": 1.0642446961253882,
-      "epoch": 0.64,
-      "grad_norm": 1.4160140752792358,
-      "learning_rate": 8.726027397260275e-06,
-      "loss": 1.0196,
-      "mean_token_accuracy": 0.7551121093332768,
-      "num_tokens": 277099.0,
-      "step": 280
-    },
-    {
-      "entropy": 1.1535940799862145,
-      "epoch": 0.6628571428571428,
-      "grad_norm": 1.6847327947616577,
-      "learning_rate": 8.680365296803653e-06,
-      "loss": 1.0552,
-      "mean_token_accuracy": 0.7411838915199042,
-      "num_tokens": 283815.0,
-      "step": 290
-    },
-    {
-      "entropy": 1.1651097811758517,
-      "epoch": 0.6857142857142857,
-      "grad_norm": 2.285900592803955,
-      "learning_rate": 8.634703196347033e-06,
-      "loss": 1.0267,
-      "mean_token_accuracy": 0.7420264776796103,
-      "num_tokens": 288831.0,
-      "step": 300
-    },
-    {
-      "entropy": 0.7967551020905376,
-      "epoch": 0.7085714285714285,
-      "grad_norm": 1.1952354907989502,
-      "learning_rate": 8.589041095890412e-06,
-      "loss": 0.8013,
-      "mean_token_accuracy": 0.803380336984992,
-      "num_tokens": 305255.0,
-      "step": 310
-    },
-    {
-      "entropy": 0.9132619671523571,
-      "epoch": 0.7314285714285714,
-      "grad_norm": 1.2885327339172363,
-      "learning_rate": 8.54337899543379e-06,
-      "loss": 0.8516,
-      "mean_token_accuracy": 0.7869276314973831,
-      "num_tokens": 316828.0,
-      "step": 320
-    },
-    {
-      "entropy": 1.020271310582757,
-      "epoch": 0.7542857142857143,
-      "grad_norm": 1.46247398853302,
-      "learning_rate": 8.497716894977169e-06,
-      "loss": 0.9943,
-      "mean_token_accuracy": 0.7584239929914475,
-      "num_tokens": 325557.0,
-      "step": 330
-    },
-    {
-      "entropy": 1.1350981347262858,
-      "epoch": 0.7771428571428571,
-      "grad_norm": 1.8628100156784058,
-      "learning_rate": 8.45205479452055e-06,
-      "loss": 1.0646,
-      "mean_token_accuracy": 0.7359312813729048,
-      "num_tokens": 332224.0,
-      "step": 340
-    },
-    {
-      "entropy": 1.1172638952732086,
-      "epoch": 0.8,
-      "grad_norm": 2.362889289855957,
-      "learning_rate": 8.406392694063928e-06,
-      "loss": 0.974,
-      "mean_token_accuracy": 0.7591060597449542,
-      "num_tokens": 337193.0,
-      "step": 350
-    },
-    {
-      "entropy": 0.7974750218912959,
-      "epoch": 0.8228571428571428,
-      "grad_norm": 1.1142443418502808,
-      "learning_rate": 8.360730593607306e-06,
-      "loss": 0.7776,
-      "mean_token_accuracy": 0.8041313651949167,
-      "num_tokens": 353595.0,
-      "step": 360
-    },
-    {
-      "entropy": 0.8807439863681793,
-      "epoch": 0.8457142857142858,
-      "grad_norm": 1.4073342084884644,
-      "learning_rate": 8.315068493150685e-06,
-      "loss": 0.8367,
-      "mean_token_accuracy": 0.7917833589017391,
-      "num_tokens": 365258.0,
-      "step": 370
-    },
-    {
-      "entropy": 1.0210479736328124,
-      "epoch": 0.8685714285714285,
-      "grad_norm": 1.506361484527588,
-      "learning_rate": 8.269406392694065e-06,
-      "loss": 0.9998,
-      "mean_token_accuracy": 0.753864735737443,
-      "num_tokens": 373694.0,
-      "step": 380
-    },
-    {
-      "entropy": 1.0973432060331105,
-      "epoch": 0.8914285714285715,
-      "grad_norm": 1.8420379161834717,
-      "learning_rate": 8.223744292237444e-06,
-      "loss": 1.0189,
-      "mean_token_accuracy": 0.7553424458950758,
-      "num_tokens": 380272.0,
-      "step": 390
-    },
-    {
-      "entropy": 1.1227020058780908,
-      "epoch": 0.9142857142857143,
-      "grad_norm": 2.7073919773101807,
-      "learning_rate": 8.178082191780822e-06,
-      "loss": 0.939,
-      "mean_token_accuracy": 0.755080708488822,
-      "num_tokens": 385214.0,
-      "step": 400
-    },
-    {
-      "entropy": 0.7808640262112021,
-      "epoch": 0.9371428571428572,
-      "grad_norm": 1.3160523176193237,
-      "learning_rate": 8.1324200913242e-06,
-      "loss": 0.7749,
-      "mean_token_accuracy": 0.8022231217473745,
-      "num_tokens": 400292.0,
-      "step": 410
-    },
-    {
-      "entropy": 0.9076132765039802,
-      "epoch": 0.96,
-      "grad_norm": 1.5909351110458374,
-      "learning_rate": 8.08675799086758e-06,
-      "loss": 0.8487,
-      "mean_token_accuracy": 0.78284947052598,
-      "num_tokens": 410576.0,
-      "step": 420
-    },
-    {
-      "entropy": 1.03597099930048,
-      "epoch": 0.9828571428571429,
-      "grad_norm": 1.9207741022109985,
-      "learning_rate": 8.04109589041096e-06,
-      "loss": 0.9651,
-      "mean_token_accuracy": 0.7578862871974706,
-      "num_tokens": 417570.0,
-      "step": 430
-    },
-    {
-      "epoch": 1.0,
-      "eval_accuracy": 0.008388412892696859,
-      "eval_entropy": 0.9678071262063207,
-      "eval_loss": 1.2615772485733032,
-      "eval_mean_token_accuracy": 0.7336987892172971,
-      "eval_num_tokens": 421194.0,
-      "eval_runtime": 298.9526,
-      "eval_samples_per_second": 3.459,
-      "eval_steps_per_second": 0.866,
-      "step": 438
-    },
-    {
-      "entropy": 0.9707203358411789,
-      "epoch": 1.0045714285714287,
-      "grad_norm": 1.046062707901001,
-      "learning_rate": 7.99543378995434e-06,
-      "loss": 0.8608,
-      "mean_token_accuracy": 0.7782945515293824,
-      "num_tokens": 425275.0,
-      "step": 440
-    },
-    {
-      "entropy": 0.7523622503504157,
-      "epoch": 1.0274285714285714,
-      "grad_norm": 1.2251920700073242,
-      "learning_rate": 7.949771689497718e-06,
-      "loss": 0.7473,
-      "mean_token_accuracy": 0.8092356324195862,
-      "num_tokens": 439850.0,
-      "step": 450
-    },
-    {
-      "entropy": 0.84888547193259,
-      "epoch": 1.0502857142857143,
-      "grad_norm": 1.325343370437622,
-      "learning_rate": 7.904109589041097e-06,
-      "loss": 0.7907,
-      "mean_token_accuracy": 0.7977306388318539,
-      "num_tokens": 451136.0,
-      "step": 460
-    },
-    {
-      "entropy": 0.9735806178301573,
-      "epoch": 1.0731428571428572,
-      "grad_norm": 1.6439032554626465,
-      "learning_rate": 7.858447488584475e-06,
-      "loss": 0.9513,
-      "mean_token_accuracy": 0.7617030199617147,
-      "num_tokens": 459486.0,
-      "step": 470
-    },
-    {
-      "entropy": 1.0277788739651441,
-      "epoch": 1.096,
-      "grad_norm": 1.8182581663131714,
-      "learning_rate": 7.812785388127855e-06,
-      "loss": 0.9526,
-      "mean_token_accuracy": 0.7630892738699913,
-      "num_tokens": 465975.0,
-      "step": 480
-    },
-    {
-      "entropy": 0.943339848332107,
-      "epoch": 1.1188571428571428,
-      "grad_norm": 1.1697943210601807,
-      "learning_rate": 7.767123287671234e-06,
-      "loss": 0.851,
-      "mean_token_accuracy": 0.7827658370137215,
-      "num_tokens": 473929.0,
-      "step": 490
-    },
-    {
-      "entropy": 0.7733788685873151,
-      "epoch": 1.1417142857142857,
-      "grad_norm": 1.2411632537841797,
-      "learning_rate": 7.721461187214612e-06,
-      "loss": 0.7691,
-      "mean_token_accuracy": 0.8111804500222206,
-      "num_tokens": 488306.0,
-      "step": 500
-    },
-    {
-      "entropy": 0.824394048191607,
-      "epoch": 1.1645714285714286,
-      "grad_norm": 1.3971821069717407,
-      "learning_rate": 7.675799086757991e-06,
-      "loss": 0.7429,
-      "mean_token_accuracy": 0.8088308341801167,
-      "num_tokens": 499208.0,
-      "step": 510
-    },
-    {
-      "entropy": 0.9718045836314559,
-      "epoch": 1.1874285714285715,
-      "grad_norm": 1.7808269262313843,
-      "learning_rate": 7.630136986301371e-06,
-      "loss": 0.9365,
-      "mean_token_accuracy": 0.762304800376296,
-      "num_tokens": 507299.0,
-      "step": 520
-    },
-    {
-      "entropy": 0.9984221205115318,
-      "epoch": 1.2102857142857144,
-      "grad_norm": 2.0445668697357178,
-      "learning_rate": 7.58447488584475e-06,
-      "loss": 0.9299,
-      "mean_token_accuracy": 0.7667763099074364,
-      "num_tokens": 513558.0,
-      "step": 530
-    },
-    {
-      "entropy": 0.9143548993393779,
-      "epoch": 1.233142857142857,
-      "grad_norm": 1.4540475606918335,
-      "learning_rate": 7.538812785388129e-06,
-      "loss": 0.7977,
-      "mean_token_accuracy": 0.7940818261355161,
-      "num_tokens": 521477.0,
-      "step": 540
-    },
-    {
-      "entropy": 0.7225218357518315,
-      "epoch": 1.256,
-      "grad_norm": 1.4530831575393677,
-      "learning_rate": 7.4931506849315075e-06,
-      "loss": 0.7282,
-      "mean_token_accuracy": 0.8143456902354955,
-      "num_tokens": 536156.0,
-      "step": 550
-    },
-    {
-      "entropy": 0.826616644486785,
-      "epoch": 1.278857142857143,
-      "grad_norm": 1.307198166847229,
-      "learning_rate": 7.447488584474887e-06,
-      "loss": 0.7509,
-      "mean_token_accuracy": 0.8082947298884392,
-      "num_tokens": 547250.0,
-      "step": 560
-    },
-    {
-      "entropy": 0.9597857438027859,
-      "epoch": 1.3017142857142856,
-      "grad_norm": 2.1991994380950928,
-      "learning_rate": 7.401826484018265e-06,
-      "loss": 0.9294,
-      "mean_token_accuracy": 0.7654231000691653,
-      "num_tokens": 555523.0,
-      "step": 570
-    },
-    {
-      "entropy": 1.0113731533288957,
-      "epoch": 1.3245714285714285,
-      "grad_norm": 2.2134881019592285,
-      "learning_rate": 7.356164383561645e-06,
-      "loss": 0.9149,
-      "mean_token_accuracy": 0.7716960549354553,
-      "num_tokens": 561790.0,
-      "step": 580
-    },
-    {
-      "entropy": 0.89712286721915,
-      "epoch": 1.3474285714285714,
-      "grad_norm": 1.2084845304489136,
-      "learning_rate": 7.310502283105023e-06,
-      "loss": 0.7891,
-      "mean_token_accuracy": 0.7911178763955832,
-      "num_tokens": 569844.0,
-      "step": 590
-    },
-    {
-      "entropy": 0.7331022916361689,
-      "epoch": 1.3702857142857143,
-      "grad_norm": 1.3023542165756226,
-      "learning_rate": 7.269406392694065e-06,
-      "loss": 0.7457,
-      "mean_token_accuracy": 0.8113049529492855,
-      "num_tokens": 584459.0,
-      "step": 600
-    },
-    {
-      "entropy": 0.7879349924623966,
-      "epoch": 1.3931428571428572,
-      "grad_norm": 1.555379867553711,
-      "learning_rate": 7.223744292237444e-06,
-      "loss": 0.7306,
-      "mean_token_accuracy": 0.8145640216767788,
-      "num_tokens": 595804.0,
-      "step": 610
-    },
-    {
-      "entropy": 0.9201723251491785,
-      "epoch": 1.416,
-      "grad_norm": 2.0131261348724365,
-      "learning_rate": 7.178082191780823e-06,
-      "loss": 0.881,
-      "mean_token_accuracy": 0.7761596899479628,
-      "num_tokens": 604098.0,
-      "step": 620
-    },
-    {
-      "entropy": 1.0043058268725873,
-      "epoch": 1.4388571428571428,
-      "grad_norm": 1.952837586402893,
-      "learning_rate": 7.132420091324202e-06,
-      "loss": 0.9229,
-      "mean_token_accuracy": 0.7723097205162048,
-      "num_tokens": 610481.0,
-      "step": 630
-    },
-    {
-      "entropy": 0.8940085913985968,
-      "epoch": 1.4617142857142857,
-      "grad_norm": 1.2801399230957031,
-      "learning_rate": 7.086757990867581e-06,
-      "loss": 0.8006,
-      "mean_token_accuracy": 0.7930479496717453,
-      "num_tokens": 618699.0,
-      "step": 640
-    },
-    {
-      "entropy": 0.6966889450326562,
-      "epoch": 1.4845714285714287,
-      "grad_norm": 1.557562232017517,
-      "learning_rate": 7.0410958904109596e-06,
-      "loss": 0.665,
-      "mean_token_accuracy": 0.8264754865318537,
-      "num_tokens": 632856.0,
-      "step": 650
-    },
-    {
-      "entropy": 0.8100471086800098,
-      "epoch": 1.5074285714285716,
-      "grad_norm": 1.7616751194000244,
-      "learning_rate": 6.995433789954339e-06,
-      "loss": 0.7669,
-      "mean_token_accuracy": 0.8096333492547274,
-      "num_tokens": 643712.0,
-      "step": 660
-    },
-    {
-      "entropy": 0.9476521443575621,
-      "epoch": 1.5302857142857142,
-      "grad_norm": 1.97320556640625,
-      "learning_rate": 6.9497716894977175e-06,
-      "loss": 0.8769,
-      "mean_token_accuracy": 0.7822451706975698,
-      "num_tokens": 651732.0,
-      "step": 670
-    },
-    {
-      "entropy": 0.9541807420551777,
-      "epoch": 1.5531428571428572,
-      "grad_norm": 2.2813711166381836,
-      "learning_rate": 6.904109589041097e-06,
-      "loss": 0.8731,
-      "mean_token_accuracy": 0.7764547783881426,
-      "num_tokens": 658104.0,
-      "step": 680
-    },
-    {
-      "entropy": 0.8891686601564288,
-      "epoch": 1.576,
-      "grad_norm": 1.2347137928009033,
-      "learning_rate": 6.858447488584475e-06,
-      "loss": 0.8099,
-      "mean_token_accuracy": 0.795854776352644,
-      "num_tokens": 666681.0,
-      "step": 690
-    },
-    {
-      "entropy": 0.7062053712084889,
-      "epoch": 1.5988571428571428,
-      "grad_norm": 1.505817174911499,
-      "learning_rate": 6.812785388127855e-06,
-      "loss": 0.6689,
-      "mean_token_accuracy": 0.8225430808961391,
-      "num_tokens": 681161.0,
-      "step": 700
-    },
-    {
-      "entropy": 0.7627910353243351,
-      "epoch": 1.6217142857142857,
-      "grad_norm": 1.7354750633239746,
-      "learning_rate": 6.767123287671233e-06,
-      "loss": 0.7217,
-      "mean_token_accuracy": 0.8088484812527895,
-      "num_tokens": 692262.0,
-      "step": 710
-    },
-    {
-      "entropy": 0.9181301448494196,
-      "epoch": 1.6445714285714286,
-      "grad_norm": 1.9427331686019897,
-      "learning_rate": 6.721461187214613e-06,
-      "loss": 0.8664,
-      "mean_token_accuracy": 0.7764203164726495,
-      "num_tokens": 700252.0,
-      "step": 720
-    },
-    {
-      "entropy": 0.970825233310461,
-      "epoch": 1.6674285714285715,
-      "grad_norm": 2.231489419937134,
-      "learning_rate": 6.675799086757991e-06,
-      "loss": 0.8727,
-      "mean_token_accuracy": 0.77991351634264,
-      "num_tokens": 706466.0,
-      "step": 730
-    },
-    {
-      "entropy": 0.8769128978252411,
-      "epoch": 1.6902857142857144,
-      "grad_norm": 1.3580577373504639,
-      "learning_rate": 6.630136986301371e-06,
-      "loss": 0.7826,
-      "mean_token_accuracy": 0.7997685220092535,
-      "num_tokens": 714701.0,
-      "step": 740
-    },
-    {
-      "entropy": 0.6923451218754053,
-      "epoch": 1.713142857142857,
-      "grad_norm": 1.4095361232757568,
-      "learning_rate": 6.584474885844749e-06,
-      "loss": 0.6984,
-      "mean_token_accuracy": 0.8204937841743231,
-      "num_tokens": 729622.0,
-      "step": 750
-    },
-    {
-      "entropy": 0.7426450593397022,
-      "epoch": 1.736,
-      "grad_norm": 1.5736570358276367,
-      "learning_rate": 6.538812785388129e-06,
-      "loss": 0.667,
-      "mean_token_accuracy": 0.8291565012186766,
-      "num_tokens": 740772.0,
-      "step": 760
-    },
-    {
-      "entropy": 0.910079357214272,
-      "epoch": 1.758857142857143,
-      "grad_norm": 2.1047656536102295,
-      "learning_rate": 6.493150684931508e-06,
-      "loss": 0.875,
-      "mean_token_accuracy": 0.7781037461012602,
-      "num_tokens": 748857.0,
-      "step": 770
-    },
-    {
-      "entropy": 0.9749910116195679,
-      "epoch": 1.7817142857142856,
-      "grad_norm": 2.2609705924987793,
-      "learning_rate": 6.447488584474887e-06,
-      "loss": 0.9058,
-      "mean_token_accuracy": 0.7749961122870446,
-      "num_tokens": 755273.0,
-      "step": 780
-    },
-    {
-      "entropy": 0.8688624935224653,
-      "epoch": 1.8045714285714287,
-      "grad_norm": 2.156954765319824,
-      "learning_rate": 6.401826484018266e-06,
-      "loss": 0.7568,
-      "mean_token_accuracy": 0.8001648161560297,
-      "num_tokens": 763404.0,
-      "step": 790
-    },
-    {
-      "entropy": 0.6553533479571343,
-      "epoch": 1.8274285714285714,
-      "grad_norm": 1.5286246538162231,
-      "learning_rate": 6.356164383561645e-06,
-      "loss": 0.6357,
-      "mean_token_accuracy": 0.8322514686733484,
-      "num_tokens": 777652.0,
-      "step": 800
-    },
-    {
-      "entropy": 0.7381465582177043,
-      "epoch": 1.8502857142857143,
-      "grad_norm": 1.889930248260498,
-      "learning_rate": 6.3105022831050235e-06,
-      "loss": 0.6995,
-      "mean_token_accuracy": 0.8194405883550644,
-      "num_tokens": 788541.0,
-      "step": 810
-    },
-    {
-      "entropy": 0.9207667458802462,
-      "epoch": 1.8731428571428572,
-      "grad_norm": 2.3677663803100586,
-      "learning_rate": 6.264840182648403e-06,
-      "loss": 0.876,
-      "mean_token_accuracy": 0.7714111492037773,
-      "num_tokens": 796574.0,
-      "step": 820
-    },
-    {
-      "entropy": 0.9494761880487204,
-      "epoch": 1.896,
-      "grad_norm": 2.424638032913208,
-      "learning_rate": 6.219178082191781e-06,
-      "loss": 0.8548,
-      "mean_token_accuracy": 0.7811690699309111,
-      "num_tokens": 802836.0,
-      "step": 830
-    },
-    {
-      "entropy": 0.8909835416823626,
-      "epoch": 1.9188571428571428,
-      "grad_norm": 1.3449039459228516,
-      "learning_rate": 6.173515981735161e-06,
-      "loss": 0.7825,
-      "mean_token_accuracy": 0.7954777158796787,
-      "num_tokens": 810726.0,
-      "step": 840
-    },
-    {
-      "entropy": 0.6921561988070607,
-      "epoch": 1.9417142857142857,
-      "grad_norm": 1.490689992904663,
-      "learning_rate": 6.127853881278539e-06,
-      "loss": 0.6554,
-      "mean_token_accuracy": 0.8262197155505419,
-      "num_tokens": 824145.0,
-      "step": 850
-    },
-    {
-      "entropy": 0.8137379666790366,
-      "epoch": 1.9645714285714284,
-      "grad_norm": 2.0120434761047363,
-      "learning_rate": 6.082191780821919e-06,
-      "loss": 0.8024,
-      "mean_token_accuracy": 0.7950452182441949,
-      "num_tokens": 833220.0,
-      "step": 860
-    },
-    {
-      "entropy": 0.9502449594438076,
-      "epoch": 1.9874285714285715,
-      "grad_norm": 2.679570198059082,
-      "learning_rate": 6.036529680365297e-06,
-      "loss": 0.8545,
-      "mean_token_accuracy": 0.781839894503355,
-      "num_tokens": 839758.0,
-      "step": 870
-    },
-    {
-      "epoch": 2.0,
-      "eval_accuracy": 0.00894328845369237,
-      "eval_entropy": 0.9275054344799528,
-      "eval_loss": 1.3659894466400146,
-      "eval_mean_token_accuracy": 0.7276899333626147,
-      "eval_num_tokens": 842388.0,
-      "eval_runtime": 299.6651,
-      "eval_samples_per_second": 3.451,
-      "eval_steps_per_second": 0.864,
-      "step": 876
-    },
-    {
-      "entropy": 0.5898892944678664,
-      "epoch": 2.0091428571428573,
-      "grad_norm": 1.3116419315338135,
-      "learning_rate": 5.990867579908676e-06,
-      "loss": 0.5677,
-      "mean_token_accuracy": 0.8516398537904024,
-      "num_tokens": 7804.0,
-      "step": 880
-    },
-    {
-      "entropy": 0.6767458073794842,
-      "epoch": 2.032,
-      "grad_norm": 1.4093241691589355,
-      "learning_rate": 5.945205479452055e-06,
-      "loss": 0.6416,
-      "mean_token_accuracy": 0.8335339192301034,
-      "num_tokens": 21400.0,
-      "step": 890
-    },
-    {
-      "entropy": 0.7389240754768253,
-      "epoch": 2.0548571428571427,
-      "grad_norm": 1.8839222192764282,
-      "learning_rate": 5.8995433789954336e-06,
-      "loss": 0.6714,
-      "mean_token_accuracy": 0.8232478138059378,
-      "num_tokens": 31986.0,
-      "step": 900
-    },
-    {
-      "entropy": 0.8426970480009913,
-      "epoch": 2.077714285714286,
-      "grad_norm": 2.546990394592285,
-      "learning_rate": 5.853881278538813e-06,
-      "loss": 0.773,
-      "mean_token_accuracy": 0.7947055101394653,
-      "num_tokens": 39598.0,
-      "step": 910
-    },
-    {
-      "entropy": 0.9150473427027463,
-      "epoch": 2.1005714285714285,
-      "grad_norm": 2.7457187175750732,
-      "learning_rate": 5.8082191780821915e-06,
-      "loss": 0.8326,
-      "mean_token_accuracy": 0.7829385627061128,
-      "num_tokens": 45726.0,
-      "step": 920
-    },
-    {
-      "entropy": 0.7779074914753437,
-      "epoch": 2.123428571428571,
-      "grad_norm": 1.6818033456802368,
-      "learning_rate": 5.762557077625572e-06,
-      "loss": 0.6953,
-      "mean_token_accuracy": 0.8222391355782748,
-      "num_tokens": 55996.0,
-      "step": 930
-    },
-    {
-      "entropy": 0.6338619258254766,
-      "epoch": 2.1462857142857144,
-      "grad_norm": 1.8489398956298828,
-      "learning_rate": 5.716894977168949e-06,
-      "loss": 0.5875,
-      "mean_token_accuracy": 0.8410690952092409,
-      "num_tokens": 69720.0,
-      "step": 940
-    },
-    {
-      "entropy": 0.7125289073213935,
-      "epoch": 2.169142857142857,
-      "grad_norm": 1.8807828426361084,
-      "learning_rate": 5.6712328767123296e-06,
-      "loss": 0.6763,
-      "mean_token_accuracy": 0.8223242565989495,
-      "num_tokens": 80327.0,
-      "step": 950
-    },
-    {
-      "entropy": 0.8898364685475826,
-      "epoch": 2.192,
-      "grad_norm": 2.358139753341675,
-      "learning_rate": 5.625570776255708e-06,
-      "loss": 0.8491,
-      "mean_token_accuracy": 0.7844949930906295,
-      "num_tokens": 88242.0,
-      "step": 960
-    },
-    {
-      "entropy": 0.8897234506905078,
-      "epoch": 2.214857142857143,
-      "grad_norm": 2.5401251316070557,
-      "learning_rate": 5.5799086757990874e-06,
-      "loss": 0.786,
-      "mean_token_accuracy": 0.795602411031723,
-      "num_tokens": 94381.0,
-      "step": 970
-    },
-    {
-      "entropy": 0.7333379179239273,
-      "epoch": 2.2377142857142855,
-      "grad_norm": 1.7071613073349,
-      "learning_rate": 5.534246575342466e-06,
-      "loss": 0.665,
-      "mean_token_accuracy": 0.8248766608536243,
-      "num_tokens": 105014.0,
-      "step": 980
-    },
-    {
-      "entropy": 0.6419918712228536,
-      "epoch": 2.2605714285714287,
-      "grad_norm": 1.5552293062210083,
-      "learning_rate": 5.488584474885845e-06,
-      "loss": 0.6026,
-      "mean_token_accuracy": 0.8414491657167673,
-      "num_tokens": 118550.0,
-      "step": 990
-    },
-    {
-      "entropy": 0.7310339482501149,
-      "epoch": 2.2834285714285714,
-      "grad_norm": 2.278031587600708,
-      "learning_rate": 5.442922374429224e-06,
-      "loss": 0.6805,
-      "mean_token_accuracy": 0.8224124182015657,
-      "num_tokens": 128693.0,
-      "step": 1000
-    },
-    {
-      "entropy": 0.880455293878913,
-      "epoch": 2.306285714285714,
-      "grad_norm": 2.459608554840088,
-      "learning_rate": 5.397260273972603e-06,
-      "loss": 0.8336,
-      "mean_token_accuracy": 0.7825992915779352,
-      "num_tokens": 136122.0,
-      "step": 1010
-    },
-    {
-      "entropy": 0.8820295415818691,
-      "epoch": 2.329142857142857,
-      "grad_norm": 2.6355550289154053,
-      "learning_rate": 5.351598173515982e-06,
-      "loss": 0.7887,
-      "mean_token_accuracy": 0.7964828334748745,
-      "num_tokens": 142194.0,
-      "step": 1020
-    },
-    {
-      "entropy": 0.7468231266364456,
-      "epoch": 2.352,
-      "grad_norm": 1.3787378072738647,
-      "learning_rate": 5.305936073059361e-06,
-      "loss": 0.6603,
-      "mean_token_accuracy": 0.8238665115088224,
-      "num_tokens": 152715.0,
-      "step": 1030
-    },
-    {
-      "entropy": 0.6317665258422493,
-      "epoch": 2.374857142857143,
-      "grad_norm": 1.7379688024520874,
-      "learning_rate": 5.26027397260274e-06,
-      "loss": 0.6007,
-      "mean_token_accuracy": 0.8413413379341363,
-      "num_tokens": 166545.0,
-      "step": 1040
-    },
-    {
-      "entropy": 0.7211008200421929,
-      "epoch": 2.3977142857142857,
-      "grad_norm": 2.5596601963043213,
-      "learning_rate": 5.214611872146119e-06,
-      "loss": 0.656,
-      "mean_token_accuracy": 0.8258481413125992,
-      "num_tokens": 177411.0,
-      "step": 1050
-    },
-    {
-      "entropy": 0.8763142567127943,
-      "epoch": 2.420571428571429,
-      "grad_norm": 2.739737033843994,
-      "learning_rate": 5.1689497716894975e-06,
-      "loss": 0.8354,
-      "mean_token_accuracy": 0.7820440270006657,
-      "num_tokens": 185340.0,
-      "step": 1060
-    },
-    {
-      "entropy": 0.8908181961625814,
-      "epoch": 2.4434285714285715,
-      "grad_norm": 3.223233222961426,
-      "learning_rate": 5.123287671232877e-06,
-      "loss": 0.7972,
-      "mean_token_accuracy": 0.79459448158741,
-      "num_tokens": 191540.0,
-      "step": 1070
-    },
-    {
-      "entropy": 0.7510069858282804,
-      "epoch": 2.466285714285714,
-      "grad_norm": 1.703507661819458,
-      "learning_rate": 5.077625570776255e-06,
-      "loss": 0.6854,
-      "mean_token_accuracy": 0.8139259118586779,
-      "num_tokens": 201907.0,
-      "step": 1080
-    },
-    {
-      "entropy": 0.6569937597960234,
-      "epoch": 2.4891428571428573,
-      "grad_norm": 1.7559291124343872,
-      "learning_rate": 5.031963470319635e-06,
-      "loss": 0.6111,
-      "mean_token_accuracy": 0.84332409016788,
-      "num_tokens": 215231.0,
-      "step": 1090
-    },
-    {
-      "entropy": 0.7254189381375908,
-      "epoch": 2.512,
-      "grad_norm": 1.9119453430175781,
-      "learning_rate": 4.986301369863014e-06,
-      "loss": 0.6814,
-      "mean_token_accuracy": 0.8200360022485256,
-      "num_tokens": 225539.0,
-      "step": 1100
-    },
-    {
-      "entropy": 0.8703744746744633,
-      "epoch": 2.5348571428571427,
-      "grad_norm": 2.812936305999756,
-      "learning_rate": 4.9406392694063935e-06,
-      "loss": 0.8006,
-      "mean_token_accuracy": 0.7889351420104503,
-      "num_tokens": 233013.0,
-      "step": 1110
-    },
-    {
-      "entropy": 0.8793116342276335,
-      "epoch": 2.557714285714286,
-      "grad_norm": 3.227419137954712,
-      "learning_rate": 4.894977168949772e-06,
-      "loss": 0.7693,
-      "mean_token_accuracy": 0.7998475536704064,
-      "num_tokens": 238974.0,
-      "step": 1120
-    },
-    {
-      "entropy": 0.7303921280428767,
-      "epoch": 2.5805714285714285,
-      "grad_norm": 1.6300448179244995,
-      "learning_rate": 4.849315068493151e-06,
-      "loss": 0.6538,
-      "mean_token_accuracy": 0.8278157886117696,
-      "num_tokens": 249037.0,
-      "step": 1130
-    },
-    {
-      "entropy": 0.6417382193729282,
-      "epoch": 2.603428571428571,
-      "grad_norm": 1.6912339925765991,
-      "learning_rate": 4.80365296803653e-06,
-      "loss": 0.6108,
-      "mean_token_accuracy": 0.8381971474736929,
-      "num_tokens": 262395.0,
-      "step": 1140
-    },
-    {
-      "entropy": 0.7339965717867016,
-      "epoch": 2.6262857142857143,
-      "grad_norm": 2.330716371536255,
-      "learning_rate": 4.757990867579909e-06,
-      "loss": 0.6772,
-      "mean_token_accuracy": 0.8194692388176918,
-      "num_tokens": 272694.0,
-      "step": 1150
-    },
-    {
-      "entropy": 0.8351591594517231,
-      "epoch": 2.649142857142857,
-      "grad_norm": 2.8293557167053223,
-      "learning_rate": 4.712328767123288e-06,
-      "loss": 0.7642,
-      "mean_token_accuracy": 0.7966304961591959,
-      "num_tokens": 280231.0,
-      "step": 1160
-    },
-    {
-      "entropy": 0.8861342877149582,
-      "epoch": 2.672,
-      "grad_norm": 2.9575674533843994,
-      "learning_rate": 4.666666666666667e-06,
-      "loss": 0.7999,
-      "mean_token_accuracy": 0.7918535027652979,
-      "num_tokens": 286161.0,
-      "step": 1170
-    },
-    {
-      "entropy": 0.7180219950154424,
-      "epoch": 2.694857142857143,
-      "grad_norm": 1.5886666774749756,
-      "learning_rate": 4.6210045662100465e-06,
-      "loss": 0.6434,
-      "mean_token_accuracy": 0.8283716265112162,
-      "num_tokens": 296388.0,
-      "step": 1180
-    },
-    {
-      "entropy": 0.6133397184312344,
-      "epoch": 2.717714285714286,
-      "grad_norm": 1.8250149488449097,
-      "learning_rate": 4.575342465753425e-06,
-      "loss": 0.6059,
-      "mean_token_accuracy": 0.8467012654989958,
-      "num_tokens": 310217.0,
-      "step": 1190
-    },
-    {
-      "entropy": 0.6812281895428896,
-      "epoch": 2.7405714285714287,
-      "grad_norm": 2.336768627166748,
-      "learning_rate": 4.529680365296804e-06,
-      "loss": 0.6216,
-      "mean_token_accuracy": 0.8333451252430677,
-      "num_tokens": 320782.0,
-      "step": 1200
-    },
-    {
-      "entropy": 0.8352824920788408,
-      "epoch": 2.7634285714285713,
-      "grad_norm": 2.4751791954040527,
-      "learning_rate": 4.484018264840183e-06,
-      "loss": 0.7816,
-      "mean_token_accuracy": 0.7953334752470255,
-      "num_tokens": 328520.0,
-      "step": 1210
-    },
-    {
-      "entropy": 0.8553074564784765,
-      "epoch": 2.7862857142857145,
-      "grad_norm": 3.6519722938537598,
-      "learning_rate": 4.438356164383562e-06,
-      "loss": 0.7807,
-      "mean_token_accuracy": 0.7970743294805288,
-      "num_tokens": 334489.0,
-      "step": 1220
-    },
-    {
-      "entropy": 0.7267135815694928,
-      "epoch": 2.809142857142857,
-      "grad_norm": 1.6625852584838867,
-      "learning_rate": 4.392694063926941e-06,
-      "loss": 0.6521,
-      "mean_token_accuracy": 0.8255741696804761,
-      "num_tokens": 344543.0,
-      "step": 1230
-    },
-    {
-      "entropy": 0.6338536148890853,
-      "epoch": 2.832,
-      "grad_norm": 1.9026601314544678,
-      "learning_rate": 4.34703196347032e-06,
-      "loss": 0.5962,
-      "mean_token_accuracy": 0.8427884597331285,
-      "num_tokens": 357986.0,
-      "step": 1240
-    },
-    {
-      "entropy": 0.7158672915771603,
-      "epoch": 2.854857142857143,
-      "grad_norm": 2.288316488265991,
-      "learning_rate": 4.301369863013699e-06,
-      "loss": 0.6478,
-      "mean_token_accuracy": 0.8221574258059263,
-      "num_tokens": 368102.0,
-      "step": 1250
-    },
-    {
-      "entropy": 0.8342319210991264,
-      "epoch": 2.8777142857142857,
-      "grad_norm": 2.675821542739868,
-      "learning_rate": 4.255707762557078e-06,
-      "loss": 0.7634,
-      "mean_token_accuracy": 0.8008246626704931,
-      "num_tokens": 375682.0,
-      "step": 1260
-    },
-    {
-      "entropy": 0.8324337769299746,
-      "epoch": 2.9005714285714284,
-      "grad_norm": 3.794491767883301,
-      "learning_rate": 4.2100456621004574e-06,
-      "loss": 0.7409,
-      "mean_token_accuracy": 0.8102138575166464,
-      "num_tokens": 381707.0,
-      "step": 1270
-    },
-    {
-      "entropy": 0.7096458308398723,
-      "epoch": 2.9234285714285715,
-      "grad_norm": 1.945020318031311,
-      "learning_rate": 4.164383561643836e-06,
-      "loss": 0.6394,
-      "mean_token_accuracy": 0.8287061709910631,
-      "num_tokens": 391884.0,
-      "step": 1280
-    },
-    {
-      "entropy": 0.6416196620091796,
-      "epoch": 2.946285714285714,
-      "grad_norm": 2.1223883628845215,
-      "learning_rate": 4.118721461187215e-06,
-      "loss": 0.613,
-      "mean_token_accuracy": 0.837811603397131,
-      "num_tokens": 404730.0,
-      "step": 1290
-    },
-    {
-      "entropy": 0.7876641971990466,
-      "epoch": 2.9691428571428573,
-      "grad_norm": 3.030888795852661,
-      "learning_rate": 4.073059360730594e-06,
-      "loss": 0.7228,
-      "mean_token_accuracy": 0.8122910633683205,
-      "num_tokens": 413461.0,
-      "step": 1300
-    },
-    {
-      "entropy": 0.8694131746888161,
-      "epoch": 2.992,
-      "grad_norm": 2.9641623497009277,
-      "learning_rate": 4.027397260273973e-06,
-      "loss": 0.793,
-      "mean_token_accuracy": 0.7905130475759506,
-      "num_tokens": 419636.0,
-      "step": 1310
-    },
-    {
-      "epoch": 3.0,
-      "eval_accuracy": 0.009547123623011015,
-      "eval_entropy": 0.8601508936826787,
-      "eval_loss": 1.4408637285232544,
-      "eval_mean_token_accuracy": 0.7276363938931792,
-      "eval_num_tokens": 421194.0,
-      "eval_runtime": 323.9597,
-      "eval_samples_per_second": 3.192,
-      "eval_steps_per_second": 0.799,
-      "step": 1314
-    },
-    {
-      "entropy": 0.5390967978164554,
-      "epoch": 3.013714285714286,
-      "grad_norm": 1.706173300743103,
-      "learning_rate": 3.9863013698630135e-06,
-      "loss": 0.5134,
-      "mean_token_accuracy": 0.8650896890709797,
-      "num_tokens": 10877.0,
-      "step": 1320
-    },
-    {
-      "entropy": 0.5993562566116453,
-      "epoch": 3.0365714285714285,
-      "grad_norm": 1.8674507141113281,
-      "learning_rate": 3.940639269406393e-06,
-      "loss": 0.5419,
-      "mean_token_accuracy": 0.8520117592066526,
-      "num_tokens": 23811.0,
-      "step": 1330
-    },
-    {
-      "entropy": 0.6931641317903996,
-      "epoch": 3.0594285714285716,
-      "grad_norm": 3.045653820037842,
-      "learning_rate": 3.8949771689497714e-06,
-      "loss": 0.6581,
-      "mean_token_accuracy": 0.8223089531064034,
-      "num_tokens": 33518.0,
-      "step": 1340
-    },
-    {
-      "entropy": 0.8081557534635067,
-      "epoch": 3.0822857142857143,
-      "grad_norm": 3.7819550037384033,
-      "learning_rate": 3.849315068493151e-06,
-      "loss": 0.7526,
-      "mean_token_accuracy": 0.8033633768558502,
-      "num_tokens": 40731.0,
-      "step": 1350
-    },
-    {
-      "entropy": 0.8427179055288434,
-      "epoch": 3.105142857142857,
-      "grad_norm": 3.792132616043091,
-      "learning_rate": 3.8036529680365297e-06,
-      "loss": 0.7482,
-      "mean_token_accuracy": 0.8013022668659687,
-      "num_tokens": 46454.0,
-      "step": 1360
-    },
-    {
-      "entropy": 0.649565021879971,
-      "epoch": 3.128,
-      "grad_norm": 1.703001618385315,
-      "learning_rate": 3.7579908675799087e-06,
-      "loss": 0.575,
-      "mean_token_accuracy": 0.8482304524630309,
-      "num_tokens": 58387.0,
-      "step": 1370
-    },
-    {
-      "entropy": 0.6079548856243491,
-      "epoch": 3.150857142857143,
-      "grad_norm": 2.0299415588378906,
-      "learning_rate": 3.7123287671232876e-06,
-      "loss": 0.5657,
-      "mean_token_accuracy": 0.8562000930309296,
-      "num_tokens": 71361.0,
-      "step": 1380
-    },
-    {
-      "entropy": 0.6868046056479216,
-      "epoch": 3.1737142857142855,
-      "grad_norm": 2.8305366039276123,
-      "learning_rate": 3.6666666666666666e-06,
-      "loss": 0.6482,
-      "mean_token_accuracy": 0.8264093812555074,
-      "num_tokens": 81416.0,
-      "step": 1390
-    },
-    {
-      "entropy": 0.7875820865854621,
-      "epoch": 3.1965714285714286,
-      "grad_norm": 2.7740256786346436,
-      "learning_rate": 3.6210045662100455e-06,
-      "loss": 0.692,
-      "mean_token_accuracy": 0.8111964620649814,
-      "num_tokens": 88744.0,
-      "step": 1400
-    },
-    {
-      "entropy": 0.8337188992649317,
-      "epoch": 3.2194285714285713,
-      "grad_norm": 3.5716171264648438,
-      "learning_rate": 3.575342465753425e-06,
-      "loss": 0.7367,
-      "mean_token_accuracy": 0.8026115108281374,
-      "num_tokens": 94445.0,
-      "step": 1410
-    },
-    {
-      "entropy": 0.6171189024113118,
-      "epoch": 3.2422857142857144,
-      "grad_norm": 2.167130708694458,
-      "learning_rate": 3.529680365296804e-06,
-      "loss": 0.5768,
-      "mean_token_accuracy": 0.8457576856017113,
-      "num_tokens": 106447.0,
-      "step": 1420
-    },
-    {
-      "entropy": 0.588103704340756,
-      "epoch": 3.265142857142857,
-      "grad_norm": 1.812523603439331,
-      "learning_rate": 3.4840182648401828e-06,
-      "loss": 0.5576,
-      "mean_token_accuracy": 0.850422840192914,
-      "num_tokens": 119288.0,
-      "step": 1430
-    },
-    {
-      "entropy": 0.7199772633612156,
-      "epoch": 3.288,
-      "grad_norm": 2.936997890472412,
-      "learning_rate": 3.4383561643835617e-06,
-      "loss": 0.6583,
-      "mean_token_accuracy": 0.8228961959481239,
-      "num_tokens": 129120.0,
-      "step": 1440
-    },
-    {
-      "entropy": 0.8470261815935374,
-      "epoch": 3.310857142857143,
-      "grad_norm": 3.213144302368164,
-      "learning_rate": 3.3926940639269407e-06,
-      "loss": 0.7691,
-      "mean_token_accuracy": 0.7942754574120044,
-      "num_tokens": 136353.0,
-      "step": 1450
-    },
-    {
-      "entropy": 0.7986762724816799,
-      "epoch": 3.3337142857142856,
-      "grad_norm": 3.557766914367676,
-      "learning_rate": 3.3470319634703196e-06,
-      "loss": 0.6934,
-      "mean_token_accuracy": 0.8079280138015748,
-      "num_tokens": 142146.0,
-      "step": 1460
-    },
-    {
-      "entropy": 0.6144793089479208,
-      "epoch": 3.3565714285714288,
-      "grad_norm": 1.8922325372695923,
-      "learning_rate": 3.3013698630136985e-06,
-      "loss": 0.5531,
-      "mean_token_accuracy": 0.8498249996453524,
-      "num_tokens": 154573.0,
-      "step": 1470
-    },
-    {
-      "entropy": 0.6026311848312617,
-      "epoch": 3.3794285714285714,
-      "grad_norm": 2.3621621131896973,
-      "learning_rate": 3.2557077625570775e-06,
-      "loss": 0.5788,
-      "mean_token_accuracy": 0.843865205347538,
-      "num_tokens": 167191.0,
-      "step": 1480
-    },
-    {
-      "entropy": 0.692712034098804,
-      "epoch": 3.402285714285714,
-      "grad_norm": 2.5524520874023438,
-      "learning_rate": 3.210045662100457e-06,
-      "loss": 0.6331,
-      "mean_token_accuracy": 0.8313357140868902,
-      "num_tokens": 176759.0,
-      "step": 1490
-    },
-    {
-      "entropy": 0.8092848775908351,
-      "epoch": 3.4251428571428573,
-      "grad_norm": 2.9528560638427734,
-      "learning_rate": 3.164383561643836e-06,
-      "loss": 0.729,
-      "mean_token_accuracy": 0.8075098406523467,
-      "num_tokens": 183859.0,
-      "step": 1500
-    },
-    {
-      "entropy": 0.8049974404275417,
-      "epoch": 3.448,
-      "grad_norm": 4.05062198638916,
-      "learning_rate": 3.1187214611872147e-06,
-      "loss": 0.7243,
-      "mean_token_accuracy": 0.8051667951047421,
-      "num_tokens": 189583.0,
-      "step": 1510
-    },
-    {
-      "entropy": 0.6506396448239684,
-      "epoch": 3.4708571428571426,
-      "grad_norm": 1.689829707145691,
-      "learning_rate": 3.0730593607305937e-06,
-      "loss": 0.5978,
-      "mean_token_accuracy": 0.8373685766011476,
-      "num_tokens": 201924.0,
-      "step": 1520
-    },
-    {
-      "entropy": 0.5990774085745215,
-      "epoch": 3.4937142857142858,
-      "grad_norm": 1.9814187288284302,
-      "learning_rate": 3.0273972602739726e-06,
-      "loss": 0.5512,
-      "mean_token_accuracy": 0.8501433119177818,
-      "num_tokens": 215022.0,
-      "step": 1530
-    },
-    {
-      "entropy": 0.6897100256755948,
-      "epoch": 3.5165714285714285,
-      "grad_norm": 2.9063243865966797,
-      "learning_rate": 2.9817351598173516e-06,
-      "loss": 0.6379,
-      "mean_token_accuracy": 0.8311983995139599,
-      "num_tokens": 225193.0,
-      "step": 1540
-    },
-    {
-      "entropy": 0.8148755021393299,
-      "epoch": 3.5394285714285716,
-      "grad_norm": 3.0410537719726562,
-      "learning_rate": 2.9360730593607305e-06,
-      "loss": 0.7679,
-      "mean_token_accuracy": 0.8003406222909689,
-      "num_tokens": 232748.0,
-      "step": 1550
-    },
-    {
-      "entropy": 0.829321713745594,
-      "epoch": 3.5622857142857143,
-      "grad_norm": 3.606051206588745,
-      "learning_rate": 2.8904109589041095e-06,
-      "loss": 0.7419,
-      "mean_token_accuracy": 0.8075093895196914,
-      "num_tokens": 238583.0,
-      "step": 1560
-    },
-    {
-      "entropy": 0.6521147439256311,
-      "epoch": 3.5851428571428574,
-      "grad_norm": 2.431035280227661,
-      "learning_rate": 2.8447488584474884e-06,
-      "loss": 0.5798,
-      "mean_token_accuracy": 0.8457587130367756,
-      "num_tokens": 251053.0,
-      "step": 1570
-    },
-    {
-      "entropy": 0.59049033485353,
-      "epoch": 3.608,
-      "grad_norm": 2.3326406478881836,
-      "learning_rate": 2.7990867579908678e-06,
-      "loss": 0.5584,
-      "mean_token_accuracy": 0.8490381717681885,
-      "num_tokens": 263796.0,
-      "step": 1580
-    },
-    {
-      "entropy": 0.6822433460503816,
-      "epoch": 3.630857142857143,
-      "grad_norm": 2.796164035797119,
-      "learning_rate": 2.7534246575342467e-06,
-      "loss": 0.6184,
-      "mean_token_accuracy": 0.8322674740105868,
-      "num_tokens": 273597.0,
-      "step": 1590
-    },
-    {
-      "entropy": 0.802175448089838,
-      "epoch": 3.653714285714286,
-      "grad_norm": 3.028571367263794,
-      "learning_rate": 2.7077625570776257e-06,
-      "loss": 0.7123,
-      "mean_token_accuracy": 0.810053302720189,
-      "num_tokens": 280717.0,
-      "step": 1600
-    },
-    {
-      "entropy": 0.834137250110507,
-      "epoch": 3.6765714285714286,
-      "grad_norm": 3.7012383937835693,
-      "learning_rate": 2.6621004566210046e-06,
-      "loss": 0.7221,
-      "mean_token_accuracy": 0.8077411744743586,
-      "num_tokens": 286435.0,
-      "step": 1610
-    },
-    {
-      "entropy": 0.6269000029191375,
-      "epoch": 3.6994285714285713,
-      "grad_norm": 1.8523389101028442,
-      "learning_rate": 2.6164383561643835e-06,
-      "loss": 0.5711,
-      "mean_token_accuracy": 0.8481807161122561,
-      "num_tokens": 299317.0,
-      "step": 1620
-    },
-    {
-      "entropy": 0.5854253606870771,
-      "epoch": 3.7222857142857144,
-      "grad_norm": 2.166839361190796,
-      "learning_rate": 2.5707762557077625e-06,
-      "loss": 0.5558,
-      "mean_token_accuracy": 0.8550698190927506,
-      "num_tokens": 312515.0,
-      "step": 1630
-    },
-    {
-      "entropy": 0.6648679519072175,
-      "epoch": 3.745142857142857,
-      "grad_norm": 2.547304391860962,
-      "learning_rate": 2.5251141552511414e-06,
-      "loss": 0.6132,
-      "mean_token_accuracy": 0.8365901987999678,
-      "num_tokens": 322556.0,
-      "step": 1640
-    },
-    {
-      "entropy": 0.7738912200555206,
-      "epoch": 3.768,
-      "grad_norm": 3.1837921142578125,
-      "learning_rate": 2.479452054794521e-06,
-      "loss": 0.7081,
-      "mean_token_accuracy": 0.8078534632921219,
-      "num_tokens": 329999.0,
-      "step": 1650
-    },
-    {
-      "entropy": 0.8226759549230337,
-      "epoch": 3.790857142857143,
-      "grad_norm": 3.3705971240997314,
-      "learning_rate": 2.4337899543378997e-06,
-      "loss": 0.718,
-      "mean_token_accuracy": 0.8068133030086756,
-      "num_tokens": 335854.0,
-      "step": 1660
-    },
-    {
-      "entropy": 0.6193161474540829,
-      "epoch": 3.8137142857142856,
-      "grad_norm": 1.969228982925415,
-      "learning_rate": 2.3881278538812787e-06,
-      "loss": 0.5679,
-      "mean_token_accuracy": 0.8450891207903624,
-      "num_tokens": 348492.0,
-      "step": 1670
-    },
-    {
-      "entropy": 0.5715032175183297,
-      "epoch": 3.8365714285714283,
-      "grad_norm": 2.211939573287964,
-      "learning_rate": 2.3424657534246576e-06,
-      "loss": 0.5261,
-      "mean_token_accuracy": 0.860519690066576,
-      "num_tokens": 361386.0,
-      "step": 1680
-    },
-    {
-      "entropy": 0.6757410081103444,
-      "epoch": 3.8594285714285714,
-      "grad_norm": 3.2903149127960205,
-      "learning_rate": 2.296803652968037e-06,
-      "loss": 0.6431,
-      "mean_token_accuracy": 0.825245127826929,
-      "num_tokens": 371053.0,
-      "step": 1690
-    },
-    {
-      "entropy": 0.7908247765153646,
-      "epoch": 3.8822857142857146,
-      "grad_norm": 2.754093885421753,
-      "learning_rate": 2.251141552511416e-06,
-      "loss": 0.7062,
-      "mean_token_accuracy": 0.8119447190314532,
-      "num_tokens": 378222.0,
-      "step": 1700
-    },
-    {
-      "entropy": 0.8256376493722201,
-      "epoch": 3.9051428571428572,
-      "grad_norm": 3.919424295425415,
-      "learning_rate": 2.205479452054795e-06,
-      "loss": 0.7394,
-      "mean_token_accuracy": 0.8005702033638954,
-      "num_tokens": 383976.0,
-      "step": 1710
-    },
-    {
-      "entropy": 0.6268141394481063,
-      "epoch": 3.928,
-      "grad_norm": 2.0268919467926025,
-      "learning_rate": 2.159817351598174e-06,
-      "loss": 0.5669,
-      "mean_token_accuracy": 0.8454550232738256,
-      "num_tokens": 395553.0,
-      "step": 1720
-    },
-    {
-      "entropy": 0.6326769331470132,
-      "epoch": 3.950857142857143,
-      "grad_norm": 2.593083381652832,
-      "learning_rate": 2.1141552511415528e-06,
-      "loss": 0.5813,
-      "mean_token_accuracy": 0.8418777663260698,
-      "num_tokens": 407215.0,
-      "step": 1730
-    },
-    {
-      "entropy": 0.7661219704896212,
-      "epoch": 3.9737142857142858,
-      "grad_norm": 2.932962417602539,
-      "learning_rate": 2.0684931506849317e-06,
-      "loss": 0.7034,
-      "mean_token_accuracy": 0.8150413926690817,
-      "num_tokens": 414979.0,
-      "step": 1740
-    },
-    {
-      "entropy": 0.8078423272818327,
-      "epoch": 3.9965714285714284,
-      "grad_norm": 4.143108367919922,
-      "learning_rate": 2.0228310502283106e-06,
-      "loss": 0.6825,
-      "mean_token_accuracy": 0.8139274593442678,
-      "num_tokens": 420607.0,
-      "step": 1750
-    },
-    {
-      "epoch": 4.0,
-      "eval_accuracy": 0.00985720114239086,
-      "eval_entropy": 0.833712039652018,
-      "eval_loss": 1.498316764831543,
-      "eval_mean_token_accuracy": 0.72518339603564,
-      "eval_num_tokens": 421194.0,
-      "eval_runtime": 325.9415,
-      "eval_samples_per_second": 3.172,
-      "eval_steps_per_second": 0.795,
-      "step": 1752
-    },
-    {
-      "entropy": 0.5481636302643701,
-      "epoch": 4.018285714285715,
-      "grad_norm": 1.847604513168335,
-      "learning_rate": 1.9771689497716896e-06,
-      "loss": 0.5007,
-      "mean_token_accuracy": 0.8671465257280752,
-      "num_tokens": 434791.0,
-      "step": 1760
-    },
-    {
-      "entropy": 0.5742504514753819,
-      "epoch": 4.041142857142857,
-      "grad_norm": 2.2244632244110107,
-      "learning_rate": 1.931506849315069e-06,
-      "loss": 0.5271,
-      "mean_token_accuracy": 0.8591625761240721,
-      "num_tokens": 447092.0,
-      "step": 1770
-    },
-    {
-      "entropy": 0.6953230138868094,
-      "epoch": 4.064,
-      "grad_norm": 3.42199444770813,
-      "learning_rate": 1.8858447488584477e-06,
-      "loss": 0.6469,
-      "mean_token_accuracy": 0.8285580322146415,
-      "num_tokens": 456254.0,
-      "step": 1780
-    },
-    {
-      "entropy": 0.7736836820840836,
-      "epoch": 4.086857142857143,
-      "grad_norm": 3.351454257965088,
-      "learning_rate": 1.8401826484018268e-06,
-      "loss": 0.6909,
-      "mean_token_accuracy": 0.8115375626832246,
-      "num_tokens": 463064.0,
-      "step": 1790
-    },
-    {
-      "entropy": 0.771463468298316,
-      "epoch": 4.109714285714285,
-      "grad_norm": 4.134479522705078,
-      "learning_rate": 1.7945205479452058e-06,
-      "loss": 0.6807,
-      "mean_token_accuracy": 0.8126782298088073,
-      "num_tokens": 468431.0,
-      "step": 1800
-    },
-    {
-      "entropy": 0.5649335160851479,
-      "epoch": 4.132571428571429,
-      "grad_norm": 2.1762540340423584,
-      "learning_rate": 1.7488584474885847e-06,
-      "loss": 0.5221,
-      "mean_token_accuracy": 0.8567749988287687,
-      "num_tokens": 482534.0,
-      "step": 1810
-    },
-    {
-      "entropy": 0.5837410872802138,
-      "epoch": 4.155428571428572,
-      "grad_norm": 2.349236011505127,
-      "learning_rate": 1.7031963470319637e-06,
-      "loss": 0.5371,
-      "mean_token_accuracy": 0.8581233065575361,
-      "num_tokens": 494845.0,
-      "step": 1820
-    },
-    {
-      "entropy": 0.6922593496739864,
-      "epoch": 4.178285714285714,
-      "grad_norm": 2.9896738529205322,
-      "learning_rate": 1.6575342465753428e-06,
-      "loss": 0.6648,
-      "mean_token_accuracy": 0.8241277992725372,
-      "num_tokens": 504153.0,
-      "step": 1830
-    },
-    {
-      "entropy": 0.7702124075964093,
-      "epoch": 4.201142857142857,
-      "grad_norm": 3.322385549545288,
-      "learning_rate": 1.6118721461187218e-06,
-      "loss": 0.6712,
-      "mean_token_accuracy": 0.8188040722161531,
-      "num_tokens": 511101.0,
-      "step": 1840
-    },
-    {
-      "entropy": 0.8012370727956295,
-      "epoch": 4.224,
-      "grad_norm": 4.359086036682129,
-      "learning_rate": 1.5662100456621007e-06,
-      "loss": 0.6748,
-      "mean_token_accuracy": 0.8118188168853522,
-      "num_tokens": 516460.0,
-      "step": 1850
-    },
-    {
-      "entropy": 0.5523576781153678,
-      "epoch": 4.246857142857142,
-      "grad_norm": 2.107539176940918,
-      "learning_rate": 1.5205479452054797e-06,
-      "loss": 0.5081,
-      "mean_token_accuracy": 0.8673421230167151,
-      "num_tokens": 530848.0,
-      "step": 1860
-    },
-    {
-      "entropy": 0.5729602897539735,
-      "epoch": 4.269714285714286,
-      "grad_norm": 2.5580902099609375,
-      "learning_rate": 1.4748858447488584e-06,
-      "loss": 0.5319,
-      "mean_token_accuracy": 0.8585442833602428,
-      "num_tokens": 543148.0,
-      "step": 1870
-    },
-    {
-      "entropy": 0.6956869766116143,
-      "epoch": 4.292571428571429,
-      "grad_norm": 3.1137397289276123,
-      "learning_rate": 1.4292237442922373e-06,
-      "loss": 0.6509,
-      "mean_token_accuracy": 0.8259295519441366,
-      "num_tokens": 552615.0,
-      "step": 1880
-    },
-    {
-      "entropy": 0.7814504994079471,
-      "epoch": 4.315428571428572,
-      "grad_norm": 3.9837899208068848,
-      "learning_rate": 1.3835616438356165e-06,
-      "loss": 0.6732,
-      "mean_token_accuracy": 0.8206901982426643,
-      "num_tokens": 559644.0,
-      "step": 1890
-    },
-    {
-      "entropy": 0.8006520505994559,
-      "epoch": 4.338285714285714,
-      "grad_norm": 4.293622016906738,
-      "learning_rate": 1.3378995433789954e-06,
-      "loss": 0.687,
-      "mean_token_accuracy": 0.8162542518228293,
-      "num_tokens": 565049.0,
-      "step": 1900
-    },
-    {
-      "entropy": 0.5543251828290522,
-      "epoch": 4.361142857142857,
-      "grad_norm": 2.0004706382751465,
-      "learning_rate": 1.2922374429223744e-06,
-      "loss": 0.5105,
-      "mean_token_accuracy": 0.8651686757802963,
-      "num_tokens": 579511.0,
-      "step": 1910
-    },
-    {
-      "entropy": 0.5778749627992511,
-      "epoch": 4.384,
-      "grad_norm": 2.4198007583618164,
-      "learning_rate": 1.2465753424657535e-06,
-      "loss": 0.5216,
-      "mean_token_accuracy": 0.8559423860162496,
-      "num_tokens": 591873.0,
-      "step": 1920
-    },
-    {
-      "entropy": 0.6606394873932004,
-      "epoch": 4.406857142857143,
-      "grad_norm": 3.485213279724121,
-      "learning_rate": 1.2009132420091325e-06,
-      "loss": 0.6086,
-      "mean_token_accuracy": 0.8372324761003256,
-      "num_tokens": 601337.0,
-      "step": 1930
-    },
-    {
-      "entropy": 0.7576876068487763,
-      "epoch": 4.429714285714286,
-      "grad_norm": 3.3860034942626953,
-      "learning_rate": 1.1552511415525116e-06,
-      "loss": 0.7,
-      "mean_token_accuracy": 0.8150843985378742,
-      "num_tokens": 608326.0,
-      "step": 1940
-    },
-    {
-      "entropy": 0.7779752794653177,
-      "epoch": 4.452571428571429,
-      "grad_norm": 4.422528266906738,
-      "learning_rate": 1.1095890410958906e-06,
-      "loss": 0.6755,
-      "mean_token_accuracy": 0.8102645222097635,
-      "num_tokens": 613860.0,
-      "step": 1950
-    },
-    {
-      "entropy": 0.5644787142053247,
-      "epoch": 4.475428571428571,
-      "grad_norm": 2.2278542518615723,
-      "learning_rate": 1.0639269406392695e-06,
-      "loss": 0.5188,
-      "mean_token_accuracy": 0.8587661664932966,
-      "num_tokens": 627853.0,
-      "step": 1960
-    },
-    {
-      "entropy": 0.5740617036819458,
-      "epoch": 4.498285714285714,
-      "grad_norm": 2.4340105056762695,
-      "learning_rate": 1.0182648401826485e-06,
-      "loss": 0.5167,
-      "mean_token_accuracy": 0.8595852922648192,
-      "num_tokens": 640020.0,
-      "step": 1970
-    },
-    {
-      "entropy": 0.671654068864882,
-      "epoch": 4.521142857142857,
-      "grad_norm": 3.127539873123169,
-      "learning_rate": 9.726027397260274e-07,
-      "loss": 0.6331,
-      "mean_token_accuracy": 0.8256836850196123,
-      "num_tokens": 649058.0,
-      "step": 1980
-    },
-    {
-      "entropy": 0.7583655359223485,
-      "epoch": 4.5440000000000005,
-      "grad_norm": 3.5964298248291016,
-      "learning_rate": 9.269406392694065e-07,
-      "loss": 0.679,
-      "mean_token_accuracy": 0.8139733098447323,
-      "num_tokens": 655831.0,
-      "step": 1990
-    },
-    {
-      "entropy": 0.7816360153257846,
-      "epoch": 4.566857142857143,
-      "grad_norm": 4.389492511749268,
-      "learning_rate": 8.812785388127855e-07,
-      "loss": 0.6784,
-      "mean_token_accuracy": 0.8120625615119934,
-      "num_tokens": 661256.0,
-      "step": 2000
-    },
-    {
-      "entropy": 0.5732687024399639,
-      "epoch": 4.589714285714286,
-      "grad_norm": 2.0767221450805664,
-      "learning_rate": 8.356164383561644e-07,
-      "loss": 0.5335,
-      "mean_token_accuracy": 0.8624513667076826,
-      "num_tokens": 675612.0,
-      "step": 2010
-    },
-    {
-      "entropy": 0.5804870082065463,
-      "epoch": 4.612571428571428,
-      "grad_norm": 2.554534673690796,
-      "learning_rate": 7.899543378995435e-07,
-      "loss": 0.5238,
-      "mean_token_accuracy": 0.8590863507241011,
-      "num_tokens": 687684.0,
-      "step": 2020
-    },
-    {
-      "entropy": 0.6967678766697645,
-      "epoch": 4.635428571428571,
-      "grad_norm": 3.255140542984009,
-      "learning_rate": 7.442922374429224e-07,
-      "loss": 0.6487,
-      "mean_token_accuracy": 0.8244634248316288,
-      "num_tokens": 696675.0,
-      "step": 2030
-    },
-    {
-      "entropy": 0.7526340587064624,
-      "epoch": 4.658285714285714,
-      "grad_norm": 3.69323992729187,
-      "learning_rate": 6.986301369863015e-07,
-      "loss": 0.6719,
-      "mean_token_accuracy": 0.8216490592807532,
-      "num_tokens": 703456.0,
-      "step": 2040
-    },
-    {
-      "entropy": 0.7950498787686229,
-      "epoch": 4.6811428571428575,
-      "grad_norm": 4.715794563293457,
-      "learning_rate": 6.529680365296804e-07,
-      "loss": 0.6808,
-      "mean_token_accuracy": 0.8184644509106874,
-      "num_tokens": 708782.0,
-      "step": 2050
-    },
-    {
-      "entropy": 0.5505135927349329,
-      "epoch": 4.704,
-      "grad_norm": 2.3146073818206787,
-      "learning_rate": 6.073059360730594e-07,
-      "loss": 0.507,
-      "mean_token_accuracy": 0.8652824487537145,
-      "num_tokens": 723247.0,
-      "step": 2060
-    },
-    {
-      "entropy": 0.5807493371888995,
-      "epoch": 4.726857142857143,
-      "grad_norm": 2.615732192993164,
-      "learning_rate": 5.616438356164384e-07,
-      "loss": 0.5342,
-      "mean_token_accuracy": 0.854337964951992,
-      "num_tokens": 735283.0,
-      "step": 2070
-    },
-    {
-      "entropy": 0.7081154704093933,
-      "epoch": 4.749714285714286,
-      "grad_norm": 3.0795960426330566,
-      "learning_rate": 5.159817351598174e-07,
-      "loss": 0.6499,
-      "mean_token_accuracy": 0.8241405732929706,
-      "num_tokens": 744298.0,
-      "step": 2080
-    },
-    {
-      "entropy": 0.783203998953104,
-      "epoch": 4.772571428571428,
-      "grad_norm": 3.7807230949401855,
-      "learning_rate": 4.7031963470319636e-07,
-      "loss": 0.6948,
-      "mean_token_accuracy": 0.8167315106838942,
-      "num_tokens": 751212.0,
-      "step": 2090
-    },
-    {
-      "entropy": 0.7742167858406901,
-      "epoch": 4.795428571428571,
-      "grad_norm": 4.185308933258057,
-      "learning_rate": 4.2465753424657536e-07,
-      "loss": 0.6705,
-      "mean_token_accuracy": 0.8145616598427295,
-      "num_tokens": 756648.0,
-      "step": 2100
-    },
-    {
-      "entropy": 0.5554421614855528,
-      "epoch": 4.8182857142857145,
-      "grad_norm": 2.0456132888793945,
-      "learning_rate": 3.7899543378995436e-07,
-      "loss": 0.4982,
-      "mean_token_accuracy": 0.8656269229948521,
-      "num_tokens": 771047.0,
-      "step": 2110
-    },
-    {
-      "entropy": 0.5584687992930413,
-      "epoch": 4.841142857142858,
-      "grad_norm": 2.591322422027588,
-      "learning_rate": 3.3333333333333335e-07,
-      "loss": 0.5038,
-      "mean_token_accuracy": 0.8639244794845581,
-      "num_tokens": 783484.0,
-      "step": 2120
-    },
-    {
-      "entropy": 0.6443877406418324,
-      "epoch": 4.864,
-      "grad_norm": 3.1148664951324463,
-      "learning_rate": 2.8767123287671235e-07,
-      "loss": 0.5898,
-      "mean_token_accuracy": 0.8393101956695318,
-      "num_tokens": 793159.0,
-      "step": 2130
-    },
-    {
-      "entropy": 0.7694938328117132,
-      "epoch": 4.886857142857143,
-      "grad_norm": 3.860647201538086,
-      "learning_rate": 2.4200913242009135e-07,
-      "loss": 0.6661,
-      "mean_token_accuracy": 0.8193999473005533,
-      "num_tokens": 800322.0,
-      "step": 2140
-    },
-    {
-      "entropy": 0.7594648722559214,
-      "epoch": 4.909714285714285,
-      "grad_norm": 4.108844757080078,
-      "learning_rate": 1.9634703196347034e-07,
-      "loss": 0.6513,
-      "mean_token_accuracy": 0.820786502957344,
-      "num_tokens": 805782.0,
-      "step": 2150
-    },
-    {
-      "entropy": 0.5718974178656936,
-      "epoch": 4.932571428571428,
-      "grad_norm": 2.0961592197418213,
-      "learning_rate": 1.5068493150684934e-07,
-      "loss": 0.5321,
-      "mean_token_accuracy": 0.8587038304656744,
-      "num_tokens": 819399.0,
-      "step": 2160
-    },
-    {
-      "entropy": 0.6284356378018856,
-      "epoch": 4.9554285714285715,
-      "grad_norm": 2.863541841506958,
-      "learning_rate": 1.0502283105022832e-07,
-      "loss": 0.5916,
-      "mean_token_accuracy": 0.8409151379019022,
-      "num_tokens": 830244.0,
-      "step": 2170
-    },
-    {
-      "entropy": 0.7672561943531037,
-      "epoch": 4.978285714285715,
-      "grad_norm": 3.52964186668396,
-      "learning_rate": 5.936073059360731e-08,
-      "loss": 0.6968,
-      "mean_token_accuracy": 0.8189954232424498,
-      "num_tokens": 837492.0,
-      "step": 2180
-    },
-    {
-      "entropy": 0.7868584784630098,
-      "epoch": 5.0,
-      "grad_norm": 6.119595050811768,
-      "learning_rate": 1.3698630136986303e-08,
-      "loss": 0.6771,
-      "mean_token_accuracy": 0.8141911743502868,
-      "num_tokens": 842388.0,
-      "step": 2190
-    },
-    {
-      "epoch": 5.0,
-      "eval_accuracy": 0.009840881272949817,
-      "eval_entropy": 0.8240771901193272,
-      "eval_loss": 1.5346653461456299,
-      "eval_mean_token_accuracy": 0.7217774188656605,
-      "eval_num_tokens": 842388.0,
-      "eval_runtime": 325.5092,
-      "eval_samples_per_second": 3.177,
-      "eval_steps_per_second": 0.796,
-      "step": 2190
-    }
-  ],
-  "logging_steps": 10,
-  "max_steps": 2190,
-  "num_input_tokens_seen": 0,
-  "num_train_epochs": 5,
-  "save_steps": 500,
-  "stateful_callbacks": {
-    "TrainerControl": {
-      "args": {
-        "should_epoch_stop": false,
-        "should_evaluate": false,
-        "should_log": false,
-        "should_save": true,
-        "should_training_stop": true
-      },
-      "attributes": {}
-    }
-  },
-  "total_flos": 1.470014665030656e+17,
-  "train_batch_size": 1,
-  "trial_name": null,
-  "trial_params": null
-}

last-checkpoint/training_args.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:4f69888a63d7dbdc14452efa835b06908606f8c384200e78ac65b432c94b2f31
-size 6353