Upload task output goof-intercode-synth

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,209 @@
+---
+base_model: Jordansky/augmented-f560e4e6ee71e78d
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/cache/models/Jordansky--augmented-f560e4e6ee71e78d
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Jordansky/augmented-f560e4e6ee71e78d",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 512,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "up_proj",
+    "k_proj",
+    "v_proj",
+    "gate_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3598fd5fa956f092a7a1b00b7771250d547a1dfc2ef483619747b42524202575
+size 1057033224

added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,61 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

loss.txt

@@ -0,0 +1 @@
+869,no_eval

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 1010000,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

trainer_state.json

@@ -0,0 +1,1591 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9988505747126437,
+  "eval_steps": 500,
+  "global_step": 869,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.005747126436781609,
+      "grad_norm": 11.50942325592041,
+      "learning_rate": 3.2463636363636365e-06,
+      "loss": 1.9136,
+      "mean_token_accuracy": 0.6757451295852661,
+      "num_tokens": 403129.0,
+      "step": 5
+    },
+    {
+      "epoch": 0.011494252873563218,
+      "grad_norm": 3.601832866668701,
+      "learning_rate": 7.3043181818181825e-06,
+      "loss": 1.6064,
+      "mean_token_accuracy": 0.6915940046310425,
+      "num_tokens": 808163.0,
+      "step": 10
+    },
+    {
+      "epoch": 0.017241379310344827,
+      "grad_norm": 2.5387675762176514,
+      "learning_rate": 1.1362272727272727e-05,
+      "loss": 1.178,
+      "mean_token_accuracy": 0.7406479597091675,
+      "num_tokens": 1212678.0,
+      "step": 15
+    },
+    {
+      "epoch": 0.022988505747126436,
+      "grad_norm": 2.6335058212280273,
+      "learning_rate": 1.5420227272727274e-05,
+      "loss": 0.752,
+      "mean_token_accuracy": 0.8273701429367065,
+      "num_tokens": 1615888.0,
+      "step": 20
+    },
+    {
+      "epoch": 0.028735632183908046,
+      "grad_norm": 1.3575574159622192,
+      "learning_rate": 1.9478181818181816e-05,
+      "loss": 0.3227,
+      "mean_token_accuracy": 0.9204407930374146,
+      "num_tokens": 2020108.0,
+      "step": 25
+    },
+    {
+      "epoch": 0.034482758620689655,
+      "grad_norm": 4.603687763214111,
+      "learning_rate": 2.3536136363636365e-05,
+      "loss": 0.0922,
+      "mean_token_accuracy": 0.9775937318801879,
+      "num_tokens": 2425497.0,
+      "step": 30
+    },
+    {
+      "epoch": 0.040229885057471264,
+      "grad_norm": 0.3694731295108795,
+      "learning_rate": 2.759409090909091e-05,
+      "loss": 0.0651,
+      "mean_token_accuracy": 0.983700180053711,
+      "num_tokens": 2828765.0,
+      "step": 35
+    },
+    {
+      "epoch": 0.04597701149425287,
+      "grad_norm": 0.312744677066803,
+      "learning_rate": 3.1652045454545456e-05,
+      "loss": 0.0291,
+      "mean_token_accuracy": 0.991856825351715,
+      "num_tokens": 3233303.0,
+      "step": 40
+    },
+    {
+      "epoch": 0.05172413793103448,
+      "grad_norm": 0.1357622891664505,
+      "learning_rate": 3.571e-05,
+      "loss": 0.0213,
+      "mean_token_accuracy": 0.9935701847076416,
+      "num_tokens": 3637991.0,
+      "step": 45
+    },
+    {
+      "epoch": 0.05747126436781609,
+      "grad_norm": 0.11526472866535187,
+      "learning_rate": 3.570757864931307e-05,
+      "loss": 0.0188,
+      "mean_token_accuracy": 0.9941024422645569,
+      "num_tokens": 4042692.0,
+      "step": 50
+    },
+    {
+      "epoch": 0.06321839080459771,
+      "grad_norm": 0.06420823186635971,
+      "learning_rate": 3.570031547288962e-05,
+      "loss": 0.0187,
+      "mean_token_accuracy": 0.9939285755157471,
+      "num_tokens": 4446677.0,
+      "step": 55
+    },
+    {
+      "epoch": 0.06896551724137931,
+      "grad_norm": 0.07142060250043869,
+      "learning_rate": 3.568821309732495e-05,
+      "loss": 0.0179,
+      "mean_token_accuracy": 0.9940606236457825,
+      "num_tokens": 4851040.0,
+      "step": 60
+    },
+    {
+      "epoch": 0.07471264367816093,
+      "grad_norm": 0.14311493933200836,
+      "learning_rate": 3.567127589922249e-05,
+      "loss": 0.0181,
+      "mean_token_accuracy": 0.9940086245536804,
+      "num_tokens": 5254512.0,
+      "step": 65
+    },
+    {
+      "epoch": 0.08045977011494253,
+      "grad_norm": 0.1732994168996811,
+      "learning_rate": 3.5649510003611104e-05,
+      "loss": 0.0163,
+      "mean_token_accuracy": 0.9943243384361267,
+      "num_tokens": 5659208.0,
+      "step": 70
+    },
+    {
+      "epoch": 0.08620689655172414,
+      "grad_norm": 0.03773409128189087,
+      "learning_rate": 3.562292328173002e-05,
+      "loss": 0.0154,
+      "mean_token_accuracy": 0.9946787118911743,
+      "num_tokens": 6064767.0,
+      "step": 75
+    },
+    {
+      "epoch": 0.09195402298850575,
+      "grad_norm": 0.04296102374792099,
+      "learning_rate": 3.5591525348182415e-05,
+      "loss": 0.0174,
+      "mean_token_accuracy": 0.9942841291427612,
+      "num_tokens": 6467687.0,
+      "step": 80
+    },
+    {
+      "epoch": 0.09770114942528736,
+      "grad_norm": 0.052675988525152206,
+      "learning_rate": 3.5555327557458406e-05,
+      "loss": 0.0168,
+      "mean_token_accuracy": 0.9943979263305665,
+      "num_tokens": 6870874.0,
+      "step": 85
+    },
+    {
+      "epoch": 0.10344827586206896,
+      "grad_norm": 0.04074929654598236,
+      "learning_rate": 3.551434299982895e-05,
+      "loss": 0.0162,
+      "mean_token_accuracy": 0.9944364666938782,
+      "num_tokens": 7274927.0,
+      "step": 90
+    },
+    {
+      "epoch": 0.10919540229885058,
+      "grad_norm": 0.053142838180065155,
+      "learning_rate": 3.546858649661196e-05,
+      "loss": 0.0166,
+      "mean_token_accuracy": 0.994316291809082,
+      "num_tokens": 7678668.0,
+      "step": 95
+    },
+    {
+      "epoch": 0.11494252873563218,
+      "grad_norm": 0.05124868080019951,
+      "learning_rate": 3.5418074594812456e-05,
+      "loss": 0.0156,
+      "mean_token_accuracy": 0.9945274949073791,
+      "num_tokens": 8082319.0,
+      "step": 100
+    },
+    {
+      "epoch": 0.1206896551724138,
+      "grad_norm": 0.05799630656838417,
+      "learning_rate": 3.536282556113864e-05,
+      "loss": 0.0153,
+      "mean_token_accuracy": 0.9946353793144226,
+      "num_tokens": 8485887.0,
+      "step": 105
+    },
+    {
+      "epoch": 0.12643678160919541,
+      "grad_norm": 0.048035357147455215,
+      "learning_rate": 3.530285937539609e-05,
+      "loss": 0.0153,
+      "mean_token_accuracy": 0.9946409940719605,
+      "num_tokens": 8888964.0,
+      "step": 110
+    },
+    {
+      "epoch": 0.13218390804597702,
+      "grad_norm": 0.040226273238658905,
+      "learning_rate": 3.5238197723262395e-05,
+      "loss": 0.0151,
+      "mean_token_accuracy": 0.9946888566017151,
+      "num_tokens": 9291928.0,
+      "step": 115
+    },
+    {
+      "epoch": 0.13793103448275862,
+      "grad_norm": 0.05495241656899452,
+      "learning_rate": 3.5168863988444935e-05,
+      "loss": 0.0149,
+      "mean_token_accuracy": 0.9946548223495484,
+      "num_tokens": 9695699.0,
+      "step": 120
+    },
+    {
+      "epoch": 0.14367816091954022,
+      "grad_norm": 0.06851889193058014,
+      "learning_rate": 3.509488324422462e-05,
+      "loss": 0.0152,
+      "mean_token_accuracy": 0.9944992899894715,
+      "num_tokens": 10099209.0,
+      "step": 125
+    },
+    {
+      "epoch": 0.14942528735632185,
+      "grad_norm": 0.04976602643728256,
+      "learning_rate": 3.5016282244388565e-05,
+      "loss": 0.0143,
+      "mean_token_accuracy": 0.9948261499404907,
+      "num_tokens": 10503188.0,
+      "step": 130
+    },
+    {
+      "epoch": 0.15517241379310345,
+      "grad_norm": 0.05081531032919884,
+      "learning_rate": 3.493308941355507e-05,
+      "loss": 0.0155,
+      "mean_token_accuracy": 0.994414746761322,
+      "num_tokens": 10906288.0,
+      "step": 135
+    },
+    {
+      "epoch": 0.16091954022988506,
+      "grad_norm": 0.050771500915288925,
+      "learning_rate": 3.48453348368944e-05,
+      "loss": 0.0143,
+      "mean_token_accuracy": 0.9947004079818725,
+      "num_tokens": 11311117.0,
+      "step": 140
+    },
+    {
+      "epoch": 0.16666666666666666,
+      "grad_norm": 0.06757643818855286,
+      "learning_rate": 3.475305024924899e-05,
+      "loss": 0.0142,
+      "mean_token_accuracy": 0.994769024848938,
+      "num_tokens": 11715201.0,
+      "step": 145
+    },
+    {
+      "epoch": 0.1724137931034483,
+      "grad_norm": 0.050865691155195236,
+      "learning_rate": 3.465626902365717e-05,
+      "loss": 0.0148,
+      "mean_token_accuracy": 0.9945730566978455,
+      "num_tokens": 12117386.0,
+      "step": 150
+    },
+    {
+      "epoch": 0.1781609195402299,
+      "grad_norm": 0.038052380084991455,
+      "learning_rate": 3.455502615928437e-05,
+      "loss": 0.0143,
+      "mean_token_accuracy": 0.9947114586830139,
+      "num_tokens": 12519849.0,
+      "step": 155
+    },
+    {
+      "epoch": 0.1839080459770115,
+      "grad_norm": 0.045970458537340164,
+      "learning_rate": 3.4449358268766333e-05,
+      "loss": 0.0149,
+      "mean_token_accuracy": 0.9944195628166199,
+      "num_tokens": 12922528.0,
+      "step": 160
+    },
+    {
+      "epoch": 0.1896551724137931,
+      "grad_norm": 0.03716587647795677,
+      "learning_rate": 3.433930356496882e-05,
+      "loss": 0.0141,
+      "mean_token_accuracy": 0.994634211063385,
+      "num_tokens": 13325018.0,
+      "step": 165
+    },
+    {
+      "epoch": 0.19540229885057472,
+      "grad_norm": 0.039773862808942795,
+      "learning_rate": 3.422490184716862e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9949192404747009,
+      "num_tokens": 13729095.0,
+      "step": 170
+    },
+    {
+      "epoch": 0.20114942528735633,
+      "grad_norm": 0.058626800775527954,
+      "learning_rate": 3.4106194486660854e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9947888255119324,
+      "num_tokens": 14132517.0,
+      "step": 175
+    },
+    {
+      "epoch": 0.20689655172413793,
+      "grad_norm": 0.04217752069234848,
+      "learning_rate": 3.398322441179785e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9947295308113098,
+      "num_tokens": 14537265.0,
+      "step": 180
+    },
+    {
+      "epoch": 0.21264367816091953,
+      "grad_norm": 0.029374336823821068,
+      "learning_rate": 3.385603609246481e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9946742534637452,
+      "num_tokens": 14941978.0,
+      "step": 185
+    },
+    {
+      "epoch": 0.21839080459770116,
+      "grad_norm": 0.043314963579177856,
+      "learning_rate": 3.372467552399816e-05,
+      "loss": 0.0141,
+      "mean_token_accuracy": 0.9947606563568115,
+      "num_tokens": 15345327.0,
+      "step": 190
+    },
+    {
+      "epoch": 0.22413793103448276,
+      "grad_norm": 0.03939647227525711,
+      "learning_rate": 3.358919021055213e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9946980714797974,
+      "num_tokens": 15749804.0,
+      "step": 195
+    },
+    {
+      "epoch": 0.22988505747126436,
+      "grad_norm": 0.02790178172290325,
+      "learning_rate": 3.3449629147919754e-05,
+      "loss": 0.0143,
+      "mean_token_accuracy": 0.9946450471878052,
+      "num_tokens": 16154369.0,
+      "step": 200
+    },
+    {
+      "epoch": 0.23563218390804597,
+      "grad_norm": 0.051178909838199615,
+      "learning_rate": 3.330604280581446e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9946406483650208,
+      "num_tokens": 16558165.0,
+      "step": 205
+    },
+    {
+      "epoch": 0.2413793103448276,
+      "grad_norm": 0.08488579094409943,
+      "learning_rate": 3.315848310961857e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.994620680809021,
+      "num_tokens": 16962046.0,
+      "step": 210
+    },
+    {
+      "epoch": 0.2471264367816092,
+      "grad_norm": 0.06223829835653305,
+      "learning_rate": 3.3007003421605485e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9947610378265381,
+      "num_tokens": 17365830.0,
+      "step": 215
+    },
+    {
+      "epoch": 0.25287356321839083,
+      "grad_norm": 0.04801960662007332,
+      "learning_rate": 3.285165852164214e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9947750210762024,
+      "num_tokens": 17769736.0,
+      "step": 220
+    },
+    {
+      "epoch": 0.25862068965517243,
+      "grad_norm": 0.06384305655956268,
+      "learning_rate": 3.2692504587378904e-05,
+      "loss": 0.0142,
+      "mean_token_accuracy": 0.9945329785346985,
+      "num_tokens": 18173470.0,
+      "step": 225
+    },
+    {
+      "epoch": 0.26436781609195403,
+      "grad_norm": 0.041138093918561935,
+      "learning_rate": 3.252959917393394e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9946693658828736,
+      "num_tokens": 18577196.0,
+      "step": 230
+    },
+    {
+      "epoch": 0.27011494252873564,
+      "grad_norm": 0.03739451244473457,
+      "learning_rate": 3.236300119307945e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9948052048683167,
+      "num_tokens": 18981159.0,
+      "step": 235
+    },
+    {
+      "epoch": 0.27586206896551724,
+      "grad_norm": 0.03280220553278923,
+      "learning_rate": 3.219277089193731e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9946678638458252,
+      "num_tokens": 19384054.0,
+      "step": 240
+    },
+    {
+      "epoch": 0.28160919540229884,
+      "grad_norm": 0.032432641834020615,
+      "learning_rate": 3.201896983119178e-05,
+      "loss": 0.0141,
+      "mean_token_accuracy": 0.9946418523788452,
+      "num_tokens": 19786850.0,
+      "step": 245
+    },
+    {
+      "epoch": 0.28735632183908044,
+      "grad_norm": 0.03479701280593872,
+      "learning_rate": 3.184166086282724e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9946959257125855,
+      "num_tokens": 20190209.0,
+      "step": 250
+    },
+    {
+      "epoch": 0.29310344827586204,
+      "grad_norm": 0.04647434875369072,
+      "learning_rate": 3.166090810739887e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9949136853218079,
+      "num_tokens": 20593987.0,
+      "step": 255
+    },
+    {
+      "epoch": 0.2988505747126437,
+      "grad_norm": 0.05208832398056984,
+      "learning_rate": 3.147677693084465e-05,
+      "loss": 0.0144,
+      "mean_token_accuracy": 0.9945999264717102,
+      "num_tokens": 20996490.0,
+      "step": 260
+    },
+    {
+      "epoch": 0.3045977011494253,
+      "grad_norm": 0.027506856247782707,
+      "learning_rate": 3.128933392084697e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9948423385620118,
+      "num_tokens": 21400522.0,
+      "step": 265
+    },
+    {
+      "epoch": 0.3103448275862069,
+      "grad_norm": 0.04659893363714218,
+      "learning_rate": 3.109864686275239e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948711514472961,
+      "num_tokens": 21805437.0,
+      "step": 270
+    },
+    {
+      "epoch": 0.3160919540229885,
+      "grad_norm": 0.03459516912698746,
+      "learning_rate": 3.0904784715058294e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9946945190429688,
+      "num_tokens": 22210298.0,
+      "step": 275
+    },
+    {
+      "epoch": 0.3218390804597701,
+      "grad_norm": 0.04663790389895439,
+      "learning_rate": 3.0707817584475316e-05,
+      "loss": 0.0141,
+      "mean_token_accuracy": 0.9947023153305053,
+      "num_tokens": 22614052.0,
+      "step": 280
+    },
+    {
+      "epoch": 0.3275862068965517,
+      "grad_norm": 0.0350983701646328,
+      "learning_rate": 3.0507816700574574e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9948037743568421,
+      "num_tokens": 23017388.0,
+      "step": 285
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 0.041611939668655396,
+      "learning_rate": 3.030485439002871e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9947538614273072,
+      "num_tokens": 23421547.0,
+      "step": 290
+    },
+    {
+      "epoch": 0.3390804597701149,
+      "grad_norm": 0.044495340436697006,
+      "learning_rate": 3.009900405045633e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9946321249008179,
+      "num_tokens": 23824441.0,
+      "step": 295
+    },
+    {
+      "epoch": 0.3448275862068966,
+      "grad_norm": 0.0457838736474514,
+      "learning_rate": 2.989034012387908e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9946706891059875,
+      "num_tokens": 24228304.0,
+      "step": 300
+    },
+    {
+      "epoch": 0.3505747126436782,
+      "grad_norm": 0.05629606544971466,
+      "learning_rate": 2.9678938069801016e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9949870824813842,
+      "num_tokens": 24633847.0,
+      "step": 305
+    },
+    {
+      "epoch": 0.3563218390804598,
+      "grad_norm": 0.03772628307342529,
+      "learning_rate": 2.9464874337920103e-05,
+      "loss": 0.0137,
+      "mean_token_accuracy": 0.9947891712188721,
+      "num_tokens": 25037659.0,
+      "step": 310
+    },
+    {
+      "epoch": 0.3620689655172414,
+      "grad_norm": 0.04441123828291893,
+      "learning_rate": 2.924822634048154e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9948173046112061,
+      "num_tokens": 25441857.0,
+      "step": 315
+    },
+    {
+      "epoch": 0.367816091954023,
+      "grad_norm": 0.032479654997587204,
+      "learning_rate": 2.9029072424283028e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9949279308319092,
+      "num_tokens": 25847022.0,
+      "step": 320
+    },
+    {
+      "epoch": 0.3735632183908046,
+      "grad_norm": 0.03903662785887718,
+      "learning_rate": 2.8807491842342106e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.9946053504943848,
+      "num_tokens": 26250421.0,
+      "step": 325
+    },
+    {
+      "epoch": 0.3793103448275862,
+      "grad_norm": 0.03149571642279625,
+      "learning_rate": 2.858356472523572e-05,
+      "loss": 0.0137,
+      "mean_token_accuracy": 0.9947115182876587,
+      "num_tokens": 26653852.0,
+      "step": 330
+    },
+    {
+      "epoch": 0.3850574712643678,
+      "grad_norm": 0.05093760043382645,
+      "learning_rate": 2.8357372052122475e-05,
+      "loss": 0.0137,
+      "mean_token_accuracy": 0.99478600025177,
+      "num_tokens": 27056647.0,
+      "step": 335
+    },
+    {
+      "epoch": 0.39080459770114945,
+      "grad_norm": 0.04730261117219925,
+      "learning_rate": 2.8128995621458e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9948318123817443,
+      "num_tokens": 27460001.0,
+      "step": 340
+    },
+    {
+      "epoch": 0.39655172413793105,
+      "grad_norm": 0.035402730107307434,
+      "learning_rate": 2.7898518021414006e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9948247194290161,
+      "num_tokens": 27864720.0,
+      "step": 345
+    },
+    {
+      "epoch": 0.40229885057471265,
+      "grad_norm": 0.038121409714221954,
+      "learning_rate": 2.766602260001181e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.994686484336853,
+      "num_tokens": 28268488.0,
+      "step": 350
+    },
+    {
+      "epoch": 0.40804597701149425,
+      "grad_norm": 0.04125091806054115,
+      "learning_rate": 2.7431593434981018e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948553442955017,
+      "num_tokens": 28672697.0,
+      "step": 355
+    },
+    {
+      "epoch": 0.41379310344827586,
+      "grad_norm": 0.03373762220144272,
+      "learning_rate": 2.719531530335436e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.994598388671875,
+      "num_tokens": 29076416.0,
+      "step": 360
+    },
+    {
+      "epoch": 0.41954022988505746,
+      "grad_norm": 0.05404042452573776,
+      "learning_rate": 2.695727365080962e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9946176290512085,
+      "num_tokens": 29479332.0,
+      "step": 365
+    },
+    {
+      "epoch": 0.42528735632183906,
+      "grad_norm": 0.040191181004047394,
+      "learning_rate": 2.671755456076977e-05,
+      "loss": 0.0137,
+      "mean_token_accuracy": 0.9947783350944519,
+      "num_tokens": 29883389.0,
+      "step": 370
+    },
+    {
+      "epoch": 0.43103448275862066,
+      "grad_norm": 0.05008212476968765,
+      "learning_rate": 2.6476244723272458e-05,
+      "loss": 0.0141,
+      "mean_token_accuracy": 0.9946029782295227,
+      "num_tokens": 30287635.0,
+      "step": 375
+    },
+    {
+      "epoch": 0.4367816091954023,
+      "grad_norm": 0.03571382537484169,
+      "learning_rate": 2.6233431403620137e-05,
+      "loss": 0.014,
+      "mean_token_accuracy": 0.99459388256073,
+      "num_tokens": 30691089.0,
+      "step": 380
+    },
+    {
+      "epoch": 0.4425287356321839,
+      "grad_norm": 0.03077247366309166,
+      "learning_rate": 2.598920241082219e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948281884193421,
+      "num_tokens": 31095704.0,
+      "step": 385
+    },
+    {
+      "epoch": 0.4482758620689655,
+      "grad_norm": 0.02985704317688942,
+      "learning_rate": 2.5743646065840346e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9947766423225403,
+      "num_tokens": 31499708.0,
+      "step": 390
+    },
+    {
+      "epoch": 0.4540229885057471,
+      "grad_norm": 0.03301994130015373,
+      "learning_rate": 2.549685116964907e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9949437975883484,
+      "num_tokens": 31904187.0,
+      "step": 395
+    },
+    {
+      "epoch": 0.45977011494252873,
+      "grad_norm": 0.04135014861822128,
+      "learning_rate": 2.5248906971122276e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948192358016967,
+      "num_tokens": 32307081.0,
+      "step": 400
+    },
+    {
+      "epoch": 0.46551724137931033,
+      "grad_norm": 0.05234241113066673,
+      "learning_rate": 2.4999903134758113e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9947088360786438,
+      "num_tokens": 32712091.0,
+      "step": 405
+    },
+    {
+      "epoch": 0.47126436781609193,
+      "grad_norm": 0.03200434520840645,
+      "learning_rate": 2.4749929708253427e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9949337005615234,
+      "num_tokens": 33115832.0,
+      "step": 410
+    },
+    {
+      "epoch": 0.47701149425287354,
+      "grad_norm": 0.025907520204782486,
+      "learning_rate": 2.4499077089939677e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948367953300477,
+      "num_tokens": 33520665.0,
+      "step": 415
+    },
+    {
+      "epoch": 0.4827586206896552,
+      "grad_norm": 0.026905179023742676,
+      "learning_rate": 2.4247435996091983e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9948036670684814,
+      "num_tokens": 33924164.0,
+      "step": 420
+    },
+    {
+      "epoch": 0.4885057471264368,
+      "grad_norm": 0.0450819693505764,
+      "learning_rate": 2.3995097428123274e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9949346303939819,
+      "num_tokens": 34329060.0,
+      "step": 425
+    },
+    {
+      "epoch": 0.4942528735632184,
+      "grad_norm": 0.0394849069416523,
+      "learning_rate": 2.3742152639675284e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9948944926261902,
+      "num_tokens": 34733190.0,
+      "step": 430
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.02788865938782692,
+      "learning_rate": 2.3488693103618318e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9949518084526062,
+      "num_tokens": 35137740.0,
+      "step": 435
+    },
+    {
+      "epoch": 0.5057471264367817,
+      "grad_norm": 0.03805901110172272,
+      "learning_rate": 2.3234810478971772e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.994696569442749,
+      "num_tokens": 35542288.0,
+      "step": 440
+    },
+    {
+      "epoch": 0.5114942528735632,
+      "grad_norm": 0.032415661960840225,
+      "learning_rate": 2.2980596577757305e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9947757482528686,
+      "num_tokens": 35946058.0,
+      "step": 445
+    },
+    {
+      "epoch": 0.5172413793103449,
+      "grad_norm": 0.053135160356760025,
+      "learning_rate": 2.2726143331796738e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948907971382142,
+      "num_tokens": 36349724.0,
+      "step": 450
+    },
+    {
+      "epoch": 0.5229885057471264,
+      "grad_norm": 0.03707072511315346,
+      "learning_rate": 2.247154275946651e-05,
+      "loss": 0.0137,
+      "mean_token_accuracy": 0.9947781324386596,
+      "num_tokens": 36752494.0,
+      "step": 455
+    },
+    {
+      "epoch": 0.5287356321839081,
+      "grad_norm": 0.032205525785684586,
+      "learning_rate": 2.2216886932420993e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9945814847946167,
+      "num_tokens": 37156356.0,
+      "step": 460
+    },
+    {
+      "epoch": 0.5344827586206896,
+      "grad_norm": 0.0344030000269413,
+      "learning_rate": 2.19622679422964e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9949215650558472,
+      "num_tokens": 37560139.0,
+      "step": 465
+    },
+    {
+      "epoch": 0.5402298850574713,
+      "grad_norm": 0.024279696866869926,
+      "learning_rate": 2.170777786740754e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9947062373161316,
+      "num_tokens": 37965641.0,
+      "step": 470
+    },
+    {
+      "epoch": 0.5459770114942529,
+      "grad_norm": 0.030384989455342293,
+      "learning_rate": 2.14535087394494e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9948716759681702,
+      "num_tokens": 38369757.0,
+      "step": 475
+    },
+    {
+      "epoch": 0.5517241379310345,
+      "grad_norm": 0.04183518514037132,
+      "learning_rate": 2.1199552510215515e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9947738170623779,
+      "num_tokens": 38772941.0,
+      "step": 480
+    },
+    {
+      "epoch": 0.5574712643678161,
+      "grad_norm": 0.03046581894159317,
+      "learning_rate": 2.0946001018345306e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9949072957038879,
+      "num_tokens": 39178203.0,
+      "step": 485
+    },
+    {
+      "epoch": 0.5632183908045977,
+      "grad_norm": 0.09169202297925949,
+      "learning_rate": 2.0692945956112318e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9949472784996033,
+      "num_tokens": 39581967.0,
+      "step": 490
+    },
+    {
+      "epoch": 0.5689655172413793,
+      "grad_norm": 0.029076358303427696,
+      "learning_rate": 2.0440478836265358e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948386192321778,
+      "num_tokens": 39984021.0,
+      "step": 495
+    },
+    {
+      "epoch": 0.5747126436781609,
+      "grad_norm": 0.05347994714975357,
+      "learning_rate": 2.0188690958934603e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9947713971138,
+      "num_tokens": 40387785.0,
+      "step": 500
+    },
+    {
+      "epoch": 0.5804597701149425,
+      "grad_norm": 0.023038016632199287,
+      "learning_rate": 1.993767337861455e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948361039161682,
+      "num_tokens": 40791443.0,
+      "step": 505
+    },
+    {
+      "epoch": 0.5862068965517241,
+      "grad_norm": 0.03382337838411331,
+      "learning_rate": 1.9687516871235824e-05,
+      "loss": 0.0136,
+      "mean_token_accuracy": 0.9947430968284607,
+      "num_tokens": 41194964.0,
+      "step": 510
+    },
+    {
+      "epoch": 0.5919540229885057,
+      "grad_norm": 0.03841397538781166,
+      "learning_rate": 1.9438311901337744e-05,
+      "loss": 0.0139,
+      "mean_token_accuracy": 0.9947336792945862,
+      "num_tokens": 41599118.0,
+      "step": 515
+    },
+    {
+      "epoch": 0.5977011494252874,
+      "grad_norm": 0.03508458286523819,
+      "learning_rate": 1.9190148589353425e-05,
+      "loss": 0.0138,
+      "mean_token_accuracy": 0.9947627902030944,
+      "num_tokens": 42001873.0,
+      "step": 520
+    },
+    {
+      "epoch": 0.603448275862069,
+      "grad_norm": 0.0344594344496727,
+      "learning_rate": 1.8943116679019407e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9947917938232422,
+      "num_tokens": 42405413.0,
+      "step": 525
+    },
+    {
+      "epoch": 0.6091954022988506,
+      "grad_norm": 0.024883072823286057,
+      "learning_rate": 1.869730550492144e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948460817337036,
+      "num_tokens": 42809113.0,
+      "step": 530
+    },
+    {
+      "epoch": 0.6149425287356322,
+      "grad_norm": 0.03987499698996544,
+      "learning_rate": 1.8452803960188304e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9949107050895691,
+      "num_tokens": 43213404.0,
+      "step": 535
+    },
+    {
+      "epoch": 0.6206896551724138,
+      "grad_norm": 0.04916604235768318,
+      "learning_rate": 1.8209700464345162e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9949909687042237,
+      "num_tokens": 43616989.0,
+      "step": 540
+    },
+    {
+      "epoch": 0.6264367816091954,
+      "grad_norm": 0.025707852095365524,
+      "learning_rate": 1.796808293133832e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9949591755867004,
+      "num_tokens": 44020557.0,
+      "step": 545
+    },
+    {
+      "epoch": 0.632183908045977,
+      "grad_norm": 0.03211604803800583,
+      "learning_rate": 1.7728038737742696e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948301553726197,
+      "num_tokens": 44424704.0,
+      "step": 550
+    },
+    {
+      "epoch": 0.6379310344827587,
+      "grad_norm": 0.03153684735298157,
+      "learning_rate": 1.748965469116373e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9949110507965088,
+      "num_tokens": 44828304.0,
+      "step": 555
+    },
+    {
+      "epoch": 0.6436781609195402,
+      "grad_norm": 0.044283442199230194,
+      "learning_rate": 1.7253016998844934e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.994849705696106,
+      "num_tokens": 45231864.0,
+      "step": 560
+    },
+    {
+      "epoch": 0.6494252873563219,
+      "grad_norm": 0.03939127177000046,
+      "learning_rate": 1.7018211236492662e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948228478431702,
+      "num_tokens": 45636143.0,
+      "step": 565
+    },
+    {
+      "epoch": 0.6551724137931034,
+      "grad_norm": 0.02582518756389618,
+      "learning_rate": 1.678532231732921e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9948286533355712,
+      "num_tokens": 46041434.0,
+      "step": 570
+    },
+    {
+      "epoch": 0.6609195402298851,
+      "grad_norm": 0.03828392177820206,
+      "learning_rate": 1.655443446138546e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9949392199516296,
+      "num_tokens": 46445765.0,
+      "step": 575
+    },
+    {
+      "epoch": 0.6666666666666666,
+      "grad_norm": 0.026919830590486526,
+      "learning_rate": 1.632563116504431e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9947931051254273,
+      "num_tokens": 46850075.0,
+      "step": 580
+    },
+    {
+      "epoch": 0.6724137931034483,
+      "grad_norm": 0.031082892790436745,
+      "learning_rate": 1.6098995170845647e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.99483882188797,
+      "num_tokens": 47254907.0,
+      "step": 585
+    },
+    {
+      "epoch": 0.6781609195402298,
+      "grad_norm": 0.0375247485935688,
+      "learning_rate": 1.5874608437564096e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948214530944824,
+      "num_tokens": 47659312.0,
+      "step": 590
+    },
+    {
+      "epoch": 0.6839080459770115,
+      "grad_norm": 0.0356401652097702,
+      "learning_rate": 1.565255211057006e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9949233651161193,
+      "num_tokens": 48062760.0,
+      "step": 595
+    },
+    {
+      "epoch": 0.6896551724137931,
+      "grad_norm": 0.046727504581213,
+      "learning_rate": 1.5432906492485005e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9949892401695252,
+      "num_tokens": 48466976.0,
+      "step": 600
+    },
+    {
+      "epoch": 0.6954022988505747,
+      "grad_norm": 0.056539956480264664,
+      "learning_rate": 1.52157510141415e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9949196696281433,
+      "num_tokens": 48870906.0,
+      "step": 605
+    },
+    {
+      "epoch": 0.7011494252873564,
+      "grad_norm": 0.049220457673072815,
+      "learning_rate": 1.5001164205858465e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.994917893409729,
+      "num_tokens": 49275039.0,
+      "step": 610
+    },
+    {
+      "epoch": 0.7068965517241379,
+      "grad_norm": 0.04120581969618797,
+      "learning_rate": 1.4789223669042143e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.994994330406189,
+      "num_tokens": 49678668.0,
+      "step": 615
+    },
+    {
+      "epoch": 0.7126436781609196,
+      "grad_norm": 0.02625158056616783,
+      "learning_rate": 1.4580006048122943e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9949682116508484,
+      "num_tokens": 50083729.0,
+      "step": 620
+    },
+    {
+      "epoch": 0.7183908045977011,
+      "grad_norm": 0.05274219810962677,
+      "learning_rate": 1.43735870028384e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948260664939881,
+      "num_tokens": 50488215.0,
+      "step": 625
+    },
+    {
+      "epoch": 0.7241379310344828,
+      "grad_norm": 0.03466196358203888,
+      "learning_rate": 1.4170041180872212e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9949446439743042,
+      "num_tokens": 50892168.0,
+      "step": 630
+    },
+    {
+      "epoch": 0.7298850574712644,
+      "grad_norm": 0.03565268591046333,
+      "learning_rate": 1.3969442190859236e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9947892189025879,
+      "num_tokens": 51296446.0,
+      "step": 635
+    },
+    {
+      "epoch": 0.735632183908046,
+      "grad_norm": 0.03092004917562008,
+      "learning_rate": 1.3771862575766306e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9949315309524536,
+      "num_tokens": 51699117.0,
+      "step": 640
+    },
+    {
+      "epoch": 0.7413793103448276,
+      "grad_norm": 0.05381985753774643,
+      "learning_rate": 1.3577373786658359e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.995068883895874,
+      "num_tokens": 52103973.0,
+      "step": 645
+    },
+    {
+      "epoch": 0.7471264367816092,
+      "grad_norm": 0.03023788332939148,
+      "learning_rate": 1.33860461568595e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9947940468788147,
+      "num_tokens": 52507158.0,
+      "step": 650
+    },
+    {
+      "epoch": 0.7528735632183908,
+      "grad_norm": 0.04271196573972702,
+      "learning_rate": 1.3197948876518184e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9950397610664368,
+      "num_tokens": 52910307.0,
+      "step": 655
+    },
+    {
+      "epoch": 0.7586206896551724,
+      "grad_norm": 0.03678242862224579,
+      "learning_rate": 1.301314996758591e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9949288129806518,
+      "num_tokens": 53313505.0,
+      "step": 660
+    },
+    {
+      "epoch": 0.764367816091954,
+      "grad_norm": 0.024950511753559113,
+      "learning_rate": 1.283171625921829e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.994904899597168,
+      "num_tokens": 53718020.0,
+      "step": 665
+    },
+    {
+      "epoch": 0.7701149425287356,
+      "grad_norm": 0.0270620696246624,
+      "learning_rate": 1.2653713363607528e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948266863822937,
+      "num_tokens": 54119480.0,
+      "step": 670
+    },
+    {
+      "epoch": 0.7758620689655172,
+      "grad_norm": 0.027166401967406273,
+      "learning_rate": 1.2479205652254954e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9950602531433106,
+      "num_tokens": 54523011.0,
+      "step": 675
+    },
+    {
+      "epoch": 0.7816091954022989,
+      "grad_norm": 0.037310510873794556,
+      "learning_rate": 1.2308256232692228e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948535680770874,
+      "num_tokens": 54926247.0,
+      "step": 680
+    },
+    {
+      "epoch": 0.7873563218390804,
+      "grad_norm": 0.030097058042883873,
+      "learning_rate": 1.2140926925659696e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9949084520339966,
+      "num_tokens": 55329313.0,
+      "step": 685
+    },
+    {
+      "epoch": 0.7931034482758621,
+      "grad_norm": 0.031420208513736725,
+      "learning_rate": 1.1977278242749991e-05,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9948212742805481,
+      "num_tokens": 55732944.0,
+      "step": 690
+    },
+    {
+      "epoch": 0.7988505747126436,
+      "grad_norm": 0.04165807366371155,
+      "learning_rate": 1.1817369364525179e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9948533535003662,
+      "num_tokens": 56137191.0,
+      "step": 695
+    },
+    {
+      "epoch": 0.8045977011494253,
+      "grad_norm": 0.04301629588007927,
+      "learning_rate": 1.1661258119115136e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9947885513305664,
+      "num_tokens": 56540742.0,
+      "step": 700
+    },
+    {
+      "epoch": 0.8103448275862069,
+      "grad_norm": 0.030160659924149513,
+      "learning_rate": 1.1509000961305077e-05,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9950136542320251,
+      "num_tokens": 56944001.0,
+      "step": 705
+    },
+    {
+      "epoch": 0.8160919540229885,
+      "grad_norm": 0.03499389439821243,
+      "learning_rate": 1.1360652952119732e-05,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9947500705718995,
+      "num_tokens": 57348147.0,
+      "step": 710
+    },
+    {
+      "epoch": 0.8218390804597702,
+      "grad_norm": 0.024081142619252205,
+      "learning_rate": 1.1216267738911505e-05,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9946637511253357,
+      "num_tokens": 57751838.0,
+      "step": 715
+    },
+    {
+      "epoch": 0.8275862068965517,
+      "grad_norm": 0.027669720351696014,
+      "learning_rate": 1.1075897535959896e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9950794219970703,
+      "num_tokens": 58155972.0,
+      "step": 720
+    },
+    {
+      "epoch": 0.8333333333333334,
+      "grad_norm": 0.030040688812732697,
+      "learning_rate": 1.0939593105589159e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9951125979423523,
+      "num_tokens": 58561406.0,
+      "step": 725
+    },
+    {
+      "epoch": 0.8390804597701149,
+      "grad_norm": 0.0329650342464447,
+      "learning_rate": 1.0807403739811037e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.994926369190216,
+      "num_tokens": 58964866.0,
+      "step": 730
+    },
+    {
+      "epoch": 0.8448275862068966,
+      "grad_norm": 0.024679485708475113,
+      "learning_rate": 1.0679377242499197e-05,
+      "loss": 0.0125,
+      "mean_token_accuracy": 0.9951718211174011,
+      "num_tokens": 59368847.0,
+      "step": 735
+    },
+    {
+      "epoch": 0.8505747126436781,
+      "grad_norm": 0.06430592387914658,
+      "learning_rate": 1.0555559912101832e-05,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9949762463569641,
+      "num_tokens": 59772552.0,
+      "step": 740
+    },
+    {
+      "epoch": 0.8563218390804598,
+      "grad_norm": 0.028835134580731392,
+      "learning_rate": 1.043599652489869e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9949678659439087,
+      "num_tokens": 60176799.0,
+      "step": 745
+    },
+    {
+      "epoch": 0.8620689655172413,
+      "grad_norm": 0.026038020849227905,
+      "learning_rate": 1.0320730318808507e-05,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9948094844818115,
+      "num_tokens": 60581056.0,
+      "step": 750
+    },
+    {
+      "epoch": 0.867816091954023,
+      "grad_norm": 0.02826244942843914,
+      "learning_rate": 1.0209802977752852e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9948521971702575,
+      "num_tokens": 60985012.0,
+      "step": 755
+    },
+    {
+      "epoch": 0.8735632183908046,
+      "grad_norm": 0.037073444575071335,
+      "learning_rate": 1.0103254616581863e-05,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9949637532234192,
+      "num_tokens": 61388779.0,
+      "step": 760
+    },
+    {
+      "epoch": 0.8793103448275862,
+      "grad_norm": 0.032884031534194946,
+      "learning_rate": 1.0001123766567458e-05,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9949699401855469,
+      "num_tokens": 61793039.0,
+      "step": 765
+    },
+    {
+      "epoch": 0.8850574712643678,
+      "grad_norm": 0.03468252718448639,
+      "learning_rate": 9.90344736146919e-06,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9949258089065551,
+      "num_tokens": 62196261.0,
+      "step": 770
+    },
+    {
+      "epoch": 0.8908045977011494,
+      "grad_norm": 0.024934224784374237,
+      "learning_rate": 9.810260724177834e-06,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9948127031326294,
+      "num_tokens": 62599709.0,
+      "step": 775
+    },
+    {
+      "epoch": 0.896551724137931,
+      "grad_norm": 0.03356098756194115,
+      "learning_rate": 9.721597553941526e-06,
+      "loss": 0.0128,
+      "mean_token_accuracy": 0.9949464678764344,
+      "num_tokens": 63004030.0,
+      "step": 780
+    },
+    {
+      "epoch": 0.9022988505747126,
+      "grad_norm": 0.038369644433259964,
+      "learning_rate": 9.637489914179006e-06,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9949608087539673,
+      "num_tokens": 63408554.0,
+      "step": 785
+    },
+    {
+      "epoch": 0.9080459770114943,
+      "grad_norm": 0.03118038922548294,
+      "learning_rate": 9.557968220884502e-06,
+      "loss": 0.0134,
+      "mean_token_accuracy": 0.9947882890701294,
+      "num_tokens": 63811332.0,
+      "step": 790
+    },
+    {
+      "epoch": 0.9137931034482759,
+      "grad_norm": 0.036854732781648636,
+      "learning_rate": 9.483061231628312e-06,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9949264883995056,
+      "num_tokens": 64214657.0,
+      "step": 795
+    },
+    {
+      "epoch": 0.9195402298850575,
+      "grad_norm": 0.04122990742325783,
+      "learning_rate": 9.412796035157183e-06,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9948729038238525,
+      "num_tokens": 64617148.0,
+      "step": 800
+    },
+    {
+      "epoch": 0.9252873563218391,
+      "grad_norm": 0.05959238111972809,
+      "learning_rate": 9.347198041598134e-06,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9949862718582153,
+      "num_tokens": 65022073.0,
+      "step": 805
+    },
+    {
+      "epoch": 0.9310344827586207,
+      "grad_norm": 0.03828701004385948,
+      "learning_rate": 9.286290973269366e-06,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9947553157806397,
+      "num_tokens": 65425185.0,
+      "step": 810
+    },
+    {
+      "epoch": 0.9367816091954023,
+      "grad_norm": 0.03720645606517792,
+      "learning_rate": 9.230096856101493e-06,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9950950860977172,
+      "num_tokens": 65827745.0,
+      "step": 815
+    },
+    {
+      "epoch": 0.9425287356321839,
+      "grad_norm": 0.028984155505895615,
+      "learning_rate": 9.178636011672301e-06,
+      "loss": 0.013,
+      "mean_token_accuracy": 0.9948612570762634,
+      "num_tokens": 66231352.0,
+      "step": 820
+    },
+    {
+      "epoch": 0.9482758620689655,
+      "grad_norm": 0.029562933370471,
+      "learning_rate": 9.131927049857791e-06,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.9949148774147034,
+      "num_tokens": 66634894.0,
+      "step": 825
+    },
+    {
+      "epoch": 0.9540229885057471,
+      "grad_norm": 0.03893503174185753,
+      "learning_rate": 9.089986862102262e-06,
+      "loss": 0.0129,
+      "mean_token_accuracy": 0.994910728931427,
+      "num_tokens": 67038555.0,
+      "step": 830
+    },
+    {
+      "epoch": 0.9597701149425287,
+      "grad_norm": 0.025236055254936218,
+      "learning_rate": 9.052830615309845e-06,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.994949197769165,
+      "num_tokens": 67441898.0,
+      "step": 835
+    },
+    {
+      "epoch": 0.9655172413793104,
+      "grad_norm": 0.040895842015743256,
+      "learning_rate": 9.020471746359651e-06,
+      "loss": 0.0133,
+      "mean_token_accuracy": 0.9949271559715271,
+      "num_tokens": 67845282.0,
+      "step": 840
+    },
+    {
+      "epoch": 0.9712643678160919,
+      "grad_norm": 0.030874410644173622,
+      "learning_rate": 8.992921957246571e-06,
+      "loss": 0.0135,
+      "mean_token_accuracy": 0.9948145151138306,
+      "num_tokens": 68248415.0,
+      "step": 845
+    },
+    {
+      "epoch": 0.9770114942528736,
+      "grad_norm": 0.02959192916750908,
+      "learning_rate": 8.970191210849467e-06,
+      "loss": 0.0132,
+      "mean_token_accuracy": 0.9948099732398987,
+      "num_tokens": 68652350.0,
+      "step": 850
+    },
+    {
+      "epoch": 0.9827586206896551,
+      "grad_norm": 0.022801557555794716,
+      "learning_rate": 8.952287727328286e-06,
+      "loss": 0.0127,
+      "mean_token_accuracy": 0.995022428035736,
+      "num_tokens": 69057474.0,
+      "step": 855
+    },
+    {
+      "epoch": 0.9885057471264368,
+      "grad_norm": 0.02625376358628273,
+      "learning_rate": 8.939217981151367e-06,
+      "loss": 0.0131,
+      "mean_token_accuracy": 0.9948750615119935,
+      "num_tokens": 69460959.0,
+      "step": 860
+    },
+    {
+      "epoch": 0.9942528735632183,
+      "grad_norm": 0.027628207579255104,
+      "learning_rate": 8.930986698754095e-06,
+      "loss": 0.0126,
+      "mean_token_accuracy": 0.9950263857841491,
+      "num_tokens": 69864907.0,
+      "step": 865
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 870,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.641453571692036e+18,
+  "train_batch_size": 20,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:431015aa236dbffeeb367355d0c9213146daf6fa5414dba3344b2bd4cc0080b7
+size 6545