Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,202 @@
+---
+base_model: togethercomputer/Llama-4-Scout-17B-16E-Instruct_bnb_4bit
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.1

adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "togethercomputer/Llama-4-Scout-17B-16E-Instruct_bnb_4bit",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": [],
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "feed_forward.gate_proj",
+    "o_proj",
+    "q_proj",
+    "feed_forward.up_proj",
+    "shared_expert.down_proj",
+    "k_proj",
+    "v_proj",
+    "shared_expert.gate_proj",
+    "shared_expert.up_proj",
+    "feed_forward.down_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55b50915c65906fd14a222e91df6ca5b6f5c8a814d10a131dbdfc2b7eb48da37
+size 27262976

chat_template.jinja

@@ -0,0 +1,123 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- if strftime_now is defined %}
+        {%- set date_string = strftime_now("%d %b %Y") %}
+    {%- else %}
+        {%- set date_string = "26 Jul 2024" %}
+    {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}    
+    {%- if messages[0]['content'] is string %}
+        {%- set system_message = messages[0]['content']|trim %}
+    {%- else %}
+        {#- FIXME: The processor requires an array, always. #}
+        {%- set system_message = messages[0]['content'][0]['text']|trim %}
+    {%- endif %}
+    {%- set messages = messages[1:] %}
+    {%- set user_supplied_system_message = true %}
+{%- else %}
+    {%- set system_message = "" %}
+    {%- set user_supplied_system_message = false %}
+{%- endif %}
+
+{#- System message if the user supplied one #}
+{%- if user_supplied_system_message %}
+    {{- "<|header_start|>system<|header_end|>\n\n" }}
+    {%- if tools is not none %}
+        {{- "Environment: ipython\n" }}
+    {%- endif %}
+    {%- if tools is not none and not tools_in_user_message %}
+        {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+        {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+        {{- "Do not use variables.\n\n" }}
+        {%- for t in tools %}
+            {{- t | tojson(indent=4) }}
+            {{- "\n\n" }}
+        {%- endfor %}
+    {%- endif %}
+    {{- system_message }}
+    {{- "<|eot|>" }}
+{%- endif %}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|header_start|>user<|header_end|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+    {{- '<|header_start|>' + message['role'] + '<|header_end|>\n\n' }}
+        {%- if message['content'] is string %}
+            {{- message['content'] }}
+        {%- else %}
+            {%- for content in message['content'] %}
+                {%- if content['type'] == 'image' %}
+                    {{- '<|image|>' }}
+                {%- elif content['type'] == 'text' %}
+                    {{- content['text'] }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- "<|eot|>" }}
+    {%- elif 'tool_calls' in message and message.tool_calls|length > 0 %}
+       {{- '<|header_start|>assistant<|header_end|>\n\n' -}}
+       {{- '<|python_start|>' }}
+        {%- if message['content'] is string %}
+            {{- message['content'] }}
+        {%- else %}
+            {%- for content in message['content'] %}
+                {%- if content['type'] == 'image' %}
+                    {{- '<|image|>' }}
+                {%- elif content['type'] == 'text' %}
+                    {{- content['text'] }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+       {{- '<|python_end|>' }}
+        {%- for tool_call in message.tool_calls %}
+           {{- '{"name": "' + tool_call.function.name + '", ' }}
+           {{- '"parameters": ' }}
+           {{- tool_call.function.arguments | tojson }}
+           {{- "}" }}
+        {%- endfor %}
+       {{- "<|eot|>" }}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|header_start|>ipython<|header_end|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|header_start|>assistant<|header_end|>\n\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,215 @@
+{
+  "architectures": [
+    "Llama4ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_chunk_size": 8192,
+  "attention_dropout": 0.0,
+  "attn_scale": 0.1,
+  "attn_temperature_tuning": true,
+  "bos_token_id": 200000,
+  "cache_implementation": "hybrid",
+  "dtype": "bfloat16",
+  "eos_token_id": 200008,
+  "floor_scale": 8192,
+  "for_llm_compressor": false,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "interleave_moe_layer_step": 1,
+  "intermediate_size": 8192,
+  "intermediate_size_mlp": 16384,
+  "layer_types": [
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "chunked_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 10485760,
+  "model_type": "llama4_text",
+  "moe_layers": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    33,
+    34,
+    35,
+    36,
+    37,
+    38,
+    39,
+    40,
+    41,
+    42,
+    43,
+    44,
+    45,
+    46,
+    47
+  ],
+  "no_rope_layer_interval": 4,
+  "no_rope_layers": [
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0
+  ],
+  "num_attention_heads": 40,
+  "num_experts_per_tok": 1,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 8,
+  "num_local_experts": 16,
+  "output_router_logits": false,
+  "pad_token_id": 200018,
+  "quantization_config": {
+    "_load_in_4bit": true,
+    "_load_in_8bit": false,
+    "bnb_4bit_compute_dtype": "bfloat16",
+    "bnb_4bit_quant_storage": "bfloat16",
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": true,
+    "load_in_8bit": false,
+    "quant_method": "bitsandbytes"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "factor": 16.0,
+    "high_freq_factor": 1.0,
+    "low_freq_factor": 1.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000.0,
+    "rope_type": "llama3"
+  },
+  "router_aux_loss_coef": 0.001,
+  "router_jitter_noise": 0.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.10.1",
+  "use_cache": false,
+  "use_qk_norm": true,
+  "vocab_size": 202048,
+  "torch_dtype": "bfloat16"
+}

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|begin_of_text|>",
+  "eos_token": "<|eot|>",
+  "pad_token": "<|finetune_right_pad|>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:172c9eb4beafc72601690da3ccfcede5c2e6806a8d5ec1fca33e22acea8023a4
+size 27948578

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eot|>",
+  "is_local": false,
+  "local_files_only": true,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 10485760,
+  "pad_token": "<|finetune_right_pad|>",
+  "padding_side": "right",
+  "processor_class": "Llama4Processor",
+  "tokenizer_class": "TokenizersBackend"
+}

trainer_state.json

@@ -0,0 +1,1719 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 5.0,
+  "eval_steps": 47,
+  "global_step": 235,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.02127659574468085,
+      "grad_norm": 0.23393794894218445,
+      "learning_rate": 0.0,
+      "loss": 1.498046875,
+      "step": 1
+    },
+    {
+      "epoch": 0.0425531914893617,
+      "grad_norm": 0.35629600286483765,
+      "learning_rate": 2.5e-05,
+      "loss": 1.4755859375,
+      "step": 2
+    },
+    {
+      "epoch": 0.06382978723404255,
+      "grad_norm": 0.23825904726982117,
+      "learning_rate": 5e-05,
+      "loss": 1.46875,
+      "step": 3
+    },
+    {
+      "epoch": 0.0851063829787234,
+      "grad_norm": 0.2110234946012497,
+      "learning_rate": 7.500000000000001e-05,
+      "loss": 1.4609375,
+      "step": 4
+    },
+    {
+      "epoch": 0.10638297872340426,
+      "grad_norm": 0.11191354691982269,
+      "learning_rate": 0.0001,
+      "loss": 1.283203125,
+      "step": 5
+    },
+    {
+      "epoch": 0.1276595744680851,
+      "grad_norm": 0.10534299165010452,
+      "learning_rate": 0.000125,
+      "loss": 1.40625,
+      "step": 6
+    },
+    {
+      "epoch": 0.14893617021276595,
+      "grad_norm": 0.2014382779598236,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 1.341796875,
+      "step": 7
+    },
+    {
+      "epoch": 0.1702127659574468,
+      "grad_norm": 0.18545843660831451,
+      "learning_rate": 0.000175,
+      "loss": 1.21185302734375,
+      "step": 8
+    },
+    {
+      "epoch": 0.19148936170212766,
+      "grad_norm": 0.10173580050468445,
+      "learning_rate": 0.0002,
+      "loss": 1.2802734375,
+      "step": 9
+    },
+    {
+      "epoch": 0.2127659574468085,
+      "grad_norm": 0.07516055554151535,
+      "learning_rate": 0.00019999185989458338,
+      "loss": 1.357421875,
+      "step": 10
+    },
+    {
+      "epoch": 0.23404255319148937,
+      "grad_norm": 0.06378420442342758,
+      "learning_rate": 0.00019996744113742325,
+      "loss": 1.0709228515625,
+      "step": 11
+    },
+    {
+      "epoch": 0.2553191489361702,
+      "grad_norm": 0.07495936006307602,
+      "learning_rate": 0.00019992674840549045,
+      "loss": 1.283203125,
+      "step": 12
+    },
+    {
+      "epoch": 0.2765957446808511,
+      "grad_norm": 0.07314153015613556,
+      "learning_rate": 0.00019986978949274083,
+      "loss": 1.2734375,
+      "step": 13
+    },
+    {
+      "epoch": 0.2978723404255319,
+      "grad_norm": 0.05656423047184944,
+      "learning_rate": 0.00019979657530862276,
+      "loss": 1.1826171875,
+      "step": 14
+    },
+    {
+      "epoch": 0.3191489361702128,
+      "grad_norm": 0.0377945713698864,
+      "learning_rate": 0.00019970711987598753,
+      "loss": 1.2021484375,
+      "step": 15
+    },
+    {
+      "epoch": 0.3404255319148936,
+      "grad_norm": 0.03924940899014473,
+      "learning_rate": 0.00019960144032840334,
+      "loss": 1.2109375,
+      "step": 16
+    },
+    {
+      "epoch": 0.3617021276595745,
+      "grad_norm": 0.051506128162145615,
+      "learning_rate": 0.00019947955690687388,
+      "loss": 1.15380859375,
+      "step": 17
+    },
+    {
+      "epoch": 0.3829787234042553,
+      "grad_norm": 0.05868421867489815,
+      "learning_rate": 0.00019934149295596157,
+      "loss": 1.16259765625,
+      "step": 18
+    },
+    {
+      "epoch": 0.40425531914893614,
+      "grad_norm": 0.06149609386920929,
+      "learning_rate": 0.0001991872749193161,
+      "loss": 1.1240234375,
+      "step": 19
+    },
+    {
+      "epoch": 0.425531914893617,
+      "grad_norm": 0.039777159690856934,
+      "learning_rate": 0.00019901693233460987,
+      "loss": 1.224609375,
+      "step": 20
+    },
+    {
+      "epoch": 0.44680851063829785,
+      "grad_norm": 0.04213375225663185,
+      "learning_rate": 0.00019883049782788045,
+      "loss": 1.15869140625,
+      "step": 21
+    },
+    {
+      "epoch": 0.46808510638297873,
+      "grad_norm": 0.04403379186987877,
+      "learning_rate": 0.00019862800710728168,
+      "loss": 1.1796875,
+      "step": 22
+    },
+    {
+      "epoch": 0.48936170212765956,
+      "grad_norm": 0.03600035607814789,
+      "learning_rate": 0.00019840949895624438,
+      "loss": 1.2626953125,
+      "step": 23
+    },
+    {
+      "epoch": 0.5106382978723404,
+      "grad_norm": 0.03431978449225426,
+      "learning_rate": 0.0001981750152260482,
+      "loss": 1.2353515625,
+      "step": 24
+    },
+    {
+      "epoch": 0.5319148936170213,
+      "grad_norm": 0.0372556671500206,
+      "learning_rate": 0.00019792460082780568,
+      "loss": 1.1591796875,
+      "step": 25
+    },
+    {
+      "epoch": 0.5531914893617021,
+      "grad_norm": 0.028302719816565514,
+      "learning_rate": 0.0001976583037238603,
+      "loss": 1.197265625,
+      "step": 26
+    },
+    {
+      "epoch": 0.574468085106383,
+      "grad_norm": 0.026456288993358612,
+      "learning_rate": 0.00019737617491860028,
+      "loss": 1.1591796875,
+      "step": 27
+    },
+    {
+      "epoch": 0.5957446808510638,
+      "grad_norm": 0.03277663141489029,
+      "learning_rate": 0.00019707826844868952,
+      "loss": 1.166015625,
+      "step": 28
+    },
+    {
+      "epoch": 0.6170212765957447,
+      "grad_norm": 0.03094353713095188,
+      "learning_rate": 0.00019676464137271786,
+      "loss": 1.2197265625,
+      "step": 29
+    },
+    {
+      "epoch": 0.6382978723404256,
+      "grad_norm": 0.026416977867484093,
+      "learning_rate": 0.00019643535376027263,
+      "loss": 1.1787109375,
+      "step": 30
+    },
+    {
+      "epoch": 0.6595744680851063,
+      "grad_norm": 0.02268395945429802,
+      "learning_rate": 0.00019609046868043328,
+      "loss": 0.9776611328125,
+      "step": 31
+    },
+    {
+      "epoch": 0.6808510638297872,
+      "grad_norm": 0.027079889550805092,
+      "learning_rate": 0.00019573005218969175,
+      "loss": 1.19140625,
+      "step": 32
+    },
+    {
+      "epoch": 0.7021276595744681,
+      "grad_norm": 0.03761419281363487,
+      "learning_rate": 0.00019535417331930047,
+      "loss": 1.1455078125,
+      "step": 33
+    },
+    {
+      "epoch": 0.723404255319149,
+      "grad_norm": 0.029582127928733826,
+      "learning_rate": 0.0001949629040620507,
+      "loss": 1.03955078125,
+      "step": 34
+    },
+    {
+      "epoch": 0.7446808510638298,
+      "grad_norm": 0.027478355914354324,
+      "learning_rate": 0.0001945563193584836,
+      "loss": 1.15234375,
+      "step": 35
+    },
+    {
+      "epoch": 0.7659574468085106,
+      "grad_norm": 0.043285392224788666,
+      "learning_rate": 0.0001941344970825368,
+      "loss": 1.1416015625,
+      "step": 36
+    },
+    {
+      "epoch": 0.7872340425531915,
+      "grad_norm": 0.02737746201455593,
+      "learning_rate": 0.000193697518026629,
+      "loss": 1.0283203125,
+      "step": 37
+    },
+    {
+      "epoch": 0.8085106382978723,
+      "grad_norm": 0.027979858219623566,
+      "learning_rate": 0.00019324546588618552,
+      "loss": 1.169921875,
+      "step": 38
+    },
+    {
+      "epoch": 0.8297872340425532,
+      "grad_norm": 0.02747548371553421,
+      "learning_rate": 0.00019277842724360825,
+      "loss": 1.1865234375,
+      "step": 39
+    },
+    {
+      "epoch": 0.851063829787234,
+      "grad_norm": 0.0261751189827919,
+      "learning_rate": 0.00019229649155169207,
+      "loss": 1.1396484375,
+      "step": 40
+    },
+    {
+      "epoch": 0.8723404255319149,
+      "grad_norm": 0.026329027488827705,
+      "learning_rate": 0.00019179975111649196,
+      "loss": 1.177734375,
+      "step": 41
+    },
+    {
+      "epoch": 0.8936170212765957,
+      "grad_norm": 0.026858177036046982,
+      "learning_rate": 0.00019128830107964338,
+      "loss": 1.1123046875,
+      "step": 42
+    },
+    {
+      "epoch": 0.9148936170212766,
+      "grad_norm": 0.02628260664641857,
+      "learning_rate": 0.00019076223940013966,
+      "loss": 1.1318359375,
+      "step": 43
+    },
+    {
+      "epoch": 0.9361702127659575,
+      "grad_norm": 0.027040669694542885,
+      "learning_rate": 0.00019022166683556964,
+      "loss": 1.1669921875,
+      "step": 44
+    },
+    {
+      "epoch": 0.9574468085106383,
+      "grad_norm": 0.030067605897784233,
+      "learning_rate": 0.00018966668692281947,
+      "loss": 1.09228515625,
+      "step": 45
+    },
+    {
+      "epoch": 0.9787234042553191,
+      "grad_norm": 0.024822646751999855,
+      "learning_rate": 0.00018909740595824193,
+      "loss": 1.068359375,
+      "step": 46
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.04047278314828873,
+      "learning_rate": 0.00018851393297729732,
+      "loss": 1.095703125,
+      "step": 47
+    },
+    {
+      "epoch": 1.0,
+      "eval_loss": 1.130859375,
+      "eval_runtime": 5.5329,
+      "eval_samples_per_second": 0.542,
+      "eval_steps_per_second": 0.181,
+      "step": 47
+    },
+    {
+      "epoch": 1.0212765957446808,
+      "grad_norm": 0.02252831682562828,
+      "learning_rate": 0.0001879163797336698,
+      "loss": 1.0732421875,
+      "step": 48
+    },
+    {
+      "epoch": 1.0425531914893618,
+      "grad_norm": 0.03408103808760643,
+      "learning_rate": 0.00018730486067786265,
+      "loss": 1.04296875,
+      "step": 49
+    },
+    {
+      "epoch": 1.0638297872340425,
+      "grad_norm": 0.04057957977056503,
+      "learning_rate": 0.00018667949293527783,
+      "loss": 1.033203125,
+      "step": 50
+    },
+    {
+      "epoch": 1.0851063829787233,
+      "grad_norm": 0.03971686214208603,
+      "learning_rate": 0.0001860403962837823,
+      "loss": 1.068359375,
+      "step": 51
+    },
+    {
+      "epoch": 1.1063829787234043,
+      "grad_norm": 0.0383610762655735,
+      "learning_rate": 0.000185387693130767,
+      "loss": 1.01318359375,
+      "step": 52
+    },
+    {
+      "epoch": 1.127659574468085,
+      "grad_norm": 0.031908098608255386,
+      "learning_rate": 0.0001847215084897019,
+      "loss": 1.1396484375,
+      "step": 53
+    },
+    {
+      "epoch": 1.148936170212766,
+      "grad_norm": 0.0396200455725193,
+      "learning_rate": 0.00018404196995619175,
+      "loss": 1.1015625,
+      "step": 54
+    },
+    {
+      "epoch": 1.1702127659574468,
+      "grad_norm": 0.0316191166639328,
+      "learning_rate": 0.0001833492076835377,
+      "loss": 1.0084228515625,
+      "step": 55
+    },
+    {
+      "epoch": 1.1914893617021276,
+      "grad_norm": 0.023904742673039436,
+      "learning_rate": 0.0001826433543578087,
+      "loss": 1.0810546875,
+      "step": 56
+    },
+    {
+      "epoch": 1.2127659574468086,
+      "grad_norm": 0.03784968703985214,
+      "learning_rate": 0.00018192454517242768,
+      "loss": 1.1533203125,
+      "step": 57
+    },
+    {
+      "epoch": 1.2340425531914894,
+      "grad_norm": 0.03177719935774803,
+      "learning_rate": 0.00018119291780227816,
+      "loss": 0.9163818359375,
+      "step": 58
+    },
+    {
+      "epoch": 1.2553191489361701,
+      "grad_norm": 0.029503269121050835,
+      "learning_rate": 0.00018044861237733462,
+      "loss": 1.115234375,
+      "step": 59
+    },
+    {
+      "epoch": 1.2765957446808511,
+      "grad_norm": 0.03916515037417412,
+      "learning_rate": 0.00017969177145582346,
+      "loss": 1.10498046875,
+      "step": 60
+    },
+    {
+      "epoch": 1.297872340425532,
+      "grad_norm": 0.0334615521132946,
+      "learning_rate": 0.00017892253999691842,
+      "loss": 1.03173828125,
+      "step": 61
+    },
+    {
+      "epoch": 1.3191489361702127,
+      "grad_norm": 0.02631629817187786,
+      "learning_rate": 0.00017814106533297632,
+      "loss": 1.056640625,
+      "step": 62
+    },
+    {
+      "epoch": 1.3404255319148937,
+      "grad_norm": 0.03638143837451935,
+      "learning_rate": 0.00017734749714131827,
+      "loss": 1.0673828125,
+      "step": 63
+    },
+    {
+      "epoch": 1.3617021276595744,
+      "grad_norm": 0.03113693557679653,
+      "learning_rate": 0.0001765419874155616,
+      "loss": 1.00830078125,
+      "step": 64
+    },
+    {
+      "epoch": 1.3829787234042552,
+      "grad_norm": 0.03078601136803627,
+      "learning_rate": 0.00017572469043650808,
+      "loss": 1.03125,
+      "step": 65
+    },
+    {
+      "epoch": 1.4042553191489362,
+      "grad_norm": 0.02543475851416588,
+      "learning_rate": 0.00017489576274259472,
+      "loss": 1.0009765625,
+      "step": 66
+    },
+    {
+      "epoch": 1.425531914893617,
+      "grad_norm": 0.031025195494294167,
+      "learning_rate": 0.0001740553630999111,
+      "loss": 1.0927734375,
+      "step": 67
+    },
+    {
+      "epoch": 1.4468085106382977,
+      "grad_norm": 0.02687178924679756,
+      "learning_rate": 0.000173203652471791,
+      "loss": 1.03076171875,
+      "step": 68
+    },
+    {
+      "epoch": 1.4680851063829787,
+      "grad_norm": 0.02888372913002968,
+      "learning_rate": 0.0001723407939879826,
+      "loss": 1.05419921875,
+      "step": 69
+    },
+    {
+      "epoch": 1.4893617021276595,
+      "grad_norm": 0.0319877564907074,
+      "learning_rate": 0.00017146695291340392,
+      "loss": 1.140625,
+      "step": 70
+    },
+    {
+      "epoch": 1.5106382978723403,
+      "grad_norm": 0.0288713201880455,
+      "learning_rate": 0.00017058229661648947,
+      "loss": 1.1103515625,
+      "step": 71
+    },
+    {
+      "epoch": 1.5319148936170213,
+      "grad_norm": 0.033843640238046646,
+      "learning_rate": 0.00016968699453713367,
+      "loss": 1.0419921875,
+      "step": 72
+    },
+    {
+      "epoch": 1.5531914893617023,
+      "grad_norm": 0.02779719978570938,
+      "learning_rate": 0.00016878121815423782,
+      "loss": 1.08154296875,
+      "step": 73
+    },
+    {
+      "epoch": 1.574468085106383,
+      "grad_norm": 0.029009222984313965,
+      "learning_rate": 0.00016786514095286646,
+      "loss": 1.04931640625,
+      "step": 74
+    },
+    {
+      "epoch": 1.5957446808510638,
+      "grad_norm": 0.03975093364715576,
+      "learning_rate": 0.0001669389383910193,
+      "loss": 1.05126953125,
+      "step": 75
+    },
+    {
+      "epoch": 1.6170212765957448,
+      "grad_norm": 0.030499164015054703,
+      "learning_rate": 0.00016600278786602549,
+      "loss": 1.1044921875,
+      "step": 76
+    },
+    {
+      "epoch": 1.6382978723404256,
+      "grad_norm": 0.03535850718617439,
+      "learning_rate": 0.00016505686868056646,
+      "loss": 1.072265625,
+      "step": 77
+    },
+    {
+      "epoch": 1.6595744680851063,
+      "grad_norm": 0.028383029624819756,
+      "learning_rate": 0.00016410136200833365,
+      "loss": 0.89642333984375,
+      "step": 78
+    },
+    {
+      "epoch": 1.6808510638297873,
+      "grad_norm": 0.02772560343146324,
+      "learning_rate": 0.0001631364508593279,
+      "loss": 1.08740234375,
+      "step": 79
+    },
+    {
+      "epoch": 1.702127659574468,
+      "grad_norm": 0.0280919149518013,
+      "learning_rate": 0.00016216232004480745,
+      "loss": 1.0400390625,
+      "step": 80
+    },
+    {
+      "epoch": 1.7234042553191489,
+      "grad_norm": 0.03165218606591225,
+      "learning_rate": 0.0001611791561418903,
+      "loss": 0.950927734375,
+      "step": 81
+    },
+    {
+      "epoch": 1.7446808510638299,
+      "grad_norm": 0.031245512887835503,
+      "learning_rate": 0.00016018714745781912,
+      "loss": 1.0517578125,
+      "step": 82
+    },
+    {
+      "epoch": 1.7659574468085106,
+      "grad_norm": 0.02875831164419651,
+      "learning_rate": 0.00015918648399389424,
+      "loss": 1.03857421875,
+      "step": 83
+    },
+    {
+      "epoch": 1.7872340425531914,
+      "grad_norm": 0.02935316413640976,
+      "learning_rate": 0.00015817735740908227,
+      "loss": 0.9404296875,
+      "step": 84
+    },
+    {
+      "epoch": 1.8085106382978724,
+      "grad_norm": 0.028770264238119125,
+      "learning_rate": 0.00015715996098330742,
+      "loss": 1.07666015625,
+      "step": 85
+    },
+    {
+      "epoch": 1.8297872340425532,
+      "grad_norm": 0.03135351464152336,
+      "learning_rate": 0.00015613448958043223,
+      "loss": 1.09716796875,
+      "step": 86
+    },
+    {
+      "epoch": 1.851063829787234,
+      "grad_norm": 0.030160393565893173,
+      "learning_rate": 0.0001551011396109348,
+      "loss": 1.04296875,
+      "step": 87
+    },
+    {
+      "epoch": 1.872340425531915,
+      "grad_norm": 0.028330618515610695,
+      "learning_rate": 0.00015406010899429007,
+      "loss": 1.08642578125,
+      "step": 88
+    },
+    {
+      "epoch": 1.8936170212765957,
+      "grad_norm": 0.029394598677754402,
+      "learning_rate": 0.00015301159712106187,
+      "loss": 1.021484375,
+      "step": 89
+    },
+    {
+      "epoch": 1.9148936170212765,
+      "grad_norm": 0.029200712218880653,
+      "learning_rate": 0.0001519558048147132,
+      "loss": 1.037109375,
+      "step": 90
+    },
+    {
+      "epoch": 1.9361702127659575,
+      "grad_norm": 0.02774479240179062,
+      "learning_rate": 0.00015089293429314216,
+      "loss": 1.0791015625,
+      "step": 91
+    },
+    {
+      "epoch": 1.9574468085106385,
+      "grad_norm": 0.028711751103401184,
+      "learning_rate": 0.00014982318912995095,
+      "loss": 1.0048828125,
+      "step": 92
+    },
+    {
+      "epoch": 1.978723404255319,
+      "grad_norm": 0.02777498960494995,
+      "learning_rate": 0.00014874677421545477,
+      "loss": 0.98388671875,
+      "step": 93
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 0.029884032905101776,
+      "learning_rate": 0.00014766389571743894,
+      "loss": 0.990234375,
+      "step": 94
+    },
+    {
+      "epoch": 2.0,
+      "eval_loss": 1.08203125,
+      "eval_runtime": 5.516,
+      "eval_samples_per_second": 0.544,
+      "eval_steps_per_second": 0.181,
+      "step": 94
+    },
+    {
+      "epoch": 2.021276595744681,
+      "grad_norm": 0.029040010645985603,
+      "learning_rate": 0.00014657476104167115,
+      "loss": 0.97216796875,
+      "step": 95
+    },
+    {
+      "epoch": 2.0425531914893615,
+      "grad_norm": 0.0317298099398613,
+      "learning_rate": 0.00014547957879217668,
+      "loss": 0.966796875,
+      "step": 96
+    },
+    {
+      "epoch": 2.0638297872340425,
+      "grad_norm": 0.03343191742897034,
+      "learning_rate": 0.000144378558731284,
+      "loss": 0.94873046875,
+      "step": 97
+    },
+    {
+      "epoch": 2.0851063829787235,
+      "grad_norm": 0.030764257535338402,
+      "learning_rate": 0.00014327191173944874,
+      "loss": 0.984375,
+      "step": 98
+    },
+    {
+      "epoch": 2.106382978723404,
+      "grad_norm": 0.031886182725429535,
+      "learning_rate": 0.00014215984977486315,
+      "loss": 0.93408203125,
+      "step": 99
+    },
+    {
+      "epoch": 2.127659574468085,
+      "grad_norm": 0.029551995918154716,
+      "learning_rate": 0.00014104258583285946,
+      "loss": 1.05908203125,
+      "step": 100
+    },
+    {
+      "epoch": 2.148936170212766,
+      "grad_norm": 0.038718681782484055,
+      "learning_rate": 0.00013992033390511458,
+      "loss": 1.02294921875,
+      "step": 101
+    },
+    {
+      "epoch": 2.1702127659574466,
+      "grad_norm": 0.029679151251912117,
+      "learning_rate": 0.0001387933089386637,
+      "loss": 0.937591552734375,
+      "step": 102
+    },
+    {
+      "epoch": 2.1914893617021276,
+      "grad_norm": 0.041849151253700256,
+      "learning_rate": 0.00013766172679473114,
+      "loss": 1.00146484375,
+      "step": 103
+    },
+    {
+      "epoch": 2.2127659574468086,
+      "grad_norm": 0.030312562361359596,
+      "learning_rate": 0.00013652580420738607,
+      "loss": 1.07080078125,
+      "step": 104
+    },
+    {
+      "epoch": 2.2340425531914896,
+      "grad_norm": 0.030917160212993622,
+      "learning_rate": 0.00013538575874203095,
+      "loss": 0.8511962890625,
+      "step": 105
+    },
+    {
+      "epoch": 2.25531914893617,
+      "grad_norm": 0.03268856555223465,
+      "learning_rate": 0.00013424180875373093,
+      "loss": 1.03759765625,
+      "step": 106
+    },
+    {
+      "epoch": 2.276595744680851,
+      "grad_norm": 0.030307801440358162,
+      "learning_rate": 0.00013309417334539183,
+      "loss": 1.02734375,
+      "step": 107
+    },
+    {
+      "epoch": 2.297872340425532,
+      "grad_norm": 0.029849864542484283,
+      "learning_rate": 0.00013194307232579488,
+      "loss": 0.95849609375,
+      "step": 108
+    },
+    {
+      "epoch": 2.3191489361702127,
+      "grad_norm": 0.030392149463295937,
+      "learning_rate": 0.00013078872616749642,
+      "loss": 0.98291015625,
+      "step": 109
+    },
+    {
+      "epoch": 2.3404255319148937,
+      "grad_norm": 0.03271440789103508,
+      "learning_rate": 0.00012963135596460036,
+      "loss": 0.99072265625,
+      "step": 110
+    },
+    {
+      "epoch": 2.3617021276595747,
+      "grad_norm": 0.029442068189382553,
+      "learning_rate": 0.0001284711833904116,
+      "loss": 0.93798828125,
+      "step": 111
+    },
+    {
+      "epoch": 2.382978723404255,
+      "grad_norm": 0.03272737190127373,
+      "learning_rate": 0.00012730843065497847,
+      "loss": 0.96240234375,
+      "step": 112
+    },
+    {
+      "epoch": 2.404255319148936,
+      "grad_norm": 0.03141280636191368,
+      "learning_rate": 0.00012614332046253254,
+      "loss": 0.93408203125,
+      "step": 113
+    },
+    {
+      "epoch": 2.425531914893617,
+      "grad_norm": 0.033171191811561584,
+      "learning_rate": 0.00012497607596883352,
+      "loss": 1.01904296875,
+      "step": 114
+    },
+    {
+      "epoch": 2.4468085106382977,
+      "grad_norm": 0.03286304324865341,
+      "learning_rate": 0.0001238069207384279,
+      "loss": 0.95947265625,
+      "step": 115
+    },
+    {
+      "epoch": 2.4680851063829787,
+      "grad_norm": 0.12366930395364761,
+      "learning_rate": 0.00012263607870182918,
+      "loss": 0.982421875,
+      "step": 116
+    },
+    {
+      "epoch": 2.4893617021276597,
+      "grad_norm": 0.0373450331389904,
+      "learning_rate": 0.00012146377411262806,
+      "loss": 1.06884765625,
+      "step": 117
+    },
+    {
+      "epoch": 2.5106382978723403,
+      "grad_norm": 0.03353523090481758,
+      "learning_rate": 0.00012029023150454064,
+      "loss": 1.0361328125,
+      "step": 118
+    },
+    {
+      "epoch": 2.5319148936170213,
+      "grad_norm": 0.03350302577018738,
+      "learning_rate": 0.00011911567564840325,
+      "loss": 0.974609375,
+      "step": 119
+    },
+    {
+      "epoch": 2.5531914893617023,
+      "grad_norm": 0.03674326464533806,
+      "learning_rate": 0.00011794033150912135,
+      "loss": 1.01123046875,
+      "step": 120
+    },
+    {
+      "epoch": 2.574468085106383,
+      "grad_norm": 0.03349175676703453,
+      "learning_rate": 0.00011676442420258202,
+      "loss": 0.984375,
+      "step": 121
+    },
+    {
+      "epoch": 2.595744680851064,
+      "grad_norm": 0.03776804730296135,
+      "learning_rate": 0.00011558817895253658,
+      "loss": 0.9833984375,
+      "step": 122
+    },
+    {
+      "epoch": 2.617021276595745,
+      "grad_norm": 0.037562087178230286,
+      "learning_rate": 0.00011441182104746344,
+      "loss": 1.03369140625,
+      "step": 123
+    },
+    {
+      "epoch": 2.6382978723404253,
+      "grad_norm": 0.03163403272628784,
+      "learning_rate": 0.00011323557579741797,
+      "loss": 1.00439453125,
+      "step": 124
+    },
+    {
+      "epoch": 2.6595744680851063,
+      "grad_norm": 0.03091254085302353,
+      "learning_rate": 0.00011205966849087864,
+      "loss": 0.84625244140625,
+      "step": 125
+    },
+    {
+      "epoch": 2.6808510638297873,
+      "grad_norm": 0.043524835258722305,
+      "learning_rate": 0.00011088432435159679,
+      "loss": 1.0205078125,
+      "step": 126
+    },
+    {
+      "epoch": 2.702127659574468,
+      "grad_norm": 0.0412999652326107,
+      "learning_rate": 0.00010970976849545936,
+      "loss": 0.97412109375,
+      "step": 127
+    },
+    {
+      "epoch": 2.723404255319149,
+      "grad_norm": 0.03663191571831703,
+      "learning_rate": 0.00010853622588737198,
+      "loss": 0.89208984375,
+      "step": 128
+    },
+    {
+      "epoch": 2.74468085106383,
+      "grad_norm": 0.049419160932302475,
+      "learning_rate": 0.00010736392129817083,
+      "loss": 0.9853515625,
+      "step": 129
+    },
+    {
+      "epoch": 2.7659574468085104,
+      "grad_norm": 0.11463593691587448,
+      "learning_rate": 0.0001061930792615721,
+      "loss": 0.974609375,
+      "step": 130
+    },
+    {
+      "epoch": 2.7872340425531914,
+      "grad_norm": 0.032455358654260635,
+      "learning_rate": 0.0001050239240311665,
+      "loss": 0.88623046875,
+      "step": 131
+    },
+    {
+      "epoch": 2.8085106382978724,
+      "grad_norm": 0.03574486821889877,
+      "learning_rate": 0.00010385667953746748,
+      "loss": 1.01123046875,
+      "step": 132
+    },
+    {
+      "epoch": 2.829787234042553,
+      "grad_norm": 0.03459080681204796,
+      "learning_rate": 0.00010269156934502154,
+      "loss": 1.03564453125,
+      "step": 133
+    },
+    {
+      "epoch": 2.851063829787234,
+      "grad_norm": 0.031877003610134125,
+      "learning_rate": 0.00010152881660958842,
+      "loss": 0.98046875,
+      "step": 134
+    },
+    {
+      "epoch": 2.872340425531915,
+      "grad_norm": 0.03113487735390663,
+      "learning_rate": 0.00010036864403539965,
+      "loss": 1.0205078125,
+      "step": 135
+    },
+    {
+      "epoch": 2.8936170212765955,
+      "grad_norm": 0.03436438366770744,
+      "learning_rate": 9.921127383250361e-05,
+      "loss": 0.9609375,
+      "step": 136
+    },
+    {
+      "epoch": 2.9148936170212765,
+      "grad_norm": 0.032232243567705154,
+      "learning_rate": 9.805692767420514e-05,
+      "loss": 0.97265625,
+      "step": 137
+    },
+    {
+      "epoch": 2.9361702127659575,
+      "grad_norm": 0.033736906945705414,
+      "learning_rate": 9.690582665460819e-05,
+      "loss": 1.01611328125,
+      "step": 138
+    },
+    {
+      "epoch": 2.9574468085106385,
+      "grad_norm": 0.030821099877357483,
+      "learning_rate": 9.575819124626905e-05,
+      "loss": 0.94287109375,
+      "step": 139
+    },
+    {
+      "epoch": 2.978723404255319,
+      "grad_norm": 0.03239232301712036,
+      "learning_rate": 9.461424125796907e-05,
+      "loss": 0.9228515625,
+      "step": 140
+    },
+    {
+      "epoch": 3.0,
+      "grad_norm": 0.035801757127046585,
+      "learning_rate": 9.347419579261398e-05,
+      "loss": 0.91064453125,
+      "step": 141
+    },
+    {
+      "epoch": 3.0,
+      "eval_loss": 1.068359375,
+      "eval_runtime": 5.5027,
+      "eval_samples_per_second": 0.545,
+      "eval_steps_per_second": 0.182,
+      "step": 141
+    },
+    {
+      "epoch": 3.021276595744681,
+      "grad_norm": 0.031308405101299286,
+      "learning_rate": 9.233827320526887e-05,
+      "loss": 0.89306640625,
+      "step": 142
+    },
+    {
+      "epoch": 3.0425531914893615,
+      "grad_norm": 0.042995061725378036,
+      "learning_rate": 9.12066910613363e-05,
+      "loss": 0.912109375,
+      "step": 143
+    },
+    {
+      "epoch": 3.0638297872340425,
+      "grad_norm": 0.04316018521785736,
+      "learning_rate": 9.007966609488546e-05,
+      "loss": 0.89013671875,
+      "step": 144
+    },
+    {
+      "epoch": 3.0851063829787235,
+      "grad_norm": 0.03554543852806091,
+      "learning_rate": 8.895741416714056e-05,
+      "loss": 0.92919921875,
+      "step": 145
+    },
+    {
+      "epoch": 3.106382978723404,
+      "grad_norm": 0.04152313992381096,
+      "learning_rate": 8.78401502251369e-05,
+      "loss": 0.87744140625,
+      "step": 146
+    },
+    {
+      "epoch": 3.127659574468085,
+      "grad_norm": 0.039277493953704834,
+      "learning_rate": 8.672808826055129e-05,
+      "loss": 1.001953125,
+      "step": 147
+    },
+    {
+      "epoch": 3.148936170212766,
+      "grad_norm": 0.0363825261592865,
+      "learning_rate": 8.562144126871601e-05,
+      "loss": 0.96533203125,
+      "step": 148
+    },
+    {
+      "epoch": 3.1702127659574466,
+      "grad_norm": 0.040966104716062546,
+      "learning_rate": 8.452042120782332e-05,
+      "loss": 0.886871337890625,
+      "step": 149
+    },
+    {
+      "epoch": 3.1914893617021276,
+      "grad_norm": 0.041720908135175705,
+      "learning_rate": 8.342523895832885e-05,
+      "loss": 0.943359375,
+      "step": 150
+    },
+    {
+      "epoch": 3.2127659574468086,
+      "grad_norm": 0.03947775438427925,
+      "learning_rate": 8.23361042825611e-05,
+      "loss": 1.01416015625,
+      "step": 151
+    },
+    {
+      "epoch": 3.2340425531914896,
+      "grad_norm": 0.03244869410991669,
+      "learning_rate": 8.125322578454524e-05,
+      "loss": 0.8028564453125,
+      "step": 152
+    },
+    {
+      "epoch": 3.25531914893617,
+      "grad_norm": 0.04258759692311287,
+      "learning_rate": 8.017681087004904e-05,
+      "loss": 0.98095703125,
+      "step": 153
+    },
+    {
+      "epoch": 3.276595744680851,
+      "grad_norm": 0.03907298296689987,
+      "learning_rate": 7.910706570685784e-05,
+      "loss": 0.97314453125,
+      "step": 154
+    },
+    {
+      "epoch": 3.297872340425532,
+      "grad_norm": 0.03476712107658386,
+      "learning_rate": 7.804419518528682e-05,
+      "loss": 0.904296875,
+      "step": 155
+    },
+    {
+      "epoch": 3.3191489361702127,
+      "grad_norm": 0.042200688272714615,
+      "learning_rate": 7.698840287893813e-05,
+      "loss": 0.92822265625,
+      "step": 156
+    },
+    {
+      "epoch": 3.3404255319148937,
+      "grad_norm": 0.03728483244776726,
+      "learning_rate": 7.593989100570993e-05,
+      "loss": 0.9365234375,
+      "step": 157
+    },
+    {
+      "epoch": 3.3617021276595747,
+      "grad_norm": 0.0334622859954834,
+      "learning_rate": 7.48988603890652e-05,
+      "loss": 0.88623046875,
+      "step": 158
+    },
+    {
+      "epoch": 3.382978723404255,
+      "grad_norm": 0.03617995232343674,
+      "learning_rate": 7.386551041956778e-05,
+      "loss": 0.911865234375,
+      "step": 159
+    },
+    {
+      "epoch": 3.404255319148936,
+      "grad_norm": 0.03311219438910484,
+      "learning_rate": 7.284003901669259e-05,
+      "loss": 0.88427734375,
+      "step": 160
+    },
+    {
+      "epoch": 3.425531914893617,
+      "grad_norm": 0.037256091833114624,
+      "learning_rate": 7.182264259091774e-05,
+      "loss": 0.96630859375,
+      "step": 161
+    },
+    {
+      "epoch": 3.4468085106382977,
+      "grad_norm": 0.0344235897064209,
+      "learning_rate": 7.081351600610575e-05,
+      "loss": 0.908203125,
+      "step": 162
+    },
+    {
+      "epoch": 3.4680851063829787,
+      "grad_norm": 0.04023665934801102,
+      "learning_rate": 6.981285254218084e-05,
+      "loss": 0.9306640625,
+      "step": 163
+    },
+    {
+      "epoch": 3.4893617021276597,
+      "grad_norm": 0.038979027420282364,
+      "learning_rate": 6.882084385810971e-05,
+      "loss": 1.01513671875,
+      "step": 164
+    },
+    {
+      "epoch": 3.5106382978723403,
+      "grad_norm": 0.036239344626665115,
+      "learning_rate": 6.783767995519255e-05,
+      "loss": 0.982421875,
+      "step": 165
+    },
+    {
+      "epoch": 3.5319148936170213,
+      "grad_norm": 0.041207022964954376,
+      "learning_rate": 6.686354914067206e-05,
+      "loss": 0.9228515625,
+      "step": 166
+    },
+    {
+      "epoch": 3.5531914893617023,
+      "grad_norm": 0.03745558485388756,
+      "learning_rate": 6.589863799166639e-05,
+      "loss": 0.95849609375,
+      "step": 167
+    },
+    {
+      "epoch": 3.574468085106383,
+      "grad_norm": 0.03531312942504883,
+      "learning_rate": 6.494313131943356e-05,
+      "loss": 0.93505859375,
+      "step": 168
+    },
+    {
+      "epoch": 3.595744680851064,
+      "grad_norm": 0.03886643052101135,
+      "learning_rate": 6.399721213397453e-05,
+      "loss": 0.93212890625,
+      "step": 169
+    },
+    {
+      "epoch": 3.617021276595745,
+      "grad_norm": 0.038001321256160736,
+      "learning_rate": 6.306106160898076e-05,
+      "loss": 0.97900390625,
+      "step": 170
+    },
+    {
+      "epoch": 3.6382978723404253,
+      "grad_norm": 0.03643008694052696,
+      "learning_rate": 6.213485904713357e-05,
+      "loss": 0.95556640625,
+      "step": 171
+    },
+    {
+      "epoch": 3.6595744680851063,
+      "grad_norm": 0.036596424877643585,
+      "learning_rate": 6.121878184576219e-05,
+      "loss": 0.81085205078125,
+      "step": 172
+    },
+    {
+      "epoch": 3.6808510638297873,
+      "grad_norm": 0.042928438633680344,
+      "learning_rate": 6.031300546286637e-05,
+      "loss": 0.96875,
+      "step": 173
+    },
+    {
+      "epoch": 3.702127659574468,
+      "grad_norm": 0.041720643639564514,
+      "learning_rate": 5.941770338351056e-05,
+      "loss": 0.92431640625,
+      "step": 174
+    },
+    {
+      "epoch": 3.723404255319149,
+      "grad_norm": 0.041083648800849915,
+      "learning_rate": 5.853304708659608e-05,
+      "loss": 0.849609375,
+      "step": 175
+    },
+    {
+      "epoch": 3.74468085106383,
+      "grad_norm": 0.04229078069329262,
+      "learning_rate": 5.76592060120174e-05,
+      "loss": 0.93701171875,
+      "step": 176
+    },
+    {
+      "epoch": 3.7659574468085104,
+      "grad_norm": 0.052078451961278915,
+      "learning_rate": 5.6796347528209006e-05,
+      "loss": 0.92578125,
+      "step": 177
+    },
+    {
+      "epoch": 3.7872340425531914,
+      "grad_norm": 0.033227503299713135,
+      "learning_rate": 5.594463690008891e-05,
+      "loss": 0.84375,
+      "step": 178
+    },
+    {
+      "epoch": 3.8085106382978724,
+      "grad_norm": 0.042091649025678635,
+      "learning_rate": 5.51042372574053e-05,
+      "loss": 0.962890625,
+      "step": 179
+    },
+    {
+      "epoch": 3.829787234042553,
+      "grad_norm": 0.039128612726926804,
+      "learning_rate": 5.427530956349194e-05,
+      "loss": 0.9873046875,
+      "step": 180
+    },
+    {
+      "epoch": 3.851063829787234,
+      "grad_norm": 0.04030850902199745,
+      "learning_rate": 5.3458012584438444e-05,
+      "loss": 0.93359375,
+      "step": 181
+    },
+    {
+      "epoch": 3.872340425531915,
+      "grad_norm": 0.03772120922803879,
+      "learning_rate": 5.265250285868172e-05,
+      "loss": 0.97412109375,
+      "step": 182
+    },
+    {
+      "epoch": 3.8936170212765955,
+      "grad_norm": 0.03561572730541229,
+      "learning_rate": 5.1858934667023686e-05,
+      "loss": 0.9150390625,
+      "step": 183
+    },
+    {
+      "epoch": 3.9148936170212765,
+      "grad_norm": 0.040115874260663986,
+      "learning_rate": 5.10774600030816e-05,
+      "loss": 0.92724609375,
+      "step": 184
+    },
+    {
+      "epoch": 3.9361702127659575,
+      "grad_norm": 0.039529941976070404,
+      "learning_rate": 5.030822854417656e-05,
+      "loss": 0.9697265625,
+      "step": 185
+    },
+    {
+      "epoch": 3.9574468085106385,
+      "grad_norm": 0.0371088832616806,
+      "learning_rate": 4.955138762266541e-05,
+      "loss": 0.89892578125,
+      "step": 186
+    },
+    {
+      "epoch": 3.978723404255319,
+      "grad_norm": 0.04574158415198326,
+      "learning_rate": 4.880708219772186e-05,
+      "loss": 0.880859375,
+      "step": 187
+    },
+    {
+      "epoch": 4.0,
+      "grad_norm": 0.04366738349199295,
+      "learning_rate": 4.8075454827572316e-05,
+      "loss": 0.85009765625,
+      "step": 188
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 1.060546875,
+      "eval_runtime": 5.504,
+      "eval_samples_per_second": 0.545,
+      "eval_steps_per_second": 0.182,
+      "step": 188
+    },
+    {
+      "epoch": 4.0212765957446805,
+      "grad_norm": 0.03930988535284996,
+      "learning_rate": 4.7356645642191354e-05,
+      "loss": 0.8349609375,
+      "step": 189
+    },
+    {
+      "epoch": 4.042553191489362,
+      "grad_norm": 0.04266972094774246,
+      "learning_rate": 4.665079231646232e-05,
+      "loss": 0.87353515625,
+      "step": 190
+    },
+    {
+      "epoch": 4.0638297872340425,
+      "grad_norm": 0.05122610181570053,
+      "learning_rate": 4.5958030043808264e-05,
+      "loss": 0.84716796875,
+      "step": 191
+    },
+    {
+      "epoch": 4.085106382978723,
+      "grad_norm": 0.04046507924795151,
+      "learning_rate": 4.527849151029813e-05,
+      "loss": 0.88720703125,
+      "step": 192
+    },
+    {
+      "epoch": 4.1063829787234045,
+      "grad_norm": 0.04002128913998604,
+      "learning_rate": 4.461230686923302e-05,
+      "loss": 0.83642578125,
+      "step": 193
+    },
+    {
+      "epoch": 4.127659574468085,
+      "grad_norm": 0.04339519888162613,
+      "learning_rate": 4.395960371621773e-05,
+      "loss": 0.9609375,
+      "step": 194
+    },
+    {
+      "epoch": 4.148936170212766,
+      "grad_norm": 0.044231873005628586,
+      "learning_rate": 4.3320507064722186e-05,
+      "loss": 0.9248046875,
+      "step": 195
+    },
+    {
+      "epoch": 4.170212765957447,
+      "grad_norm": 0.03891272470355034,
+      "learning_rate": 4.269513932213737e-05,
+      "loss": 0.84954833984375,
+      "step": 196
+    },
+    {
+      "epoch": 4.191489361702128,
+      "grad_norm": 0.03978855907917023,
+      "learning_rate": 4.208362026633024e-05,
+      "loss": 0.89892578125,
+      "step": 197
+    },
+    {
+      "epoch": 4.212765957446808,
+      "grad_norm": 0.04313793033361435,
+      "learning_rate": 4.148606702270267e-05,
+      "loss": 0.97216796875,
+      "step": 198
+    },
+    {
+      "epoch": 4.23404255319149,
+      "grad_norm": 0.037154123187065125,
+      "learning_rate": 4.09025940417581e-05,
+      "loss": 0.77178955078125,
+      "step": 199
+    },
+    {
+      "epoch": 4.25531914893617,
+      "grad_norm": 0.04250561445951462,
+      "learning_rate": 4.033331307718056e-05,
+      "loss": 0.94140625,
+      "step": 200
+    },
+    {
+      "epoch": 4.276595744680851,
+      "grad_norm": 0.038513898849487305,
+      "learning_rate": 3.9778333164430374e-05,
+      "loss": 0.935546875,
+      "step": 201
+    },
+    {
+      "epoch": 4.297872340425532,
+      "grad_norm": 0.03767329454421997,
+      "learning_rate": 3.923776059986035e-05,
+      "loss": 0.8671875,
+      "step": 202
+    },
+    {
+      "epoch": 4.319148936170213,
+      "grad_norm": 0.03793912008404732,
+      "learning_rate": 3.871169892035663e-05,
+      "loss": 0.88818359375,
+      "step": 203
+    },
+    {
+      "epoch": 4.340425531914893,
+      "grad_norm": 0.03997759893536568,
+      "learning_rate": 3.820024888350806e-05,
+      "loss": 0.8994140625,
+      "step": 204
+    },
+    {
+      "epoch": 4.361702127659575,
+      "grad_norm": 0.036675214767456055,
+      "learning_rate": 3.7703508448307947e-05,
+      "loss": 0.84814453125,
+      "step": 205
+    },
+    {
+      "epoch": 4.382978723404255,
+      "grad_norm": 0.03600461781024933,
+      "learning_rate": 3.722157275639177e-05,
+      "loss": 0.879150390625,
+      "step": 206
+    },
+    {
+      "epoch": 4.404255319148936,
+      "grad_norm": 0.036929093301296234,
+      "learning_rate": 3.675453411381449e-05,
+      "loss": 0.85205078125,
+      "step": 207
+    },
+    {
+      "epoch": 4.425531914893617,
+      "grad_norm": 0.044605664908885956,
+      "learning_rate": 3.630248197337103e-05,
+      "loss": 0.92919921875,
+      "step": 208
+    },
+    {
+      "epoch": 4.446808510638298,
+      "grad_norm": 0.03827337920665741,
+      "learning_rate": 3.586550291746319e-05,
+      "loss": 0.8720703125,
+      "step": 209
+    },
+    {
+      "epoch": 4.468085106382979,
+      "grad_norm": 0.03790247067809105,
+      "learning_rate": 3.5443680641516394e-05,
+      "loss": 0.89404296875,
+      "step": 210
+    },
+    {
+      "epoch": 4.48936170212766,
+      "grad_norm": 0.03629126027226448,
+      "learning_rate": 3.503709593794933e-05,
+      "loss": 0.978515625,
+      "step": 211
+    },
+    {
+      "epoch": 4.51063829787234,
+      "grad_norm": 0.040744345635175705,
+      "learning_rate": 3.464582668069955e-05,
+      "loss": 0.9482421875,
+      "step": 212
+    },
+    {
+      "epoch": 4.531914893617021,
+      "grad_norm": 0.03703819215297699,
+      "learning_rate": 3.426994781030826e-05,
+      "loss": 0.89111328125,
+      "step": 213
+    },
+    {
+      "epoch": 4.553191489361702,
+      "grad_norm": 0.037714630365371704,
+      "learning_rate": 3.3909531319566716e-05,
+      "loss": 0.92578125,
+      "step": 214
+    },
+    {
+      "epoch": 4.574468085106383,
+      "grad_norm": 0.03681576997041702,
+      "learning_rate": 3.3564646239727386e-05,
+      "loss": 0.9033203125,
+      "step": 215
+    },
+    {
+      "epoch": 4.595744680851064,
+      "grad_norm": 0.039367783814668655,
+      "learning_rate": 3.323535862728216e-05,
+      "loss": 0.900390625,
+      "step": 216
+    },
+    {
+      "epoch": 4.617021276595745,
+      "grad_norm": 0.03795965760946274,
+      "learning_rate": 3.292173155131051e-05,
+      "loss": 0.9453125,
+      "step": 217
+    },
+    {
+      "epoch": 4.638297872340425,
+      "grad_norm": 0.03927023336291313,
+      "learning_rate": 3.262382508139974e-05,
+      "loss": 0.92431640625,
+      "step": 218
+    },
+    {
+      "epoch": 4.659574468085106,
+      "grad_norm": 0.03417414799332619,
+      "learning_rate": 3.234169627613971e-05,
+      "loss": 0.785308837890625,
+      "step": 219
+    },
+    {
+      "epoch": 4.680851063829787,
+      "grad_norm": 0.039658013731241226,
+      "learning_rate": 3.2075399172194336e-05,
+      "loss": 0.935546875,
+      "step": 220
+    },
+    {
+      "epoch": 4.702127659574468,
+      "grad_norm": 0.041840773075819016,
+      "learning_rate": 3.18249847739518e-05,
+      "loss": 0.89453125,
+      "step": 221
+    },
+    {
+      "epoch": 4.723404255319149,
+      "grad_norm": 0.036156002432107925,
+      "learning_rate": 3.1590501043755626e-05,
+      "loss": 0.82373046875,
+      "step": 222
+    },
+    {
+      "epoch": 4.74468085106383,
+      "grad_norm": 0.039113543927669525,
+      "learning_rate": 3.1371992892718335e-05,
+      "loss": 0.9052734375,
+      "step": 223
+    },
+    {
+      "epoch": 4.76595744680851,
+      "grad_norm": 0.037734437733888626,
+      "learning_rate": 3.116950217211955e-05,
+      "loss": 0.89794921875,
+      "step": 224
+    },
+    {
+      "epoch": 4.787234042553192,
+      "grad_norm": 0.03511641174554825,
+      "learning_rate": 3.098306766539013e-05,
+      "loss": 0.8203125,
+      "step": 225
+    },
+    {
+      "epoch": 4.808510638297872,
+      "grad_norm": 0.041939668357372284,
+      "learning_rate": 3.081272508068391e-05,
+      "loss": 0.93359375,
+      "step": 226
+    },
+    {
+      "epoch": 4.829787234042553,
+      "grad_norm": 0.04047448933124542,
+      "learning_rate": 3.065850704403845e-05,
+      "loss": 0.9580078125,
+      "step": 227
+    },
+    {
+      "epoch": 4.851063829787234,
+      "grad_norm": 0.03897187486290932,
+      "learning_rate": 3.0520443093126134e-05,
+      "loss": 0.9052734375,
+      "step": 228
+    },
+    {
+      "epoch": 4.872340425531915,
+      "grad_norm": 0.03869474306702614,
+      "learning_rate": 3.0398559671596688e-05,
+      "loss": 0.94677734375,
+      "step": 229
+    },
+    {
+      "epoch": 4.8936170212765955,
+      "grad_norm": 0.03831680491566658,
+      "learning_rate": 3.0292880124012486e-05,
+      "loss": 0.88671875,
+      "step": 230
+    },
+    {
+      "epoch": 4.914893617021277,
+      "grad_norm": 0.03840392827987671,
+      "learning_rate": 3.0203424691377225e-05,
+      "loss": 0.90087890625,
+      "step": 231
+    },
+    {
+      "epoch": 4.9361702127659575,
+      "grad_norm": 0.03766055777668953,
+      "learning_rate": 3.013021050725918e-05,
+      "loss": 0.9443359375,
+      "step": 232
+    },
+    {
+      "epoch": 4.957446808510638,
+      "grad_norm": 0.039559632539749146,
+      "learning_rate": 3.0073251594509578e-05,
+      "loss": 0.8740234375,
+      "step": 233
+    },
+    {
+      "epoch": 4.9787234042553195,
+      "grad_norm": 0.05103755742311478,
+      "learning_rate": 3.0032558862576754e-05,
+      "loss": 0.85498046875,
+      "step": 234
+    },
+    {
+      "epoch": 5.0,
+      "grad_norm": 0.046121180057525635,
+      "learning_rate": 3.000814010541664e-05,
+      "loss": 0.814453125,
+      "step": 235
+    },
+    {
+      "epoch": 5.0,
+      "eval_loss": 1.0634765625,
+      "eval_runtime": 5.5227,
+      "eval_samples_per_second": 0.543,
+      "eval_steps_per_second": 0.181,
+      "step": 235
+    }
+  ],
+  "logging_steps": 1.0,
+  "max_steps": 235,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 0,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.046024116602929e+19,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}