Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoint-1/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-10986/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-16479/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-21972/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-27465/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-32958/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-38451/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-43939/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoint-5493/tokenizer.json filter=lfs diff=lfs merge=lfs -text

checkpoint-1/README.md

@@ -0,0 +1,208 @@
+---
+base_model: mistralai/Mistral-Nemo-Instruct-2407
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- axolotl
+- base_model:adapter:mistralai/Mistral-Nemo-Instruct-2407
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-1/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-Nemo-Instruct-2407",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "down_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": [],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-1/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e89dd0c5654ee8cd416433ae7c9e815653326cb541212da803c5cf5d0a8d5e6
+size 456206152

checkpoint-1/chat_template.jinja

@@ -0,0 +1,87 @@
+{%- if messages[0]["role"] == "system" %}
+    {%- set system_message = messages[0]["content"] %}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
+{%- set ns = namespace() %}
+{%- set ns.index = 0 %}
+{%- for message in loop_messages %}
+    {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
+        {%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
+            {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+        {%- endif %}
+        {%- set ns.index = ns.index + 1 %}
+    {%- endif %}
+{%- endfor %}
+
+{{- bos_token }}
+{%- for message in loop_messages %}
+    {%- if message["role"] == "user" %}
+        {%- if tools is not none and (message == user_messages[-1]) %}
+            {{- "[AVAILABLE_TOOLS][" }}
+            {%- for tool in tools %}
+                {%- set tool = tool.function %}
+                {{- '{"type": "function", "function": {' }}
+                {%- for key, val in tool.items() if key != "return" %}
+                    {%- if val is string %}
+                        {{- '"' + key + '": "' + val + '"' }}
+                    {%- else %}
+                        {{- '"' + key + '": ' + val|tojson }}
+                    {%- endif %}
+                    {%- if not loop.last %}
+                        {{- ", " }}
+                    {%- endif %}
+                {%- endfor %}
+                {{- "}}" }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- else %}
+                    {{- "]" }}
+                {%- endif %}
+            {%- endfor %}
+            {{- "[/AVAILABLE_TOOLS]" }}
+            {%- endif %}
+        {%- if loop.last and system_message is defined %}
+            {{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
+        {%- else %}
+            {{- "[INST]" + message["content"] + "[/INST]" }}
+        {%- endif %}
+    {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
+        {{- "[TOOL_CALLS][" }}
+        {%- for tool_call in message.tool_calls %}
+            {%- set out = tool_call.function|tojson %}
+            {{- out[:-1] }}
+            {%- if not tool_call.id is defined or tool_call.id|length != 9 %}
+                {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+            {%- endif %}
+            {{- ', "id": "' + tool_call.id + '"}' }}
+            {%- if not loop.last %}
+                {{- ", " }}
+            {%- else %}
+                {{- "]" + eos_token }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif message["role"] == "assistant" %}
+        {{- message["content"] + eos_token}}
+    {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+        {%- if message.content is defined and message.content.content is defined %}
+            {%- set content = message.content.content %}
+        {%- else %}
+            {%- set content = message.content %}
+        {%- endif %}
+        {{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
+        {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
+            {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+        {%- endif %}
+        {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
+    {%- else %}
+        {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+    {%- endif %}
+{%- endfor %}

checkpoint-1/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2020bdc677e8437ee65e80d24ee17709a4210e5d2875236fd6f91fea9d6c8ad
+size 232287141

checkpoint-1/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f6887d8424819036310a2e84d986db1032a31ea5b39b32a3ea2f1773e3211c5
+size 14645

checkpoint-1/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3019aa1e0ed410fa59cea14f0b1996ce34ccac1bc0015d8a304fe483015927b8
+size 1465

checkpoint-1/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0240ce510f08e6c2041724e9043e33be9d251d1e4a4d94eb68cd47b954b61d2
+size 17078292

checkpoint-1/trainer_state.json

@@ -0,0 +1,45 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 9.103322712790169e-05,
+  "eval_steps": 2197,
+  "global_step": 1,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0,
+      "eval_loss": 1.9759739637374878,
+      "eval_runtime": 197.0423,
+      "eval_samples_per_second": 5.07,
+      "eval_steps_per_second": 0.365,
+      "memory/device_reserved (GiB)": 77.68,
+      "memory/max_active (GiB)": 76.86,
+      "memory/max_allocated (GiB)": 76.86,
+      "step": 0
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 43939,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 5493,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1898948070604800.0,
+  "train_batch_size": 9,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34a581bf9b76f74a5a4e25651fbaa01e61b75b19cbbfa8df2370c772924bc5d7
+size 11409

checkpoint-10986/README.md

@@ -0,0 +1,208 @@
+---
+base_model: mistralai/Mistral-Nemo-Instruct-2407
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- axolotl
+- base_model:adapter:mistralai/Mistral-Nemo-Instruct-2407
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-10986/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-Nemo-Instruct-2407",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "down_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": [],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-10986/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17511e1eacc55416b1c06b20fa7a42eb3a1a1f1ba4bca9822b44be3812400ffb
+size 456206152

checkpoint-10986/chat_template.jinja

@@ -0,0 +1,87 @@
+{%- if messages[0]["role"] == "system" %}
+    {%- set system_message = messages[0]["content"] %}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
+{%- set ns = namespace() %}
+{%- set ns.index = 0 %}
+{%- for message in loop_messages %}
+    {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
+        {%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
+            {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+        {%- endif %}
+        {%- set ns.index = ns.index + 1 %}
+    {%- endif %}
+{%- endfor %}
+
+{{- bos_token }}
+{%- for message in loop_messages %}
+    {%- if message["role"] == "user" %}
+        {%- if tools is not none and (message == user_messages[-1]) %}
+            {{- "[AVAILABLE_TOOLS][" }}
+            {%- for tool in tools %}
+                {%- set tool = tool.function %}
+                {{- '{"type": "function", "function": {' }}
+                {%- for key, val in tool.items() if key != "return" %}
+                    {%- if val is string %}
+                        {{- '"' + key + '": "' + val + '"' }}
+                    {%- else %}
+                        {{- '"' + key + '": ' + val|tojson }}
+                    {%- endif %}
+                    {%- if not loop.last %}
+                        {{- ", " }}
+                    {%- endif %}
+                {%- endfor %}
+                {{- "}}" }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- else %}
+                    {{- "]" }}
+                {%- endif %}
+            {%- endfor %}
+            {{- "[/AVAILABLE_TOOLS]" }}
+            {%- endif %}
+        {%- if loop.last and system_message is defined %}
+            {{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
+        {%- else %}
+            {{- "[INST]" + message["content"] + "[/INST]" }}
+        {%- endif %}
+    {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
+        {{- "[TOOL_CALLS][" }}
+        {%- for tool_call in message.tool_calls %}
+            {%- set out = tool_call.function|tojson %}
+            {{- out[:-1] }}
+            {%- if not tool_call.id is defined or tool_call.id|length != 9 %}
+                {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+            {%- endif %}
+            {{- ', "id": "' + tool_call.id + '"}' }}
+            {%- if not loop.last %}
+                {{- ", " }}
+            {%- else %}
+                {{- "]" + eos_token }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif message["role"] == "assistant" %}
+        {{- message["content"] + eos_token}}
+    {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+        {%- if message.content is defined and message.content.content is defined %}
+            {%- set content = message.content.content %}
+        {%- else %}
+            {%- set content = message.content %}
+        {%- endif %}
+        {{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
+        {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
+            {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+        {%- endif %}
+        {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
+    {%- else %}
+        {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+    {%- endif %}
+{%- endfor %}

checkpoint-10986/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be39a2e92b5a3c1cfcc66e38e255f6e12ebe66815afe5b4bf230eaa1eb0a087d
+size 232287717

checkpoint-10986/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8929b95342e38b16a2a31247a473d14ad624ba7d1384229ec5b7cd6725e55df
+size 14645

checkpoint-10986/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1bdf5ba82a58c2dfaf9897817a6beac0a5b8b6d81f717f715327cbba1bbb725
+size 1465

checkpoint-10986/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-10986/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0240ce510f08e6c2041724e9043e33be9d251d1e4a4d94eb68cd47b954b61d2
+size 17078292

checkpoint-10986/trainer_state.json

@@ -0,0 +1,2509 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.000091033227128,
+  "eval_steps": 2197,
+  "global_step": 10986,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0,
+      "eval_loss": 1.9759739637374878,
+      "eval_runtime": 197.0423,
+      "eval_samples_per_second": 5.07,
+      "eval_steps_per_second": 0.365,
+      "memory/device_reserved (GiB)": 77.68,
+      "memory/max_active (GiB)": 76.86,
+      "memory/max_allocated (GiB)": 76.86,
+      "step": 0
+    },
+    {
+      "epoch": 0.004551661356395084,
+      "grad_norm": 1.4988383054733276,
+      "learning_rate": 5.577054404734805e-07,
+      "loss": 1.9687,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 50,
+      "tokens_per_second_per_gpu": 1534.89
+    },
+    {
+      "epoch": 0.009103322712790168,
+      "grad_norm": 1.205540657043457,
+      "learning_rate": 1.1267926246300935e-06,
+      "loss": 1.9557,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 100,
+      "tokens_per_second_per_gpu": 512.71
+    },
+    {
+      "epoch": 0.013654984069185253,
+      "grad_norm": 1.4495829343795776,
+      "learning_rate": 1.6958798087867062e-06,
+      "loss": 1.8251,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 150,
+      "tokens_per_second_per_gpu": 492.64
+    },
+    {
+      "epoch": 0.018206645425580335,
+      "grad_norm": 0.9070694446563721,
+      "learning_rate": 2.264966992943319e-06,
+      "loss": 1.6389,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 200,
+      "tokens_per_second_per_gpu": 513.79
+    },
+    {
+      "epoch": 0.02275830678197542,
+      "grad_norm": 1.1880543231964111,
+      "learning_rate": 2.834054177099932e-06,
+      "loss": 1.4838,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 250,
+      "tokens_per_second_per_gpu": 489.92
+    },
+    {
+      "epoch": 0.027309968138370506,
+      "grad_norm": 1.474748134613037,
+      "learning_rate": 3.4031413612565448e-06,
+      "loss": 1.4042,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 300,
+      "tokens_per_second_per_gpu": 485.95
+    },
+    {
+      "epoch": 0.03186162949476559,
+      "grad_norm": 1.490429401397705,
+      "learning_rate": 3.972228545413158e-06,
+      "loss": 1.3435,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 350,
+      "tokens_per_second_per_gpu": 482.48
+    },
+    {
+      "epoch": 0.03641329085116067,
+      "grad_norm": 1.9503854513168335,
+      "learning_rate": 4.54131572956977e-06,
+      "loss": 1.3146,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 400,
+      "tokens_per_second_per_gpu": 472.38
+    },
+    {
+      "epoch": 0.040964952207555756,
+      "grad_norm": 1.4551211595535278,
+      "learning_rate": 5.110402913726383e-06,
+      "loss": 1.2806,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 450,
+      "tokens_per_second_per_gpu": 484.66
+    },
+    {
+      "epoch": 0.04551661356395084,
+      "grad_norm": 1.7765358686447144,
+      "learning_rate": 5.679490097882996e-06,
+      "loss": 1.2785,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 500,
+      "tokens_per_second_per_gpu": 459.31
+    },
+    {
+      "epoch": 0.05006827492034593,
+      "grad_norm": 1.6205016374588013,
+      "learning_rate": 6.248577282039609e-06,
+      "loss": 1.2962,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 550,
+      "tokens_per_second_per_gpu": 496.89
+    },
+    {
+      "epoch": 0.05461993627674101,
+      "grad_norm": 2.360264539718628,
+      "learning_rate": 6.8176644661962216e-06,
+      "loss": 1.2698,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 600,
+      "tokens_per_second_per_gpu": 535.06
+    },
+    {
+      "epoch": 0.05917159763313609,
+      "grad_norm": 1.9422169923782349,
+      "learning_rate": 7.386751650352834e-06,
+      "loss": 1.2719,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 650,
+      "tokens_per_second_per_gpu": 469.37
+    },
+    {
+      "epoch": 0.06372325898953118,
+      "grad_norm": 1.6929277181625366,
+      "learning_rate": 7.955838834509448e-06,
+      "loss": 1.2423,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 700,
+      "tokens_per_second_per_gpu": 469.51
+    },
+    {
+      "epoch": 0.06827492034592626,
+      "grad_norm": 2.481524705886841,
+      "learning_rate": 8.52492601866606e-06,
+      "loss": 1.2189,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 750,
+      "tokens_per_second_per_gpu": 443.79
+    },
+    {
+      "epoch": 0.07282658170232134,
+      "grad_norm": 1.7966183423995972,
+      "learning_rate": 9.094013202822673e-06,
+      "loss": 1.2364,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 800,
+      "tokens_per_second_per_gpu": 471.56
+    },
+    {
+      "epoch": 0.07737824305871643,
+      "grad_norm": 1.7296600341796875,
+      "learning_rate": 9.663100386979286e-06,
+      "loss": 1.183,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 850,
+      "tokens_per_second_per_gpu": 476.21
+    },
+    {
+      "epoch": 0.08192990441511151,
+      "grad_norm": 1.7888613939285278,
+      "learning_rate": 1.0232187571135898e-05,
+      "loss": 1.1902,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 900,
+      "tokens_per_second_per_gpu": 496.55
+    },
+    {
+      "epoch": 0.0864815657715066,
+      "grad_norm": 1.9669216871261597,
+      "learning_rate": 1.0801274755292511e-05,
+      "loss": 1.2042,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 950,
+      "tokens_per_second_per_gpu": 494.8
+    },
+    {
+      "epoch": 0.09103322712790168,
+      "grad_norm": 2.7225213050842285,
+      "learning_rate": 1.1370361939449125e-05,
+      "loss": 1.1681,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1000,
+      "tokens_per_second_per_gpu": 469.73
+    },
+    {
+      "epoch": 0.09558488848429676,
+      "grad_norm": 2.091661214828491,
+      "learning_rate": 1.1939449123605736e-05,
+      "loss": 1.1957,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1050,
+      "tokens_per_second_per_gpu": 501.85
+    },
+    {
+      "epoch": 0.10013654984069185,
+      "grad_norm": 2.423203945159912,
+      "learning_rate": 1.250853630776235e-05,
+      "loss": 1.2117,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1100,
+      "tokens_per_second_per_gpu": 475.58
+    },
+    {
+      "epoch": 0.10468821119708693,
+      "grad_norm": 1.9933820962905884,
+      "learning_rate": 1.3077623491918964e-05,
+      "loss": 1.2049,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1150,
+      "tokens_per_second_per_gpu": 484.91
+    },
+    {
+      "epoch": 0.10923987255348203,
+      "grad_norm": 1.9259192943572998,
+      "learning_rate": 1.3646710676075574e-05,
+      "loss": 1.1552,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1200,
+      "tokens_per_second_per_gpu": 464.56
+    },
+    {
+      "epoch": 0.1137915339098771,
+      "grad_norm": 1.8878772258758545,
+      "learning_rate": 1.4215797860232188e-05,
+      "loss": 1.1847,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1250,
+      "tokens_per_second_per_gpu": 532.02
+    },
+    {
+      "epoch": 0.11834319526627218,
+      "grad_norm": 2.4258198738098145,
+      "learning_rate": 1.4784885044388803e-05,
+      "loss": 1.1844,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1300,
+      "tokens_per_second_per_gpu": 480.37
+    },
+    {
+      "epoch": 0.12289485662266728,
+      "grad_norm": 1.9142098426818848,
+      "learning_rate": 1.5353972228545412e-05,
+      "loss": 1.1706,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1350,
+      "tokens_per_second_per_gpu": 484.91
+    },
+    {
+      "epoch": 0.12744651797906237,
+      "grad_norm": 1.8054569959640503,
+      "learning_rate": 1.5923059412702026e-05,
+      "loss": 1.1494,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1400,
+      "tokens_per_second_per_gpu": 469.8
+    },
+    {
+      "epoch": 0.13199817933545743,
+      "grad_norm": 1.9961028099060059,
+      "learning_rate": 1.649214659685864e-05,
+      "loss": 1.1767,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1450,
+      "tokens_per_second_per_gpu": 494.13
+    },
+    {
+      "epoch": 0.13654984069185253,
+      "grad_norm": 1.7714760303497314,
+      "learning_rate": 1.7061233781015252e-05,
+      "loss": 1.1622,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1500,
+      "tokens_per_second_per_gpu": 482.39
+    },
+    {
+      "epoch": 0.14110150204824762,
+      "grad_norm": 1.944865345954895,
+      "learning_rate": 1.7630320965171866e-05,
+      "loss": 1.1754,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1550,
+      "tokens_per_second_per_gpu": 482.41
+    },
+    {
+      "epoch": 0.14565316340464268,
+      "grad_norm": 1.8010004758834839,
+      "learning_rate": 1.819940814932848e-05,
+      "loss": 1.1597,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1600,
+      "tokens_per_second_per_gpu": 477.14
+    },
+    {
+      "epoch": 0.15020482476103778,
+      "grad_norm": 2.0305533409118652,
+      "learning_rate": 1.8768495333485092e-05,
+      "loss": 1.1533,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1650,
+      "tokens_per_second_per_gpu": 484.22
+    },
+    {
+      "epoch": 0.15475648611743287,
+      "grad_norm": 1.8290001153945923,
+      "learning_rate": 1.9337582517641702e-05,
+      "loss": 1.1548,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1700,
+      "tokens_per_second_per_gpu": 492.04
+    },
+    {
+      "epoch": 0.15930814747382796,
+      "grad_norm": 1.9180041551589966,
+      "learning_rate": 1.9906669701798315e-05,
+      "loss": 1.1421,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1750,
+      "tokens_per_second_per_gpu": 446.31
+    },
+    {
+      "epoch": 0.16385980883022302,
+      "grad_norm": 1.6121618747711182,
+      "learning_rate": 2.047575688595493e-05,
+      "loss": 1.1166,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1800,
+      "tokens_per_second_per_gpu": 454.94
+    },
+    {
+      "epoch": 0.16841147018661812,
+      "grad_norm": 1.5404553413391113,
+      "learning_rate": 2.1044844070111542e-05,
+      "loss": 1.1582,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1850,
+      "tokens_per_second_per_gpu": 507.31
+    },
+    {
+      "epoch": 0.1729631315430132,
+      "grad_norm": 1.6776846647262573,
+      "learning_rate": 2.1613931254268155e-05,
+      "loss": 1.1591,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1900,
+      "tokens_per_second_per_gpu": 503.53
+    },
+    {
+      "epoch": 0.17751479289940827,
+      "grad_norm": 1.8899917602539062,
+      "learning_rate": 2.218301843842477e-05,
+      "loss": 1.1293,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 1950,
+      "tokens_per_second_per_gpu": 467.41
+    },
+    {
+      "epoch": 0.18206645425580337,
+      "grad_norm": 1.4181222915649414,
+      "learning_rate": 2.275210562258138e-05,
+      "loss": 1.1541,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2000,
+      "tokens_per_second_per_gpu": 494.36
+    },
+    {
+      "epoch": 0.18661811561219846,
+      "grad_norm": 1.6970280408859253,
+      "learning_rate": 2.3321192806737992e-05,
+      "loss": 1.1207,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2050,
+      "tokens_per_second_per_gpu": 457.57
+    },
+    {
+      "epoch": 0.19116977696859352,
+      "grad_norm": 1.7740880250930786,
+      "learning_rate": 2.3890279990894605e-05,
+      "loss": 1.1484,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2100,
+      "tokens_per_second_per_gpu": 484.6
+    },
+    {
+      "epoch": 0.19572143832498862,
+      "grad_norm": 1.6202048063278198,
+      "learning_rate": 2.445936717505122e-05,
+      "loss": 1.1451,
+      "memory/device_reserved (GiB)": 76.76,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2150,
+      "tokens_per_second_per_gpu": 532.16
+    },
+    {
+      "epoch": 0.2,
+      "eval_loss": 1.119379997253418,
+      "eval_runtime": 195.6152,
+      "eval_samples_per_second": 5.107,
+      "eval_steps_per_second": 0.368,
+      "memory/device_reserved (GiB)": 77.96,
+      "memory/max_active (GiB)": 77.11,
+      "memory/max_allocated (GiB)": 77.11,
+      "step": 2197
+    },
+    {
+      "epoch": 0.2002730996813837,
+      "grad_norm": 1.9262148141860962,
+      "learning_rate": 2.502845435920783e-05,
+      "loss": 1.1345,
+      "memory/device_reserved (GiB)": 77.06,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2200,
+      "tokens_per_second_per_gpu": 24.4
+    },
+    {
+      "epoch": 0.2048247610377788,
+      "grad_norm": 1.6392292976379395,
+      "learning_rate": 2.5597541543364445e-05,
+      "loss": 1.1275,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2250,
+      "tokens_per_second_per_gpu": 462.24
+    },
+    {
+      "epoch": 0.20937642239417387,
+      "grad_norm": 1.8397603034973145,
+      "learning_rate": 2.6166628727521058e-05,
+      "loss": 1.1246,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2300,
+      "tokens_per_second_per_gpu": 490.72
+    },
+    {
+      "epoch": 0.21392808375056896,
+      "grad_norm": 1.2947629690170288,
+      "learning_rate": 2.673571591167767e-05,
+      "loss": 1.1322,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2350,
+      "tokens_per_second_per_gpu": 475.82
+    },
+    {
+      "epoch": 0.21847974510696405,
+      "grad_norm": 1.5403435230255127,
+      "learning_rate": 2.730480309583428e-05,
+      "loss": 1.1028,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2400,
+      "tokens_per_second_per_gpu": 503.72
+    },
+    {
+      "epoch": 0.22303140646335912,
+      "grad_norm": 1.2514817714691162,
+      "learning_rate": 2.7873890279990895e-05,
+      "loss": 1.1469,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2450,
+      "tokens_per_second_per_gpu": 525.63
+    },
+    {
+      "epoch": 0.2275830678197542,
+      "grad_norm": 1.5391417741775513,
+      "learning_rate": 2.8442977464147508e-05,
+      "loss": 1.1068,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2500,
+      "tokens_per_second_per_gpu": 480.06
+    },
+    {
+      "epoch": 0.2321347291761493,
+      "grad_norm": 1.393747091293335,
+      "learning_rate": 2.901206464830412e-05,
+      "loss": 1.1453,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2550,
+      "tokens_per_second_per_gpu": 498.26
+    },
+    {
+      "epoch": 0.23668639053254437,
+      "grad_norm": 1.5987119674682617,
+      "learning_rate": 2.9581151832460735e-05,
+      "loss": 1.116,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2600,
+      "tokens_per_second_per_gpu": 489.22
+    },
+    {
+      "epoch": 0.24123805188893946,
+      "grad_norm": 1.2702186107635498,
+      "learning_rate": 3.0150239016617348e-05,
+      "loss": 1.1107,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2650,
+      "tokens_per_second_per_gpu": 482.9
+    },
+    {
+      "epoch": 0.24578971324533455,
+      "grad_norm": 1.4149582386016846,
+      "learning_rate": 3.071932620077396e-05,
+      "loss": 1.1302,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2700,
+      "tokens_per_second_per_gpu": 495.9
+    },
+    {
+      "epoch": 0.2503413746017296,
+      "grad_norm": 3.761045455932617,
+      "learning_rate": 3.128841338493057e-05,
+      "loss": 1.1311,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2750,
+      "tokens_per_second_per_gpu": 489.72
+    },
+    {
+      "epoch": 0.25489303595812474,
+      "grad_norm": 1.2627049684524536,
+      "learning_rate": 3.185750056908718e-05,
+      "loss": 1.1243,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2800,
+      "tokens_per_second_per_gpu": 472.22
+    },
+    {
+      "epoch": 0.2594446973145198,
+      "grad_norm": 1.3173185586929321,
+      "learning_rate": 3.24265877532438e-05,
+      "loss": 1.1066,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2850,
+      "tokens_per_second_per_gpu": 467.07
+    },
+    {
+      "epoch": 0.26399635867091487,
+      "grad_norm": 1.4638679027557373,
+      "learning_rate": 3.2995674937400414e-05,
+      "loss": 1.1183,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2900,
+      "tokens_per_second_per_gpu": 495.32
+    },
+    {
+      "epoch": 0.26854802002731,
+      "grad_norm": 1.3660715818405151,
+      "learning_rate": 3.3564762121557024e-05,
+      "loss": 1.0697,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 2950,
+      "tokens_per_second_per_gpu": 474.36
+    },
+    {
+      "epoch": 0.27309968138370505,
+      "grad_norm": 1.175022840499878,
+      "learning_rate": 3.413384930571364e-05,
+      "loss": 1.1047,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3000,
+      "tokens_per_second_per_gpu": 501.11
+    },
+    {
+      "epoch": 0.2776513427401001,
+      "grad_norm": 1.247564673423767,
+      "learning_rate": 3.470293648987025e-05,
+      "loss": 1.1115,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3050,
+      "tokens_per_second_per_gpu": 491.44
+    },
+    {
+      "epoch": 0.28220300409649524,
+      "grad_norm": 1.1579777002334595,
+      "learning_rate": 3.527202367402686e-05,
+      "loss": 1.1257,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3100,
+      "tokens_per_second_per_gpu": 512.81
+    },
+    {
+      "epoch": 0.2867546654528903,
+      "grad_norm": 1.1780682802200317,
+      "learning_rate": 3.584111085818348e-05,
+      "loss": 1.1078,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3150,
+      "tokens_per_second_per_gpu": 476.72
+    },
+    {
+      "epoch": 0.29130632680928537,
+      "grad_norm": 1.224279761314392,
+      "learning_rate": 3.641019804234009e-05,
+      "loss": 1.1464,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3200,
+      "tokens_per_second_per_gpu": 524.15
+    },
+    {
+      "epoch": 0.2958579881656805,
+      "grad_norm": 1.2794740200042725,
+      "learning_rate": 3.6979285226496704e-05,
+      "loss": 1.0947,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3250,
+      "tokens_per_second_per_gpu": 517.16
+    },
+    {
+      "epoch": 0.30040964952207555,
+      "grad_norm": 1.2177249193191528,
+      "learning_rate": 3.7548372410653314e-05,
+      "loss": 1.1261,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3300,
+      "tokens_per_second_per_gpu": 467.35
+    },
+    {
+      "epoch": 0.3049613108784706,
+      "grad_norm": 1.1074647903442383,
+      "learning_rate": 3.811745959480993e-05,
+      "loss": 1.1125,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3350,
+      "tokens_per_second_per_gpu": 505.32
+    },
+    {
+      "epoch": 0.30951297223486574,
+      "grad_norm": 1.2007449865341187,
+      "learning_rate": 3.8686546778966534e-05,
+      "loss": 1.1102,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3400,
+      "tokens_per_second_per_gpu": 500.46
+    },
+    {
+      "epoch": 0.3140646335912608,
+      "grad_norm": 1.5369510650634766,
+      "learning_rate": 3.925563396312315e-05,
+      "loss": 1.0961,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3450,
+      "tokens_per_second_per_gpu": 461.95
+    },
+    {
+      "epoch": 0.3186162949476559,
+      "grad_norm": 0.9427760243415833,
+      "learning_rate": 3.982472114727977e-05,
+      "loss": 1.1108,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3500,
+      "tokens_per_second_per_gpu": 515.4
+    },
+    {
+      "epoch": 0.323167956304051,
+      "grad_norm": 1.334630012512207,
+      "learning_rate": 4.039380833143638e-05,
+      "loss": 1.1014,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3550,
+      "tokens_per_second_per_gpu": 498.55
+    },
+    {
+      "epoch": 0.32771961766044605,
+      "grad_norm": 1.1430654525756836,
+      "learning_rate": 4.0962895515592994e-05,
+      "loss": 1.11,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3600,
+      "tokens_per_second_per_gpu": 437.81
+    },
+    {
+      "epoch": 0.33227127901684117,
+      "grad_norm": 1.4943946599960327,
+      "learning_rate": 4.1531982699749604e-05,
+      "loss": 1.0991,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3650,
+      "tokens_per_second_per_gpu": 476.41
+    },
+    {
+      "epoch": 0.33682294037323623,
+      "grad_norm": 1.2534953355789185,
+      "learning_rate": 4.210106988390622e-05,
+      "loss": 1.0669,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3700,
+      "tokens_per_second_per_gpu": 463.29
+    },
+    {
+      "epoch": 0.3413746017296313,
+      "grad_norm": 1.0308473110198975,
+      "learning_rate": 4.267015706806283e-05,
+      "loss": 1.0984,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3750,
+      "tokens_per_second_per_gpu": 455.95
+    },
+    {
+      "epoch": 0.3459262630860264,
+      "grad_norm": 1.2468562126159668,
+      "learning_rate": 4.323924425221944e-05,
+      "loss": 1.0733,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3800,
+      "tokens_per_second_per_gpu": 460.65
+    },
+    {
+      "epoch": 0.3504779244424215,
+      "grad_norm": 1.1006118059158325,
+      "learning_rate": 4.380833143637606e-05,
+      "loss": 1.098,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3850,
+      "tokens_per_second_per_gpu": 482.27
+    },
+    {
+      "epoch": 0.35502958579881655,
+      "grad_norm": 1.1994727849960327,
+      "learning_rate": 4.437741862053267e-05,
+      "loss": 1.1089,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3900,
+      "tokens_per_second_per_gpu": 527.68
+    },
+    {
+      "epoch": 0.35958124715521167,
+      "grad_norm": 1.166426181793213,
+      "learning_rate": 4.4946505804689283e-05,
+      "loss": 1.0693,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 3950,
+      "tokens_per_second_per_gpu": 447.81
+    },
+    {
+      "epoch": 0.36413290851160673,
+      "grad_norm": 1.003192663192749,
+      "learning_rate": 4.551559298884589e-05,
+      "loss": 1.1004,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4000,
+      "tokens_per_second_per_gpu": 481.48
+    },
+    {
+      "epoch": 0.3686845698680018,
+      "grad_norm": 0.9891825318336487,
+      "learning_rate": 4.60846801730025e-05,
+      "loss": 1.0936,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4050,
+      "tokens_per_second_per_gpu": 498.97
+    },
+    {
+      "epoch": 0.3732362312243969,
+      "grad_norm": 1.1373133659362793,
+      "learning_rate": 4.665376735715912e-05,
+      "loss": 1.0909,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4100,
+      "tokens_per_second_per_gpu": 504.57
+    },
+    {
+      "epoch": 0.377787892580792,
+      "grad_norm": 1.149749517440796,
+      "learning_rate": 4.722285454131573e-05,
+      "loss": 1.0913,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4150,
+      "tokens_per_second_per_gpu": 497.49
+    },
+    {
+      "epoch": 0.38233955393718705,
+      "grad_norm": 1.1793601512908936,
+      "learning_rate": 4.7791941725472346e-05,
+      "loss": 1.0951,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4200,
+      "tokens_per_second_per_gpu": 532.82
+    },
+    {
+      "epoch": 0.38689121529358217,
+      "grad_norm": 0.982142448425293,
+      "learning_rate": 4.8361028909628956e-05,
+      "loss": 1.1117,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4250,
+      "tokens_per_second_per_gpu": 518.2
+    },
+    {
+      "epoch": 0.39144287664997723,
+      "grad_norm": 1.0042027235031128,
+      "learning_rate": 4.893011609378557e-05,
+      "loss": 1.0869,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4300,
+      "tokens_per_second_per_gpu": 487.27
+    },
+    {
+      "epoch": 0.3959945380063723,
+      "grad_norm": 1.2012032270431519,
+      "learning_rate": 4.949920327794218e-05,
+      "loss": 1.0682,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4350,
+      "tokens_per_second_per_gpu": 467.47
+    },
+    {
+      "epoch": 0.4,
+      "eval_loss": 1.0708580017089844,
+      "eval_runtime": 195.2836,
+      "eval_samples_per_second": 5.116,
+      "eval_steps_per_second": 0.369,
+      "memory/device_reserved (GiB)": 77.96,
+      "memory/max_active (GiB)": 77.11,
+      "memory/max_allocated (GiB)": 77.11,
+      "step": 4394
+    },
+    {
+      "epoch": 0.4005461993627674,
+      "grad_norm": 1.1110295057296753,
+      "learning_rate": 4.999999716007333e-05,
+      "loss": 1.1033,
+      "memory/device_reserved (GiB)": 77.06,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4400,
+      "tokens_per_second_per_gpu": 62.64
+    },
+    {
+      "epoch": 0.4050978607191625,
+      "grad_norm": 1.0224543809890747,
+      "learning_rate": 4.999975261123525e-05,
+      "loss": 1.088,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4450,
+      "tokens_per_second_per_gpu": 512.07
+    },
+    {
+      "epoch": 0.4096495220755576,
+      "grad_norm": 0.8717514872550964,
+      "learning_rate": 4.999911363255113e-05,
+      "loss": 1.0761,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4500,
+      "tokens_per_second_per_gpu": 494.38
+    },
+    {
+      "epoch": 0.41420118343195267,
+      "grad_norm": 1.2397717237472534,
+      "learning_rate": 4.999808023410233e-05,
+      "loss": 1.0455,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4550,
+      "tokens_per_second_per_gpu": 460.23
+    },
+    {
+      "epoch": 0.41875284478834773,
+      "grad_norm": 1.0875898599624634,
+      "learning_rate": 4.999665243219317e-05,
+      "loss": 1.0789,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4600,
+      "tokens_per_second_per_gpu": 488.7
+    },
+    {
+      "epoch": 0.42330450614474285,
+      "grad_norm": 0.9245534539222717,
+      "learning_rate": 4.9994830249350555e-05,
+      "loss": 1.0736,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4650,
+      "tokens_per_second_per_gpu": 480.91
+    },
+    {
+      "epoch": 0.4278561675011379,
+      "grad_norm": 1.1447221040725708,
+      "learning_rate": 4.9992613714323723e-05,
+      "loss": 1.0639,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4700,
+      "tokens_per_second_per_gpu": 492.85
+    },
+    {
+      "epoch": 0.432407828857533,
+      "grad_norm": 1.0307674407958984,
+      "learning_rate": 4.99900028620837e-05,
+      "loss": 1.077,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4750,
+      "tokens_per_second_per_gpu": 477.88
+    },
+    {
+      "epoch": 0.4369594902139281,
+      "grad_norm": 0.9278285503387451,
+      "learning_rate": 4.9986997733822826e-05,
+      "loss": 1.088,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4800,
+      "tokens_per_second_per_gpu": 480.35
+    },
+    {
+      "epoch": 0.44151115157032317,
+      "grad_norm": 1.2356767654418945,
+      "learning_rate": 4.998359837695407e-05,
+      "loss": 1.0781,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4850,
+      "tokens_per_second_per_gpu": 472.56
+    },
+    {
+      "epoch": 0.44606281292671823,
+      "grad_norm": 0.9927319884300232,
+      "learning_rate": 4.997980484511027e-05,
+      "loss": 1.1009,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4900,
+      "tokens_per_second_per_gpu": 488.86
+    },
+    {
+      "epoch": 0.45061447428311335,
+      "grad_norm": 0.9227324724197388,
+      "learning_rate": 4.997561719814329e-05,
+      "loss": 1.0898,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 4950,
+      "tokens_per_second_per_gpu": 520.64
+    },
+    {
+      "epoch": 0.4551661356395084,
+      "grad_norm": 1.0663466453552246,
+      "learning_rate": 4.9971035502123126e-05,
+      "loss": 1.0729,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5000,
+      "tokens_per_second_per_gpu": 478.18
+    },
+    {
+      "epoch": 0.4597177969959035,
+      "grad_norm": 0.9325670599937439,
+      "learning_rate": 4.9966059829336786e-05,
+      "loss": 1.069,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5050,
+      "tokens_per_second_per_gpu": 484.12
+    },
+    {
+      "epoch": 0.4642694583522986,
+      "grad_norm": 1.1680859327316284,
+      "learning_rate": 4.9960690258287204e-05,
+      "loss": 1.0801,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5100,
+      "tokens_per_second_per_gpu": 469.73
+    },
+    {
+      "epoch": 0.46882111970869367,
+      "grad_norm": 1.132150411605835,
+      "learning_rate": 4.995492687369198e-05,
+      "loss": 1.0803,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5150,
+      "tokens_per_second_per_gpu": 483.93
+    },
+    {
+      "epoch": 0.47337278106508873,
+      "grad_norm": 1.0051616430282593,
+      "learning_rate": 4.994876976648204e-05,
+      "loss": 1.0829,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5200,
+      "tokens_per_second_per_gpu": 491.51
+    },
+    {
+      "epoch": 0.47792444242148385,
+      "grad_norm": 1.000447392463684,
+      "learning_rate": 4.994221903380024e-05,
+      "loss": 1.0709,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5250,
+      "tokens_per_second_per_gpu": 510.0
+    },
+    {
+      "epoch": 0.4824761037778789,
+      "grad_norm": 1.1662852764129639,
+      "learning_rate": 4.993527477899977e-05,
+      "loss": 1.0676,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5300,
+      "tokens_per_second_per_gpu": 512.04
+    },
+    {
+      "epoch": 0.48702776513427404,
+      "grad_norm": 1.0618816614151,
+      "learning_rate": 4.992793711164256e-05,
+      "loss": 1.0825,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5350,
+      "tokens_per_second_per_gpu": 495.62
+    },
+    {
+      "epoch": 0.4915794264906691,
+      "grad_norm": 1.1399046182632446,
+      "learning_rate": 4.992020614749758e-05,
+      "loss": 1.0664,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5400,
+      "tokens_per_second_per_gpu": 494.69
+    },
+    {
+      "epoch": 0.49613108784706417,
+      "grad_norm": 1.2065247297286987,
+      "learning_rate": 4.991208200853894e-05,
+      "loss": 1.0657,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5450,
+      "tokens_per_second_per_gpu": 493.34
+    },
+    {
+      "epoch": 0.5006827492034592,
+      "grad_norm": 1.0179318189620972,
+      "learning_rate": 4.9903564822944024e-05,
+      "loss": 1.0677,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5500,
+      "tokens_per_second_per_gpu": 506.23
+    },
+    {
+      "epoch": 0.5052344105598543,
+      "grad_norm": 1.192871332168579,
+      "learning_rate": 4.989465472509145e-05,
+      "loss": 1.0534,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5550,
+      "tokens_per_second_per_gpu": 487.98
+    },
+    {
+      "epoch": 0.5097860719162495,
+      "grad_norm": 1.072653889656067,
+      "learning_rate": 4.988535185555896e-05,
+      "loss": 1.0625,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5600,
+      "tokens_per_second_per_gpu": 473.49
+    },
+    {
+      "epoch": 0.5143377332726445,
+      "grad_norm": 1.1457641124725342,
+      "learning_rate": 4.987565636112117e-05,
+      "loss": 1.0596,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5650,
+      "tokens_per_second_per_gpu": 489.03
+    },
+    {
+      "epoch": 0.5188893946290396,
+      "grad_norm": 0.9638017416000366,
+      "learning_rate": 4.986556839474729e-05,
+      "loss": 1.0502,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5700,
+      "tokens_per_second_per_gpu": 485.55
+    },
+    {
+      "epoch": 0.5234410559854347,
+      "grad_norm": 1.1078736782073975,
+      "learning_rate": 4.98550881155987e-05,
+      "loss": 1.0695,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5750,
+      "tokens_per_second_per_gpu": 485.32
+    },
+    {
+      "epoch": 0.5279927173418297,
+      "grad_norm": 1.3936585187911987,
+      "learning_rate": 4.984421568902643e-05,
+      "loss": 1.0253,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5800,
+      "tokens_per_second_per_gpu": 463.27
+    },
+    {
+      "epoch": 0.5325443786982249,
+      "grad_norm": 1.12526535987854,
+      "learning_rate": 4.983295128656856e-05,
+      "loss": 1.0549,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5850,
+      "tokens_per_second_per_gpu": 480.14
+    },
+    {
+      "epoch": 0.53709604005462,
+      "grad_norm": 1.0340914726257324,
+      "learning_rate": 4.9821295085947515e-05,
+      "loss": 1.0723,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5900,
+      "tokens_per_second_per_gpu": 497.14
+    },
+    {
+      "epoch": 0.541647701411015,
+      "grad_norm": 1.0414705276489258,
+      "learning_rate": 4.980924727106725e-05,
+      "loss": 1.0622,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 5950,
+      "tokens_per_second_per_gpu": 491.2
+    },
+    {
+      "epoch": 0.5461993627674101,
+      "grad_norm": 1.0546574592590332,
+      "learning_rate": 4.979680803201034e-05,
+      "loss": 1.1061,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6000,
+      "tokens_per_second_per_gpu": 528.09
+    },
+    {
+      "epoch": 0.5507510241238052,
+      "grad_norm": 1.0350837707519531,
+      "learning_rate": 4.978397756503504e-05,
+      "loss": 1.0554,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6050,
+      "tokens_per_second_per_gpu": 489.79
+    },
+    {
+      "epoch": 0.5553026854802002,
+      "grad_norm": 0.9223297238349915,
+      "learning_rate": 4.97707560725721e-05,
+      "loss": 1.056,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6100,
+      "tokens_per_second_per_gpu": 504.58
+    },
+    {
+      "epoch": 0.5598543468365954,
+      "grad_norm": 1.9479364156723022,
+      "learning_rate": 4.9757143763221635e-05,
+      "loss": 1.0438,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6150,
+      "tokens_per_second_per_gpu": 482.52
+    },
+    {
+      "epoch": 0.5644060081929905,
+      "grad_norm": 1.0383278131484985,
+      "learning_rate": 4.9743140851749814e-05,
+      "loss": 1.0378,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6200,
+      "tokens_per_second_per_gpu": 490.36
+    },
+    {
+      "epoch": 0.5689576695493855,
+      "grad_norm": 1.022878646850586,
+      "learning_rate": 4.972874755908548e-05,
+      "loss": 1.0385,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6250,
+      "tokens_per_second_per_gpu": 469.72
+    },
+    {
+      "epoch": 0.5735093309057806,
+      "grad_norm": 1.0296788215637207,
+      "learning_rate": 4.971396411231664e-05,
+      "loss": 1.0658,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6300,
+      "tokens_per_second_per_gpu": 507.78
+    },
+    {
+      "epoch": 0.5780609922621757,
+      "grad_norm": 1.2907683849334717,
+      "learning_rate": 4.9698790744686903e-05,
+      "loss": 1.0536,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6350,
+      "tokens_per_second_per_gpu": 474.21
+    },
+    {
+      "epoch": 0.5826126536185707,
+      "grad_norm": 1.1980242729187012,
+      "learning_rate": 4.968322769559181e-05,
+      "loss": 1.0722,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6400,
+      "tokens_per_second_per_gpu": 497.21
+    },
+    {
+      "epoch": 0.5871643149749659,
+      "grad_norm": 1.0062897205352783,
+      "learning_rate": 4.9667275210575036e-05,
+      "loss": 1.0533,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6450,
+      "tokens_per_second_per_gpu": 490.36
+    },
+    {
+      "epoch": 0.591715976331361,
+      "grad_norm": 1.1628015041351318,
+      "learning_rate": 4.9650933541324506e-05,
+      "loss": 1.0687,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6500,
+      "tokens_per_second_per_gpu": 502.69
+    },
+    {
+      "epoch": 0.596267637687756,
+      "grad_norm": 1.0880138874053955,
+      "learning_rate": 4.9634202945668476e-05,
+      "loss": 1.0512,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6550,
+      "tokens_per_second_per_gpu": 508.51
+    },
+    {
+      "epoch": 0.6,
+      "eval_loss": 1.0370653867721558,
+      "eval_runtime": 195.0721,
+      "eval_samples_per_second": 5.121,
+      "eval_steps_per_second": 0.369,
+      "memory/device_reserved (GiB)": 77.96,
+      "memory/max_active (GiB)": 77.11,
+      "memory/max_allocated (GiB)": 77.11,
+      "step": 6591
+    },
+    {
+      "epoch": 0.6008192990441511,
+      "grad_norm": 1.089520812034607,
+      "learning_rate": 4.961708368757138e-05,
+      "loss": 1.0435,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6600,
+      "tokens_per_second_per_gpu": 77.93
+    },
+    {
+      "epoch": 0.6053709604005462,
+      "grad_norm": 0.904808759689331,
+      "learning_rate": 4.959957603712977e-05,
+      "loss": 1.0353,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6650,
+      "tokens_per_second_per_gpu": 478.54
+    },
+    {
+      "epoch": 0.6099226217569412,
+      "grad_norm": 1.1445884704589844,
+      "learning_rate": 4.9581680270567957e-05,
+      "loss": 1.0425,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6700,
+      "tokens_per_second_per_gpu": 502.08
+    },
+    {
+      "epoch": 0.6144742831133364,
+      "grad_norm": 1.157138466835022,
+      "learning_rate": 4.956339667023373e-05,
+      "loss": 1.0563,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6750,
+      "tokens_per_second_per_gpu": 511.61
+    },
+    {
+      "epoch": 0.6190259444697315,
+      "grad_norm": 0.8729771375656128,
+      "learning_rate": 4.954472552459383e-05,
+      "loss": 1.0364,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6800,
+      "tokens_per_second_per_gpu": 481.17
+    },
+    {
+      "epoch": 0.6235776058261265,
+      "grad_norm": 1.065958857536316,
+      "learning_rate": 4.9525667128229474e-05,
+      "loss": 1.0605,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6850,
+      "tokens_per_second_per_gpu": 497.49
+    },
+    {
+      "epoch": 0.6281292671825216,
+      "grad_norm": 0.9055165648460388,
+      "learning_rate": 4.950622178183164e-05,
+      "loss": 1.0743,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6900,
+      "tokens_per_second_per_gpu": 484.48
+    },
+    {
+      "epoch": 0.6326809285389167,
+      "grad_norm": 1.157254934310913,
+      "learning_rate": 4.948638979219637e-05,
+      "loss": 1.035,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 6950,
+      "tokens_per_second_per_gpu": 497.23
+    },
+    {
+      "epoch": 0.6372325898953118,
+      "grad_norm": 1.0744335651397705,
+      "learning_rate": 4.9466171472219904e-05,
+      "loss": 1.0365,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7000,
+      "tokens_per_second_per_gpu": 484.74
+    },
+    {
+      "epoch": 0.6417842512517069,
+      "grad_norm": 0.9064004421234131,
+      "learning_rate": 4.944556714089374e-05,
+      "loss": 1.0769,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7050,
+      "tokens_per_second_per_gpu": 527.16
+    },
+    {
+      "epoch": 0.646335912608102,
+      "grad_norm": 0.9930420517921448,
+      "learning_rate": 4.942457712329964e-05,
+      "loss": 1.0223,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7100,
+      "tokens_per_second_per_gpu": 462.81
+    },
+    {
+      "epoch": 0.650887573964497,
+      "grad_norm": 0.9996587634086609,
+      "learning_rate": 4.940320175060446e-05,
+      "loss": 1.0566,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7150,
+      "tokens_per_second_per_gpu": 501.44
+    },
+    {
+      "epoch": 0.6554392353208921,
+      "grad_norm": 1.7110569477081299,
+      "learning_rate": 4.9381441360054915e-05,
+      "loss": 1.0522,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7200,
+      "tokens_per_second_per_gpu": 499.36
+    },
+    {
+      "epoch": 0.6599908966772872,
+      "grad_norm": 1.2259682416915894,
+      "learning_rate": 4.9359296294972315e-05,
+      "loss": 1.0541,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7250,
+      "tokens_per_second_per_gpu": 473.99
+    },
+    {
+      "epoch": 0.6645425580336823,
+      "grad_norm": 1.290621042251587,
+      "learning_rate": 4.933676690474711e-05,
+      "loss": 1.0551,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7300,
+      "tokens_per_second_per_gpu": 484.9
+    },
+    {
+      "epoch": 0.6690942193900774,
+      "grad_norm": 0.8518486022949219,
+      "learning_rate": 4.931385354483336e-05,
+      "loss": 1.0512,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7350,
+      "tokens_per_second_per_gpu": 499.75
+    },
+    {
+      "epoch": 0.6736458807464725,
+      "grad_norm": 1.0784375667572021,
+      "learning_rate": 4.929055657674316e-05,
+      "loss": 1.0463,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7400,
+      "tokens_per_second_per_gpu": 451.58
+    },
+    {
+      "epoch": 0.6781975421028675,
+      "grad_norm": 1.1245126724243164,
+      "learning_rate": 4.9266876368040954e-05,
+      "loss": 1.0281,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7450,
+      "tokens_per_second_per_gpu": 488.61
+    },
+    {
+      "epoch": 0.6827492034592626,
+      "grad_norm": 1.037510633468628,
+      "learning_rate": 4.924281329233766e-05,
+      "loss": 1.036,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7500,
+      "tokens_per_second_per_gpu": 502.11
+    },
+    {
+      "epoch": 0.6873008648156577,
+      "grad_norm": 0.9862679839134216,
+      "learning_rate": 4.921836772928484e-05,
+      "loss": 1.0504,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7550,
+      "tokens_per_second_per_gpu": 492.9
+    },
+    {
+      "epoch": 0.6918525261720528,
+      "grad_norm": 1.153113603591919,
+      "learning_rate": 4.9193540064568724e-05,
+      "loss": 1.0229,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7600,
+      "tokens_per_second_per_gpu": 458.22
+    },
+    {
+      "epoch": 0.6964041875284479,
+      "grad_norm": 1.1115753650665283,
+      "learning_rate": 4.916833068990404e-05,
+      "loss": 1.0183,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7650,
+      "tokens_per_second_per_gpu": 484.87
+    },
+    {
+      "epoch": 0.700955848884843,
+      "grad_norm": 1.1606495380401611,
+      "learning_rate": 4.914274000302793e-05,
+      "loss": 1.0284,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7700,
+      "tokens_per_second_per_gpu": 496.9
+    },
+    {
+      "epoch": 0.705507510241238,
+      "grad_norm": 0.9907020330429077,
+      "learning_rate": 4.911676840769361e-05,
+      "loss": 1.0377,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7750,
+      "tokens_per_second_per_gpu": 472.59
+    },
+    {
+      "epoch": 0.7100591715976331,
+      "grad_norm": 1.0251307487487793,
+      "learning_rate": 4.9090416313664036e-05,
+      "loss": 1.0421,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7800,
+      "tokens_per_second_per_gpu": 472.37
+    },
+    {
+      "epoch": 0.7146108329540283,
+      "grad_norm": 1.1614022254943848,
+      "learning_rate": 4.9063684136705404e-05,
+      "loss": 1.0792,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7850,
+      "tokens_per_second_per_gpu": 518.05
+    },
+    {
+      "epoch": 0.7191624943104233,
+      "grad_norm": 0.9551727175712585,
+      "learning_rate": 4.903657229858063e-05,
+      "loss": 1.0356,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7900,
+      "tokens_per_second_per_gpu": 482.4
+    },
+    {
+      "epoch": 0.7237141556668184,
+      "grad_norm": 0.9774997234344482,
+      "learning_rate": 4.900908122704269e-05,
+      "loss": 1.0267,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 7950,
+      "tokens_per_second_per_gpu": 507.2
+    },
+    {
+      "epoch": 0.7282658170232135,
+      "grad_norm": 1.1599117517471313,
+      "learning_rate": 4.898121135582782e-05,
+      "loss": 1.0426,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8000,
+      "tokens_per_second_per_gpu": 494.25
+    },
+    {
+      "epoch": 0.7328174783796085,
+      "grad_norm": 1.0983076095581055,
+      "learning_rate": 4.895296312464874e-05,
+      "loss": 1.0496,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8050,
+      "tokens_per_second_per_gpu": 506.5
+    },
+    {
+      "epoch": 0.7373691397360036,
+      "grad_norm": 1.1329846382141113,
+      "learning_rate": 4.892433697918768e-05,
+      "loss": 0.9954,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8100,
+      "tokens_per_second_per_gpu": 459.45
+    },
+    {
+      "epoch": 0.7419208010923988,
+      "grad_norm": 1.291815161705017,
+      "learning_rate": 4.889533337108935e-05,
+      "loss": 1.039,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8150,
+      "tokens_per_second_per_gpu": 459.98
+    },
+    {
+      "epoch": 0.7464724624487938,
+      "grad_norm": 0.8809608817100525,
+      "learning_rate": 4.886595275795383e-05,
+      "loss": 1.0431,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8200,
+      "tokens_per_second_per_gpu": 503.41
+    },
+    {
+      "epoch": 0.7510241238051889,
+      "grad_norm": 1.0669974088668823,
+      "learning_rate": 4.8836195603329335e-05,
+      "loss": 1.047,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8250,
+      "tokens_per_second_per_gpu": 513.85
+    },
+    {
+      "epoch": 0.755575785161584,
+      "grad_norm": 1.1232683658599854,
+      "learning_rate": 4.880606237670491e-05,
+      "loss": 1.0282,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8300,
+      "tokens_per_second_per_gpu": 456.41
+    },
+    {
+      "epoch": 0.760127446517979,
+      "grad_norm": 1.0202189683914185,
+      "learning_rate": 4.877555355350301e-05,
+      "loss": 1.0415,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8350,
+      "tokens_per_second_per_gpu": 512.26
+    },
+    {
+      "epoch": 0.7646791078743741,
+      "grad_norm": 1.2844234704971313,
+      "learning_rate": 4.8744669615072015e-05,
+      "loss": 1.0322,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8400,
+      "tokens_per_second_per_gpu": 474.13
+    },
+    {
+      "epoch": 0.7692307692307693,
+      "grad_norm": 1.028136134147644,
+      "learning_rate": 4.8713411048678635e-05,
+      "loss": 1.0353,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8450,
+      "tokens_per_second_per_gpu": 508.77
+    },
+    {
+      "epoch": 0.7737824305871643,
+      "grad_norm": 0.9930169582366943,
+      "learning_rate": 4.868177834750022e-05,
+      "loss": 1.0397,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8500,
+      "tokens_per_second_per_gpu": 508.0
+    },
+    {
+      "epoch": 0.7783340919435594,
+      "grad_norm": 1.0289788246154785,
+      "learning_rate": 4.8649772010616955e-05,
+      "loss": 1.043,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8550,
+      "tokens_per_second_per_gpu": 526.03
+    },
+    {
+      "epoch": 0.7828857532999545,
+      "grad_norm": 0.923629879951477,
+      "learning_rate": 4.8617392543004014e-05,
+      "loss": 1.0374,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8600,
+      "tokens_per_second_per_gpu": 479.04
+    },
+    {
+      "epoch": 0.7874374146563495,
+      "grad_norm": 1.0967282056808472,
+      "learning_rate": 4.8584640455523604e-05,
+      "loss": 1.0222,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8650,
+      "tokens_per_second_per_gpu": 464.9
+    },
+    {
+      "epoch": 0.7919890760127446,
+      "grad_norm": 1.5993574857711792,
+      "learning_rate": 4.855151626491686e-05,
+      "loss": 1.0443,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8700,
+      "tokens_per_second_per_gpu": 486.03
+    },
+    {
+      "epoch": 0.7965407373691398,
+      "grad_norm": 1.1585549116134644,
+      "learning_rate": 4.851802049379574e-05,
+      "loss": 1.0213,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8750,
+      "tokens_per_second_per_gpu": 467.88
+    },
+    {
+      "epoch": 0.8,
+      "eval_loss": 1.0147367715835571,
+      "eval_runtime": 195.7565,
+      "eval_samples_per_second": 5.103,
+      "eval_steps_per_second": 0.368,
+      "memory/device_reserved (GiB)": 77.96,
+      "memory/max_active (GiB)": 77.11,
+      "memory/max_allocated (GiB)": 77.11,
+      "step": 8788
+    },
+    {
+      "epoch": 0.8010923987255348,
+      "grad_norm": 1.144241213798523,
+      "learning_rate": 4.8484153670634735e-05,
+      "loss": 1.0558,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8800,
+      "tokens_per_second_per_gpu": 109.97
+    },
+    {
+      "epoch": 0.8056440600819299,
+      "grad_norm": 0.8619866371154785,
+      "learning_rate": 4.844991632976257e-05,
+      "loss": 1.033,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8850,
+      "tokens_per_second_per_gpu": 518.75
+    },
+    {
+      "epoch": 0.810195721438325,
+      "grad_norm": 1.1444193124771118,
+      "learning_rate": 4.841530901135376e-05,
+      "loss": 1.0518,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8900,
+      "tokens_per_second_per_gpu": 518.64
+    },
+    {
+      "epoch": 0.81474738279472,
+      "grad_norm": 0.937384843826294,
+      "learning_rate": 4.838033226142008e-05,
+      "loss": 1.0305,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 8950,
+      "tokens_per_second_per_gpu": 516.59
+    },
+    {
+      "epoch": 0.8192990441511152,
+      "grad_norm": 0.9430196285247803,
+      "learning_rate": 4.8344986631801944e-05,
+      "loss": 1.0417,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9000,
+      "tokens_per_second_per_gpu": 515.31
+    },
+    {
+      "epoch": 0.8238507055075103,
+      "grad_norm": 1.0032052993774414,
+      "learning_rate": 4.8309272680159723e-05,
+      "loss": 1.0416,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9050,
+      "tokens_per_second_per_gpu": 546.53
+    },
+    {
+      "epoch": 0.8284023668639053,
+      "grad_norm": 1.0585726499557495,
+      "learning_rate": 4.8273190969964936e-05,
+      "loss": 1.0054,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9100,
+      "tokens_per_second_per_gpu": 482.0
+    },
+    {
+      "epoch": 0.8329540282203004,
+      "grad_norm": 1.1669644117355347,
+      "learning_rate": 4.8236742070491356e-05,
+      "loss": 1.0343,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9150,
+      "tokens_per_second_per_gpu": 534.53
+    },
+    {
+      "epoch": 0.8375056895766955,
+      "grad_norm": 1.120493769645691,
+      "learning_rate": 4.819992655680601e-05,
+      "loss": 1.0371,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9200,
+      "tokens_per_second_per_gpu": 499.88
+    },
+    {
+      "epoch": 0.8420573509330905,
+      "grad_norm": 0.9619265198707581,
+      "learning_rate": 4.816274500976016e-05,
+      "loss": 1.0289,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9250,
+      "tokens_per_second_per_gpu": 483.97
+    },
+    {
+      "epoch": 0.8466090122894857,
+      "grad_norm": 1.0850692987442017,
+      "learning_rate": 4.812519801598007e-05,
+      "loss": 1.029,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9300,
+      "tokens_per_second_per_gpu": 489.02
+    },
+    {
+      "epoch": 0.8511606736458808,
+      "grad_norm": 1.7269978523254395,
+      "learning_rate": 4.808728616785781e-05,
+      "loss": 1.0358,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9350,
+      "tokens_per_second_per_gpu": 479.95
+    },
+    {
+      "epoch": 0.8557123350022758,
+      "grad_norm": 1.0903490781784058,
+      "learning_rate": 4.804901006354188e-05,
+      "loss": 1.0154,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9400,
+      "tokens_per_second_per_gpu": 459.61
+    },
+    {
+      "epoch": 0.8602639963586709,
+      "grad_norm": 1.1368602514266968,
+      "learning_rate": 4.801037030692776e-05,
+      "loss": 1.0186,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9450,
+      "tokens_per_second_per_gpu": 502.83
+    },
+    {
+      "epoch": 0.864815657715066,
+      "grad_norm": 1.1895757913589478,
+      "learning_rate": 4.7971367507648424e-05,
+      "loss": 0.9895,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9500,
+      "tokens_per_second_per_gpu": 464.56
+    },
+    {
+      "epoch": 0.869367319071461,
+      "grad_norm": 0.9769192934036255,
+      "learning_rate": 4.793200228106468e-05,
+      "loss": 1.0405,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9550,
+      "tokens_per_second_per_gpu": 512.92
+    },
+    {
+      "epoch": 0.8739189804278562,
+      "grad_norm": 1.0490390062332153,
+      "learning_rate": 4.789227524825549e-05,
+      "loss": 1.0499,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9600,
+      "tokens_per_second_per_gpu": 495.91
+    },
+    {
+      "epoch": 0.8784706417842513,
+      "grad_norm": 1.029001235961914,
+      "learning_rate": 4.785218703600815e-05,
+      "loss": 1.0322,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9650,
+      "tokens_per_second_per_gpu": 484.91
+    },
+    {
+      "epoch": 0.8830223031406463,
+      "grad_norm": 1.1553659439086914,
+      "learning_rate": 4.7811738276808415e-05,
+      "loss": 1.0196,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9700,
+      "tokens_per_second_per_gpu": 478.05
+    },
+    {
+      "epoch": 0.8875739644970414,
+      "grad_norm": 1.1080563068389893,
+      "learning_rate": 4.777092960883051e-05,
+      "loss": 1.0324,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9750,
+      "tokens_per_second_per_gpu": 513.85
+    },
+    {
+      "epoch": 0.8921256258534365,
+      "grad_norm": 1.3158994913101196,
+      "learning_rate": 4.772976167592708e-05,
+      "loss": 1.04,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9800,
+      "tokens_per_second_per_gpu": 461.81
+    },
+    {
+      "epoch": 0.8966772872098316,
+      "grad_norm": 1.4326947927474976,
+      "learning_rate": 4.768823512761901e-05,
+      "loss": 1.0311,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9850,
+      "tokens_per_second_per_gpu": 508.24
+    },
+    {
+      "epoch": 0.9012289485662267,
+      "grad_norm": 1.200109839439392,
+      "learning_rate": 4.764635061908517e-05,
+      "loss": 1.0129,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9900,
+      "tokens_per_second_per_gpu": 475.06
+    },
+    {
+      "epoch": 0.9057806099226218,
+      "grad_norm": 1.0471111536026,
+      "learning_rate": 4.7604108811152126e-05,
+      "loss": 1.0137,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 9950,
+      "tokens_per_second_per_gpu": 459.57
+    },
+    {
+      "epoch": 0.9103322712790168,
+      "grad_norm": 1.168687343597412,
+      "learning_rate": 4.7561510370283654e-05,
+      "loss": 1.0504,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10000,
+      "tokens_per_second_per_gpu": 510.16
+    },
+    {
+      "epoch": 0.9148839326354119,
+      "grad_norm": 1.0843098163604736,
+      "learning_rate": 4.7518555968570276e-05,
+      "loss": 1.0268,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10050,
+      "tokens_per_second_per_gpu": 480.03
+    },
+    {
+      "epoch": 0.919435593991807,
+      "grad_norm": 1.1069607734680176,
+      "learning_rate": 4.7475246283718614e-05,
+      "loss": 1.0117,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10100,
+      "tokens_per_second_per_gpu": 486.0
+    },
+    {
+      "epoch": 0.9239872553482021,
+      "grad_norm": 1.1321187019348145,
+      "learning_rate": 4.743158199904072e-05,
+      "loss": 1.0196,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10150,
+      "tokens_per_second_per_gpu": 496.42
+    },
+    {
+      "epoch": 0.9285389167045972,
+      "grad_norm": 1.0236319303512573,
+      "learning_rate": 4.73875638034433e-05,
+      "loss": 1.0377,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10200,
+      "tokens_per_second_per_gpu": 483.49
+    },
+    {
+      "epoch": 0.9330905780609923,
+      "grad_norm": 1.1202757358551025,
+      "learning_rate": 4.734319239141682e-05,
+      "loss": 1.0076,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10250,
+      "tokens_per_second_per_gpu": 486.17
+    },
+    {
+      "epoch": 0.9376422394173873,
+      "grad_norm": 1.2959187030792236,
+      "learning_rate": 4.72984684630246e-05,
+      "loss": 1.0061,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10300,
+      "tokens_per_second_per_gpu": 496.26
+    },
+    {
+      "epoch": 0.9421939007737824,
+      "grad_norm": 1.0744752883911133,
+      "learning_rate": 4.725339272389168e-05,
+      "loss": 1.0143,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10350,
+      "tokens_per_second_per_gpu": 473.73
+    },
+    {
+      "epoch": 0.9467455621301775,
+      "grad_norm": 1.035398006439209,
+      "learning_rate": 4.72079658851938e-05,
+      "loss": 1.0254,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10400,
+      "tokens_per_second_per_gpu": 476.32
+    },
+    {
+      "epoch": 0.9512972234865726,
+      "grad_norm": 1.0589100122451782,
+      "learning_rate": 4.716218866364606e-05,
+      "loss": 0.9991,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10450,
+      "tokens_per_second_per_gpu": 489.27
+    },
+    {
+      "epoch": 0.9558488848429677,
+      "grad_norm": 1.072708010673523,
+      "learning_rate": 4.711606178149173e-05,
+      "loss": 1.034,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10500,
+      "tokens_per_second_per_gpu": 513.81
+    },
+    {
+      "epoch": 0.9604005461993628,
+      "grad_norm": 1.2344636917114258,
+      "learning_rate": 4.706958596649076e-05,
+      "loss": 1.0254,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10550,
+      "tokens_per_second_per_gpu": 492.35
+    },
+    {
+      "epoch": 0.9649522075557578,
+      "grad_norm": 0.9406333565711975,
+      "learning_rate": 4.7022761951908365e-05,
+      "loss": 1.0171,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10600,
+      "tokens_per_second_per_gpu": 474.51
+    },
+    {
+      "epoch": 0.9695038689121529,
+      "grad_norm": 1.1612303256988525,
+      "learning_rate": 4.6975590476503396e-05,
+      "loss": 1.017,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10650,
+      "tokens_per_second_per_gpu": 493.48
+    },
+    {
+      "epoch": 0.9740555302685481,
+      "grad_norm": 1.5133899450302124,
+      "learning_rate": 4.692807228451672e-05,
+      "loss": 1.0212,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10700,
+      "tokens_per_second_per_gpu": 481.52
+    },
+    {
+      "epoch": 0.9786071916249431,
+      "grad_norm": 1.1254411935806274,
+      "learning_rate": 4.68802081256595e-05,
+      "loss": 1.009,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10750,
+      "tokens_per_second_per_gpu": 483.76
+    },
+    {
+      "epoch": 0.9831588529813382,
+      "grad_norm": 1.0411556959152222,
+      "learning_rate": 4.683199875510132e-05,
+      "loss": 1.0262,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10800,
+      "tokens_per_second_per_gpu": 493.64
+    },
+    {
+      "epoch": 0.9877105143377333,
+      "grad_norm": 0.979972243309021,
+      "learning_rate": 4.6783444933458284e-05,
+      "loss": 1.0246,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10850,
+      "tokens_per_second_per_gpu": 477.13
+    },
+    {
+      "epoch": 0.9922621756941283,
+      "grad_norm": 1.1155357360839844,
+      "learning_rate": 4.673454742678102e-05,
+      "loss": 1.0532,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10900,
+      "tokens_per_second_per_gpu": 505.17
+    },
+    {
+      "epoch": 0.9968138370505234,
+      "grad_norm": 1.1054545640945435,
+      "learning_rate": 4.668530700654262e-05,
+      "loss": 1.0041,
+      "memory/device_reserved (GiB)": 77.07,
+      "memory/max_active (GiB)": 75.64,
+      "memory/max_allocated (GiB)": 75.64,
+      "step": 10950,
+      "tokens_per_second_per_gpu": 488.33
+    },
+    {
+      "epoch": 1.0,
+      "eval_loss": 0.9990127086639404,
+      "eval_runtime": 196.2105,
+      "eval_samples_per_second": 5.091,
+      "eval_steps_per_second": 0.367,
+      "memory/device_reserved (GiB)": 77.96,
+      "memory/max_active (GiB)": 77.11,
+      "memory/max_allocated (GiB)": 77.11,
+      "step": 10985
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 43939,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 5493,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.086121052097413e+19,
+  "train_batch_size": 9,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-10986/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34a581bf9b76f74a5a4e25651fbaa01e61b75b19cbbfa8df2370c772924bc5d7
+size 11409

checkpoint-16479/README.md

@@ -0,0 +1,208 @@
+---
+base_model: mistralai/Mistral-Nemo-Instruct-2407
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- axolotl
+- base_model:adapter:mistralai/Mistral-Nemo-Instruct-2407
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-16479/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-Nemo-Instruct-2407",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "down_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": [],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-16479/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2447934c6f97cd035c3bdaee1fcf0cfd77d676fb1de72c73265cf319e8826aea
+size 456206152

checkpoint-16479/chat_template.jinja

@@ -0,0 +1,87 @@
+{%- if messages[0]["role"] == "system" %}
+    {%- set system_message = messages[0]["content"] %}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
+{%- set ns = namespace() %}
+{%- set ns.index = 0 %}
+{%- for message in loop_messages %}
+    {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
+        {%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
+            {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+        {%- endif %}
+        {%- set ns.index = ns.index + 1 %}
+    {%- endif %}
+{%- endfor %}
+
+{{- bos_token }}
+{%- for message in loop_messages %}
+    {%- if message["role"] == "user" %}
+        {%- if tools is not none and (message == user_messages[-1]) %}
+            {{- "[AVAILABLE_TOOLS][" }}
+            {%- for tool in tools %}
+                {%- set tool = tool.function %}
+                {{- '{"type": "function", "function": {' }}
+                {%- for key, val in tool.items() if key != "return" %}
+                    {%- if val is string %}
+                        {{- '"' + key + '": "' + val + '"' }}
+                    {%- else %}
+                        {{- '"' + key + '": ' + val|tojson }}
+                    {%- endif %}
+                    {%- if not loop.last %}
+                        {{- ", " }}
+                    {%- endif %}
+                {%- endfor %}
+                {{- "}}" }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- else %}
+                    {{- "]" }}
+                {%- endif %}
+            {%- endfor %}
+            {{- "[/AVAILABLE_TOOLS]" }}
+            {%- endif %}
+        {%- if loop.last and system_message is defined %}
+            {{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
+        {%- else %}
+            {{- "[INST]" + message["content"] + "[/INST]" }}
+        {%- endif %}
+    {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
+        {{- "[TOOL_CALLS][" }}
+        {%- for tool_call in message.tool_calls %}
+            {%- set out = tool_call.function|tojson %}
+            {{- out[:-1] }}
+            {%- if not tool_call.id is defined or tool_call.id|length != 9 %}
+                {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+            {%- endif %}
+            {{- ', "id": "' + tool_call.id + '"}' }}
+            {%- if not loop.last %}
+                {{- ", " }}
+            {%- else %}
+                {{- "]" + eos_token }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif message["role"] == "assistant" %}
+        {{- message["content"] + eos_token}}
+    {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+        {%- if message.content is defined and message.content.content is defined %}
+            {%- set content = message.content.content %}
+        {%- else %}
+            {%- set content = message.content %}
+        {%- endif %}
+        {{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
+        {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
+            {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+        {%- endif %}
+        {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
+    {%- else %}
+        {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+    {%- endif %}
+{%- endfor %}

checkpoint-16479/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd851ec28b5c0c4dd7afd96c5004aaf4a3696dbe3436d1b697d210e4aaf26b5c
+size 232287717

checkpoint-16479/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8854e0fa10e1b45cfa48c7feb900f4b8546450d9fd413714c4084f1d9515c84
+size 14645

checkpoint-16479/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96b1320f9beac9c44d55aa2e8aea84b773a5c68097bede396981fa8122cd86bd
+size 1465

checkpoint-16479/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-16479/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0240ce510f08e6c2041724e9043e33be9d251d1e4a4d94eb68cd47b954b61d2
+size 17078292

checkpoint-16479/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34a581bf9b76f74a5a4e25651fbaa01e61b75b19cbbfa8df2370c772924bc5d7
+size 11409

checkpoint-21972/README.md

@@ -0,0 +1,208 @@
+---
+base_model: mistralai/Mistral-Nemo-Instruct-2407
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- axolotl
+- base_model:adapter:mistralai/Mistral-Nemo-Instruct-2407
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

checkpoint-21972/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-Nemo-Instruct-2407",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "down_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "gate_proj",
+    "up_proj"
+  ],
+  "target_parameters": [],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-21972/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d05f26e62b3a8a1de3967e655ff60545d195d6163cd2ba59e5c4a2d601dafc2
+size 456206152

checkpoint-21972/chat_template.jinja

@@ -0,0 +1,87 @@
+{%- if messages[0]["role"] == "system" %}
+    {%- set system_message = messages[0]["content"] %}
+    {%- set loop_messages = messages[1:] %}
+{%- else %}
+    {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
+{%- set ns = namespace() %}
+{%- set ns.index = 0 %}
+{%- for message in loop_messages %}
+    {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
+        {%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
+            {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+        {%- endif %}
+        {%- set ns.index = ns.index + 1 %}
+    {%- endif %}
+{%- endfor %}
+
+{{- bos_token }}
+{%- for message in loop_messages %}
+    {%- if message["role"] == "user" %}
+        {%- if tools is not none and (message == user_messages[-1]) %}
+            {{- "[AVAILABLE_TOOLS][" }}
+            {%- for tool in tools %}
+                {%- set tool = tool.function %}
+                {{- '{"type": "function", "function": {' }}
+                {%- for key, val in tool.items() if key != "return" %}
+                    {%- if val is string %}
+                        {{- '"' + key + '": "' + val + '"' }}
+                    {%- else %}
+                        {{- '"' + key + '": ' + val|tojson }}
+                    {%- endif %}
+                    {%- if not loop.last %}
+                        {{- ", " }}
+                    {%- endif %}
+                {%- endfor %}
+                {{- "}}" }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- else %}
+                    {{- "]" }}
+                {%- endif %}
+            {%- endfor %}
+            {{- "[/AVAILABLE_TOOLS]" }}
+            {%- endif %}
+        {%- if loop.last and system_message is defined %}
+            {{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
+        {%- else %}
+            {{- "[INST]" + message["content"] + "[/INST]" }}
+        {%- endif %}
+    {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
+        {{- "[TOOL_CALLS][" }}
+        {%- for tool_call in message.tool_calls %}
+            {%- set out = tool_call.function|tojson %}
+            {{- out[:-1] }}
+            {%- if not tool_call.id is defined or tool_call.id|length != 9 %}
+                {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+            {%- endif %}
+            {{- ', "id": "' + tool_call.id + '"}' }}
+            {%- if not loop.last %}
+                {{- ", " }}
+            {%- else %}
+                {{- "]" + eos_token }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif message["role"] == "assistant" %}
+        {{- message["content"] + eos_token}}
+    {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+        {%- if message.content is defined and message.content.content is defined %}
+            {%- set content = message.content.content %}
+        {%- else %}
+            {%- set content = message.content %}
+        {%- endif %}
+        {{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
+        {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
+            {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
+        {%- endif %}
+        {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
+    {%- else %}
+        {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+    {%- endif %}
+{%- endfor %}

checkpoint-21972/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a658036cf46757a9cbefa25fd612796286c01268c03c37ce24ca4ba039f4cac
+size 232287717

checkpoint-21972/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab8eb9b74a62058fa9e389d926e774dcfd8ee2df1bc2991789e217fe550eb0eb
+size 14645

checkpoint-21972/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31c64b0ecfc0176480a9e0daacd26cc1b5ac7c881653e0ea1d0f0d8a29b232b8
+size 1465

checkpoint-21972/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-21972/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0240ce510f08e6c2041724e9043e33be9d251d1e4a4d94eb68cd47b954b61d2
+size 17078292

checkpoint-21972/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34a581bf9b76f74a5a4e25651fbaa01e61b75b19cbbfa8df2370c772924bc5d7
+size 11409

checkpoint-27465/README.md

@@ -0,0 +1,208 @@
+---
+base_model: mistralai/Mistral-Nemo-Instruct-2407
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- axolotl
+- base_model:adapter:mistralai/Mistral-Nemo-Instruct-2407
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1