Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+chatbot.gguf filter=lfs diff=lfs merge=lfs -text

adapter/README.md

@@ -0,0 +1,202 @@
+---
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

adapter/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "up_proj",
+    "o_proj",
+    "gate_proj",
+    "k_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80638e126558371903ea539b62031fab8a64f1f81d71a0e9b8ed3f940c7c3d35
+size 50503544

adapter/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

adapter/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

adapter/tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 2048,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

all_results.json

@@ -0,0 +1,11 @@
+{
+    "epoch": 2.0,
+    "eval_loss": 1.0187978744506836,
+    "eval_runtime": 229.1117,
+    "eval_samples_per_second": 4.365,
+    "eval_steps_per_second": 0.546,
+    "train_loss": 1.089675091592889,
+    "train_runtime": 22210.9127,
+    "train_samples_per_second": 1.711,
+    "train_steps_per_second": 0.214
+}

chatbot.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f068420d316d998e88097d8eeb7f61d52d59411a106b4d03d8da8258fa83e5f2
+size 2201017760

checkpoint-2000/README.md

@@ -0,0 +1,202 @@
+---
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-2000/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "up_proj",
+    "o_proj",
+    "gate_proj",
+    "k_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-2000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eea5111ad8851fc8b52a655d65ee1dbfa3ebf00fe029a418d53fe9b97709c8b3
+size 50503544

checkpoint-2000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19f244ce106a76c51877ed4264142c56462bb2a821126fae9dd9340e9181a8eb
+size 101184122

checkpoint-2000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce27cc7a82bd52c322534b379fa4bb758484904d51b3c1fc97e7883d6bae8111
+size 14244

checkpoint-2000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68c7719d2775cc469e286a97411a0b98fe5718c741fc31967879d0a0f23def8b
+size 1064

checkpoint-2000/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-2000/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

checkpoint-2000/tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 2048,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-2000/trainer_state.json

@@ -0,0 +1,1436 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.8421052631578947,
+  "eval_steps": 2000,
+  "global_step": 2000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0,
+      "grad_norm": 1.0191779136657715,
+      "learning_rate": 8.403361344537815e-07,
+      "loss": 1.4757,
+      "step": 1
+    },
+    {
+      "epoch": 0.0,
+      "grad_norm": 0.947059690952301,
+      "learning_rate": 8.403361344537817e-06,
+      "loss": 1.5145,
+      "step": 10
+    },
+    {
+      "epoch": 0.01,
+      "grad_norm": 0.9549775719642639,
+      "learning_rate": 1.6806722689075634e-05,
+      "loss": 1.5546,
+      "step": 20
+    },
+    {
+      "epoch": 0.01,
+      "grad_norm": 1.1984186172485352,
+      "learning_rate": 2.5210084033613446e-05,
+      "loss": 1.6809,
+      "step": 30
+    },
+    {
+      "epoch": 0.02,
+      "grad_norm": 0.9854737520217896,
+      "learning_rate": 3.361344537815127e-05,
+      "loss": 1.8074,
+      "step": 40
+    },
+    {
+      "epoch": 0.02,
+      "grad_norm": 1.3563014268875122,
+      "learning_rate": 4.201680672268908e-05,
+      "loss": 1.699,
+      "step": 50
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.6414602398872375,
+      "learning_rate": 5.042016806722689e-05,
+      "loss": 1.3419,
+      "step": 60
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.580333948135376,
+      "learning_rate": 5.882352941176471e-05,
+      "loss": 1.2524,
+      "step": 70
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.6972444653511047,
+      "learning_rate": 6.722689075630254e-05,
+      "loss": 1.4261,
+      "step": 80
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 0.8142390847206116,
+      "learning_rate": 7.563025210084033e-05,
+      "loss": 1.5472,
+      "step": 90
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 1.9213963747024536,
+      "learning_rate": 8.403361344537815e-05,
+      "loss": 1.5581,
+      "step": 100
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.7875838279724121,
+      "learning_rate": 9.243697478991598e-05,
+      "loss": 1.2596,
+      "step": 110
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.6198441982269287,
+      "learning_rate": 0.00010084033613445378,
+      "loss": 1.2901,
+      "step": 120
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.6708433032035828,
+      "learning_rate": 0.00010924369747899159,
+      "loss": 1.3087,
+      "step": 130
+    },
+    {
+      "epoch": 0.06,
+      "grad_norm": 0.8415197730064392,
+      "learning_rate": 0.00011764705882352942,
+      "loss": 1.4806,
+      "step": 140
+    },
+    {
+      "epoch": 0.06,
+      "grad_norm": 1.7113947868347168,
+      "learning_rate": 0.00012605042016806722,
+      "loss": 1.5082,
+      "step": 150
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.7543832659721375,
+      "learning_rate": 0.00013445378151260507,
+      "loss": 1.1818,
+      "step": 160
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.577403724193573,
+      "learning_rate": 0.00014285714285714287,
+      "loss": 1.2308,
+      "step": 170
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.6748159527778625,
+      "learning_rate": 0.00015126050420168066,
+      "loss": 1.3738,
+      "step": 180
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.8842242360115051,
+      "learning_rate": 0.0001596638655462185,
+      "loss": 1.4544,
+      "step": 190
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 1.7435425519943237,
+      "learning_rate": 0.0001680672268907563,
+      "loss": 1.4997,
+      "step": 200
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 0.5861243009567261,
+      "learning_rate": 0.00017647058823529413,
+      "loss": 1.1774,
+      "step": 210
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 0.5415711998939514,
+      "learning_rate": 0.00018487394957983195,
+      "loss": 1.0529,
+      "step": 220
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.6118600964546204,
+      "learning_rate": 0.00019327731092436975,
+      "loss": 1.3117,
+      "step": 230
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.8368746042251587,
+      "learning_rate": 0.00019999990304024961,
+      "loss": 1.4745,
+      "step": 240
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 1.086878776550293,
+      "learning_rate": 0.00019999650946872738,
+      "loss": 1.3995,
+      "step": 250
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 0.6036991477012634,
+      "learning_rate": 0.00019998826809770645,
+      "loss": 1.2325,
+      "step": 260
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 0.5606744289398193,
+      "learning_rate": 0.0001999751793267259,
+      "loss": 1.2043,
+      "step": 270
+    },
+    {
+      "epoch": 0.12,
+      "grad_norm": 0.6112542748451233,
+      "learning_rate": 0.00019995724379032526,
+      "loss": 1.2512,
+      "step": 280
+    },
+    {
+      "epoch": 0.12,
+      "grad_norm": 0.7892420887947083,
+      "learning_rate": 0.00019993446235801367,
+      "loss": 1.3604,
+      "step": 290
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 1.2104417085647583,
+      "learning_rate": 0.00019990683613422787,
+      "loss": 1.4282,
+      "step": 300
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 0.5284091234207153,
+      "learning_rate": 0.0001998743664582786,
+      "loss": 1.1223,
+      "step": 310
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 0.5215238928794861,
+      "learning_rate": 0.0001998370549042855,
+      "loss": 1.1399,
+      "step": 320
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 0.6061808466911316,
+      "learning_rate": 0.0001997949032811011,
+      "loss": 1.2432,
+      "step": 330
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 0.7220178246498108,
+      "learning_rate": 0.0001997479136322229,
+      "loss": 1.4737,
+      "step": 340
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 1.035914421081543,
+      "learning_rate": 0.00019969608823569433,
+      "loss": 1.5087,
+      "step": 350
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 0.5120232701301575,
+      "learning_rate": 0.00019963942960399446,
+      "loss": 1.1583,
+      "step": 360
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.5209968686103821,
+      "learning_rate": 0.00019957794048391602,
+      "loss": 1.1494,
+      "step": 370
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.5688252449035645,
+      "learning_rate": 0.0001995116238564323,
+      "loss": 1.2648,
+      "step": 380
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.7350272536277771,
+      "learning_rate": 0.00019944048293655265,
+      "loss": 1.3836,
+      "step": 390
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 1.117912769317627,
+      "learning_rate": 0.00019936452117316663,
+      "loss": 1.3725,
+      "step": 400
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 0.5087815523147583,
+      "learning_rate": 0.00019928374224887678,
+      "loss": 1.0783,
+      "step": 410
+    },
+    {
+      "epoch": 0.18,
+      "grad_norm": 0.4944276213645935,
+      "learning_rate": 0.00019919815007982003,
+      "loss": 1.0936,
+      "step": 420
+    },
+    {
+      "epoch": 0.18,
+      "grad_norm": 0.5578716397285461,
+      "learning_rate": 0.000199107748815478,
+      "loss": 1.1705,
+      "step": 430
+    },
+    {
+      "epoch": 0.19,
+      "grad_norm": 0.670518696308136,
+      "learning_rate": 0.00019901254283847573,
+      "loss": 1.3099,
+      "step": 440
+    },
+    {
+      "epoch": 0.19,
+      "grad_norm": 0.9550499320030212,
+      "learning_rate": 0.00019891253676436915,
+      "loss": 1.3971,
+      "step": 450
+    },
+    {
+      "epoch": 0.19,
+      "grad_norm": 0.48883694410324097,
+      "learning_rate": 0.00019880773544142148,
+      "loss": 1.1091,
+      "step": 460
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.5331695675849915,
+      "learning_rate": 0.00019869814395036807,
+      "loss": 1.1169,
+      "step": 470
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.6103072166442871,
+      "learning_rate": 0.0001985837676041701,
+      "loss": 1.2462,
+      "step": 480
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 0.7295002937316895,
+      "learning_rate": 0.00019846461194775712,
+      "loss": 1.3711,
+      "step": 490
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 1.003199815750122,
+      "learning_rate": 0.000198340682757758,
+      "loss": 1.3987,
+      "step": 500
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 0.49524402618408203,
+      "learning_rate": 0.00019821198604222113,
+      "loss": 1.1002,
+      "step": 510
+    },
+    {
+      "epoch": 0.22,
+      "grad_norm": 0.4883529543876648,
+      "learning_rate": 0.00019807852804032305,
+      "loss": 1.0895,
+      "step": 520
+    },
+    {
+      "epoch": 0.22,
+      "grad_norm": 0.5645517110824585,
+      "learning_rate": 0.00019794031522206594,
+      "loss": 1.1673,
+      "step": 530
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 0.7130540609359741,
+      "learning_rate": 0.00019779735428796395,
+      "loss": 1.2775,
+      "step": 540
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 1.035349726676941,
+      "learning_rate": 0.00019764965216871846,
+      "loss": 1.3272,
+      "step": 550
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.5434763431549072,
+      "learning_rate": 0.00019749721602488207,
+      "loss": 1.1483,
+      "step": 560
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.472807914018631,
+      "learning_rate": 0.0001973400532465113,
+      "loss": 1.0666,
+      "step": 570
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.575896143913269,
+      "learning_rate": 0.00019717817145280844,
+      "loss": 1.122,
+      "step": 580
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 0.6578025817871094,
+      "learning_rate": 0.00019701157849175228,
+      "loss": 1.2858,
+      "step": 590
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 1.2908504009246826,
+      "learning_rate": 0.0001968402824397174,
+      "loss": 1.3983,
+      "step": 600
+    },
+    {
+      "epoch": 0.26,
+      "grad_norm": 0.5685085654258728,
+      "learning_rate": 0.0001966642916010829,
+      "loss": 1.0992,
+      "step": 610
+    },
+    {
+      "epoch": 0.26,
+      "grad_norm": 0.4999242126941681,
+      "learning_rate": 0.0001964836145078295,
+      "loss": 1.1434,
+      "step": 620
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.5502079129219055,
+      "learning_rate": 0.00019629825991912623,
+      "loss": 1.1573,
+      "step": 630
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.6397935748100281,
+      "learning_rate": 0.0001961082368209055,
+      "loss": 1.3418,
+      "step": 640
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.9650578498840332,
+      "learning_rate": 0.00019591355442542775,
+      "loss": 1.3298,
+      "step": 650
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 0.49895137548446655,
+      "learning_rate": 0.00019571422217083454,
+      "loss": 1.0907,
+      "step": 660
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 0.49023300409317017,
+      "learning_rate": 0.00019551024972069126,
+      "loss": 1.1305,
+      "step": 670
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 0.5746453404426575,
+      "learning_rate": 0.00019530164696351848,
+      "loss": 1.1865,
+      "step": 680
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 0.6834538578987122,
+      "learning_rate": 0.00019508842401231257,
+      "loss": 1.3113,
+      "step": 690
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 1.0113998651504517,
+      "learning_rate": 0.0001948705912040556,
+      "loss": 1.3686,
+      "step": 700
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.5399815440177917,
+      "learning_rate": 0.0001946481590992139,
+      "loss": 1.1602,
+      "step": 710
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.5135788321495056,
+      "learning_rate": 0.00019442113848122636,
+      "loss": 1.0881,
+      "step": 720
+    },
+    {
+      "epoch": 0.31,
+      "grad_norm": 0.5519896149635315,
+      "learning_rate": 0.00019418954035598152,
+      "loss": 1.1885,
+      "step": 730
+    },
+    {
+      "epoch": 0.31,
+      "grad_norm": 0.7329438328742981,
+      "learning_rate": 0.000193953375951284,
+      "loss": 1.2668,
+      "step": 740
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 1.0599132776260376,
+      "learning_rate": 0.00019371265671631037,
+      "loss": 1.3524,
+      "step": 750
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.5613999962806702,
+      "learning_rate": 0.00019346739432105374,
+      "loss": 1.0983,
+      "step": 760
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.5169478058815002,
+      "learning_rate": 0.00019321760065575834,
+      "loss": 1.1699,
+      "step": 770
+    },
+    {
+      "epoch": 0.33,
+      "grad_norm": 0.5800842642784119,
+      "learning_rate": 0.00019296328783034294,
+      "loss": 1.2105,
+      "step": 780
+    },
+    {
+      "epoch": 0.33,
+      "grad_norm": 0.7089204788208008,
+      "learning_rate": 0.00019270446817381377,
+      "loss": 1.3458,
+      "step": 790
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 0.8912138342857361,
+      "learning_rate": 0.00019244115423366678,
+      "loss": 1.3386,
+      "step": 800
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 0.5033462047576904,
+      "learning_rate": 0.00019217335877527947,
+      "loss": 1.0824,
+      "step": 810
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.5139713287353516,
+      "learning_rate": 0.0001919010947812918,
+      "loss": 1.0122,
+      "step": 820
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.5905170440673828,
+      "learning_rate": 0.00019162437545097719,
+      "loss": 1.2924,
+      "step": 830
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.715568482875824,
+      "learning_rate": 0.00019134321419960203,
+      "loss": 1.2982,
+      "step": 840
+    },
+    {
+      "epoch": 0.36,
+      "grad_norm": 1.046240210533142,
+      "learning_rate": 0.00019105762465777598,
+      "loss": 1.4013,
+      "step": 850
+    },
+    {
+      "epoch": 0.36,
+      "grad_norm": 0.49822917580604553,
+      "learning_rate": 0.00019076762067079059,
+      "loss": 1.0633,
+      "step": 860
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 0.47567543387413025,
+      "learning_rate": 0.0001904732162979485,
+      "loss": 1.0497,
+      "step": 870
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 0.5331007242202759,
+      "learning_rate": 0.0001901744258118816,
+      "loss": 1.166,
+      "step": 880
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 0.7400739789009094,
+      "learning_rate": 0.00018987126369785916,
+      "loss": 1.3527,
+      "step": 890
+    },
+    {
+      "epoch": 0.38,
+      "grad_norm": 1.1264495849609375,
+      "learning_rate": 0.00018956374465308563,
+      "loss": 1.3475,
+      "step": 900
+    },
+    {
+      "epoch": 0.38,
+      "grad_norm": 0.5124080777168274,
+      "learning_rate": 0.00018925188358598813,
+      "loss": 1.05,
+      "step": 910
+    },
+    {
+      "epoch": 0.39,
+      "grad_norm": 0.522303581237793,
+      "learning_rate": 0.00018893569561549364,
+      "loss": 1.156,
+      "step": 920
+    },
+    {
+      "epoch": 0.39,
+      "grad_norm": 0.5782234072685242,
+      "learning_rate": 0.00018861519607029606,
+      "loss": 1.158,
+      "step": 930
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.7259439826011658,
+      "learning_rate": 0.00018829040048811305,
+      "loss": 1.306,
+      "step": 940
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.9230518937110901,
+      "learning_rate": 0.0001879613246149329,
+      "loss": 1.4171,
+      "step": 950
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.5070952773094177,
+      "learning_rate": 0.00018762798440425096,
+      "loss": 1.0647,
+      "step": 960
+    },
+    {
+      "epoch": 0.41,
+      "grad_norm": 0.540289044380188,
+      "learning_rate": 0.00018729039601629634,
+      "loss": 1.0823,
+      "step": 970
+    },
+    {
+      "epoch": 0.41,
+      "grad_norm": 0.6044334173202515,
+      "learning_rate": 0.00018694857581724855,
+      "loss": 1.2258,
+      "step": 980
+    },
+    {
+      "epoch": 0.42,
+      "grad_norm": 0.6775342226028442,
+      "learning_rate": 0.00018660254037844388,
+      "loss": 1.3045,
+      "step": 990
+    },
+    {
+      "epoch": 0.42,
+      "grad_norm": 1.0051116943359375,
+      "learning_rate": 0.00018625230647557217,
+      "loss": 1.3275,
+      "step": 1000
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.522710382938385,
+      "learning_rate": 0.00018589789108786344,
+      "loss": 1.1284,
+      "step": 1010
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.527962327003479,
+      "learning_rate": 0.00018553931139726485,
+      "loss": 1.0688,
+      "step": 1020
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.5457212924957275,
+      "learning_rate": 0.0001851765847876076,
+      "loss": 1.1434,
+      "step": 1030
+    },
+    {
+      "epoch": 0.44,
+      "grad_norm": 0.6518785357475281,
+      "learning_rate": 0.00018480972884376434,
+      "loss": 1.2961,
+      "step": 1040
+    },
+    {
+      "epoch": 0.44,
+      "grad_norm": 1.0324338674545288,
+      "learning_rate": 0.00018443876135079638,
+      "loss": 1.407,
+      "step": 1050
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.6092559099197388,
+      "learning_rate": 0.0001840637002930918,
+      "loss": 1.0535,
+      "step": 1060
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.5149545073509216,
+      "learning_rate": 0.00018368456385349334,
+      "loss": 1.0546,
+      "step": 1070
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.6056516766548157,
+      "learning_rate": 0.0001833013704124169,
+      "loss": 1.1735,
+      "step": 1080
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 0.7382489442825317,
+      "learning_rate": 0.00018291413854696077,
+      "loss": 1.3044,
+      "step": 1090
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 0.9548698663711548,
+      "learning_rate": 0.0001825228870300046,
+      "loss": 1.3571,
+      "step": 1100
+    },
+    {
+      "epoch": 0.47,
+      "grad_norm": 0.5410871505737305,
+      "learning_rate": 0.0001821276348292995,
+      "loss": 1.097,
+      "step": 1110
+    },
+    {
+      "epoch": 0.47,
+      "grad_norm": 0.5245351195335388,
+      "learning_rate": 0.00018172840110654862,
+      "loss": 1.0923,
+      "step": 1120
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.6045931577682495,
+      "learning_rate": 0.00018132520521647785,
+      "loss": 1.2115,
+      "step": 1130
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.6429124474525452,
+      "learning_rate": 0.00018091806670589788,
+      "loss": 1.2736,
+      "step": 1140
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.9760797619819641,
+      "learning_rate": 0.0001805070053127563,
+      "loss": 1.2964,
+      "step": 1150
+    },
+    {
+      "epoch": 0.49,
+      "grad_norm": 0.48860272765159607,
+      "learning_rate": 0.0001800920409651809,
+      "loss": 1.0942,
+      "step": 1160
+    },
+    {
+      "epoch": 0.49,
+      "grad_norm": 0.5090013146400452,
+      "learning_rate": 0.00017967319378051343,
+      "loss": 1.0925,
+      "step": 1170
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.5822442770004272,
+      "learning_rate": 0.00017925048406433437,
+      "loss": 1.1583,
+      "step": 1180
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.7363468408584595,
+      "learning_rate": 0.00017882393230947855,
+      "loss": 1.3283,
+      "step": 1190
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.840218186378479,
+      "learning_rate": 0.00017839355919504164,
+      "loss": 1.3098,
+      "step": 1200
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.5360991954803467,
+      "learning_rate": 0.00017795938558537753,
+      "loss": 1.1028,
+      "step": 1210
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.5553500056266785,
+      "learning_rate": 0.00017752143252908705,
+      "loss": 1.0737,
+      "step": 1220
+    },
+    {
+      "epoch": 0.52,
+      "grad_norm": 0.5366541743278503,
+      "learning_rate": 0.00017707972125799735,
+      "loss": 1.1918,
+      "step": 1230
+    },
+    {
+      "epoch": 0.52,
+      "grad_norm": 0.7474969029426575,
+      "learning_rate": 0.0001766342731861327,
+      "loss": 1.2756,
+      "step": 1240
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 1.0241633653640747,
+      "learning_rate": 0.00017618510990867624,
+      "loss": 1.298,
+      "step": 1250
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 0.5135468244552612,
+      "learning_rate": 0.00017573225320092323,
+      "loss": 0.9818,
+      "step": 1260
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 0.5308533906936646,
+      "learning_rate": 0.00017527572501722512,
+      "loss": 1.0729,
+      "step": 1270
+    },
+    {
+      "epoch": 0.54,
+      "grad_norm": 0.6265894174575806,
+      "learning_rate": 0.00017481554748992553,
+      "loss": 1.1791,
+      "step": 1280
+    },
+    {
+      "epoch": 0.54,
+      "grad_norm": 0.7500415444374084,
+      "learning_rate": 0.00017435174292828695,
+      "loss": 1.2722,
+      "step": 1290
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 1.0609970092773438,
+      "learning_rate": 0.00017388433381740952,
+      "loss": 1.3034,
+      "step": 1300
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 0.5286604166030884,
+      "learning_rate": 0.00017341334281714064,
+      "loss": 0.9932,
+      "step": 1310
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.5054761171340942,
+      "learning_rate": 0.00017293879276097677,
+      "loss": 1.0386,
+      "step": 1320
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.6337370872497559,
+      "learning_rate": 0.00017246070665495608,
+      "loss": 1.1341,
+      "step": 1330
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.7288526892662048,
+      "learning_rate": 0.00017197910767654352,
+      "loss": 1.2551,
+      "step": 1340
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 1.0159952640533447,
+      "learning_rate": 0.0001714940191735069,
+      "loss": 1.3048,
+      "step": 1350
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 0.5907754898071289,
+      "learning_rate": 0.00017100546466278504,
+      "loss": 1.0322,
+      "step": 1360
+    },
+    {
+      "epoch": 0.58,
+      "grad_norm": 0.5079711675643921,
+      "learning_rate": 0.00017051346782934786,
+      "loss": 1.0071,
+      "step": 1370
+    },
+    {
+      "epoch": 0.58,
+      "grad_norm": 0.6100701689720154,
+      "learning_rate": 0.00017001805252504795,
+      "loss": 1.1588,
+      "step": 1380
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 0.7802088260650635,
+      "learning_rate": 0.00016951924276746425,
+      "loss": 1.2911,
+      "step": 1390
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 1.0676313638687134,
+      "learning_rate": 0.00016901706273873777,
+      "loss": 1.2942,
+      "step": 1400
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 0.527285099029541,
+      "learning_rate": 0.00016851153678439922,
+      "loss": 1.0107,
+      "step": 1410
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.5518642663955688,
+      "learning_rate": 0.00016800268941218876,
+      "loss": 1.0683,
+      "step": 1420
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.6224582195281982,
+      "learning_rate": 0.0001674905452908678,
+      "loss": 1.1685,
+      "step": 1430
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 0.6205934882164001,
+      "learning_rate": 0.0001669751292490231,
+      "loss": 1.302,
+      "step": 1440
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 1.0423789024353027,
+      "learning_rate": 0.0001664564662738632,
+      "loss": 1.2654,
+      "step": 1450
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 0.5314890742301941,
+      "learning_rate": 0.00016593458151000688,
+      "loss": 1.083,
+      "step": 1460
+    },
+    {
+      "epoch": 0.62,
+      "grad_norm": 0.5361500978469849,
+      "learning_rate": 0.00016540950025826422,
+      "loss": 1.0952,
+      "step": 1470
+    },
+    {
+      "epoch": 0.62,
+      "grad_norm": 0.5990237593650818,
+      "learning_rate": 0.00016488124797441004,
+      "loss": 1.1923,
+      "step": 1480
+    },
+    {
+      "epoch": 0.63,
+      "grad_norm": 0.7383199334144592,
+      "learning_rate": 0.00016434985026794987,
+      "loss": 1.288,
+      "step": 1490
+    },
+    {
+      "epoch": 0.63,
+      "grad_norm": 1.0500961542129517,
+      "learning_rate": 0.0001638153329008782,
+      "loss": 1.318,
+      "step": 1500
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.5643898248672485,
+      "learning_rate": 0.00016327772178642986,
+      "loss": 1.0641,
+      "step": 1510
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.509543776512146,
+      "learning_rate": 0.0001627370429878235,
+      "loss": 1.0216,
+      "step": 1520
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.6048777103424072,
+      "learning_rate": 0.0001621933227169981,
+      "loss": 1.136,
+      "step": 1530
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 0.7480033040046692,
+      "learning_rate": 0.0001616465873333423,
+      "loss": 1.2798,
+      "step": 1540
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 1.0061513185501099,
+      "learning_rate": 0.00016109686334241655,
+      "loss": 1.2482,
+      "step": 1550
+    },
+    {
+      "epoch": 0.66,
+      "grad_norm": 0.5828456282615662,
+      "learning_rate": 0.0001605441773946679,
+      "loss": 1.1134,
+      "step": 1560
+    },
+    {
+      "epoch": 0.66,
+      "grad_norm": 0.5483405590057373,
+      "learning_rate": 0.00015998855628413823,
+      "loss": 1.0308,
+      "step": 1570
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.6139534711837769,
+      "learning_rate": 0.00015943002694716519,
+      "loss": 1.1093,
+      "step": 1580
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.6477651000022888,
+      "learning_rate": 0.0001588686164610763,
+      "loss": 1.2047,
+      "step": 1590
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.9526233077049255,
+      "learning_rate": 0.00015830435204287628,
+      "loss": 1.3359,
+      "step": 1600
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 0.5442110896110535,
+      "learning_rate": 0.00015773726104792768,
+      "loss": 1.0236,
+      "step": 1610
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 0.5667725801467896,
+      "learning_rate": 0.0001571673709686245,
+      "loss": 0.9956,
+      "step": 1620
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.567449688911438,
+      "learning_rate": 0.00015659470943305955,
+      "loss": 1.1698,
+      "step": 1630
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.7154501080513,
+      "learning_rate": 0.00015601930420368486,
+      "loss": 1.2682,
+      "step": 1640
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.9630643725395203,
+      "learning_rate": 0.00015544118317596605,
+      "loss": 1.3124,
+      "step": 1650
+    },
+    {
+      "epoch": 0.7,
+      "grad_norm": 0.5698636770248413,
+      "learning_rate": 0.0001548603743770296,
+      "loss": 1.0047,
+      "step": 1660
+    },
+    {
+      "epoch": 0.7,
+      "grad_norm": 0.538144588470459,
+      "learning_rate": 0.0001542769059643045,
+      "loss": 0.9859,
+      "step": 1670
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 0.6153010725975037,
+      "learning_rate": 0.00015369080622415683,
+      "loss": 1.1059,
+      "step": 1680
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 0.7482182383537292,
+      "learning_rate": 0.00015310210357051863,
+      "loss": 1.2589,
+      "step": 1690
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 1.014577031135559,
+      "learning_rate": 0.00015251082654351044,
+      "loss": 1.2322,
+      "step": 1700
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.5590934157371521,
+      "learning_rate": 0.00015191700380805752,
+      "loss": 1.0283,
+      "step": 1710
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.5307114124298096,
+      "learning_rate": 0.00015132066415250042,
+      "loss": 1.0196,
+      "step": 1720
+    },
+    {
+      "epoch": 0.73,
+      "grad_norm": 0.6487598419189453,
+      "learning_rate": 0.000150721836487199,
+      "loss": 1.0595,
+      "step": 1730
+    },
+    {
+      "epoch": 0.73,
+      "grad_norm": 0.8052285313606262,
+      "learning_rate": 0.00015012054984313116,
+      "loss": 1.2083,
+      "step": 1740
+    },
+    {
+      "epoch": 0.74,
+      "grad_norm": 1.1172990798950195,
+      "learning_rate": 0.00014951683337048537,
+      "loss": 1.2899,
+      "step": 1750
+    },
+    {
+      "epoch": 0.74,
+      "grad_norm": 0.5224259495735168,
+      "learning_rate": 0.00014891071633724737,
+      "loss": 1.0475,
+      "step": 1760
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.5798863768577576,
+      "learning_rate": 0.00014830222812778126,
+      "loss": 0.9953,
+      "step": 1770
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.6029157638549805,
+      "learning_rate": 0.00014769139824140516,
+      "loss": 1.1619,
+      "step": 1780
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.7881869673728943,
+      "learning_rate": 0.00014707825629096084,
+      "loss": 1.2333,
+      "step": 1790
+    },
+    {
+      "epoch": 0.76,
+      "grad_norm": 0.9496088624000549,
+      "learning_rate": 0.00014646283200137828,
+      "loss": 1.2424,
+      "step": 1800
+    },
+    {
+      "epoch": 0.76,
+      "grad_norm": 0.5425153374671936,
+      "learning_rate": 0.00014584515520823453,
+      "loss": 0.9886,
+      "step": 1810
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.5590393543243408,
+      "learning_rate": 0.0001452252558563073,
+      "loss": 1.0041,
+      "step": 1820
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.5956026911735535,
+      "learning_rate": 0.00014460316399812325,
+      "loss": 1.1507,
+      "step": 1830
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.7304853796958923,
+      "learning_rate": 0.00014397890979250108,
+      "loss": 1.2183,
+      "step": 1840
+    },
+    {
+      "epoch": 0.78,
+      "grad_norm": 1.0130630731582642,
+      "learning_rate": 0.0001433525235030894,
+      "loss": 1.2944,
+      "step": 1850
+    },
+    {
+      "epoch": 0.78,
+      "grad_norm": 0.5265572667121887,
+      "learning_rate": 0.00014272403549689955,
+      "loss": 1.0166,
+      "step": 1860
+    },
+    {
+      "epoch": 0.79,
+      "grad_norm": 0.5423105955123901,
+      "learning_rate": 0.0001420934762428335,
+      "loss": 0.9559,
+      "step": 1870
+    },
+    {
+      "epoch": 0.79,
+      "grad_norm": 0.6281871199607849,
+      "learning_rate": 0.00014146087631020665,
+      "loss": 1.064,
+      "step": 1880
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.7130801677703857,
+      "learning_rate": 0.00014082626636726582,
+      "loss": 1.1955,
+      "step": 1890
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 1.038220763206482,
+      "learning_rate": 0.00014018967717970254,
+      "loss": 1.2621,
+      "step": 1900
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.5577895045280457,
+      "learning_rate": 0.00013955113960916146,
+      "loss": 0.9577,
+      "step": 1910
+    },
+    {
+      "epoch": 0.81,
+      "grad_norm": 0.526157796382904,
+      "learning_rate": 0.0001389106846117443,
+      "loss": 1.0109,
+      "step": 1920
+    },
+    {
+      "epoch": 0.81,
+      "grad_norm": 0.6196286678314209,
+      "learning_rate": 0.000138268343236509,
+      "loss": 1.1256,
+      "step": 1930
+    },
+    {
+      "epoch": 0.82,
+      "grad_norm": 0.7702537178993225,
+      "learning_rate": 0.00013762414662396444,
+      "loss": 1.2524,
+      "step": 1940
+    },
+    {
+      "epoch": 0.82,
+      "grad_norm": 0.8746676445007324,
+      "learning_rate": 0.00013697812600456093,
+      "loss": 1.288,
+      "step": 1950
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.5723572969436646,
+      "learning_rate": 0.000136330312697176,
+      "loss": 0.995,
+      "step": 1960
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.5687806606292725,
+      "learning_rate": 0.00013568073810759615,
+      "loss": 1.042,
+      "step": 1970
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.5694392323493958,
+      "learning_rate": 0.00013502943372699426,
+      "loss": 1.0513,
+      "step": 1980
+    },
+    {
+      "epoch": 0.84,
+      "grad_norm": 0.770982563495636,
+      "learning_rate": 0.00013437643113040301,
+      "loss": 1.2595,
+      "step": 1990
+    },
+    {
+      "epoch": 0.84,
+      "grad_norm": 1.0096752643585205,
+      "learning_rate": 0.000133721761975184,
+      "loss": 1.2178,
+      "step": 2000
+    },
+    {
+      "epoch": 0.84,
+      "eval_loss": 1.0813658237457275,
+      "eval_runtime": 229.3235,
+      "eval_samples_per_second": 4.361,
+      "eval_steps_per_second": 0.545,
+      "step": 2000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4750,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 2000,
+  "total_flos": 2.0110035829948416e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32056b300030c8a314c8642f76d2f1834a17c0eb91fd7f49ef60c14a2491fb22
+size 4856

checkpoint-4000/README.md

@@ -0,0 +1,202 @@
+---
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-4000/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "up_proj",
+    "o_proj",
+    "gate_proj",
+    "k_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-4000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a0ee7a966c1ff9041167a1b59365e1454808603ea96b806f11100ab525729e6
+size 50503544

checkpoint-4000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fac9dfafdbe257c27d126ebc64c36f5002cbcc102e11f7c26dbbd8877543a18
+size 101184122

checkpoint-4000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c7a809b4cac00f4b5d15b016ea856622b58fbd0af257b00186cb58f545a8b32
+size 14244

checkpoint-4000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19341e45d577af9db7d2e9fc7df789efc6373c9623da8fc18c0bd22cbfa2470f
+size 1064

checkpoint-4000/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-4000/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

checkpoint-4000/tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 2048,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-4000/trainer_state.json

@@ -0,0 +1,2844 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.6842105263157894,
+  "eval_steps": 2000,
+  "global_step": 4000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0,
+      "grad_norm": 1.0191779136657715,
+      "learning_rate": 8.403361344537815e-07,
+      "loss": 1.4757,
+      "step": 1
+    },
+    {
+      "epoch": 0.0,
+      "grad_norm": 0.947059690952301,
+      "learning_rate": 8.403361344537817e-06,
+      "loss": 1.5145,
+      "step": 10
+    },
+    {
+      "epoch": 0.01,
+      "grad_norm": 0.9549775719642639,
+      "learning_rate": 1.6806722689075634e-05,
+      "loss": 1.5546,
+      "step": 20
+    },
+    {
+      "epoch": 0.01,
+      "grad_norm": 1.1984186172485352,
+      "learning_rate": 2.5210084033613446e-05,
+      "loss": 1.6809,
+      "step": 30
+    },
+    {
+      "epoch": 0.02,
+      "grad_norm": 0.9854737520217896,
+      "learning_rate": 3.361344537815127e-05,
+      "loss": 1.8074,
+      "step": 40
+    },
+    {
+      "epoch": 0.02,
+      "grad_norm": 1.3563014268875122,
+      "learning_rate": 4.201680672268908e-05,
+      "loss": 1.699,
+      "step": 50
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.6414602398872375,
+      "learning_rate": 5.042016806722689e-05,
+      "loss": 1.3419,
+      "step": 60
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.580333948135376,
+      "learning_rate": 5.882352941176471e-05,
+      "loss": 1.2524,
+      "step": 70
+    },
+    {
+      "epoch": 0.03,
+      "grad_norm": 0.6972444653511047,
+      "learning_rate": 6.722689075630254e-05,
+      "loss": 1.4261,
+      "step": 80
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 0.8142390847206116,
+      "learning_rate": 7.563025210084033e-05,
+      "loss": 1.5472,
+      "step": 90
+    },
+    {
+      "epoch": 0.04,
+      "grad_norm": 1.9213963747024536,
+      "learning_rate": 8.403361344537815e-05,
+      "loss": 1.5581,
+      "step": 100
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.7875838279724121,
+      "learning_rate": 9.243697478991598e-05,
+      "loss": 1.2596,
+      "step": 110
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.6198441982269287,
+      "learning_rate": 0.00010084033613445378,
+      "loss": 1.2901,
+      "step": 120
+    },
+    {
+      "epoch": 0.05,
+      "grad_norm": 0.6708433032035828,
+      "learning_rate": 0.00010924369747899159,
+      "loss": 1.3087,
+      "step": 130
+    },
+    {
+      "epoch": 0.06,
+      "grad_norm": 0.8415197730064392,
+      "learning_rate": 0.00011764705882352942,
+      "loss": 1.4806,
+      "step": 140
+    },
+    {
+      "epoch": 0.06,
+      "grad_norm": 1.7113947868347168,
+      "learning_rate": 0.00012605042016806722,
+      "loss": 1.5082,
+      "step": 150
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.7543832659721375,
+      "learning_rate": 0.00013445378151260507,
+      "loss": 1.1818,
+      "step": 160
+    },
+    {
+      "epoch": 0.07,
+      "grad_norm": 0.577403724193573,
+      "learning_rate": 0.00014285714285714287,
+      "loss": 1.2308,
+      "step": 170
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.6748159527778625,
+      "learning_rate": 0.00015126050420168066,
+      "loss": 1.3738,
+      "step": 180
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.8842242360115051,
+      "learning_rate": 0.0001596638655462185,
+      "loss": 1.4544,
+      "step": 190
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 1.7435425519943237,
+      "learning_rate": 0.0001680672268907563,
+      "loss": 1.4997,
+      "step": 200
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 0.5861243009567261,
+      "learning_rate": 0.00017647058823529413,
+      "loss": 1.1774,
+      "step": 210
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 0.5415711998939514,
+      "learning_rate": 0.00018487394957983195,
+      "loss": 1.0529,
+      "step": 220
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.6118600964546204,
+      "learning_rate": 0.00019327731092436975,
+      "loss": 1.3117,
+      "step": 230
+    },
+    {
+      "epoch": 0.1,
+      "grad_norm": 0.8368746042251587,
+      "learning_rate": 0.00019999990304024961,
+      "loss": 1.4745,
+      "step": 240
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 1.086878776550293,
+      "learning_rate": 0.00019999650946872738,
+      "loss": 1.3995,
+      "step": 250
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 0.6036991477012634,
+      "learning_rate": 0.00019998826809770645,
+      "loss": 1.2325,
+      "step": 260
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 0.5606744289398193,
+      "learning_rate": 0.0001999751793267259,
+      "loss": 1.2043,
+      "step": 270
+    },
+    {
+      "epoch": 0.12,
+      "grad_norm": 0.6112542748451233,
+      "learning_rate": 0.00019995724379032526,
+      "loss": 1.2512,
+      "step": 280
+    },
+    {
+      "epoch": 0.12,
+      "grad_norm": 0.7892420887947083,
+      "learning_rate": 0.00019993446235801367,
+      "loss": 1.3604,
+      "step": 290
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 1.2104417085647583,
+      "learning_rate": 0.00019990683613422787,
+      "loss": 1.4282,
+      "step": 300
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 0.5284091234207153,
+      "learning_rate": 0.0001998743664582786,
+      "loss": 1.1223,
+      "step": 310
+    },
+    {
+      "epoch": 0.13,
+      "grad_norm": 0.5215238928794861,
+      "learning_rate": 0.0001998370549042855,
+      "loss": 1.1399,
+      "step": 320
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 0.6061808466911316,
+      "learning_rate": 0.0001997949032811011,
+      "loss": 1.2432,
+      "step": 330
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 0.7220178246498108,
+      "learning_rate": 0.0001997479136322229,
+      "loss": 1.4737,
+      "step": 340
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 1.035914421081543,
+      "learning_rate": 0.00019969608823569433,
+      "loss": 1.5087,
+      "step": 350
+    },
+    {
+      "epoch": 0.15,
+      "grad_norm": 0.5120232701301575,
+      "learning_rate": 0.00019963942960399446,
+      "loss": 1.1583,
+      "step": 360
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.5209968686103821,
+      "learning_rate": 0.00019957794048391602,
+      "loss": 1.1494,
+      "step": 370
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.5688252449035645,
+      "learning_rate": 0.0001995116238564323,
+      "loss": 1.2648,
+      "step": 380
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.7350272536277771,
+      "learning_rate": 0.00019944048293655265,
+      "loss": 1.3836,
+      "step": 390
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 1.117912769317627,
+      "learning_rate": 0.00019936452117316663,
+      "loss": 1.3725,
+      "step": 400
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 0.5087815523147583,
+      "learning_rate": 0.00019928374224887678,
+      "loss": 1.0783,
+      "step": 410
+    },
+    {
+      "epoch": 0.18,
+      "grad_norm": 0.4944276213645935,
+      "learning_rate": 0.00019919815007982003,
+      "loss": 1.0936,
+      "step": 420
+    },
+    {
+      "epoch": 0.18,
+      "grad_norm": 0.5578716397285461,
+      "learning_rate": 0.000199107748815478,
+      "loss": 1.1705,
+      "step": 430
+    },
+    {
+      "epoch": 0.19,
+      "grad_norm": 0.670518696308136,
+      "learning_rate": 0.00019901254283847573,
+      "loss": 1.3099,
+      "step": 440
+    },
+    {
+      "epoch": 0.19,
+      "grad_norm": 0.9550499320030212,
+      "learning_rate": 0.00019891253676436915,
+      "loss": 1.3971,
+      "step": 450
+    },
+    {
+      "epoch": 0.19,
+      "grad_norm": 0.48883694410324097,
+      "learning_rate": 0.00019880773544142148,
+      "loss": 1.1091,
+      "step": 460
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.5331695675849915,
+      "learning_rate": 0.00019869814395036807,
+      "loss": 1.1169,
+      "step": 470
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 0.6103072166442871,
+      "learning_rate": 0.0001985837676041701,
+      "loss": 1.2462,
+      "step": 480
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 0.7295002937316895,
+      "learning_rate": 0.00019846461194775712,
+      "loss": 1.3711,
+      "step": 490
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 1.003199815750122,
+      "learning_rate": 0.000198340682757758,
+      "loss": 1.3987,
+      "step": 500
+    },
+    {
+      "epoch": 0.21,
+      "grad_norm": 0.49524402618408203,
+      "learning_rate": 0.00019821198604222113,
+      "loss": 1.1002,
+      "step": 510
+    },
+    {
+      "epoch": 0.22,
+      "grad_norm": 0.4883529543876648,
+      "learning_rate": 0.00019807852804032305,
+      "loss": 1.0895,
+      "step": 520
+    },
+    {
+      "epoch": 0.22,
+      "grad_norm": 0.5645517110824585,
+      "learning_rate": 0.00019794031522206594,
+      "loss": 1.1673,
+      "step": 530
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 0.7130540609359741,
+      "learning_rate": 0.00019779735428796395,
+      "loss": 1.2775,
+      "step": 540
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 1.035349726676941,
+      "learning_rate": 0.00019764965216871846,
+      "loss": 1.3272,
+      "step": 550
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.5434763431549072,
+      "learning_rate": 0.00019749721602488207,
+      "loss": 1.1483,
+      "step": 560
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.472807914018631,
+      "learning_rate": 0.0001973400532465113,
+      "loss": 1.0666,
+      "step": 570
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.575896143913269,
+      "learning_rate": 0.00019717817145280844,
+      "loss": 1.122,
+      "step": 580
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 0.6578025817871094,
+      "learning_rate": 0.00019701157849175228,
+      "loss": 1.2858,
+      "step": 590
+    },
+    {
+      "epoch": 0.25,
+      "grad_norm": 1.2908504009246826,
+      "learning_rate": 0.0001968402824397174,
+      "loss": 1.3983,
+      "step": 600
+    },
+    {
+      "epoch": 0.26,
+      "grad_norm": 0.5685085654258728,
+      "learning_rate": 0.0001966642916010829,
+      "loss": 1.0992,
+      "step": 610
+    },
+    {
+      "epoch": 0.26,
+      "grad_norm": 0.4999242126941681,
+      "learning_rate": 0.0001964836145078295,
+      "loss": 1.1434,
+      "step": 620
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.5502079129219055,
+      "learning_rate": 0.00019629825991912623,
+      "loss": 1.1573,
+      "step": 630
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.6397935748100281,
+      "learning_rate": 0.0001961082368209055,
+      "loss": 1.3418,
+      "step": 640
+    },
+    {
+      "epoch": 0.27,
+      "grad_norm": 0.9650578498840332,
+      "learning_rate": 0.00019591355442542775,
+      "loss": 1.3298,
+      "step": 650
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 0.49895137548446655,
+      "learning_rate": 0.00019571422217083454,
+      "loss": 1.0907,
+      "step": 660
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 0.49023300409317017,
+      "learning_rate": 0.00019551024972069126,
+      "loss": 1.1305,
+      "step": 670
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 0.5746453404426575,
+      "learning_rate": 0.00019530164696351848,
+      "loss": 1.1865,
+      "step": 680
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 0.6834538578987122,
+      "learning_rate": 0.00019508842401231257,
+      "loss": 1.3113,
+      "step": 690
+    },
+    {
+      "epoch": 0.29,
+      "grad_norm": 1.0113998651504517,
+      "learning_rate": 0.0001948705912040556,
+      "loss": 1.3686,
+      "step": 700
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.5399815440177917,
+      "learning_rate": 0.0001946481590992139,
+      "loss": 1.1602,
+      "step": 710
+    },
+    {
+      "epoch": 0.3,
+      "grad_norm": 0.5135788321495056,
+      "learning_rate": 0.00019442113848122636,
+      "loss": 1.0881,
+      "step": 720
+    },
+    {
+      "epoch": 0.31,
+      "grad_norm": 0.5519896149635315,
+      "learning_rate": 0.00019418954035598152,
+      "loss": 1.1885,
+      "step": 730
+    },
+    {
+      "epoch": 0.31,
+      "grad_norm": 0.7329438328742981,
+      "learning_rate": 0.000193953375951284,
+      "loss": 1.2668,
+      "step": 740
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 1.0599132776260376,
+      "learning_rate": 0.00019371265671631037,
+      "loss": 1.3524,
+      "step": 750
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.5613999962806702,
+      "learning_rate": 0.00019346739432105374,
+      "loss": 1.0983,
+      "step": 760
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.5169478058815002,
+      "learning_rate": 0.00019321760065575834,
+      "loss": 1.1699,
+      "step": 770
+    },
+    {
+      "epoch": 0.33,
+      "grad_norm": 0.5800842642784119,
+      "learning_rate": 0.00019296328783034294,
+      "loss": 1.2105,
+      "step": 780
+    },
+    {
+      "epoch": 0.33,
+      "grad_norm": 0.7089204788208008,
+      "learning_rate": 0.00019270446817381377,
+      "loss": 1.3458,
+      "step": 790
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 0.8912138342857361,
+      "learning_rate": 0.00019244115423366678,
+      "loss": 1.3386,
+      "step": 800
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 0.5033462047576904,
+      "learning_rate": 0.00019217335877527947,
+      "loss": 1.0824,
+      "step": 810
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.5139713287353516,
+      "learning_rate": 0.0001919010947812918,
+      "loss": 1.0122,
+      "step": 820
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.5905170440673828,
+      "learning_rate": 0.00019162437545097719,
+      "loss": 1.2924,
+      "step": 830
+    },
+    {
+      "epoch": 0.35,
+      "grad_norm": 0.715568482875824,
+      "learning_rate": 0.00019134321419960203,
+      "loss": 1.2982,
+      "step": 840
+    },
+    {
+      "epoch": 0.36,
+      "grad_norm": 1.046240210533142,
+      "learning_rate": 0.00019105762465777598,
+      "loss": 1.4013,
+      "step": 850
+    },
+    {
+      "epoch": 0.36,
+      "grad_norm": 0.49822917580604553,
+      "learning_rate": 0.00019076762067079059,
+      "loss": 1.0633,
+      "step": 860
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 0.47567543387413025,
+      "learning_rate": 0.0001904732162979485,
+      "loss": 1.0497,
+      "step": 870
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 0.5331007242202759,
+      "learning_rate": 0.0001901744258118816,
+      "loss": 1.166,
+      "step": 880
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 0.7400739789009094,
+      "learning_rate": 0.00018987126369785916,
+      "loss": 1.3527,
+      "step": 890
+    },
+    {
+      "epoch": 0.38,
+      "grad_norm": 1.1264495849609375,
+      "learning_rate": 0.00018956374465308563,
+      "loss": 1.3475,
+      "step": 900
+    },
+    {
+      "epoch": 0.38,
+      "grad_norm": 0.5124080777168274,
+      "learning_rate": 0.00018925188358598813,
+      "loss": 1.05,
+      "step": 910
+    },
+    {
+      "epoch": 0.39,
+      "grad_norm": 0.522303581237793,
+      "learning_rate": 0.00018893569561549364,
+      "loss": 1.156,
+      "step": 920
+    },
+    {
+      "epoch": 0.39,
+      "grad_norm": 0.5782234072685242,
+      "learning_rate": 0.00018861519607029606,
+      "loss": 1.158,
+      "step": 930
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.7259439826011658,
+      "learning_rate": 0.00018829040048811305,
+      "loss": 1.306,
+      "step": 940
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.9230518937110901,
+      "learning_rate": 0.0001879613246149329,
+      "loss": 1.4171,
+      "step": 950
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.5070952773094177,
+      "learning_rate": 0.00018762798440425096,
+      "loss": 1.0647,
+      "step": 960
+    },
+    {
+      "epoch": 0.41,
+      "grad_norm": 0.540289044380188,
+      "learning_rate": 0.00018729039601629634,
+      "loss": 1.0823,
+      "step": 970
+    },
+    {
+      "epoch": 0.41,
+      "grad_norm": 0.6044334173202515,
+      "learning_rate": 0.00018694857581724855,
+      "loss": 1.2258,
+      "step": 980
+    },
+    {
+      "epoch": 0.42,
+      "grad_norm": 0.6775342226028442,
+      "learning_rate": 0.00018660254037844388,
+      "loss": 1.3045,
+      "step": 990
+    },
+    {
+      "epoch": 0.42,
+      "grad_norm": 1.0051116943359375,
+      "learning_rate": 0.00018625230647557217,
+      "loss": 1.3275,
+      "step": 1000
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.522710382938385,
+      "learning_rate": 0.00018589789108786344,
+      "loss": 1.1284,
+      "step": 1010
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.527962327003479,
+      "learning_rate": 0.00018553931139726485,
+      "loss": 1.0688,
+      "step": 1020
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 0.5457212924957275,
+      "learning_rate": 0.0001851765847876076,
+      "loss": 1.1434,
+      "step": 1030
+    },
+    {
+      "epoch": 0.44,
+      "grad_norm": 0.6518785357475281,
+      "learning_rate": 0.00018480972884376434,
+      "loss": 1.2961,
+      "step": 1040
+    },
+    {
+      "epoch": 0.44,
+      "grad_norm": 1.0324338674545288,
+      "learning_rate": 0.00018443876135079638,
+      "loss": 1.407,
+      "step": 1050
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.6092559099197388,
+      "learning_rate": 0.0001840637002930918,
+      "loss": 1.0535,
+      "step": 1060
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.5149545073509216,
+      "learning_rate": 0.00018368456385349334,
+      "loss": 1.0546,
+      "step": 1070
+    },
+    {
+      "epoch": 0.45,
+      "grad_norm": 0.6056516766548157,
+      "learning_rate": 0.0001833013704124169,
+      "loss": 1.1735,
+      "step": 1080
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 0.7382489442825317,
+      "learning_rate": 0.00018291413854696077,
+      "loss": 1.3044,
+      "step": 1090
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 0.9548698663711548,
+      "learning_rate": 0.0001825228870300046,
+      "loss": 1.3571,
+      "step": 1100
+    },
+    {
+      "epoch": 0.47,
+      "grad_norm": 0.5410871505737305,
+      "learning_rate": 0.0001821276348292995,
+      "loss": 1.097,
+      "step": 1110
+    },
+    {
+      "epoch": 0.47,
+      "grad_norm": 0.5245351195335388,
+      "learning_rate": 0.00018172840110654862,
+      "loss": 1.0923,
+      "step": 1120
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.6045931577682495,
+      "learning_rate": 0.00018132520521647785,
+      "loss": 1.2115,
+      "step": 1130
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.6429124474525452,
+      "learning_rate": 0.00018091806670589788,
+      "loss": 1.2736,
+      "step": 1140
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.9760797619819641,
+      "learning_rate": 0.0001805070053127563,
+      "loss": 1.2964,
+      "step": 1150
+    },
+    {
+      "epoch": 0.49,
+      "grad_norm": 0.48860272765159607,
+      "learning_rate": 0.0001800920409651809,
+      "loss": 1.0942,
+      "step": 1160
+    },
+    {
+      "epoch": 0.49,
+      "grad_norm": 0.5090013146400452,
+      "learning_rate": 0.00017967319378051343,
+      "loss": 1.0925,
+      "step": 1170
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.5822442770004272,
+      "learning_rate": 0.00017925048406433437,
+      "loss": 1.1583,
+      "step": 1180
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.7363468408584595,
+      "learning_rate": 0.00017882393230947855,
+      "loss": 1.3283,
+      "step": 1190
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.840218186378479,
+      "learning_rate": 0.00017839355919504164,
+      "loss": 1.3098,
+      "step": 1200
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.5360991954803467,
+      "learning_rate": 0.00017795938558537753,
+      "loss": 1.1028,
+      "step": 1210
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 0.5553500056266785,
+      "learning_rate": 0.00017752143252908705,
+      "loss": 1.0737,
+      "step": 1220
+    },
+    {
+      "epoch": 0.52,
+      "grad_norm": 0.5366541743278503,
+      "learning_rate": 0.00017707972125799735,
+      "loss": 1.1918,
+      "step": 1230
+    },
+    {
+      "epoch": 0.52,
+      "grad_norm": 0.7474969029426575,
+      "learning_rate": 0.0001766342731861327,
+      "loss": 1.2756,
+      "step": 1240
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 1.0241633653640747,
+      "learning_rate": 0.00017618510990867624,
+      "loss": 1.298,
+      "step": 1250
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 0.5135468244552612,
+      "learning_rate": 0.00017573225320092323,
+      "loss": 0.9818,
+      "step": 1260
+    },
+    {
+      "epoch": 0.53,
+      "grad_norm": 0.5308533906936646,
+      "learning_rate": 0.00017527572501722512,
+      "loss": 1.0729,
+      "step": 1270
+    },
+    {
+      "epoch": 0.54,
+      "grad_norm": 0.6265894174575806,
+      "learning_rate": 0.00017481554748992553,
+      "loss": 1.1791,
+      "step": 1280
+    },
+    {
+      "epoch": 0.54,
+      "grad_norm": 0.7500415444374084,
+      "learning_rate": 0.00017435174292828695,
+      "loss": 1.2722,
+      "step": 1290
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 1.0609970092773438,
+      "learning_rate": 0.00017388433381740952,
+      "loss": 1.3034,
+      "step": 1300
+    },
+    {
+      "epoch": 0.55,
+      "grad_norm": 0.5286604166030884,
+      "learning_rate": 0.00017341334281714064,
+      "loss": 0.9932,
+      "step": 1310
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.5054761171340942,
+      "learning_rate": 0.00017293879276097677,
+      "loss": 1.0386,
+      "step": 1320
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.6337370872497559,
+      "learning_rate": 0.00017246070665495608,
+      "loss": 1.1341,
+      "step": 1330
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.7288526892662048,
+      "learning_rate": 0.00017197910767654352,
+      "loss": 1.2551,
+      "step": 1340
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 1.0159952640533447,
+      "learning_rate": 0.0001714940191735069,
+      "loss": 1.3048,
+      "step": 1350
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 0.5907754898071289,
+      "learning_rate": 0.00017100546466278504,
+      "loss": 1.0322,
+      "step": 1360
+    },
+    {
+      "epoch": 0.58,
+      "grad_norm": 0.5079711675643921,
+      "learning_rate": 0.00017051346782934786,
+      "loss": 1.0071,
+      "step": 1370
+    },
+    {
+      "epoch": 0.58,
+      "grad_norm": 0.6100701689720154,
+      "learning_rate": 0.00017001805252504795,
+      "loss": 1.1588,
+      "step": 1380
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 0.7802088260650635,
+      "learning_rate": 0.00016951924276746425,
+      "loss": 1.2911,
+      "step": 1390
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 1.0676313638687134,
+      "learning_rate": 0.00016901706273873777,
+      "loss": 1.2942,
+      "step": 1400
+    },
+    {
+      "epoch": 0.59,
+      "grad_norm": 0.527285099029541,
+      "learning_rate": 0.00016851153678439922,
+      "loss": 1.0107,
+      "step": 1410
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.5518642663955688,
+      "learning_rate": 0.00016800268941218876,
+      "loss": 1.0683,
+      "step": 1420
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 0.6224582195281982,
+      "learning_rate": 0.0001674905452908678,
+      "loss": 1.1685,
+      "step": 1430
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 0.6205934882164001,
+      "learning_rate": 0.0001669751292490231,
+      "loss": 1.302,
+      "step": 1440
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 1.0423789024353027,
+      "learning_rate": 0.0001664564662738632,
+      "loss": 1.2654,
+      "step": 1450
+    },
+    {
+      "epoch": 0.61,
+      "grad_norm": 0.5314890742301941,
+      "learning_rate": 0.00016593458151000688,
+      "loss": 1.083,
+      "step": 1460
+    },
+    {
+      "epoch": 0.62,
+      "grad_norm": 0.5361500978469849,
+      "learning_rate": 0.00016540950025826422,
+      "loss": 1.0952,
+      "step": 1470
+    },
+    {
+      "epoch": 0.62,
+      "grad_norm": 0.5990237593650818,
+      "learning_rate": 0.00016488124797441004,
+      "loss": 1.1923,
+      "step": 1480
+    },
+    {
+      "epoch": 0.63,
+      "grad_norm": 0.7383199334144592,
+      "learning_rate": 0.00016434985026794987,
+      "loss": 1.288,
+      "step": 1490
+    },
+    {
+      "epoch": 0.63,
+      "grad_norm": 1.0500961542129517,
+      "learning_rate": 0.0001638153329008782,
+      "loss": 1.318,
+      "step": 1500
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.5643898248672485,
+      "learning_rate": 0.00016327772178642986,
+      "loss": 1.0641,
+      "step": 1510
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.509543776512146,
+      "learning_rate": 0.0001627370429878235,
+      "loss": 1.0216,
+      "step": 1520
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.6048777103424072,
+      "learning_rate": 0.0001621933227169981,
+      "loss": 1.136,
+      "step": 1530
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 0.7480033040046692,
+      "learning_rate": 0.0001616465873333423,
+      "loss": 1.2798,
+      "step": 1540
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 1.0061513185501099,
+      "learning_rate": 0.00016109686334241655,
+      "loss": 1.2482,
+      "step": 1550
+    },
+    {
+      "epoch": 0.66,
+      "grad_norm": 0.5828456282615662,
+      "learning_rate": 0.0001605441773946679,
+      "loss": 1.1134,
+      "step": 1560
+    },
+    {
+      "epoch": 0.66,
+      "grad_norm": 0.5483405590057373,
+      "learning_rate": 0.00015998855628413823,
+      "loss": 1.0308,
+      "step": 1570
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.6139534711837769,
+      "learning_rate": 0.00015943002694716519,
+      "loss": 1.1093,
+      "step": 1580
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.6477651000022888,
+      "learning_rate": 0.0001588686164610763,
+      "loss": 1.2047,
+      "step": 1590
+    },
+    {
+      "epoch": 0.67,
+      "grad_norm": 0.9526233077049255,
+      "learning_rate": 0.00015830435204287628,
+      "loss": 1.3359,
+      "step": 1600
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 0.5442110896110535,
+      "learning_rate": 0.00015773726104792768,
+      "loss": 1.0236,
+      "step": 1610
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 0.5667725801467896,
+      "learning_rate": 0.0001571673709686245,
+      "loss": 0.9956,
+      "step": 1620
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.567449688911438,
+      "learning_rate": 0.00015659470943305955,
+      "loss": 1.1698,
+      "step": 1630
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.7154501080513,
+      "learning_rate": 0.00015601930420368486,
+      "loss": 1.2682,
+      "step": 1640
+    },
+    {
+      "epoch": 0.69,
+      "grad_norm": 0.9630643725395203,
+      "learning_rate": 0.00015544118317596605,
+      "loss": 1.3124,
+      "step": 1650
+    },
+    {
+      "epoch": 0.7,
+      "grad_norm": 0.5698636770248413,
+      "learning_rate": 0.0001548603743770296,
+      "loss": 1.0047,
+      "step": 1660
+    },
+    {
+      "epoch": 0.7,
+      "grad_norm": 0.538144588470459,
+      "learning_rate": 0.0001542769059643045,
+      "loss": 0.9859,
+      "step": 1670
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 0.6153010725975037,
+      "learning_rate": 0.00015369080622415683,
+      "loss": 1.1059,
+      "step": 1680
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 0.7482182383537292,
+      "learning_rate": 0.00015310210357051863,
+      "loss": 1.2589,
+      "step": 1690
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 1.014577031135559,
+      "learning_rate": 0.00015251082654351044,
+      "loss": 1.2322,
+      "step": 1700
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.5590934157371521,
+      "learning_rate": 0.00015191700380805752,
+      "loss": 1.0283,
+      "step": 1710
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.5307114124298096,
+      "learning_rate": 0.00015132066415250042,
+      "loss": 1.0196,
+      "step": 1720
+    },
+    {
+      "epoch": 0.73,
+      "grad_norm": 0.6487598419189453,
+      "learning_rate": 0.000150721836487199,
+      "loss": 1.0595,
+      "step": 1730
+    },
+    {
+      "epoch": 0.73,
+      "grad_norm": 0.8052285313606262,
+      "learning_rate": 0.00015012054984313116,
+      "loss": 1.2083,
+      "step": 1740
+    },
+    {
+      "epoch": 0.74,
+      "grad_norm": 1.1172990798950195,
+      "learning_rate": 0.00014951683337048537,
+      "loss": 1.2899,
+      "step": 1750
+    },
+    {
+      "epoch": 0.74,
+      "grad_norm": 0.5224259495735168,
+      "learning_rate": 0.00014891071633724737,
+      "loss": 1.0475,
+      "step": 1760
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.5798863768577576,
+      "learning_rate": 0.00014830222812778126,
+      "loss": 0.9953,
+      "step": 1770
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.6029157638549805,
+      "learning_rate": 0.00014769139824140516,
+      "loss": 1.1619,
+      "step": 1780
+    },
+    {
+      "epoch": 0.75,
+      "grad_norm": 0.7881869673728943,
+      "learning_rate": 0.00014707825629096084,
+      "loss": 1.2333,
+      "step": 1790
+    },
+    {
+      "epoch": 0.76,
+      "grad_norm": 0.9496088624000549,
+      "learning_rate": 0.00014646283200137828,
+      "loss": 1.2424,
+      "step": 1800
+    },
+    {
+      "epoch": 0.76,
+      "grad_norm": 0.5425153374671936,
+      "learning_rate": 0.00014584515520823453,
+      "loss": 0.9886,
+      "step": 1810
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.5590393543243408,
+      "learning_rate": 0.0001452252558563073,
+      "loss": 1.0041,
+      "step": 1820
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.5956026911735535,
+      "learning_rate": 0.00014460316399812325,
+      "loss": 1.1507,
+      "step": 1830
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 0.7304853796958923,
+      "learning_rate": 0.00014397890979250108,
+      "loss": 1.2183,
+      "step": 1840
+    },
+    {
+      "epoch": 0.78,
+      "grad_norm": 1.0130630731582642,
+      "learning_rate": 0.0001433525235030894,
+      "loss": 1.2944,
+      "step": 1850
+    },
+    {
+      "epoch": 0.78,
+      "grad_norm": 0.5265572667121887,
+      "learning_rate": 0.00014272403549689955,
+      "loss": 1.0166,
+      "step": 1860
+    },
+    {
+      "epoch": 0.79,
+      "grad_norm": 0.5423105955123901,
+      "learning_rate": 0.0001420934762428335,
+      "loss": 0.9559,
+      "step": 1870
+    },
+    {
+      "epoch": 0.79,
+      "grad_norm": 0.6281871199607849,
+      "learning_rate": 0.00014146087631020665,
+      "loss": 1.064,
+      "step": 1880
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.7130801677703857,
+      "learning_rate": 0.00014082626636726582,
+      "loss": 1.1955,
+      "step": 1890
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 1.038220763206482,
+      "learning_rate": 0.00014018967717970254,
+      "loss": 1.2621,
+      "step": 1900
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.5577895045280457,
+      "learning_rate": 0.00013955113960916146,
+      "loss": 0.9577,
+      "step": 1910
+    },
+    {
+      "epoch": 0.81,
+      "grad_norm": 0.526157796382904,
+      "learning_rate": 0.0001389106846117443,
+      "loss": 1.0109,
+      "step": 1920
+    },
+    {
+      "epoch": 0.81,
+      "grad_norm": 0.6196286678314209,
+      "learning_rate": 0.000138268343236509,
+      "loss": 1.1256,
+      "step": 1930
+    },
+    {
+      "epoch": 0.82,
+      "grad_norm": 0.7702537178993225,
+      "learning_rate": 0.00013762414662396444,
+      "loss": 1.2524,
+      "step": 1940
+    },
+    {
+      "epoch": 0.82,
+      "grad_norm": 0.8746676445007324,
+      "learning_rate": 0.00013697812600456093,
+      "loss": 1.288,
+      "step": 1950
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.5723572969436646,
+      "learning_rate": 0.000136330312697176,
+      "loss": 0.995,
+      "step": 1960
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.5687806606292725,
+      "learning_rate": 0.00013568073810759615,
+      "loss": 1.042,
+      "step": 1970
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 0.5694392323493958,
+      "learning_rate": 0.00013502943372699426,
+      "loss": 1.0513,
+      "step": 1980
+    },
+    {
+      "epoch": 0.84,
+      "grad_norm": 0.770982563495636,
+      "learning_rate": 0.00013437643113040301,
+      "loss": 1.2595,
+      "step": 1990
+    },
+    {
+      "epoch": 0.84,
+      "grad_norm": 1.0096752643585205,
+      "learning_rate": 0.000133721761975184,
+      "loss": 1.2178,
+      "step": 2000
+    },
+    {
+      "epoch": 0.84,
+      "eval_loss": 1.0813658237457275,
+      "eval_runtime": 229.3235,
+      "eval_samples_per_second": 4.361,
+      "eval_steps_per_second": 0.545,
+      "step": 2000
+    },
+    {
+      "epoch": 0.85,
+      "grad_norm": 0.5128979086875916,
+      "learning_rate": 0.00013306545799949304,
+      "loss": 0.9791,
+      "step": 2010
+    },
+    {
+      "epoch": 0.85,
+      "grad_norm": 0.6104242205619812,
+      "learning_rate": 0.00013240755102074162,
+      "loss": 0.9958,
+      "step": 2020
+    },
+    {
+      "epoch": 0.85,
+      "grad_norm": 0.634865403175354,
+      "learning_rate": 0.00013174807293405428,
+      "loss": 1.1336,
+      "step": 2030
+    },
+    {
+      "epoch": 0.86,
+      "grad_norm": 0.7030709385871887,
+      "learning_rate": 0.00013108705571072233,
+      "loss": 1.2373,
+      "step": 2040
+    },
+    {
+      "epoch": 0.86,
+      "grad_norm": 0.9466651082038879,
+      "learning_rate": 0.00013042453139665397,
+      "loss": 1.2789,
+      "step": 2050
+    },
+    {
+      "epoch": 0.87,
+      "grad_norm": 0.5502979159355164,
+      "learning_rate": 0.00012976053211082074,
+      "loss": 1.0581,
+      "step": 2060
+    },
+    {
+      "epoch": 0.87,
+      "grad_norm": 0.5557164549827576,
+      "learning_rate": 0.00012909509004370028,
+      "loss": 0.9754,
+      "step": 2070
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.6184536218643188,
+      "learning_rate": 0.00012842823745571588,
+      "loss": 1.0759,
+      "step": 2080
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.7712015509605408,
+      "learning_rate": 0.0001277600066756724,
+      "loss": 1.199,
+      "step": 2090
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 1.0351948738098145,
+      "learning_rate": 0.00012709043009918896,
+      "loss": 1.3047,
+      "step": 2100
+    },
+    {
+      "epoch": 0.89,
+      "grad_norm": 0.5212281346321106,
+      "learning_rate": 0.00012641954018712863,
+      "loss": 0.9635,
+      "step": 2110
+    },
+    {
+      "epoch": 0.89,
+      "grad_norm": 0.5956359505653381,
+      "learning_rate": 0.0001257473694640244,
+      "loss": 0.9984,
+      "step": 2120
+    },
+    {
+      "epoch": 0.9,
+      "grad_norm": 0.649682343006134,
+      "learning_rate": 0.00012507395051650268,
+      "loss": 1.1465,
+      "step": 2130
+    },
+    {
+      "epoch": 0.9,
+      "grad_norm": 0.7536346912384033,
+      "learning_rate": 0.00012439931599170342,
+      "loss": 1.2214,
+      "step": 2140
+    },
+    {
+      "epoch": 0.91,
+      "grad_norm": 0.926645040512085,
+      "learning_rate": 0.00012372349859569737,
+      "loss": 1.197,
+      "step": 2150
+    },
+    {
+      "epoch": 0.91,
+      "grad_norm": 0.5298001766204834,
+      "learning_rate": 0.0001230465310919004,
+      "loss": 1.0572,
+      "step": 2160
+    },
+    {
+      "epoch": 0.91,
+      "grad_norm": 0.5461262464523315,
+      "learning_rate": 0.00012236844629948538,
+      "loss": 1.0131,
+      "step": 2170
+    },
+    {
+      "epoch": 0.92,
+      "grad_norm": 0.6402174234390259,
+      "learning_rate": 0.00012168927709179094,
+      "loss": 1.1128,
+      "step": 2180
+    },
+    {
+      "epoch": 0.92,
+      "grad_norm": 0.8170130252838135,
+      "learning_rate": 0.00012100905639472779,
+      "loss": 1.1808,
+      "step": 2190
+    },
+    {
+      "epoch": 0.93,
+      "grad_norm": 1.1390860080718994,
+      "learning_rate": 0.00012032781718518257,
+      "loss": 1.2739,
+      "step": 2200
+    },
+    {
+      "epoch": 0.93,
+      "grad_norm": 0.5590136647224426,
+      "learning_rate": 0.00011964559248941903,
+      "loss": 0.9234,
+      "step": 2210
+    },
+    {
+      "epoch": 0.93,
+      "grad_norm": 0.5282031297683716,
+      "learning_rate": 0.00011896241538147703,
+      "loss": 0.9832,
+      "step": 2220
+    },
+    {
+      "epoch": 0.94,
+      "grad_norm": 0.6679378151893616,
+      "learning_rate": 0.00011827831898156905,
+      "loss": 1.1028,
+      "step": 2230
+    },
+    {
+      "epoch": 0.94,
+      "grad_norm": 0.7357510328292847,
+      "learning_rate": 0.00011759333645447455,
+      "loss": 1.1958,
+      "step": 2240
+    },
+    {
+      "epoch": 0.95,
+      "grad_norm": 0.9115193486213684,
+      "learning_rate": 0.0001169075010079322,
+      "loss": 1.2317,
+      "step": 2250
+    },
+    {
+      "epoch": 0.95,
+      "grad_norm": 0.5533779263496399,
+      "learning_rate": 0.00011622084589102986,
+      "loss": 1.0045,
+      "step": 2260
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.6029343008995056,
+      "learning_rate": 0.00011553340439259286,
+      "loss": 0.9566,
+      "step": 2270
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.5945916771888733,
+      "learning_rate": 0.00011484520983956997,
+      "loss": 1.0913,
+      "step": 2280
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.8109087944030762,
+      "learning_rate": 0.00011415629559541782,
+      "loss": 1.2433,
+      "step": 2290
+    },
+    {
+      "epoch": 0.97,
+      "grad_norm": 0.9386932849884033,
+      "learning_rate": 0.00011346669505848353,
+      "loss": 1.2307,
+      "step": 2300
+    },
+    {
+      "epoch": 0.97,
+      "grad_norm": 0.6317028403282166,
+      "learning_rate": 0.00011277644166038534,
+      "loss": 1.0105,
+      "step": 2310
+    },
+    {
+      "epoch": 0.98,
+      "grad_norm": 0.5305240750312805,
+      "learning_rate": 0.00011208556886439216,
+      "loss": 0.9351,
+      "step": 2320
+    },
+    {
+      "epoch": 0.98,
+      "grad_norm": 0.621440589427948,
+      "learning_rate": 0.00011139411016380093,
+      "loss": 1.0558,
+      "step": 2330
+    },
+    {
+      "epoch": 0.99,
+      "grad_norm": 0.7868437170982361,
+      "learning_rate": 0.00011070209908031322,
+      "loss": 1.1889,
+      "step": 2340
+    },
+    {
+      "epoch": 0.99,
+      "grad_norm": 0.9640358686447144,
+      "learning_rate": 0.00011000956916240985,
+      "loss": 1.2697,
+      "step": 2350
+    },
+    {
+      "epoch": 0.99,
+      "grad_norm": 0.6120818853378296,
+      "learning_rate": 0.00010931655398372462,
+      "loss": 0.9262,
+      "step": 2360
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.7230446934700012,
+      "learning_rate": 0.00010862308714141661,
+      "loss": 1.1753,
+      "step": 2370
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.5455570816993713,
+      "learning_rate": 0.00010792920225454139,
+      "loss": 1.0577,
+      "step": 2380
+    },
+    {
+      "epoch": 1.01,
+      "grad_norm": 0.5207303762435913,
+      "learning_rate": 0.00010723493296242129,
+      "loss": 0.8919,
+      "step": 2390
+    },
+    {
+      "epoch": 1.01,
+      "grad_norm": 0.6713350415229797,
+      "learning_rate": 0.00010654031292301432,
+      "loss": 0.9547,
+      "step": 2400
+    },
+    {
+      "epoch": 1.01,
+      "grad_norm": 0.7119928598403931,
+      "learning_rate": 0.00010584537581128276,
+      "loss": 1.1245,
+      "step": 2410
+    },
+    {
+      "epoch": 1.02,
+      "grad_norm": 0.9546805024147034,
+      "learning_rate": 0.00010515015531756036,
+      "loss": 1.1268,
+      "step": 2420
+    },
+    {
+      "epoch": 1.02,
+      "grad_norm": 0.7098658084869385,
+      "learning_rate": 0.00010445468514591925,
+      "loss": 1.0389,
+      "step": 2430
+    },
+    {
+      "epoch": 1.03,
+      "grad_norm": 0.5891678333282471,
+      "learning_rate": 0.00010375899901253572,
+      "loss": 0.9701,
+      "step": 2440
+    },
+    {
+      "epoch": 1.03,
+      "grad_norm": 0.646004855632782,
+      "learning_rate": 0.0001030631306440559,
+      "loss": 0.9574,
+      "step": 2450
+    },
+    {
+      "epoch": 1.04,
+      "grad_norm": 0.7500150203704834,
+      "learning_rate": 0.00010236711377596072,
+      "loss": 1.0919,
+      "step": 2460
+    },
+    {
+      "epoch": 1.04,
+      "grad_norm": 0.9092882871627808,
+      "learning_rate": 0.00010167098215093009,
+      "loss": 1.0797,
+      "step": 2470
+    },
+    {
+      "epoch": 1.04,
+      "grad_norm": 0.6681928038597107,
+      "learning_rate": 0.00010097476951720752,
+      "loss": 0.9011,
+      "step": 2480
+    },
+    {
+      "epoch": 1.05,
+      "grad_norm": 0.6693964004516602,
+      "learning_rate": 0.00010027850962696368,
+      "loss": 0.911,
+      "step": 2490
+    },
+    {
+      "epoch": 1.05,
+      "grad_norm": 0.6667538285255432,
+      "learning_rate": 9.958223623466025e-05,
+      "loss": 0.9444,
+      "step": 2500
+    },
+    {
+      "epoch": 1.06,
+      "grad_norm": 0.7424312829971313,
+      "learning_rate": 9.888598309541347e-05,
+      "loss": 1.0328,
+      "step": 2510
+    },
+    {
+      "epoch": 1.06,
+      "grad_norm": 1.0285619497299194,
+      "learning_rate": 9.818978396335772e-05,
+      "loss": 1.1074,
+      "step": 2520
+    },
+    {
+      "epoch": 1.07,
+      "grad_norm": 0.7485564351081848,
+      "learning_rate": 9.749367259000918e-05,
+      "loss": 0.9986,
+      "step": 2530
+    },
+    {
+      "epoch": 1.07,
+      "grad_norm": 0.6146641969680786,
+      "learning_rate": 9.679768272262949e-05,
+      "loss": 0.9217,
+      "step": 2540
+    },
+    {
+      "epoch": 1.07,
+      "grad_norm": 0.714490532875061,
+      "learning_rate": 9.610184810258963e-05,
+      "loss": 0.9297,
+      "step": 2550
+    },
+    {
+      "epoch": 1.08,
+      "grad_norm": 0.7230827212333679,
+      "learning_rate": 9.540620246373439e-05,
+      "loss": 1.0436,
+      "step": 2560
+    },
+    {
+      "epoch": 1.08,
+      "grad_norm": 0.9164597392082214,
+      "learning_rate": 9.471077953074674e-05,
+      "loss": 1.1374,
+      "step": 2570
+    },
+    {
+      "epoch": 1.09,
+      "grad_norm": 0.6540518999099731,
+      "learning_rate": 9.401561301751286e-05,
+      "loss": 1.011,
+      "step": 2580
+    },
+    {
+      "epoch": 1.09,
+      "grad_norm": 0.6425168514251709,
+      "learning_rate": 9.332073662548784e-05,
+      "loss": 0.899,
+      "step": 2590
+    },
+    {
+      "epoch": 1.09,
+      "grad_norm": 0.6701865196228027,
+      "learning_rate": 9.262618404206191e-05,
+      "loss": 0.9899,
+      "step": 2600
+    },
+    {
+      "epoch": 1.1,
+      "grad_norm": 0.7741339206695557,
+      "learning_rate": 9.193198893892698e-05,
+      "loss": 1.0721,
+      "step": 2610
+    },
+    {
+      "epoch": 1.1,
+      "grad_norm": 0.9790834188461304,
+      "learning_rate": 9.123818497044446e-05,
+      "loss": 1.1014,
+      "step": 2620
+    },
+    {
+      "epoch": 1.11,
+      "grad_norm": 0.672179639339447,
+      "learning_rate": 9.05448057720138e-05,
+      "loss": 0.934,
+      "step": 2630
+    },
+    {
+      "epoch": 1.11,
+      "grad_norm": 0.6310063004493713,
+      "learning_rate": 8.98518849584416e-05,
+      "loss": 0.8713,
+      "step": 2640
+    },
+    {
+      "epoch": 1.12,
+      "grad_norm": 0.6647841334342957,
+      "learning_rate": 8.915945612231214e-05,
+      "loss": 0.9242,
+      "step": 2650
+    },
+    {
+      "epoch": 1.12,
+      "grad_norm": 0.7702280879020691,
+      "learning_rate": 8.846755283235874e-05,
+      "loss": 1.0557,
+      "step": 2660
+    },
+    {
+      "epoch": 1.12,
+      "grad_norm": 1.0556683540344238,
+      "learning_rate": 8.777620863183657e-05,
+      "loss": 1.1401,
+      "step": 2670
+    },
+    {
+      "epoch": 1.13,
+      "grad_norm": 0.651453971862793,
+      "learning_rate": 8.708545703689613e-05,
+      "loss": 1.0147,
+      "step": 2680
+    },
+    {
+      "epoch": 1.13,
+      "grad_norm": 0.6781310439109802,
+      "learning_rate": 8.639533153495867e-05,
+      "loss": 0.9664,
+      "step": 2690
+    },
+    {
+      "epoch": 1.14,
+      "grad_norm": 0.6900728344917297,
+      "learning_rate": 8.57058655830927e-05,
+      "loss": 0.9194,
+      "step": 2700
+    },
+    {
+      "epoch": 1.14,
+      "grad_norm": 0.7449643015861511,
+      "learning_rate": 8.501709260639186e-05,
+      "loss": 1.0693,
+      "step": 2710
+    },
+    {
+      "epoch": 1.15,
+      "grad_norm": 0.9307679533958435,
+      "learning_rate": 8.432904599635459e-05,
+      "loss": 1.0927,
+      "step": 2720
+    },
+    {
+      "epoch": 1.15,
+      "grad_norm": 0.6419615745544434,
+      "learning_rate": 8.364175910926528e-05,
+      "loss": 0.9482,
+      "step": 2730
+    },
+    {
+      "epoch": 1.15,
+      "grad_norm": 0.6502748727798462,
+      "learning_rate": 8.295526526457732e-05,
+      "loss": 0.9289,
+      "step": 2740
+    },
+    {
+      "epoch": 1.16,
+      "grad_norm": 0.683403491973877,
+      "learning_rate": 8.226959774329747e-05,
+      "loss": 0.9169,
+      "step": 2750
+    },
+    {
+      "epoch": 1.16,
+      "grad_norm": 0.7855057716369629,
+      "learning_rate": 8.158478978637269e-05,
+      "loss": 1.0693,
+      "step": 2760
+    },
+    {
+      "epoch": 1.17,
+      "grad_norm": 1.0051300525665283,
+      "learning_rate": 8.090087459307854e-05,
+      "loss": 1.17,
+      "step": 2770
+    },
+    {
+      "epoch": 1.17,
+      "grad_norm": 0.704396665096283,
+      "learning_rate": 8.021788531940967e-05,
+      "loss": 1.0218,
+      "step": 2780
+    },
+    {
+      "epoch": 1.17,
+      "grad_norm": 0.6089404821395874,
+      "learning_rate": 7.953585507647239e-05,
+      "loss": 1.0019,
+      "step": 2790
+    },
+    {
+      "epoch": 1.18,
+      "grad_norm": 0.7013250589370728,
+      "learning_rate": 7.885481692887947e-05,
+      "loss": 0.8923,
+      "step": 2800
+    },
+    {
+      "epoch": 1.18,
+      "grad_norm": 0.7909386157989502,
+      "learning_rate": 7.817480389314738e-05,
+      "loss": 0.9581,
+      "step": 2810
+    },
+    {
+      "epoch": 1.19,
+      "grad_norm": 0.9857104420661926,
+      "learning_rate": 7.749584893609532e-05,
+      "loss": 1.0834,
+      "step": 2820
+    },
+    {
+      "epoch": 1.19,
+      "grad_norm": 0.6876636147499084,
+      "learning_rate": 7.681798497324716e-05,
+      "loss": 1.033,
+      "step": 2830
+    },
+    {
+      "epoch": 1.2,
+      "grad_norm": 0.6592469811439514,
+      "learning_rate": 7.614124486723588e-05,
+      "loss": 0.9588,
+      "step": 2840
+    },
+    {
+      "epoch": 1.2,
+      "grad_norm": 0.725791871547699,
+      "learning_rate": 7.54656614262101e-05,
+      "loss": 0.9681,
+      "step": 2850
+    },
+    {
+      "epoch": 1.2,
+      "grad_norm": 0.7910930514335632,
+      "learning_rate": 7.47912674022437e-05,
+      "loss": 1.073,
+      "step": 2860
+    },
+    {
+      "epoch": 1.21,
+      "grad_norm": 1.0530593395233154,
+      "learning_rate": 7.411809548974792e-05,
+      "loss": 1.127,
+      "step": 2870
+    },
+    {
+      "epoch": 1.21,
+      "grad_norm": 0.7734891176223755,
+      "learning_rate": 7.344617832388662e-05,
+      "loss": 0.948,
+      "step": 2880
+    },
+    {
+      "epoch": 1.22,
+      "grad_norm": 0.6848577260971069,
+      "learning_rate": 7.277554847899378e-05,
+      "loss": 0.9217,
+      "step": 2890
+    },
+    {
+      "epoch": 1.22,
+      "grad_norm": 0.7144964933395386,
+      "learning_rate": 7.210623846699445e-05,
+      "loss": 0.9335,
+      "step": 2900
+    },
+    {
+      "epoch": 1.23,
+      "grad_norm": 0.7606580257415771,
+      "learning_rate": 7.14382807358287e-05,
+      "loss": 1.0414,
+      "step": 2910
+    },
+    {
+      "epoch": 1.23,
+      "grad_norm": 0.9773916602134705,
+      "learning_rate": 7.077170766787835e-05,
+      "loss": 1.1292,
+      "step": 2920
+    },
+    {
+      "epoch": 1.23,
+      "grad_norm": 0.6572744846343994,
+      "learning_rate": 7.010655157839726e-05,
+      "loss": 0.9491,
+      "step": 2930
+    },
+    {
+      "epoch": 1.24,
+      "grad_norm": 0.705321192741394,
+      "learning_rate": 6.944284471394452e-05,
+      "loss": 0.9468,
+      "step": 2940
+    },
+    {
+      "epoch": 1.24,
+      "grad_norm": 0.7025087475776672,
+      "learning_rate": 6.878061925082137e-05,
+      "loss": 0.9031,
+      "step": 2950
+    },
+    {
+      "epoch": 1.25,
+      "grad_norm": 0.7313035726547241,
+      "learning_rate": 6.811990729351109e-05,
+      "loss": 1.004,
+      "step": 2960
+    },
+    {
+      "epoch": 1.25,
+      "grad_norm": 1.1327190399169922,
+      "learning_rate": 6.746074087312262e-05,
+      "loss": 1.0948,
+      "step": 2970
+    },
+    {
+      "epoch": 1.25,
+      "grad_norm": 0.6862003207206726,
+      "learning_rate": 6.680315194583794e-05,
+      "loss": 0.9841,
+      "step": 2980
+    },
+    {
+      "epoch": 1.26,
+      "grad_norm": 0.65215003490448,
+      "learning_rate": 6.614717239136246e-05,
+      "loss": 0.8705,
+      "step": 2990
+    },
+    {
+      "epoch": 1.26,
+      "grad_norm": 0.7134726047515869,
+      "learning_rate": 6.549283401137982e-05,
+      "loss": 0.9343,
+      "step": 3000
+    },
+    {
+      "epoch": 1.27,
+      "grad_norm": 0.8117490410804749,
+      "learning_rate": 6.484016852800993e-05,
+      "loss": 1.017,
+      "step": 3010
+    },
+    {
+      "epoch": 1.27,
+      "grad_norm": 1.1070729494094849,
+      "learning_rate": 6.418920758227135e-05,
+      "loss": 1.1044,
+      "step": 3020
+    },
+    {
+      "epoch": 1.28,
+      "grad_norm": 0.6767369508743286,
+      "learning_rate": 6.353998273254706e-05,
+      "loss": 0.9423,
+      "step": 3030
+    },
+    {
+      "epoch": 1.28,
+      "grad_norm": 0.6971580386161804,
+      "learning_rate": 6.28925254530547e-05,
+      "loss": 0.9172,
+      "step": 3040
+    },
+    {
+      "epoch": 1.28,
+      "grad_norm": 0.7322612404823303,
+      "learning_rate": 6.224686713232059e-05,
+      "loss": 0.9127,
+      "step": 3050
+    },
+    {
+      "epoch": 1.29,
+      "grad_norm": 0.7999578714370728,
+      "learning_rate": 6.160303907165822e-05,
+      "loss": 1.0778,
+      "step": 3060
+    },
+    {
+      "epoch": 1.29,
+      "grad_norm": 0.9508334398269653,
+      "learning_rate": 6.0961072483650526e-05,
+      "loss": 1.028,
+      "step": 3070
+    },
+    {
+      "epoch": 1.3,
+      "grad_norm": 0.6923615336418152,
+      "learning_rate": 6.0320998490636884e-05,
+      "loss": 0.9482,
+      "step": 3080
+    },
+    {
+      "epoch": 1.3,
+      "grad_norm": 0.6519889831542969,
+      "learning_rate": 5.9682848123204305e-05,
+      "loss": 0.8825,
+      "step": 3090
+    },
+    {
+      "epoch": 1.31,
+      "grad_norm": 0.6621166467666626,
+      "learning_rate": 5.904665231868294e-05,
+      "loss": 0.9616,
+      "step": 3100
+    },
+    {
+      "epoch": 1.31,
+      "grad_norm": 0.8509411215782166,
+      "learning_rate": 5.841244191964637e-05,
+      "loss": 1.0519,
+      "step": 3110
+    },
+    {
+      "epoch": 1.31,
+      "grad_norm": 1.0802364349365234,
+      "learning_rate": 5.778024767241632e-05,
+      "loss": 1.0827,
+      "step": 3120
+    },
+    {
+      "epoch": 1.32,
+      "grad_norm": 0.679560661315918,
+      "learning_rate": 5.71501002255722e-05,
+      "loss": 0.9085,
+      "step": 3130
+    },
+    {
+      "epoch": 1.32,
+      "grad_norm": 0.6764241456985474,
+      "learning_rate": 5.652203012846513e-05,
+      "loss": 0.8935,
+      "step": 3140
+    },
+    {
+      "epoch": 1.33,
+      "grad_norm": 0.7432907223701477,
+      "learning_rate": 5.589606782973683e-05,
+      "loss": 0.948,
+      "step": 3150
+    },
+    {
+      "epoch": 1.33,
+      "grad_norm": 0.8105322122573853,
+      "learning_rate": 5.5272243675843896e-05,
+      "loss": 1.0448,
+      "step": 3160
+    },
+    {
+      "epoch": 1.33,
+      "grad_norm": 1.0853699445724487,
+      "learning_rate": 5.4650587909586025e-05,
+      "loss": 1.1097,
+      "step": 3170
+    },
+    {
+      "epoch": 1.34,
+      "grad_norm": 0.6888929009437561,
+      "learning_rate": 5.4031130668640376e-05,
+      "loss": 0.9453,
+      "step": 3180
+    },
+    {
+      "epoch": 1.34,
+      "grad_norm": 0.6341966986656189,
+      "learning_rate": 5.341390198410019e-05,
+      "loss": 0.8856,
+      "step": 3190
+    },
+    {
+      "epoch": 1.35,
+      "grad_norm": 0.6854045391082764,
+      "learning_rate": 5.2798931779019026e-05,
+      "loss": 0.8839,
+      "step": 3200
+    },
+    {
+      "epoch": 1.35,
+      "grad_norm": 0.8016719222068787,
+      "learning_rate": 5.21862498669601e-05,
+      "loss": 1.0946,
+      "step": 3210
+    },
+    {
+      "epoch": 1.36,
+      "grad_norm": 1.1072624921798706,
+      "learning_rate": 5.157588595055073e-05,
+      "loss": 1.0541,
+      "step": 3220
+    },
+    {
+      "epoch": 1.36,
+      "grad_norm": 0.6596867442131042,
+      "learning_rate": 5.0967869620042794e-05,
+      "loss": 0.9648,
+      "step": 3230
+    },
+    {
+      "epoch": 1.36,
+      "grad_norm": 0.6851956844329834,
+      "learning_rate": 5.0362230351877706e-05,
+      "loss": 0.8897,
+      "step": 3240
+    },
+    {
+      "epoch": 1.37,
+      "grad_norm": 0.7606251835823059,
+      "learning_rate": 4.9758997507257763e-05,
+      "loss": 0.8958,
+      "step": 3250
+    },
+    {
+      "epoch": 1.37,
+      "grad_norm": 0.7487627267837524,
+      "learning_rate": 4.915820033072257e-05,
+      "loss": 0.9596,
+      "step": 3260
+    },
+    {
+      "epoch": 1.38,
+      "grad_norm": 1.0656851530075073,
+      "learning_rate": 4.855986794873132e-05,
+      "loss": 1.1342,
+      "step": 3270
+    },
+    {
+      "epoch": 1.38,
+      "grad_norm": 0.6128414273262024,
+      "learning_rate": 4.796402936825074e-05,
+      "loss": 0.9758,
+      "step": 3280
+    },
+    {
+      "epoch": 1.39,
+      "grad_norm": 0.6946419477462769,
+      "learning_rate": 4.737071347534873e-05,
+      "loss": 0.8761,
+      "step": 3290
+    },
+    {
+      "epoch": 1.39,
+      "grad_norm": 0.6466770172119141,
+      "learning_rate": 4.677994903379432e-05,
+      "loss": 0.9176,
+      "step": 3300
+    },
+    {
+      "epoch": 1.39,
+      "grad_norm": 0.8571310043334961,
+      "learning_rate": 4.6191764683662744e-05,
+      "loss": 1.0479,
+      "step": 3310
+    },
+    {
+      "epoch": 1.4,
+      "grad_norm": 1.091630220413208,
+      "learning_rate": 4.560618893994735e-05,
+      "loss": 1.1288,
+      "step": 3320
+    },
+    {
+      "epoch": 1.4,
+      "grad_norm": 0.599025309085846,
+      "learning_rate": 4.5023250191176925e-05,
+      "loss": 0.9917,
+      "step": 3330
+    },
+    {
+      "epoch": 1.41,
+      "grad_norm": 0.6568591594696045,
+      "learning_rate": 4.444297669803981e-05,
+      "loss": 0.8653,
+      "step": 3340
+    },
+    {
+      "epoch": 1.41,
+      "grad_norm": 0.7494246959686279,
+      "learning_rate": 4.38653965920133e-05,
+      "loss": 1.0219,
+      "step": 3350
+    },
+    {
+      "epoch": 1.41,
+      "grad_norm": 0.8404898643493652,
+      "learning_rate": 4.3290537874000284e-05,
+      "loss": 1.0938,
+      "step": 3360
+    },
+    {
+      "epoch": 1.42,
+      "grad_norm": 1.154548168182373,
+      "learning_rate": 4.271842841297169e-05,
+      "loss": 1.0831,
+      "step": 3370
+    },
+    {
+      "epoch": 1.42,
+      "grad_norm": 0.7497602701187134,
+      "learning_rate": 4.214909594461511e-05,
+      "loss": 0.9233,
+      "step": 3380
+    },
+    {
+      "epoch": 1.43,
+      "grad_norm": 0.616992712020874,
+      "learning_rate": 4.158256806999059e-05,
+      "loss": 0.8713,
+      "step": 3390
+    },
+    {
+      "epoch": 1.43,
+      "grad_norm": 0.7367027401924133,
+      "learning_rate": 4.1018872254192166e-05,
+      "loss": 0.9085,
+      "step": 3400
+    },
+    {
+      "epoch": 1.44,
+      "grad_norm": 0.9179362654685974,
+      "learning_rate": 4.0458035825016806e-05,
+      "loss": 1.0192,
+      "step": 3410
+    },
+    {
+      "epoch": 1.44,
+      "grad_norm": 1.1351600885391235,
+      "learning_rate": 3.9900085971639036e-05,
+      "loss": 1.025,
+      "step": 3420
+    },
+    {
+      "epoch": 1.44,
+      "grad_norm": 0.7443174719810486,
+      "learning_rate": 3.934504974329326e-05,
+      "loss": 0.9504,
+      "step": 3430
+    },
+    {
+      "epoch": 1.45,
+      "grad_norm": 0.6519255638122559,
+      "learning_rate": 3.879295404796222e-05,
+      "loss": 0.821,
+      "step": 3440
+    },
+    {
+      "epoch": 1.45,
+      "grad_norm": 0.7432305216789246,
+      "learning_rate": 3.824382565107249e-05,
+      "loss": 0.9678,
+      "step": 3450
+    },
+    {
+      "epoch": 1.46,
+      "grad_norm": 0.8053915500640869,
+      "learning_rate": 3.769769117419703e-05,
+      "loss": 0.9972,
+      "step": 3460
+    },
+    {
+      "epoch": 1.46,
+      "grad_norm": 1.1109815835952759,
+      "learning_rate": 3.7154577093764334e-05,
+      "loss": 1.1064,
+      "step": 3470
+    },
+    {
+      "epoch": 1.47,
+      "grad_norm": 0.7476344108581543,
+      "learning_rate": 3.661450973977526e-05,
+      "loss": 0.9627,
+      "step": 3480
+    },
+    {
+      "epoch": 1.47,
+      "grad_norm": 0.7136977314949036,
+      "learning_rate": 3.607751529452609e-05,
+      "loss": 0.9483,
+      "step": 3490
+    },
+    {
+      "epoch": 1.47,
+      "grad_norm": 0.724554181098938,
+      "learning_rate": 3.554361979133959e-05,
+      "loss": 0.9112,
+      "step": 3500
+    },
+    {
+      "epoch": 1.48,
+      "grad_norm": 0.7963750958442688,
+      "learning_rate": 3.501284911330274e-05,
+      "loss": 1.0281,
+      "step": 3510
+    },
+    {
+      "epoch": 1.48,
+      "grad_norm": 1.048077940940857,
+      "learning_rate": 3.448522899201198e-05,
+      "loss": 1.0543,
+      "step": 3520
+    },
+    {
+      "epoch": 1.49,
+      "grad_norm": 0.6887779235839844,
+      "learning_rate": 3.396078500632576e-05,
+      "loss": 0.9685,
+      "step": 3530
+    },
+    {
+      "epoch": 1.49,
+      "grad_norm": 0.6389910578727722,
+      "learning_rate": 3.3439542581124374e-05,
+      "loss": 0.9018,
+      "step": 3540
+    },
+    {
+      "epoch": 1.49,
+      "grad_norm": 0.8143815994262695,
+      "learning_rate": 3.292152698607768e-05,
+      "loss": 0.9103,
+      "step": 3550
+    },
+    {
+      "epoch": 1.5,
+      "grad_norm": 0.8993640542030334,
+      "learning_rate": 3.240676333441955e-05,
+      "loss": 1.0309,
+      "step": 3560
+    },
+    {
+      "epoch": 1.5,
+      "grad_norm": 1.1748642921447754,
+      "learning_rate": 3.189527658173087e-05,
+      "loss": 1.1086,
+      "step": 3570
+    },
+    {
+      "epoch": 1.51,
+      "grad_norm": 0.6727201342582703,
+      "learning_rate": 3.138709152472938e-05,
+      "loss": 0.9665,
+      "step": 3580
+    },
+    {
+      "epoch": 1.51,
+      "grad_norm": 0.6939325332641602,
+      "learning_rate": 3.088223280006769e-05,
+      "loss": 0.8796,
+      "step": 3590
+    },
+    {
+      "epoch": 1.52,
+      "grad_norm": 0.8073004484176636,
+      "learning_rate": 3.0380724883138856e-05,
+      "loss": 0.8609,
+      "step": 3600
+    },
+    {
+      "epoch": 1.52,
+      "grad_norm": 0.8646635413169861,
+      "learning_rate": 2.9882592086889736e-05,
+      "loss": 1.0229,
+      "step": 3610
+    },
+    {
+      "epoch": 1.52,
+      "grad_norm": 1.1847567558288574,
+      "learning_rate": 2.93878585606426e-05,
+      "loss": 1.033,
+      "step": 3620
+    },
+    {
+      "epoch": 1.53,
+      "grad_norm": 0.6628533601760864,
+      "learning_rate": 2.889654828892393e-05,
+      "loss": 0.9475,
+      "step": 3630
+    },
+    {
+      "epoch": 1.53,
+      "grad_norm": 0.7180255651473999,
+      "learning_rate": 2.8408685090302044e-05,
+      "loss": 0.7856,
+      "step": 3640
+    },
+    {
+      "epoch": 1.54,
+      "grad_norm": 0.7089993953704834,
+      "learning_rate": 2.7924292616232218e-05,
+      "loss": 0.9597,
+      "step": 3650
+    },
+    {
+      "epoch": 1.54,
+      "grad_norm": 0.8458394408226013,
+      "learning_rate": 2.7443394349910045e-05,
+      "loss": 1.0683,
+      "step": 3660
+    },
+    {
+      "epoch": 1.55,
+      "grad_norm": 1.1439454555511475,
+      "learning_rate": 2.6966013605133088e-05,
+      "loss": 1.088,
+      "step": 3670
+    },
+    {
+      "epoch": 1.55,
+      "grad_norm": 0.6164907813072205,
+      "learning_rate": 2.649217352517044e-05,
+      "loss": 0.9503,
+      "step": 3680
+    },
+    {
+      "epoch": 1.55,
+      "grad_norm": 0.7188657522201538,
+      "learning_rate": 2.6021897081641055e-05,
+      "loss": 0.8759,
+      "step": 3690
+    },
+    {
+      "epoch": 1.56,
+      "grad_norm": 0.7899062633514404,
+      "learning_rate": 2.5555207073399733e-05,
+      "loss": 0.9169,
+      "step": 3700
+    },
+    {
+      "epoch": 1.56,
+      "grad_norm": 0.8519218564033508,
+      "learning_rate": 2.5092126125432136e-05,
+      "loss": 1.0466,
+      "step": 3710
+    },
+    {
+      "epoch": 1.57,
+      "grad_norm": 1.1700925827026367,
+      "learning_rate": 2.463267668775767e-05,
+      "loss": 1.1155,
+      "step": 3720
+    },
+    {
+      "epoch": 1.57,
+      "grad_norm": 0.6643344759941101,
+      "learning_rate": 2.417688103434146e-05,
+      "loss": 0.9251,
+      "step": 3730
+    },
+    {
+      "epoch": 1.57,
+      "grad_norm": 0.6496443748474121,
+      "learning_rate": 2.3724761262014182e-05,
+      "loss": 0.8856,
+      "step": 3740
+    },
+    {
+      "epoch": 1.58,
+      "grad_norm": 0.6989282965660095,
+      "learning_rate": 2.3276339289400895e-05,
+      "loss": 0.94,
+      "step": 3750
+    },
+    {
+      "epoch": 1.58,
+      "grad_norm": 0.9367401599884033,
+      "learning_rate": 2.2831636855858663e-05,
+      "loss": 1.0337,
+      "step": 3760
+    },
+    {
+      "epoch": 1.59,
+      "grad_norm": 1.117208480834961,
+      "learning_rate": 2.2390675520422278e-05,
+      "loss": 1.0989,
+      "step": 3770
+    },
+    {
+      "epoch": 1.59,
+      "grad_norm": 0.6940779685974121,
+      "learning_rate": 2.1953476660759398e-05,
+      "loss": 0.9401,
+      "step": 3780
+    },
+    {
+      "epoch": 1.6,
+      "grad_norm": 0.7155762910842896,
+      "learning_rate": 2.1520061472133902e-05,
+      "loss": 0.9211,
+      "step": 3790
+    },
+    {
+      "epoch": 1.6,
+      "grad_norm": 0.7470029592514038,
+      "learning_rate": 2.109045096637865e-05,
+      "loss": 0.9259,
+      "step": 3800
+    },
+    {
+      "epoch": 1.6,
+      "grad_norm": 0.9242197871208191,
+      "learning_rate": 2.0664665970876496e-05,
+      "loss": 1.0221,
+      "step": 3810
+    },
+    {
+      "epoch": 1.61,
+      "grad_norm": 1.1893744468688965,
+      "learning_rate": 2.0242727127550865e-05,
+      "loss": 1.0464,
+      "step": 3820
+    },
+    {
+      "epoch": 1.61,
+      "grad_norm": 0.6893196702003479,
+      "learning_rate": 1.982465489186489e-05,
+      "loss": 0.9307,
+      "step": 3830
+    },
+    {
+      "epoch": 1.62,
+      "grad_norm": 0.7716548442840576,
+      "learning_rate": 1.9410469531829777e-05,
+      "loss": 0.8919,
+      "step": 3840
+    },
+    {
+      "epoch": 1.62,
+      "grad_norm": 0.6742382645606995,
+      "learning_rate": 1.9000191127022237e-05,
+      "loss": 0.8667,
+      "step": 3850
+    },
+    {
+      "epoch": 1.63,
+      "grad_norm": 0.8534088134765625,
+      "learning_rate": 1.8593839567610928e-05,
+      "loss": 1.0633,
+      "step": 3860
+    },
+    {
+      "epoch": 1.63,
+      "grad_norm": 0.9838773608207703,
+      "learning_rate": 1.8191434553392428e-05,
+      "loss": 1.1161,
+      "step": 3870
+    },
+    {
+      "epoch": 1.63,
+      "grad_norm": 0.6817989945411682,
+      "learning_rate": 1.779299559283588e-05,
+      "loss": 0.8697,
+      "step": 3880
+    },
+    {
+      "epoch": 1.64,
+      "grad_norm": 0.63358473777771,
+      "learning_rate": 1.7398542002137475e-05,
+      "loss": 0.8897,
+      "step": 3890
+    },
+    {
+      "epoch": 1.64,
+      "grad_norm": 0.7668142318725586,
+      "learning_rate": 1.7008092904283913e-05,
+      "loss": 0.8808,
+      "step": 3900
+    },
+    {
+      "epoch": 1.65,
+      "grad_norm": 0.833977222442627,
+      "learning_rate": 1.6621667228125302e-05,
+      "loss": 1.0321,
+      "step": 3910
+    },
+    {
+      "epoch": 1.65,
+      "grad_norm": 1.2148581743240356,
+      "learning_rate": 1.6239283707457576e-05,
+      "loss": 1.0653,
+      "step": 3920
+    },
+    {
+      "epoch": 1.65,
+      "grad_norm": 0.6781288385391235,
+      "learning_rate": 1.586096088011414e-05,
+      "loss": 0.9533,
+      "step": 3930
+    },
+    {
+      "epoch": 1.66,
+      "grad_norm": 0.7431279420852661,
+      "learning_rate": 1.5486717087067393e-05,
+      "loss": 0.8971,
+      "step": 3940
+    },
+    {
+      "epoch": 1.66,
+      "grad_norm": 0.7349860668182373,
+      "learning_rate": 1.5116570471539293e-05,
+      "loss": 0.8705,
+      "step": 3950
+    },
+    {
+      "epoch": 1.67,
+      "grad_norm": 0.8228428363800049,
+      "learning_rate": 1.4750538978121987e-05,
+      "loss": 1.075,
+      "step": 3960
+    },
+    {
+      "epoch": 1.67,
+      "grad_norm": 1.0865501165390015,
+      "learning_rate": 1.4388640351907801e-05,
+      "loss": 1.085,
+      "step": 3970
+    },
+    {
+      "epoch": 1.68,
+      "grad_norm": 0.698239803314209,
+      "learning_rate": 1.4030892137628905e-05,
+      "loss": 0.9638,
+      "step": 3980
+    },
+    {
+      "epoch": 1.68,
+      "grad_norm": 0.728515625,
+      "learning_rate": 1.3677311678806838e-05,
+      "loss": 0.8978,
+      "step": 3990
+    },
+    {
+      "epoch": 1.68,
+      "grad_norm": 0.7482006549835205,
+      "learning_rate": 1.33279161169116e-05,
+      "loss": 0.971,
+      "step": 4000
+    },
+    {
+      "epoch": 1.68,
+      "eval_loss": 1.0210562944412231,
+      "eval_runtime": 228.9321,
+      "eval_samples_per_second": 4.368,
+      "eval_steps_per_second": 0.546,
+      "step": 4000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 4750,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 2000,
+  "total_flos": 4.027274639002829e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-4000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32056b300030c8a314c8642f76d2f1834a17c0eb91fd7f49ef60c14a2491fb22
+size 4856

eval_results.json

@@ -0,0 +1,7 @@
+{
+    "epoch": 2.0,
+    "eval_loss": 1.0187978744506836,
+    "eval_runtime": 229.1117,
+    "eval_samples_per_second": 4.365,
+    "eval_steps_per_second": 0.546
+}

merged/config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 5632,
+  "max_position_embeddings": 2048,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 22,
+  "num_key_value_heads": 4,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.38.2",
+  "use_cache": true,
+  "vocab_size": 32000
+}

merged/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "max_length": 2048,
+  "pad_token_id": 0,
+  "transformers_version": "4.38.2"
+}

merged/model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9da1cd0c14c537c97c5b079dde1683225e23c58a81050bd349714a4857675ec2
+size 1976658592

merged/model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d357d68a15606e6356b649a46d82f83dbf18e10aa11705ec5c6770f2984405b4
+size 1984007376

merged/model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccd14aa1ebd67924ff2f702bde33c93dfe84d31a3df21008deab9de58aba4e88
+size 177222184

merged/model.safetensors.index.json

@@ -0,0 +1,208 @@
+{
+  "metadata": {
+    "total_size": 4137865216
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00003-of-00003.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.norm.weight": "model-00003-of-00003.safetensors"
+  }
+}

merged/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

merged/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

merged/tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 2048,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

tokenizer/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer/tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 2048,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

train_results.json

@@ -0,0 +1,7 @@
+{
+    "epoch": 2.0,
+    "train_loss": 1.089675091592889,
+    "train_runtime": 22210.9127,
+    "train_samples_per_second": 1.711,
+    "train_steps_per_second": 0.214
+}