Add initial model files

.ipynb_checkpoints/README-checkpoint.md

@@ -0,0 +1,202 @@
+---
+base_model: Hugofernandez/Mistral-7B-v0.1-colab-sharded
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

.ipynb_checkpoints/adapter_config-checkpoint.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Hugofernandez/Mistral-7B-v0.1-colab-sharded",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

README.md

@@ -1,3 +1,202 @@
----
-license: mit
----
+---
+base_model: Hugofernandez/Mistral-7B-v0.1-colab-sharded
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Hugofernandez/Mistral-7B-v0.1-colab-sharded",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7930def8af856948c5b74f5635b93b87627c5bd2f4bed249222e8e8b7e26795
+size 54560368

checkpoint-1500/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Hugofernandez/Mistral-7B-v0.1-colab-sharded
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

checkpoint-1500/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Hugofernandez/Mistral-7B-v0.1-colab-sharded",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-1500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e78463f7540f0c82bdfa0c5eb20db7bd7bcc8e459e34c648eb43cc2c892fc2a
+size 54560368

checkpoint-1500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2af83a00c0923b9092021625f9f182a553bf77dca1d7acf7f9e130242ea61e15
+size 109267450

checkpoint-1500/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16088195464da3c1c5d391b01b5864881718c722adaa71cc9ff6b8dd9a2c75c9
+size 14244

checkpoint-1500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a688c7a01f640638ea76ca8f1b61d33d9df4d7062ff4db5c09dd32efbf01229
+size 1064

checkpoint-1500/trainer_state.json

@@ -0,0 +1,1203 @@
+{
+  "best_metric": 1.6504695415496826,
+  "best_model_checkpoint": "models/mistral-7b-finetuned/checkpoint-600",
+  "epoch": 4.7936,
+  "eval_steps": 100,
+  "global_step": 1500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.032,
+      "grad_norm": 21.417774200439453,
+      "learning_rate": 2e-05,
+      "loss": 12.9874,
+      "step": 10
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 10.754881858825684,
+      "learning_rate": 4e-05,
+      "loss": 10.8458,
+      "step": 20
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 5.602188587188721,
+      "learning_rate": 6e-05,
+      "loss": 9.5793,
+      "step": 30
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 6.1397199630737305,
+      "learning_rate": 8e-05,
+      "loss": 9.1163,
+      "step": 40
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 5.41762113571167,
+      "learning_rate": 0.0001,
+      "loss": 8.9351,
+      "step": 50
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 6.518367290496826,
+      "learning_rate": 0.00012,
+      "loss": 8.5776,
+      "step": 60
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 8.664566040039062,
+      "learning_rate": 0.00013000000000000002,
+      "loss": 10.0047,
+      "step": 70
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 5.126749515533447,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 7.2974,
+      "step": 80
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 4.876817226409912,
+      "learning_rate": 0.00017,
+      "loss": 7.2398,
+      "step": 90
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 4.667001247406006,
+      "learning_rate": 0.00019,
+      "loss": 7.2189,
+      "step": 100
+    },
+    {
+      "epoch": 0.32,
+      "eval_loss": 1.8090190887451172,
+      "eval_runtime": 284.2204,
+      "eval_samples_per_second": 17.592,
+      "eval_steps_per_second": 1.101,
+      "step": 100
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 4.669983386993408,
+      "learning_rate": 0.0001993150684931507,
+      "loss": 7.1553,
+      "step": 110
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 4.542459487915039,
+      "learning_rate": 0.00019794520547945205,
+      "loss": 7.1573,
+      "step": 120
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 4.285698890686035,
+      "learning_rate": 0.00019657534246575345,
+      "loss": 7.1514,
+      "step": 130
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 4.1678619384765625,
+      "learning_rate": 0.0001952054794520548,
+      "loss": 7.1323,
+      "step": 140
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 4.345493793487549,
+      "learning_rate": 0.00019383561643835617,
+      "loss": 7.0651,
+      "step": 150
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 4.271751403808594,
+      "learning_rate": 0.00019246575342465752,
+      "loss": 7.1547,
+      "step": 160
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 4.395969390869141,
+      "learning_rate": 0.00019109589041095893,
+      "loss": 7.1948,
+      "step": 170
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 4.7001824378967285,
+      "learning_rate": 0.00018972602739726027,
+      "loss": 6.933,
+      "step": 180
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 4.359734058380127,
+      "learning_rate": 0.00018835616438356165,
+      "loss": 7.0293,
+      "step": 190
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 4.372984886169434,
+      "learning_rate": 0.00018698630136986303,
+      "loss": 7.112,
+      "step": 200
+    },
+    {
+      "epoch": 0.64,
+      "eval_loss": 1.7474249601364136,
+      "eval_runtime": 284.175,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 200
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 4.544906139373779,
+      "learning_rate": 0.0001856164383561644,
+      "loss": 7.026,
+      "step": 210
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 4.632802963256836,
+      "learning_rate": 0.00018424657534246578,
+      "loss": 6.8293,
+      "step": 220
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 4.6305460929870605,
+      "learning_rate": 0.00018287671232876713,
+      "loss": 7.0001,
+      "step": 230
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 4.706456661224365,
+      "learning_rate": 0.0001815068493150685,
+      "loss": 6.8841,
+      "step": 240
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 4.436362266540527,
+      "learning_rate": 0.00018013698630136988,
+      "loss": 6.8762,
+      "step": 250
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 4.542873859405518,
+      "learning_rate": 0.00017876712328767126,
+      "loss": 7.0296,
+      "step": 260
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 4.713407516479492,
+      "learning_rate": 0.0001773972602739726,
+      "loss": 6.8233,
+      "step": 270
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 4.381680965423584,
+      "learning_rate": 0.00017602739726027398,
+      "loss": 6.9384,
+      "step": 280
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 4.34271240234375,
+      "learning_rate": 0.00017465753424657536,
+      "loss": 6.8376,
+      "step": 290
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 4.777706623077393,
+      "learning_rate": 0.00017328767123287673,
+      "loss": 6.846,
+      "step": 300
+    },
+    {
+      "epoch": 0.96,
+      "eval_loss": 1.7063945531845093,
+      "eval_runtime": 284.2454,
+      "eval_samples_per_second": 17.59,
+      "eval_steps_per_second": 1.101,
+      "step": 300
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 4.728296279907227,
+      "learning_rate": 0.00017191780821917808,
+      "loss": 6.9672,
+      "step": 310
+    },
+    {
+      "epoch": 1.0224,
+      "grad_norm": 4.8477911949157715,
+      "learning_rate": 0.00017054794520547946,
+      "loss": 6.1095,
+      "step": 320
+    },
+    {
+      "epoch": 1.0544,
+      "grad_norm": 4.551374912261963,
+      "learning_rate": 0.00016917808219178083,
+      "loss": 6.1668,
+      "step": 330
+    },
+    {
+      "epoch": 1.0864,
+      "grad_norm": 5.062366008758545,
+      "learning_rate": 0.0001678082191780822,
+      "loss": 6.1749,
+      "step": 340
+    },
+    {
+      "epoch": 1.1184,
+      "grad_norm": 5.494389057159424,
+      "learning_rate": 0.00016643835616438356,
+      "loss": 6.0886,
+      "step": 350
+    },
+    {
+      "epoch": 1.1504,
+      "grad_norm": 5.126116752624512,
+      "learning_rate": 0.00016506849315068493,
+      "loss": 6.1965,
+      "step": 360
+    },
+    {
+      "epoch": 1.1824,
+      "grad_norm": 5.569834232330322,
+      "learning_rate": 0.0001636986301369863,
+      "loss": 6.14,
+      "step": 370
+    },
+    {
+      "epoch": 1.2144,
+      "grad_norm": 5.212077617645264,
+      "learning_rate": 0.00016232876712328768,
+      "loss": 6.0993,
+      "step": 380
+    },
+    {
+      "epoch": 1.2464,
+      "grad_norm": 5.771147727966309,
+      "learning_rate": 0.00016095890410958906,
+      "loss": 6.1471,
+      "step": 390
+    },
+    {
+      "epoch": 1.2784,
+      "grad_norm": 5.495727062225342,
+      "learning_rate": 0.0001595890410958904,
+      "loss": 6.0653,
+      "step": 400
+    },
+    {
+      "epoch": 1.2784,
+      "eval_loss": 1.6909687519073486,
+      "eval_runtime": 288.2891,
+      "eval_samples_per_second": 17.344,
+      "eval_steps_per_second": 1.086,
+      "step": 400
+    },
+    {
+      "epoch": 1.3104,
+      "grad_norm": 5.4256510734558105,
+      "learning_rate": 0.00015821917808219179,
+      "loss": 6.0756,
+      "step": 410
+    },
+    {
+      "epoch": 1.3424,
+      "grad_norm": 5.075507640838623,
+      "learning_rate": 0.00015684931506849316,
+      "loss": 6.1188,
+      "step": 420
+    },
+    {
+      "epoch": 1.3744,
+      "grad_norm": 5.522802352905273,
+      "learning_rate": 0.00015547945205479454,
+      "loss": 6.0036,
+      "step": 430
+    },
+    {
+      "epoch": 1.4064,
+      "grad_norm": 5.155237674713135,
+      "learning_rate": 0.00015410958904109589,
+      "loss": 6.0873,
+      "step": 440
+    },
+    {
+      "epoch": 1.4384000000000001,
+      "grad_norm": 5.6655144691467285,
+      "learning_rate": 0.0001527397260273973,
+      "loss": 6.0004,
+      "step": 450
+    },
+    {
+      "epoch": 1.4704,
+      "grad_norm": 5.642876148223877,
+      "learning_rate": 0.00015136986301369864,
+      "loss": 6.1436,
+      "step": 460
+    },
+    {
+      "epoch": 1.5024,
+      "grad_norm": 5.669251918792725,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 5.9854,
+      "step": 470
+    },
+    {
+      "epoch": 1.5344,
+      "grad_norm": 5.512977600097656,
+      "learning_rate": 0.00014863013698630136,
+      "loss": 6.0923,
+      "step": 480
+    },
+    {
+      "epoch": 1.5664,
+      "grad_norm": 5.692931652069092,
+      "learning_rate": 0.00014726027397260274,
+      "loss": 6.1155,
+      "step": 490
+    },
+    {
+      "epoch": 1.5984,
+      "grad_norm": 5.3102922439575195,
+      "learning_rate": 0.00014589041095890411,
+      "loss": 6.043,
+      "step": 500
+    },
+    {
+      "epoch": 1.5984,
+      "eval_loss": 1.6654105186462402,
+      "eval_runtime": 284.2701,
+      "eval_samples_per_second": 17.589,
+      "eval_steps_per_second": 1.101,
+      "step": 500
+    },
+    {
+      "epoch": 1.6303999999999998,
+      "grad_norm": 6.025744438171387,
+      "learning_rate": 0.0001445205479452055,
+      "loss": 6.0334,
+      "step": 510
+    },
+    {
+      "epoch": 1.6623999999999999,
+      "grad_norm": 6.030339241027832,
+      "learning_rate": 0.00014315068493150684,
+      "loss": 5.9134,
+      "step": 520
+    },
+    {
+      "epoch": 1.6944,
+      "grad_norm": 5.629039287567139,
+      "learning_rate": 0.00014178082191780822,
+      "loss": 6.0717,
+      "step": 530
+    },
+    {
+      "epoch": 1.7264,
+      "grad_norm": 5.687103271484375,
+      "learning_rate": 0.0001404109589041096,
+      "loss": 5.9897,
+      "step": 540
+    },
+    {
+      "epoch": 1.7584,
+      "grad_norm": 5.512781143188477,
+      "learning_rate": 0.00013904109589041097,
+      "loss": 5.9935,
+      "step": 550
+    },
+    {
+      "epoch": 1.7904,
+      "grad_norm": 5.138786315917969,
+      "learning_rate": 0.00013767123287671234,
+      "loss": 5.9327,
+      "step": 560
+    },
+    {
+      "epoch": 1.8224,
+      "grad_norm": 5.621225833892822,
+      "learning_rate": 0.0001363013698630137,
+      "loss": 6.077,
+      "step": 570
+    },
+    {
+      "epoch": 1.8544,
+      "grad_norm": 5.893583297729492,
+      "learning_rate": 0.00013493150684931507,
+      "loss": 5.9342,
+      "step": 580
+    },
+    {
+      "epoch": 1.8864,
+      "grad_norm": 5.9357781410217285,
+      "learning_rate": 0.00013356164383561644,
+      "loss": 6.0448,
+      "step": 590
+    },
+    {
+      "epoch": 1.9184,
+      "grad_norm": 6.081833839416504,
+      "learning_rate": 0.00013219178082191782,
+      "loss": 5.9358,
+      "step": 600
+    },
+    {
+      "epoch": 1.9184,
+      "eval_loss": 1.6504695415496826,
+      "eval_runtime": 284.2059,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 600
+    },
+    {
+      "epoch": 1.9504000000000001,
+      "grad_norm": 5.667593002319336,
+      "learning_rate": 0.00013082191780821917,
+      "loss": 5.9483,
+      "step": 610
+    },
+    {
+      "epoch": 1.9824000000000002,
+      "grad_norm": 5.708164691925049,
+      "learning_rate": 0.00012945205479452057,
+      "loss": 5.8574,
+      "step": 620
+    },
+    {
+      "epoch": 2.0128,
+      "grad_norm": 5.627442836761475,
+      "learning_rate": 0.00012808219178082192,
+      "loss": 5.3508,
+      "step": 630
+    },
+    {
+      "epoch": 2.0448,
+      "grad_norm": 6.404241561889648,
+      "learning_rate": 0.0001267123287671233,
+      "loss": 5.0398,
+      "step": 640
+    },
+    {
+      "epoch": 2.0768,
+      "grad_norm": 6.678513050079346,
+      "learning_rate": 0.00012534246575342465,
+      "loss": 5.0185,
+      "step": 650
+    },
+    {
+      "epoch": 2.1088,
+      "grad_norm": 6.974586486816406,
+      "learning_rate": 0.00012397260273972605,
+      "loss": 5.0075,
+      "step": 660
+    },
+    {
+      "epoch": 2.1408,
+      "grad_norm": 6.707793235778809,
+      "learning_rate": 0.0001226027397260274,
+      "loss": 4.8282,
+      "step": 670
+    },
+    {
+      "epoch": 2.1728,
+      "grad_norm": 7.186903476715088,
+      "learning_rate": 0.00012123287671232877,
+      "loss": 5.019,
+      "step": 680
+    },
+    {
+      "epoch": 2.2048,
+      "grad_norm": 7.059166431427002,
+      "learning_rate": 0.00011986301369863014,
+      "loss": 4.9958,
+      "step": 690
+    },
+    {
+      "epoch": 2.2368,
+      "grad_norm": 7.017621994018555,
+      "learning_rate": 0.00011849315068493152,
+      "loss": 4.9351,
+      "step": 700
+    },
+    {
+      "epoch": 2.2368,
+      "eval_loss": 1.6881011724472046,
+      "eval_runtime": 284.1907,
+      "eval_samples_per_second": 17.594,
+      "eval_steps_per_second": 1.101,
+      "step": 700
+    },
+    {
+      "epoch": 2.2688,
+      "grad_norm": 7.222373008728027,
+      "learning_rate": 0.00011712328767123289,
+      "loss": 4.9442,
+      "step": 710
+    },
+    {
+      "epoch": 2.3008,
+      "grad_norm": 6.619111061096191,
+      "learning_rate": 0.00011575342465753425,
+      "loss": 5.042,
+      "step": 720
+    },
+    {
+      "epoch": 2.3327999999999998,
+      "grad_norm": 7.72958517074585,
+      "learning_rate": 0.00011438356164383561,
+      "loss": 4.9984,
+      "step": 730
+    },
+    {
+      "epoch": 2.3648,
+      "grad_norm": 7.93172550201416,
+      "learning_rate": 0.000113013698630137,
+      "loss": 4.9636,
+      "step": 740
+    },
+    {
+      "epoch": 2.3968,
+      "grad_norm": 6.98663330078125,
+      "learning_rate": 0.00011164383561643836,
+      "loss": 5.1007,
+      "step": 750
+    },
+    {
+      "epoch": 2.4288,
+      "grad_norm": 7.263844966888428,
+      "learning_rate": 0.00011027397260273973,
+      "loss": 5.0368,
+      "step": 760
+    },
+    {
+      "epoch": 2.4608,
+      "grad_norm": 7.00322151184082,
+      "learning_rate": 0.00010890410958904109,
+      "loss": 4.9507,
+      "step": 770
+    },
+    {
+      "epoch": 2.4928,
+      "grad_norm": 7.051172733306885,
+      "learning_rate": 0.00010753424657534248,
+      "loss": 4.9565,
+      "step": 780
+    },
+    {
+      "epoch": 2.5248,
+      "grad_norm": 7.020392894744873,
+      "learning_rate": 0.00010616438356164384,
+      "loss": 4.9394,
+      "step": 790
+    },
+    {
+      "epoch": 2.5568,
+      "grad_norm": 6.7947258949279785,
+      "learning_rate": 0.0001047945205479452,
+      "loss": 5.0271,
+      "step": 800
+    },
+    {
+      "epoch": 2.5568,
+      "eval_loss": 1.6769148111343384,
+      "eval_runtime": 284.2014,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 800
+    },
+    {
+      "epoch": 2.5888,
+      "grad_norm": 7.476090431213379,
+      "learning_rate": 0.00010342465753424658,
+      "loss": 4.943,
+      "step": 810
+    },
+    {
+      "epoch": 2.6208,
+      "grad_norm": 7.626896381378174,
+      "learning_rate": 0.00010205479452054795,
+      "loss": 5.0302,
+      "step": 820
+    },
+    {
+      "epoch": 2.6528,
+      "grad_norm": 8.080647468566895,
+      "learning_rate": 0.00010068493150684932,
+      "loss": 5.044,
+      "step": 830
+    },
+    {
+      "epoch": 2.6848,
+      "grad_norm": 7.787088394165039,
+      "learning_rate": 9.931506849315069e-05,
+      "loss": 4.9441,
+      "step": 840
+    },
+    {
+      "epoch": 2.7168,
+      "grad_norm": 6.886526584625244,
+      "learning_rate": 9.794520547945207e-05,
+      "loss": 4.9945,
+      "step": 850
+    },
+    {
+      "epoch": 2.7488,
+      "grad_norm": 7.360565662384033,
+      "learning_rate": 9.657534246575343e-05,
+      "loss": 4.9781,
+      "step": 860
+    },
+    {
+      "epoch": 2.7808,
+      "grad_norm": 6.702237606048584,
+      "learning_rate": 9.520547945205481e-05,
+      "loss": 4.9239,
+      "step": 870
+    },
+    {
+      "epoch": 2.8128,
+      "grad_norm": 7.556909561157227,
+      "learning_rate": 9.383561643835617e-05,
+      "loss": 4.947,
+      "step": 880
+    },
+    {
+      "epoch": 2.8448,
+      "grad_norm": 8.256214141845703,
+      "learning_rate": 9.246575342465755e-05,
+      "loss": 4.9977,
+      "step": 890
+    },
+    {
+      "epoch": 2.8768000000000002,
+      "grad_norm": 7.51767110824585,
+      "learning_rate": 9.109589041095891e-05,
+      "loss": 5.033,
+      "step": 900
+    },
+    {
+      "epoch": 2.8768000000000002,
+      "eval_loss": 1.6717288494110107,
+      "eval_runtime": 284.177,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 900
+    },
+    {
+      "epoch": 2.9088000000000003,
+      "grad_norm": 7.163748741149902,
+      "learning_rate": 8.972602739726028e-05,
+      "loss": 5.0054,
+      "step": 910
+    },
+    {
+      "epoch": 2.9408,
+      "grad_norm": 7.538811206817627,
+      "learning_rate": 8.835616438356165e-05,
+      "loss": 4.8454,
+      "step": 920
+    },
+    {
+      "epoch": 2.9728,
+      "grad_norm": 7.5668511390686035,
+      "learning_rate": 8.698630136986302e-05,
+      "loss": 4.921,
+      "step": 930
+    },
+    {
+      "epoch": 3.0032,
+      "grad_norm": 6.434643268585205,
+      "learning_rate": 8.561643835616438e-05,
+      "loss": 4.5925,
+      "step": 940
+    },
+    {
+      "epoch": 3.0352,
+      "grad_norm": 9.08491325378418,
+      "learning_rate": 8.424657534246576e-05,
+      "loss": 4.0337,
+      "step": 950
+    },
+    {
+      "epoch": 3.0672,
+      "grad_norm": 7.357309818267822,
+      "learning_rate": 8.287671232876712e-05,
+      "loss": 4.0417,
+      "step": 960
+    },
+    {
+      "epoch": 3.0992,
+      "grad_norm": 9.104633331298828,
+      "learning_rate": 8.15068493150685e-05,
+      "loss": 4.1111,
+      "step": 970
+    },
+    {
+      "epoch": 3.1312,
+      "grad_norm": 9.187715530395508,
+      "learning_rate": 8.013698630136986e-05,
+      "loss": 4.1661,
+      "step": 980
+    },
+    {
+      "epoch": 3.1632,
+      "grad_norm": 8.985525131225586,
+      "learning_rate": 7.876712328767124e-05,
+      "loss": 3.9923,
+      "step": 990
+    },
+    {
+      "epoch": 3.1952,
+      "grad_norm": 8.606496810913086,
+      "learning_rate": 7.73972602739726e-05,
+      "loss": 4.0144,
+      "step": 1000
+    },
+    {
+      "epoch": 3.1952,
+      "eval_loss": 1.770361304283142,
+      "eval_runtime": 284.1856,
+      "eval_samples_per_second": 17.594,
+      "eval_steps_per_second": 1.101,
+      "step": 1000
+    },
+    {
+      "epoch": 3.2272,
+      "grad_norm": 9.081073760986328,
+      "learning_rate": 7.602739726027398e-05,
+      "loss": 4.1956,
+      "step": 1010
+    },
+    {
+      "epoch": 3.2592,
+      "grad_norm": 8.82044792175293,
+      "learning_rate": 7.465753424657535e-05,
+      "loss": 4.048,
+      "step": 1020
+    },
+    {
+      "epoch": 3.2912,
+      "grad_norm": 8.417832374572754,
+      "learning_rate": 7.328767123287671e-05,
+      "loss": 4.0966,
+      "step": 1030
+    },
+    {
+      "epoch": 3.3232,
+      "grad_norm": 8.612914085388184,
+      "learning_rate": 7.191780821917809e-05,
+      "loss": 4.1092,
+      "step": 1040
+    },
+    {
+      "epoch": 3.3552,
+      "grad_norm": 9.172372817993164,
+      "learning_rate": 7.054794520547947e-05,
+      "loss": 4.1355,
+      "step": 1050
+    },
+    {
+      "epoch": 3.3872,
+      "grad_norm": 9.245715141296387,
+      "learning_rate": 6.917808219178083e-05,
+      "loss": 4.062,
+      "step": 1060
+    },
+    {
+      "epoch": 3.4192,
+      "grad_norm": 8.667610168457031,
+      "learning_rate": 6.78082191780822e-05,
+      "loss": 4.1739,
+      "step": 1070
+    },
+    {
+      "epoch": 3.4512,
+      "grad_norm": 8.895543098449707,
+      "learning_rate": 6.643835616438357e-05,
+      "loss": 4.0495,
+      "step": 1080
+    },
+    {
+      "epoch": 3.4832,
+      "grad_norm": 8.715056419372559,
+      "learning_rate": 6.506849315068494e-05,
+      "loss": 4.1382,
+      "step": 1090
+    },
+    {
+      "epoch": 3.5152,
+      "grad_norm": 7.614181041717529,
+      "learning_rate": 6.36986301369863e-05,
+      "loss": 4.1006,
+      "step": 1100
+    },
+    {
+      "epoch": 3.5152,
+      "eval_loss": 1.7618277072906494,
+      "eval_runtime": 284.175,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 1100
+    },
+    {
+      "epoch": 3.5472,
+      "grad_norm": 9.246039390563965,
+      "learning_rate": 6.232876712328768e-05,
+      "loss": 4.0609,
+      "step": 1110
+    },
+    {
+      "epoch": 3.5792,
+      "grad_norm": 8.759495735168457,
+      "learning_rate": 6.095890410958904e-05,
+      "loss": 4.1395,
+      "step": 1120
+    },
+    {
+      "epoch": 3.6112,
+      "grad_norm": 9.344980239868164,
+      "learning_rate": 5.958904109589042e-05,
+      "loss": 4.0981,
+      "step": 1130
+    },
+    {
+      "epoch": 3.6432,
+      "grad_norm": 9.564950942993164,
+      "learning_rate": 5.821917808219178e-05,
+      "loss": 4.1304,
+      "step": 1140
+    },
+    {
+      "epoch": 3.6752000000000002,
+      "grad_norm": 8.611926078796387,
+      "learning_rate": 5.684931506849316e-05,
+      "loss": 4.0112,
+      "step": 1150
+    },
+    {
+      "epoch": 3.7072000000000003,
+      "grad_norm": 8.46778678894043,
+      "learning_rate": 5.5479452054794526e-05,
+      "loss": 4.1906,
+      "step": 1160
+    },
+    {
+      "epoch": 3.7392,
+      "grad_norm": 8.935091972351074,
+      "learning_rate": 5.4109589041095895e-05,
+      "loss": 4.1618,
+      "step": 1170
+    },
+    {
+      "epoch": 3.7712,
+      "grad_norm": 9.01773452758789,
+      "learning_rate": 5.2739726027397264e-05,
+      "loss": 4.0465,
+      "step": 1180
+    },
+    {
+      "epoch": 3.8032,
+      "grad_norm": 8.965126991271973,
+      "learning_rate": 5.136986301369864e-05,
+      "loss": 4.175,
+      "step": 1190
+    },
+    {
+      "epoch": 3.8352,
+      "grad_norm": 8.331424713134766,
+      "learning_rate": 5e-05,
+      "loss": 4.1717,
+      "step": 1200
+    },
+    {
+      "epoch": 3.8352,
+      "eval_loss": 1.7625691890716553,
+      "eval_runtime": 284.1586,
+      "eval_samples_per_second": 17.596,
+      "eval_steps_per_second": 1.101,
+      "step": 1200
+    },
+    {
+      "epoch": 3.8672,
+      "grad_norm": 9.159274101257324,
+      "learning_rate": 4.863013698630137e-05,
+      "loss": 4.0633,
+      "step": 1210
+    },
+    {
+      "epoch": 3.8992,
+      "grad_norm": 8.6594820022583,
+      "learning_rate": 4.726027397260274e-05,
+      "loss": 4.1609,
+      "step": 1220
+    },
+    {
+      "epoch": 3.9312,
+      "grad_norm": 9.011448860168457,
+      "learning_rate": 4.589041095890411e-05,
+      "loss": 4.1451,
+      "step": 1230
+    },
+    {
+      "epoch": 3.9632,
+      "grad_norm": 8.665167808532715,
+      "learning_rate": 4.452054794520548e-05,
+      "loss": 4.1671,
+      "step": 1240
+    },
+    {
+      "epoch": 3.9952,
+      "grad_norm": 8.256596565246582,
+      "learning_rate": 4.3150684931506855e-05,
+      "loss": 4.0105,
+      "step": 1250
+    },
+    {
+      "epoch": 4.0256,
+      "grad_norm": 9.888619422912598,
+      "learning_rate": 4.1780821917808224e-05,
+      "loss": 3.4033,
+      "step": 1260
+    },
+    {
+      "epoch": 4.0576,
+      "grad_norm": 11.095122337341309,
+      "learning_rate": 4.041095890410959e-05,
+      "loss": 3.4214,
+      "step": 1270
+    },
+    {
+      "epoch": 4.0896,
+      "grad_norm": 8.760441780090332,
+      "learning_rate": 3.904109589041096e-05,
+      "loss": 3.399,
+      "step": 1280
+    },
+    {
+      "epoch": 4.1216,
+      "grad_norm": 9.515835762023926,
+      "learning_rate": 3.767123287671233e-05,
+      "loss": 3.454,
+      "step": 1290
+    },
+    {
+      "epoch": 4.1536,
+      "grad_norm": 9.442548751831055,
+      "learning_rate": 3.63013698630137e-05,
+      "loss": 3.464,
+      "step": 1300
+    },
+    {
+      "epoch": 4.1536,
+      "eval_loss": 1.9104056358337402,
+      "eval_runtime": 284.2013,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 1300
+    },
+    {
+      "epoch": 4.1856,
+      "grad_norm": 10.060050010681152,
+      "learning_rate": 3.493150684931507e-05,
+      "loss": 3.4107,
+      "step": 1310
+    },
+    {
+      "epoch": 4.2176,
+      "grad_norm": 10.52231216430664,
+      "learning_rate": 3.356164383561644e-05,
+      "loss": 3.5689,
+      "step": 1320
+    },
+    {
+      "epoch": 4.2496,
+      "grad_norm": 9.943062782287598,
+      "learning_rate": 3.219178082191781e-05,
+      "loss": 3.4895,
+      "step": 1330
+    },
+    {
+      "epoch": 4.2816,
+      "grad_norm": 10.587417602539062,
+      "learning_rate": 3.082191780821918e-05,
+      "loss": 3.4161,
+      "step": 1340
+    },
+    {
+      "epoch": 4.3136,
+      "grad_norm": 9.584128379821777,
+      "learning_rate": 2.945205479452055e-05,
+      "loss": 3.4561,
+      "step": 1350
+    },
+    {
+      "epoch": 4.3456,
+      "grad_norm": 10.216991424560547,
+      "learning_rate": 2.808219178082192e-05,
+      "loss": 3.4157,
+      "step": 1360
+    },
+    {
+      "epoch": 4.3776,
+      "grad_norm": 9.239152908325195,
+      "learning_rate": 2.671232876712329e-05,
+      "loss": 3.3716,
+      "step": 1370
+    },
+    {
+      "epoch": 4.4096,
+      "grad_norm": 10.390013694763184,
+      "learning_rate": 2.534246575342466e-05,
+      "loss": 3.5381,
+      "step": 1380
+    },
+    {
+      "epoch": 4.4416,
+      "grad_norm": 10.115812301635742,
+      "learning_rate": 2.3972602739726026e-05,
+      "loss": 3.3905,
+      "step": 1390
+    },
+    {
+      "epoch": 4.4736,
+      "grad_norm": 10.157413482666016,
+      "learning_rate": 2.2602739726027396e-05,
+      "loss": 3.5456,
+      "step": 1400
+    },
+    {
+      "epoch": 4.4736,
+      "eval_loss": 1.91524338722229,
+      "eval_runtime": 284.249,
+      "eval_samples_per_second": 17.59,
+      "eval_steps_per_second": 1.101,
+      "step": 1400
+    },
+    {
+      "epoch": 4.5056,
+      "grad_norm": 9.1631498336792,
+      "learning_rate": 2.1232876712328768e-05,
+      "loss": 3.4429,
+      "step": 1410
+    },
+    {
+      "epoch": 4.5376,
+      "grad_norm": 10.102991104125977,
+      "learning_rate": 1.9863013698630137e-05,
+      "loss": 3.4436,
+      "step": 1420
+    },
+    {
+      "epoch": 4.5696,
+      "grad_norm": 9.654899597167969,
+      "learning_rate": 1.8493150684931506e-05,
+      "loss": 3.4385,
+      "step": 1430
+    },
+    {
+      "epoch": 4.6016,
+      "grad_norm": 9.90255069732666,
+      "learning_rate": 1.7123287671232875e-05,
+      "loss": 3.4765,
+      "step": 1440
+    },
+    {
+      "epoch": 4.6336,
+      "grad_norm": 9.965128898620605,
+      "learning_rate": 1.5753424657534248e-05,
+      "loss": 3.3879,
+      "step": 1450
+    },
+    {
+      "epoch": 4.6655999999999995,
+      "grad_norm": 9.81021499633789,
+      "learning_rate": 1.4383561643835617e-05,
+      "loss": 3.4569,
+      "step": 1460
+    },
+    {
+      "epoch": 4.6975999999999996,
+      "grad_norm": 9.7155179977417,
+      "learning_rate": 1.3013698630136986e-05,
+      "loss": 3.4842,
+      "step": 1470
+    },
+    {
+      "epoch": 4.7296,
+      "grad_norm": 8.976916313171387,
+      "learning_rate": 1.1643835616438355e-05,
+      "loss": 3.503,
+      "step": 1480
+    },
+    {
+      "epoch": 4.7616,
+      "grad_norm": 9.62564468383789,
+      "learning_rate": 1.0273972602739726e-05,
+      "loss": 3.3785,
+      "step": 1490
+    },
+    {
+      "epoch": 4.7936,
+      "grad_norm": 9.098535537719727,
+      "learning_rate": 8.904109589041095e-06,
+      "loss": 3.4574,
+      "step": 1500
+    },
+    {
+      "epoch": 4.7936,
+      "eval_loss": 1.9100289344787598,
+      "eval_runtime": 284.167,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 1500
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1560,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.0982376941811139e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95d7fb00a7bada41b9936c9f81c1d2fb99b98284b2c0118daa233b4b6c364fed
+size 5304

checkpoint-1560/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Hugofernandez/Mistral-7B-v0.1-colab-sharded
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

checkpoint-1560/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Hugofernandez/Mistral-7B-v0.1-colab-sharded",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-1560/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fea8b7aad2a2baf6a9594b193d8c7c9fb48495b3b78d027638b174e20a483b62
+size 54560368

checkpoint-1560/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdac10e39eb4bc33e247956167949f4c6a82eb32ad4ddac7bbd39698d9474ee6
+size 109267450

checkpoint-1560/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cb51717d8263ca0f9ffff53643981961fe27e9558c28ec0587a3bcfee30fb3e
+size 14244

checkpoint-1560/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1ddacaaea23ab3a27021353b2a005c2e7ddde74af82009a64fb63596d4d69e6
+size 1064

checkpoint-1560/trainer_state.json

@@ -0,0 +1,1245 @@
+{
+  "best_metric": 1.6504695415496826,
+  "best_model_checkpoint": "models/mistral-7b-finetuned/checkpoint-600",
+  "epoch": 4.9856,
+  "eval_steps": 100,
+  "global_step": 1560,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.032,
+      "grad_norm": 21.417774200439453,
+      "learning_rate": 2e-05,
+      "loss": 12.9874,
+      "step": 10
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 10.754881858825684,
+      "learning_rate": 4e-05,
+      "loss": 10.8458,
+      "step": 20
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 5.602188587188721,
+      "learning_rate": 6e-05,
+      "loss": 9.5793,
+      "step": 30
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 6.1397199630737305,
+      "learning_rate": 8e-05,
+      "loss": 9.1163,
+      "step": 40
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 5.41762113571167,
+      "learning_rate": 0.0001,
+      "loss": 8.9351,
+      "step": 50
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 6.518367290496826,
+      "learning_rate": 0.00012,
+      "loss": 8.5776,
+      "step": 60
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 8.664566040039062,
+      "learning_rate": 0.00013000000000000002,
+      "loss": 10.0047,
+      "step": 70
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 5.126749515533447,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 7.2974,
+      "step": 80
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 4.876817226409912,
+      "learning_rate": 0.00017,
+      "loss": 7.2398,
+      "step": 90
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 4.667001247406006,
+      "learning_rate": 0.00019,
+      "loss": 7.2189,
+      "step": 100
+    },
+    {
+      "epoch": 0.32,
+      "eval_loss": 1.8090190887451172,
+      "eval_runtime": 284.2204,
+      "eval_samples_per_second": 17.592,
+      "eval_steps_per_second": 1.101,
+      "step": 100
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 4.669983386993408,
+      "learning_rate": 0.0001993150684931507,
+      "loss": 7.1553,
+      "step": 110
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 4.542459487915039,
+      "learning_rate": 0.00019794520547945205,
+      "loss": 7.1573,
+      "step": 120
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 4.285698890686035,
+      "learning_rate": 0.00019657534246575345,
+      "loss": 7.1514,
+      "step": 130
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 4.1678619384765625,
+      "learning_rate": 0.0001952054794520548,
+      "loss": 7.1323,
+      "step": 140
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 4.345493793487549,
+      "learning_rate": 0.00019383561643835617,
+      "loss": 7.0651,
+      "step": 150
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 4.271751403808594,
+      "learning_rate": 0.00019246575342465752,
+      "loss": 7.1547,
+      "step": 160
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 4.395969390869141,
+      "learning_rate": 0.00019109589041095893,
+      "loss": 7.1948,
+      "step": 170
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 4.7001824378967285,
+      "learning_rate": 0.00018972602739726027,
+      "loss": 6.933,
+      "step": 180
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 4.359734058380127,
+      "learning_rate": 0.00018835616438356165,
+      "loss": 7.0293,
+      "step": 190
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 4.372984886169434,
+      "learning_rate": 0.00018698630136986303,
+      "loss": 7.112,
+      "step": 200
+    },
+    {
+      "epoch": 0.64,
+      "eval_loss": 1.7474249601364136,
+      "eval_runtime": 284.175,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 200
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 4.544906139373779,
+      "learning_rate": 0.0001856164383561644,
+      "loss": 7.026,
+      "step": 210
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 4.632802963256836,
+      "learning_rate": 0.00018424657534246578,
+      "loss": 6.8293,
+      "step": 220
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 4.6305460929870605,
+      "learning_rate": 0.00018287671232876713,
+      "loss": 7.0001,
+      "step": 230
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 4.706456661224365,
+      "learning_rate": 0.0001815068493150685,
+      "loss": 6.8841,
+      "step": 240
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 4.436362266540527,
+      "learning_rate": 0.00018013698630136988,
+      "loss": 6.8762,
+      "step": 250
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 4.542873859405518,
+      "learning_rate": 0.00017876712328767126,
+      "loss": 7.0296,
+      "step": 260
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 4.713407516479492,
+      "learning_rate": 0.0001773972602739726,
+      "loss": 6.8233,
+      "step": 270
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 4.381680965423584,
+      "learning_rate": 0.00017602739726027398,
+      "loss": 6.9384,
+      "step": 280
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 4.34271240234375,
+      "learning_rate": 0.00017465753424657536,
+      "loss": 6.8376,
+      "step": 290
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 4.777706623077393,
+      "learning_rate": 0.00017328767123287673,
+      "loss": 6.846,
+      "step": 300
+    },
+    {
+      "epoch": 0.96,
+      "eval_loss": 1.7063945531845093,
+      "eval_runtime": 284.2454,
+      "eval_samples_per_second": 17.59,
+      "eval_steps_per_second": 1.101,
+      "step": 300
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 4.728296279907227,
+      "learning_rate": 0.00017191780821917808,
+      "loss": 6.9672,
+      "step": 310
+    },
+    {
+      "epoch": 1.0224,
+      "grad_norm": 4.8477911949157715,
+      "learning_rate": 0.00017054794520547946,
+      "loss": 6.1095,
+      "step": 320
+    },
+    {
+      "epoch": 1.0544,
+      "grad_norm": 4.551374912261963,
+      "learning_rate": 0.00016917808219178083,
+      "loss": 6.1668,
+      "step": 330
+    },
+    {
+      "epoch": 1.0864,
+      "grad_norm": 5.062366008758545,
+      "learning_rate": 0.0001678082191780822,
+      "loss": 6.1749,
+      "step": 340
+    },
+    {
+      "epoch": 1.1184,
+      "grad_norm": 5.494389057159424,
+      "learning_rate": 0.00016643835616438356,
+      "loss": 6.0886,
+      "step": 350
+    },
+    {
+      "epoch": 1.1504,
+      "grad_norm": 5.126116752624512,
+      "learning_rate": 0.00016506849315068493,
+      "loss": 6.1965,
+      "step": 360
+    },
+    {
+      "epoch": 1.1824,
+      "grad_norm": 5.569834232330322,
+      "learning_rate": 0.0001636986301369863,
+      "loss": 6.14,
+      "step": 370
+    },
+    {
+      "epoch": 1.2144,
+      "grad_norm": 5.212077617645264,
+      "learning_rate": 0.00016232876712328768,
+      "loss": 6.0993,
+      "step": 380
+    },
+    {
+      "epoch": 1.2464,
+      "grad_norm": 5.771147727966309,
+      "learning_rate": 0.00016095890410958906,
+      "loss": 6.1471,
+      "step": 390
+    },
+    {
+      "epoch": 1.2784,
+      "grad_norm": 5.495727062225342,
+      "learning_rate": 0.0001595890410958904,
+      "loss": 6.0653,
+      "step": 400
+    },
+    {
+      "epoch": 1.2784,
+      "eval_loss": 1.6909687519073486,
+      "eval_runtime": 288.2891,
+      "eval_samples_per_second": 17.344,
+      "eval_steps_per_second": 1.086,
+      "step": 400
+    },
+    {
+      "epoch": 1.3104,
+      "grad_norm": 5.4256510734558105,
+      "learning_rate": 0.00015821917808219179,
+      "loss": 6.0756,
+      "step": 410
+    },
+    {
+      "epoch": 1.3424,
+      "grad_norm": 5.075507640838623,
+      "learning_rate": 0.00015684931506849316,
+      "loss": 6.1188,
+      "step": 420
+    },
+    {
+      "epoch": 1.3744,
+      "grad_norm": 5.522802352905273,
+      "learning_rate": 0.00015547945205479454,
+      "loss": 6.0036,
+      "step": 430
+    },
+    {
+      "epoch": 1.4064,
+      "grad_norm": 5.155237674713135,
+      "learning_rate": 0.00015410958904109589,
+      "loss": 6.0873,
+      "step": 440
+    },
+    {
+      "epoch": 1.4384000000000001,
+      "grad_norm": 5.6655144691467285,
+      "learning_rate": 0.0001527397260273973,
+      "loss": 6.0004,
+      "step": 450
+    },
+    {
+      "epoch": 1.4704,
+      "grad_norm": 5.642876148223877,
+      "learning_rate": 0.00015136986301369864,
+      "loss": 6.1436,
+      "step": 460
+    },
+    {
+      "epoch": 1.5024,
+      "grad_norm": 5.669251918792725,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 5.9854,
+      "step": 470
+    },
+    {
+      "epoch": 1.5344,
+      "grad_norm": 5.512977600097656,
+      "learning_rate": 0.00014863013698630136,
+      "loss": 6.0923,
+      "step": 480
+    },
+    {
+      "epoch": 1.5664,
+      "grad_norm": 5.692931652069092,
+      "learning_rate": 0.00014726027397260274,
+      "loss": 6.1155,
+      "step": 490
+    },
+    {
+      "epoch": 1.5984,
+      "grad_norm": 5.3102922439575195,
+      "learning_rate": 0.00014589041095890411,
+      "loss": 6.043,
+      "step": 500
+    },
+    {
+      "epoch": 1.5984,
+      "eval_loss": 1.6654105186462402,
+      "eval_runtime": 284.2701,
+      "eval_samples_per_second": 17.589,
+      "eval_steps_per_second": 1.101,
+      "step": 500
+    },
+    {
+      "epoch": 1.6303999999999998,
+      "grad_norm": 6.025744438171387,
+      "learning_rate": 0.0001445205479452055,
+      "loss": 6.0334,
+      "step": 510
+    },
+    {
+      "epoch": 1.6623999999999999,
+      "grad_norm": 6.030339241027832,
+      "learning_rate": 0.00014315068493150684,
+      "loss": 5.9134,
+      "step": 520
+    },
+    {
+      "epoch": 1.6944,
+      "grad_norm": 5.629039287567139,
+      "learning_rate": 0.00014178082191780822,
+      "loss": 6.0717,
+      "step": 530
+    },
+    {
+      "epoch": 1.7264,
+      "grad_norm": 5.687103271484375,
+      "learning_rate": 0.0001404109589041096,
+      "loss": 5.9897,
+      "step": 540
+    },
+    {
+      "epoch": 1.7584,
+      "grad_norm": 5.512781143188477,
+      "learning_rate": 0.00013904109589041097,
+      "loss": 5.9935,
+      "step": 550
+    },
+    {
+      "epoch": 1.7904,
+      "grad_norm": 5.138786315917969,
+      "learning_rate": 0.00013767123287671234,
+      "loss": 5.9327,
+      "step": 560
+    },
+    {
+      "epoch": 1.8224,
+      "grad_norm": 5.621225833892822,
+      "learning_rate": 0.0001363013698630137,
+      "loss": 6.077,
+      "step": 570
+    },
+    {
+      "epoch": 1.8544,
+      "grad_norm": 5.893583297729492,
+      "learning_rate": 0.00013493150684931507,
+      "loss": 5.9342,
+      "step": 580
+    },
+    {
+      "epoch": 1.8864,
+      "grad_norm": 5.9357781410217285,
+      "learning_rate": 0.00013356164383561644,
+      "loss": 6.0448,
+      "step": 590
+    },
+    {
+      "epoch": 1.9184,
+      "grad_norm": 6.081833839416504,
+      "learning_rate": 0.00013219178082191782,
+      "loss": 5.9358,
+      "step": 600
+    },
+    {
+      "epoch": 1.9184,
+      "eval_loss": 1.6504695415496826,
+      "eval_runtime": 284.2059,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 600
+    },
+    {
+      "epoch": 1.9504000000000001,
+      "grad_norm": 5.667593002319336,
+      "learning_rate": 0.00013082191780821917,
+      "loss": 5.9483,
+      "step": 610
+    },
+    {
+      "epoch": 1.9824000000000002,
+      "grad_norm": 5.708164691925049,
+      "learning_rate": 0.00012945205479452057,
+      "loss": 5.8574,
+      "step": 620
+    },
+    {
+      "epoch": 2.0128,
+      "grad_norm": 5.627442836761475,
+      "learning_rate": 0.00012808219178082192,
+      "loss": 5.3508,
+      "step": 630
+    },
+    {
+      "epoch": 2.0448,
+      "grad_norm": 6.404241561889648,
+      "learning_rate": 0.0001267123287671233,
+      "loss": 5.0398,
+      "step": 640
+    },
+    {
+      "epoch": 2.0768,
+      "grad_norm": 6.678513050079346,
+      "learning_rate": 0.00012534246575342465,
+      "loss": 5.0185,
+      "step": 650
+    },
+    {
+      "epoch": 2.1088,
+      "grad_norm": 6.974586486816406,
+      "learning_rate": 0.00012397260273972605,
+      "loss": 5.0075,
+      "step": 660
+    },
+    {
+      "epoch": 2.1408,
+      "grad_norm": 6.707793235778809,
+      "learning_rate": 0.0001226027397260274,
+      "loss": 4.8282,
+      "step": 670
+    },
+    {
+      "epoch": 2.1728,
+      "grad_norm": 7.186903476715088,
+      "learning_rate": 0.00012123287671232877,
+      "loss": 5.019,
+      "step": 680
+    },
+    {
+      "epoch": 2.2048,
+      "grad_norm": 7.059166431427002,
+      "learning_rate": 0.00011986301369863014,
+      "loss": 4.9958,
+      "step": 690
+    },
+    {
+      "epoch": 2.2368,
+      "grad_norm": 7.017621994018555,
+      "learning_rate": 0.00011849315068493152,
+      "loss": 4.9351,
+      "step": 700
+    },
+    {
+      "epoch": 2.2368,
+      "eval_loss": 1.6881011724472046,
+      "eval_runtime": 284.1907,
+      "eval_samples_per_second": 17.594,
+      "eval_steps_per_second": 1.101,
+      "step": 700
+    },
+    {
+      "epoch": 2.2688,
+      "grad_norm": 7.222373008728027,
+      "learning_rate": 0.00011712328767123289,
+      "loss": 4.9442,
+      "step": 710
+    },
+    {
+      "epoch": 2.3008,
+      "grad_norm": 6.619111061096191,
+      "learning_rate": 0.00011575342465753425,
+      "loss": 5.042,
+      "step": 720
+    },
+    {
+      "epoch": 2.3327999999999998,
+      "grad_norm": 7.72958517074585,
+      "learning_rate": 0.00011438356164383561,
+      "loss": 4.9984,
+      "step": 730
+    },
+    {
+      "epoch": 2.3648,
+      "grad_norm": 7.93172550201416,
+      "learning_rate": 0.000113013698630137,
+      "loss": 4.9636,
+      "step": 740
+    },
+    {
+      "epoch": 2.3968,
+      "grad_norm": 6.98663330078125,
+      "learning_rate": 0.00011164383561643836,
+      "loss": 5.1007,
+      "step": 750
+    },
+    {
+      "epoch": 2.4288,
+      "grad_norm": 7.263844966888428,
+      "learning_rate": 0.00011027397260273973,
+      "loss": 5.0368,
+      "step": 760
+    },
+    {
+      "epoch": 2.4608,
+      "grad_norm": 7.00322151184082,
+      "learning_rate": 0.00010890410958904109,
+      "loss": 4.9507,
+      "step": 770
+    },
+    {
+      "epoch": 2.4928,
+      "grad_norm": 7.051172733306885,
+      "learning_rate": 0.00010753424657534248,
+      "loss": 4.9565,
+      "step": 780
+    },
+    {
+      "epoch": 2.5248,
+      "grad_norm": 7.020392894744873,
+      "learning_rate": 0.00010616438356164384,
+      "loss": 4.9394,
+      "step": 790
+    },
+    {
+      "epoch": 2.5568,
+      "grad_norm": 6.7947258949279785,
+      "learning_rate": 0.0001047945205479452,
+      "loss": 5.0271,
+      "step": 800
+    },
+    {
+      "epoch": 2.5568,
+      "eval_loss": 1.6769148111343384,
+      "eval_runtime": 284.2014,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 800
+    },
+    {
+      "epoch": 2.5888,
+      "grad_norm": 7.476090431213379,
+      "learning_rate": 0.00010342465753424658,
+      "loss": 4.943,
+      "step": 810
+    },
+    {
+      "epoch": 2.6208,
+      "grad_norm": 7.626896381378174,
+      "learning_rate": 0.00010205479452054795,
+      "loss": 5.0302,
+      "step": 820
+    },
+    {
+      "epoch": 2.6528,
+      "grad_norm": 8.080647468566895,
+      "learning_rate": 0.00010068493150684932,
+      "loss": 5.044,
+      "step": 830
+    },
+    {
+      "epoch": 2.6848,
+      "grad_norm": 7.787088394165039,
+      "learning_rate": 9.931506849315069e-05,
+      "loss": 4.9441,
+      "step": 840
+    },
+    {
+      "epoch": 2.7168,
+      "grad_norm": 6.886526584625244,
+      "learning_rate": 9.794520547945207e-05,
+      "loss": 4.9945,
+      "step": 850
+    },
+    {
+      "epoch": 2.7488,
+      "grad_norm": 7.360565662384033,
+      "learning_rate": 9.657534246575343e-05,
+      "loss": 4.9781,
+      "step": 860
+    },
+    {
+      "epoch": 2.7808,
+      "grad_norm": 6.702237606048584,
+      "learning_rate": 9.520547945205481e-05,
+      "loss": 4.9239,
+      "step": 870
+    },
+    {
+      "epoch": 2.8128,
+      "grad_norm": 7.556909561157227,
+      "learning_rate": 9.383561643835617e-05,
+      "loss": 4.947,
+      "step": 880
+    },
+    {
+      "epoch": 2.8448,
+      "grad_norm": 8.256214141845703,
+      "learning_rate": 9.246575342465755e-05,
+      "loss": 4.9977,
+      "step": 890
+    },
+    {
+      "epoch": 2.8768000000000002,
+      "grad_norm": 7.51767110824585,
+      "learning_rate": 9.109589041095891e-05,
+      "loss": 5.033,
+      "step": 900
+    },
+    {
+      "epoch": 2.8768000000000002,
+      "eval_loss": 1.6717288494110107,
+      "eval_runtime": 284.177,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 900
+    },
+    {
+      "epoch": 2.9088000000000003,
+      "grad_norm": 7.163748741149902,
+      "learning_rate": 8.972602739726028e-05,
+      "loss": 5.0054,
+      "step": 910
+    },
+    {
+      "epoch": 2.9408,
+      "grad_norm": 7.538811206817627,
+      "learning_rate": 8.835616438356165e-05,
+      "loss": 4.8454,
+      "step": 920
+    },
+    {
+      "epoch": 2.9728,
+      "grad_norm": 7.5668511390686035,
+      "learning_rate": 8.698630136986302e-05,
+      "loss": 4.921,
+      "step": 930
+    },
+    {
+      "epoch": 3.0032,
+      "grad_norm": 6.434643268585205,
+      "learning_rate": 8.561643835616438e-05,
+      "loss": 4.5925,
+      "step": 940
+    },
+    {
+      "epoch": 3.0352,
+      "grad_norm": 9.08491325378418,
+      "learning_rate": 8.424657534246576e-05,
+      "loss": 4.0337,
+      "step": 950
+    },
+    {
+      "epoch": 3.0672,
+      "grad_norm": 7.357309818267822,
+      "learning_rate": 8.287671232876712e-05,
+      "loss": 4.0417,
+      "step": 960
+    },
+    {
+      "epoch": 3.0992,
+      "grad_norm": 9.104633331298828,
+      "learning_rate": 8.15068493150685e-05,
+      "loss": 4.1111,
+      "step": 970
+    },
+    {
+      "epoch": 3.1312,
+      "grad_norm": 9.187715530395508,
+      "learning_rate": 8.013698630136986e-05,
+      "loss": 4.1661,
+      "step": 980
+    },
+    {
+      "epoch": 3.1632,
+      "grad_norm": 8.985525131225586,
+      "learning_rate": 7.876712328767124e-05,
+      "loss": 3.9923,
+      "step": 990
+    },
+    {
+      "epoch": 3.1952,
+      "grad_norm": 8.606496810913086,
+      "learning_rate": 7.73972602739726e-05,
+      "loss": 4.0144,
+      "step": 1000
+    },
+    {
+      "epoch": 3.1952,
+      "eval_loss": 1.770361304283142,
+      "eval_runtime": 284.1856,
+      "eval_samples_per_second": 17.594,
+      "eval_steps_per_second": 1.101,
+      "step": 1000
+    },
+    {
+      "epoch": 3.2272,
+      "grad_norm": 9.081073760986328,
+      "learning_rate": 7.602739726027398e-05,
+      "loss": 4.1956,
+      "step": 1010
+    },
+    {
+      "epoch": 3.2592,
+      "grad_norm": 8.82044792175293,
+      "learning_rate": 7.465753424657535e-05,
+      "loss": 4.048,
+      "step": 1020
+    },
+    {
+      "epoch": 3.2912,
+      "grad_norm": 8.417832374572754,
+      "learning_rate": 7.328767123287671e-05,
+      "loss": 4.0966,
+      "step": 1030
+    },
+    {
+      "epoch": 3.3232,
+      "grad_norm": 8.612914085388184,
+      "learning_rate": 7.191780821917809e-05,
+      "loss": 4.1092,
+      "step": 1040
+    },
+    {
+      "epoch": 3.3552,
+      "grad_norm": 9.172372817993164,
+      "learning_rate": 7.054794520547947e-05,
+      "loss": 4.1355,
+      "step": 1050
+    },
+    {
+      "epoch": 3.3872,
+      "grad_norm": 9.245715141296387,
+      "learning_rate": 6.917808219178083e-05,
+      "loss": 4.062,
+      "step": 1060
+    },
+    {
+      "epoch": 3.4192,
+      "grad_norm": 8.667610168457031,
+      "learning_rate": 6.78082191780822e-05,
+      "loss": 4.1739,
+      "step": 1070
+    },
+    {
+      "epoch": 3.4512,
+      "grad_norm": 8.895543098449707,
+      "learning_rate": 6.643835616438357e-05,
+      "loss": 4.0495,
+      "step": 1080
+    },
+    {
+      "epoch": 3.4832,
+      "grad_norm": 8.715056419372559,
+      "learning_rate": 6.506849315068494e-05,
+      "loss": 4.1382,
+      "step": 1090
+    },
+    {
+      "epoch": 3.5152,
+      "grad_norm": 7.614181041717529,
+      "learning_rate": 6.36986301369863e-05,
+      "loss": 4.1006,
+      "step": 1100
+    },
+    {
+      "epoch": 3.5152,
+      "eval_loss": 1.7618277072906494,
+      "eval_runtime": 284.175,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 1100
+    },
+    {
+      "epoch": 3.5472,
+      "grad_norm": 9.246039390563965,
+      "learning_rate": 6.232876712328768e-05,
+      "loss": 4.0609,
+      "step": 1110
+    },
+    {
+      "epoch": 3.5792,
+      "grad_norm": 8.759495735168457,
+      "learning_rate": 6.095890410958904e-05,
+      "loss": 4.1395,
+      "step": 1120
+    },
+    {
+      "epoch": 3.6112,
+      "grad_norm": 9.344980239868164,
+      "learning_rate": 5.958904109589042e-05,
+      "loss": 4.0981,
+      "step": 1130
+    },
+    {
+      "epoch": 3.6432,
+      "grad_norm": 9.564950942993164,
+      "learning_rate": 5.821917808219178e-05,
+      "loss": 4.1304,
+      "step": 1140
+    },
+    {
+      "epoch": 3.6752000000000002,
+      "grad_norm": 8.611926078796387,
+      "learning_rate": 5.684931506849316e-05,
+      "loss": 4.0112,
+      "step": 1150
+    },
+    {
+      "epoch": 3.7072000000000003,
+      "grad_norm": 8.46778678894043,
+      "learning_rate": 5.5479452054794526e-05,
+      "loss": 4.1906,
+      "step": 1160
+    },
+    {
+      "epoch": 3.7392,
+      "grad_norm": 8.935091972351074,
+      "learning_rate": 5.4109589041095895e-05,
+      "loss": 4.1618,
+      "step": 1170
+    },
+    {
+      "epoch": 3.7712,
+      "grad_norm": 9.01773452758789,
+      "learning_rate": 5.2739726027397264e-05,
+      "loss": 4.0465,
+      "step": 1180
+    },
+    {
+      "epoch": 3.8032,
+      "grad_norm": 8.965126991271973,
+      "learning_rate": 5.136986301369864e-05,
+      "loss": 4.175,
+      "step": 1190
+    },
+    {
+      "epoch": 3.8352,
+      "grad_norm": 8.331424713134766,
+      "learning_rate": 5e-05,
+      "loss": 4.1717,
+      "step": 1200
+    },
+    {
+      "epoch": 3.8352,
+      "eval_loss": 1.7625691890716553,
+      "eval_runtime": 284.1586,
+      "eval_samples_per_second": 17.596,
+      "eval_steps_per_second": 1.101,
+      "step": 1200
+    },
+    {
+      "epoch": 3.8672,
+      "grad_norm": 9.159274101257324,
+      "learning_rate": 4.863013698630137e-05,
+      "loss": 4.0633,
+      "step": 1210
+    },
+    {
+      "epoch": 3.8992,
+      "grad_norm": 8.6594820022583,
+      "learning_rate": 4.726027397260274e-05,
+      "loss": 4.1609,
+      "step": 1220
+    },
+    {
+      "epoch": 3.9312,
+      "grad_norm": 9.011448860168457,
+      "learning_rate": 4.589041095890411e-05,
+      "loss": 4.1451,
+      "step": 1230
+    },
+    {
+      "epoch": 3.9632,
+      "grad_norm": 8.665167808532715,
+      "learning_rate": 4.452054794520548e-05,
+      "loss": 4.1671,
+      "step": 1240
+    },
+    {
+      "epoch": 3.9952,
+      "grad_norm": 8.256596565246582,
+      "learning_rate": 4.3150684931506855e-05,
+      "loss": 4.0105,
+      "step": 1250
+    },
+    {
+      "epoch": 4.0256,
+      "grad_norm": 9.888619422912598,
+      "learning_rate": 4.1780821917808224e-05,
+      "loss": 3.4033,
+      "step": 1260
+    },
+    {
+      "epoch": 4.0576,
+      "grad_norm": 11.095122337341309,
+      "learning_rate": 4.041095890410959e-05,
+      "loss": 3.4214,
+      "step": 1270
+    },
+    {
+      "epoch": 4.0896,
+      "grad_norm": 8.760441780090332,
+      "learning_rate": 3.904109589041096e-05,
+      "loss": 3.399,
+      "step": 1280
+    },
+    {
+      "epoch": 4.1216,
+      "grad_norm": 9.515835762023926,
+      "learning_rate": 3.767123287671233e-05,
+      "loss": 3.454,
+      "step": 1290
+    },
+    {
+      "epoch": 4.1536,
+      "grad_norm": 9.442548751831055,
+      "learning_rate": 3.63013698630137e-05,
+      "loss": 3.464,
+      "step": 1300
+    },
+    {
+      "epoch": 4.1536,
+      "eval_loss": 1.9104056358337402,
+      "eval_runtime": 284.2013,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 1300
+    },
+    {
+      "epoch": 4.1856,
+      "grad_norm": 10.060050010681152,
+      "learning_rate": 3.493150684931507e-05,
+      "loss": 3.4107,
+      "step": 1310
+    },
+    {
+      "epoch": 4.2176,
+      "grad_norm": 10.52231216430664,
+      "learning_rate": 3.356164383561644e-05,
+      "loss": 3.5689,
+      "step": 1320
+    },
+    {
+      "epoch": 4.2496,
+      "grad_norm": 9.943062782287598,
+      "learning_rate": 3.219178082191781e-05,
+      "loss": 3.4895,
+      "step": 1330
+    },
+    {
+      "epoch": 4.2816,
+      "grad_norm": 10.587417602539062,
+      "learning_rate": 3.082191780821918e-05,
+      "loss": 3.4161,
+      "step": 1340
+    },
+    {
+      "epoch": 4.3136,
+      "grad_norm": 9.584128379821777,
+      "learning_rate": 2.945205479452055e-05,
+      "loss": 3.4561,
+      "step": 1350
+    },
+    {
+      "epoch": 4.3456,
+      "grad_norm": 10.216991424560547,
+      "learning_rate": 2.808219178082192e-05,
+      "loss": 3.4157,
+      "step": 1360
+    },
+    {
+      "epoch": 4.3776,
+      "grad_norm": 9.239152908325195,
+      "learning_rate": 2.671232876712329e-05,
+      "loss": 3.3716,
+      "step": 1370
+    },
+    {
+      "epoch": 4.4096,
+      "grad_norm": 10.390013694763184,
+      "learning_rate": 2.534246575342466e-05,
+      "loss": 3.5381,
+      "step": 1380
+    },
+    {
+      "epoch": 4.4416,
+      "grad_norm": 10.115812301635742,
+      "learning_rate": 2.3972602739726026e-05,
+      "loss": 3.3905,
+      "step": 1390
+    },
+    {
+      "epoch": 4.4736,
+      "grad_norm": 10.157413482666016,
+      "learning_rate": 2.2602739726027396e-05,
+      "loss": 3.5456,
+      "step": 1400
+    },
+    {
+      "epoch": 4.4736,
+      "eval_loss": 1.91524338722229,
+      "eval_runtime": 284.249,
+      "eval_samples_per_second": 17.59,
+      "eval_steps_per_second": 1.101,
+      "step": 1400
+    },
+    {
+      "epoch": 4.5056,
+      "grad_norm": 9.1631498336792,
+      "learning_rate": 2.1232876712328768e-05,
+      "loss": 3.4429,
+      "step": 1410
+    },
+    {
+      "epoch": 4.5376,
+      "grad_norm": 10.102991104125977,
+      "learning_rate": 1.9863013698630137e-05,
+      "loss": 3.4436,
+      "step": 1420
+    },
+    {
+      "epoch": 4.5696,
+      "grad_norm": 9.654899597167969,
+      "learning_rate": 1.8493150684931506e-05,
+      "loss": 3.4385,
+      "step": 1430
+    },
+    {
+      "epoch": 4.6016,
+      "grad_norm": 9.90255069732666,
+      "learning_rate": 1.7123287671232875e-05,
+      "loss": 3.4765,
+      "step": 1440
+    },
+    {
+      "epoch": 4.6336,
+      "grad_norm": 9.965128898620605,
+      "learning_rate": 1.5753424657534248e-05,
+      "loss": 3.3879,
+      "step": 1450
+    },
+    {
+      "epoch": 4.6655999999999995,
+      "grad_norm": 9.81021499633789,
+      "learning_rate": 1.4383561643835617e-05,
+      "loss": 3.4569,
+      "step": 1460
+    },
+    {
+      "epoch": 4.6975999999999996,
+      "grad_norm": 9.7155179977417,
+      "learning_rate": 1.3013698630136986e-05,
+      "loss": 3.4842,
+      "step": 1470
+    },
+    {
+      "epoch": 4.7296,
+      "grad_norm": 8.976916313171387,
+      "learning_rate": 1.1643835616438355e-05,
+      "loss": 3.503,
+      "step": 1480
+    },
+    {
+      "epoch": 4.7616,
+      "grad_norm": 9.62564468383789,
+      "learning_rate": 1.0273972602739726e-05,
+      "loss": 3.3785,
+      "step": 1490
+    },
+    {
+      "epoch": 4.7936,
+      "grad_norm": 9.098535537719727,
+      "learning_rate": 8.904109589041095e-06,
+      "loss": 3.4574,
+      "step": 1500
+    },
+    {
+      "epoch": 4.7936,
+      "eval_loss": 1.9100289344787598,
+      "eval_runtime": 284.167,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 1500
+    },
+    {
+      "epoch": 4.8256,
+      "grad_norm": 11.281402587890625,
+      "learning_rate": 7.5342465753424655e-06,
+      "loss": 3.4439,
+      "step": 1510
+    },
+    {
+      "epoch": 4.8576,
+      "grad_norm": 11.415246963500977,
+      "learning_rate": 6.1643835616438354e-06,
+      "loss": 3.4374,
+      "step": 1520
+    },
+    {
+      "epoch": 4.8896,
+      "grad_norm": 9.063279151916504,
+      "learning_rate": 4.7945205479452054e-06,
+      "loss": 3.3243,
+      "step": 1530
+    },
+    {
+      "epoch": 4.9216,
+      "grad_norm": 9.895434379577637,
+      "learning_rate": 3.4246575342465754e-06,
+      "loss": 3.432,
+      "step": 1540
+    },
+    {
+      "epoch": 4.9536,
+      "grad_norm": 10.165241241455078,
+      "learning_rate": 2.054794520547945e-06,
+      "loss": 3.5225,
+      "step": 1550
+    },
+    {
+      "epoch": 4.9856,
+      "grad_norm": 10.477776527404785,
+      "learning_rate": 6.849315068493151e-07,
+      "loss": 3.4495,
+      "step": 1560
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1560,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.1822792573659382e+18,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1560/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95d7fb00a7bada41b9936c9f81c1d2fb99b98284b2c0118daa233b4b6c364fed
+size 5304

checkpoint-600/.ipynb_checkpoints/trainer_state-checkpoint.json

@@ -0,0 +1,501 @@
+{
+  "best_metric": 1.6504695415496826,
+  "best_model_checkpoint": "models/mistral-7b-finetuned/checkpoint-600",
+  "epoch": 1.9184,
+  "eval_steps": 100,
+  "global_step": 600,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.032,
+      "grad_norm": 21.417774200439453,
+      "learning_rate": 2e-05,
+      "loss": 12.9874,
+      "step": 10
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 10.754881858825684,
+      "learning_rate": 4e-05,
+      "loss": 10.8458,
+      "step": 20
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 5.602188587188721,
+      "learning_rate": 6e-05,
+      "loss": 9.5793,
+      "step": 30
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 6.1397199630737305,
+      "learning_rate": 8e-05,
+      "loss": 9.1163,
+      "step": 40
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 5.41762113571167,
+      "learning_rate": 0.0001,
+      "loss": 8.9351,
+      "step": 50
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 6.518367290496826,
+      "learning_rate": 0.00012,
+      "loss": 8.5776,
+      "step": 60
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 8.664566040039062,
+      "learning_rate": 0.00013000000000000002,
+      "loss": 10.0047,
+      "step": 70
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 5.126749515533447,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 7.2974,
+      "step": 80
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 4.876817226409912,
+      "learning_rate": 0.00017,
+      "loss": 7.2398,
+      "step": 90
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 4.667001247406006,
+      "learning_rate": 0.00019,
+      "loss": 7.2189,
+      "step": 100
+    },
+    {
+      "epoch": 0.32,
+      "eval_loss": 1.8090190887451172,
+      "eval_runtime": 284.2204,
+      "eval_samples_per_second": 17.592,
+      "eval_steps_per_second": 1.101,
+      "step": 100
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 4.669983386993408,
+      "learning_rate": 0.0001993150684931507,
+      "loss": 7.1553,
+      "step": 110
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 4.542459487915039,
+      "learning_rate": 0.00019794520547945205,
+      "loss": 7.1573,
+      "step": 120
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 4.285698890686035,
+      "learning_rate": 0.00019657534246575345,
+      "loss": 7.1514,
+      "step": 130
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 4.1678619384765625,
+      "learning_rate": 0.0001952054794520548,
+      "loss": 7.1323,
+      "step": 140
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 4.345493793487549,
+      "learning_rate": 0.00019383561643835617,
+      "loss": 7.0651,
+      "step": 150
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 4.271751403808594,
+      "learning_rate": 0.00019246575342465752,
+      "loss": 7.1547,
+      "step": 160
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 4.395969390869141,
+      "learning_rate": 0.00019109589041095893,
+      "loss": 7.1948,
+      "step": 170
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 4.7001824378967285,
+      "learning_rate": 0.00018972602739726027,
+      "loss": 6.933,
+      "step": 180
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 4.359734058380127,
+      "learning_rate": 0.00018835616438356165,
+      "loss": 7.0293,
+      "step": 190
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 4.372984886169434,
+      "learning_rate": 0.00018698630136986303,
+      "loss": 7.112,
+      "step": 200
+    },
+    {
+      "epoch": 0.64,
+      "eval_loss": 1.7474249601364136,
+      "eval_runtime": 284.175,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 200
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 4.544906139373779,
+      "learning_rate": 0.0001856164383561644,
+      "loss": 7.026,
+      "step": 210
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 4.632802963256836,
+      "learning_rate": 0.00018424657534246578,
+      "loss": 6.8293,
+      "step": 220
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 4.6305460929870605,
+      "learning_rate": 0.00018287671232876713,
+      "loss": 7.0001,
+      "step": 230
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 4.706456661224365,
+      "learning_rate": 0.0001815068493150685,
+      "loss": 6.8841,
+      "step": 240
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 4.436362266540527,
+      "learning_rate": 0.00018013698630136988,
+      "loss": 6.8762,
+      "step": 250
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 4.542873859405518,
+      "learning_rate": 0.00017876712328767126,
+      "loss": 7.0296,
+      "step": 260
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 4.713407516479492,
+      "learning_rate": 0.0001773972602739726,
+      "loss": 6.8233,
+      "step": 270
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 4.381680965423584,
+      "learning_rate": 0.00017602739726027398,
+      "loss": 6.9384,
+      "step": 280
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 4.34271240234375,
+      "learning_rate": 0.00017465753424657536,
+      "loss": 6.8376,
+      "step": 290
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 4.777706623077393,
+      "learning_rate": 0.00017328767123287673,
+      "loss": 6.846,
+      "step": 300
+    },
+    {
+      "epoch": 0.96,
+      "eval_loss": 1.7063945531845093,
+      "eval_runtime": 284.2454,
+      "eval_samples_per_second": 17.59,
+      "eval_steps_per_second": 1.101,
+      "step": 300
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 4.728296279907227,
+      "learning_rate": 0.00017191780821917808,
+      "loss": 6.9672,
+      "step": 310
+    },
+    {
+      "epoch": 1.0224,
+      "grad_norm": 4.8477911949157715,
+      "learning_rate": 0.00017054794520547946,
+      "loss": 6.1095,
+      "step": 320
+    },
+    {
+      "epoch": 1.0544,
+      "grad_norm": 4.551374912261963,
+      "learning_rate": 0.00016917808219178083,
+      "loss": 6.1668,
+      "step": 330
+    },
+    {
+      "epoch": 1.0864,
+      "grad_norm": 5.062366008758545,
+      "learning_rate": 0.0001678082191780822,
+      "loss": 6.1749,
+      "step": 340
+    },
+    {
+      "epoch": 1.1184,
+      "grad_norm": 5.494389057159424,
+      "learning_rate": 0.00016643835616438356,
+      "loss": 6.0886,
+      "step": 350
+    },
+    {
+      "epoch": 1.1504,
+      "grad_norm": 5.126116752624512,
+      "learning_rate": 0.00016506849315068493,
+      "loss": 6.1965,
+      "step": 360
+    },
+    {
+      "epoch": 1.1824,
+      "grad_norm": 5.569834232330322,
+      "learning_rate": 0.0001636986301369863,
+      "loss": 6.14,
+      "step": 370
+    },
+    {
+      "epoch": 1.2144,
+      "grad_norm": 5.212077617645264,
+      "learning_rate": 0.00016232876712328768,
+      "loss": 6.0993,
+      "step": 380
+    },
+    {
+      "epoch": 1.2464,
+      "grad_norm": 5.771147727966309,
+      "learning_rate": 0.00016095890410958906,
+      "loss": 6.1471,
+      "step": 390
+    },
+    {
+      "epoch": 1.2784,
+      "grad_norm": 5.495727062225342,
+      "learning_rate": 0.0001595890410958904,
+      "loss": 6.0653,
+      "step": 400
+    },
+    {
+      "epoch": 1.2784,
+      "eval_loss": 1.6909687519073486,
+      "eval_runtime": 288.2891,
+      "eval_samples_per_second": 17.344,
+      "eval_steps_per_second": 1.086,
+      "step": 400
+    },
+    {
+      "epoch": 1.3104,
+      "grad_norm": 5.4256510734558105,
+      "learning_rate": 0.00015821917808219179,
+      "loss": 6.0756,
+      "step": 410
+    },
+    {
+      "epoch": 1.3424,
+      "grad_norm": 5.075507640838623,
+      "learning_rate": 0.00015684931506849316,
+      "loss": 6.1188,
+      "step": 420
+    },
+    {
+      "epoch": 1.3744,
+      "grad_norm": 5.522802352905273,
+      "learning_rate": 0.00015547945205479454,
+      "loss": 6.0036,
+      "step": 430
+    },
+    {
+      "epoch": 1.4064,
+      "grad_norm": 5.155237674713135,
+      "learning_rate": 0.00015410958904109589,
+      "loss": 6.0873,
+      "step": 440
+    },
+    {
+      "epoch": 1.4384000000000001,
+      "grad_norm": 5.6655144691467285,
+      "learning_rate": 0.0001527397260273973,
+      "loss": 6.0004,
+      "step": 450
+    },
+    {
+      "epoch": 1.4704,
+      "grad_norm": 5.642876148223877,
+      "learning_rate": 0.00015136986301369864,
+      "loss": 6.1436,
+      "step": 460
+    },
+    {
+      "epoch": 1.5024,
+      "grad_norm": 5.669251918792725,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 5.9854,
+      "step": 470
+    },
+    {
+      "epoch": 1.5344,
+      "grad_norm": 5.512977600097656,
+      "learning_rate": 0.00014863013698630136,
+      "loss": 6.0923,
+      "step": 480
+    },
+    {
+      "epoch": 1.5664,
+      "grad_norm": 5.692931652069092,
+      "learning_rate": 0.00014726027397260274,
+      "loss": 6.1155,
+      "step": 490
+    },
+    {
+      "epoch": 1.5984,
+      "grad_norm": 5.3102922439575195,
+      "learning_rate": 0.00014589041095890411,
+      "loss": 6.043,
+      "step": 500
+    },
+    {
+      "epoch": 1.5984,
+      "eval_loss": 1.6654105186462402,
+      "eval_runtime": 284.2701,
+      "eval_samples_per_second": 17.589,
+      "eval_steps_per_second": 1.101,
+      "step": 500
+    },
+    {
+      "epoch": 1.6303999999999998,
+      "grad_norm": 6.025744438171387,
+      "learning_rate": 0.0001445205479452055,
+      "loss": 6.0334,
+      "step": 510
+    },
+    {
+      "epoch": 1.6623999999999999,
+      "grad_norm": 6.030339241027832,
+      "learning_rate": 0.00014315068493150684,
+      "loss": 5.9134,
+      "step": 520
+    },
+    {
+      "epoch": 1.6944,
+      "grad_norm": 5.629039287567139,
+      "learning_rate": 0.00014178082191780822,
+      "loss": 6.0717,
+      "step": 530
+    },
+    {
+      "epoch": 1.7264,
+      "grad_norm": 5.687103271484375,
+      "learning_rate": 0.0001404109589041096,
+      "loss": 5.9897,
+      "step": 540
+    },
+    {
+      "epoch": 1.7584,
+      "grad_norm": 5.512781143188477,
+      "learning_rate": 0.00013904109589041097,
+      "loss": 5.9935,
+      "step": 550
+    },
+    {
+      "epoch": 1.7904,
+      "grad_norm": 5.138786315917969,
+      "learning_rate": 0.00013767123287671234,
+      "loss": 5.9327,
+      "step": 560
+    },
+    {
+      "epoch": 1.8224,
+      "grad_norm": 5.621225833892822,
+      "learning_rate": 0.0001363013698630137,
+      "loss": 6.077,
+      "step": 570
+    },
+    {
+      "epoch": 1.8544,
+      "grad_norm": 5.893583297729492,
+      "learning_rate": 0.00013493150684931507,
+      "loss": 5.9342,
+      "step": 580
+    },
+    {
+      "epoch": 1.8864,
+      "grad_norm": 5.9357781410217285,
+      "learning_rate": 0.00013356164383561644,
+      "loss": 6.0448,
+      "step": 590
+    },
+    {
+      "epoch": 1.9184,
+      "grad_norm": 6.081833839416504,
+      "learning_rate": 0.00013219178082191782,
+      "loss": 5.9358,
+      "step": 600
+    },
+    {
+      "epoch": 1.9184,
+      "eval_loss": 1.6504695415496826,
+      "eval_runtime": 284.2059,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 600
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1560,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.397152854883697e+17,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-600/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Hugofernandez/Mistral-7B-v0.1-colab-sharded
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.12.0

checkpoint-600/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Hugofernandez/Mistral-7B-v0.1-colab-sharded",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "k_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-600/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7930def8af856948c5b74f5635b93b87627c5bd2f4bed249222e8e8b7e26795
+size 54560368

checkpoint-600/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff6f407082e0c51c80fa75fc32549b30b379c058d73ed5a7828b877e46dad77c
+size 109267450

checkpoint-600/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cf9fb5ae4358802885a59a3a3758a38b66888ab711310bb06bf0cc342d69746
+size 14244

checkpoint-600/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40f6b28960f7139ac7dccae9b82dd1b040e22dcf7fcd80c4e20d29414abf6e90
+size 1064

checkpoint-600/trainer_state.json

@@ -0,0 +1,501 @@
+{
+  "best_metric": 1.6504695415496826,
+  "best_model_checkpoint": "models/mistral-7b-finetuned/checkpoint-600",
+  "epoch": 1.9184,
+  "eval_steps": 100,
+  "global_step": 600,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.032,
+      "grad_norm": 21.417774200439453,
+      "learning_rate": 2e-05,
+      "loss": 12.9874,
+      "step": 10
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 10.754881858825684,
+      "learning_rate": 4e-05,
+      "loss": 10.8458,
+      "step": 20
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 5.602188587188721,
+      "learning_rate": 6e-05,
+      "loss": 9.5793,
+      "step": 30
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 6.1397199630737305,
+      "learning_rate": 8e-05,
+      "loss": 9.1163,
+      "step": 40
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 5.41762113571167,
+      "learning_rate": 0.0001,
+      "loss": 8.9351,
+      "step": 50
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 6.518367290496826,
+      "learning_rate": 0.00012,
+      "loss": 8.5776,
+      "step": 60
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 8.664566040039062,
+      "learning_rate": 0.00013000000000000002,
+      "loss": 10.0047,
+      "step": 70
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 5.126749515533447,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 7.2974,
+      "step": 80
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 4.876817226409912,
+      "learning_rate": 0.00017,
+      "loss": 7.2398,
+      "step": 90
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 4.667001247406006,
+      "learning_rate": 0.00019,
+      "loss": 7.2189,
+      "step": 100
+    },
+    {
+      "epoch": 0.32,
+      "eval_loss": 1.8090190887451172,
+      "eval_runtime": 284.2204,
+      "eval_samples_per_second": 17.592,
+      "eval_steps_per_second": 1.101,
+      "step": 100
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 4.669983386993408,
+      "learning_rate": 0.0001993150684931507,
+      "loss": 7.1553,
+      "step": 110
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 4.542459487915039,
+      "learning_rate": 0.00019794520547945205,
+      "loss": 7.1573,
+      "step": 120
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 4.285698890686035,
+      "learning_rate": 0.00019657534246575345,
+      "loss": 7.1514,
+      "step": 130
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 4.1678619384765625,
+      "learning_rate": 0.0001952054794520548,
+      "loss": 7.1323,
+      "step": 140
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 4.345493793487549,
+      "learning_rate": 0.00019383561643835617,
+      "loss": 7.0651,
+      "step": 150
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 4.271751403808594,
+      "learning_rate": 0.00019246575342465752,
+      "loss": 7.1547,
+      "step": 160
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 4.395969390869141,
+      "learning_rate": 0.00019109589041095893,
+      "loss": 7.1948,
+      "step": 170
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 4.7001824378967285,
+      "learning_rate": 0.00018972602739726027,
+      "loss": 6.933,
+      "step": 180
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 4.359734058380127,
+      "learning_rate": 0.00018835616438356165,
+      "loss": 7.0293,
+      "step": 190
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 4.372984886169434,
+      "learning_rate": 0.00018698630136986303,
+      "loss": 7.112,
+      "step": 200
+    },
+    {
+      "epoch": 0.64,
+      "eval_loss": 1.7474249601364136,
+      "eval_runtime": 284.175,
+      "eval_samples_per_second": 17.595,
+      "eval_steps_per_second": 1.101,
+      "step": 200
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 4.544906139373779,
+      "learning_rate": 0.0001856164383561644,
+      "loss": 7.026,
+      "step": 210
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 4.632802963256836,
+      "learning_rate": 0.00018424657534246578,
+      "loss": 6.8293,
+      "step": 220
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 4.6305460929870605,
+      "learning_rate": 0.00018287671232876713,
+      "loss": 7.0001,
+      "step": 230
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 4.706456661224365,
+      "learning_rate": 0.0001815068493150685,
+      "loss": 6.8841,
+      "step": 240
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 4.436362266540527,
+      "learning_rate": 0.00018013698630136988,
+      "loss": 6.8762,
+      "step": 250
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 4.542873859405518,
+      "learning_rate": 0.00017876712328767126,
+      "loss": 7.0296,
+      "step": 260
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 4.713407516479492,
+      "learning_rate": 0.0001773972602739726,
+      "loss": 6.8233,
+      "step": 270
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 4.381680965423584,
+      "learning_rate": 0.00017602739726027398,
+      "loss": 6.9384,
+      "step": 280
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 4.34271240234375,
+      "learning_rate": 0.00017465753424657536,
+      "loss": 6.8376,
+      "step": 290
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 4.777706623077393,
+      "learning_rate": 0.00017328767123287673,
+      "loss": 6.846,
+      "step": 300
+    },
+    {
+      "epoch": 0.96,
+      "eval_loss": 1.7063945531845093,
+      "eval_runtime": 284.2454,
+      "eval_samples_per_second": 17.59,
+      "eval_steps_per_second": 1.101,
+      "step": 300
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 4.728296279907227,
+      "learning_rate": 0.00017191780821917808,
+      "loss": 6.9672,
+      "step": 310
+    },
+    {
+      "epoch": 1.0224,
+      "grad_norm": 4.8477911949157715,
+      "learning_rate": 0.00017054794520547946,
+      "loss": 6.1095,
+      "step": 320
+    },
+    {
+      "epoch": 1.0544,
+      "grad_norm": 4.551374912261963,
+      "learning_rate": 0.00016917808219178083,
+      "loss": 6.1668,
+      "step": 330
+    },
+    {
+      "epoch": 1.0864,
+      "grad_norm": 5.062366008758545,
+      "learning_rate": 0.0001678082191780822,
+      "loss": 6.1749,
+      "step": 340
+    },
+    {
+      "epoch": 1.1184,
+      "grad_norm": 5.494389057159424,
+      "learning_rate": 0.00016643835616438356,
+      "loss": 6.0886,
+      "step": 350
+    },
+    {
+      "epoch": 1.1504,
+      "grad_norm": 5.126116752624512,
+      "learning_rate": 0.00016506849315068493,
+      "loss": 6.1965,
+      "step": 360
+    },
+    {
+      "epoch": 1.1824,
+      "grad_norm": 5.569834232330322,
+      "learning_rate": 0.0001636986301369863,
+      "loss": 6.14,
+      "step": 370
+    },
+    {
+      "epoch": 1.2144,
+      "grad_norm": 5.212077617645264,
+      "learning_rate": 0.00016232876712328768,
+      "loss": 6.0993,
+      "step": 380
+    },
+    {
+      "epoch": 1.2464,
+      "grad_norm": 5.771147727966309,
+      "learning_rate": 0.00016095890410958906,
+      "loss": 6.1471,
+      "step": 390
+    },
+    {
+      "epoch": 1.2784,
+      "grad_norm": 5.495727062225342,
+      "learning_rate": 0.0001595890410958904,
+      "loss": 6.0653,
+      "step": 400
+    },
+    {
+      "epoch": 1.2784,
+      "eval_loss": 1.6909687519073486,
+      "eval_runtime": 288.2891,
+      "eval_samples_per_second": 17.344,
+      "eval_steps_per_second": 1.086,
+      "step": 400
+    },
+    {
+      "epoch": 1.3104,
+      "grad_norm": 5.4256510734558105,
+      "learning_rate": 0.00015821917808219179,
+      "loss": 6.0756,
+      "step": 410
+    },
+    {
+      "epoch": 1.3424,
+      "grad_norm": 5.075507640838623,
+      "learning_rate": 0.00015684931506849316,
+      "loss": 6.1188,
+      "step": 420
+    },
+    {
+      "epoch": 1.3744,
+      "grad_norm": 5.522802352905273,
+      "learning_rate": 0.00015547945205479454,
+      "loss": 6.0036,
+      "step": 430
+    },
+    {
+      "epoch": 1.4064,
+      "grad_norm": 5.155237674713135,
+      "learning_rate": 0.00015410958904109589,
+      "loss": 6.0873,
+      "step": 440
+    },
+    {
+      "epoch": 1.4384000000000001,
+      "grad_norm": 5.6655144691467285,
+      "learning_rate": 0.0001527397260273973,
+      "loss": 6.0004,
+      "step": 450
+    },
+    {
+      "epoch": 1.4704,
+      "grad_norm": 5.642876148223877,
+      "learning_rate": 0.00015136986301369864,
+      "loss": 6.1436,
+      "step": 460
+    },
+    {
+      "epoch": 1.5024,
+      "grad_norm": 5.669251918792725,
+      "learning_rate": 0.00015000000000000001,
+      "loss": 5.9854,
+      "step": 470
+    },
+    {
+      "epoch": 1.5344,
+      "grad_norm": 5.512977600097656,
+      "learning_rate": 0.00014863013698630136,
+      "loss": 6.0923,
+      "step": 480
+    },
+    {
+      "epoch": 1.5664,
+      "grad_norm": 5.692931652069092,
+      "learning_rate": 0.00014726027397260274,
+      "loss": 6.1155,
+      "step": 490
+    },
+    {
+      "epoch": 1.5984,
+      "grad_norm": 5.3102922439575195,
+      "learning_rate": 0.00014589041095890411,
+      "loss": 6.043,
+      "step": 500
+    },
+    {
+      "epoch": 1.5984,
+      "eval_loss": 1.6654105186462402,
+      "eval_runtime": 284.2701,
+      "eval_samples_per_second": 17.589,
+      "eval_steps_per_second": 1.101,
+      "step": 500
+    },
+    {
+      "epoch": 1.6303999999999998,
+      "grad_norm": 6.025744438171387,
+      "learning_rate": 0.0001445205479452055,
+      "loss": 6.0334,
+      "step": 510
+    },
+    {
+      "epoch": 1.6623999999999999,
+      "grad_norm": 6.030339241027832,
+      "learning_rate": 0.00014315068493150684,
+      "loss": 5.9134,
+      "step": 520
+    },
+    {
+      "epoch": 1.6944,
+      "grad_norm": 5.629039287567139,
+      "learning_rate": 0.00014178082191780822,
+      "loss": 6.0717,
+      "step": 530
+    },
+    {
+      "epoch": 1.7264,
+      "grad_norm": 5.687103271484375,
+      "learning_rate": 0.0001404109589041096,
+      "loss": 5.9897,
+      "step": 540
+    },
+    {
+      "epoch": 1.7584,
+      "grad_norm": 5.512781143188477,
+      "learning_rate": 0.00013904109589041097,
+      "loss": 5.9935,
+      "step": 550
+    },
+    {
+      "epoch": 1.7904,
+      "grad_norm": 5.138786315917969,
+      "learning_rate": 0.00013767123287671234,
+      "loss": 5.9327,
+      "step": 560
+    },
+    {
+      "epoch": 1.8224,
+      "grad_norm": 5.621225833892822,
+      "learning_rate": 0.0001363013698630137,
+      "loss": 6.077,
+      "step": 570
+    },
+    {
+      "epoch": 1.8544,
+      "grad_norm": 5.893583297729492,
+      "learning_rate": 0.00013493150684931507,
+      "loss": 5.9342,
+      "step": 580
+    },
+    {
+      "epoch": 1.8864,
+      "grad_norm": 5.9357781410217285,
+      "learning_rate": 0.00013356164383561644,
+      "loss": 6.0448,
+      "step": 590
+    },
+    {
+      "epoch": 1.9184,
+      "grad_norm": 6.081833839416504,
+      "learning_rate": 0.00013219178082191782,
+      "loss": 5.9358,
+      "step": 600
+    },
+    {
+      "epoch": 1.9184,
+      "eval_loss": 1.6504695415496826,
+      "eval_runtime": 284.2059,
+      "eval_samples_per_second": 17.593,
+      "eval_steps_per_second": 1.101,
+      "step": 600
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1560,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.397152854883697e+17,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-600/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95d7fb00a7bada41b9936c9f81c1d2fb99b98284b2c0118daa233b4b6c364fed
+size 5304

runs/Jan25_17-31-36_c23hpda2/events.out.tfevents.1737822696.c23hpda2.1130850.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:697bd0d56ff975d9e9789287288901089abf298babd61a7c813c250080425268
+size 42818

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95d7fb00a7bada41b9936c9f81c1d2fb99b98284b2c0118daa233b4b6c364fed
+size 5304