Upload fine-tuned troubleshooting-codellama-7b-lora

README.md

@@ -0,0 +1,207 @@
+---
+base_model: codellama/CodeLlama-7b-Instruct-hf
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:codellama/CodeLlama-7b-Instruct-hf
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

adapter_config.json

@@ -0,0 +1,39 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "codellama/CodeLlama-7b-Instruct-hf",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:103b04d33b59fd6ac2823ac9d1374f1e8e4a35f5b066839bb2dc0fa0ea5d0e6e
+size 268470272

chat_template.jinja

@@ -0,0 +1 @@
+{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\n' + system_message + '\n<</SYS>>\n\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + '[INST] ' + content | trim + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' '  + content | trim + ' ' + eos_token }}{% endif %}{% endfor %}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45ccb9c8b6b561889acea59191d66986d314e7cbd6a78abc6e49b139ca91c1e6
+size 500058

tokenizer_config.json

@@ -0,0 +1,85 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "▁<PRE>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "▁<SUF>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "▁<MID>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "▁<EOT>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "▁<PRE>",
+    "▁<MID>",
+    "▁<SUF>",
+    "▁<EOT>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "eot_token": "▁<EOT>",
+  "extra_special_tokens": {},
+  "fill_token": "<FILL_ME>",
+  "legacy": null,
+  "middle_token": "▁<MID>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "prefix_token": "▁<PRE>",
+  "sp_model_kwargs": {},
+  "suffix_token": "▁<SUF>",
+  "tokenizer_class": "CodeLlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f209be5f8fa4dc7e8fb3d8969c4558eef0c3379040ebe8a5f0916ef2cbef847
+size 5969

training_metrics.json

@@ -0,0 +1,1085 @@
+[
+  {
+    "loss": 1.0338,
+    "grad_norm": 0.20301783084869385,
+    "learning_rate": 4e-05,
+    "epoch": 0.02,
+    "step": 10
+  },
+  {
+    "loss": 0.8106,
+    "grad_norm": 0.29791900515556335,
+    "learning_rate": 8.444444444444444e-05,
+    "epoch": 0.04,
+    "step": 20
+  },
+  {
+    "loss": 0.4477,
+    "grad_norm": 0.2857895791530609,
+    "learning_rate": 0.00012888888888888892,
+    "epoch": 0.06,
+    "step": 30
+  },
+  {
+    "loss": 0.3043,
+    "grad_norm": 0.1130324974656105,
+    "learning_rate": 0.00017333333333333334,
+    "epoch": 0.08,
+    "step": 40
+  },
+  {
+    "loss": 0.2469,
+    "grad_norm": 0.06668497622013092,
+    "learning_rate": 0.00019999627041039135,
+    "epoch": 0.1,
+    "step": 50
+  },
+  {
+    "loss": 0.2065,
+    "grad_norm": 0.04713931679725647,
+    "learning_rate": 0.00019995431572214454,
+    "epoch": 0.12,
+    "step": 60
+  },
+  {
+    "loss": 0.2862,
+    "grad_norm": 0.05631931498646736,
+    "learning_rate": 0.00019986576398242566,
+    "epoch": 0.14,
+    "step": 70
+  },
+  {
+    "loss": 0.2329,
+    "grad_norm": 0.027118006721138954,
+    "learning_rate": 0.00019973065647259348,
+    "epoch": 0.16,
+    "step": 80
+  },
+  {
+    "loss": 0.2511,
+    "grad_norm": 0.019656436517834663,
+    "learning_rate": 0.00019954905617753814,
+    "epoch": 0.18,
+    "step": 90
+  },
+  {
+    "loss": 0.2152,
+    "grad_norm": 0.11979807168245316,
+    "learning_rate": 0.00019932104775631846,
+    "epoch": 0.2,
+    "step": 100
+  },
+  {
+    "loss": 0.2556,
+    "grad_norm": 0.025218302384018898,
+    "learning_rate": 0.00019904673750269537,
+    "epoch": 0.22,
+    "step": 110
+  },
+  {
+    "loss": 0.2348,
+    "grad_norm": 0.02096831239759922,
+    "learning_rate": 0.00019872625329557953,
+    "epoch": 0.24,
+    "step": 120
+  },
+  {
+    "loss": 0.2584,
+    "grad_norm": 0.02073720097541809,
+    "learning_rate": 0.0001983597445394162,
+    "epoch": 0.26,
+    "step": 130
+  },
+  {
+    "loss": 0.2241,
+    "grad_norm": 0.04057011008262634,
+    "learning_rate": 0.00019794738209453545,
+    "epoch": 0.28,
+    "step": 140
+  },
+  {
+    "loss": 0.2463,
+    "grad_norm": 0.024945076555013657,
+    "learning_rate": 0.00019748935819749987,
+    "epoch": 0.3,
+    "step": 150
+  },
+  {
+    "loss": 0.2115,
+    "grad_norm": 0.041034162044525146,
+    "learning_rate": 0.00019698588637148703,
+    "epoch": 0.32,
+    "step": 160
+  },
+  {
+    "loss": 0.2357,
+    "grad_norm": 0.03297119215130806,
+    "learning_rate": 0.00019643720132674856,
+    "epoch": 0.34,
+    "step": 170
+  },
+  {
+    "loss": 0.1961,
+    "grad_norm": 0.03046344220638275,
+    "learning_rate": 0.00019584355885119196,
+    "epoch": 0.36,
+    "step": 180
+  },
+  {
+    "loss": 0.2405,
+    "grad_norm": 0.04020893946290016,
+    "learning_rate": 0.00019520523569113677,
+    "epoch": 0.38,
+    "step": 190
+  },
+  {
+    "loss": 0.2361,
+    "grad_norm": 0.044725269079208374,
+    "learning_rate": 0.0001945225294222997,
+    "epoch": 0.4,
+    "step": 200
+  },
+  {
+    "loss": 0.2507,
+    "grad_norm": 0.03552557900547981,
+    "learning_rate": 0.00019379575831106994,
+    "epoch": 0.42,
+    "step": 210
+  },
+  {
+    "loss": 0.205,
+    "grad_norm": 0.030593285337090492,
+    "learning_rate": 0.00019302526116613864,
+    "epoch": 0.44,
+    "step": 220
+  },
+  {
+    "loss": 0.2271,
+    "grad_norm": 0.027227604761719704,
+    "learning_rate": 0.0001922113971805517,
+    "epoch": 0.46,
+    "step": 230
+  },
+  {
+    "loss": 0.2382,
+    "grad_norm": 0.03209502622485161,
+    "learning_rate": 0.0001913545457642601,
+    "epoch": 0.48,
+    "step": 240
+  },
+  {
+    "loss": 0.2138,
+    "grad_norm": 0.042515285313129425,
+    "learning_rate": 0.0001904551063672452,
+    "epoch": 0.5,
+    "step": 250
+  },
+  {
+    "loss": 0.1812,
+    "grad_norm": 0.030030420050024986,
+    "learning_rate": 0.00018951349829330168,
+    "epoch": 0.52,
+    "step": 260
+  },
+  {
+    "loss": 0.2127,
+    "grad_norm": 0.03021136485040188,
+    "learning_rate": 0.0001885301605045651,
+    "epoch": 0.54,
+    "step": 270
+  },
+  {
+    "loss": 0.1924,
+    "grad_norm": 0.013924747705459595,
+    "learning_rate": 0.000187505551416875,
+    "epoch": 0.56,
+    "step": 280
+  },
+  {
+    "loss": 0.216,
+    "grad_norm": 0.04422696679830551,
+    "learning_rate": 0.00018644014868606895,
+    "epoch": 0.58,
+    "step": 290
+  },
+  {
+    "loss": 0.1981,
+    "grad_norm": 0.012003812938928604,
+    "learning_rate": 0.0001853344489853074,
+    "epoch": 0.6,
+    "step": 300
+  },
+  {
+    "loss": 0.2289,
+    "grad_norm": 0.021777210757136345,
+    "learning_rate": 0.0001841889677735327,
+    "epoch": 0.62,
+    "step": 310
+  },
+  {
+    "loss": 0.2247,
+    "grad_norm": 0.01970461569726467,
+    "learning_rate": 0.0001830042390551708,
+    "epoch": 0.64,
+    "step": 320
+  },
+  {
+    "loss": 0.2155,
+    "grad_norm": 0.037736713886260986,
+    "learning_rate": 0.00018178081513118706,
+    "epoch": 0.66,
+    "step": 330
+  },
+  {
+    "loss": 0.204,
+    "grad_norm": 0.01495356485247612,
+    "learning_rate": 0.00018051926634161282,
+    "epoch": 0.68,
+    "step": 340
+  },
+  {
+    "loss": 0.2122,
+    "grad_norm": 0.03197380155324936,
+    "learning_rate": 0.0001792201807996622,
+    "epoch": 0.7,
+    "step": 350
+  },
+  {
+    "loss": 0.2156,
+    "grad_norm": 0.008430073969066143,
+    "learning_rate": 0.00017788416411756338,
+    "epoch": 0.72,
+    "step": 360
+  },
+  {
+    "loss": 0.2389,
+    "grad_norm": 0.028168736025691032,
+    "learning_rate": 0.00017651183912423228,
+    "epoch": 0.74,
+    "step": 370
+  },
+  {
+    "loss": 0.2244,
+    "grad_norm": 0.03992301598191261,
+    "learning_rate": 0.00017510384557492,
+    "epoch": 0.76,
+    "step": 380
+  },
+  {
+    "loss": 0.23,
+    "grad_norm": 0.02907286398112774,
+    "learning_rate": 0.00017366083985296947,
+    "epoch": 0.78,
+    "step": 390
+  },
+  {
+    "loss": 0.2205,
+    "grad_norm": 0.19837206602096558,
+    "learning_rate": 0.00017218349466382023,
+    "epoch": 0.8,
+    "step": 400
+  },
+  {
+    "loss": 0.2106,
+    "grad_norm": 0.08533652871847153,
+    "learning_rate": 0.0001706724987214045,
+    "epoch": 0.82,
+    "step": 410
+  },
+  {
+    "loss": 0.2328,
+    "grad_norm": 0.030324826017022133,
+    "learning_rate": 0.00016912855642708,
+    "epoch": 0.84,
+    "step": 420
+  },
+  {
+    "loss": 0.1786,
+    "grad_norm": 0.020158372819423676,
+    "learning_rate": 0.00016755238754124965,
+    "epoch": 0.86,
+    "step": 430
+  },
+  {
+    "loss": 0.2266,
+    "grad_norm": 0.018949558958411217,
+    "learning_rate": 0.0001659447268478212,
+    "epoch": 0.88,
+    "step": 440
+  },
+  {
+    "loss": 0.2138,
+    "grad_norm": 0.024497846141457558,
+    "learning_rate": 0.00016430632381166305,
+    "epoch": 0.9,
+    "step": 450
+  },
+  {
+    "loss": 0.2029,
+    "grad_norm": 0.014030157588422298,
+    "learning_rate": 0.0001626379422292162,
+    "epoch": 0.92,
+    "step": 460
+  },
+  {
+    "loss": 0.2452,
+    "grad_norm": 0.016469834372401237,
+    "learning_rate": 0.00016094035987242484,
+    "epoch": 0.94,
+    "step": 470
+  },
+  {
+    "loss": 0.2299,
+    "grad_norm": 0.01868318021297455,
+    "learning_rate": 0.00015921436812615204,
+    "epoch": 0.96,
+    "step": 480
+  },
+  {
+    "loss": 0.2318,
+    "grad_norm": 0.013339920900762081,
+    "learning_rate": 0.00015746077161924905,
+    "epoch": 0.98,
+    "step": 490
+  },
+  {
+    "loss": 0.2186,
+    "grad_norm": 0.013921337202191353,
+    "learning_rate": 0.00015568038784945077,
+    "epoch": 1.0,
+    "step": 500
+  },
+  {
+    "eval_loss": 0.22092734277248383,
+    "eval_runtime": 242.1965,
+    "eval_samples_per_second": 4.129,
+    "eval_steps_per_second": 1.032,
+    "epoch": 1.0,
+    "step": 500
+  },
+  {
+    "loss": 0.244,
+    "grad_norm": 0.031378380954265594,
+    "learning_rate": 0.00015387404680227175,
+    "epoch": 1.02,
+    "step": 510
+  },
+  {
+    "loss": 0.2587,
+    "grad_norm": 0.0316997729241848,
+    "learning_rate": 0.00015204259056408046,
+    "epoch": 1.04,
+    "step": 520
+  },
+  {
+    "loss": 0.1868,
+    "grad_norm": 0.020641034469008446,
+    "learning_rate": 0.00015018687292953293,
+    "epoch": 1.06,
+    "step": 530
+  },
+  {
+    "loss": 0.2146,
+    "grad_norm": 0.018853699788451195,
+    "learning_rate": 0.00014830775900354735,
+    "epoch": 1.08,
+    "step": 540
+  },
+  {
+    "loss": 0.1969,
+    "grad_norm": 0.012189121916890144,
+    "learning_rate": 0.00014640612479800686,
+    "epoch": 1.1,
+    "step": 550
+  },
+  {
+    "loss": 0.2358,
+    "grad_norm": 0.029771337285637856,
+    "learning_rate": 0.00014448285682337682,
+    "epoch": 1.12,
+    "step": 560
+  },
+  {
+    "loss": 0.1948,
+    "grad_norm": 0.015213917009532452,
+    "learning_rate": 0.00014253885167542866,
+    "epoch": 1.1400000000000001,
+    "step": 570
+  },
+  {
+    "loss": 0.1775,
+    "grad_norm": 0.019172601401805878,
+    "learning_rate": 0.00014057501561726157,
+    "epoch": 1.16,
+    "step": 580
+  },
+  {
+    "loss": 0.2086,
+    "grad_norm": 0.017535991966724396,
+    "learning_rate": 0.0001385922641568175,
+    "epoch": 1.18,
+    "step": 590
+  },
+  {
+    "loss": 0.2183,
+    "grad_norm": 0.01955597475171089,
+    "learning_rate": 0.00013659152162008676,
+    "epoch": 1.2,
+    "step": 600
+  },
+  {
+    "loss": 0.2203,
+    "grad_norm": 0.021044399589300156,
+    "learning_rate": 0.0001345737207202023,
+    "epoch": 1.22,
+    "step": 610
+  },
+  {
+    "loss": 0.2472,
+    "grad_norm": 0.014098984189331532,
+    "learning_rate": 0.0001325398021226242,
+    "epoch": 1.24,
+    "step": 620
+  },
+  {
+    "loss": 0.2092,
+    "grad_norm": 0.00962766446173191,
+    "learning_rate": 0.00013049071400661716,
+    "epoch": 1.26,
+    "step": 630
+  },
+  {
+    "loss": 0.2226,
+    "grad_norm": 0.016245288774371147,
+    "learning_rate": 0.00012842741162322487,
+    "epoch": 1.28,
+    "step": 640
+  },
+  {
+    "loss": 0.1859,
+    "grad_norm": 0.016099464148283005,
+    "learning_rate": 0.00012635085684994767,
+    "epoch": 1.3,
+    "step": 650
+  },
+  {
+    "loss": 0.2022,
+    "grad_norm": 0.028731973841786385,
+    "learning_rate": 0.00012426201774233135,
+    "epoch": 1.32,
+    "step": 660
+  },
+  {
+    "loss": 0.213,
+    "grad_norm": 0.013753650709986687,
+    "learning_rate": 0.00012216186808267546,
+    "epoch": 1.34,
+    "step": 670
+  },
+  {
+    "loss": 0.2161,
+    "grad_norm": 0.019580556079745293,
+    "learning_rate": 0.0001200513869260721,
+    "epoch": 1.3599999999999999,
+    "step": 680
+  },
+  {
+    "loss": 0.2166,
+    "grad_norm": 0.022302748635411263,
+    "learning_rate": 0.00011793155814398674,
+    "epoch": 1.38,
+    "step": 690
+  },
+  {
+    "loss": 0.2005,
+    "grad_norm": 0.01767272688448429,
+    "learning_rate": 0.00011580336996559343,
+    "epoch": 1.4,
+    "step": 700
+  },
+  {
+    "loss": 0.2049,
+    "grad_norm": 0.04267221689224243,
+    "learning_rate": 0.00011366781451707879,
+    "epoch": 1.42,
+    "step": 710
+  },
+  {
+    "loss": 0.2102,
+    "grad_norm": 0.00890172366052866,
+    "learning_rate": 0.0001115258873591291,
+    "epoch": 1.44,
+    "step": 720
+  },
+  {
+    "loss": 0.2413,
+    "grad_norm": 0.021932514384388924,
+    "learning_rate": 0.00010937858702281631,
+    "epoch": 1.46,
+    "step": 730
+  },
+  {
+    "loss": 0.2287,
+    "grad_norm": 0.013405244797468185,
+    "learning_rate": 0.00010722691454409943,
+    "epoch": 1.48,
+    "step": 740
+  },
+  {
+    "loss": 0.2189,
+    "grad_norm": 0.016190968453884125,
+    "learning_rate": 0.00010507187299715815,
+    "epoch": 1.5,
+    "step": 750
+  },
+  {
+    "loss": 0.2109,
+    "grad_norm": 0.01780977100133896,
+    "learning_rate": 0.00010291446702677599,
+    "epoch": 1.52,
+    "step": 760
+  },
+  {
+    "loss": 0.2475,
+    "grad_norm": 0.02242923714220524,
+    "learning_rate": 0.0001007557023799917,
+    "epoch": 1.54,
+    "step": 770
+  },
+  {
+    "loss": 0.239,
+    "grad_norm": 0.01664547249674797,
+    "learning_rate": 9.859658543723659e-05,
+    "epoch": 1.56,
+    "step": 780
+  },
+  {
+    "loss": 0.2223,
+    "grad_norm": 0.018891215324401855,
+    "learning_rate": 9.643812274317644e-05,
+    "epoch": 1.58,
+    "step": 790
+  },
+  {
+    "loss": 0.207,
+    "grad_norm": 0.012406964786350727,
+    "learning_rate": 9.428132053747712e-05,
+    "epoch": 1.6,
+    "step": 800
+  },
+  {
+    "loss": 0.2257,
+    "grad_norm": 0.017393184825778008,
+    "learning_rate": 9.212718428571231e-05,
+    "epoch": 1.62,
+    "step": 810
+  },
+  {
+    "loss": 0.2283,
+    "grad_norm": 0.013160837814211845,
+    "learning_rate": 8.997671821063191e-05,
+    "epoch": 1.6400000000000001,
+    "step": 820
+  },
+  {
+    "loss": 0.213,
+    "grad_norm": 0.014801936224102974,
+    "learning_rate": 8.783092482401005e-05,
+    "epoch": 1.6600000000000001,
+    "step": 830
+  },
+  {
+    "loss": 0.2217,
+    "grad_norm": 0.013826963491737843,
+    "learning_rate": 8.569080445929073e-05,
+    "epoch": 1.6800000000000002,
+    "step": 840
+  },
+  {
+    "loss": 0.2179,
+    "grad_norm": 0.01602191850543022,
+    "learning_rate": 8.355735480524874e-05,
+    "epoch": 1.7,
+    "step": 850
+  },
+  {
+    "loss": 0.1996,
+    "grad_norm": 0.009928937070071697,
+    "learning_rate": 8.143157044088377e-05,
+    "epoch": 1.72,
+    "step": 860
+  },
+  {
+    "loss": 0.1899,
+    "grad_norm": 0.021860240027308464,
+    "learning_rate": 7.931444237176398e-05,
+    "epoch": 1.74,
+    "step": 870
+  },
+  {
+    "loss": 0.2296,
+    "grad_norm": 0.025594869628548622,
+    "learning_rate": 7.72069575680357e-05,
+    "epoch": 1.76,
+    "step": 880
+  },
+  {
+    "loss": 0.2014,
+    "grad_norm": 0.013402123004198074,
+    "learning_rate": 7.5110098504314e-05,
+    "epoch": 1.78,
+    "step": 890
+  },
+  {
+    "loss": 0.1979,
+    "grad_norm": 0.017627021297812462,
+    "learning_rate": 7.30248427016697e-05,
+    "epoch": 1.8,
+    "step": 900
+  },
+  {
+    "loss": 0.2586,
+    "grad_norm": 0.012801664881408215,
+    "learning_rate": 7.095216227192467e-05,
+    "epoch": 1.8199999999999998,
+    "step": 910
+  },
+  {
+    "loss": 0.2306,
+    "grad_norm": 0.0071096778847277164,
+    "learning_rate": 6.889302346446969e-05,
+    "epoch": 1.8399999999999999,
+    "step": 920
+  },
+  {
+    "loss": 0.2675,
+    "grad_norm": 0.018459230661392212,
+    "learning_rate": 6.684838621581478e-05,
+    "epoch": 1.8599999999999999,
+    "step": 930
+  },
+  {
+    "loss": 0.2198,
+    "grad_norm": 0.01876717247068882,
+    "learning_rate": 6.481920370208274e-05,
+    "epoch": 1.88,
+    "step": 940
+  },
+  {
+    "loss": 0.2406,
+    "grad_norm": 0.008501769043505192,
+    "learning_rate": 6.28064218946542e-05,
+    "epoch": 1.9,
+    "step": 950
+  },
+  {
+    "loss": 0.2338,
+    "grad_norm": 0.012678908184170723,
+    "learning_rate": 6.0810979119171254e-05,
+    "epoch": 1.92,
+    "step": 960
+  },
+  {
+    "loss": 0.2433,
+    "grad_norm": 0.012930098921060562,
+    "learning_rate": 5.883380561810563e-05,
+    "epoch": 1.94,
+    "step": 970
+  },
+  {
+    "loss": 0.2206,
+    "grad_norm": 0.010908545926213264,
+    "learning_rate": 5.6875823117095025e-05,
+    "epoch": 1.96,
+    "step": 980
+  },
+  {
+    "loss": 0.2132,
+    "grad_norm": 0.00644224788993597,
+    "learning_rate": 5.493794439524979e-05,
+    "epoch": 1.98,
+    "step": 990
+  },
+  {
+    "loss": 0.2192,
+    "grad_norm": 0.007400419097393751,
+    "learning_rate": 5.302107285963045e-05,
+    "epoch": 2.0,
+    "step": 1000
+  },
+  {
+    "eval_loss": 0.22025924921035767,
+    "eval_runtime": 241.3764,
+    "eval_samples_per_second": 4.143,
+    "eval_steps_per_second": 1.036,
+    "epoch": 2.0,
+    "step": 1000
+  },
+  {
+    "loss": 0.2088,
+    "grad_norm": 0.016288993880152702,
+    "learning_rate": 5.1126102124094064e-05,
+    "epoch": 2.02,
+    "step": 1010
+  },
+  {
+    "loss": 0.2158,
+    "grad_norm": 0.012635517865419388,
+    "learning_rate": 4.9253915592706515e-05,
+    "epoch": 2.04,
+    "step": 1020
+  },
+  {
+    "loss": 0.2463,
+    "grad_norm": 0.011536939069628716,
+    "learning_rate": 4.74053860479137e-05,
+    "epoch": 2.06,
+    "step": 1030
+  },
+  {
+    "loss": 0.2681,
+    "grad_norm": 0.016435137018561363,
+    "learning_rate": 4.558137524366533e-05,
+    "epoch": 2.08,
+    "step": 1040
+  },
+  {
+    "loss": 0.2011,
+    "grad_norm": 0.016080299392342567,
+    "learning_rate": 4.3782733503678886e-05,
+    "epoch": 2.1,
+    "step": 1050
+  },
+  {
+    "loss": 0.206,
+    "grad_norm": 0.013971799053251743,
+    "learning_rate": 4.2010299325033034e-05,
+    "epoch": 2.12,
+    "step": 1060
+  },
+  {
+    "loss": 0.1889,
+    "grad_norm": 0.007345122750848532,
+    "learning_rate": 4.026489898727419e-05,
+    "epoch": 2.14,
+    "step": 1070
+  },
+  {
+    "loss": 0.2159,
+    "grad_norm": 0.01497857179492712,
+    "learning_rate": 3.854734616721852e-05,
+    "epoch": 2.16,
+    "step": 1080
+  },
+  {
+    "loss": 0.226,
+    "grad_norm": 0.018126606941223145,
+    "learning_rate": 3.6858441559629306e-05,
+    "epoch": 2.18,
+    "step": 1090
+  },
+  {
+    "loss": 0.1763,
+    "grad_norm": 0.01388466078788042,
+    "learning_rate": 3.519897250394612e-05,
+    "epoch": 2.2,
+    "step": 1100
+  },
+  {
+    "loss": 0.2085,
+    "grad_norm": 0.015727991238236427,
+    "learning_rate": 3.3569712617240435e-05,
+    "epoch": 2.22,
+    "step": 1110
+  },
+  {
+    "loss": 0.1822,
+    "grad_norm": 0.007142161950469017,
+    "learning_rate": 3.197142143356787e-05,
+    "epoch": 2.24,
+    "step": 1120
+  },
+  {
+    "loss": 0.2494,
+    "grad_norm": 0.019193366169929504,
+    "learning_rate": 3.040484404988614e-05,
+    "epoch": 2.26,
+    "step": 1130
+  },
+  {
+    "loss": 0.2451,
+    "grad_norm": 0.014046960510313511,
+    "learning_rate": 2.8870710778703103e-05,
+    "epoch": 2.2800000000000002,
+    "step": 1140
+  },
+  {
+    "loss": 0.2359,
+    "grad_norm": 0.01773045025765896,
+    "learning_rate": 2.736973680761702e-05,
+    "epoch": 2.3,
+    "step": 1150
+  },
+  {
+    "loss": 0.2242,
+    "grad_norm": 0.019559817388653755,
+    "learning_rate": 2.590262186590805e-05,
+    "epoch": 2.32,
+    "step": 1160
+  },
+  {
+    "loss": 0.2079,
+    "grad_norm": 0.011157271452248096,
+    "learning_rate": 2.447004989833599e-05,
+    "epoch": 2.34,
+    "step": 1170
+  },
+  {
+    "loss": 0.2273,
+    "grad_norm": 0.013317988254129887,
+    "learning_rate": 2.307268874629649e-05,
+    "epoch": 2.36,
+    "step": 1180
+  },
+  {
+    "loss": 0.194,
+    "grad_norm": 0.014325031079351902,
+    "learning_rate": 2.1711189836484314e-05,
+    "epoch": 2.38,
+    "step": 1190
+  },
+  {
+    "loss": 0.2181,
+    "grad_norm": 0.008949311450123787,
+    "learning_rate": 2.038618787720925e-05,
+    "epoch": 2.4,
+    "step": 1200
+  },
+  {
+    "loss": 0.2337,
+    "grad_norm": 0.008357610553503036,
+    "learning_rate": 1.9098300562505266e-05,
+    "epoch": 2.42,
+    "step": 1210
+  },
+  {
+    "loss": 0.2239,
+    "grad_norm": 0.010320372879505157,
+    "learning_rate": 1.784812828417197e-05,
+    "epoch": 2.44,
+    "step": 1220
+  },
+  {
+    "loss": 0.2114,
+    "grad_norm": 0.0153227923437953,
+    "learning_rate": 1.663625385188182e-05,
+    "epoch": 2.46,
+    "step": 1230
+  },
+  {
+    "loss": 0.2242,
+    "grad_norm": 0.015742843970656395,
+    "learning_rate": 1.5463242221483743e-05,
+    "epoch": 2.48,
+    "step": 1240
+  },
+  {
+    "loss": 0.2042,
+    "grad_norm": 0.00864436011761427,
+    "learning_rate": 1.432964023163028e-05,
+    "epoch": 2.5,
+    "step": 1250
+  },
+  {
+    "loss": 0.1973,
+    "grad_norm": 0.01347821019589901,
+    "learning_rate": 1.3235976348850165e-05,
+    "epoch": 2.52,
+    "step": 1260
+  },
+  {
+    "loss": 0.1884,
+    "grad_norm": 0.012478268705308437,
+    "learning_rate": 1.218276042118629e-05,
+    "epoch": 2.54,
+    "step": 1270
+  },
+  {
+    "loss": 0.2072,
+    "grad_norm": 0.01012639794498682,
+    "learning_rate": 1.1170483440512614e-05,
+    "epoch": 2.56,
+    "step": 1280
+  },
+  {
+    "loss": 0.2106,
+    "grad_norm": 0.01367532555013895,
+    "learning_rate": 1.0199617313642063e-05,
+    "epoch": 2.58,
+    "step": 1290
+  },
+  {
+    "loss": 0.2102,
+    "grad_norm": 0.009128044359385967,
+    "learning_rate": 9.270614642331376e-06,
+    "epoch": 2.6,
+    "step": 1300
+  },
+  {
+    "loss": 0.2267,
+    "grad_norm": 0.010528783313930035,
+    "learning_rate": 8.383908512285555e-06,
+    "epoch": 2.62,
+    "step": 1310
+  },
+  {
+    "loss": 0.2162,
+    "grad_norm": 0.012414328753948212,
+    "learning_rate": 7.5399122912605095e-06,
+    "epoch": 2.64,
+    "step": 1320
+  },
+  {
+    "loss": 0.2309,
+    "grad_norm": 0.013732027262449265,
+    "learning_rate": 6.739019436357774e-06,
+    "epoch": 2.66,
+    "step": 1330
+  },
+  {
+    "loss": 0.1852,
+    "grad_norm": 0.007979297079145908,
+    "learning_rate": 5.981603310601414e-06,
+    "epoch": 2.68,
+    "step": 1340
+  },
+  {
+    "loss": 0.2223,
+    "grad_norm": 0.011046427302062511,
+    "learning_rate": 5.2680170088822425e-06,
+    "epoch": 2.7,
+    "step": 1350
+  },
+  {
+    "loss": 0.2161,
+    "grad_norm": 0.012683490291237831,
+    "learning_rate": 4.5985931933508754e-06,
+    "epoch": 2.7199999999999998,
+    "step": 1360
+  },
+  {
+    "loss": 0.2497,
+    "grad_norm": 0.01153184100985527,
+    "learning_rate": 3.973643938336113e-06,
+    "epoch": 2.74,
+    "step": 1370
+  },
+  {
+    "loss": 0.2206,
+    "grad_norm": 0.010045906528830528,
+    "learning_rate": 3.393460584861008e-06,
+    "epoch": 2.76,
+    "step": 1380
+  },
+  {
+    "loss": 0.2235,
+    "grad_norm": 0.007768252864480019,
+    "learning_rate": 2.8583136048245697e-06,
+    "epoch": 2.7800000000000002,
+    "step": 1390
+  },
+  {
+    "loss": 0.2335,
+    "grad_norm": 0.01499921828508377,
+    "learning_rate": 2.368452474912153e-06,
+    "epoch": 2.8,
+    "step": 1400
+  },
+  {
+    "loss": 0.2029,
+    "grad_norm": 0.010377482511103153,
+    "learning_rate": 1.9241055602935877e-06,
+    "epoch": 2.82,
+    "step": 1410
+  },
+  {
+    "loss": 0.2328,
+    "grad_norm": 0.012535168789327145,
+    "learning_rate": 1.5254800081630826e-06,
+    "epoch": 2.84,
+    "step": 1420
+  },
+  {
+    "loss": 0.2131,
+    "grad_norm": 0.019476955756545067,
+    "learning_rate": 1.1727616511706508e-06,
+    "epoch": 2.86,
+    "step": 1430
+  },
+  {
+    "loss": 0.2109,
+    "grad_norm": 0.010667297057807446,
+    "learning_rate": 8.661149207899844e-07,
+    "epoch": 2.88,
+    "step": 1440
+  },
+  {
+    "loss": 0.2272,
+    "grad_norm": 0.008616985753178596,
+    "learning_rate": 6.056827706632185e-07,
+    "epoch": 2.9,
+    "step": 1450
+  },
+  {
+    "loss": 0.2327,
+    "grad_norm": 0.02126043103635311,
+    "learning_rate": 3.9158660995830545e-07,
+    "epoch": 2.92,
+    "step": 1460
+  },
+  {
+    "loss": 0.2323,
+    "grad_norm": 0.009215867146849632,
+    "learning_rate": 2.2392624677004536e-07,
+    "epoch": 2.94,
+    "step": 1470
+  },
+  {
+    "loss": 0.2039,
+    "grad_norm": 0.009469298645853996,
+    "learning_rate": 1.0277984159122733e-07,
+    "epoch": 2.96,
+    "step": 1480
+  },
+  {
+    "loss": 0.2254,
+    "grad_norm": 0.0077147288247942924,
+    "learning_rate": 2.820387087548726e-08,
+    "epoch": 2.98,
+    "step": 1490
+  },
+  {
+    "loss": 0.2507,
+    "grad_norm": 0.014298005029559135,
+    "learning_rate": 2.331007089351189e-10,
+    "epoch": 3.0,
+    "step": 1500
+  },
+  {
+    "eval_loss": 0.21910983324050903,
+    "eval_runtime": 241.4388,
+    "eval_samples_per_second": 4.142,
+    "eval_steps_per_second": 1.035,
+    "epoch": 3.0,
+    "step": 1500
+  },
+  {
+    "train_runtime": 20992.5211,
+    "train_samples_per_second": 1.143,
+    "train_steps_per_second": 0.071,
+    "total_flos": 1.968395457134592e+18,
+    "train_loss": 0.2320040352344513,
+    "epoch": 3.0,
+    "step": 1500
+  }
+]