Upload 18 files

first time training

README.md

@@ -0,0 +1,202 @@
+---
+base_model: microsoft/Phi-3-mini-4k-instruct
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "microsoft/Phi-3-mini-4k-instruct",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "qkv_proj",
+    "o_proj",
+    "down_proj",
+    "gate_up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b0f4d1be2d30f8d739c7de33ac8884c3189294518cfd14bf78ebe2cc8809e02
+size 100697728

checkpoint-4400/README.md

@@ -0,0 +1,202 @@
+---
+base_model: microsoft/Phi-3-mini-4k-instruct
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-4400/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "microsoft/Phi-3-mini-4k-instruct",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "qkv_proj",
+    "o_proj",
+    "down_proj",
+    "gate_up_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-4400/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:860422fed9192d449bbaad799da52902bcea634b2470df0c171a7d422f11f958
+size 100697728

checkpoint-4400/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f371a2a4e4dafea8ac084da51d8c649b2abaad28bdc8fafdbcd3d6184181de0d
+size 201541754

checkpoint-4400/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c80389514dd2139bf381c1cc48778a2407a96731c7238bb38f039e9982ddf96e
+size 14180

checkpoint-4400/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3800e192756712ddbce2a75e8345dab77b19b165456e08cfc198eb6798665bfc
+size 1064

checkpoint-4400/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<unk>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-4400/tokenizer_config.json

@@ -0,0 +1,130 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "32000": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32001": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32002": {
+      "content": "<|placeholder1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32003": {
+      "content": "<|placeholder2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32004": {
+      "content": "<|placeholder3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32005": {
+      "content": "<|placeholder4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32006": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "<|end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "<|placeholder5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "<|placeholder6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') %}{{'<|user|>' + '\n' + message['content'] + '<|end|>' + '\n' + '<|assistant|>' + '\n'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '<|end|>' + '\n'}}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "legacy": false,
+  "model_max_length": 4096,
+  "pad_token": "<unk>",
+  "padding_side": "left",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-4400/trainer_state.json

@@ -0,0 +1,1573 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9952499434517078,
+  "eval_steps": 500,
+  "global_step": 4400,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.004523863379325944,
+      "grad_norm": 1.2934283018112183,
+      "learning_rate": 1.1299435028248588e-07,
+      "loss": 1.8133,
+      "step": 20
+    },
+    {
+      "epoch": 0.009047726758651889,
+      "grad_norm": 1.8428480625152588,
+      "learning_rate": 2.2598870056497177e-07,
+      "loss": 1.6188,
+      "step": 40
+    },
+    {
+      "epoch": 0.013571590137977834,
+      "grad_norm": 1.6636091470718384,
+      "learning_rate": 3.3898305084745766e-07,
+      "loss": 1.6899,
+      "step": 60
+    },
+    {
+      "epoch": 0.018095453517303777,
+      "grad_norm": 2.0820469856262207,
+      "learning_rate": 4.5197740112994353e-07,
+      "loss": 1.703,
+      "step": 80
+    },
+    {
+      "epoch": 0.022619316896629722,
+      "grad_norm": 2.4241092205047607,
+      "learning_rate": 5.649717514124295e-07,
+      "loss": 1.7203,
+      "step": 100
+    },
+    {
+      "epoch": 0.027143180275955667,
+      "grad_norm": 4.02646017074585,
+      "learning_rate": 6.779661016949153e-07,
+      "loss": 1.6147,
+      "step": 120
+    },
+    {
+      "epoch": 0.03166704365528161,
+      "grad_norm": 2.3496134281158447,
+      "learning_rate": 7.909604519774013e-07,
+      "loss": 1.6992,
+      "step": 140
+    },
+    {
+      "epoch": 0.036190907034607554,
+      "grad_norm": 2.2215747833251953,
+      "learning_rate": 9.039548022598871e-07,
+      "loss": 1.4976,
+      "step": 160
+    },
+    {
+      "epoch": 0.040714770413933496,
+      "grad_norm": 4.31510066986084,
+      "learning_rate": 1.016949152542373e-06,
+      "loss": 1.4859,
+      "step": 180
+    },
+    {
+      "epoch": 0.045238633793259445,
+      "grad_norm": 2.950899600982666,
+      "learning_rate": 1.129943502824859e-06,
+      "loss": 1.3391,
+      "step": 200
+    },
+    {
+      "epoch": 0.049762497172585386,
+      "grad_norm": 4.141122341156006,
+      "learning_rate": 1.2429378531073449e-06,
+      "loss": 1.3377,
+      "step": 220
+    },
+    {
+      "epoch": 0.054286360551911335,
+      "grad_norm": 2.346282720565796,
+      "learning_rate": 1.3559322033898307e-06,
+      "loss": 1.2225,
+      "step": 240
+    },
+    {
+      "epoch": 0.05881022393123728,
+      "grad_norm": 1.5768523216247559,
+      "learning_rate": 1.4689265536723166e-06,
+      "loss": 1.1489,
+      "step": 260
+    },
+    {
+      "epoch": 0.06333408731056323,
+      "grad_norm": 1.6714472770690918,
+      "learning_rate": 1.5819209039548026e-06,
+      "loss": 1.2574,
+      "step": 280
+    },
+    {
+      "epoch": 0.06785795068988916,
+      "grad_norm": 1.2765991687774658,
+      "learning_rate": 1.6949152542372882e-06,
+      "loss": 1.0654,
+      "step": 300
+    },
+    {
+      "epoch": 0.07238181406921511,
+      "grad_norm": 0.9537855386734009,
+      "learning_rate": 1.8079096045197741e-06,
+      "loss": 1.0468,
+      "step": 320
+    },
+    {
+      "epoch": 0.07690567744854106,
+      "grad_norm": 0.8233529329299927,
+      "learning_rate": 1.92090395480226e-06,
+      "loss": 1.0045,
+      "step": 340
+    },
+    {
+      "epoch": 0.08142954082786699,
+      "grad_norm": 0.977318286895752,
+      "learning_rate": 2.033898305084746e-06,
+      "loss": 1.0425,
+      "step": 360
+    },
+    {
+      "epoch": 0.08595340420719294,
+      "grad_norm": 0.7735580205917358,
+      "learning_rate": 2.146892655367232e-06,
+      "loss": 0.8498,
+      "step": 380
+    },
+    {
+      "epoch": 0.09047726758651889,
+      "grad_norm": 0.9134358763694763,
+      "learning_rate": 2.259887005649718e-06,
+      "loss": 0.8149,
+      "step": 400
+    },
+    {
+      "epoch": 0.09500113096584484,
+      "grad_norm": 1.034114956855774,
+      "learning_rate": 2.372881355932204e-06,
+      "loss": 0.9009,
+      "step": 420
+    },
+    {
+      "epoch": 0.09952499434517077,
+      "grad_norm": 0.8178341388702393,
+      "learning_rate": 2.4858757062146898e-06,
+      "loss": 0.8489,
+      "step": 440
+    },
+    {
+      "epoch": 0.10404885772449672,
+      "grad_norm": 0.9051856398582458,
+      "learning_rate": 2.5988700564971753e-06,
+      "loss": 0.8824,
+      "step": 460
+    },
+    {
+      "epoch": 0.10857272110382267,
+      "grad_norm": 0.7567653059959412,
+      "learning_rate": 2.7118644067796613e-06,
+      "loss": 0.8307,
+      "step": 480
+    },
+    {
+      "epoch": 0.1130965844831486,
+      "grad_norm": 0.800140917301178,
+      "learning_rate": 2.8248587570621473e-06,
+      "loss": 0.7441,
+      "step": 500
+    },
+    {
+      "epoch": 0.11762044786247455,
+      "grad_norm": 0.6388680338859558,
+      "learning_rate": 2.9378531073446333e-06,
+      "loss": 0.7953,
+      "step": 520
+    },
+    {
+      "epoch": 0.1221443112418005,
+      "grad_norm": 1.6128122806549072,
+      "learning_rate": 3.0508474576271192e-06,
+      "loss": 0.7497,
+      "step": 540
+    },
+    {
+      "epoch": 0.12666817462112645,
+      "grad_norm": 0.7322567105293274,
+      "learning_rate": 3.163841807909605e-06,
+      "loss": 0.7005,
+      "step": 560
+    },
+    {
+      "epoch": 0.1311920380004524,
+      "grad_norm": 0.678276777267456,
+      "learning_rate": 3.2768361581920903e-06,
+      "loss": 0.678,
+      "step": 580
+    },
+    {
+      "epoch": 0.13571590137977832,
+      "grad_norm": 0.5271691679954529,
+      "learning_rate": 3.3898305084745763e-06,
+      "loss": 0.6971,
+      "step": 600
+    },
+    {
+      "epoch": 0.14023976475910427,
+      "grad_norm": 1.0406115055084229,
+      "learning_rate": 3.5028248587570623e-06,
+      "loss": 0.6432,
+      "step": 620
+    },
+    {
+      "epoch": 0.14476362813843022,
+      "grad_norm": 1.0237056016921997,
+      "learning_rate": 3.6158192090395483e-06,
+      "loss": 0.6278,
+      "step": 640
+    },
+    {
+      "epoch": 0.14928749151775617,
+      "grad_norm": 0.4786810278892517,
+      "learning_rate": 3.7288135593220342e-06,
+      "loss": 0.6545,
+      "step": 660
+    },
+    {
+      "epoch": 0.15381135489708211,
+      "grad_norm": 0.9167463183403015,
+      "learning_rate": 3.84180790960452e-06,
+      "loss": 0.5941,
+      "step": 680
+    },
+    {
+      "epoch": 0.15833521827640806,
+      "grad_norm": 0.764803409576416,
+      "learning_rate": 3.954802259887006e-06,
+      "loss": 0.5993,
+      "step": 700
+    },
+    {
+      "epoch": 0.16285908165573398,
+      "grad_norm": 0.5060482025146484,
+      "learning_rate": 4.067796610169492e-06,
+      "loss": 0.7018,
+      "step": 720
+    },
+    {
+      "epoch": 0.16738294503505993,
+      "grad_norm": 1.2368682622909546,
+      "learning_rate": 4.180790960451978e-06,
+      "loss": 0.656,
+      "step": 740
+    },
+    {
+      "epoch": 0.17190680841438588,
+      "grad_norm": 0.4092175364494324,
+      "learning_rate": 4.293785310734464e-06,
+      "loss": 0.6578,
+      "step": 760
+    },
+    {
+      "epoch": 0.17643067179371183,
+      "grad_norm": 1.4677420854568481,
+      "learning_rate": 4.40677966101695e-06,
+      "loss": 0.6246,
+      "step": 780
+    },
+    {
+      "epoch": 0.18095453517303778,
+      "grad_norm": 0.56260085105896,
+      "learning_rate": 4.519774011299436e-06,
+      "loss": 0.6593,
+      "step": 800
+    },
+    {
+      "epoch": 0.18547839855236373,
+      "grad_norm": 0.6638891696929932,
+      "learning_rate": 4.632768361581922e-06,
+      "loss": 0.6385,
+      "step": 820
+    },
+    {
+      "epoch": 0.19000226193168968,
+      "grad_norm": 0.9223029613494873,
+      "learning_rate": 4.745762711864408e-06,
+      "loss": 0.6267,
+      "step": 840
+    },
+    {
+      "epoch": 0.1945261253110156,
+      "grad_norm": 0.666900634765625,
+      "learning_rate": 4.8587570621468936e-06,
+      "loss": 0.5532,
+      "step": 860
+    },
+    {
+      "epoch": 0.19904998869034154,
+      "grad_norm": 0.7605287432670593,
+      "learning_rate": 4.9717514124293796e-06,
+      "loss": 0.6349,
+      "step": 880
+    },
+    {
+      "epoch": 0.2035738520696675,
+      "grad_norm": 0.7302986979484558,
+      "learning_rate": 4.9997779957257335e-06,
+      "loss": 0.6167,
+      "step": 900
+    },
+    {
+      "epoch": 0.20809771544899344,
+      "grad_norm": 0.9552111625671387,
+      "learning_rate": 4.998791389568115e-06,
+      "loss": 0.6314,
+      "step": 920
+    },
+    {
+      "epoch": 0.2126215788283194,
+      "grad_norm": 1.1077909469604492,
+      "learning_rate": 4.997015825567898e-06,
+      "loss": 0.6525,
+      "step": 940
+    },
+    {
+      "epoch": 0.21714544220764534,
+      "grad_norm": 0.6709986925125122,
+      "learning_rate": 4.994451864334162e-06,
+      "loss": 0.5795,
+      "step": 960
+    },
+    {
+      "epoch": 0.22166930558697126,
+      "grad_norm": 0.5601279735565186,
+      "learning_rate": 4.991100315401203e-06,
+      "loss": 0.6191,
+      "step": 980
+    },
+    {
+      "epoch": 0.2261931689662972,
+      "grad_norm": 1.1237138509750366,
+      "learning_rate": 4.986962236972929e-06,
+      "loss": 0.6329,
+      "step": 1000
+    },
+    {
+      "epoch": 0.23071703234562316,
+      "grad_norm": 0.987455427646637,
+      "learning_rate": 4.982038935588749e-06,
+      "loss": 0.572,
+      "step": 1020
+    },
+    {
+      "epoch": 0.2352408957249491,
+      "grad_norm": 1.0568902492523193,
+      "learning_rate": 4.976331965711056e-06,
+      "loss": 0.6261,
+      "step": 1040
+    },
+    {
+      "epoch": 0.23976475910427505,
+      "grad_norm": 0.5604099035263062,
+      "learning_rate": 4.969843129234422e-06,
+      "loss": 0.554,
+      "step": 1060
+    },
+    {
+      "epoch": 0.244288622483601,
+      "grad_norm": 0.9130414128303528,
+      "learning_rate": 4.962574474916678e-06,
+      "loss": 0.5794,
+      "step": 1080
+    },
+    {
+      "epoch": 0.24881248586292695,
+      "grad_norm": 1.5077348947525024,
+      "learning_rate": 4.954528297732047e-06,
+      "loss": 0.6414,
+      "step": 1100
+    },
+    {
+      "epoch": 0.2533363492422529,
+      "grad_norm": 0.5713602900505066,
+      "learning_rate": 4.94570713814654e-06,
+      "loss": 0.6231,
+      "step": 1120
+    },
+    {
+      "epoch": 0.2578602126215788,
+      "grad_norm": 1.139797568321228,
+      "learning_rate": 4.93611378131584e-06,
+      "loss": 0.6535,
+      "step": 1140
+    },
+    {
+      "epoch": 0.2623840760009048,
+      "grad_norm": 1.476881980895996,
+      "learning_rate": 4.925751256205924e-06,
+      "loss": 0.561,
+      "step": 1160
+    },
+    {
+      "epoch": 0.2669079393802307,
+      "grad_norm": 0.9005554914474487,
+      "learning_rate": 4.914622834636717e-06,
+      "loss": 0.6006,
+      "step": 1180
+    },
+    {
+      "epoch": 0.27143180275955664,
+      "grad_norm": 0.6789364218711853,
+      "learning_rate": 4.902732030249057e-06,
+      "loss": 0.6368,
+      "step": 1200
+    },
+    {
+      "epoch": 0.2759556661388826,
+      "grad_norm": 0.8574463725090027,
+      "learning_rate": 4.890082597395316e-06,
+      "loss": 0.6138,
+      "step": 1220
+    },
+    {
+      "epoch": 0.28047952951820854,
+      "grad_norm": 1.226874589920044,
+      "learning_rate": 4.876678529954011e-06,
+      "loss": 0.618,
+      "step": 1240
+    },
+    {
+      "epoch": 0.2850033928975345,
+      "grad_norm": 0.49481701850891113,
+      "learning_rate": 4.8625240600688e-06,
+      "loss": 0.5962,
+      "step": 1260
+    },
+    {
+      "epoch": 0.28952725627686043,
+      "grad_norm": 0.8259846568107605,
+      "learning_rate": 4.847623656812241e-06,
+      "loss": 0.5725,
+      "step": 1280
+    },
+    {
+      "epoch": 0.2940511196561864,
+      "grad_norm": 1.3541384935379028,
+      "learning_rate": 4.831982024774742e-06,
+      "loss": 0.6237,
+      "step": 1300
+    },
+    {
+      "epoch": 0.29857498303551233,
+      "grad_norm": 0.6489030718803406,
+      "learning_rate": 4.815604102579159e-06,
+      "loss": 0.6559,
+      "step": 1320
+    },
+    {
+      "epoch": 0.30309884641483825,
+      "grad_norm": 0.6666083931922913,
+      "learning_rate": 4.798495061321492e-06,
+      "loss": 0.5905,
+      "step": 1340
+    },
+    {
+      "epoch": 0.30762270979416423,
+      "grad_norm": 1.033155918121338,
+      "learning_rate": 4.780660302938188e-06,
+      "loss": 0.5638,
+      "step": 1360
+    },
+    {
+      "epoch": 0.31214657317349015,
+      "grad_norm": 0.6378616690635681,
+      "learning_rate": 4.762105458500556e-06,
+      "loss": 0.5433,
+      "step": 1380
+    },
+    {
+      "epoch": 0.3166704365528161,
+      "grad_norm": 1.3384662866592407,
+      "learning_rate": 4.742836386436831e-06,
+      "loss": 0.5535,
+      "step": 1400
+    },
+    {
+      "epoch": 0.32119429993214205,
+      "grad_norm": 0.6671473979949951,
+      "learning_rate": 4.722859170682468e-06,
+      "loss": 0.612,
+      "step": 1420
+    },
+    {
+      "epoch": 0.32571816331146797,
+      "grad_norm": 0.98403000831604,
+      "learning_rate": 4.702180118759219e-06,
+      "loss": 0.5735,
+      "step": 1440
+    },
+    {
+      "epoch": 0.33024202669079394,
+      "grad_norm": 1.0076704025268555,
+      "learning_rate": 4.680805759783622e-06,
+      "loss": 0.5879,
+      "step": 1460
+    },
+    {
+      "epoch": 0.33476589007011986,
+      "grad_norm": 0.9248991012573242,
+      "learning_rate": 4.6587428424055324e-06,
+      "loss": 0.5747,
+      "step": 1480
+    },
+    {
+      "epoch": 0.33928975344944584,
+      "grad_norm": 0.7047035098075867,
+      "learning_rate": 4.635998332677325e-06,
+      "loss": 0.5827,
+      "step": 1500
+    },
+    {
+      "epoch": 0.34381361682877176,
+      "grad_norm": 0.9783876538276672,
+      "learning_rate": 4.612579411854468e-06,
+      "loss": 0.5671,
+      "step": 1520
+    },
+    {
+      "epoch": 0.34833748020809774,
+      "grad_norm": 0.8747379183769226,
+      "learning_rate": 4.588493474128137e-06,
+      "loss": 0.5934,
+      "step": 1540
+    },
+    {
+      "epoch": 0.35286134358742366,
+      "grad_norm": 0.8138028383255005,
+      "learning_rate": 4.563748124290612e-06,
+      "loss": 0.7145,
+      "step": 1560
+    },
+    {
+      "epoch": 0.3573852069667496,
+      "grad_norm": 0.5983314514160156,
+      "learning_rate": 4.538351175334167e-06,
+      "loss": 0.5961,
+      "step": 1580
+    },
+    {
+      "epoch": 0.36190907034607556,
+      "grad_norm": 0.7765645980834961,
+      "learning_rate": 4.512310645984231e-06,
+      "loss": 0.6588,
+      "step": 1600
+    },
+    {
+      "epoch": 0.3664329337254015,
+      "grad_norm": 0.6621695756912231,
+      "learning_rate": 4.485634758167589e-06,
+      "loss": 0.6608,
+      "step": 1620
+    },
+    {
+      "epoch": 0.37095679710472745,
+      "grad_norm": 0.5030168294906616,
+      "learning_rate": 4.458331934416427e-06,
+      "loss": 0.5647,
+      "step": 1640
+    },
+    {
+      "epoch": 0.3754806604840534,
+      "grad_norm": 0.7155460715293884,
+      "learning_rate": 4.430410795209031e-06,
+      "loss": 0.5815,
+      "step": 1660
+    },
+    {
+      "epoch": 0.38000452386337935,
+      "grad_norm": 1.0417250394821167,
+      "learning_rate": 4.401880156248006e-06,
+      "loss": 0.6304,
+      "step": 1680
+    },
+    {
+      "epoch": 0.38452838724270527,
+      "grad_norm": 1.251464605331421,
+      "learning_rate": 4.372749025676829e-06,
+      "loss": 0.5852,
+      "step": 1700
+    },
+    {
+      "epoch": 0.3890522506220312,
+      "grad_norm": 0.53974848985672,
+      "learning_rate": 4.343026601235666e-06,
+      "loss": 0.5883,
+      "step": 1720
+    },
+    {
+      "epoch": 0.39357611400135717,
+      "grad_norm": 0.720723569393158,
+      "learning_rate": 4.312722267357311e-06,
+      "loss": 0.6516,
+      "step": 1740
+    },
+    {
+      "epoch": 0.3980999773806831,
+      "grad_norm": 0.8760417103767395,
+      "learning_rate": 4.281845592204188e-06,
+      "loss": 0.5926,
+      "step": 1760
+    },
+    {
+      "epoch": 0.40262384076000907,
+      "grad_norm": 0.7676084637641907,
+      "learning_rate": 4.2504063246473395e-06,
+      "loss": 0.6594,
+      "step": 1780
+    },
+    {
+      "epoch": 0.407147704139335,
+      "grad_norm": 0.6235904097557068,
+      "learning_rate": 4.218414391188353e-06,
+      "loss": 0.6249,
+      "step": 1800
+    },
+    {
+      "epoch": 0.41167156751866096,
+      "grad_norm": 0.5731521248817444,
+      "learning_rate": 4.185879892825222e-06,
+      "loss": 0.613,
+      "step": 1820
+    },
+    {
+      "epoch": 0.4161954308979869,
+      "grad_norm": 0.8284727334976196,
+      "learning_rate": 4.152813101863087e-06,
+      "loss": 0.6487,
+      "step": 1840
+    },
+    {
+      "epoch": 0.4207192942773128,
+      "grad_norm": 0.5617169141769409,
+      "learning_rate": 4.119224458670905e-06,
+      "loss": 0.5878,
+      "step": 1860
+    },
+    {
+      "epoch": 0.4252431576566388,
+      "grad_norm": 0.647582471370697,
+      "learning_rate": 4.085124568385057e-06,
+      "loss": 0.6037,
+      "step": 1880
+    },
+    {
+      "epoch": 0.4297670210359647,
+      "grad_norm": 0.7462798357009888,
+      "learning_rate": 4.050524197560921e-06,
+      "loss": 0.5911,
+      "step": 1900
+    },
+    {
+      "epoch": 0.4342908844152907,
+      "grad_norm": 0.6552990674972534,
+      "learning_rate": 4.015434270773485e-06,
+      "loss": 0.6064,
+      "step": 1920
+    },
+    {
+      "epoch": 0.4388147477946166,
+      "grad_norm": 0.8512873649597168,
+      "learning_rate": 3.979865867168071e-06,
+      "loss": 0.5766,
+      "step": 1940
+    },
+    {
+      "epoch": 0.4433386111739425,
+      "grad_norm": 0.6242995262145996,
+      "learning_rate": 3.943830216962254e-06,
+      "loss": 0.603,
+      "step": 1960
+    },
+    {
+      "epoch": 0.4478624745532685,
+      "grad_norm": 0.933954119682312,
+      "learning_rate": 3.9073386979000715e-06,
+      "loss": 0.6316,
+      "step": 1980
+    },
+    {
+      "epoch": 0.4523863379325944,
+      "grad_norm": 0.5889476537704468,
+      "learning_rate": 3.870402831659676e-06,
+      "loss": 0.5686,
+      "step": 2000
+    },
+    {
+      "epoch": 0.4569102013119204,
+      "grad_norm": 0.5107278227806091,
+      "learning_rate": 3.833034280215518e-06,
+      "loss": 0.6149,
+      "step": 2020
+    },
+    {
+      "epoch": 0.4614340646912463,
+      "grad_norm": 0.546362042427063,
+      "learning_rate": 3.795244842156246e-06,
+      "loss": 0.6593,
+      "step": 2040
+    },
+    {
+      "epoch": 0.4659579280705723,
+      "grad_norm": 0.7260658740997314,
+      "learning_rate": 3.757046448959474e-06,
+      "loss": 0.576,
+      "step": 2060
+    },
+    {
+      "epoch": 0.4704817914498982,
+      "grad_norm": 0.5332072377204895,
+      "learning_rate": 3.718451161224584e-06,
+      "loss": 0.6064,
+      "step": 2080
+    },
+    {
+      "epoch": 0.47500565482922413,
+      "grad_norm": 0.5485090613365173,
+      "learning_rate": 3.679471164864763e-06,
+      "loss": 0.6147,
+      "step": 2100
+    },
+    {
+      "epoch": 0.4795295182085501,
+      "grad_norm": 0.6777561902999878,
+      "learning_rate": 3.640118767259474e-06,
+      "loss": 0.5975,
+      "step": 2120
+    },
+    {
+      "epoch": 0.48405338158787603,
+      "grad_norm": 0.5616791248321533,
+      "learning_rate": 3.600406393368577e-06,
+      "loss": 0.5684,
+      "step": 2140
+    },
+    {
+      "epoch": 0.488577244967202,
+      "grad_norm": 0.7273005843162537,
+      "learning_rate": 3.560346581809328e-06,
+      "loss": 0.5439,
+      "step": 2160
+    },
+    {
+      "epoch": 0.4931011083465279,
+      "grad_norm": 0.399677574634552,
+      "learning_rate": 3.51995198089748e-06,
+      "loss": 0.6252,
+      "step": 2180
+    },
+    {
+      "epoch": 0.4976249717258539,
+      "grad_norm": 0.5627871155738831,
+      "learning_rate": 3.479235344653767e-06,
+      "loss": 0.6454,
+      "step": 2200
+    },
+    {
+      "epoch": 0.5021488351051798,
+      "grad_norm": 0.7673916816711426,
+      "learning_rate": 3.4382095287769962e-06,
+      "loss": 0.5485,
+      "step": 2220
+    },
+    {
+      "epoch": 0.5066726984845058,
+      "grad_norm": 0.8236356377601624,
+      "learning_rate": 3.396887486585047e-06,
+      "loss": 0.6054,
+      "step": 2240
+    },
+    {
+      "epoch": 0.5111965618638317,
+      "grad_norm": 0.9059731364250183,
+      "learning_rate": 3.355282264925047e-06,
+      "loss": 0.5705,
+      "step": 2260
+    },
+    {
+      "epoch": 0.5157204252431576,
+      "grad_norm": 0.8679450750350952,
+      "learning_rate": 3.3134070000540114e-06,
+      "loss": 0.5901,
+      "step": 2280
+    },
+    {
+      "epoch": 0.5202442886224836,
+      "grad_norm": 0.5972412824630737,
+      "learning_rate": 3.2712749134912582e-06,
+      "loss": 0.5561,
+      "step": 2300
+    },
+    {
+      "epoch": 0.5247681520018096,
+      "grad_norm": 0.830877959728241,
+      "learning_rate": 3.2288993078439034e-06,
+      "loss": 0.6464,
+      "step": 2320
+    },
+    {
+      "epoch": 0.5292920153811355,
+      "grad_norm": 1.2185938358306885,
+      "learning_rate": 3.1862935626067492e-06,
+      "loss": 0.5784,
+      "step": 2340
+    },
+    {
+      "epoch": 0.5338158787604614,
+      "grad_norm": 0.6374582648277283,
+      "learning_rate": 3.1434711299379025e-06,
+      "loss": 0.5661,
+      "step": 2360
+    },
+    {
+      "epoch": 0.5383397421397874,
+      "grad_norm": 0.7829556465148926,
+      "learning_rate": 3.100445530411442e-06,
+      "loss": 0.5385,
+      "step": 2380
+    },
+    {
+      "epoch": 0.5428636055191133,
+      "grad_norm": 0.8443902134895325,
+      "learning_rate": 3.0572303487484935e-06,
+      "loss": 0.5565,
+      "step": 2400
+    },
+    {
+      "epoch": 0.5473874688984393,
+      "grad_norm": 1.1685762405395508,
+      "learning_rate": 3.013839229528046e-06,
+      "loss": 0.5781,
+      "step": 2420
+    },
+    {
+      "epoch": 0.5519113322777652,
+      "grad_norm": 0.9711598753929138,
+      "learning_rate": 2.9702858728788714e-06,
+      "loss": 0.6448,
+      "step": 2440
+    },
+    {
+      "epoch": 0.5564351956570912,
+      "grad_norm": 0.43809574842453003,
+      "learning_rate": 2.9265840301538995e-06,
+      "loss": 0.558,
+      "step": 2460
+    },
+    {
+      "epoch": 0.5609590590364171,
+      "grad_norm": 0.6405979990959167,
+      "learning_rate": 2.8827474995884304e-06,
+      "loss": 0.6875,
+      "step": 2480
+    },
+    {
+      "epoch": 0.565482922415743,
+      "grad_norm": 0.6583331823348999,
+      "learning_rate": 2.8387901219435298e-06,
+      "loss": 0.5485,
+      "step": 2500
+    },
+    {
+      "epoch": 0.570006785795069,
+      "grad_norm": 0.7614380121231079,
+      "learning_rate": 2.7947257761360136e-06,
+      "loss": 0.5632,
+      "step": 2520
+    },
+    {
+      "epoch": 0.5745306491743949,
+      "grad_norm": 0.69844651222229,
+      "learning_rate": 2.750568374856372e-06,
+      "loss": 0.6059,
+      "step": 2540
+    },
+    {
+      "epoch": 0.5790545125537209,
+      "grad_norm": 0.7071551084518433,
+      "learning_rate": 2.7063318601760386e-06,
+      "loss": 0.5265,
+      "step": 2560
+    },
+    {
+      "epoch": 0.5835783759330468,
+      "grad_norm": 0.6122928857803345,
+      "learning_rate": 2.6620301991453755e-06,
+      "loss": 0.5629,
+      "step": 2580
+    },
+    {
+      "epoch": 0.5881022393123728,
+      "grad_norm": 0.8555611968040466,
+      "learning_rate": 2.617677379383781e-06,
+      "loss": 0.5967,
+      "step": 2600
+    },
+    {
+      "epoch": 0.5926261026916987,
+      "grad_norm": 0.5617034435272217,
+      "learning_rate": 2.5732874046632938e-06,
+      "loss": 0.6009,
+      "step": 2620
+    },
+    {
+      "epoch": 0.5971499660710247,
+      "grad_norm": 0.6981135606765747,
+      "learning_rate": 2.528874290487102e-06,
+      "loss": 0.5641,
+      "step": 2640
+    },
+    {
+      "epoch": 0.6016738294503506,
+      "grad_norm": 0.5575506687164307,
+      "learning_rate": 2.4844520596643495e-06,
+      "loss": 0.5759,
+      "step": 2660
+    },
+    {
+      "epoch": 0.6061976928296765,
+      "grad_norm": 0.7685838341712952,
+      "learning_rate": 2.4400347378826296e-06,
+      "loss": 0.5343,
+      "step": 2680
+    },
+    {
+      "epoch": 0.6107215562090025,
+      "grad_norm": 0.7420554161071777,
+      "learning_rate": 2.395636349279577e-06,
+      "loss": 0.6218,
+      "step": 2700
+    },
+    {
+      "epoch": 0.6152454195883285,
+      "grad_norm": 0.43521085381507874,
+      "learning_rate": 2.3512709120149436e-06,
+      "loss": 0.5303,
+      "step": 2720
+    },
+    {
+      "epoch": 0.6197692829676543,
+      "grad_norm": 0.6557121276855469,
+      "learning_rate": 2.3069524338445666e-06,
+      "loss": 0.5794,
+      "step": 2740
+    },
+    {
+      "epoch": 0.6242931463469803,
+      "grad_norm": 0.7263210415840149,
+      "learning_rate": 2.2626949076976153e-06,
+      "loss": 0.6239,
+      "step": 2760
+    },
+    {
+      "epoch": 0.6288170097263063,
+      "grad_norm": 0.5805336833000183,
+      "learning_rate": 2.2185123072585253e-06,
+      "loss": 0.5821,
+      "step": 2780
+    },
+    {
+      "epoch": 0.6333408731056323,
+      "grad_norm": 0.8271047472953796,
+      "learning_rate": 2.174418582555005e-06,
+      "loss": 0.5924,
+      "step": 2800
+    },
+    {
+      "epoch": 0.6378647364849581,
+      "grad_norm": 0.49003851413726807,
+      "learning_rate": 2.1304276555535123e-06,
+      "loss": 0.5567,
+      "step": 2820
+    },
+    {
+      "epoch": 0.6423885998642841,
+      "grad_norm": 0.6948228478431702,
+      "learning_rate": 2.086553415763597e-06,
+      "loss": 0.541,
+      "step": 2840
+    },
+    {
+      "epoch": 0.6469124632436101,
+      "grad_norm": 0.7717708945274353,
+      "learning_rate": 2.0428097158524768e-06,
+      "loss": 0.6128,
+      "step": 2860
+    },
+    {
+      "epoch": 0.6514363266229359,
+      "grad_norm": 0.8652535676956177,
+      "learning_rate": 1.999210367271261e-06,
+      "loss": 0.5484,
+      "step": 2880
+    },
+    {
+      "epoch": 0.6559601900022619,
+      "grad_norm": 0.48044684529304504,
+      "learning_rate": 1.9557691358941796e-06,
+      "loss": 0.6335,
+      "step": 2900
+    },
+    {
+      "epoch": 0.6604840533815879,
+      "grad_norm": 0.6042224764823914,
+      "learning_rate": 1.9124997376722038e-06,
+      "loss": 0.5719,
+      "step": 2920
+    },
+    {
+      "epoch": 0.6650079167609139,
+      "grad_norm": 0.45004037022590637,
+      "learning_rate": 1.869415834302429e-06,
+      "loss": 0.5828,
+      "step": 2940
+    },
+    {
+      "epoch": 0.6695317801402397,
+      "grad_norm": 0.7315308451652527,
+      "learning_rate": 1.826531028914593e-06,
+      "loss": 0.5809,
+      "step": 2960
+    },
+    {
+      "epoch": 0.6740556435195657,
+      "grad_norm": 0.4399823546409607,
+      "learning_rate": 1.7838588617760733e-06,
+      "loss": 0.5933,
+      "step": 2980
+    },
+    {
+      "epoch": 0.6785795068988917,
+      "grad_norm": 1.064976692199707,
+      "learning_rate": 1.7414128060167457e-06,
+      "loss": 0.5469,
+      "step": 3000
+    },
+    {
+      "epoch": 0.6831033702782175,
+      "grad_norm": 0.7710141539573669,
+      "learning_rate": 1.6992062633750346e-06,
+      "loss": 0.585,
+      "step": 3020
+    },
+    {
+      "epoch": 0.6876272336575435,
+      "grad_norm": 0.673542857170105,
+      "learning_rate": 1.657252559966502e-06,
+      "loss": 0.5436,
+      "step": 3040
+    },
+    {
+      "epoch": 0.6921510970368695,
+      "grad_norm": 0.6577861309051514,
+      "learning_rate": 1.6155649420763214e-06,
+      "loss": 0.6201,
+      "step": 3060
+    },
+    {
+      "epoch": 0.6966749604161955,
+      "grad_norm": 0.9896159768104553,
+      "learning_rate": 1.5741565719769447e-06,
+      "loss": 0.6079,
+      "step": 3080
+    },
+    {
+      "epoch": 0.7011988237955213,
+      "grad_norm": 0.7078260183334351,
+      "learning_rate": 1.5330405237723078e-06,
+      "loss": 0.6218,
+      "step": 3100
+    },
+    {
+      "epoch": 0.7057226871748473,
+      "grad_norm": 1.2103734016418457,
+      "learning_rate": 1.4922297792698645e-06,
+      "loss": 0.5955,
+      "step": 3120
+    },
+    {
+      "epoch": 0.7102465505541733,
+      "grad_norm": 0.6530666947364807,
+      "learning_rate": 1.4517372238817668e-06,
+      "loss": 0.5647,
+      "step": 3140
+    },
+    {
+      "epoch": 0.7147704139334992,
+      "grad_norm": 0.6449224352836609,
+      "learning_rate": 1.4115756425564798e-06,
+      "loss": 0.6194,
+      "step": 3160
+    },
+    {
+      "epoch": 0.7192942773128251,
+      "grad_norm": 0.5960292220115662,
+      "learning_rate": 1.3717577157421169e-06,
+      "loss": 0.5926,
+      "step": 3180
+    },
+    {
+      "epoch": 0.7238181406921511,
+      "grad_norm": 0.6450957655906677,
+      "learning_rate": 1.3322960153827632e-06,
+      "loss": 0.6137,
+      "step": 3200
+    },
+    {
+      "epoch": 0.7283420040714771,
+      "grad_norm": 0.5211399793624878,
+      "learning_rate": 1.293203000949072e-06,
+      "loss": 0.5796,
+      "step": 3220
+    },
+    {
+      "epoch": 0.732865867450803,
+      "grad_norm": 0.6112457513809204,
+      "learning_rate": 1.2544910155043545e-06,
+      "loss": 0.6187,
+      "step": 3240
+    },
+    {
+      "epoch": 0.7373897308301289,
+      "grad_norm": 0.5904879570007324,
+      "learning_rate": 1.2161722818074412e-06,
+      "loss": 0.5708,
+      "step": 3260
+    },
+    {
+      "epoch": 0.7419135942094549,
+      "grad_norm": 0.5986872911453247,
+      "learning_rate": 1.1782588984535129e-06,
+      "loss": 0.5455,
+      "step": 3280
+    },
+    {
+      "epoch": 0.7464374575887808,
+      "grad_norm": 0.716018795967102,
+      "learning_rate": 1.14076283605415e-06,
+      "loss": 0.6109,
+      "step": 3300
+    },
+    {
+      "epoch": 0.7509613209681067,
+      "grad_norm": 0.5863745212554932,
+      "learning_rate": 1.1036959334577708e-06,
+      "loss": 0.5875,
+      "step": 3320
+    },
+    {
+      "epoch": 0.7554851843474327,
+      "grad_norm": 0.8205773830413818,
+      "learning_rate": 1.0670698940116937e-06,
+      "loss": 0.6015,
+      "step": 3340
+    },
+    {
+      "epoch": 0.7600090477267587,
+      "grad_norm": 0.6910251975059509,
+      "learning_rate": 1.030896281866964e-06,
+      "loss": 0.5625,
+      "step": 3360
+    },
+    {
+      "epoch": 0.7645329111060846,
+      "grad_norm": 0.42625927925109863,
+      "learning_rate": 9.951865183271443e-07,
+      "loss": 0.5823,
+      "step": 3380
+    },
+    {
+      "epoch": 0.7690567744854105,
+      "grad_norm": 0.742891788482666,
+      "learning_rate": 9.599518782421954e-07,
+      "loss": 0.5209,
+      "step": 3400
+    },
+    {
+      "epoch": 0.7735806378647365,
+      "grad_norm": 0.6565542817115784,
+      "learning_rate": 9.252034864486062e-07,
+      "loss": 0.5896,
+      "step": 3420
+    },
+    {
+      "epoch": 0.7781045012440624,
+      "grad_norm": 0.7679634690284729,
+      "learning_rate": 8.909523142568807e-07,
+      "loss": 0.6889,
+      "step": 3440
+    },
+    {
+      "epoch": 0.7826283646233884,
+      "grad_norm": 0.8022564649581909,
+      "learning_rate": 8.572091759875081e-07,
+      "loss": 0.5643,
+      "step": 3460
+    },
+    {
+      "epoch": 0.7871522280027143,
+      "grad_norm": 1.0336594581604004,
+      "learning_rate": 8.239847255564889e-07,
+      "loss": 0.6212,
+      "step": 3480
+    },
+    {
+      "epoch": 0.7916760913820403,
+      "grad_norm": 0.6547170877456665,
+      "learning_rate": 7.912894531115184e-07,
+      "loss": 0.5685,
+      "step": 3500
+    },
+    {
+      "epoch": 0.7961999547613662,
+      "grad_norm": 0.725557267665863,
+      "learning_rate": 7.591336817198683e-07,
+      "loss": 0.5953,
+      "step": 3520
+    },
+    {
+      "epoch": 0.8007238181406922,
+      "grad_norm": 0.8147058486938477,
+      "learning_rate": 7.275275641090238e-07,
+      "loss": 0.628,
+      "step": 3540
+    },
+    {
+      "epoch": 0.8052476815200181,
+      "grad_norm": 0.5000850558280945,
+      "learning_rate": 6.964810794611093e-07,
+      "loss": 0.5696,
+      "step": 3560
+    },
+    {
+      "epoch": 0.809771544899344,
+      "grad_norm": 0.9064733386039734,
+      "learning_rate": 6.660040302620974e-07,
+      "loss": 0.6752,
+      "step": 3580
+    },
+    {
+      "epoch": 0.81429540827867,
+      "grad_norm": 0.7876561284065247,
+      "learning_rate": 6.361060392068177e-07,
+      "loss": 0.6045,
+      "step": 3600
+    },
+    {
+      "epoch": 0.818819271657996,
+      "grad_norm": 0.5467254519462585,
+      "learning_rate": 6.067965461607275e-07,
+      "loss": 0.5963,
+      "step": 3620
+    },
+    {
+      "epoch": 0.8233431350373219,
+      "grad_norm": 0.5970877408981323,
+      "learning_rate": 5.780848051794066e-07,
+      "loss": 0.6115,
+      "step": 3640
+    },
+    {
+      "epoch": 0.8278669984166478,
+      "grad_norm": 0.809258759021759,
+      "learning_rate": 5.499798815867208e-07,
+      "loss": 0.53,
+      "step": 3660
+    },
+    {
+      "epoch": 0.8323908617959738,
+      "grad_norm": 0.9609009027481079,
+      "learning_rate": 5.224906491125778e-07,
+      "loss": 0.5172,
+      "step": 3680
+    },
+    {
+      "epoch": 0.8369147251752997,
+      "grad_norm": 0.5807494521141052,
+      "learning_rate": 4.95625787091169e-07,
+      "loss": 0.5784,
+      "step": 3700
+    },
+    {
+      "epoch": 0.8414385885546256,
+      "grad_norm": 0.8828917145729065,
+      "learning_rate": 4.693937777205973e-07,
+      "loss": 0.6114,
+      "step": 3720
+    },
+    {
+      "epoch": 0.8459624519339516,
+      "grad_norm": 1.2154064178466797,
+      "learning_rate": 4.438029033847402e-07,
+      "loss": 0.5837,
+      "step": 3740
+    },
+    {
+      "epoch": 0.8504863153132776,
+      "grad_norm": 0.5537241101264954,
+      "learning_rate": 4.1886124403820864e-07,
+      "loss": 0.5766,
+      "step": 3760
+    },
+    {
+      "epoch": 0.8550101786926034,
+      "grad_norm": 0.9004805088043213,
+      "learning_rate": 3.94576674655211e-07,
+      "loss": 0.6033,
+      "step": 3780
+    },
+    {
+      "epoch": 0.8595340420719294,
+      "grad_norm": 0.7978100776672363,
+      "learning_rate": 3.7095686274314757e-07,
+      "loss": 0.5284,
+      "step": 3800
+    },
+    {
+      "epoch": 0.8640579054512554,
+      "grad_norm": 0.7304428219795227,
+      "learning_rate": 3.4800926592169995e-07,
+      "loss": 0.6011,
+      "step": 3820
+    },
+    {
+      "epoch": 0.8685817688305814,
+      "grad_norm": 0.7252260446548462,
+      "learning_rate": 3.2574112956819613e-07,
+      "loss": 0.5691,
+      "step": 3840
+    },
+    {
+      "epoch": 0.8731056322099072,
+      "grad_norm": 0.6625013947486877,
+      "learning_rate": 3.041594845299856e-07,
+      "loss": 0.5782,
+      "step": 3860
+    },
+    {
+      "epoch": 0.8776294955892332,
+      "grad_norm": 0.5668479800224304,
+      "learning_rate": 2.832711449045461e-07,
+      "loss": 0.5781,
+      "step": 3880
+    },
+    {
+      "epoch": 0.8821533589685592,
+      "grad_norm": 0.6604365110397339,
+      "learning_rate": 2.630827058880345e-07,
+      "loss": 0.5579,
+      "step": 3900
+    },
+    {
+      "epoch": 0.886677222347885,
+      "grad_norm": 0.9126925468444824,
+      "learning_rate": 2.4360054169294377e-07,
+      "loss": 0.6684,
+      "step": 3920
+    },
+    {
+      "epoch": 0.891201085727211,
+      "grad_norm": 0.5072206854820251,
+      "learning_rate": 2.2483080353553537e-07,
+      "loss": 0.6443,
+      "step": 3940
+    },
+    {
+      "epoch": 0.895724949106537,
+      "grad_norm": 0.5561603307723999,
+      "learning_rate": 2.067794176936838e-07,
+      "loss": 0.6012,
+      "step": 3960
+    },
+    {
+      "epoch": 0.900248812485863,
+      "grad_norm": 0.5377748012542725,
+      "learning_rate": 1.8945208363573553e-07,
+      "loss": 0.5513,
+      "step": 3980
+    },
+    {
+      "epoch": 0.9047726758651888,
+      "grad_norm": 0.6415522694587708,
+      "learning_rate": 1.7285427222098388e-07,
+      "loss": 0.6421,
+      "step": 4000
+    },
+    {
+      "epoch": 0.9092965392445148,
+      "grad_norm": 0.7068675756454468,
+      "learning_rate": 1.569912239723237e-07,
+      "loss": 0.6059,
+      "step": 4020
+    },
+    {
+      "epoch": 0.9138204026238408,
+      "grad_norm": 0.5292190909385681,
+      "learning_rate": 1.4186794742162918e-07,
+      "loss": 0.5233,
+      "step": 4040
+    },
+    {
+      "epoch": 0.9183442660031667,
+      "grad_norm": 0.7200678586959839,
+      "learning_rate": 1.27489217528384e-07,
+      "loss": 0.607,
+      "step": 4060
+    },
+    {
+      "epoch": 0.9228681293824926,
+      "grad_norm": 0.5470794439315796,
+      "learning_rate": 1.138595741720519e-07,
+      "loss": 0.6102,
+      "step": 4080
+    },
+    {
+      "epoch": 0.9273919927618186,
+      "grad_norm": 0.6571109294891357,
+      "learning_rate": 1.009833207186775e-07,
+      "loss": 0.5965,
+      "step": 4100
+    },
+    {
+      "epoch": 0.9319158561411446,
+      "grad_norm": 0.534355640411377,
+      "learning_rate": 8.886452266215817e-08,
+      "loss": 0.6624,
+      "step": 4120
+    },
+    {
+      "epoch": 0.9364397195204704,
+      "grad_norm": 0.7418869137763977,
+      "learning_rate": 7.750700634062303e-08,
+      "loss": 0.6008,
+      "step": 4140
+    },
+    {
+      "epoch": 0.9409635828997964,
+      "grad_norm": 0.6004541516304016,
+      "learning_rate": 6.691435772831933e-08,
+      "loss": 0.582,
+      "step": 4160
+    },
+    {
+      "epoch": 0.9454874462791224,
+      "grad_norm": 0.48619237542152405,
+      "learning_rate": 5.708992130339458e-08,
+      "loss": 0.5868,
+      "step": 4180
+    },
+    {
+      "epoch": 0.9500113096584483,
+      "grad_norm": 0.5165609121322632,
+      "learning_rate": 4.8036798991923925e-08,
+      "loss": 0.5558,
+      "step": 4200
+    },
+    {
+      "epoch": 0.9545351730377742,
+      "grad_norm": 0.7379537224769592,
+      "learning_rate": 3.9757849188520046e-08,
+      "loss": 0.617,
+      "step": 4220
+    },
+    {
+      "epoch": 0.9590590364171002,
+      "grad_norm": 0.6974282264709473,
+      "learning_rate": 3.2255685853837815e-08,
+      "loss": 0.5584,
+      "step": 4240
+    },
+    {
+      "epoch": 0.9635828997964262,
+      "grad_norm": 0.6777763366699219,
+      "learning_rate": 2.553267768925116e-08,
+      "loss": 0.553,
+      "step": 4260
+    },
+    {
+      "epoch": 0.9681067631757521,
+      "grad_norm": 1.0349258184432983,
+      "learning_rate": 1.959094738897216e-08,
+      "loss": 0.5674,
+      "step": 4280
+    },
+    {
+      "epoch": 0.972630626555078,
+      "grad_norm": 1.0187406539916992,
+      "learning_rate": 1.4432370969836851e-08,
+      "loss": 0.6268,
+      "step": 4300
+    },
+    {
+      "epoch": 0.977154489934404,
+      "grad_norm": 0.4671124219894409,
+      "learning_rate": 1.0058577178982954e-08,
+      "loss": 0.6393,
+      "step": 4320
+    },
+    {
+      "epoch": 0.9816783533137299,
+      "grad_norm": 0.6963977217674255,
+      "learning_rate": 6.4709469795945505e-09,
+      "loss": 0.6028,
+      "step": 4340
+    },
+    {
+      "epoch": 0.9862022166930559,
+      "grad_norm": 0.7137852907180786,
+      "learning_rate": 3.67061311488226e-09,
+      "loss": 0.5654,
+      "step": 4360
+    },
+    {
+      "epoch": 0.9907260800723818,
+      "grad_norm": 0.6567050218582153,
+      "learning_rate": 1.6584597504365475e-09,
+      "loss": 0.5807,
+      "step": 4380
+    },
+    {
+      "epoch": 0.9952499434517078,
+      "grad_norm": 0.9630624651908875,
+      "learning_rate": 4.3512219506519626e-10,
+      "loss": 0.5787,
+      "step": 4400
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 4421,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.1225101215714509e+17,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-4400/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aea02cd52a6bd221ffa620d5ceaa1c146062e7958ebf17d7c66581378628b22a
+size 5368

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<unk>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,130 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "32000": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32001": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32002": {
+      "content": "<|placeholder1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32003": {
+      "content": "<|placeholder2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32004": {
+      "content": "<|placeholder3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32005": {
+      "content": "<|placeholder4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32006": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32007": {
+      "content": "<|end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32008": {
+      "content": "<|placeholder5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32009": {
+      "content": "<|placeholder6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "32010": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') %}{{'<|user|>' + '\n' + message['content'] + '<|end|>' + '\n' + '<|assistant|>' + '\n'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '<|end|>' + '\n'}}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "legacy": false,
+  "model_max_length": 4096,
+  "pad_token": "<unk>",
+  "padding_side": "left",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aea02cd52a6bd221ffa620d5ceaa1c146062e7958ebf17d7c66581378628b22a
+size 5368